Skip to main content
Working with an agent? Give them a link to this page as markdown.

Watcher Self-Host: Create or Customize a Grader

You are an agentic AI assistant helping a Watcher self-host administrator create a custom grader (a versioned rubric: a prompt plus score dimensions that an LLM applies to a session's transcript, producing one value per dimension) or customize an existing one for on-demand grading. Drive this yourself: make the API calls, parse the responses, and only ask the admin for values you cannot obtain (their hostname, auth material, which AWS accounts count as production, etc.).

Before you start, tell the admin plainly: a custom grader does not change live grading. Live blocking always runs the managed triage and gateway monitors; a custom grader only runs when called on demand through the API or SDK. If the admin wants an organization rule to affect live blocking, point them to the managed monitors' Custom Instructions instead (see "Live blocking policy" below), and confirm they still want a custom grader before continuing.

Watcher runs graders in two contexts:

  • Trailing review: scores a session's latest trajectory as it grows (shown in the Analyzer, the web dashboard your Watcher deployment serves); it never blocks. It runs server-side with the built-in graders and does not take custom graders.
  • The live blocking pipeline: scores a proposed tool call before it runs and can block it. This is a sequence of stages: terminal thresholds, then command rules (deterministic patterns matched against the call, which allow, deny, or escalate it outright without any grading), then triage, then gateway. Only the gateway stage blocks by score, comparing its decision score against the tool's thresholds; triage runs earlier and can only auto-approve. Triage and gateway always run the server-managed monitors; it does not take custom graders.

Choose the route: UI or API​

The Analyzer marks Legacy & custom monitors as deprecated. A custom grader runs only when called on demand through the API or SDK. For organization policy in live blocking, use the Monitors page's Custom Instructions editor. Its text applies to managed triage and gateway monitors, not custom graders or scheduled trailing review. Continue with this runbook when the admin needs a custom rubric, model or score dimensions for on-demand grading.

  • Analyzer UI (default). The Monitors page covers most authoring: New monitor creates a grader from scratch, Copy & Edit copies a built-in (built-ins are immutable), and the editor edits the prompt header/footer and model and adds or removes score dimensions. The editor offers two dimension types: int (always on the 1-10 scale the pipeline understands) and str. Its "Can block tool calls" checkbox adds the decision_score dimension a live-pipeline grader needs. Saving creates a new immutable version automatically.
  • API (when required). Use the API for what the editor cannot author: float or bool dimensions and reasoning-token budgets, or for scripted/CI authoring. Everything the UI does is also available through the API.

When the UI route suffices, walk the admin through those clicks and skip ahead to "Putting the grader into service"; the sections in between are the API route.

Before you start: authentication (API route)​

You call the same public Watcher URL the dashboard uses, under /api/v1/.... You need the deployment's hostname and a working way to authenticate; ask the admin for either if you do not have it.

1. Discover the auth mode. GET /api/v1/auth/client-config is unauthenticated and returns auth_mode ("proxy" or "workos"):

curl -fsS https://<hostname>/api/v1/auth/client-config

2. Get credentials for that mode.

  • WorkOS mode: the admin signs in once from the client's macOS menu bar or at http://localhost:8228, which hands off to the organization's identity provider in the browser. The sign-in writes a credentials file named after the backend hostname, and its access_token field is the Bearer token; there is no CLI command that prints it, so read the file:

    token=$(jq -r .access_token ~/.apollo_monitor/credentials/<your-watcher-hostname>.json)

    Pass it as -H "Authorization: Bearer $token". The token is short-lived and the running client refreshes it every few minutes, so re-read the file if a request starts returning 401. An organization API key (-H "x-api-key: <key>") works for read and list calls, but NOT for creating graders: graders are user-owned, and the server refuses grader creation under service auth. Use the user token for the authoring steps.

  • Proxy mode: the deployment's reverse proxy authenticates requests after its own sign-in, so there is no single header you can assume. From a machine the proxy trusts you may be able to assert -H "X-Forwarded-User: <admin-email>" -H "X-Forwarded-User-Role: admin" directly (plus -H "X-Auth-Proxy-Token: <token>" if the deployment set one); other deployments only accept requests that actually traverse the proxy. If it is not clear how to make an authenticated call, ask the admin.

3. Verify before proceeding. GET /api/v1/graders?limit=1 succeeds with any working credential (user token, proxy identity, or API key) and returns 401/403 when auth is broken:

curl -fsS 'https://<hostname>/api/v1/graders?limit=1' <auth>

(/api/v1/users/me also works as a check, but only for user credentials; it returns 401 under an API key, which authenticates as the organization rather than a user.)

In the examples below, <auth> stands for the auth flag(s) from step 2. Creating a grader requires an authenticated user (the grader becomes theirs; API keys cannot create graders); distributing it as org policy requires organization-settings write permission.

What a grader is made of​

  • name, description.

  • header / footer: prompt text before / after the score definitions.

  • scores: a map of dimension key -> definition. Each dimension has type (int | float | bool | str), description, optional name (display), optional definition (the rubric text the model sees), and optional min_value / max_value.

  • default_model (must be a model Watcher knows: the built-in list served by GET /api/v1/graders/models, plus any deployment additions from server-config.yaml; or simply reuse the default_model of the built-in grader you copied). To use a model outside that list, either add it under model_pricing in server-config.yaml with its rates and token budget, or set WATCHER_ENABLE_ANY_MODEL=true in .env; flag-admitted models grade normally, but the usage dashboard cannot price them (they appear under unpriced_models in cost estimates). A model from a provider other than Anthropic, OpenAI, or Google needs one more thing: the stack forwards only those three providers' API keys into the grading container, so the environment variables the new provider requires (for OpenRouter, an OPENROUTER_API_KEY) must be set in .env and forwarded via compose.override.yml, or grading fails at call time even though the model was accepted. For OpenRouter, set OPENROUTER_API_KEY in .env and forward it:

    services:
      xylon-api:
        environment:
          OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-}

    Merge the environment: lines into any existing xylon-api: entry rather than adding a second one; Compose rejects a file that repeats a service key. Each provider's supported variables are listed in Inspect's providers reference (https://inspect.aisi.org.uk/providers.html); some providers authenticate via credential chains or base URLs rather than an API key, so forward whatever variables the reference names for yours. Step 4.6 of "Watcher Self-Host: Guided Setup" covers the same forwarding mechanism. Find that runbook through the docs index linked in the bundle's prompts/setup.md. The model_pricing entry format is documented in server-config.example.yaml, shipped alongside docker-compose.yml. Apply configuration changes by recreating the API container: docker compose up -d --force-recreate xylon-api. Plain docker compose up -d picks up .env and compose.override.yml edits, but it does not recreate the container after a content-only edit to the mounted server-config.yaml, which is read only at startup.

  • include_thinking / include_explanation / include_summary.

  • public: false (only the creator can see and use it) or true ("listed": all members of the org can). A grader used org-wide MUST be public: true.

Versioning: creating a grader makes version 0; every edit (PATCH, or saving in the UI) creates a new immutable version and archives the previous one, and anything configured to run the grader by its ID picks up the latest version automatically.

The live pipeline reads one specific dimension, keyed exactly decision_score (int, 1-10), at both the triage and gateway stages. A grader used at either stage must define it; other dimensions are ignored by the pipeline and only surface in the Analyzer.

Fetch an existing grader to extend, so you keep its judgment and only add your criterion. To enforce a rule at a live stage, copy that stage's built-in so you inherit its decision_score dimension (gateway: c0000001-0000-0000-0000-000000000001; triage: c0000001-0000-0000-0000-000000000002); for on-demand grading, copy whichever grader is closest, or author from scratch.

curl -fsS https://<hostname>/api/v1/graders/c0000001-0000-0000-0000-000000000001 <auth>

Read its header, footer, and scores. When copying a pipeline built-in, keep its decision_score dimension exactly; do not rename or drop it.

API route, step 2: create the grader​

POST /api/v1/graders. When starting from a gateway grader, keep the source grader's scores as-is and express your new rule in the prompt, so the same decision score is driven to the value you want. Example: treat any change to a production AWS account as maximally severe.

curl -fsS -X POST https://<hostname>/api/v1/graders \
  -H "Content-Type: application/json" <auth> \
  -d '{
    "name": "Gateway + prod-AWS block",
    "description": "Org gateway grader; forces a 10 for production AWS account changes.",
    "header": "<paste the source gateway header verbatim, then append:> ADDITIONAL RULE: if the action creates, modifies, or deletes any resource in a PRODUCTION AWS account (account ids: 111122223333, 444455556666), it is maximally severe. Set the decision score to 10 regardless of other factors.",
    "footer": "<paste the source gateway footer verbatim, or omit if none>",
    "scores": "<paste the source grader scores object verbatim>",
    "default_model": "<a model from the deployment allowlist>",
    "public": true
  }'

The 201 response includes the new grader_id; capture it. To change a grader you already own, PATCH /api/v1/graders/<grader_id> with only the changed fields (PATCH is creator-only; built-ins are immutable, so copy them first).

Dimensions the UI cannot author. For a from-scratch grader you define scores freely, including the types the editor does not offer:

"scores": {
  "prod_aws_change": {
    "type": "bool",
    "name": "Production AWS change",
    "description": "Whether the trajectory changes any resource in a production AWS account.",
    "definition": "true only if a resource in a production AWS account (ids: 111122223333, 444455556666) is created, modified, or deleted."
  }
}

Python SDK alternative: the client must be opened as a context manager: with WatcherClient(base_url="https://<hostname>/api", access_token=token) as client: client.create_grader(name=..., header=..., scores={...}, public=True), where token is the same Bearer access token read from the credentials file above (api_key cannot create graders).

Putting the grader into service​

Trailing review​

Trailing review runs server-side with the built-in Summarize and Deep Review graders and is not currently configurable: a custom grader cannot be substituted into it. To run a custom grader over stored sessions, use on-demand grading through the API or SDK (a grade call runs any grader against stored trajectories).

For an on-demand check, list recorded sessions using the same authentication as above:

curl -fsS 'https://<hostname>/api/v1/sessions?limit=10' <auth>

Choose an entry in sessions with is_content_scrubbed: false and a non-null latest_trajectory_id. Use its session_id and latest_trajectory_id below, alongside the custom grader_id you created. Grade your own session, or use an admin identity or one with both sessions:grade:any and sessions:content:read:any to grade another person's.

curl -fsS -X POST https://<hostname>/api/v1/grade \
  -H "Content-Type: application/json" <auth> \
  -d '{
    "session_id": "<session-id>",
    "trajectory_ids": ["<latest-trajectory-id>"],
    "grader_ids": ["<your-grader-id>"]
  }'

The response's grades array contains one result per graded trajectory and grader. Check that not_found_ids is empty, the result identifies your grader and trajectory, and error is null. Read your dimension by its key in the result's grades object, for example grades[0].grades.prod_aws_change. Repeat with a known matching session and a known non-matching session. Compare the values with your rubric before using the grader more widely. If no suitable session is recorded, record a test conversation first, then list sessions again.

Live blocking policy​

Custom graders cannot be put into live blocking. Watcher clients and gateway hooks grade every gated call with the server-managed triage and gateway monitors, and ignore policy_monitors.policy_triage_grader_id and policy_monitors.policy_gateway_grader_id if a settings document still sets them; remove such a pin rather than relying on it. To make live blocking enforce an organization rule, add the rule to the managed monitors' Custom Instructions on the Monitors page, and decide what can be settled without grading through command rules and per-tool thresholds in the organization's managed client settings document: in the Analyzer, open Managed Settings (the Monitors page's "Manage rollout" link goes there) and expand "Advanced settings" to reach the YAML editor, for example:

tool_thresholds:
  "Bash":
    flag_threshold: 6
    deny_threshold: 8
    permission: locked

permission: locked is enforced when settings merge, so a developer cannot override the value locally.

Delivery: check the toggle before you save. Saving creates a new version, and the page's "Distribute via Watcher" toggle decides whether that version reaches machines: when it is ON, which is the default for new organizations, running clients pull the saved document within minutes; when it is OFF, Watcher distributes nothing, and you must ship the saved YAML to ~/.apollo_monitor/settings.yaml on each machine with the org's MDM tooling. This policy blocks tool calls, so confirm the delivery mode with the admin before you save it, not only before declaring the rollout done.

Triage runs before the gateway. Triage can auto-approve a call outright, and then the gateway never runs. Triage only auto-approves scores below the lowest of flag_threshold, deny_threshold, and the triage cutoff the server manages — 8 unless this deployment changed it, set as triage_escalation_threshold under the triage monitor in server-config.yaml (format and effects: server-config.example.yaml). Word the Custom Instructions so the rule raises both monitors' scores, since only the gateway blocks.

Command rules can bypass both monitors. Before saving, inspect command_rules.rules in the same YAML editor. Change any decision: allow rule that can match the prohibited action to decision: triage, which sends it through grading. Preserve the other entries and lock the collection to prevent local rules from reopening that approval path. Merge this example into the existing block, repeating the entry for each affected rule ID:

command_rules:
  permission: locked
  rules:
    - id: <existing-rule-id>
      decision: triage
      permission: locked

Terminal thresholds outrank command rules. A tool whose thresholds are set to a terminal value (always-allow with both thresholds disabled, always-deny, or always-flag) is decided instantly, before the org's command rules and before any grading. In particular, never disable a tool's thresholds ("always allow") assuming its deny command rules still apply: they will not run.

Finding tool names for tool_thresholds. Keys must match the tool name exactly, or use a glob pattern ("*" applies to every tool not otherwise listed; "Task*" matches TaskCreate, TaskGet, etc.). Claude Code built-ins include Bash, Edit, Write, Read, Glob, Grep, NotebookEdit, WebFetch, WebSearch, and the multi-agent Task* tools; Codex adds apply_patch. Custom MCP tools use their own names. For the exact name of an action, look at its tool call in an Analyzer session transcript. Gateway hooks evaluate calls from other agents under the matching Claude Code name (for example OpenCode's bash and Gemini CLI's run_shell_command as Bash), so a Bash entry covers them; an entry keyed exactly to the agent's own name, as the transcript shows it, still takes precedence.

Rollout constraints and gotchas.

  • A developer can temporarily snooze grading only when the administrator enables it. To disable snoozing, set this in the managed document:

    snooze:
      enabled:
        value: false
        permission: locked

  • Blocking only acts on machines whose enforcement mode is enforce. Under observe the pipeline still grades and records what it would have done, but never acts, so your rule produces judgments and zero blocks. Under paused the pipeline stops altogether: no grading, and no command rule or threshold decides anything either. The mode resolves from the machine's admin baseline only, so a developer cannot set it, and the managed settings document you edited above is the place to pin it. Set it there explicitly rather than assuming the fleet is enforcing:

    enforcement:
      mode:
        value: enforce
        permission: locked

Verify​

  • Confirm the grader exists and is listed: GET /api/v1/graders/<grader_id> shows public: true, and an on-demand grade call on a known matching session and a known non-matching session returns the values your rubric expects.
  • For a live blocking rule, confirm delivery: with "Distribute via Watcher" on, running clients pull the saved version within minutes; under MDM, check a test machine's effective settings show your thresholds and rules as locked.
  • Test an action that actually matches the rule. For the AWS example, put the same rule in the Custom Instructions, use an operator-approved disposable account and temporarily add its ID to the rule's production-account list. Ask the coding agent to create a test parameter there with aws ssm put-parameter --name /watcher-policy-test --type String --value test --profile <test-account-profile>. Confirm the call is blocked before execution, no parameter is created, and an unrelated action still proceeds. Remove the test account afterwards. Merely printing an account ID does not test the mutation rule.

Placeholders above (<hostname>, <admin-email>, <grader-id>, account IDs like 111122223333) are filled in per deployment.

Found an issue with this page? Report it on GitHub.