Watcher Self-Host: Create or Customize a Grader

You are an agentic AI assistant helping a Watcher self-host administrator create a custom grader (a versioned rubric: a prompt plus score dimensions that an LLM applies to a session's transcript, producing one value per dimension) or customize an existing one and, optionally, put it into service as the mandatory live blocking policy. Drive this yourself: make the API calls, parse the responses, and only ask the admin for values you cannot obtain (their hostname, auth material, which AWS accounts count as production, etc.).

Watcher runs graders in two contexts:

• Trailing review: scores a session's latest trajectory as it grows (shown in the Analyzer, the web dashboard your Watcher deployment serves); it never blocks. It runs server-side with the built-in graders and does not take custom graders.
• The live blocking pipeline: scores a proposed tool call before it runs and can block it. This is a sequence of stages: terminal thresholds, then command rules (deterministic patterns matched against the call, which allow, deny, or escalate it outright without any grading), then triage, then gateway. Only the gateway stage blocks by score, comparing its decision score against the tool's thresholds; triage runs earlier and can only auto-approve.

Choose the route: UI or API

• Analyzer UI (default). The Monitors page covers most authoring: New monitor creates a grader from scratch, Copy & Edit copies a built-in (built-ins are immutable), and the editor edits the prompt header/footer and model and adds or removes score dimensions. The editor offers two dimension types: int (always on the 1-10 scale the pipeline understands) and str. Its "Can block tool calls" checkbox adds the decision_score dimension a live-pipeline grader needs. Saving creates a new immutable version automatically.
• API (when required). Use the API for what the editor cannot author: float or bool dimensions and reasoning-token budgets, or for scripted/CI authoring. Everything the UI does is also available through the API.

When the UI route suffices, walk the admin through those clicks and skip ahead to "Putting the grader into service"; the sections in between are the API route.

Before you start: authentication (API route)

You call the same public Watcher URL the dashboard uses, under /api/v1/.... You need the deployment's hostname and a working way to authenticate; ask the admin for either if you do not have it.

1. Discover the auth mode. GET /api/v1/auth/client-config is unauthenticated and returns auth_mode ("proxy" or "workos"):

curl -fsS https://<hostname>/api/v1/auth/client-config

2. Get credentials for that mode.

• WorkOS mode: the admin signs in once from the client's local UI at http://localhost:8228, which hands off to the organization's identity provider in the browser. The sign-in writes a credentials file named after the backend hostname, and its access_token field is the Bearer token; there is no CLI command that prints it, so read the file:

  token=$(jq -r .access_token ~/.apollo_monitor/credentials/<your-watcher-hostname>.json)

  Pass it as -H "Authorization: Bearer $token". The token is short-lived and the running client refreshes it every few minutes, so re-read the file if a request starts returning 401. An organization API key (-H "x-api-key: <key>") works for read and list calls, but NOT for creating graders: graders are user-owned, and the server refuses grader creation under service auth. Use the user token for the authoring steps.

• Proxy mode: the deployment's reverse proxy authenticates requests after its own login, so there is no single header you can assume. From a machine the proxy trusts you may be able to assert -H "X-Forwarded-User: <admin-email>" -H "X-Forwarded-User-Role: admin" directly (plus -H "X-Auth-Proxy-Token: <token>" if the deployment set one); other deployments only accept requests that actually traverse the proxy. If it is not clear how to make an authenticated call, ask the admin.

3. Verify before proceeding. GET /api/v1/graders?limit=1 succeeds with any working credential (user token, proxy identity, or API key) and returns 401/403 when auth is broken:

curl -fsS 'https://<hostname>/api/v1/graders?limit=1' <auth>

(/api/v1/users/me also works as a check, but only for user credentials; it returns 401 under an API key, which authenticates as the organization rather than a user.)

In the examples below, <auth> stands for the auth flag(s) from step 2. Creating a grader requires an authenticated user (the grader becomes theirs; API keys cannot create graders); distributing it as org policy requires organization-settings write permission.

What a grader is made of

• name, description.
• header / footer: prompt text before / after the score definitions.
• scores: a map of dimension key -> definition. Each dimension has type (int | float | bool | str), description, optional name (display), optional definition (the rubric text the model sees), and optional min_value / max_value.
• default_model (must be a model Watcher knows: the built-in list served by GET /api/v1/graders/models, plus any deployment additions from server-config.yaml; or simply reuse the default_model of the built-in grader you copied). To use a model outside that list, either add it under model_pricing in server-config.yaml with its rates and token budget, or set WATCHER_ENABLE_ANY_MODEL=true in .env; flag-admitted models grade normally, but the usage dashboard cannot price them (they appear under unpriced_models in cost estimates).
• include_thinking / include_explanation / include_summary.
• public: false (only the creator can see and use it) or true ("listed": all members of the org can). A grader used org-wide MUST be public: true.

Versioning: creating a grader makes version 0; every edit (PATCH, or saving in the UI) creates a new immutable version and archives the previous one, and anything configured to run the grader by its ID picks up the latest version automatically.

The live pipeline reads one specific dimension, keyed exactly decision_score (int, 1-10), at both the triage and gateway stages. A grader used at either stage must define it; other dimensions are ignored by the pipeline and only surface in the Analyzer.

API route, step 1: (recommended) start from an existing grader

Fetch an existing grader to extend, so you keep its judgment and only add your criterion. To enforce a rule at a live stage, copy that stage's built-in so you inherit its decision_score dimension (gateway: c0000001-0000-0000-0000-000000000001; triage: c0000001-0000-0000-0000-000000000002); for on-demand grading, copy whichever grader is closest, or author from scratch.

curl -fsS https://<hostname>/api/v1/graders/c0000001-0000-0000-0000-000000000001 <auth>

Read its header, footer, and scores. When copying a pipeline built-in, keep its decision_score dimension exactly; do not rename or drop it.

API route, step 2: create the grader

POST /api/v1/graders. For a live gateway grader, keep the source grader's scores as-is and express your new rule in the prompt, so the same decision score is driven to the value you want. Example: treat any change to a production AWS account as maximally severe.

curl -fsS -X POST https://<hostname>/api/v1/graders \
  -H "Content-Type: application/json" <auth> \
  -d '{
    "name": "Gateway + prod-AWS block",
    "description": "Org gateway grader; forces a 10 for production AWS account changes.",
    "header": "<paste the source gateway header verbatim, then append:> ADDITIONAL RULE: if the action creates, modifies, or deletes any resource in a PRODUCTION AWS account (account ids: 111122223333, 444455556666), it is maximally severe. Set the decision score to 10 regardless of other factors.",
    "footer": "<paste the source gateway footer verbatim, or omit if none>",
    "scores": "<paste the source grader scores object verbatim>",
    "default_model": "<a model from the deployment allowlist>",
    "public": true
  }'

The 201 response includes the new grader_id; capture it. To change a grader you already own, PATCH /api/v1/graders/<grader_id> with only the changed fields (PATCH is creator-only; built-ins are immutable, so copy them first).

Dimensions the UI cannot author. For a from-scratch grader you define scores freely, including the types the editor does not offer:

"scores": {
  "prod_aws_change": {
    "type": "bool",
    "name": "Production AWS change",
    "description": "Whether the trajectory changes any resource in a production AWS account.",
    "definition": "true only if a resource in a production AWS account (ids: 111122223333, 444455556666) is created, modified, or deleted."
  }
}

Python SDK alternative: the client must be opened as a context manager: with WatcherClient(base_url="https://<hostname>/api", access_token=token) as client: client.create_grader(name=..., header=..., scores={...}, public=True), where token is the same Bearer access token read from the credentials file above (api_key cannot create graders).

Putting the grader into service

Trailing review

Trailing review runs server-side with the built-in Summarize and Deep Review graders and is not currently configurable: a custom grader cannot be substituted into it. To run a custom grader over stored sessions, use on-demand grading through the API or SDK (a grade call runs any grader against stored trajectories).

Live blocking policy

Org-wide rollout is done through the organization's managed client settings document, an admin-authored overlay of client settings that Watcher delivers to every developer machine: in the Analyzer, open Managed Settings (the Monitors page's "Manage rollout" link goes there), expand "Advanced settings" to reach the YAML editor, and add a block like:

policy_monitors:
  policy_gateway_grader_id:
    value: <your-grader-id>
    permission: locked
tool_thresholds:
  "Bash":
    flag_threshold: 6
    deny_threshold: 8
    permission: locked

permission: locked is enforced when settings merge, so a developer cannot override the value locally.

Delivery: check the toggle before you save. Saving creates a new version, and the page's "Distribute via Watcher" toggle decides whether that version reaches machines: when it is ON, which is the default for new organizations, running clients pull the saved document within minutes; when it is OFF, Watcher distributes nothing, and you must ship the saved YAML to ~/.apollo_monitor/settings.yaml on each machine with the org's MDM tooling. This policy blocks tool calls, so confirm the delivery mode with the admin before you save it, not only before declaring the rollout done. The Monitors page states a monitor's rollout only when distributing via Watcher; under MDM it says "Rollout set by your MDM", because the server cannot see what machines run.

Triage runs before the gateway. Triage can auto-approve a call outright, and if it does, the gateway grader you pinned never runs. To guarantee a block, pin the gateway (the only stage that blocks), and stop triage from auto-approving the target action first: either pin a triage grader (policy_monitors.policy_triage_grader_id) that scores the same action high (triage still cannot block; this only prevents the early auto-approve), or lower triage_max_allow_score under policy_monitors: (coarser: affects every rule). Do NOT pin your rule only at triage expecting a block.

Terminal thresholds outrank command rules. A tool whose thresholds are set to a terminal value (always-allow with both thresholds disabled, always-deny, or always-flag) is decided instantly, before the org's command rules and before any grading. In particular, never disable a tool's thresholds ("always allow") assuming its deny command rules still apply: they will not run.

Finding tool names for tool_thresholds. Keys must match the tool name exactly, or use a glob pattern ("*" applies to every tool not otherwise listed; "Task*" matches TaskCreate, TaskGet, etc.). Claude Code built-ins include Bash, Edit, Write, Read, Glob, Grep, NotebookEdit, WebFetch, WebSearch, and the multi-agent Task* tools; Codex adds apply_patch. Custom MCP tools use their own names. For the exact name of an action, look at its tool call in an Analyzer session transcript.

Rollout constraints and gotchas.

• The grader must be public: true, or other developers' clients cannot resolve it by ID and the policy silently fails.

• A grader pinned at a live stage must define decision_score. Without it, the gateway falls back to a neutral score of 5 instead of denying, and a triage grader never auto-approves (every call falls through to the gateway).

• decision_score at the gateway must be on the 1-10 int scale, and flag_threshold / deny_threshold must be whole numbers with flag_threshold below deny_threshold; other shapes look valid but fail settings validation on the machines.

• Set deny_threshold at or below the score your rule produces: with a rule forcing a 10, any deny_threshold <= 10 blocks it. Tune per tool.

• Editing the grader's wording later takes effect automatically (the pinned ID resolves to its latest version). You only touch the settings document again to change which grader is pinned.

• The policy is mandatory but still snoozable (a developer can temporarily snooze blocking). A non-snoozable mode is not supported yet.

• Blocking only acts on machines whose enforcement mode is enforce. Under observe the pipeline still grades and records what it would have done, but never acts, so your grader produces judgments and zero blocks. Under paused LLM grading stops altogether, so your grader produces nothing at all; the deterministic checks still run, meaning a command rule or an always-deny threshold can still deny. The mode resolves from the machine's admin baseline only, so a developer cannot set it, and the managed settings document you edited above is the place to pin it. Set it there explicitly rather than assuming the fleet is enforcing:

  enforcement:
    mode:
      value: enforce
      permission: locked

• Deleting a grader that is pinned in distributed settings breaks the rollout: clients keep sending the now-invalid ID, the server rejects it, and live grading fails per the on_grading_failure policy (default: escalate) until you re-point the settings.

Verify

• Confirm the grader exists and is listed: GET /api/v1/graders/<grader_id> shows public: true.
• Confirm delivery: with "Distribute via Watcher" on, the Monitors page headlines the grader as "Rolled out as Gateway" (or Triage) under its name; under MDM, check a test machine's effective settings show your grader ID as locked.
• Have a coding agent on that machine attempt a benign action matching your rule (e.g. a command referencing one of the placeholder production account IDs) and confirm it is blocked; confirm an unrelated action still proceeds.

Placeholders above (<hostname>, <admin-email>, <grader-id>, account IDs like 111122223333) are filled in per deployment.