--- description: >- The agent runbook for creating or customizing an LLM grader on a self-hosted Watcher deployment, and rolling it out for live blocking or on-demand grading. --- > Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt > Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md > Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20Watcher%20Self-Host%3A%20Create%20or%20Customize%20a%20Grader&body=Page%3A%20Watcher%20Self-Host%3A%20Create%20or%20Customize%20a%20Grader%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fself-hosting%2Fprompts%2Fcustom-grader%2F%0A%0A # Watcher Self-Host: Create or Customize a Grader You are an agentic AI assistant helping a Watcher self-host administrator create a custom grader (a versioned rubric: a prompt plus score dimensions that an LLM applies to a session's transcript, producing one value per dimension) or customize an existing one and, optionally, put it into service as the mandatory live blocking policy. Drive this yourself: make the API calls, parse the responses, and only ask the admin for values you cannot obtain (their hostname, auth material, which AWS accounts count as production, etc.). Watcher runs graders in two contexts: - **Trailing review:** scores a session's latest trajectory as it grows (shown in the Analyzer, the web dashboard your Watcher deployment serves); it never blocks. It runs server-side with the built-in graders and does not take custom graders. - **The live blocking pipeline:** scores a proposed tool call before it runs and can block it. This is a sequence of stages: terminal thresholds, then command rules (deterministic patterns matched against the call, which allow, deny, or escalate it outright without any grading), then triage, then gateway. Only the gateway stage blocks by score, comparing its decision score against the tool's thresholds; triage runs earlier and can only auto-approve. ## Choose the route: UI or API The Analyzer marks Legacy & custom monitors as deprecated. Existing custom graders still run where configured. For organization policy on managed live monitors, prefer the Monitors page's Custom Instructions editor. Its text applies to managed triage and gateway monitors, not custom graders or scheduled trailing review. Continue with this runbook when the admin needs a custom rubric, model, score dimensions, or on-demand grader. - **Analyzer UI (default).** The Monitors page covers most authoring: **New monitor** creates a grader from scratch, **Copy & Edit** copies a built-in (built-ins are immutable), and the editor edits the prompt header/footer and model and adds or removes score dimensions. The editor offers two dimension types: `int` (always on the 1-10 scale the pipeline understands) and `str`. Its "Can block tool calls" checkbox adds the `decision_score` dimension a live-pipeline grader needs. Saving creates a new immutable version automatically. - **API (when required).** Use the API for what the editor cannot author: `float` or `bool` dimensions and reasoning-token budgets, or for scripted/CI authoring. Everything the UI does is also available through the API. When the UI route suffices, walk the admin through those clicks and skip ahead to "Putting the grader into service"; the sections in between are the API route. ## Before you start: authentication (API route) You call the same public Watcher URL the dashboard uses, under `/api/v1/...`. You need the deployment's hostname and a working way to authenticate; ask the admin for either if you do not have it. **1. Discover the auth mode.** `GET /api/v1/auth/client-config` is unauthenticated and returns `auth_mode` (`"proxy"` or `"workos"`): ``` curl -fsS https:///api/v1/auth/client-config ``` **2. Get credentials for that mode.** - **WorkOS mode:** the admin signs in once from the client's local UI at `http://localhost:8228`, which hands off to the organization's identity provider in the browser. The sign-in writes a credentials file named after the backend hostname, and its `access_token` field is the Bearer token; there is no CLI command that prints it, so read the file: ``` token=$(jq -r .access_token ~/.apollo_monitor/credentials/.json) ``` Pass it as `-H "Authorization: Bearer $token"`. The token is short-lived and the running client refreshes it every few minutes, so re-read the file if a request starts returning 401. An organization API key (`-H "x-api-key: "`) works for read and list calls, but NOT for creating graders: graders are user-owned, and the server refuses grader creation under service auth. Use the user token for the authoring steps. - **Proxy mode:** the deployment's reverse proxy authenticates requests after its own login, so there is no single header you can assume. From a machine the proxy trusts you may be able to assert `-H "X-Forwarded-User: " -H "X-Forwarded-User-Role: admin"` directly (plus `-H "X-Auth-Proxy-Token: "` if the deployment set one); other deployments only accept requests that actually traverse the proxy. If it is not clear how to make an authenticated call, ask the admin. **3. Verify before proceeding.** `GET /api/v1/graders?limit=1` succeeds with any working credential (user token, proxy identity, or API key) and returns 401/403 when auth is broken: ``` curl -fsS 'https:///api/v1/graders?limit=1' ``` (`/api/v1/users/me` also works as a check, but only for user credentials; it returns 401 under an API key, which authenticates as the organization rather than a user.) In the examples below, `` stands for the auth flag(s) from step 2. Creating a grader requires an authenticated user (the grader becomes theirs; API keys cannot create graders); distributing it as org policy requires organization-settings write permission. ## What a grader is made of - `name`, `description`. - `header` / `footer`: prompt text before / after the score definitions. - `scores`: a map of dimension key -> definition. Each dimension has `type` (`int` | `float` | `bool` | `str`), `description`, optional `name` (display), optional `definition` (the rubric text the model sees), and optional `min_value` / `max_value`. - `default_model` (must be a model Watcher knows: the built-in list served by `GET /api/v1/graders/models`, plus any deployment additions from `server-config.yaml`; or simply reuse the `default_model` of the built-in grader you copied). To use a model outside that list, either add it under `model_pricing` in `server-config.yaml` with its rates and token budget, or set `WATCHER_ENABLE_ANY_MODEL=true` in `.env`; flag-admitted models grade normally, but the usage dashboard cannot price them (they appear under `unpriced_models` in cost estimates). A model from a provider other than Anthropic, OpenAI, or Google needs one more thing: the stack forwards only those three providers' API keys into the grading container, so the environment variables the new provider requires (for OpenRouter, an `OPENROUTER_API_KEY`) must be set in `.env` and forwarded via `compose.override.yml`, or grading fails at call time even though the model was accepted. For OpenRouter, set `OPENROUTER_API_KEY` in `.env` and forward it: ```yaml services: xylon-api: environment: OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-} ``` Merge the `environment:` lines into any existing `xylon-api:` entry rather than adding a second one; Compose rejects a file that repeats a service key. Each provider's supported variables are listed in Inspect's providers reference (https://inspect.aisi.org.uk/providers.html); some providers authenticate via credential chains or base URLs rather than an API key, so forward whatever variables the reference names for yours. Step 4.6 of "Watcher Self-Host: Guided Setup" covers the same forwarding mechanism. Find that runbook through the docs index linked in the bundle's `prompts/setup.md`. The `model_pricing` entry format is documented in `server-config.example.yaml`, shipped alongside `docker-compose.yml`. Apply configuration changes by recreating the API container: `docker compose up -d --force-recreate xylon-api`. Plain `docker compose up -d` picks up `.env` and `compose.override.yml` edits, but it does not recreate the container after a content-only edit to the mounted `server-config.yaml`, which is read only at startup. - `include_thinking` / `include_explanation` / `include_summary`. - `public`: `false` (only the creator can see and use it) or `true` ("listed": all members of the org can). A grader used org-wide MUST be `public: true`. Versioning: creating a grader makes version 0; every edit (`PATCH`, or saving in the UI) creates a new immutable version and archives the previous one, and anything configured to run the grader by its ID picks up the latest version automatically. The live pipeline reads one specific dimension, keyed exactly `decision_score` (int, 1-10), at both the triage and gateway stages. A grader used at either stage must define it; other dimensions are ignored by the pipeline and only surface in the Analyzer. ## API route, step 1: (recommended) start from an existing grader Fetch an existing grader to extend, so you keep its judgment and only add your criterion. To enforce a rule at a live stage, copy that stage's built-in so you inherit its `decision_score` dimension (gateway: `c0000001-0000-0000-0000-000000000001`; triage: `c0000001-0000-0000-0000-000000000002`); for on-demand grading, copy whichever grader is closest, or author from scratch. ``` curl -fsS https:///api/v1/graders/c0000001-0000-0000-0000-000000000001 ``` Read its `header`, `footer`, and `scores`. When copying a pipeline built-in, keep its `decision_score` dimension exactly; do not rename or drop it. ## API route, step 2: create the grader `POST /api/v1/graders`. For a live gateway grader, keep the source grader's `scores` as-is and express your new rule in the prompt, so the same decision score is driven to the value you want. Example: treat any change to a production AWS account as maximally severe. ``` curl -fsS -X POST https:///api/v1/graders \ -H "Content-Type: application/json" \ -d '{ "name": "Gateway + prod-AWS block", "description": "Org gateway grader; forces a 10 for production AWS account changes.", "header": " ADDITIONAL RULE: if the action creates, modifies, or deletes any resource in a PRODUCTION AWS account (account ids: 111122223333, 444455556666), it is maximally severe. Set the decision score to 10 regardless of other factors.", "footer": "", "scores": "", "default_model": "", "public": true }' ``` The 201 response includes the new `grader_id`; capture it. To change a grader you already own, `PATCH /api/v1/graders/` with only the changed fields (PATCH is creator-only; built-ins are immutable, so copy them first). **Dimensions the UI cannot author.** For a from-scratch grader you define `scores` freely, including the types the editor does not offer: ``` "scores": { "prod_aws_change": { "type": "bool", "name": "Production AWS change", "description": "Whether the trajectory changes any resource in a production AWS account.", "definition": "true only if a resource in a production AWS account (ids: 111122223333, 444455556666) is created, modified, or deleted." } } ``` Python SDK alternative: the client must be opened as a context manager: `with WatcherClient(base_url="https:///api", access_token=token) as client: client.create_grader(name=..., header=..., scores={...}, public=True)`, where `token` is the same Bearer access token read from the credentials file above (`api_key` cannot create graders). ## Putting the grader into service ### Trailing review Trailing review runs server-side with the built-in Summarize and Deep Review graders and is not currently configurable: a custom grader cannot be substituted into it. To run a custom grader over stored sessions, use on-demand grading through the API or SDK (a `grade` call runs any grader against stored trajectories). For an on-demand check, list recorded sessions using the same authentication as above: ```bash curl -fsS 'https:///api/v1/sessions?limit=10' ``` Choose an entry in `sessions` with `is_content_scrubbed: false` and a non-null `latest_trajectory_id`. Use its `session_id` and `latest_trajectory_id` below, alongside the custom `grader_id` you created. Grade your own session, or use an admin identity or one with `sessions:write:any` to grade another person's. ```bash curl -fsS -X POST https:///api/v1/grade \ -H "Content-Type: application/json" \ -d '{ "session_id": "", "trajectory_ids": [""], "grader_ids": [""] }' ``` The response's `grades` array contains one result per graded trajectory and grader. Check that `not_found_ids` is empty, the result identifies your grader and trajectory, and `error` is null. Read your dimension by its key in the result's `grades` object, for example `grades[0].grades.prod_aws_change`. Repeat with a known matching session and a known non-matching session. Compare the values with your rubric before using the grader more widely. If no suitable session is recorded, record a test conversation first, then list sessions again. ### Live blocking policy Org-wide rollout is done through the organization's managed client settings document, an admin-authored overlay of client settings that Watcher delivers to every developer machine: in the Analyzer, open Managed Settings (the Monitors page's "Manage rollout" link goes there), expand "Advanced settings" to reach the YAML editor, and add a block like: ```yaml policy_monitors: policy_gateway_grader_id: value: permission: locked tool_thresholds: "Bash": flag_threshold: 6 deny_threshold: 8 permission: locked ``` `permission: locked` is enforced when settings merge, so a developer cannot override the value locally. **Delivery: check the toggle before you save.** Saving creates a new version, and the page's "Distribute via Watcher" toggle decides whether that version reaches machines: when it is ON, which is the default for new organizations, running clients pull the saved document within minutes; when it is OFF, Watcher distributes nothing, and you must ship the saved YAML to `~/.apollo_monitor/settings.yaml` on each machine with the org's MDM tooling. This policy blocks tool calls, so confirm the delivery mode with the admin before you save it, not only before declaring the rollout done. The Monitors page states a monitor's rollout only when distributing via Watcher; under MDM it says "Rollout set by your MDM", because the server cannot see what machines run. **Triage runs before the gateway.** Triage can auto-approve a call outright, and if it does, the gateway grader you pinned never runs. To enforce your rule, pin the gateway (the only stage that blocks) and a custom triage grader (`policy_monitors.policy_triage_grader_id`) that scores the same action 10. Triage only auto-approves scores below the lowest of `flag_threshold`, `deny_threshold`, and the Triage cutoff, so 10 reaches the gateway under the thresholds above. The client setting `policy_monitors.triage_max_allow_score` only applies when a custom triage grader is pinned. Built-in triage uses its server-managed cutoff — 8 unless this deployment changed it, set as `triage_escalation_threshold` under the `triage` monitor in `server-config.yaml` (format and effects: `server-config.example.yaml`) — so lowering the client setting alone cannot enforce your rule. Do NOT pin your rule only at triage expecting a block. **Command rules can bypass both graders.** Before saving, inspect `command_rules.rules` in the same YAML editor. Change any `decision: allow` rule that can match the prohibited action to `decision: triage`, which sends it through grading. Preserve the other entries and lock the collection to prevent local rules from reopening that approval path. Merge this example into the existing block, repeating the entry for each affected rule ID: ```yaml command_rules: permission: locked rules: - id: decision: triage permission: locked ``` **Terminal thresholds outrank command rules.** A tool whose thresholds are set to a terminal value (always-allow with both thresholds disabled, always-deny, or always-flag) is decided instantly, before the org's command rules and before any grading. In particular, never disable a tool's thresholds ("always allow") assuming its deny command rules still apply: they will not run. **Finding tool names for `tool_thresholds`.** Keys must match the tool name exactly, or use a glob pattern (`"*"` applies to every tool not otherwise listed; `"Task*"` matches `TaskCreate`, `TaskGet`, etc.). Claude Code built-ins include `Bash`, `Edit`, `Write`, `Read`, `Glob`, `Grep`, `NotebookEdit`, `WebFetch`, `WebSearch`, and the multi-agent `Task*` tools; Codex adds `apply_patch`. Custom MCP tools use their own names. For the exact name of an action, look at its tool call in an Analyzer session transcript. Gateway hooks evaluate calls from other agents under the matching Claude Code name (for example OpenCode's `bash` and Gemini CLI's `run_shell_command` as `Bash`), so a `Bash` entry covers them; an entry keyed exactly to the agent's own name, as the transcript shows it, still takes precedence. **Rollout constraints and gotchas.** - The grader must be `public: true`, or other developers' clients cannot resolve it by ID and the policy silently fails. - A grader pinned at a live stage must define `decision_score`. Without it, the gateway falls back to a neutral score of 5 instead of denying, and a triage grader never auto-approves (every call falls through to the gateway). - `decision_score` at the gateway must be on the 1-10 int scale, and `flag_threshold` / `deny_threshold` must be whole numbers with `flag_threshold` below `deny_threshold`; other shapes look valid but fail settings validation on the machines. - Set `deny_threshold` at or below the score your rule produces: with a rule forcing a 10, any `deny_threshold` <= 10 blocks it. Tune per tool. - Editing the grader's wording later takes effect automatically (the pinned ID resolves to its latest version). You only touch the settings document again to change which grader is pinned. - A developer can temporarily snooze grading only when the administrator enables it. To disable snoozing, set this in the managed document: ```yaml snooze: enabled: value: false permission: locked ``` - Blocking only acts on machines whose enforcement mode is `enforce`. Under `observe` the pipeline still grades and records what it would have done, but never acts, so your grader produces judgments and zero blocks. Under `paused` LLM grading stops altogether, so your grader produces nothing at all; the deterministic checks still run, meaning a command rule or an always-deny threshold can still deny. The mode resolves from the machine's admin baseline only, so a developer cannot set it, and the managed settings document you edited above is the place to pin it. Set it there explicitly rather than assuming the fleet is enforcing: ```yaml enforcement: mode: value: enforce permission: locked ``` - Deleting a grader that is pinned in distributed settings breaks the rollout: clients keep sending the now-invalid ID, the server rejects it, and live grading fails per the `on_grading_failure` policy (default: escalate) until you re-point the settings. ## Verify - Confirm the grader exists and is listed: `GET /api/v1/graders/` shows `public: true`. - Confirm delivery: with "Distribute via Watcher" on, the Monitors page headlines the grader as "Rolled out as Gateway" (or Triage) under its name; under MDM, check a test machine's effective settings show your grader ID as locked. - Test an action that actually matches the rule. For the AWS example, use an operator-approved disposable account and temporarily add its ID to both graders' production-account lists. Ask the coding agent to create a test parameter there with `aws ssm put-parameter --name /watcher-policy-test --type String --value test --profile `. Confirm the call is blocked before execution, no parameter is created, and an unrelated action still proceeds. Remove the test account from both graders after verification. Merely printing an account ID does not test the mutation rule. Placeholders above (``, ``, ``, account IDs like `111122223333`) are filled in per deployment.