Graders

A grader is a versioned grading rubric. It has prompt text plus a set of score dimensions. An LLM applies these to a trajectory. Running a grader produces a grade. This includes one value per score dimension, plus optional explanation text. Grades are what you review in the Analyzer and query through the API.

What a grader defines

• Prompt text. A header before the score definitions and an optional footer after them. Watcher composes the full grading prompt from these parts. It adds a response-format section it generates from the grader's configuration. The rubric text and the output contract cannot drift apart.
• Score dimensions. Each dimension has a key, a display name, and a type. The type is integer, float, boolean, or string. It also has a description and rubric text that becomes part of the prompt. The built-in graders score each dimension from 1 to 10, where higher is worse. A grader may also define no dimensions. It can produce only prose output, such as a title and summary.
• A default model. The LLM the grader runs on. The caller can override it. It includes optional reasoning-effort settings.
• Output options. Whether a grade includes the grading model's thinking, a written explanation, and a summary. These are in addition to the dimension values.

What a grade records

A grade is one grader version applied to one trajectory. It stores the value of each score dimension. It also stores the optional thinking, explanation, and summary text. It records the model that produced it and the exact grader version used. You can always recover the configuration behind a score. The same trajectory can hold many grades. They come from different graders, from different models, or from re-grading. See what a grade covers for how grades attach to trajectories.

Versioning

Grader versions are immutable. Editing a grader never changes an existing version. It creates a new version and archives the previous one. Existing grades keep pointing at the version that produced them. Anything configured to run a grader by its ID picks up the latest version automatically.

Where graders run

• Trailing review grades a session's latest trajectory while the session runs and after it ends. It runs on the Watcher server with the built-in Summarize and Deep Review graders. There is nothing to configure on developer machines.
• Blocking review uses graders to score a proposed tool call before it runs. The pipeline stages read a single integer dimension keyed decision_score from their grader.
• On demand, through the API or the Python SDK. A grade call runs any grader against stored trajectories.

Custom graders run when called through the API or SDK. They do not run in blocking review, which always uses the managed monitors below, nor in server-side trailing review.

Monitors

Watcher manages monitor prompts and models. It updates them with releases. Before upgrading, check the release pages for required model-access or configuration steps. Clients address monitors by role. The roles are triage and gateway. summarize and deep_review run server-side on a schedule. The Watcher client sends each gated call to the gateway hook endpoint, which runs triage and gateway by role. Other clients can use GET /v1/monitors and POST /v1/monitors/{role}/grade rather than grader UUID.

Their customization surface is custom instructions, shared across the organization. Use them to add policies such as "Never allow AWS calls by the agent." An admin sets them in the Analyzer's Custom Instructions editor or through the monitor customization API.

Instructions apply when the managed triage and gateway monitors run, including through gateway hooks. They do not affect custom graders or the scheduled summarize and deep_review monitors. They add organization policy without replacing a monitor's rubric or output format. Calls resolved by deterministic checks do not reach these monitors.

Built-in graders

Watcher ships built-in graders. They cannot be edited or deleted, but you can copy one as the starting point for your own.

Grader	Role
Summarize	Prose only. A title and short summary of the trajectory, surfacing any security concerns. No score dimensions. The default trailing-review grader.
Default - Security	Scores a trajectory across nine security failure-mode dimensions. Examples are secret exposure, data exfiltration, and prompt injection.
Agent Reliability	Scores a trajectory across nine reliability dimensions. Examples are premature completion claims, fabricated information, and ignoring user corrections.
Policy Triage	Blocking review's fast pre-filter. Scores a proposed tool call cheaply and can only auto-approve it.
Policy Gateway	Blocking review's deep evaluator. Its decision score is compared against per-tool thresholds to allow, deny, or escalate the call.

Custom graders

The Analyzer marks Legacy & custom monitors as deprecated. Custom graders remain available through the API and SDK, but blocking review no longer runs them: a grader still pinned in policy_monitors.policy_triage_grader_id or policy_gateway_grader_id is ignored. For organization policy in blocking review, use custom instructions above. They do not replace bespoke score dimensions or on-demand graders.

The Analyzer's Monitors page can create a grader from scratch. It can also copy an existing one as a starting point. You can edit its prompt, model, and score dimensions. This includes adding and removing dimensions. New dimensions created in the page are integer or string typed. Float and boolean dimensions are created through the API or SDK. They offer the same authoring otherwise. The custom grader prompt is a runbook. It walks a coding agent through the API route. A custom grader is either visible to your whole organization or unlisted. If unlisted, only its creator sees it in grader lists. It still runs, and its scores are visible like any grader's.