Graders

A grader is a versioned grading rubric: prompt text plus a set of score dimensions that an LLM applies to a trajectory. Running a grader produces a grade: one value per score dimension, plus optional explanation text. Grades are what you review in the Analyzer and query through the API.

What a grader defines

• Prompt text. A header before the score definitions and an optional footer after them. Watcher composes the full grading prompt from these parts together with a response-format section it generates from the grader's configuration, so the rubric text and the output contract cannot drift apart.
• Score dimensions. Each dimension has a key, a display name, a type (integer, float, boolean, or string), a description, and rubric text that becomes part of the prompt. The built-in graders score each dimension from 1 to 10, where higher is worse. A grader may also define no dimensions and produce only prose output, such as a title and summary.
• A default model. The LLM the grader runs on unless the caller overrides it, with optional reasoning-effort settings.
• Output options. Whether a grade includes the grading model's thinking, a written explanation, and a summary, in addition to the dimension values.

What a grade records

A grade is one grader version applied to one trajectory. It stores the value of each score dimension, the optional thinking, explanation, and summary text, the model that produced it, and the exact grader version used, so you can always recover the configuration behind a score. The same trajectory can hold many grades: from different graders, from different models, or from re-grading. See what a grade covers for how grades attach to trajectories.

Versioning

Grader versions are immutable. Editing a grader never changes an existing version: it creates a new version and archives the previous one. Existing grades keep pointing at the version that produced them, and anything configured to run a grader by its ID picks up the latest version automatically.

Where graders run

• Trailing review grades a session's latest trajectory while the session runs and after it ends. It runs on the Watcher server with the built-in Summarize and Deep Review graders; there is nothing to configure on developer machines.
• Blocking review uses graders to score a proposed tool call before it runs. The pipeline stages read a single integer dimension keyed decision_score from their grader.
• On demand, through the API or the Python SDK: a grade call runs any grader against stored trajectories.

Monitors

Monitors are managed end to end by Watcher and improve with releases, without any action on your side. Clients address them by role (triage, gateway; summarize and deep_review run server-side on a schedule) through GET /v1/monitors and POST /v1/monitors/{role}/grade rather than by grader UUID.

Their customization surface is custom instructions: Watcher's monitors already keep you safe from security incidents, and the instructions are for enforcing policies unique to your organization — for example "Never allow AWS calls by the agent." An admin sets them once (the Custom Instructions entry on the Analyzer's Monitors page, or PUT /v1/monitors/customization). Today they apply to the on-demand monitors (gateway and triage, via POST /v1/monitors/{role}/grade); the scheduled monitors (summarize and deep_review) pick them up when monitoring switches over to the managed path. The text is length-capped and is applied as organization policy; it cannot replace a monitor's rubric or output format.

Built-in graders

Watcher ships built-in graders. They cannot be edited or deleted, but you can copy one as the starting point for your own.

Grader	Role
Summarize	Prose only: a title and short summary of the trajectory, surfacing any security concerns. No score dimensions. The default trailing-review grader.
Default - Security	Scores a trajectory across nine security failure-mode dimensions, such as secret exposure, data exfiltration, and prompt injection.
Agent Reliability	Scores a trajectory across nine reliability dimensions, such as premature completion claims, fabricated information, and ignoring user corrections.
Policy Triage	Blocking review's fast pre-filter: scores a proposed tool call cheaply and can only auto-approve it.
Policy Gateway	Blocking review's deep evaluator: its decision score is compared against per-tool thresholds to allow, deny, or escalate the call.

Custom graders

The Analyzer's Monitors page can create a grader from scratch or copy an existing one as a starting point, and edit its prompt, model, and score dimensions, including adding and removing dimensions. New dimensions created in the page are integer or string typed; float and boolean dimensions are created through the API or SDK, which offer the same authoring otherwise. The custom grader prompt is a runbook that walks a coding agent through the API route. A custom grader is either visible to your whole organization or unlisted, in which case only its creator sees it in grader lists (it still runs, and its scores are visible like any grader's).