---
description: >-
  What a Watcher grader is: a versioned LLM grading rubric with score
  dimensions, the built-in graders, where they run, and how to create your own.
---

> Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt
> Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md
> Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20Graders&body=Page%3A%20Graders%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fconcepts%2Fgraders%2F%0A%0A

# Graders

A grader is a versioned grading rubric. It has prompt text plus a set of score
dimensions. An LLM applies these to a
[trajectory](sessions-and-trajectories.md). Running a grader produces a grade.
This includes one value per score dimension, plus optional explanation text.
Grades are what you review in the [Analyzer](analyzer.md) and query through the
API.

## What a grader defines

- **Prompt text.** A header before the score definitions and an optional footer
  after them. Watcher composes the full grading prompt from these parts. It adds
  a response-format section it generates from the grader's configuration. The
  rubric text and the output contract cannot drift apart.
- **Score dimensions.** Each dimension has a key, a display name, and a type.
  The type is integer, float, boolean, or string. It also has a description and
  rubric text that becomes part of the prompt. The built-in graders score each
  dimension from 1 to 10, where higher is worse. A grader may also define no
  dimensions. It can produce only prose output, such as a title and summary.
- **A default model.** The LLM the grader runs on. The caller can override it.
  It includes optional reasoning-effort settings.
- **Output options.** Whether a grade includes the grading model's thinking, a
  written explanation, and a summary. These are in addition to the dimension
  values.

## What a grade records

A grade is one grader version applied to one trajectory. It stores the value of
each score dimension. It also stores the optional thinking, explanation, and
summary text. It records the model that produced it and the exact grader version
used. You can always recover the configuration behind a score. The same
trajectory can hold many grades. They come from different graders, from
different models, or from re-grading. See
[what a grade covers](sessions-and-trajectories.md#what-a-grade-covers) for how
grades attach to trajectories.

## Versioning

Grader versions are immutable. Editing a grader never changes an existing
version. It creates a new version and archives the previous one. Existing grades
keep pointing at the version that produced them. Anything configured to run a
grader by its ID picks up the latest version automatically.

## Where graders run

- **[Trailing review](blocking-and-trailing-review.md)** grades a session's
  latest trajectory while the session runs and after it ends. It runs on the
  Watcher server with the built-in Summarize and Deep Review graders. There is
  nothing to configure on developer machines.
- **[Blocking review](blocking-and-trailing-review.md)** uses graders to score a
  proposed tool call before it runs. The pipeline stages read a single integer
  dimension keyed `decision_score` from their grader.
- **On demand**, through the [API](../api/index.md) or the
  [Python SDK](../api/python-sdk.md). A `grade` call runs any grader against
  stored trajectories.

Custom graders run when a blocking-review stage selects them. They also run when
called through the API or SDK. They do not automatically run in server-side
trailing review.

## Monitors

Watcher manages monitor prompts and models. It updates them with releases.
Before upgrading, check the [release pages](../releases/index.md) for required
model-access or configuration steps. Clients address monitors by **role**. The
roles are `triage` and `gateway`. `summarize` and `deep_review` run server-side
on a schedule. Clients use `GET /v1/monitors` and
`POST /v1/monitors/{role}/grade` rather than grader UUID.

Their customization surface is **custom instructions**, shared across the
organization. Use them to add policies such as "Never allow AWS calls by the
agent." An admin sets them in the Analyzer's Custom Instructions editor or
through the [monitor customization API](../api/monitor-customization.md).

Instructions apply when the managed `triage` and `gateway` monitors run,
including through [gateway hooks](../api/hooks.md). They do not affect custom
graders or the scheduled `summarize` and `deep_review` monitors. They add
organization policy without replacing a monitor's rubric or output format.
Calls resolved by deterministic checks do not reach these monitors.

## Built-in graders

Watcher ships built-in graders. They cannot be edited or deleted, but you can
copy one as the starting point for your own.

| Grader | Role |
| --- | --- |
| Summarize | Prose only. A title and short summary of the trajectory, surfacing any security concerns. No score dimensions. The default trailing-review grader. |
| Default - Security | Scores a trajectory across nine security failure-mode dimensions. Examples are secret exposure, data exfiltration, and prompt injection. |
| Agent Reliability | Scores a trajectory across nine reliability dimensions. Examples are premature completion claims, fabricated information, and ignoring user corrections. |
| Policy Triage | Blocking review's fast pre-filter. Scores a proposed tool call cheaply and can only auto-approve it. |
| Policy Gateway | Blocking review's deep evaluator. Its decision score is compared against per-tool thresholds to allow, deny, or escalate the call. |

## Custom graders

The Analyzer marks **Legacy & custom monitors** as deprecated. Existing custom
graders still run where configured and remain available through the API and SDK.
For organization policy on managed live monitors, prefer custom instructions
above. They do not replace bespoke score dimensions or on-demand graders.

The Analyzer's Monitors page can create a grader from scratch. It can also copy
an existing one as a starting point. You can edit its prompt, model, and score
dimensions. This includes adding and removing dimensions. New dimensions created
in the page are integer or string typed. Float and boolean dimensions are
created through the API or SDK. They offer the same authoring otherwise. The
[custom grader prompt](../self-hosting/prompts/custom-grader.md) is a runbook.
It walks a coding agent through the API route. A custom grader is either visible
to your whole organization or unlisted. If unlisted, only its creator sees it in
grader lists. It still runs, and its scores are visible like any grader's.