---
description: >-
  Verify a self-hosted Watcher server update on one test machine before updating the client fleet.
---

> Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt
> Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md
> Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20Verify%20an%20update&body=Page%3A%20Verify%20an%20update%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fself-hosting%2Fupdate-verification%2F%0A%0A

# Verify an update

Run these checks after updating the Watcher server. Run them before approving
the release for the rest of the [Watcher client](../client/index.md) fleet. They
test the deployed release through the external URL and grading paths used in
normal operation.

We recommend giving this page to a coding agent. The agent must be able to
inspect your deployment definitions, workload status, and server logs. Tell it
the known [Analyzer](../concepts/analyzer.md) URL, the target version, when the
update started, and which deployment system you use. Do not give it secret
values. The agent should translate the functional checks below into the commands
for your orchestration and logging systems.

## Check the deployment

1. Open [Releases](../releases/index.md#required-upgrade-steps). Compare the
   required steps with the installed version recorded before the update and the
   target version. Confirm that every applicable self-hosted and all-deployment
   step was completed for each crossed release, oldest first. Keep the expected
   target version, configuration, and model access available for comparison. Do
   not copy secret values into your notes.
2. Confirm that the target release's one-shot bootstrap workload completed
   successfully. Confirm that the API, Analyzer, and retention cleanup workloads
   are healthy. If async grading is enabled, confirm that the grading scheduler
   is also healthy.
3. Request the health and version endpoints through the known Analyzer URL:

   ```sh
   curl -fsS "<Analyzer URL>/api/v1/health"
   curl -fsS "<Analyzer URL>/api/v1/version"
   ```

   The shipped proxy configuration allows the health request without
   authentication. If your operator-owned proxy protects that endpoint, send the
   same request through its normal authentication mechanism. The health
   response's `status` field must be `healthy`. A successful HTTP response alone
   is not enough. The version response's `version` field must contain the target
   release. If proxy authentication blocks the version request, sign in to the
   Analyzer. Confirm that the build version at the bottom of the expanded
   navigation sidebar contains the target release.
4. Sign in to the Analyzer through its normal external URL. Confirm that the
   [session](../concepts/sessions-and-trajectories.md) list loads.
5. Review logs from the bootstrap and long-running workloads. Start at the
   update time. Resolve new migration, startup, configuration, authentication,
   permission, model-not-found, and repeated grading errors. Check scheduler
   errors when async grading is enabled.

## Test one client

Choose one test machine that is connected to this deployment. The update path
depends on how the client was installed:

- For a public-installer client, update the test machine immediately:

  ```sh
  watcher update
  watcher --version
  ```

  A public-installer client running through its login agent also checks for a
  matching server release automatically. It does this every two to
  two-and-a-half hours. Other such clients can update while this verification is
  running. This is an immediate test rather than a strict canary gate. A strict
  canary gate requires mirror-backed client distribution.
- For a mirror-backed client, publish the target client release to a canary
  group in your fleet tooling. Redeploy only the test machine. Do not run
  `watcher update`. Mirror-backed clients do not self-update.

1. Run `watcher --version` on the test machine. It must report the target
   release.

2. Open Organization Settings in the Analyzer. Download its
   [red-team drill](../concepts/organization-settings.md#red-team-drill). Give
   the prompt to a coding agent on the test machine. Agree which threat to
   probe. The drill exercises recording and
   [blocking review](../concepts/blocking-and-trailing-review.md#blocking-review)
   against the organization's real settings. Let it restore every local setting
   it changed before continuing.
3. In the Analyzer, confirm that the new session appears. Confirm that the
   tested tool calls show their blocking-review decisions and grades.
4. If async grading is enabled, read the scheduler interval from your deployment
   configuration. Use the scheduler logs to confirm that a
   [trailing-review](../concepts/blocking-and-trailing-review.md#trailing-review)
   pass started after the test session was ingested and completed. Then confirm
   that the session has a summary. A session does not need a Deep Review grade
   to pass this check. Deep Review runs only for qualifying high-scored
   sessions.
5. Review the server and client logs covering the test. Resolve new provider,
   grading, ingestion, authentication, or connection errors.

## Decide whether to continue

For mirror-backed clients and foreground-only public-installer clients, update
the remaining clients only when every applicable check above passes.
Public-installer clients running through their login agents can already have
reconciled to the target release while verification was in progress.

If a check fails, do not update any remaining mirror-backed clients or
foreground-only public-installer clients. Correct the deployment or
configuration and repeat the full verification. If recovery requires returning
to the previous server release, stop the updated workloads. Select the previous
release's definitions and image tag before following the restore procedure with
the pre-update database backup. This ensures the restore procedure starts only
the previous release. Do not run the previous server release against the
migrated database. See the [update procedure](updates.md#update-the-server) for
the full rollback context.

After the previous server release is healthy, return each client distribution to
the matching release. Run `watcher update` on public-installer clients that need
immediate recovery. For mirror-backed clients, publish the previous client
release to the mirror. Redeploy it through your fleet tooling.