Update Watcher

Self-hosted deployments should update the server first. Watcher Cloud users can go straight to the release checks and client update steps.

Watcher clients installed with the public installer follow the server version automatically. You can also update them. Run watcher update to do this.

We recommend working through the update with a coding agent. Let it inspect your deployment definitions, workload status, and server logs. Do not share secret values. Ask it to check every required action. Ask it to run the verification steps below. This catches configuration regressions before you update the client fleet. Use Verify an update for the complete go/no-go check on one test machine.

For ongoing maintenance, set up automatic updates. Use your host scheduler or deployment pipeline. Keep this page's release checks and update sequence in that workflow.

Check release-specific upgrade steps

Before updating, open /api/v1/version on your known Analyzer URL. For example, use https://watcher.example.com/api/v1/version. Record the version value. If proxy authentication blocks an unauthenticated request to that endpoint, sign in to the Analyzer. Read the build version at the bottom of the expanded navigation sidebar.

Before every upgrade, read the release pages. Also read their linked full changelogs. Check every version newer than the installed version and no newer than the target. Read them even when no required steps are listed. This helps you understand behavior changes and compatibility before acting. Complete applicable required steps oldest first. This includes versions you skip.

For a client upgrade, record watcher --version before updating. Use that as the starting version. If different machines run different versions, check the releases crossed by each machine. Each release guide labels self-hosted, cloud, and all-deployment requirements.

Self-hosted operators must complete the self-hosted steps. They must also complete any steps that apply to all deployments. Watcher Cloud users only need the cloud or all-deployment steps. Apollo operates the cloud server. Cloud users can skip Update the server and follow Update Watcher clients.

Upgrade PostgreSQL

Watcher requires PostgreSQL 17 or later. The background-job worker uses PostgreSQL 17's transaction timeout and refuses an older server. Check the version with SHOW server_version_num; (170000 or higher is PostgreSQL 17 or later).

• Bundled database (Docker Compose or the Helm chart's evaluation database): it runs PostgreSQL 18. Nothing to do.
• Amazon RDS created by the Watcher Terraform module: databases created before PostgreSQL 17 became the module's default run PostgreSQL 16. Follow Upgrade an existing database.
• Any other managed database (RDS outside the module, Cloud SQL, Azure Database for PostgreSQL): use your provider's major-version upgrade. Take a backup, rehearse the upgrade on a restored copy and test Watcher against it, then stop Watcher's writers (docker compose stop, or scale the Helm workloads to zero), upgrade, check the version, and start Watcher again.
• A database you run yourself: use pg_upgrade or dump and restore into a new PostgreSQL 17 or later server, with the same backup, rehearsal and downtime steps.

Upgrading the database is separate from updating Watcher. It can happen before or after a Watcher update. Until it does, the background-job worker cannot run, and requests for its jobs (such as Generate report) fail.

Update the server

1. Back up the database and your deployment configuration. Database migrations only move forward. A rollback requires the pre-update database backup.

2. Verify and unpack the target release beside the current release. Use its deployment definitions and image tag together. If you use an internal registry, import the target tag for the xylon-api, bootstrap, and analyzer images before continuing.

3. Refresh any release-owned utilities you use. This includes the support bundle, scripts, authentication proxy, prompts, and infrastructure modules.

4. Apply your operator-owned configuration to the target release's deployment definitions. Preserve these items:

   • Environment values
   • Secret references
   • server-config.yaml
   • managed-settings.yaml
   • Certificates
   • Storage settings
   • Proxy and network settings

   Do not carry the previous release's deployment definitions forward.

5. Run the target release's one-shot bootstrap image against the deployment's database and configuration. It must complete successfully before the long-running workloads update. The shipped Compose configuration enforces this with service dependencies. The Helm chart uses a pre-upgrade Job.

6. Deploy the target image tag for the API, Analyzer, and retention cleanup workloads. Use your orchestration system. Deploy the grading scheduler only when async grading is enabled in your operator-owned configuration.

7. Complete Verify an update on one test machine. Follow its separate rollout guidance for public-installer and mirror-backed clients.

If you use the bundled Terraform module, update it separately. Follow Manage the RDS version for an existing managed database, then review terraform plan. Read Replace the instance before applying a plan that replaces the host.

To roll back, follow these steps:

1. Stop the updated workloads.
2. Select the previous release's deployment definitions and image tag.
3. Reapply your operator-owned configuration to them.
4. Follow the restore procedure with the pre-update database backup.

The restore procedure's final start must launch the previous release, not the target release. Do not run the previous server release against the migrated database.

Update Watcher clients

A client installed with the public installer and running through its login agent checks every two to two-and-a-half hours. It reads the configured server's version. It installs that client release. This includes moving back to an older release after a server rollback. Sign-in and local settings are kept.

Rolling back to a release earlier than 0.36.0 isn't automatic. The client refuses those releases and retries about every hour without success. After such a server rollback, reinstall the client at the server's release on each machine:

watcher uninstall
curl -fsSL https://github.com/ApolloResearch/watcher-bin/releases/download/v<version>/install.sh | bash -s -- --watcher-api-url https://watcher.example.com
watcher doctor

To update immediately on one machine, run:

watcher update
watcher --version

Public-installer clients running only in the foreground do not check automatically. Run watcher update for those clients.

Do not update by re-running the installer. On a machine that already has a client, it refuses and exits. It prints Watcher is already installed. For a client from a release that predates watcher update, run watcher uninstall, then re-run the installer. Use watcher update from then on. Sign-in and local settings survive. Uninstall keeps the client's data directory.

Clients installed from an internal mirror do not update themselves. Publish the approved client release to the mirror. Redeploy it through your fleet tooling. An MDM installation that uses the public installer can rely on the built-in update check. The MDM continues to repair the installation.

See Client deployment for installation paths and MDM rollout for managed fleets.