Required updates for Watcher

Watcher clients now send each gated call to your backend, which grades it with the managed triage and gateway monitors against the organization's saved managed settings. A custom grader pinned for live blocking stops taking effect.

New sessions keep otherwise identical tool calls separate when their IDs differ. This is now enabled by default. Self-hosted upgrades must finish replacing old ingestion code before using that default.

Session access is split into aggregates, findings, and content.

Clients check each release with GitHub before installing it. They need HTTPS access to api.github.com, and a server rollback to a release earlier than 0.36.0 no longer carries clients with it.

What's new?

• A grader pinned in policy_monitors.policy_triage_grader_id or policy_monitors.policy_gateway_grader_id is ignored; the managed monitors run instead.
• Custom graders still run when called on demand through the API or SDK.
• Your backend judges each call by the managed settings saved in the Analyzer, plus what that machine's local settings and profile change. A settings file a machine received another way no longer decides on its own: if your MDM ships a file that differs from the saved document, the saved document wins, and with no saved document your backend uses the built-in defaults.
• In paused enforcement mode, Watcher clients now stop the whole pipeline, as the API does: no threshold, command rule or monitor decides a call, so an always-deny threshold or a deny rule no longer blocks anything while paused. Previously a paused client still applied thresholds and command rules and skipped only the monitors.
• 1a. The server, Docker Compose and Helm enable distinct tool invocations by default. An explicit false override still disables the setting.
• 1b. Existing sessions keep their stored identity version. This change needs no database migration and does not repair earlier collisions.
• Admins retain full access by default. The new org-viewer role reads organization metrics and all findings, but only its own session content. Members read their own sessions. Org-viewers and members cannot manage settings.
• sessions:read:any is replaced by three independent grants: sessions:aggregates:read:any for organization metrics, sessions:findings:read:any for browsing sessions and reading grades, and sessions:content:read:any for summaries and transcripts. See API access rules.
• Grading another user's stored session requires sessions:grade:any. Standard monitors and deep reviews need no content grant; caller-selected graders also require content access. sessions:write:any no longer authorizes this. Owners can still grade their own sessions, and automatic monitoring and grading supplied messages are unaffected.
• Findings may include transcript excerpts, but session summaries require content access.
• Slack alerts default to full details, independently of recipients' Watcher content permissions. Choose Findings and aggregates only in organization settings to omit session summaries and raw tool content. Findings may still contain transcript excerpts; control access through channel membership.
• Analyzer hides Dashboard when neither aggregates nor findings are accessible.
• Users can request temporary access to a session's content. Admins approve or deny requests in Analyzer, including their own; approvals last eight hours and can be revoked early. Custom WorkOS roles can also receive sessions:access-requests:review. The API also allows shorter approvals. Each decision records the reviewer and timestamps.
• The optional organization content restriction overrides blanket content grants, including admins' and API keys', while preserving ownership and active approvals. It defaults to off. Viewing it requires org:settings:read; changing it requires org:settings:write. Aggregates, findings, and automatic monitoring are unaffected.
• Before it changes anything, watcher update reads GitHub's record of the target release from api.github.com. It installs the release only when the release is immutable and the downloaded installer matches the SHA-256 digest in that record. The login agent's automatic check runs the same command.
• If the check cannot run or does not pass, the client stays on its installed release and the command says why. A client running through its login agent tries again about an hour later.
• Releases earlier than 0.36.0 are not immutable. watcher update and watcher downgrade refuse them.

Required upgrade steps

Live blocking grades with the managed monitors and your saved settings

These steps apply to self-hosted deployments and Watcher Cloud organizations.

1. If your managed settings pin a custom triage or gateway grader, move the rule it enforced into the managed monitors' Custom Instructions, then remove the pin from your managed settings.
2. If you distribute managed settings through your MDM ("Distribute via Watcher" off), save the exact document your MDM ships in the Analyzer, and keep shipping that saved version. See Delivering managed settings.
3. If you run any machines in paused mode and rely on an always-deny threshold or a deny rule to block calls there, switch those machines to observe (to record without blocking) or enforce (to block) in your managed settings before updating.
4. Follow Update Watcher.
5. After the update, trigger a call your settings or Custom Instructions target on an updated client and confirm it receives the expected decision.

Distinct tool invocations enabled by default

These steps apply to self-hosted deployments. Watcher Cloud users do not need to change server settings. No developer-machine changes or upload-cursor resets are needed for this change.

Use Update Watcher for the general update, backup and verification procedure. Server release 0.34.0 is the first release that supports distinct tool invocations. Check the steps for every release you skip.

Self-hosted: replace old writers before activation

If you already run 0.34.0 or later with the setting unset, the first deploy of this release activates it. In Compose, an empty value also counts as unset. Before that deploy, check every writer and restart path listed in step 2 and take a database backup. The pre-update backup is also the pre-activation backup in this case. Then deploy and complete step 4. Alternatively, set an explicit false before deploying and use the staged rollout below to defer activation.

1. If any server or ingestion job sharing the database runs pre-0.34.0 code, explicitly disable the setting for the first compatible rollout. This includes upgrading straight to this release without installing 0.34.0 first. Set WATCHER_DISTINCT_TOOL_INVOCATIONS_ENABLED=false in the Compose .env, or api.distinctToolInvocationsEnabled: false in your Helm values. For custom servers and ingestion jobs, set XYLON_DISTINCT_TOOL_INVOCATIONS_ENABLED=false in each process environment. Do not rely on the new release's default during this stage. If you cannot verify that all writers are compatible, keep the explicit false setting.
2. Deploy the target release with the setting disabled. Confirm every API replica and ingestion job sharing the database runs compatible code. On each Compose host, run docker compose ps --all. Use docker ps to check for other running writers. In Kubernetes, run kubectl rollout status deployment/xylon-api in its namespace. Inspect kubectl get pods,jobs,cronjobs and each workload's image. Also update stopped workloads, scheduled jobs, and automation that could restart an old image. One healthy new replica does not prove this step is complete.
3. Back up the database before activation. Once every writer is compatible, set the same setting to true everywhere. Redeploy the same target release as a separate step. For Compose, run docker compose up -d xylon-api on each API host after editing .env. For Helm, run your normal helm upgrade with the updated values and the same chart and image tag. Restart custom ingestion jobs with the new environment. The setting is read when each process starts.
4. Follow Verify an update. Continue an existing watched session and start a new one. Check that both receive messages in the Analyzer, and that tool-call decisions link to the expected calls. Keep your chosen value explicit in operator-owned configuration for later updates.

An existing explicit false override stays false on upgrade. If every writer already runs 0.34.0 or later with that override, verify step 2, then complete steps 3 and 4 when ready to activate. A new installation with an empty database can keep the true default.

Sessions created before activation keep their previous matching behavior. Calls with equal content but different IDs can still share a message in those sessions. New sessions need a different tool-call ID for each invocation; integrations must preserve that ID across retries and in tool results.

Rollback after activation

Once any session uses the new identity version, disabling the setting does not make pre-0.34.0 server code safe against that database. Use compatible rollback code, or restore the database backup taken before activation and its matching release. Follow the rollback procedure. Restoring a backup discards writes made after it was taken.

Session read access and viewer role

For WorkOS-backed self-hosted deployments and Watcher Cloud:

1. Before upgrading, create the read and grading grants above in WorkOS Authorization if absent. Assign each to custom roles and API keys that need it. To retain all previous session-read access, grant both findings and content; add aggregates for metrics. For cross-user stored-session grading, add sessions:grade:any; add content for caller-selected graders and findings to read the resulting grades. Enable the grants under Organization API key permissions and replace keys as needed. Admins need no extra grant. Keep sessions:read:any until the upgrade completes.
2. WorkOS environment setup provisions org-viewer with its default grants. For existing environments, create it in WorkOS Authorization if absent, then assign it to the intended users. Built-in role defaults apply even if their WorkOS grants are removed; additional grants can extend access. Refresh user credentials after changing roles or grants by signing out and back in.
3. After upgrading, remove obsolete sessions:read:any grants.

For self-hosted deployments using proxy authentication, keep the trusted X-Forwarded-User-Role header. Use admin, org-viewer, or member for the access described above. Strip client-supplied values and inject the role from the authenticated user's identity.

Apply the self-hosted server update using Update Watcher. For Watcher Cloud, prepare role and API-key assignments before rollout.

Verify access using sessions from two owners. An org-viewer should see organization metrics and both sessions' findings, but only their own summaries and transcripts. A member should see only their own sessions. Test API keys against the same boundaries for their assigned grants.

To require approval for admins too, enable the content restriction after upgrading. Verify that an admin can read another user's content only while an approval is active.

Verified client updates

These steps apply to developer machines that use the public installer, for self-hosted deployments and Watcher Cloud. Mirror-backed installs do not self-update and are unaffected.

Allow api.github.com

Clients need HTTPS access to api.github.com as well as github.com. On any network that restricts outbound traffic from developer machines, allow api.github.com before the server moves to this release. Where it is blocked, clients stop updating until it is allowed.

GitHub also limits requests per source IP. A busy shared network can hit that limit even when the host is allowed. Wait for the limit to reset, then retry.

Before rollout, run this read-only check from a representative developer machine, with the network and proxy settings used by its Watcher client. Set release_version to an existing published target version; 0.39.0 is an example:

release_version='0.39.0'
curl -fsS "https://api.github.com/repos/ApolloResearch/watcher-bin/releases/tags/v$release_version" >/dev/null

The command must exit successfully before rollout. If it fails, resolve the API access failure first. This checks access with curl; the client's own certificate configuration must also trust any HTTPS proxy on the network.

After the server update, run watcher update and then watcher --version on one machine. The version should match the server's release. This confirms installation only: the old client's initial upgrade does not use the new API check, and an up-to-date client does not contact GitHub.

Self-hosted: roll back to a release earlier than 0.36.0

Once clients have updated to this release, rolling the server back to a release earlier than 0.36.0 isn't automatic. Clients refuse the older release, and the login agent retries about every hour without success.

After such a rollback, reinstall the client at the server's release on each machine. Run watcher uninstall, run the installer from the server's release, then check the client with watcher doctor. The exact commands are in Update Watcher clients. Follow Update the server for the server rollback itself.