--- description: >- Review self-hosted Watcher data collection, storage, network connections, telemetry, auditing, and retention. --- > Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt > Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md > Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20Security&body=Page%3A%20Security%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fself-hosting%2Fsecurity%2F%0A%0A # Security This page describes what a self-hosted Watcher deployment collects, stores, and sends. Use it for security and compliance review. ## What is collected The [Watcher client](../client/index.md) records coding-agent [sessions](../concepts/sessions-and-trajectories.md). Each recorded session contains: - The full conversation. This includes user prompts, assistant replies, the system prompt, and tool calls with their arguments and results. Tool results include whatever the agent read or produced. Transcripts can contain source code, file contents, and any secrets the agent encountered. - The agent that produced the session ([Claude Code or Codex](../supported-agents.md)), timestamps, and the developer's identity. In proxy mode, this is the identity header your proxy injects. In SSO mode, it comes from the SSO sign-in. See [authentication mode](setup.md#choose-an-authentication-mode). - [Blocking review](../concepts/blocking-and-trailing-review.md) decisions. This includes the tool context, score, and outcome for each reviewed call. - Grades. These are the scores, explanations, and summary text produced by [graders](../concepts/graders.md). Watcher does not record the screen, keystrokes, or terminal contents. The client captures only terminal identifiers (terminal program, tty, session IDs). These let the UI focus the right window. Watcher records what the agent did, not what the developer did outside the agent. ## Where it is stored The durable copy lives in your deployment's Postgres database. This is either the bundled Docker Postgres or the managed Postgres you configured. The terminal-focus identifiers stay on the client. They are not uploaded. [Operations](operations.md#backups) covers server backups. The state on each developer's machine is covered on the [client page](../client/index.md#per-machine-settings). ## Which LLM providers are called All LLM grading runs from your Watcher deployment, never from developer machines. Blocking review's triage and gateway stages send the transcript excerpt from the client to your deployment. Your deployment calls the grading model. [Trailing review](../concepts/blocking-and-trailing-review.md) grades server-side the same way. The API makes provider calls. The grading scheduler triggers server-side grading and needs database access. It does not hold provider keys. The scheduler does hold database credentials. Each pass connects to Postgres to list the organizations to grade. Then it calls the API for each one. It reads no session content. Include it when you inventory which containers receive database credentials. Allow it a route to Postgres when you restrict network or database access. If a policy blocks that connection, the scheduler logs `async grading sweep failed`. It retries on the next interval. Trailing review stays stopped until you restore access. The `xylon-retention-cleanup` container also holds database credentials. It runs the `xylon-api` image with a dedicated entrypoint. It contacts no LLM provider. It connects to Postgres to delete expired unlinked grader inputs. It also enforces any enabled organization session-content retention policy. That policy is disabled by default. Organization administrators configure it through the Analyzer's [Data Retention controls](../concepts/organization-settings.md#data-retention) or the organization settings API. If the worker cannot reach Postgres, it logs the failed sweep and retries on the next interval. The default [managed monitors](../concepts/graders.md#monitors) use three providers across four monitors: - Triage uses OpenAI's `gpt-5.6-luna` - Gateway uses Anthropic's `claude-opus-4-7` - Deep Review uses Anthropic's `claude-sonnet-5` - Session summaries use Google's `gemini-2.5-flash` Obtain access to these exact models before setup. They are Watcher's recommended and tested configuration. Grading calls send session content to the provider. This includes prompts, agent output, and tool results. You supply the provider API keys. Provider choice, accounts, and data-processing terms are yours. You can override monitor models in `server-config.yaml` when your deployment uses a different provider set. Alternative models are not Watcher's recommended, currently tested configuration. They may impede monitor performance. A missing key or unavailable model makes the affected grading path fail. Watcher does not select another model automatically. ## Self-hosted Watcher does not phone home The stack sends no usage analytics, error reporting, or other telemetry to Apollo. It makes these outbound connections when configured: - **LLM APIs:** grading sends session content to the configured model providers. - **WorkOS:** in [SSO mode](setup.md#sso-mode), the API uses WorkOS for sign-in, token management, and organization membership. Proxy mode does not use it. - **Slack:** if an administrator configures a webhook, notifications send the grader's summary, scores, and an Analyzer link. Blocking-review alerts can also contain tool arguments and paths. Watcher replaces credentials in common formats with `[REDACTED]` before sending, but a secret in another format can still appear. To keep this traffic inside your network boundary, set the `SLACK_WEBHOOK_URL` environment variable on the API to an internal relay that forwards to Slack: it overrides the webhook URL saved in organization settings (see [Organization settings](../concepts/organization-settings.md#deployment-managed-slack-webhook)). A URL saved in organization settings must use `https://`. The `SLACK_WEBHOOK_URL` variable accepts `http://` as well, for a relay on a trusted network that does not terminate TLS. Over `http://` the notification and the webhook URL itself are sent in cleartext and no certificate is verified, and the URL's path is the credential that lets anyone holding it post to the channel, so use `http://` only on a network you trust to carry both. An `https://` destination has its certificate verified on every send, so an internal relay reached over `https://` needs a certificate that chains to a publicly trusted root (the trust requirement is spelled out on the linked page). Installation and upgrades pull server images from a container registry. They also pull client files from a public release or internal mirror. These requests contain no session data. See [Supply chain security](supply-chain.md) for internal mirror options. The Watcher client sends session data only to the backend recorded during installation. The [deployment example](deployment-example.md) shows these connections. ## Admin access auditing Admins can see every user's sessions through the [Analyzer](../concepts/analyzer.md). The deployment keeps the reciprocal record. It maintains an [admin audit log](operations.md#admin-audit-log), on by default. The log appends one JSON line for each of these actions: - Reads of another user's session data - Session list or search requests - User-directory reads - Organization-level settings writes - Session-content scrubs (always recorded) Administrator actions carry the actor's email. Automatic retention uses `retention-policy` as the actor and `system` as the authentication method. Records carry resource identifiers, never session content. The logs stay in a Docker volume for Compose. For Kubernetes, they stay in [persistent volume claims](kubernetes-infrastructure.md#scale-and-operate). They contain personal data and are not collected by the support bundle. [Operations](operations.md#admin-audit-log) covers enabling, reviewing, and pruning it. ## Retention and deletion By default, Watcher retains session content indefinitely. An administrator can remove the stored content of one session. Use `DELETE /api/v1/sessions/{session_id}/content`. This operation removes the session's messages, tool-call records, and linked blocking-review decisions and grades from the active Postgres database. The retained records described below can still contain session-derived prose. The server also supports an organization-level inactive-session retention policy. It is disabled by default. Session content is retained indefinitely. Organization administrators can enable it from the Analyzer's [Data Retention controls](../concepts/organization-settings.md#data-retention) or with `PATCH /api/v1/organization/settings`. The configured inactivity period must be between 7 and 365 days. Setting it to `null` disables the policy. The always-on retention worker removes content from sessions whose latest recorded activity is at least the configured number of days old. The worker runs immediately at startup and every 24 hours thereafter. The scrub retains trajectory grades. This includes their explanations and summaries. It also retains notification titles and bodies, in their original form rather than the redacted form sent to Slack. These can contain sensitive excerpts from the session. Scrubbing does not redact them. The scrub also retains the session record. This includes its historical activity dates, message count, and content-free activity history. For each tool call, it retains one content-free fact. This fact includes: - The tool-call ID - When the call happened - What Watcher decided (allowed, denied, or escalated to a person, and under which enforcement mode) - The numeric scores that decision was based on These scores are recorded on the fact as the call is decided. They outlive the grades they came from. The fact never carries the tool's name, its arguments, or any prose. These facts keep tool-call counts, filters, and [severity](../concepts/analyzer.md#severity) accurate for a scrubbed session in the [Analyzer](../concepts/analyzer.md). A later full upload of the same session can restore its messages and reconnect retained trajectory grades. Scrubbing does not prevent re-upload. Sessions created after [distinct tool invocations](setup.md#distinct-tool-invocations) are enabled work like this. Restoring the same conversation with the same tool-call IDs reconnects retained grades to the corresponding messages. This works even when branches are uploaded in a different order. Missing or reused IDs cannot distinguish otherwise identical invocations. Sessions created before this setting was enabled work differently. They can treat equal-content tool calls at the same conversation position as the same message. This happens even when their IDs differ. Restoring such content after a scrub can reconnect a grade from an earlier invocation. That association does not establish that the restored invocation was graded. Upgrading or enabling the setting does not change these existing sessions. It does not repair earlier conflations. A blocking-review grade whose corresponding decision has not yet arrived is removed by the scrub. This happens when the grade recorded which session it was graded for. Current Watcher clients write grades this way. Unlinked grades without that session reference cannot be removed by this session-specific operation. These come from older clients or were graded before the session's first upload. Watcher's always-on retention worker makes these unlinked grader inputs eligible for deletion after eight days. It deletes them on its next sweep. Linked grades and session content outside an enabled policy remain until an administrator scrubs the session. Scrubbing and the retention worker do not alter source files, client-local state, external copies, or existing backups. Saved [managed client settings](../concepts/organization-settings.md#managed-client-settings) documents are also retained. They are kept as an append-only version history. Retention beyond these session-content and orphan-grade policies is yours to implement. Use standard database tooling. Backups remain under your control. See [Operations](operations.md#backups). On developer machines, `watcher uninstall` removes the hooks and the binary. It intentionally keeps the local state under `~/.apollo_monitor/`. It prints a reminder to remove it manually. Uninstalling a client never touches the data already uploaded to the deployment. ## Client credential helpers An [external credential helper](../client/credential-helpers.md) is trusted local code that runs with the developer account's privileges. Watcher validates its JSON output, extracts a bearer token, and limits invocation time and output. The gateway must validate token identity, audience, expiry, and authenticity before injecting identity or role headers. Block direct backend access and strip client-supplied versions of those headers. Helper output itself grants no authority, and owner-only state does not protect against unrestricted processes running as the same user.