--- description: >- Run the Watcher client inside containers with coding-agent hooks, authentication, and persistent settings. --- > Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt > Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md > Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20Containers&body=Page%3A%20Containers%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fclient%2Fcontainers%2F%0A%0A # Containers Teams run coding agents in containers, unattended in CI or in an agent fleet. The [Watcher client](index.md) runs there too, with three differences from a laptop install that this page covers: how the container authenticates, which review posture makes sense when nobody is watching, and what to keep on a volume so a rebuild does not wipe it. For a dev container on a developer's machine that is already signed in, [Dev containers](devcontainer.md) has a copyable template that reuses that sign-in instead of issuing a key. If the workspace must be treated as untrusted, see [Hardening dev containers](devcontainer-hardening.md) for design options rather than a ready-made template. This page is written for Claude Code. Codex works the same way with one extra consent step, described at the end. ## What has to be true inside the container The client is not only a set of hooks. The hooks ask the client's local backend for each [blocking review](../concepts/blocking-and-trailing-review.md) decision, so the client has to be running as a process in the container. If it is not, the hooks fall through to the coding agent's own permission flow: the agent keeps working, nothing is recorded, and nothing warns you. Three consequences shape everything below: - **The client starts from your entrypoint.** `watcher autostart` supports macOS login agents and Linux systemd user services; a container has neither. Use `watcher --foreground` to run without installing a login agent. - **The client must be detached from the terminal.** A background process that touches a TTY is stopped by the operating system. In a container started with `docker run -it`, a plain `watcher &` dies seconds later, silently, and you get the fall-through above. Start it with `setsid` and no stdin. - **State belongs on a volume.** Credentials, settings, and the local session and decision history live in `~/.apollo_monitor`. Without a volume, every rebuild starts from nothing. ## Prerequisites - A Watcher deployment the container can reach over the network. - An organization API key, unless the container will use one of the other credential options below. Admins create keys on the Analyzer's Organization page (see [Organization settings](../concepts/organization-settings.md#api-keys)). The key needs these permissions: | Permission | Why | | --- | --- | | `sessions:write:any` | Required for full Watcher operation under an API key because decision writes need authority over the session. It also permits cross-user ingestion. The narrower `sessions:ingest` permission is sufficient only for observe-only, unattributed collection. | | `org:settings:read` | Required only if the container should follow [managed client settings](../concepts/organization-settings.md#managed-client-settings). | | `sessions:read:any` | Optional. Needed only if something in the container reads sessions back from the [Watcher API](../api/index.md). | A `sessions:ingest` key can add turns to any member's Claude Code or Codex conversation whose ID it sends, though it cannot read sessions. Revoke it if it leaks. See [Organization settings](../concepts/organization-settings.md#api-keys). - An x86_64 image. The Linux client is published for x86_64 only, so on Apple Silicon the image is built and run with `--platform linux/amd64` under emulation. Emulation costs build and runtime speed; it does not change behavior. ## The image ```dockerfile # syntax=docker/dockerfile:1 FROM --platform=linux/amd64 node:22-bookworm-slim # The backend this machine reports to. The installer bakes it onto the agent # hook command lines, so it is a build argument rather than a runtime variable. ARG WATCHER_API_URL=https://watcher.example.com RUN apt-get update && apt-get install -y --no-install-recommends \ ca-certificates curl git tar procps socat \ && rm -rf /var/lib/apt/lists/* RUN npm install -g @anthropic-ai/claude-code RUN useradd --create-home --shell /bin/bash dev USER dev WORKDIR /home/dev ENV PATH="/home/dev/.local/bin:${PATH}" RUN curl -fsSL https://github.com/ApolloResearch/watcher-bin/releases/latest/download/install.sh \ | bash -s -- --watcher-api-url "${WATCHER_API_URL}" COPY --chown=dev:dev --chmod=0755 entrypoint.sh /home/dev/entrypoint.sh ENTRYPOINT ["/home/dev/entrypoint.sh"] CMD ["bash"] ``` Notes on this file: - The client is installed as a normal user, not root, because the agent hooks and `~/.apollo_monitor` belong to a user home. - The installer asks for consent before changing Codex's configuration. With no terminal attached, as during a build, it proceeds without prompting. - `socat` is only needed if you want to reach the client's local UI from outside the container. See [Reaching the review UI](#reaching-the-review-ui). ## The entrypoint ```bash #!/usr/bin/env bash set -euo pipefail PORT="${WATCHER_BACKEND_PORT:-8228}" # Seed the admin settings baseline from a mounted file, if one is given. # Mounting directly onto ~/.apollo_monitor/settings.yaml would have Docker # create the parent directory as root, leaving the client unable to write its # own state into it. if [ -n "${WATCHER_SETTINGS_FILE:-}" ]; then mkdir -p /home/dev/.apollo_monitor cp "${WATCHER_SETTINGS_FILE}" /home/dev/.apollo_monitor/settings.yaml fi # setsid and /home/dev/watcher-client.log 2>&1 & # Wait for the client before handing over: an agent that starts first races it, # and its early tool calls are neither reviewed nor recorded. Failing here is # deliberate. A container that starts the agent anyway runs it unmonitored. for _ in $(seq 1 60); do if curl -fsS "http://localhost:${PORT}/api/health" > /dev/null 2>&1; then healthy=1 break fi sleep 1 done if [ -z "${healthy:-}" ]; then echo "watcher did not become healthy; see /home/dev/watcher-client.log" >&2 exit 1 fi # Optional: expose the local UI on a second port bound to all interfaces. if [ -n "${WATCHER_PUBLISH_UI:-}" ]; then socat "TCP-LISTEN:${WATCHER_PUBLISH_UI},fork,reuseaddr,bind=0.0.0.0" \ "TCP:127.0.0.1:${PORT}" & fi exec "$@" ``` ## Authenticating the container The client has to prove who it is to your deployment. Pick by what the container is: | Option | Use it when | How | | --- | --- | --- | | Organization API key | The container is unattended, or nobody is signed in on the host. | Pass `WATCHER_API_KEY` at run time. The hooks and the client both read it from the environment. | | Terminal sign-in | A person owns the container but there is no browser in or near it. | Run `watcher login` inside the container: it prints a URL and a code, and you approve from a browser on any other device. Sessions are attributed to the person who approved. See [Signing in without a browser](headless-sign-in.md). | | Browser sign-in | A person is at the machine and the container is long-lived. | Publish the client's UI port (see below) and sign in from a browser on the host. | | Reuse the host's sign-in | The container runs on a developer machine that is already set up. | Bind-mount the host's `~/.apollo_monitor/credentials` directory, and nothing else from it. [Dev containers](devcontainer.md) has a complete template that does this. | | External credential helper | Your proxy requires renewable bearer tokens from a company CLI. | Install the helper inside the container and configure it as the container user. Keep its writable state owned by that effective UID; mismatched bind-mount ownership is rejected. See [External credential helpers](credential-helpers.md). | | Proxy headers | Your deployment authenticates through a reverse proxy. | Set the `WATCHER_HEADER_*` variables in the environment that runs `watcher hooks install`, as on any other machine. See [Sending additional headers](../self-hosting/client-deployment.md#sending-additional-headers). | Two traps with the bind-mount option: - **The URLs must match exactly.** Credentials are stored per backend hostname, so a host signed in to `https://watcher.example.com` produces nothing a container pointed at a different hostname for the same deployment can use. - **Share the credentials directory only.** The rest of `~/.apollo_monitor` assumes one client per directory: it holds which client is running and on which port, the local session and decision database, and per-deployment health state. If you mount the whole directory, the host and container clients interfere with each other. Give the container its own copy of the rest, on a volume. Sessions recorded under an API key are attributed to the organization, not to a person: the Analyzer shows no user for them. If you need to tell fleets apart, issue a key per fleet or per purpose. The coding agent needs its own credentials, separately from Watcher. Pass its API key in the environment, or mount the agent's credentials directory, as you would for any containerized agent. ## Review posture without a human An escalated tool call waits for a person to approve it. A headless agent has nowhere to show that request, so it treats the escalation as a refusal and moves on: the call is lost and nobody is asked. Never leave escalation enabled in an unattended container. Use the **Auto** posture instead: nothing escalates, and calls scoring at or above 8 out of 10 are denied outright. The agent gets a clear "auto-denied by policy" result it can act on, and dangerous calls are still stopped. `watcher profile set auto` selects it at run time; to bake it into an image, mount this file and point `WATCHER_SETTINGS_FILE` at it: ```yaml version: 1 enforcement: mode: enforce policy_monitors: on_grading_failure: allow tool_thresholds: "*": flag_threshold: 11 deny_threshold: 8 unknown: flag_threshold: 11 deny_threshold: 8 Bash: flag_threshold: 11 deny_threshold: 8 PowerShell: flag_threshold: 11 deny_threshold: 8 apply_patch: flag_threshold: 11 deny_threshold: 8 Edit: flag_threshold: 11 deny_threshold: 8 Write: flag_threshold: 11 deny_threshold: 8 NotebookEdit: flag_threshold: 11 deny_threshold: 8 WebFetch: flag_threshold: 11 deny_threshold: 8 Skill: flag_threshold: 11 deny_threshold: 8 Monitor: flag_threshold: 11 deny_threshold: 8 AskUserQuestion: flag_threshold: 11 deny_threshold: 11 EnterPlanMode: flag_threshold: 11 deny_threshold: 11 ExitPlanMode: flag_threshold: 11 deny_threshold: 11 command_rules: rules: [] ``` What the keys mean: - A `flag_threshold` of 11 never escalates and a `deny_threshold` of 11 never denies, so the two plan-mode entries and `AskUserQuestion` are always allowed. - Every tool is listed by name on purpose. A named entry takes precedence over the `"*"` glob, so setting the glob alone leaves the tools that ship with a lower flag threshold still escalating. `unknown` is a reserved key, not a tool: it covers any tool that no named entry and no glob matches, which is how a container handles tools added after the file was written. - `on_grading_failure` decides what happens when grading itself fails, for example when the grading provider is unreachable. The default is `escalate`, which is useless in an unattended container, so this file sets `allow`: ungraded calls run. Set it to `deny` instead if you would rather a grading outage stop the agent than let it work unreviewed. If you would rather watch before you block, set `enforcement.mode` to `observe`: every call is graded and recorded, and the agent's own permission rules decide what runs. Nothing is ever blocked in that mode, so treat it as a tuning step rather than a destination. The modes are described in full under [enforcement modes](../concepts/blocking-and-trailing-review.md#enforcement-modes). ### Tune the graders to what containers get wrong The thresholds decide what happens to a score; the [graders](../concepts/graders.md) decide the score. An unattended container has a different risk profile from a developer's laptop: no one notices a quietly disabled test, a scope creep into unrelated files, credentials read out of the environment, or an agent editing its own guardrails. Write those failure modes into a grader's prompt so they score high enough to hit the deny threshold, and distribute it as organization policy. The [custom grader prompt](../self-hosting/prompts/custom-grader.md) walks an agent through creating one against your deployment. ### Managed settings overwrite the file Where your organization distributes [managed client settings](../concepts/organization-settings.md#managed-client-settings) through Watcher, the client replaces `~/.apollo_monitor/settings.yaml` with the organization's document within a few minutes of starting. A posture baked into an image is therefore temporary: put anything that must survive into the organization's document instead. An API-key container can pull managed settings without browser sign-in. Give its key `org:settings:read`; `WATCHER_DISABLE_WORKOS` is not required for these pulls. ## Running it Build, then run. This example is the unattended shape: an API key, the Auto posture, and a named volume for state. ```bash docker build --platform linux/amd64 \ --build-arg WATCHER_API_URL=https://watcher.example.com \ -t agent-with-watcher . docker volume create watcher-state docker run --rm -it --platform linux/amd64 \ -v watcher-state:/home/dev/.apollo_monitor \ -v "$PWD/settings-auto.yaml:/preset/settings-auto.yaml:ro" \ -e WATCHER_SETTINGS_FILE=/preset/settings-auto.yaml \ -e WATCHER_API_KEY="$WATCHER_API_KEY" \ agent-with-watcher bash ``` Mount the settings file anywhere except inside `~/.apollo_monitor`; the entrypoint copies it into place. That example drops you in a shell, which is what you want while setting the container up. For an unattended run, replace `-it ... bash` with the agent invocation the container exists to run, for example: ```bash docker run --rm --platform linux/amd64 \ -v watcher-state:/home/dev/.apollo_monitor \ -v "$PWD/settings-auto.yaml:/preset/settings-auto.yaml:ro" \ -e WATCHER_SETTINGS_FILE=/preset/settings-auto.yaml \ -e WATCHER_API_KEY="$WATCHER_API_KEY" \ agent-with-watcher claude -p "run the test suite and fix any failures" ``` The entrypoint starts the client and waits for it before the agent command runs, so the agent's first tool call is already covered. ### Reaching the review UI The client serves its UI on port 8228 bound to `127.0.0.1`, and it accepts requests only for `localhost`. That is deliberate, and it means publishing 8228 with `-p` reaches nothing. To open the UI from the host, run the container with `-e WATCHER_PUBLISH_UI=8229 -p 8229:8229`: the entrypoint relays the published port to the loopback one, and `http://localhost:8229` on the host serves the UI, including pending approvals. Only do this where you would be willing to expose the UI: anything that can reach that port can approve calls. ## Verifying it works Inside the container: ```bash watcher doctor ``` This confirms the target, the agent hooks, and that the client is running. **It does not confirm that an API key works.** On the API-key path, doctor checks only that a key is present, so an install with a wrong or unpermissioned key reports a clean bill of health while every upload fails. Do not treat a green report as proof that recording works. Prove recording end to end instead. Run one agent turn, then check both ends: ```bash watcher logs hooks -n 5 ``` A working container logs a line per tool call, such as `[pre_tool_use] Permission evaluated: Bash -> allow`. Then open the [Analyzer](../concepts/analyzer.md) and confirm the session is there. Three failure signatures worth recognizing: | In the hooks log | Meaning | | --- | --- | | `watcher not running ... falling through to native permission flow` | The client is not running. In a container with a TTY this is usually a client started without `setsid`. Nothing is being recorded or reviewed. | | `watcher signed out ... falling through to native permission flow` | The container has nothing that authenticates to its backend target: no sign-in credentials and no API key. The hooks decide nothing, so your coding agent's own permission prompt gates each call. Nothing is being recorded or reviewed. | | `watcher auth rejected ... falling through to native permission flow` | The backend has rejected a run of the container's requests, on a deployment that authenticates through a reverse proxy or with an `x-api-key` captured into the hook headers. For a reverse proxy the fix is the gateway or the identity headers, not a sign-in; a rejected captured key needs replacing. `watcher doctor` names the specifics. The hooks decide nothing until a request is accepted, and nothing is being recorded or reviewed in the meantime. A key supplied through `WATCHER_API_KEY` is different once the client has learned from the backend that it runs WorkOS mode: the hooks stay enabled and the key's rejection surfaces as grading failures, handled by your organization's `on_grading_failure` setting rather than by this signature. Against a server too old to report its auth mode, repeated rejections disable the hooks and log this signature for the environment key too. | | `403 Service ingestion requires the 'sessions:ingest' or 'sessions:write:any' permission` | The key authenticates but cannot write sessions. For a normal Watcher container, grant `sessions:write:any` (see [Prerequisites](#prerequisites)). | ## Codex Everything above applies to Codex, with one addition. Watcher manages Codex's approval policy in `config.toml` (and in enforce mode the sandbox and who answers an approval request as well), and Codex asks you to confirm the new hooks the next time it starts, running without them until you accept. In a container that means an unattended Codex run records nothing unless the confirmation was already accepted in an image layer or in mounted state. See [Supported agents](../supported-agents.md) for what the hooks do in each agent.