---
description: >-
  Deploy self-hosted Watcher into an existing Kubernetes cluster with a Helm example, architecture diagram, and verification steps.
---

> Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt
> Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md
> Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20Kubernetes%20infrastructure&body=Page%3A%20Kubernetes%20infrastructure%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fself-hosting%2Fkubernetes-infrastructure%2F%0A%0A

# Kubernetes infrastructure

The Watcher Helm chart deploys the [Analyzer](../concepts/analyzer.md),
[Watcher API](../api/index.md), background workers, and a database bootstrap
Job into your cluster. This example uses managed PostgreSQL and proxy
authentication. You provide the cluster, database, storage, and authenticated
HTTPS entry point.

The chart comes separately from the standard self-host release bundle.
You need the chart and its matching Watcher image tag to follow this page.
The chart's `values.yaml` lists its settings.
For a single Docker host instead, use [AWS infrastructure](aws-infrastructure.md)
and [Setup](setup.md).

## Architecture

![Watcher on Kubernetes with proxy authentication. Developer clients and browsers connect through a TLS-terminating identity-aware proxy to the Analyzer service on port 80, whose pods listen on 8080. The Analyzer forwards API requests to the internal API service on port 8000. The API calls LLM providers. The API, grading scheduler, retention worker, and bootstrap Job access managed PostgreSQL. Separate persistent volume claims hold the API and retention audit logs. Cluster nodes pull the release images from a registry.](/watcher-kubernetes-architecture.svg)

All external paths, including `/api/*`, go to the `analyzer` Service on port
80. Its pods listen on port 8080 and proxy API requests to `xylon-api:8000`.
Keep the API Service private. An IP-target load balancer must target the
Analyzer pod port, 8080.

The scheduler triggers [trailing review](../concepts/blocking-and-trailing-review.md).
The retention worker applies cleanup policies. The bootstrap Job migrates and
seeds the database after installation and before upgrades.

## Prerequisites

- An existing cluster, Helm, and a `kubectl` context with permission to create
  workloads, Secrets, Services, NetworkPolicies, and persistent volume claims.
  Use one Watcher release per namespace because the chart uses fixed resource
  names.
- A managed PostgreSQL database reachable from the pods. Give its user
  ownership of an empty database for the first install. The chart's bundled
  PostgreSQL is optional and intended for evaluation.
- A StorageClass that creates volumes. This example enables the
  [admin audit log](operations.md#admin-audit-log). It uses separate API and
  retention-worker claims. Both start with `ReadWriteOnce`.
- A network plugin that enforces NetworkPolicy. Creating policy objects alone
  does not prove enforcement.
- Registry access for the `xylon-api`, `bootstrap`, and `analyzer` images,
  including any pull Secret in the release namespace. For private ECR pulls,
  set up token renewal before installing. ECR login tokens expire after
  12 hours. [Internal mirrors](supply-chain.md#use-internal-mirrors) are another
  option. The chart does not create a token refresh job.
- An identity-aware proxy and HTTPS hostname. The proxy must authenticate
  requests and strip client-supplied `X-Forwarded-User` and
  `X-Forwarded-User-Role` headers before adding trusted values. It must be the
  only reachable path into the Analyzer, including from other cluster
  workloads. The chart's NetworkPolicy protects the API, not the Analyzer.
- LLM access from the API pods for all default
  [managed monitors](../concepts/graders.md#monitors): Anthropic for Gateway
  and Deep Review, OpenAI for Triage, and Google for session summaries.
  This example uses keys for all three providers. See
  [Which LLM providers are called](security.md#which-llm-providers-are-called)
  for the required model names.

For EKS, provide a CSI driver and StorageClass for the audit volumes.
Your network plugin must enforce NetworkPolicy. Your security groups must
allow traffic from pods to the database. The [EKS reference](eks-reference.md)
covers these steps, an existing ALB, and registry token renewal. It uses the
Terraform files in the Kubernetes deployment bundle.

## 1. Select the cluster and prepare secrets

Set the context and local chart directory:

```sh
WATCHER_CONTEXT=your-cluster-context
WATCHER_NAMESPACE=watcher
WATCHER_CHART=./chart
kubectl --context "$WATCHER_CONTEXT" create namespace "$WATCHER_NAMESPACE"
```

Skip namespace creation if it already exists. Use your secret manager to
create an Opaque Secret named `watcher-external` in that namespace:

| Secret key | Value for this example |
| --- | --- |
| `DATABASE_URL` | `postgresql+asyncpg://USER:PASSWORD@HOST:PORT/DBNAME`; URL-encode special characters in credentials. |
| `ANTHROPIC_API_KEY` | Access to the Gateway and Deep Review models. |
| `OPENAI_API_KEY` | Access to the Triage model. |
| `GOOGLE_API_KEY` | Access to the session-summary model. |
| `WATCHER_INTERNAL_AUTH_TOKEN` | A random shared secret used by the scheduler to authenticate to the API. |

Keep secret values out of `watcher-values.yaml` and shell arguments.
Rendering the chart does not check the contents of an existing Secret.
Create the Secret and image-pull credentials before installing.

The API pods can start without the OpenAI or Google keys, but their default
monitors still need access to those providers. Changing `gradingModel` does
not override the models chosen by managed monitors.

## 2. Configure the Helm release

Save this as `watcher-values.yaml`, replacing the placeholder values:

```yaml
image:
  tag: "<matching-watcher-release-tag>"
  pullSecrets:
    - watcher-registry
publicUrl: "https://watcher.example.com"
organization:
  id: "<stable-organization-id>"
  name: "Example organization"
auth:
  mode: proxy
existingSecret: watcher-external
postgresql:
  enabled: false
api:
  replicas: 1
asyncGrading:
  enabled: true
auditLog:
  storageClassName: "<your-storage-class>"
networkPolicy:
  enabled: true
ingress:
  enabled: false
```

Set `image.pullSecrets` to your pull Secret names. Use `[]` if the nodes
already have registry access. For a mirror, set `image.registry` to the full
prefix of the three image repositories, such as
`registry.example.com/watcher-self-host`. The chart adds `/xylon-api`,
`/bootstrap`, or `/analyzer` and the tag to that prefix.

Choose `organization.id` once and keep it across upgrades. Set `publicUrl`
to the external HTTPS URL users and clients will reach.

This example disables the chart's Ingress. Set up your own authenticated
ingress to send all traffic to `analyzer:80`. Keep the original host and
forwarded HTTPS scheme. Client request timeouts must cover the server's
grading budget. Set the load-balancer idle timeout above the client timeouts.
Check `https://watcher.example.com/api/v1/health` using your proxy's
authentication.

To use [WorkOS authentication](setup.md#sso-mode), set `auth.mode: workos`
and `auth.workos.orgId`. Add `WATCHER_WORKOS_CLIENT_ID` and
`WATCHER_WORKOS_API_KEY` to the Secret. The API then authenticates users
through WorkOS. The chart's `ingress` values can set up routing and TLS with
your existing ingress controller. Match its host and TLS settings to
`publicUrl`. It still routes every path through the Analyzer.

### Model gateways

To use an Anthropic-compatible gateway that authenticates through your
network or workload identity, add:

```yaml
modelGateway:
  anthropicBaseUrl: "https://models.example.com"
```

The Anthropic base URL has no `/v1` suffix. For OpenAI-compatible calls,
`modelGateway.openaiBaseUrl` includes `/v1`, for example
`https://models.example.com/v1`. Each URL routes calls for that provider only.
With the Anthropic gateway above, keep `OPENAI_API_KEY` for Triage and
`GOOGLE_API_KEY` for summaries. If you also set `modelGateway.openaiBaseUrl`,
you can omit the OpenAI key under the same gateway rules. These settings do
not route Google calls; this example still needs `GOOGLE_API_KEY`.

With `existingSecret`, the chart sends the provider key `unused` to each
configured gateway. You can omit that provider key from the Secret.
Adding a real key to the Secret does not change this behavior. If your gateway
needs a real provider key, you must adapt this example. Review the chart's
`modelGateway` settings first. Set up the gateway and its authentication
outside the chart.

## 3. Render and install

Inspect the resources before applying them:

```sh
helm lint "$WATCHER_CHART" -f watcher-values.yaml
helm template watcher "$WATCHER_CHART" \
  --namespace "$WATCHER_NAMESPACE" -f watcher-values.yaml > watcher-rendered.yaml
helm install watcher "$WATCHER_CHART" \
  --kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE" \
  -f watcher-values.yaml --timeout 10m
```

The install runs bootstrap as a `post-install` hook. Helm deletes the
bootstrap Job when it succeeds. If the install fails, read the Job's logs.
Fix the reported database, image-pull, or settings error before continuing:

```sh
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" logs job/watcher-bootstrap
```

## 4. Verify before connecting developers

Check the release, deployments, and claims:

```sh
helm status watcher --kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE"
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" get pods,pvc
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/xylon-api --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/analyzer --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/xylon-grading-scheduler --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/xylon-retention-cleanup --timeout=300s
```

Both audit claims must be `Bound`. A Pending claim usually needs a working
provisioner or a corrected StorageClass. Confirm HTTP 200 from
`/api/v1/health` through your authenticated HTTPS entry point. Check that
unauthenticated requests cannot access session data by supplying forged
identity headers.

Test the API NetworkPolicy with a pod that has none of Watcher's allowed
labels. Set `PROBE_IMAGE` to an approved image containing `curl`, reachable
from the cluster:

```sh
PROBE_IMAGE=your-registry.example.com/curl:approved-tag
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  run watcher-network-probe --rm -i --restart=Never --image="$PROBE_IMAGE" -- \
  curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \
  http://xylon-api:8000/api/v1/health
```

Expect a connection timeout and HTTP code `000`, while the authenticated
entry point still works. Check the curl error: a DNS failure also returns
`000`. A DNS error, image-pull failure, or HTTP 404 does not prove that the
policy blocks traffic.

Finally, follow [Client deployment](client-deployment.md) to connect a
[Watcher client](../client/index.md) to `publicUrl`. Run a coding-agent
[session](../concepts/sessions-and-trajectories.md) and confirm it appears
with grades in the Analyzer. This tests both recording and model completion
access. A health check alone does not test grading.

## Scale and operate

Install metrics-server before enabling API autoscaling. To keep auditing
with multiple API pods:

1. Choose a StorageClass that supports `ReadWriteMany` for API audit logs.
   It must also support `ReadWriteOnce` for the retention worker's claim.
2. Set `auditLog.accessModes: [ReadWriteMany]` before enabling
   `api.autoscaling.enabled` or increasing `api.replicas`.

Each API pod writes its own audit file. Read all of these files for the API
audit trail. See `values.yaml` for replica bounds and the CPU target.
CPU usage is measured against the API container's CPU request.

Setting `adminAuditLogFile: ""` disables auditing and removes the claims
from the rendered resources. Helm keeps existing audit claims. Changing a
claim's access modes requires a new claim. Save the data before replacing or
deleting any claim.

With the default single-replica audit storage, API upgrades use `Recreate`.
Service stops while the new pod starts.

Back up managed PostgreSQL, both audit volumes, and your deployment
configuration. Collect container logs with your cluster's logging system.
After rotating `watcher-external`, restart the API, scheduler, and retention
deployments to load the new values:

```sh
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" rollout restart \
  deployment/xylon-api deployment/xylon-grading-scheduler deployment/xylon-retention-cleanup
```

For upgrades, use the target chart with its matching image tag and follow
[Update Watcher](updates.md#update-the-server). The chart runs bootstrap as a
`pre-upgrade` hook. It must complete before workloads update. If you use
GitOps, check that your controller follows this order. Follow
[CI/CD and Kubernetes/GitOps](automatic-updates.md#cicd-and-kubernetesgitops)
for deployment automation, including database recovery requirements.