Skip to main content
Working with an agent? Give them a link to this page as markdown.

Kubernetes infrastructure

The Watcher Helm chart deploys the Analyzer, Watcher API, background workers, and a database bootstrap Job into your cluster. This example uses managed PostgreSQL and proxy authentication. You provide the cluster, database, storage, and authenticated HTTPS entry point.

Download the self-host ZIP from Watcher releases and verify its checksum. The chart is in watcher-self-host/chart/, alongside the Compose files. Its chart version, app version, and default image tag match the release. The chart's values.yaml lists its settings. Run this guide's commands from the extracted watcher-self-host/ directory. For a single Docker host instead, use AWS infrastructure and Setup.

Architecture​

Watcher on Kubernetes with proxy authentication. Developer clients and browsers connect through a TLS-terminating identity-aware proxy to the Analyzer service on port 80, whose pods listen on 8080. The Analyzer forwards API requests to the internal API service on port 8000. The API and the operation worker call LLM providers. The API, grading scheduler, operation worker, retention worker, and bootstrap Job access managed PostgreSQL. Separate persistent volume claims hold the API and retention audit logs. Cluster nodes pull the release images from a registry.

All external paths, including /api/*, go to the analyzer Service on port 80. Its pods listen on port 8080 and proxy API requests to xylon-api:8000. Keep the API Service private. An IP-target load balancer must target the Analyzer pod port, 8080.

The scheduler triggers trailing review. The operation worker (xylon-operation-worker) runs background jobs: session summaries and deep reviews, both the ones the scheduler queues and the ones users request with Generate report in the Analyzer or through the API. It calls the same model providers as the API. The retention worker applies cleanup policies. The bootstrap Job migrates and seeds the database after installation and before upgrades.

Prerequisites​

  • An existing cluster, Helm, and a kubectl context with permission to create workloads, Secrets, Services, NetworkPolicies, and persistent volume claims. Use one Watcher release per namespace because the chart uses fixed resource names.
  • A managed PostgreSQL 17 or later database reachable from the pods. The operation worker refuses older servers. Give its user ownership of an empty database for the first install. The chart's bundled PostgreSQL is optional and intended for evaluation.
  • A StorageClass that creates volumes. This example enables the admin audit log. It uses separate API and retention-worker claims. Both start with ReadWriteOnce.
  • A network plugin that enforces NetworkPolicy. Creating policy objects alone does not prove enforcement.
  • Registry access for the xylon-api, bootstrap, and analyzer images, including any pull Secret in the release namespace. For private ECR pulls, set up token renewal before installing. ECR login tokens expire after 12 hours. Internal mirrors are another option. The chart does not create a token refresh job.
  • An identity-aware proxy and HTTPS hostname. The proxy must authenticate requests and strip client-supplied X-Forwarded-User and X-Forwarded-User-Role headers before adding trusted values. It must be the only reachable path into the Analyzer, including from other cluster workloads. The chart's NetworkPolicy protects the API, not the Analyzer.
  • LLM access from the API and operation-worker pods for all default managed monitors: Anthropic for Gateway and Deep Review, and OpenAI for Triage and session summaries. This example uses keys for both providers. See Which LLM providers are called for the required model names.

For EKS, provide a CSI driver and StorageClass for the audit volumes. Your network plugin must enforce NetworkPolicy. Your security groups must allow traffic from pods to the database. The EKS infrastructure checklist covers these requirements, ALB routing and registry token renewal. Provision them with your own infrastructure tooling; the chart does not include EKS Terraform.

1. Select the cluster and prepare secrets​

Set the context and local chart directory:

WATCHER_CONTEXT=your-cluster-context
WATCHER_NAMESPACE=watcher
WATCHER_CHART=./chart
kubectl --context "$WATCHER_CONTEXT" create namespace "$WATCHER_NAMESPACE"

Skip namespace creation if it already exists. Use your secret manager to create an Opaque Secret named watcher-external in that namespace:

Secret keyValue for this example
DATABASE_URLpostgresql+asyncpg://USER:PASSWORD@HOST:PORT/DBNAME; URL-encode special characters in credentials.
ANTHROPIC_API_KEYAccess to the Gateway and Deep Review models.
OPENAI_API_KEYAccess to the Triage model.
WATCHER_INTERNAL_AUTH_TOKENA random shared secret used by the scheduler to authenticate to the API.

Keep secret values out of watcher-values.yaml and shell arguments. Rendering the chart does not check the contents of an existing Secret. Create the Secret and image-pull credentials before installing.

The API and operation-worker pods can start without the OpenAI key, but their default monitors still need access to OpenAI. Add GOOGLE_API_KEY to the Secret only for graders on a google/ model: a monitor you override, a custom grader, or the built-in Policy Triage grader, which Watcher clients older than 0.41 use to grade every gated tool call. Changing gradingModel does not override the models chosen by managed monitors.

2. Configure the Helm release​

Save this as watcher-values.yaml, replacing the placeholder values:

image:
tag: "<matching-watcher-release-tag>"
pullSecrets:
- watcher-registry
publicUrl: "https://watcher.example.com"
organization:
id: "<stable-organization-id>"
name: "Example organization"
auth:
mode: proxy
existingSecret: watcher-external
postgresql:
enabled: false
api:
replicas: 1
asyncGrading:
enabled: true
auditLog:
storageClassName: "<your-storage-class>"
networkPolicy:
enabled: true
ingress:
enabled: false

Set image.pullSecrets to your pull Secret names. Use [] if the nodes already have registry access. For a mirror, set image.registry to the full prefix of the three image repositories, such as registry.example.com/watcher-self-host. The chart adds /xylon-api, /bootstrap, or /analyzer and the tag to that prefix.

Choose organization.id once and keep it across upgrades. Set publicUrl to the external HTTPS URL users and clients will reach.

This example disables the chart's Ingress. Set up your own authenticated ingress to send all traffic to analyzer:80. Keep the original host and forwarded HTTPS scheme. Client request timeouts must cover the server's grading budget. Set the load-balancer idle timeout above the client timeouts. Check https://watcher.example.com/api/v1/health using your proxy's authentication.

To use WorkOS authentication, set auth.mode: workos and auth.workos.orgId. Add WATCHER_WORKOS_CLIENT_ID and WATCHER_WORKOS_API_KEY to the Secret. The API then authenticates users through WorkOS. The chart's ingress values can set up routing and TLS with your existing ingress controller. Match its host and TLS settings to publicUrl. It still routes every path through the Analyzer.

Health probes​

Configure probes in your Helm values so GitOps retains the settings across syncs and upgrades. Each value accepts the standard Kubernetes probe fields. Partial overrides keep the other chart defaults. Lists are replaced, so include an accepted Host header when overriding an API probe's httpHeaders.

ComponentProbe values
APIapi.readinessProbe, api.livenessProbe
Analyzeranalyzer.readinessProbe
Grading schedulerasyncGrading.livenessProbe
Bundled PostgreSQLpostgresql.readinessProbe
Optional gatewaylitellm.readinessProbe, litellm.livenessProbe

API probes send Host: 127.0.0.1 while connecting to the pod IP. This header passes the API's host validation, including when the cluster uses pod IPs from 100.64.0.0/10. Keep the header on both probes. It does not select a connection address or require a DNS lookup.

For example, allow more time for each API health request:

api:
readinessProbe:
timeoutSeconds: 5
failureThreshold: 6
livenessProbe:
timeoutSeconds: 5
failureThreshold: 6

Set an individual probe to null to disable it. An empty map keeps the defaults. To switch probe type, remove the default handler with null as you add the new handler, such as httpGet: null alongside exec.

With Istio probe rewriting enabled, inspect both the Deployment and the injected Pod. The sidecar records the original probes in ISTIO_KUBE_APP_PROBERS; check the original headers for both readiness and liveness there.

Server configuration​

Use the same server-config.yaml as a Docker Compose installation for role permissions, model pricing, and monitor configuration. Add --set-file serverConfig=server-config.yaml to the Helm lint, template, and install commands below, and keep it on subsequent upgrades. The chart stores the file in a ConfigMap, mounts it read-only at /etc/watcher/server-config.yaml in the API and operation worker, and restarts both when its contents change or the file is removed. Keep credentials in the Secret. Leave serverConfig unset to use the shipped defaults.

To apply an updated file to an existing release:

helm upgrade watcher "$WATCHER_CHART" \
--kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE" \
-f watcher-values.yaml --set-file serverConfig=server-config.yaml --timeout 10m
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-api --timeout=300s
# If the operation worker is enabled (the default):
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-operation-worker --timeout=300s

For a server upgrade, follow the target release's required steps in Update Watcher.

Managed client settings from the release​

The chart can also supply the organization's managed client settings document, so no admin can edit or stop its distribution from the Analyzer. Set managedSettings.existingConfigMap to a ConfigMap that holds the document under the key managed-settings.yaml, or set managedSettings.yaml to the document itself. Set one, not both. See Supplying the document from your deployment for how the API loads and reloads it.

Optional native Copilot gateway​

To send native Copilot traffic through Watcher, follow Native Copilot through LiteLLM. The optional litellm values add two gateway replicas and a separate migration Job using your image, database and identity mapping. This is separate from routing Watcher's own grading requests through a model gateway below.

Model gateways​

To use an Anthropic-compatible gateway that authenticates through your network or workload identity, add:

modelGateway:
anthropicBaseUrl: "https://models.example.com"

The Anthropic base URL has no /v1 suffix. For OpenAI-compatible calls, modelGateway.openaiBaseUrl includes /v1, for example https://models.example.com/v1. Each URL routes calls for that provider only. With the Anthropic gateway above, keep OPENAI_API_KEY for Triage and session summaries. If you also set modelGateway.openaiBaseUrl, you can omit the OpenAI key under the same gateway rules. These settings do not route Google calls.

With existingSecret, the chart sends the provider key unused to each configured gateway. You can omit that provider key from the Secret. Adding a real key to the Secret does not change this behavior. If your gateway needs a real provider key, you must adapt this example. Review the chart's modelGateway settings first. Set up the gateway and its authentication outside the chart.

3. Render and install​

Inspect the resources before applying them:

helm lint "$WATCHER_CHART" -f watcher-values.yaml
helm template watcher "$WATCHER_CHART" \
--namespace "$WATCHER_NAMESPACE" -f watcher-values.yaml > watcher-rendered.yaml
helm install watcher "$WATCHER_CHART" \
--kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE" \
-f watcher-values.yaml --timeout 10m

The install runs bootstrap as a post-install hook. Helm deletes the bootstrap Job when it succeeds. If the install fails, read the Job's logs. Fix the reported database, image-pull, or settings error before continuing:

kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" logs job/watcher-bootstrap

4. Verify before connecting developers​

Check the release, deployments, and claims:

helm status watcher --kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE"
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" get pods,pvc
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-api --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/analyzer --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-grading-scheduler --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-retention-cleanup --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-operation-worker --timeout=300s

Both audit claims must be Bound. A Pending claim usually needs a working provisioner or a corrected StorageClass. Confirm HTTP 200 from /api/v1/health through your authenticated HTTPS entry point. Check that unauthenticated requests cannot access session data by supplying forged identity headers.

Test the API NetworkPolicy with a pod that has none of Watcher's allowed labels. Set PROBE_IMAGE to an approved image containing curl, reachable from the cluster:

PROBE_IMAGE=your-registry.example.com/curl:approved-tag
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
run watcher-network-probe --rm -i --restart=Never --image="$PROBE_IMAGE" -- \
curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \
http://xylon-api:8000/api/v1/health

Expect a connection timeout and HTTP code 000, while the authenticated entry point still works. Check the curl error: a DNS failure also returns 000. A DNS error, image-pull failure, or HTTP 404 does not prove that the policy blocks traffic.

Finally, follow Client deployment to connect a Watcher client to publicUrl. Run a coding-agent session and confirm it appears with grades in the Analyzer. This tests both recording and model completion access. A health check alone does not test grading.

Scale and operate​

Install metrics-server before enabling API autoscaling. To keep auditing with multiple API pods:

  1. Choose a StorageClass that supports ReadWriteMany for API audit logs. It must also support ReadWriteOnce for the retention worker's claim.
  2. Set auditLog.accessModes: [ReadWriteMany] before enabling api.autoscaling.enabled or increasing api.replicas.

Each API pod writes its own audit file. Read all of these files for the API audit trail. See values.yaml for replica bounds and the CPU target. CPU usage is measured against the API container's CPU request.

Setting adminAuditLogFile: "" disables auditing and removes the claims from the rendered resources. Helm keeps existing audit claims. Changing a claim's access modes requires a new claim. Save the data before replacing or deleting any claim.

With the default single-replica audit storage, API upgrades use Recreate. Service stops while the new pod starts.

Back up managed PostgreSQL, both audit volumes, and your deployment configuration. Collect container logs with your cluster's logging system. After rotating watcher-external, restart the API, scheduler, operation worker, and retention deployments to load the new values:

kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" rollout restart \
deployment/xylon-api deployment/xylon-grading-scheduler \
deployment/xylon-operation-worker deployment/xylon-retention-cleanup

For upgrades, use the target chart with its matching image tag and follow Update Watcher. The chart runs bootstrap as a pre-upgrade hook. It must complete before workloads update. If you use GitOps, check that your controller follows this order. Follow CI/CD and Kubernetes/GitOps for deployment automation, including database recovery requirements.