Kubernetes infrastructure
The Watcher Helm chart deploys the Analyzer, Watcher API, background workers, and a database bootstrap Job into your cluster. This example uses managed PostgreSQL and proxy authentication. You provide the cluster, database, storage, and authenticated HTTPS entry point.
Download the self-host ZIP from Watcher releases
and verify its checksum.
The chart is in watcher-self-host/chart/, alongside the Compose files.
Its chart version, app version, and default image tag match the release.
The chart's values.yaml lists its settings. Run this guide's commands from
the extracted watcher-self-host/ directory.
For a single Docker host instead, use AWS infrastructure
and Setup.
Architecture
All external paths, including /api/*, go to the analyzer Service on port
80. Its pods listen on port 8080 and proxy API requests to xylon-api:8000.
Keep the API Service private. An IP-target load balancer must target the
Analyzer pod port, 8080.
The scheduler triggers trailing review.
The operation worker (xylon-operation-worker) runs background jobs: session
summaries and deep reviews, both the ones the scheduler queues and the ones
users request with Generate report in the Analyzer or through the API. It
calls the same model providers as the API. The retention worker
applies cleanup policies. The bootstrap Job migrates and
seeds the database after installation and before upgrades.
Prerequisites
- An existing cluster, Helm, and a
kubectlcontext with permission to create workloads, Secrets, Services, NetworkPolicies, and persistent volume claims. Use one Watcher release per namespace because the chart uses fixed resource names. - A managed PostgreSQL 17 or later database reachable from the pods. The operation worker refuses older servers. Give its user ownership of an empty database for the first install. The chart's bundled PostgreSQL is optional and intended for evaluation.
- A StorageClass that creates volumes. This example enables the
admin audit log. It uses separate API and
retention-worker claims. Both start with
ReadWriteOnce. - A network plugin that enforces NetworkPolicy. Creating policy objects alone does not prove enforcement.
- Registry access for the
xylon-api,bootstrap, andanalyzerimages, including any pull Secret in the release namespace. For private ECR pulls, set up token renewal before installing. ECR login tokens expire after 12 hours. Internal mirrors are another option. The chart does not create a token refresh job. - An identity-aware proxy and HTTPS hostname. The proxy must authenticate
requests and strip client-supplied
X-Forwarded-UserandX-Forwarded-User-Roleheaders before adding trusted values. It must be the only reachable path into the Analyzer, including from other cluster workloads. The chart's NetworkPolicy protects the API, not the Analyzer. - LLM access from the API and operation-worker pods for all default managed monitors: Anthropic for Gateway and Deep Review, and OpenAI for Triage and session summaries. This example uses keys for both providers. See Which LLM providers are called for the required model names.
For EKS, provide a CSI driver and StorageClass for the audit volumes. Your network plugin must enforce NetworkPolicy. Your security groups must allow traffic from pods to the database. The EKS infrastructure checklist covers these requirements, ALB routing and registry token renewal. Provision them with your own infrastructure tooling; the chart does not include EKS Terraform.
1. Select the cluster and prepare secrets
Set the context and local chart directory:
WATCHER_CONTEXT=your-cluster-context
WATCHER_NAMESPACE=watcher
WATCHER_CHART=./chart
kubectl --context "$WATCHER_CONTEXT" create namespace "$WATCHER_NAMESPACE"
Skip namespace creation if it already exists. Use your secret manager to
create an Opaque Secret named watcher-external in that namespace:
| Secret key | Value for this example |
|---|---|
DATABASE_URL | postgresql+asyncpg://USER:PASSWORD@HOST:PORT/DBNAME; URL-encode special characters in credentials. |
ANTHROPIC_API_KEY | Access to the Gateway and Deep Review models. |
OPENAI_API_KEY | Access to the Triage model. |
WATCHER_INTERNAL_AUTH_TOKEN | A random shared secret used by the scheduler to authenticate to the API. |
Keep secret values out of watcher-values.yaml and shell arguments.
Rendering the chart does not check the contents of an existing Secret.
Create the Secret and image-pull credentials before installing.
The API and operation-worker pods can start without the OpenAI key, but their
default monitors still need access to OpenAI. Add GOOGLE_API_KEY to the
Secret only for graders on a google/ model: a monitor you override, a custom
grader, or the built-in Policy Triage grader, which Watcher clients older than
0.41 use to grade every gated tool call. Changing gradingModel does
not override the models chosen by managed monitors.
2. Configure the Helm release
Save this as watcher-values.yaml, replacing the placeholder values:
image:
tag: "<matching-watcher-release-tag>"
pullSecrets:
- watcher-registry
publicUrl: "https://watcher.example.com"
organization:
id: "<stable-organization-id>"
name: "Example organization"
auth:
mode: proxy
existingSecret: watcher-external
postgresql:
enabled: false
api:
replicas: 1
asyncGrading:
enabled: true
auditLog:
storageClassName: "<your-storage-class>"
networkPolicy:
enabled: true
ingress:
enabled: false
Set image.pullSecrets to your pull Secret names. Use [] if the nodes
already have registry access. For a mirror, set image.registry to the full
prefix of the three image repositories, such as
registry.example.com/watcher-self-host. The chart adds /xylon-api,
/bootstrap, or /analyzer and the tag to that prefix.
Choose organization.id once and keep it across upgrades. Set publicUrl
to the external HTTPS URL users and clients will reach.
This example disables the chart's Ingress. Set up your own authenticated
ingress to send all traffic to analyzer:80. Keep the original host and
forwarded HTTPS scheme. Client request timeouts must cover the server's
grading budget. Set the load-balancer idle timeout above the client timeouts.
Check https://watcher.example.com/api/v1/health using your proxy's
authentication.
To use WorkOS authentication, set auth.mode: workos
and auth.workos.orgId. Add WATCHER_WORKOS_CLIENT_ID and
WATCHER_WORKOS_API_KEY to the Secret. The API then authenticates users
through WorkOS. The chart's ingress values can set up routing and TLS with
your existing ingress controller. Match its host and TLS settings to
publicUrl. It still routes every path through the Analyzer.
Health probes
Configure probes in your Helm values so GitOps retains the settings across
syncs and upgrades. Each value accepts the standard Kubernetes probe fields.
Partial overrides keep the other chart defaults. Lists are replaced, so
include an accepted Host header when overriding an API probe's httpHeaders.
| Component | Probe values |
|---|---|
| API | api.readinessProbe, api.livenessProbe |
| Analyzer | analyzer.readinessProbe |
| Grading scheduler | asyncGrading.livenessProbe |
| Bundled PostgreSQL | postgresql.readinessProbe |
| Optional gateway | litellm.readinessProbe, litellm.livenessProbe |
API probes send Host: 127.0.0.1 while connecting to the pod IP. This header
passes the API's host validation, including when the cluster uses pod IPs
from 100.64.0.0/10. Keep the header on both probes. It does not select a
connection address or require a DNS lookup.
For example, allow more time for each API health request:
api:
readinessProbe:
timeoutSeconds: 5
failureThreshold: 6
livenessProbe:
timeoutSeconds: 5
failureThreshold: 6
Set an individual probe to null to disable it. An empty map keeps the
defaults. To switch probe type, remove the default handler with null as
you add the new handler, such as httpGet: null alongside exec.
With Istio probe rewriting enabled, inspect both the Deployment and the
injected Pod. The sidecar records the original probes in
ISTIO_KUBE_APP_PROBERS; check the original headers for both readiness and
liveness there.
Server configuration
Use the same server-config.yaml as a Docker Compose installation for
role permissions, model pricing, and monitor
configuration. Add --set-file serverConfig=server-config.yaml to the Helm
lint, template, and install commands below, and keep it on subsequent upgrades.
The chart stores the file in a ConfigMap, mounts it read-only at
/etc/watcher/server-config.yaml in the API and operation worker, and restarts
both when its contents change or the file is removed. Keep credentials in the
Secret. Leave serverConfig unset to use the shipped defaults.
To apply an updated file to an existing release:
helm upgrade watcher "$WATCHER_CHART" \
--kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE" \
-f watcher-values.yaml --set-file serverConfig=server-config.yaml --timeout 10m
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-api --timeout=300s
# If the operation worker is enabled (the default):
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-operation-worker --timeout=300s
For a server upgrade, follow the target release's required steps in Update Watcher.
Managed client settings from the release
The chart can also supply the organization's
managed client settings
document, so no admin can edit or stop its distribution from the Analyzer.
Set managedSettings.existingConfigMap to a ConfigMap that holds the document
under the key managed-settings.yaml, or set managedSettings.yaml to the
document itself. Set one, not both. See
Supplying the document from your deployment
for how the API loads and reloads it.
Optional native Copilot gateway
To send native Copilot traffic through Watcher, follow
Native Copilot through LiteLLM. The optional
litellm values add two gateway replicas and a separate migration Job using
your image, database and identity mapping. This is separate from routing
Watcher's own grading requests through a model gateway below.
Model gateways
To use an Anthropic-compatible gateway that authenticates through your network or workload identity, add:
modelGateway:
anthropicBaseUrl: "https://models.example.com"
The Anthropic base URL has no /v1 suffix. For OpenAI-compatible calls,
modelGateway.openaiBaseUrl includes /v1, for example
https://models.example.com/v1. Each URL routes calls for that provider only.
With the Anthropic gateway above, keep OPENAI_API_KEY for Triage and
session summaries. If you also set modelGateway.openaiBaseUrl, you can omit
the OpenAI key under the same gateway rules. These settings do not route
Google calls.
With existingSecret, the chart sends the provider key unused to each
configured gateway. You can omit that provider key from the Secret.
Adding a real key to the Secret does not change this behavior. If your gateway
needs a real provider key, you must adapt this example. Review the chart's
modelGateway settings first. Set up the gateway and its authentication
outside the chart.
3. Render and install
Inspect the resources before applying them:
helm lint "$WATCHER_CHART" -f watcher-values.yaml
helm template watcher "$WATCHER_CHART" \
--namespace "$WATCHER_NAMESPACE" -f watcher-values.yaml > watcher-rendered.yaml
helm install watcher "$WATCHER_CHART" \
--kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE" \
-f watcher-values.yaml --timeout 10m
The install runs bootstrap as a post-install hook. Helm deletes the
bootstrap Job when it succeeds. If the install fails, read the Job's logs.
Fix the reported database, image-pull, or settings error before continuing:
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" logs job/watcher-bootstrap
4. Verify before connecting developers
Check the release, deployments, and claims:
helm status watcher --kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE"
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" get pods,pvc
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-api --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/analyzer --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-grading-scheduler --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-retention-cleanup --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
rollout status deployment/xylon-operation-worker --timeout=300s
Both audit claims must be Bound. A Pending claim usually needs a working
provisioner or a corrected StorageClass. Confirm HTTP 200 from
/api/v1/health through your authenticated HTTPS entry point. Check that
unauthenticated requests cannot access session data by supplying forged
identity headers.
Test the API NetworkPolicy with a pod that has none of Watcher's allowed
labels. Set PROBE_IMAGE to an approved image containing curl, reachable
from the cluster:
PROBE_IMAGE=your-registry.example.com/curl:approved-tag
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
run watcher-network-probe --rm -i --restart=Never --image="$PROBE_IMAGE" -- \
curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \
http://xylon-api:8000/api/v1/health
Expect a connection timeout and HTTP code 000, while the authenticated
entry point still works. Check the curl error: a DNS failure also returns
000. A DNS error, image-pull failure, or HTTP 404 does not prove that the
policy blocks traffic.
Finally, follow Client deployment to connect a
Watcher client to publicUrl. Run a coding-agent
session and confirm it appears
with grades in the Analyzer. This tests both recording and model completion
access. A health check alone does not test grading.
Scale and operate
Install metrics-server before enabling API autoscaling. To keep auditing with multiple API pods:
- Choose a StorageClass that supports
ReadWriteManyfor API audit logs. It must also supportReadWriteOncefor the retention worker's claim. - Set
auditLog.accessModes: [ReadWriteMany]before enablingapi.autoscaling.enabledor increasingapi.replicas.
Each API pod writes its own audit file. Read all of these files for the API
audit trail. See values.yaml for replica bounds and the CPU target.
CPU usage is measured against the API container's CPU request.
Setting adminAuditLogFile: "" disables auditing and removes the claims
from the rendered resources. Helm keeps existing audit claims. Changing a
claim's access modes requires a new claim. Save the data before replacing or
deleting any claim.
With the default single-replica audit storage, API upgrades use Recreate.
Service stops while the new pod starts.
Back up managed PostgreSQL, both audit volumes, and your deployment
configuration. Collect container logs with your cluster's logging system.
After rotating watcher-external, restart the API, scheduler, operation
worker, and retention deployments to load the new values:
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" rollout restart \
deployment/xylon-api deployment/xylon-grading-scheduler \
deployment/xylon-operation-worker deployment/xylon-retention-cleanup
For upgrades, use the target chart with its matching image tag and follow
Update Watcher. The chart runs bootstrap as a
pre-upgrade hook. It must complete before workloads update. If you use
GitOps, check that your controller follows this order. Follow
CI/CD and Kubernetes/GitOps
for deployment automation, including database recovery requirements.