Kubernetes infrastructure

The Watcher Helm chart deploys the Analyzer, Watcher API, background workers, and a database bootstrap Job into your cluster. This example uses managed PostgreSQL and proxy authentication. You provide the cluster, database, storage, and authenticated HTTPS entry point.

The chart comes separately from the standard self-host release bundle. You need the chart and its matching Watcher image tag to follow this page. The chart's values.yaml lists its settings. For a single Docker host instead, use AWS infrastructure and Setup.

Architecture

All external paths, including /api/*, go to the analyzer Service on port 80. Its pods listen on port 8080 and proxy API requests to xylon-api:8000. Keep the API Service private. An IP-target load balancer must target the Analyzer pod port, 8080.

The scheduler triggers trailing review. The retention worker applies cleanup policies. The bootstrap Job migrates and seeds the database after installation and before upgrades.

Prerequisites

• An existing cluster, Helm, and a kubectl context with permission to create workloads, Secrets, Services, NetworkPolicies, and persistent volume claims. Use one Watcher release per namespace because the chart uses fixed resource names.
• A managed PostgreSQL database reachable from the pods. Give its user ownership of an empty database for the first install. The chart's bundled PostgreSQL is optional and intended for evaluation.
• A StorageClass that creates volumes. This example enables the admin audit log. It uses separate API and retention-worker claims. Both start with ReadWriteOnce.
• A network plugin that enforces NetworkPolicy. Creating policy objects alone does not prove enforcement.
• Registry access for the xylon-api, bootstrap, and analyzer images, including any pull Secret in the release namespace. For private ECR pulls, set up token renewal before installing. ECR login tokens expire after 12 hours. Internal mirrors are another option. The chart does not create a token refresh job.
• An identity-aware proxy and HTTPS hostname. The proxy must authenticate requests and strip client-supplied X-Forwarded-User and X-Forwarded-User-Role headers before adding trusted values. It must be the only reachable path into the Analyzer, including from other cluster workloads. The chart's NetworkPolicy protects the API, not the Analyzer.
• LLM access from the API pods for all default managed monitors: Anthropic for Gateway and Deep Review, OpenAI for Triage, and Google for session summaries. This example uses keys for all three providers. See Which LLM providers are called for the required model names.

For EKS, provide a CSI driver and StorageClass for the audit volumes. Your network plugin must enforce NetworkPolicy. Your security groups must allow traffic from pods to the database. The EKS reference covers these steps, an existing ALB, and registry token renewal. It uses the Terraform files in the Kubernetes deployment bundle.

1. Select the cluster and prepare secrets

Set the context and local chart directory:

WATCHER_CONTEXT=your-cluster-context
WATCHER_NAMESPACE=watcher
WATCHER_CHART=./chart
kubectl --context "$WATCHER_CONTEXT" create namespace "$WATCHER_NAMESPACE"

Skip namespace creation if it already exists. Use your secret manager to create an Opaque Secret named watcher-external in that namespace:

Secret key	Value for this example
DATABASE_URL	postgresql+asyncpg://USER:PASSWORD@HOST:PORT/DBNAME; URL-encode special characters in credentials.
ANTHROPIC_API_KEY	Access to the Gateway and Deep Review models.
OPENAI_API_KEY	Access to the Triage model.
GOOGLE_API_KEY	Access to the session-summary model.
WATCHER_INTERNAL_AUTH_TOKEN	A random shared secret used by the scheduler to authenticate to the API.

Keep secret values out of watcher-values.yaml and shell arguments. Rendering the chart does not check the contents of an existing Secret. Create the Secret and image-pull credentials before installing.

The API pods can start without the OpenAI or Google keys, but their default monitors still need access to those providers. Changing gradingModel does not override the models chosen by managed monitors.

2. Configure the Helm release

Save this as watcher-values.yaml, replacing the placeholder values:

image:
  tag: "<matching-watcher-release-tag>"
  pullSecrets:
    - watcher-registry
publicUrl: "https://watcher.example.com"
organization:
  id: "<stable-organization-id>"
  name: "Example organization"
auth:
  mode: proxy
existingSecret: watcher-external
postgresql:
  enabled: false
api:
  replicas: 1
asyncGrading:
  enabled: true
auditLog:
  storageClassName: "<your-storage-class>"
networkPolicy:
  enabled: true
ingress:
  enabled: false

Set image.pullSecrets to your pull Secret names. Use [] if the nodes already have registry access. For a mirror, set image.registry to the full prefix of the three image repositories, such as registry.example.com/watcher-self-host. The chart adds /xylon-api, /bootstrap, or /analyzer and the tag to that prefix.

Choose organization.id once and keep it across upgrades. Set publicUrl to the external HTTPS URL users and clients will reach.

This example disables the chart's Ingress. Set up your own authenticated ingress to send all traffic to analyzer:80. Keep the original host and forwarded HTTPS scheme. Client request timeouts must cover the server's grading budget. Set the load-balancer idle timeout above the client timeouts. Check https://watcher.example.com/api/v1/health using your proxy's authentication.

To use WorkOS authentication, set auth.mode: workos and auth.workos.orgId. Add WATCHER_WORKOS_CLIENT_ID and WATCHER_WORKOS_API_KEY to the Secret. The API then authenticates users through WorkOS. The chart's ingress values can set up routing and TLS with your existing ingress controller. Match its host and TLS settings to publicUrl. It still routes every path through the Analyzer.

Model gateways

To use an Anthropic-compatible gateway that authenticates through your network or workload identity, add:

modelGateway:
  anthropicBaseUrl: "https://models.example.com"

The Anthropic base URL has no /v1 suffix. For OpenAI-compatible calls, modelGateway.openaiBaseUrl includes /v1, for example https://models.example.com/v1. Each URL routes calls for that provider only. With the Anthropic gateway above, keep OPENAI_API_KEY for Triage and GOOGLE_API_KEY for summaries. If you also set modelGateway.openaiBaseUrl, you can omit the OpenAI key under the same gateway rules. These settings do not route Google calls; this example still needs GOOGLE_API_KEY.

With existingSecret, the chart sends the provider key unused to each configured gateway. You can omit that provider key from the Secret. Adding a real key to the Secret does not change this behavior. If your gateway needs a real provider key, you must adapt this example. Review the chart's modelGateway settings first. Set up the gateway and its authentication outside the chart.

3. Render and install

Inspect the resources before applying them:

helm lint "$WATCHER_CHART" -f watcher-values.yaml
helm template watcher "$WATCHER_CHART" \
  --namespace "$WATCHER_NAMESPACE" -f watcher-values.yaml > watcher-rendered.yaml
helm install watcher "$WATCHER_CHART" \
  --kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE" \
  -f watcher-values.yaml --timeout 10m

The install runs bootstrap as a post-install hook. Helm deletes the bootstrap Job when it succeeds. If the install fails, read the Job's logs. Fix the reported database, image-pull, or settings error before continuing:

kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" logs job/watcher-bootstrap

4. Verify before connecting developers

Check the release, deployments, and claims:

helm status watcher --kube-context "$WATCHER_CONTEXT" --namespace "$WATCHER_NAMESPACE"
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" get pods,pvc
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/xylon-api --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/analyzer --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/xylon-grading-scheduler --timeout=300s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  rollout status deployment/xylon-retention-cleanup --timeout=300s

Both audit claims must be Bound. A Pending claim usually needs a working provisioner or a corrected StorageClass. Confirm HTTP 200 from /api/v1/health through your authenticated HTTPS entry point. Check that unauthenticated requests cannot access session data by supplying forged identity headers.

Test the API NetworkPolicy with a pod that has none of Watcher's allowed labels. Set PROBE_IMAGE to an approved image containing curl, reachable from the cluster:

PROBE_IMAGE=your-registry.example.com/curl:approved-tag
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  run watcher-network-probe --rm -i --restart=Never --image="$PROBE_IMAGE" -- \
  curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \
  http://xylon-api:8000/api/v1/health

Expect a connection timeout and HTTP code 000, while the authenticated entry point still works. Check the curl error: a DNS failure also returns 000. A DNS error, image-pull failure, or HTTP 404 does not prove that the policy blocks traffic.

Finally, follow Client deployment to connect a Watcher client to publicUrl. Run a coding-agent session and confirm it appears with grades in the Analyzer. This tests both recording and model completion access. A health check alone does not test grading.

Scale and operate

Install metrics-server before enabling API autoscaling. To keep auditing with multiple API pods:

1. Choose a StorageClass that supports ReadWriteMany for API audit logs. It must also support ReadWriteOnce for the retention worker's claim.
2. Set auditLog.accessModes: [ReadWriteMany] before enabling api.autoscaling.enabled or increasing api.replicas.

Each API pod writes its own audit file. Read all of these files for the API audit trail. See values.yaml for replica bounds and the CPU target. CPU usage is measured against the API container's CPU request.

Setting adminAuditLogFile: "" disables auditing and removes the claims from the rendered resources. Helm keeps existing audit claims. Changing a claim's access modes requires a new claim. Save the data before replacing or deleting any claim.

With the default single-replica audit storage, API upgrades use Recreate. Service stops while the new pod starts.

Back up managed PostgreSQL, both audit volumes, and your deployment configuration. Collect container logs with your cluster's logging system. After rotating watcher-external, restart the API, scheduler, and retention deployments to load the new values:

kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" rollout restart \
  deployment/xylon-api deployment/xylon-grading-scheduler deployment/xylon-retention-cleanup

For upgrades, use the target chart with its matching image tag and follow Update Watcher. The chart runs bootstrap as a pre-upgrade hook. It must complete before workloads update. If you use GitOps, check that your controller follows this order. Follow CI/CD and Kubernetes/GitOps for deployment automation, including database recovery requirements.