Skip to main content
Working with an agent? Give them a link to this page as markdown.

EKS reference

Use this example when your Kubernetes deployment bundle contains both chart/ and reference-terraform/. It creates a dedicated EKS cluster and RDS database in an existing VPC. An existing ALB routes traffic to the Analyzer. The Watcher API authenticates users through WorkOS.

Keep the two directories beside each other: the Terraform release loads ../chart. For a cluster you already operate, start with the Kubernetes infrastructure guide instead.

What you provide​

  • Terraform 1.13 or later, AWS CLI, Helm, and kubectl.
  • AWS credentials that can create the reference's resources. Run Terraform and kubectl from a host that can reach the private EKS API.
  • An existing VPC and private subnets in at least two availability zones. Provide egress or private endpoints for registry pulls and AWS services. Allow API pods to reach your model providers or gateway.
  • An ALB, its HTTPS listener and certificate, DNS, and permission to change the listener's forwarding rule and security-group rules.
  • The matching Watcher chart and image tag, registry access, WorkOS credentials, and an approved secret manager.
  • Access from API pods to the Anthropic, OpenAI, and Google models used by the default managed monitors. Follow the provider requirements for all three, even if you use a gateway for one of them.

The reference uses the default namespace for Watcher and the registry refresher. Keep your Secret, pull credentials, Helm release, and TargetGroupBinding in that namespace. If you move Watcher to another namespace, update those resources and every refresher Kubernetes resource. Update the refresher RoleBinding's subject namespace and IAM trust policy's service-account subject too.

Infrastructure and routing​

An EKS deployment using WorkOS authentication. Operator tooling reaches the private EKS API. Users connect over HTTPS to an existing ALB. Its IP target group routes to Analyzer pods on port 8080, selected through the Analyzer Service on port 80. The Analyzer forwards requests to the API, which connects to WorkOS and private RDS. The load-balancer controller maintains the target registrations.

The diagram focuses on cluster access and ingress. The Kubernetes architecture also shows the background workloads, model calls, and audit volumes.

Reference fileWhat it manages
eks.tfPrivate EKS API, managed nodes, Kubernetes Secret encryption, control-plane logs, network-policy enforcement, EBS CSI, and IAM roles for service accounts.
rds.tfEncrypted private PostgreSQL, its pod-access rule, generated database password, backups, and deletion protection.
watcher.tfAn encrypted gp3 StorageClass and the Watcher Helm release.
ingress-alb.tfAn IP target group, security-group rules, AWS Load Balancer Controller, and a TargetGroupBinding.
registry-refresher.tfA service account, scoped Secret permissions, and a CronJob that renews registry credentials every eight hours.

The ALB listener, certificate, and DNS stay in your existing infrastructure. Configure the listener to forward the Watcher hostname to the <cluster_name>-analyzer target group created by the reference. Its health-check path is /api/v1/health. Allow TCP 8080 from the ALB security group to the pod security group in both the ALB's egress rules and the pods' ingress rules. The TargetGroupBinding selects analyzer:80; the target group registers pod IPs on port 8080.

This route uses WorkOS authentication. If you set auth.mode to proxy, use the identity-aware proxy described in Kubernetes prerequisites. It must be the only path into the Analyzer. An ALB alone cannot authenticate proxy-mode requests.

1. Adapt the reference​

Review the Terraform files before running terraform init. Replace every <PLACEHOLDER>, including organization values, the ALB security group, and third-party image repositories and digests. Set the inputs in reference-terraform/terraform.tfvars:

region             = "eu-west-2"
cluster_name       = "watcher-k8s"
vpc_id             = "vpc-xxxxxxxx"
private_subnet_ids = ["subnet-aaaaaaaa", "subnet-bbbbbbbb"]
public_url         = "https://watcher.example.com"
eks_version        = "<supported-eks-version>"
watcher_image_tag  = "<matching-watcher-release-tag>"

Choose an EKS version in standard support. In providers.tf, replace the region placeholder in both exec-auth argument lists. If you change cluster_name, update the literal cluster name in both lists too. These provider settings must match terraform.tfvars.

The cluster creator gets admin access. Add explicit EKS access entries for the operator and CI roles that will manage it. A kubeconfig context alone does not grant access.

Protect your Terraform backend and restrict access to its state. Terraform stores the generated RDS password in state. Marking the output sensitive hides it from normal display; it stays in state. Store application and WorkOS credentials in your secret manager.

Review the database settings against your availability and recovery needs. The RDS example uses a single availability zone, seven-day backups, deletion protection, and a final snapshot.

The supplied gp3 class uses encrypted EBS volumes. With WaitForFirstConsumer, a claim can stay Pending until a pod is scheduled. The reclaim policy is Delete. Helm keeps audit claims, but deleting a claim yourself can delete its volume.

Match controller and refresher images to your node architecture when mirroring them. Pin each image to the digest from the registry you mirror into. The example's m6i nodes use linux/amd64. Use the load-balancer controller chart and bundled IAM policy from the same version.

2. Provision AWS resources​

Run from reference-terraform/. Split the first apply into stages. The Kubernetes providers need a reachable cluster. Terraform also needs the TargetGroupBinding CRD before it can plan that manifest.

terraform init
terraform apply \
  -target=aws_eks_cluster.this \
  -target=aws_eks_node_group.default \
  -target=aws_eks_addon.vpc_cni \
  -target=aws_eks_addon.ebs_csi \
  -target=aws_iam_role_policy_attachment.ebs_csi \
  -target=aws_db_instance.watcher \
  -target=aws_vpc_security_group_ingress_rule.rds_from_pods

Include the EBS driver policy attachment and database ingress rule as separate targets. The driver needs the policy to manage volumes. The bootstrap Job needs the rule to reach the database on its first run.

Create a context using the same cluster and region as the Terraform inputs:

WATCHER_AWS_REGION=eu-west-2
WATCHER_CLUSTER=watcher-k8s
WATCHER_CONTEXT=watcher-eks
WATCHER_NAMESPACE=default
aws eks update-kubeconfig --name "$WATCHER_CLUSTER" \
  --region "$WATCHER_AWS_REGION" --alias "$WATCHER_CONTEXT"
kubectl --context "$WATCHER_CONTEXT" get nodes

Fix network or EKS access failures before continuing. This reference keeps the cluster API's public endpoint disabled.

3. Install controllers and registry refresh​

Before pulling images from Apollo's registry, allow the refresher's IAM role to assume the registry pull role. The pull role must also trust the refresher.

The refresher must pull its own AWS CLI and kubectl images before it creates apollo-ecr. Use your node role's registry access or a separate credential for those images.

terraform apply \
  -target=helm_release.lb_controller \
  -target=aws_iam_role_policy.lb_controller \
  -target=kubernetes_cron_job_v1.ecr_refresher \
  -target=aws_iam_role_policy.ecr_refresher \
  -target=kubernetes_role_binding.ecr_refresher
kubectl --context "$WATCHER_CONTEXT" wait --for=condition=Established \
  crd/targetgroupbindings.elbv2.k8s.aws --timeout=180s

Run the refresher once before installing Watcher, then verify completion and the pull Secret:

kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  create job --from=cronjob/ecr-refresher ecr-refresher-init
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \
  wait --for=condition=complete job/ecr-refresher-init --timeout=180s
kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" get secret apollo-ecr

The CronJob renews the Secret every eight hours because ECR tokens expire after twelve. Its Kubernetes role can create Secrets in the namespace, but its read and patch permissions name only apollo-ecr. Monitor refresh failures. An expired token can stop new pods from starting.

For an internal mirror, omit the refresher resources and remove their targets from the apply command above. Set image.registry to the mirror's full repository prefix. Replace image.pullSecrets with your own pull Secret names, or use [] when nodes already have access. Install the load-balancer controller and verify its CRD before proceeding.

4. Create the application Secret and install Watcher​

Read the database endpoint without exposing the password:

terraform output -raw rds_endpoint

The endpoint includes its port. Transfer rds_master_password directly to your secret manager. Set DATABASE_URL to postgresql+asyncpg://watcher:PASSWORD@HOST:PORT/watcher. Create watcher-external in the release namespace with these keys:

Secret keyValue
DATABASE_URLThe database URL above.
WATCHER_INTERNAL_AUTH_TOKENA random shared value.
WATCHER_WORKOS_CLIENT_IDYour WorkOS client ID.
WATCHER_WORKOS_API_KEYYour WorkOS API key.
ANTHROPIC_API_KEYAccess to Gateway and Deep Review. Omit only with an Anthropic-compatible gateway.
OPENAI_API_KEYAccess to Triage. Omit only with an OpenAI-compatible gateway.
GOOGLE_API_KEYAccess to session summaries. Keep this key in this example.

Follow the model gateway settings for each provider you route through a gateway. With only an Anthropic gateway configured, keep the OpenAI and Google keys.

Keep secret values out of Terraform values, shell arguments, and committed manifests.

In watcher.tf, set organization.id, organization.name, and auth.workos.orgId. Check existingSecret: watcher-external, the matching image tag, and the external publicUrl. Keep the organization ID after installing.

With the CRD, pull credentials, and application Secret ready, review and run the full apply:

terraform plan
terraform apply

Use full plans for later changes. Targeted applies are for the first install.

5. Verify and operate​

Complete the checks in Verify before connecting developers. Check that the ALB target group is healthy and the listener points to it. Test that WorkOS login returns to publicUrl. Set the ALB idle timeout above client timeouts for synchronous grading calls.

Back up RDS, the audit volumes, and deployment settings. Collect pod logs outside the nodes. Install metrics-server before enabling API autoscaling; the reference does not include it. EBS gp3 cannot provide shared audit storage for multiple API replicas. Follow Scale and operate to choose shared storage and keep your audit logs.

Found an issue with this page? Report it on GitHub.