--- description: >- Adapt the supplied EKS Terraform reference with private cluster access, RDS, an existing ALB, and staged Helm installation. --- > Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt > Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md > Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20EKS%20reference&body=Page%3A%20EKS%20reference%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fself-hosting%2Feks-reference%2F%0A%0A # EKS reference Use this example when your Kubernetes deployment bundle contains both `chart/` and `reference-terraform/`. It creates a dedicated EKS cluster and RDS database in an existing VPC. An existing ALB routes traffic to the [Analyzer](../concepts/analyzer.md). The [Watcher API](../api/index.md) authenticates users through [WorkOS](setup.md#sso-mode). Keep the two directories beside each other: the Terraform release loads `../chart`. For a cluster you already operate, start with the [Kubernetes infrastructure guide](kubernetes-infrastructure.md) instead. ## What you provide - Terraform 1.13 or later, AWS CLI, Helm, and `kubectl`. - AWS credentials that can create the reference's resources. Run Terraform and `kubectl` from a host that can reach the private EKS API. - An existing VPC and private subnets in at least two availability zones. Provide egress or private endpoints for registry pulls and AWS services. Allow API pods to reach your model providers or gateway. - An ALB, its HTTPS listener and certificate, DNS, and permission to change the listener's forwarding rule and security-group rules. - The matching Watcher chart and image tag, registry access, WorkOS credentials, and an approved secret manager. - Access from API pods to the Anthropic, OpenAI, and Google models used by the default [managed monitors](../concepts/graders.md#monitors). Follow the [provider requirements](kubernetes-infrastructure.md#prerequisites) for all three, even if you use a gateway for one of them. The reference uses the `default` namespace for Watcher and the registry refresher. Keep your Secret, pull credentials, Helm release, and TargetGroupBinding in that namespace. If you move Watcher to another namespace, update those resources and every refresher Kubernetes resource. Update the refresher RoleBinding's subject namespace and IAM trust policy's service-account subject too. ## Infrastructure and routing ![An EKS deployment using WorkOS authentication. Operator tooling reaches the private EKS API. Users connect over HTTPS to an existing ALB. Its IP target group routes to Analyzer pods on port 8080, selected through the Analyzer Service on port 80. The Analyzer forwards requests to the API, which connects to WorkOS and private RDS. The load-balancer controller maintains the target registrations.](/watcher-eks-reference.svg) The diagram focuses on cluster access and ingress. The [Kubernetes architecture](kubernetes-infrastructure.md#architecture) also shows the background workloads, model calls, and audit volumes. | Reference file | What it manages | | --- | --- | | `eks.tf` | Private EKS API, managed nodes, Kubernetes Secret encryption, control-plane logs, network-policy enforcement, EBS CSI, and IAM roles for service accounts. | | `rds.tf` | Encrypted private PostgreSQL, its pod-access rule, generated database password, backups, and deletion protection. | | `watcher.tf` | An encrypted `gp3` StorageClass and the Watcher Helm release. | | `ingress-alb.tf` | An IP target group, security-group rules, AWS Load Balancer Controller, and a TargetGroupBinding. | | `registry-refresher.tf` | A service account, scoped Secret permissions, and a CronJob that renews registry credentials every eight hours. | The ALB listener, certificate, and DNS stay in your existing infrastructure. Configure the listener to forward the Watcher hostname to the `-analyzer` target group created by the reference. Its health-check path is `/api/v1/health`. Allow TCP 8080 from the ALB security group to the pod security group in both the ALB's egress rules and the pods' ingress rules. The TargetGroupBinding selects `analyzer:80`; the target group registers pod IPs on port 8080. This route uses WorkOS authentication. If you set `auth.mode` to `proxy`, use the identity-aware proxy described in [Kubernetes prerequisites](kubernetes-infrastructure.md#prerequisites). It must be the only path into the Analyzer. An ALB alone cannot authenticate proxy-mode requests. ## 1. Adapt the reference Review the Terraform files before running `terraform init`. Replace every ``, including organization values, the ALB security group, and third-party image repositories and digests. Set the inputs in `reference-terraform/terraform.tfvars`: ```hcl region = "eu-west-2" cluster_name = "watcher-k8s" vpc_id = "vpc-xxxxxxxx" private_subnet_ids = ["subnet-aaaaaaaa", "subnet-bbbbbbbb"] public_url = "https://watcher.example.com" eks_version = "" watcher_image_tag = "" ``` Choose an EKS version in standard support. In `providers.tf`, replace the region placeholder in both exec-auth argument lists. If you change `cluster_name`, update the literal cluster name in both lists too. These provider settings must match `terraform.tfvars`. The cluster creator gets admin access. Add explicit EKS access entries for the operator and CI roles that will manage it. A kubeconfig context alone does not grant access. Protect your Terraform backend and restrict access to its state. Terraform stores the generated RDS password in state. Marking the output `sensitive` hides it from normal display; it stays in state. Store application and WorkOS credentials in your secret manager. Review the database settings against your availability and recovery needs. The RDS example uses a single availability zone, seven-day backups, deletion protection, and a final snapshot. The supplied `gp3` class uses encrypted EBS volumes. With `WaitForFirstConsumer`, a claim can stay Pending until a pod is scheduled. The reclaim policy is `Delete`. Helm keeps audit claims, but deleting a claim yourself can delete its volume. Match controller and refresher images to your node architecture when mirroring them. Pin each image to the digest from the registry you mirror into. The example's `m6i` nodes use `linux/amd64`. Use the load-balancer controller chart and bundled IAM policy from the same version. ## 2. Provision AWS resources Run from `reference-terraform/`. Split the first apply into stages. The Kubernetes providers need a reachable cluster. Terraform also needs the TargetGroupBinding CRD before it can plan that manifest. ```sh terraform init terraform apply \ -target=aws_eks_cluster.this \ -target=aws_eks_node_group.default \ -target=aws_eks_addon.vpc_cni \ -target=aws_eks_addon.ebs_csi \ -target=aws_iam_role_policy_attachment.ebs_csi \ -target=aws_db_instance.watcher \ -target=aws_vpc_security_group_ingress_rule.rds_from_pods ``` Include the EBS driver policy attachment and database ingress rule as separate targets. The driver needs the policy to manage volumes. The bootstrap Job needs the rule to reach the database on its first run. Create a context using the same cluster and region as the Terraform inputs: ```sh WATCHER_AWS_REGION=eu-west-2 WATCHER_CLUSTER=watcher-k8s WATCHER_CONTEXT=watcher-eks WATCHER_NAMESPACE=default aws eks update-kubeconfig --name "$WATCHER_CLUSTER" \ --region "$WATCHER_AWS_REGION" --alias "$WATCHER_CONTEXT" kubectl --context "$WATCHER_CONTEXT" get nodes ``` Fix network or EKS access failures before continuing. This reference keeps the cluster API's public endpoint disabled. ## 3. Install controllers and registry refresh Before pulling images from Apollo's registry, allow the refresher's IAM role to assume the registry pull role. The pull role must also trust the refresher. The refresher must pull its own AWS CLI and kubectl images before it creates `apollo-ecr`. Use your node role's registry access or a separate credential for those images. ```sh terraform apply \ -target=helm_release.lb_controller \ -target=aws_iam_role_policy.lb_controller \ -target=kubernetes_cron_job_v1.ecr_refresher \ -target=aws_iam_role_policy.ecr_refresher \ -target=kubernetes_role_binding.ecr_refresher kubectl --context "$WATCHER_CONTEXT" wait --for=condition=Established \ crd/targetgroupbindings.elbv2.k8s.aws --timeout=180s ``` Run the refresher once before installing Watcher, then verify completion and the pull Secret: ```sh kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \ create job --from=cronjob/ecr-refresher ecr-refresher-init kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" \ wait --for=condition=complete job/ecr-refresher-init --timeout=180s kubectl --context "$WATCHER_CONTEXT" -n "$WATCHER_NAMESPACE" get secret apollo-ecr ``` The CronJob renews the Secret every eight hours because ECR tokens expire after twelve. Its Kubernetes role can create Secrets in the namespace, but its read and patch permissions name only `apollo-ecr`. Monitor refresh failures. An expired token can stop new pods from starting. For an [internal mirror](supply-chain.md#use-internal-mirrors), omit the refresher resources and remove their targets from the apply command above. Set `image.registry` to the mirror's full repository prefix. Replace `image.pullSecrets` with your own pull Secret names, or use `[]` when nodes already have access. Install the load-balancer controller and verify its CRD before proceeding. ## 4. Create the application Secret and install Watcher Read the database endpoint without exposing the password: ```sh terraform output -raw rds_endpoint ``` The endpoint includes its port. Transfer `rds_master_password` directly to your secret manager. Set `DATABASE_URL` to `postgresql+asyncpg://watcher:PASSWORD@HOST:PORT/watcher`. Create `watcher-external` in the release namespace with these keys: | Secret key | Value | | --- | --- | | `DATABASE_URL` | The database URL above. | | `WATCHER_INTERNAL_AUTH_TOKEN` | A random shared value. | | `WATCHER_WORKOS_CLIENT_ID` | Your WorkOS client ID. | | `WATCHER_WORKOS_API_KEY` | Your WorkOS API key. | | `ANTHROPIC_API_KEY` | Access to Gateway and Deep Review. Omit only with an Anthropic-compatible gateway. | | `OPENAI_API_KEY` | Access to Triage. Omit only with an OpenAI-compatible gateway. | | `GOOGLE_API_KEY` | Access to session summaries. Keep this key in this example. | Follow the [model gateway settings](kubernetes-infrastructure.md#model-gateways) for each provider you route through a gateway. With only an Anthropic gateway configured, keep the OpenAI and Google keys. Keep secret values out of Terraform values, shell arguments, and committed manifests. In `watcher.tf`, set `organization.id`, `organization.name`, and `auth.workos.orgId`. Check `existingSecret: watcher-external`, the matching image tag, and the external `publicUrl`. Keep the organization ID after installing. With the CRD, pull credentials, and application Secret ready, review and run the full apply: ```sh terraform plan terraform apply ``` Use full plans for later changes. Targeted applies are for the first install. ## 5. Verify and operate Complete the checks in [Verify before connecting developers](kubernetes-infrastructure.md#4-verify-before-connecting-developers). Check that the ALB target group is healthy and the listener points to it. Test that WorkOS login returns to `publicUrl`. Set the ALB idle timeout above client timeouts for synchronous grading calls. Back up RDS, the audit volumes, and deployment settings. Collect pod logs outside the nodes. Install metrics-server before enabling API autoscaling; the reference does not include it. EBS `gp3` cannot provide shared audit storage for multiple API replicas. Follow [Scale and operate](kubernetes-infrastructure.md#scale-and-operate) to choose shared storage and keep your audit logs.