Skip to main content
Working with an agent? Give them a link to this page as markdown.

EKS infrastructure checklist

Use this checklist alongside the Kubernetes setup guide when deploying Watcher on EKS. Provision AWS resources with your own infrastructure tooling. The Watcher chart does not include an EKS Terraform module or create the cluster, database, registry refresh job or load balancer.

Cluster access and networking​

Provide an EKS cluster and a host that can reach its Kubernetes API. With a private endpoint, run deployment tooling from inside the VPC or a connected network. Give the operator and deployment roles EKS access entries with the permissions needed to install the chart. A kubeconfig context alone does not grant access.

Choose an EKS version in standard support. Provide private subnets in at least two availability zones and enough addresses for nodes and pods. Configure egress or VPC endpoints for registry pulls and AWS services. The API and operation-worker pods also need access to their model providers or gateway.

Select the cluster before following the Kubernetes setup guide:

WATCHER_AWS_REGION=eu-west-2
WATCHER_CLUSTER=your-cluster-name
WATCHER_CONTEXT=watcher-eks
aws eks update-kubeconfig --name "$WATCHER_CLUSTER" \
--region "$WATCHER_AWS_REGION" --alias "$WATCHER_CONTEXT"
kubectl --context "$WATCHER_CONTEXT" get nodes

Enable NetworkPolicy enforcement in the VPC CNI, or provide another network plugin that enforces it. For the VPC CNI, configure enableNetworkPolicy: true. Policy objects alone do not prove enforcement. Run the network probe in the Kubernetes setup guide before connecting developers.

Storage and database​

Install the EBS CSI driver and grant its IAM role the permissions to manage volumes. Create an encrypted gp3 StorageClass and select it through auditLog.storageClassName. An existing gp2 class alone does not establish that the cluster can provision the chart's audit volumes.

With WaitForFirstConsumer, a claim can remain Pending until a pod is scheduled. Check the class's reclaim policy before deleting claims. Helm keeps audit claims, but a class with Delete can delete the volume when an operator deletes the claim.

EBS volumes do not provide shared audit storage for API replicas on different nodes. For multiple API replicas, use shared storage such as EFS with its CSI driver, or disable the admin audit log. Follow the audit-storage requirements. Install metrics-server before enabling API autoscaling.

Provide private PostgreSQL 17 or later, and let the database user own an empty database for the first installation. Allow access from the bootstrap Job and all Watcher workloads that use it. Configure security groups as well as Kubernetes policies; a reachable node does not prove that pods can connect. Choose backups, availability and deletion protection to meet your recovery requirements.

Store the database credentials in your secret manager. Set DATABASE_URL in watcher-external as described in the Kubernetes setup guide. If Terraform manages the database password, protect its state: marking an output sensitive does not remove the password from state.

Registry access before installation​

Provide registry access for all images before installing the chart. You can use an internal mirror, node credentials, or image-pull Secrets in the Watcher namespace.

For an image-pull Secret containing an ECR login token, arrange renewal before the token expires after twelve hours. An eight-hour refresh interval leaves margin. Give the refreshing workload permission to obtain the registry credentials and update the intended Secret. It must be able to pull its own image before that Secret exists. Run it once and verify the Secret before Helm installation; waiting for the first scheduled run can leave the bootstrap Job unable to start.

For Apollo's registry, the customer's workload role must be allowed to assume Apollo's registry pull role, and that pull role must trust it. See registry access for the two-role relationship. For your own mirror, follow your registry's authentication process. The chart does not install a token refresher.

Ingress and authentication​

An EKS deployment using WorkOS authentication. Operator tooling reaches the private EKS API. Users connect over HTTPS to an existing ALB. Its IP target group routes to Analyzer pods on port 8080, selected through the Analyzer Service on port 80. The Analyzer forwards requests to the API, which connects to WorkOS and private RDS. The load-balancer controller maintains the target registrations.

The diagram shows one WorkOS deployment. The Analyzer Service listens on port 80, while its pods listen on port 8080. For an ALB with IP targets, allow traffic to pod port 8080 in both the relevant security groups and network policies. Route all Watcher paths, including /api/*, through the Analyzer. Its health-check path is /api/v1/health.

An existing ALB can use a TargetGroupBinding to keep the target group aligned with Analyzer pods. Install the AWS Load Balancer Controller, its IAM permissions and its CRDs before creating the binding. Configure the listener, certificate and DNS in your own infrastructure. Alternatively, use the chart's Ingress with your existing controller.

Match publicUrl to the HTTPS hostname. Set the load-balancer idle timeout above the client timeout, and the client timeout high enough for the server's grading budget.

For proxy authentication, an identity-aware proxy must authenticate every request and strip client-supplied identity headers before adding verified values. An ALB alone is not that boundary. Keep the proxy as the only path into the Analyzer. For WorkOS, the Watcher API authenticates users through WorkOS.

Installation order​

  1. Provision the cluster, database, network rules and required IAM permissions.
  2. Install the storage driver, StorageClass and ingress controller. Wait for any required CRDs before creating custom resources such as TargetGroupBinding.
  3. Create the Watcher namespace, application Secret and registry credentials. Complete the initial registry refresh before installing workloads.
  4. Follow Render and install, then Verify before connecting developers.

If you use Terraform, separate the AWS provisioning stage from Kubernetes resources that need a reachable cluster at plan time. Resources using a CRD also need that CRD installed before Terraform can plan them.

Keep the database and audit volumes across chart upgrades. Back them up, and collect pod logs outside the nodes. For upgrades, use the matching chart and image tag and follow Update Watcher.