--- description: >- Provision self-hosted Watcher on AWS with the bundled Terraform modules. --- > Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt > Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md > Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20AWS%20infrastructure&body=Page%3A%20AWS%20infrastructure%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fself-hosting%2Faws-infrastructure%2F%0A%0A # AWS infrastructure The self-host bundle includes Terraform modules for the EC2 host and an optional ACM certificate. The host module also creates security groups, CloudWatch Logs integration, and an optional RDS Postgres instance. It creates the host's IAM role by default, or uses an existing instance profile. You provide the VPC and subnets. For an existing Kubernetes cluster, use the [Kubernetes infrastructure example](kubernetes-infrastructure.md). To provision a dedicated AWS cluster from the Kubernetes bundle, use the [EKS reference](eks-reference.md). The [infrastructure prerequisites prompt](prompts/prerequisites.md) covers the network and certificate decisions before provisioning. ## Prepare registry access The default image registry uses two IAM roles: | Role | Account | Purpose | | --- | --- | --- | | Customer source role | Your AWS account | Gives the host credentials to request image-pull access. | | Apollo destination role | Apollo's AWS account | `arn:aws:iam::456354208905:role/WatcherSelfHostPuller`, which grants image-pull access. | The host calls `aws sts assume-role` with the Apollo ARN as `--role-arn`. That option selects the destination. The host's existing AWS credentials choose the source role. `--role-session-name` names the resulting session; it does not choose the source role. Before provisioning, confirm which source role the host will use: - With the module defaults, it creates `arn:aws:iam:::role/watcher-self-host-instance` and grants it `sts:AssumeRole` on the Apollo destination ARN. - With a custom `instance_role_name`, use that role's ARN. - With `create_iam = false`, set `existing_instance_profile_name`. Your IAM owner must grant the role behind that profile `sts:AssumeRole` on the Apollo destination ARN. The module does not manage that role's policies. Apollo's pull role must trust the exact source IAM role ARN, including any path. Confirm this before the host starts pulling images. The default role can be allowlisted before it exists, using your AWS account ID and its name. An allowlist entry for that role does not cover another role in your account. A custom role needs its own entry, such as `arn:aws:iam:::role/platform/WatcherHost`. If you use an [internal registry](supply-chain.md#use-internal-mirrors), grant access through your own IAM policy and set `configure_apollo_ecr = false`. The host does not need Apollo role access; the linked page covers the registry login you provide instead. ## Use the module Copy `modules/` from the release bundle into your Terraform repository, then add a module call: ```hcl module "watcher" { source = "./modules/watcher-self-host" vpc_id = "vpc-xxxxxxxx" instance_subnet_id = "subnet-xxxxxxxx" ami_id = "ami-xxxxxxxx" # configure_apollo_ecr = false # Internal registry only. analyzer_ingress_security_group_ids = ["sg-xxxxxxxx"] # Optional: managed RDS instead of bundled Postgres. # create_rds = true # rds_subnet_ids = ["subnet-aaaa", "subnet-bbbb"] tags = { environment = "production" } } output "watcher_instance_id" { value = module.watcher.instance_id } output "watcher_instance_role_arn" { value = module.watcher.instance_role_arn } output "watcher_rds_endpoint" { value = module.watcher.rds_endpoint } output "watcher_rds_database_name" { value = module.watcher.rds_database_name } output "watcher_rds_master_username" { value = module.watcher.rds_master_username } output "watcher_rds_master_password" { value = module.watcher.rds_master_password sensitive = true } ``` Terraform does not surface a module's outputs at the root, so declare a root output for every value you need later. The four RDS outputs are `null` when `create_rds` is false. `watcher_instance_role_arn` is `null` when `create_iam = false`; get that role's ARN from your IAM owner instead. Use bundled Postgres for the smallest infrastructure surface. Use RDS when you want managed backups and restore testing. The setup prompt explains the operational ownership of both choices. The host must reach the image registry, LLM endpoints, and SSM unless you provide private endpoints. The module expects an Ubuntu 24.04 LTS AMI for the deployment region. See `modules/watcher-self-host/variables.tf` in the bundle for the full input and output reference. ## Manage the RDS version With `create_rds = true`, new databases default to PostgreSQL 17. Set `rds_engine_version` in your module call to choose another supported RDS version. The module does not permit major-version upgrades. Changing this input is not an upgrade procedure. For an existing database, pin its current major version before applying an updated module. Read the actual version with: ```sh aws rds describe-db-instances --db-instance-identifier \ --query 'DBInstances[0].EngineVersion' --output text ``` For example, a database running 16.x needs this input inside the module call: ```hcl rds_engine_version = "16" ``` Review the complete `terraform plan`. The first plan may add `allow_major_version_upgrade = false` without changing `engine_version`; that policy change is expected and does not change the database version. Do not apply unexpected engine-version changes, database replacement or host replacement. A rejected database upgrade does not roll back other resources Terraform has already changed. ### Upgrade an existing database Existing PostgreSQL 16 installations can keep their API, grading and ingestion workloads. The background-job worker generates the deep reviews that users request with **Generate report** in the Analyzer's session view; scheduled summaries and deep reviews do not use it. The worker ships as the `xylon-operation-worker` Compose service behind the `operation-worker` profile. Compose installs created from this release's `.env.example` run the worker; on an existing install, add `operation-worker` to `COMPOSE_PROFILES`. Without the worker, requested reviews stay queued and never finish. Updating this Terraform module alone does not start the worker. The worker requires PostgreSQL 17 or later and refuses to start on an older server. No database upgrade is required to keep the existing workloads. Before enabling the worker on PostgreSQL 16, qualify the upgrade separately: 1. Choose a supported target for the database's exact source version and region using the [RDS major-upgrade guide](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.MajorVersion.html). 2. Take a snapshot and rehearse the upgrade on a restored copy. Follow the [RDS upgrade procedure](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.MajorVersion.Process.html), including extension, parameter-group, instance-class and pre-upgrade checks. Test the application against the upgraded copy. 3. Suspend automated Terraform applies until the version input is reconciled in the next step. Schedule downtime and stop application writers. Take a final pre-upgrade backup, then perform the reviewed RDS upgrade outside this module. 4. Verify `SHOW server_version_num` returns at least `170000`. Check application reads and writes before resuming traffic. Set `rds_engine_version` to the upgraded major version and review a refreshed Terraform plan; it must not propose undoing or replacing the database. Rollback requires restoring the pre-upgrade backup to another database and reconciling the application connection and Terraform state. Setting the version input back cannot downgrade PostgreSQL. Restoration discards writes made after the backup unless you separately recover them. ## Connect after provisioning Connect through SSM, then continue with [Setup](setup.md): ```sh aws ssm start-session --target $(terraform output -raw watcher_instance_id) ``` For Apollo ECR, verify the source identity inside the host shell: ```sh aws sts get-caller-identity --region eu-west-2 ``` The ARN for a role session has the form `arn:aws:sts:::assumed-role//`. This is not the IAM role ARN needed for the allowlist. The session ARN omits the IAM role's path. Read `terraform output -raw watcher_instance_role_arn` from the Terraform working directory for a module-created role. For an existing role, read its full ARN in your account with: ```sh aws iam get-role --role-name --query 'Role.Arn' --output text ``` Use an IAM administrator's credentials for that lookup if the host cannot read IAM. Match its account and role name to the host identity, then confirm the full IAM ARN is allowlisted. Complete the [setup prompt's access test](prompts/setup.md#step-1-confirm-prerequisites-and-delivery-inputs) on the host before Docker login. A successful test from a laptop using different credentials does not verify the host's access. With `create_rds = true`, read the connection values first. Setup asks for a `DATABASE_URL` of the form `postgresql+asyncpg://:@/`, and the endpoint output already carries `host:port`: ```sh terraform output -raw watcher_rds_endpoint terraform output -raw watcher_rds_master_username terraform output -raw watcher_rds_database_name terraform output -raw watcher_rds_master_password ``` Store the password in your secret manager before closing the session. ## Replace the instance Changing `ami_id`, or changing module input that affects instance user data, replaces the EC2 instance and destroys its root volume. This removes: - files under `/opt/watcher-self-host`, including deployment configuration; - the [admin audit log](operations.md#admin-audit-log) volume; - the bundled Postgres volume when `create_rds` is false. RDS and resources outside the instance survive. Back up every applicable item under [Backups](operations.md#backups) before applying a replacement. Restore the saved configuration and data on the new host.