AWS infrastructure

The self-host bundle includes Terraform modules for the EC2 host and an optional ACM certificate. The host module also creates security groups, CloudWatch Logs integration, and an optional RDS Postgres instance. It creates the host's IAM role by default, or uses an existing instance profile. You provide the VPC and subnets. For an existing Kubernetes cluster, use the Kubernetes infrastructure example. To provision a dedicated AWS cluster from the Kubernetes bundle, use the EKS reference.

The infrastructure prerequisites prompt covers the network and certificate decisions before provisioning.

Prepare registry access

The default image registry uses two IAM roles:

Role	Account	Purpose
Customer source role	Your AWS account	Gives the host credentials to request image-pull access.
Apollo destination role	Apollo's AWS account	arn:aws:iam::456354208905:role/WatcherSelfHostPuller, which grants image-pull access.

The host calls aws sts assume-role with the Apollo ARN as --role-arn. That option selects the destination. The host's existing AWS credentials choose the source role. --role-session-name names the resulting session; it does not choose the source role.

Before provisioning, confirm which source role the host will use:

• With the module defaults, it creates arn:aws:iam::<your-account-id>:role/watcher-self-host-instance and grants it sts:AssumeRole on the Apollo destination ARN.
• With a custom instance_role_name, use that role's ARN.
• With create_iam = false, set existing_instance_profile_name. Your IAM owner must grant the role behind that profile sts:AssumeRole on the Apollo destination ARN. The module does not manage that role's policies.

Apollo's pull role must trust the exact source IAM role ARN, including any path. Confirm this before the host starts pulling images. The default role can be allowlisted before it exists, using your AWS account ID and its name. An allowlist entry for that role does not cover another role in your account. A custom role needs its own entry, such as arn:aws:iam::<your-account-id>:role/platform/WatcherHost.

If you use an internal registry, grant access through your own IAM policy and set configure_apollo_ecr = false. The host does not need Apollo role access; the linked page covers the registry login you provide instead.

Use the module

Copy modules/ from the release bundle into your Terraform repository, then add a module call:

module "watcher" {
  source = "./modules/watcher-self-host"

  vpc_id             = "vpc-xxxxxxxx"
  instance_subnet_id = "subnet-xxxxxxxx"
  ami_id              = "ami-xxxxxxxx"

  # configure_apollo_ecr = false  # Internal registry only.

  analyzer_ingress_security_group_ids = ["sg-xxxxxxxx"]

  # Optional: managed RDS instead of bundled Postgres.
  # create_rds     = true
  # rds_subnet_ids = ["subnet-aaaa", "subnet-bbbb"]

  tags = {
    environment = "production"
  }
}

output "watcher_instance_id" {
  value = module.watcher.instance_id
}

output "watcher_instance_role_arn" {
  value = module.watcher.instance_role_arn
}

output "watcher_rds_endpoint" {
  value = module.watcher.rds_endpoint
}

output "watcher_rds_database_name" {
  value = module.watcher.rds_database_name
}

output "watcher_rds_master_username" {
  value = module.watcher.rds_master_username
}

output "watcher_rds_master_password" {
  value     = module.watcher.rds_master_password
  sensitive = true
}

Terraform does not surface a module's outputs at the root, so declare a root output for every value you need later. The four RDS outputs are null when create_rds is false. watcher_instance_role_arn is null when create_iam = false; get that role's ARN from your IAM owner instead.

Use bundled Postgres for the smallest infrastructure surface. Use RDS when you want managed backups and restore testing. The setup prompt explains the operational ownership of both choices.

The host must reach the image registry, LLM endpoints, and SSM unless you provide private endpoints. The module expects an Ubuntu 24.04 LTS AMI for the deployment region. See modules/watcher-self-host/variables.tf in the bundle for the full input and output reference.

Manage the RDS version

With create_rds = true, new databases default to PostgreSQL 17. Set rds_engine_version in your module call to choose another supported RDS version. The module does not permit major-version upgrades. Changing this input is not an upgrade procedure.

For an existing database, pin its current major version before applying an updated module. Read the actual version with:

aws rds describe-db-instances --db-instance-identifier <database identifier> \
  --query 'DBInstances[0].EngineVersion' --output text

For example, a database running 16.x needs this input inside the module call:

rds_engine_version = "16"

Review the complete terraform plan. The first plan may add allow_major_version_upgrade = false without changing engine_version; that policy change is expected and does not change the database version. Do not apply unexpected engine-version changes, database replacement or host replacement. A rejected database upgrade does not roll back other resources Terraform has already changed.

Upgrade an existing database

Existing PostgreSQL 16 installations can keep their API, grading and ingestion workloads. The background-job worker generates the deep reviews that users request with Generate report in the Analyzer's session view; scheduled summaries and deep reviews do not use it. The worker ships as the xylon-operation-worker Compose service behind the operation-worker profile. Compose installs created from this release's .env.example run the worker; on an existing install, add operation-worker to COMPOSE_PROFILES. Without the worker, requested reviews stay queued and never finish. Updating this Terraform module alone does not start the worker.

The worker requires PostgreSQL 17 or later and refuses to start on an older server. No database upgrade is required to keep the existing workloads. Before enabling the worker on PostgreSQL 16, qualify the upgrade separately:

1. Choose a supported target for the database's exact source version and region using the RDS major-upgrade guide.
2. Take a snapshot and rehearse the upgrade on a restored copy. Follow the RDS upgrade procedure, including extension, parameter-group, instance-class and pre-upgrade checks. Test the application against the upgraded copy.
3. Suspend automated Terraform applies until the version input is reconciled in the next step. Schedule downtime and stop application writers. Take a final pre-upgrade backup, then perform the reviewed RDS upgrade outside this module.
4. Verify SHOW server_version_num returns at least 170000. Check application reads and writes before resuming traffic. Set rds_engine_version to the upgraded major version and review a refreshed Terraform plan; it must not propose undoing or replacing the database.

Rollback requires restoring the pre-upgrade backup to another database and reconciling the application connection and Terraform state. Setting the version input back cannot downgrade PostgreSQL. Restoration discards writes made after the backup unless you separately recover them.

Connect after provisioning

Connect through SSM, then continue with Setup:

aws ssm start-session --target $(terraform output -raw watcher_instance_id)

For Apollo ECR, verify the source identity inside the host shell:

aws sts get-caller-identity --region eu-west-2

The ARN for a role session has the form arn:aws:sts::<your-account-id>:assumed-role/<role-name>/<session>. This is not the IAM role ARN needed for the allowlist. The session ARN omits the IAM role's path. Read terraform output -raw watcher_instance_role_arn from the Terraform working directory for a module-created role. For an existing role, read its full ARN in your account with:

aws iam get-role --role-name <source-role-name> --query 'Role.Arn' --output text

Use an IAM administrator's credentials for that lookup if the host cannot read IAM. Match its account and role name to the host identity, then confirm the full IAM ARN is allowlisted. Complete the setup prompt's access test on the host before Docker login. A successful test from a laptop using different credentials does not verify the host's access.

With create_rds = true, read the connection values first. Setup asks for a DATABASE_URL of the form postgresql+asyncpg://<username>:<password>@<endpoint>/<database name>, and the endpoint output already carries host:port:

terraform output -raw watcher_rds_endpoint
terraform output -raw watcher_rds_master_username
terraform output -raw watcher_rds_database_name
terraform output -raw watcher_rds_master_password

Store the password in your secret manager before closing the session.

Replace the instance

Changing ami_id, or changing module input that affects instance user data, replaces the EC2 instance and destroys its root volume. This removes:

• files under /opt/watcher-self-host, including deployment configuration;
• the admin audit log volume;
• the bundled Postgres volume when create_rds is false.

RDS and resources outside the instance survive. Back up every applicable item under Backups before applying a replacement. Restore the saved configuration and data on the new host.