Operations

This page covers backups and restore, database timeouts, support bundles, the admin audit log, and prompt-driven operations. Examples include creating a grader.

The commands below use Docker Compose. For Kubernetes storage and backup requirements, see Scale and operate.

Backups

Watcher does not schedule backups. Back up:

• the Postgres database;
• deployment-owned configuration, including .env, server-config.yaml, Compose overrides, certificates, and proxy configuration;
• the admin audit log volume, if the log is enabled.

Copy the audit log out through the container, and back the same way when restoring onto a new host:

docker compose cp xylon-api:/var/log/watcher/. ./watcher-audit-backup
docker compose cp ./watcher-audit-backup/. xylon-api:/var/log/watcher

For managed Postgres, use your database provider's backup and restore features. For bundled Postgres, schedule a dump and copy it off the host:

docker compose exec -T xylon-postgres pg_dump -U xylon -Fc xylon \
  > watcher-$(date +%F).dump

Database dumps and audit logs contain sensitive data. Apply the same access controls and encryption used for the deployment.

Restoring

Restore into the same or a newer Watcher release than the one that wrote the dump. Database migrations only run forward.

These steps restore the database. When recovering a whole host (for example, after replacing the instance), put the deployment-owned configuration back beside docker-compose.yml before step

1. Copy the audit log back after step 3, once xylon-api is running.

2. Stop the stack, then start only bundled Postgres:

   docker compose stop
   docker compose start xylon-postgres

3. Drop and recreate the database, then restore into the empty database:

   docker compose exec xylon-postgres dropdb --force -U xylon xylon
   docker compose exec xylon-postgres createdb -U xylon xylon
   docker compose exec -T xylon-postgres \
     pg_restore -U xylon -d xylon < watcher-<date>.dump

   Do not restore over the existing database. Objects created after the dump can survive. They can cause the next migration to fail.

4. Start the stack. The bootstrap container applies pending migrations:

   docker compose up -d

Confirm that bootstrap exits successfully. Then open the Analyzer. Check recent sessions and grades. Test restores regularly.

For managed Postgres, stop the stack, restore through your provider, update DATABASE_URL if the endpoint changed, and start the stack again.

Database timeouts

The API sets two server-side PostgreSQL guards on every database connection. These prevent any single query or abandoned transaction from holding a connection and its memory indefinitely:

• WATCHER_DB_STATEMENT_TIMEOUT (default 120): maximum runtime of one SQL statement, in seconds.
• WATCHER_DB_IDLE_IN_TRANSACTION_TIMEOUT (default 900): maximum time a connection may sit inside an open transaction with no statement running, in seconds. Grading holds a transaction open while it waits on model providers. This value must stay above your grading pass budget. The default has headroom above the standard budgets.

Set either variable in .env for Docker Compose deployments; the Helm chart exposes the same values as api.dbStatementTimeout and api.dbIdleInTransactionTimeout. A value of 0 disables that guard.

The defaults leave generous headroom above the slowest queries Watcher runs. These are analytics views and grading selection. Most deployments never need to change them. Raise WATCHER_DB_STATEMENT_TIMEOUT if a very large deployment sees dashboard, usage, or export queries fail. The error will be canceling statement due to statement timeout. Lower either value for tighter protection against runaway queries.

Admin audit log

The API writes one JSON line for each audited API request. This includes:

• Reads of other users' session data
• Session lists and searches
• Session-content scrubs
• Session-content access requests
• User-directory access
• Organization settings changes

The retention worker writes the same kind of line for every session an enabled retention policy scrubs. Each record identifies the actor and affected resources. Records do not contain session content. Administrator actions carry the actor's email. Automatic retention uses retention-policy as the actor and system as the authentication method.

Access-request decisions and revocations include target_user_id (the session owner), requesting_user_id, access_request_id, and valid_until for approvals and revocations. Session reads include content_access_basis: permission, approval, or none for findings-only reads. Approval-based reads include access_request_id. Routine reads of your own sessions remain excluded.

Enabling and disabling

The audit log is on by default. It is stored in the xylon_audit_log Docker volume. The file is created on the first audited request.

To verify it is live, open the session list in the Analyzer. List requests are always audited. Then run:

docker compose exec xylon-api tail -n 5 /var/log/watcher/admin-audit.log

A fresh entry confirms it. If the file is missing, trigger an audited request before troubleshooting the log.

To disable it, set the variable to an empty value in .env and restart:

echo 'WATCHER_ADMIN_AUDIT_LOG_FILE=' >> .env
docker compose up -d

Audit-log write failures do not block API requests. Check API logs for errors.

Reviewing and retaining it

Each line is a JSON object, so standard log tools work directly:

docker compose exec xylon-api tail -n 100 /var/log/watcher/admin-audit.log

The log contains personal data. It stays on the deployment host. It is excluded from support bundles. Watcher does not rotate or prune it. Apply your normal log retention policy. Copy any archives off the Docker volume.

Lines are 300 to 600 bytes. Human triage activity keeps the file small. Enabling a retention policy on a deployment that already holds data is the exception. The first sweep can write one entry per eligible historical session. Check the volume after enabling a policy. Rotate accordingly.

Collecting a support bundle

Run the bundled diagnostics script from the deployment directory:

./support-bundle.sh

The archive contains service logs, redacted configuration, image tags, and aggregate database health information. It does not collect session or message content. Secret redaction is best effort. Review every file before sharing the archive.

Run ./support-bundle.sh --help for available options.

Prompt-driven operations

Some operations are driven by pasting a prompt into a coding assistant. The assistant must have access to your deployment. This is the same pattern as guided setup:

• Create or customize a grader: walks an assistant through authoring a grader on your deployment. Optionally, it can roll out the grader as the organization's live blocking policy.