--- description: >- Back up, restore, audit, and collect diagnostics from a self-hosted Watcher deployment. --- > Part of the Watcher docs. Index of all pages, with summaries: https://watcher-docs.apolloresearch.ai/llms.txt > Before upgrading Watcher, read the release pages and full changelogs for every version crossed: https://watcher-docs.apolloresearch.ai/releases/index.md > Found an issue with this page? Report it: https://github.com/ApolloResearch/watcher-bin/issues/new?title=Docs%20feedback%3A%20Operations&body=Page%3A%20Operations%0AURL%3A%20https%3A%2F%2Fwatcher-docs.apolloresearch.ai%2Fself-hosting%2Foperations%2F%0A%0A # Operations This page covers backups and restore, [database timeouts](#database-timeouts), support bundles, the [admin audit log](#admin-audit-log), and prompt-driven operations. Examples include creating a [grader](../concepts/graders.md). The commands below use Docker Compose. For Kubernetes storage and backup requirements, see [Scale and operate](kubernetes-infrastructure.md#scale-and-operate). ## Backups Watcher does not schedule backups. Back up: - the Postgres database; - deployment-owned configuration, including `.env`, `server-config.yaml`, Compose overrides, certificates, and proxy configuration; - the [admin audit log](#admin-audit-log) volume, if the log is enabled. Copy the audit log out through the container, and back the same way when restoring onto a new host: ```sh docker compose cp xylon-api:/var/log/watcher/. ./watcher-audit-backup docker compose cp ./watcher-audit-backup/. xylon-api:/var/log/watcher ``` For managed Postgres, use your database provider's backup and restore features. For bundled Postgres, schedule a dump and copy it off the host: ```sh docker compose exec -T xylon-postgres pg_dump -U xylon -Fc xylon \ > watcher-$(date +%F).dump ``` Database dumps and audit logs contain sensitive data. Apply the same access controls and encryption used for the deployment. ## Restoring Restore into the same or a newer Watcher release than the one that wrote the dump. Database migrations only run forward. These steps restore the database. When recovering a whole host (for example, after [replacing the instance](aws-infrastructure.md#replace-the-instance)), put the deployment-owned configuration back beside `docker-compose.yml` before step 1. Copy the audit log back after step 3, once `xylon-api` is running. 1. Stop the stack, then start only bundled Postgres: ```sh docker compose stop docker compose start xylon-postgres ``` 2. Drop and recreate the database, then restore into the empty database: ```sh docker compose exec xylon-postgres dropdb --force -U xylon xylon docker compose exec xylon-postgres createdb -U xylon xylon docker compose exec -T xylon-postgres \ pg_restore -U xylon -d xylon < watcher-.dump ``` Do not restore over the existing database. Objects created after the dump can survive. They can cause the next migration to fail. 3. Start the stack. The bootstrap container applies pending migrations: ```sh docker compose up -d ``` Confirm that bootstrap exits successfully. Then open the [Analyzer](../concepts/analyzer.md). Check recent [sessions](../concepts/sessions-and-trajectories.md) and grades. Test restores regularly. For managed Postgres, stop the stack, restore through your provider, update `DATABASE_URL` if the endpoint changed, and start the stack again. ## Database timeouts The API sets two server-side PostgreSQL guards on every database connection. These prevent any single query or abandoned transaction from holding a connection and its memory indefinitely: - `WATCHER_DB_STATEMENT_TIMEOUT` (default 120): maximum runtime of one SQL statement, in seconds. - `WATCHER_DB_IDLE_IN_TRANSACTION_TIMEOUT` (default 900): maximum time a connection may sit inside an open transaction with no statement running, in seconds. Grading holds a transaction open while it waits on model providers. This value must stay above your grading pass budget. The default has headroom above the standard budgets. Set either variable in `.env` for Docker Compose deployments; the Helm chart exposes the same values as `api.dbStatementTimeout` and `api.dbIdleInTransactionTimeout`. A value of 0 disables that guard. The defaults leave generous headroom above the slowest queries Watcher runs. These are analytics views and grading selection. Most deployments never need to change them. Raise `WATCHER_DB_STATEMENT_TIMEOUT` if a very large deployment sees dashboard, usage, or export queries fail. The error will be `canceling statement due to statement timeout`. Lower either value for tighter protection against runaway queries. ## Admin audit log The API writes one JSON line for each audited API request. This includes: - Reads of other users' session data - Session lists and searches - Session-content scrubs - User-directory access - Organization settings changes The retention worker writes the same kind of line for every session an enabled retention policy scrubs. Each record identifies the actor and affected resources. Records do not contain session content. Administrator actions carry the actor's email. Automatic retention uses `retention-policy` as the actor and `system` as the authentication method. ### Enabling and disabling The audit log is on by default. It is stored in the `xylon_audit_log` Docker volume. The file is created on the first audited request. To verify it is live, open the session list in the [Analyzer](../concepts/analyzer.md). List requests are always audited. Then run: ```sh docker compose exec xylon-api tail -n 5 /var/log/watcher/admin-audit.log ``` A fresh entry confirms it. If the file is missing, trigger an audited request before troubleshooting the log. To disable it, set the variable to an empty value in `.env` and restart: ```sh echo 'WATCHER_ADMIN_AUDIT_LOG_FILE=' >> .env docker compose up -d ``` Audit-log write failures do not block API requests. Check API logs for errors. ### Reviewing and retaining it Each line is a JSON object, so standard log tools work directly: ```sh docker compose exec xylon-api tail -n 100 /var/log/watcher/admin-audit.log ``` The log contains personal data. It stays on the deployment host. It is excluded from support bundles. Watcher does not rotate or prune it. Apply your normal log retention policy. Copy any archives off the Docker volume. Lines are 300 to 600 bytes. Human triage activity keeps the file small. Enabling a retention policy on a deployment that already holds data is the exception. The first sweep can write one entry per eligible historical session. Check the volume after enabling a policy. Rotate accordingly. ## Collecting a support bundle Run the bundled diagnostics script from the deployment directory: ```sh ./support-bundle.sh ``` The archive contains service logs, redacted configuration, image tags, and aggregate database health information. It does not collect session or message content. Secret redaction is best effort. Review every file before sharing the archive. Run `./support-bundle.sh --help` for available options. ## Prompt-driven operations Some operations are driven by pasting a prompt into a coding assistant. The assistant must have access to your deployment. This is the same pattern as [guided setup](setup.md): - [Create or customize a grader](prompts/custom-grader.md): walks an assistant through authoring a grader on your deployment. Optionally, it can roll out the grader as the organization's live blocking policy.