Operations

This page covers backups and restore, support bundles, the admin audit log, and prompt-driven operations such as creating a grader.

Backups

Watcher does not schedule backups. Back up:

• the Postgres database;
• deployment-owned configuration, including .env, server-config.yaml, Compose overrides, certificates, and proxy configuration;
• the admin audit log volume, if the log is enabled.

Copy the audit log out through the container, and back the same way when restoring onto a new host:

docker compose cp xylon-api:/var/log/watcher/. ./watcher-audit-backup
docker compose cp ./watcher-audit-backup/. xylon-api:/var/log/watcher

Use your database provider's backup and restore features for managed Postgres. For bundled Postgres, schedule a dump and copy it off the host:

docker compose exec -T xylon-postgres pg_dump -U xylon -Fc xylon \
  > watcher-$(date +%F).dump

Database dumps and audit logs contain sensitive data. Apply the same access controls and encryption used for the deployment.

Restoring

Restore into the same or a newer Watcher release than the one that wrote the dump. Database migrations only run forward.

These steps restore the database. When recovering a whole host, for example after replacing the instance, put the deployment-owned configuration back beside docker-compose.yml before step 1, and copy the audit log back after step 3, once xylon-api is running.

1. Stop the stack, then start only bundled Postgres:

   docker compose stop
   docker compose start xylon-postgres

2. Drop and recreate the database, then restore into the empty database:

   docker compose exec xylon-postgres dropdb --force -U xylon xylon
   docker compose exec xylon-postgres createdb -U xylon xylon
   docker compose exec -T xylon-postgres \
     pg_restore -U xylon -d xylon < watcher-<date>.dump

   Do not restore over the existing database. Objects created after the dump can survive and cause the next migration to fail.

3. Start the stack. The bootstrap container applies pending migrations:

   docker compose up -d

Confirm that bootstrap exits successfully, then open the Analyzer and check recent sessions and grades. Test restores regularly.

For managed Postgres, stop the stack, restore through your provider, update DATABASE_URL if the endpoint changed, and start the stack again.

Admin audit log

The API writes one JSON line for audited API requests, including reads of other users' session data, session lists and searches, session-content scrubs, user-directory access, and organization settings changes. The retention worker writes the same kind of line for every session an enabled retention policy scrubs. Each record identifies the actor and affected resources, but does not contain session content. Administrator actions carry the actor's email; automatic retention uses retention-policy as the actor and system as the authentication method.

Enabling and disabling

The audit log is on by default and stored in the xylon_audit_log Docker volume. The file is created on the first audited request.

To verify it is live, open the session list in the Analyzer (list requests are always audited), then:

docker compose exec xylon-api tail -n 5 /var/log/watcher/admin-audit.log

A fresh entry confirms it. If the file is missing, trigger an audited request before troubleshooting the log.

To disable it, set the variable to an empty value in .env and restart:

echo 'WATCHER_ADMIN_AUDIT_LOG_FILE=' >> .env
docker compose up -d

Audit-log write failures do not block API requests. Check API logs for errors.

Reviewing and retaining it

Each line is a JSON object, so standard log tools work directly:

docker compose exec xylon-api tail -n 100 /var/log/watcher/admin-audit.log

The log contains personal data, stays on the deployment host, and is excluded from support bundles. Watcher does not rotate or prune it. Apply your normal log retention policy and copy any archives off the Docker volume.

Lines are 300 to 600 bytes, and human triage activity keeps the file small. Enabling a retention policy on a deployment that already holds data is the exception: the first sweep can write one entry per eligible historical session. Check the volume after enabling a policy and rotate accordingly.

Collecting a support bundle

Run the bundled diagnostics script from the deployment directory:

./support-bundle.sh

The archive contains service logs, redacted configuration, image tags, and aggregate database health information. It does not collect session or message content. Secret redaction is best effort, so review every file before sharing the archive.

Run ./support-bundle.sh --help for available options.

Prompt-driven operations

Some operations are driven by pasting a prompt into a coding assistant that has access to your deployment, the same pattern as guided setup:

• Create or customize a grader: walks an assistant through authoring a grader on your deployment and, optionally, rolling it out as the organization's live blocking policy.