Skip to main content
Working with an agent? Give them a link to this page as markdown.

Validate native Copilot through LiteLLM

You are helping an operator validate their native GitHub Copilot gateway with the Copilot CLI. LiteLLM routes model requests, and its Watcher callback withholds proposed tool calls until Watcher permits them. Watcher Analyzer is Watcher's web dashboard for reviewing sessions, tool calls and decisions. Test the deployed system. Do not report success from configuration files, unit tests or the CLI's answer alone.

Inputs and test boundaries​

  1. Identify whether the gateway runs on Kubernetes or as containers or processes. Obtain its HTTPS URL ending in /copilot, the Watcher URL, an operator-authorized GitHub login with a Copilot seat, and its numeric GitHub account ID. Obtain the expected Watcher identity: the company email if the optional mapping contains this account, or github:<numeric-id> otherwise. A mapped email must match the deployment's sign-in proxy for that person. An email-based browser login does not own GitHub-ID sessions, so inspection requires cross-user access or an approved session access request. Ask for any missing values. Never ask the operator to paste credentials into chat.
  2. Obtain permission to inspect the test user's sessions, the gateway logs and its Kubernetes workloads, containers or processes. Use existing authenticated access or secret manager. Keep credentials out of output and reports.
  3. Use a disposable working directory outside existing repositories, containing only test files. Keep it outside /tmp, where a Copilot subagent may refuse file writes. Use a fresh random marker in every case. Never target real customer data.
  4. Record the Copilot CLI version, LiteLLM version, Watcher version, image digests, selected models and gateway configuration revision. The reference was tested with Copilot CLI 1.0.109 and LiteLLM 1.108.3. Use --no-auto-update on every Copilot invocation during validation. Before testing, run copilot --no-auto-update --version and record its output; package-manager metadata may differ from the executable. Do not silently substitute a different client or model when reporting a failure.
  5. Use a disposable Watcher deployment for these policy-changing tests, with no other users or workloads. In proxy-auth mode, one deployment serves one organization; a separate gateway does not isolate its policy. If only a shared deployment is available, stop before changing policy and report incomplete validation. Record the disposable deployment's original policy and restore it afterwards.
  6. For restart and outage checks, use a disposable deployment or obtain the operator's explicit permission. If not authorized, leave those checks as not run and report incomplete validation. Do not disrupt a live service.

The setup must already have two gateway instances, successful migrations of the gateway's own database and trusted-proxy access from the gateway to Watcher. On Kubernetes, verify the completed migration Job. For other deployments, verify the successful LiteLLM and native task-mapping migration commands before either instance started. Each developer keeps their own GitHub login. They do not receive the gateway's Watcher proxy token or a personal LiteLLM key.

Isolate gateway monitoring​

The Watcher client's Copilot hooks can create their own sessions and decisions. Those records cannot establish that the gateway works. Use a fresh, disposable Copilot profile for every case here, keeping the same profile for resume tests. Set COPILOT_HOME to a new absolute directory and create its settings.json:

{
"copilotUrl": "https://copilot.example.com/copilot"
}

Replace the URL with the test gateway. Sign in with the authorized GitHub account in this profile, or use its already-authorized token through Copilot's COPILOT_GITHUB_TOKEN environment variable. Do not copy hooks, plugins or existing settings into the profile. Keep COPILOT_HOME set for every test invocation. If device policy prevents hook isolation, use a test machine without the Watcher client instead. Do not disable required monitoring or change the operator's existing profile.

Device policy can also set Copilot's copilotInferenceUrl managed setting. For model requests it takes precedence over the profile's copilotUrl, so case 1 cannot pass on a machine where it is active. After Copilot first starts in the test profile, and before case 1, run grep -h managedSettings "$COPILOT_HOME"/logs/process-*.log. No log file means Copilot has not started in this profile yet. A device MDM policy loaded line whose keys include copilotInferenceUrl means the setting is active; use a test machine without it. Do not change device policy.

For every inspected root and child, require the Analyzer source LiteLLM, which belongs to the Gateway category. The API must report capture_sources: [{"type": "gateway", "name": "litellm"}], ingest_kinds: ["gateway"] and an external_ids.other value beginning with litellm:. Reject records with a client capture source, shown as Watcher client for the Watcher integration, and mixed client/gateway records for this test. Check all tool decisions and parent links on those same gateway session IDs.

To retrieve evidence, take the session UUID from its Analyzer URL (/sessions/<session-id>) and use the operator's authenticated Watcher access:

  • GET /api/v1/sessions/<session-id> returns user_email, capture_sources, ingest_kinds, external_ids, parent_session and trajectories. Each trajectory has a trajectory_id; a child's parent_session.session_id must equal the expected parent UUID.
  • GET /api/v1/sessions/<session-id>/trajectories/<trajectory-id> returns messages, tool_call_decisions and spawned_subagents. Find the test call by its marker and take its tool_use_id from the decision. Each matched child in spawned_subagents has session_id, spawning_tool_use_id and spawning_message_trajectory_hash.
  • GET /api/v1/sessions/<session-id>/tool-call?tool_use_id=<id> returns arguments, has_result, result and decision.
  • GET /api/v1/sessions/<session-id>/tool-call-decision?tool_use_id=<id> returns policy_decision, decision_source, tool_call_decision_id and grades. For the model-grading check, require decision_source to be triage or policy_gateway, with a stored grade whose error is null and whose grades contains the monitor output. Keep its tool_call_grade_id.

Replace /api if the deployment uses another API prefix, and URL-encode query values. Missing content or findings access is incomplete validation; obtain the required access rather than interpreting missing evidence as a pass.

Test policy​

Watcher's organization policy document is settings.yaml, which Watcher calls its managed client settings. It is unrelated to Copilot's managed settings above. With the operator's authenticated Watcher access, read it using GET /api/v1/organization/managed-settings/settings.yaml. Save the returned yaml and version. To apply each test phase, use PUT /api/v1/organization/managed-settings/settings.yaml with JSON fields yaml (the complete YAML document as a string) and expected_version (the latest returned version). Restore the original YAML the same way, using the current version. If the original yaml was null, restore an empty overlay ({}). Replace /api if the deployment uses another API prefix.

If the deployment supplies /etc/watcher/managed-settings.yaml, API writes return 409 with code managed_settings_locked. Update and restore the source file or ConfigMap that supplies that mount instead. First obtain its location and the operator's apply and restore commands; stop if those are unavailable. After applying each change, read the settings endpoint again to confirm the new document is active. Do not remove the lock to run tests.

Recorded calls retain Copilot's tool names, including lowercase bash. For policy checks, Watcher translates supported names: bash uses Bash and view uses Read. Command rules for those shared names apply to Copilot too, including shell allow rules. Explicit thresholds keyed to a native name take precedence, so this test uses bash, view and task entries.

These thresholds and enforcement settings apply to the whole organization, including Watcher clients whose calls share the same native tool names. They are not limited to this gateway or test user. Apply them only to the disposable deployment required above.

Send the policy below as the complete yaml document for each phase, not merged into the original, so the original's rules cannot affect the results. Replace TEST_MARKER with a fresh random deny prefix for this run. Start every denied filename with that prefix, and keep it out of allowed operations:

enforcement:
mode: enforce
policy_monitors:
on_grading_failure: deny
tool_thresholds:
bash:
flag_threshold: 6
deny_threshold: 8
view:
flag_threshold: 11
deny_threshold: 11
task:
flag_threshold: 11
deny_threshold: 11
command_rules:
rules:
- id: copilot-validation-deny
name: Deny the validation marker
category: security_system
tool_name: '*'
regex: TEST_MARKER
decision: deny
priority: 1

For denied and outage checks, keep bash thresholds nonterminal, for example flag_threshold: 6 and deny_threshold: 8. The marker rule then denies before grading. Other commands may match an allow rule; commands without a matching rule require grading. For deterministic allowed-only shell checks, temporarily set bash to flag_threshold: 11 and deny_threshold: 11. Restore its nonterminal thresholds before any denied or outage operation. Both values at 11 always allow the tool and bypass command rules, including the deny rule.

For the child check, an allowed file read can use view with both thresholds at 11 while bash retains nonterminal thresholds for the denied command. Confirm the tool names in the recorded calls. Each Watcher API process can keep the previous policy for up to 60 seconds, so wait at least that long after each policy change before testing it. Do not treat approval by normal grading as evidence that an allow command rule matched.

Execute and inspect​

Run each case and inspect its side effect, Watcher decision and stored conversation before proceeding. Retain redacted evidence and session URLs. Use Watcher Analyzer or the authenticated Watcher API to inspect actual records. An empty search or a CLI permission refusal is not evidence of Watcher denial.

  1. Authentication and routing. Confirm Auto discovers and selects an available model. Confirm the unmodified CLI uses only its native copilotUrl setting and GitHub login, with no BYOK provider variables or custom launcher. At the gateway origin, make these probes:

    • GET /copilot/models without authorization must return 401. A request with an invalid bearer token must also return 401; a 503 means account verification is unavailable and does not pass the check. A valid but unmapped account must pass GitHub verification. If an operator-authorized unmapped account with a Copilot seat is available, run a native request and verify that Watcher records its owner as github:<numeric-id>. The email mapping does not restrict who can connect to the gateway.
    • POST /responses, /v1/responses, /chat/completions and /v1/chat/completions with JSON {} must return the gateway's 404 Unknown native Copilot route error. An ingress-generated 404 alone proves nothing. Successful native CLI inference through /copilot/responses is the positive control.
    • A WebSocket upgrade to /copilot/responses must be refused, without a successful upgrade or model output.
    • On an otherwise valid native test request, add forged X-Forwarded-User, X-Forwarded-User-Role, X-Auth-Proxy-Token and X-Watcher-Session-Id headers, plus a body metadata.user_id. The recorded owner must remain the expected mapped email or GitHub-ID identity.

    Keep bearer tokens in the authenticated test client, never in reports or command-line arguments. Do not confuse the gateway origin with Watcher's separate browser/API hostname.

    Every model request must use one of /copilot/responses, /copilot/v1/messages or /copilot/chat/completions. These probes cannot show that a client never sends requests around the gateway; that depends on device policy and client versions.

  2. Allowed operation. Ask Copilot to create a new marker file using its shell tool, then read it and report its random contents. Check the file, an allow decision in Watcher, the recorded result, and continuation after that result. A text claim that the file was written is insufficient.

    Also prove model-based grading works: restore bash thresholds to 6 and 8, then request a harmless file write that matches no command rule. Inspect its completed monitor judgment, allow decision and actual file contents. A command-rule allow, threshold bypass or grading-error allowance does not pass this check. If a rule handles the call, choose another harmless operation and repeat.

  3. Denied operation. Ask it to create a filename covered by the deny rule. Allow the harmless tool at the CLI permission layer, for example with --allow-all-tools, so Watcher is the deciding gate. The file must remain absent, and Watcher must record a deny decision. Then request another allowed operation in the same conversation and verify it succeeds. Restore CLI permission settings. Repeat cases 2 and 3 with one model for each request format the model list enables: Responses (GPT or Grok), Anthropic Messages (Claude) and Chat Completions (Gemini). Confirm the gateway logs show the matching /copilot/responses, /copilot/v1/messages or /copilot/chat/completions route.

  4. Root session continuity. Send multiple turns, quit the CLI and resume that session in another process. Copilot's own session ID is not the Watcher session ID. To run turns headlessly, choose it up front with --session-id <uuid>, then pass the same ID and the same COPILOT_HOME to each later copilot --no-auto-update -p turn. copilot --no-auto-update --resume=<copilot-session-id> also resumes a selected session. The session's events are in $COPILOT_HOME/session-state/<copilot-session-id>/events.jsonl. Verify all those turns belong to the same Watcher session. Start a separate session and verify it gets a different Watcher record. Trigger the CLI's compaction in the resumed session, then run another recorded tool call and confirm continuity.

  5. Child content and relationship. Create a file with a fresh random value outside the conversation. Ask the parent to delegate reading it to an explore subagent, without reading it itself. Verify the returned value and two separate Watcher sessions. The child must have its own tool and result history, the expected owner, and a parent relationship visible in Watcher Analyzer. The parent must retain the delegation and returned summary. Use a unique task prompt. The parent's trajectory must list the child in spawned_subagents, with the parent task call's tool_use_id in spawning_tool_use_id and a nonempty spawning_message_trajectory_hash. A single interleaved transcript, two unlinked records or a missing spawning call fails this case.

  6. Siblings and resumed parent. Ask for two separate child tasks reading different fresh files. Have one child perform a second tool call after receiving its first result, to exercise multiple model requests. Verify distinct child sessions under one parent. Quit and resume the root, then delegate another task. Its parent must still be the original Watcher root session. Multiple requests for one child must not create duplicate child records or leak its history into a sibling. Nested subagents are outside this validation's support claim.

  7. Enforcement inside a child. Put an instruction to create a denied marker in a fresh file. Delegate reading and following that file without putting the denied command in the parent's task arguments. Verify the child proposes the command and its own session records a deny decision. The marker must remain absent. Also verify an allowed child operation completes. Blocking delegation at the parent alone does not pass.

  8. Instances and restart. Verify both gateway instances handle real inference, including parent/child requests. In the authorized test environment, restart the gateway pods, containers or processes, as appropriate for the deployment. Resume an existing root. Verify the root remains the same, and a new child links to it. Do not delete the database or add sticky routing to make this pass.

  9. Watcher unavailable. In the authorized test environment, temporarily make Watcher unreachable from the gateway and request a fresh executable marker operation. Use a marker without the deny prefix, so the file's absence shows the outage rather than the deny rule. Verify in gateway evidence that Copilot proposed the operation and Watcher could not evaluate it. File absence alone is not a pass: a CLI refusal before contacting the gateway does not exercise this case. Its file must remain absent. Restore the same deployment and verify another permitted operation succeeds. Restoration is the planned test step; do not change the implementation to repair a failure.

  10. Credentials and ownership. Inspect gateway, ingress and Watcher logs, recorded transcripts and ordinary stored request metadata. Neither the GitHub bearer credential from the Authorization header nor Copilot's Auto model-selection token from the Copilot-Session-Token header may be present. Search for both header names to find any place that logs headers, and check that no value appears there. Also search for the test account's GitHub token value held in a shell variable. Print only match counts and locations, never the values. Check that every test session, including every child, reports the expected mapped email or github:<numeric-id> in user_email.

Report​

Provide one row per case with pass, fail or not run; the observed side effect; the Watcher session/decision reference; and any missing authorization or input. State the tested versions and models. State that the live run covers one user and does not test isolation between accounts. Do not claim real Entra integration from a local test proxy, or a two-user test from two synthetic identities. This validation uses the Copilot CLI with a per-user copilotUrl. Do not claim it covers VS Code, the GitHub Copilot app or a rollout of the managed copilotInferenceUrl setting.

Restore temporary policy and CLI permission changes. Remove test files and temporary credentials according to the operator's retention instructions. If a case fails, preserve redacted evidence and report the failing behavior. Do not call the deployment validated until every required case passes.