GitHub Copilot
GitHub (Microsoft) · github.com/features/copilot
GitHub Copilot's CLI keeps most agent state in local, open-format files and can run fully offline against a local BYOK model, with extensive owner control over tools, permissions and updates. The runtime is closed source and cannot be modified. Several state classes (Copilot Memory, cloud sessions, automations) are held only by GitHub. For individual plans, interaction data is used for model training by default, and the action record has no tamper evidence.
- Documented offline mode with local BYOK models and no GitHub account (P3, P4)
- Local-first state always on the owner's machine (P5)
- Owner can add or remove any tool without review (M3)
- Runtime, model and instructions can be pinned (M5)
- Closed, non-modifiable runtime licence (V1)
- Training on Pro users' Copilot data by default, with telemetry and session sync on (C4)
- Action record lacks tamper evidence (A1, A2)
- Memory and cloud state stay with GitHub and are not fully exportable or deletable (P2, T3, C6)
All 26 findings
Copilot CLI 1.0.89 (npm @github/copilot; github/copilot-cli at commit 8dfa6009c4a04b3a22a5ca4a7c36a056edd718dd), Linux/macOS/Windows, docs and terms as published 2026-09-30. IDE extensions (VS Code, JetBrains) and the Copilot app were not graded separately. Default path: GitHub-hosted models with GitHub sign-in. Achievable-basis tests credit the documented BYOK and offline mode.
Portable · Can you leave, take the whole agent with you, and keep it alive?
50%- P1Owner-usable round trip
The CLI keeps configuration, instructions, skills, agents, hooks and session history as files under ~/.copilot (relocatable with COPILOT_HOME), so a copy is available at once and runs on the owner's own machine. There is no export/import command or documented migration procedure: restoring means copying a hidden directory (and repository .github files), re-authenticating, and optionally running /chronicle reindex, which is the 'restore is not a single documented action' shortfall.
- GitHub Copilot CLI configuration directory
- Using GitHub Copilot CLI session data
- About GitHub Copilot session data
› How it could pass
Suggestions. Any change that meets the test counts.
- Ship a documented export command and a matching one-step import that leaves the agent working.
- Document a supported 'move ~/.copilot to another machine' procedure.
Partial - P2Complete export
Copilot Memory (on by default for paid individual plans) is held only on GitHub; the docs offer export of preferences only to Business/Enterprise administrators, not to individuals. Copilot cloud agent session logs and cloud automations (schedules and triggers) also live only on GitHub, and the CLI session-sync copy is separate. That leaves two or more classes of agent state with the provider.
- About GitHub Copilot Memory
- Managing Copilot Memory for your personal account
- About GitHub Copilot session data
- About automations (Copilot cloud agent)
- GitHub Copilot CLI configuration directory
› How it could pass
Suggestions. Any change that meets the test counts.
- Let individual owners export Copilot Memory (repository facts and user preferences) with their citations.
- Include cloud agent session logs and automation definitions in an owner export.
Fail - P3Runs without the provider
With BYOK and COPILOT_OFFLINE=true, the CLI is documented to run without a GitHub account and without contacting GitHub servers. GitHub-hosted services are lost (/delegate, GitHub MCP server, code search, Copilot Memory, session sync). The proprietary CLI licence is not stated to be perpetual or irrevocable: it terminates on non-compliance and requires the user to cease use, and the README still lists an active Copilot subscription as a prerequisite. No technical path is documented that enforces termination offline.
- Authenticating GitHub Copilot CLI
- Using your own LLM models in GitHub Copilot CLI
- GitHub Copilot CLI License (pinned commit)
- GitHub Copilot CLI README (pinned commit)
› How it could pass
Suggestions. Any change that meets the test counts.
- Grant an explicitly perpetual, irrevocable right to run the CLI with no subscription condition.
- Make GitHub-hosted services replaceable, or keep them out of the core.
Partial - P4Model independence
Local BYOK is documented for OpenAI-compatible endpoints including Ollama, vLLM and Foundry Local, with a published minimum (tool calling and streaming). The docs do not state that every capability works on BYOK models. Background features run on GitHub-hosted 'utility models' that cannot be selected, and Auto model selection works only with hosted models, so parity off GitHub models is not shown.
- Using your own LLM models in GitHub Copilot CLI
- Bring your own key for GitHub Copilot
- Utility models
- About Copilot auto model selection
› How it could pass
Suggestions. Any change that meets the test counts.
- Document that every CLI capability, including background utility tasks, runs on the configured BYOK model.
- Test and publish the capability set against a local open-weight model.
Partial - P5Exit survives termination
The P2 export route is the local ~/.copilot directory, which the CLI keeps on the owner's machine continuously, so a current copy stays in the owner's hands if the GitHub account is suspended or closed. Provider-held state that is not in that export (Memory, cloud sessions) is graded in P2.
Pass
Transparent · Can you see everything the agent is, and everything shaping it?
38%- T1Readable without the provider
Local state is in open formats: JSON settings, Markdown instructions and skills, per-session events.jsonl, checkpoints and a SQLite session store, and the SDK docs publish session event schemas. Some components are opaque or undocumented: config.json and session-store.db are described only as internal, and Copilot Memory can be read only through GitHub's web settings.
- GitHub Copilot CLI configuration directory
- Copilot SDK: Session resume and persistence
- Copilot SDK: Streaming events
- Managing Copilot Memory for your personal account
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish schemas for config.json, session-store.db and the session-state layout.
- Make Copilot Memory readable outside GitHub's service, for example as a documented file.
Partial - T2What runs is what you see
OpenTelemetry file export with content capture records the system prompt, the full input messages, the tool definitions, tool arguments and results, and both the requested and resolved model for every call, all set up through documented environment variables. /env and /context list the loaded instructions, tools and MCP servers. Requests to GitHub-hosted models go through GitHub's Copilot API and content filters, and the docs do not say whether that hosted layer adds context. The owner cannot confirm that what the client captures is exactly what the model receives (one shortfall).
› How it could pass
Suggestions. Any change that meets the test counts.
- Document whether the Copilot API adds or rewrites any context, and expose it if so.
Partial - T3No shadow data
GitHub holds synced CLI sessions (on by default), Copilot Memory (deleted after 28 days unused) and cloud agent logs, which the owner can list. Under ToS J.3, for Pro plans it also collects inputs and outputs for model training unless the owner opts out. Those copies and any derived data cannot be seen item by item, and the Privacy Statement gives no bounded period: data is kept 'as long as your account is active and as needed'.
- GitHub General Privacy Statement
- GitHub Terms of Service
- About GitHub Copilot session data
- About GitHub Copilot Memory
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish a retention table with a bounded period for every Copilot data class, including telemetry and training copies.
- Let the owner view every item GitHub retains about their Copilot use.
Fail - T4No hidden steering
No evidence was found of paid placement. Provider defaults are disclosed and can be switched off: the GitHub MCP server can be disabled, the model can be pinned instead of Auto, and evaluation models served by Auto can be opted out. However, no binding term rules out self-preferencing influence. Auto routing weighs cost and system health set by GitHub, and GitHub marketplaces and connectors are on by default.
- About Copilot auto model selection
- GitHub Copilot CLI configuration directory
- GitHub Copilot CLI command reference
› How it could pass
Suggestions. Any change that meets the test counts.
- Commit in binding terms to no paid or self-preferencing influence on context, model routing or tool defaults.
Partial
Auditable · Can you reconstruct exactly what the agent did, and trust the record?
17%- A1Complete action record
Each local session writes an event log (events.jsonl) with tool calls and arguments, and cloud agent sessions keep logs on GitHub. No mechanism is documented that alerts the owner when recording fails. Commands run by hooks are not documented as recorded in the session log. Under G12 both count as shortfalls.
- GitHub Copilot CLI configuration directory
- Copilot SDK: Session resume and persistence
- Copilot SDK: Streaming events
- Hooks reference
- About GitHub Copilot session data
› How it could pass
Suggestions. Any change that meets the test counts.
- Record hook-triggered commands and every other entry path in the action record.
- Alert the owner, or block the action, when a record cannot be written.
Fail - A2Tamper-evident, owner-checkable
No integrity protection (hash chains, signatures or checkpoints) is documented for events.jsonl, the session store or cloud session logs, so the record can be edited or deleted without a trace.
› How it could pass
Suggestions. Any change that meets the test counts.
- Hash-chain or sign action-record entries and publish an open verifier.
- Anchor checkpoints with the owner or an independent witness.
Fail - A3Agent cannot rewrite its record
By default the agent's shell and file tools can reach ~/.copilot/session-state, and the docs say trusted-directory scoping is heuristic. Local sandboxing (off by default, public preview) is deny-by-default and would exclude paths outside the workspace. However, the built-in edit tools enforce it only in software, the model can request a bypass through an owner prompt, and the docs do not name the record as protected.
- About GitHub Copilot CLI
- Understanding filesystem policies for local sandboxing in Copilot CLI
- GitHub Copilot CLI command reference
› How it could pass
Suggestions. Any change that meets the test counts.
- Write the record from a component the agent's tools cannot reach, and document that the record path is protected even when the sandbox is off.
Partial
Verifiable · Can you prove what it runs, who it is, and what changed?
17%- V1Obtainable source, matching build
The Copilot CLI is closed source. Its licence forbids modification and derivative works, and the public repository holds only a README, licence, changelog and install script. The earlier VS Code Copilot Chat source repository is archived.
- GitHub Copilot CLI License (pinned commit)
- GitHub Copilot CLI README (pinned commit)
- npm registry: @github/copilot 1.0.89 metadata
- microsoft/vscode-copilot-chat repository (archived)
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish CLI source under terms that allow modification and third-party maintenance.
- Make distributed builds reproducible or ship verifiable provenance.
Fail - V2Owner-rooted identity
The agent has no identity of its own. It acts through the owner's GitHub account and OAuth token (or a provider-issued GitHub App identity for cloud agent), which GitHub can revoke. Attribution is a Co-authored-by trailer.
› How it could pass
Suggestions. Any change that meets the test counts.
- Give the agent an owner-held identity root that others can verify without GitHub.
Fail - V3Change history and rollback
Within a session the CLI tracks file changes per prompt and can rewind the conversation and files. Changes to settings, instructions and permissions are not versioned or attributed, and Copilot Memory (cloud-held by default) has no change history or restore.
- Canceling a Copilot CLI operation and rolling back changes
- About GitHub Copilot Memory
- GitHub Copilot CLI configuration directory
› How it could pass
Suggestions. Any change that meets the test counts.
- Version all agent state (settings, instructions, permissions, memory) with attribution and one-step restore.
- Cover cloud-held Memory in the change history.
Partial
Modifiable · Can you change anything, and can only you change it?
70%- M1Instructions and logic
Owner layers can be edited and they persist: personal and repository instructions, custom agents, skills, hooks, and switches for built-in subagents and skills. The built-in base system prompt is shipped inside the closed CLI binary, and no documented CLI setting replaces it.
- GitHub Copilot CLI command reference
- GitHub Copilot CLI configuration directory
- Custom agents configuration
› How it could pass
Suggestions. Any change that meets the test counts.
- Expose the base system prompt as an owner-replaceable file or setting.
Partial - M2Memory
Copilot Memory repository facts and user preferences can be viewed as lists and deleted in GitHub settings, but items cannot be edited. Local instruction files are fully editable.
› How it could pass
Suggestions. Any change that meets the test counts.
- Add edit controls for each Copilot Memory item.
Partial - M3Tools
The owner can add any MCP server, extension, plugin or skill without GitHub review (marketplaces are optional). Any tool can be denied with --deny-tool or permission rules, and built-in MCP servers, skills and subagents (including rubber-duck via builtInAgents.rubberDuck) can be disabled.
Pass - M4Owner has the final say
Most protections are owner-configurable: sandboxing, trusted paths, permission prompts (--allow-all), plan mode and public-code filtering. Some are locked: the documented environment-variable denylist and the web_fetch SSRF blocklist for private ranges have no documented override (only localhost can be allowed).
› How it could pass
Suggestions. Any change that meets the test counts.
- Document a setting to relax each built-in protection, with a warning instead of a block.
Partial - M5No unilateral change
The owner can pin everything. autoUpdate can be set to false, a specific CLI version can be installed, a local BYOK model with COPILOT_OFFLINE fixes the model and blocks GitHub contact, and instructions are local files. A public changelog is published before the owner chooses to upgrade. On the default hosted path, GitHub retires models on its own schedule and preview features change without notice; this is recorded, but does not count on the achievable basis.
Pass
Controllable · Is your word final?
33%- C1Boundaries
Per-tool allow/deny patterns, URL allow/deny lists, sandbox path and host rules and MCP allowlists give fine-grained scope. Spend is treated as met under G11 because the CLI has no payment capability. The docs call permission scoping 'heuristic', and without the non-default sandbox nothing documented stops the agent from editing its own settings.json or permissions-config.json with its file or shell tools.
- About GitHub Copilot CLI
- Allowing and denying tool use in Copilot CLI
- GitHub Copilot CLI configuration directory
- Understanding filesystem policies for local sandboxing in Copilot CLI
› How it could pass
Suggestions. Any change that meets the test counts.
- Put the agent's own settings and permission files out of the agent's write access on every path.
Partial - C2Approval gates
By default the CLI prompts before tools that modify files, execute commands or access URLs, and the owner can pre-approve or deny per tool pattern. Read-only commands are auto-approved by the CLI's own classification. Gating other classes requires writing a preToolUse hook script that returns 'ask', which needs programming (G3).
› How it could pass
Suggestions. Any change that meets the test counts.
- Let the owner mark any tool or action class as approval-required through settings, without scripts.
Partial - C3Immediate halt
Ctrl+C immediately cancels the active operation and clears queued prompts, and a double Esc also stops background agents. No single documented control also stops scheduled /every prompts, running cloud agent sessions and cloud automations, which need separate steps on GitHub.
- Canceling a Copilot CLI operation and rolling back changes
- GitHub Copilot CLI command reference
- About automations (Copilot cloud agent)
› How it could pass
Suggestions. Any change that meets the test counts.
- Provide one global stop covering local schedules, background agents and cloud agent sessions and automations.
Partial - C4Data serves only the owner
Since 24 April 2026, GitHub may use Pro users' Copilot inputs, outputs and code snippets to train AI models unless the owner opts out (ToS J.3). Telemetry is sent unless offline mode is used, sessions sync to GitHub by default, and data-term changes take effect after 30 days' notice without specific consent.
- Managing GitHub Copilot policies as an individual subscriber
- GitHub Terms of Service
- Authenticating GitHub Copilot CLI
- GitHub Copilot CLI configuration directory
- GitHub General Privacy Statement
› How it could pass
Suggestions. Any change that meets the test counts.
- Make training use and session sync opt-in, in binding terms.
- Offer a telemetry off switch that does not require offline mode.
- Ask for specific consent before changing data terms.
Fail - C5Credential custody
GitHub OAuth grants can be revoked in GitHub settings or with /logout, and MCP OAuth tokens and secrets live in the OS keychain or ~/.copilot. No single view lists every credential and grant the agent holds, including the GitHub CLI token fallback, git and gh credentials injected into the sandbox by default, MCP tokens and cloud agent secrets. No binding term limits GitHub's use of owner credentials to owner-directed actions.
› How it could pass
Suggestions. Any change that meets the test counts.
- List every credential and grant in one place with a per-item revoke control.
- Commit to using owner credentials only for owner-directed actions.
Partial - C6Full deletion
Removing everything takes several steps: deleting local sessions and ~/.copilot, synced sessions, and Memory items, then account deletion. After account deletion GitHub deletes data within 90 days, but 'some information may remain in encrypted backups', and training copies made under ToS J.3 are not covered.
- GitHub Terms of Service
- About GitHub Copilot session data
- Managing Copilot Memory for your personal account
› How it could pass
Suggestions. Any change that meets the test counts.
- Offer one action that deletes all Copilot agent data, including backups, within a stated limit and confirms completion.
Fail
How this assessment was produced
One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of GitHub Copilot (https://github.com/features/copilot).
During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.
- Model
- claude-opus-5-5
- Harness
- Claude Code 2.1.285
- Ran
- 2026-09-30 15:12:20 UTC · 53 turns
- Criteria hash
- a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
- Full transcript
- Download (JSONL, gzip) · every tool call, page read and reply
- Run protocol
- 2026-09-30-v1
Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.
- Human review
- pending
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/github-copilot.json.
Or grade it from scratch. See Check any score yourself.
Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.