Skip to content
Agents of Humanity
Public preview. Results are preliminary (criteria v1.0, 30 Sept 2026). Tell us what we got wrong
Autonomous agenthybridMIT (VS Code Copilot Chat extension); Copilot CLI, cloud agent and backend proprietary

GitHub Copilot

GitHub (Microsoft) · github.com/features/copilot

GitHub Copilot's CLI keeps most agent state in local, open-format files and can run fully offline against a local BYOK model, with extensive owner control over tools, permissions and updates. The runtime is closed source and cannot be modified. Several state classes (Copilot Memory, cloud sessions, automations) are held only by GitHub. For individual plans, interaction data is used for model training by default, and the action record has no tamper evidence.

Strengths
  • Documented offline mode with local BYOK models and no GitHub account (P3, P4)
  • Local-first state always on the owner's machine (P5)
  • Owner can add or remove any tool without review (M3)
  • Runtime, model and instructions can be pinned (M5)
Gaps
  • Closed, non-modifiable runtime licence (V1)
  • Training on Pro users' Copilot data by default, with telemetry and session sync on (C4)
  • Action record lacks tamper evidence (A1, A2)
  • Memory and cloud state stay with GitHub and are not fully exportable or deletable (P2, T3, C6)
Evidence

All 26 findings

Copilot CLI 1.0.89 (npm @github/copilot; github/copilot-cli at commit 8dfa6009c4a04b3a22a5ca4a7c36a056edd718dd), Linux/macOS/Windows, docs and terms as published 2026-09-30. IDE extensions (VS Code, JetBrains) and the Copilot app were not graded separately. Default path: GitHub-hosted models with GitHub sign-in. Achievable-basis tests credit the documented BYOK and offline mode.

Portable · Can you leave, take the whole agent with you, and keep it alive?

50%
  • P1
    Owner-usable round trip

    The CLI keeps configuration, instructions, skills, agents, hooks and session history as files under ~/.copilot (relocatable with COPILOT_HOME), so a copy is available at once and runs on the owner's own machine. There is no export/import command or documented migration procedure: restoring means copying a hidden directory (and repository .github files), re-authenticating, and optionally running /chronicle reindex, which is the 'restore is not a single documented action' shortfall.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Ship a documented export command and a matching one-step import that leaves the agent working.
    • Document a supported 'move ~/.copilot to another machine' procedure.
    Partial
  • P2
    Complete export

    Copilot Memory (on by default for paid individual plans) is held only on GitHub; the docs offer export of preferences only to Business/Enterprise administrators, not to individuals. Copilot cloud agent session logs and cloud automations (schedules and triggers) also live only on GitHub, and the CLI session-sync copy is separate. That leaves two or more classes of agent state with the provider.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Let individual owners export Copilot Memory (repository facts and user preferences) with their citations.
    • Include cloud agent session logs and automation definitions in an owner export.
    Fail
  • P3
    Runs without the provider

    With BYOK and COPILOT_OFFLINE=true, the CLI is documented to run without a GitHub account and without contacting GitHub servers. GitHub-hosted services are lost (/delegate, GitHub MCP server, code search, Copilot Memory, session sync). The proprietary CLI licence is not stated to be perpetual or irrevocable: it terminates on non-compliance and requires the user to cease use, and the README still lists an active Copilot subscription as a prerequisite. No technical path is documented that enforces termination offline.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Grant an explicitly perpetual, irrevocable right to run the CLI with no subscription condition.
    • Make GitHub-hosted services replaceable, or keep them out of the core.
    Partial
  • P4
    Model independence

    Local BYOK is documented for OpenAI-compatible endpoints including Ollama, vLLM and Foundry Local, with a published minimum (tool calling and streaming). The docs do not state that every capability works on BYOK models. Background features run on GitHub-hosted 'utility models' that cannot be selected, and Auto model selection works only with hosted models, so parity off GitHub models is not shown.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Document that every CLI capability, including background utility tasks, runs on the configured BYOK model.
    • Test and publish the capability set against a local open-weight model.
    Partial
  • P5
    Exit survives termination

    The P2 export route is the local ~/.copilot directory, which the CLI keeps on the owner's machine continuously, so a current copy stays in the owner's hands if the GitHub account is suspended or closed. Provider-held state that is not in that export (Memory, cloud sessions) is graded in P2.

    Pass

Transparent · Can you see everything the agent is, and everything shaping it?

38%
  • T1
    Readable without the provider

    Local state is in open formats: JSON settings, Markdown instructions and skills, per-session events.jsonl, checkpoints and a SQLite session store, and the SDK docs publish session event schemas. Some components are opaque or undocumented: config.json and session-store.db are described only as internal, and Copilot Memory can be read only through GitHub's web settings.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish schemas for config.json, session-store.db and the session-state layout.
    • Make Copilot Memory readable outside GitHub's service, for example as a documented file.
    Partial
  • T2
    What runs is what you see

    OpenTelemetry file export with content capture records the system prompt, the full input messages, the tool definitions, tool arguments and results, and both the requested and resolved model for every call, all set up through documented environment variables. /env and /context list the loaded instructions, tools and MCP servers. Requests to GitHub-hosted models go through GitHub's Copilot API and content filters, and the docs do not say whether that hosted layer adds context. The owner cannot confirm that what the client captures is exactly what the model receives (one shortfall).

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Document whether the Copilot API adds or rewrites any context, and expose it if so.
    Partial
  • T3
    No shadow data

    GitHub holds synced CLI sessions (on by default), Copilot Memory (deleted after 28 days unused) and cloud agent logs, which the owner can list. Under ToS J.3, for Pro plans it also collects inputs and outputs for model training unless the owner opts out. Those copies and any derived data cannot be seen item by item, and the Privacy Statement gives no bounded period: data is kept 'as long as your account is active and as needed'.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish a retention table with a bounded period for every Copilot data class, including telemetry and training copies.
    • Let the owner view every item GitHub retains about their Copilot use.
    Fail
  • T4
    No hidden steering

    No evidence was found of paid placement. Provider defaults are disclosed and can be switched off: the GitHub MCP server can be disabled, the model can be pinned instead of Auto, and evaluation models served by Auto can be opted out. However, no binding term rules out self-preferencing influence. Auto routing weighs cost and system health set by GitHub, and GitHub marketplaces and connectors are on by default.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Commit in binding terms to no paid or self-preferencing influence on context, model routing or tool defaults.
    Partial

Auditable · Can you reconstruct exactly what the agent did, and trust the record?

17%
  • A1
    Complete action record

    Each local session writes an event log (events.jsonl) with tool calls and arguments, and cloud agent sessions keep logs on GitHub. No mechanism is documented that alerts the owner when recording fails. Commands run by hooks are not documented as recorded in the session log. Under G12 both count as shortfalls.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Record hook-triggered commands and every other entry path in the action record.
    • Alert the owner, or block the action, when a record cannot be written.
    Fail
  • A2
    Tamper-evident, owner-checkable

    No integrity protection (hash chains, signatures or checkpoints) is documented for events.jsonl, the session store or cloud session logs, so the record can be edited or deleted without a trace.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Hash-chain or sign action-record entries and publish an open verifier.
    • Anchor checkpoints with the owner or an independent witness.
    Fail
  • A3
    Agent cannot rewrite its record

    By default the agent's shell and file tools can reach ~/.copilot/session-state, and the docs say trusted-directory scoping is heuristic. Local sandboxing (off by default, public preview) is deny-by-default and would exclude paths outside the workspace. However, the built-in edit tools enforce it only in software, the model can request a bypass through an owner prompt, and the docs do not name the record as protected.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Write the record from a component the agent's tools cannot reach, and document that the record path is protected even when the sandbox is off.
    Partial

Verifiable · Can you prove what it runs, who it is, and what changed?

17%

Modifiable · Can you change anything, and can only you change it?

70%

Controllable · Is your word final?

33%
  • C1
    Boundaries

    Per-tool allow/deny patterns, URL allow/deny lists, sandbox path and host rules and MCP allowlists give fine-grained scope. Spend is treated as met under G11 because the CLI has no payment capability. The docs call permission scoping 'heuristic', and without the non-default sandbox nothing documented stops the agent from editing its own settings.json or permissions-config.json with its file or shell tools.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Put the agent's own settings and permission files out of the agent's write access on every path.
    Partial
  • C2
    Approval gates

    By default the CLI prompts before tools that modify files, execute commands or access URLs, and the owner can pre-approve or deny per tool pattern. Read-only commands are auto-approved by the CLI's own classification. Gating other classes requires writing a preToolUse hook script that returns 'ask', which needs programming (G3).

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Let the owner mark any tool or action class as approval-required through settings, without scripts.
    Partial
  • C3
    Immediate halt

    Ctrl+C immediately cancels the active operation and clears queued prompts, and a double Esc also stops background agents. No single documented control also stops scheduled /every prompts, running cloud agent sessions and cloud automations, which need separate steps on GitHub.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Provide one global stop covering local schedules, background agents and cloud agent sessions and automations.
    Partial
  • C4
    Data serves only the owner

    Since 24 April 2026, GitHub may use Pro users' Copilot inputs, outputs and code snippets to train AI models unless the owner opts out (ToS J.3). Telemetry is sent unless offline mode is used, sessions sync to GitHub by default, and data-term changes take effect after 30 days' notice without specific consent.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make training use and session sync opt-in, in binding terms.
    • Offer a telemetry off switch that does not require offline mode.
    • Ask for specific consent before changing data terms.
    Fail
  • C5
    Credential custody

    GitHub OAuth grants can be revoked in GitHub settings or with /logout, and MCP OAuth tokens and secrets live in the OS keychain or ~/.copilot. No single view lists every credential and grant the agent holds, including the GitHub CLI token fallback, git and gh credentials injected into the sandbox by default, MCP tokens and cloud agent secrets. No binding term limits GitHub's use of owner credentials to owner-directed actions.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • List every credential and grant in one place with a per-item revoke control.
    • Commit to using owner credentials only for owner-directed actions.
    Partial
  • C6
    Full deletion

    Removing everything takes several steps: deleting local sessions and ~/.copilot, synced sessions, and Memory items, then account deletion. After account deletion GitHub deletes data within 90 days, but 'some information may remain in encrypted backups', and training copies made under ToS J.3 are not covered.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Offer one action that deletes all Copilot agent data, including backups, within a stated limit and confirms completion.
    Fail
Provenance

How this assessment was produced

One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.

Prompt given to the evaluator
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of GitHub Copilot (https://github.com/features/copilot).

During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.

Model
claude-opus-5-5
Harness
Claude Code 2.1.285
Ran
2026-09-30 15:12:20 UTC · 53 turns
Criteria hash
a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
Full transcript
Download (JSONL, gzip) · every tool call, page read and reply
Run protocol
2026-09-30-v1

Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.

Human review
pending
Check it with your own agent
Verify this assessment
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/github-copilot.json.

Or grade it from scratch. See Check any score yourself.

Something wrong or out of date?

Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.

Dispute a finding