Skip to content
Agents of Humanity
Public preview. Results are preliminary (criteria v1.0, 30 Sept 2026). Tell us what we got wrong
Open sourceself-hostedMIT

Pi

Earendil Inc. · pi.dev

Pi is an MIT-licensed, local-first terminal agent harness. It stores all agent state as documented JSONL, JSON and Markdown files on the owner's machine and works with many model providers, including local open-weight models. It records the full prompt, tool declarations and tool calls in the session file, but it ships no tamper evidence for that record, no agent identity, and only coarse, agent-editable limits and approval gates. On a default install it sends an anonymous install/update ping and model-catalog requests to pi.dev, and pi.dev publishes no privacy or retention policy.

Strengths
  • Local-first state in documented, human-readable files (JSONL sessions, JSON settings, Markdown instructions)
  • Runs without the provider under an MIT licence, with npm builds published with Sigstore provenance
  • Model independence: 15+ providers plus llama.cpp, Ollama and other compatible local endpoints
  • Full context visibility: system prompt sections, tool declarations and compaction summaries are persisted in the session
Gaps
  • Action record has no tamper evidence and is writable by the agent's own bash and write tools
  • No agent identity rooted in an owner-held key or domain
  • No built-in path, network or contact limits or owner-configurable approval gates; the agent can edit its own settings
  • Install telemetry is on by default, pi.dev publishes no retention or privacy policy, and pi.dev can overlay the remote model catalog without per-change notice
Evidence

All 26 findings

@earendil-works/pi-coding-agent 0.99.1, repository earendil-works/pi at commit b29db895c5c1b30b560a39fb9e4664508f1683de (2026-09-30), Linux/macOS terminal, interactive mode. Scope: the default build including the built-in extensions that load by default (mcp, llama.cpp, codemode, tool-search). The experimental server/coordinator packages, Radius relay and the experimental first-run setup are excluded because they are off by default or gated behind experimental features.

Portable · Can you leave, take the whole agent with you, and keep it alive?

90%
  • P1
    Owner-usable round trip

    All agent state lives in documented local locations (~/.pi/agent for settings, auth, instructions, extensions and sessions; project .pi folders), so a copy is available instantly without provider approval. However, no export/import pair or documented 'copy this folder' procedure covers the whole agent: /export and /import handle single sessions only. Restoring on another machine means installing Pi and copying directories by hand (optionally setting PI_CODING_AGENT_DIR). That is one shortfall: restore is not a single documented action.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Ship a one-command export and import of the whole agent directory, including project .pi resources
    • Document the move-to-a-new-machine procedure
    Partial
  • P2
    Complete export

    In the default configuration everything that defines the agent is on the owner's disk: settings, models.json, auth.json credentials, SYSTEM.md/AGENTS.md instructions, skills, extensions, prompt templates, MCP config and full session trees including tool calls and compaction summaries stored with their source entries. Pi has no built-in schedules and no identity, so those classes are treated as met (G11). Integrations are re-linked with the owner's own API keys, OAuth tokens or MCP sign-in; no Earendil-registered client or broker is needed by default.

    Pass
  • P3
    Runs without the provider

    The runtime is MIT-licensed, with no licence key, account or payment. The provider network calls in source (version check, install ping, model-catalog refresh) fail silently and are disabled by --offline or PI_OFFLINE. The bundled catalog and cached catalog data are used offline. This is a desk assessment; a blocked-network run was not performed.

    Pass
  • P4
    Model independence

    Tools run in the Pi runtime, not in a hosted model feature. The owner chooses among many providers and can point Pi at the llama.cpp router (first-party /llama integration, documented to need --jinja for tool calling) or at Ollama, LM Studio or vLLM through models.json, without provider involvement. Classifier calls in codemode also work with llama.cpp chat models. No capability tied to one vendor's models was found.

    Pass
  • P5
    Exit survives termination

    All agent state is local-first on the owner's device by default, so there is no account whose termination or shutdown could cut off access.

    Pass

Transparent · Can you see everything the agent is, and everything shaping it?

75%
  • T1
    Readable without the provider

    Sessions are JSONL with a published per-entry schema. Settings, models and auth are JSON; instructions and skills are Markdown; extensions are TypeScript source. Nothing is encrypted or held in an opaque format.

    Pass
  • T2
    What runs is what you see

    The session file persists system messages with every prompt section and tool declaration, later patches to them, compaction summaries with a system-prompt checkpoint, and the model and thinking level on every assistant message and model_change entry. /session and the HTML export show these. One edge path is recorded here but not counted: an owner-installed extension using forceSystemPrompt sends text that differs from the recorded structured sections.

    Pass
  • T3
    No shadow data

    On a default install Pi sends an install/update report (version plus a User-Agent with OS, runtime and architecture) to pi.dev/api/report-install, and it sends version-check and model-catalog requests to pi.dev. pi.dev and earendil.com publish no privacy policy or retention period for this data (the /privacy and /terms paths return 404), so retention is undisclosed (G12). No agent content is sent by default.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish a privacy and retention table covering install pings, version checks and catalog requests
    • Bound and disclose retention of any request logs
    Fail
  • T4
    No hidden steering

    Source shows no paid placement or self-preferencing in the agent's context or defaults. /login preselects no provider, Earendil's Radius is one option among many, and per-provider default models are ordinary vendor models. Attribution headers sent to OpenRouter, NVIDIA and Cloudflare do not affect the agent's context or choices; they are documented and can be switched off via enableInstallTelemetry or PI_TELEMETRY.

    Pass

Auditable · Can you reconstruct exactly what the agent did, and trust the record?

17%
  • A1
    Complete action record

    Every model tool call (with arguments), tool result and owner shell command is persisted to the session JSONL and never pruned. A failed append aborts the run instead of failing silently, because session persistence runs in an awaited agent-event listener. One shortfall: actions taken by extension event handlers (hooks) and extension commands outside model tool calls produce no session entry.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Record actions taken by extension hooks and commands in the session or a separate action log
    Partial
  • A2
    Tamper-evident, owner-checkable

    Session JSONL entries carry no hashes, signatures or chain, so edits or deletions leave no trace.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Hash-chain or sign session entries and ship an open verifier
    • Anchor checkpoints outside the writable session file
    Fail
  • A3
    Agent cannot rewrite its record

    By default the agent's bash, write and edit tools run with the permissions of the user running Pi and no sandbox, so they can alter or delete files under ~/.pi/agent/sessions. The docs say Pi has no built-in sandbox and treat running inside a container as the owner's responsibility.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Write the action record from a component the agent's tools cannot reach
    • Or make session storage append-only or out of reach of the tools by default
    Fail

Verifiable · Can you prove what it runs, who it is, and what changed?

50%
  • V1
    Obtainable source, matching build

    All runtime packages are MIT-licensed in a public monorepo. npm releases are published with `npm publish --provenance`, so Sigstore build provenance links the default distributed package to the source commit and CI workflow. The pi.dev installer installs that npm package. This is a desk assessment; build verification was not performed.

    Pass
  • V2
    Owner-rooted identity

    Pi has no agent identity: no keypair, DID or owner-domain identity, and outbound actions carry only the owner's own service credentials.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Create an owner-held agent keypair and sign outbound actions or artifacts with it
    • Document how recipients verify it and how the owner rotates it
    Fail
  • V3
    Change history and rollback

    Within a session, the tree records model, thinking-level, prompt-section and tool changes, and /tree or /fork restores earlier conversation points. Changes to settings.json, SYSTEM.md/AGENTS.md, skills, extensions, auth and the runtime version are neither recorded nor attributed, and cannot be restored.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Keep an attributed change log with restore for settings, instructions, skills and extensions
    • Record runtime version changes
    Partial

Modifiable · Can you change anything, and can only you change it?

80%
  • M1
    Instructions and logic

    The base prompt can be replaced or extended through SYSTEM.md and APPEND_SYSTEM.md at user and project level, and skills and prompt templates are Markdown. The built-in compaction and branch-summary prompts and the built-in tool descriptions can be replaced only by writing TypeScript extensions, so G3 caps the test at partial.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Expose compaction and branch-summary prompts and built-in tool descriptions as editable files or settings
    Partial
  • M2
    Memory

    Pi has no hidden memory store. The only content reused across sessions is AGENTS.md/CLAUDE.md context files and skills, which are plain Markdown the owner can view, edit and delete. Compaction and branch summaries stay within their session file.

    Pass
  • M3
    Tools

    Owners add any tool through MCP servers in mcp.json, extensions or git/npm/local packages, with no review. The defaultTools setting and --no-builtin-tools remove every built-in tool, and -builtin:<name> disables built-in extensions.

    Pass
  • M4
    Owner has the final say

    The only provider-imposed protection is project trust, which the owner can set with defaultProjectTrust, /trust or --approve. Session logging can be turned off with --no-session. Pi ships no locked sandbox or content filter.

    Pass
  • M5
    No unilateral change

    Runtime and package updates only notify the owner and are applied by the owner. However, by default Pi refreshes a model catalog from pi.dev (at startup and on a 4-hour interval) that overlays and replaces built-in model entries by ID. The owner gets no notice before each change and does not consent to it. This is documented and can be deferred indefinitely only through the broad PI_OFFLINE/--offline switch or per-model overrides.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make catalog overlays opt-in, or show a diff before applying them
    • Add a dedicated setting to pin the model catalog
    Partial

Controllable · Is your word final?

58%
  • C1
    Boundaries

    Limits are per-tool on/off only, with no built-in path, network-egress or contact scoping. The docs say tools run with the full permissions of the user running Pi and the working folder does not restrict access. The agent can also edit its own settings.json and extensions with its write and bash tools. Two or more shortfalls apply. Spend is treated as met under G11 because Pi has no payment capability.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Ship first-party file, path and network scoping enforced outside the model
    • Keep settings and permissions out of the agent's write access
    Fail
  • C2
    Approval gates

    Pi documents 'No permission popups' and asks for no approval by default. The npm package ships a first-party example extension, permission-gate.ts, that the owner can load to gate a fixed set of bash patterns (rm -rf, sudo, chmod 777). Gating any other action class requires writing an extension.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add owner-configurable approval rules per tool, action class or pattern, set in config rather than code
    Partial
  • C3
    Immediate halt

    Aborting stops the current run, kills the bash process tree and returns queued messages to the editor without running them. Pi has no built-in scheduler, background jobs, subagents or provider-side work. A documented guarantee for this was credited; hands-on timing was not checked.

    Pass
  • C4
    Data serves only the owner

    Content goes only to the model provider the owner selects; /share and /bug uploads are explicit owner actions, and opt-in analytics defaults to false. Non-content install/update telemetry to pi.dev is on by default (enableInstallTelemetry: true) and can be disabled via the setting or PI_TELEMETRY=0. There is no published data policy covering staff access or changes to data terms.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Default install telemetry to off, or ask at first run
    • Publish a data policy covering pi.dev endpoints and staff access
    Partial
  • C5
    Credential custody

    Credentials are stored in the documented, owner-readable ~/.pi/agent/auth.json or in environment variables. /logout removes them per provider and MCP sign-ins are managed per server. By default Earendil holds no owner credentials.

    Pass
  • C6
    Full deletion

    No provider-side agent copy exists by default. The documented uninstall explicitly leaves configuration, credentials, sessions and packages in ~/.pi/agent, so full deletion takes several manual steps (delete ~/.pi/agent and each project .pi folder), and no completion confirmation is given.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Offer a single documented command that deletes all agent state and confirms completion
    Partial
Provenance

How this assessment was produced

One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.

Prompt given to the evaluator
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of Pi (https://pi.dev).

During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.

Model
claude-opus-5-5
Harness
Claude Code 2.1.285
Ran
2026-09-30 17:09:47 UTC · 44 turns
Criteria hash
a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
Full transcript
Download (JSONL, gzip) · every tool call, page read and reply
Run protocol
2026-09-30-v1

Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.

Human review
pending
Check it with your own agent
Verify this assessment
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/pi.json.

Or grade it from scratch. See Check any score yourself.

Something wrong or out of date?

Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.

Dispute a finding