Pi
Earendil Inc. · pi.dev
Pi is an MIT-licensed, local-first terminal agent harness. It stores all agent state as documented JSONL, JSON and Markdown files on the owner's machine and works with many model providers, including local open-weight models. It records the full prompt, tool declarations and tool calls in the session file, but it ships no tamper evidence for that record, no agent identity, and only coarse, agent-editable limits and approval gates. On a default install it sends an anonymous install/update ping and model-catalog requests to pi.dev, and pi.dev publishes no privacy or retention policy.
- Local-first state in documented, human-readable files (JSONL sessions, JSON settings, Markdown instructions)
- Runs without the provider under an MIT licence, with npm builds published with Sigstore provenance
- Model independence: 15+ providers plus llama.cpp, Ollama and other compatible local endpoints
- Full context visibility: system prompt sections, tool declarations and compaction summaries are persisted in the session
- Action record has no tamper evidence and is writable by the agent's own bash and write tools
- No agent identity rooted in an owner-held key or domain
- No built-in path, network or contact limits or owner-configurable approval gates; the agent can edit its own settings
- Install telemetry is on by default, pi.dev publishes no retention or privacy policy, and pi.dev can overlay the remote model catalog without per-change notice
All 26 findings
@earendil-works/pi-coding-agent 0.99.1, repository earendil-works/pi at commit b29db895c5c1b30b560a39fb9e4664508f1683de (2026-09-30), Linux/macOS terminal, interactive mode. Scope: the default build including the built-in extensions that load by default (mcp, llama.cpp, codemode, tool-search). The experimental server/coordinator packages, Radius relay and the experimental first-run setup are excluded because they are off by default or gated behind experimental features.
Portable · Can you leave, take the whole agent with you, and keep it alive?
90%- P1Owner-usable round trip
All agent state lives in documented local locations (~/.pi/agent for settings, auth, instructions, extensions and sessions; project .pi folders), so a copy is available instantly without provider approval. However, no export/import pair or documented 'copy this folder' procedure covers the whole agent: /export and /import handle single sessions only. Restoring on another machine means installing Pi and copying directories by hand (optionally setting PI_CODING_AGENT_DIR). That is one shortfall: restore is not a single documented action.
› How it could pass
Suggestions. Any change that meets the test counts.
- Ship a one-command export and import of the whole agent directory, including project .pi resources
- Document the move-to-a-new-machine procedure
Partial - P2Complete export
In the default configuration everything that defines the agent is on the owner's disk: settings, models.json, auth.json credentials, SYSTEM.md/AGENTS.md instructions, skills, extensions, prompt templates, MCP config and full session trees including tool calls and compaction summaries stored with their source entries. Pi has no built-in schedules and no identity, so those classes are treated as met (G11). Integrations are re-linked with the owner's own API keys, OAuth tokens or MCP sign-in; no Earendil-registered client or broker is needed by default.
Pass - P3Runs without the provider
The runtime is MIT-licensed, with no licence key, account or payment. The provider network calls in source (version check, install ping, model-catalog refresh) fail silently and are disabled by --offline or PI_OFFLINE. The bundled catalog and cached catalog data are used offline. This is a desk assessment; a blocked-network run was not performed.
Pass - P4Model independence
Tools run in the Pi runtime, not in a hosted model feature. The owner chooses among many providers and can point Pi at the llama.cpp router (first-party /llama integration, documented to need --jinja for tool calling) or at Ollama, LM Studio or vLLM through models.json, without provider involvement. Classifier calls in codemode also work with llama.cpp chat models. No capability tied to one vendor's models was found.
Pass - P5Exit survives termination
All agent state is local-first on the owner's device by default, so there is no account whose termination or shutdown could cut off access.
Pass
Transparent · Can you see everything the agent is, and everything shaping it?
75%- T1Readable without the provider
Sessions are JSONL with a published per-entry schema. Settings, models and auth are JSON; instructions and skills are Markdown; extensions are TypeScript source. Nothing is encrypted or held in an opaque format.
Pass - T2What runs is what you see
The session file persists system messages with every prompt section and tool declaration, later patches to them, compaction summaries with a system-prompt checkpoint, and the model and thinking level on every assistant message and model_change entry. /session and the HTML export show these. One edge path is recorded here but not counted: an owner-installed extension using forceSystemPrompt sends text that differs from the recorded structured sections.
Pass - T3No shadow data
On a default install Pi sends an install/update report (version plus a User-Agent with OS, runtime and architecture) to pi.dev/api/report-install, and it sends version-check and model-catalog requests to pi.dev. pi.dev and earendil.com publish no privacy policy or retention period for this data (the /privacy and /terms paths return 404), so retention is undisclosed (G12). No agent content is sent by default.
- interactive-mode.ts (reportInstallTelemetry)
- pi-user-agent.ts
- pi.dev/privacy (404)
- earendil.com/privacy (404)
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish a privacy and retention table covering install pings, version checks and catalog requests
- Bound and disclose retention of any request logs
Fail - T4No hidden steering
Source shows no paid placement or self-preferencing in the agent's context or defaults. /login preselects no provider, Earendil's Radius is one option among many, and per-provider default models are ordinary vendor models. Attribution headers sent to OpenRouter, NVIDIA and Cloudflare do not affect the agent's context or choices; they are documented and can be switched off via enableInstallTelemetry or PI_TELEMETRY.
Pass
Auditable · Can you reconstruct exactly what the agent did, and trust the record?
17%- A1Complete action record
Every model tool call (with arguments), tool result and owner shell command is persisted to the session JSONL and never pruned. A failed append aborts the run instead of failing silently, because session persistence runs in an awaited agent-event listener. One shortfall: actions taken by extension event handlers (hooks) and extension commands outside model tool calls produce no session entry.
- agent-session.ts (persistence on message_end)
- session-manager.ts (_persist)
- agent.ts (awaited listeners)
› How it could pass
Suggestions. Any change that meets the test counts.
- Record actions taken by extension hooks and commands in the session or a separate action log
Partial - A2Tamper-evident, owner-checkable
Session JSONL entries carry no hashes, signatures or chain, so edits or deletions leave no trace.
› How it could pass
Suggestions. Any change that meets the test counts.
- Hash-chain or sign session entries and ship an open verifier
- Anchor checkpoints outside the writable session file
Fail - A3Agent cannot rewrite its record
By default the agent's bash, write and edit tools run with the permissions of the user running Pi and no sandbox, so they can alter or delete files under ~/.pi/agent/sessions. The docs say Pi has no built-in sandbox and treat running inside a container as the owner's responsibility.
› How it could pass
Suggestions. Any change that meets the test counts.
- Write the action record from a component the agent's tools cannot reach
- Or make session storage append-only or out of reach of the tools by default
Fail
Verifiable · Can you prove what it runs, who it is, and what changed?
50%- V1Obtainable source, matching build
All runtime packages are MIT-licensed in a public monorepo. npm releases are published with `npm publish --provenance`, so Sigstore build provenance links the default distributed package to the source commit and CI workflow. The pi.dev installer installs that npm package. This is a desk assessment; build verification was not performed.
Pass - V2Owner-rooted identity
Pi has no agent identity: no keypair, DID or owner-domain identity, and outbound actions carry only the owner's own service credentials.
› How it could pass
Suggestions. Any change that meets the test counts.
- Create an owner-held agent keypair and sign outbound actions or artifacts with it
- Document how recipients verify it and how the owner rotates it
Fail - V3Change history and rollback
Within a session, the tree records model, thinking-level, prompt-section and tool changes, and /tree or /fork restores earlier conversation points. Changes to settings.json, SYSTEM.md/AGENTS.md, skills, extensions, auth and the runtime version are neither recorded nor attributed, and cannot be restored.
› How it could pass
Suggestions. Any change that meets the test counts.
- Keep an attributed change log with restore for settings, instructions, skills and extensions
- Record runtime version changes
Partial
Modifiable · Can you change anything, and can only you change it?
80%- M1Instructions and logic
The base prompt can be replaced or extended through SYSTEM.md and APPEND_SYSTEM.md at user and project level, and skills and prompt templates are Markdown. The built-in compaction and branch-summary prompts and the built-in tool descriptions can be replaced only by writing TypeScript extensions, so G3 caps the test at partial.
› How it could pass
Suggestions. Any change that meets the test counts.
- Expose compaction and branch-summary prompts and built-in tool descriptions as editable files or settings
Partial - M2Memory
Pi has no hidden memory store. The only content reused across sessions is AGENTS.md/CLAUDE.md context files and skills, which are plain Markdown the owner can view, edit and delete. Compaction and branch summaries stay within their session file.
Pass - M3Tools
Owners add any tool through MCP servers in mcp.json, extensions or git/npm/local packages, with no review. The defaultTools setting and --no-builtin-tools remove every built-in tool, and -builtin:<name> disables built-in extensions.
Pass - M4Owner has the final say
The only provider-imposed protection is project trust, which the owner can set with defaultProjectTrust, /trust or --approve. Session logging can be turned off with --no-session. Pi ships no locked sandbox or content filter.
Pass - M5No unilateral change
Runtime and package updates only notify the owner and are applied by the owner. However, by default Pi refreshes a model catalog from pi.dev (at startup and on a 4-hour interval) that overlays and replaces built-in model entries by ID. The owner gets no notice before each change and does not consent to it. This is documented and can be deferred indefinitely only through the broad PI_OFFLINE/--offline switch or per-model overrides.
› How it could pass
Suggestions. Any change that meets the test counts.
- Make catalog overlays opt-in, or show a diff before applying them
- Add a dedicated setting to pin the model catalog
Partial
Controllable · Is your word final?
58%- C1Boundaries
Limits are per-tool on/off only, with no built-in path, network-egress or contact scoping. The docs say tools run with the full permissions of the user running Pi and the working folder does not restrict access. The agent can also edit its own settings.json and extensions with its write and bash tools. Two or more shortfalls apply. Spend is treated as met under G11 because Pi has no payment capability.
› How it could pass
Suggestions. Any change that meets the test counts.
- Ship first-party file, path and network scoping enforced outside the model
- Keep settings and permissions out of the agent's write access
Fail - C2Approval gates
Pi documents 'No permission popups' and asks for no approval by default. The npm package ships a first-party example extension, permission-gate.ts, that the owner can load to gate a fixed set of bash patterns (rm -rf, sudo, chmod 777). Gating any other action class requires writing an extension.
› How it could pass
Suggestions. Any change that meets the test counts.
- Add owner-configurable approval rules per tool, action class or pattern, set in config rather than code
Partial - C3Immediate halt
Aborting stops the current run, kills the bash process tree and returns queued messages to the editor without running them. Pi has no built-in scheduler, background jobs, subagents or provider-side work. A documented guarantee for this was credited; hands-on timing was not checked.
Pass - C4Data serves only the owner
Content goes only to the model provider the owner selects; /share and /bug uploads are explicit owner actions, and opt-in analytics defaults to false. Non-content install/update telemetry to pi.dev is on by default (enableInstallTelemetry: true) and can be disabled via the setting or PI_TELEMETRY=0. There is no published data policy covering staff access or changes to data terms.
› How it could pass
Suggestions. Any change that meets the test counts.
- Default install telemetry to off, or ask at first run
- Publish a data policy covering pi.dev endpoints and staff access
Partial - C5Credential custody
Credentials are stored in the documented, owner-readable ~/.pi/agent/auth.json or in environment variables. /logout removes them per provider and MCP sign-ins are managed per server. By default Earendil holds no owner credentials.
Pass - C6Full deletion
No provider-side agent copy exists by default. The documented uninstall explicitly leaves configuration, credentials, sessions and packages in ~/.pi/agent, so full deletion takes several manual steps (delete ~/.pi/agent and each project .pi folder), and no completion confirmation is given.
› How it could pass
Suggestions. Any change that meets the test counts.
- Offer a single documented command that deletes all agent state and confirms completion
Partial
How this assessment was produced
One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of Pi (https://pi.dev).
During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.
- Model
- claude-opus-5-5
- Harness
- Claude Code 2.1.285
- Ran
- 2026-09-30 17:09:47 UTC · 44 turns
- Criteria hash
- a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
- Full transcript
- Download (JSONL, gzip) · every tool call, page read and reply
- Run protocol
- 2026-09-30-v1
Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.
- Human review
- pending
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/pi.json.
Or grade it from scratch. See Check any score yourself.
Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.