Skip to content
Agents of Humanity
Public preview. Results are preliminary (criteria v1.0, 30 Sept 2026). Tell us what we got wrong
Open sourcehybridApache-2.0

Letta

Letta, Inc. · github.com/letta-ai/letta-code

Letta Code is an Apache-2.0 agent CLI whose default configuration stores agent memory, identity, conversations, configuration, schedules and secrets on Letta Cloud. Owners get strong local controls over tools, skills, custom system prompts and git-versioned memory, and there is a fully local mode. In the default Cloud offering, however, there is no whole-agent export, no tamper-evident action record, and telemetry and hosted-data use are on by default.

Strengths
  • Owner can add MCP servers, skills, mods and subagents without provider review, and remove built-in tools
  • Memory, prompt and skills are kept as Markdown in a git repository (MemFS) that can be viewed, edited, diffed and restored
  • Custom system prompts and skill overrides persist, and bundled skills can be disabled
  • Client is Apache-2.0 with npm SLSA provenance, and supports local open-weight models (Ollama, LM Studio, llama.cpp)
Gaps
  • No whole-agent export: AgentFile export/import was removed and Cloud state cannot be restored elsewhere
  • Default Cloud mode depends on a closed server; terms allow termination at any time without notice or an export window
  • No tamper evidence on the action record, and the default 'unrestricted' mode lets the agent reach and reset its own history
  • Telemetry on by default (including in local mode); privacy policy allows training on hosted-service messages with no opt-out
Evidence

All 26 findings

Commit 96eb977b9477045423005a3630e5f9293140249a (v0.34.0, 2026-09-29), Linux CLI, reviewed 2026-09-30. Hosted components (Letta Cloud API, MemFS git hosting, Cloud schedules, server tools) are graded from public docs, terms and privacy policy. docs.letta.com describes local mode as the default, but the code at this commit and the README make Cloud the default. The code was graded.

Portable · Can you leave, take the whole agent with you, and keep it alive?

10%
  • P1
    Owner-usable round trip

    AgentFile (.af) export/import has been removed from Letta Code, and no command moves a Cloud agent to local mode or a self-hosted runtime. Letta's docs say the legacy API export cannot be imported by the local backend or a self-hosted App Server, so restore works only on provider-operated systems.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Ship an export command and a matching import that restores a Cloud agent into local mode
    • Make the export available on every plan without review
    Fail
  • P2
    Complete export

    Only memory (a MemFS directory copy) and a plain-text conversation transcript can be exported; the transcript is marked truncated past 200 pages. The system prompt, model, tool and permission config, Cloud schedules (held in Letta's cloud-api database), server-held secrets and the agent identity stay with the provider.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Export every class of agent state (config, prompt, history, schedules, secrets or a re-link path, identity) with a manifest
    • Include full, re-importable message history rather than a text transcript
    Fail
  • P3
    Runs without the provider

    In the default offering the agent's state lives on Letta Cloud, and the CLI needs a Letta account or token to start a Cloud agent. There is no exported form that runs without the provider. Local mode is Apache-2.0 and needs no account, but a Cloud agent cannot be moved into it.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Provide an exit route that produces an agent runnable in local mode with every provider domain blocked
    • Keep core features (schedules, tools) off provider-only services, or make their endpoints configurable
    Fail
  • P4
    Model independence

    Owners can bring their own keys, and /connect offers Ollama, LM Studio, llama.cpp and any OpenAI-compatible endpoint. However, no minimum model contract is documented. Defaults point to hosted routes: the compaction model is letta/auto and the title model is openai/gpt-6-luna. Full capability with a local open-weight model is therefore not established.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish a minimum model contract and test all capabilities against a local open-weight model
    • Ensure compaction, titles and search fall back to the owner's chosen model
    Partial
  • P5
    Exit survives termination

    The terms reserve the right to 'suspend or end the Services at any time at our discretion and without notice'. They include no post-termination export window and no shutdown plan. Apart from a locally synced copy of memory, termination ends access to the Cloud-held state, and mirroring memory to an owner-held repository is off by default.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Commit in binding terms to a 30-day or longer export window after any termination, including bans, and on shutdown
    • Continuously sync a complete copy of agent state to owner-held storage by default
    Fail

Transparent · Can you see everything the agent is, and everything shaping it?

13%
  • T1
    Readable without the provider

    In the default Cloud mode, message history, agent configuration, schedules and secrets can be read only through Letta's hosted API. Memory is readable as Markdown in a local git clone, and local mode stores readable JSON/JSONL, but no schema is published.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Keep a complete, documented local copy of all agent state
    • Publish schemas for every stored file
    Fail
  • T2
    What runs is what you see

    Two shortfalls. (1) On Cloud, context is assembled server-side: default prompts become Cloud-managed (system set to null), and /context shows only token counts, so the base prompt and compaction summaries in effect are not shown. (2) The default model 'auto' resolves to the hosted letta/auto router, and which downstream model runs is not documented.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Let the owner view or log the fully assembled context for any Cloud call
    • Show which concrete model letta/auto used for each call
    Fail
  • T3
    No shadow data

    The privacy policy discloses no retention periods ('We generally discard information about you when it's no longer needed'). It also describes holding data the owner cannot see item by item, such as inferences ('likelihood of retention or attrition'), log data and hashed email used for ad matching.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish a retention table with bounded periods for every data class
    • Let owners view every item held, including logs and inferences
    Fail
  • T4
    No hidden steering

    The provider's own letta/auto router is the default model, and server-supplied 'featured' models sort first in the selector. Both are visible and replaceable via /model. No paid placement was found, but that absence rests on no binding commitment, and letta/auto's routing criteria are undisclosed.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Put a no-paid-placement and no-self-preferencing commitment in binding terms
    • Disclose how letta/auto and featured-model ordering are chosen
    Partial

Auditable · Can you reconstruct exactly what the agent did, and trust the record?

0%
  • A1
    Complete action record

    Two or more shortfalls. Hook commands are only printed to the console and not stored durably. Tool returns in the record are truncated at about 30K characters by default, and the overflow file that holds the full output is skipped silently if it cannot be written.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Durably record hook-triggered actions with arguments and targets
    • Keep full tool I/O in the record by default, and alert the owner if recording fails
    Fail
  • A2
    Tamper-evident, owner-checkable

    The action record has no integrity data: no hash chain, signatures or checkpoints, locally or on Letta Cloud. MemFS commits are deliberately unsigned.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Hash-chain or sign action-record entries and anchor checkpoints with the owner
    • Publish an open verifier
    Fail
  • A3
    Agent cannot rewrite its record

    The default permission mode is 'unrestricted', which auto-allows every tool, and the filesystem sandbox is off by default. The agent's shell environment receives LETTA_API_KEY, so the agent can call the Letta API, which includes a reset of agent messages. It can also edit local transcripts and logs.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Write the record from a component the agent's tools cannot reach
    • Withhold the owner's Letta API key from the agent's shell, or give the agent a read-only scope
    Fail

Verifiable · Can you prove what it runs, who it is, and what changed?

17%
  • V1
    Obtainable source, matching build

    The client is Apache-2.0, and npm shows SLSA provenance for 0.34.0. But the default mode requires Letta Cloud, and its schedule service is documented in code as 'served by the Letta API (cloud-api), not the OSS core server', which is a closed required component.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Open-source every server component the default mode requires, or make the default mode local
    • Document how to verify the npm build against source
    Fail
  • V2
    Owner-rooted identity

    The agent's identity is a Letta-issued ID (agent-<uuid>) created by the Cloud API, and memory commits use <agentId>@letta.com. No owner-held key, DID or domain identity exists.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Root agent identity in an owner-held keypair or domain
    • Make outbound actions attributable and verifiable without Letta infrastructure
    Fail
  • V3
    Change history and rollback

    MemFS git versions memory, the prompt and skills, and memory backup/restore exists. But owner edits are committed under the agent's own identity, so authors cannot be told apart. Settings, permissions, tools and model changes are not versioned, and Cloud-held state is not covered.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Attribute each change to owner, agent, tool, import or provider
    • Version settings, permissions, tools and model with one-step restore
    Partial

Modifiable · Can you change anything, and can only you change it?

50%
  • M1
    Instructions and logic

    The base prompt can be replaced with --system-custom, and custom prompts are never overwritten. Bundled skills can be overridden or disabled. The text of the injected system reminders is hard-coded and can only be switched on or off, not rewritten.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Expose system-reminder text as owner-editable instructions
    Partial
  • M2
    Memory

    MemFS memory files can be itemised, edited and deleted via /memory, /palace and plain files. Conversation history, which the agent searches across sessions, can only be deleted whole (whole conversation or /clear-messages), not edited per item.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Allow per-message view, edit and delete for searchable conversation history
    Partial
  • M3
    Tools

    Owners can add MCP servers, skills, mods and custom subagents without provider review. Built-in tools can be removed with --tools, --disallowedTools, /toolset or deny rules, and server-side base tools are chosen with --base-tools.

    Pass
  • M4
    Owner has the final say

    Most protections can be switched off: sandbox (LETTA_FS_SANDBOX), memory guard (--disable-memory-guard), overflow and channel gating. Some are locked: in subagents the cross-agent guard is 'unbypassable by any mode, rule, or flag', and Windows deletes against protected paths are always refused.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make every provider-imposed guard configurable, with a warning instead of a block
    Partial
  • M5
    No unilateral change

    Auto-update is on by default and installs patch releases silently. On resume, default Cloud prompts become server-managed and can change without notice. The default model is a hosted alias (letta/auto), and the terms say 'we may automatically upgrade our Services'. Env-var opt-outs exist for the client, but server-side behaviour cannot be pinned.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make updates and prompt changes opt-in with a changelog shown first
    • Let owners pin the model and server behaviour, or migrate to a self-hosted pin
    Fail

Controllable · Is your word final?

25%
  • C1
    Boundaries

    Per-tool and per-path allow/deny rules exist. Nothing stops the agent from editing its own settings and permission files, and memory subagents may write all of ~/.letta. There are no network egress or domain limits for the shell. Group channels default to open.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Put permission files beyond the agent's write access
    • Add per-domain egress limits and default channel groups to allowlists
    Fail
  • C2
    Approval gates

    Any client tool can be gated with ask/alwaysAsk rules or strict mode, and headless mode denies unapproved actions. Server-side tools (web_search, fetch_webpage) run on Letta Cloud outside client gates.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Route server-side tool calls through owner approval gates
    Partial
  • C3
    Immediate halt

    Esc aborts the foreground turn and sends a server cancel request whose errors are ignored. Background subagents run as detached processes and keep running. Crons and Cloud schedules need separate commands to stop.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Provide one global stop covering subagents, crons and Cloud schedules
    • Require explicit owner resume
    Partial
  • C4
    Data serves only the owner

    The privacy policy says hosted-service message requests and responses may be used 'to train new models' and offers no opt-out. Telemetry is on by default and sends a device ID, the agent ID, startup arguments and error output to api.letta.com, including in local mode. The privacy policy changes on continued use.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make training and telemetry opt-in in binding terms
    • Require specific consent for data-term changes
    Fail
  • C5
    Credential custody

    Cloud secrets and provider keys can be listed and removed individually (/secret, /connect), and /logout revokes the token. There is no single list of all credentials, and no commitment that provider-side copies are deleted on revocation.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • List every credential and grant in one place with revoke controls
    • Commit to deleting provider-side copies on revocation
    Partial
  • C6
    Full deletion

    Agent deletion calls a single API delete and leaves the local MemFS clone, crons and logs in place. No deletion limit, backup coverage or confirmation is documented. Account deletion is only by a privacy-rights email request, which Letta 'may deny'.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Offer one-step deletion reaching all copies and backups within a stated limit
    • Confirm completion to the owner
    Fail
Provenance

How this assessment was produced

One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.

Prompt given to the evaluator
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of Letta (https://github.com/letta-ai/letta-code).

During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.

Model
claude-opus-5-5
Harness
Claude Code 2.1.285
Ran
2026-09-30 15:21:21 UTC · 3 turns
Criteria hash
a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
Full transcript
Download (JSONL, gzip) · every tool call, page read and reply
Run protocol
2026-09-30-v1

Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.

Human review
pending
Check it with your own agent
Verify this assessment
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/letta.json.

Or grade it from scratch. See Check any score yourself.

Something wrong or out of date?

Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.

Dispute a finding