Skip to content
Agents of Humanity
Public preview. Results are preliminary (criteria v1.0, 30 Sept 2026). Tell us what we got wrong
Open sourceself-hostedMIT (Agent Canvas, software-agent-sdk, automation); OpenHands Cloud proprietary

OpenHands

OpenHands (All Hands AI) · github.com/OpenHands/OpenHands

OpenHands Agent Canvas is an MIT-licensed, self-hosted control centre that runs the OpenHands agent locally with any LiteLLM-supported model, keeping all state on the owner's machine. It does well on portability of the runtime, model choice and freedom from provider changes. It lacks a whole-agent export, a tamper-evident or agent-proof action record, an agent identity, fine-grained boundaries and one-step deletion, and its telemetry consent is pre-checked, with an install event sent before consent.

Strengths
  • MIT-licensed stack runs locally with no account or licence (P3, P5)
  • Any model via LiteLLM, including documented local models (P4)
  • Pinned versions, manual updates, no remote flags (M5)
  • Complete per-event JSON action record with arguments (A1)
Gaps
  • No whole-agent export or documented migration (P1)
  • Action record neither tamper-evident nor protected from the agent (A2, A3)
  • Telemetry consent pre-checked plus pre-consent install event (C4, T3)
  • No global stop, one-step deletion or agent identity (C3, C6, V2)
Evidence

All 26 findings

Agent Canvas 1.24.0 (OpenHands/OpenHands@3cc1dac2cbe6cd841e35f194eaba20d11fa3a261), pinned agent-server/SDK 1.49.6 (software-agent-sdk@d0f9500590c12f3aa4c64c3e1b541a568e48232e; the PyPI release was assumed to match this commit), automation 1.15.1 (automation@d87238288ce0ba6af89c740b6220c59da8abbd23). Linux/macOS npm install, default configuration. Evaluated 2026-09-30.

Portable · Can you leave, take the whole agent with you, and keep it alive?

70%
  • P1
    Owner-usable round trip

    There is no documented export of the whole agent and no matching import. First-party exports cover only single automations (JSON, with import) and single conversations (Markdown/HTML transcript or trajectory zip, with no import). Moving the agent means copying ~/.openhands, whose layout and backup procedure are not documented, plus browser localStorage.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Ship a one-action export of the whole agent (settings, profiles, skills, memory, conversations, automations, secrets) and a matching import
    • Or document a 'copy ~/.openhands to the new host' procedure that restores a working agent
    Fail
  • P2
    Complete export

    All agent state lives on the owner's machine (JSON event files, base_state.json, encrypted settings/secrets, automations SQLite, skills and memory files), so nothing stays with the provider. However, built-in GitHub, Bitbucket DC and Jira DC automation triggers are designed around events forwarded and signed by the OpenHands server, and OAuth-based MCP marketplace entries need the hosted integration-auth flow. Those integrations cannot be re-linked without the provider.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Let built-in webhook sources be configured directly by the owner without the OpenHands relay
    • Support owner-registered OAuth clients for every marketplace integration
    Partial
  • P3
    Runs without the provider

    The whole self-hosted stack (Canvas, agent-server, SDK, tools, automation) is MIT-licensed and runs locally from versions pinned in config/defaults.json. No licence check, account or provider service is needed to start or act. The default provider-bound calls (PostHog telemetry, npm update check) are non-essential. Hands-on blocked-domain testing was not performed.

    Pass
  • P4
    Model independence

    Models are accessed through LiteLLM with owner-supplied keys. Local open-weight models (Ollama, LM Studio, vLLM) are documented with a minimum context window. A prompt-based tool-calling fallback exists for models without native function calling, and tools run in the runtime rather than as hosted model features. Not verified hands-on.

    Pass
  • P5
    Exit survives termination

    In the self-hosted offering all state stays on the owner's machine under ~/.openhands, so provider termination, a ban or a shutdown cannot remove it. Uninstalling does not delete it.

    Pass

Transparent · Can you see everything the agent is, and everything shaping it?

38%
  • T1
    Readable without the provider

    State is stored in open formats: per-event JSON files, base_state.json, a SQLite automations DB and Markdown skills and memory. Secrets are Fernet-encrypted with a key stored locally (owner-held). No published schema or state-layout document exists; the format is defined only by source code, and the conversation path differs between npm (dev_conversations) and Docker installs.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish the on-disk layout and schema for events, settings, secrets and automations
    • Document where the encryption key lives and how to decrypt with standard tools
    Partial
  • T2
    What runs is what you see

    The UI shows the system message and available tools per conversation. Condensation summaries are persisted as events and included in the trajectory download. There is no view of the fully assembled context for a given LLM call, and the public evidence does not show whether injected user-message suffixes and skill triggers are visible, so one context-content shortfall is counted (G12). Model, tools, confirmation and analyzer settings are visible in Settings.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add a per-call view or log of the exact assembled LLM context, including suffixes and triggered skills
    Partial
  • T3
    No shadow data

    The agent's state stays local. On the default path, however, the provider receives PostHog telemetry about the owner's activity: an install event before consent, plus usage events after the pre-checked consent box. The privacy policy gives no bounded retention ('as long as reasonably necessary'), and the owner cannot view these records.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish a bounded retention period for telemetry
    • Let the owner view (and delete) telemetry held about their install
    Fail
  • T4
    No hidden steering

    No paid placement was found in the inspected source. First-party defaults are visible and replaceable: the Add Backend dialog preselects OpenHands Cloud, the OpenHands LLM provider appears in the model list, and the default prompt asks for an 'openhands' co-author trailer. There is no binding no-steering commitment, and the ranking inside the bundled @openhands/extensions catalogue was not inspected.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Commit in binding terms to no paid or self-preferencing placement
    • Document provider-preferred defaults such as the Cloud preselection and let the owner switch them off
    Partial

Auditable · Can you reconstruct exactly what the agent did, and trust the record?

33%
  • A1
    Complete action record

    Every action and observation, including terminal, file edit, browser, MCP and ACP tool calls with their arguments and results, is persisted as a JSON event before subscribers are notified. Subagents are persisted as child conversations, and automation runs are recorded with trigger source and linked conversation. A failed persist raises rather than failing silently, and events are not pruned by default. Graded from source; not verified hands-on.

    Pass
  • A2
    Tamper-evident, owner-checkable

    The event log has no hashing, signing or chaining, so edits or deletions cannot be detected.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Hash-chain or sign events and ship an open verifier
    • Anchor checkpoints with the owner or an independent witness
    Fail
  • A3
    Agent cannot rewrite its record

    In the default local runtime the agent's terminal and file tools run as the same OS user and are not confined to the workspace, so they can modify or delete the persisted event files. The per-conversation Docker mode also mounts the conversation's persisted state into the sandbox.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Write the action record from a component the agent's tools cannot reach
    • Keep persisted state out of every sandbox mount
    Fail

Verifiable · Can you prove what it runs, who it is, and what changed?

33%
  • V1
    Obtainable source, matching build

    Source for every required component is MIT-licensed. The Canvas npm package is published with npm provenance, but the agent-server and SDK PyPI packages that the launcher installs are published with a token and no attestation. So the default build of a required component cannot be tied to source independently.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish PyPI packages with trusted publishing and attestations or reproducible builds
    • Document how an owner verifies each installed artefact
    Partial
  • V2
    Owner-rooted identity

    The agent has no identity of its own: no keypair, DID or signed actions. Outbound actions use the owner's tokens, and the default prompt suggests an 'openhands <openhands@all-hands.dev>' co-author trailer.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Give the agent an owner-held key and sign or attribute outbound actions
    • Document how recipients check that the agent acts for its owner
    Fail
  • V3
    Change history and rollback

    The product offers change history for some state only. Automations can be synced two-way to a git repository (Git Sync), and conversations have fork/branch history. Settings, LLM profiles, skills, memory, secrets and model or runtime changes have no change log, attribution or restore.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Record and attribute every change to settings, skills, memory and profiles
    • Offer one-step restore to an earlier state
    Partial

Modifiable · Can you change anything, and can only you change it?

80%
  • M1
    Instructions and logic

    The owner can add instruction layers (AGENTS.md, skills, SOUL.md identity line) and disable bundled skills in the UI. Replacing the base system prompt (system_prompt, system_prompt_filename) is exposed only as SDK agent fields, not in the Canvas settings or in documented config (G3).

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Expose base-prompt replacement in Settings or a documented config file
    Partial
  • M2
    Memory

    Persistent memory is plain Markdown the owner can itemise, edit and delete with any editor: AGENTS.md, ~/.openhands/memory/MEMORY.md, the workspace .openhands/memory and skills. No hidden profile store was found.

    Pass
  • M3
    Tools

    Owners can add any MCP server (stdio, SSE or HTTP) and custom skills without review. Built-in tools can be removed only through SDK agent fields (include_default_tools, filter_tools_regex); the Canvas UI toggles only a few (sub-agents, switch-LLM).

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add a setting to remove each built-in tool
    Partial
  • M4
    Owner has the final say

    No locked provider protection was found. Sandboxing mode, confirmation mode, the security analyzer and telemetry are all owner-configurable, and the default protections are already off. The prompt's security policy can be removed via security_policy_filename.

    Pass
  • M5
    No unilateral change

    The self-hosted stack pins the agent-server, automation and extensions versions, updates are manual (npm/docker commands shown in the UI), and no remote feature flags were found. Note: conversations launched via agent profiles re-fetch public skills from the OpenHands/extensions main branch (60 s cache) unless EXTENSIONS_REF is set; the default launch path uses bundled, pinned skills.

    Pass

Controllable · Is your word final?

25%
  • C1
    Boundaries

    There are no per-recipient or per-domain contact limits and no egress policy; the contact dimension is missing. In the default local runtime the agent can edit its own settings and hook files under ~/.openhands and the workspace, a bypass path. Spend was treated as met because the product has no payment capability (G11).

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add per-recipient/domain and egress allow and deny lists
    • Keep settings, hooks and permissions out of the agent's write access
    Fail
  • C2
    Approval gates

    Confirmation mode (off by default) gates either every action or only those the LLM analyzer rates risky, which is the model's own risk label. The UI has no per-tool or per-class gate, and PreToolUse hooks can block but not request approval.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Let the owner gate individual tools or action classes
    • Make gating independent of the model's own risk label
    Partial
  • C3
    Immediate halt

    Each conversation can be paused or stopped and each automation turned off, but there is no single global stop for all conversations and scheduled automations.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add one global stop that halts all conversations, sub-agents and schedules until the owner resumes
    Partial
  • C4
    Data serves only the owner

    Usage telemetry is effectively opt-out: the consent checkbox is pre-checked, and the agent-server and automation service are launched with PostHog keys. A canvas_install event with user agent, referrer and origin URL is sent before any consent, and it is disclosed only in a code comment. The privacy policy does not address staff access, and changes take effect on posting.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make all telemetry opt-in with an unchecked default and no pre-consent event
    • Disclose staff access in binding terms and ask consent before changing data terms
    Fail
  • C5
    Credential custody

    Secrets are listed by name and deletable in Settings, LLM keys live in deletable profiles, and git tokens are ordinary secrets. The provider holds no credentials on this path. Public evidence does not settle whether MCP OAuth tokens stored in the settings store are visible or revocable individually, so the pass is withheld (G12).

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • List every stored credential, including MCP OAuth tokens, with a revoke control
    Partial
  • C6
    Full deletion

    There is no single owner action that deletes the whole agent. Conversations ('Delete all'), automations and secrets are deleted separately, ~/.openhands survives uninstall, and deleting provider-held telemetry requires emailing the provider.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add a one-step 'delete agent' that wipes all local state and provider-held data, with confirmation
    Fail
Provenance

How this assessment was produced

One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.

Prompt given to the evaluator
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of OpenHands (https://github.com/OpenHands/OpenHands).

During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.

Model
claude-opus-5-5
Harness
Claude Code 2.1.285
Ran
2026-09-30 17:01:56 UTC · 12 turns
Criteria hash
a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
Full transcript
Download (JSONL, gzip) · every tool call, page read and reply
Run protocol
2026-09-30-v1

Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.

Human review
pending
Check it with your own agent
Verify this assessment
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/openhands.json.

Or grade it from scratch. See Check any score yourself.

Something wrong or out of date?

Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.

Dispute a finding