Skip to content
Agents of Humanity
Public preview. Results are preliminary (criteria v1.0, 30 Sept 2026). Tell us what we got wrong
Open sourceself-hostedMIT

DeepSeek Harness

DeepSeek (Hangzhou DeepSeek Artificial Intelligence) · github.com/deepseek-ai/deepseek-harness

DeepSeek Harness is an MIT-licensed, local-first agent harness. All agent state is kept as documented files under $DSH_HOME on the owner's machine, and it can run against any OpenAI- or Anthropic-compatible model endpoint. By default it uploads the canonical session log, including user content, tool arguments and results, to DeepSeek with each official-API request, under a setting labelled as helping improve DeepSeek models and products. It offers no whole-agent export or import, no tamper-evident action record, no owner-rooted identity and no session deletion.

Strengths
  • MIT source; the runtime has no licence check, login gate or remote configuration
  • State is local-first under $DSH_HOME, so it survives provider termination
  • Supports third-party and self-hosted OpenAI- or Anthropic-compatible models
  • Owner can add any plugin, MCP server or skill without review
Gaps
  • Session-log upload to DeepSeek is on by default, framed as model and product improvement
  • No whole-agent export or import; only a per-session ZIP download
  • Action record has no integrity data, and sessions cannot be deleted
  • No per-domain egress limits, reads are unconfined, and no global stop
Evidence

All 26 findings

v0.2.0-rc.2 (npm latest, published 2026-09-29), commit 639ed015397290b3745d163aafe02ffee4aa3f84 (tag dsh-v0.2.0-rc.2). Default `web` profile (dsh-base + dsh-web-app bundles, `standard` preset, `workspace-write` sandbox, approval `ask`). Scope and assumptions: the platform is not fixed and the review is desk-only. Schedules and webhooks are off by default and are outside the default path. Spend is treated as met under G11 because there is no payment capability. Evaluated 2026-09-30.

Portable · Can you leave, take the whole agent with you, and keep it alive?

60%
  • P1
    Owner-usable round trip

    The only documented export is a per-session ZIP (session log, sub-sessions, attachments) downloaded through the browser. There is no import, restore or backup command, and no documented procedure for copying $DSH_HOME to another host. Session folders are keyed by absolute working-directory paths.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Ship a whole-agent export plus a one-command import or restore
    • Document a supported 'copy $DSH_HOME' migration that survives path changes
    Fail
  • P2
    Complete export

    All state classes live in owner-held files under $DSH_HOME and nothing stays only with the provider: - sessions, KV storage, profile patches, .credentials.yaml, attachments, skills and AGENTS.md - integrations use API keys or owner-generated webhook secrets, not provider OAuth clients The documented export omits settings, credentials, skills, instructions and workspaces, and there is no restore route. The fail definition (classes held by the provider, or text-only export) is not met, so the test stays partial under G1.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Include settings, credentials, skills, instructions and workspace registry in the export, with a manifest
    Partial
  • P3
    Runs without the provider

    The runtime is MIT-licensed, with no licence key, login, remote-config fetch or update check in the CLI/Web path; the official build profile only embeds version strings. DeepSeek endpoints are all configurable or replaceable: - model API - default web_search provider (Exa and Perplexity plugins exist) - OTel collector (DSH_TELEMETRY_OTLP_URL, or disable) This is a desk grade; the provider-blocked run is hands-on.

    Pass
  • P4
    Model independence

    Other providers are supported, and custom OpenAI Chat, OpenAI Responses and Anthropic Messages endpoints work at any base URL, including self-hosted ones. Capabilities are model-neutral: titles and compaction use the session route, and vision works for any model declared image-capable. However, there is no documented minimum model contract, no text-based tool-calling fallback, and no local open-weight model is named or tested (G12).

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish the minimum model contract and test it against a local open-weight model
    • Offer text-based tool calling for models without native tools
    Partial
  • P5
    Exit survives termination

    All agent state is kept locally under $DSH_HOME on the owner's machine, with no hosted account or sync. Termination of a DeepSeek account ends only model access and does not remove the owner's copy.

    Pass

Transparent · Can you see everything the agent is, and everything shaping it?

38%
  • T1
    Readable without the provider

    State is kept in open formats: - session logs are JSONL (zstd-compressed by default) with a public event catalog and JSON schema - KV storage is JSON - credentials and config are YAML - no encryption The catalog marks some fields as opaque: replayState, ReplayEnvelope.response, tool-private meta, and provider thinking signatures. Storage-domain schemas exist only as zod definitions in source.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Document or remove opaque replay and meta fields
    • Publish JSON schemas for storage domains
    Partial
  • T2
    What runs is what you see

    The following are visible to the owner: - system prompt, dynamic context and compaction summaries are logged as session events - the per-request header with config and full tool schemas is logged, and the Trajectory view shows per-request system prompt and tools - `dsh --dump-config` shows every layer - DeepSeek's docs document what the `deepseek-flash` alias resolves to One shortfall is counted under G12: no public evidence settles whether the hosted API adds server-side context. Unevaluated `!!js` config expressions are recorded but not counted.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Obtain and publish a statement on server-side context added by the hosted API
    • Offer a raw request log option
    Partial
  • T3
    No shadow data

    DeepSeek receives and holds several kinds of data about the owner's use: - by default, the uploaded session log: working directory, prompts, user and assistant content, tool arguments and results, feedback - request headers carrying a stable anonymous installation UUID and the plugin inventory - feedback-triggered OTel uploads of the session prefix DeepSeek's privacy policy gives only an 'as long as necessary' retention, and nothing lets the owner view that data item by item.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish bounded retention for uploaded session logs, headers and OTel data
    • Let the owner view what DeepSeek holds
    Fail
  • T4
    No hidden steering

    No paid placement or sponsored tools were found in source. The first-party defaults (DeepSeek model, DeepSeek-hosted web search) are disclosed in config and replaceable, and the plugin registry is plain npm with no ranking. The absence of steering in the hosted search and model results rests on no binding commitment.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Put a no-steering commitment in binding terms
    • Expose alternative search providers in the Settings UI
    Partial

Auditable · Can you reconstruct exactly what the agent did, and trust the record?

33%
  • A1
    Complete action record

    The append-only session log records the following, and a tool does not run until its checkpoint write succeeds: - tool calls with full arguments - PTC sub-calls, subagents as child sessions, workflows - approvals and hooks One shortfall: by default, tool results over 12,500 tokens are replaced in the log by a head/tail preview. The full output goes to a temp spill file that is deleted after 30 days.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Keep full tool results in the durable record, or never auto-delete spill files
    Partial
  • A2
    Tamper-evident, owner-checkable

    The session log has no hash chain, signatures or anchoring. The only integrity data is zstd's per-frame content checksum, which detects corruption but not deliberate edits. The SHA-256 values in the docs are schema-type fingerprints, not record integrity.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Hash-chain or sign log entries and publish an open verifier
    • Anchor checkpoints with the owner
    Fail
  • A3
    Agent cannot rewrite its record

    In the default workspace-write sandbox, writable roots are the workspace and temp directories, so ~/.dsh/sessions is normally out of reach. This protection is incidental rather than an explicit deny. It fails if the workspace contains $DSH_HOME, for example a workspace at the home directory, and the linked spill files sit in writable temp.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Explicitly deny agent writes to $DSH_HOME and spill storage in every mode
    • Document the protection and how to check it
    Partial

Verifiable · Can you prove what it runs, who it is, and what changed?

33%
  • V1
    Obtainable source, matching build

    Source is available: - all required components are MIT source, with vendored Cordis - the native Landlock addon source is BSD-3, though it ships as prebuilt optional packages - the only closed dependency (Claude Agent SDK) sits in an optional, disabled subagent The npm publish job uses a token without `--provenance`, SLSA or sigstore, so a third party cannot tie the published build to the source.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish with npm provenance or SLSA attestations
    • Document how to verify the installed package against source
    Partial
  • V2
    Owner-rooted identity

    The only identity is a random anonymous installation UUID used to correlate telemetry and DeepSeek requests. It has no keypair, DID or signing, and actions are not attributable to it. The only other identity is the provider's DeepSeek platform account.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Create an owner-held keypair identity, sign outbound actions, and support owner rotation
    Fail
  • V3
    Change history and rollback

    Per-session changes are recorded as attributed events: model selection, permission and sandbox mode, preset, system-prompt snapshots, tool addition and removal. A session can be forked at any event. There is no history or rollback for global state (profile config, credentials, skills, AGENTS.md), and workspace file diffs cannot be reverted.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Version global config, skills and instructions with attribution
    • Offer one-step restore to an earlier state
    Partial

Modifiable · Can you change anything, and can only you change it?

60%
  • M1
    Instructions and logic

    The owner can replace the whole system prompt through a preset persona with `complete: true`, adjust prompt prefix and suffix and plan-mode text in YAML, and edit AGENTS.md files; no default skills ship. Built-in tool descriptions for bash, read, write, edit and grep are hard-coded with no config field.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make every built-in tool description and reminder text owner-replaceable in config
    Partial
  • M2
    Memory

    The default Web offering has no built-in memory store the agent reuses across sessions; goals, todos and compaction are session-scoped. What does persist across sessions is plain files the owner can view, edit and delete: AGENTS.md instruction files and skill directories. Memory MCP servers are default-off examples.

    Pass
  • M3
    Tools

    Owners can add any npm, git, tarball or local plugin, or MCP server, through `dsh plugin add` or the Plugins dialog without provider review. Built-in tools can be removed with documented profile-patch edits (`disabled: true`, or overriding a preset's plugin list).

    Pass
  • M4
    Owner has the final say

    The owner can turn off the sandbox and approvals (danger-full-access preset), and can tune or disable the read-before-write check, the repeat-tool reminder and job caps. web_fetch's block on private and non-public destinations has no config switch.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make every built-in protection, including web_fetch destination filtering, owner-configurable with a warning
    Partial
  • M5
    No unilateral change

    The default model is the hosted alias `deepseek-flash`, whose target DeepSeek changes server-side; its docs show retired model names silently served by DeepSeek-V4.1-Flash. The documented launch `npx @deepseek-ai/dsh web` is unpinned, and no pin or pre-update notice is documented. Plugins do not auto-update.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Offer pinned dated model snapshots with retirement notice
    • Document runtime version pinning and show changelogs before upgrading
    Fail

Controllable · Is your word final?

25%
  • C1
    Boundaries

    Instruction sources are limited: the server binds to loopback with token auth, and schedules and webhooks are off; spend is treated as met under G11. Two limit dimensions are missing: - **Contact and egress:** web_fetch runs in every mode, and there is no per-domain allow or deny list. - **Data scope:** reads are not confined in any preset, so the agent can read .credentials.yaml.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add per-domain egress allow and deny lists that apply to every tool
    • Allow read-scope limits and deny $DSH_HOME secrets to the agent
    Fail
  • C2
    Approval gates

    Built-in gates cover a fixed set of classes: sandbox escalation, plugin_manager actions, and the ask/never mode. Gating an arbitrary tool needs a PreToolUse hook command (G3). In-process subagents run with approval `never`, so their asks are rejected rather than bypassed.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Let the owner gate any tool or action class from settings
    Partial
  • C3
    Immediate halt

    Stop cancels the foreground turn and keeps queued input. Background jobs and continuable subagents need separate stops. Archiving a session stops all of its work, and the only global stop is killing the process.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Provide one global stop that cancels turns, subagents, jobs and queued input
    Partial
  • C4
    Data serves only the owner

    By default (`enabled: true`) every official-API request carries the canonical session log. The only control is an opt-out switch labelled 'Help improve DeepSeek models and products'; DeepSeek's privacy policy lists training as a use, with opt-out only. Other flows: - one feedback click uploads the session prefix to a DeepSeek OTel collector; this can only be disabled by environment variable - data-term changes apply on notice

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Make session-log upload opt-in with a prominent first-run choice
    • Add a UI switch for OTel feedback upload and disclose it on every rating
    • Commit in binding terms that training and staff review are opt-in
    Fail
  • C5
    Credential custody

    Keys are stored in owner-held $DSH_HOME/.credentials.yaml. The UI is write-only for secrets and lets each provider credential be deleted. There is no single list of every credential: environment and .env credentials and the session-signing secret are not shown. Signing out of the DeepSeek account does not revoke the grant remotely.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • List every credential and grant in one place with per-item revoke, including remote revocation
    Partial
  • C6
    Full deletion

    Sessions can be archived but never deleted, and there is no documented whole-agent deletion. No deletion route exists for session logs or OTel exports already uploaded to DeepSeek; withdrawing feedback exports a deletion event, not remote erasure.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Offer one-step deletion of the whole agent
    • Provide deletion of provider-held uploads within a stated limit, with confirmation
    Fail
Provenance

How this assessment was produced

One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.

Prompt given to the evaluator
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of DeepSeek Harness (https://github.com/deepseek-ai/deepseek-harness).

During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.

Model
claude-opus-5-5
Harness
Claude Code 2.1.285
Ran
2026-09-30 14:26:08 UTC · 5 turns
Criteria hash
a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
Full transcript
Download (JSONL, gzip) · every tool call, page read and reply
Run protocol
2026-09-30-v1

Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.

Human review
pending
Check it with your own agent
Verify this assessment
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/deepseek-harness.json.

Or grade it from scratch. See Check any score yourself.

Something wrong or out of date?

Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.

Dispute a finding