Skip to content
Agents of Humanity
Public preview. Results are preliminary (criteria v1.0, 30 Sept 2026). Tell us what we got wrong
Open sourceself-hostedMIT

AnythingLLM

Mintplex Labs · anythingllm.com

AnythingLLM Desktop is a local-first workspace and agent app. All state sits in a documented storage folder on the owner's machine, it runs on local open-weight models by default, and the owner can add custom skills, MCP servers and flows without review. It has no documented whole-agent export or restore, no durable tool-call record with integrity protection, and no agent identity. The Desktop app itself ships under a proprietary licence that forbids modification, while its server core is MIT.

Strengths
  • Local-first: all agent state stays in an owner-held folder (P5)
  • Runs on local open-weight models by default, with a text-based tool-calling fallback (P4)
  • Owner can add skills, MCP tools and flows without review, and disable built-in skills (M3)
  • No auto-update or remote flags; owner-initiated updates act as a version pin (M5)
Gaps
  • No documented export/import of the whole agent; only chat logs can be exported (P1, P2)
  • Agent tool calls and their arguments are not durably recorded, and no record has integrity protection (A1, A2)
  • Desktop wrapper is closed source under a no-modification licence (V1)
  • Telemetry with a stable instance ID is on by default, with no disclosed retention period (T3, C4)
Evidence

All 26 findings

AnythingLLM Desktop v1.16.2 (released 2026-09-22). Open-source server core read at commit 067806d4bf8211699265f82302fa07a5722ea461 (server package version 1.16.2) of github.com/Mintplex-Labs/anything-llm. The Desktop Electron wrapper is not public and could not be inspected. Desk review on Linux, 2026-09-30. First-run defaults per docs: built-in local LLM (Ollama engine, labelled 'Built-in (default)'), native embedder (all-MiniLM-L6-v2), LanceDB. Agent web browsing is on by default with You.com search.

Portable · Can you leave, take the whole agent with you, and keep it alive?

50%

Transparent · Can you see everything the agent is, and everything shaping it?

63%
  • T1
    Readable without the provider

    State is kept in SQLite with a public Prisma schema, JSON documents and vector cache, LanceDB (an open format), JSON flow definitions, JavaScript skill source and a plaintext .env, all in an owner-held folder. Built-in skill and prompt logic is published as MIT source, and no provider-held keys are involved.

    Pass
  • T2
    What runs is what you see

    The UI shows the workspace system prompt, the model, and enabled skills and MCP tools. Some context additions can be seen only in source code: the agent role prompt, the appended clarifying-questions instruction, the UnTooled function-picker prompt and the memory header. No view or log of the fully assembled context was found, so G3 caps this test at partial.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add a view or log of the full assembled context for each model call
    • Surface hardcoded prompt additions in the UI
    Partial
  • T3
    No shadow data

    On by default, the app sends usage events (chat sent, agent tool name, documents uploaded, LLM provider and model, etc.) to PostHog with a stable per-instance UUID. The Desktop privacy policy does not disclose how long this data is kept. The README says sent events appear in the local output log, but retention is undisclosed.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish a bounded retention period for telemetry and any other data held
    • Or keep telemetry off by default so the provider holds nothing
    Fail
  • T4
    No hidden steering

    The one provider-chosen default found is You.com as the agent web-search provider, which the changelog and blog disclose as a partnership. The owner can switch the search provider or disable web browsing. Source code shows no sponsored or paid placement in agent context or tool defaults. The Community Hub pages could not be opened (HTTP 403).

    Pass

Auditable · Can you reconstruct exactly what the agent did, and trust the record?

17%
  • A1
    Complete action record

    Interactive agent sessions store only the final response in workspace_chats. Tool calls and their arguments go only to the live websocket and are not stored (scheduled-job runs do store a full trace). No mechanism tells the owner when recording fails. Two shortfalls apply: entries lack arguments and targets, and recording can fail silently.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Persist every tool call with its arguments and target, from every entry path
    • Tell the owner when an action could not be recorded
    Fail
  • A2
    Tamper-evident, owner-checkable

    Chats, event logs, invocations and job runs are plain SQLite rows with no hashing, signing or chaining. An admin endpoint can clear all event logs.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Hash-chain or sign action-record entries
    • Anchor checkpoints with the owner and publish an open verifier
    Fail
  • A3
    Agent cannot rewrite its record

    Built-in default skills cannot write the chat or event tables. The SQL agent connects only to external databases, and the filesystem agent is sandboxed and available only in Docker. However, owner-enabled custom skills (in-process Node handlers) and stdio MCP servers run arbitrary code with access to anythingllm.db, and nothing separates the record from them. The protection holds by default except for this one path.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Write the record from a component that skills and MCP processes cannot reach
    • Document the protection
    Partial

Verifiable · Can you prove what it runs, who it is, and what changed?

33%
  • V1
    Obtainable source, matching build

    The Desktop Electron wrapper, a required component of the evaluated offering, has no public source. Its Terms of Use call the software's source code a trade secret and forbid modification and reverse engineering. The server core is MIT, and release binaries ship without checksums or provenance.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Publish the Desktop wrapper source under a licence that allows modification
    • Provide reproducible builds or verifiable provenance for release binaries
    Fail
  • V2
    Owner-rooted identity

    The agent has no identity of its own. Outbound actions go through channels the owner holds, such as a Gmail Apps Script deployment the owner runs or an Outlook OAuth app the owner registers, with no Mintplex client or broker. Attribution is therefore per channel only.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Give the agent an owner-held key or identity that recipients can check across channels
    Partial
  • V3
    Change history and rollback

    Workspace system prompt changes are kept in a prompt history with author and one-click restore, and the event log records that the LLM, embedder or vector DB changed (without values). Other settings, skills, memories and model changes have no change history or restore.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Log and attribute every state change (settings, skills, memories, model)
    • Offer restore to a chosen earlier state
    Partial

Modifiable · Can you change anything, and can only you change it?

70%
  • M1
    Instructions and logic

    The workspace system prompt and the instance default system prompt can be edited in the UI, and custom skills and flows are owner-defined. Several hardcoded instructions (agent role prompt, clarifying-questions instruction, UnTooled tool-selection prompt, memory header) can be changed only by editing source.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Expose every hardcoded agent instruction as an owner-editable setting
    Partial
  • M2
    Memory

    Memories (workspace and global) are listed with create, edit and delete controls. However, the default-on rag-memory skill's 'store' action embeds text straight into the workspace vector DB with no document record, so those items are not itemised or individually editable in the UI.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • List agent-stored vector memories as items with edit and delete controls
    Partial
  • M3
    Tools

    The owner can add custom skills (plugins/agent-skills handler.js), MCP servers and Agent Flows without provider review. Every built-in skill, including the four default-on ones, can be disabled, and individual MCP tools can be suppressed.

    Pass
  • M4
    Owner has the final say

    Some protections are configurable (tool-call chain limit, approval timeout, MCP cooldown, auto-approved skills), but on Desktop only through environment variables that the docs tell Desktop users not to edit. Others are hardcoded: the 100-round agent limit, the 5-minute agent websocket timeout, and the memory count limits.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Expose every protection as a documented Desktop setting with a warning
    Partial
  • M5
    No unilateral change

    Desktop updates are owner-initiated reinstalls (an in-app banner only links to downloads), and no remote flag or server-side configuration code was found. The default model is local. Keeping the installed version acts as an indefinite pin.

    Pass

Controllable · Is your word final?

42%
  • C1
    Boundaries

    Contact and web reach are on/off only: the Gmail/Outlook skills and web browsing/scraping have no per-recipient or per-domain allow or deny lists. There is also no owner control over which sources the agent takes instructions from. Spend is treated as met under G11 because there is no payment capability. File scope has per-folder grants, but only in Docker.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add per-recipient and per-domain allow/deny lists for messaging and web tools
    • Let the owner limit which sources may instruct the agent
    Fail
  • C2
    Approval gates

    Approval prompts exist only for a fixed set of built-in write actions (filesystem writes, file creation, Gmail/Outlook/Calendar writes, job creation) and for custom skills that request them. MCP tools, SQL, web and flows cannot be gated, and scheduled-job runs auto-approve every tool.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Let the owner gate any tool, including MCP and flows
    • Make scheduled runs honour gates, for example by queuing actions for approval
    Partial
  • C3
    Immediate halt

    The stop button fully aborts the foreground agent session and inference (v1.16.0). Scheduled jobs must be stopped per run or disabled separately, and background jobs such as memory extraction keep running. There is no single global stop.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Add one global stop that cancels in-flight, scheduled and background agent work until the owner resumes
    Partial
  • C4
    Data serves only the owner

    Non-content telemetry is on by default and can be disabled in Settings > Privacy. By default, agent web-search queries go to You.com, a destination the owner did not specifically choose, though web browsing can be turned off. The privacy policy says data is not sold. Policy changes take effect after posted notice, without specific consent.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Turn telemetry off by default
    • Make third-party web search opt-in
    • Ask for specific consent before changing data terms
    Partial
  • C5
    Credential custody

    All credentials are held locally: provider keys in the plaintext storage .env, integration configs in SQLite. Mintplex holds none and uses no broker. The UI shows only whether each key is set, across several settings pages, and LLM provider keys can be overwritten but not individually revoked or listed in one place.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • List every credential and grant in one view with a revoke control for each
    Partial
  • C6
    Full deletion

    No provider-side copy of the agent exists. Full deletion is a documented multi-step manual process (uninstall, then delete the storage folder) with no in-app single action or completion confirmation. Telemetry already sent to PostHog is not agent state.

    › How it could pass

    Suggestions. Any change that meets the test counts.

    • Offer a one-step in-app 'delete everything' action with confirmation
    Partial
Provenance

How this assessment was produced

One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.

Prompt given to the evaluator
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of AnythingLLM (https://anythingllm.com).

During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.

Model
claude-opus-5-5
Harness
Claude Code 2.1.285
Ran
2026-09-30 13:59:33 UTC · 10 turns
Criteria hash
a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
Full transcript
Download (JSONL, gzip) · every tool call, page read and reply
Run protocol
2026-09-30-v1

Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.

Human review
pending
Check it with your own agent
Verify this assessment
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/anythingllm.json.

Or grade it from scratch. See Check any score yourself.

Something wrong or out of date?

Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.

Dispute a finding