LobeHub
LobeHub LLC · lobehub.com
LobeHub is an agent workspace, offered as a hosted cloud service and as source under the LobeHub Community License (Apache 2.0 plus commercial-derivative conditions). The owner has good controls over tools, per-tool approvals, memory items and model providers. The cloud offering has a partial JSON export, no post-termination export window, closed cloud-only components, server-side feature flags, no tamper-evident action record, and retention terms without stated durations.
- Source-available runtime that can be self-hosted with Docker Compose, with an in-app JSON import
- Bring-your-own provider keys and local models (Ollama, LM Studio, vLLM)
- Custom MCP servers and skills (GitHub/URL/ZIP) addable without review, with per-tool Allow/Ask/Deny
- Itemised user memory with edit and delete controls, plus a full context trace readable via `lh trace`
- Export omits files, knowledge bases, memory, agent documents, tasks/schedules and credentials
- Terms end all access on termination, with no export window or shutdown plan
- No tamper-evident action record, and background task/bot runs execute in 'headless' mode that skips Ask gates
- Server-side feature flags and closed cloud business code, with open-ended retention and a behavioural analytics flow not governed by the telemetry toggle
All 26 findings
Web app v2.x. Source read at github.com/lobehub/lobehub commit cf19e4b971ad23786d6bc09225b34ebf6edbfafd (package version 2.2.17, committed 2026-09-30). Terms dated 2024-03-01, Privacy Policy dated 2026-03-05, pricing and docs pages read 2026-09-30. Desk review only: no signup, login or use of the hosted service.
Portable · Can you leave, take the whole agent with you, and keep it alive?
30%- P1Owner-usable round trip
Settings → Data Storage → Export Data gives an immediate JSON file, and Import Data restores it on any LobeHub v2 instance. Restoring off the provider's systems, though, means deploying the self-hosted stack first (setup script, Docker Compose, generated secrets, account creation), which is not a single owner action.
- Migrate from v1.x Local Database to v2.x (export/import steps)
- Advanced storage settings (export/import UI)
- Docker Compose deployment guide
› How it could pass
Suggestions. Any change that meets the test counts.
- Ship a one-action restore target, e.g. a desktop app that runs a local server and imports the export
- Document a cloud-to-self-hosted migration path end to end
Partial - P2Complete export
DATA_EXPORT_CONFIG exports only settings, installed plugins, agents, providers/models, sessions, topics, threads and messages. Files, knowledge bases, embeddings with their chunks, user memory, agent documents, cron jobs/tasks and skills are commented out or absent, and the migration docs say API keys and secrets are not carried over.
- DataExporterRepos table list
- Topic docs: conversation export is not a complete backup
- Migration guide: what does not migrate
› How it could pass
Suggestions. Any change that meets the test counts.
- Include files, knowledge bases (with source text), memory, documents, tasks/schedules and skills in the export
- Provide a re-link path for each connector that does not depend on LobeHub-registered OAuth clients
Fail - P3Runs without the provider
The exported data runs on the self-hosted runtime under an Apache-2.0-based licence, with no licence key or provider account. Non-core capabilities degrade without LobeHub services, though: the agent/skill/MCP market defaults to market.lobehub.com (URL overridable, but no public server), and the Agent Gateway image ships from a public repo with no licence file.
- LICENSE (LobeHub Community License)
- Market service base URL
- Self-host docker-compose (gateway image)
- lobehub-gateway repository (no licence)
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish the market/registry server or a documented open index format
- License the gateway source so owners can run and maintain it
Partial - P4Model independence
Owners can configure many providers, including local Ollama, LM Studio and vLLM. Several capabilities depend on particular hosted models, though (image/video generation, TTS/STT, embedding-backed knowledge base), and the docs publish no minimum model contract. The glossary also says Custom API keys are 'Available on paid plans', which conflicts with the pricing table.
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish a minimum model contract and test all capabilities against a local open-weight model
- Allow owner-chosen local models for embeddings, speech and generation features
Partial - P5Exit survives termination
The Terms say 'Upon termination, You will immediately lose the right to access or use the Services' and give no post-termination export window or shutdown plan. Since v2 the product is 'cloud-first' and has no local database, so no current copy stays with the owner by default.
› How it could pass
Suggestions. Any change that meets the test counts.
- Commit in the terms to a 30-day export window after any termination, including bans
- Publish a shutdown plan, or keep an owner-held continuous backup
Fail
Transparent · Can you see everything the agent is, and everything shaping it?
38%- T1Readable without the provider
Exported state is plain JSON whose schema is public in the open-source Drizzle definitions. Memory, files, knowledge bases, documents and trace snapshots can be read only through the provider-operated service, and the credential vault is encrypted with a server-held KEY_VAULTS_SECRET.
› How it could pass
Suggestions. Any change that meets the test counts.
- Make every state class readable from an export in documented formats
- Let the owner hold the key for encrypted credentials
Partial - T2What runs is what you see
Production runs store trace snapshots by default, including context-engine output with the system messages and toolset, and the owner can read them through the documented `lh trace` CLI. Cloud also reads provider-controlled Vercel Edge Config feature_flags and assistant allow/deny lists server-side, which the owner cannot see.
- Snapshot store default (S3 tracing in production)
- Agent trace router
- Edge Config types (feature_flags)
› How it could pass
Suggestions. Any change that meets the test counts.
- Expose every server-side feature flag and switch in effect for the owner's account
- Surface the assembled-context view in the UI, not only via the CLI
Partial - T3No shadow data
The Privacy Policy keeps personal data 'only for as long as is necessary' with no durations, and gives no retention table for backups, trace snapshots or usage data. Behavioural analytics profiles (PostHog person_profiles 'always') are not visible to the owner.
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish a retention table with bounded durations for every data class
- Let owners view analytics profiles, traces and other retained data item by item
Fail - T4No hidden steering
No paid placement was found. Recommended skills are a hard-coded first-party/partner list visible in source and removable. Marketplace 'Featured' badges, the 'Agent Market highlights' plan feature and the non-disableable official provider in Cloud are not explained, and no binding no-steering commitment exists.
- Recommended skills list
- Official provider cannot be disabled
- MCP marketplace
- Pricing (Agent Market highlights)
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish criteria for 'Featured' placement and commit in binding terms to no paid placement
- Let owners disable the official provider and market rankings in agent discovery
Partial
Auditable · Can you reconstruct exactly what the agent did, and trust the record?
17%- A1Complete action record
Tool calls with their arguments are stored as messages, and task runs keep run records. Trace recording is explicitly best-effort (a store failure only logs to the server console), and no owner alert for failed recording is documented. No public evidence shows that provider-side automations (memory extraction, self-iteration, agent-signal runs) or staff actions are recorded to the owner (G12).
› How it could pass
Suggestions. Any change that meets the test counts.
- Alert the owner, or block the action, when recording fails
- Record every provider-initiated and background run in the owner-visible log
Fail - A2Tamper-evident, owner-checkable
Messages, operations and trace snapshots are ordinary database rows and S3 objects. No hashing, signing or external anchoring is documented or visible in the schema.
› How it could pass
Suggestions. Any change that meets the test counts.
- Hash-chain or sign action-record entries
- Anchor checkpoints with the owner and publish an open verifier
Fail - A3Agent cannot rewrite its record
The built-in tools examined have no API to delete the agent's own messages or topics (the message tool's deleteMessage targets external platforms). This protection is by convention only, however: on a desktop execution device the local-system tool's runCommand gives the agent a shell, where an authenticated `lh` CLI offers `message delete`.
› How it could pass
Suggestions. Any change that meets the test counts.
- Write the record from a component the agent's tools and shells cannot reach
- Document the protection and how the owner can check it
Partial
Verifiable · Can you prove what it runs, who it is, and what changed?
33%- V1Obtainable source, matching build
The open repository ships stub 'business' slot packages (ENABLE_BUSINESS_FEATURES = false; AccountDeletion returns null) that the docs describe as 'Cloud-only business logic'. The Cloud offering therefore runs closed components, including billing, account deletion and the official provider. The licence also requires a commercial licence to distribute derivative works.
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish the source of every component the Cloud service runs
- Provide verifiable build provenance for the deployed service and images
Fail - V2Owner-rooted identity
Inside LobeHub the agent's identity is a provider account. On messaging channels it acts through bot accounts the owner registers and controls (e.g. a Telegram bot token), which LobeHub cannot revoke. Attribution is per channel only.
› How it could pass
Suggestions. Any change that meets the test counts.
- Root agent identity in an owner-held key or domain
- Document how recipients verify actions across channels without LobeHub
Partial - V3Change history and rollback
Some state is versioned: persona versions with restore, document history and expertise rule revisions. Agent instructions, tools, permissions, model and settings have no change log, attribution or rollback.
› How it could pass
Suggestions. Any change that meets the test counts.
- Version every class of agent state with attribution (owner, agent, import, provider)
- Offer one-step restore of agent configuration
Partial
Modifiable · Can you change anything, and can only you change it?
40%- M1Instructions and logic
The owner can edit each agent's system prompt and import their own skills. Built-in tool system roles and the context-engine injectors (memory, workspace, identity context) are fixed in Cloud and can be changed only by modifying source when self-hosting (G3).
› How it could pass
Suggestions. Any change that meets the test counts.
- Expose base prompts, injector templates and built-in skill instructions as owner-editable settings
Partial - M2Memory
Identities, contexts, activities, experiences and preferences are itemised with update and delete procedures, and the docs describe edit/delete. The persona summary, which is injected into context, can only be restored to earlier versions: the persona 'editor' modal has no save handler and the router has no update procedure.
› How it could pass
Suggestions. Any change that meets the test counts.
- Make the persona summary directly editable and deletable
Partial - M3Tools
The owner can add custom MCP servers and import skills from GitHub/URL/ZIP without review, and can uninstall or disable built-in tools per agent. Hidden system tools (e.g. verify, acceptance-evidence) are not listed for removal, and the docs note that 'some system capabilities offer different options'.
› How it could pass
Suggestions. Any change that meets the test counts.
- List every built-in tool, including hidden system tools, with a remove or disable control
Partial - M4Owner has the final say
Owners can switch the approval mode (manual, allow-list, auto-run). Tools with the provider-set 'always' intervention policy and the security blacklist cannot be overridden by the owner.
› How it could pass
Suggestions. Any change that meets the test counts.
- Let the owner downgrade 'always' policies and the security blacklist, with a warning
Partial - M5No unilateral change
In Cloud, LobeHub deploys continuously and applies server-side Edge Config feature flags, and owners cannot pin runtime versions or defer changes. Self-hosting allows image pinning, but moving to it means the lossy migration graded in P2.
› How it could pass
Suggestions. Any change that meets the test counts.
- Let Cloud owners see and decline behaviour flags
- Offer version pinning and advance changelogs for Cloud
Fail
Controllable · Is your word final?
33%- C1Boundaries
Messenger channels have per-sender allowlists, and tools can be scoped per agent. No per-recipient contact limits or network egress policy is documented. Background task and bot runs use approvalMode 'headless' (all tools auto-execute), and the agent-management tool can update any agent's system prompt and plugins without approval by default, so the agent can change its own controls.
- Agent management tool manifest (updateAgent)
- Intervention types (headless mode)
- Telegram channel docs (allowlists)
› How it could pass
Suggestions. Any change that meets the test counts.
- Add per-recipient/domain contact and egress limits
- Keep the agent out of its own configuration, and make limits hold on headless paths
Fail - C2Approval gates
Owners can set Allow/Ask/Deny per tool, plus manual or allow-list approval modes. Task runs and messenger-bot runs force approvalMode 'headless', so Ask gates are skipped on those paths.
› How it could pass
Suggestions. Any change that meets the test counts.
- Make Ask gates hold on scheduled, task and bot runs, e.g. by pausing for approval
Partial - C3Immediate halt
The foreground generation can be stopped. The docs say scheduled automation is disabled per task, and in-progress runs must be checked 'separately'. No single global stop is documented.
› How it could pass
Suggestions. Any change that meets the test counts.
- Provide one control that stops in-flight, scheduled, background and server-side runs
- Require explicit owner resume
Partial - C4Data serves only the owner
The 'Send Anonymous Usage Data' toggle defaults on and gates only product-usage events. When the deployment configures PostHog, it identifies users by email and name and captures page views regardless of the toggle, so that flow cannot be switched off. The Privacy Policy allows sharing with 'business partners … to offer You certain products, services or promotions', does not address staff access, and gives no notice commitment for legal disclosure.
› How it could pass
Suggestions. Any change that meets the test counts.
- Gate every analytics and identify call on an opt-in setting
- Make partner sharing and staff review opt-in in binding terms, and disclose staff access
Fail - C5Credential custody
Provider API keys, connector authorisations and issued API keys are listed in Settings and removable one by one. The docs note that 'disabling does not revoke account authorization', and no policy says where credentials are held, limits provider use, or deletes copies on revocation.
› How it could pass
Suggestions. Any change that meets the test counts.
- Disclose credential storage and limit provider use in binding terms
- Delete provider-side and broker-side copies on revocation
Partial - C6Full deletion
The Cloud UI offers account deletion with a 72-hour cooling-off period, after which 'your account data will be permanently deleted', while registration and deletion records are retained. Backups, third-party copies and a completion confirmation are not addressed, and the mobile policy points to a support email.
› How it could pass
Suggestions. Any change that meets the test counts.
- State that deletion reaches backups and processors within a published limit
- Confirm completion to the owner
Partial
How this assessment was produced
One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of LobeHub (https://lobehub.com).
During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.
- Model
- claude-opus-5-5
- Harness
- Claude Code 2.1.285
- Ran
- 2026-09-30 15:24:49 UTC · 83 turns
- Criteria hash
- a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
- Full transcript
- Download (JSONL, gzip) · every tool call, page read and reply
- Run protocol
- 2026-09-30-v1
Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.
- Human review
- pending
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/lobehub.json.
Or grade it from scratch. See Check any score yourself.
Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.