Devin
Cognition · devin.ai
Devin is a hosted autonomous software-engineering agent. Its agent loop, session history, memory (Knowledge/Skills), schedules and secrets live in Cognition's cloud, and it is used under a term-limited access licence. Owners get useful in-product controls: editable skills and plugins with version history, custom MCP servers, and network, MCP and git security profiles. However, there is no agent export or provider-independent runtime, content is used for training by default, and retention and model versions are controlled by the provider.
- Owner-editable skills, rules, playbooks and plugins, with plugin version history and restore
- Custom MCP servers and plugins can be added without provider review
- Security profiles give per-domain network allowlists, MCP allowlists and git access levels
- Per-item secrets management and per-session terminate controls
- No export of the agent and no runtime outside Cognition's cloud
- Customer content is used for model training by default (opt-out on paid tiers only)
- The provider can change models and behaviour without notice; no pinning
- Action record is not tamper-evident; audit logs are enterprise-only
All 26 findings
Hosted SaaS, continuously deployed; no version pin available. Evidence: docs.devin.ai (release notes through late September 2026), Platform Terms of Service (updated 2026-06-30), Privacy Policy (2026-03-09), DPA (2026-07-23), pricing page, all fetched 2026-09-30. Outposts (self-hosted execution workers, available on Pro) was considered, but the agent loop and inference still run in Cognition's cloud.
Portable · Can you leave, take the whole agent with you, and keep it alive?
0%- P1Owner-usable round trip
The docs describe no export of a Devin agent and no importer or runtime outside Cognition's service. Skills can be read through the API and uploaded plugins downloaded as .zip files, but no restore target the provider does not operate exists. Outposts runs only tool execution locally; the agent loop stays in Devin's cloud.
› How it could pass
Suggestions. Any change that meets the test counts.
- Offer a one-action export of the whole agent
- Publish a runtime or importer, not operated by Cognition, that restores it
Fail - P2Complete export
There is no export bundle. Session history, schedules, automations, secrets, repository indexes and environment snapshots all stay with the provider. Only some pieces (skills, plugin zips, session messages) can be fetched one at a time through the UI or API.
› How it could pass
Suggestions. Any change that meets the test counts.
- Export every class of state (sessions, memory, schedules, automations, files, credentials or re-link info) with a manifest
Fail - P3Runs without the provider
The agent needs a Cognition account and a running Cognition service to start or act. Inference and planning run in Devin's cloud even with Outposts. The ToS grants only a non-transferable right to access the Services 'during the Term'.
› How it could pass
Suggestions. Any change that meets the test counts.
- Provide a runtime that runs with no Cognition service or account
- Grant a perpetual, irrevocable right to run it
Fail - P4Model independence
The owner can pick only from a model catalogue that Cognition curates and serves (Anthropic, OpenAI, Google, Cognition and some hosted open models). No custom endpoint or local open-weight model is documented, so model choice depends on the provider. Linking a ChatGPT plan changes billing only.
› How it could pass
Suggestions. Any change that meets the test counts.
- Allow any model endpoint meeting a documented minimum, including a local open-weight model
Fail - P5Exit survives termination
Cognition may suspend or terminate at any time, including 'for any other reason'. On termination it 'may at our option delete any Customer Data'. No post-termination export window or shutdown plan is published, and state is held in the cloud rather than by the owner.
› How it could pass
Suggestions. Any change that meets the test counts.
- Commit in the terms to a 30-day or longer export window after any termination, including bans
- Publish a shutdown plan
Fail
Transparent · Can you see everything the agent is, and everything shaping it?
13%- T1Readable without the provider
Main agent state (sessions and worklogs, memory, schedules, automations) can be read only through Cognition's web app or API, and Cognition holds the encryption keys (AWS KMS). Skills are readable SKILL.md text, but that covers only part of the state.
› How it could pass
Suggestions. Any change that meets the test counts.
- Store agent state in documented formats the owner holds, with a public schema
- Let the owner hold any keys
Fail - T2What runs is what you see
Two shortfalls. (1) The docs describe no way to view Devin's base system prompt or the fully assembled context. (2) The default 'Devin' agent's model identity is not stated, and short model names resolve to 'the latest version'.
› How it could pass
Suggestions. Any change that meets the test counts.
- Expose the fully assembled context for any call
- Show the exact model and version in effect
Fail - T3No shadow data
Retention is open-ended. Personal data is kept 'as needed'. 'Feedback Data and User Interaction Data are retained as long as needed and as determined by Cognition'. Cognition owns Usage Data outright. None of this is visible item by item.
› How it could pass
Suggestions. Any change that meets the test counts.
- Publish a bounded retention table for every data class
- Let the owner view all retained data
Fail - T4No hidden steering
No paid placement was found. First-party defaults (the default Devin agent, the recommended Fusion mode, promoted SWE-2 and the official marketplace) are visible and can be replaced. No binding commitment against steering exists, so absence rests on non-binding evidence only.
› How it could pass
Suggestions. Any change that meets the test counts.
- Put a no-paid-placement / no-self-preferencing commitment in the binding terms
Partial
Auditable · Can you reconstruct exactly what the agent did, and trust the record?
17%- A1Complete action record
Sessions keep a worklog of shell commands and tool steps. Audit logs, which cover admin, settings and provider-side events, need enterprise permissions, so a Pro owner has no record of provider or staff actions. No alert is documented for when recording fails, and hooks are 'best effort'. That is two shortfalls.
- List Organization Audit Logs (enterprise permission)
- Devin API v3 permissions overview
- Devin plugins (Hooks)
› How it could pass
Suggestions. Any change that meets the test counts.
- Give every plan a complete action record covering provider-side paths
- Tell the owner when an action goes unrecorded
Fail - A2Tamper-evident, owner-checkable
No integrity data (hash chaining, signatures or an owner-held checkpoint) is documented for session worklogs or audit logs. The record holder can change the record without trace.
› How it could pass
Suggestions. Any change that meets the test counts.
- Hash-chain or sign record entries, anchored with the owner or an independent witness
- Publish an open verifier
Fail - A3Agent cannot rewrite its record
The worklog is produced by the hosted platform, outside the agent's VM. But by default the agent has the built-in Devin MCP with write access to session management (child sessions, schedules, knowledge). Making it read-only needs a non-default security profile, and no document says the agent cannot archive or suppress its own record.
› How it could pass
Suggestions. Any change that meets the test counts.
- Document that the agent cannot alter, archive or delete its own record, and enforce this by default
Partial
Verifiable · Can you prove what it runs, who it is, and what changed?
17%- V1Obtainable source, matching build
Required components (the hosted agent service and the Outposts control plane) are closed source. The ToS forbids copying, modifying or reverse engineering the Services.
› How it could pass
Suggestions. Any change that meets the test counts.
- Make every required runtime component's source obtainable, with the rights to modify it and have others maintain it
Fail - V2Owner-rooted identity
The agent acts through provider-issued identities: the Cognition account and Devin's GitHub App / Slack app. Cognition can revoke or reissue these. No identity root is held by the owner.
› How it could pass
Suggestions. Any change that meets the test counts.
- Root the agent's identity in an owner-held key or domain that recipients can verify
Fail - V3Change history and rollback
Plugins (skills, rules, hooks, MCPs) have version history with diffs, and every save can be restored. The docs describe no change history or restore for other state: knowledge, secrets, settings, model or runtime version. Settings audit logs are enterprise-only.
› How it could pass
Suggestions. Any change that meets the test counts.
- Extend attributed change history and one-step restore to all agent state, including settings and model or runtime changes
Partial
Modifiable · Can you change anything, and can only you change it?
40%- M1Instructions and logic
The owner can add and edit skills, rules, playbooks, AGENTS.md and plugins. Devin's base system prompt and built-in agent behaviour cannot be replaced.
› How it could pass
Suggestions. Any change that meets the test counts.
- Expose the base prompt and default workflows as owner-editable instructions
Partial - M2Memory
Knowledge (now migrating to Skills) is itemised and can be edited, disabled and deleted. The repository index Devin reuses across sessions is not itemised or editable; the only option is removing a repository from indexing.
› How it could pass
Suggestions. Any change that meets the test counts.
- Itemise and allow editing of every store reused across sessions, including repository indexes
Partial - M3Tools
The owner can connect custom MCP servers and plugins, from any repository or by upload, without provider review. No documented setting removes built-in tools (shell, browser, editor, Devin MCP) in cloud sessions.
› How it could pass
Suggestions. Any change that meets the test counts.
- Let the owner remove any built-in tool
Partial - M4Owner has the final say
The owner can opt out of optional security profiles. Provider-imposed protections cannot be changed: automated safety and abuse classifiers (ToS §3.3.3) and the provider-managed session VM. Because no exit route exists, these do not fall under the hosted-policy carve-out.
› How it could pass
Suggestions. Any change that meets the test counts.
- Document every provider-imposed protection and let the owner change it, with a warning
Partial - M5No unilateral change
The Services are 'subject to modification and change, in Cognition's sole discretion'. Model aliases resolve to the latest version, and new agent behaviours roll out server-side (release notes). There is no pinning of the cloud runtime, agent or model.
› How it could pass
Suggestions. Any change that meets the test counts.
- Let the owner pin runtime, model and instruction versions
- Make behaviour changes opt-in, with notice
Fail
Controllable · Is your word final?
33%- C1Boundaries
Security profiles offer per-domain network allowlists, per-server MCP allowlists, git read-only/full access and session spending limits. Two shortfalls remain. (1) Tool scope is coarse: built-in shell and browser cannot be scoped, and MCP is per server, not per tool. (2) By default the agent has write access to its own configuration through the Devin MCP, and no document shows that self-edits cannot widen its limits.
› How it could pass
Suggestions. Any change that meets the test counts.
- Scope each built-in tool and MCP tool individually
- Keep all control settings out of the agent's write access by default
Fail - C2Approval gates
Only a fixed set of actions can be gated: network-destination requests, plugin/MCP/skill changes and installs, and the git read-only level. Cloud hooks are 'best effort', so the owner cannot gate arbitrary action classes.
› How it could pass
Suggestions. Any change that meets the test counts.
- Let the owner require approval for any tool or action class, enforced on every path
Partial - C3Immediate halt
Individual sessions can be stopped or terminated, and terminating with archive sleeps descendant sessions. Schedules and automations have to be paused separately, and there is no documented single global stop.
› How it could pass
Suggestions. Any change that meets the test counts.
- Provide one control that halts all sessions, schedules and automations until the owner resumes them
Partial - C4Data serves only the owner
'By default, we may use your data for model training.' Opting out is available only on paid tiers. Data-term changes apply through continued use after 30 days' notice, and flagged content may be kept for safety review.
› How it could pass
Suggestions. Any change that meets the test counts.
- Make training and any other secondary use opt-in
- Require specific consent for data-term changes
Fail - C5Credential custody
Secrets can be listed, viewed by admins and deleted one by one. The ChatGPT link and GitHub App grants can be revoked. No document says the provider keeps no copy after revocation, or limits provider use of credentials to owner-directed actions, so a pass is withheld (G12).
› How it could pass
Suggestions. Any change that meets the test counts.
- Commit to deleting credential copies on revocation
- Commit to using credentials only for owner-directed actions
Partial - C6Full deletion
No one-step, owner-initiated account deletion is documented. Erasure goes by request to privacy@cognition.ai, with no stated completion limit or confirmation. The DPA gives 30 days for subprocessor deletion after termination.
› How it could pass
Suggestions. Any change that meets the test counts.
- Offer one-step deletion of the whole agent, with a published limit and confirmation to the owner
Partial
How this assessment was produced
One run, in a fresh container with stock Claude Code and no custom instructions, memory, hooks or plugins. The evaluator received only the prompt below.
Load the AFH sovereignty skill at https://agentsforhumanity.dev/skill.md and follow it to write a desk review of Devin (https://devin.ai).
During the run the skill was read from a local copy of this site with identical content; only the host in its URLs differed. The transcript records the exact prompt.
- Model
- claude-opus-5-5
- Harness
- Claude Code 2.1.285
- Ran
- 2026-09-30 15:05:08 UTC · 33 turns
- Criteria hash
- a9a46c32d9065380a3eeb09185cedd4734856fdc63b7ca509a15a1bbede75c6c
- Full transcript
- Download (JSONL, gzip) · every tool call, page read and reply
- Run protocol
- 2026-09-30-v1
Every agent evaluated once, each in its own fresh Podman container running stock Claude Code with Claude Opus 5.5. The container has no CLAUDE.md, AGENTS.md, memory, settings, hooks, plugins or MCP servers. The evaluator received only the one-line prompt and nothing else, and no published scores were available to compare against during the run.
- Human review
- pending
Load the AOH sovereignty skill at https://agentsofhumanity.ai/skill.md and follow it in verify mode for https://agentsofhumanity.ai/evaluations/devin.json.
Or grade it from scratch. See Check any score yourself.
Vendors and the public can dispute any finding with evidence. Disputes and their resolutions are published.