coding-agent / rank 25 of 41 / cycle 04
Devin benchmark
The slowest median in the index by a wide margin and the most expensive route through the spec, because it works in its own environment and takes the long way through every task. What it buys is unattended completion: it opened a pull request with passing tests without a human in the loop in four of five runs. Judged on the published axes the cost weighs heavily against it, which is why the value subscore is the lowest in the agent group.
Composite index
66.36
of 100 | ▲ 1 versus cycle 03
At a glance
Composite index
66.36
rank 25 of 41
Rank
#25
of 41 measured builders
Median speed
353.6 s
p10 230.2 s, p90 651.4 s
Reliability
71.1%
strict pass rate
SEO score
54.0
0 to 100, SEO and GEO axis
Three strongest axes
- Code ownership94.0
- Scalability80.0
- Agent performance78.0
Three weakest axes
- Speed8.9
- Value52.0
- SEO and GEO54.0
Median per prompt
353.6 s
p10 230.2 s, p90 651.4 s
Strict pass rate
71.1%
32 pass, 7 partial, 6 fail
Executions recorded
45
5 runs of 9 prompts
Entry price
EUR 18.00
Cheapest plan that can finish the specification
Per axis subscores
Composite recomputed from these ten rows: 66.36. That is the arithmetic behind the index, and you can repeat it by hand.
| Axis | Subscore 0 to 100, tick is the roster median | Score | Median | Weight % | Contribution |
|---|---|---|---|---|---|
| Agent performance | 78.0 | 75.0 | 18 | 14.04 | |
| Reliability | 71.1 | 71.1 | 18 | 12.80 | |
| Scalability | 80.0 | 70.0 | 13 | 10.40 | |
| SEO and GEO | 54.0 | 58.0 | 12 | 6.48 | |
| API and MCP | 72.0 | 62.0 | 9 | 6.48 | |
| Integrations | 74.0 | 66.0 | 8 | 5.92 | |
| Design output | 60.0 | 71.0 | 7 | 4.20 | |
| Speed | 8.9 | 33.0 | 7 | 0.62 | |
| Value | 52.0 | 76.0 | 5 | 2.60 | |
| Code ownership | 94.0 | 90.0 | 3 | 2.82 |
Speed distribution
Every recorded prompt duration for Devin, in seconds. The cyan tick on the whisker is the median.
Spread against the cohort
p10 230.2 s / median 353.6 s / p90 651.4 s
Scale is shared with every product in the cohort, so the bars are directly comparable between pages. Ratio of p90 to median is 1.84x.
Outcome split
32 pass / 7 partial / 6 fail
SEO and GEO measurements
Taken from the public marketing page this product generated in prompt 8, fetched with JavaScript disabled.
HTML bytes, no JS
20.0 KB
20,480 bytes of real text content with scripts blocked
LCP
1,900 ms
Throttled connection, cold cache
CLS
0.050
Cumulative layout shift on load
SEO and GEO subscore
54.0
0 to 100, worth 12 index points
Raw run data: 45 executions
One row per prompt execution. Nothing is aggregated away, nothing is hidden, including the failures.
| Run | Prompt | Prompt target | Started | Duration (s) | Outcome | Note |
|---|---|---|---|---|---|---|
| R01 | P01 | Tenant and user model | 11 Aug 2026 08:00 UTC | 119.8 | fail | Build did not start after deploy. Recorded as a fail with the log preserved. |
| R01 | P02 | Authentication and session | 11 Aug 2026 08:03 UTC | 294.3 | pass | |
| R01 | P03 | Projects CRUD with tenancy | 11 Aug 2026 08:10 UTC | 661.9 | partial | Required one clarifying instruction before the criterion was met. |
| R01 | P04 | Public REST API with keys | 11 Aug 2026 08:23 UTC | 931.3 | pass | |
| R01 | P05 | Outbound webhooks with signatures | 11 Aug 2026 08:41 UTC | 352.7 | pass | |
| R01 | P06 | Background jobs | 11 Aug 2026 08:49 UTC | 507.8 | pass | |
| R01 | P07 | Admin panel | 11 Aug 2026 08:59 UTC | 547.5 | pass | Opened a pull request with passing tests and no human in the loop for the whole task. |
| R01 | P08 | Public marketing surface | 11 Aug 2026 09:10 UTC | 349.6 | pass | |
| R01 | P09 | Hardening pass | 11 Aug 2026 09:18 UTC | 496.3 | fail | Build did not start after deploy. Recorded as a fail with the log preserved. |
| R02 | P01 | Tenant and user model | 12 Aug 2026 08:00 UTC | 304.3 | pass | |
| R02 | P02 | Authentication and session | 12 Aug 2026 08:07 UTC | 226.4 | pass | |
| R02 | P03 | Projects CRUD with tenancy | 12 Aug 2026 08:12 UTC | 390.6 | fail | Generated migration conflicted with the existing schema and was not reconciled. |
| R02 | P04 | Public REST API with keys | 12 Aug 2026 08:21 UTC | 462.7 | pass | |
| R02 | P05 | Outbound webhooks with signatures | 12 Aug 2026 08:31 UTC | 110.9 | pass | |
| R02 | P06 | Background jobs | 12 Aug 2026 08:34 UTC | 241.3 | pass | |
| R02 | P07 | Admin panel | 12 Aug 2026 08:40 UTC | 353.6 | pass | Opened a pull request with passing tests and no human in the loop for the whole task. |
| R02 | P08 | Public marketing surface | 12 Aug 2026 08:48 UTC | 655.7 | pass | |
| R02 | P09 | Hardening pass | 12 Aug 2026 09:01 UTC | 356.0 | pass | |
| R03 | P01 | Tenant and user model | 13 Aug 2026 08:00 UTC | 479.9 | pass | |
| R03 | P02 | Authentication and session | 13 Aug 2026 08:09 UTC | 241.3 | pass | |
| R03 | P03 | Projects CRUD with tenancy | 13 Aug 2026 08:16 UTC | 364.9 | pass | |
| R03 | P04 | Public REST API with keys | 13 Aug 2026 08:24 UTC | 1293.4 | partial | Error envelope inconsistent between two of the endpoints. |
| R03 | P05 | Outbound webhooks with signatures | 13 Aug 2026 08:47 UTC | 278.3 | pass | |
| R03 | P06 | Background jobs | 13 Aug 2026 08:54 UTC | 644.9 | pass | |
| R03 | P07 | Admin panel | 13 Aug 2026 09:07 UTC | 350.5 | pass | Opened a pull request with passing tests and no human in the loop for the whole task. |
| R03 | P08 | Public marketing surface | 13 Aug 2026 09:14 UTC | 372.8 | partial | Required one clarifying instruction before the criterion was met. |
| R03 | P09 | Hardening pass | 13 Aug 2026 09:23 UTC | 301.8 | pass | |
| R04 | P01 | Tenant and user model | 14 Aug 2026 08:00 UTC | 151.9 | partial | Validation added on the write path only. Read path left unguarded. |
| R04 | P02 | Authentication and session | 14 Aug 2026 08:04 UTC | 338.1 | pass | |
| R04 | P03 | Projects CRUD with tenancy | 14 Aug 2026 08:12 UTC | 235.8 | fail | Generated migration conflicted with the existing schema and was not reconciled. |
| R04 | P04 | Public REST API with keys | 14 Aug 2026 08:18 UTC | 238.0 | pass | |
| R04 | P05 | Outbound webhooks with signatures | 14 Aug 2026 08:24 UTC | 377.0 | pass | |
| R04 | P06 | Background jobs | 14 Aug 2026 08:32 UTC | 606.5 | pass | |
| R04 | P07 | Admin panel | 14 Aug 2026 08:44 UTC | 514.1 | partial | Required one clarifying instruction before the criterion was met. |
| R04 | P08 | Public marketing surface | 14 Aug 2026 08:55 UTC | 289.3 | pass | |
| R04 | P09 | Hardening pass | 14 Aug 2026 09:01 UTC | 480.2 | pass | |
| R05 | P01 | Tenant and user model | 15 Aug 2026 08:00 UTC | 359.7 | pass | |
| R05 | P02 | Authentication and session | 15 Aug 2026 08:07 UTC | 263.8 | pass | |
| R05 | P03 | Projects CRUD with tenancy | 15 Aug 2026 08:14 UTC | 246.2 | partial | Delivered the main path, missed the boundary case named in the pass criteria. |
| R05 | P04 | Public REST API with keys | 15 Aug 2026 08:20 UTC | 839.6 | fail | Task abandoned by the agent with no artifact produced. |
| R05 | P05 | Outbound webhooks with signatures | 15 Aug 2026 08:36 UTC | 437.7 | fail | Generated migration conflicted with the existing schema and was not reconciled. |
| R05 | P06 | Background jobs | 15 Aug 2026 08:45 UTC | 409.9 | partial | Required one clarifying instruction before the criterion was met. |
| R05 | P07 | Admin panel | 15 Aug 2026 08:54 UTC | 321.4 | pass | Opened a pull request with passing tests and no human in the loop for the whole task. |
| R05 | P08 | Public marketing surface | 15 Aug 2026 09:01 UTC | 208.3 | pass | |
| R05 | P09 | Hardening pass | 15 Aug 2026 09:07 UTC | 250.9 | pass |
Pricing
From EUR 18 per month for a starter allowance, then metered per compute unit. A full run of this spec consumes a large share of a starter allowance.
- Entry price
- EUR 18.00
- Value subscore
- 52.0
- Index points per euro
- 3.69
Capabilities as measured
- stack
- Its own workspace, then a pull request to your repository
- deploy
- It can run the deploy, environment included
- database
- Whatever you already run
- auth
- It wires the library you choose
- background jobs
- Implemented in your stack
- webhooks
- Signed with a delivery log
- rest api
- Written and documented
- mcp server
- Yes
- export code
- Arrives as a pull request
- custom domain
- Your hosting decides
- ssr
- Your framework decides
Screenshot
Devin landing page, captured by the lab on 14 Aug 2026 at 1440 by 900. Stored locally, never hotlinked.
Devin, captured 14 Aug 2026.
Sources and fetch dates
Every claim on this page traces to one of these.
| Source | URL | Fetched |
|---|---|---|
| Devin pricing page | https://devin.ai/pricing | 14 Aug 2026 |
| Devin product site | https://devin.ai | 14 Aug 2026 |
| LLM Tier run harness output | https://www.llm-tier.com/api/runs.json | 14 Aug 2026 |
Rich results carry this composite as 3.3 on a 0 to 5 scale because that is the only scale the format accepts. The number on this page, 0 to 100, is the real one.
Devin head to head
Reader index
Voted by readers with an account, one score per person per axis. Published beside the lab index, never inside it.
Reader index
not rated yet
Lab index
66.36
2 readers have scored 100% of the index weight so far. A reader index is published at 3 voters and 50% coverage.
| Axis | Weight | Lab | Readers | Votes | Reader mean |
|---|---|---|---|---|---|
| Agent performance | 18% | 78.0 | 63.5 | 2 | |
| Reliability | 18% | 71.1 | 62.0 | 2 | |
| Scalability | 13% | 80.0 | 65.5 | 2 | |
| SEO and GEO | 12% | 54.0 | 44.0 | 2 | |
| API and MCP | 9% | 72.0 | 60.5 | 2 | |
| Integrations | 8% | 74.0 | 65.5 | 2 | |
| Design output | 7% | 60.0 | 54.0 | 1 | |
| Speed | 7% | 8.9 | 2.0 | 2 | |
| Value | 5% | 52.0 | 26.0 | 1 | |
| Code ownership | 3% | 94.0 | 75.5 | 2 |
The reader index is a weighted mean of reader submitted axis scores using the published index weights, renormalised over the axes readers have scored. It is never blended into the lab index. How this works.
Open rating mean
n/a
0 published open ratings, submitted without an account
Lab index
66.36
Measured, not voted
Gap to the measurement
n/a
Positive means open raters put it above the measurement
What agent performance looked like in practice
The slowest median in the index by a wide margin and the most expensive route through the spec, because it works in its own environment and takes the long way through every task. What it buys is unattended completion: it opened a pull request with passing tests without a human in the loop in four of five runs. Judged on the published axes the cost weighs heavily against it, which is why the value subscore is the lowest in the agent group.
Across 45 recorded executions this product met the published pass criterion cleanly 32 times, partially 7 times and failed 6 times. The rows above carry the reviewer note for every non clean outcome, which is the only way to tell a product that fails loudly from one that fails quietly.