Methodology

Value is index points per euro. The published entry price never is: it stays exactly what the vendor states, in the vendor's own currency, with its own unit and billing period, sourced. So before it can rank products that bill in dollars against products that bill in euros, each entry price is converted once to a euro figure that is used only for this axis and only for the "index points per euro" derivative next to it. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. That euro figure is never the number printed as the entry price anywhere on this site, never appears in a JSON-LD Offer, and never appears in an export without this rate and this date printed beside it.

The Value subscore itself is a rank preserving normalisation, not the ratio read off directly. Within a board's own cohort, products are ordered by index points per euro, most efficient first, and that order is mapped onto the cohort's subscore distribution. A product with no licence fee at all sits at the top of that order on price alone, because the ratio is unbounded as price approaches zero; where two or more products share a zero price, they are then ordered against each other by index score, highest first, since price can no longer separate them.

Where a vendor publishes no entry price at all, rather than a genuine zero price, the Value axis is not scored for that product. It is not scored as zero and it is not guessed: the composite index for that product is renormalised over the remaining nine weights instead, so a missing price cannot silently drag the whole index down the way a zero would. YouWare is the current instance of this: its vendor pricing page returns no published figure, so its Value axis reads "not scored" rather than a number, and its composite is computed from the other nine axes with their weights rescaled to still sum to 100.

Median wall clock seconds for one prompt, not for a whole build. The test specification is nine prompts, run five times per product, so a single row here is the middle value of 45 recorded prompt executions.

Relabelling did not change a single stored number: every speed_seconds_p10/median/p90 and every score_speed value on this site is exactly what it was before the rename. Only the words above the numbers changed, because the numbers were never the part that was wrong.

Wherever a turn latency figure is printed, we also print a "full specification, derived" minutes figure: nine times the per prompt median, converted to minutes. Derived, not separately measured. It is nine times the median prompt latency, converted to minutes. It is not a measured end-to-end wall clock for one continuous session, and it excludes any operator time between prompts.

Worked example: Zed's median is 31.4 seconds per prompt, and 31.4 times nine is 282.6 seconds, which is 4.7 min. That figure was never itself timed end to end: it is the per-prompt median multiplied by nine, not nine separate prompt medians added together.

This derivation understates the real end-to-end wall clock. Taking each run's own nine-prompt total and then the median across that product's five runs gives a board median of 16.5 minutes for the full-stack app builders and 12.9 minutes for the coding agents, against the 14.7 and 11.8 minutes the derived column medians to. Multiplying a median by nine is not the same as the median of nine summed durations, and the difference goes in one direction: the derived figure is the optimistic one. Both are recomputable from runs.csv.

This figure covers the entire nine-prompt vibeOps specification, not a single build task. A reader asking how long one prompt takes should read the median column instead.

p10, median and p90 are linear interpolated over all 45 recorded prompt executions per product (nine prompts, five runs), regardless of outcome. A partial or an outright fail is not excluded, not conditioned out and not truncated at some cap: its duration counts exactly like a clean pass does. This is deliberate. A percentile computed only over successes would understate how long the product actually takes, because it would quietly delete the slowest attempts from the sample.

You can check any row by hand: sort a product's 45 published run durations from the raw export at /api/runs.json and the 23rd value is the median we print. The 5th and 41st values, interpolated, are the p10 and p90 we print. There is no other step and no adjustment between the sort and the published figure.

Across the cycle-04 dataset, 1,232 prompt executions were recorded as a clean pass, with a mean duration of 113.3 seconds and a longest execution of 931.3 seconds. 261 executions were recorded as a partial, mean 121.2 seconds, longest 1,293.4 seconds. 217 executions were recorded as an outright fail, mean 119.3 seconds, longest 839.6 seconds.

Read plainly: a partial or a fail in this dataset runs about 5 percent longer than a pass on average, not several times longer. The tail is where the classes separate, not the centre, which is exactly why the percentiles above are computed over every outcome rather than only over passes.

Devin's prompt 4 execution in cycle 04 ran 1,293.4 seconds, 21.6 minutes, and is recorded as a partial outcome, not a fail. An earlier version of this page stated that executions running past 20 minutes are recorded as a fail. Devin's row contradicts that rule as written: it ran past 20 minutes and was not recorded as a fail.

We left the row exactly as recorded rather than change a data point to fit a rule written after the fact. Instead we corrected the rule: executions running past 20 minutes are flagged for manual review and scored on the outcome the reviewer actually recorded, and the full duration is always retained and always counts toward the turn latency percentiles above, regardless of which outcome the review lands on. The reliability axis method text earlier on this page reflects that correction.

Anthropic published a research post on 18 Feb 2026 describing its own telemetry for coding-agent workloads, with tail latencies around 60 times the median. The published spread on this site's turn latency axis, p90 divided by median across the roster, ranges from 2.03x to 4.40x.

Those two figures are not measuring the same thing and we do not want a reader to assume they are. Anthropic's number describes one production deployment's tail across all of its real traffic, at whatever volume that traffic runs. Ours describes five runs of one fixed nine-prompt specification per product, which is a far smaller and far more uniform sample than a production tail can ever be. Five runs cannot surface a 60x event even if the underlying system is capable of one: a benchmark of this size will structurally under report the true tail of any product it measures, and this site's spread should be read as a lower bound on variability, not as the full tail.

The reader index, the reader lab notes and the open ratings were seeded by this lab in August 2026, before the site had readers. Every reader row currently published came from that batch and is labelled as such wherever it appears. Reader submissions made since then come from registered accounts and carry no label. None of it enters the lab index, which is computed only from logged runs on the vibeOps spec.

The weight tables, the nine prompts and the axis measurements on this page describe the vibeOps composites for full-stack app builders and coding agents only. The site also publishes a table of large language models at /leaderboard/llms, and that table is built on an entirely different basis. We do not run a language-model harness. We previously published a per-model composite index there, entered directly as editorial judgement with no logged runs behind it, and we have removed it: this lab holds no model-level run records, so it could not show the arithmetic behind that number, and a site whose whole premise is reproducible measurement should not publish a figure it cannot show the work for. Nothing on this site will publish a model composite index again unless and until we actually run models through a harness and store the executions. The removal itself, with the Arena rank evidence that showed the old rows were stale as well as unsourced, is written up in this lab note.

What replaces it is a sourced specification and benchmark table. Every cell is either a vendor-published fact (context window, output limit, price, licence), an attributed third party benchmark result (Arena WebDev Elo, the Artificial Analysis Intelligence Index, or Terminal-Bench v2.1), or an explicit blank where no source states a number. None of those cells is an estimate, an average of two sources, or a figure carried over from a price aggregator. Full attribution, the benchmark provenance box and a documented pricing error we found on Arena itself are all published on the table page.

Coverage boundary, stated rather than left implicit: Arena's WebDev board carries strong models from vendors we do not list, because we could not source a vendor-published price for them from that vendor's own page. That includes Z.ai (GLM 5.3-max at 1599 Elo, MIT licence, ahead of eight proprietary flagships we do list), Bytedance, Tencent, Xiaomi, MiniMax, Thinky, Poolside and Upstage, plus Alibaba's own open-weight qwen3.8-27b at 1595 Elo. We would rather publish 26 fully sourced rows than 40 rows with an estimated price on some of them. These are the first candidates for the next update.

Several of the axes above could not apply to a raw model even if we tried to measure them the same way: code ownership, integrations, and SEO and GEO all describe a deployed application or an edited repository, not a model responding to a prompt. That is a second, independent reason the model table is not comparable to either board, on top of the first: the model table carries no index or rank of any kind, ours or anyone else's conflated with ours.

Reproducing a composite

Take the subscores from any product page, multiply each by its board's weight from the tables above, divide the sum by one hundred and round to two decimals. That is the index. A coding agent has nine subscores to multiply, a full-stack app builder has ten. There is no other step, and no step where a figure from one board enters the other board's sum. A worked example is published in the lab notes.

Reproducing a timing

Start a monotonic clock when the prompt is submitted. For a full-stack app builder, stop it on the first 200 response from the deployed URL. For a coding agent, stop it on the agent's returned diff or completion signal for that prompt, since most coding agents never produce a hosted URL. Do not read the timer the product displays, because those timers differ in what they include. Queue time counts, cold start counts where it applies, and if the product reports completion while the deployed URL is not serving, the clock keeps running. Repeat the full nine prompt sequence five times from an empty project, then compute p10, median and p90 by linear interpolation over all 45 executions, regardless of outcome. Multiply the median by nine and divide by sixty for the derived full specification minutes figure; do not present that figure as a separately measured one.

Reproducing a pass or fail

Each prompt has one published pass criterion. Check it against the artifact and the diff, not the chat transcript. Score 100 for a clean pass, 50 where part of the criterion is met or one clarifying instruction was needed, and 0 otherwise. Two reviewers score independently, blind to which product produced the artifact, and a full band disagreement goes to a third. An execution running past 20 minutes goes to manual review rather than an automatic fail, and the outcome the reviewer records is the one that stands.

Reproducing the SEO and GEO figures

Fetch the generated public marketing page with scripts disabled, strip tags, count the bytes of remaining text and record that as the no JavaScript HTML weight. Measure LCP and CLS on a throttled connection with a cold cache. Check for a title, meta description, canonical link, robots policy, sitemap and any structured data present in the first response.

The data behind it

Every figure on this site is downloadable. Rankings for both boards and the sourced LLM reference table are at /api/rankings.json and /data/rankings.csv, and every individual execution for a currently published product is at /api/runs.json. All three carry an explicit CC-BY-4.0 licence field, so you may republish them with attribution to LLM Tier.

Conflicts and funding

No product can pay to be measured, re measured, promoted or removed, on either board. There are no sponsored rows, no paid placements and no affiliate links in any index table. The lab is funded by data licensing and consulting on measurement design, never by the products it measures. Every row is produced by the same harness on its board's own specification, so no product receives different treatment from any other product on its board.

Corrections

Send evidence through the contact form. If a measurement is wrong we correct it, mark the correction on the product page and keep the original value in the record. Where the rule was wrong rather than the data, as with the 20 minute reliability rule above, we correct the rule and leave the recorded row standing. Three corrections have been issued since the index launched, most recently the rule text above.