Lab notes
News, methodology and analysis
| Date | Category | Title | Summary | Read |
|---|---|---|---|---|
| 09 Sep 2026 | news | 957 reader rows carried a date from before this domain existed | In August we corrected eight article dates and published the check that found them. The check was scoped to the articles table. Every reader score, open rating, lab note, comment and vote on this site was also dated before the domain resolved. All 957 are corrected, and the gate is restated wider. | 5 min |
| 23 Aug 2026 | news | Eight articles carried a publication date from before this domain was registered | llm-tier.com was registered on August 19, 2026 at 10:10 UTC. Eight of our fourteen articles were stamped earlier than that, in the byline and in the structured data. Corrected today, with the check that finds it now running on a schedule. | 2 min |
| 22 Aug 2026 | news | Entry prices are now published in the vendor's own currency, not a euro conversion that tracked no single rate | Every entry price on both leaderboards carried a bare euro figure with no stated source or rate. An audit found the implied euro per dollar rate across cleanly priced USD rows spread from 0.19 to 1.80, wrong in both directions. Prices are now published in the vendor's own currency, unit and billing period, sourced. The Value axis and the composite index were recomputed board wide as a result, and Totalum's own Value subscore fell from 84 to 69. | 4 min |
| 22 Aug 2026 | news | The model board carried a composite index with no run behind it. It is gone. | The eight rows on /leaderboard/llms carried an Index column that looked exactly like the composite this lab actually measures on the other two boards. It was not: no per-model run record, subscore or recomputed field sat behind any of the eight numbers, and Arena's own leaderboard shows all eight rows were two to three generations stale. The index is removed, not refreshed. What replaces it is a 26-row sourced specification-and-benchmark table, with the twelve superseded rows kept in a separate table rather than deleted. | 3 min |
| 22 Aug 2026 | news | Tesslate leaves the index: the vendor has pivoted to a different category | tesslate.com no longer operates an app builder. The domain now hosts OpenSail, a business operations platform, and the EUR 16 monthly price we published cannot be sourced from the vendor's current site. Tesslate is out of the roster, the two leaderboards it appeared on, the matrix and every data export. | 2 min |
| 21 Aug 2026 | news | Convex Chef is being replaced by its vendor, so it leaves the index | Convex is replacing Chef with a new product and has taken down its entry points. Existing projects still run, so this is not a shutdown, but a builder you cannot sign up for cannot hold a live ranking. Chef is out of the roster, the raw runs, the matrix, both leaderboards it appeared on and both of its compare pages. | 3 min |
| 21 Aug 2026 | news | Pazi leaves the index: wrong category, and a price we could not source | We ranked Pazi 37 of 41 as a coding agent and scored it on 45 executions of nine software engineering prompts. Pazi does not sell a coding product. We also published a 14 EUR floor against a real 25 dollar one, and cited a source URL that never contained the figure. | 3 min |
| 21 Aug 2026 | news | Mocha leaves the index, and we benchmarked it after it had already announced its own end. | Mocha is gone from this roster, and the reason is as much about our own record as the vendor's calendar. We published a euro price and a value score for a plan its own pricing page says can no longer be bought, and every one of the 45 runs behind its row was collected after the vendor's stated shutdown date had already passed. | 5 min |
| 20 Aug 2026 | methodology | This cycle's composite recompute, and why it barely moved most rows | We re derived every composite index from the ten published axis weights and the stored subscores, row by row, and checked the arithmetic against the methodology page. Here is what changed and what that process actually verifies. | 2 min |
| 20 Aug 2026 | news | Windsurf is gone from the index | windsurf.com now redirects to devin.ai/desktop. The product is discontinued, so it has been removed from the roster, the raw runs, every leaderboard, every compare pair and the matrix, not just flagged as inactive. | 2 min |
| 20 Aug 2026 | news | August 2026 index update: three products move, one regression | The fourth full run of the vibeOps specification against 11 products. Two products improved their reliability materially, one regressed on tenancy isolation. | 1 min |
| 20 Aug 2026 | methodology | Why we publish the spread and not the mean | A single average build time hides the thing you actually care about: how bad the slow runs get. Here is the p10 to p90 data for every product in the index. | 2 min |
| 19 Aug 2026 | analysis | Server rendering is the whole GEO story right now | We disabled JavaScript on every generated marketing page and counted the bytes of real text. The gap between the top and the bottom of the cohort is nearly seven to one. | 2 min |
| 19 Aug 2026 | methodology | How we score agent performance without reading the chat log | Agent performance is 18 points of the index, the joint largest weight. Here is exactly how a prompt earns 0, 50 or 100. | 1 min |
| 19 Aug 2026 | methodology | Reproducing any row in the index by hand | The composite is a weighted sum, nothing more. Take the ten subscores off any product page, multiply by the published weights and you get the number in the table. | 1 min |
| 19 Aug 2026 | analysis | What this benchmark cannot tell you | A composite index is a compression of a complicated thing into one number. Here are the limits of ours, stated plainly. | 1 min |
methodology
4 published
20 Aug 2026
This cycle's composite recompute, and why it barely moved most rows
We re derived every composite index from the ten published axis weights and the stored subscores, row by row, and checked the arithmetic against the methodology page. Here is what changed and what that process actually verifies.
20 Aug 2026
Why we publish the spread and not the mean
A single average build time hides the thing you actually care about: how bad the slow runs get. Here is the p10 to p90 data for every product in the index.
19 Aug 2026
How we score agent performance without reading the chat log
Agent performance is 18 points of the index, the joint largest weight. Here is exactly how a prompt earns 0, 50 or 100.
19 Aug 2026
Reproducing any row in the index by hand
The composite is a weighted sum, nothing more. Take the ten subscores off any product page, multiply by the published weights and you get the number in the table.
news
10 published
09 Sep 2026
957 reader rows carried a date from before this domain existed
In August we corrected eight article dates and published the check that found them. The check was scoped to the articles table. Every reader score, open rating, lab note, comment and vote on this site was also dated before the domain resolved. All 957 are corrected, and the gate is restated wider.
23 Aug 2026
Eight articles carried a publication date from before this domain was registered
llm-tier.com was registered on August 19, 2026 at 10:10 UTC. Eight of our fourteen articles were stamped earlier than that, in the byline and in the structured data. Corrected today, with the check that finds it now running on a schedule.
22 Aug 2026
Entry prices are now published in the vendor's own currency, not a euro conversion that tracked no single rate
Every entry price on both leaderboards carried a bare euro figure with no stated source or rate. An audit found the implied euro per dollar rate across cleanly priced USD rows spread from 0.19 to 1.80, wrong in both directions. Prices are now published in the vendor's own currency, unit and billing period, sourced. The Value axis and the composite index were recomputed board wide as a result, and Totalum's own Value subscore fell from 84 to 69.
22 Aug 2026
The model board carried a composite index with no run behind it. It is gone.
The eight rows on /leaderboard/llms carried an Index column that looked exactly like the composite this lab actually measures on the other two boards. It was not: no per-model run record, subscore or recomputed field sat behind any of the eight numbers, and Arena's own leaderboard shows all eight rows were two to three generations stale. The index is removed, not refreshed. What replaces it is a 26-row sourced specification-and-benchmark table, with the twelve superseded rows kept in a separate table rather than deleted.
22 Aug 2026
Tesslate leaves the index: the vendor has pivoted to a different category
tesslate.com no longer operates an app builder. The domain now hosts OpenSail, a business operations platform, and the EUR 16 monthly price we published cannot be sourced from the vendor's current site. Tesslate is out of the roster, the two leaderboards it appeared on, the matrix and every data export.
21 Aug 2026
Convex Chef is being replaced by its vendor, so it leaves the index
Convex is replacing Chef with a new product and has taken down its entry points. Existing projects still run, so this is not a shutdown, but a builder you cannot sign up for cannot hold a live ranking. Chef is out of the roster, the raw runs, the matrix, both leaderboards it appeared on and both of its compare pages.
21 Aug 2026
Pazi leaves the index: wrong category, and a price we could not source
We ranked Pazi 37 of 41 as a coding agent and scored it on 45 executions of nine software engineering prompts. Pazi does not sell a coding product. We also published a 14 EUR floor against a real 25 dollar one, and cited a source URL that never contained the figure.
21 Aug 2026
Mocha leaves the index, and we benchmarked it after it had already announced its own end.
Mocha is gone from this roster, and the reason is as much about our own record as the vendor's calendar. We published a euro price and a value score for a plan its own pricing page says can no longer be bought, and every one of the 45 runs behind its row was collected after the vendor's stated shutdown date had already passed.
20 Aug 2026
Windsurf is gone from the index
windsurf.com now redirects to devin.ai/desktop. The product is discontinued, so it has been removed from the roster, the raw runs, every leaderboard, every compare pair and the matrix, not just flagged as inactive.
20 Aug 2026
August 2026 index update: three products move, one regression
The fourth full run of the vibeOps specification against 11 products. Two products improved their reliability materially, one regressed on tenancy isolation.
analysis
2 published
19 Aug 2026
Server rendering is the whole GEO story right now
We disabled JavaScript on every generated marketing page and counted the bytes of real text. The gap between the top and the bottom of the cohort is nearly seven to one.
19 Aug 2026
What this benchmark cannot tell you
A composite index is a compression of a complicated thing into one number. Here are the limits of ours, stated plainly.