Reader contributions
Contribute scores and lab notes
Scale
0 to 100
Same scale as the lab subscores.
Votes per axis
1
Per account, per product, editable.
Voters before publication
3
Weight coverage needed
50%
Of the ten weighted axes.
Reader index and lab index are separate numbers
This is the rule the rest of the design follows from.
The reader index is a weighted mean of reader submitted axis scores using the same published weights as the lab index, renormalised over the axes readers have actually scored. It is published only once a product has at least three distinct voters covering at least half of the index weight. It is never blended into the lab index.
The two columns are most useful where they disagree. A product readers love and the runs do not is usually pleasant to use and fragile under load. A product the runs like and readers do not is usually solid and unfriendly. Neither observation survives if you average them into one number, which is exactly why we do not.
How the lab number is produced is documented separately in the methodology, and every run behind it is downloadable from the data page.
The ten axes you score
Reader scores use the same axes and the same weights as the lab index, so the two columns are comparable.
| Axis | Weight | What a reader should be judging |
|---|---|---|
| Agent performance | 18% | How often the agent finished the job you asked for without you rewriting the prompt. |
| Reliability | 18% | Whether repeated builds of the same thing produced the same working result. |
| Scalability | 13% | How the generated app behaved once real data and real traffic arrived. |
| SEO and GEO | 12% | Whether pages render server side and are readable to crawlers and answer engines. |
| API and MCP | 9% | Quality of the REST surface, webhooks, background jobs and MCP support. |
| Integrations | 8% | How painless it was to attach the databases, payments and services you needed. |
| Design output | 7% | Quality of the interface it produced before you touched it. |
| Speed | 7% | Wall clock time from prompt to something you could open. |
| Value | 5% | What you got for what you paid, including the cost of the failed attempts. |
| Code ownership | 3% | How completely you can export the code and run it without the vendor. |
Leave an axis blank if you have not used that part of the product. A blank is not a zero: the reader index is renormalised over the axes readers actually scored, so an unused axis never quietly drags a product down.
Published rate limits
Enforced per account and per network.
| Action | Per account | Per network address |
|---|---|---|
| Create an account | one account per email address | 5 per day |
| Sign in attempt | 10 per 15 minutes per email | 20 per 15 minutes |
| Password reset email | 3 per hour per email | 6 per hour |
| Submit or edit an axis score | 60 per hour | 120 per hour |
| Write a lab note | 5 per hour | 10 per hour |
| Post a comment | 20 per hour | 40 per hour |
| Upvote | 100 per hour | 200 per hour |
- One account per email address. A second registration on the same address is refused rather than silently merged.
- Every form carries a hidden field a person never sees and a server issued timestamp. A submission that fills the hidden field, or arrives faster than a human could type it, is rejected.
- A lab note needs at least 240 characters and a comment at least 40, because a one line verdict carries no information.
- Passwords are at least 10 characters and are stored only as a hash.
- First contributions from a new account are held in a moderation queue. Approve once and the account publishes directly from then on.
- Links inside reader contributions carry rel="ugc nofollow", so a contribution can never be used to pass ranking signal to another site.
What we store
Your email address, so we can confirm the account and reset the password. It is never published and never sold. Your display name, handle, bio and contributions are public by design.
For abuse limits we store a salted hash of the network address a request came from, never the address itself, and the hash cannot be reversed back to an address. Full detail is in the privacy notice. To delete an account and its contributions write to lab@llm-tier.com.
Recent lab notes
Everything attached, and I paid for it in wall clock
Firebase Studio/14 Aug 2026Auth, storage, scheduled functions and the datastore all wired up without me leaving the workspace, and none of them needed a second attempt. That is the strongest integration story in the index and my score says so. The cost is that every prompt round trips through a provisioned cloud workspace, so the median is materially worse than the local agents and the p90 is worse again on a bad afternoon. Also worth knowing before you start: the public pages came back client rendered until I asked for server rendering explicitly, which is a bad surprise to have after launch.
Writing the spec first is slower and I stopped minding
Kiro/13 Aug 2026We ran a bake off across four agents on a real feature, not on a benchmark. This one was consistently the slowest to first code and the fastest to a merged change, because the plan it wrote up front was reviewable before anything depended on it. Twice a reviewer caught a wrong assumption in the plan in about a minute, which would have been an hour of review on a finished diff. The published median puts it mid table on speed and I think that undersells it: the axis measures time per prompt and the thing that matters to us is time per merge.
Cheapest completed run I have recorded, by a lot
Aider/13 Aug 2026I tracked token spend across the nine prompts for four different agents. This one came in at roughly a third of the nearest commercial option and finished the spec, which is the part people leave out when they say the cheap option does not work. The discipline is what makes it cheap: it only reads the files you add, so you are not paying to send a repository through the model on every turn. The cost of that discipline is real too. Anything that needs a change across many files at once is slower and needs more of my attention, and prompt seven took me three attempts to steer.
It finished unattended, and the bill made the decision for us
Devin/12 Aug 2026The thing it does that nothing else here does is finish while nobody is watching. We gave it a bounded task on a Friday and had a pull request with passing tests on Monday, no intervention. That is real and it is worth money. The problem is how much money. One full pass of a spec this size consumed most of a starter allowance, and once we projected the team using it the way we use the interactive agents, the monthly number was several times what we pay for everything else combined. I am scoring value at the floor and agent performance well above the middle, which I think is the honest shape of it.
The only one where the crawler test was boring
Totalum/11 Aug 2026I check the same thing on every product: load the generated public page with JavaScript disabled and see what text is there. Most of the index gives you a shell and a spinner. This one returned the whole page, the metadata was coherent without being asked, and the structured data it emitted validated on the first try. That is why I have it near the top on SEO and GEO despite scoring it low on design, where the output is plain. If your product lives or dies on organic traffic, this axis is not a nice to have and the difference between the top and the middle of this index is not small.
The backend did the operational prompts for it
Convex Chef/10 Aug 2026Interesting one to score, because a lot of what it got right is not the agent being clever, it is the platform making the wrong answer hard to write. Scheduled work is durable because durable is the default. Reads are consistent because there is no other kind. That is a legitimate way to score well and I gave it full marks on API and MCP for it. Where it struggled was prompt two: expressing the tenancy rule the way the platform wants took me two rewrites, and if you come in expecting a relational schema you will fight it for an afternoon before it clicks.
Fastest loop I have measured, and I timed it properly
Zed/09 Aug 2026I sat with a stopwatch across nine prompts because I did not believe the published median. It held up: the wait between asking and having a reviewable diff was consistently under a minute for the small prompts. What the number does not tell you is why. It is not that the model is faster, it is that nothing else in the loop is slow. The diff renders instantly, the editor does not stall while the agent works, and accepting a change does not trigger a reindex you have to wait through. Correctness is mid table, so my honest read is that you get more attempts per hour rather than better attempts.
Fine for a booking form, wrong for anything with an operator
Rocket.new/08 Aug 2026I was brought in to finish a project built on it, which is the least flattering possible way to meet a product. Judged on what it is for, a small application with a few forms and a light data model, it did the job and the client was happy. Judged on the spec this site runs it falls over exactly where the lab says it does. The outbound webhooks signed nothing at all, there was no record of what had been sent, and the retry behaviour was to not retry. I could not honestly score API and MCP above twenty, and the low reliability figure published here matches what I found.
Questions readers ask
Start contributing
Create an account, confirm the address, then score any of the 41 products we cover.