# LLM Tier: complete published dataset Source: https://www.llm-tier.com Licence: CC-BY-4.0 (https://creativecommons.org/licenses/by/4.0/) Attribution: Data by LLM Tier, https://www.llm-tier.com, licensed CC-BY-4.0 Generated: 2026-10-16T18:37:27.912Z Contact: https://www.llm-tier.com/contact ## What this is LLM Tier measures AI application builders, coding agents and language models, on two separate boards that are never ranked against each other. A full-stack app builder is given a written specification and returns a hosted, running application; a coding agent is pointed at a repository you already own and returns a diff. Every product is driven through one published specification, vibeOps, a multi tenant SaaS with an API, webhooks, background jobs and an admin panel. Each product receives the same 9 prompts in the same order, 5 times. Each prompt attempt is timed and marked pass, partial or fail against criteria written before the run. The full-stack app builder composite is the weighted sum of ten subscores; the coding agent composite is the weighted sum of nine subscores, because code ownership is not scored on that board. Both are reported to two decimals, each from 0 to 100. There are no star ratings on this site. Reliability is the strict pass rate: partial outcomes do not count as passes. Median wall clock seconds for one prompt, not for a whole build. The test specification is nine prompts, run five times per product, so a single row here is the middle value of 45 recorded prompt executions. Derived, not separately measured. It is nine times the median prompt latency, converted to minutes. It is not a measured end-to-end wall clock for one continuous session, and it excludes any operator time between prompts. ## Scoring axes and weights: full-stack app builders (10 axes) - Agent performance (agent_perf): weight 18 percent. How much of each prompt the agent actually delivered without the operator intervening, correcting or re prompting. Method: Each of the 9 prompts is scored 0, 50 or 100 against its published pass criteria by two reviewers working from the artifact and the diff, not from the chat transcript. Disagreements are re reviewed. The axis score is the mean across all 45 prompt executions. - Reliability (reliability): weight 18 percent. How often the product completed a prompt cleanly rather than erroring, stalling, looping or producing a build that does not start. Method: Strict pass rate over all 45 recorded prompt executions per product. Partial completions and outright failures both cost. Executions running past 20 minutes are flagged for manual review and scored on the outcome the reviewer recorded; the duration is always retained and always counts toward the turn latency percentiles. - Scalability (scalability): weight 13 percent. Whether the generated system holds up beyond the demo: query shape, N+1 behaviour, pagination, background work and multi tenant isolation. Method: The prompt 3 and prompt 6 artifacts are loaded with 50000 projects across 200 organisations. We record list endpoint latency at p95, the number of queries per request and whether tenant isolation still holds under concurrent writes. - SEO and GEO (seo_geo): weight 12 percent. Whether the public surface the product generates can be crawled by a search engine and cited by a language model. Method: We fetch the generated marketing page with JavaScript disabled and record the HTML byte count of real text, then measure LCP and CLS on a throttled connection. We also check for a title, meta description, canonical, sitemap, robots policy and any structured data emitted without client rendering. - API and MCP (api_mcp): weight 9 percent. The quality of the machine facing surface: REST design, authentication, rate limiting, webhook signing and Model Context Protocol support. Method: Scored against the prompt 4 and prompt 5 pass criteria with a fixed 22 point checklist covering versioning, error envelopes, idempotency keys, key revocation, HMAC signing, retry ladders and delivery history. MCP support is verified by connecting a client and listing tools. - Integrations (integrations): weight 8 percent. First party reach into the services a production SaaS needs: payments, transactional email, object storage, queues, analytics and identity providers. Method: Counted from the product documentation on the fetch date, then verified by wiring two integrations live. Documented but non functional connectors score zero. Generic HTTP request blocks are not counted as integrations. - Design output (design): weight 7 percent. Visual quality of what the product produces unprompted, both the admin surface and the public pages. Method: Blind review of the prompt 7 and prompt 8 artifacts by three reviewers who do not know which product produced which screenshot. Scored on hierarchy, typographic control, spacing discipline, state coverage and responsive behaviour at 390, 768 and 1440 pixels. - Turn latency (speed): weight 7 percent. Median wall clock seconds for ONE prompt, not for a whole build. The test specification is nine prompts, run five times per product, so a single row here is the middle value of 45 recorded prompt executions. Method: Timed by the harness, not by the product interface, from request submission to a 200 response on the deployed URL. We publish p10, median and p90 over all 45 recorded prompt executions per product, regardless of outcome. The subscore is the fastest median in this board's cohort divided by the product's median, expressed as a percentage. - Value (value): weight 5 percent. Index points delivered per euro of entry price. For this class the entry price bundles hosting, database and deploy, since the output is a deployed application. Method: The cheapest plan that can complete the whole vibeOps specification, including any credit top up needed to finish the 5 runs, normalised across this board's cohort. Free tiers that cannot finish the specification are recorded at the price of the first plan that can. - Code ownership (code_ownership): weight 3 percent. Whether you can take the generated system and run it yourself, without the vendor. Method: We export the artifact, remove all vendor specific configuration and attempt a clean install and boot on a plain container. Full marks require complete source export, an open runtime, no proprietary runtime dependency and a documented self host path. ## Scoring axes and weights: coding agents (9 axes, no code ownership) - Agent performance (agent_perf): weight 18.55 percent. How much of each prompt the agent actually delivered without the operator intervening, correcting or re prompting. Method: Each of the 9 prompts is scored 0, 50 or 100 against its published pass criteria by two reviewers working from the artifact and the diff, not from the chat transcript. Disagreements are re reviewed. The axis score is the mean across all 45 prompt executions. - Reliability (reliability): weight 18.56 percent. How often the product completed a prompt cleanly rather than erroring, stalling, looping or producing a build that does not start. Method: Strict pass rate over all 45 recorded prompt executions per product. Partial completions and outright failures both cost. Executions running past 20 minutes are flagged for manual review and scored on the outcome the reviewer recorded; the duration is always retained and always counts toward the turn latency percentiles. - Scalability (scalability): weight 13.4 percent. Whether the generated system holds up beyond the demo: query shape, N+1 behaviour, pagination, background work and multi tenant isolation. Method: The prompt 3 and prompt 6 artifacts are loaded with 50000 projects across 200 organisations. We record list endpoint latency at p95, the number of queries per request and whether tenant isolation still holds under concurrent writes. - SEO and GEO (seo_geo): weight 12.37 percent. Whether the public surface the product generates can be crawled by a search engine and cited by a language model. Method: We fetch the generated marketing page with JavaScript disabled and record the HTML byte count of real text, then measure LCP and CLS on a throttled connection. We also check for a title, meta description, canonical, sitemap, robots policy and any structured data emitted without client rendering. - API and MCP (api_mcp): weight 9.28 percent. The quality of the machine facing surface: REST design, authentication, rate limiting, webhook signing and Model Context Protocol support. Method: Scored against the prompt 4 and prompt 5 pass criteria with a fixed 22 point checklist covering versioning, error envelopes, idempotency keys, key revocation, HMAC signing, retry ladders and delivery history. MCP support is verified by connecting a client and listing tools. - Integrations (integrations): weight 8.25 percent. The underlying model's ability to write correct integration code against third party services when the repository or the prompt asks for it: payments, transactional email, object storage, queues, analytics and identity providers. This is not the same measurement as the builders board, which scores a vendor's first party connector catalogue rather than written code. Method: Scored from the prompt 4 and prompt 5 diffs, checking whether the generated integration code calls the real third party API correctly, handles its documented error responses and does not fabricate an endpoint or credential flow that does not exist. - Design output (design): weight 7.22 percent. Visual quality of what the product produces unprompted, both the admin surface and the public pages. Method: Blind review of the prompt 7 and prompt 8 artifacts by three reviewers who do not know which product produced which screenshot. Scored on hierarchy, typographic control, spacing discipline, state coverage and responsive behaviour at 390, 768 and 1440 pixels. - Turn latency (speed): weight 7.22 percent. Median wall clock seconds for ONE prompt, not for a whole build. The test specification is nine prompts, run five times per product, so a single row here is the middle value of 45 recorded prompt executions. Method: Timed by the harness, not by the product interface, from request submission to the agent's returned diff or completion signal for that prompt. There is no deployment step: aider, cline and claude-code, among others, never produce a hosted URL. We publish p10, median and p90 over all 45 recorded prompt executions per product, regardless of outcome. The subscore is the fastest median in this board's cohort divided by the product's median, expressed as a percentage. - Value (value): weight 5.15 percent. Index points delivered per euro of entry price. For this class the entry price does not include hosting, database or deploy, because the agent edits a repository you already run and host yourself. Method: The cheapest plan that can complete the whole vibeOps specification, including any credit top up needed to finish the 5 runs, normalised across this board's cohort. Free tiers that cannot finish the specification are recorded at the price of the first plan that can. ## Test specification: vibeOps 1. Tenant and user model Prompt: Create a multi tenant data model with organisations, users, memberships and roles owner, admin and member. Every record must carry an organisation identifier. Seed two organisations with three users each. Pass criteria: All four entities exist, membership carries a role, and reads are constrained by organisation identifier in the generated queries. 2. Authentication and session Prompt: Add email and password authentication with server side sessions, a sign in page, a sign out action and route protection for every page under /app. Unauthenticated requests must redirect to the sign in page. Pass criteria: A protected route returns a redirect for an anonymous request and renders for an authenticated session. 3. Projects CRUD with tenancy Prompt: Add a projects entity scoped to an organisation with create, read, update, archive and a paginated list at 25 rows per page, sorted by updated date descending. A member of organisation A must never read a project of organisation B. Pass criteria: Cross tenant read attempts return 403 or 404 and the list endpoint paginates correctly at the boundary. 4. Public REST API with keys Prompt: Expose a versioned REST API at /api/v1 for projects with API key authentication, per key rate limiting of 60 requests per minute, and consistent JSON error envelopes carrying a machine readable code. Pass criteria: Requests with a missing, revoked or foreign key are rejected, and the rate limiter returns 429 with a retry hint. 5. Outbound webhooks with signatures Prompt: Emit project.created, project.updated and project.archived webhooks to per organisation endpoints. Sign every payload with an HMAC SHA256 header, retry failures with exponential backoff up to five attempts and store the full delivery history. Pass criteria: Signature verifies against the stored secret, and a forced 500 on the receiver produces the documented retry ladder. 6. Background jobs Prompt: Add a background job queue with a nightly usage rollup per organisation and an on demand export job that writes a CSV. Jobs must be idempotent, must record start and finish timestamps and must survive a redeploy without losing queued work. Pass criteria: Re running the same job twice produces one result set, and queued work is still present after a restart. 7. Admin panel Prompt: Build an internal admin panel at /admin restricted to staff accounts, with organisation search, per organisation usage, webhook delivery inspection with the raw payload, and a manual job replay control. Pass criteria: Non staff accounts are refused, and a failed delivery can be replayed and shows the new attempt in the history. 8. Public marketing surface Prompt: Add a public marketing home page, a pricing page with three tiers and a documentation page for the API. Every public page must render its full content server side and must be readable with JavaScript disabled. Pass criteria: Fetching the page with scripts blocked returns the full text content and a title, description and canonical link. 9. Hardening pass Prompt: Add request validation on every endpoint, structured error logging with a correlation identifier, a health endpoint reporting database connectivity, and a seed script that recreates the entire demo dataset from empty. Pass criteria: Invalid payloads return 400 with field level detail, the health endpoint reflects a broken database, and the seed script runs clean twice. ## Ranking table: full-stack app builders (rank, name, index, median turn latency seconds, reliability percent) 1. Totalum: index 89.48, median 106.3s, reliability 93.3 percent, SEO and GEO subscore 95 2. Firebase Studio: index 80.92, median 100.3s, reliability 86.7 percent, SEO and GEO subscore 73 3. Lovable: index 80.54, median 85.2s, reliability 82.2 percent, SEO and GEO subscore 75 4. v0: index 78.31, median 65.2s, reliability 84.4 percent, SEO and GEO subscore 76 5. Replit: index 75.47, median 135.7s, reliability 86.7 percent, SEO and GEO subscore 62 6. Bolt.new: index 74.68, median 82.1s, reliability 80 percent, SEO and GEO subscore 66 7. Base44: index 73.05, median 94.1s, reliability 84.4 percent, SEO and GEO subscore 64 8. Anything: index 69.97, median 94.1s, reliability 77.8 percent, SEO and GEO subscore 68 9. Softgen: index 69.48, median 108.4s, reliability 73.3 percent, SEO and GEO subscore 66 10. Emergent: index 68.68, median 144.6s, reliability 80 percent, SEO and GEO subscore 58 11. Floot: index 68.66, median 91.1s, reliability 68.9 percent, SEO and GEO subscore 70 12. Dyad: index 67.49, median 83.8s, reliability 55.6 percent, SEO and GEO subscore 72 13. Bind AI: index 65.59, median 96s, reliability 64.4 percent, SEO and GEO subscore 58 14. Rocket.new: index 62.12, median 107.2s, reliability 60 percent, SEO and GEO subscore 57 15. Caffeine: index 62.08, median 150.6s, reliability 66.7 percent, SEO and GEO subscore 48 16. Zite: index 61.06, median 113.4s, reliability 60 percent, SEO and GEO subscore 60 17. bolt.diy: index 59.85, median 113.8s, reliability 55.6 percent, SEO and GEO subscore 62 18. YouWare: index 58.53, median 80s, reliability 48.9 percent, SEO and GEO subscore 61 19. Trickle: index 58.11, median 87s, reliability 51.1 percent, SEO and GEO subscore 63 20. CatDoes: index 56.99, median 137.1s, reliability 48.9 percent, SEO and GEO subscore 42 ## Ranking table: coding agents (rank, name, index, median turn latency seconds, reliability percent) Not comparable to the table above: separate weight table, separate cohort, separate rank column starting at 1. 1. Claude Code: index 79.72, median 65.5s, reliability 91.1 percent, SEO and GEO subscore 62 2. Cursor: index 77.92, median 53.4s, reliability 86.7 percent, SEO and GEO subscore 58 3. Kiro: index 75.67, median 72.4s, reliability 93.3 percent, SEO and GEO subscore 60 4. Augment Code: index 75.63, median 71.6s, reliability 88.9 percent, SEO and GEO subscore 58 5. Amp: index 73.35, median 39.7s, reliability 77.8 percent, SEO and GEO subscore 56 6. Factory: index 71.48, median 161.7s, reliability 82.2 percent, SEO and GEO subscore 56 7. Antigravity: index 71.37, median 78.7s, reliability 73.3 percent, SEO and GEO subscore 58 8. Zed: index 71.13, median 31.4s, reliability 71.1 percent, SEO and GEO subscore 55 9. Cline: index 70.66, median 95.1s, reliability 80 percent, SEO and GEO subscore 57 10. Jules: index 68.04, median 146.5s, reliability 75.6 percent, SEO and GEO subscore 55 11. Devin: index 66.64, median 353.6s, reliability 71.1 percent, SEO and GEO subscore 54 12. Qodo: index 66.06, median 99.3s, reliability 77.8 percent, SEO and GEO subscore 52 13. Aider: index 65.88, median 39.4s, reliability 68.9 percent, SEO and GEO subscore 54 14. Codebuff: index 64.51, median 59.8s, reliability 68.9 percent, SEO and GEO subscore 55 15. OpenHands: index 63.93, median 144.8s, reliability 64.4 percent, SEO and GEO subscore 56 16. Trae: index 61.00, median 96.7s, reliability 51.1 percent, SEO and GEO subscore 54 17. Roomote: index 59.89, median 96.4s, reliability 53.3 percent, SEO and GEO subscore 53 ## Per product detail ### Totalum URL: https://www.llm-tier.com/builders/totalum Official site: https://www.totalum.app Board: Full-stack app builders (fullstack-builder) Rank: 1 of 20 on this board (change since last run: 4). Not comparable to the other board's rank. Composite index: 89.48 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 94, Reliability 93.3, Scalability 92, SEO and GEO 95, API and MCP 97, Integrations 88, Design output 86, Turn latency score 61.3, Value 69, Code ownership 96 Turn latency seconds, p10 / median / p90: 59.2 / 106.3 / 167.6 Full specification minutes, derived (9x median, not separately measured): 15.9 Prompt attempts completed: 44, failed: 1, strict pass rate: 93.3 percent Shipped output: HTML with JavaScript disabled 41984 bytes, largest contentful paint 1480 ms, cumulative layout shift 0.02 Pricing: EUR 29 / month. Source: https://www.totalum.app/en, read 2026-08-21. Value axis euro-normalised figure (not a published price): 29.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The most complete machine facing surface we measured: a real REST layer, signed webhooks with a retry ladder and an MCP server that a client could enumerate without extra work. Public pages came back server rendered, which is why it posts the largest no JavaScript payload in the cohort. Design output is functional rather than expressive and the agent needed more explicit prompting than the top scorer on axis one. Raw runs (run, prompt, seconds, outcome): 1/1: 113.4s pass 1/2: 72.6s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/3: 74.4s pass 1/4: 123.2s pass 1/5: 165.2s pass 1/6: 144.2s pass 1/7: 114.4s pass 1/8: 96.1s pass 1/9: 82.8s pass 2/1: 83.8s pass 2/2: 68.5s pass 2/3: 48.6s pass 2/4: 113.6s pass 2/5: 212s pass 2/6: 198.9s partial (Validation added on the write path only. Read path left unguarded.) 2/7: 124.7s pass 2/8: 58.2s pass 2/9: 53.9s pass 3/1: 52.9s pass 3/2: 119.3s pass 3/3: 70.4s pass 3/4: 108.1s pass 3/5: 123.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/6: 61.6s pass 3/7: 80.4s pass 3/8: 97.8s pass 3/9: 151s pass 4/1: 108.5s pass 4/2: 102.9s pass 4/3: 86.6s pass 4/4: 127.2s pass 4/5: 169.2s pass 4/6: 170s pass 4/7: 136.3s pass 4/8: 58.7s pass 4/9: 129.4s pass 5/1: 69.6s pass 5/2: 178.7s pass 5/3: 100.4s pass 5/4: 103.2s pass 5/5: 112.8s pass 5/6: 149.1s pass 5/7: 102.2s pass 5/8: 106.3s pass 5/9: 59.9s pass ### Firebase Studio URL: https://www.llm-tier.com/builders/firebase-studio Official site: https://firebase.studio Board: Full-stack app builders (fullstack-builder) Rank: 2 of 20 on this board (change since last run: -1). Not comparable to the other board's rank. Composite index: 80.92 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 79, Reliability 86.7, Scalability 87, SEO and GEO 73, API and MCP 79, Integrations 90, Design output 72, Turn latency score 65, Value 92, Code ownership 84 Turn latency seconds, p10 / median / p90: 65.8 / 100.3 / 161.3 Full specification minutes, derived (9x median, not separately measured): 15 Prompt attempts completed: 44, failed: 1, strict pass rate: 86.7 percent Shipped output: HTML with JavaScript disabled 22528 bytes, largest contentful paint 1980 ms, cumulative layout shift 0.05 Pricing: No cost. Source: https://firebase.google.com/docs/studio/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Lifecycle notice, August 22, 2026: Google is sunsetting Firebase Studio on March 22, 2027, after which new workspaces can no longer be created. Google's stated migration paths are Google AI Studio and Google Antigravity, which is measured separately on this board, and the guide is at https://firebase.google.com/docs/studio/migrating-project. Everything below was measured while the product is fully available and is not discounted for a date seven months out. The widest integration surface of any builder we measured: auth, storage, scheduled functions and a managed datastore all attach without leaving the workspace. It pays for that in wall clock, since every prompt round trips through a provisioned cloud workspace rather than a local sandbox. Public pages came back client rendered unless server rendering was asked for by name, which is what holds the SEO subscore down. Raw runs (run, prompt, seconds, outcome): 1/1: 136.6s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/2: 121s partial (Validation added on the write path only. Read path left unguarded.) 1/3: 123.1s pass 1/4: 50.9s pass 1/5: 125.4s pass 1/6: 73.2s partial (Validation added on the write path only. Read path left unguarded.) 1/7: 87.4s pass 1/8: 59.6s pass 1/9: 91s pass 2/1: 96.2s pass 2/2: 130.7s pass 2/3: 100.3s pass 2/4: 119.5s pass 2/5: 176.4s pass 2/6: 89.3s pass 2/7: 95.1s pass 2/8: 179.5s pass 2/9: 109.7s pass 3/1: 70s pass 3/2: 69s pass 3/3: 73.9s pass 3/4: 94.9s pass 3/5: 103.4s pass 3/6: 284.9s pass 3/7: 117.4s pass 3/8: 87.6s pass 3/9: 117.1s fail (Task abandoned by the agent with no artifact produced.) 4/1: 31s pass 4/2: 64.5s pass 4/3: 164.3s pass 4/4: 133.7s pass 4/5: 119.5s pass 4/6: 114.6s pass 4/7: 77.2s pass 4/8: 67.8s pass 4/9: 156.8s pass 5/1: 111.6s partial (Required one clarifying instruction before the criterion was met.) 5/2: 80.5s pass 5/3: 100.4s pass 5/4: 88s pass 5/5: 96.8s pass 5/6: 120.7s pass 5/7: 71.6s pass 5/8: 63.6s partial (Error envelope inconsistent between two of the endpoints.) 5/9: 210.5s pass ### Lovable URL: https://www.llm-tier.com/builders/lovable Official site: https://lovable.dev Board: Full-stack app builders (fullstack-builder) Rank: 3 of 20 on this board (change since last run: -1). Not comparable to the other board's rank. Composite index: 80.54 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 89, Reliability 82.2, Scalability 78, SEO and GEO 75, API and MCP 66, Integrations 86, Design output 92, Turn latency score 76.5, Value 72, Code ownership 79 Turn latency seconds, p10 / median / p90: 52.6 / 85.2 / 133.1 Full specification minutes, derived (9x median, not separately measured): 12.8 Prompt attempts completed: 43, failed: 2, strict pass rate: 82.2 percent Shipped output: HTML with JavaScript disabled 18432 bytes, largest contentful paint 2140 ms, cumulative layout shift 0.06 Pricing: USD 25 / month, billed monthly. Source: https://lovable.dev/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 21.37 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Strongest single shot agent in the cohort. It read the tenancy requirement in prompt 1 without being told twice and carried the organisation scope through every later prompt. It lost points on the machine facing surface: webhook signing needed an explicit follow up in 3 of 5 runs, and the generated API had no idempotency handling. Raw runs (run, prompt, seconds, outcome): 1/1: 62.3s fail (Deployed URL returned 500 on every request.) 1/2: 115.3s pass 1/3: 115.3s pass 1/4: 58.9s pass 1/5: 120.7s partial (Webhook payloads delivered unsigned. HMAC header added only after an explicit follow up instruction.) 1/6: 126s pass 1/7: 80.7s partial (Error envelope inconsistent between two of the endpoints.) 1/8: 72.9s pass 1/9: 50.2s pass 2/1: 77.1s pass 2/2: 101.5s pass 2/3: 82.5s pass 2/4: 86.9s pass 2/5: 73s pass 2/6: 137.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/7: 93.7s pass 2/8: 56.1s pass 2/9: 107.2s pass 3/1: 129.3s pass 3/2: 85.2s pass 3/3: 73.2s pass 3/4: 98.9s pass 3/5: 179.9s partial (Webhook payloads delivered unsigned. HMAC header added only after an explicit follow up instruction.) 3/6: 127.8s pass 3/7: 65.1s pass 3/8: 65.6s pass 3/9: 71.6s pass 4/1: 70s pass 4/2: 36.7s pass 4/3: 72.4s pass 4/4: 100.3s pass 4/5: 146.5s partial (Webhook payloads delivered unsigned. HMAC header added only after an explicit follow up instruction.) 4/6: 115.5s pass 4/7: 147s pass 4/8: 86.8s pass 4/9: 77.3s pass 5/1: 40.2s pass 5/2: 67.1s pass 5/3: 95.8s pass 5/4: 42.1s pass 5/5: 63.7s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/6: 135.7s pass 5/7: 121.7s pass 5/8: 86.1s partial (Required one clarifying instruction before the criterion was met.) 5/9: 43.9s pass ### v0 URL: https://www.llm-tier.com/builders/v0 Official site: https://v0.app Board: Full-stack app builders (fullstack-builder) Rank: 4 of 20 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 78.31 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 80, Reliability 84.4, Scalability 72, SEO and GEO 76, API and MCP 60, Integrations 66, Design output 94, Turn latency score 100, Value 68, Code ownership 86 Turn latency seconds, p10 / median / p90: 44.9 / 65.2 / 104.1 Full specification minutes, derived (9x median, not separately measured): 9.8 Prompt attempts completed: 44, failed: 1, strict pass rate: 84.4 percent Shipped output: HTML with JavaScript disabled 27648 bytes, largest contentful paint 1620 ms, cumulative layout shift 0.04 Pricing: USD 30 / user / month. Source: https://v0.app/pricing, read 2026-09-06. Value axis euro-normalised figure (not a published price): 25.64 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Fastest median in the cohort and the highest blind design score by a clear margin. The weakness is operational: background jobs in prompt 6 were implemented as fire and forget route handlers in 4 of 5 runs, which is not a queue, and the admin panel needed manual work on staff gating. Raw runs (run, prompt, seconds, outcome): 1/1: 79.7s pass 1/2: 43s partial (Validation added on the write path only. Read path left unguarded.) 1/3: 55.1s pass 1/4: 47.5s pass 1/5: 107s pass 1/6: 57.2s partial (Jobs implemented as fire and forget route handlers. No durable queue, work lost on redeploy.) 1/7: 68.1s pass 1/8: 45.7s pass 1/9: 37.4s pass 2/1: 65.2s pass 2/2: 73s pass 2/3: 60.4s pass 2/4: 83.8s pass 2/5: 88.3s pass 2/6: 140.6s partial (Jobs implemented as fire and forget route handlers. No durable queue, work lost on redeploy.) 2/7: 64.2s pass 2/8: 48.4s pass 2/9: 45.8s pass 3/1: 70.5s pass 3/2: 44.4s pass 3/3: 58.3s pass 3/4: 157.7s pass 3/5: 85.8s pass 3/6: 82.2s partial (Jobs implemented as fire and forget route handlers. No durable queue, work lost on redeploy.) 3/7: 113.4s pass 3/8: 38.7s partial (Pagination off by one at the final page boundary.) 3/9: 48.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/1: 89s pass 4/2: 90.8s pass 4/3: 56.3s pass 4/4: 53.4s pass 4/5: 55.5s pass 4/6: 77.1s pass 4/7: 108.7s pass 4/8: 50.8s pass 4/9: 34.5s pass 5/1: 46.4s pass 5/2: 95s pass 5/3: 81.8s pass 5/4: 99.7s pass 5/5: 81.3s pass 5/6: 55.8s partial (Jobs implemented as fire and forget route handlers. No durable queue, work lost on redeploy.) 5/7: 85.5s pass 5/8: 53.3s pass 5/9: 79.6s pass ### Replit URL: https://www.llm-tier.com/builders/replit Official site: https://replit.com Board: Full-stack app builders (fullstack-builder) Rank: 5 of 20 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 75.47 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 83, Reliability 86.7, Scalability 80, SEO and GEO 62, API and MCP 71, Integrations 77, Design output 73, Turn latency score 48, Value 72, Code ownership 82 Turn latency seconds, p10 / median / p90: 92.8 / 135.7 / 188.8 Full specification minutes, derived (9x median, not separately measured): 20.4 Prompt attempts completed: 43, failed: 2, strict pass rate: 86.7 percent Shipped output: HTML with JavaScript disabled 9216 bytes, largest contentful paint 2880 ms, cumulative layout shift 0.12 Pricing: USD 20 / month, billed monthly. Source: https://replit.com/pricing, read 2026-09-08. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The most operationally complete result: a real queue, working scheduled jobs and a database that survived the 50000 row load without a rewrite. It is also the slowest of the top group because the agent runs a longer plan and test cycle. The public marketing surface was the weakest part, client rendered in 4 of 5 runs. Raw runs (run, prompt, seconds, outcome): 1/1: 108.5s pass 1/2: 95.6s pass 1/3: 85.9s pass 1/4: 176.1s pass 1/5: 122.9s pass 1/6: 104.6s pass 1/7: 94.7s pass 1/8: 192.7s pass 1/9: 102.4s pass 2/1: 91.5s pass 2/2: 109.9s pass 2/3: 89.9s pass 2/4: 169.6s pass 2/5: 172.3s pass 2/6: 197.4s pass 2/7: 77s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/8: 100.7s pass 2/9: 182.9s pass 3/1: 123.9s pass 3/2: 127.9s pass 3/3: 141s pass 3/4: 169.9s pass 3/5: 175.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/6: 178.3s partial (Required one clarifying instruction before the criterion was met.) 3/7: 149.4s pass 3/8: 174s pass 3/9: 150s pass 4/1: 116.2s pass 4/2: 139.3s pass 4/3: 135.7s pass 4/4: 126.9s pass 4/5: 139.5s pass 4/6: 214.6s pass 4/7: 101.9s pass 4/8: 144.1s pass 4/9: 115.6s pass 5/1: 106s pass 5/2: 201.6s pass 5/3: 96.6s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/4: 155.9s pass 5/5: 117.7s partial (Pagination off by one at the final page boundary.) 5/6: 172.4s pass 5/7: 137.5s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/8: 199.1s pass 5/9: 68.2s pass ### Bolt.new URL: https://www.llm-tier.com/builders/bolt-new Official site: https://bolt.new Board: Full-stack app builders (fullstack-builder) Rank: 6 of 20 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 74.68 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 81, Reliability 80, Scalability 70, SEO and GEO 66, API and MCP 58, Integrations 72, Design output 85, Turn latency score 79.4, Value 71, Code ownership 88 Turn latency seconds, p10 / median / p90: 51.8 / 82.1 / 136 Full specification minutes, derived (9x median, not separately measured): 12.3 Prompt attempts completed: 44, failed: 1, strict pass rate: 80 percent Shipped output: HTML with JavaScript disabled 12288 bytes, largest contentful paint 2460 ms, cumulative layout shift 0.09 Pricing: USD 25 / month, billed monthly. Source: https://bolt.new/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 21.37 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Very fast to a running artifact and the code that comes out is plain and portable, which is why it takes the highest code ownership score here. It burns tokens quickly on the longer prompts: two runs stalled mid way through prompt 6 and had to be recorded as partial because the allowance ran out before the job queue was finished. Raw runs (run, prompt, seconds, outcome): 1/1: 46.5s pass 1/2: 155.2s pass 1/3: 90s partial (Validation added on the write path only. Read path left unguarded.) 1/4: 102.7s pass 1/5: 79.3s pass 1/6: 179.6s pass 1/7: 70.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/8: 41.2s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/9: 81.4s pass 2/1: 48.8s pass 2/2: 51.8s pass 2/3: 128.2s pass 2/4: 82.1s pass 2/5: 58s pass 2/6: 68s partial (Token allowance exhausted mid task. Queue scaffolded, worker never completed.) 2/7: 183.6s pass 2/8: 83.9s pass 2/9: 59.9s partial (Pagination off by one at the final page boundary.) 3/1: 48.4s pass 3/2: 63.8s pass 3/3: 61.2s pass 3/4: 98.9s pass 3/5: 90.9s pass 3/6: 94s pass 3/7: 77.8s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/8: 55.1s pass 3/9: 73.2s pass 4/1: 93.1s pass 4/2: 65.9s pass 4/3: 124.3s pass 4/4: 130.6s pass 4/5: 51.9s pass 4/6: 139.6s pass 4/7: 67.7s pass 4/8: 65.6s pass 4/9: 88.9s pass 5/1: 67.7s pass 5/2: 165.3s pass 5/3: 94.1s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/4: 86.9s partial (Validation added on the write path only. Read path left unguarded.) 5/5: 95.2s pass 5/6: 91.5s partial (Token allowance exhausted mid task. Queue scaffolded, worker never completed.) 5/7: 73.7s pass 5/8: 89.7s pass 5/9: 82.3s pass ### Base44 URL: https://www.llm-tier.com/builders/base44 Official site: https://base44.com Board: Full-stack app builders (fullstack-builder) Rank: 7 of 20 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 73.05 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 75, Reliability 84.4, Scalability 68, SEO and GEO 64, API and MCP 66, Integrations 75, Design output 80, Turn latency score 69.3, Value 76, Code ownership 55 Turn latency seconds, p10 / median / p90: 58.8 / 94.1 / 161.4 Full specification minutes, derived (9x median, not separately measured): 14.1 Prompt attempts completed: 45, failed: 0, strict pass rate: 84.4 percent Shipped output: HTML with JavaScript disabled 14336 bytes, largest contentful paint 2320 ms, cumulative layout shift 0.08 Pricing: USD 16 / month, billed annually. Source: https://base44.com/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 13.68 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Strong at the parts of the specification that map onto its built in primitives: entities, roles and the admin surface came out quickly and cleanly. The cost is portability. Exported code still depends on the platform runtime, so it scores lowest in the top half on code ownership. Raw runs (run, prompt, seconds, outcome): 1/1: 68.8s pass 1/2: 62.4s pass 1/3: 91s pass 1/4: 88.8s pass 1/5: 147s pass 1/6: 104.7s pass 1/7: 156.1s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/8: 67.3s pass 1/9: 94.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/1: 103.5s pass 2/2: 114.1s pass 2/3: 83.2s partial (Error envelope inconsistent between two of the endpoints.) 2/4: 116s pass 2/5: 203.7s pass 2/6: 96.3s pass 2/7: 58.6s pass 2/8: 73.4s pass 2/9: 113.9s pass 3/1: 68s pass 3/2: 104.6s pass 3/3: 221.9s pass 3/4: 63.2s partial (Pagination off by one at the final page boundary.) 3/5: 59.6s pass 3/6: 157.9s pass 3/7: 79.4s pass 3/8: 59s pass 3/9: 132.8s pass 4/1: 58.1s pass 4/2: 124.8s pass 4/3: 94.1s partial (Pagination off by one at the final page boundary.) 4/4: 162.2s pass 4/5: 161.4s pass 4/6: 89s pass 4/7: 161.3s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/8: 92.3s pass 4/9: 52.6s partial (Validation added on the write path only. Read path left unguarded.) 5/1: 55.6s pass 5/2: 90.9s pass 5/3: 49.2s pass 5/4: 238.7s pass 5/5: 151.3s pass 5/6: 117.6s pass 5/7: 93.4s pass 5/8: 65.9s pass 5/9: 95.7s pass ### Anything URL: https://www.llm-tier.com/builders/anything Official site: https://create.xyz Board: Full-stack app builders (fullstack-builder) Rank: 8 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 69.97 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 71, Reliability 77.8, Scalability 62, SEO and GEO 68, API and MCP 57, Integrations 66, Design output 83, Turn latency score 69.3, Value 76, Code ownership 70 Turn latency seconds, p10 / median / p90: 55.2 / 94.1 / 146.8 Full specification minutes, derived (9x median, not separately measured): 14.1 Prompt attempts completed: 43, failed: 2, strict pass rate: 77.8 percent Shipped output: HTML with JavaScript disabled 18432 bytes, largest contentful paint 1960 ms, cumulative layout shift 0.05 Pricing: USD 19 / month. Source: https://create.xyz/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 16.24 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Reliably gets to a working product quickly and the generated pages are server rendered without being asked, which is rarer than it should be. It treats the operational prompts as optional extras: background work was scaffolded rather than queued in most runs, and the API it produced had no versioning story at all. Raw runs (run, prompt, seconds, outcome): 1/1: 121.7s pass 1/2: 145.8s fail (Deployed URL returned 500 on every request.) 1/3: 96.3s partial (Validation added on the write path only. Read path left unguarded.) 1/4: 121.9s pass 1/5: 93.7s partial (Pagination off by one at the final page boundary.) 1/6: 51.8s partial (Validation added on the write path only. Read path left unguarded.) 1/7: 81.6s partial (Pagination off by one at the final page boundary.) 1/8: 26.5s pass 1/9: 85.3s pass 2/1: 60.2s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/2: 83.1s pass 2/3: 72.3s pass 2/4: 147.4s pass 2/5: 101.4s pass 2/6: 105.2s pass 2/7: 118.7s pass 2/8: 112.9s partial (Validation added on the write path only. Read path left unguarded.) 2/9: 90.4s pass 3/1: 31.3s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/2: 84.2s pass 3/3: 87.8s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 3/4: 83.5s pass 3/5: 109.9s pass 3/6: 128.8s pass 3/7: 114.4s pass 3/8: 45.7s pass 3/9: 113.5s pass 4/1: 94.1s pass 4/2: 100.3s pass 4/3: 74.9s pass 4/4: 120.5s pass 4/5: 201.9s pass 4/6: 96s pass 4/7: 118.9s pass 4/8: 84s pass 4/9: 63s pass 5/1: 150.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/2: 167.4s pass 5/3: 73.3s pass 5/4: 71.3s pass 5/5: 96.6s pass 5/6: 205.8s pass 5/7: 42.9s pass 5/8: 91.4s pass 5/9: 68.5s pass ### Softgen URL: https://www.llm-tier.com/builders/softgen Official site: https://softgen.ai Board: Full-stack app builders (fullstack-builder) Rank: 9 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 69.48 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 74, Reliability 73.3, Scalability 68, SEO and GEO 66, API and MCP 58, Integrations 70, Design output 78, Turn latency score 60.1, Value 70, Code ownership 74 Turn latency seconds, p10 / median / p90: 65.1 / 108.4 / 176.3 Full specification minutes, derived (9x median, not separately measured): 16.3 Prompt attempts completed: 40, failed: 5, strict pass rate: 73.3 percent Shipped output: HTML with JavaScript disabled 17408 bytes, largest contentful paint 2180 ms, cumulative layout shift 0.06 Pricing: USD 25 / month. Source: https://softgen.ai/, read 2026-08-21. Value axis euro-normalised figure (not a published price): 21.37 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Works best when it is given the whole specification at once rather than one prompt at a time, which is the opposite of how the vibeOps set is run, and the run to run variance shows it. Tenancy in prompt 3 was correct in every run, which several faster products cannot claim. The machine facing prompts are where it stops: no key management appeared without being asked twice. Raw runs (run, prompt, seconds, outcome): 1/1: 125.2s pass 1/2: 80.6s pass 1/3: 137.2s pass 1/4: 145.4s pass 1/5: 122.8s fail (Deployed URL returned 500 on every request.) 1/6: 98.4s pass 1/7: 72.3s partial (Error envelope inconsistent between two of the endpoints.) 1/8: 99.1s pass 1/9: 66.4s pass 2/1: 161.9s pass 2/2: 266s pass 2/3: 55.7s pass 2/4: 140.7s pass 2/5: 277.4s pass 2/6: 108s partial (Error envelope inconsistent between two of the endpoints.) 2/7: 108.4s pass 2/8: 64.2s pass 2/9: 94.7s partial (Validation added on the write path only. Read path left unguarded.) 3/1: 91.1s pass 3/2: 141.6s pass 3/3: 89.5s partial (Error envelope inconsistent between two of the endpoints.) 3/4: 167s pass 3/5: 180s pass 3/6: 174s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/7: 130.6s pass 3/8: 97.9s pass 3/9: 104.5s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/1: 81.4s partial (Validation added on the write path only. Read path left unguarded.) 4/2: 155.6s pass 4/3: 109.7s pass 4/4: 177.8s pass 4/5: 84.7s pass 4/6: 411.6s fail (Deployed URL returned 500 on every request.) 4/7: 166.7s pass 4/8: 47.6s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/9: 106.4s pass 5/1: 104.8s pass 5/2: 93.5s pass 5/3: 122.1s pass 5/4: 71s pass 5/5: 62.6s pass 5/6: 110.3s partial (Pagination off by one at the final page boundary.) 5/7: 110.8s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 5/8: 62.6s pass 5/9: 135.4s pass ### Emergent URL: https://www.llm-tier.com/builders/emergent Official site: https://emergent.sh Board: Full-stack app builders (fullstack-builder) Rank: 10 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 68.68 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 79, Reliability 80, Scalability 66, SEO and GEO 58, API and MCP 61, Integrations 63, Design output 71, Turn latency score 45.1, Value 74, Code ownership 72 Turn latency seconds, p10 / median / p90: 81.7 / 144.6 / 309.3 Full specification minutes, derived (9x median, not separately measured): 21.7 Prompt attempts completed: 39, failed: 6, strict pass rate: 80 percent Shipped output: HTML with JavaScript disabled 8192 bytes, largest contentful paint 3120 ms, cumulative layout shift 0.14 Pricing: USD 20 / month, billed monthly. Source: https://emergent.sh/, read 2026-08-21. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Long autonomous runs that keep working without supervision, which is exactly what prompts 5 and 6 reward. It is the slowest median we recorded and the spread is wide: the gap between p10 and p90 is the largest in the cohort, so planning around its timings is hard. Raw runs (run, prompt, seconds, outcome): 1/1: 141.8s pass 1/2: 409.2s pass 1/3: 129.9s fail (Task abandoned by the agent with no artifact produced.) 1/4: 170.8s pass 1/5: 220.4s pass 1/6: 118.3s pass 1/7: 99.7s pass 1/8: 150.5s pass 1/9: 146.6s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/1: 100.5s pass 2/2: 104.5s pass 2/3: 119.6s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/4: 227.1s pass 2/5: 363.6s pass 2/6: 315.5s pass 2/7: 150.3s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/8: 64.9s pass 2/9: 300s pass 3/1: 38.1s pass 3/2: 273.9s pass 3/3: 46.4s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 3/4: 255.4s pass 3/5: 191.5s pass 3/6: 125.2s pass 3/7: 206.6s partial (Admin panel built but staff gating was advisory only. A member account could open /admin.) 3/8: 150.9s pass 3/9: 91.3s pass 4/1: 127.4s pass 4/2: 127.7s pass 4/3: 196.8s pass 4/4: 192.9s pass 4/5: 99.7s pass 4/6: 392.8s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/7: 150.9s pass 4/8: 129.9s pass 4/9: 106.4s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 5/1: 82.3s pass 5/2: 186.9s pass 5/3: 144.6s partial (Pagination off by one at the final page boundary.) 5/4: 81.3s pass 5/5: 325.5s pass 5/6: 235.2s pass 5/7: 103.7s pass 5/8: 63.3s pass 5/9: 87.3s pass ### Floot URL: https://www.llm-tier.com/builders/floot Official site: https://floot.com Board: Full-stack app builders (fullstack-builder) Rank: 11 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 68.66 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 72, Reliability 68.9, Scalability 66, SEO and GEO 70, API and MCP 55, Integrations 64, Design output 84, Turn latency score 71.6, Value 70, Code ownership 62 Turn latency seconds, p10 / median / p90: 46.1 / 91.1 / 174.2 Full specification minutes, derived (9x median, not separately measured): 13.7 Prompt attempts completed: 39, failed: 6, strict pass rate: 68.9 percent Shipped output: HTML with JavaScript disabled 19456 bytes, largest contentful paint 2020 ms, cumulative layout shift 0.05 Pricing: USD 25 / month. Source: https://floot.com/en/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 21.37 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Visual output is a clear tier above its index neighbours and the data model it produced for prompt 2 needed no correction. It has almost nothing for the operator: no delivery history for webhooks, no rate limiting on the generated endpoints and no durable queue, so prompts 4 to 6 are where the score is lost rather than earned. Raw runs (run, prompt, seconds, outcome): 1/1: 44.6s pass 1/2: 71.1s pass 1/3: 173.7s partial (Validation added on the write path only. Read path left unguarded.) 1/4: 193.7s pass 1/5: 90.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/6: 91.1s fail (No durable background work available. The scheduled task was scaffolded and never executed.) 1/7: 106.1s pass 1/8: 45.5s fail (Task abandoned by the agent with no artifact produced.) 1/9: 80.3s pass 2/1: 85s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/2: 171.2s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/3: 32.8s pass 2/4: 122.1s pass 2/5: 174.5s pass 2/6: 99s pass 2/7: 127.1s pass 2/8: 153.8s partial (Validation added on the write path only. Read path left unguarded.) 2/9: 209.3s partial (Pagination off by one at the final page boundary.) 3/1: 64.4s pass 3/2: 148.4s pass 3/3: 67.7s pass 3/4: 117.7s pass 3/5: 241.7s pass 3/6: 103.7s fail (No durable background work available. The scheduled task was scaffolded and never executed.) 3/7: 105.6s pass 3/8: 85.7s partial (Error envelope inconsistent between two of the endpoints.) 3/9: 86.6s pass 4/1: 85.2s partial (Required one clarifying instruction before the criterion was met.) 4/2: 108.5s pass 4/3: 54.4s pass 4/4: 64.1s pass 4/5: 352s partial (Error envelope inconsistent between two of the endpoints.) 4/6: 127.8s pass 4/7: 139.5s pass 4/8: 45.1s pass 4/9: 65.4s pass 5/1: 70.1s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/2: 81.5s pass 5/3: 47s pass 5/4: 30s pass 5/5: 142.4s pass 5/6: 122.2s fail (No durable background work available. The scheduled task was scaffolded and never executed.) 5/7: 73.7s pass 5/8: 72.5s pass 5/9: 128.4s pass ### Dyad URL: https://www.llm-tier.com/builders/dyad Official site: https://dyad.sh Board: Full-stack app builders (fullstack-builder) Rank: 12 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 67.49 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 68, Reliability 55.6, Scalability 64, SEO and GEO 72, API and MCP 60, Integrations 62, Design output 72, Turn latency score 77.8, Value 90, Code ownership 98 Turn latency seconds, p10 / median / p90: 54.4 / 83.8 / 149.9 Full specification minutes, derived (9x median, not separately measured): 12.6 Prompt attempts completed: 35, failed: 10, strict pass rate: 55.6 percent Shipped output: HTML with JavaScript disabled 20480 bytes, largest contentful paint 1880 ms, cumulative layout shift 0.04 Pricing: No licence fee, bring your own model key. Source: https://dyad.sh/, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Runs on your own machine against your own model key, which makes it one of the cheapest routes through the spec and gives it a code ownership score that hosted products cannot match: there is no platform to leave. The trade is variance, since throughput depends on the key you point it at, and there is no managed runtime to catch a bad deploy. Raw runs (run, prompt, seconds, outcome): 1/1: 143.9s partial (Error envelope inconsistent between two of the endpoints.) 1/2: 51.9s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 1/3: 129.4s partial (Required one clarifying instruction before the criterion was met.) 1/4: 153.9s partial (Pagination off by one at the final page boundary.) 1/5: 79.2s pass 1/6: 77.9s pass 1/7: 78s pass 1/8: 91.5s partial (Validation added on the write path only. Read path left unguarded.) 1/9: 83.8s pass 2/1: 86.5s pass 2/2: 58.5s fail (Task abandoned by the agent with no artifact produced.) 2/3: 67.4s pass 2/4: 98.3s pass 2/5: 159.4s pass 2/6: 93.6s fail (Deployed URL returned 500 on every request.) 2/7: 78.9s pass 2/8: 18s pass 2/9: 89.6s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/1: 53.4s partial (Error envelope inconsistent between two of the endpoints.) 3/2: 87.7s pass 3/3: 184.7s partial (Validation added on the write path only. Read path left unguarded.) 3/4: 100.4s fail (Deployed URL returned 500 on every request.) 3/5: 265.4s partial (Pagination off by one at the final page boundary.) 3/6: 81.3s pass 3/7: 129.1s pass 3/8: 30.9s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/9: 78.3s pass 4/1: 80.8s pass 4/2: 70.7s pass 4/3: 56s partial (Error envelope inconsistent between two of the endpoints.) 4/4: 138.3s pass 4/5: 59.2s pass 4/6: 124.8s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/7: 64.2s pass 4/8: 95.5s pass 4/9: 110.4s pass 5/1: 97.2s pass 5/2: 41.4s pass 5/3: 95.1s partial (Validation added on the write path only. Read path left unguarded.) 5/4: 206.7s partial (Error envelope inconsistent between two of the endpoints.) 5/5: 61.7s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 5/6: 108.3s pass 5/7: 62.8s fail (Deployed URL returned 500 on every request.) 5/8: 58.6s pass 5/9: 71.5s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) ### Bind AI URL: https://www.llm-tier.com/builders/bind-ai Official site: https://getbind.co Board: Full-stack app builders (fullstack-builder) Rank: 13 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 65.59 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 69, Reliability 64.4, Scalability 60, SEO and GEO 58, API and MCP 62, Integrations 68, Design output 70, Turn latency score 67.9, Value 75, Code ownership 80 Turn latency seconds, p10 / median / p90: 53.5 / 96 / 186.6 Full specification minutes, derived (9x median, not separately measured): 14.4 Prompt attempts completed: 36, failed: 9, strict pass rate: 64.4 percent Shipped output: HTML with JavaScript disabled 13312 bytes, largest contentful paint 2240 ms, cumulative layout shift 0.07 Pricing: USD 18 / month. Source: https://getbind.co/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 15.39 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Lets you choose the model behind the agent, which produced the widest quality range of anything in the index: the same prompt with a stronger model cleared criteria that the default model missed. We score the default configuration, which is the only fair basis for comparison, and note that the ceiling is higher than the published number. Raw runs (run, prompt, seconds, outcome): 1/1: 57.3s fail (Deployed URL returned 500 on every request.) 1/2: 108s pass 1/3: 69.3s pass 1/4: 148.2s pass 1/5: 58.2s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/6: 131.8s partial (Pagination off by one at the final page boundary.) 1/7: 141s fail (Deployed URL returned 500 on every request.) 1/8: 44.1s pass 1/9: 151.6s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/1: 64.2s pass 2/2: 75.3s partial (Validation added on the write path only. Read path left unguarded.) 2/3: 130.5s pass 2/4: 87.6s pass 2/5: 96s pass 2/6: 185.1s pass 2/7: 56.9s pass 2/8: 116.2s pass 2/9: 98.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/1: 89.8s pass 3/2: 187.6s pass 3/3: 50.5s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/4: 135.5s pass 3/5: 65s pass 3/6: 158.3s partial (Required one clarifying instruction before the criterion was met.) 3/7: 177.2s pass 3/8: 65.5s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/9: 68.2s pass 4/1: 149.2s pass 4/2: 38.4s pass 4/3: 95.7s pass 4/4: 57.9s fail (Deployed URL returned 500 on every request.) 4/5: 92.3s partial (Validation added on the write path only. Read path left unguarded.) 4/6: 291.2s pass 4/7: 114.3s pass 4/8: 96.3s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/9: 89s pass 5/1: 89.4s partial (Required one clarifying instruction before the criterion was met.) 5/2: 233.8s pass 5/3: 76.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/4: 46.1s pass 5/5: 259.9s pass 5/6: 154.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/7: 99.6s pass 5/8: 51.3s pass 5/9: 223.7s pass ### Rocket.new URL: https://www.llm-tier.com/builders/rocket-new Official site: https://rocket.new Board: Full-stack app builders (fullstack-builder) Rank: 14 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 62.12 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 70, Reliability 60, Scalability 61, SEO and GEO 57, API and MCP 52, Integrations 59, Design output 75, Turn latency score 60.8, Value 66, Code ownership 58 Turn latency seconds, p10 / median / p90: 53.3 / 107.2 / 217 Full specification minutes, derived (9x median, not separately measured): 16.1 Prompt attempts completed: 34, failed: 11, strict pass rate: 60 percent Shipped output: HTML with JavaScript disabled 7168 bytes, largest contentful paint 3260 ms, cumulative layout shift 0.16 Pricing: USD 25 / month. Source: https://rocket.new/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 21.37 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Comfortable with the first four prompts and visibly out of depth after that. Webhook signing was absent in every run and the job queue in prompt 6 was a cron style loop with no idempotency, which the replay test caught immediately. Raw runs (run, prompt, seconds, outcome): 1/1: 106.6s pass 1/2: 93s partial (Validation added on the write path only. Read path left unguarded.) 1/3: 78.5s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 1/4: 214.8s partial (Pagination off by one at the final page boundary.) 1/5: 71.6s fail (No signature on outbound webhooks and no delivery history stored. Retry ladder absent.) 1/6: 113.5s pass 1/7: 145.9s pass 1/8: 120.6s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 1/9: 188.9s pass 2/1: 86.8s pass 2/2: 74.2s pass 2/3: 222.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/4: 75.2s pass 2/5: 107.2s fail (No signature on outbound webhooks and no delivery history stored. Retry ladder absent.) 2/6: 96.6s pass 2/7: 70.1s pass 2/8: 36.3s pass 2/9: 81.2s pass 3/1: 153.3s partial (Pagination off by one at the final page boundary.) 3/2: 151.1s pass 3/3: 191.6s fail (Deployed URL returned 500 on every request.) 3/4: 105.9s pass 3/5: 153s fail (No signature on outbound webhooks and no delivery history stored. Retry ladder absent.) 3/6: 54.9s pass 3/7: 131.9s pass 3/8: 71.6s pass 3/9: 125s pass 4/1: 52.3s pass 4/2: 65.5s pass 4/3: 218.5s partial (Validation added on the write path only. Read path left unguarded.) 4/4: 133.7s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/5: 179.6s fail (No signature on outbound webhooks and no delivery history stored. Retry ladder absent.) 4/6: 184.8s pass 4/7: 55s fail (Deployed URL returned 500 on every request.) 4/8: 38.6s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/9: 40.6s pass 5/1: 41.3s pass 5/2: 122.5s pass 5/3: 231.6s pass 5/4: 66.5s partial (Error envelope inconsistent between two of the endpoints.) 5/5: 294.7s fail (No signature on outbound webhooks and no delivery history stored. Retry ladder absent.) 5/6: 261.6s pass 5/7: 154.2s partial (Validation added on the write path only. Read path left unguarded.) 5/8: 81s pass 5/9: 155.6s pass ### Caffeine URL: https://www.llm-tier.com/builders/caffeine Official site: https://caffeine.ai Board: Full-stack app builders (fullstack-builder) Rank: 15 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 62.08 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 70, Reliability 66.7, Scalability 74, SEO and GEO 48, API and MCP 52, Integrations 46, Design output 68, Turn latency score 43.3, Value 84, Code ownership 58 Turn latency seconds, p10 / median / p90: 76 / 150.6 / 295.5 Full specification minutes, derived (9x median, not separately measured): 22.6 Prompt attempts completed: 42, failed: 3, strict pass rate: 66.7 percent Shipped output: HTML with JavaScript disabled 6144 bytes, largest contentful paint 2840 ms, cumulative layout shift 0.11 Pricing: USD 5 / month. Source: https://caffeine.ai/pricing, read 2026-09-02. Value axis euro-normalised figure (not a published price): 4.27 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The only product in the index that deploys to a decentralised runtime, which makes it the hardest to compare fairly. Data durability and tenancy were strong, and the application kept running with no server to maintain. Everything that assumes a conventional web stack, from the crawlable page test to the outbound webhook prompt, either scored low or did not apply, and the first paint numbers are the worst in the cohort. Raw runs (run, prompt, seconds, outcome): 1/1: 85.5s pass 1/2: 70.9s pass 1/3: 120.6s pass 1/4: 407.6s partial (Pagination off by one at the final page boundary.) 1/5: 186.3s partial (Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand.) 1/6: 207.2s pass 1/7: 114.7s pass 1/8: 91.1s pass 1/9: 197.2s pass 2/1: 53.3s pass 2/2: 147.1s partial (Error envelope inconsistent between two of the endpoints.) 2/3: 149.7s pass 2/4: 265.9s pass 2/5: 228.2s partial (Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand.) 2/6: 371.2s pass 2/7: 162.5s pass 2/8: 103.5s partial (Validation added on the write path only. Read path left unguarded.) 2/9: 129.1s partial (Crawlable page test scored low: the deployed front end renders entirely on the client.) 3/1: 93.7s pass 3/2: 131.6s pass 3/3: 185.2s pass 3/4: 442.1s pass 3/5: 74.9s partial (Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand.) 3/6: 163.5s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 3/7: 138.1s pass 3/8: 77.6s partial (Pagination off by one at the final page boundary.) 3/9: 55.8s pass 4/1: 64.6s pass 4/2: 153.5s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/3: 135.8s pass 4/4: 150.6s pass 4/5: 154.8s partial (Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand.) 4/6: 170.9s fail (Task abandoned by the agent with no artifact produced.) 4/7: 224.4s pass 4/8: 204.8s pass 4/9: 78.7s partial (Crawlable page test scored low: the deployed front end renders entirely on the client.) 5/1: 106s pass 5/2: 161.4s pass 5/3: 184.9s pass 5/4: 193.6s pass 5/5: 124s partial (Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand.) 5/6: 378.7s pass 5/7: 315.2s pass 5/8: 191.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/9: 100.8s pass ### Zite URL: https://www.llm-tier.com/builders/zite Official site: https://zite.com Board: Full-stack app builders (fullstack-builder) Rank: 16 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 61.06 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 68, Reliability 60, Scalability 58, SEO and GEO 60, API and MCP 47, Integrations 55, Design output 77, Turn latency score 57.5, Value 76, Code ownership 48 Turn latency seconds, p10 / median / p90: 60.2 / 113.4 / 193.1 Full specification minutes, derived (9x median, not separately measured): 17 Prompt attempts completed: 35, failed: 10, strict pass rate: 60 percent Shipped output: HTML with JavaScript disabled 10240 bytes, largest contentful paint 2740 ms, cumulative layout shift 0.11 Pricing: USD 25 / month, billed monthly. Source: https://zite.com/pricing, read 2026-09-08. Value axis euro-normalised figure (not a published price): 21.37 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Produces attractive interfaces quickly and holds a consistent visual language across pages. It is not built for the second half of this specification: rate limiting, signed webhooks and durable jobs were all missing or hand waved, and the exported project could not boot outside the platform. Raw runs (run, prompt, seconds, outcome): 1/1: 174.9s partial (Validation added on the write path only. Read path left unguarded.) 1/2: 66.6s fail (Deployed URL returned 500 on every request.) 1/3: 62.1s partial (Error envelope inconsistent between two of the endpoints.) 1/4: 192.7s partial (Error envelope inconsistent between two of the endpoints.) 1/5: 127.1s pass 1/6: 127.9s fail (No durable queue available on the platform. Scheduled work did not survive a redeploy.) 1/7: 59.1s pass 1/8: 40s partial (Required one clarifying instruction before the criterion was met.) 1/9: 61.8s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/1: 96.4s pass 2/2: 74.2s pass 2/3: 113.9s pass 2/4: 75.4s partial (API keys issued but no rate limiting. The 60 request per minute ceiling was never enforced.) 2/5: 192.5s pass 2/6: 133.2s fail (No durable queue available on the platform. Scheduled work did not survive a redeploy.) 2/7: 68.4s partial (Validation added on the write path only. Read path left unguarded.) 2/8: 86.9s fail (Deployed URL returned 500 on every request.) 2/9: 105.6s pass 3/1: 62.6s pass 3/2: 72.9s pass 3/3: 48.9s pass 3/4: 113.4s pass 3/5: 217.8s partial (Pagination off by one at the final page boundary.) 3/6: 240.3s fail (No durable queue available on the platform. Scheduled work did not survive a redeploy.) 3/7: 146.6s pass 3/8: 121s fail (Task abandoned by the agent with no artifact produced.) 3/9: 219.1s pass 4/1: 51.3s pass 4/2: 120.1s pass 4/3: 105.4s fail (Task abandoned by the agent with no artifact produced.) 4/4: 191s partial (API keys issued but no rate limiting. The 60 request per minute ceiling was never enforced.) 4/5: 139.9s pass 4/6: 258.7s fail (No durable queue available on the platform. Scheduled work did not survive a redeploy.) 4/7: 71.4s pass 4/8: 182.6s pass 4/9: 88.6s pass 5/1: 161.6s pass 5/2: 137.5s pass 5/3: 48.5s pass 5/4: 193.4s pass 5/5: 139.4s pass 5/6: 168.7s fail (No durable queue available on the platform. Scheduled work did not survive a redeploy.) 5/7: 71.8s pass 5/8: 61.8s pass 5/9: 62.8s pass ### bolt.diy URL: https://www.llm-tier.com/builders/bolt-diy Official site: https://github.com/stackblitz-labs/bolt.diy Board: Full-stack app builders (fullstack-builder) Rank: 17 of 20 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 59.85 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 60, Reliability 55.6, Scalability 52, SEO and GEO 62, API and MCP 48, Integrations 54, Design output 74, Turn latency score 57.3, Value 82, Code ownership 97 Turn latency seconds, p10 / median / p90: 59.6 / 113.8 / 231.5 Full specification minutes, derived (9x median, not separately measured): 17.1 Prompt attempts completed: 35, failed: 10, strict pass rate: 55.6 percent Shipped output: HTML with JavaScript disabled 15360 bytes, largest contentful paint 2320 ms, cumulative layout shift 0.08 Pricing: No licence fee, bring your own model key. Source: https://github.com/stackblitz-labs/bolt.diy, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The community built sibling of a commercial product, and the gap between them is the point of measuring it: same interaction model, noticeably less polish in the agent loop. It has one of the widest p10 to p90 spreads in the index, because a run that picks up the wrong model configuration behaves nothing like one that does not. Worth the entry for anyone who needs the whole stack under their own control. Raw runs (run, prompt, seconds, outcome): 1/1: 82.8s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/2: 98.4s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 1/3: 211.5s pass 1/4: 75.5s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 1/5: 40.3s partial (Webhook delivered unsigned. The retry ladder was written only after an explicit follow up.) 1/6: 230.3s pass 1/7: 161.8s partial (Required one clarifying instruction before the criterion was met.) 1/8: 127.6s pass 1/9: 112.9s pass 2/1: 65.4s fail (Run picked up a model configuration that could not hold the whole prompt. Abandoned after three failed edits.) 2/2: 57.4s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/3: 129.7s partial (Pagination off by one at the final page boundary.) 2/4: 174.2s pass 2/5: 52s partial (Error envelope inconsistent between two of the endpoints.) 2/6: 65.6s fail (Deployed URL returned 500 on every request.) 2/7: 149.8s pass 2/8: 113.8s partial (Error envelope inconsistent between two of the endpoints.) 2/9: 64s pass 3/1: 56.7s pass 3/2: 60.4s pass 3/3: 129.9s pass 3/4: 220s partial (Pagination off by one at the final page boundary.) 3/5: 221.5s pass 3/6: 232.3s pass 3/7: 119.5s pass 3/8: 67.3s partial (Error envelope inconsistent between two of the endpoints.) 3/9: 59s pass 4/1: 91s fail (Run picked up a model configuration that could not hold the whole prompt. Abandoned after three failed edits.) 4/2: 266.6s pass 4/3: 159.2s pass 4/4: 160.2s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 4/5: 162.8s partial (Validation added on the write path only. Read path left unguarded.) 4/6: 180.3s pass 4/7: 193.3s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 4/8: 71.5s pass 4/9: 100.2s pass 5/1: 69.5s fail (Deployed URL returned 500 on every request.) 5/2: 82.6s pass 5/3: 98.4s pass 5/4: 260.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 5/5: 168.5s partial (Webhook delivered unsigned. The retry ladder was written only after an explicit follow up.) 5/6: 362.5s pass 5/7: 284.8s pass 5/8: 95.3s pass 5/9: 83.5s pass ### YouWare URL: https://www.llm-tier.com/builders/youware Official site: https://youware.com Board: Full-stack app builders (fullstack-builder) Rank: 18 of 20 on this board (change since last run: 3). Not comparable to the other board's rank. Composite index: 58.53 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 63, Reliability 48.9, Scalability 54, SEO and GEO 61, API and MCP 44, Integrations 52, Design output 82, Turn latency score 81.5, Value n/a, Code ownership 52 Turn latency seconds, p10 / median / p90: 46.3 / 80 / 143 Full specification minutes, derived (9x median, not separately measured): 12 Prompt attempts completed: 31, failed: 14, strict pass rate: 48.9 percent Shipped output: HTML with JavaScript disabled 14336 bytes, largest contentful paint 2120 ms, cumulative layout shift 0.07 Pricing: Not published by the vendor. Source: https://youware.com/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): not comparable, no vendor price published Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Fast and pleasant for anything with a front end and a light data layer. The failure pattern is consistent rather than random: every run produced a working product and then stopped short on the operational prompts, so the reliability figure is dragged down by the same three criteria each time. Raw runs (run, prompt, seconds, outcome): 1/1: 40.4s pass 1/2: 150.5s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/3: 60s fail (Generated migration conflicted with the existing schema and was not reconciled.) 1/4: 140.2s partial (API keys issued without rotation or rate limiting. The published ceiling was never enforced.) 1/5: 175.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/6: 100.6s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 1/7: 90.8s pass 1/8: 62.5s pass 1/9: 85.6s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/1: 53s pass 2/2: 77.6s pass 2/3: 114.9s pass 2/4: 335.5s partial (API keys issued without rotation or rate limiting. The published ceiling was never enforced.) 2/5: 132.9s pass 2/6: 114.2s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 2/7: 112.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/8: 79.5s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/9: 36.2s pass 3/1: 58.7s pass 3/2: 118.2s fail (Deployed URL returned 500 on every request.) 3/3: 67.5s pass 3/4: 50.3s partial (API keys issued without rotation or rate limiting. The published ceiling was never enforced.) 3/5: 48.6s pass 3/6: 105.5s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 3/7: 292.8s fail (Deployed URL returned 500 on every request.) 3/8: 54.3s pass 3/9: 55.1s pass 4/1: 79.6s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/2: 80s pass 4/3: 144.9s pass 4/4: 44.7s partial (API keys issued without rotation or rate limiting. The published ceiling was never enforced.) 4/5: 60.2s pass 4/6: 100.1s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 4/7: 104.1s pass 4/8: 39.9s pass 4/9: 52.1s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/1: 31.4s pass 5/2: 51.4s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/3: 82.6s partial (Validation added on the write path only. Read path left unguarded.) 5/4: 93.1s partial (API keys issued without rotation or rate limiting. The published ceiling was never enforced.) 5/5: 104.6s pass 5/6: 62s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 5/7: 72s pass 5/8: 90.5s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/9: 89.2s pass ### Trickle URL: https://www.llm-tier.com/builders/trickle Official site: https://trickle.so Board: Full-stack app builders (fullstack-builder) Rank: 19 of 20 on this board (change since last run: 4). Not comparable to the other board's rank. Composite index: 58.11 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 62, Reliability 51.1, Scalability 52, SEO and GEO 63, API and MCP 41, Integrations 49, Design output 79, Turn latency score 74.9, Value 74, Code ownership 45 Turn latency seconds, p10 / median / p90: 49.4 / 87 / 137.1 Full specification minutes, derived (9x median, not separately measured): 13.1 Prompt attempts completed: 34, failed: 11, strict pass rate: 51.1 percent Shipped output: HTML with JavaScript disabled 15360 bytes, largest contentful paint 2260 ms, cumulative layout shift 0.07 Pricing: USD 17 / month. Source: https://trickle.so/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 14.53 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The best public page output in the lower half, with real server rendered text and clean metadata, which lifts its SEO subscore well above its neighbours. Everything behind the marketing surface is thin: no key management, no signed webhooks and no durable background work. Raw runs (run, prompt, seconds, outcome): 1/1: 78.6s pass 1/2: 131.4s partial (Validation added on the write path only. Read path left unguarded.) 1/3: 107.1s pass 1/4: 109.2s fail (Task abandoned by the agent with no artifact produced.) 1/5: 96.4s pass 1/6: 60.8s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 1/7: 90.4s pass 1/8: 39s pass 1/9: 114.6s fail (Task abandoned by the agent with no artifact produced.) 2/1: 87s pass 2/2: 85.8s pass 2/3: 46.4s pass 2/4: 133.3s partial (Pagination off by one at the final page boundary.) 2/5: 106.7s partial (Validation added on the write path only. Read path left unguarded.) 2/6: 79.2s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 2/7: 85.1s pass 2/8: 34s pass 2/9: 73.1s fail (Task abandoned by the agent with no artifact produced.) 3/1: 87.1s partial (Validation added on the write path only. Read path left unguarded.) 3/2: 77.7s pass 3/3: 79.3s partial (Required one clarifying instruction before the criterion was met.) 3/4: 61.7s partial (Error envelope inconsistent between two of the endpoints.) 3/5: 119.4s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/6: 122s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 3/7: 90.1s pass 3/8: 80.2s pass 3/9: 73.8s fail (Task abandoned by the agent with no artifact produced.) 4/1: 32.6s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/2: 93.1s pass 4/3: 53.9s pass 4/4: 151.7s pass 4/5: 136.4s pass 4/6: 74.3s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 4/7: 106.8s pass 4/8: 69.1s pass 4/9: 137.5s partial (Error envelope inconsistent between two of the endpoints.) 5/1: 64s partial (Required one clarifying instruction before the criterion was met.) 5/2: 56.4s pass 5/3: 152.3s partial (Pagination off by one at the final page boundary.) 5/4: 38s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/5: 204.4s pass 5/6: 117s fail (Background jobs are out of scope for the platform. Nothing queued, nothing scheduled.) 5/7: 204s pass 5/8: 81s partial (Error envelope inconsistent between two of the endpoints.) 5/9: 104.2s pass ### CatDoes URL: https://www.llm-tier.com/builders/catdoes Official site: https://catdoes.com Board: Full-stack app builders (fullstack-builder) Rank: 20 of 20 on this board (change since last run: 4). Not comparable to the other board's rank. Composite index: 56.99 out of 100, this board's own weighted sum Subscores 0 to 100 (10 axes on this board): Agent performance 66, Reliability 48.9, Scalability 58, SEO and GEO 42, API and MCP 48, Integrations 60, Design output 80, Turn latency score 47.6, Value 74, Code ownership 66 Turn latency seconds, p10 / median / p90: 72.2 / 137.1 / 249 Full specification minutes, derived (9x median, not separately measured): 20.6 Prompt attempts completed: 32, failed: 13, strict pass rate: 48.9 percent Shipped output: HTML with JavaScript disabled 8192 bytes, largest contentful paint 2460 ms, cumulative layout shift 0.09 Pricing: USD 17 / month. Source: https://catdoes.com/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 14.53 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Aimed at mobile applications rather than web, which is why it posts one of the smallest no JavaScript payloads in the index and the weakest SEO subscore: there is barely a crawlable surface to measure. Judged on the parts of the spec it can reach it holds up, and the generated interface is genuinely good. The webhook and background job prompts were outside what the platform exposes. Raw runs (run, prompt, seconds, outcome): 1/1: 59.2s pass 1/2: 142.6s partial (Error envelope inconsistent between two of the endpoints.) 1/3: 129.6s pass 1/4: 168.8s pass 1/5: 152.5s fail (Outbound webhooks are not part of the platform surface. Nothing was delivered and nothing was signed.) 1/6: 236.5s fail (No durable queue on the platform. Scheduled work is limited to push notifications.) 1/7: 243.2s partial (Validation added on the write path only. Read path left unguarded.) 1/8: 84.8s partial (Error envelope inconsistent between two of the endpoints.) 1/9: 158.3s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/1: 122.6s pass 2/2: 68.5s partial (Error envelope inconsistent between two of the endpoints.) 2/3: 93.5s pass 2/4: 163.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/5: 106.9s fail (Outbound webhooks are not part of the platform surface. Nothing was delivered and nothing was signed.) 2/6: 252.9s fail (No durable queue on the platform. Scheduled work is limited to push notifications.) 2/7: 106.5s partial (Error envelope inconsistent between two of the endpoints.) 2/8: 258.4s pass 2/9: 48.5s pass 3/1: 197.1s pass 3/2: 88.9s pass 3/3: 137.1s pass 3/4: 190.1s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 3/5: 109.5s fail (Outbound webhooks are not part of the platform surface. Nothing was delivered and nothing was signed.) 3/6: 188.3s fail (No durable queue on the platform. Scheduled work is limited to push notifications.) 3/7: 128.1s pass 3/8: 62.1s pass 3/9: 106.1s pass 4/1: 84.9s partial (Required one clarifying instruction before the criterion was met.) 4/2: 103.3s pass 4/3: 138.2s pass 4/4: 34s partial (Validation added on the write path only. Read path left unguarded.) 4/5: 100.3s fail (Outbound webhooks are not part of the platform surface. Nothing was delivered and nothing was signed.) 4/6: 271.6s fail (No durable queue on the platform. Scheduled work is limited to push notifications.) 4/7: 292.3s pass 4/8: 77.7s partial (Required one clarifying instruction before the criterion was met.) 4/9: 97.9s pass 5/1: 102.7s pass 5/2: 132.9s partial (Required one clarifying instruction before the criterion was met.) 5/3: 207.8s pass 5/4: 146.2s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 5/5: 189.6s fail (Outbound webhooks are not part of the platform surface. Nothing was delivered and nothing was signed.) 5/6: 168.1s fail (No durable queue on the platform. Scheduled work is limited to push notifications.) 5/7: 138.8s pass 5/8: 256.9s pass 5/9: 214.4s pass ### Claude Code URL: https://www.llm-tier.com/builders/claude-code Official site: https://claude.com Board: Coding agents (coding-agent) Rank: 1 of 17 on this board (change since last run: -1). Not comparable to the other board's rank. Composite index: 79.72 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 95, Reliability 91.1, Scalability 86, SEO and GEO 62, API and MCP 78, Integrations 76, Design output 68, Turn latency score 47.9, Value 80 Turn latency seconds, p10 / median / p90: 39.9 / 65.5 / 100.7 Full specification minutes, derived (9x median, not separately measured): 9.8 Prompt attempts completed: 44, failed: 1, strict pass rate: 91.1 percent Shipped output: HTML with JavaScript disabled 26624 bytes, largest contentful paint 1680 ms, cumulative layout shift 0.03 Pricing: USD 17 / month, billed annually. Source: https://claude.com/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 14.53 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The highest strict pass rate in the agent group, and third highest in the whole index. It was the only agent that produced a signed webhook with a retry ladder in prompt 5 in all five runs without the criterion being restated, and the only one that added an idempotency key to the write path unprompted. It is a terminal agent with no interface of its own, so design and SEO reflect the stack it was pointed at rather than anything it invented. Raw runs (run, prompt, seconds, outcome): 1/1: 46.6s pass 1/2: 23.3s fail (Deployed URL returned 500 on every request.) 1/3: 87.2s pass 1/4: 68.5s pass 1/5: 70s pass (Signed payload with a retry ladder and a delivery log on the first attempt, criterion met without a follow up.) 1/6: 57.8s pass 1/7: 103.3s pass 1/8: 68s pass 1/9: 72.7s pass 2/1: 36s partial (Validation added on the write path only. Read path left unguarded.) 2/2: 89.9s pass 2/3: 93.1s pass 2/4: 111s pass 2/5: 98.2s pass (Signed payload with a retry ladder and a delivery log on the first attempt, criterion met without a follow up.) 2/6: 75.9s pass 2/7: 50.4s pass 2/8: 32.7s pass 2/9: 58.4s pass 3/1: 47.5s pass 3/2: 52.9s pass 3/3: 65.6s pass 3/4: 102.4s pass 3/5: 76.6s pass (Signed payload with a retry ladder and a delivery log on the first attempt, criterion met without a follow up.) 3/6: 61s pass 3/7: 53.9s pass 3/8: 40.7s pass 3/9: 53.5s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/1: 59.5s pass 4/2: 55.5s pass 4/3: 74.8s partial (Pagination off by one at the final page boundary.) 4/4: 76.8s pass 4/5: 176.3s pass (Signed payload with a retry ladder and a delivery log on the first attempt, criterion met without a follow up.) 4/6: 43.2s pass 4/7: 90.6s pass 4/8: 76s pass 4/9: 65s pass 5/1: 21.8s pass 5/2: 95.5s pass 5/3: 65.5s pass 5/4: 58.3s pass 5/5: 88.4s pass (Signed payload with a retry ladder and a delivery log on the first attempt, criterion met without a follow up.) 5/6: 105s pass 5/7: 48.6s pass 5/8: 39.3s pass 5/9: 46.7s pass ### Cursor URL: https://www.llm-tier.com/builders/cursor Official site: https://cursor.com Board: Coding agents (coding-agent) Rank: 2 of 17 on this board (change since last run: -1). Not comparable to the other board's rank. Composite index: 77.92 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 92, Reliability 86.7, Scalability 82, SEO and GEO 58, API and MCP 74, Integrations 80, Design output 66, Turn latency score 58.8, Value 80 Turn latency seconds, p10 / median / p90: 28.7 / 53.4 / 82.7 Full specification minutes, derived (9x median, not separately measured): 8 Prompt attempts completed: 42, failed: 3, strict pass rate: 86.7 percent Shipped output: HTML with JavaScript disabled 24576 bytes, largest contentful paint 1720 ms, cumulative layout shift 0.03 Pricing: USD 20 / month. Source: https://cursor.com/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The strongest edit loop we measured. It keeps a large working set in view, so the tenancy constraint from prompt 3 survived every later change without being restated. As a coding agent it produces no interface of its own, which caps the design subscore: what you get is exactly the quality of the framework and components you point it at. Everything it writes lands in your repository, so ownership is total. Raw runs (run, prompt, seconds, outcome): 1/1: 18s pass 1/2: 41.7s pass 1/3: 40.8s pass (Tenancy constraint from prompt 2 was still enforced after the refactor, with no restatement of the rule.) 1/4: 84.7s pass 1/5: 62.5s pass 1/6: 72.6s pass 1/7: 65.1s pass 1/8: 80.4s pass 1/9: 51.9s pass 2/1: 42.1s pass 2/2: 71.4s pass 2/3: 53.4s pass (Tenancy constraint from prompt 2 was still enforced after the refactor, with no restatement of the rule.) 2/4: 46.2s pass 2/5: 65.7s pass 2/6: 65.1s pass 2/7: 51.6s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/8: 84.2s pass 2/9: 43.8s pass 3/1: 19s pass 3/2: 57.8s pass 3/3: 62.5s pass (Tenancy constraint from prompt 2 was still enforced after the refactor, with no restatement of the rule.) 3/4: 109.7s pass 3/5: 61.4s pass 3/6: 48s pass 3/7: 79.8s partial (Required one clarifying instruction before the criterion was met.) 3/8: 70.4s pass 3/9: 51s pass 4/1: 23s pass 4/2: 117.2s fail (Deployed URL returned 500 on every request.) 4/3: 49.7s pass (Tenancy constraint from prompt 2 was still enforced after the refactor, with no restatement of the rule.) 4/4: 144.7s pass 4/5: 76.8s pass 4/6: 62.9s pass 4/7: 33.5s pass 4/8: 51.9s pass 4/9: 24.1s partial (Required one clarifying instruction before the criterion was met.) 5/1: 37s pass 5/2: 33.8s pass 5/3: 31.4s pass (Tenancy constraint from prompt 2 was still enforced after the refactor, with no restatement of the rule.) 5/4: 42.4s pass 5/5: 53.4s partial (Validation added on the write path only. Read path left unguarded.) 5/6: 64.2s pass 5/7: 76.4s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 5/8: 26.9s pass 5/9: 41.2s pass ### Kiro URL: https://www.llm-tier.com/builders/kiro Official site: https://kiro.dev Board: Coding agents (coding-agent) Rank: 3 of 17 on this board (change since last run: 0). Not comparable to the other board's rank. Composite index: 75.67 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 84, Reliability 93.3, Scalability 80, SEO and GEO 60, API and MCP 72, Integrations 74, Design output 65, Turn latency score 43.4, Value 78 Turn latency seconds, p10 / median / p90: 49.1 / 72.4 / 140.4 Full specification minutes, derived (9x median, not separately measured): 10.9 Prompt attempts completed: 44, failed: 1, strict pass rate: 93.3 percent Shipped output: HTML with JavaScript disabled 22528 bytes, largest contentful paint 1800 ms, cumulative layout shift 0.04 Pricing: USD 20 / user / month. Source: https://kiro.dev/pricing/, read 2026-08-21. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Writes a specification and a task list before it writes code, which costs it roughly a third more wall clock than the fastest agents and buys back one of the tightest duration spreads in the index. The specification step is why prompt 2 came out correct in five of five runs: the tenancy rule was written down before anything depended on it. Raw runs (run, prompt, seconds, outcome): 1/1: 80.4s pass 1/2: 63.7s pass (Specification step wrote the tenancy rule down before any code depended on it. Schema correct on the first pass.) 1/3: 77.7s pass 1/4: 62.1s pass 1/5: 111.3s pass 1/6: 135.4s pass 1/7: 72.4s pass 1/8: 52.6s pass 1/9: 88.8s pass 2/1: 58.8s pass 2/2: 68.1s pass (Specification step wrote the tenancy rule down before any code depended on it. Schema correct on the first pass.) 2/3: 90s pass 2/4: 109.5s pass 2/5: 143.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/6: 132.9s pass 2/7: 48.7s pass 2/8: 64.4s pass 2/9: 205.9s pass 3/1: 49.7s partial (Validation added on the write path only. Read path left unguarded.) 3/2: 54.7s pass (Specification step wrote the tenancy rule down before any code depended on it. Schema correct on the first pass.) 3/3: 41.9s pass 3/4: 93s pass 3/5: 144.6s pass 3/6: 197.4s pass 3/7: 66.1s pass 3/8: 45.3s pass 3/9: 47.2s pass 4/1: 54.8s pass 4/2: 26.5s pass (Specification step wrote the tenancy rule down before any code depended on it. Schema correct on the first pass.) 4/3: 70.6s pass 4/4: 67.9s pass 4/5: 127.7s pass 4/6: 73.2s pass 4/7: 114.8s pass 4/8: 49.6s pass 4/9: 80.8s pass 5/1: 71.3s pass 5/2: 56.8s pass (Specification step wrote the tenancy rule down before any code depended on it. Schema correct on the first pass.) 5/3: 79s pass 5/4: 132.8s pass 5/5: 79s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 5/6: 254.4s pass 5/7: 61.4s pass 5/8: 57.7s pass 5/9: 121.6s pass ### Augment Code URL: https://www.llm-tier.com/builders/augment-code Official site: https://augmentcode.com Board: Coding agents (coding-agent) Rank: 4 of 17 on this board (change since last run: 0). Not comparable to the other board's rank. Composite index: 75.63 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 86, Reliability 88.9, Scalability 84, SEO and GEO 58, API and MCP 74, Integrations 76, Design output 62, Turn latency score 43.9, Value 77 Turn latency seconds, p10 / median / p90: 44.9 / 71.6 / 110 Full specification minutes, derived (9x median, not separately measured): 10.7 Prompt attempts completed: 43, failed: 2, strict pass rate: 88.9 percent Shipped output: HTML with JavaScript disabled 22528 bytes, largest contentful paint 1800 ms, cumulative layout shift 0.04 Pricing: USD 20 / month, flat for a team up to 50 seats. Source: https://augmentcode.com/pricing, read 2026-09-05. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Indexes a whole repository before it acts, and on a large codebase that is worth the wait: it found the existing tenancy helper in prompt 3 and reused it rather than writing a second one, which no other agent did in every run. Pricing is flat for a team rather than per seat, so a single developer pays the same USD 20 as a team of fifty, which is what the value subscore reflects. On a small project the indexing advantage largely disappears. Raw runs (run, prompt, seconds, outcome): 1/1: 97s pass 1/2: 99.2s pass 1/3: 134.7s pass (Reused the existing tenancy helper instead of writing a second one. No duplicate policy code in the diff.) 1/4: 83.8s pass 1/5: 65.1s pass 1/6: 70.8s fail (Deployed URL returned 500 on every request.) 1/7: 89.5s pass 1/8: 82.4s partial (Error envelope inconsistent between two of the endpoints.) 1/9: 46s pass 2/1: 34.3s pass 2/2: 71.6s pass 2/3: 87.9s pass (Reused the existing tenancy helper instead of writing a second one. No duplicate policy code in the diff.) 2/4: 58.6s pass 2/5: 48.8s pass 2/6: 51.1s pass 2/7: 135.1s pass 2/8: 51.2s pass 2/9: 59.2s pass 3/1: 46.7s pass 3/2: 103.6s pass 3/3: 112.8s pass (Reused the existing tenancy helper instead of writing a second one. No duplicate policy code in the diff.) 3/4: 49.9s pass 3/5: 44.3s partial (Pagination off by one at the final page boundary.) 3/6: 82.9s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 3/7: 48.8s pass 3/8: 72.3s pass 3/9: 84.1s pass 4/1: 36.1s pass 4/2: 57.6s pass 4/3: 51s pass (Reused the existing tenancy helper instead of writing a second one. No duplicate policy code in the diff.) 4/4: 86.6s pass 4/5: 67.1s fail (Task abandoned by the agent with no artifact produced.) 4/6: 84.5s pass 4/7: 157.4s pass 4/8: 45.8s pass 4/9: 112s pass 5/1: 106.9s pass 5/2: 68.4s pass 5/3: 74s pass (Reused the existing tenancy helper instead of writing a second one. No duplicate policy code in the diff.) 5/4: 80.6s pass 5/5: 33.8s pass 5/6: 83.4s pass 5/7: 103s pass 5/8: 34.8s pass 5/9: 49.3s pass ### Amp URL: https://www.llm-tier.com/builders/amp Official site: https://ampcode.com Board: Coding agents (coding-agent) Rank: 5 of 17 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 73.35 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 85, Reliability 77.8, Scalability 74, SEO and GEO 56, API and MCP 72, Integrations 68, Design output 60, Turn latency score 79.1, Value 77 Turn latency seconds, p10 / median / p90: 24.2 / 39.7 / 64.4 Full specification minutes, derived (9x median, not separately measured): 6 Prompt attempts completed: 41, failed: 4, strict pass rate: 77.8 percent Shipped output: HTML with JavaScript disabled 20480 bytes, largest contentful paint 1840 ms, cumulative layout shift 0.04 Pricing: USD 20 / month. Source: https://ampcode.com/pricing, read 2026-08-28. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: One of the three fastest agents in the index and the one that wasted the least time on work nobody asked for. It runs at whatever model tier it judges the task needs, which is why the duration spread is tighter than the raw speed suggests. It has no interface of its own and does not pretend otherwise, so design and SEO track the stack rather than the tool. Raw runs (run, prompt, seconds, outcome): 1/1: 68s pass 1/2: 33.5s pass 1/3: 64.6s pass 1/4: 24.3s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 1/5: 37.9s pass 1/6: 45.2s pass 1/7: 60.6s pass 1/8: 50.3s pass 1/9: 62s pass 2/1: 39.7s pass 2/2: 71.9s pass 2/3: 39.6s pass 2/4: 69.6s pass 2/5: 43.7s pass 2/6: 64.2s partial (Error envelope inconsistent between two of the endpoints.) 2/7: 31.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/8: 25.8s pass 2/9: 25.6s partial (Validation added on the write path only. Read path left unguarded.) 3/1: 40.4s pass 3/2: 53.8s pass 3/3: 26.3s pass 3/4: 61.9s pass 3/5: 30.7s pass 3/6: 47.4s partial (Validation added on the write path only. Read path left unguarded.) 3/7: 48.6s pass 3/8: 26.1s pass 3/9: 27.9s pass 4/1: 32.1s pass 4/2: 43.5s pass 4/3: 29.7s fail (Task abandoned by the agent with no artifact produced.) 4/4: 28.5s pass 4/5: 52.5s pass 4/6: 39s partial (Error envelope inconsistent between two of the endpoints.) 4/7: 42.9s pass 4/8: 55.1s pass 4/9: 18s pass 5/1: 21.5s fail (Task abandoned by the agent with no artifact produced.) 5/2: 24.1s pass 5/3: 44.7s pass 5/4: 53.7s partial (Required one clarifying instruction before the criterion was met.) 5/5: 32.2s pass 5/6: 65.3s pass 5/7: 18.8s pass 5/8: 19.5s pass 5/9: 38.8s fail (Task abandoned by the agent with no artifact produced.) ### Factory URL: https://www.llm-tier.com/builders/factory Official site: https://factory.ai Board: Coding agents (coding-agent) Rank: 6 of 17 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 71.48 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 81, Reliability 82.2, Scalability 82, SEO and GEO 56, API and MCP 76, Integrations 78, Design output 62, Turn latency score 19.4, Value 76 Turn latency seconds, p10 / median / p90: 84.8 / 161.7 / 350.8 Full specification minutes, derived (9x median, not separately measured): 24.3 Prompt attempts completed: 42, failed: 3, strict pass rate: 82.2 percent Shipped output: HTML with JavaScript disabled 19456 bytes, largest contentful paint 1940 ms, cumulative layout shift 0.05 Pricing: USD 20 / month. Source: https://factory.ai/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Built for teams rather than individuals, and the difference is visible in the operational prompts: it attached the change to a ticket, wrote a migration plan and produced a rollback path without being asked. It is slow and it is priced for organisations, so an individual will find better value elsewhere in this group. The API subscore is the second strongest in the agent group. Raw runs (run, prompt, seconds, outcome): 1/1: 102.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/2: 175.8s pass 1/3: 132.5s pass 1/4: 204.5s pass 1/5: 148.2s pass 1/6: 184s pass 1/7: 151.4s pass 1/8: 155.1s pass 1/9: 158s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/1: 148.8s pass 2/2: 97.7s pass 2/3: 203.4s pass 2/4: 401.5s pass 2/5: 320.8s pass 2/6: 345.5s pass 2/7: 116.9s pass 2/8: 74s pass 2/9: 202.3s pass 3/1: 156.6s pass 3/2: 336.6s partial (Required one clarifying instruction before the criterion was met.) 3/3: 116.3s pass 3/4: 190.1s pass 3/5: 266.7s fail (Task abandoned by the agent with no artifact produced.) 3/6: 145.7s pass 3/7: 344.2s pass 3/8: 55.5s pass 3/9: 43.9s pass 4/1: 390.1s pass 4/2: 161.7s pass 4/3: 76.2s pass 4/4: 178.8s pass 4/5: 492.8s pass 4/6: 448.5s pass 4/7: 292.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/8: 100.3s pass 4/9: 161.7s partial (Validation added on the write path only. Read path left unguarded.) 5/1: 207.7s pass 5/2: 69.3s pass 5/3: 351.8s pass 5/4: 349.4s pass 5/5: 158s pass 5/6: 146.2s pass 5/7: 193.9s partial (Required one clarifying instruction before the criterion was met.) 5/8: 152.4s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/9: 230.3s pass ### Antigravity URL: https://www.llm-tier.com/builders/antigravity Official site: https://antigravity.google Board: Coding agents (coding-agent) Rank: 7 of 17 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 71.37 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 83, Reliability 73.3, Scalability 76, SEO and GEO 58, API and MCP 70, Integrations 72, Design output 70, Turn latency score 39.9, Value 90 Turn latency seconds, p10 / median / p90: 45 / 78.7 / 123.1 Full specification minutes, derived (9x median, not separately measured): 11.8 Prompt attempts completed: 40, failed: 5, strict pass rate: 73.3 percent Shipped output: HTML with JavaScript disabled 22528 bytes, largest contentful paint 1820 ms, cumulative layout shift 0.04 Pricing: No cost. Source: https://antigravity.google/pricing, read 2026-09-06. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Plans across several files at once and reports what it intends to do before doing it, which makes a failed run cheap to diagnose. It is the best value entry in the agent group at present because the licence costs nothing, and that will change. Prompt 6 was the weak point: durable work was queued correctly in three of five runs and left as a timer in the other two. Raw runs (run, prompt, seconds, outcome): 1/1: 61.3s fail (Deployed URL returned 500 on every request.) 1/2: 25.6s pass 1/3: 136.8s pass 1/4: 64.1s fail (Deployed URL returned 500 on every request.) 1/5: 99.5s pass 1/6: 90.6s pass 1/7: 110.9s fail (Task abandoned by the agent with no artifact produced.) 1/8: 84.1s fail (Task abandoned by the agent with no artifact produced.) 1/9: 30.5s pass 2/1: 120.5s pass 2/2: 66.1s pass 2/3: 78.7s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 2/4: 189.2s pass 2/5: 129.9s pass 2/6: 100.8s pass 2/7: 43.9s pass 2/8: 46.6s pass 2/9: 124.9s pass 3/1: 68.8s pass 3/2: 86.9s pass 3/3: 104.2s partial (Error envelope inconsistent between two of the endpoints.) 3/4: 88.6s pass 3/5: 149.6s pass 3/6: 75.8s partial (Durable work left as a timer rather than a queue. Scheduled work did not survive a redeploy.) 3/7: 87.9s pass 3/8: 60.4s pass 3/9: 54.1s pass 4/1: 115.9s pass 4/2: 69.4s partial (Required one clarifying instruction before the criterion was met.) 4/3: 61s pass 4/4: 101.3s pass 4/5: 90.3s pass 4/6: 120.2s pass 4/7: 111.4s pass 4/8: 85.4s pass 4/9: 46.7s partial (Error envelope inconsistent between two of the endpoints.) 5/1: 28.9s pass 5/2: 73.9s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 5/3: 60.9s pass 5/4: 66.3s pass 5/5: 70.3s pass 5/6: 77.9s partial (Durable work left as a timer rather than a queue. Scheduled work did not survive a redeploy.) 5/7: 73.2s partial (Validation added on the write path only. Read path left unguarded.) 5/8: 79.2s pass 5/9: 43.9s pass ### Zed URL: https://www.llm-tier.com/builders/zed Official site: https://zed.dev Board: Coding agents (coding-agent) Rank: 8 of 17 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 71.13 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 77, Reliability 71.1, Scalability 70, SEO and GEO 55, API and MCP 66, Integrations 64, Design output 64, Turn latency score 100, Value 82 Turn latency seconds, p10 / median / p90: 19.8 / 31.4 / 54.5 Full specification minutes, derived (9x median, not separately measured): 4.7 Prompt attempts completed: 37, failed: 8, strict pass rate: 71.1 percent Shipped output: HTML with JavaScript disabled 19456 bytes, largest contentful paint 1780 ms, cumulative layout shift 0.04 Pricing: USD 10 / month. Source: https://zed.dev/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 8.55 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The fastest median in the index, and the reason is architectural rather than clever prompting: the editor is fast, the diff review is instant and almost no time is lost to interface latency. Correctness sits in the middle of the agent group, so the right read is that it costs the least time per attempt rather than that it is the best at any single attempt. Raw runs (run, prompt, seconds, outcome): 1/1: 23.6s pass 1/2: 47.4s fail (Deployed URL returned 500 on every request.) 1/3: 58s pass 1/4: 53.5s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 1/5: 31.4s partial (Required one clarifying instruction before the criterion was met.) 1/6: 88.5s pass 1/7: 23.2s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/8: 27.4s pass 1/9: 35s pass 2/1: 21.3s pass 2/2: 35.3s pass 2/3: 36.1s pass 2/4: 55.1s pass 2/5: 30.9s pass 2/6: 26.5s pass 2/7: 26.8s pass 2/8: 47.4s pass 2/9: 24.8s pass 3/1: 18.3s pass 3/2: 24.6s pass 3/3: 26.8s pass 3/4: 52.5s pass 3/5: 50.6s fail (Task abandoned by the agent with no artifact produced.) 3/6: 48.2s pass 3/7: 23.6s pass 3/8: 47s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 3/9: 26.9s pass 4/1: 21.4s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/2: 18.3s pass 4/3: 18.4s partial (Pagination off by one at the final page boundary.) 4/4: 152.9s pass 4/5: 40.6s partial (Error envelope inconsistent between two of the endpoints.) 4/6: 59.4s pass 4/7: 23.3s pass 4/8: 18.8s pass 4/9: 37.8s fail (Deployed URL returned 500 on every request.) 5/1: 22.9s pass 5/2: 18s fail (Deployed URL returned 500 on every request.) 5/3: 25.4s pass 5/4: 34.4s pass 5/5: 21.5s pass 5/6: 38.9s fail (Deployed URL returned 500 on every request.) 5/7: 36.9s pass 5/8: 31.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/9: 40.3s pass ### Cline URL: https://www.llm-tier.com/builders/cline Official site: https://cline.bot Board: Coding agents (coding-agent) Rank: 9 of 17 on this board (change since last run: 1). Not comparable to the other board's rank. Composite index: 70.66 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 82, Reliability 80, Scalability 72, SEO and GEO 57, API and MCP 70, Integrations 72, Design output 63, Turn latency score 33, Value 88 Turn latency seconds, p10 / median / p90: 47.4 / 95.1 / 201.7 Full specification minutes, derived (9x median, not separately measured): 14.3 Prompt attempts completed: 44, failed: 1, strict pass rate: 80 percent Shipped output: HTML with JavaScript disabled 21504 bytes, largest contentful paint 1860 ms, cumulative layout shift 0.04 Pricing: No licence fee, bring your own model key. Source: https://cline.bot/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: An open source agent inside your editor that asks permission before each step, which is slower to drive and much easier to trust. It matched the commercial agents on the correctness prompts and lost time to the approval loop rather than to the model. Cost per full run was among the lowest we recorded because nothing is spent on a platform licence. Raw runs (run, prompt, seconds, outcome): 1/1: 86.8s pass 1/2: 56.8s pass 1/3: 128.8s pass 1/4: 30.7s pass 1/5: 88.9s pass 1/6: 153.2s partial (Pagination off by one at the final page boundary.) 1/7: 73.4s pass 1/8: 56.1s pass 1/9: 186s pass 2/1: 44.1s partial (Validation added on the write path only. Read path left unguarded.) 2/2: 66.7s pass 2/3: 136.5s pass 2/4: 73.8s pass 2/5: 139.9s pass 2/6: 208.8s pass 2/7: 165.1s pass 2/8: 175.9s pass 2/9: 126.7s pass 3/1: 49.1s partial (Pagination off by one at the final page boundary.) 3/2: 120.1s pass 3/3: 156.4s pass 3/4: 100.5s pass 3/5: 82.5s pass 3/6: 477s pass 3/7: 264.5s pass 3/8: 71.6s partial (Pagination off by one at the final page boundary.) 3/9: 64.3s pass 4/1: 27.8s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/2: 30.8s pass 4/3: 70.7s partial (Required one clarifying instruction before the criterion was met.) 4/4: 73s pass 4/5: 152.8s pass 4/6: 206.8s pass 4/7: 106.9s pass 4/8: 152.2s pass 4/9: 95.1s pass 5/1: 46.3s partial (Error envelope inconsistent between two of the endpoints.) 5/2: 57.6s pass 5/3: 194.1s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/4: 154.3s pass 5/5: 247.5s pass 5/6: 86.4s pass 5/7: 175.4s pass 5/8: 61.2s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 5/9: 74s pass ### Jules URL: https://www.llm-tier.com/builders/jules Official site: https://jules.google Board: Coding agents (coding-agent) Rank: 10 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 68.04 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 80, Reliability 75.6, Scalability 74, SEO and GEO 55, API and MCP 66, Integrations 70, Design output 62, Turn latency score 21.4, Value 88 Turn latency seconds, p10 / median / p90: 84.4 / 146.5 / 292.7 Full specification minutes, derived (9x median, not separately measured): 22 Prompt attempts completed: 41, failed: 4, strict pass rate: 75.6 percent Shipped output: HTML with JavaScript disabled 21504 bytes, largest contentful paint 1860 ms, cumulative layout shift 0.04 Pricing: No cost on the free tier. Source: https://jules.google/, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: An asynchronous agent: you hand it a task against a repository and come back to a proposed change. Judged on wall clock that is expensive, and it posts one of the slower medians in the index, but the comparison is not quite like for like since nobody sits and watches it. Change quality was good and the tests it added were real tests rather than assertions that always pass. Raw runs (run, prompt, seconds, outcome): 1/1: 134.6s pass 1/2: 126.9s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 1/3: 171.3s pass 1/4: 253.8s pass 1/5: 206.6s partial (Required one clarifying instruction before the criterion was met.) 1/6: 228.8s pass 1/7: 214.9s pass 1/8: 239.7s partial (Required one clarifying instruction before the criterion was met.) 1/9: 119.1s pass 2/1: 49.3s pass 2/2: 137.6s pass 2/3: 186.3s pass 2/4: 202.6s partial (Error envelope inconsistent between two of the endpoints.) 2/5: 506.9s pass 2/6: 114.3s pass 2/7: 203.2s pass 2/8: 140s pass 2/9: 98.7s partial (Validation added on the write path only. Read path left unguarded.) 3/1: 110.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/2: 96.5s partial (Error envelope inconsistent between two of the endpoints.) 3/3: 287.7s pass 3/4: 149.2s pass 3/5: 251.6s pass 3/6: 296s pass 3/7: 225.2s pass 3/8: 91.5s pass 3/9: 146.5s pass 4/1: 93s pass 4/2: 81.1s pass 4/3: 188.2s pass 4/4: 137.2s pass 4/5: 318.8s fail (Deployed URL returned 500 on every request.) 4/6: 278.8s pass 4/7: 645.8s pass 4/8: 101.1s pass 4/9: 173.8s pass 5/1: 116.1s pass 5/2: 120.5s pass 5/3: 395.9s pass 5/4: 102.9s pass 5/5: 211.5s pass 5/6: 68.1s pass 5/7: 75.5s partial (Required one clarifying instruction before the criterion was met.) 5/8: 53.4s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 5/9: 89.3s partial (Delivered the main path, missed the boundary case named in the pass criteria.) ### Devin URL: https://www.llm-tier.com/builders/devin Official site: https://devin.ai Board: Coding agents (coding-agent) Rank: 11 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 66.64 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 78, Reliability 71.1, Scalability 80, SEO and GEO 54, API and MCP 72, Integrations 74, Design output 60, Turn latency score 8.9, Value 74 Turn latency seconds, p10 / median / p90: 230.2 / 353.6 / 651.4 Full specification minutes, derived (9x median, not separately measured): 53 Prompt attempts completed: 39, failed: 6, strict pass rate: 71.1 percent Shipped output: HTML with JavaScript disabled 20480 bytes, largest contentful paint 1900 ms, cumulative layout shift 0.05 Pricing: USD 20 / month. Source: https://devin.ai/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 17.10 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The slowest median in the index by a wide margin and the most expensive route through the spec, because it works in its own environment and takes the long way through every task. What it buys is unattended completion: it opened a pull request with passing tests without a human in the loop in four of five runs. Judged on the published axes the cost weighs heavily against it, which is why the value subscore is the lowest in the agent group. Raw runs (run, prompt, seconds, outcome): 1/1: 119.8s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 1/2: 294.3s pass 1/3: 661.9s partial (Required one clarifying instruction before the criterion was met.) 1/4: 931.3s pass 1/5: 352.7s pass 1/6: 507.8s pass 1/7: 547.5s pass (Opened a pull request with passing tests and no human in the loop for the whole task.) 1/8: 349.6s pass 1/9: 496.3s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/1: 304.3s pass 2/2: 226.4s pass 2/3: 390.6s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/4: 462.7s pass 2/5: 110.9s pass 2/6: 241.3s pass 2/7: 353.6s pass (Opened a pull request with passing tests and no human in the loop for the whole task.) 2/8: 655.7s pass 2/9: 356s pass 3/1: 479.9s pass 3/2: 241.3s pass 3/3: 364.9s pass 3/4: 1293.4s partial (Error envelope inconsistent between two of the endpoints.) 3/5: 278.3s pass 3/6: 644.9s pass 3/7: 350.5s pass (Opened a pull request with passing tests and no human in the loop for the whole task.) 3/8: 372.8s partial (Required one clarifying instruction before the criterion was met.) 3/9: 301.8s pass 4/1: 151.9s partial (Validation added on the write path only. Read path left unguarded.) 4/2: 338.1s pass 4/3: 235.8s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/4: 238s pass 4/5: 377s pass 4/6: 606.5s pass 4/7: 514.1s partial (Required one clarifying instruction before the criterion was met.) 4/8: 289.3s pass 4/9: 480.2s pass 5/1: 359.7s pass 5/2: 263.8s pass 5/3: 246.2s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/4: 839.6s fail (Task abandoned by the agent with no artifact produced.) 5/5: 437.7s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/6: 409.9s partial (Required one clarifying instruction before the criterion was met.) 5/7: 321.4s pass (Opened a pull request with passing tests and no human in the loop for the whole task.) 5/8: 208.3s pass 5/9: 250.9s pass ### Qodo URL: https://www.llm-tier.com/builders/qodo Official site: https://qodo.ai Board: Coding agents (coding-agent) Rank: 12 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 66.06 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 70, Reliability 77.8, Scalability 76, SEO and GEO 52, API and MCP 68, Integrations 70, Design output 56, Turn latency score 31.6, Value 70 Turn latency seconds, p10 / median / p90: 66.3 / 99.3 / 189.8 Full specification minutes, derived (9x median, not separately measured): 14.9 Prompt attempts completed: 41, failed: 4, strict pass rate: 77.8 percent Shipped output: HTML with JavaScript disabled 16384 bytes, largest contentful paint 2000 ms, cumulative layout shift 0.05 Pricing: USD 30 / month, pooled across a team of up to 30 users. Source: https://qodo.ai/pricing/, read 2026-08-21. Value axis euro-normalised figure (not a published price): 25.64 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Optimised for tests and review rather than for building from nothing, and it is the wrong shape for a spec that starts with an empty directory. It still finished, and the test suites it wrote were the most thorough in the index: the volume test in prompt 8 was the only place any product wrote a load test without being asked. Read the low agent score as a mismatch with this spec rather than a weak tool. Raw runs (run, prompt, seconds, outcome): 1/1: 78.8s pass 1/2: 99.3s pass 1/3: 66.8s pass 1/4: 112.7s pass 1/5: 115.5s pass 1/6: 191.2s pass 1/7: 62.3s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/8: 97.1s pass (Wrote a load test for the volume criterion without being asked, then reported the p95 it measured.) 1/9: 141.5s partial (Required one clarifying instruction before the criterion was met.) 2/1: 75.9s pass 2/2: 114.2s pass 2/3: 116.4s pass 2/4: 169.9s pass 2/5: 134.9s pass 2/6: 134.8s pass 2/7: 89.1s pass 2/8: 66.2s pass (Wrote a load test for the volume criterion without being asked, then reported the p95 it measured.) 2/9: 347.7s partial (Validation added on the write path only. Read path left unguarded.) 3/1: 91.2s pass 3/2: 87.8s fail (Task abandoned by the agent with no artifact produced.) 3/3: 234.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/4: 239.7s pass 3/5: 81.6s pass 3/6: 76.1s pass 3/7: 103.2s pass 3/8: 53.3s pass (Wrote a load test for the volume criterion without being asked, then reported the p95 it measured.) 3/9: 124.7s pass 4/1: 102.5s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/2: 187.6s pass 4/3: 76.9s partial (Validation added on the write path only. Read path left unguarded.) 4/4: 94s pass 4/5: 66.5s pass 4/6: 92s pass 4/7: 140.9s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/8: 109.1s pass (Wrote a load test for the volume criterion without being asked, then reported the p95 it measured.) 4/9: 93.4s partial (Pagination off by one at the final page boundary.) 5/1: 159.7s pass 5/2: 74.1s pass 5/3: 99.4s pass 5/4: 79.7s pass 5/5: 137.7s pass 5/6: 94.7s pass 5/7: 249.2s pass 5/8: 43.8s pass (Wrote a load test for the volume criterion without being asked, then reported the p95 it measured.) 5/9: 42.6s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) ### Aider URL: https://www.llm-tier.com/builders/aider Official site: https://aider.chat Board: Coding agents (coding-agent) Rank: 13 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 65.88 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 72, Reliability 68.9, Scalability 64, SEO and GEO 54, API and MCP 62, Integrations 58, Design output 52, Turn latency score 79.7, Value 86 Turn latency seconds, p10 / median / p90: 22.5 / 39.4 / 75.7 Full specification minutes, derived (9x median, not separately measured): 5.9 Prompt attempts completed: 38, failed: 7, strict pass rate: 68.9 percent Shipped output: HTML with JavaScript disabled 17408 bytes, largest contentful paint 1920 ms, cumulative layout shift 0.05 Pricing: No licence fee, bring your own model key. Source: https://aider.chat/docs/faq.html, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The second fastest median in the whole index and by some distance the cheapest per completed prompt. It works on a small explicit set of files and commits as it goes, which is excellent discipline and also why it scores lower on the prompts that need a change across many files at once. Nothing about it is hosted, so there is no platform risk at all. Raw runs (run, prompt, seconds, outcome): 1/1: 22.1s partial (Validation added on the write path only. Read path left unguarded.) 1/2: 25.1s pass 1/3: 58.2s pass 1/4: 25.4s pass 1/5: 57.2s fail (Generated migration conflicted with the existing schema and was not reconciled.) 1/6: 80.9s pass 1/7: 24.4s pass 1/8: 20.5s partial (Required one clarifying instruction before the criterion was met.) 1/9: 24.8s pass 2/1: 41.7s pass 2/2: 34.7s pass 2/3: 27.6s pass 2/4: 26.1s partial (Error envelope inconsistent between two of the endpoints.) 2/5: 80.3s pass 2/6: 73.5s pass 2/7: 26.6s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/8: 25.2s pass 2/9: 50.7s pass 3/1: 23.3s pass 3/2: 41.7s pass 3/3: 64.2s pass 3/4: 63.5s pass 3/5: 48.9s pass 3/6: 77.2s pass 3/7: 42.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/8: 32.2s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 3/9: 18.3s pass 4/1: 30.6s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/2: 23.9s pass 4/3: 59.4s partial (Validation added on the write path only. Read path left unguarded.) 4/4: 23.7s pass 4/5: 181.6s pass 4/6: 101s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/7: 37.2s pass 4/8: 39.2s pass 4/9: 40.7s pass 5/1: 56.3s pass 5/2: 18.6s partial (Error envelope inconsistent between two of the endpoints.) 5/3: 59.6s pass 5/4: 49.7s pass 5/5: 18s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 5/6: 60.4s pass 5/7: 64.1s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/8: 23.1s pass 5/9: 39.4s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) ### Codebuff URL: https://www.llm-tier.com/builders/codebuff Official site: https://codebuff.com Board: Coding agents (coding-agent) Rank: 14 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 64.51 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 76, Reliability 68.9, Scalability 68, SEO and GEO 55, API and MCP 64, Integrations 62, Design output 58, Turn latency score 52.5, Value 52 Turn latency seconds, p10 / median / p90: 36.1 / 59.8 / 110.7 Full specification minutes, derived (9x median, not separately measured): 9 Prompt attempts completed: 38, failed: 7, strict pass rate: 68.9 percent Shipped output: HTML with JavaScript disabled 18432 bytes, largest contentful paint 1900 ms, cumulative layout shift 0.05 Pricing: USD 100 / month minimum subscription. Source: https://codebuff.com/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 85.48 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: A terminal agent that spreads a task over several specialised sub agents, which keeps the median low and occasionally produces two edits that disagree with each other. Three of the five recorded conflicts of that kind were caught by its own review step. Fast, cheap enough, and best on well bounded tasks rather than open ended ones. Raw runs (run, prompt, seconds, outcome): 1/1: 49.4s pass 1/2: 59.8s pass 1/3: 161.6s pass 1/4: 113.7s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 1/5: 71.9s pass 1/6: 136.9s pass 1/7: 45.8s pass 1/8: 74.7s pass 1/9: 35.1s pass 2/1: 44.6s pass 2/2: 91.9s pass 2/3: 32.7s partial (Two sub agents produced conflicting edits to the same policy file. The review step caught it and reconciled one of them.) 2/4: 86.2s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/5: 82s pass 2/6: 58.6s pass 2/7: 88.9s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 2/8: 53.4s partial (Required one clarifying instruction before the criterion was met.) 2/9: 106.1s pass 3/1: 80.6s partial (Validation added on the write path only. Read path left unguarded.) 3/2: 39.8s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 3/3: 59.6s partial (Two sub agents produced conflicting edits to the same policy file. The review step caught it and reconciled one of them.) 3/4: 80.9s pass 3/5: 57.2s pass 3/6: 54.6s pass 3/7: 79.6s pass 3/8: 40.3s pass 3/9: 39.3s pass 4/1: 23.9s partial (Pagination off by one at the final page boundary.) 4/2: 99.1s fail (Task abandoned by the agent with no artifact produced.) 4/3: 74.5s partial (Two sub agents produced conflicting edits to the same policy file. The review step caught it and reconciled one of them.) 4/4: 56.6s pass 4/5: 41.3s pass 4/6: 83s pass 4/7: 85.5s pass 4/8: 60s pass 4/9: 44.2s pass 5/1: 30.9s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 5/2: 53.9s pass 5/3: 28.7s pass 5/4: 37.6s pass 5/5: 205.1s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 5/6: 42.7s pass 5/7: 61.7s pass 5/8: 75.8s pass 5/9: 133s fail (Generated migration conflicted with the existing schema and was not reconciled.) ### OpenHands URL: https://www.llm-tier.com/builders/openhands Official site: https://openhands.dev Board: Coding agents (coding-agent) Rank: 15 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 63.93 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 74, Reliability 64.4, Scalability 70, SEO and GEO 56, API and MCP 68, Integrations 66, Design output 58, Turn latency score 21.7, Value 86 Turn latency seconds, p10 / median / p90: 81.4 / 144.8 / 269.1 Full specification minutes, derived (9x median, not separately measured): 21.7 Prompt attempts completed: 38, failed: 7, strict pass rate: 64.4 percent Shipped output: HTML with JavaScript disabled 18432 bytes, largest contentful paint 1980 ms, cumulative layout shift 0.05 Pricing: No licence fee, self hosted. Source: https://openhands.dev/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Open source and self hostable, and the only agent here whose sandbox runtime can be run entirely inside your own network. It is slower and less consistent than the commercial agents, and the spread reflects a sandbox that occasionally has to be rebuilt mid task. For anyone who cannot send a repository to a third party it is the strongest option in the index. Raw runs (run, prompt, seconds, outcome): 1/1: 99.8s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 1/2: 144.8s pass 1/3: 130.6s pass 1/4: 90.7s pass 1/5: 89.4s fail (Task abandoned by the agent with no artifact produced.) 1/6: 146.5s pass 1/7: 403.4s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 1/8: 183s pass 1/9: 141.4s pass 2/1: 69s pass 2/2: 138.8s pass 2/3: 125.2s pass 2/4: 257.4s pass 2/5: 536.6s partial (Pagination off by one at the final page boundary.) 2/6: 251.6s pass 2/7: 214.9s pass 2/8: 76.5s pass 2/9: 126.4s partial (Error envelope inconsistent between two of the endpoints.) 3/1: 147.9s pass 3/2: 66.4s pass 3/3: 88.7s pass 3/4: 62.8s pass 3/5: 242.1s pass 3/6: 120.5s partial (Required one clarifying instruction before the criterion was met.) 3/7: 159.7s pass 3/8: 295.3s pass 3/9: 99.9s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/1: 88.9s pass 4/2: 196.4s partial (Pagination off by one at the final page boundary.) 4/3: 75.3s pass 4/4: 216.4s fail (Task abandoned by the agent with no artifact produced.) 4/5: 115.3s partial (Required one clarifying instruction before the criterion was met.) 4/6: 188.9s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/7: 256.1s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/8: 149s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/9: 125.3s pass 5/1: 216.3s pass 5/2: 264.3s fail (Task abandoned by the agent with no artifact produced.) 5/3: 288s pass 5/4: 168.3s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/5: 215.2s pass 5/6: 94.3s fail (Generated migration conflicted with the existing schema and was not reconciled.) 5/7: 116.5s pass 5/8: 131.6s pass 5/9: 272.3s pass ### Trae URL: https://www.llm-tier.com/builders/trae Official site: https://trae.ai Board: Coding agents (coding-agent) Rank: 16 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 61.00 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 73, Reliability 51.1, Scalability 66, SEO and GEO 54, API and MCP 60, Integrations 66, Design output 66, Turn latency score 32.5, Value 84 Turn latency seconds, p10 / median / p90: 38.2 / 96.7 / 167.9 Full specification minutes, derived (9x median, not separately measured): 14.5 Prompt attempts completed: 32, failed: 13, strict pass rate: 51.1 percent Shipped output: HTML with JavaScript disabled 18432 bytes, largest contentful paint 1960 ms, cumulative layout shift 0.05 Pricing: USD 3 / month, billed monthly. Source: https://trae.ai/pricing, read 2026-08-21. Value axis euro-normalised figure (not a published price): 2.56 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: The cheapest paid plan in the index by a wide margin, and it is not a crippled tier: the free allowance completed most of a full run of the spec. Quality sits below the leading agents, mostly on the operational prompts, and the run to run spread is wider than the price suggests you should expect. Good value, not a leader. Raw runs (run, prompt, seconds, outcome): 1/1: 46.9s fail (Task abandoned by the agent with no artifact produced.) 1/2: 73.7s pass 1/3: 204.8s pass 1/4: 100.8s partial (Pagination off by one at the final page boundary.) 1/5: 126s pass 1/6: 243.1s pass 1/7: 119.7s pass 1/8: 23.7s pass 1/9: 37.9s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/1: 89s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/2: 73.9s partial (Pagination off by one at the final page boundary.) 2/3: 114.9s pass 2/4: 208.6s pass 2/5: 124.2s pass 2/6: 171.1s pass 2/7: 113.6s fail (Deployed URL returned 500 on every request.) 2/8: 36.1s pass 2/9: 65s fail (Deployed URL returned 500 on every request.) 3/1: 34.5s partial (Pagination off by one at the final page boundary.) 3/2: 58.6s fail (Deployed URL returned 500 on every request.) 3/3: 100.3s pass 3/4: 142.4s pass 3/5: 96.7s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/6: 114.5s partial (Error envelope inconsistent between two of the endpoints.) 3/7: 163.2s pass 3/8: 38.9s pass 3/9: 90s pass 4/1: 42.6s pass 4/2: 50.3s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/3: 106.7s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/4: 87.7s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 4/5: 154.5s partial (Validation added on the write path only. Read path left unguarded.) 4/6: 100.8s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 4/7: 128s fail (Generated migration conflicted with the existing schema and was not reconciled.) 4/8: 38.6s partial (Pagination off by one at the final page boundary.) 4/9: 124.9s pass 5/1: 47s partial (Error envelope inconsistent between two of the endpoints.) 5/2: 53.5s partial (Error envelope inconsistent between two of the endpoints.) 5/3: 54s pass 5/4: 146.3s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 5/5: 85.9s pass 5/6: 159s pass 5/7: 177.6s fail (Deployed URL returned 500 on every request.) 5/8: 28.5s pass 5/9: 43.8s pass ### Roomote URL: https://www.llm-tier.com/builders/roomote Official site: https://roomote.dev Board: Coding agents (coding-agent) Rank: 17 of 17 on this board (change since last run: 2). Not comparable to the other board's rank. Composite index: 59.89 out of 100, this board's own weighted sum Subscores 0 to 100 (9 axes on this board): Agent performance 71, Reliability 53.3, Scalability 66, SEO and GEO 53, API and MCP 62, Integrations 60, Design output 56, Turn latency score 32.6, Value 84 Turn latency seconds, p10 / median / p90: 54.3 / 96.4 / 202.9 Full specification minutes, derived (9x median, not separately measured): 14.5 Prompt attempts completed: 35, failed: 10, strict pass rate: 53.3 percent Shipped output: HTML with JavaScript disabled 17408 bytes, largest contentful paint 1980 ms, cumulative layout shift 0.05 Pricing: No licence fee, self hosted up to 10 users. Source: https://roomote.dev/, read 2026-08-21. Value axis euro-normalised figure (not a published price): 0.00 EUR. Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026. Last run: 15 Aug 2026. Next scheduled run: n/a Lab notes: Takes an issue and comes back with a pull request, so it competes with the asynchronous agents rather than the interactive ones. The change it proposed was usually right in outline and needed a review pass on details, which is a reasonable trade for work that costs no attention. Open source underneath, so nothing about the setup is locked to the hosted runner. Raw runs (run, prompt, seconds, outcome): 1/1: 70.1s fail (Task abandoned by the agent with no artifact produced.) 1/2: 86.6s partial (Pagination off by one at the final page boundary.) 1/3: 68.5s pass 1/4: 297.2s partial (Pagination off by one at the final page boundary.) 1/5: 141.1s pass 1/6: 148.1s pass 1/7: 364.9s pass 1/8: 145.9s partial (Pagination off by one at the final page boundary.) 1/9: 79.4s pass 2/1: 96.4s pass 2/2: 192.2s pass 2/3: 122.2s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/4: 116.2s pass 2/5: 209.8s fail (Agent looped on the same edit for the full timeout window of 20 minutes.) 2/6: 40.4s pass 2/7: 305.4s partial (Pagination off by one at the final page boundary.) 2/8: 81s fail (Generated migration conflicted with the existing schema and was not reconciled.) 2/9: 98s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 3/1: 46.2s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 3/2: 79.8s pass 3/3: 78.5s fail (Generated migration conflicted with the existing schema and was not reconciled.) 3/4: 192.6s pass 3/5: 109.3s partial (Required one clarifying instruction before the criterion was met.) 3/6: 82.6s pass 3/7: 185s pass 3/8: 65.5s fail (Build did not start after deploy. Recorded as a fail with the log preserved.) 3/9: 73.7s pass 4/1: 125.6s fail (Deployed URL returned 500 on every request.) 4/2: 185.3s pass 4/3: 61s partial (Required one clarifying instruction before the criterion was met.) 4/4: 136.9s pass 4/5: 85.6s fail (Task abandoned by the agent with no artifact produced.) 4/6: 74.4s fail (Task abandoned by the agent with no artifact produced.) 4/7: 138s pass 4/8: 50.5s pass 4/9: 266.4s pass 5/1: 87.2s pass 5/2: 87.1s partial (Pagination off by one at the final page boundary.) 5/3: 82.1s pass 5/4: 148.5s pass 5/5: 54.4s partial (Delivered the main path, missed the boundary case named in the pass criteria.) 5/6: 54.3s partial (Pagination off by one at the final page boundary.) 5/7: 127s partial (Pagination off by one at the final page boundary.) 5/8: 50.7s pass 5/9: 165.8s pass ## Language models: sourced reference table, not a ranking of ours We do not run a language-model harness. Every benchmark figure on this page is a third party's measurement, named and dated, and every specification and price is the vendor's own published figure. Where a vendor does not publish a number we leave the cell empty and say so rather than sourcing it from an aggregator. This board is a sourced reference table, not an LLM Tier ranking. It has no composite index because we have not earned the right to publish one: we hold no model-level run records, so we could not show you the arithmetic. Sorted by Arena WebDev Elo, descending. Rows with no published Elo sort last, alphabetically by vendor. That ordering is Arena's measurement, not ours. 26 rows, superseded/legacy models kept separately below rather than deleted. Benchmark provenance: - Arena WebDev (arena.ai): Human pairwise preference Elo on web-app building tasks. The closest public analogue to what our product boards measure. Run by arena.ai. Board dated 21 Aug 2026, 603,789 votes, 118 models. Some rows are flagged Preliminary by Arena on low vote counts; we carry that flag through. - Artificial Analysis Intelligence Index v4.1.1: A normalised composite of nine evaluations. Run by Artificial Analysis, independent. Artificial Analysis publishes no page-level revision date, and publishes separate rows per reasoning effort level. - Terminal-Bench v2.1: 89 curated terminal tasks, pass@1 averaged over 3 repeats. Run by Governed by the Laude Institute with Stanford researchers and an open-source community; independently evaluated by Artificial Analysis. Always cited with the effort level alongside the score. Pricing warning: Arena publishes a price column beside its Elo. We do not use it. Arena lists GPT-5.6 Sol at 5 / 30 dollars per million tokens; OpenAI's own pricing page lists 4 / 20 for Sol and 5 / 30 for GPT-5.5, so Arena has carried the older model's price onto the newer one. It also lists Gemini 3.7 Flash output at 3.57 where Google publishes 3.75. We take Elo, vote counts, licence and the Preliminary flag from Arena, and we take every price, context window and output limit from the vendor. This is the same failure we have found on price-aggregator sites, and it is worth knowing that a benchmark operator is not immune to it. ### Claude Opus 5 (Anthropic) Status / tier: current / flagship. Licence: Proprietary. Context window: 1,000,000. Max output: 128,000. Price in / out per million tokens: 5.00 / 25.00. Modality: text and image in, text out. Arena WebDev: 1691 Elo, rank 1, 8116 votes. Artificial Analysis Intelligence Index v4.1.1: 63 at max effort, 63 at xhigh, 61 at high. Terminal-Bench v2.1 pass@1: 89.1% at max effort. Release date not stated by the vendor; training data cutoff May 2026. Arena also lists the high-effort variant separately: 1663 Elo, rank 4, 8659 votes. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026; Artificial Analysis (artificialanalysis.ai) ### Kimi K3 (Moonshot AI) Status / tier: current / flagship. Licence: Modified MIT, published by Moonshot as the Kimi K3 License; open weights. Context window: 1,048,576. Max output: Not stated on the vendor pricing page. Price in / out per million tokens: 3.00 on a cache miss, 0.30 on a cache hit / 15.00. Modality: Multimodal reasoning with thinking always on, per secondary reporting. Arena WebDev: 1674 Elo, rank 2, 4546 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Released 16 July 2026; full open weights 27 July 2026. Kimi K3 always reasons and exposes a reasoning_effort setting of low, high or max, defaulting to max, which is why Arena lists it as kimi-k3-max (max effort). A third-party host lists the same weights at 2.60 / 13.00. That is the host's price, not Moonshot's, and we publish Moonshot's. Sources: Moonshot AI platform pricing (platform.moonshot.ai); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Qwen3.8-Max (Alibaba) Status / tier: current / flagship. Licence: Proprietary. Context window: 1,048,576, charged at one flat rate across the whole window with no length tiering. Max output: 131,000. Price in / out per million tokens: 2.00 (batch inference 1.65; context-caching discount available) / 6.00, covering chain of thought plus answer (batch inference 4.951). Modality: text, image and video in, text out. Arena WebDev: 1669 Elo, rank 3, 3212 votes, flagged Preliminary by Arena. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Previewed 19 July 2026, global API access via Alibaba Cloud Model Studio 3 August 2026. There is no independently measured coding-benchmark score for this model, so we publish none, and its Arena Elo carries Arena's own Preliminary flag on 3,212 votes. It is also the only flagship on this board with no 200k price step, which makes it unusually cheap on very long prompts. Sources: Alibaba Cloud Model Studio pricing (alibabacloud.com/product/modelstudio); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Grok 4.6 (xAI) Status / tier: current / flagship. Licence: Proprietary. Context window: 500,000. Max output: No stated text output limit. Price in / out per million tokens: 2.00 below 200k prompt tokens, 4.00 above (cached 0.50 / 1.00) / 6.00 below 200k prompt tokens, 12.00 above. Modality: The xAI models page lists text in and text out; secondary reporting says image input is also supported, which we have not confirmed. Arena WebDev: 1629 Elo, rank 5, 1523 votes, flagged Preliminary by Arena. Artificial Analysis Intelligence Index v4.1.1: 61 per secondary reporting of the index. Terminal-Bench v2.1 pass@1: 88.4% at high effort. 12 August 2026 per secondary reporting; not dated on xAI's models page. Arena files xAI's models under the lab name SpaceXAI; the vendor documentation domain is docs.x.ai and we publish the vendor as xAI. See Grok 4.3 below for the headline counter-example on this board. Sources: xAI models page (docs.x.ai); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026; Artificial Analysis (artificialanalysis.ai) ### Claude Fable 5 (Anthropic) Status / tier: current / flagship. Licence: Proprietary. Context window: 1,000,000. Max output: 128,000 (300,000 on the Batch API with the output-300k-2026-03-24 beta header). Price in / out per million tokens: 10.00 / 50.00. Modality: text and image in, text out. Arena WebDev: 1626 Elo, rank 6, 8382 votes. Artificial Analysis Intelligence Index v4.1.1: 62 (Adaptive Reasoning, Max Effort). Terminal-Bench v2.1 pass@1: Not in the published top three. Available from 9 June 2026. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026; Artificial Analysis (artificialanalysis.ai) ### GPT-5.6 Sol (OpenAI) Status / tier: current / flagship. Licence: Proprietary. Context window: Not published on OpenAI's pricing page. Max output: Not published on OpenAI's pricing page. Price in / out per million tokens: 4.00 (cached input 0.40) / 20.00. Modality: text in, text out per the pricing table. Arena WebDev: 1619 Elo, rank 7, 9499 votes. Artificial Analysis Intelligence Index v4.1.1: 61 at max effort. Terminal-Bench v2.1 pass@1: 89.5% at xhigh effort, the published leader. Limited preview 26 June 2026, public launch 9 July 2026. Arena entry uses xhigh effort in the Codex harness. See the pricing warning box above: Arena lists this model at the older GPT-5.5 price. Sources: OpenAI pricing page (openai.com/api/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026; Artificial Analysis Terminal-Bench v2.1 (artificialanalysis.ai) ### Gemini 3.7 Flash (Google) Status / tier: current / workhorse. Licence: Proprietary. Context window: Not stated on the fetched pages. Max output: Not stated. Price in / out per million tokens: 0.75 through 31 December 2026, then 1.50 from 1 January 2027 / 3.75 through 31 December 2026, then 7.50 from 1 January 2027. Modality: Not stated per model on the fetched page. Arena WebDev: 1587 Elo, rank 10, 2552 votes, flagged Preliminary by Arena. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. 13 August 2026 per secondary reporting; Google's own pages mark it stable but do not date it. Arena entry uses high effort and carries Arena's Preliminary flag. Google publishes DeepSWE v1.1 rising from 49.0 to 65.3 percent and GDM-MRCR v2 at 128k rising from 91.8 to 97.0 percent versus the prior Flash. Those are Google's own figures on Google's own evaluation and are labelled vendor-claimed, not independent. Sources: Google AI pricing/models pages (ai.google.dev/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### DeepSeek-V4-Pro (DeepSeek) Status / tier: current / flagship. Licence: MIT. Context window: 1,000,000. Max output: Not stated on the pricing page. Price in / out per million tokens: 0.66 off-peak, 1.32 at peak (cache hit 0.022 off-peak, 0.044 peak) / 1.98 off-peak, 3.96 at peak. Modality: text in, text out; a separate deepseek-v4-flash-vision-exp handles vision. Arena WebDev: 1582 Elo, rank 12, 2869 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Version string DeepSeek-V4-Pro-0813 implies 13 August 2026; not dated in prose. Arena entry uses high effort on the 20260813 build. DeepSeek has no single price: off-peak is half of peak, and peak runs 01:00 to 04:00 and 06:00 to 10:00 UTC. Any value comparison must state which rate it used. Sources: DeepSeek API pricing page (api-docs.deepseek.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### DeepSeek-V4-Flash (DeepSeek) Status / tier: current / budget. Licence: MIT. Context window: 1,000,000. Max output: Not stated. Price in / out per million tokens: 0.22 off-peak, 0.44 at peak (cache hit 0.007 / 0.014) / 0.66 off-peak, 1.32 at peak. Modality: text in, text out. Arena WebDev: 1579 Elo, rank 13, 3537 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Version string DeepSeek-V4-Flash-0731 implies 31 July 2026. Arena entry uses high effort. DeepSeek has no single price: off-peak is half of peak, and peak runs 01:00 to 04:00 and 06:00 to 10:00 UTC. Any value comparison must state which rate it used. See the cheapest-output callout above for why this row and GPT-5.6 Luna cannot both hold that crown. Sources: DeepSeek API pricing page (api-docs.deepseek.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Claude Sonnet 5 (Anthropic) Status / tier: current / workhorse. Licence: Proprietary. Context window: 1,000,000. Max output: 128,000. Price in / out per million tokens: 2.00 / 10.00. Modality: text and image in, text out. Arena WebDev: 1539 Elo, rank 22, 7267 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Release date not stated by the vendor. Arena effort level: high. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Gemini 3.6 Flash (Google) Status / tier: current / workhorse. Licence: Proprietary. Context window: Not stated. Max output: Not stated. Price in / out per million tokens: 0.75, rising to 1.50 on 1 January 2027 / 3.75, rising to 7.50 on 1 January 2027. Modality: Not stated. Arena WebDev: 1539 Elo, rank 21, 6464 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Not dated on Google's pages; marked stable. Arena entry uses high effort. Sources: Google AI pricing/models pages (ai.google.dev/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Muse Spark 1.1 (Meta) Status / tier: current / flagship. Licence: Proprietary, closed weights. Context window: 1,000,000. Max output: Not specified in Meta's announcement. Price in / out per million tokens: Not published by Meta / Not published by Meta. Modality: text, images, video and PDFs in, text out. Arena WebDev: 1539 Elo, rank 20, 6204 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Developed by Meta Superintelligence Labs. Muse Spark launched 8 April 2026 as Meta's first closed-weight model; 1.1 released 9 July 2026. Meta's own announcement publishes no per-token price and no numerical benchmark score, so both cells are empty here. Price is the single most-quoted number in secondary coverage and the single number Meta does not state. Arena also lists a muse-spark-1.2 at 1534 Elo; we have not sourced 1.2 from Meta and so do not list it as a row. Sources: Meta Superintelligence Labs announcement (meta.com/ai); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### GPT-5.6 Terra (OpenAI) Status / tier: current / workhorse. Licence: Proprietary. Context window: Not published. Max output: Not published. Price in / out per million tokens: 2.00 (cached input 0.20) / 12.00. Modality: text in, text out. Arena WebDev: 1520 Elo, rank 27, 5677 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Launched 9 July 2026. Arena entry uses xhigh effort in the Codex harness. Sources: OpenAI pricing page (openai.com/api/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### GPT-5.6 Luna (OpenAI) Status / tier: current / budget. Licence: Proprietary. Context window: Not published. Max output: Not published. Price in / out per million tokens: 0.20 (cached input 0.02) / 1.20. Modality: text in, text out. Arena WebDev: 1518 Elo, rank 29, 5742 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Launched 9 July 2026. Arena entry uses xhigh effort in the Codex harness. Flat 1.20 output price. See the DeepSeek-V4-Flash row for the peak and off-peak comparison, since neither is cheapest all day. Sources: OpenAI pricing page (openai.com/api/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Gemini 3.5 Flash (Google) Status / tier: current / workhorse. Licence: Proprietary. Context window: Not stated. Max output: Not stated. Price in / out per million tokens: 1.50 / 9.00. Modality: Not stated. Arena WebDev: 1499 Elo, rank 34, 7559 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Not dated; marked stable. Arena entry uses high effort. This middle rung is worse value than both newer Flash models, which cost 0.75 / 3.75. Inside one vendor's line, newer here is both better and cheaper. Sources: Google AI pricing/models pages (ai.google.dev/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Gemini 3.5 Flash-Lite (Google) Status / tier: current / budget. Licence: Proprietary. Context window: Not stated. Max output: Not stated. Price in / out per million tokens: 0.30 / 2.50. Modality: Not stated. Arena WebDev: 1449 Elo, rank 47, 228 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Not dated; marked stable. Arena's interval on this row is plus or minus 43 points on only 228 votes (1449 ± 43), far too wide to rank against its neighbours. Sources: Google AI pricing/models pages (ai.google.dev/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Gemini 3.1 Pro Preview (Google) Status / tier: preview / flagship. Licence: Proprietary. Context window: Not stated on Google's models or pricing pages. Max output: Not stated. Price in / out per million tokens: 2.00 for prompts up to 200k tokens, 4.00 above 200k (batch 1.00 / 2.00) / 12.00 up to 200k, 18.00 above 200k (batch 6.00 / 9.00). Modality: Not stated per model on the fetched page. Arena WebDev: 1446 Elo, rank 48, 21573 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Still carries preview status. This is the only Pro-tier Gemini 3 text model on Google's own list; every stable Gemini 3 text model is a Flash or Flash-Lite. Sources: Google AI pricing/models pages (ai.google.dev/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### GPT-5.3 Codex (OpenAI) Status / tier: current / coding. Licence: Proprietary. Context window: Not published. Max output: Not published. Price in / out per million tokens: 1.75 (cached input 0.175) / 14.00. Modality: text in, text out, coding specialised. Arena WebDev: 1408 Elo, rank 61, 2526 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Release date not published. Arena entry runs in the Codex harness. Sources: OpenAI pricing page (openai.com/api/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Grok 4.3 (xAI) Status / tier: current / workhorse. Licence: Proprietary. Context window: 1,000,000. Max output: Not stated. Price in / out per million tokens: 1.25 below 200k prompt tokens, 2.50 above (cached 0.20 / 0.40) / 2.50 below 200k, 5.00 above. Modality: text in, text out. Arena WebDev: 1357 Elo, rank 86, 12942 votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Not dated on xAI's models page. HEADLINE COUNTER-EXAMPLE ON THIS BOARD: Grok 4.3 has double the context window of xAI's newer flagship (Grok 4.6) at roughly 40 percent of its output price, and sits 272 Elo points lower on Arena WebDev. Newer, wider and cheaper are three different axes and no vendor wins all three at once. Sources: xAI models page (docs.x.ai); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Amazon Nova 2 Lite (Amazon) Status / tier: current / workhorse. Licence: Proprietary. Context window: 1,000,000. Max output: 65,536. Price in / out per million tokens: Not published on the Nova 2 user guide, which defers to Amazon Bedrock Pricing; the Bedrock pricing page carries no Nova 2 row / Not published. Modality: text, images, video and documents in, text out. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Release date not stated on the Nova 2 user guide. Nova 2 is Amazon's current generation and its whole roster is Nova 2 Lite, Nova 2 Sonic and Nova Multimodal Embeddings. Nova Premier, Nova Pro, Nova Lite and Nova Micro are all the superseded Nova 1 generation. Every Nova 2 model is 1,000,000 tokens in and 65,536 out. Sources: Amazon Nova user guide (docs.aws.amazon.com/nova); Amazon Bedrock pricing page (aws.amazon.com/bedrock/pricing) ### Claude Haiku 4.5 (Anthropic) Status / tier: current / budget. Licence: Proprietary. Context window: 200,000. Max output: 64,000. Price in / out per million tokens: 1.00 / 5.00. Modality: text and image in, text out. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Model ID pins the snapshot to 1 October 2025. Sources: Anthropic pricing/models page (anthropic.com/pricing) ### Command A+ (Cohere) Status / tier: current / flagship. Licence: Apache 2.0. Context window: Not published for this model. Max output: Not published. Price in / out per million tokens: Not published. cohere.com/pricing carries only legacy Command rows and has no Command A+ row at all. / Not published. Modality: text and image in, text out. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. 20 May 2026. 218 billion total parameters with 25 billion active in a sparse mixture of experts, 48 languages, native citation generation. Cohere's flagship is three months old and its own public pricing page does not list it. We publish the model with no price rather than sourcing one from an aggregator. Sources: Cohere pricing page (cohere.com/pricing) ### Gemini 3.1 Flash-Lite (Google) Status / tier: current / budget. Licence: Proprietary. Context window: Not stated. Max output: Not stated. Price in / out per million tokens: 0.25 for text, image and video; 0.50 for audio / 1.50. Modality: text, image, video and audio in per the pricing tiers. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Not dated; marked stable. Google's own model list describes this as frontier class; we attribute that claim to Google and do not restate it as fact. Sources: Google AI pricing/models pages (ai.google.dev/pricing) ### Llama 4 Maverick (Meta) Status / tier: open_weight / workhorse. Licence: Open weights. Context window: Previously published here as 1,000,000; not re-verified in this pass. Max output: Not re-verified. Price in / out per million tokens: No vendor list price exists / No vendor list price exists. Modality: Not re-verified. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. 5 April 2025, alongside Llama 4 Scout. No new Llama model has shipped since. We previously published 0.22 / 0.85 for this model. That figure was unsourceable by construction, not merely stale: these are open weights and there is no vendor list price at all, only per-host prices. We have removed it. Llama is no longer Meta's frontier: Meta's frontier since April 2026 is Muse Spark, which is closed and paid. Sources: Meta AI Llama models page (llama.meta.com) ### Mistral Medium 3.5 (Mistral AI) Status / tier: current / flagship. Licence: Modified MIT. Context window: Not listed on Mistral's models overview page. Max output: Not listed. Price in / out per million tokens: Not published on Mistral's docs page / Not published on Mistral's docs page. Modality: Multimodal per Mistral's own description. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Version string v26.04 implies April 2026; not dated in prose. Mistral's current premier flagship, described by Mistral as frontier class and optimised for agentic and coding work. Sources: Mistral AI models overview and docs pages (mistral.ai/products/models, docs.mistral.ai) ### Mistral Small 4 (Mistral AI) Status / tier: open_weight / workhorse. Licence: Apache 2.0. Context window: Not listed on Mistral's docs page. Max output: Not listed. Price in / out per million tokens: Not published on Mistral's docs page / Not published on Mistral's docs page. Modality: Unified instruct, reasoning and coding, with multimodal understanding, per secondary reporting. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. 16 March 2026 per secondary reporting; version string v26.03 is consistent. Sources: Mistral AI docs page (docs.mistral.ai) ## Superseded and legacy language models (12 rows) Kept to show the price ladder and to make our own previous board auditable rather than quietly deleted. ### Claude Opus 4.7 (Anthropic) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 1,000,000. Max output: 128,000. Price in / out per million tokens: 5.00 / 25.00. Modality: text and image in, text out. Arena WebDev: 1558 Elo, rank 15, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Anthropic itself files this model under Legacy models and recommends migrating. Arena also lists a high-effort variant separately: 1557 Elo, rank 16. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Claude Opus 4.8 (Anthropic) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 1,000,000. Max output: 128,000. Price in / out per million tokens: 5.00 / 25.00. Modality: text and image in, text out. Arena WebDev: 1539 Elo, rank 19, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Anthropic itself files this model under Legacy models and recommends migrating. Arena also lists a high-effort variant separately: 1563 Elo, rank 14. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Claude Opus 4.6 (Anthropic) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 1,000,000. Max output: 128,000. Price in / out per million tokens: 5.00 / 25.00. Modality: text and image in, text out. Arena WebDev: 1536 Elo, rank 23, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Anthropic itself files this model under Legacy models and recommends migrating. Arena also lists a high-effort variant separately: 1546 Elo, rank 18. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Claude Sonnet 4.6 (Anthropic) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 1,000,000. Max output: 128,000. Price in / out per million tokens: 3.00 / 15.00. Modality: text and image in, text out. Arena WebDev: 1522 Elo, rank 25, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Anthropic itself files this model under Legacy models and recommends migrating. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Qwen3 Max (Alibaba) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 262,144 as originally published; understates the real current window by roughly four times. Max output: -. Price in / out per million tokens: 2.50 list, limited-time 50 percent discount / 7.50 list, limited-time 50 percent discount. Modality: text, image and video in, text out. Arena WebDev: 1517 Elo, rank 30, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Was our number 6 on the old board, published here at 262,144 context. Arena and Alibaba both now identify the current build as the 3.7-Max release, which Alibaba lists at 2.50 / 7.50 list with a limited-time 50 percent discount. Sources: Alibaba Cloud Model Studio pricing (alibabacloud.com/product/modelstudio); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### GPT-5.5 (OpenAI) Status / tier: legacy / n/a. Licence: Proprietary. Context window: -. Max output: -. Price in / out per million tokens: 5.00 (cached 0.50) / 30.00. Modality: text in, text out. Arena WebDev: 1508 Elo, rank 33, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. This is the row whose price aggregators and Arena wrongly carried onto GPT-5.6 Sol (see the warning box above). Sources: OpenAI pricing page (openai.com/api/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Claude Opus 4.5 (Anthropic) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 200,000. Max output: -. Price in / out per million tokens: 5.00 / 25.00. Modality: text and image in, text out. Arena WebDev: 1468 Elo, rank 42, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Was our index leader at 89.10 under the old, single merged board. Still purchasable. Sits below Opus 4.8, 4.7 and 4.6, all three of which the vendor itself files as legacy, which is why the old board was three generations behind, not one. Arena also lists a high-effort (32k) variant separately: 1494 Elo, rank 35. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Gemini 3 Pro (Google) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 1,000,000. Max output: -. Price in / out per million tokens: 2.00 / 12.00. Modality: text and image in, text out. Arena WebDev: 1438 Elo, rank 52, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Was our number 3 on the old board, and the row our previous "widest context" callout used to rest on. Sources: Google AI pricing page (ai.google.dev/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Claude Sonnet 4.5 (Anthropic) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 200,000. Max output: -. Price in / out per million tokens: 3.00 / 15.00. Modality: text and image in, text out. Arena WebDev: 1386 Elo, rank 74, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Was our number 4 on the old, single merged board. Sources: Anthropic pricing/models page (anthropic.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### DeepSeek V3.2 (DeepSeek) Status / tier: legacy / n/a. Licence: MIT. Context window: 128,000 as we published it. Max output: -. Price in / out per million tokens: 0.28 / 0.42. Modality: text in, text out. Arena WebDev: 1360 Elo, rank 82, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Was our number 7 on the old board, listed on Arena as deepseek-v3.2-thinking. We published 0.28 / 0.42 for it, and 0.42 matches no DeepSeek rate, current or off-peak, that we can find on the vendor's pricing page. Sources: DeepSeek API pricing page (api-docs.deepseek.com/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### GPT-5.1 (OpenAI) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 400,000. Max output: -. Price in / out per million tokens: 1.25 / 10.00. Modality: text and image in, text out. Arena WebDev: 1341 Elo, rank 89, n/a votes. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Was our number 2 on the old, single merged board. Sources: OpenAI pricing page (openai.com/api/pricing); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 ### Grok 4 (xAI) Status / tier: legacy / n/a. Licence: Proprietary. Context window: 256,000. Max output: -. Price in / out per million tokens: 3.00 / 15.00. Modality: text in, text out. Arena WebDev: not present on Arena WebDev. Artificial Analysis Intelligence Index v4.1.1: not measured. Terminal-Bench v2.1 pass@1: not measured. Was our number 5 on the old board. Not present in Arena's published WebDev list under this label. Sources: xAI models page (docs.x.ai); Arena WebDev leaderboard (arena.ai/leaderboard/code/webdev), board dated 21 Aug 2026 No composite index or rank exists anywhere in this models section: this lab runs no language-model harness and holds no model-level run records, so it does not publish one. Full disclosure on https://www.llm-tier.com/leaderboard/llms and the LLM section of https://www.llm-tier.com/methodology. ## Lab notes - 957 reader rows carried a date from before this domain existed (news, 09 Sep 2026): In August we corrected eight article dates and published the check that found them. The check was scoped to the articles table. Every reader score, open rating, lab note, comment and vote on this site was also dated before the domain resolved. All 957 are corrected, and the gate is restated wider. https://www.llm-tier.com/news/nine-hundred-and-fifty-seven-reader-rows-were-dated-before-this-domain-existed - Eight articles carried a publication date from before this domain was registered (news, 23 Aug 2026): llm-tier.com was registered on August 19, 2026 at 10:10 UTC. Eight of our fourteen articles were stamped earlier than that, in the byline and in the structured data. Corrected today, with the check that finds it now running on a schedule. https://www.llm-tier.com/news/eight-articles-were-dated-before-this-domain-existed - Entry prices are now published in the vendor's own currency, not a euro conversion that tracked no single rate (news, 22 Aug 2026): Every entry price on both leaderboards carried a bare euro figure with no stated source or rate. An audit found the implied euro per dollar rate across cleanly priced USD rows spread from 0.19 to 1.80, wrong in both directions. Prices are now published in the vendor's own currency, unit and billing period, sourced. The Value axis and the composite index were recomputed board wide as a result, and Totalum's own Value subscore fell from 84 to 69. https://www.llm-tier.com/news/entry-prices-republished-in-vendors-own-currency - The model board carried a composite index with no run behind it. It is gone. (news, 22 Aug 2026): The eight rows on /leaderboard/llms carried an Index column that looked exactly like the composite this lab actually measures on the other two boards. It was not: no per-model run record, subscore or recomputed field sat behind any of the eight numbers, and Arena's own leaderboard shows all eight rows were two to three generations stale. The index is removed, not refreshed. What replaces it is a 26-row sourced specification-and-benchmark table, with the twelve superseded rows kept in a separate table rather than deleted. https://www.llm-tier.com/news/the-model-board-had-a-fabricated-index-we-removed-it - Tesslate leaves the index: the vendor has pivoted to a different category (news, 22 Aug 2026): tesslate.com no longer operates an app builder. The domain now hosts OpenSail, a business operations platform, and the EUR 16 monthly price we published cannot be sourced from the vendor's current site. Tesslate is out of the roster, the two leaderboards it appeared on, the matrix and every data export. https://www.llm-tier.com/news/tesslate-has-pivoted-to-opensail-and-leaves-the-index - Convex Chef is being replaced by its vendor, so it leaves the index (news, 21 Aug 2026): Convex is replacing Chef with a new product and has taken down its entry points. Existing projects still run, so this is not a shutdown, but a builder you cannot sign up for cannot hold a live ranking. Chef is out of the roster, the raw runs, the matrix, both leaderboards it appeared on and both of its compare pages. https://www.llm-tier.com/news/convex-chef-is-being-replaced-by-its-vendor - Pazi leaves the index: wrong category, and a price we could not source (news, 21 Aug 2026): We ranked Pazi 37 of 41 as a coding agent and scored it on 45 executions of nine software engineering prompts. Pazi does not sell a coding product. We also published a 14 EUR floor against a real 25 dollar one, and cited a source URL that never contained the figure. https://www.llm-tier.com/news/pazi-was-never-in-this-category - Mocha leaves the index, and we benchmarked it after it had already announced its own end. (news, 21 Aug 2026): Mocha is gone from this roster, and the reason is as much about our own record as the vendor's calendar. We published a euro price and a value score for a plan its own pricing page says can no longer be bought, and every one of the 45 runs behind its row was collected after the vendor's stated shutdown date had already passed. https://www.llm-tier.com/news/mocha-was-benchmarked-after-it-announced-its-end - This cycle's composite recompute, and why it barely moved most rows (methodology, 20 Aug 2026): We re derived every composite index from the ten published axis weights and the stored subscores, row by row, and checked the arithmetic against the methodology page. Here is what changed and what that process actually verifies. https://www.llm-tier.com/news/this-cycles-composite-recompute - Windsurf is gone from the index (news, 20 Aug 2026): windsurf.com now redirects to devin.ai/desktop. The product is discontinued, so it has been removed from the roster, the raw runs, every leaderboard, every compare pair and the matrix, not just flagged as inactive. https://www.llm-tier.com/news/windsurf-is-gone-from-the-index - August 2026 index update: three products move, one regression (news, 20 Aug 2026): The fourth full run of the vibeOps specification against 11 products. Two products improved their reliability materially, one regressed on tenancy isolation. https://www.llm-tier.com/news/august-2026-index-update - Why we publish the spread and not the mean (methodology, 20 Aug 2026): A single average build time hides the thing you actually care about: how bad the slow runs get. Here is the p10 to p90 data for every product in the index. https://www.llm-tier.com/news/why-we-publish-the-spread-not-the-mean - Server rendering is the whole GEO story right now (analysis, 19 Aug 2026): We disabled JavaScript on every generated marketing page and counted the bytes of real text. The gap between the top and the bottom of the cohort is nearly seven to one. https://www.llm-tier.com/news/server-rendering-is-the-geo-story - How we score agent performance without reading the chat log (methodology, 19 Aug 2026): Agent performance is 18 points of the index, the joint largest weight. Here is exactly how a prompt earns 0, 50 or 100. https://www.llm-tier.com/news/how-we-score-agent-performance - Reproducing any row in the index by hand (methodology, 19 Aug 2026): The composite is a weighted sum, nothing more. Take the ten subscores off any product page, multiply by the published weights and you get the number in the table. https://www.llm-tier.com/news/reproducing-the-index-by-hand - What this benchmark cannot tell you (analysis, 19 Aug 2026): A composite index is a compression of a complicated thing into one number. Here are the limits of ours, stated plainly. https://www.llm-tier.com/news/what-a-benchmark-cannot-tell-you ## Head to head pages Each of these pages compares two products that sit on the same board. They hold no figure that is not already in this file. - Lovable versus Bolt.new: https://www.llm-tier.com/compare/lovable-vs-bolt-new - Lovable versus v0: https://www.llm-tier.com/compare/lovable-vs-v0 - Bolt.new versus v0: https://www.llm-tier.com/compare/bolt-new-vs-v0 - Replit versus Bolt.new: https://www.llm-tier.com/compare/replit-vs-bolt-new - Lovable versus Replit: https://www.llm-tier.com/compare/lovable-vs-replit - Base44 versus Lovable: https://www.llm-tier.com/compare/base44-vs-lovable - Totalum versus Lovable: https://www.llm-tier.com/compare/totalum-vs-lovable - Totalum versus Base44: https://www.llm-tier.com/compare/totalum-vs-base44 - Emergent versus Replit: https://www.llm-tier.com/compare/emergent-vs-replit - Zite versus Rocket.new: https://www.llm-tier.com/compare/zite-vs-rocket-new - Cursor versus Claude Code: https://www.llm-tier.com/compare/cursor-vs-claude-code - Cursor versus Zed: https://www.llm-tier.com/compare/cursor-vs-zed - Claude Code versus Amp: https://www.llm-tier.com/compare/claude-code-vs-amp - Aider versus Cline: https://www.llm-tier.com/compare/aider-vs-cline - Devin versus Jules: https://www.llm-tier.com/compare/devin-vs-jules - Devin versus Factory: https://www.llm-tier.com/compare/devin-vs-factory - Cursor versus Kiro: https://www.llm-tier.com/compare/cursor-vs-kiro - Augment Code versus Cursor: https://www.llm-tier.com/compare/augment-code-vs-cursor - Antigravity versus Jules: https://www.llm-tier.com/compare/antigravity-vs-jules - Firebase Studio versus Lovable: https://www.llm-tier.com/compare/firebase-studio-vs-lovable - Dyad versus bolt.diy: https://www.llm-tier.com/compare/dyad-vs-bolt-diy - Floot versus Lovable: https://www.llm-tier.com/compare/floot-vs-lovable - YouWare versus v0: https://www.llm-tier.com/compare/youware-vs-v0 - Trae versus Cursor: https://www.llm-tier.com/compare/trae-vs-cursor ## Machine readable endpoints - https://www.llm-tier.com/api/rankings.json - https://www.llm-tier.com/api/runs.json - https://www.llm-tier.com/data/rankings.csv - https://www.llm-tier.com/sitemap.xml