The AI app builder and coding agent benchmark, published in full

Most published comparisons of AI app builders are written from a free trial and a screenshot. Someone signs up, types a prompt about a to do list, likes what appears, and writes five hundred words about the future of software. That process cannot separate a product that is genuinely good from a product that is good at the first ninety seconds, and it produces rankings that change with the mood of the writer rather than with the behaviour of the software. This site exists because we wanted the other kind of comparison: one specification, applied identically, repeated enough times to see variance, with every measurement published so the conclusion can be checked.

The specification is called vibeOps. It describes a multi tenant SaaS product: a public REST API with key based authentication and rate limiting, outbound webhooks that are signed and retried, background jobs that survive a redeploy, an internal admin panel gated to staff accounts, and a public marketing surface that has to render without JavaScript. It is deliberately not a landing page and not a to do list. It is the shape of work people actually pay for, and it forces every product to touch data modelling, authentication, asynchronous work and an operations surface rather than just layout.

Both classes are given the same nine prompts, but a merged ranking across them was never measuring one thing: the two products start from a different state (a cold empty project versus a warm existing repository), produce a different unit of work (a deployed URL versus a diff), and are judged by a different success criterion (does the app work versus do the existing tests still pass). Turn latency score, value, integrations and code ownership together carry 23 of the 100 index points on the builders board, and none of those four survives a merge across the two classes unchanged. That is why this site publishes two boards, each with its own weight table, its own rank column starting at 1, and no page that mixes them into one ordering.

We issue 9 prompts in a fixed order. Each prompt is phrased precisely, in the language a working engineer would use, and each has a published pass criterion written before the run. The criteria are concrete rather than aesthetic: a cross tenant read must be refused, a rate limiter must return 429 with a retry hint, a webhook signature must verify against the stored secret, a job must be idempotent when replayed and must still be queued after a restart. The prompts are never rephrased to help a product that is struggling, and a product that asks a clarifying question gets an answer without being told what it forgot.

Every product runs the full sequence 5 times, which gives 45 recorded executions per product and 1,665 across the published cohort this cycle. Repetition is the part most benchmarks skip, and it is the part that matters most, because these systems are not deterministic. The same prompt to the same product on the same day can produce a clean result once and a stalled agent the next time. Publishing a single build time from a single lucky run is not a measurement, it is an anecdote. So for every product we publish the tenth percentile, the median and the ninetieth percentile of per-prompt turn latency over all 45 recorded executions regardless of outcome, and we publish the outcome of every individual execution as pass, partial or fail.

Turn latency score is worth less than most readers expect, 7 of 100 points on the builders board and 7.22 of 100 on the coding agent board, and the seconds figure behind it measures one prompt, not a whole build: a product that returns a 200 response in thirty seconds and leaves you to write the queue yourself has not saved you an afternoon. Code ownership is worth only 3 points on the builders board. Code ownership is not scored at all on the coding agent board, because a coding agent edits a repository you already own, so the property was never the vendor's to grant and a full mark measures nothing. The other nine weights are rescaled proportionally so they still sum to 100; their relative importance to each other is unchanged.

Each board's composite is a plain weighted sum with no curve, no normalisation against a moving cohort mean and no editorial adjustment. Take the subscores from any product page, multiply each by its board's published weight, divide by one hundred, and you get the number in the index column to two decimal places. Where two products on the same board sit within about one and a half index points of each other, treat them as tied: five runs is enough for a usable median and not enough for a confident ordering at that distance. When a number here is wrong, we would rather hear it with evidence and correct it in public, which has already happened twice. The whole dataset, including every individual run, is available under a CC BY 4.0 licence for anyone who wants to check the arithmetic or disagree with the weights. Derived, not separately measured. It is nine times the median prompt latency, converted to minutes. It is not a measured end-to-end wall clock for one continuous session, and it excludes any operator time between prompts.