AI app builder benchmark

Most published comparisons of AI app builders are written from a free trial and a screenshot. Someone signs up, types a prompt about a to do list, likes what appears, and writes five hundred words about the future of software. That process cannot separate a product that is genuinely good from a product that is good at the first ninety seconds, and it produces rankings that change with the mood of the writer rather than with the behaviour of the software. This site exists because we wanted the other kind of comparison: one specification, applied identically, repeated enough times to see variance, with every measurement published so the conclusion can be checked.

The specification is called vibeOps. It describes a multi tenant SaaS product: a public REST API with key based authentication and rate limiting, outbound webhooks that are signed and retried, background jobs that survive a redeploy, an internal admin panel gated to staff accounts, and a public marketing surface that has to render without JavaScript. It is deliberately not a landing page and not a to do list. It is the shape of work people actually pay for, and it forces every product to touch data modelling, authentication, asynchronous work and an operations surface rather than just layout.

We issue 9 prompts in a fixed order. Each prompt is phrased precisely, in the language a working engineer would use, and each has a published pass criterion written before the run. The criteria are concrete rather than aesthetic: a cross tenant read must be refused, a rate limiter must return 429 with a retry hint, a webhook signature must verify against the stored secret, a job must be idempotent when replayed and must still be queued after a restart. The prompts are never rephrased to help a product that is struggling, and a product that asks a clarifying question gets an answer without being told what it forgot.

Every product runs the full sequence 5 times, which gives 45 recorded executions per product and 1,845 in this cycle. Repetition is the part most benchmarks skip, and it is the part that matters most, because these systems are not deterministic. The same prompt to the same product on the same day can produce a clean result once and a stalled agent the next time. Publishing a single build time from a single lucky run is not a measurement, it is an anecdote. So we publish the tenth percentile, the median and the ninetieth percentile of wall clock time, and we publish the outcome of every individual execution as pass, partial or fail with the note the reviewer wrote at the time.

Scoring uses ten axes. Agent performance and reliability carry the most weight, 18 points each, because the two questions that decide whether one of these products is usable are how much of the work it actually did and how often it did it without breaking. Scalability follows at 13, measured by loading the generated system with fifty thousand records across two hundred organisations and watching what happens to query counts and tenant isolation. SEO and GEO carries 12, which is higher than a traditional review would give it, because the public pages these tools generate decide whether the thing you build can be found by a search engine or cited by a language model at all. We fetch the generated marketing page with scripts disabled and count the bytes of real text: across the current cohort that number ranges by nearly seven to one.

The remaining weight goes to the machine facing surface at 9 points, first party integrations at 8, blind reviewed design output at 7, speed at 7, value at 5 and code ownership at 3. Speed is worth less than most readers expect. A product that reaches a deployed URL in sixty seconds and leaves you to write the queue yourself has not saved you an afternoon. Code ownership is worth only 3 points but it is the axis that decides whether a project can outlive its vendor, so we publish it separately rather than folding it into a vague platform score.

The composite is a plain weighted sum with no curve, no normalisation against a moving cohort mean and no editorial adjustment. Take the ten subscores from any product page, multiply each by its published weight, divide by one hundred, and you get the number in the index column to two decimal places. Where two products sit within about one and a half index points of each other, treat them as tied: five runs is enough for a usable median and not enough for a confident ordering at that distance. When a number here is wrong, we would rather hear it with evidence and correct it in public, which has already happened twice. The whole dataset, including every individual run, is available under a CC BY 4.0 licence for anyone who wants to check the arithmetic or disagree with the weights.