Methodology / cycle 04 / published before the run
Prompts per run
9
Fixed order, verbatim, no coaching
Runs per product
5
Full sequence, repeated from empty
Executions this cycle
1,845
All published individually
Weights total
100
Sums to exactly 100 by construction
The composite index is a weighted sum of these ten subscores. No curve, no cohort normalisation, no editorial adjustment.
| Axis | Key | Weight | % | What it means |
|---|---|---|---|---|
| Agent performance | agent_perf | 18 | How much of each prompt the agent actually delivered without the operator intervening, correcting or re prompting. | |
| Reliability | reliability | 18 | How often the product completed a prompt cleanly rather than erroring, stalling, looping or producing a build that does not start. | |
| Scalability | scalability | 13 | Whether the generated system holds up beyond the demo: query shape, N+1 behaviour, pagination, background work and multi tenant isolation. | |
| SEO and GEO | seo_geo | 12 | Whether the public surface the product generates can be crawled by a search engine and cited by a language model. | |
| API and MCP | api_mcp | 9 | The quality of the machine facing surface: REST design, authentication, rate limiting, webhook signing and Model Context Protocol support. | |
| Integrations | integrations | 8 | First party reach into the services a production SaaS needs: payments, transactional email, object storage, queues, analytics and identity providers. | |
| Design output | design | 7 | Visual quality of what the product produces unprompted, both the admin surface and the public pages. | |
| Speed | speed | 7 | Wall clock time from submitting a prompt to a deployed, reachable artifact. | |
| Value | value | 5 | Index points delivered per euro of entry price for a single working developer. | |
| Code ownership | code_ownership | 3 | Whether you can take the generated system and run it yourself, without the vendor. | |
| Total | sum | 100 | Bounded at 0 and 100 by construction |
vibeOps is a multi tenant SaaS with a public REST API, outbound webhooks, background jobs and an internal admin panel. It is deliberately not a landing page: it forces every product to touch data modelling, authentication, async work and an operations surface.
| # | Target | Prompt as issued | Pass criterion |
|---|---|---|---|
| P01 | Tenant and user model | Create a multi tenant data model with organisations, users, memberships and roles owner, admin and member. Every record must carry an organisation identifier. Seed two organisations with three users each. | All four entities exist, membership carries a role, and reads are constrained by organisation identifier in the generated queries. |
| P02 | Authentication and session | Add email and password authentication with server side sessions, a sign in page, a sign out action and route protection for every page under /app. Unauthenticated requests must redirect to the sign in page. | A protected route returns a redirect for an anonymous request and renders for an authenticated session. |
| P03 | Projects CRUD with tenancy | Add a projects entity scoped to an organisation with create, read, update, archive and a paginated list at 25 rows per page, sorted by updated date descending. A member of organisation A must never read a project of organisation B. | Cross tenant read attempts return 403 or 404 and the list endpoint paginates correctly at the boundary. |
| P04 | Public REST API with keys | Expose a versioned REST API at /api/v1 for projects with API key authentication, per key rate limiting of 60 requests per minute, and consistent JSON error envelopes carrying a machine readable code. | Requests with a missing, revoked or foreign key are rejected, and the rate limiter returns 429 with a retry hint. |
| P05 | Outbound webhooks with signatures | Emit project.created, project.updated and project.archived webhooks to per organisation endpoints. Sign every payload with an HMAC SHA256 header, retry failures with exponential backoff up to five attempts and store the full delivery history. | Signature verifies against the stored secret, and a forced 500 on the receiver produces the documented retry ladder. |
| P06 | Background jobs | Add a background job queue with a nightly usage rollup per organisation and an on demand export job that writes a CSV. Jobs must be idempotent, must record start and finish timestamps and must survive a redeploy without losing queued work. | Re running the same job twice produces one result set, and queued work is still present after a restart. |
| P07 | Admin panel | Build an internal admin panel at /admin restricted to staff accounts, with organisation search, per organisation usage, webhook delivery inspection with the raw payload, and a manual job replay control. | Non staff accounts are refused, and a failed delivery can be replayed and shows the new attempt in the history. |
| P08 | Public marketing surface | Add a public marketing home page, a pricing page with three tiers and a documentation page for the API. Every public page must render its full content server side and must be readable with JavaScript disabled. | Fetching the page with scripts blocked returns the full text content and a title, description and canonical link. |
| P09 | Hardening pass | Add request validation on every endpoint, structured error logging with a correlation identifier, a health endpoint reporting database connectivity, and a seed script that recreates the entire demo dataset from empty. | Invalid payloads return 400 with field level detail, the health endpoint reflects a broken database, and the seed script runs clean twice. |
Prompts are issued in this order with no rephrasing. A clarifying question from the agent gets a factual answer. Telling an agent what it forgot counts as intervention and caps that execution at a partial result.
One block per axis, in weight order.
How much of each prompt the agent actually delivered without the operator intervening, correcting or re prompting.
Method
Each of the 9 prompts is scored 0, 50 or 100 against its published pass criteria by two reviewers working from the artifact and the diff, not from the chat transcript. Disagreements are re reviewed. The axis score is the mean across all 45 prompt executions.
How often the product completed a prompt cleanly rather than erroring, stalling, looping or producing a build that does not start.
Method
Strict pass rate over all 45 recorded prompt executions per product. Partial completions and outright failures both cost. Timeouts above 20 minutes are recorded as a fail with the note preserved.
Whether the generated system holds up beyond the demo: query shape, N+1 behaviour, pagination, background work and multi tenant isolation.
Method
The prompt 3 and prompt 6 artifacts are loaded with 50000 projects across 200 organisations. We record list endpoint latency at p95, the number of queries per request and whether tenant isolation still holds under concurrent writes.
Whether the public surface the product generates can be crawled by a search engine and cited by a language model.
Method
We fetch the generated marketing page with JavaScript disabled and record the HTML byte count of real text, then measure LCP and CLS on a throttled connection. We also check for a title, meta description, canonical, sitemap, robots policy and any structured data emitted without client rendering.
The quality of the machine facing surface: REST design, authentication, rate limiting, webhook signing and Model Context Protocol support.
Method
Scored against the prompt 4 and prompt 5 pass criteria with a fixed 22 point checklist covering versioning, error envelopes, idempotency keys, key revocation, HMAC signing, retry ladders and delivery history. MCP support is verified by connecting a client and listing tools.
First party reach into the services a production SaaS needs: payments, transactional email, object storage, queues, analytics and identity providers.
Method
Counted from the product documentation on the fetch date, then verified by wiring two integrations live. Documented but non functional connectors score zero. Generic HTTP request blocks are not counted as integrations.
Visual quality of what the product produces unprompted, both the admin surface and the public pages.
Method
Blind review of the prompt 7 and prompt 8 artifacts by three reviewers who do not know which product produced which screenshot. Scored on hierarchy, typographic control, spacing discipline, state coverage and responsive behaviour at 390, 768 and 1440 pixels.
Wall clock time from submitting a prompt to a deployed, reachable artifact.
Method
Timed by the harness, not by the product interface, from request submission to a 200 response on the deployed URL. We publish p10, median and p90 across all recorded executions. The subscore is the fastest median in the cohort divided by the product median, expressed as a percentage.
Index points delivered per euro of entry price for a single working developer.
Method
The cheapest plan that can complete the whole vibeOps specification, including any credit top up needed to finish the 5 runs, normalised across the cohort. Free tiers that cannot finish the specification are recorded at the price of the first plan that can.
Whether you can take the generated system and run it yourself, without the vendor.
Method
We export the artifact, remove all vendor specific configuration and attempt a clean install and boot on a plain container. Full marks require complete source export, an open runtime, no proprietary runtime dependency and a documented self host path.
Take the ten subscores from any product page, multiply each by its weight from the table above, divide the sum by one hundred and round to two decimals. That is the index. There is no other step. A worked example is published in the lab notes.
Start a monotonic clock when the prompt is submitted. Stop it on the first 200 response from the deployed URL. Do not read the timer the product displays, because those timers differ in what they include. Queue time counts, cold start counts, and if the product reports completion while the URL is not serving, the clock keeps running. Repeat the full nine prompt sequence five times from an empty project.
Each prompt has one published pass criterion. Check it against the artifact and the diff, not the chat transcript. Score 100 for a clean pass, 50 where part of the criterion is met or one clarifying instruction was needed, and 0 otherwise. Two reviewers score independently, blind to which product produced the artifact, and a full band disagreement goes to a third.
Fetch the generated public marketing page with scripts disabled, strip tags, count the bytes of remaining text and record that as the no JavaScript HTML weight. Measure LCP and CLS on a throttled connection with a cold cache. Check for a title, meta description, canonical link, robots policy, sitemap and any structured data present in the first response.
Every figure on this site is downloadable. Rankings are at /api/rankings.json and /data/rankings.csv, and every individual execution is at /api/runs.json. All three carry an explicit CC-BY-4.0 licence field, so you may republish them with attribution to LLM Tier.
No product can pay to be measured, re measured, promoted or removed. There are no sponsored rows, no paid placements and no affiliate links in the index tables. The lab is funded by data licensing and consulting on measurement design, never by the products it measures. Every row is produced by the same harness on the same specification, so no product receives different treatment from any other.
Send evidence to lab@llm-tier.com. If a measurement is wrong we correct it, mark the correction on the product page and keep the original value in the record. Two corrections have been issued since the index launched.