Full-stack app builders / rank 15 of 20 / cycle 04
Caffeine benchmark
The only product in the index that deploys to a decentralised runtime, which makes it the hardest to compare fairly. Data durability and tenancy were strong, and the application kept running with no server to maintain. Everything that assumes a conventional web stack, from the crawlable page test to the outbound webhook prompt, either scored low or did not apply, and the first paint numbers are the worst in the cohort.
Composite index
62.08
of 100 | withheld vs. cycle 03
Delta not shown for cycle 04: the roster was re-based from one merged ranking into two class boards, so cycle 03 and cycle 04 ranks are not on the same scale. Resumes cycle 05.
At a glance
Composite index
62.08
rank 15 of 20
Rank
#15
of 20 on the full-stack app builders board
Median turn latency
150.6 s per prompt
22.6 min across the full nine prompt spec, derived
Reliability
66.7%
strict pass rate
SEO score
48.0
0 to 100, SEO and GEO axis
Three strongest axes
- Value84.0
- Scalability74.0
- Agent performance70.0
Three weakest axes
- Turn latency score43.3
- Integrations46.0
- SEO and GEO48.0
Median turn latency
150.6 s per prompt
p10 76.0 s, p90 295.5 s
Strict pass rate
66.7%
30 pass, 12 partial, 3 fail
Executions recorded
45
5 runs of 9 prompts
Entry price
USD 5 / month
Vendor's own unit and billing period. Never a currency conversion
Per axis subscores
Composite recomputed from these 10 rows: 62.08. That is the arithmetic behind the index, and you can repeat it by hand. Rounding rule: half up at two decimals, applied uniformly to all 37 published products.
| Axis | Subscore 0 to 100, tick is the roster median | Score | Median | Weight % | Contribution |
|---|---|---|---|---|---|
| Agent performance | 70.0 | 71.5 | 18 | 12.60 | |
| Reliability | 66.7 | 71.1 | 18 | 12.01 | |
| Scalability | 74.0 | 66.0 | 13 | 9.62 | |
| SEO and GEO | 48.0 | 63.5 | 12 | 5.76 | |
| API and MCP | 52.0 | 58.0 | 9 | 4.68 | |
| Integrations | 46.0 | 65.0 | 8 | 3.68 | |
| Design output | 68.0 | 78.5 | 7 | 4.76 | |
| Turn latency score | 43.3 | 66.5 | 7 | 3.03 | |
| Value | 84.0 | 74.0 | 5 | 4.20 | |
| Code ownership | 58.0 | 73.0 | 3 | 1.74 |
Turn latency distribution
Every recorded prompt duration for Caffeine, in seconds. Median wall clock time for one prompt, not for a whole build. The cyan tick on the whisker is the median.
Spread against the cohort
p10 76.0 s / median 150.6 s / p90 295.5 s
Scale is shared with every product in the cohort, so the bars are directly comparable between pages. Ratio of p90 to median is 1.96x.
Outcome split
30 pass / 12 partial / 3 fail
SEO and GEO measurements
Taken from the public marketing page this product generated in prompt 8, fetched with JavaScript disabled.
HTML bytes, no JS
6.0 KB
6,144 bytes of real text content with scripts blocked
LCP
2,840 ms
Throttled connection, cold cache
CLS
0.110
Cumulative layout shift on load
SEO and GEO subscore
48.0
0 to 100, worth 12 index points on this board
Raw run data: 45 executions
One row per prompt execution. Nothing is aggregated away, nothing is hidden, including the failures.
| Run | Prompt | Prompt target | Started | Duration (s) | Outcome | Note |
|---|---|---|---|---|---|---|
| R01 | P01 | Tenant and user model | 11 Aug 2026 08:00 UTC | 85.5 | pass | |
| R01 | P02 | Authentication and session | 11 Aug 2026 08:03 UTC | 70.9 | pass | |
| R01 | P03 | Projects CRUD with tenancy | 11 Aug 2026 08:06 UTC | 120.6 | pass | |
| R01 | P04 | Public REST API with keys | 11 Aug 2026 08:10 UTC | 407.6 | partial | Pagination off by one at the final page boundary. |
| R01 | P05 | Outbound webhooks with signatures | 11 Aug 2026 08:19 UTC | 186.3 | partial | Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand. |
| R01 | P06 | Background jobs | 11 Aug 2026 08:24 UTC | 207.2 | pass | |
| R01 | P07 | Admin panel | 11 Aug 2026 08:29 UTC | 114.7 | pass | |
| R01 | P08 | Public marketing surface | 11 Aug 2026 08:33 UTC | 91.1 | pass | |
| R01 | P09 | Hardening pass | 11 Aug 2026 08:37 UTC | 197.2 | pass | |
| R02 | P01 | Tenant and user model | 12 Aug 2026 08:00 UTC | 53.3 | pass | |
| R02 | P02 | Authentication and session | 12 Aug 2026 08:02 UTC | 147.1 | partial | Error envelope inconsistent between two of the endpoints. |
| R02 | P03 | Projects CRUD with tenancy | 12 Aug 2026 08:07 UTC | 149.7 | pass | |
| R02 | P04 | Public REST API with keys | 12 Aug 2026 08:11 UTC | 265.9 | pass | |
| R02 | P05 | Outbound webhooks with signatures | 12 Aug 2026 08:18 UTC | 228.2 | partial | Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand. |
| R02 | P06 | Background jobs | 12 Aug 2026 08:24 UTC | 371.2 | pass | |
| R02 | P07 | Admin panel | 12 Aug 2026 08:32 UTC | 162.5 | pass | |
| R02 | P08 | Public marketing surface | 12 Aug 2026 08:36 UTC | 103.5 | partial | Validation added on the write path only. Read path left unguarded. |
| R02 | P09 | Hardening pass | 12 Aug 2026 08:40 UTC | 129.1 | partial | Crawlable page test scored low: the deployed front end renders entirely on the client. |
| R03 | P01 | Tenant and user model | 13 Aug 2026 08:00 UTC | 93.7 | pass | |
| R03 | P02 | Authentication and session | 13 Aug 2026 08:03 UTC | 131.6 | pass | |
| R03 | P03 | Projects CRUD with tenancy | 13 Aug 2026 08:07 UTC | 185.2 | pass | |
| R03 | P04 | Public REST API with keys | 13 Aug 2026 08:12 UTC | 442.1 | pass | |
| R03 | P05 | Outbound webhooks with signatures | 13 Aug 2026 08:22 UTC | 74.9 | partial | Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand. |
| R03 | P06 | Background jobs | 13 Aug 2026 08:25 UTC | 163.5 | partial | Delivered the main path, missed the boundary case named in the pass criteria. |
| R03 | P07 | Admin panel | 13 Aug 2026 08:30 UTC | 138.1 | pass | |
| R03 | P08 | Public marketing surface | 13 Aug 2026 08:34 UTC | 77.6 | partial | Pagination off by one at the final page boundary. |
| R03 | P09 | Hardening pass | 13 Aug 2026 08:37 UTC | 55.8 | pass | |
| R04 | P01 | Tenant and user model | 14 Aug 2026 08:00 UTC | 64.6 | pass | |
| R04 | P02 | Authentication and session | 14 Aug 2026 08:03 UTC | 153.5 | fail | Agent looped on the same edit for the full timeout window of 20 minutes. |
| R04 | P03 | Projects CRUD with tenancy | 14 Aug 2026 08:07 UTC | 135.8 | pass | |
| R04 | P04 | Public REST API with keys | 14 Aug 2026 08:11 UTC | 150.6 | pass | |
| R04 | P05 | Outbound webhooks with signatures | 14 Aug 2026 08:16 UTC | 154.8 | partial | Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand. |
| R04 | P06 | Background jobs | 14 Aug 2026 08:20 UTC | 170.9 | fail | Task abandoned by the agent with no artifact produced. |
| R04 | P07 | Admin panel | 14 Aug 2026 08:25 UTC | 224.4 | pass | |
| R04 | P08 | Public marketing surface | 14 Aug 2026 08:31 UTC | 204.8 | pass | |
| R04 | P09 | Hardening pass | 14 Aug 2026 08:36 UTC | 78.7 | partial | Crawlable page test scored low: the deployed front end renders entirely on the client. |
| R05 | P01 | Tenant and user model | 15 Aug 2026 08:00 UTC | 106.0 | pass | |
| R05 | P02 | Authentication and session | 15 Aug 2026 08:03 UTC | 161.4 | pass | |
| R05 | P03 | Projects CRUD with tenancy | 15 Aug 2026 08:08 UTC | 184.9 | pass | |
| R05 | P04 | Public REST API with keys | 15 Aug 2026 08:13 UTC | 193.6 | pass | |
| R05 | P05 | Outbound webhooks with signatures | 15 Aug 2026 08:18 UTC | 124.0 | partial | Outbound calls are constrained by the runtime. Delivery succeeded through a gateway, signing was added by hand. |
| R05 | P06 | Background jobs | 15 Aug 2026 08:22 UTC | 378.7 | pass | |
| R05 | P07 | Admin panel | 15 Aug 2026 08:31 UTC | 315.2 | pass | |
| R05 | P08 | Public marketing surface | 15 Aug 2026 08:38 UTC | 191.8 | fail | Generated migration conflicted with the existing schema and was not reconciled. |
| R05 | P09 | Hardening pass | 15 Aug 2026 08:43 UTC | 100.8 | pass |
Pricing
Four monthly plans: Free at USD 0, Host at USD 5, Studio at USD 25 with a USD 10 first month, and Business at USD 250. Each carries a fixed monthly credit allowance rather than open ended usage billing, at 15 welcome credits on the free plan, then 20, 100 and 1000 a month. The free plan remixes existing app market apps only, so USD 5 is the entry price for building from scratch. Annual billing is offered at a stated 20 percent discount. The plan buys capacity on a decentralised runtime, so there is no separate server to size or maintain alongside it.
Pricing note
Every figure on caffeine.ai/pricing is a bare dollar sign, and the page names no currency at all. The USD label here is this lab's reading of that symbol rather than a vendor statement, and the euro figure is converted on the same basis.
- Entry price
- USD 5 / month
- Priced from
- vendor pricing page
- Value subscore
- 84.0
- Index points per euro
- 14.54
Entry price is the vendor's own published figure, fetched 02 Sep 2026, never a currency conversion. The currency label is the vendor's own except where the pricing note above records that this lab inferred it. Index points per euro is computed from a separate, euro-normalised figure used only for the Value axis: Converted at the European Central Bank euro reference rate of 1.1699 US dollars to the euro, August 21, 2026.
Capabilities as measured
- stack
- Canister backend with a generated web front end
- deploy
- Deployed to a decentralised runtime
- database
- On chain persistent storage
- auth
- Cryptographic identity
- background jobs
- Timers, no external queue
- webhooks
- Outbound is constrained by the runtime
- rest api
- Gateway exposed
- mcp server
- Yes, vendor hosted at mcp.caffeine.ai, OAuth sign in, published as alpha
- export code
- Yes, source is portable
- custom domain
- Yes
- ssr
- No
Screenshot
Caffeine landing page, captured by the lab on 14 Aug 2026 at 1440 by 900. Stored locally, never hotlinked.
Caffeine, captured 14 Aug 2026.
Sources and fetch dates
Every claim on this page traces to one of these.
| Source | URL | Fetched |
|---|---|---|
| Caffeine pricing page | https://caffeine.ai/pricing | 14 Aug 2026 |
| Caffeine product site | https://caffeine.ai | 14 Aug 2026 |
| LLM Tier run harness output | https://www.llm-tier.com/api/runs.json | 14 Aug 2026 |
| Caffeine llms.txt capability index | https://caffeine.ai/llms.txt | 04 Sep 2026 |
Rich results carry this composite as 3.1 on a 0 to 5 scale because that is the only scale the format accepts. The number on this page, 0 to 100, is the real one.
Reader index
Voted by readers with an account, one score per person per axis. Published beside the lab index, never inside it.
Reader index
not rated yet
Lab index
62.08
No reader has scored Caffeine yet. The first 3 voters put the column on the board.
Seeded at launch
The reader scores, lab notes and open ratings published here were written by this lab in August 2026 to populate the reader column before the site had readers. They are not customer submissions, they are not vendor supplied and nobody was paid for them. Anything submitted since then comes from a registered account. The lab index is measured from logged runs and is unaffected either way.
The reader index is a weighted mean of reader submitted axis scores using the published index weights, renormalised over the axes readers have scored. It is never blended into the lab index. How this works.
Open rating mean
n/a
0 published open ratings, submitted without an account
Lab index
62.08
Measured, not voted
Gap to the measurement
n/a
Positive means open raters put it above the measurement
What agent performance looked like in practice
The only product in the index that deploys to a decentralised runtime, which makes it the hardest to compare fairly. Data durability and tenancy were strong, and the application kept running with no server to maintain. Everything that assumes a conventional web stack, from the crawlable page test to the outbound webhook prompt, either scored low or did not apply, and the first paint numbers are the worst in the cohort.
Across 45 recorded executions this product met the published pass criterion cleanly 30 times, partially 12 times and failed 3 times. The rows above carry the reviewer note for every non clean outcome, which is the only way to tell a product that fails loudly from one that fails quietly.