Reliability

Full-stack app builders. Strict pass rate over all 45 recorded prompt executions per product. Partial completions and outright failures both cost. Executions running past 20 minutes are flagged for manual review and scored on the outcome the reviewer recorded; the duration is always retained and always counts toward the turn latency percentiles.

Coding agents. Strict pass rate over all 45 recorded prompt executions per product. Partial completions and outright failures both cost. Executions running past 20 minutes are flagged for manual review and scored on the outcome the reviewer recorded; the duration is always retained and always counts toward the turn latency percentiles.

This axis carries 18 of the 100 index points on the builders board and 18.56 of the 100 index points on the coding agent board. Each product completed 5 runs of 9 prompts in this cycle, and the underlying per run data behind every figure on this page is downloadable at /api/runs.json under a CC BY 4.0 licence.

A high score on one axis is not a recommendation. Read it against the composite column on the right and against the axes that matter for the work you have in front of you. Reliability at 93.3% for the leader of the builders cohort is the context for everything else on the page.