Why we publish the spread and not the mean

The mean is the least useful number we collect

Every vendor benchmark we have read reports one build time. One number, usually a good one, usually with no sample size next to it. That number tells you almost nothing, because the distribution of agent build times is not symmetric. It has a long right tail, and the right tail is where your afternoon goes.

So we publish three numbers for every product: the tenth percentile, the median and the ninetieth percentile of wall clock seconds per prompt, measured across 45 recorded executions.

What the spread shows

The fastest median in the current cohort is 68 seconds. The slowest is 148. That ratio is a little over two to one, and if you stopped there you would conclude that the fast products are twice as good on speed.

Look at p90 instead and the picture changes. Some products are tight: their p90 sits within 60 percent of their median, so planning around them is easy. Others have a p90 more than three times their median, which means one execution in ten takes long enough that you will switch tasks and lose the context you were holding.

How the timing is taken

The harness starts a monotonic clock when the prompt is submitted and stops it on the first 200 response from the deployed URL. We do not read the timer the product shows in its own interface, because those timers vary in what they include. Queue time counts. Cold start counts. If the product told us it was finished and the URL was not serving, the clock kept running.

What we do with it

The speed axis is worth 7 points of the composite index, and it uses the median rather than the mean, so one pathological run cannot move a product up or down the table. The spread is published next to it so you can make your own judgement about variance, which is a different question from speed and deserves its own column.

Every duration behind these figures is downloadable as JSON at /api/runs.json under a CC BY licence.