analysis / 03 Aug 2026 / 1 min read
What this benchmark cannot tell you
A composite index is a compression of a complicated thing into one number. Here are the limits of ours, stated plainly.
By Priya Anand | Updated 19 Aug 2026
One specification is one specification
vibeOps is a multi tenant SaaS with an API, webhooks, background jobs and an admin panel. It is representative of a large class of real work and it is not representative of all work. A product that scores poorly here may be excellent at building a marketing site, a data dashboard or a mobile front end.
Read the axis subscores rather than the composite if your work looks different from the specification. That is why every subscore is published separately.
Five runs is five runs
Five runs per product gives us a usable median and a rough sense of spread. It does not give us tight confidence intervals. Where two products are within about one and a half index points of each other, treat them as tied.
Models move underneath the products
Most of these products are wrappers around frontier models that change without notice. A result from August is a measurement of August. This is why every product page carries the last run date and the next scheduled run date, and why we keep the historical runs rather than overwriting them.
We do not measure support, roadmap or company health
None of those fit in a reproducible harness, and all of them matter when you are choosing where to build. Use the numbers for what they are good for, which is narrowing the field, then run your own trial on the two or three that survive.