methodology / 09 Aug 2026 / 1 min read
How we score agent performance without reading the chat log
Agent performance is 18 points of the index, the joint largest weight. Here is exactly how a prompt earns 0, 50 or 100.
By Naomi Kessler | Updated 19 Aug 2026
The rule
Each of the 9 prompts has a published pass criterion. A prompt execution scores 100 if the artifact meets the criterion with no operator intervention, 50 if it meets part of it or required one clarifying instruction, and 0 otherwise.
Reviewers work from the artifact and the diff. They do not read the conversation. A product that explains what it is going to do beautifully and then does not do it scores zero, which is the correct outcome and the reason we do not score transcripts.
Two reviewers, blind to the product
Screenshots and diffs are stripped of product branding before review. Two reviewers score each execution independently. Where they disagree by a full band the execution is re reviewed by a third.
What counts as intervention
Clicking a deploy button the product presents is not intervention. Answering a question the agent asks is not intervention. Telling the agent it forgot to sign the webhook is intervention, and it caps that execution at 50.
Why not just count tests passing
We tried it. Automated criteria reward products that write tests for what they built rather than what was asked, and they miss the failures that matter most, like a tenancy boundary that leaks only under a specific read path. The pass criteria are written to be checked by a human in under two minutes each, and the check itself is published on the methodology page so you can repeat it.