The Evaluation Ladder describes five levels of rigour in how a team proves its AI system works. Level 1 is vibes. Level 2 is spot checks. Level 3 is a fixed test set. Level 4 is a calibrated judge. Level 5 is evaluation running in continuous integration.
Level 3 is the first level an enterprise buyer will accept as evidence, and the level most teams skip on their way to building an automated judge that confidently measures the wrong thing.
Almost every stalled AI pilot I have been asked to look at had the same thing missing, and it was not a better model. Nobody could say what "working" meant, in writing, before the build started.
That sounds like a small omission. It is the whole problem. Without a written definition of correct, a pilot cannot be defended to a finance committee, cannot be compared against the previous release, and cannot be handed to a risk function. It can only be demonstrated, and a demonstration convinces exactly the people who were already convinced.
This is the ladder I use to find out where a team actually is. Most people put themselves a level higher than the evidence supports.
The five levels
Vibes
Someone on the team tries it and says it feels good. Somebody else tries it and disagrees. The argument is settled by seniority.
Cost: nothing. Value: nothing you can defend to anyone outside the room. This is where most teams genuinely are, including teams who would tell you otherwise.
Spot checks
Ten or twenty examples pasted into a spreadsheet before each release, looked at by whoever is free that afternoon.
Better than nothing, and genuinely useful for catching catastrophes. But the examples change every time, so there is no baseline, and you cannot tell an improvement from noise.
A fixed test set
50 to 200 real traces from your own system, hand-labelled once against a written definition of correct, and re-run unchanged on every release.
This is the first level at which you have evidence rather than opinion, and the first level an enterprise buyer's risk function will accept. It is also boring, manual, and unglamorous, which is precisely why it gets skipped.
Tools: a spreadsheet is genuinely enough to start.
A calibrated judge
A model scores the outputs automatically, and you have measured how often it agrees with your human labels. Without that measurement it is not a judge, it is a second opinion of unknown quality.
Ask for binary verdicts rather than scores out of ten. Nobody, human or model, can calibrate a 7.
Tools: Braintrust, LangSmith, Langfuse, Arize, or a script you write yourself.
Evals in CI
A regression fails the build before a customer sees it. The evaluation set is a test suite, versioned with the code, and changing it requires a review like any other change.
Tools: your existing CI, plus the judge from level 4.
The mistake almost everybody makes
Teams jump from level 1 to level 4.
It is an understandable jump. Level 4 is the fun one. It is automated, it produces a dashboard, and there are good products that will sell it to you this week. Level 3 is a person reading two hundred of their own outputs and deciding, one at a time, whether each was right.
But a judge has to be calibrated against something, and the only thing available to calibrate it against is human labels. Skip level 3 and you get a judge that is precise, fast, automated, and measuring something nobody cares about. Worse, it now produces a number, and numbers get quoted in board papers.
You cannot calibrate a judge without labels. You cannot get labels without reading your own production data.
The reading is the point. Almost every team that sits down to label two hundred real traces discovers a failure mode they did not know they had, within the first thirty.
Accuracy is a vanity metric
The other common trap is at the top of the ladder rather than the bottom. A team gets to level 4, produces a single headline accuracy figure, and puts it on a slide.
Accuracy averages across cases that carry wildly different consequences. Consider a support agent that handles three kinds of request: answering a policy question, updating a record, and issuing a credit. Suppose it is 94 percent accurate overall. That number is meaningless until you know how the 6 percent is distributed.
| Decision type | Share of volume | Accuracy | Cost of being wrong |
|---|---|---|---|
| Answer a policy question | 70% | 97% | Low. Customer re-asks. |
| Update a record | 25% | 92% | Medium. Reversible, needs a human. |
| Issue a credit | 5% | 78% | High. Money leaves. Not reversible. |
Scroll the table sideways to compare →
The blended figure is about 94 percent, and it is the least useful number on the page. The real finding is that the highest-consequence decision is the least reliable one, which is an autonomy decision rather than a modelling one: that row should not be automated yet.
Measure per decision type, weighted by what it costs to be wrong. A single accuracy number is a metric designed to be reported rather than used.
How to climb one level this month
If you are at level 1 or 2, the whole of the next step is level 3, and it is a week of work rather than a quarter.
- Write down what correct means. One paragraph per decision type, specific enough that two people labelling separately would agree. If they would not agree, the definition is not finished.
- Pull 100 real traces. From your own pilot or production traffic, not synthetic examples. Include the ugly ones. Especially the ugly ones.
- Label them by hand, once. Someone who understands the domain, not whoever has capacity. Budget a day.
- Re-run the set on every release, unchanged. The moment you start editing the set to make the numbers look better, you are back at level 2.
- Only then build a judge. And measure its agreement with your labels before you trust a single one of its scores.
Common questions
How many examples do I actually need?
50 to 200 real traces is enough to start and far more useful than thousands of synthetic ones. What matters is that they come from your own traffic, cover the failure modes you actually see, and are labelled by someone who understands the domain. You can grow the set later; you cannot retrofit relevance.
Do I need an evaluation platform?
Not to start. Level 3 works in a spreadsheet, and starting there means you learn what you are measuring before you buy a tool to measure it. Platforms earn their place at level 4 and 5, when you are running a judge on every release and want the history, tracing and CI hooks.
Our outputs are open-ended text. How do we label that?
Do not score quality on a scale. Decompose it into binary questions you can actually answer: did it use the right source, did it answer the question that was asked, did it state anything unsupported, did it stay within policy. Four binary checks are more reliable and more actionable than one score out of ten.
What if the correct answer is genuinely ambiguous?
Then that is your finding, and it is an important one. Ambiguity in the label means ambiguity in the product decision. Either tighten the definition of correct, or accept that this decision type should not be automated, which is a legitimate and often correct outcome.
Can I use this framework in my own work?
Yes. Use it, teach it, put it in your internal docs. Attribution to AIAUTOMIC is appreciated and not required.
The short version
Your pilot did not fail. It was never defined well enough to succeed.
Level 3 is boring and it is where evidence starts. Everything above it is leverage on work you have not done yet.
Related frameworks
The 8-Layer Agentic Stack places evaluation in context: it is part of layer eight, governance, and it is the layer that decides whether anything ships.
Fractional CPO vs consultancy vs full-time hire covers who owns evaluation, which is a product decision more often than an engineering one.