Evals are a Compass
A quality gate tells you if you passed. The better question: where are you going?
The natural mental model for AI evals is unit tests: given some contexts and queries, a model must produce certain outputs. Keeping these “green” in continuous integration is the whole game. This is a useful way to manage quality, and there are still a lot of teams who aren’t doing it. According to Inngest’s AI in Production Report, “35% of respondents building AI in production are not doing AI evals at all.”
The “Green CI” mode of evals is a great first step, but if you stop there, you are leaving an actionable signal on the table.
Measurements vs. Verdicts
A unit test returns a verdict: pass or fail, and fail means something is broken. That’s the right model for most software: there’s a correct answer, and anything else is a defect.
Most of what we want to know about an AI product doesn’t have that shape, though. Consider a code review tool aimed at security. The question isn’t “did it pass,” it’s “when a diff introduces a SQL-injection-class vulnerability, how often does the reviewer catch it?” The answer is a number, and most of the time it isn’t green (perfect) or red (broken). It’s a measurement telling you where your product currently sits on an axis you care about.
Accuracy rate isn’t the only metric available. It might be task duration, costs, escalations to human reviewers, or other non-binary quality measures.
We hear about the benchmarks frontier labs are using to evaluate their models, but you can apply the same concept to your domain. How are you benchmarking your agents?
Where Are We Going?
A single performance measurement is a snapshot: reviewer agent identified 68% of vulnerabilities. It’s interesting, and if you peel back the logs you can get a sense for where your failures are. But it gets more interesting when you do something and measure again.
Run the same benchmark after you change the prompt, and it’s 74%. Swap the underlying model, and it falls to 71%. Add a retrieval step, and it drops to 66%, telling you more context isn’t the answer. The measurements are directional: their movement tells you whether you’re building toward the product you want.
A verdict can’t give you this. Green-to-green tells you nothing about whether you improved. And worse, the eval-as-gate disposition will incentivize you to write evals for only the most certain cases. Aspirational cases are to be avoided, because they’re “flaky”.
It’s even more powerful when you layer multiple performance metrics. The model swap that saved cash drove escalations up. It allows you to make tradeoffs intentionally instead of guessing.
The Suite Life
Building a good suite is important. You want to ensure coverage across a range of scenarios, including some that are aspirational. This allows you to capture performance tradeoffs, and get a handle on the model’s jagged edges.
If the reviewer agent identifies 68% across all vulnerabilities, that’s a useful average, but there’s more structure. Break the benchmark out by class (injection, auth, secrets in code, unsafe deserialization) and the flat number resolves into a profile. Suppose the reviewer catches 90% of injection bugs and 30% of auth flaws. That’s far more actionable than “68% identified”, because it’s a different assessment: the reviewer is missing a certain category.
Results like these point toward action. Maybe it’s a prompt issue, or maybe the auth flaws need context beyond the diff and no amount of prompt-tuning will fix it. You don’t have to tweak things and hope for the best. You can actually experiment.
The Price of Direction
It’s not free. A benchmark suite takes time to build and to maintain. You need labeled cases, scored outputs, and experts to calibrate judges. The engineering isn’t the hard part. You can’t write a good suite until you know what your product actually promises: which scenarios matter, which tradeoffs you will accept, and what “good” actually means for the thing you’re building. If you’re skipping evals, is it because you’re dodging data labeling, or because you’re dodging those harder questions?
This is the real reason to do it. A unit test checks whether you kept a promise you already made. A good eval suite makes you state your promise, and then tells you release over release how close you are to keeping it. CI tells you when something breaks. A compass tells you if you’re going the right way.