Evals can be a useful benchmark for decision-making when you’re updating harnesses or prompts. You can compare two or more variants, and choose the one that performs best, something like “Harness A showed 90% accuracy vs. Harness B’s 88%”. That 90% figure is also an anchor for your expectations: after shipping it you expect 90% accuracy. If you’ve done this, you were probably disappointed; the variant never seems to do as well as it did at the time of selection. What gives?
To make sense of it, we can look to classical stats and ML.
Old habits
Data scientists learn this rule early: don’t use the same dataset for performance reporting and selection. The act of choosing based on a dataset has contaminated your measurement! Cawley and Talbot wrote the definitive paper on selection bias in model selection long before we were tuning prompts, back in 2010 (ancient history!). Unfortunately, this is one of the habits I’ve noticed teams setting aside in the AI era. Maybe tuning a prompt doesn’t feel like training a model, but a prompt is basically a hyperparameter in a black-box model, so the same selection bias applies. In doing this, teams are setting themselves up for disappointment when things hit prod.
Illustrating a winner’s curse
Each variant’s evaluation score is the combination of two things: its actual “goodness”, and how lucky it got (noise). Even if goodness is stable, luck is fleeting, so scoring the same variant against different samples will yield different scores. This is just a fact of life, but the trouble is in the selection rule itself.
To illustrate this, we’ll be omniscient for a moment. Consider two prompts, and an eval set of 50. Both have “true” scores of 88% and are equally good. But every statistic has sampling noise, and in this case we’d expect a score to typically wind up between 84% and 92%.1 So if you run the eval on these two equally-good variants, one of them’s going to score higher. Statistically, they’ll differ by two or more points 85% of the time! The higher score is probably an overestimate of true performance.
This gets worse with more variants. If we did the same thought experiment with twenty equally-good (88%) variants, we’d expect the winner to score around 96%, overestimating its performance by 8 percentage points!2 If you use this to set expectations about performance in production, you’re going to be very disappointed. This is a variant of the winner’s curse, a perennial problem in applied statistics.
Holdout hope
In the example, we were omniscient. In real life, we just have the measured results. What can we do when we don’t know everything? The answer is pretty boring: we perform selection and final performance benchmarking on different data. It’s just a holdout set. As we saw in the last section, this is even more important as the number of variants increases.
In scenarios where the stakes are high, there are more formal corrections to be considered. Xu et al.'s SIREN measured bias on tuning runs and found it was large enough to flip conclusions. They developed a protocol to handle reporting in light of this. But in lower-stakes contexts, measurement against the holdout is a low-cost way to get a number you can trust.
Takeaways
The field is advancing fast, and with the proliferation of nondeterministic systems, the rigor of classic stats is more important than ever. Practitioners have been guarding against this in A/B testing, biostatistics, and applied ML for decades. Building with LLMs doesn’t feel like model selection, but it is. We can build better software by carrying these good habits forward into the new world.
Given a binomial distribution with p=0.88 and n=50. This has a standard deviation of 2.3 successes, or 4.6% of 50 trials.
Based on Monte Carlo simulation, choosing the max of 20 i.i.d. draws of Binomial(50, 0.88). E[max] ≈ 96%.