When a large language model is used as a judge or classifier, it’s common to request a confidence score in the output. The score is usually something like a probability, so it gets treated like one. It gets compared against some threshold and routed accordingly. There’s a problem, though. The LLM’s confidence is a coarse, ordinal judgement disguised as a continuous one.
This one goes to eleven eight
A recent paper from a team at Amazon measured what models emit when you ask for confidence on a 0 to 100 scale. They found that, on a typical sentiment analysis task, the model produced eight distinct values across the whole dataset. More than half of the responses were exactly 95! This same pattern showed up across every model and dataset they tested. When you ask for a score from 0 to 100, you think you’re giving the LLM control over a precise dial, but it’s actually just got 8 notches we can tell which one is the model’s favorite.
When the score of 95 doesn’t reliably mean “95% correct”, or “probability of 0.95”, we have to rethink the way we make decisions down the line. What we think are finely-tuned thresholds lose their meaning. This isn’t an exotic failure case. Any time you’re asking an LLM for a number, you have to wonder if you’re getting the calibrated precision you want. At best, you’re making bulk decisions when you think you’re routing carefully. At worst, you’re letting the illusion of precision lead you toward false confidence.
What can we do?
The model’s confidence judgements aren’t useless, though. Researchers found they are pretty good at ordering. Given cases that need to be ranked into coarse categories (this one is good, this one is ambiguous, this one is bad), the model’s ordering is meaningful. But they’re bad at attaching precise, calibrated values to those rankings. The problems arise when we ask for the latter thing when the model can only reliably deliver the former.
Approach the problem in a different way. Instead of a 0 to 100 scale, ask for a small set of discrete levels that you design as labels plus criteria. In a multi-class classification problem, you might ask it to report one of three options: “High: the record clearly matches one category and no other. Medium: the record matches one category best but has features of another. Low: multiple categories are plausible.”
You might worry: is it losing precision? No, because the model was going to collapse your scale into a couple of bins anyway! This way, you get buckets with criteria you designed, instead of whatever coarse distribution the model falls into. This is also why judges anchored in rubrics outperform judges that just output scores. A rubric is just a set of tiers designed around qualities instead of confidence.
What does “high confidence” actually mean?
Thoughtfully-designed tiers fix what the model emits, but they don’t tell you what “High” confidence actually means. For that, we can go back to our roots in classical ML: calibration. Take a sample of data for which you have ground truth labels, and run your classifier/judge over it. Group those by the model’s reported tier, and compute the accuracy within each. A few lines of code is all it takes to get a routing table.
Something like: “High” is 94% accurate and covers 67% of traffic. “Medium” is 81% accurate and 21% of traffic. “Low” is 55% accurate with 12% of traffic. That allows us to make real decisions: if the bar for auto-accepting is 90%, “High” passes and everything else goes to review. We know to expect review load to be 33% of volume. If we’re surprised by the accuracy-to-tier mappings, we learn something about the task.
The model can supply the ordering, but the accuracy figures are an empirical question. Plus, when the model changes, or behavior drifts, we can rerun the slice and update the table.
What if you need precision?
Sometimes, you genuinely need finer grained scores than a handful of tiers can give. There are still two options. If your model provider exposes token logprobs, you can read the probability of the answer token directly. If not, consider running the same input several times and using the agreement rate. (This does multiply your inference costs). But start by asking if the precision is actually enabling better decision-making. Most routing problems don’t need it.
Tuned at the factory?
There’s an old joke about an amateur musician declining a tune up, saying "No thanks, mine was tuned at the factory."
I’ve written previously about how evals are an instrument to tell you where you are going. Like any instrument, they need tuning. An LLM’s confidence lacks calibration, and better prompting cannot get you there. Instead, the way forward is to stop asking for what it can’t give you. Ask for tiered judgement and supply the numbers yourself, based on real data. You can ask the model’s opinion, just don’t trust its fidelity.