Not Every Ticket Is Your Opus
Bigger isn't always better, and you don't have to guess which model you need.
If you’re running an LLM workload, you’ve probably heard two pieces of standard advice. On one hand: when in doubt, default to the more capable model. It’s the safest choice, and the quality is worth the price. On the other: optimize your costs by being clever instead. Run a cheap model, check its confidence, and escalate the hard cases to a bigger model.
I tested a few approaches on a common problem: routing support-ticket-style data. In this case, both bits of standard advice failed. The most capable model bought no detectable advantage, and cascading by model confidence managed to be more expensive and no better.
Neither approach is necessarily bad advice, but ultimately the better advice is: you don’t have to guess! Build a model selection strategy based on performance on a baseline set of tasks. This post walks through the process. I don’t have support ticket data I can write about publicly, so I’m using a public data set from the CFPB.1
Data and Methods
The task is ticket sorting: given the text of a consumer complaint, file it under the right product category. The CFPB consumer complaint database contains real complaints about financial products. I’ve used the consumer’s own category choice as the label.2 There are eleven categories, and the model predicts the category based on the complaint narrative.
I sampled 750 complaints from a recent three-month window stratified proportional to category share. I ran four Anthropic models across three price tiers (Haiku, Sonnet, and both same-priced Opus generations: 4.8, and Opus 5, released while I was writing this). I cached and logged all responses.3
Bang for Buck
I compared the labeling task against a floor that used the most common category (30%). As we’d expect, every model does far better than that, with accuracies over 80%.4 But the 2.5x pricing for Opus buys us no detectable boost. The confidence intervals suggest no accuracy gain with Opus, and if anything the point estimate points us the other way, toward Sonnet.
While I was preparing this post, Opus 5 was released, so I retested using the latest model. It cost about $3 and moved nothing: both generations land on an identical 628 of 750.5 This is what it looks like in practice to know how a changing model lineup changes performance.
Paired tests show the same: Sonnet and Opus are both clearly superior to Haiku, but we don’t see any gains from Opus. Paying for the large tier is just burning cash. If we were handling 100,000 tickets a month, Sonnet costs about $78/month more than Haiku and it buys us 4 points of accuracy. That’s a tradeoff you can make depending on the cost of a misfiled ticket. But Opus runs us about $184/month more than Sonnet, and it doesn’t get us anything. That’s no tradeoff at all, it’s just a matter of informed spending.
Is Cascading Smarter?
We might wonder if we could do better with a mix of models. A confidence cascade starts by running the cheap model on everything and requesting a confidence score. If the confidence is below a threshold, it sends the question to a stronger model. This is the standard recommendation in most LLM cost-optimization posts, including the one I was planning to write!
Asking an LLM to self-report confidence can work reasonably well. Haiku’s self-reported confidence separates its correct answers from its wrong ones with an AUC of 0.74, and correct answers carry higher confidence on every model I tested. So the premise is good.
But it falls apart in practice. With the model cascade, Haiku is highly confident on only 21% of tickets. It scores 96.8% on those, so keeping them cheap is right. But the other 79% are low-confidence, so the majority get sent upstream anyway. There’s not a “confident-and-cheap” bucket of tickets big enough to make the cascade work for us. 6
The best Haiku-to-Sonnet cascade (threshold chosen with full hindsight) reaches 85.1% at $1.40 per thousand tickets. Flat Sonnet is 84.9% at $1.23. At this point we’re using Sonnet for four out of five tickets anyway, and every one of those has a Haiku “tax” too. We’re paying more for a statistical tie.
Takeaways and Code
On this task: defaulting to Haiku seems like the good answer. It is only 4 points off the best accuracy for a third of the price, and some of its “misses” are just disagreements with a noisy human label. If those last 4 points matter, buy flat Sonnet. The model cascade isn’t worth it, and neither is Opus.
But that’s not something we can generalize, which is the actual point! These numbers describe one task and setup. The same measurement on your workload could come out totally differently, so the process is what matters here. Define your task, run models against a representative sample and evaluate performance, then compare and decide what to buy. In this case, it took a few hours and less than $20 in tokens. Before recommending high-tier pricing or building a routing layer, this is the sort of audit I start with. It’s a lot cheaper than building blindly!
The question “which model should we use” can be answered empirically and inexpensively, and there’s no one-size-fits-all rule.
Sample code and data are available on GitHub.
But if we’re talking cheap, every model in this post charges by the token. In the next piece, I run the same 750 tickets through the sort of model we all fine-tuned before GPT-3. The inference cost for those? Roughly zero. Subscribe to see how it holds up next week!
The labels are chosen by the person filing the complaint, and people can be inconsistent. For example, debt-collection issues may get filed under credit reporting. This means we have to assume there are some built-in errors, and think of accuracy as “agreement with a human choice” rather than correctness. In a high stakes scenario, we would devote time to human annotation and cleanup. But we’re focused on the relative comparison here.
I disabled thinking for all models. Costs presented are Batch API pricing per 1,000 tickets.
A better model is probably possible with feature engineering and tuning. The goal of this post was to compare raw model performance against a representative task, not to build the best model.
The identical score is a coincidence. The two Opus generations disagree on 29 individual tickets. On 13 of those, only Opus 4.8 is right. On another 13, only Opus 5 is right. On the remaining 3 both are wrong and just picked different wrong labels. It’s a wash.
We can measure the ceiling for routing using the results across all models. An omniscient oracle that sent each ticket to the cheapest model that gets it right scored 88.3% at $0.53 per thousand, escalating only 7.5% of tickets. This suggests that there is some routing strategy that can increase accuracy, but self-reported confidence isn’t enough.