The LLM is a jackhammer
Overkill if your problem is a nail.
Last week I compared three Claude tiers on a ticket-sorting task. From the available models, the mid-tier model was the right buy for that scenario. But as I mentioned at the end of the post, we’ve got other tools in our toolbelt for a tiny sliver of the cost.
To demonstrate this, I fit an encoder model1 like the ones we used in ancient times (circa 2019, pre-GPT-3). The model was fine-tuned on about seventeen thousand tickets that weren’t used last week. This took a little over an hour on my laptop, and cost effectively zero dollars.
Data and Methods
The task and evaluation was the same as last week. We are predicting the CFPB Product category (11 options) from the user-submitted complaint. The evaluation set is the same 750 tickets from the last post. This time, we added two new models to the mix: the fine-tuned ModernBERT encoder and a DistilBERT model as the floor.
Bang for Buck, Revisited
The encoder is significantly better than Haiku (+4.7 points, p < 0.001). When compared to Sonnet and Opus, the encoder performed equally well with 85.5% accuracy.2 So it’s no improvement, but cost is the key. If we estimate the costs of the encoder model based on compute costs, it comes out to something like $0.0041 per 1,000 tickets. This is about 1/300th the cost of Sonnet! 3
Even the most naive option, the DistilBERT model, is statistically indistinguishable from every LLM tier.4 The question we should be asking goes beyond “which LLM works best”, it’s “should we be using an LLM at all?”
Revisiting the Cascade
Last week, the confidence cascade wasn’t a winning strategy because Haiku escalated too many tickets. The encoder does better here. Its confidence is a statistic produced during prediction instead of a subjective judgement. Producing it is free, and doesn’t require a prompt or judge model.
Having the encoder escalate its lowest-confidence 10%5 to Sonnet scores 87.5% accuracy, with a cost of $0.12 per 1,000 tickets. This is significantly better than Sonnet (p = 0.018) or the encoder alone (p = 0.011), at a tenth of Sonnet's price. The encoder is a good judge of what it will get wrong, and a tenth of the tickets carry most of the recoverable error, so its escalations are well-earned.6
How Many Labels Does This Take?
Of course, nothing is truly free. In this case, the catch is that the encoder needs labeled training data and the LLMs need none. The practical question is what labels would cost you. I priced outsourced labeling at $0.15/label,7 but you could plug in your own costs.
In this case, one thousand labels gets the encoder to parity with Haiku. Five thousand is enough to get within a couple points of Sonnet, with no statistical difference in this analysis. If you have the tickets, a couple thousand labels is a day or two of annotation. With the 10% escalation policy, even the 1,000-label encoder is already doing as well as Sonnet at a tenth of the price. The compute costs for training are noise throughout the process: under a dollar per training run.
Rare Category Weakness
This isn’t a silver bullet. The encoder performs worse on rare categories.8 When we fit our model to the empirical distribution, it learns not to focus on classes it rarely sees. A prompted LLM can reason without that prior. In scenarios where tail recall is key, the bias toward LLM may be well-deserved. But a hybrid model, rather than flat encoder, could still be an option.
Takeaways
Last week there were two layers of takeaway. The first was for the task: skip the cascade, default to Haiku if it’s low stakes, otherwise use Sonnet for a few points of accuracy. With other modeling approaches in the mix, our conclusions change. The new routing question isn’t “Haiku versus Sonnet”, but “classifier versus LLM”. For a defined category list with labels available, a fine-tuned encoder is a better default. We can wire an LLM in as the escalation path for the tickets the encoder flags.
This actually reinforces the overall takeaway from the last post: these are empirical questions, and we can bring our full toolkit to bear on them. This includes the tools from the pre-LLM era. By quantifying these tradeoffs, your team can make more informed decisions, grounded in the costs and outcomes you care about. You don’t have to default to the most powerful tool
Sample code and data are available on GitHub.
The model is ModernBERT-base, a ~150M parameter encoder. I fine-tuned it on 17,274 labeled CFPB complaints that weren’t part of our evaluation set. It’s a refresh of the approach used in 2019 BERT models.
Results of paired McNemar test are: encoder vs Sonnet +0.5 pp (95% CI −1.9 to +3.0, p = 0.75); vs Opus 5 +1.7 pp (p = 0.21).
I ran encoder training and inference locally, so to calculate this I priced with rental rates at $0.80/GPU-hour for training. This is actually higher than RunPod's published on-demand rate for an RTX 4090-class card, so if anything the encoder costs are overstated. Inference is priced at measured CPU throughput across 6 threads × $0.04/vCPU-hour each. LLM costs are metered Batch API prices like last week. The ratios are large enough that the pricing assumptions aren’t really a point of sensitivity, but if you want to tinker with different price points you can change them in the linked repository.
The DistilBERT model scored 83.5% accuracy. This includes a token cap that truncated 14% of inputs.
With the way I split the data, choosing a cutoff that scores best on the eval set would leak data and overstate accuracy. Instead, I just fixed a rule in advance: commit to escalating 10% of tickets. In this set, this is not actually the best-scoring budget (15% scores higher), but it is a more honest strategy.
The encoder is a far better judge of its own errors than the LLMs are of theirs: AUC of confidence predicting correctness is 0.873, against 0.735–0.806 across the LLM tiers. An omniscient router that uses the cheapest correct source per ticket (when available) gets 93.1% at $0.04/1k, but that’s cheating!
This assumes 30 seconds to read the complaint and label it, at $18/hour contractor rates.
The encoder has an F1 score of 0.36 on the Prepaid card class, whereas Sonnet scores 0.63. Sonnet’s macro-F1 (0.820) score beats the encoder’s (0.791) despite lower accuracy