Starting with the LLM

In the early stages of a classification project, we often suffer from a lack of real-world data. Collecting, purchasing, or building the data to train models like these is costly and time-consuming, and expending those resources before you’ve even proven the concept is a non-starter. LLMs excel at few-shot classification tasks like these, and are a solid place to start. It’s not just getting the feature out the door for its own sake, it’s producing the dataset you were missing! Every time GitHub’s LLM raised a secret alert, it was a human decision-point, and after two years it’s a heck of a training set that you didn’t have to go out and build.

Moving on down

It can be tempting to stop there. Things are working, the business is humming along. But the reason we started with the LLM was because we had little data, and that’s no longer true. The LLM is a jackhammer, and this problem is a nail. With a dataset like this, there are more options. On the same traffic, GitHub’s fine-tuned encoder showed 13% higher precision and 19% higher recall than the LLM. It was trained on their actual decisions, and used the relevant features directly instead of reasoning them out every time.

When is the bootstrap over?

GitHub’s guidance here is “smaller fine-tuned classifier can be the right answer when the task is narrow, the decision boundary is measurable, and serving efficiency is critical.” These criteria tell you if an encoder can do the job. But the question of whether now is the time to replace the LLM involves a few others.

First, can you enumerate your labels? If you don’t yet understand your categories, or you are still refining their hierarchy, then you are still in task discovery. The flexibility of the generalist LLM is an advantage. Once your labels have settled, it’s time to start looking. Second, are you reading the rationale returned by your LLM classifiers? If you are reviewing these logs, or presenting the explanatory text to your users, then you are still in LLM territory. Prose justifications and explanations aren’t something you will get from an encoder. But you have to be honest here. If you’re logging the rationale because you hope to review them “some day”, there’s a good chance you’re paying for reasoning that you will never use. Finally, how much prompt iteration are you doing? If there’s still a lot of churn in your prompts, the task (and associated data) has probably not settled enough yet.

Takeaways

An LLM-based classification pipeline can be a great fit when you are starting from limited data. By shipping it, you are building your product’s learning loop without slowing down. But the justification for starting with an LLM breaks down as your corpus grows. GitHub’s example shows that moving on can be cheaper, faster, and better!

You are ready to move on from an LLM-based pipeline when: you can enumerate your labels, you no longer need to read the rationale, and your task definition has settled. If all these are true, and you have thousands of labels, you are ready to try other approaches.

The only way to know which model is best for you is to measure it, though. Once you have the data, you can try fine-tuning an encoder, or calibrating a decision model. Score them all against your holdout set, and pick the winner.

Illusion of precision, revisited

In The Illusion of Precision, I cited research showing that while LLMs can provide seemingly precise confidence scores, the scores collapse to a small number of tiers. The recommendation was to ask your judges to provide a score from a small ordinal scale, and to calibrate them on a hand-labeled data set. Jev outputs a probability directly, rather than generating text. So the question is: does that number actually behave like a probability, or is it also a handful of tiers with more precise window-dressing?

I pulled 300 entries from the Nebius SWE-agent data set, and asked two judges (Jev and Sonnet 5) whether the run solved its issue, given the patch the agent submitted (a code diff), and the issue text. Both judges achieved about 77% accuracy, but the numbers in those verdicts were very different. Sonnet returned 32 distinct values, while Jev returned 85. If we compare the reliability curves, we can see that Jev’s scores stay close to the diagonal. This means that Jev’s confidence scores corresponded closely to the actual probability that the run succeeded. Sonnet’s results are much more poorly calibrated, and many are clumped on the lower end.

The advice for small, ordinal scales seems inapplicable to Jev. But while the Jev judge’s calibration was better than Sonnet, its calibration is imperfect, and I only know how it did because I checked. You should still calibrate your Jev judges, but there’s a good chance you’re starting much closer to the truth.

A million token bad habit, revisited

In A Million Token Bad Habit we explored how context is a design decision, and how just because context fits in the window doesn’t mean the model will use it well. I also discussed the importance of location within the context window. To test these on Jev, I sent the same SWE-agent prompts in three ways: the patch diff alone (Patch); the patch code plus the last ten steps of the agent’s run (Tail); and the full trajectory (Full).

The Tail did best on both accuracy and calibration. Full trajectory performed well on accuracy, but was less well-calibrated with probabilities about ten points too low across the board.

I figured this was the junk-drawer effect from the original post: too much irrelevant context in the window. But it wasn’t! To explore this, I used 153 runs that had ten steps or fewer, since for those runs Tail and Full include the same steps with different surrounding context. Cutting the inputs by two-thirds didn’t change anything. Stripping out the agent’s thoughts closed the gap a little bit. But moving the patch and exit status from the end of the state to right after the issue closed most of the gap.

You can see that even when we have the same context, the score jumps nine points solely because of where the patch was in the context window. This is one of the position effects from that post. It appears that for this task, putting the core content of the question at the top of the context window matters most. Changing the structure of context means that judge calibration should be rechecked, too.

Six jobs, or five?

In Should PMs Own Evals, I described the six jobs of an eval program. One of them was calibration:

Calibrate. If you’re using models to evaluate models (LLM-judges, classifiers, etc.), then these measures need to be calibrated themselves because their native outputs may not represent what you expect. Your investment here should be calibrated to your level of risk. Curate smaller, human-labeled sets to check your scorers. Output: scoring models calibrated against a validation set (evals in miniature!)

A decision model like Jev doesn’t eliminate this job, but it may make for a much smaller job! In the previous sections, we saw how our Jev judges started off better calibrated. If you start with a model that’s well-calibrated out of the box, you have much less work to do. But the only way to know for sure is to measure it. And as we saw in the last section, you still have context decisions to make.

Takeaways

Decision models like Jev and OpenAI’s new Decisions API provide an alternative to LLM judges that can be well-calibrated, inexpensive, and fast. I’ve previously argued that you should choose the right model for the job (and that sometimes it’s not an LLM at all), and now we have another in our toolbelt. But a generally well-calibrated judge isn’t necessarily well-calibrated to your problem. And as we saw above, you may find that its performance is still dependent on the design of your context. So there’s still work to do, but these models may change how much time we spend actively tuning and calibrating judges and classifiers vs. just checking them.

Last week, Anthropic released Opus 5.5. They tout a 40% reduction in cost relative to Opus 5. That sounds like a very good deal, but when they report a number like that they’re computing an average over a set of representative workloads. Should you expect to see a 40% reduction in your inference costs?

If you are using evals as a compass, you already have what you need to answer that question. I previously shared an analysis of verbosity in Opus 5 vs. 4.8, and I used that data and code to examine the changes with 5.5.

Data and methods

I have a small collection of twenty-five data science prompts that I use to evaluate the stylistic characteristics of LLM responses. These are designed to be relatively straightforward across a range of task types including factual questions, coding, recommendations, explanation, and text-condensing. I ran each prompt five times against each model: Opus 4.8, Opus 5, and Opus 5.5.1 Reasoning tokens are billed as output tokens, but I track them separately because they aren’t visible as part of the response.

Right on the money

On my workload, Anthropic’s claimed reduction holds up. A median response that costs 5.1 cents of output on Opus 5 costs 3.1 cents with Opus 5.5. That’s right on target with a 40% reduction. Very little of those savings come from the model writing less in its response. The responses themselves are the same length (within noise) as Opus 5’s. The 40% savings breaks down into a few parts. First, the obvious: Opus 5.5’s output price dropped from $25 to $20 per million tokens. Second, we see a notable shift in reasoning: Opus 5.5 spends roughly half the thinking tokens Opus 5 did on the same prompts.

Cheaper than what?

Opus 5.5 is cheaper than Opus 5. But the cost trajectory over the last three months isn’t all downward. When Anthropic released Opus 5, they carried forward Opus 4.8’s pricing structure. But they weren’t equally costly to run: against my workload, Opus 5 billed almost two and a half times the output tokens compared to 4.8. So even if the pricing is fixed, your COGS is moving.

Takeaways

If you were running Opus 4.8 earlier this summer, last week’s drop in unit cost didn’t close that gap. Your unit costs are still up by over a third since June! Of course, pricing changes and new models are released all the time. How well do you understand what those shifts are going to cost you?

At last week’s Copilot launch, which I covered for Every, Satya Nadella called for ‘evals specific to your company,’ because general benchmarks can’t tell you how a model changes outcomes in your organization. A cost quote like Anthropic’s is the same kind of average. The only way to know how model updates impact your workloads is to measure it yourself. A few bits of practical advice:

• When you track tokens, break out reasoning and response text separately, even if they are priced the same. There can be invisible shifts in model behavior that you will miss if you simply track in/out.

• Price model swaps on your own workloads. You’re already building evals for benchmarking and quality (right?), use them to forecast cost as well.

The code and data used in this analysis are available on GitHub.

Trajectories from software engineering

As an example, we can look to the trajectories from the Nebius SWE-agent data set. In this data set, the creators ran a software engineering agent against real GitHub issues with matching PRs, and graded each run’s result on whether the agent’s changes passed the tests from the actual fix. They produced many runs for each issue (dozens in some cases), so we can see what happens when an agent keeps trying the same task.

For example, in pydantic issue 2496 it was reported that a model field with the empty tuple type always generated a broken JSON schema. The agent tried to resolve this issue 16 times. On one attempt it gave up, but the other fifteen times it submitted a change. All fifteen were unique, and all failed their tests. Each of them introduced some sort of special-case for the empty tuple, and the real fix was more general.

Contrast this with issue 2290 from the same repo. Here, the reporter noted that the copy method didn’t properly flag changed fields. This was also attempted sixteen times, and passed seven of them. Each run changed the same line of code. Seven got the change right, and the others failed. This one really was a coin flip.

Retrying only helps if the task failed due to bad luck: a transient failure, a system error, or some mangled output. We just saw examples of both, so the question is: what’s typical?

In the Nebius data set, the coin flips were the minority. In the plot below, we can see the distribution of pass rates by issue.1 If the failures were due to bad luck, we’d expect to see a distribution like the dashed line.2 But the blue distribution reveals a different pattern: most of the issues that failed never succeeded.

In the data I analyzed, once a task failed twice the next run’s pass rate was about 2.5%. If you know this number for your domain, it’s the number to build a retry policy on: what are the chances the next run will succeed?

What about better models?

This research was conducted with 2024-era models, a generation or two behind today’s. Better models can solve more tasks, meaning their pass rates increase. But when they fail, retrying doesn’t help much. Two recent papers3 reported pass rates using current models, both for single runs and for the best of several (“pass@k”). In every case, we saw the same pattern with the extra runs. Tasks that failed once mostly failed again.

Try something else?

Retrying is cheap, and sometimes it works! It’s a reasonable first-step for those “bad luck” situations. The second failure isn’t bad luck, though. There is something about the task as presented that is not going to pass. Running again with the same prompt, context, and tools is the “cut-and-paste” retry. This usually fails.

The fix is to change something. Try giving the agent different context, different examples, or a different sort of prompt. You might have more luck splitting the task into pieces. This is where you consider handing it to a human with the failing trajectories attached. This is how you learn about your agent’s failure modes.

If you’re dealing with this problem, you don’t need fancy analysis right away. Start with a table containing task ids and pass/fail for each run. Sort your tasks into those that always failed, vs. those that sometimes failed. If the first bucket is large, you know where to start to break out of the retry loop.

To make sense of it, we can look to classical stats and ML.

Old habits

Data scientists learn this rule early: don’t use the same dataset for performance reporting and selection. The act of choosing based on a dataset has contaminated your measurement! Cawley and Talbot wrote the definitive paper on selection bias in model selection long before we were tuning prompts, back in 2010 (ancient history!). Unfortunately, this is one of the habits I’ve noticed teams setting aside in the AI era. Maybe tuning a prompt doesn’t feel like training a model, but a prompt is basically a hyperparameter in a black-box model, so the same selection bias applies. In doing this, teams are setting themselves up for disappointment when things hit prod.

Illustrating a winner’s curse

Each variant’s evaluation score is the combination of two things: its actual “goodness”, and how lucky it got (noise). Even if goodness is stable, luck is fleeting, so scoring the same variant against different samples will yield different scores. This is just a fact of life, but the trouble is in the selection rule itself.

To illustrate this, we’ll be omniscient for a moment. Consider two prompts, and an eval set of 50. Both have “true” scores of 88% and are equally good. But every statistic has sampling noise, and in this case we’d expect a score to typically wind up between 84% and 92%.1 So if you run the eval on these two equally-good variants, one of them’s going to score higher. Statistically, they’ll differ by two or more points 85% of the time! The higher score is probably an overestimate of true performance.

This gets worse with more variants. If we did the same thought experiment with twenty equally-good (88%) variants, we’d expect the winner to score around 96%, overestimating its performance by 8 percentage points!2 If you use this to set expectations about performance in production, you’re going to be very disappointed. This is a variant of the winner’s curse, a perennial problem in applied statistics.

Holdout hope

In the example, we were omniscient. In real life, we just have the measured results. What can we do when we don’t know everything? The answer is pretty boring: we perform selection and final performance benchmarking on different data. It’s just a holdout set. As we saw in the last section, this is even more important as the number of variants increases.

In scenarios where the stakes are high, there are more formal corrections to be considered. Xu et al.'s SIREN measured bias on tuning runs and found it was large enough to flip conclusions. They developed a protocol to handle reporting in light of this. But in lower-stakes contexts, measurement against the holdout is a low-cost way to get a number you can trust.

Takeaways

The field is advancing fast, and with the proliferation of nondeterministic systems, the rigor of classic stats is more important than ever. Practitioners have been guarding against this in A/B testing, biostatistics, and applied ML for decades. Building with LLMs doesn’t feel like model selection, but it is. We can build better software by carrying these good habits forward into the new world.

Two common patterns here are to blend the criteria into one rubric or average the results between judges. Both of these miss the point, though. The disagreements can actually tell you something, and they represent the stakeholder tensions built-in to your product. In this post, I demonstrate the concept with a simple example.

Setup

A 2025 paper (MedScore) assembled a public dataset of patient questions and physician responses. They used the physician’s responses as context and had AI rewrite them. The authors made that dataset public (AskDocsAI). It includes 300 patient questions, each with the physician’s actual reply and an AI rewrite.

Patient communications present a good example of the multiple-viewpoints problem, because doctors and patients prioritize different qualities. I setup LLM judges to represent each of these viewpoints: a patient rubric (addresses my concern, provides me clear next steps) and a clinician rubric (appropriate urgency and certainty, no unsupported claims)1. Each judge outputs a score from 1-5 based on its criteria. 2

I sampled 150 of the patient question/response pairs, and used these judges to evaluate the responses. Both judges used the same underlying model and input context, only the prompt/rubric was changed. For each response, I ran three replications of each judge.

A difference of opinion

We see considerable differences between judge “personas”. The patient and clinician judge scores are at least one point apart on 53% of responses. LLMs are probabilistic, so we should expect some variation. The same judge re-run on the same answer differs by that much only 11% of the time. We’re capturing two distinct viewpoints.

Most of the disagreements are one point out of five, but ten had differences of two points. This goes in both directions: there are answers the patient judge likes and the clinician judge dislikes, and vice versa. Looking at the LLM judge rationales we can see differences in priority.

In one example, someone with a single stitch asks if they can remove it themselves instead of going back to the doctor. The response asks clarifying questions (how long the stitch has been in), and defers its recommendation. The clinician judge gives this response a 4, with rationale like “appropriately asks for more information…rather than guessing.” On the other hand, the patient judge scored it a 2, saying the response “deflects by asking for more info…without providing any interim advice or self-removal instructions.”

Neither judge is “wrong” about its rubric criteria, they were just designed around different priorities. This illustrates a conflict between stakeholders. If we were building this system, this would be a fundamental tradeoff between actionable and cautious repsonses that we’d have to reckon with.

A rubric smoothie?

We might be tempted to simply roll these up into one combined rubric. Then we don’t have to think about tradeoffs at all! I did this, too. I ran a third judge with a combined patient+clinician rubric. Perhaps surprisingly, it didn’t act as a compromise. It scored lower than either of the targeted judges. Merging the criteria didn’t average the two viewpoints. It made an even stricter judge that never awarded a five! This is likely because the combined rubric covers so much it finds more reasons to dock points.

On the ten answers where patient and clinician judges differed by two points, the combined judge sides with the patient judge on four, the clinician judge on two, “compromises” on three, and scores lower than either on one. We can’t distinguish between “bad responses” and responses that expose an underlying tension. Worse, we lose visibility into how that tradeoff is being decided. It’s no longer a product decision, we’ve just tossed it to the model.

The other obvious solution is to average the two judge scores. But this has the same problem of flattening meaningful disagreements. In the stitch example, the patient judge’s score of 2 and the clinician judge’s score of 4 average to 3, hiding disagreement.

Handling disagreement

So what can we do? There are a few practical pieces of advice.

• Report judge scores separately. If you have sometimes-conflicting priorities to balance, reporting performance in each dimension gives you a clearer view into the tradeoffs you are making.

• Route disagreement for human review. When judges disagree, escalate those items for review. In this case, I looked at the ten answers that produced large differences in scores to understand how judgements might conflict.

• Decide your priority explicitly. If you have to force multiple judges into a single verdict, make an intentional decision about your priority (e.g., clinician must pass, patient ranks the rest). Social choice theorists have been studying how to aggregate conflicting preferences for over a hundred years, and offers several techniques. Averaging scores across judges is an implicit prioritization, and it’s a strong assumption! You can likely do better for your product.

Takeaways

The ten answers where the judges showed marked disagreement revealed a fundamental tension in this domain. To get to that, I had to go a little deeper than the scores, and read the rationale from each judge. It would’ve been easy to “smooth” this disagreement away and miss the insight and tradeoff entirely.

If you find yourself balancing competing priorities or stakeholders, consider splitting your judges. If you read their rationale when they disagree, you will learn something about your product and its fundamental tradeoffs.

The six jobs of an eval program

An eval program is a build-measure-learn loop in miniature. We can think about each job as having a required output that feeds the next phase, with the whole cycle forming a learning loop.

1. Define “good”. Off-the-shelf evals rarely perform well in product-specific scenarios, so the bulk of every product’s evals should be tailored to its goals and moment of value. Building a useful eval program requires you to define what good outputs look like for your product. The output of this job is a framework for evaluation: categories of test cases, with criteria for evaluating them.

2. Define what to test. With a definition of “good”, we have what we need to construct a test set. The best test cases are a blend of real-world samples and intentionally constructed edge- or adversarial-cases. These should tie back directly to your definition of “good”. The output of this job is: specific test cases with evaluation criteria.

3. Measure performance. Build a scorer with rubrics, judges, and code for deterministic evaluation. As with any measurement program, we have to consider the right statistical methods and required sample size. The output of this job is: runnable evals that produce interpretable results.

4. Calibrate. If you’re using models to evaluate models (LLM-judges, classifiers, etc.), then these measures need to be calibrated themselves because their native outputs may not represent what you expect. Your investment here should be calibrated to your level of risk. Curate smaller, human-labeled sets to check your scorers. Output: scoring models calibrated against a validation set (evals in miniature!)

5. Monitor performance. One-time snapshot evals are useful, but ultimately this is a part of your CI process. At a minimum, you’re testing for regressions and evaluating model upgrades. If you’re using evals as a compass, you need ongoing measurement to know where you’re going. Where possible, you also want to be running evaluations against production traffic, or at least logging for future eval development. This job is classic MLE, and its output is a harness with logs and metrics.

6. Learn. This applies to the product and to the eval set itself. For the eval set: read failures as they come in, and use error analysis to refine and build on your eval framework and test cases. Evals serve as a yardstick for product decisions that connect to the models (version changes, reasoning budgets, etc.). The evals are a benchmark to guide judgement. Output: expanding eval-set (and confidence), data-informed product changes.

Mapping the jobs to your team

With these jobs defined, you can map out concrete accountabilities and ownership. The way they fit into your org depends on the structure of your team, but it’s helpful to think about them in terms of the expertise they require.

If you’re an AI startup with no data science team, you have gaps to fill in measurement and calibration. No matter what roles you have on your team, you need domain expertise. The PM fills this role in a pinch, but without a subject-matter expert, evals will tend to be defined by what’s easiest to measure. Scores might be green, but if an expert would reject the results, you’re just playing make believe.

Falling through the cracks

Calibration and monitoring often fall through the cracks because they seem optional. The models are getting better all the time, right? If you’re renting these models without running calibrated scorers on an ongoing basis, you’re putting the performance of your product in someone else’s hands on little more than faith. The same goes for your LLM judges: when the vendor’s model update comes out, your scores will shift, regardless of whether your product did.

The frontier labs are incentivized to make their models better, but better against their benchmarks does not necessarily mean better at your tasks.

So, should PMs own evals?

Sure, at least in part. The first two jobs and the learning loop clearly land in PM territory. But it takes a village to build an effective eval program. Start with the six jobs, find out who on your team is well-positioned to do them, assign accountabilities, and see what’s left. If you’ve still got a gap, now you know what you’re hiring for.

This one goes to eleven eight

A recent paper from a team at Amazon measured what models emit when you ask for confidence on a 0 to 100 scale. They found that, on a typical sentiment analysis task, the model produced eight distinct values across the whole dataset. More than half of the responses were exactly 95! This same pattern showed up across every model and dataset they tested. When you ask for a score from 0 to 100, you think you’re giving the LLM control over a precise dial, but it’s actually just got 8 notches we can tell which one is the model’s favorite.

When the score of 95 doesn’t reliably mean “95% correct”, or “probability of 0.95”, we have to rethink the way we make decisions down the line. What we think are finely-tuned thresholds lose their meaning. This isn’t an exotic failure case. Any time you’re asking an LLM for a number, you have to wonder if you’re getting the calibrated precision you want. At best, you’re making bulk decisions when you think you’re routing carefully. At worst, you’re letting the illusion of precision lead you toward false confidence.

What can we do?

The model’s confidence judgements aren’t useless, though. Researchers found they are pretty good at ordering. Given cases that need to be ranked into coarse categories (this one is good, this one is ambiguous, this one is bad), the model’s ordering is meaningful. But they’re bad at attaching precise, calibrated values to those rankings. The problems arise when we ask for the latter thing when the model can only reliably deliver the former.

Approach the problem in a different way. Instead of a 0 to 100 scale, ask for a small set of discrete levels that you design as labels plus criteria. In a multi-class classification problem, you might ask it to report one of three options: “High: the record clearly matches one category and no other. Medium: the record matches one category best but has features of another. Low: multiple categories are plausible.”

You might worry: is it losing precision? No, because the model was going to collapse your scale into a couple of bins anyway! This way, you get buckets with criteria you designed, instead of whatever coarse distribution the model falls into. This is also why judges anchored in rubrics outperform judges that just output scores. A rubric is just a set of tiers designed around qualities instead of confidence.

What does “high confidence” actually mean?

Thoughtfully-designed tiers fix what the model emits, but they don’t tell you what “High” confidence actually means. For that, we can go back to our roots in classical ML: calibration. Take a sample of data for which you have ground truth labels, and run your classifier/judge over it. Group those by the model’s reported tier, and compute the accuracy within each. A few lines of code is all it takes to get a routing table.

Something like: “High” is 94% accurate and covers 67% of traffic. “Medium” is 81% accurate and 21% of traffic. “Low” is 55% accurate with 12% of traffic. That allows us to make real decisions: if the bar for auto-accepting is 90%, “High” passes and everything else goes to review. We know to expect review load to be 33% of volume. If we’re surprised by the accuracy-to-tier mappings, we learn something about the task.

The model can supply the ordering, but the accuracy figures are an empirical question. Plus, when the model changes, or behavior drifts, we can rerun the slice and update the table.

What if you need precision?

Sometimes, you genuinely need finer grained scores than a handful of tiers can give. There are still two options. If your model provider exposes token logprobs, you can read the probability of the answer token directly. If not, consider running the same input several times and using the agreement rate. (This does multiply your inference costs). But start by asking if the precision is actually enabling better decision-making. Most routing problems don’t need it.

Tuned at the factory?

There’s an old joke about an amateur musician declining a tune up, saying "No thanks, mine was tuned at the factory."

I’ve written previously about how evals are an instrument to tell you where you are going. Like any instrument, they need tuning. An LLM’s confidence lacks calibration, and better prompting cannot get you there. Instead, the way forward is to stop asking for what it can’t give you. Ask for tiered judgement and supply the numbers yourself, based on real data. You can ask the model’s opinion, just don’t trust its fidelity.

The junk drawer becomes the junk drawer because it’s bigger than all the little odds and ends that we toss inside. Nothing gets lost or thrown out, and we can always dig things out later. The problem is that the model is doing the digging, and it is just not that good at it.

The quantity, form and sequence of context is a design decision, and like all design decisions it has consequences. This deserves the same care you would give a database schema or API contract.

“It fits” vs. “it works”

Chroma’s Context Rot report tested 18 frontier models (including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3)1 on simple tasks. Things like fact retrieval, conversational Q&A, and reproducing sequences. Every model they tested degraded as input length grew. This started long before they hit the context window’s limit. Even when the context window is a million tokens, the model’s performance starts to decrease at tens of thousands of tokens. The tasks in the study were designed to be simple so that length was the only variable. When we start talking about real workloads with reasoning and ambiguity, we can only expect it to get worse.

We’re not talking 80% or even 50% context here. In many cases this is 10% or less. We should think of the number on the context window as a (marketable) capacity limit more than a performance guarantee. You can fit a lot of clowns in a VW Beetle, but I wouldn’t drive it to IKEA.

It’s not just about too much, there are also risks to plopping in loosely related content. If you’ve ever designed multiple-choice tests for humans, you’re probably familiar with “distractors” - answers that are wrong but look plausibly correct. In the Chroma study, they planted distractors (passages that look relevant but are not) and found performance degraded as a result.

Position matters, too

In Liu et al.’s Lost in the Middle, researchers provided the same set of documents in context, but moved one relevant document around. Their accuracy curves were U-shaped: good when the answer is at the beginning or end of context, and worse when it is buried in the middle. The middle of your context window is where information goes to be ignored. And in a way, the middle gets “bigger” the more we stuff the window.

This applies on a smaller scale, too. Chen et al. showed that just reordering the premises of a logic problem significantly changes reasoning accuracy. Nothing was added or removed, only rearranged. If shuffling identical information changes the output, then context engineering becomes akin to a modeling decision, whether we’re making it intentionally or not.

What earns its keep?

Anthropic’s context engineering guidance has a useful framing for this: models have an attention budget. Every token spends it, and the budget depletes faster than the context window fills. Anthropic’s goal for context assembly is the smallest set of high-signal tokens to maximize the likelihood of the outcome you want.

If we apply this lens, it changes how we engineer context. It’s no longer a question of “could this be relevant?”, but “does this context earn its tokens?” We have to weigh the value added by each bit of added context.

In a way, junk drawer thinking optimizes for the person assembling the context. If nothing is left out, we can’t miss anything, and we can avoid making tough decisions. It’s a shortcut akin to tech debt, but we often take it without proper consideration. Instead, we should optimize for what the model actually needs. What context will actually get used?

Practical tips

In practice, there are a few patterns I recommend:

Curate first. When the quality of retrieved content is disappointing, the path of least resistance is to raise “k” in the top-k retrieved. This is usually the wrong move. Twenty mediocre chunks are worse than three good ones because the other seventeen are distractors. Instead, rerank and make cuts. Treat every chunk you include as a fact that must be relevant

Put critical context at the edges. Instructions, the user’s prompt, and the most critical facts go at the start or the end.

Choose your representation deliberately. You can provide the same underlying data as raw records, prose summaries, structured relationships, etc. But the model will likely not perform equally well across them. I’ve written previously about getting better results by changing how data reaches the model rather than how much of it does. A good evaluation set can help you tune representation.

Prune agent histories. Long-running agents accumulate tool outputs, dead-end explorations, and assorted junk state. Trim your context as you go using compaction and structured notes, and drop tool output once it’s no longer necessary. Treat the agent’s context window as a working set instead of an archive.

Takeaways

We might hope that frontier models will just get better at long context, and none of this will matter. It’s possible, but the trend suggests the opposite. Windows keep getting longer but attention doesn’t. Every model Chroma tested degraded including the ones explicitly built for long context. As agents start running longer, they’ll accumulate more and more junk.

Fortunately, this is a tractable problem, and there are some easy wins. You don’t need a new model or a bigger window, you need to think of the context as a piece of engineering: something worth testing, curating, and designing.

Go open your junk drawer and peek inside. Do you really need all those rubber bands?

To demonstrate this, I fit an encoder model1 like the ones we used in ancient times (circa 2019, pre-GPT-3). The model was fine-tuned on about seventeen thousand tickets that weren’t used last week. This took a little over an hour on my laptop, and cost effectively zero dollars.

Data and Methods

The task and evaluation was the same as last week. We are predicting the CFPB Product category (11 options) from the user-submitted complaint. The evaluation set is the same 750 tickets from the last post. This time, we added two new models to the mix: the fine-tuned ModernBERT encoder and a DistilBERT model as the floor.

Bang for Buck, Revisited

The encoder is significantly better than Haiku (+4.7 points, p < 0.001). When compared to Sonnet and Opus, the encoder performed equally well with 85.5% accuracy.2 So it’s no improvement, but cost is the key. If we estimate the costs of the encoder model based on compute costs, it comes out to something like $0.0041 per 1,000 tickets. This is about 1/300th the cost of Sonnet! 3

Even the most naive option, the DistilBERT model, is statistically indistinguishable from every LLM tier.4 The question we should be asking goes beyond “which LLM works best”, it’s “should we be using an LLM at all?”

Revisiting the Cascade

Last week, the confidence cascade wasn’t a winning strategy because Haiku escalated too many tickets. The encoder does better here. Its confidence is a statistic produced during prediction instead of a subjective judgement. Producing it is free, and doesn’t require a prompt or judge model.

Having the encoder escalate its lowest-confidence 10%5 to Sonnet scores 87.5% accuracy, with a cost of $0.12 per 1,000 tickets. This is significantly better than Sonnet (p = 0.018) or the encoder alone (p = 0.011), at a tenth of Sonnet's price. The encoder is a good judge of what it will get wrong, and a tenth of the tickets carry most of the recoverable error, so its escalations are well-earned.6

How Many Labels Does This Take?

Of course, nothing is truly free. In this case, the catch is that the encoder needs labeled training data and the LLMs need none. The practical question is what labels would cost you. I priced outsourced labeling at $0.15/label,7 but you could plug in your own costs.

In this case, one thousand labels gets the encoder to parity with Haiku. Five thousand is enough to get within a couple points of Sonnet, with no statistical difference in this analysis. If you have the tickets, a couple thousand labels is a day or two of annotation. With the 10% escalation policy, even the 1,000-label encoder is already doing as well as Sonnet at a tenth of the price. The compute costs for training are noise throughout the process: under a dollar per training run.

Rare Category Weakness

This isn’t a silver bullet. The encoder performs worse on rare categories.8 When we fit our model to the empirical distribution, it learns not to focus on classes it rarely sees. A prompted LLM can reason without that prior. In scenarios where tail recall is key, the bias toward LLM may be well-deserved. But a hybrid model, rather than flat encoder, could still be an option.

Takeaways

Last week there were two layers of takeaway. The first was for the task: skip the cascade, default to Haiku if it’s low stakes, otherwise use Sonnet for a few points of accuracy. With other modeling approaches in the mix, our conclusions change. The new routing question isn’t “Haiku versus Sonnet”, but “classifier versus LLM”. For a defined category list with labels available, a fine-tuned encoder is a better default. We can wire an LLM in as the escalation path for the tickets the encoder flags.

This actually reinforces the overall takeaway from the last post: these are empirical questions, and we can bring our full toolkit to bear on them. This includes the tools from the pre-LLM era. By quantifying these tradeoffs, your team can make more informed decisions, grounded in the costs and outcomes you care about. You don’t have to default to the most powerful tool

Sample code and data are available on GitHub.

I tested a few approaches on a common problem: routing support-ticket-style data. In this case, both bits of standard advice failed. The most capable model bought no detectable advantage, and cascading by model confidence managed to be more expensive and no better.

Neither approach is necessarily bad advice, but ultimately the better advice is: you don’t have to guess! Build a model selection strategy based on performance on a baseline set of tasks. This post walks through the process. I don’t have support ticket data I can write about publicly, so I’m using a public data set from the CFPB.1

Data and Methods

The task is ticket sorting: given the text of a consumer complaint, file it under the right product category. The CFPB consumer complaint database contains real complaints about financial products. I’ve used the consumer’s own category choice as the label.2 There are eleven categories, and the model predicts the category based on the complaint narrative.

I sampled 750 complaints from a recent three-month window stratified proportional to category share. I ran four Anthropic models across three price tiers (Haiku, Sonnet, and both same-priced Opus generations: 4.8, and Opus 5, released while I was writing this). I cached and logged all responses.3

Bang for Buck

I compared the labeling task against a floor that used the most common category (30%). As we’d expect, every model does far better than that, with accuracies over 80%.4 But the 2.5x pricing for Opus buys us no detectable boost. The confidence intervals suggest no accuracy gain with Opus, and if anything the point estimate points us the other way, toward Sonnet.

While I was preparing this post, Opus 5 was released, so I retested using the latest model. It cost about $3 and moved nothing: both generations land on an identical 628 of 750.5 This is what it looks like in practice to know how a changing model lineup changes performance.

Paired tests show the same: Sonnet and Opus are both clearly superior to Haiku, but we don’t see any gains from Opus. Paying for the large tier is just burning cash. If we were handling 100,000 tickets a month, Sonnet costs about $78/month more than Haiku and it buys us 4 points of accuracy. That’s a tradeoff you can make depending on the cost of a misfiled ticket. But Opus runs us about $184/month more than Sonnet, and it doesn’t get us anything. That’s no tradeoff at all, it’s just a matter of informed spending.

Is Cascading Smarter?

We might wonder if we could do better with a mix of models. A confidence cascade starts by running the cheap model on everything and requesting a confidence score. If the confidence is below a threshold, it sends the question to a stronger model. This is the standard recommendation in most LLM cost-optimization posts, including the one I was planning to write!

Asking an LLM to self-report confidence can work reasonably well. Haiku’s self-reported confidence separates its correct answers from its wrong ones with an AUC of 0.74, and correct answers carry higher confidence on every model I tested. So the premise is good.

But it falls apart in practice. With the model cascade, Haiku is highly confident on only 21% of tickets. It scores 96.8% on those, so keeping them cheap is right. But the other 79% are low-confidence, so the majority get sent upstream anyway. There’s not a “confident-and-cheap” bucket of tickets big enough to make the cascade work for us. 6

The best Haiku-to-Sonnet cascade (threshold chosen with full hindsight) reaches 85.1% at $1.40 per thousand tickets. Flat Sonnet is 84.9% at $1.23. At this point we’re using Sonnet for four out of five tickets anyway, and every one of those has a Haiku “tax” too. We’re paying more for a statistical tie.

Takeaways and Code

On this task: defaulting to Haiku seems like the good answer. It is only 4 points off the best accuracy for a third of the price, and some of its “misses” are just disagreements with a noisy human label. If those last 4 points matter, buy flat Sonnet. The model cascade isn’t worth it, and neither is Opus.

But that’s not something we can generalize, which is the actual point! These numbers describe one task and setup. The same measurement on your workload could come out totally differently, so the process is what matters here. Define your task, run models against a representative sample and evaluate performance, then compare and decide what to buy. In this case, it took a few hours and less than $20 in tokens. Before recommending high-tier pricing or building a routing layer, this is the sort of audit I start with. It’s a lot cheaper than building blindly!

The question “which model should we use” can be answered empirically and inexpensively, and there’s no one-size-fits-all rule.

Sample code and data are available on GitHub.

But if we’re talking cheap, every model in this post charges by the token. In the next piece, I run the same 750 tickets through the sort of model we all fine-tuned before GPT-3. The inference cost for those? Roughly zero. Subscribe to see how it holds up next week!

The failure

I posed the same small business question to the first version of the agent. It produced an answer that was on the right track but blundered into major factual errors. It got the themes right, identifying a family of bills focused on relief and the opposite family that imposed new fees or mandates. It got the roster of sponsors right. But when it transitioned to the meat of the response: the “cross-aisle Democrats worth cultivating”, it named four Democrats as Nay votes on SB 5525.

Unfortuntaely, zero Democrats voted Nay on SB 5525.

The same response also described a “consistent bloc” of 56 legislators voting Nay on “all or nearly all” of four mandate bills. The real all-four bloc is 22 members. The 56 names were all genuine Nay voters somewhere, but the story overstated how much their votes overlapped.

This verges on worst-case scenario. The response looked credible to someone familiar with the domain. There were specific names, specific bills, and specific vote positions. But many of them were false. Without manual validation, this sort of error can easily slip through unnoticed, undermining the entire analysis.

The diagnosis

Of course, we know LLMs can hallucinate. But the claims were very structured, and almost grounded in the facts: real Democrats with crossover reputations, but the wrong bill. I explored a different hypothesis: that the model didn’t invent the votes, but misassigned real ones.

The working set for that response contained other employer-mandate bills, including HB 1213, a paid-leave bill. In the actual record, Springer, Chapman, and Rule all voted Nay on HB 1213. These were real votes, breaking with their caucus. Richards voted Nay on HB 1213 and HB 2020. All the agent’s “fabricated” SB 5525 Nay votes were a true HB 1213 Nay. One bill-label swap between two adjacent lists accounts for the entire failure.

This specific class of error happens a lot. When the context holds many similar tuples, the model will get “mixed up”. The reason it showed up here and not earlier is instructive: previous runs only drilled down to one bill, so the evidence was compact and easy to attribute. This run was the first exploring a broad set of legislation: fourteen bills and five very similar Nay lists. Summarizing these (”who voted Nay on all four?”) required the model to intersect those lists. It got the list of dissenters right, but got the specific votes mixed up. The 56-member bloc overstatement was the same failure in a different way: set intersection done by the model instead of in code.

Tools maketh the agent

Examining the tools I defined, I discovered I’d made the problem worse. My bill_votes tool returned rows like:

Jim Walsh (R-House): Nay — 3rd Reading & Final Passage

There’s no bill ID here. It only appears in the tool-call arguments, one line above forty rows of results. The graph even stores bill_id denormalized on every VOTED edge, but my initial tool call (in the name of brevity), dropped it. So the model was forced to map rows to bills by where they were in the output. This kind of positional binding across adjacent, similar lists is precisely where things go wrong.

Self-describing rows — HB 1213 | Jim Walsh (R-House): Nay — ... — is a one-line fix that tailors the context for the model.

The real fix

The formatting fix helps, but we’re still just treating a symptom. The real insight lines up with what I discussed in the last post:

This is a failure mode of flat RAG, and while context design goes a long way, ultimately we’re fighting an uphill battle against the model.

For claims like these, the right trust boundary is the boundary of the data layer. Move aggregation into tools:

• voting_bloc(bill_ids, vote) - members who voted a given way on all of these bills

• party_crossovers(bill_ids) - members who voted against their party’s majority, per bill

• verify_claims(claims) - check a list of specific (member, bill, vote, motion) tuples before they’re used in a response.

A “find themes across bills” question needs to do these things, and the tools allow the agent to run them as direct operations on the graph. The new agent chooses to use them. The transcript behind the last post shows voting_bloc and party_crossovers are used to explore the cross bill space, and verify_claims was used to check the named votes before the response was generated. When I added tools to run these operations in the data layer, all the resulting claims could be verified

The takeaway

There’s an obvious takeaway here: give your agents verification tools! The less obvious takeaway is a design principle for agentic retrieval: let the model decide what to compute, but let the data layer compute it. The model’s job is planning and interpretation: which bills constitute the concept, what the crossover pattern means. If the model is also doing all the joins, you risk one of the common failure modes of flat RAG.

This sort of structured, interactive data layer makes agents auditable against a class of “credible but incorrect” hallucinations. Your agent is as good as its tools.

One of my favorite publicly available data sources is the Washington State Legislature1. In the legislature every bill, sponsor, and roll-call vote is public record, yet the questions an advocate or analyst actually cares about are more strategic. “Who is cross-pressured on this issue?” “Where could a coalition form?” There’s no record in the data that directly answers these questions. Answers exist only as patterns blending semantics and relationships: bill text, sponsorships, votes, and oppositions. Strictly speaking, everything is stored; what’s missing is any record that addresses the question. The answer is derivable, but it’s not something retrievable in a particular record.

I built a system to demonstrate how an agent over a structured data layer can derive answers to questions like these.

Data and Methods

The dataset is one biennium of Washington State legislative activity (2025–26): 713 bills that reached a floor vote, 154 members, ~152k individual roll-call votes, ingested via the WA Legislature’s web services. I organized this data in HelixDB as a property graph with Bill and Member nodes connected by sponsored and voted edges. Every Bill node has an embedding of its digest. This allows retrieval in two ways: semantic search over bill digests, and traversal over relationships. I built a PydanticAI agent with tools for both kinds of search.

Some benefits to this structure are obvious, so I kept the evaluation set short: I wrote 12 questions in three categories. Four each of semantic (“which bills deal with unemployment insurance?”), relational (“which cosponsors of SB 5041 voted against it on final passage?”), and hybrid (“find the major rent-stabilization bill and report how close its Senate vote was”). I evaluated two agents with different tools: the control gets flat vector search over bill digests, and the graph agent gets the same embeddings search plus traversal tools.

The control (vector search only) scored 4/4 on the semantic queries. It’s not a big surprise - retrieval is good as long as there’s a relevant passage with the answer. Also as we’d expect, vector search performed poorly on the relational and hybrid questions, succeeding 0/8 times. It could find the right bill and then say (correctly) that no digest could tell it about voting relationships. The graph agent scored 12/12. This could be useful, but a SQL database with a vector index could tell the same story. The interesting question is more strategic.

Exploring Questions with No Record

I chose a test case without a specific bill of its own. The idea of “regulatory burden” is scattered across legislation for taxes/fees and mandates, so there’s not a single bill to pin it to. Washington State Legislature is under strong party control, which means nearly everything that reaches the floor passes. The interesting signal then is who deviates from party patterns. The query: “I represent a coalition of small business owners concerned about growing regulatory and compliance burden. Which WA legislators should we prioritize for outreach? Focus on themes across bills rather than any single bill.”

The “regulatory burden” idea shows up indirectly: there are mandate and fee bills the coalition would oppose, and relief bills it would support. This means to answer, the agent has to construct the concept before it can query for an answer. It does a series of semantic searches and targeted graph traversals, splitting the landscape into two clusters:

• Cost and mandate bills. The relevant bills were HB 2081 (a B&O tax surcharge), SB 5786 (broad license and fee increases), SB 5525 (mass-layoff benefit requirements), and HB 1524 (new workplace-standards duties). They’d increase costs or overhead, so our hypothetical coalition would fight them.

• Relief bills. These were near-unanimous: HB 2575 (reduced reporting obligations), SB 5611 and HB 2418 (streamlining permits). Unanimous votes carry no signal, but sponsorship of these bills does. The members who authored these bills chose to be champions for small business relief.

The agent surfaced the swing Democrats who repeatedly crossed party lines on the Cost bills (six House members voted Nay on both HB 2081 and SB 5786) as the highest-value outreach targets, the 22-member bloc that voted Nay on all four mandate bills as the reliable allies, and the Democratic sponsors of the relief bills. For example, it flagged Rep. Edwin Obras, who authored one of the mandate bills and crossed over to vote Nay on another. This sort of signal requires both sponsorships and votes.

The same approach works multi-hop. Provided a query asking who could reach the opposition, the agent walked sponsorship ties outward to identify the crossover Democrats who cosponsor bills with the opposition’s members. Then it searched what those crossovers strongly support, traversing to their sponsored bills and searching the digests for bills related to relief.. In this case, it identified Rep. Adison Richards, who was primary sponsor on bills with Republicans. The clearest thread was HB 1902, a bill creating a permitting-streamlining work group, sponsored by Richards and cosponsored by three of the other crossover Democrats.

Aggregation across bills (voting blocs, crossover counts) ran as structured operations on the data using tool calls, not in the model’s working memory. This allows for structured, grounded reasoning. Each of the agent’s claims can be decomposed into specific edges, and I verified these against the graph.

There was no record the agent could search for stating “Adison Richards is a potential ally on cost-of-doing-business bills.”2 Instead, the agent derived this from semantic similarity, opposition dynamics, and the presence (or absence) of support along party lines.

Why Structure Matters

A flat-vector system can’t do this. It’s not a problem with the embeddings (we use the same embeddings for both agents). But the evidence is distributed across relationships that aren’t encoded in text digests, so this is where the the graph structure comes into play. This isn’t not about using a specific database or tool. I used a graph+vector database, but you could replicate this structure with a SQL database using vector indices and a junction table. The key is that the computation across bills runs as operations against the data instead of in the model’s “head”.

Another benefit is auditability. An LLM asked this question cold may produce a confident answer without grounding. This agent’s answer decomposes into specific graph edges representing how members relate to particular legislation through votes and sponsorship. Each is something I can (and did) verify against the ingested data. In one early experiment, I caught the agent confidently misattributing real votes to the wrong bills (more on that failure and fix in a follow-up post). The resulting data source is rich and auditable, a double win.

Limitations

This is just a demonstration. We’re looking at data from a single legislative session, and I’ve included only floor-vote bills (so anything that failed in committee isn’t present). The step where the agent clusters concepts is interpretive, so the agent's judgment of what counts as "cost" or "relief" is only as good as the model’s reasoning. Ideally, a well-guided system should surface that judgment rather than gloss over it.

Unsurprisingly, prompting also matters: the naive phrasing of the same question produced a single-bill tally. The agent uses these capabilities when the question invited analysis across multiple bills.

Takeaways & Code

The combination of graph and vector structure is usually pitched as a way of extending RAG to relational facts. It’s true, but it undersells the benefits. Stored fact lookup (relational or not) is something databases can always do. But the benefits go one step higher, making the corpus’s latent structure queryable. With the right context and structure, we can explore those unstated patterns and audit what comes out.

The code is available on GitHub. This repository includes the ingest pipeline, graph model, agent tooling, and the eval set with the sample transcripts referenced in this post. The dataset is public record, and the repo has code to retrieve it from Washington State Legislative Web Services.

The “Green CI” mode of evals is a great first step, but if you stop there, you are leaving an actionable signal on the table.

Measurements vs. Verdicts

A unit test returns a verdict: pass or fail, and fail means something is broken. That’s the right model for most software: there’s a correct answer, and anything else is a defect.

Most of what we want to know about an AI product doesn’t have that shape, though. Consider a code review tool aimed at security. The question isn’t “did it pass,” it’s “when a diff introduces a SQL-injection-class vulnerability, how often does the reviewer catch it?” The answer is a number, and most of the time it isn’t green (perfect) or red (broken). It’s a measurement telling you where your product currently sits on an axis you care about.

Accuracy rate isn’t the only metric available. It might be task duration, costs, escalations to human reviewers, or other non-binary quality measures.

We hear about the benchmarks frontier labs are using to evaluate their models, but you can apply the same concept to your domain. How are you benchmarking your agents?

Where Are We Going?

A single performance measurement is a snapshot: reviewer agent identified 68% of vulnerabilities. It’s interesting, and if you peel back the logs you can get a sense for where your failures are. But it gets more interesting when you do something and measure again.

Run the same benchmark after you change the prompt, and it’s 74%. Swap the underlying model, and it falls to 71%. Add a retrieval step, and it drops to 66%, telling you more context isn’t the answer. The measurements are directional: their movement tells you whether you’re building toward the product you want.

A verdict can’t give you this. Green-to-green tells you nothing about whether you improved. And worse, the eval-as-gate disposition will incentivize you to write evals for only the most certain cases. Aspirational cases are to be avoided, because they’re “flaky”.

It’s even more powerful when you layer multiple performance metrics. The model swap that saved cash drove escalations up. It allows you to make tradeoffs intentionally instead of guessing.

The Suite Life

Building a good suite is important. You want to ensure coverage across a range of scenarios, including some that are aspirational. This allows you to capture performance tradeoffs, and get a handle on the model’s jagged edges.

If the reviewer agent identifies 68% across all vulnerabilities, that’s a useful average, but there’s more structure. Break the benchmark out by class (injection, auth, secrets in code, unsafe deserialization) and the flat number resolves into a profile. Suppose the reviewer catches 90% of injection bugs and 30% of auth flaws. That’s far more actionable than “68% identified”, because it’s a different assessment: the reviewer is missing a certain category.

Results like these point toward action. Maybe it’s a prompt issue, or maybe the auth flaws need context beyond the diff and no amount of prompt-tuning will fix it. You don’t have to tweak things and hope for the best. You can actually experiment.

The Price of Direction

It’s not free. A benchmark suite takes time to build and to maintain. You need labeled cases, scored outputs, and experts to calibrate judges. The engineering isn’t the hard part. You can’t write a good suite until you know what your product actually promises: which scenarios matter, which tradeoffs you will accept, and what “good” actually means for the thing you’re building. If you’re skipping evals, is it because you’re dodging data labeling, or because you’re dodging those harder questions?

This is the real reason to do it. A unit test checks whether you kept a promise you already made. A good eval suite makes you state your promise, and then tells you release over release how close you are to keeping it. CI tells you when something breaks. A compass tells you if you’re going the right way.

Here’s the uncomfortable truth: in one week you’ve already lost most of them. I don’t know who you are, but I can tell you that your weekly retention curve looks like this:

In Mixpanel’s 2024 Benchmarks Report, they reported that the average one-week retention rate was 28%.3 And it’s getting worse; the amount varied by vertical but every industry had a year-over-year decrease in retention. This tracks with Li’s finding that the first day is the modal point for conversions by an order of magnitude. By the end of the trial, it’s too late for most users.

We Need to Go Deeper…

The single-biggest hurdle in a free trial is about time, but it’s not the duration of the trial: it’s the time to the user’s activation. “Activation” is product analytics jargon for the moment a user gets to the moment of value – the “aha” moment that demonstrates why they need what you’re selling. Activation is a matter of minutes, hours, maybe days, but not almost certainly not weeks. Before you worry about habits, cannibalizing your paid business, or any of that, you have to go a layer deeper than retention.

The starting point, if you haven’t already, is to define and measure your funnel. The funnel is like a zoomed-in version of the retention curve. The retention curve measures the funnel’s global, end-to-end transition percent:

Your global problem is the aggregate of local problems, and the funnel captures local transitions between the four stages.4 Breaking data out in this way provides metrics to optimize, but I think the most productive use of the funnel is actually as a diagnostic tool. Defining and measuring a funnel requires you to be clear about things like:

• What’s our product’s moment of value? If you have to distill the new user experience into one “aha” moment that leads to conversion, what would it be?5 How is it measured? Your product has a many meaningful interactions, but trying to make sense of them all at once is overwhelming and rarely yields actionable insight. Start with focus.

• What steps are absolutely necessary to get to that moment? As products grow in complexity, their setup processes do too. The funnel provides a lens through which you can focus on what’s most essential, and can give you the confidence you need to defer (or cut!) the rest of the setup process.

• How often do you actually expect users to come back? You’ll probably be measuring Daily Active Users, and you should be, but there’s a good chance you don’t expect most users to return every day. Do you expect them to be there within three days? A week? A month?

If you’re looking for a framework to help you get started, I’ve collected some pointers and first-level diagnostics in the Funnel Diagnostic Worksheet. The worksheet is for subscribers only, but I’ve got great news: subscriptions are free!

Two Common Diagnostic Questions

There are loads of diagnostic approaches to take with a funnel, but there are two high-impact questions to start with:

How many steps are in setup? A Userpilot study found that only 81% of users don’t complete their onboarding checklists.6 Remember: your goal is to get them to “aha”. Cut ruthlessly.

What transition has the lowest conversion percent? This is where you have to roll up your sleeves and dig into the data. There are a few questions to ask your data:

• Have we seen any changes in conversion percent in the last 30 days? 90 days? Year?

• What groups of users are converting well? This can give an enlightening view into your product’s personas, and a pointer for product or UX research.

• When they’re not converting, what are they doing instead? Grab your favorite analytics software (or analytics person) and wade into the data marshes. Sam Zaiss wrote a great piece on this. To quote Sam, the common challenges to address at this step are:

• The most insightful user paths aren’t the most common, so we need another way to identify that signal amidst all the noise.

• The most meaningful events aren’t tracked, because they sit on the seams between features or product teams.

• The most common visualizations aren’t designed to let the key insights pop, making them harder to discover.

Sam Zaiss, Deliberate Data Culture

When you view the funnel as a diagnostic tool, it can shine a light on the best places to put your data-savvy folks to work. As you take action on problems, the local transition rates give you a responsive and measurable indicator of progress.

Ready to Dig In?

If you’ve been through the basics and aren’t sure how to go deeper, I’d love to help. Full Rank is a data science consultancy that works with small teams that don’t have full-time data science support. If you complete the Funnel Diagnostic Worksheet, reach out for a free 30-minute consultation.

Last week, I read that a company in Finland is using AI to find the “perfect coffee blend”. And here I am buying an imperfect blend of beans from a local coffee roaster like a sucker.

There’s little question that AI is everywhere. I have a wide network of product managers and data scientists, and the vast majority of them are working on an AI integration or product of some sort. Companies talk about AI with enthusiasm – who doesn’t want “perfect” coffee? The market’s roaring enthusiasm for AI technology doesn’t transfer to individuals, though. Gallup’s 2024 survey found that only 13% of Americans believe AI does more good than harm. On top of that, 77% don’t trust businesses to use AI responsibly.

I’m a data scientist in Seattle, so I wondered: how are local companies approaching the AI trend? Is it really everywhere in the startup bubble, or am I in an even smaller AI bubble? Is the hype out of control? Are they publicly committed to responsibility? (Spoiler: You’re not going to like all the answers). I took a deep dive into the public-facing content of some of the fastest-growing startups in the Pacific Northwest to analyze their AI-related language.

Data and Methods

The Geekwire 200 is a ranked list of 200 fast-growing startups in the Pacific Northwest. I built a crawler to scan the public-facing websites of the companies on that list, and extract their text. After filtering out sites that prohibit automated crawling, I was left with 187 startups. I pulled down the first two “layers” of content from their websites (the homepage, and all pages linked from it), and extracted user-facing text to analyze common phrases.1

After an initial analysis of frequent phrases, I built three lists:

• Markers for AI. This includes things like “AI”, “artificial intelligence”, and “Generative Models”.

• Markers for hyperbole. This includes things like “visionary”, “bleeding-edge”, “revolutionary”, and “perfect”.2

• Markers for AI responsibility. Things like “Responsible AI”, “bias mitigation”, and “AI ethics”.

The way these three lists overlap (or don’t) is revealing. Onto the findings!

All Aboard the B2B AI Train

The first major difference is clear: 73% of B2B companies are pitching AI, but less than half of consumer and R&D startups are.3 B2B startups have focused these pitches on efficiency. When they talk about AI they use words like “faster” and “maximize engagement”. Consumer companies focus on terms like “conversation” and “personalization”, emphasizing the interactivity enabled by Generative AI.

Transformational. Revolutionary. Super Stoked.

I don’t go to company websites expecting humility, but the language associated with AI often goes a few steps beyond optimism. I looked at the overlap of hyperbolic language and text about AI. Pre-market R&D companies use the most of it ( and that might be where it’s most warranted)4, but this language isn’t reserved for the lab. B2B and B2C companies alike are on the “cutting-edge”, building “revolutionary” tech. Speaking just for myself, this language almost always rings hollow, whether it’s about a spreadsheet or the “perfect” AI-generated coffee blend. 🥱

Who’s Taking Responsible AI Seriously?

Gallup’s findings revealed that people don’t have high hopes that companies will use AI responsibly. But there’s a light at the end of the anxiety-tunnel: 57% of respondents reported that their concerns would be reduced if businesses were transparent about how AI is used. I looked at how companies discuss their commitments and actions to responsible AI.

The current state of the world won’t inspire hope in those 57% of Americans. Only 19% of the companies talking about AI shared anything at all about their commitments to responsible AI, model evaluation, or bias mitigation. I didn’t expect a majority, but I was a little taken aback. There wasn’t a high bar for inclusion here: merely stating “StartupCo believes in responsible AI” would’ve counted. When the vast majority of companies aren’t even paying lip-service to responsibility, is it any wonder that people don’t trust them?

It’s Not All Bleak: Bright Spots and Actions

It doesn’t paint a rosy picture, but there are bright spots. Some PNW startups doing this well. Responsive AI’s Committment to Ethical AI and Humanly’s Ethical AI Manifesto outline the design principles and frameworks used to develop their product. Textio’s Building Responsibly page describes some of the procedural and statistical methods used to evaluate models and mitigate bias.5 If you’re a team building an AI product and you’re not sure where to start with responsible AI, you can look to your peers working on this already.

If you’re ready to roll up your sleeves, Data Statements provide a clear place to start. This form of documentation was originally proposed by Emily M. Bender and Batya Friedman in a 2018 paper at UW. As Gallup reported, transparency is a key first step to winning consumer trust. But documentation about the provenance, characteristics, and handlers of the data provide benefits that go beyond building customer confidence. Every data set is biased, whether you’ve quantified it or not. Data statements like these also provide the builders with a framework for identifying gaps and risks. They’re fertile soil for growing your responsible AI efforts.

Faced with overstocked aisles of AI products, consumers will vote with their wallets. When the hype fades, unease about responsible AI may remain. Why should they trust your company? Have you earned that trust? What do you owe to your customers and neighbors, and how can you get started today?

What About Your Product?

If you haven’t started rigorously evaluating your models, now’s the best time to start. If you could use help measuring and mitigating bias in your AI products, I’ve done it before, and I’d love to work with your team. Reach out and let’s talk about how we can work together to understand the performance of your AI features, identify and address biases, and give you (and your customers) peace of mind.