The six jobs of an eval program

An eval program is a build-measure-learn loop in miniature. We can think about each job as having a required output that feeds the next phase, with the whole cycle forming a learning loop.

1. Define “good”. Off-the-shelf evals rarely perform well in product-specific scenarios, so the bulk of every product’s evals should be tailored to its goals and moment of value. Building a useful eval program requires you to define what good outputs look like for your product. The output of this job is a framework for evaluation: categories of test cases, with criteria for evaluating them.

2. Define what to test. With a definition of “good”, we have what we need to construct a test set. The best test cases are a blend of real-world samples and intentionally constructed edge- or adversarial-cases. These should tie back directly to your definition of “good”. The output of this job is: specific test cases with evaluation criteria.

3. Measure performance. Build a scorer with rubrics, judges, and code for deterministic evaluation. As with any measurement program, we have to consider the right statistical methods and required sample size. The output of this job is: runnable evals that produce interpretable results.

4. Calibrate. If you’re using models to evaluate models (LLM-judges, classifiers, etc.), then these measures need to be calibrated themselves because their native outputs may not represent what you expect. Your investment here should be calibrated to your level of risk. Curate smaller, human-labeled sets to check your scorers. Output: scoring models calibrated against a validation set (evals in miniature!)

5. Monitor performance. One-time snapshot evals are useful, but ultimately this is a part of your CI process. At a minimum, you’re testing for regressions and evaluating model upgrades. If you’re using evals as a compass, you need ongoing measurement to know where you’re going. Where possible, you also want to be running evaluations against production traffic, or at least logging for future eval development. This job is classic MLE, and its output is a harness with logs and metrics.

6. Learn. This applies to the product and to the eval set itself. For the eval set: read failures as they come in, and use error analysis to refine and build on your eval framework and test cases. Evals serve as a yardstick for product decisions that connect to the models (version changes, reasoning budgets, etc.). The evals are a benchmark to guide judgement. Output: expanding eval-set (and confidence), data-informed product changes.

Mapping the jobs to your team

With these jobs defined, you can map out concrete accountabilities and ownership. The way they fit into your org depends on the structure of your team, but it’s helpful to think about them in terms of the expertise they require.

If you’re an AI startup with no data science team, you have gaps to fill in measurement and calibration. No matter what roles you have on your team, you need domain expertise. The PM fills this role in a pinch, but without a subject-matter expert, evals will tend to be defined by what’s easiest to measure. Scores might be green, but if an expert would reject the results, you’re just playing make believe.

Falling through the cracks

Calibration and monitoring often fall through the cracks because they seem optional. The models are getting better all the time, right? If you’re renting these models without running calibrated scorers on an ongoing basis, you’re putting the performance of your product in someone else’s hands on little more than faith. The same goes for your LLM judges: when the vendor’s model update comes out, your scores will shift, regardless of whether your product did.

The frontier labs are incentivized to make their models better, but better against their benchmarks does not necessarily mean better at your tasks.

So, should PMs own evals?

Sure, at least in part. The first two jobs and the learning loop clearly land in PM territory. But it takes a village to build an effective eval program. Start with the six jobs, find out who on your team is well-positioned to do them, assign accountabilities, and see what’s left. If you’ve still got a gap, now you know what you’re hiring for.

This one goes to eleven eight

A recent paper from a team at Amazon measured what models emit when you ask for confidence on a 0 to 100 scale. They found that, on a typical sentiment analysis task, the model produced eight distinct values across the whole dataset. More than half of the responses were exactly 95! This same pattern showed up across every model and dataset they tested. When you ask for a score from 0 to 100, you think you’re giving the LLM control over a precise dial, but it’s actually just got 8 notches we can tell which one is the model’s favorite.

When the score of 95 doesn’t reliably mean “95% correct”, or “probability of 0.95”, we have to rethink the way we make decisions down the line. What we think are finely-tuned thresholds lose their meaning. This isn’t an exotic failure case. Any time you’re asking an LLM for a number, you have to wonder if you’re getting the calibrated precision you want. At best, you’re making bulk decisions when you think you’re routing carefully. At worst, you’re letting the illusion of precision lead you toward false confidence.

What can we do?

The model’s confidence judgements aren’t useless, though. Researchers found they are pretty good at ordering. Given cases that need to be ranked into coarse categories (this one is good, this one is ambiguous, this one is bad), the model’s ordering is meaningful. But they’re bad at attaching precise, calibrated values to those rankings. The problems arise when we ask for the latter thing when the model can only reliably deliver the former.

Approach the problem in a different way. Instead of a 0 to 100 scale, ask for a small set of discrete levels that you design as labels plus criteria. In a multi-class classification problem, you might ask it to report one of three options: “High: the record clearly matches one category and no other. Medium: the record matches one category best but has features of another. Low: multiple categories are plausible.”

You might worry: is it losing precision? No, because the model was going to collapse your scale into a couple of bins anyway! This way, you get buckets with criteria you designed, instead of whatever coarse distribution the model falls into. This is also why judges anchored in rubrics outperform judges that just output scores. A rubric is just a set of tiers designed around qualities instead of confidence.

What does “high confidence” actually mean?

Thoughtfully-designed tiers fix what the model emits, but they don’t tell you what “High” confidence actually means. For that, we can go back to our roots in classical ML: calibration. Take a sample of data for which you have ground truth labels, and run your classifier/judge over it. Group those by the model’s reported tier, and compute the accuracy within each. A few lines of code is all it takes to get a routing table.

Something like: “High” is 94% accurate and covers 67% of traffic. “Medium” is 81% accurate and 21% of traffic. “Low” is 55% accurate with 12% of traffic. That allows us to make real decisions: if the bar for auto-accepting is 90%, “High” passes and everything else goes to review. We know to expect review load to be 33% of volume. If we’re surprised by the accuracy-to-tier mappings, we learn something about the task.

The model can supply the ordering, but the accuracy figures are an empirical question. Plus, when the model changes, or behavior drifts, we can rerun the slice and update the table.

What if you need precision?

Sometimes, you genuinely need finer grained scores than a handful of tiers can give. There are still two options. If your model provider exposes token logprobs, you can read the probability of the answer token directly. If not, consider running the same input several times and using the agreement rate. (This does multiply your inference costs). But start by asking if the precision is actually enabling better decision-making. Most routing problems don’t need it.

Tuned at the factory?

There’s an old joke about an amateur musician declining a tune up, saying "No thanks, mine was tuned at the factory."

I’ve written previously about how evals are an instrument to tell you where you are going. Like any instrument, they need tuning. An LLM’s confidence lacks calibration, and better prompting cannot get you there. Instead, the way forward is to stop asking for what it can’t give you. Ask for tiered judgement and supply the numbers yourself, based on real data. You can ask the model’s opinion, just don’t trust its fidelity.

The junk drawer becomes the junk drawer because it’s bigger than all the little odds and ends that we toss inside. Nothing gets lost or thrown out, and we can always dig things out later. The problem is that the model is doing the digging, and it is just not that good at it.

The quantity, form and sequence of context is a design decision, and like all design decisions it has consequences. This deserves the same care you would give a database schema or API contract.

“It fits” vs. “it works”

Chroma’s Context Rot report tested 18 frontier models (including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3)1 on simple tasks. Things like fact retrieval, conversational Q&A, and reproducing sequences. Every model they tested degraded as input length grew. This started long before they hit the context window’s limit. Even when the context window is a million tokens, the model’s performance starts to decrease at tens of thousands of tokens. The tasks in the study were designed to be simple so that length was the only variable. When we start talking about real workloads with reasoning and ambiguity, we can only expect it to get worse.

We’re not talking 80% or even 50% context here. In many cases this is 10% or less. We should think of the number on the context window as a (marketable) capacity limit more than a performance guarantee. You can fit a lot of clowns in a VW Beetle, but I wouldn’t drive it to IKEA.

It’s not just about too much, there are also risks to plopping in loosely related content. If you’ve ever designed multiple-choice tests for humans, you’re probably familiar with “distractors” - answers that are wrong but look plausibly correct. In the Chroma study, they planted distractors (passages that look relevant but are not) and found performance degraded as a result.

Position matters, too

In Liu et al.’s Lost in the Middle, researchers provided the same set of documents in context, but moved one relevant document around. Their accuracy curves were U-shaped: good when the answer is at the beginning or end of context, and worse when it is buried in the middle. The middle of your context window is where information goes to be ignored. And in a way, the middle gets “bigger” the more we stuff the window.

This applies on a smaller scale, too. Chen et al. showed that just reordering the premises of a logic problem significantly changes reasoning accuracy. Nothing was added or removed, only rearranged. If shuffling identical information changes the output, then context engineering becomes akin to a modeling decision, whether we’re making it intentionally or not.

What earns its keep?

Anthropic’s context engineering guidance has a useful framing for this: models have an attention budget. Every token spends it, and the budget depletes faster than the context window fills. Anthropic’s goal for context assembly is the smallest set of high-signal tokens to maximize the likelihood of the outcome you want.

If we apply this lens, it changes how we engineer context. It’s no longer a question of “could this be relevant?”, but “does this context earn its tokens?” We have to weigh the value added by each bit of added context.

In a way, junk drawer thinking optimizes for the person assembling the context. If nothing is left out, we can’t miss anything, and we can avoid making tough decisions. It’s a shortcut akin to tech debt, but we often take it without proper consideration. Instead, we should optimize for what the model actually needs. What context will actually get used?

Practical tips

In practice, there are a few patterns I recommend:

Curate first. When the quality of retrieved content is disappointing, the path of least resistance is to raise “k” in the top-k retrieved. This is usually the wrong move. Twenty mediocre chunks are worse than three good ones because the other seventeen are distractors. Instead, rerank and make cuts. Treat every chunk you include as a fact that must be relevant

Put critical context at the edges. Instructions, the user’s prompt, and the most critical facts go at the start or the end.

Choose your representation deliberately. You can provide the same underlying data as raw records, prose summaries, structured relationships, etc. But the model will likely not perform equally well across them. I’ve written previously about getting better results by changing how data reaches the model rather than how much of it does. A good evaluation set can help you tune representation.

Prune agent histories. Long-running agents accumulate tool outputs, dead-end explorations, and assorted junk state. Trim your context as you go using compaction and structured notes, and drop tool output once it’s no longer necessary. Treat the agent’s context window as a working set instead of an archive.

Takeaways

We might hope that frontier models will just get better at long context, and none of this will matter. It’s possible, but the trend suggests the opposite. Windows keep getting longer but attention doesn’t. Every model Chroma tested degraded including the ones explicitly built for long context. As agents start running longer, they’ll accumulate more and more junk.

Fortunately, this is a tractable problem, and there are some easy wins. You don’t need a new model or a bigger window, you need to think of the context as a piece of engineering: something worth testing, curating, and designing.

Go open your junk drawer and peek inside. Do you really need all those rubber bands?

To demonstrate this, I fit an encoder model1 like the ones we used in ancient times (circa 2019, pre-GPT-3). The model was fine-tuned on about seventeen thousand tickets that weren’t used last week. This took a little over an hour on my laptop, and cost effectively zero dollars.

Data and Methods

The task and evaluation was the same as last week. We are predicting the CFPB Product category (11 options) from the user-submitted complaint. The evaluation set is the same 750 tickets from the last post. This time, we added two new models to the mix: the fine-tuned ModernBERT encoder and a DistilBERT model as the floor.

Bang for Buck, Revisited

The encoder is significantly better than Haiku (+4.7 points, p < 0.001). When compared to Sonnet and Opus, the encoder performed equally well with 85.5% accuracy.2 So it’s no improvement, but cost is the key. If we estimate the costs of the encoder model based on compute costs, it comes out to something like $0.0041 per 1,000 tickets. This is about 1/300th the cost of Sonnet! 3

Even the most naive option, the DistilBERT model, is statistically indistinguishable from every LLM tier.4 The question we should be asking goes beyond “which LLM works best”, it’s “should we be using an LLM at all?”

Revisiting the Cascade

Last week, the confidence cascade wasn’t a winning strategy because Haiku escalated too many tickets. The encoder does better here. Its confidence is a statistic produced during prediction instead of a subjective judgement. Producing it is free, and doesn’t require a prompt or judge model.

Having the encoder escalate its lowest-confidence 10%5 to Sonnet scores 87.5% accuracy, with a cost of $0.12 per 1,000 tickets. This is significantly better than Sonnet (p = 0.018) or the encoder alone (p = 0.011), at a tenth of Sonnet's price. The encoder is a good judge of what it will get wrong, and a tenth of the tickets carry most of the recoverable error, so its escalations are well-earned.6

How Many Labels Does This Take?

Of course, nothing is truly free. In this case, the catch is that the encoder needs labeled training data and the LLMs need none. The practical question is what labels would cost you. I priced outsourced labeling at $0.15/label,7 but you could plug in your own costs.

In this case, one thousand labels gets the encoder to parity with Haiku. Five thousand is enough to get within a couple points of Sonnet, with no statistical difference in this analysis. If you have the tickets, a couple thousand labels is a day or two of annotation. With the 10% escalation policy, even the 1,000-label encoder is already doing as well as Sonnet at a tenth of the price. The compute costs for training are noise throughout the process: under a dollar per training run.

Rare Category Weakness

This isn’t a silver bullet. The encoder performs worse on rare categories.8 When we fit our model to the empirical distribution, it learns not to focus on classes it rarely sees. A prompted LLM can reason without that prior. In scenarios where tail recall is key, the bias toward LLM may be well-deserved. But a hybrid model, rather than flat encoder, could still be an option.

Takeaways

Last week there were two layers of takeaway. The first was for the task: skip the cascade, default to Haiku if it’s low stakes, otherwise use Sonnet for a few points of accuracy. With other modeling approaches in the mix, our conclusions change. The new routing question isn’t “Haiku versus Sonnet”, but “classifier versus LLM”. For a defined category list with labels available, a fine-tuned encoder is a better default. We can wire an LLM in as the escalation path for the tickets the encoder flags.

This actually reinforces the overall takeaway from the last post: these are empirical questions, and we can bring our full toolkit to bear on them. This includes the tools from the pre-LLM era. By quantifying these tradeoffs, your team can make more informed decisions, grounded in the costs and outcomes you care about. You don’t have to default to the most powerful tool

Sample code and data are available on GitHub.

I tested a few approaches on a common problem: routing support-ticket-style data. In this case, both bits of standard advice failed. The most capable model bought no detectable advantage, and cascading by model confidence managed to be more expensive and no better.

Neither approach is necessarily bad advice, but ultimately the better advice is: you don’t have to guess! Build a model selection strategy based on performance on a baseline set of tasks. This post walks through the process. I don’t have support ticket data I can write about publicly, so I’m using a public data set from the CFPB.1

Data and Methods

The task is ticket sorting: given the text of a consumer complaint, file it under the right product category. The CFPB consumer complaint database contains real complaints about financial products. I’ve used the consumer’s own category choice as the label.2 There are eleven categories, and the model predicts the category based on the complaint narrative.

I sampled 750 complaints from a recent three-month window stratified proportional to category share. I ran four Anthropic models across three price tiers (Haiku, Sonnet, and both same-priced Opus generations: 4.8, and Opus 5, released while I was writing this). I cached and logged all responses.3

Bang for Buck

I compared the labeling task against a floor that used the most common category (30%). As we’d expect, every model does far better than that, with accuracies over 80%.4 But the 2.5x pricing for Opus buys us no detectable boost. The confidence intervals suggest no accuracy gain with Opus, and if anything the point estimate points us the other way, toward Sonnet.

While I was preparing this post, Opus 5 was released, so I retested using the latest model. It cost about $3 and moved nothing: both generations land on an identical 628 of 750.5 This is what it looks like in practice to know how a changing model lineup changes performance.

Paired tests show the same: Sonnet and Opus are both clearly superior to Haiku, but we don’t see any gains from Opus. Paying for the large tier is just burning cash. If we were handling 100,000 tickets a month, Sonnet costs about $78/month more than Haiku and it buys us 4 points of accuracy. That’s a tradeoff you can make depending on the cost of a misfiled ticket. But Opus runs us about $184/month more than Sonnet, and it doesn’t get us anything. That’s no tradeoff at all, it’s just a matter of informed spending.

Is Cascading Smarter?

We might wonder if we could do better with a mix of models. A confidence cascade starts by running the cheap model on everything and requesting a confidence score. If the confidence is below a threshold, it sends the question to a stronger model. This is the standard recommendation in most LLM cost-optimization posts, including the one I was planning to write!

Asking an LLM to self-report confidence can work reasonably well. Haiku’s self-reported confidence separates its correct answers from its wrong ones with an AUC of 0.74, and correct answers carry higher confidence on every model I tested. So the premise is good.

But it falls apart in practice. With the model cascade, Haiku is highly confident on only 21% of tickets. It scores 96.8% on those, so keeping them cheap is right. But the other 79% are low-confidence, so the majority get sent upstream anyway. There’s not a “confident-and-cheap” bucket of tickets big enough to make the cascade work for us. 6

The best Haiku-to-Sonnet cascade (threshold chosen with full hindsight) reaches 85.1% at $1.40 per thousand tickets. Flat Sonnet is 84.9% at $1.23. At this point we’re using Sonnet for four out of five tickets anyway, and every one of those has a Haiku “tax” too. We’re paying more for a statistical tie.

Takeaways and Code

On this task: defaulting to Haiku seems like the good answer. It is only 4 points off the best accuracy for a third of the price, and some of its “misses” are just disagreements with a noisy human label. If those last 4 points matter, buy flat Sonnet. The model cascade isn’t worth it, and neither is Opus.

But that’s not something we can generalize, which is the actual point! These numbers describe one task and setup. The same measurement on your workload could come out totally differently, so the process is what matters here. Define your task, run models against a representative sample and evaluate performance, then compare and decide what to buy. In this case, it took a few hours and less than $20 in tokens. Before recommending high-tier pricing or building a routing layer, this is the sort of audit I start with. It’s a lot cheaper than building blindly!

The question “which model should we use” can be answered empirically and inexpensively, and there’s no one-size-fits-all rule.

Sample code and data are available on GitHub.

But if we’re talking cheap, every model in this post charges by the token. In the next piece, I run the same 750 tickets through the sort of model we all fine-tuned before GPT-3. The inference cost for those? Roughly zero. Subscribe to see how it holds up next week!

The failure

I posed the same small business question to the first version of the agent. It produced an answer that was on the right track but blundered into major factual errors. It got the themes right, identifying a family of bills focused on relief and the opposite family that imposed new fees or mandates. It got the roster of sponsors right. But when it transitioned to the meat of the response: the “cross-aisle Democrats worth cultivating”, it named four Democrats as Nay votes on SB 5525.

Unfortuntaely, zero Democrats voted Nay on SB 5525.

The same response also described a “consistent bloc” of 56 legislators voting Nay on “all or nearly all” of four mandate bills. The real all-four bloc is 22 members. The 56 names were all genuine Nay voters somewhere, but the story overstated how much their votes overlapped.

This verges on worst-case scenario. The response looked credible to someone familiar with the domain. There were specific names, specific bills, and specific vote positions. But many of them were false. Without manual validation, this sort of error can easily slip through unnoticed, undermining the entire analysis.

The diagnosis

Of course, we know LLMs can hallucinate. But the claims were very structured, and almost grounded in the facts: real Democrats with crossover reputations, but the wrong bill. I explored a different hypothesis: that the model didn’t invent the votes, but misassigned real ones.

The working set for that response contained other employer-mandate bills, including HB 1213, a paid-leave bill. In the actual record, Springer, Chapman, and Rule all voted Nay on HB 1213. These were real votes, breaking with their caucus. Richards voted Nay on HB 1213 and HB 2020. All the agent’s “fabricated” SB 5525 Nay votes were a true HB 1213 Nay. One bill-label swap between two adjacent lists accounts for the entire failure.

This specific class of error happens a lot. When the context holds many similar tuples, the model will get “mixed up”. The reason it showed up here and not earlier is instructive: previous runs only drilled down to one bill, so the evidence was compact and easy to attribute. This run was the first exploring a broad set of legislation: fourteen bills and five very similar Nay lists. Summarizing these (”who voted Nay on all four?”) required the model to intersect those lists. It got the list of dissenters right, but got the specific votes mixed up. The 56-member bloc overstatement was the same failure in a different way: set intersection done by the model instead of in code.

Tools maketh the agent

Examining the tools I defined, I discovered I’d made the problem worse. My bill_votes tool returned rows like:

Jim Walsh (R-House): Nay — 3rd Reading & Final Passage

There’s no bill ID here. It only appears in the tool-call arguments, one line above forty rows of results. The graph even stores bill_id denormalized on every VOTED edge, but my initial tool call (in the name of brevity), dropped it. So the model was forced to map rows to bills by where they were in the output. This kind of positional binding across adjacent, similar lists is precisely where things go wrong.

Self-describing rows — HB 1213 | Jim Walsh (R-House): Nay — ... — is a one-line fix that tailors the context for the model.

The real fix

The formatting fix helps, but we’re still just treating a symptom. The real insight lines up with what I discussed in the last post:

This is a failure mode of flat RAG, and while context design goes a long way, ultimately we’re fighting an uphill battle against the model.

For claims like these, the right trust boundary is the boundary of the data layer. Move aggregation into tools:

• voting_bloc(bill_ids, vote) - members who voted a given way on all of these bills

• party_crossovers(bill_ids) - members who voted against their party’s majority, per bill

• verify_claims(claims) - check a list of specific (member, bill, vote, motion) tuples before they’re used in a response.

A “find themes across bills” question needs to do these things, and the tools allow the agent to run them as direct operations on the graph. The new agent chooses to use them. The transcript behind the last post shows voting_bloc and party_crossovers are used to explore the cross bill space, and verify_claims was used to check the named votes before the response was generated. When I added tools to run these operations in the data layer, all the resulting claims could be verified

The takeaway

There’s an obvious takeaway here: give your agents verification tools! The less obvious takeaway is a design principle for agentic retrieval: let the model decide what to compute, but let the data layer compute it. The model’s job is planning and interpretation: which bills constitute the concept, what the crossover pattern means. If the model is also doing all the joins, you risk one of the common failure modes of flat RAG.

This sort of structured, interactive data layer makes agents auditable against a class of “credible but incorrect” hallucinations. Your agent is as good as its tools.

One of my favorite publicly available data sources is the Washington State Legislature1. In the legislature every bill, sponsor, and roll-call vote is public record, yet the questions an advocate or analyst actually cares about are more strategic. “Who is cross-pressured on this issue?” “Where could a coalition form?” There’s no record in the data that directly answers these questions. Answers exist only as patterns blending semantics and relationships: bill text, sponsorships, votes, and oppositions. Strictly speaking, everything is stored; what’s missing is any record that addresses the question. The answer is derivable, but it’s not something retrievable in a particular record.

I built a system to demonstrate how an agent over a structured data layer can derive answers to questions like these.

Data and Methods

The dataset is one biennium of Washington State legislative activity (2025–26): 713 bills that reached a floor vote, 154 members, ~152k individual roll-call votes, ingested via the WA Legislature’s web services. I organized this data in HelixDB as a property graph with Bill and Member nodes connected by sponsored and voted edges. Every Bill node has an embedding of its digest. This allows retrieval in two ways: semantic search over bill digests, and traversal over relationships. I built a PydanticAI agent with tools for both kinds of search.

Some benefits to this structure are obvious, so I kept the evaluation set short: I wrote 12 questions in three categories. Four each of semantic (“which bills deal with unemployment insurance?”), relational (“which cosponsors of SB 5041 voted against it on final passage?”), and hybrid (“find the major rent-stabilization bill and report how close its Senate vote was”). I evaluated two agents with different tools: the control gets flat vector search over bill digests, and the graph agent gets the same embeddings search plus traversal tools.

The control (vector search only) scored 4/4 on the semantic queries. It’s not a big surprise - retrieval is good as long as there’s a relevant passage with the answer. Also as we’d expect, vector search performed poorly on the relational and hybrid questions, succeeding 0/8 times. It could find the right bill and then say (correctly) that no digest could tell it about voting relationships. The graph agent scored 12/12. This could be useful, but a SQL database with a vector index could tell the same story. The interesting question is more strategic.

Exploring Questions with No Record

I chose a test case without a specific bill of its own. The idea of “regulatory burden” is scattered across legislation for taxes/fees and mandates, so there’s not a single bill to pin it to. Washington State Legislature is under strong party control, which means nearly everything that reaches the floor passes. The interesting signal then is who deviates from party patterns. The query: “I represent a coalition of small business owners concerned about growing regulatory and compliance burden. Which WA legislators should we prioritize for outreach? Focus on themes across bills rather than any single bill.”

The “regulatory burden” idea shows up indirectly: there are mandate and fee bills the coalition would oppose, and relief bills it would support. This means to answer, the agent has to construct the concept before it can query for an answer. It does a series of semantic searches and targeted graph traversals, splitting the landscape into two clusters:

• Cost and mandate bills. The relevant bills were HB 2081 (a B&O tax surcharge), SB 5786 (broad license and fee increases), SB 5525 (mass-layoff benefit requirements), and HB 1524 (new workplace-standards duties). They’d increase costs or overhead, so our hypothetical coalition would fight them.

• Relief bills. These were near-unanimous: HB 2575 (reduced reporting obligations), SB 5611 and HB 2418 (streamlining permits). Unanimous votes carry no signal, but sponsorship of these bills does. The members who authored these bills chose to be champions for small business relief.

The agent surfaced the swing Democrats who repeatedly crossed party lines on the Cost bills (six House members voted Nay on both HB 2081 and SB 5786) as the highest-value outreach targets, the 22-member bloc that voted Nay on all four mandate bills as the reliable allies, and the Democratic sponsors of the relief bills. For example, it flagged Rep. Edwin Obras, who authored one of the mandate bills and crossed over to vote Nay on another. This sort of signal requires both sponsorships and votes.

The same approach works multi-hop. Provided a query asking who could reach the opposition, the agent walked sponsorship ties outward to identify the crossover Democrats who cosponsor bills with the opposition’s members. Then it searched what those crossovers strongly support, traversing to their sponsored bills and searching the digests for bills related to relief.. In this case, it identified Rep. Adison Richards, who was primary sponsor on bills with Republicans. The clearest thread was HB 1902, a bill creating a permitting-streamlining work group, sponsored by Richards and cosponsored by three of the other crossover Democrats.

Aggregation across bills (voting blocs, crossover counts) ran as structured operations on the data using tool calls, not in the model’s working memory. This allows for structured, grounded reasoning. Each of the agent’s claims can be decomposed into specific edges, and I verified these against the graph.

There was no record the agent could search for stating “Adison Richards is a potential ally on cost-of-doing-business bills.”2 Instead, the agent derived this from semantic similarity, opposition dynamics, and the presence (or absence) of support along party lines.

Why Structure Matters

A flat-vector system can’t do this. It’s not a problem with the embeddings (we use the same embeddings for both agents). But the evidence is distributed across relationships that aren’t encoded in text digests, so this is where the the graph structure comes into play. This isn’t not about using a specific database or tool. I used a graph+vector database, but you could replicate this structure with a SQL database using vector indices and a junction table. The key is that the computation across bills runs as operations against the data instead of in the model’s “head”.

Another benefit is auditability. An LLM asked this question cold may produce a confident answer without grounding. This agent’s answer decomposes into specific graph edges representing how members relate to particular legislation through votes and sponsorship. Each is something I can (and did) verify against the ingested data. In one early experiment, I caught the agent confidently misattributing real votes to the wrong bills (more on that failure and fix in a follow-up post). The resulting data source is rich and auditable, a double win.

Limitations

This is just a demonstration. We’re looking at data from a single legislative session, and I’ve included only floor-vote bills (so anything that failed in committee isn’t present). The step where the agent clusters concepts is interpretive, so the agent's judgment of what counts as "cost" or "relief" is only as good as the model’s reasoning. Ideally, a well-guided system should surface that judgment rather than gloss over it.

Unsurprisingly, prompting also matters: the naive phrasing of the same question produced a single-bill tally. The agent uses these capabilities when the question invited analysis across multiple bills.

Takeaways & Code

The combination of graph and vector structure is usually pitched as a way of extending RAG to relational facts. It’s true, but it undersells the benefits. Stored fact lookup (relational or not) is something databases can always do. But the benefits go one step higher, making the corpus’s latent structure queryable. With the right context and structure, we can explore those unstated patterns and audit what comes out.

The code is available on GitHub. This repository includes the ingest pipeline, graph model, agent tooling, and the eval set with the sample transcripts referenced in this post. The dataset is public record, and the repo has code to retrieve it from Washington State Legislative Web Services.

The “Green CI” mode of evals is a great first step, but if you stop there, you are leaving an actionable signal on the table.

Measurements vs. Verdicts

A unit test returns a verdict: pass or fail, and fail means something is broken. That’s the right model for most software: there’s a correct answer, and anything else is a defect.

Most of what we want to know about an AI product doesn’t have that shape, though. Consider a code review tool aimed at security. The question isn’t “did it pass,” it’s “when a diff introduces a SQL-injection-class vulnerability, how often does the reviewer catch it?” The answer is a number, and most of the time it isn’t green (perfect) or red (broken). It’s a measurement telling you where your product currently sits on an axis you care about.

Accuracy rate isn’t the only metric available. It might be task duration, costs, escalations to human reviewers, or other non-binary quality measures.

We hear about the benchmarks frontier labs are using to evaluate their models, but you can apply the same concept to your domain. How are you benchmarking your agents?

Where Are We Going?

A single performance measurement is a snapshot: reviewer agent identified 68% of vulnerabilities. It’s interesting, and if you peel back the logs you can get a sense for where your failures are. But it gets more interesting when you do something and measure again.

Run the same benchmark after you change the prompt, and it’s 74%. Swap the underlying model, and it falls to 71%. Add a retrieval step, and it drops to 66%, telling you more context isn’t the answer. The measurements are directional: their movement tells you whether you’re building toward the product you want.

A verdict can’t give you this. Green-to-green tells you nothing about whether you improved. And worse, the eval-as-gate disposition will incentivize you to write evals for only the most certain cases. Aspirational cases are to be avoided, because they’re “flaky”.

It’s even more powerful when you layer multiple performance metrics. The model swap that saved cash drove escalations up. It allows you to make tradeoffs intentionally instead of guessing.

The Suite Life

Building a good suite is important. You want to ensure coverage across a range of scenarios, including some that are aspirational. This allows you to capture performance tradeoffs, and get a handle on the model’s jagged edges.

If the reviewer agent identifies 68% across all vulnerabilities, that’s a useful average, but there’s more structure. Break the benchmark out by class (injection, auth, secrets in code, unsafe deserialization) and the flat number resolves into a profile. Suppose the reviewer catches 90% of injection bugs and 30% of auth flaws. That’s far more actionable than “68% identified”, because it’s a different assessment: the reviewer is missing a certain category.

Results like these point toward action. Maybe it’s a prompt issue, or maybe the auth flaws need context beyond the diff and no amount of prompt-tuning will fix it. You don’t have to tweak things and hope for the best. You can actually experiment.

The Price of Direction

It’s not free. A benchmark suite takes time to build and to maintain. You need labeled cases, scored outputs, and experts to calibrate judges. The engineering isn’t the hard part. You can’t write a good suite until you know what your product actually promises: which scenarios matter, which tradeoffs you will accept, and what “good” actually means for the thing you’re building. If you’re skipping evals, is it because you’re dodging data labeling, or because you’re dodging those harder questions?

This is the real reason to do it. A unit test checks whether you kept a promise you already made. A good eval suite makes you state your promise, and then tells you release over release how close you are to keeping it. CI tells you when something breaks. A compass tells you if you’re going the right way.

Here’s the uncomfortable truth: in one week you’ve already lost most of them. I don’t know who you are, but I can tell you that your weekly retention curve looks like this:

In Mixpanel’s 2024 Benchmarks Report, they reported that the average one-week retention rate was 28%.3 And it’s getting worse; the amount varied by vertical but every industry had a year-over-year decrease in retention. This tracks with Li’s finding that the first day is the modal point for conversions by an order of magnitude. By the end of the trial, it’s too late for most users.

We Need to Go Deeper…

The single-biggest hurdle in a free trial is about time, but it’s not the duration of the trial: it’s the time to the user’s activation. “Activation” is product analytics jargon for the moment a user gets to the moment of value – the “aha” moment that demonstrates why they need what you’re selling. Activation is a matter of minutes, hours, maybe days, but not almost certainly not weeks. Before you worry about habits, cannibalizing your paid business, or any of that, you have to go a layer deeper than retention.

The starting point, if you haven’t already, is to define and measure your funnel. The funnel is like a zoomed-in version of the retention curve. The retention curve measures the funnel’s global, end-to-end transition percent:

Your global problem is the aggregate of local problems, and the funnel captures local transitions between the four stages.4 Breaking data out in this way provides metrics to optimize, but I think the most productive use of the funnel is actually as a diagnostic tool. Defining and measuring a funnel requires you to be clear about things like:

• What’s our product’s moment of value? If you have to distill the new user experience into one “aha” moment that leads to conversion, what would it be?5 How is it measured? Your product has a many meaningful interactions, but trying to make sense of them all at once is overwhelming and rarely yields actionable insight. Start with focus.

• What steps are absolutely necessary to get to that moment? As products grow in complexity, their setup processes do too. The funnel provides a lens through which you can focus on what’s most essential, and can give you the confidence you need to defer (or cut!) the rest of the setup process.

• How often do you actually expect users to come back? You’ll probably be measuring Daily Active Users, and you should be, but there’s a good chance you don’t expect most users to return every day. Do you expect them to be there within three days? A week? A month?

If you’re looking for a framework to help you get started, I’ve collected some pointers and first-level diagnostics in the Funnel Diagnostic Worksheet. The worksheet is for subscribers only, but I’ve got great news: subscriptions are free!

Two Common Diagnostic Questions

There are loads of diagnostic approaches to take with a funnel, but there are two high-impact questions to start with:

How many steps are in setup? A Userpilot study found that only 81% of users don’t complete their onboarding checklists.6 Remember: your goal is to get them to “aha”. Cut ruthlessly.

What transition has the lowest conversion percent? This is where you have to roll up your sleeves and dig into the data. There are a few questions to ask your data:

• Have we seen any changes in conversion percent in the last 30 days? 90 days? Year?

• What groups of users are converting well? This can give an enlightening view into your product’s personas, and a pointer for product or UX research.

• When they’re not converting, what are they doing instead? Grab your favorite analytics software (or analytics person) and wade into the data marshes. Sam Zaiss wrote a great piece on this. To quote Sam, the common challenges to address at this step are:

• The most insightful user paths aren’t the most common, so we need another way to identify that signal amidst all the noise.

• The most meaningful events aren’t tracked, because they sit on the seams between features or product teams.

• The most common visualizations aren’t designed to let the key insights pop, making them harder to discover.

Sam Zaiss, Deliberate Data Culture

When you view the funnel as a diagnostic tool, it can shine a light on the best places to put your data-savvy folks to work. As you take action on problems, the local transition rates give you a responsive and measurable indicator of progress.

Ready to Dig In?

If you’ve been through the basics and aren’t sure how to go deeper, I’d love to help. Full Rank is a data science consultancy that works with small teams that don’t have full-time data science support. If you complete the Funnel Diagnostic Worksheet, reach out for a free 30-minute consultation.

Last week, I read that a company in Finland is using AI to find the “perfect coffee blend”. And here I am buying an imperfect blend of beans from a local coffee roaster like a sucker.

There’s little question that AI is everywhere. I have a wide network of product managers and data scientists, and the vast majority of them are working on an AI integration or product of some sort. Companies talk about AI with enthusiasm – who doesn’t want “perfect” coffee? The market’s roaring enthusiasm for AI technology doesn’t transfer to individuals, though. Gallup’s 2024 survey found that only 13% of Americans believe AI does more good than harm. On top of that, 77% don’t trust businesses to use AI responsibly.

I’m a data scientist in Seattle, so I wondered: how are local companies approaching the AI trend? Is it really everywhere in the startup bubble, or am I in an even smaller AI bubble? Is the hype out of control? Are they publicly committed to responsibility? (Spoiler: You’re not going to like all the answers). I took a deep dive into the public-facing content of some of the fastest-growing startups in the Pacific Northwest to analyze their AI-related language.

Data and Methods

The Geekwire 200 is a ranked list of 200 fast-growing startups in the Pacific Northwest. I built a crawler to scan the public-facing websites of the companies on that list, and extract their text. After filtering out sites that prohibit automated crawling, I was left with 187 startups. I pulled down the first two “layers” of content from their websites (the homepage, and all pages linked from it), and extracted user-facing text to analyze common phrases.1

After an initial analysis of frequent phrases, I built three lists:

• Markers for AI. This includes things like “AI”, “artificial intelligence”, and “Generative Models”.

• Markers for hyperbole. This includes things like “visionary”, “bleeding-edge”, “revolutionary”, and “perfect”.2

• Markers for AI responsibility. Things like “Responsible AI”, “bias mitigation”, and “AI ethics”.

The way these three lists overlap (or don’t) is revealing. Onto the findings!

All Aboard the B2B AI Train

The first major difference is clear: 73% of B2B companies are pitching AI, but less than half of consumer and R&D startups are.3 B2B startups have focused these pitches on efficiency. When they talk about AI they use words like “faster” and “maximize engagement”. Consumer companies focus on terms like “conversation” and “personalization”, emphasizing the interactivity enabled by Generative AI.

Transformational. Revolutionary. Super Stoked.

I don’t go to company websites expecting humility, but the language associated with AI often goes a few steps beyond optimism. I looked at the overlap of hyperbolic language and text about AI. Pre-market R&D companies use the most of it ( and that might be where it’s most warranted)4, but this language isn’t reserved for the lab. B2B and B2C companies alike are on the “cutting-edge”, building “revolutionary” tech. Speaking just for myself, this language almost always rings hollow, whether it’s about a spreadsheet or the “perfect” AI-generated coffee blend. 🥱

Who’s Taking Responsible AI Seriously?

Gallup’s findings revealed that people don’t have high hopes that companies will use AI responsibly. But there’s a light at the end of the anxiety-tunnel: 57% of respondents reported that their concerns would be reduced if businesses were transparent about how AI is used. I looked at how companies discuss their commitments and actions to responsible AI.

The current state of the world won’t inspire hope in those 57% of Americans. Only 19% of the companies talking about AI shared anything at all about their commitments to responsible AI, model evaluation, or bias mitigation. I didn’t expect a majority, but I was a little taken aback. There wasn’t a high bar for inclusion here: merely stating “StartupCo believes in responsible AI” would’ve counted. When the vast majority of companies aren’t even paying lip-service to responsibility, is it any wonder that people don’t trust them?

It’s Not All Bleak: Bright Spots and Actions

It doesn’t paint a rosy picture, but there are bright spots. Some PNW startups doing this well. Responsive AI’s Committment to Ethical AI and Humanly’s Ethical AI Manifesto outline the design principles and frameworks used to develop their product. Textio’s Building Responsibly page describes some of the procedural and statistical methods used to evaluate models and mitigate bias.5 If you’re a team building an AI product and you’re not sure where to start with responsible AI, you can look to your peers working on this already.

If you’re ready to roll up your sleeves, Data Statements provide a clear place to start. This form of documentation was originally proposed by Emily M. Bender and Batya Friedman in a 2018 paper at UW. As Gallup reported, transparency is a key first step to winning consumer trust. But documentation about the provenance, characteristics, and handlers of the data provide benefits that go beyond building customer confidence. Every data set is biased, whether you’ve quantified it or not. Data statements like these also provide the builders with a framework for identifying gaps and risks. They’re fertile soil for growing your responsible AI efforts.

Faced with overstocked aisles of AI products, consumers will vote with their wallets. When the hype fades, unease about responsible AI may remain. Why should they trust your company? Have you earned that trust? What do you owe to your customers and neighbors, and how can you get started today?

What About Your Product?

If you haven’t started rigorously evaluating your models, now’s the best time to start. If you could use help measuring and mitigating bias in your AI products, I’ve done it before, and I’d love to work with your team. Reach out and let’s talk about how we can work together to understand the performance of your AI features, identify and address biases, and give you (and your customers) peace of mind.