A Million-Token Bad Habit
Leave the junk drawer for your kitchen
The ever-lengthening context windows might be making us lazy. When context windows were a few thousand tokens, we had to think really hard about what went in. But the windows got huge, and we no longer have to be quite so judicious. It’s tempting. Every document that might be relevant gets tossed in. Bump those top-5 retrievals to 50, why not? Full tool output from a few sessions ago? Couldn’t hurt…
The junk drawer becomes the junk drawer because it’s bigger than all the little odds and ends that we toss inside. Nothing gets lost or thrown out, and we can always dig things out later. The problem is that the model is doing the digging, and it is just not that good at it.
The quantity, form and sequence of context is a design decision, and like all design decisions it has consequences. This deserves the same care you would give a database schema or API contract.
“It fits” vs. “it works”
Chroma’s Context Rot report tested 18 frontier models (including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3)1 on simple tasks. Things like fact retrieval, conversational Q&A, and reproducing sequences. Every model they tested degraded as input length grew. This started long before they hit the context window’s limit. Even when the context window is a million tokens, the model’s performance starts to decrease at tens of thousands of tokens. The tasks in the study were designed to be simple so that length was the only variable. When we start talking about real workloads with reasoning and ambiguity, we can only expect it to get worse.
We’re not talking 80% or even 50% context here. In many cases this is 10% or less. We should think of the number on the context window as a (marketable) capacity limit more than a performance guarantee. You can fit a lot of clowns in a VW Beetle, but I wouldn’t drive it to IKEA.
It’s not just about too much, there are also risks to plopping in loosely related content. If you’ve ever designed multiple-choice tests for humans, you’re probably familiar with “distractors” - answers that are wrong but look plausibly correct. In the Chroma study, they planted distractors (passages that look relevant but are not) and found performance degraded as a result.
Position matters, too
In Liu et al.’s Lost in the Middle, researchers provided the same set of documents in context, but moved one relevant document around. Their accuracy curves were U-shaped: good when the answer is at the beginning or end of context, and worse when it is buried in the middle. The middle of your context window is where information goes to be ignored. And in a way, the middle gets “bigger” the more we stuff the window.
This applies on a smaller scale, too. Chen et al. showed that just reordering the premises of a logic problem significantly changes reasoning accuracy. Nothing was added or removed, only rearranged. If shuffling identical information changes the output, then context engineering becomes akin to a modeling decision, whether we’re making it intentionally or not.
What earns its keep?
Anthropic’s context engineering guidance has a useful framing for this: models have an attention budget. Every token spends it, and the budget depletes faster than the context window fills. Anthropic’s goal for context assembly is the smallest set of high-signal tokens to maximize the likelihood of the outcome you want.
If we apply this lens, it changes how we engineer context. It’s no longer a question of “could this be relevant?”, but “does this context earn its tokens?” We have to weigh the value added by each bit of added context.
In a way, junk drawer thinking optimizes for the person assembling the context. If nothing is left out, we can’t miss anything, and we can avoid making tough decisions. It’s a shortcut akin to tech debt, but we often take it without proper consideration. Instead, we should optimize for what the model actually needs. What context will actually get used?
Practical tips
In practice, there are a few patterns I recommend:
Curate first. When the quality of retrieved content is disappointing, the path of least resistance is to raise “k” in the top-k retrieved. This is usually the wrong move. Twenty mediocre chunks are worse than three good ones because the other seventeen are distractors. Instead, rerank and make cuts. Treat every chunk you include as a fact that must be relevant
Put critical context at the edges. Instructions, the user’s prompt, and the most critical facts go at the start or the end.
Choose your representation deliberately. You can provide the same underlying data as raw records, prose summaries, structured relationships, etc. But the model will likely not perform equally well across them. I’ve written previously about getting better results by changing how data reaches the model rather than how much of it does. A good evaluation set can help you tune representation.
Prune agent histories. Long-running agents accumulate tool outputs, dead-end explorations, and assorted junk state. Trim your context as you go using compaction and structured notes, and drop tool output once it’s no longer necessary. Treat the agent’s context window as a working set instead of an archive.
Takeaways
We might hope that frontier models will just get better at long context, and none of this will matter. It’s possible, but the trend suggests the opposite. Windows keep getting longer but attention doesn’t. Every model Chroma tested degraded including the ones explicitly built for long context. As agents start running longer, they’ll accumulate more and more junk.
Fortunately, this is a tractable problem, and there are some easy wins. You don’t need a new model or a bigger window, you need to think of the context as a piece of engineering: something worth testing, curating, and designing.
Go open your junk drawer and peek inside. Do you really need all those rubber bands?
The Chroma study is from mid-2025, so these models are no longer the latest and greatest. Measurements on current models yielded similar results, though. A recent study found Opus 4.6’s recall dropping ten points after 800k tokens of activity. They even found that this did not require tricky test cases: blatant examples that a low-context model did not miss were sometimes missed due to context rot. (Classifier Context Rot: Monitor Performance Degrades with Context Length. Sam Martin and Fabien Roger. 2026)