For a few years, GitHub used an LLM to scan repos for secrets that had been inadvertently committed. Last week, they announced that they’ve replaced the LLM-based scanner with a custom classifier. Their new classifier is 98% cheaper, 86% faster, and has 11.5% higher user acceptance. A big win in pretty much every dimension! One interpretation of this is that they never should’ve used an LLM at all, but I have a different take. Starting with an LLM-based pipeline makes the zero-to-one step more accessible. But this is just the first step: the costly, slow generalist model gets you the most valuable asset for v2: labeled decisions from real users. The data you collect here provides the training set for a faster, cheaper model tailored to your problem.
Starting with the LLM
In the early stages of a classification project, we often suffer from a lack of real-world data. Collecting, purchasing, or building the data to train models like these is costly and time-consuming, and expending those resources before you’ve even proven the concept is a non-starter. LLMs excel at few-shot classification tasks like these, and are a solid place to start. It’s not just getting the feature out the door for its own sake, it’s producing the dataset you were missing! Every time GitHub’s LLM raised a secret alert, it was a human decision-point, and after two years it’s a heck of a training set that you didn’t have to go out and build.
Moving on down
It can be tempting to stop there. Things are working, the business is humming along. But the reason we started with the LLM was because we had little data, and that’s no longer true. The LLM is a jackhammer, and this problem is a nail. With a dataset like this, there are more options. On the same traffic, GitHub’s fine-tuned encoder showed 13% higher precision and 19% higher recall than the LLM. It was trained on their actual decisions, and used the relevant features directly instead of reasoning them out every time.
When is the bootstrap over?
GitHub’s guidance here is “smaller fine-tuned classifier can be the right answer when the task is narrow, the decision boundary is measurable, and serving efficiency is critical.” These criteria tell you if an encoder can do the job. But the question of whether now is the time to replace the LLM involves a few others.
First, can you enumerate your labels? If you don’t yet understand your categories, or you are still refining their hierarchy, then you are still in task discovery. The flexibility of the generalist LLM is an advantage. Once your labels have settled, it’s time to start looking. Second, are you reading the rationale returned by your LLM classifiers? If you are reviewing these logs, or presenting the explanatory text to your users, then you are still in LLM territory. Prose justifications and explanations aren’t something you will get from an encoder. But you have to be honest here. If you’re logging the rationale because you hope to review them “some day”, there’s a good chance you’re paying for reasoning that you will never use. Finally, how much prompt iteration are you doing? If there’s still a lot of churn in your prompts, the task (and associated data) has probably not settled enough yet.
Takeaways
An LLM-based classification pipeline can be a great fit when you are starting from limited data. By shipping it, you are building your product’s learning loop without slowing down. But the justification for starting with an LLM breaks down as your corpus grows. GitHub’s example shows that moving on can be cheaper, faster, and better!
You are ready to move on from an LLM-based pipeline when: you can enumerate your labels, you no longer need to read the rationale, and your task definition has settled. If all these are true, and you have thousands of labels, you are ready to try other approaches.
The only way to know which model is best for you is to measure it, though. Once you have the data, you can try fine-tuning an encoder, or calibrating a decision model. Score them all against your holdout set, and pick the winner.