Last week, I was talking with an applied researcher about some of the gotchas they’d encountered in building evals. One of the things she mentioned was managing dissent between LLM judges representing different stakeholders. Their “consumer” judge and “subject-matter expert” judge don’t always see eye-to-eye.
Two common patterns here are to blend the criteria into one rubric or average the results between judges. Both of these miss the point, though. The disagreements can actually tell you something, and they represent the stakeholder tensions built-in to your product. In this post, I demonstrate the concept with a simple example.
Setup
A 2025 paper (MedScore) assembled a public dataset of patient questions and physician responses. They used the physician’s responses as context and had AI rewrite them. The authors made that dataset public (AskDocsAI). It includes 300 patient questions, each with the physician’s actual reply and an AI rewrite.
Patient communications present a good example of the multiple-viewpoints problem, because doctors and patients prioritize different qualities. I setup LLM judges to represent each of these viewpoints: a patient rubric (addresses my concern, provides me clear next steps) and a clinician rubric (appropriate urgency and certainty, no unsupported claims)1. Each judge outputs a score from 1-5 based on its criteria. 2
I sampled 150 of the patient question/response pairs, and used these judges to evaluate the responses. Both judges used the same underlying model and input context, only the prompt/rubric was changed. For each response, I ran three replications of each judge.
A difference of opinion
We see considerable differences between judge “personas”. The patient and clinician judge scores are at least one point apart on 53% of responses. LLMs are probabilistic, so we should expect some variation. The same judge re-run on the same answer differs by that much only 11% of the time. We’re capturing two distinct viewpoints.
Most of the disagreements are one point out of five, but ten had differences of two points. This goes in both directions: there are answers the patient judge likes and the clinician judge dislikes, and vice versa. Looking at the LLM judge rationales we can see differences in priority.
In one example, someone with a single stitch asks if they can remove it themselves instead of going back to the doctor. The response asks clarifying questions (how long the stitch has been in), and defers its recommendation. The clinician judge gives this response a 4, with rationale like “appropriately asks for more information…rather than guessing.” On the other hand, the patient judge scored it a 2, saying the response “deflects by asking for more info…without providing any interim advice or self-removal instructions.”
Neither judge is “wrong” about its rubric criteria, they were just designed around different priorities. This illustrates a conflict between stakeholders. If we were building this system, this would be a fundamental tradeoff between actionable and cautious repsonses that we’d have to reckon with.
A rubric smoothie?
We might be tempted to simply roll these up into one combined rubric. Then we don’t have to think about tradeoffs at all! I did this, too. I ran a third judge with a combined patient+clinician rubric. Perhaps surprisingly, it didn’t act as a compromise. It scored lower than either of the targeted judges. Merging the criteria didn’t average the two viewpoints. It made an even stricter judge that never awarded a five! This is likely because the combined rubric covers so much it finds more reasons to dock points.
On the ten answers where patient and clinician judges differed by two points, the combined judge sides with the patient judge on four, the clinician judge on two, “compromises” on three, and scores lower than either on one. We can’t distinguish between “bad responses” and responses that expose an underlying tension. Worse, we lose visibility into how that tradeoff is being decided. It’s no longer a product decision, we’ve just tossed it to the model.
The other obvious solution is to average the two judge scores. But this has the same problem of flattening meaningful disagreements. In the stitch example, the patient judge’s score of 2 and the clinician judge’s score of 4 average to 3, hiding disagreement.
Handling disagreement
So what can we do? There are a few practical pieces of advice.
Report judge scores separately. If you have sometimes-conflicting priorities to balance, reporting performance in each dimension gives you a clearer view into the tradeoffs you are making.
Route disagreement for human review. When judges disagree, escalate those items for review. In this case, I looked at the ten answers that produced large differences in scores to understand how judgements might conflict.
Decide your priority explicitly. If you have to force multiple judges into a single verdict, make an intentional decision about your priority (e.g., clinician must pass, patient ranks the rest). Social choice theorists have been studying how to aggregate conflicting preferences for over a hundred years, and offers several techniques. Averaging scores across judges is an implicit prioritization, and it’s a strong assumption! You can likely do better for your product.
Takeaways
The ten answers where the judges showed marked disagreement revealed a fundamental tension in this domain. To get to that, I had to go a little deeper than the scores, and read the rationale from each judge. It would’ve been easy to “smooth” this disagreement away and miss the insight and tradeoff entirely.
If you find yourself balancing competing priorities or stakeholders, consider splitting your judges. If you read their rationale when they disagree, you will learn something about your product and its fundamental tradeoffs.
Important to note that I’m not actually evaluating “correctness” of medical advice here. This demonstration is focused on the way judges can represent different viewpoints, not on validating medical advice, so the rubrics do not focus on diagnostics.
In a real application, I would calibrate these judges vs. labels provided by human experts. For the purposes of this demonstration, I just made light adjustments to prompts based on my non-expert judgement.