LLM as a judge: the biases that skew AI evaluation, and the protocol we use against them

Using an LLM as a judge means asking one language model to grade or compare the outputs of others, and it has become the default way to evaluate anything without a single correct answer: ad copy, hooks, captions, chat. It is fast and cheap, and it is biased in measurable ways. Judges favour answers by their position, their formatting, their length, and whether they wrote them. None of that makes judges useless. It makes the protocol around the judge the part that decides whether a result means anything.

What is LLM as a judge, and why do marketing teams rely on it?

Most marketing output has no answer key. Two headlines can both be correct and one still be better. Human review does not scale to hundreds of variants a week, so teams ask a capable model to pick the better of two, or to score each against a rubric. The two common setups are pairwise comparison (which of these two is better) and absolute scoring (rate this from 1 to 10). Most of the bias research below studies the pairwise case, which is also the one we use.

We rely on judges too. Every prose result on our public marketing benchmark is decided by blind pairwise judging, and we publish the protocol with the numbers. What follows is what the research says can go wrong, what we do about it, and where our own benchmark is still exposed.

The four LLM judge biases that matter most

Ye and colleagues catalogue twelve potential biases in LLM judges and find that even strong models show significant bias on some tasks. Four come up again and again in practice.

The two position-bias studies do not agree on how large it is. Shi and colleagues, across 15 judges and more than 150,000 evaluations, found it systematic and tied to how close the candidates are. The 2026 mitigation study measured it at 0.04 or lower on its benchmarks, far below style bias, and found that swapping positions helped one model on one benchmark while hurting every model on an adversarial set. The honest reading is that position bias depends on the judge and the task, so you measure it rather than assume it away.

How we control for position bias: both orders, every time

Our rule is that every prose pair is judged twice, once in each order, by three judges. A verdict counts only when it holds in both orders. We then report order consistency next to the result, so a reader can see how much of the outcome survived the swap.

On short-form video hooks, our fine-tuned model beat the untrained base 36 to 9 on a 49-item exam, with order consistency of 49 out of 49 and a one-sided binomial p of 0.000033. That combination is the point: a large margin, every verdict stable when the order flipped, and a significance test on the count. It is the only pairwise judging result in our build log that clears significance, and the reason we trust it is the protocol rather than the margin. We covered what that result means for training decisions in when to fine-tune an LLM.

The same protocol also produced results against us, which we publish just as prominently. On campaign diagnosis, the untrained base beat our adapter 9 to 1. On open-ended chat, one run had the base ahead and a later run had ours ahead 7 to 3, and we claim neither: a ten-item slice cannot separate two models when training seed alone moves ten-item scores by up to 0.34.

Self-preference: the bias our own benchmark still carries

The bias that is hardest to remove is the one where the judge, or the answer key, comes from a model in the contest. Panickssery, Bowman and Feng found that models recognise their own writing better than chance, and that the stronger the recognition, the stronger the preference for it.

Our benchmark has a version of this, and we state it on the site. The three blind annotators who wrote the original answer key for our 150-ad exam are Claude agents, not people. That means the agreement ceiling we publish is not a human ceiling, and on that version of the exam Claude's own entry was graded against a key its model family produced. It is the likeliest reason Claude scored above the annotator ceiling, an advantage no other entry had. The current version of the key was re-adjudicated by Claude Opus, so the same caveat applies to any Claude model measured on it. We label it rather than hide it, and the fix (a human-annotated key, or annotators from several model families) is still to do. Our marketing AI benchmark comparison shows how one public benchmark tested LLM judges against 678 human creatives.

If you take one thing from this section: when a vendor's model wins on an evaluation judged or keyed by the same vendor's model, discount the win until someone else reruns it.

An LLM-as-a-judge protocol you can copy

1. Freeze the exam. Fix the items before any model sees them, and keep them out of every prompt and training set.
2. Blind the labels. Call the candidates A and B, never by model or vendor name.
3. Judge both orders. Count a win only when it survives the swap, and report order consistency.
4. Normalise format and length. Strip markdown, or render both answers the same way. If length differs a lot, say so or control for it.
5. Use more than one judge, from more than one family. Three judges is a sensible minimum, and none of them should be a model in the contest.
6. Test the count, not the vibe. A binomial test on wins tells you whether 7 to 3 means anything. Usually it does not.
7. Measure your own noise. Rerun with different seeds before believing a small gap. Short, low-volume Performance Max asset tests fail for the same reason small judging sets do.
8. Check the harness before the judge. We once reported a 0.947 score that came from the evaluator accidentally sending the model the answer. Measured properly it was 0.04. A bias study cannot catch a bug like that; reading the raw prompts can. We list this and five other ways a score misleads in our post on benchmark contamination.

Where gloofy fits, and where it does not

gloofy is built for marketing judgments with a closed set of answers: labelling an ad's hook, angle, persona, offer and funnel stage. Those are scored by exact match against a key, with no judge model in the loop, which removes position, style and verbosity bias from that part of the evaluation entirely. That is a deliberate design choice, and our released 4B model is measured that way.

gloofy is not a general-purpose judge for open prose, and we do not offer it as one. For ad copy, hooks and chat we still depend on LLM judges, under the protocol above, and the answer key for our tagging exam still carries the self-preference issue described earlier. On the earlier version of that exam, Claude and Gemini scored higher than gloofy; the current version has not measured them yet, and the leaderboard says both. More on how we measure, and where we have had to correct ourselves, collects in the benchmarks and evaluation hub.

What to do this week

• Find one decision in your team that rests on an AI grader's score, and check whether the outputs were judged in both orders.
• Strip formatting from candidates before judging; markdown alone can decide a contest.
• Check who judged and who wrote the key. If it is the same vendor as the winner, rerun it with a different judge.
• Stop reporting wins on fewer than about 20 pairs without a significance test.

FAQ

What is LLM as a judge?

It is the practice of using one language model to grade or compare outputs from other models, usually by picking the better of two answers or scoring each against a rubric. It is widely used where no single correct answer exists, such as ad copy or chat, because it is far cheaper and faster than human review.

What is position bias in LLM-as-a-judge evaluation?

Position bias is a judge preferring an answer because of where it appears in the prompt, first or second, rather than its quality. Research across 15 judge models found it systematic and strongest when the two answers are close in quality. The standard control is to judge every pair in both orders.

Do LLM judges prefer their own outputs?

Often, yes. Research by Panickssery, Bowman and Feng found that models can recognise their own text better than chance, and that this self-recognition goes with rating their own outputs higher. Avoid letting any model judge, or write the answer key for, a comparison in which it or its model family is competing.

How many judges and examples does an LLM-as-a-judge evaluation need?

Use at least three judges, ideally from different model families, and judge each pair in both orders. Then run a significance test on the win count. A result like 7 to 3 on ten items is usually indistinguishable from chance. Our clearest result, 36 to 9 on 49 items, has a one-sided p of 0.000033.