LLM-as-a-judge

LLM-as-a-judge is an evaluation method in which a second large language model (LLM) scores an application's output against a rubric you write. It answers questions that code checks cannot, such as whether an answer is supported by its context or equivalent to an expected answer. Each score costs one extra model call.

How it works

The judge receives the input, the output and the rubric, and returns a score, ideally after a short reason. Known biases include position bias, verbosity bias and self-preference for its own model family. Score one answer at a time and use a judge from a different family. Judging a 2,000-token trace with a 500-token rubric costs about 2,500 input tokens, so teams judge a sample and run free code checks on every trace.

Why it matters in production

A model swap or prompt edit can leave latency, tokens and errors unchanged while answers get worse. In one of our experiments, a cheaper model returned empty answers and the evaluator score fell from 1.00 to 0.00 while every dashboard stayed green. Compare the judge with human scores before you trust its trend.