LLM evaluation (evals)
LLM evaluation, often called evals, is the practice of scoring the outputs of a large language model (LLM) application for quality. A score can come from a person, from code or from another model acting as a judge. Teams compare scores across prompt versions and models to decide whether a change made the answers better or worse.
How it works
Build a dataset of real inputs, with expected outputs where you have them.
Run each version of the application over the dataset.
Score each output with an evaluator: an exact-match or schema check in code, a rubric scored by LLM-as-a-judge, or human review.
Compare the scores before you ship.
If prompt version 1 passes 184 of 200 cases (92%) and version 2 passes 170 (85%), version 2 is a regression even when latency and error rates look the same.
Why it matters in production
Monitoring tells you a request succeeded, not that the answer was right. A model swap or prompt edit can return empty or wrong answers while every dashboard stays green. Evals turn answer quality into a number you can gate releases on, and AI observability tracks it in production.
See also
On the One2N blog
Further reading
