LLM-as-judge

The use of a language model as a scoring method within an eval, especially when a task allows multiple valid answers and the rubric covers semantic quality, usefulness, or appropriateness. The judge may assign a score, compare two answers, or classify errors, but it remains one component of the evaluation protocol.

A model judge inherits position, style, and model-preference biases and may change with its prompt or version. Clear rubrics, randomized ordering, an expert-annotated sample, and meta-evaluation measure whether it agrees with the relevant human criterion. Deterministic cases remain better served by exact validators.