Evals (AI System Evaluations)

The structured process of measuring whether an AI system meets defined criteria on representative cases. An eval combines a dataset or scenarios, an expected outcome, a scoring method, and a decision rule. It can assess quality, safety, cost, latency, and behavior under failure as long as each dimension has observable evidence.

The right unit is usually the complete system: model, prompt, retrieval, tools, memory, and interface. LLM-as-judge is one possible evaluator, not the entire eval. A useful suite needs a baseline, versioning, segment analysis, and regression cases derived from real incidents.