Evals (AI System Evaluations)

A structured contract for measuring whether an AI system meets defined criteria on representative cases. An eval combines a population or scenarios, an expected outcome, a scoring method, and a decision rule. Metrics, deterministic validators, human review, and LLM-as-judge are possible instruments within that contract.

The evaluated unit follows the system and its risk. RAG evaluation specializes the contract for retrieval, generation, and end-to-end outcomes; agent evaluation includes trajectory, tools, state, and effects. A useful suite needs a baseline, versioning, segment analysis, uncertainty, and regression cases derived from real incidents.