Agent Evaluation

A specialization of evals whose unit includes both the result and the path taken by the agent. Measurement covers objective completion, trajectory, tool use, state transitions, cost, latency, and produced effects. A persuasive answer has little value when the agent chooses the wrong tool, repeats an operation, or changes a resource without authorization.

Representative cases should include tool failures, incomplete context, resumption after an interrupt, and stopping conditions. The trace supports trajectory scoring and attributes a failure to the model, retrieval, tool contract, or orchestrator. Permissions and critical effects require deterministic validators; open-ended dimensions may use a calibrated LLM-as-judge and sampled human review.