Confidence Calibration

The process of checking and adjusting whether a confidence score corresponds to the observed frequency of correctness. In a system calibrated for a given population, predictions grouped near 0.8 should be correct roughly 80% of the time. Reliability diagrams, Brier score, and Expected Calibration Error help measure that correspondence across score ranges.

Calibration depends on the distribution, time horizon, and meaning of the original score. Domain shifts can invalidate it, and aggregate ECE can hide dangerous segments. A calibrated score can support abstention, human review, and risk-routing policies; the threshold remains a system-owner decision based on the cost of errors.