Model Robustness

The ability to maintain relevant performance and behavior under variations that should not change the expected outcome. For LLMs, this includes paraphrases, option ordering, input noise, controlled sampling changes, and small context changes. The gap between accuracy and consistency appears when a model answers one formulation correctly and fails on a semantically equivalent one.

A robustness test should declare which transformations preserve meaning and which changes should alter the answer. Repeating the same prompt measures sampling variance; testing paraphrases measures sensitivity to wording. Combining these sources into one score makes diagnosis harder, so results should be segmented by perturbation, task, and error consequence.