Entropy

In information theory, a measure of uncertainty in a probability distribution. It grows when probability is spread across alternatives and reaches its maximum when all possible states are equally likely. Its value always depends on the state space: two equally likely options and one hundred thousand equally likely options have very different maximum entropies.

In language models, temperature changes the next-token distribution and generally changes its entropy: lower temperature concentrates probability; higher temperature admits more alternatives. In context engineering, “reducing entropy” is a useful metaphor for making intent less ambiguous through constraints, examples, and evidence. Formally, however, ambiguity becomes entropy only after the distribution being measured is defined. Better context does not guarantee a correct output; it only narrows, or sometimes shifts, the space of plausible behaviors.