Top-K

A sampling method that, at every next token, restricts candidates to the K most probable tokens and samples within that set. It cuts off the tail of unlikely alternatives but uses a fixed limit: K=20 behaves very differently when the distribution is concentrated versus dispersed.

Top-K belongs to model decoding, not document retrieval. A small value can make language repetitive; a large value can reintroduce weak alternatives. The setting makes sense only when evaluated against the desired behavior, not as a universal way to “improve” a model.