Token

A discrete unit that a tokenizer converts into an identifier for a model to process. A token may correspond to a whole word, part of a word, punctuation, whitespace, or a byte sequence, depending on the vocabulary and tokenization algorithm. The model receives these identifiers as vector representations and, during autoregressive generation, computes a distribution over the possible next token.

A token is not synonymous with a word. The same sentence may produce different counts across models and languages, affecting context-window capacity, latency, and cost. Character-based estimates are only approximations; operational limits should use the tokenizer associated with the model and include system messages, history, tool results, and special tokens.