Attention Mechanism

An operation that combines representations using weights computed between queries (Q) and keys (K). In the scaled dot-product attention used in Transformers, these scores pass through softmax and weight the values (V). Each position's representation incorporates information from the positions allowed by the attention mask.

In self-attention, Q, K, and V come from the same sequence; in cross-attention, queries come from one sequence and keys and values from another. A causal mask blocks access to future positions, allowing an autoregressive model to train on multiple positions in parallel. During generation, tokens still depend on previous ones. With dense attention, the number of compared pairs grows quadratically with context length, so input length and implementation directly affect cost and latency.