Transformer

A neural architecture for sequences based on attention mechanisms and feed-forward networks, introduced in 2017 without recurrence or convolution as its central structure. Self-attention lets each position combine information from other positions; positional information preserves order, while residual connections and normalization support deep training. Variants may use an encoder, a decoder, or both.

During training, sequence positions can be processed in parallel more efficiently than in recurrent architectures. Standard attention grows quadratically with context length, and autoregressive generation in decoder-only models still emits tokens sequentially. Transformer names the architecture; the term alone does not imply that a model is large, generative, or trained as a chatbot.