Grouped-Query Attention (GQA)

An attention variant in which multiple query heads share the same key and value heads within each group. In multi-head attention (MHA), each query head has its own K and V heads; in multi-query attention (MQA), all query heads share a single pair. GQA provides an intermediate point between these extremes.

Fewer KV heads reduce the KV cache size and the data read during decode. With 32 query heads and 8 KV heads, four query heads share each K/V pair; holding other dimensions and precision constant, the cache occupies roughly one quarter of its MHA equivalent. Latency gains depend on hardware and the kernel. This configuration belongs to the model architecture, and its evaluation needs to consider quality alongside memory and speed.