KV Cache (Key-Value Cache)

Inference state that stores the key (K) and value (V) vectors already computed in a causal Transformer's attention layers. During prefill, the model processes the prompt and builds this state; during decode, it adds the new token's vectors and reuses the previous ones. This avoids recomputing the entire prefix at each generation step.

This reuse carries a memory cost: with full attention, the cache grows with sequence length, layer count, KV heads, and active sequences. Long contexts can limit concurrency and throughput. Each new token still attends to the stored keys and values. Reusing prefixes across requests requires matching prefix tokens and a compatible model configuration; in CAG, this reuse supports the preloaded knowledge base.