KV cache (key-value cache)

A KV cache (key-value cache) is GPU memory where a transformer model stores the attention keys and values it has already computed for earlier tokens. Each new token reuses those stored values instead of recomputing them for the whole sequence. This makes generation faster, but the memory it uses grows with context length.

How it works

Inference has two phases. Prefill processes the whole prompt and fills the cache, and decode generates one token at a time while reading from it. Cache size per token is 2 × layers × KV heads × head dimension × bytes per value. For a 7B model with 32 layers, 32 KV heads, a head dimension of 128 and 16-bit values, that is 0.5 MiB per token, or 2 GiB for a 4,096-token context.

Why it matters in production

KV cache memory often limits how many concurrent requests a GPU can serve. Prompt caching reuses the cached prefix of a repeated prompt across requests, which cuts prefill work, lowers time to first token and is usually billed at a lower rate. Keeping prompt prefixes stable is what makes those cache hits possible.

See also

• Time to first token

• Token (LLM)

• Latency and throughput

On the One2N blog

• Token economics, paisa, prompts & production

References

1. Efficient memory management for LLM serving with PagedAttention (Kwon et al., 2023)

2. Attention is all you need (Vaswani et al., 2017)