Autoregressive decoding generates one token at a time. Recomputing keys and values for the entire prefix at every step would be wasteful, so each layer stores a key-value cache. At long context or large batch size, this cache can consume more memory than the model weights used by one request.
Let $h_q$ be the number of query heads and $h_{kv}$ the number of key-value heads, with $h_q$ divisible by $h_{kv}$. Grouped-query attention assigns $g=h_q/h_{kv}$ query heads to each shared key-value head. Standard multi-head attention has $h_{kv}=h_q$, while the single-key-value extreme has $h_{kv}=1$.
Ignoring layout overhead, cache storage across $L$ layers, batch $B$, sequence length $n$, head width $d_h$, and element size $b$ bytes is approximately $$M_{KV}=2LBnh_{kv}d_hb.$$ The factor two stores both keys and values. Reducing $h_{kv}$ lowers cache memory and memory-bandwidth demand almost proportionally.
Sharing keys and values does not make query heads identical. Each query head still has its own query projection and can attend differently to the shared key-value representation. The tradeoff is representational capacity: fewer key-value heads compress the distinct subspaces available for storing past-token information.
Tensor layout must make the grouping explicit. A key-value head can be viewed as expanding across its group of query heads without physically copying cached tensors. Accidental materialization removes much of the memory benefit. Specialized kernels should be checked for supported head counts, alignment, and precision.
Evaluate grouped-query designs with both quality and serving metrics. Report cache bytes per token, decode tokens per second, maximum batch size, and task accuracy at relevant context lengths. A configuration that improves isolated latency may behave differently under concurrent batching, where reduced cache pressure can provide the larger system-level gain.