One attention operation produces one pattern of similarity. Multi-head attention creates several smaller attention operations in parallel, allowing the model to represent different relationships at the same time.
Each head receives its own query, key, and value projections. The model width is split across heads so total computation remains manageable. Every head computes attention independently over the same sequence positions.
Head outputs are concatenated and passed through an output projection. That projection mixes information from the heads and restores the model width. A residual connection can then add the block's original input.
In language, one head might become sensitive to nearby modifiers while another tracks longer dependencies. This is an intuition, not a guarantee: head roles are learned, can overlap, and may change across layers and examples.
The model width must be compatible with the number and size of heads. More heads do not automatically improve a model because each head becomes narrower or total cost grows. Some implementations share key and value heads to reduce inference memory.
Inspect per-head entropy, similarity between heads, and head output norms. Identical heads may be redundant; a head always focused on padding indicates a mask error. Do not treat one attention picture as a complete causal explanation of a prediction.