Let an input sequence be $X\in\mathbb{R}^{n\times d_{model}}$, with one row per token. Learned projections create queries $Q=XW_Q$, keys $K=XW_K$, and values $V=XW_V$. A query describes what one token seeks, a key describes what another token offers for matching, and a value carries the information that will be mixed into the output.
For key dimension $d_k$, the score matrix is $$S=\frac{QK^T}{\sqrt{d_k}}.$$ Entry $S_{ij}$ compares query $i$ with key $j$. A row-wise softmax produces $P_{ij}=\exp(S_{ij})/\sum_l\exp(S_{il})$, and the attention output is $O=PV$. Each output row is therefore a data-dependent weighted average of value rows.
The factor $1/\sqrt{d_k}$ controls score scale. If query and key coordinates are roughly independent with variance one, their dot product has variance proportional to $d_k$. Without scaling, larger head dimensions push softmax toward near one-hot probabilities, where most score derivatives are tiny. Scaling keeps logits in a more trainable range at initialization.
Masks modify scores before softmax. A causal mask replaces forbidden future positions with a value that behaves like negative infinity, making their probabilities zero. Padding masks remove non-data positions. Applying a mask after softmax is incorrect unless the remaining probabilities are renormalized, and finite low-precision sentinel values must still be sufficiently negative.
Multi-head attention repeats this operation in several projected subspaces, concatenates the head outputs, and applies an output projection. Different heads can represent different matching patterns, but a head is not guaranteed to have a clean human interpretation. Head count also changes per-head dimension, memory layout, and kernel efficiency.
Dense self-attention stores an $n\times n$ score or probability structure per head and performs work proportional to $n^2d_k$. Sequence length can therefore dominate memory. Validate tensor shapes, ensure every softmax row sums to one on unmasked entries, confirm masked probabilities are zero, and measure peak memory as sequence length grows rather than relying only on parameter count.