Scaled dot-product attention is a short pipeline: compare queries with keys, scale the scores, apply any visibility mask, normalize each row with softmax, and use the resulting weights to blend values.
A dot product is large when two vectors point in similar directions. Computing every query against every key creates a square score matrix for self-attention. Row i describes how position i scores all candidate source positions.
As key dimension grows, unscaled dot products tend to grow in magnitude. Dividing scores by the square root of that dimension keeps softmax from becoming extremely sharp at initialization and helps gradients remain useful.
Before softmax, forbidden connections receive a very negative score. Softmax then gives them effectively zero weight. Each remaining row sums to one. Multiplying this weight matrix by the values produces one contextual output vector per query.
For scores of two and one, the first item receives more weight, but not all the weight. Raising temperature-like scale differences sharpens selection; equal scores create an equal blend. This weighted average is differentiable, allowing the projections to learn.
Verify finite scores, row sums near one, zero weight on masked locations, and expected output dimensions. Extremely one-hot weights early in training suggest missing scaling; uniform weights after training may indicate weak projections or a masking bug.