Attention compares token representations without inherently knowing which token came first. If positions are omitted, reordering tokens reorders outputs but does not express word order. Language therefore needs an explicit position signal.
Absolute position embeddings assign a vector to position zero, one, two, and so on. That vector is combined with the token embedding. The same word at different locations then enters the first transformer block with a different representation.
Fixed sinusoidal encodings use waves at several frequencies, while learned position embeddings optimize a table during training. Relative methods instead represent distance or rotation between positions, which can make nearby and far-away relationships easier to express.
In 'dog bites person' and 'person bites dog', token identities are nearly identical but roles reverse with order. Position information lets attention learn directional patterns such as looking to a previous adjective or earlier subject.
A model trained only on short sequences may not behave reliably beyond its training range. Some position schemes have a fixed table limit; others technically extend but degrade. Context length is therefore both an architectural and a training-distribution constraint.
Test a position implementation by permuting tokens, checking tensor shapes, and comparing behavior near and beyond the trained length. Confirm that padding positions do not introduce unwanted signals and that cached decoding advances position indexes correctly.