Self-attention without position information is insensitive to a shared permutation of token order. Rotary encoding adds position directly to queries and keys while leaving values unchanged. It groups feature coordinates into two-dimensional pairs and rotates each pair by an angle determined by token position and a frequency assigned to that pair.
For one coordinate pair and angular frequency $\omega$, define $$R(p,\omega)=\begin{bmatrix}\cos(p\omega)&-\sin(p\omega)\\\sin(p\omega)&\cos(p\omega)\end{bmatrix}.$$ At position $p$, transform $q_p$ to $R(p,\omega)q_p$ and transform $k_p$ similarly. Several frequencies are applied across the head dimension.
The key identity is $R(p,\omega)^TR(q,\omega)=R(q-p,\omega)$. Therefore the rotated dot product becomes $$[R(p,\omega)q_p]^T[R(q,\omega)k_q]=q_p^TR(q-p,\omega)k_q.$$ The attention score depends on relative offset $q-p$ even though each vector was transformed using its absolute position.
A rotation preserves norm because $R^TR=I$. Position changes orientation rather than vector magnitude, allowing content and position to interact inside the query-key dot product. High-frequency pairs change rapidly across nearby tokens; low-frequency pairs change slowly and can represent broader positional scales.
Long-context extension is not automatic. Frequencies trained on a limited position range may rotate through unfamiliar phases at larger offsets, and high-frequency components can alias different distances. Frequency rescaling or position interpolation changes the geometry and should be evaluated on retrieval and sequence-order tasks, not only average language loss.
During cached autoregressive decoding, a new query must use its absolute sequence position, while cached keys must retain the rotations from their original positions. Off-by-one offsets can silently damage quality without causing shape errors. Unit tests should compare cached and full-sequence logits for the same prefix and verify the coordinate-pair convention exactly.