A transformer is built by stacking similar blocks. Each block mixes information across sequence positions with attention, then transforms each position independently with a feed-forward network. Repetition gradually builds richer contextual representations.
Self-attention lets every permitted position collect information from other positions. Its output has the same sequence length and model width as its input, making it compatible with a residual connection that adds the original representation back.
The feed-forward sublayer applies the same small neural network to every position. It usually expands the feature width, applies a nonlinear activation, and projects back to the model width. Attention communicates across tokens; this sublayer computes within each token representation.
Residual connections preserve an easy information path and improve gradient flow through deep stacks. Layer normalization controls feature scale. Modern variants differ on whether normalization occurs before or after each sublayer, which changes optimization behavior.
If the input has batch size, sequence length, and model width, the main block output keeps those dimensions. Attention temporarily creates head dimensions, and the feed-forward network temporarily creates a wider hidden dimension. Shape annotations are invaluable when learning the architecture.
Log activation norms before and after each sublayer, attention entropy, and gradient norms by depth. Exploding scales suggest residual or normalization trouble; identical position outputs suggest masking or broadcast errors. Build one tested block before stacking many.