A deep sequence model repeatedly transforms a token representation. If every layer replaces its input completely, information and gradients must pass through every nonlinear transformation. A residual block instead learns a correction to a shared stream, allowing a layer to preserve its input when the correction is small.
A pre-normalized block has the form $$x_{l+1}=x_l+F_l(N(x_l)),$$ where $N$ normalizes the input and $F_l$ is an attention or feed-forward branch. A two-branch layer applies this pattern twice. Post-normalization instead normalizes after addition, producing different optimization dynamics even when parameter counts match.
The derivative contains an identity route: $$\frac{\partial x_{l+1}}{\partial x_l}=I+\frac{\partial F_l(N(x_l))}{\partial x_l}.$$ During reverse differentiation, gradients can pass through the identity term without being multiplied only by every branch derivative. This helps train deep stacks, although it does not guarantee bounded activations or gradients.
The residual stream is a running sum of contributions. Expanding the recurrence gives $x_L=x_0+\sum_{l=0}^{L-1}F_l(N(x_l))$. This additive view is useful for interventions and diagnostics, but branch outputs depend on earlier accumulated states, so the network is not simply a fixed linear sum of independent features.
As depth grows, residual updates can accumulate excessive variance. Some designs scale branch outputs by a constant or a depth-dependent factor, initialize output projections near zero, or use learned gates. These choices change signal propagation and must match the initialization, normalization placement, and learning-rate regime.
Measure the norm of the residual stream, each branch output, and their ratio across depth and training time. Also inspect gradient norms by layer and test whether removing one branch causes a plausible local change. A stable total loss can hide dead branches, exploding late-layer updates, or a stream whose scale steadily outruns normalization precision.