A deep computation can produce a gradient whose norm is far larger than its usual range. One such batch may cause a destructive parameter jump, overflow reduced-precision values, or push optimizer state into a regime that takes many updates to recover. Gradient clipping limits the update signal before the optimizer consumes it; it does not change the forward loss itself.
For a gradient vector $g$ and threshold $c>0$, global-norm clipping computes $s=\min(1,c/(\lVert g\rVert_2+\epsilon))$ and $\tilde g=sg$. If the norm is below $c$, nothing changes. If it exceeds $c$, every coordinate receives the same scale factor, so the direction is preserved while the norm is reduced to approximately $c$.
If $\lVert g\rVert_2=250$ and $c=1$, the scale factor is approximately $0.004$. A component equal to $30$ becomes $0.12$, and a component equal to $-5$ becomes $-0.02$. By contrast, clipping each component to an interval changes large coordinates by different proportions and can rotate the gradient direction substantially.
Pipeline order matters. With scaled mixed-precision gradients, unscale them before measuring or clipping the norm. With gradient accumulation, clip the final accumulated gradient if the intended object is one optimizer update; clipping every microbatch separately produces a different vector. In distributed training, the norm must represent the same globally reduced gradient that the optimizer will use.
Clipping changes the optimization problem whenever it activates. If $c$ is too low, most updates are forced to nearly constant norm, useful magnitude information disappears, and training can slow or settle differently. Clipping also does not solve vanishing gradients. Log the pre-clipping norm, post-clipping norm, scale factor, and fraction of clipped updates instead of treating the threshold as invisible plumbing.
Choose a threshold from observed stable training, then test it under the largest expected batches, sequence lengths, and loss spikes. A practical target is to leave ordinary updates untouched while limiting rare outliers. If clipping activates continuously, investigate learning rate, loss scaling, data corruption, numerical overflow, and model initialization; persistent clipping may be masking a deeper failure rather than fixing it.