For parameters $\theta$ and loss $L(\theta)$, a basic stochastic-gradient step is $\theta_t=\theta_{t-1}-\alpha g_t$, where $g_t$ is a mini-batch gradient and $\alpha$ is the learning rate. Each step reacts only to the current batch. That makes the method responsive, but it also lets sampling noise and steep directions repeatedly reverse the update.
Momentum adds a state vector $u_t$. In the normalized convention, $u_t=\beta u_{t-1}+(1-\beta)g_t$ and $\theta_t=\theta_{t-1}-\alpha u_t$. Another common implementation omits the factor $1-\beta$, which changes the scale of $u_t$. Learning rates cannot be copied blindly between these two conventions even when they use the same value of $\beta$.
Unrolling the normalized recurrence gives $u_t=(1-\beta)\sum_{k=1}^{t}\beta^{t-k}g_k$ when $u_0=0$. Recent gradients receive the most weight, while older gradients decay geometrically. A useful approximation to the memory length is $1/(1-\beta)$ updates, so $\beta=0.9$ remembers roughly ten updates and $\beta=0.99$ remembers roughly one hundred.
Curvature explains the benefit. Along a quadratic coordinate $L_i(\theta_i)=\tfrac12\lambda_i\theta_i^2$, plain gradient descent multiplies the coordinate by $1-\alpha\lambda_i$ each step. A large $\lambda_i$ forces a small learning rate to avoid unstable sign-flipping, while a small $\lambda_i$ then makes progress slow. Momentum tends to cancel gradients that alternate across a steep direction and accumulate gradients that remain aligned along a shallow direction.
Momentum is not automatic damping. The pair $(\alpha,\beta)$ defines a second-order recurrence, and an aggressive pair can create growing oscillations or divergence. Useful diagnostics include the loss, $\lVert g_t\rVert_2$, $\lVert u_t\rVert_2$, and the update-to-parameter ratio $\alpha\lVert u_t\rVert_2/(\lVert\theta_t\rVert_2+\epsilon)$. A rising update ratio often exposes instability before the loss becomes non-finite.
Treat the momentum buffer as part of the model's training state. A checkpoint that restores parameters but discards $u_t$ changes the next update and can produce a temporary loss jump. In tuning, choose the update convention first, begin with a conservative learning rate, increase momentum only while checking stability, and compare runs at equal numbers of optimizer updates rather than equal numbers of processed microbatches.