A policy update should increase the probability of actions that produced better-than-expected outcomes and decrease the probability of worse actions. Raw returns mix action quality with state difficulty: even the best action in a bad state may have a low return. The advantage $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)$ removes this state-dependent baseline.
For a value estimate $V_w$, define the one-step residual $$\delta_t=r_{t+1}+\gamma V_w(s_{t+1})-V_w(s_t).$$ If the value function were exact, the expected residual conditioned on $(s_t,a_t)$ would equal the true advantage. In practice it is an inexpensive, low-variance estimate whose accuracy depends on the value model.
A longer estimate combines future residuals: $$\hat A_t^{(\lambda)}=\sum_{l=0}^{T-t-1}(\gamma\lambda)^l\delta_{t+l},$$ where $0\leq\lambda\leq1$. At $\lambda=0$, only the one-step residual remains. As $\lambda$ approaches one, more sampled rewards influence the estimate, reducing dependence on the critic while increasing sampling variance.
The parameters $\gamma$ and $\lambda$ play different roles. Gamma defines how the task values delayed reward, while lambda controls how residuals are mixed for estimation. Their product determines the geometric decay inside the estimator, but changing lambda does not redefine the underlying reward objective in the same way changing gamma does.
Episode boundaries require masks in both the residual and the reverse-time recursion. A convenient implementation is $\hat A_t=\delta_t+\gamma\lambda(1-d_t)\hat A_{t+1}$. Normalizing advantages within a batch can stabilize optimizer scale, but it changes the relative influence of batches and does not correct a biased critic or incorrect terminal mask.
Evaluate the estimator with more than policy return. Log advantage mean, standard deviation, extreme quantiles, critic error against completed returns, and correlation between advantages and later outcomes. If advantages remain strongly shifted away from zero by state or episode position, the critic, masking, reward scale, or data collection policy may be mis-specified.