A good action can occur in an episode that ends badly, and a poor action can occur in a lucky episode. Learning becomes less noisy when each action is compared with the result normally expected from the same situation.
A state-value function predicts the future return from an observation when the current policy is followed. A value network learns this prediction. It is called the critic because it evaluates situations, while the policy is called the actor because it chooses actions.
An action value predicts return after choosing one particular action. The advantage is the action value minus the ordinary state value. Positive advantage means the action did better than expected; negative advantage means it did worse.
A one-step temporal-difference error compares the current value prediction with the reward plus the discounted next value. At a true terminal state there is no next value. At a time-limit cutoff, the task may continue conceptually, so bootstrapping can still be appropriate.
Generalized advantage estimation blends short and long prediction horizons. Its lambda setting trades lower variance against lower bias. Values near one use more sampled rewards; smaller values rely more heavily on the critic's predictions.
Monitor value loss, explained variance, and the scale of advantages. A constant critic, extreme advantage outliers, or incorrect terminal flags will mislead PPO. Normalize advantages for optimization, but retain original values for debugging.