PPO alternates between two clear phases. First freeze a rollout policy and collect fresh experience. Then compute targets and run several optimization passes over that batch. Discard it afterward because later policy versions require new on-policy data.
A rollout buffer stores observations, actions, rewards, termination and truncation flags, old values, and old log probabilities. Keep time and environment dimensions consistent. Bootstrap a nonterminal final observation with the critic, but never bootstrap beyond a true terminal state.
Work backward through the rollout to calculate temporal-difference errors and generalized advantages. Add each advantage to its old value prediction to create a critic target. Normalize advantages, shuffle samples, and split them into minibatches.
For each minibatch, recompute new log probabilities, entropy, and values. Form the probability ratio, clipped policy loss, and value loss. Clear gradients, backpropagate, clip gradient norm if needed, and take an optimizer step for only a small number of epochs.
Evaluate with separate episodes that never enter the training buffer. Plot multiple random seeds, not one lucky run. Record return, episode length, value loss, entropy, explained variance, approximate KL, clip fraction, and gradient norm.
Debug invariants before tuning. Probabilities must sum to one, arrays must have matching lengths, values must be finite, and parameters must change after an update. Check terminal handling and tensor shapes before changing PPO settings such as clip range, lambda, or minibatch size.