A complete return gives a valid learning target only after all relevant future rewards are known. Long episodes make that target delayed and noisy. Temporal-difference learning instead updates after each transition by combining one observed reward with the model's current prediction for what follows. This use of an estimate inside another estimate is called bootstrapping.
For a transition $(s_t,r_{t+1},s_{t+1})$, define the target $y_t=r_{t+1}+\gamma V_w(s_{t+1})$ and prediction error $$\delta_t=y_t-V_w(s_t).$$ A tabular update is $V(s_t)\leftarrow V(s_t)+\alpha\delta_t$. With a differentiable value model, one minimizes a loss such as $L_t(w)=\tfrac12\delta_t^2$.
The target has lower variance than a full sampled return because it replaces many random future rewards with one value estimate. The price is bias whenever $V_w(s_{t+1})$ is inaccurate. As learning improves, the target and prediction can refine one another. This moving-target structure is powerful, but it also means stability depends on step size, data distribution, and function approximation.
Terminal handling must be explicit. With terminal indicator $d_t\in\{0,1\}$, use $$y_t=r_{t+1}+\gamma(1-d_t)V_w(s_{t+1}).$$ An environment time limit is not always a true terminal event; setting $d_t=1$ for every truncation can systematically undervalue states near the limit. Store termination and truncation as separate fields.
When differentiating $L_t$, the target is commonly treated as fixed for that update. In code, the value of the next state is detached from the gradient graph, so optimization changes $V_w(s_t)$ toward the target rather than moving both sides simultaneously. Separate target networks or delayed parameters can further slow target movement in off-policy settings.
Useful diagnostics are the mean and standard deviation of $\delta_t$, value predictions, target values, and terminal masks. Also compare predicted values with completed empirical returns on held-out trajectories. A small training error alone is insufficient: a value model can fit self-consistent bootstrapped targets while remaining wrong about actual long-term outcomes.