Suppose prompt $x$ has two candidate responses, preferred response $y^+$ and rejected response $y^-$. A reward model $r_\phi(x,y)$ maps each response to a scalar. The score is not an objective fact about quality; it is a learned summary of the comparison process represented in the training data.
A common model assigns preference probability $$P(y^+\succ y^-\mid x)=\sigma(r_\phi(x,y^+)-r_\phi(x,y^-)),$$ where $\sigma(z)=1/(1+e^{-z})$. The training loss is $-\log\sigma(r^+-r^-)$. Only the score difference matters, so adding the same constant to both rewards leaves every probability unchanged.
The loss encourages correct ordering, not calibrated reward magnitude. Multiplying all score differences changes confidence even when rankings stay fixed. Accuracy at a threshold of one half therefore measures a different property from probability calibration. Both should be checked, along with held-out loss and score distributions.
Preference data can contain disagreement, position effects, inconsistent criteria, and prompt-dependent standards. Repeated labels allow estimation of disagreement rather than forcing every pair into a false deterministic truth. Randomizing response order and preserving evaluator metadata can expose systematic collection bias without placing identity information in model inputs.
A policy optimized against the reward model can move beyond the response distribution on which the reward model was trained. In that region, high predicted reward may reflect exploitable model error rather than better behavior. This is a distribution-shift problem: held-out pair accuracy from the original data does not certify scores on newly optimized outputs.
Maintain an untouched evaluation set, compare score margins by prompt type, inspect high-reward samples manually, and refresh comparisons from the evolving policy. Reward ensembles or uncertainty estimates can flag disagreement, but they do not replace evaluation. The operational target is reliable preference prediction where the policy actually generates, not merely low training loss.