Given prompt $x$, preferred response $y^+$, rejected response $y^-$, trainable policy $\pi_\theta$, and fixed reference policy $\pi_{ref}$, direct preference training compares how the trainable policy changes the relative likelihood of the pair. The reference anchors the update to a known behavior distribution.
Define the relative log-likelihood margin $$z_\theta=\beta\left[\log\frac{\pi_\theta(y^+\mid x)}{\pi_{ref}(y^+\mid x)}-\log\frac{\pi_\theta(y^-\mid x)}{\pi_{ref}(y^-\mid x)}\right].$$ The loss is $L(\theta)=-\log\sigma(z_\theta)$. Increasing the chosen response relative to the reference lowers the loss, while increasing the rejected response raises it.
The reference correction matters because a pair may already be strongly separated before training. The objective learns improvement relative to that baseline rather than optimizing raw chosen-minus-rejected likelihood alone. Changing the reference changes every margin and therefore changes the training problem, even when the preference dataset is identical.
The coefficient $\beta$ scales the margin. Large beta makes the sigmoid saturate after smaller log-ratio changes, while small beta produces gentler discrimination. Its practical effect interacts with learning rate, sequence length, label noise, and reference quality. It should be tuned with held-out preference and behavior evaluations rather than copied as a universal constant.
Sequence log probabilities are sums over response tokens. Length differences can therefore influence margins, especially when preferred and rejected responses have different verbosity. Mask prompt tokens consistently, define end-of-sequence handling, and inspect margin versus response length. Per-token normalization is a different objective and should be chosen explicitly.
This is an offline objective over observed pairs. It cannot directly judge responses absent from the comparison distribution, and repeated optimization can exploit dataset artifacts. Track reference divergence, chosen and rejected log probabilities separately, held-out pair accuracy, generation quality, and safety evaluations. A falling preference loss alone does not establish aligned behavior.