Suppose you show someone two responses to the same prompt and ask which one they prefer.
Traditional RLHF can turn many such comparisons into a reward model and then optimize a policy against that learned reward while constraining how far the policy moves from a reference model. It works, but it introduces another model and an RL optimization loop.
DPO reaches a related objective more directly.
Given a preferred response and a rejected response, it trains the policy so that the preferred response becomes relatively more likely than the rejected one, compared with a reference policy. There is no separately trained reward model and no on-policy rollout loop during DPO training.
That simplicity is the appeal.
It does not make the underlying preference problem disappear.
If annotators systematically prefer longer answers, the model can learn verbosity. If the comparisons poorly represent difficult cases, DPO does not manufacture the missing signal. If “good” actually combines correctness, safety, tone and product-specific behavior, reducing all of that to pairwise preferences can hide disagreement about what the model is being optimized for.
So the preference dataset is at least as important as the optimization method.
DPO makes preference optimization operationally simpler. It does not tell you whether the preferences themselves reward the behavior you actually want.