Skip to main content

What is DPO, and how is it different from RLHF?

LLM and transformer basics · Pro 1

Short answer

Both tune a model toward answers people prefer, using pairs of a chosen and a rejected answer. RLHF first trains a separate reward model on those pairs, then uses reinforcement learning to maximise its score. DPO skips the reward model and trains on the pairs directly, which is simpler, cheaper and more stable.

Deeper answer

RLHF has more moving parts: a reward model, a reinforcement learning loop and a limit that keeps the model close to where it started. It can work very well but is hard to tune, and the model can learn to exploit weaknesses in the reward model.

DPO turns the same goal into an ordinary training loss: raise the probability of the chosen answer and lower the rejected one, relative to a frozen reference model. For an application team with a few thousand preference pairs, DPO is usually the practical choice. Both depend on good pairs: inconsistent reviewers give inconsistent behaviour.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

How a chat model is made: pre-training, SFT and preference tuning

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.