What is DPO, and how is it different from RLHF?
LLM and transformer basics · Pro 1
Short answer
Both tune a model toward answers people prefer, using pairs of a chosen and a rejected answer. RLHF first trains a separate reward model on those pairs, then uses reinforcement learning to maximise its score. DPO skips the reward model and trains on the pairs directly, which is simpler, cheaper and more stable.
Deeper answer
RLHF has more moving parts: a reward model, a reinforcement learning loop and a limit that keeps the model close to where it started. It can work very well but is hard to tune, and the model can learn to exploit weaknesses in the reward model.
DPO turns the same goal into an ordinary training loss: raise the probability of the chosen answer and lower the rejected one, relative to a frozen reference model. For an application team with a few thousand preference pairs, DPO is usually the practical choice. Both depend on good pairs: inconsistent reviewers give inconsistent behaviour.
What a strong answer covers
- Both learn from chosen and rejected answer pairs
- RLHF trains a separate reward model
- RLHF uses reinforcement learning
- DPO trains directly on the pairs and is simpler
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- Where would you get preference pairs in a real product?
- What can go wrong if reviewers disagree a lot?
Common mistakes
- Saying DPO needs no preference data
- Claiming DPO is always better than RLHF
Learn it
How a chat model is made: pre-training, SFT and preference tuning
- At a high level, what does attention do in a transformer?
- What is a context window, and what happens when a conversation gets too long?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What does the temperature setting do?
- What is test-time compute, and how can you use it to get better answers?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?