How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
LLM and transformer basics · Intermediate 2
Short answer
In three stages. Pre-training on huge text makes a base model that predicts the next token. Supervised fine-tuning (SFT) trains it on prompt and ideal-answer pairs so it follows instructions. RLHF then uses human comparisons of two answers to train a reward model, and tunes the chat model to give answers that score well.
Deeper answer
Pre-training uses most of the compute and gives the model its knowledge. SFT and preference tuning, together called post-training, use a small share of compute but turn a text-continuer into an assistant.
SFT data is expensive because experts must write ideal answers. Preference data is cheaper, because comparing two answers is easier than writing a perfect one. RLHF keeps the tuned model close to the SFT model so it does not drift into odd text that happens to fool the reward model. Many teams now use DPO, which learns from the same pairs without a separate reward model.
What a strong answer covers
- Pre-training on large text makes a base model
- SFT uses prompt and ideal-answer examples
- RLHF learns from human comparisons of answers
- RLHF trains a reward model and tunes against it
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Why is comparison data cheaper to collect than demonstration data?
- What is DPO and why would a team choose it over RLHF?
- Why do chat models sometimes agree with users too readily?
Common mistakes
- Saying RLHF adds new knowledge to the model
- Forgetting the reward model step in RLHF
- Mixing up the base model and the instruct model
Learn it
How a chat model is made: pre-training, SFT and preference tuning
- At a high level, what does attention do in a transformer?
- What is a context window, and what happens when a conversation gets too long?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- What is DPO, and how is it different from RLHF?
- What does the temperature setting do?
- What is test-time compute, and how can you use it to get better answers?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?