Skip to main content

How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?

LLM and transformer basics · Intermediate 2

Short answer

In three stages. Pre-training on huge text makes a base model that predicts the next token. Supervised fine-tuning (SFT) trains it on prompt and ideal-answer pairs so it follows instructions. RLHF then uses human comparisons of two answers to train a reward model, and tunes the chat model to give answers that score well.

Deeper answer

Pre-training uses most of the compute and gives the model its knowledge. SFT and preference tuning, together called post-training, use a small share of compute but turn a text-continuer into an assistant.

SFT data is expensive because experts must write ideal answers. Preference data is cheaper, because comparing two answers is easier than writing a perfect one. RLHF keeps the tuned model close to the SFT model so it does not drift into odd text that happens to fool the reward model. Many teams now use DPO, which learns from the same pairs without a separate reward model.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

How a chat model is made: pre-training, SFT and preference tuning

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.