What is perplexity, and what are its limits as a metric?
LLM and transformer basics · Pro 1
Short answer
Perplexity is the exponential of cross-entropy: roughly how many equally likely next tokens a language model is choosing between. Lower means it predicts the text better. It is useful for training curves and comparing base models on the same text, but it is not comparable across tokenizers and does not measure how useful a chat model is.
Deeper answer
Cross-entropy is the average negative log probability the model gave to each actual next token; frameworks report it in nats, and the exponential is the same either way. Values depend on the text: structured text and longer context lower perplexity.
Post-training usually raises perplexity while making models more helpful, so it is a poor way to rank assistants. Useful applications include detecting contamination (memorised text has very low perplexity) and filtering broken or off-language data (very high perplexity).
What a strong answer covers
- Exponential of cross-entropy
- Roughly how many choices per token; lower is better
- Not comparable across tokenizers
- Does not measure usefulness of chat models
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you compare two models with different tokenizers?
- How can perplexity detect benchmark contamination?
Common mistakes
- Comparing perplexity across different tokenizers
- Using perplexity to rank chat assistants
Learn it
Perplexity and cross-entropy: how language models are measured
- At a high level, what does attention do in a transformer?
- What is a context window, and what happens when a conversation gets too long?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- What is DPO, and how is it different from RLHF?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What does the temperature setting do?
- What is test-time compute, and how can you use it to get better answers?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- Why do language models hallucinate, and how do you reduce it in an app?