Skip to main content

What is perplexity, and what are its limits as a metric?

LLM and transformer basics · Pro 1

Short answer

Perplexity is the exponential of cross-entropy: roughly how many equally likely next tokens a language model is choosing between. Lower means it predicts the text better. It is useful for training curves and comparing base models on the same text, but it is not comparable across tokenizers and does not measure how useful a chat model is.

Deeper answer

Cross-entropy is the average negative log probability the model gave to each actual next token; frameworks report it in nats, and the exponential is the same either way. Values depend on the text: structured text and longer context lower perplexity.

Post-training usually raises perplexity while making models more helpful, so it is a poor way to rank assistants. Useful applications include detecting contamination (memorised text has very low perplexity) and filtering broken or off-language data (very high perplexity).

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Perplexity and cross-entropy: how language models are measured

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.