Skip to main content

What is the Chinchilla scaling law, and why do labs now train models far beyond it?

Deep learning · Pro 1

Short answer

Chinchilla found that for a fixed training compute budget, the best model comes from growing parameters and training tokens together, about 20 tokens per parameter. Many earlier large models were undertrained. Labs now train smaller models far longer, because a smaller model is cheaper to run for millions of users and keeps improving with more data.

Deeper answer

DeepMind trained over 400 models to fit how loss depends on size and data. Their 70B Chinchilla, trained on 1.4 trillion tokens, beat larger models trained on less. The rule minimises training cost only.

When inference dominates, it pays to over-train: Llama 3 8B saw about 15 trillion tokens, nearly 1,900 per parameter. Training compute is roughly 6 times parameters times tokens. Returns diminish, and the next limits are high-quality data and energy.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Model size and scaling laws: parameters, training data and compute

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.