What is the Chinchilla scaling law, and why do labs now train models far beyond it?
Deep learning · Pro 1
Short answer
Chinchilla found that for a fixed training compute budget, the best model comes from growing parameters and training tokens together, about 20 tokens per parameter. Many earlier large models were undertrained. Labs now train smaller models far longer, because a smaller model is cheaper to run for millions of users and keeps improving with more data.
Deeper answer
DeepMind trained over 400 models to fit how loss depends on size and data. Their 70B Chinchilla, trained on 1.4 trillion tokens, beat larger models trained on less. The rule minimises training cost only.
When inference dominates, it pays to over-train: Llama 3 8B saw about 15 trillion tokens, nearly 1,900 per parameter. Training compute is roughly 6 times parameters times tokens. Returns diminish, and the next limits are high-quality data and energy.
What a strong answer covers
- Fixed compute budget
- Scale size and data together, about 20 tokens per parameter
- Earlier models were undertrained
- Over-training for cheaper inference
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you estimate the training compute of a model?
- Why might a newer small model beat an older large one?
Common mistakes
- Saying bigger is always better
- Treating Chinchilla as the target for product models
Learn it
Model size and scaling laws: parameters, training data and compute
- Explain backpropagation in simple terms.
- How would you estimate the compute and cost to train a language model from scratch?
- Which fine-tuning settings matter most, and how do you choose them?
- Your neural network overfits. What do you try?
- What is transfer learning, and when would you use it?
- What are vanishing and exploding gradients, and how are they addressed?
- What is model merging, and when would you use it?
- Why do convolutional neural networks work well for images?