How would you estimate the compute and cost to train a language model from scratch?
Deep learning · Pro 1
Short answer
Training compute is roughly 6 times parameters times training tokens, in floating-point operations. Divide by a GPU's peak speed times a realistic utilisation of 30 to 50% to get GPU-seconds, then convert to GPU-hours and cost. Choose tokens with a scaling rule, such as about 20 per parameter for compute-optimal training, and check you actually have that much good data.
Deeper answer
For example, a 1-billion-parameter model on 20 billion tokens needs about 1.2 × 10^20 operations, a few hundred hours on one high-end GPU at 40% utilisation. Data is often the real limit, especially for Indian languages, where good text may be a few billion tokens.
Add costs for data cleaning, experiments, failed runs and evaluation, which can exceed the final run. Very often, continuing to train an existing open model is cheaper and better than training from scratch.
What a strong answer covers
- About 6 × parameters × tokens
- Divide by GPU speed with realistic utilisation
- Chooses tokens with a scaling rule
- Notes data and other costs
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- What usually limits small teams more, compute or data?
- When is training from scratch worth it?
Common mistakes
- Forgetting utilisation is well below peak
- Ignoring the cost of experiments and data work
Learn it
Model size and scaling laws: parameters, training data and compute
- Explain backpropagation in simple terms.
- What is the Chinchilla scaling law, and why do labs now train models far beyond it?
- Which fine-tuning settings matter most, and how do you choose them?
- Your neural network overfits. What do you try?
- What is transfer learning, and when would you use it?
- What are vanishing and exploding gradients, and how are they addressed?
- What is model merging, and when would you use it?
- Why do convolutional neural networks work well for images?