Skip to main content

How would you estimate the compute and cost to train a language model from scratch?

Deep learning · Pro 1

Short answer

Training compute is roughly 6 times parameters times training tokens, in floating-point operations. Divide by a GPU's peak speed times a realistic utilisation of 30 to 50% to get GPU-seconds, then convert to GPU-hours and cost. Choose tokens with a scaling rule, such as about 20 per parameter for compute-optimal training, and check you actually have that much good data.

Deeper answer

For example, a 1-billion-parameter model on 20 billion tokens needs about 1.2 × 10^20 operations, a few hundred hours on one high-end GPU at 40% utilisation. Data is often the real limit, especially for Indian languages, where good text may be a few billion tokens.

Add costs for data cleaning, experiments, failed runs and evaluation, which can exceed the final run. Very often, continuing to train an existing open model is cheaper and better than training from scratch.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Model size and scaling laws: parameters, training data and compute

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.