Explain backpropagation in simple terms.
Deep learning · Intermediate 2
Short answer
Backpropagation works out how much each weight contributed to the error. After a forward pass computes the prediction and loss, it applies the chain rule backwards, layer by layer, to get the gradient of the loss for every weight. An optimiser such as gradient descent then nudges each weight in the direction that reduces the loss.
Deeper answer
Think of a long chain of simple functions. The chain rule says the effect of an early weight on the final loss is the product of the local effects along the path. Backpropagation computes these efficiently by reusing results from later layers as it moves back, so getting every gradient costs only a small multiple of one forward pass, not one pass per weight.
Frameworks such as PyTorch do this automatically: they record operations during the forward pass and call backward to fill in gradients. You still need to understand it to diagnose problems such as vanishing or exploding gradients, and to know why you zero gradients between steps.
What a strong answer covers
- Forward pass computes the prediction and loss
- Chain rule applied backwards layer by layer
- Produces a gradient for every weight
- Optimiser updates weights to reduce loss
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What is the role of the learning rate?
- Why do we call optimizer.zero_grad in PyTorch?
Common mistakes
- Confusing backpropagation (computing gradients) with gradient descent (updating weights)
- Saying it needs a separate pass per weight
Learn it
- What is the Chinchilla scaling law, and why do labs now train models far beyond it?
- How would you estimate the compute and cost to train a language model from scratch?
- Which fine-tuning settings matter most, and how do you choose them?
- Your neural network overfits. What do you try?
- What is transfer learning, and when would you use it?
- What are vanishing and exploding gradients, and how are they addressed?
- What is model merging, and when would you use it?
- Why do convolutional neural networks work well for images?