What are vanishing and exploding gradients, and how are they addressed?
Deep learning · Intermediate 3
Short answer
In deep networks, gradients are products of many terms. If those terms are mostly small, gradients shrink towards zero and early layers stop learning; if large, they blow up and training diverges. Fixes include ReLU-style activations, careful weight initialisation, normalisation layers, residual connections and, for exploding gradients, gradient clipping.
Deeper answer
Sigmoid and tanh activations saturate, so their derivatives are tiny for large inputs, which made deep networks and plain recurrent networks hard to train. ReLU keeps a derivative of one for positive inputs. He or Xavier initialisation keeps the scale of signals steady across layers. Batch or layer normalisation stabilises activations.
Residual connections, used in ResNets and in every transformer block, give gradients a direct path back through the network, which is a big reason very deep models train at all. LSTMs and GRUs used gates for the same reason in recurrent networks. Gradient clipping caps the norm of the gradient to stop a single bad step from wrecking training.
What a strong answer covers
- Gradients multiply through many layers and shrink or grow
- Vanishing: early layers stop learning
- Exploding: training diverges, fixed by clipping
- Fixes: ReLU, initialisation, normalisation
- Residual connections
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Why do residual connections help gradients?
- How would you notice exploding gradients during training?
Common mistakes
- Only naming one cause
- Saying ReLU solves every problem (it has dead units)
Learn it
- Explain backpropagation in simple terms.
- What is the Chinchilla scaling law, and why do labs now train models far beyond it?
- How would you estimate the compute and cost to train a language model from scratch?
- Which fine-tuning settings matter most, and how do you choose them?
- Your neural network overfits. What do you try?
- What is transfer learning, and when would you use it?
- What is model merging, and when would you use it?
- Why do convolutional neural networks work well for images?