Skip to main content

What are vanishing and exploding gradients, and how are they addressed?

Deep learning · Intermediate 3

Short answer

In deep networks, gradients are products of many terms. If those terms are mostly small, gradients shrink towards zero and early layers stop learning; if large, they blow up and training diverges. Fixes include ReLU-style activations, careful weight initialisation, normalisation layers, residual connections and, for exploding gradients, gradient clipping.

Deeper answer

Sigmoid and tanh activations saturate, so their derivatives are tiny for large inputs, which made deep networks and plain recurrent networks hard to train. ReLU keeps a derivative of one for positive inputs. He or Xavier initialisation keeps the scale of signals steady across layers. Batch or layer normalisation stabilises activations.

Residual connections, used in ResNets and in every transformer block, give gradients a direct path back through the network, which is a big reason very deep models train at all. LSTMs and GRUs used gates for the same reason in recurrent networks. Gradient clipping caps the norm of the gradient to stop a single bad step from wrecking training.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Neural networks

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.