Your neural network overfits. What do you try?
Deep learning · Intermediate 2
Short answer
Confirm it first: training loss keeps falling while validation loss rises. Then get more or augmented data, add regularisation such as dropout and weight decay, use early stopping on validation loss, shrink the model, or start from a pretrained model. Change one thing at a time and compare on the same validation set.
Deeper answer
Data augmentation is often the biggest win for images and audio: random crops, flips, colour changes or noise make the model see varied examples. Dropout randomly zeroes activations during training; weight decay penalises large weights; early stopping keeps the checkpoint with the best validation loss.
Also check for problems that look like overfitting: leakage between training and validation, a validation set from a different distribution, or a bug in evaluation mode, such as forgetting model.eval so dropout and batch norm behave as in training.
What a strong answer covers
- Confirms with training versus validation loss
- More or augmented data
- Dropout and weight decay
- Early stopping
- Smaller or pretrained model
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Why does model.eval matter for dropout and batch norm?
- How does data augmentation reduce overfitting?
Common mistakes
- Changing many things at once
- Forgetting to switch to evaluation mode
Learn it
- Explain backpropagation in simple terms.
- What is the Chinchilla scaling law, and why do labs now train models far beyond it?
- How would you estimate the compute and cost to train a language model from scratch?
- Which fine-tuning settings matter most, and how do you choose them?
- What is transfer learning, and when would you use it?
- What are vanishing and exploding gradients, and how are they addressed?
- What is model merging, and when would you use it?
- Why do convolutional neural networks work well for images?