What is transfer learning, and when would you use it?
Deep learning · Intermediate 2
Short answer
Transfer learning starts from a model pretrained on a large dataset and adapts it to your task, instead of training from scratch. You can freeze the pretrained layers and train a new head, or fine-tune some or all layers with a small learning rate. Use it whenever data is limited and a related pretrained model exists.
Deeper answer
The early layers of a vision model learn general features such as edges and textures that transfer to almost any images; later layers are more task-specific. With a few hundred labelled images you might freeze the backbone and train only a classifier; with tens of thousands you can fine-tune more layers.
The same idea drives language models: a pretrained LLM fine-tuned or prompted for your task is transfer learning. Watch for domain mismatch, such as models pretrained on photos used for X-rays, and use the same preprocessing the pretrained model expects.
What a strong answer covers
- Start from a pretrained model
- Freeze layers and train a new head, or fine-tune
- Use when data is limited
- Small learning rate and matching preprocessing
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How do you decide how many layers to unfreeze?
- Why use a smaller learning rate when fine-tuning?
Common mistakes
- Using different input normalisation from the pretrained model
- Fine-tuning everything at a high learning rate on tiny data
Learn it
- Explain backpropagation in simple terms.
- What is the Chinchilla scaling law, and why do labs now train models far beyond it?
- How would you estimate the compute and cost to train a language model from scratch?
- Which fine-tuning settings matter most, and how do you choose them?
- Your neural network overfits. What do you try?
- What are vanishing and exploding gradients, and how are they addressed?
- What is model merging, and when would you use it?
- Why do convolutional neural networks work well for images?