Which fine-tuning settings matter most, and how do you choose them?
Deep learning · Pro 1
Short answer
Learning rate matters most: too high diverges, too low barely learns; LoRA often starts around 1e-4 to 2e-4, full fine-tuning far lower. Batch size affects stability and memory, with gradient accumulation when memory is tight. Epochs depend on data size; stop when validation loss stops falling. Prompt loss weight is usually small or zero.
Deeper answer
I set them by watching curves, not by guessing: a short sweep of a few learning rates, plotting training and validation loss, then keeping the checkpoint with the best validation score.
Very large datasets may need only one or two epochs; small ones may need several. Very small batches (under about 8) can be unstable. Learning-rate schedules with a short warm-up and decay are common. Every run's settings and data version should be logged.
What a strong answer covers
- Learning rate is key, with sensible ranges
- Batch size and gradient accumulation
- Epochs set by validation loss
- Prompt loss weight or masking
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- What does gradient accumulation do?
- How would the loss curves look if the learning rate were too high?
Common mistakes
- Copying settings from another model without checking
- Training for a fixed number of epochs regardless of validation loss
Learn it
Fine-tuning in practice: tools, settings and reading the loss curve
- Explain backpropagation in simple terms.
- What is the Chinchilla scaling law, and why do labs now train models far beyond it?
- How would you estimate the compute and cost to train a language model from scratch?
- Your neural network overfits. What do you try?
- What is transfer learning, and when would you use it?
- What are vanishing and exploding gradients, and how are they addressed?
- What is model merging, and when would you use it?
- Why do convolutional neural networks work well for images?