What is model distillation, and when would you use it?
Prompting, RAG or fine-tuning · Pro 1
Short answer
Distillation trains a small student model to copy a large teacher model on a task, using the teacher's answers as training data. Use it when a large model does the task well but is too slow or costly at your volume, and the task is narrow enough for a small model to learn.
Deeper answer
The steps are: run the teacher on many real inputs, filter its outputs, fine-tune the student on those pairs, and test the student on real, hand-labelled data it has never seen. A good outcome is quality close to the teacher at a fraction of the cost and latency.
The limits matter. A student rarely beats its teacher and copies its mistakes. It can learn the teacher's style without its reasoning, producing confident wrong answers on harder or new inputs. Check the teacher provider's terms, since many forbid using outputs to train competing models.
What a strong answer covers
- A small student learns from a large teacher
- Uses the teacher's outputs as training data
- Goal is lower cost or latency at similar quality
- Tests on real, unseen data and notes limits
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you decide the student is good enough to replace the teacher?
- What would you do with inputs the student is unsure about?
Common mistakes
- Expecting the student to beat the teacher
- Evaluating the student only against the teacher's outputs
Learn it
Synthetic data and distillation: training with model-made data
- How would you build a dataset to fine-tune an LLM for a business task?
- Your fine-tuned model is no better than the base model. How do you debug it?
- Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How do you decide which model size to use for a feature?
- How would you choose a base model to continue training or fine-tune for a specialised domain or language?
- What is prompt caching, and when does it help?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- When is fine-tuning worth the effort?
- A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
- When would you use synthetic data to train or test a model, and what are the risks?