When is fine-tuning worth the effort?
Prompting, RAG or fine-tuning · Pro 3
Short answer
When a narrow, repeated task needs behaviour that prompting cannot reliably produce, you have hundreds to thousands of high-quality examples, and an eval set to prove the gain. Common wins: a strict output format, a house style, a specialised classification, or making a smaller model match a larger one to cut cost and latency.
Deeper answer
Fine-tuning has real costs: collecting and cleaning data, training runs, evaluating, hosting or paying for the tuned model, and redoing it when the base model improves. A newer base model may make your fine-tune unnecessary.
Before fine-tuning, try better prompts with examples, structured output, and retrieval. If you do fine-tune, keep a held-out test set, compare against the best prompted baseline, and watch for losing general abilities the task still needs.
What a strong answer covers
- Narrow repeated task prompting cannot handle
- Hundreds to thousands of quality examples
- An eval set to prove the gain
- Typical wins: format, style, classification, smaller model
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What would your baseline be before fine-tuning?
- What happens to your fine-tune when the provider releases a better base model?
Common mistakes
- Fine-tuning before trying a well-prompted baseline
- Ignoring the ongoing cost of maintaining training data
Learn it
- How would you build a dataset to fine-tune an LLM for a business task?
- Your fine-tuned model is no better than the base model. How do you debug it?
- Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How do you decide which model size to use for a feature?
- How would you choose a base model to continue training or fine-tune for a specialised domain or language?
- What is prompt caching, and when does it help?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- What is model distillation, and when would you use it?
- A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
- When would you use synthetic data to train or test a model, and what are the risks?