How do you decide which model size to use for a feature?
Prompting, RAG or fine-tuning · Pro 2
Short answer
Run your eval set on a few model sizes and compare quality, latency and cost per request. Choose the cheapest model that meets your quality bar for that task. Many products use a mix: a small, fast model for simple steps such as classification, and a larger one only where reasoning quality matters.
Deeper answer
Quality differences between models vary by task: on easy tasks a small model may match a large one; on hard reasoning it may not. Only your own test set tells you.
Routing is a common pattern: classify the request, send easy ones to a cheap model and hard ones to a strong model, or try the cheap model first and escalate when a check fails. Revisit the choice when new models are released, because the price and quality frontier moves quickly.
What a strong answer covers
- Runs evals on several model sizes
- Compares quality, latency and cost
- Chooses the cheapest model that meets the bar
- Mixes models by task
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you route requests between two models?
- How often would you revisit the choice?
Common mistakes
- Always defaulting to the largest model
- Choosing by public benchmarks instead of task-specific tests
Learn it
Quality, cost and speed trade-offs
- How would you build a dataset to fine-tune an LLM for a business task?
- Your fine-tuned model is no better than the base model. How do you debug it?
- Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How would you choose a base model to continue training or fine-tune for a specialised domain or language?
- What is prompt caching, and when does it help?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- What is model distillation, and when would you use it?
- When is fine-tuning worth the effort?
- A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
- When would you use synthetic data to train or test a model, and what are the risks?