A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
Prompting, RAG or fine-tuning · Pro 1
Short answer
Measure on our own test set first, against the cheaper options: a step-by-step prompt and a majority vote of a few samples. If the reasoning model still wins clearly, route only the hard cases to it, such as when cheap samples disagree or a check fails, and keep everything else on the fast model.
Deeper answer
The decision is quality, cost and latency together. I would break results down by difficulty: reasoning models usually help most on multi-step maths, code and planning and add little on simple questions.
To control cost I would cap the effort setting and output length and track cost per request. For latency I would show progress, run work in the background where possible, and measure time to the first word the user sees, because hidden thinking delays it.
What a strong answer covers
- Measures on its own test set
- Tries cheaper options first
- Routes only hard cases to the reasoning model
- Controls cost and latency
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you detect which questions are hard?
- What would you show the user while it thinks?
Common mistakes
- Switching everything to the reasoning model
- Comparing only against a lazy prompt
Learn it
Reasoning models and test-time compute: when to let the model think longer
- How would you build a dataset to fine-tune an LLM for a business task?
- Your fine-tuned model is no better than the base model. How do you debug it?
- Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How do you decide which model size to use for a feature?
- How would you choose a base model to continue training or fine-tune for a specialised domain or language?
- What is prompt caching, and when does it help?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- What is model distillation, and when would you use it?
- When is fine-tuning worth the effort?
- When would you use synthetic data to train or test a model, and what are the risks?