How would you build a dataset to fine-tune an LLM for a business task?
Prompting, RAG or fine-tuning · Pro 1
Short answer
Define the task, output format and a test set first. Write a labelling guide with hard cases and pilot it with two people until they agree. Label a varied first set of 50 to 100 excellent examples, then deduplicate, fix formats and mask personal data. Fine-tune, test, and grow the set only while scores keep rising.
Deeper answer
Quality beats volume: examples should be relevant, aligned with the task, consistent, correctly formatted, unique and allowed to be used. Coverage matters as much: list how real inputs vary (language, length, topic, format) and sample across them.
To move faster, have a strong model draft outputs and let experts correct them, while spot-checking that reviewers do not wave drafts through. A learning curve at 25%, 50% and 100% of the data tells you whether to collect more or fix quality.
What a strong answer covers
- Starts with the task, format and a test set
- Labelling guide and agreement pilot
- Covers the variety of real inputs
- Cleans: duplicates, format, personal data
- Starts small and grows with a learning curve
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you check that two labellers agree?
- Where would new examples come from after launch?
Common mistakes
- Labelling at scale before a guide and pilot
- Collecting whatever data is easiest
Learn it
Building a training dataset: quality, coverage and quantity
- Your fine-tuned model is no better than the base model. How do you debug it?
- Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How do you decide which model size to use for a feature?
- How would you choose a base model to continue training or fine-tune for a specialised domain or language?
- What is prompt caching, and when does it help?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- What is model distillation, and when would you use it?
- When is fine-tuning worth the effort?
- A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
- When would you use synthetic data to train or test a model, and what are the risks?