How would you choose a base model to continue training or fine-tune for a specialised domain or language?
Prompting, RAG or fine-tuning · Pro 1
Short answer
Shortlist open models on licence, size and hardware. Measure how well each already models your domain: bits per byte on held-out domain text, not per-token perplexity, since tokenizers differ. Check token efficiency for your language. Then run a small fine-tune or task test on real tasks, and pick on task quality, cost and speed together.
Deeper answer
Per-token perplexity rewards tokenizers that split text into many small pieces, so it misleads across models; bits per byte divides the same total surprise by the same text length. A model that already predicts your text well is a better starting point and usually needs less data.
Modelling text well does not guarantee task quality, so a pilot fine-tune on 50 to 100 examples and a task test set settle it. Token efficiency also affects training and running cost, especially for Indian languages.
What a strong answer covers
- Shortlists on licence, size and hardware
- Measures bits per byte on domain text
- Notes tokenizer differences
- Decides on real task tests, cost and speed
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Why is per-token perplexity misleading here?
- How big should the pilot be?
Common mistakes
- Picking the largest model by default
- Comparing per-token perplexity across tokenizers
Learn it
Perplexity and cross-entropy: how language models are measured
- How would you build a dataset to fine-tune an LLM for a business task?
- Your fine-tuned model is no better than the base model. How do you debug it?
- Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How do you decide which model size to use for a feature?
- What is prompt caching, and when does it help?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- What is model distillation, and when would you use it?
- When is fine-tuning worth the effort?
- A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
- When would you use synthetic data to train or test a model, and what are the risks?