Skip to main content

How would you choose a base model to continue training or fine-tune for a specialised domain or language?

Prompting, RAG or fine-tuning · Pro 1

Short answer

Shortlist open models on licence, size and hardware. Measure how well each already models your domain: bits per byte on held-out domain text, not per-token perplexity, since tokenizers differ. Check token efficiency for your language. Then run a small fine-tune or task test on real tasks, and pick on task quality, cost and speed together.

Deeper answer

Per-token perplexity rewards tokenizers that split text into many small pieces, so it misleads across models; bits per byte divides the same total surprise by the same text length. A model that already predicts your text well is a better starting point and usually needs less data.

Modelling text well does not guarantee task quality, so a pilot fine-tune on 50 to 100 examples and a task test set settle it. Token efficiency also affects training and running cost, especially for Indian languages.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Perplexity and cross-entropy: how language models are measured

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.