Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
Prompting, RAG or fine-tuning · Pro 3
Short answer
For a small, stable document set, putting it all in the prompt can be simpler and work well, especially with prompt caching. For large or changing collections, RAG is cheaper, faster and scales: you send only relevant text, can enforce access control per user, and can cite sources. Models can also miss details buried in very long inputs.
Deeper answer
Cost and latency grow with input tokens on every request, so a full-corpus prompt is expensive at volume, though caching a fixed prefix helps. Very long inputs can reduce attention to material in the middle. And a corpus of millions of documents will not fit anyway.
A practical answer is "it depends, and here is how I would decide": if the knowledge base fits comfortably and rarely changes, try long context with caching and measure; otherwise use retrieval, perhaps retrieving larger sections to give the model more context.
What a strong answer covers
- Small stable sets can go in the prompt
- Prompt caching helps that case
- RAG is cheaper and faster at scale
- RAG enables access control and citations
- Models miss details in very long inputs
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How does prompt caching change this decision?
- How would you handle per-user document permissions?
Common mistakes
- Saying RAG is obsolete because of long context
- Ignoring cost per request at scale
Learn it
- How would you build a dataset to fine-tune an LLM for a business task?
- Your fine-tuned model is no better than the base model. How do you debug it?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How do you decide which model size to use for a feature?
- How would you choose a base model to continue training or fine-tune for a specialised domain or language?
- What is prompt caching, and when does it help?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- What is model distillation, and when would you use it?
- When is fine-tuning worth the effort?
- A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
- When would you use synthetic data to train or test a model, and what are the risks?