Skip to main content

Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?

Prompting, RAG or fine-tuning · Pro 3

Short answer

For a small, stable document set, putting it all in the prompt can be simpler and work well, especially with prompt caching. For large or changing collections, RAG is cheaper, faster and scales: you send only relevant text, can enforce access control per user, and can cite sources. Models can also miss details buried in very long inputs.

Deeper answer

Cost and latency grow with input tokens on every request, so a full-corpus prompt is expensive at volume, though caching a fixed prefix helps. Very long inputs can reduce attention to material in the middle. And a corpus of millions of documents will not fit anyway.

A practical answer is "it depends, and here is how I would decide": if the knowledge base fits comfortably and rarely changes, try long context with caching and measure; otherwise use retrieval, perhaps retrieving larger sections to give the model more context.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Prompting, RAG or fine-tuning

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.