Pattern · Retrieval and RAG
Long context or RAG
Decide whether to send whole documents in a long context window or retrieve a few passages.
Say it in the interview
It depends on how much text each question needs. For one contract I would send it whole with prompt caching; for a company library I would use RAG; often the best is in between: retrieve the right two documents and send them whole.
When to use it
- A few long documents per question (one contract)
- Millions of documents across a company
- Cost or latency limits
How it works
- How much text per question? → a few documents → Long context (send them whole, cache the prefix)
- How much text per question? → a large library → RAG (retrieve the top passages)
- How much text per question? → in between → Find the right 2 documents, send them whole
Trade-offs
- Long context is simple and sees everything, but costs more and is slower
- RAG is cheap and scales, but misses what retrieval misses
Pitfalls interviewers probe
- Assuming a bigger window means better answers
- Ignoring prompt caching for repeated long documents