Skip to main content

Pattern · Retrieval and RAG

Long context or RAG

Decide whether to send whole documents in a long context window or retrieve a few passages.

Say it in the interview

It depends on how much text each question needs. For one contract I would send it whole with prompt caching; for a company library I would use RAG; often the best is in between: retrieve the right two documents and send them whole.

When to use it

  • A few long documents per question (one contract)
  • Millions of documents across a company
  • Cost or latency limits

How it works

  1. How much text per question? → a few documents → Long context (send them whole, cache the prefix)
  2. How much text per question? → a large library → RAG (retrieve the top passages)
  3. How much text per question? → in between → Find the right 2 documents, send them whole

Trade-offs

  • Long context is simple and sees everything, but costs more and is slower
  • RAG is cheap and scales, but misses what retrieval misses

Pitfalls interviewers probe

  • Assuming a bigger window means better answers
  • Ignoring prompt caching for repeated long documents

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.