1ConceptIntermediate 2
What is RAG, and why would you use it?
Short answer
Retrieval-augmented generation fetches relevant text from your own documents and gives it to the model with the question, so it answers from that text. Use it when the model needs private, recent or frequently changing information, and when you want answers with sources you can check.
The explanation
A basic pipeline: split documents into chunks, embed them, store them in an index; at question time, embed the query, retrieve the closest chunks, and prompt the model to answer only from them and cite which chunk it used.
RAG beats fine-tuning for facts that change, because you update documents instead of retraining. It also allows access control (retrieve only what the user may see) and citations. Its weak point is retrieval: if the right chunk is not found, the answer will be wrong or missing, so you evaluate retrieval separately from generation.
If an employee asks the company assistant 'How many days of work from home do I get?', a plain model does not know your policy. With RAG, the app first searches your HR documents, finds the paragraph on remote work, and gives it to the model with the question. The model answers from that paragraph and links it, and when the policy changes you update the document, not the model.
They may follow up with
- How would you handle documents a user is not allowed to see?
- When would you not use RAG?
- How do you evaluate a RAG system?
Avoid
- Describing RAG as training the model on your documents
- Ignoring retrieval quality and only tuning the prompt