Your RAG app gives wrong answers. How do you find the cause?
RAG · Intermediate 2
Short answer
Separate retrieval from generation. For a set of failing questions, check whether the right chunk was retrieved. If not, fix retrieval: chunking, embeddings, query rewriting or hybrid search. If it was retrieved but the answer is still wrong, fix the prompt, context order or model. Measure each change on a test set.
Deeper answer
A practical sequence: collect failing examples; for each, log the retrieved chunks and see whether the answer was in them. Count how many failures are retrieval misses and how many are generation errors. Retrieval misses often come from chunks cut in the wrong place, missing metadata, vocabulary mismatch, or the document not being indexed at all.
Generation errors come from too much irrelevant context, instructions that do not force grounding, or the model ignoring the context. Build a small test set with the expected source for each question, track recall at k and answer correctness, and change one thing at a time.
What a strong answer covers
- Separates retrieval from generation
- Checks whether the right chunk was retrieved
- Fixes retrieval: chunking, embeddings, hybrid search, query rewriting
- Fixes generation: prompt, context order or model
- Measures each change on a test set
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What metrics would you track for retrieval?
- How would you build the test set if you have no labelled data?
Common mistakes
- Jumping straight to a bigger model
- Changing several things at once so you cannot tell what helped
Learn it
- How do you choose a chunk size for RAG?
- Design a RAG system that lets a law firm search and ask questions about 50,000 contracts.
- How would you detect hallucinations in a production RAG application?
- What are embeddings, and how are they used in search?
- How do you make a RAG system cite sources and stay grounded?
- What are hybrid search and reranking, and when do they help?
- How do you keep a RAG index up to date as documents change?
- How do metadata filters improve retrieval?
- How would you build RAG over documents full of charts and images?
- Your RAG system must handle scanned PDFs, tables and slides. How do you ingest them?
- How do you make sure users only get answers from documents they are allowed to see?
- What are query rewriting and HyDE, and when do they help?
- Your HR assistant keeps quoting last year's leave policy. How do you fix it?
- Tell me about a RAG system you built and how you measured whether it worked.
- Can you do retrieval for long documents without a vector database?
- What is RAG, and why would you use it?
- When would you use graph RAG instead of standard vector RAG?