Tell me about a RAG system you built and how you measured whether it worked.
RAG · Intermediate 2
Short answer
Describe the documents and users, the pipeline choices (parsing, chunking, search, prompt), and above all the test set: how many questions, where they came from, and the metrics, such as how often the right passage was retrieved and how often answers were correct and grounded. Then one improvement and its measured effect.
Deeper answer
What separates strong candidates is measurement. Saying 'it worked well' is weak; saying 'on 60 real questions, the right passage was in the top five 78% of the time, and hybrid search raised that to 90%' is strong, as long as you can explain how you counted.
Cover both halves: retrieval quality and answer quality. Mention a failure you found through evaluation, such as tables parsed badly or questions needing two documents, and how you fixed it. If it was a personal project, say so and describe the real documents you used.
What a strong answer covers
- Describes the documents, users and pipeline
- Explains the test set and where it came from
- Measures retrieval quality
- Measures answer quality or grounding
- Gives an improvement with its measured effect
Answer it out loud in the mock interview about 2 min
Follow-ups to expect
- How did you build the test set?
- What was the hardest kind of question?
Common mistakes
- No numbers, or numbers with no explanation of how they were measured
- Only evaluating final answers, not retrieval
Learn it
- How do you choose a chunk size for RAG?
- Your RAG app gives wrong answers. How do you find the cause?
- Design a RAG system that lets a law firm search and ask questions about 50,000 contracts.
- How would you detect hallucinations in a production RAG application?
- What are embeddings, and how are they used in search?
- How do you make a RAG system cite sources and stay grounded?
- What are hybrid search and reranking, and when do they help?
- How do you keep a RAG index up to date as documents change?
- How do metadata filters improve retrieval?
- How would you build RAG over documents full of charts and images?
- Your RAG system must handle scanned PDFs, tables and slides. How do you ingest them?
- How do you make sure users only get answers from documents they are allowed to see?
- What are query rewriting and HyDE, and when do they help?
- Your HR assistant keeps quoting last year's leave policy. How do you fix it?
- Can you do retrieval for long documents without a vector database?
- What is RAG, and why would you use it?
- When would you use graph RAG instead of standard vector RAG?