How do you evaluate a RAG system?
Evals · Intermediate 2
Short answer
Evaluate retrieval and generation separately. For retrieval, check whether the needed chunk appears in the top results (recall at k) and how much retrieved text is relevant. For generation, check whether the answer is correct, uses only the retrieved context (faithfulness) and cites the right source. Then test end to end on real questions.
Deeper answer
Build questions with the source passage that answers them. Retrieval metrics tell you whether the right text reached the model; generation metrics tell you whether the model used it well. Frameworks such as Ragas name these metrics: context precision, context recall, faithfulness and answer relevancy.
Include questions whose answers are not in the documents, to check the system says it does not know. Track metrics by document type and question type, because averages hide problems.
What a strong answer covers
- Evaluates retrieval and generation separately
- Retrieval: recall at k
- Generation: correctness and faithfulness to context
- Checks citations and tests end to end
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What is faithfulness, and how would you measure it?
- Why include questions that have no answer in the documents?
Common mistakes
- Measuring only final answer quality, so you cannot tell where failures come from
- Leaving out unanswerable questions
Learn it
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How would you evaluate an LLM application?
- How would you build a test set for an LLM feature when you have no labelled data?
- What are the pros and cons of using an LLM as a judge?
- A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
- What is the difference between pointwise and pairwise evaluation, and when would you use each?
- How do you stop a prompt or model change from breaking things that used to work?
- How would you evaluate an LLM feature whose answers are open-ended text?