Skip to main content

How do you evaluate a RAG system?

Evals · Intermediate 2

Short answer

Evaluate retrieval and generation separately. For retrieval, check whether the needed chunk appears in the top results (recall at k) and how much retrieved text is relevant. For generation, check whether the answer is correct, uses only the retrieved context (faithfulness) and cites the right source. Then test end to end on real questions.

Deeper answer

Build questions with the source passage that answers them. Retrieval metrics tell you whether the right text reached the model; generation metrics tell you whether the model used it well. Frameworks such as Ragas name these metrics: context precision, context recall, faithfulness and answer relevancy.

Include questions whose answers are not in the documents, to check the system says it does not know. Track metrics by document type and question type, because averages hide problems.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Check whether retrieval works

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.