Skip to main content

Topic roadmap

Evaluation

8 lessons, about 25 hours in all, from the basics to advanced. Start with the first, or jump to what you need.

Lessons, easiest first

  1. Evaluation · Pro 1 · about 3 hCatching hallucinations: why models make things up and how to detect itExplain why models hallucinate, find the questions where yours does, and detect unsupported claims automatically with judge checks, claim-by-claim verification, entailment models and self-consistency.
  2. Evaluation · Pro 1 · about 2.5 hChoosing a model: benchmarks, leaderboards and your own testsShortlist models by hard requirements, read benchmark scores and leaderboards with care, spot contamination, and make the final choice with your own test set.
  3. Evaluation · Pro 1 · about 4 hEvaluating agentsBuild an eval suite for an agent: realistic tasks, graders for outcomes and for trajectories, repeated runs to measure consistency, and tracking that shows real progress.
  4. Evaluation · Pro 1 · about 3 hLLM judges and their limitsUse a model to grade answers, and check that its grades agree with a person's.
  5. Evaluation · Pro 1 · about 3 hOnline evals and feedbackMeasure quality on live traffic with automatic checks, sampled model-graded scores, user feedback and human review, and turn what you find into fixes and new test cases.
  6. Evaluation · Pro 1 · about 2 hRegression tests for promptsRun your eval set automatically on every change so a fix in one place cannot quietly break another.
  7. Evaluation · Pro 1 · about 2.5 hScoring open-ended answers: exact checks, similarity and side-by-side comparisonPick the right way to score an AI answer: run it, match it, compare it with a reference by words or meaning, or compare two answers side by side, and know each method's blind spots.
  8. Evaluation · Pro 1 · about 5 hTest sets and evalsBuild a test set for an LLM app and score every change against it.

On these role roadmaps

Interview questions on evaluation

Books on evaluation

  • AI Engineering

    Chip Huyen · 2025 · Intermediate 2 RAG builder

    The clearest single book on building products on top of foundation models: prompting, RAG, agents, evaluation, fine-tuning and serving costs. A strong companion to the GenAI Engineer roadmap.

    Affiliate link: we may earn a commission, at no extra cost to you. Disclosure

All recommended books

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.