Skip to main content

Pattern · Evaluation

Golden test set

A fixed set of real inputs with what a good output must contain, run on every change and reported by category.

Say it in the interview

Before launch I would build a golden set of 50 to 200 real inputs, grouped by category, each with what a good answer must contain. Every change runs on it, and we ship only if no category gets worse.

When to use it

  • Any LLM feature before launch
  • Comparing prompts, models or retrieval settings

How it works

  1. Real inputs from logs → Golden set, 50-200 cases
  2. Hard cases and past failures → Golden set, 50-200 cases
  3. Golden set, 50-200 cases → Run app version N
  4. Run app version N → Score by category
  5. Score by category → no category worse → Ship
  6. Production bug → becomes a case → Golden set, 50-200 cases

Trade-offs

  • Turns "seems better" into a number
  • Needs upkeep as the product changes

Pitfalls interviewers probe

  • Reporting only an average, which hides a broken category
  • Cases written from imagination, not logs

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.