Pattern · Evaluation
Golden test set
A fixed set of real inputs with what a good output must contain, run on every change and reported by category.
Say it in the interview
Before launch I would build a golden set of 50 to 200 real inputs, grouped by category, each with what a good answer must contain. Every change runs on it, and we ship only if no category gets worse.
When to use it
- Any LLM feature before launch
- Comparing prompts, models or retrieval settings
How it works
- Real inputs from logs → Golden set, 50-200 cases
- Hard cases and past failures → Golden set, 50-200 cases
- Golden set, 50-200 cases → Run app version N
- Run app version N → Score by category
- Score by category → no category worse → Ship
- Production bug → becomes a case → Golden set, 50-200 cases
Trade-offs
- Turns "seems better" into a number
- Needs upkeep as the product changes
Pitfalls interviewers probe
- Reporting only an average, which hides a broken category
- Cases written from imagination, not logs