System design answer
An evaluation pipeline for an LLM product
A golden set and scorers offline, a regression gate in CI, and online evals with tracing and feedback that feed back into the set.
The question“Design how a team would evaluate and monitor an LLM feature from first prototype to production.”
1Clarify0-5 min
Ask before you draw. These answers change the design.
- What does the feature do, and what does a good output look like?
- How often do prompts, models and retrieval change?
- Who can label examples, and how much time do they have?
- What would a bad answer cost?
2Requirements and numbers5-8 min
It must
- Score every change before it ships
- Monitor quality in production
- Turn failures into test cases
And it must be
- Fast tier under 2 minutes
- Judge agreement with people of about 90%
- Personal data masked in traces
Numbers to say out loud
- Golden set
- 150 cases in 6 categories
- Production
- 20,000 requests a day; judge on 10% plus all thumbs-down
- Eval cost
- About ₹300 per full run
3High-level design8-12 min
- Logs, hard cases, past failures → Golden set
- Golden set → Scorers (code checks, calibrated judge, human sample)
- Pull request → Fast tier in CI
- Fast tier in CI → Full tier before deploy
- Scorers (code checks, calibrated judge, human sample) → Fast tier in CI
- Full tier before deploy → Production
- Production → Traces
- Traces → Online scores and feedback
- Online scores and feedback → Dashboard and alerts
- Online scores and feedback → failures → Golden set
4Deep dives12-26 min
Scorers
Code checks for anything exact (format, citations, length). A judge for faithfulness and helpfulness, with a rubric that lists problems first, calibrated on 80 human-graded cases. A small human sample every week.
Gates
Results by category against the production baseline; a release ships only if no critical category drops beyond its tolerance. Flaky judge scores are averaged over 3 runs for close calls.
5Failures26-30 min
| What can go wrong | What stops it |
|---|---|
| Judge drift after a model upgrade | Re-calibrate the judge on the human set |
| Golden set goes stale | Add production failures weekly |
| Teams ignore a flaky gate | Keep the fast tier deterministic; judge only in the full tier |
6Evals and monitoring30-33 min
- Agreement between judge and humans
- Time from bug report to a test case
- Share of incidents caught before production
7Trade-offs33-35 min
- More cases mean more confidence but slower, costlier runs: keep two tiers
- A judge scales but must be calibrated; humans are accurate but slow
Follow-ups to expect
- How would you evaluate an agent instead of a chatbot?
- How do you pick the first 50 cases?