Skip to main content

System design answer

An evaluation pipeline for an LLM product

A golden set and scorers offline, a regression gate in CI, and online evals with tracing and feedback that feed back into the set.
The question“Design how a team would evaluate and monitor an LLM feature from first prototype to production.”

1Clarify0-5 min

Ask before you draw. These answers change the design.

  • What does the feature do, and what does a good output look like?
  • How often do prompts, models and retrieval change?
  • Who can label examples, and how much time do they have?
  • What would a bad answer cost?

2Requirements and numbers5-8 min

It must

  • Score every change before it ships
  • Monitor quality in production
  • Turn failures into test cases

And it must be

  • Fast tier under 2 minutes
  • Judge agreement with people of about 90%
  • Personal data masked in traces

Numbers to say out loud

Golden set
150 cases in 6 categories
Production
20,000 requests a day; judge on 10% plus all thumbs-down
Eval cost
About ₹300 per full run

3High-level design8-12 min

  1. Logs, hard cases, past failures → Golden set
  2. Golden set → Scorers (code checks, calibrated judge, human sample)
  3. Pull request → Fast tier in CI
  4. Fast tier in CI → Full tier before deploy
  5. Scorers (code checks, calibrated judge, human sample) → Fast tier in CI
  6. Full tier before deploy → Production
  7. Production → Traces
  8. Traces → Online scores and feedback
  9. Online scores and feedback → Dashboard and alerts
  10. Online scores and feedback → failures → Golden set

4Deep dives12-26 min

Scorers

Code checks for anything exact (format, citations, length). A judge for faithfulness and helpfulness, with a rubric that lists problems first, calibrated on 80 human-graded cases. A small human sample every week.

Gates

Results by category against the production baseline; a release ships only if no critical category drops beyond its tolerance. Flaky judge scores are averaged over 3 runs for close calls.

5Failures26-30 min

What can go wrongWhat stops it
Judge drift after a model upgradeRe-calibrate the judge on the human set
Golden set goes staleAdd production failures weekly
Teams ignore a flaky gateKeep the fast tier deterministic; judge only in the full tier

6Evals and monitoring30-33 min

  • Agreement between judge and humans
  • Time from bug report to a test case
  • Share of incidents caught before production

7Trade-offs33-35 min

  • More cases mean more confidence but slower, costlier runs: keep two tiers
  • A judge scales but must be calibrated; humans are accurate but slow

Follow-ups to expect

  • How would you evaluate an agent instead of a chatbot?
  • How do you pick the first 50 cases?

Practise out loud in the mock interview

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.