Skip to main content

Pattern · Evaluation

Online evals and tracing

Trace every request in production, score them with rules, a judge on a sample and user feedback, and turn failures into test cases.

Say it in the interview

Offline evals are not enough, so I would trace every request with prompt and model versions, run cheap rules on all of them and a judge on a sample plus every thumbs-down, review 50 traces a week, and feed failures back into the golden set.

When to use it

  • After launch, when real inputs differ from the test set
  • When users report problems you cannot reproduce

How it works

  1. Live request → Trace (prompt, retrieval, tools, tokens, cost, latency)
  2. Trace (prompt, retrieval, tools, tokens, cost, latency) → Rules on every trace
  3. Trace (prompt, retrieval, tools, tokens, cost, latency) → Judge on a sample + all thumbs-down
  4. Trace (prompt, retrieval, tools, tokens, cost, latency) → Weekly human review
  5. Rules on every trace → Dashboard and alerts
  6. Judge on a sample + all thumbs-down → Dashboard and alerts
  7. Weekly human review → New golden cases

Trade-offs

  • Finds problems the test set missed
  • Traces may hold personal data: mask them

Pitfalls interviewers probe

  • Logging without ever reading
  • Tracing that leaks customer data

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.