Pattern · Evaluation
Online evals and tracing
Trace every request in production, score them with rules, a judge on a sample and user feedback, and turn failures into test cases.
Say it in the interview
Offline evals are not enough, so I would trace every request with prompt and model versions, run cheap rules on all of them and a judge on a sample plus every thumbs-down, review 50 traces a week, and feed failures back into the golden set.
When to use it
- After launch, when real inputs differ from the test set
- When users report problems you cannot reproduce
How it works
- Live request → Trace (prompt, retrieval, tools, tokens, cost, latency)
- Trace (prompt, retrieval, tools, tokens, cost, latency) → Rules on every trace
- Trace (prompt, retrieval, tools, tokens, cost, latency) → Judge on a sample + all thumbs-down
- Trace (prompt, retrieval, tools, tokens, cost, latency) → Weekly human review
- Rules on every trace → Dashboard and alerts
- Judge on a sample + all thumbs-down → Dashboard and alerts
- Weekly human review → New golden cases
Trade-offs
- Finds problems the test set missed
- Traces may hold personal data: mask them
Pitfalls interviewers probe
- Logging without ever reading
- Tracing that leaks customer data