Pattern · Evaluation
LLM as judge
A model grades outputs against a rubric, calibrated against human grades before its scores are trusted.
Say it in the interview
For open-ended quality I would use an LLM judge with a specific rubric that lists problems before its verdict, swap answer order to cancel position bias, and trust it only after it agrees with human graders on about 90% of a sample.
When to use it
- Open-ended outputs code cannot check: faithfulness, tone, helpfulness
- Too many outputs for people to grade
How it works
- Output + rubric → Judge model (list problems, then a verdict)
- Judge model (list problems, then a verdict) → Scores
- Human grades on a sample → Calibration (agreement rate)
- Scores → Calibration (agreement rate)
- Calibration (agreement rate) → about 90% or more → Trust it for this rubric
Trade-offs
- Scales grading to thousands of outputs
- Judges have biases: position, length, self-preference
Pitfalls interviewers probe
- Trusting a judge never compared with people
- Asking for a score with no reasoning first