Skip to main content

Pattern · Evaluation

LLM as judge

A model grades outputs against a rubric, calibrated against human grades before its scores are trusted.

Say it in the interview

For open-ended quality I would use an LLM judge with a specific rubric that lists problems before its verdict, swap answer order to cancel position bias, and trust it only after it agrees with human graders on about 90% of a sample.

When to use it

  • Open-ended outputs code cannot check: faithfulness, tone, helpfulness
  • Too many outputs for people to grade

How it works

  1. Output + rubric → Judge model (list problems, then a verdict)
  2. Judge model (list problems, then a verdict) → Scores
  3. Human grades on a sample → Calibration (agreement rate)
  4. Scores → Calibration (agreement rate)
  5. Calibration (agreement rate) → about 90% or more → Trust it for this rubric

Trade-offs

  • Scales grading to thousands of outputs
  • Judges have biases: position, length, self-preference

Pitfalls interviewers probe

  • Trusting a judge never compared with people
  • Asking for a score with no reasoning first

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.