What are the pros and cons of using an LLM as a judge?
Evals · Pro 1
Short answer
A judge model can grade qualities that code cannot, such as helpfulness or tone, quickly and cheaply at scale. But judges have biases, such as preferring longer answers or the first option, and can be wrong confidently. Use them only after checking their grades agree with careful human grades, and prefer code checks where rules are exact.
Deeper answer
Make judges reliable by asking narrow questions (pass or fail on one criterion, with a reason) rather than vague 1 to 10 scores, giving clear rubrics and examples, and swapping answer order in comparisons to cancel position bias.
Measure agreement with human labels on a sample and recheck when you change the judge's model or prompt. Treat the judge as a model you also evaluate, not as ground truth.
What a strong answer covers
- Grades subjective qualities at scale
- Judges have biases such as length or position
- Checks judge agreement with human grades
- Prefers code checks for exact rules
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you measure whether a judge is trustworthy?
- Why are pass or fail judgements often better than 1 to 10 scores?
Common mistakes
- Trusting a judge without comparing it with humans
- Using the same model to generate and judge without any checks
Learn it
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How would you evaluate an LLM application?
- How do you evaluate a RAG system?
- How would you build a test set for an LLM feature when you have no labelled data?
- A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
- What is the difference between pointwise and pairwise evaluation, and when would you use each?
- How do you stop a prompt or model change from breaking things that used to work?
- How would you evaluate an LLM feature whose answers are open-ended text?