What is the difference between pointwise and pairwise evaluation, and when would you use each?
Evals · Pro 1
Short answer
Pointwise evaluation scores each answer on its own, for example 1 to 5 against a rubric. Pairwise evaluation shows two answers side by side and asks which is better. Pairwise is easier and more consistent for subjective quality and for choosing between two systems; pointwise is better for tracking one system over time and for absolute pass or fail bars.
Deeper answer
Many pairwise votes can be turned into a ranking with methods such as Elo or Bradley-Terry, which is how public chatbot arenas rank models. Its limits: the number of pairs grows quickly with more models, voters need to know enough to judge, and position and length bias must be controlled by randomising order.
Neither replaces exact checks for facts. In practice I use exact checks on every answer, pointwise judge scores for monitoring, and pairwise comparisons when deciding between two prompts or models.
What a strong answer covers
- Pointwise scores each answer alone
- Pairwise compares two answers
- Pairwise suits subjective quality and choosing systems
- Names a limit such as bias or scale
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you remove position bias in pairwise judging?
- How many comparisons would you need to decide between two prompts?
Common mistakes
- Treating preference as correctness
- Comparing answers always in the same order
Learn it
Scoring open-ended answers: exact checks, similarity and side-by-side comparison
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How would you evaluate an LLM application?
- How do you evaluate a RAG system?
- How would you build a test set for an LLM feature when you have no labelled data?
- What are the pros and cons of using an LLM as a judge?
- A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
- How do you stop a prompt or model change from breaking things that used to work?
- How would you evaluate an LLM feature whose answers are open-ended text?