How would you evaluate an LLM feature whose answers are open-ended text?
Evals · Pro 1
Short answer
Layer the checks. Test whatever can be checked exactly: facts like numbers, IDs and formats. For the wording, use a judge model with a rubric or meaning-based similarity against good reference answers, checked against human ratings. To choose between two versions, compare their answers side by side on the same questions, in random order.
Deeper answer
Word-overlap scores such as BLEU and ROUGE are cheap but mislead on modern models: a correct answer in different words scores low, and a wrong one that copies words scores high. Semantic similarity with embeddings handles paraphrase but can rate opposite facts as close, so key facts still need exact checks.
Side-by-side comparison is easier for people than absolute scores and can be turned into a ranking, but preference is not correctness, so never vote on facts the voter cannot verify. Report scores by segment, such as language and topic.
What a strong answer covers
- Checks key facts exactly or functionally
- Uses a judge or meaning similarity for wording
- Compares versions side by side
- Names the limits of word-overlap or preference
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Why are BLEU and ROUGE weak for chat answers?
- How would you set a threshold for semantic similarity?
Common mistakes
- Using one similarity score as the quality metric
- Letting preference votes decide factual questions
Learn it
Scoring open-ended answers: exact checks, similarity and side-by-side comparison
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How would you evaluate an LLM application?
- How do you evaluate a RAG system?
- How would you build a test set for an LLM feature when you have no labelled data?
- What are the pros and cons of using an LLM as a judge?
- A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
- What is the difference between pointwise and pairwise evaluation, and when would you use each?
- How do you stop a prompt or model change from breaking things that used to work?