Skip to main content

How would you evaluate an LLM feature whose answers are open-ended text?

Evals · Pro 1

Short answer

Layer the checks. Test whatever can be checked exactly: facts like numbers, IDs and formats. For the wording, use a judge model with a rubric or meaning-based similarity against good reference answers, checked against human ratings. To choose between two versions, compare their answers side by side on the same questions, in random order.

Deeper answer

Word-overlap scores such as BLEU and ROUGE are cheap but mislead on modern models: a correct answer in different words scores low, and a wrong one that copies words scores high. Semantic similarity with embeddings handles paraphrase but can rate opposite facts as close, so key facts still need exact checks.

Side-by-side comparison is easier for people than absolute scores and can be turned into a ranking, but preference is not correctness, so never vote on facts the voter cannot verify. Report scores by segment, such as language and topic.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Scoring open-ended answers: exact checks, similarity and side-by-side comparison

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.