How would you build a test set for an LLM feature when you have no labelled data?
Evals · Pro 1
Short answer
Start from real or realistic inputs: logs, support tickets, or examples written with domain experts. Cover common cases, edge cases and known failure types. For each, write what a good answer must include or avoid. Begin small, around 30 to 50 cases, review them with people who know the domain, and grow the set from production failures.
Deeper answer
Synthetic examples generated by a model can fill gaps, but review them, because they tend to be easier and more uniform than real inputs. Balance the set across categories that matter to the business, and include adversarial inputs such as prompt injection attempts and off-topic questions.
Write criteria, not just one reference answer, since many good answers are possible. Version the set, and keep a held-out part you do not tune against, so you can confirm improvements are real.
What a strong answer covers
- Starts from real or realistic inputs
- Writes cases with domain experts
- Covers common cases, edge cases and failures
- Says what a good answer must include or avoid
- Starts small and grows from production failures
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How do you stop your test set from becoming too easy?
- How large should it be at first?
Common mistakes
- Using only synthetic, model-written examples
- Writing one exact reference answer and checking for an exact match
Learn it
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How would you evaluate an LLM application?
- How do you evaluate a RAG system?
- What are the pros and cons of using an LLM as a judge?
- A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
- What is the difference between pointwise and pairwise evaluation, and when would you use each?
- How do you stop a prompt or model change from breaking things that used to work?
- How would you evaluate an LLM feature whose answers are open-ended text?