Skip to main content

How would you build a test set for an LLM feature when you have no labelled data?

Evals · Pro 1

Short answer

Start from real or realistic inputs: logs, support tickets, or examples written with domain experts. Cover common cases, edge cases and known failure types. For each, write what a good answer must include or avoid. Begin small, around 30 to 50 cases, review them with people who know the domain, and grow the set from production failures.

Deeper answer

Synthetic examples generated by a model can fill gaps, but review them, because they tend to be easier and more uniform than real inputs. Balance the set across categories that matter to the business, and include adversarial inputs such as prompt injection attempts and off-topic questions.

Write criteria, not just one reference answer, since many good answers are possible. Version the set, and keep a held-out part you do not tune against, so you can confirm improvements are real.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Test sets and evals

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.