How would you evaluate an LLM application?
Evals · Pro 1
Short answer
Build a test set of realistic inputs with what a good answer needs. Check what you can with code (format, required facts, forbidden content), use human or model review for subjective quality, and track scores for every change. In production, add user feedback and monitoring, and turn failures into new test cases.
Deeper answer
A layered approach, as Hamel Husain describes: level 1 is fast unit-test style assertions run on every change; level 2 is human and model-based review of samples, with the judge checked against human labels; level 3 is A/B testing with real users for product-level impact.
Start by looking at real outputs and writing down how they fail, then design checks for those failures. Keep the eval set versioned, add cases from production bugs, and report results by category, so a gain in one area does not hide a loss in another.
What a strong answer covers
- Builds a realistic test set
- Code checks where rules are exact
- Human or model review for subjective quality
- Tracks scores on every change
- Production feedback and monitoring
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Where do your first test cases come from?
- How do you evaluate subjective quality such as tone?
Common mistakes
- Relying on vibe checks of a few examples
- Using only generic benchmarks instead of task-specific tests
Learn it
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How do you evaluate a RAG system?
- How would you build a test set for an LLM feature when you have no labelled data?
- What are the pros and cons of using an LLM as a judge?
- A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
- What is the difference between pointwise and pairwise evaluation, and when would you use each?
- How do you stop a prompt or model change from breaking things that used to work?
- How would you evaluate an LLM feature whose answers are open-ended text?