Skip to main content

How would you evaluate an LLM application?

Evals · Pro 1

Short answer

Build a test set of realistic inputs with what a good answer needs. Check what you can with code (format, required facts, forbidden content), use human or model review for subjective quality, and track scores for every change. In production, add user feedback and monitoring, and turn failures into new test cases.

Deeper answer

A layered approach, as Hamel Husain describes: level 1 is fast unit-test style assertions run on every change; level 2 is human and model-based review of samples, with the judge checked against human labels; level 3 is A/B testing with real users for product-level impact.

Start by looking at real outputs and writing down how they fail, then design checks for those failures. Keep the eval set versioned, add cases from production bugs, and report results by category, so a gain in one area does not hide a loss in another.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Test sets and evals

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.