Skip to main content

How do you evaluate an agent, beyond checking its final answer?

Evals · Pro 1

Short answer

Build tasks with clear success criteria and check outcomes, ideally in a realistic environment. Also check the trajectory: did it use the right tools with valid arguments, how many steps and tokens it took, whether it asked for approval where required, and whether it avoided forbidden actions. Run each task several times and track the pass rate.

Deeper answer

Outcome checks can be code (the ticket was updated, the test suite passes) or model graders with rubrics. Trajectory checks catch agents that succeed by luck or by unsafe means.

Keep a regression set of tasks that once failed. Report cost and latency alongside success, because an agent that succeeds at ten times the cost may not be shippable. Anthropic's guide on evals for agents and DeepLearning.AI's course on evaluating agents cover this in depth.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Evaluating agents

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.