Skip to main content

Pattern · Evaluation

Agent trajectory evals

Judge an agent on the outcome, the path it took, and how consistently it succeeds across runs.

Say it in the interview

For an agent I would grade three things: the outcome in a seeded sandbox by code, the trajectory against rules like "approval before payment", and success on every one of several runs, with safety violations required to be zero.

When to use it

  • Agents with tools and side effects
  • Agents whose runs differ from one try to the next

How it works

  1. Task + seeded sandbox → Run 1
  2. Task + seeded sandbox → Run 2
  3. Task + seeded sandbox → Run 3
  4. Run 1 → Graders
  5. Run 2 → Graders
  6. Run 3 → Graders
  7. Graders → Outcome by code (right state in the sandbox?)
  8. Graders → Trajectory by rules (approval before payment? step cap?)
  9. Graders → Success on every run (consistency)

Trade-offs

  • Catches unsafe paths that end in the right answer
  • Sandboxes take effort to build

Pitfalls interviewers probe

  • Grading only the final message
  • One run per task, which hides flakiness

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.