Pattern · Evaluation
Agent trajectory evals
Judge an agent on the outcome, the path it took, and how consistently it succeeds across runs.
Say it in the interview
For an agent I would grade three things: the outcome in a seeded sandbox by code, the trajectory against rules like "approval before payment", and success on every one of several runs, with safety violations required to be zero.
When to use it
- Agents with tools and side effects
- Agents whose runs differ from one try to the next
How it works
- Task + seeded sandbox → Run 1
- Task + seeded sandbox → Run 2
- Task + seeded sandbox → Run 3
- Run 1 → Graders
- Run 2 → Graders
- Run 3 → Graders
- Graders → Outcome by code (right state in the sandbox?)
- Graders → Trajectory by rules (approval before payment? step cap?)
- Graders → Success on every run (consistency)
Trade-offs
- Catches unsafe paths that end in the right answer
- Sandboxes take effort to build
Pitfalls interviewers probe
- Grading only the final message
- One run per task, which hides flakiness