Skip to main content

You changed your agent's harness. How do you prove the change made it better?

Agents and tools · Pro 2

Short answer

Keep the model, settings and budget fixed. Run a fixed task set with automatic checks before and after, several times each, because agent runs vary. Compare pass rate, cost and time per task, and read failed traces. Change one thing at a time, and keep the task set in CI so later changes cannot quietly break it.

Deeper answer

A harness change is an experiment, so the only thing that should differ is the harness. Use the same model version, temperature, tools apart from the one you changed, and the same step and money limits.

The task set should be small enough to run often and real enough to matter, for example 20 tasks that each have a check a script can run, such as a test suite that must pass. Include easy, hard and long tasks. Do not tune the harness on the same tasks you report on, or keep a few tasks aside.

Run each task several times, because the same agent can pass and fail the same task. Report the pass rate with the number of runs, the cost and time per task, and the number of steps. A change that raises the pass rate but doubles cost may not be worth it. Read some traces, too, because a higher score can hide a new failure. Finally, put the task set in CI so every later harness change is checked against the baseline.

What a strong answer covers

Answer it out loud in the mock interview about 2 min

Follow-ups to expect

Common mistakes

Learn it

Advanced harness engineering

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.