You changed your agent's harness. How do you prove the change made it better?
Agents and tools · Pro 2
Short answer
Keep the model, settings and budget fixed. Run a fixed task set with automatic checks before and after, several times each, because agent runs vary. Compare pass rate, cost and time per task, and read failed traces. Change one thing at a time, and keep the task set in CI so later changes cannot quietly break it.
Deeper answer
A harness change is an experiment, so the only thing that should differ is the harness. Use the same model version, temperature, tools apart from the one you changed, and the same step and money limits.
The task set should be small enough to run often and real enough to matter, for example 20 tasks that each have a check a script can run, such as a test suite that must pass. Include easy, hard and long tasks. Do not tune the harness on the same tasks you report on, or keep a few tasks aside.
Run each task several times, because the same agent can pass and fail the same task. Report the pass rate with the number of runs, the cost and time per task, and the number of steps. A change that raises the pass rate but doubles cost may not be worth it. Read some traces, too, because a higher score can hide a new failure. Finally, put the task set in CI so every later harness change is checked against the baseline.
What a strong answer covers
- Holds the model, settings and budget fixed
- Uses a fixed task set with automatic checks
- Runs tasks several times because runs vary
- Compares pass rate, cost and time, and reads traces
- Keeps the task set in CI as a regression check
Answer it out loud in the mock interview about 2 min
Follow-ups to expect
- How many runs per task would you do, and why?
- What would you do if pass rate went up but cost doubled?
- How do you stop yourself from overfitting the harness to the task set?
Common mistakes
- Running each task once and trusting the result
- Changing the model or prompt at the same time as the harness
- Reporting pass rate without cost and time
Learn it
- What are common failure modes of agents, and how do you guard against them?
- What kinds of memory can an agent have, and how do you implement them?
- Your agent ran overnight and ran up a large API bill. What probably happened, and what do you change?
- How would you design a harness for an agent that has to work on one task for several hours?
- What is the difference between an agent and a workflow?
- What are computer-use or browser agents good for, and what are their risks?
- What is context engineering, and why does it matter for agents?
- An agent completes a task correctly most of the time but sometimes goes badly wrong. How do you debug it?
- What makes a good tool for an agent?
- Design an agent that handles refund requests for an online store.
- What are verification gates in an agent harness, and how would you add them to a coding agent?
- How does tool calling work?
- When should an AI agent ask a human for approval before acting?
- Explain state, nodes, edges and checkpoints in LangGraph.
- What are planning and reflection in agents, and when do they help?
- Two coding agents use the same model, but one solves far more of your tasks. Why could that be, and how would you find out?
- What is an agent harness, and what is harness engineering?
- What is the Model Context Protocol, and why does it matter?
- When is a multi-agent system worth it, and what are the risks?