Skip to main content

How do you stop a prompt or model change from breaking things that used to work?

Evals · Pro 1

Short answer

Keep a regression test set of past inputs, including every bug you have fixed, with checks for each. Run it automatically on every change, compare scores with the previous version, and block or review changes that make any category worse. Do the same before switching model versions.

Deeper answer

Prompts are code with unpredictable side effects: fixing one case can quietly break five others. Automating the eval set in CI turns this into a visible number. Fast deterministic checks run on every push; slower judge-based or human checks run before releases.

Pin model versions in production so behaviour does not shift under you, and treat a model upgrade as a change that must pass the same tests. Watch per-category results, not only the average.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Regression tests for prompts

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.