How do you stop a prompt or model change from breaking things that used to work?
Evals · Pro 1
Short answer
Keep a regression test set of past inputs, including every bug you have fixed, with checks for each. Run it automatically on every change, compare scores with the previous version, and block or review changes that make any category worse. Do the same before switching model versions.
Deeper answer
Prompts are code with unpredictable side effects: fixing one case can quietly break five others. Automating the eval set in CI turns this into a visible number. Fast deterministic checks run on every push; slower judge-based or human checks run before releases.
Pin model versions in production so behaviour does not shift under you, and treat a model upgrade as a change that must pass the same tests. Watch per-category results, not only the average.
What a strong answer covers
- Keeps a regression set of past inputs and fixed bugs
- Runs it automatically on every change
- Compares scores with the previous version
- Blocks or reviews changes that get worse
- Does the same before switching models
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What would you do when a new model is better on average but worse in one category?
- Which checks run on every push, and which only before release?
Common mistakes
- Testing only the case you were fixing
- Letting the provider's model updates reach production untested
Learn it
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How would you evaluate an LLM application?
- How do you evaluate a RAG system?
- How would you build a test set for an LLM feature when you have no labelled data?
- What are the pros and cons of using an LLM as a judge?
- A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
- What is the difference between pointwise and pairwise evaluation, and when would you use each?
- How would you evaluate an LLM feature whose answers are open-ended text?