Pattern · Evaluation
Regression gate in CI
Run a fast eval tier on every change and a full tier before deploy, and block merges that make a category worse.
Say it in the interview
Prompt changes are code changes, so I would gate them in CI: a 90-second tier on every push and a full tier before deploy, compared with the production baseline by category.
When to use it
- Prompts, models or retrieval change often
- Several people edit the same feature
How it works
- Pull request → Fast tier, about 90 s (format, critical cases, must-refuse cases)
- Fast tier, about 90 s (format, critical cases, must-refuse cases) → fails → Merge blocked
- Fast tier, about 90 s (format, critical cases, must-refuse cases) → passes → Full tier before deploy (all cases vs the production baseline)
- Full tier before deploy (all cases vs the production baseline) → no critical category down → Deploy
Trade-offs
- Catches breaks before users do
- Eval runs cost money and time
Pitfalls interviewers probe
- Only manual spot checks
- A gate so flaky people learn to ignore it