Skip to main content

Pattern · Evaluation

Regression gate in CI

Run a fast eval tier on every change and a full tier before deploy, and block merges that make a category worse.

Say it in the interview

Prompt changes are code changes, so I would gate them in CI: a 90-second tier on every push and a full tier before deploy, compared with the production baseline by category.

When to use it

  • Prompts, models or retrieval change often
  • Several people edit the same feature

How it works

  1. Pull request → Fast tier, about 90 s (format, critical cases, must-refuse cases)
  2. Fast tier, about 90 s (format, critical cases, must-refuse cases) → fails → Merge blocked
  3. Fast tier, about 90 s (format, critical cases, must-refuse cases) → passes → Full tier before deploy (all cases vs the production baseline)
  4. Full tier before deploy (all cases vs the production baseline) → no critical category down → Deploy

Trade-offs

  • Catches breaks before users do
  • Eval runs cost money and time

Pitfalls interviewers probe

  • Only manual spot checks
  • A gate so flaky people learn to ignore it

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.