Skip to main content

Your model scored well offline but performed badly after launch. What could explain it?

Deployment and LLMOps · Pro 1

Short answer

Common causes: data leakage made offline scores too optimistic; training data does not match live data (skew); features are computed differently in production; the population changed (drift); or the offline metric does not reflect the business goal. I would compare feature distributions offline versus online, check for leakage, and replay live inputs through the training pipeline.

Deeper answer

A gap between offline and online results usually means the evaluation did not reflect reality. Leakage, such as a feature that is only known after the event, inflates offline scores. Training-serving skew happens when production code computes a feature differently, uses stale values or fills missing data another way.

To diagnose, log production inputs and predictions, compare their distributions to training data, and recompute some live predictions offline to see if they match. Also check the metric: a model that improves AUC may not improve the decision the business makes. Fix the cause, then add monitoring so the gap is caught early.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Monitoring and drift

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.