Your model scored well offline but performed badly after launch. What could explain it?
Deployment and LLMOps · Pro 1
Short answer
Common causes: data leakage made offline scores too optimistic; training data does not match live data (skew); features are computed differently in production; the population changed (drift); or the offline metric does not reflect the business goal. I would compare feature distributions offline versus online, check for leakage, and replay live inputs through the training pipeline.
Deeper answer
A gap between offline and online results usually means the evaluation did not reflect reality. Leakage, such as a feature that is only known after the event, inflates offline scores. Training-serving skew happens when production code computes a feature differently, uses stale values or fills missing data another way.
To diagnose, log production inputs and predictions, compare their distributions to training data, and recompute some live predictions offline to see if they match. Also check the metric: a model that improves AUC may not improve the decision the business makes. Fix the cause, then add monitoring so the gap is caught early.
What a strong answer covers
- Names data leakage
- Names training-serving skew or feature differences
- Names drift or population change
- Questions whether the offline metric matches the goal
- Diagnoses by comparing distributions or replaying inputs
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you detect training-serving skew?
- How would you set up an evaluation that predicts online results better?
Common mistakes
- Retraining immediately without diagnosing
- Assuming it is drift without checking for leakage
Learn it
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- A bank wants your AI system but says no data may leave its network. What are the options?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?