How do you detect and handle model drift in production?
Deployment and LLMOps · Pro 1
Short answer
Monitor inputs and outputs, not just uptime. Data drift is when input distributions change; concept drift is when the relationship between inputs and the target changes. Track feature statistics, prediction distributions and, when labels arrive, real performance. Alert on meaningful shifts, investigate, and retrain or roll back with a tested pipeline.
Deeper answer
Labels often arrive late: you learn whether a loan defaulted months later. So use proxies in the meantime: compare live feature distributions with training data using measures such as population stability index or a statistical test, watch the share of each predicted class, and track business metrics.
Not every shift needs retraining; a festival season can shift spending without breaking the model. Set thresholds with care to avoid alert fatigue, keep a reference dataset per model version, and make retraining a routine, versioned pipeline with evaluation gates rather than a manual scramble.
What a strong answer covers
- Distinguishes data drift and concept drift
- Tracks input features and predictions
- Measures real performance when labels arrive
- Alerts with sensible thresholds
- Retrains or rolls back through a tested pipeline
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What would you monitor when labels arrive months later?
- When is retraining the wrong fix?
Common mistakes
- Monitoring only latency and errors
- Retraining automatically without an evaluation gate
Learn it
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- A bank wants your AI system but says no data may leave its network. What are the options?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?