Your model provider has an outage during peak hours. What should already be in place?
Deployment and LLMOps · Pro 1
Short answer
A gateway or client layer with timeouts and retries, a fallback to a second model or provider that has been tested on our eval set, graceful degradation such as cached answers or a clear 'try again' message, monitoring that alerts on error rates, and a runbook saying who decides to switch and how we tell users.
Deeper answer
Outages are a matter of when, not if. The response should be designed in advance, because choosing a backup model in the middle of an incident leads to surprises in quality or format.
A gateway makes switching a configuration change rather than a code change. The fallback model must be evaluated on the same tests so you know its quality, and prompts may need small adjustments. For some features, degrading gracefully is better than switching, for example showing search results without an AI summary. After the incident, review what happened and update the runbook.
What a strong answer covers
- Uses timeouts and retries
- Has a tested fallback model or provider
- Plans graceful degradation
- Monitors and alerts on errors
- Has a runbook and owner for the decision
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you test the fallback before you need it?
- Which features would you degrade rather than switch?
Common mistakes
- Having a fallback model that was never tested
- No alert until users complain
Learn it
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- A bank wants your AI system but says no data may leave its network. What are the options?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?