Skip to main content

Your model provider has an outage during peak hours. What should already be in place?

Deployment and LLMOps · Pro 1

Short answer

A gateway or client layer with timeouts and retries, a fallback to a second model or provider that has been tested on our eval set, graceful degradation such as cached answers or a clear 'try again' message, monitoring that alerts on error rates, and a runbook saying who decides to switch and how we tell users.

Deeper answer

Outages are a matter of when, not if. The response should be designed in advance, because choosing a backup model in the middle of an incident leads to surprises in quality or format.

A gateway makes switching a configuration change rather than a code change. The fallback model must be evaluated on the same tests so you know its quality, and prompts may need small adjustments. For some features, degrading gracefully is better than switching, for example showing search results without an AI summary. After the incident, review what happened and update the runbook.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

LLM gateways

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.