Pattern · Serving and cost
LLM gateway with fallbacks
One service in front of every model call: keys, budgets, retries, fallbacks, logging and model aliases.
Say it in the interview
I would put an LLM gateway in front of every call: per-feature keys and budgets, retries with backoff, model aliases so we can switch without deploys, and a tested fallback chain for outages.
When to use it
- Several teams or features call models
- You need to switch providers or survive outages
How it works
- Feature A → key A → Gateway
- Feature B → key B → Gateway
- Gateway → alias chat-default → Primary model
- Primary model → timeout or 5xx → Same model, second region
- Same model, second region → fails → Approved fallback model
- Gateway → Logs (team, feature, tokens, cost, latency)
Trade-offs
- One place for keys, budgets and switching models
- Another service to run; a single point of failure if not redundant
Pitfalls interviewers probe
- Provider keys spread across services
- Fallback models never tested on the golden set