Skip to main content

Pattern · Serving and cost

LLM gateway with fallbacks

One service in front of every model call: keys, budgets, retries, fallbacks, logging and model aliases.

Say it in the interview

I would put an LLM gateway in front of every call: per-feature keys and budgets, retries with backoff, model aliases so we can switch without deploys, and a tested fallback chain for outages.

When to use it

  • Several teams or features call models
  • You need to switch providers or survive outages

How it works

  1. Feature A → key A → Gateway
  2. Feature B → key B → Gateway
  3. Gateway → alias chat-default → Primary model
  4. Primary model → timeout or 5xx → Same model, second region
  5. Same model, second region → fails → Approved fallback model
  6. Gateway → Logs (team, feature, tokens, cost, latency)

Trade-offs

  • One place for keys, budgets and switching models
  • Another service to run; a single point of failure if not redundant

Pitfalls interviewers probe

  • Provider keys spread across services
  • Fallback models never tested on the golden set

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.