Skip to main content

What does an LLM gateway do, and when do you need one?

Deployment and LLMOps · Pro 1

Short answer

A gateway sits between your apps and model providers and gives one API for many models. It handles routing, fallbacks when a provider fails, retries, rate limits and budgets per team, caching, and central logging of cost and usage, and it keeps provider keys in one place. You need one once several teams or models are in play.

Deeper answer

Tools such as LiteLLM or managed gateways provide this. Routing can also be smart: send easy requests to a cheap model and hard ones to a strong one, as RouteLLM research explored.

The gateway is also where you enforce policy: which models are approved, data rules, and spending caps. Keep it highly available, because it becomes a single point of failure.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

LLM gateways

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.