System design answer
An internal LLM platform for many teams
A gateway with per-team keys and budgets, model aliases and fallbacks, masked logging, shared evals and approved models.
The question“Design a shared platform so 20 teams in a company can use LLMs safely, with control of cost and data.”
1Clarify0-5 min
Ask before you draw. These answers change the design.
- How many teams and requests, and which models are approved?
- What data may go to which provider?
- Who pays, and how should cost be split?
- Self-hosted models needed?
2Requirements and numbers5-8 min
It must
- One API for all approved models
- Per-team keys, budgets and usage reports
- Fallbacks and model aliases
And it must be
- Provider keys never leave the platform
- Logs masked; data stays in approved regions
- 99.9% availability
Numbers to say out loud
- Traffic
- 2 million requests a day, peak 200 a second
- Teams
- 20 teams, 60 features
- Spend
- About ₹40 lakh a month
3High-level design8-12 min
- Team feature → team key → Gateway, 3 instances
- Gateway → Budget and policy checks
- Budget and policy checks → alias → Provider A, Mumbai
- Provider A, Mumbai → fails → Provider A, second region
- Provider A, second region → fails → Self-hosted open model
- Gateway → Masked logs and usage by team
- Masked logs and usage by team → Cost reports and alerts
4Deep dives12-26 min
Policy
Each key carries the team, allowed models, data class and monthly budget. Requests with confidential data can go only to models approved for it, such as the self-hosted one.
Operations
Model aliases let the platform team upgrade models after a shared eval run, without each team changing code. Alerts at 75% of budget; a hard stop at 100% unless the owner raises it.
5Failures26-30 min
| What can go wrong | What stops it |
|---|---|
| Gateway outage stops every team | Several instances across zones; health checks |
| A team leaks data to an unapproved model | Data class on the key; policy check blocks it |
| Runaway spend | Per-key budgets and alerts |
6Evals and monitoring30-33 min
- Shared golden sets per alias, run before any model upgrade
- Latency and error rate per provider
7Trade-offs33-35 min
- A central gateway adds one hop of latency but gives control
- Self-hosting gives data control at the cost of running GPUs
Follow-ups to expect
- How would you charge teams fairly?
- How do you roll out a new model safely?