Skip to main content

System design answer

An internal LLM platform for many teams

A gateway with per-team keys and budgets, model aliases and fallbacks, masked logging, shared evals and approved models.
The question“Design a shared platform so 20 teams in a company can use LLMs safely, with control of cost and data.”

1Clarify0-5 min

Ask before you draw. These answers change the design.

  • How many teams and requests, and which models are approved?
  • What data may go to which provider?
  • Who pays, and how should cost be split?
  • Self-hosted models needed?

2Requirements and numbers5-8 min

It must

  • One API for all approved models
  • Per-team keys, budgets and usage reports
  • Fallbacks and model aliases

And it must be

  • Provider keys never leave the platform
  • Logs masked; data stays in approved regions
  • 99.9% availability

Numbers to say out loud

Traffic
2 million requests a day, peak 200 a second
Teams
20 teams, 60 features
Spend
About ₹40 lakh a month

3High-level design8-12 min

  1. Team feature → team key → Gateway, 3 instances
  2. Gateway → Budget and policy checks
  3. Budget and policy checks → alias → Provider A, Mumbai
  4. Provider A, Mumbai → fails → Provider A, second region
  5. Provider A, second region → fails → Self-hosted open model
  6. Gateway → Masked logs and usage by team
  7. Masked logs and usage by team → Cost reports and alerts

4Deep dives12-26 min

Policy

Each key carries the team, allowed models, data class and monthly budget. Requests with confidential data can go only to models approved for it, such as the self-hosted one.

Operations

Model aliases let the platform team upgrade models after a shared eval run, without each team changing code. Alerts at 75% of budget; a hard stop at 100% unless the owner raises it.

5Failures26-30 min

What can go wrongWhat stops it
Gateway outage stops every teamSeveral instances across zones; health checks
A team leaks data to an unapproved modelData class on the key; policy check blocks it
Runaway spendPer-key budgets and alerts

6Evals and monitoring30-33 min

  • Shared golden sets per alias, run before any model upgrade
  • Latency and error rate per provider

7Trade-offs33-35 min

  • A central gateway adds one hop of latency but gives control
  • Self-hosting gives data control at the cost of running GPUs

Follow-ups to expect

  • How would you charge teams fairly?
  • How do you roll out a new model safely?

Practise out loud in the mock interview

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.