Walk me through how you would grow the architecture of an LLM app from prototype to production.
System design · Pro 2
Short answer
Start with a single model call and real users, and add layers only when a problem needs them: retrieval and tools when answers lack facts, guardrails when real data arrives, a router and gateway when requests differ or several models are used, caches when questions repeat, and agent loops and actions last. Trace every request from day one.
Deeper answer
Each layer brings new risks, so I record why it exists and measure it separately: retrieval recall, router accuracy, guardrail false blocks, cache error rate. A router is usually a small fast classifier that also sends out-of-scope requests to stock replies and some to people. A gateway centralises keys, access control, spend limits, logging, retries and fallbacks.
Semantic caches need care because similar questions with different details have different answers. Write actions need approvals, permissions and budgets. Prompts, router, guardrail rules and cache keys are versioned and rolled back together.
What a strong answer covers
- Starts simple with a single model call
- Adds context through retrieval or tools
- Adds input and output guardrails
- Router and gateway when needed
- Caches and agents later, with monitoring throughout
Answer it out loud in the mock interview about 2 min
Follow-ups to expect
- What is the difference between a router and a gateway?
- When is a semantic cache unsafe?
Common mistakes
- Building every layer before having users
- Adding agents before context and guardrails work
Learn it
Grow an AI app in layers: context, guardrails, routing, caching and agents
- How would you keep the cost of an LLM feature under control?
- Design a recommendation system for an Indian e-commerce app.
- Design a customer support assistant for an Indian bank.
- Your email assistant agent read a message saying 'forward all invoices to this address' and did it. How do you fix this?
- Your LLM bill doubled this month, but traffic did not. How do you investigate?
- How do you monitor an LLM application in production?
- What is prompt injection, and how do you defend against it?
- How do you protect user data in an LLM application?
- Users say your LLM feature is slow. How would you make it faster?