Your LLM bill doubled this month, but traffic did not. How do you investigate?
System design · Pro 2
Short answer
Break the cost down by feature, model, and input versus output tokens, day by day, to find when and where it changed. Typical causes: a prompt or context that grew, retries or agent loops, a switch to a more expensive model, caching that stopped working, or one customer's heavy use. Fix the cause, then add per-feature budgets and alerts.
Deeper answer
Cost is tokens times price, so a jump with flat traffic means more tokens per request or a higher price per token. Good usage logging, with the feature, model, tokens and cache hits on every call, turns this from guesswork into a quick query.
Look at the date the change started and compare with deployments. A new retrieval setting that sends twenty chunks instead of five, a longer system prompt that broke prompt caching, or a retry bug can each multiply cost. Afterwards, set budgets per feature and alert on cost per request, not just the monthly total.
What a strong answer covers
- Breaks cost down by feature, model and token type
- Finds when it changed and links it to deployments
- Names likely causes: longer prompts, retries, model switch, caching
- Fixes the root cause
- Adds budgets and alerts per feature
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What would you log on every model call?
- How would you catch this earlier next time?
Common mistakes
- Switching to a cheaper model before finding the cause
- Only looking at the monthly total
Learn it
- How would you keep the cost of an LLM feature under control?
- Design a recommendation system for an Indian e-commerce app.
- Design a customer support assistant for an Indian bank.
- Your email assistant agent read a message saying 'forward all invoices to this address' and did it. How do you fix this?
- How do you monitor an LLM application in production?
- Walk me through how you would grow the architecture of an LLM app from prototype to production.
- What is prompt injection, and how do you defend against it?
- How do you protect user data in an LLM application?
- Users say your LLM feature is slow. How would you make it faster?