How would you keep the cost of an LLM feature under control?
System design · Pro 2
Short answer
Measure cost per request and per user, then cut it: use the cheapest model that passes your evals, shorten prompts and outputs, cache repeated prefixes and answers, batch work that can wait, and stop runaway usage with per-user limits and a provider spending cap. Recheck when traffic or prices change.
Deeper answer
Cost is tokens times price times volume, so attack each factor. Routing simple requests to a small model often gives the biggest saving. Trimming retrieved context and conversation history cuts input tokens. Batch APIs are cheaper for offline jobs.
Guardrails stop surprises: rate limits per user, maximum steps for agents, a hard monthly cap, and alerts when daily spend jumps. For a product in India, cost per active user has to fit low price points, so model the unit economics before launch.
What a strong answer covers
- Measures cost per request or user first
- Uses the cheapest model that passes evals
- Shortens prompts and outputs
- Caches and batches
- Sets per-user limits and a spend cap
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you calculate cost per active user?
- What would you do if one user generated a large share of the cost?
Common mistakes
- Only switching to a cheaper model without checking quality
- Having no hard spending limit
Learn it
- Design a recommendation system for an Indian e-commerce app.
- Design a customer support assistant for an Indian bank.
- Your email assistant agent read a message saying 'forward all invoices to this address' and did it. How do you fix this?
- Your LLM bill doubled this month, but traffic did not. How do you investigate?
- How do you monitor an LLM application in production?
- Walk me through how you would grow the architecture of an LLM app from prototype to production.
- What is prompt injection, and how do you defend against it?
- How do you protect user data in an LLM application?
- Users say your LLM feature is slow. How would you make it faster?