Skip to main content

How would you keep the cost of an LLM feature under control?

System design · Pro 2

Short answer

Measure cost per request and per user, then cut it: use the cheapest model that passes your evals, shorten prompts and outputs, cache repeated prefixes and answers, batch work that can wait, and stop runaway usage with per-user limits and a provider spending cap. Recheck when traffic or prices change.

Deeper answer

Cost is tokens times price times volume, so attack each factor. Routing simple requests to a small model often gives the biggest saving. Trimming retrieved context and conversation history cuts input tokens. Batch APIs are cheaper for offline jobs.

Guardrails stop surprises: rate limits per user, maximum steps for agents, a hard monthly cap, and alerts when daily spend jumps. For a product in India, cost per active user has to fit low price points, so model the unit economics before launch.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Cost control

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.