Users say your LLM feature is slow. How would you make it faster?
System design · Pro 2
Short answer
Measure first: find where time goes, such as retrieval, the model call or tools. Then stream the answer so text appears at once, cut input and output length, cache repeated prompt prefixes and common answers, use a smaller model for simple steps, and run independent steps in parallel.
Deeper answer
Perceived speed matters as much as total time: streaming and showing progress make an app feel fast. Output tokens are generated one by one, so asking for shorter answers often helps most. Long retrieved context adds input time and cost; rerank and send fewer, better chunks.
Architecture changes help too: precompute what you can, run retrieval and other lookups in parallel, avoid chains of sequential model calls where one call would do, and choose providers or regions close to users. Measure the effect of each change on both latency and quality.
What a strong answer covers
- Measures where the time goes first
- Streams the answer
- Shortens input and output
- Caches prefixes and common answers
- Smaller models and parallel steps
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Which usually matters more for latency: input or output length?
- How would you speed up a pipeline of three sequential model calls?
Common mistakes
- Optimising without measuring where the time goes
- Cutting quality without checking the eval set
Learn it
- How would you keep the cost of an LLM feature under control?
- Design a recommendation system for an Indian e-commerce app.
- Design a customer support assistant for an Indian bank.
- Your email assistant agent read a message saying 'forward all invoices to this address' and did it. How do you fix this?
- Your LLM bill doubled this month, but traffic did not. How do you investigate?
- How do you monitor an LLM application in production?
- Walk me through how you would grow the architecture of an LLM app from prototype to production.
- What is prompt injection, and how do you defend against it?
- How do you protect user data in an LLM application?