Skip to main content

Users say your LLM feature is slow. How would you make it faster?

System design · Pro 2

Short answer

Measure first: find where time goes, such as retrieval, the model call or tools. Then stream the answer so text appears at once, cut input and output length, cache repeated prompt prefixes and common answers, use a smaller model for simple steps, and run independent steps in parallel.

Deeper answer

Perceived speed matters as much as total time: streaming and showing progress make an app feel fast. Output tokens are generated one by one, so asking for shorter answers often helps most. Long retrieved context adds input time and cost; rerank and send fewer, better chunks.

Architecture changes help too: precompute what you can, run retrieval and other lookups in parallel, avoid chains of sequential model calls where one call would do, and choose providers or regions close to users. Measure the effect of each change on both latency and quality.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Caching and speed

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.