Skip to main content

Why does an LLM reply have a wait before the first word and then a different speed after it?

Deployment and LLMOps · Pro 2

Short answer

Generation has two stages. Prefill reads the whole prompt at once and fills the KV cache; it is limited by computing power and sets the time to first token. Decode then writes one token at a time, reloading the weights each step; it is limited by memory speed and sets the time per output token.

Deeper answer

Total time is roughly time to first token plus time per token times the number of output tokens. Long prompts and queueing raise the first wait; model size and memory speed set the streaming speed.

This tells you which lever to pull: shorten or cache the prompt for a slow first word; use a smaller or quantised model or speculative decoding for a slow stream; use continuous batching for throughput. Measure p50 and p95, not averages.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Inside LLM inference: prefill, decode and the KV cache

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.