Why does an LLM reply have a wait before the first word and then a different speed after it?
Deployment and LLMOps · Pro 2
Short answer
Generation has two stages. Prefill reads the whole prompt at once and fills the KV cache; it is limited by computing power and sets the time to first token. Decode then writes one token at a time, reloading the weights each step; it is limited by memory speed and sets the time per output token.
Deeper answer
Total time is roughly time to first token plus time per token times the number of output tokens. Long prompts and queueing raise the first wait; model size and memory speed set the streaming speed.
This tells you which lever to pull: shorten or cache the prompt for a slow first word; use a smaller or quantised model or speculative decoding for a slow stream; use continuous batching for throughput. Measure p50 and p95, not averages.
What a strong answer covers
- Prefill reads the prompt in parallel
- Decode writes one token at a time
- Names time to first token and time per token
- Prefill is compute-bound, decode memory-bound
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- What is the KV cache and why does it use so much memory?
- How does speculative decoding speed up decode?
Common mistakes
- Treating latency as one number
- Thinking decode is limited by computing power
Learn it
Inside LLM inference: prefill, decode and the KV cache
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- A bank wants your AI system but says no data may leave its network. What are the options?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?