Skip to main content

What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?

Deployment and LLMOps · Pro 2

Short answer

The KV cache stores the key and value vectors for every earlier token, so each new token does not recompute them. It saves work but uses GPU memory that grows with context length, model size and the number of requests served at once, so long chats from many users can exceed memory.

Deeper answer

A rough size is 2 times layers times model width times tokens times batch size times bytes per number. A 13-billion-parameter model with 32 requests of 2,048 tokens can need over 50 GB, more than its weights.

Fixes include shorter contexts and summarised history, serving engines with paged KV memory such as vLLM, models that share keys and values across heads, a quantised cache, and limits on concurrent requests.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Inside LLM inference: prefill, decode and the KV cache

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.