Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?
Deployment and LLMOps · Pro 1
Short answer
vLLM is built for high-throughput inference. PagedAttention manages the key-value cache in small blocks, so far less GPU memory is wasted and more requests fit at once, and continuous batching adds new requests to a running batch instead of waiting. It also offers an OpenAI-compatible server, quantisation support and tensor parallelism across GPUs.
Deeper answer
A naive generate loop serves one request at a time and wastes memory on padding, so throughput collapses under load. vLLM keeps the GPU busy.
Before self-hosting, compare total cost with a hosted API: GPUs cost money even when idle, and you take on operations. Self-hosting makes sense for data residency, steady high volume, or customised models.
What a strong answer covers
- Built for high throughput
- PagedAttention manages the KV cache efficiently
- Continuous batching
- OpenAI-compatible server and parallelism
- Compares with hosted API cost
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- When is a hosted API cheaper than self-hosting?
- What is the KV cache?
Common mistakes
- Assuming self-hosting is always cheaper
- Ignoring GPU memory planning
Learn it
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- A bank wants your AI system but says no data may leave its network. What are the options?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?