How would you serve a model that is too big for one GPU, and how would you scale it to more users?
Deployment and LLMOps · Pro 2
Short answer
First check the size: weights at the chosen precision plus KV cache room. If it fits in one server but not one GPU, use tensor parallelism across that server's GPUs, which have fast links. If it does not fit in one server, add pipeline parallelism across servers. To serve more users, run more copies behind a load balancer.
Deeper answer
Tensor parallelism splits each layer's matrices across GPUs that exchange results every layer, so it needs fast interconnects and can cut latency. Pipeline parallelism places groups of layers on different GPUs; each request passes through every stage, so it adds delay and is avoided for interactive serving when possible.
Quantising to 8-bit can halve the GPUs per copy if quality holds on the test set. I would keep at least two copies for failover, load-test with real prompt lengths at the busy hour, and use a serving engine like vLLM that handles these splits.
What a strong answer covers
- Sizes weights plus KV cache
- Tensor parallelism inside a server
- Pipeline only when spanning servers
- Replicas for more users, with failover
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Why avoid tensor parallelism across servers?
- How would quantisation change your layout?
Common mistakes
- Forgetting the KV cache when sizing
- Using pipeline parallelism for latency-sensitive chat
Learn it
Running models on many GPUs: replicas, tensor and pipeline parallelism
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- A bank wants your AI system but says no data may leave its network. What are the options?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?