Skip to main content

How would you serve a model that is too big for one GPU, and how would you scale it to more users?

Deployment and LLMOps · Pro 2

Short answer

First check the size: weights at the chosen precision plus KV cache room. If it fits in one server but not one GPU, use tensor parallelism across that server's GPUs, which have fast links. If it does not fit in one server, add pipeline parallelism across servers. To serve more users, run more copies behind a load balancer.

Deeper answer

Tensor parallelism splits each layer's matrices across GPUs that exchange results every layer, so it needs fast interconnects and can cut latency. Pipeline parallelism places groups of layers on different GPUs; each request passes through every stage, so it adds delay and is avoided for interactive serving when possible.

Quantising to 8-bit can halve the GPUs per copy if quality holds on the test set. I would keep at least two copies for failover, load-test with real prompt lengths at the busy hour, and use a serving engine like vLLM that handles these splits.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Running models on many GPUs: replicas, tensor and pipeline parallelism

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.