Skip to main content

What is the difference between data parallelism and model parallelism?

Deployment and LLMOps · Pro 2

Short answer

Data parallelism runs full copies of the model on several GPUs, each handling different batches in training or different requests in serving. Model parallelism splits one model across GPUs because it is too big or too slow for one: tensor parallelism splits work inside each layer, pipeline parallelism splits groups of layers.

Deeper answer

Data parallelism is simple and scales throughput, but each copy must fit. In training it also needs gradients and optimiser states, which is why sharded data parallelism (ZeRO, FSDP) spreads those across GPUs.

Tensor parallelism needs fast links because GPUs exchange results every layer; pipeline parallelism tolerates slower links but adds per-request delay, using micro-batches to keep stages busy in training. Large training runs combine all of these, often with context or sequence parallelism for long inputs.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Running models on many GPUs: replicas, tensor and pipeline parallelism

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.