Skip to main content

What is quantisation, and what are the trade-offs?

Deployment and LLMOps · Pro 1

Short answer

Quantisation stores model weights in fewer bits, for example 8-bit or 4-bit instead of 16-bit, which cuts memory and often speeds up inference, so a larger model fits on a smaller GPU. The cost is some loss of quality, which varies by model, method and task, so measure on your own eval set. Common methods include GPTQ, AWQ and bitsandbytes.

Deeper answer

Rough memory maths: a 7-billion-parameter model needs about 14 GB in 16-bit, about 7 GB in 8-bit and about 3.5 GB in 4-bit, plus room for the KV cache. That can be the difference between needing a data-centre GPU and fitting on a single consumer card.

Quality loss is usually small at 8-bit and more noticeable at 4-bit on reasoning-heavy tasks. Speed gains depend on hardware kernels, so benchmark rather than assume.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

GPUs and quantization

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.