Skip to main content

GPUs and quantization

Understand what limits a model on a GPU (memory for weights and cache, compute and bandwidth), choose GPU types and counts, and use quantization to fit and speed up models.

Pro 1 · Quality engineerabout 3 hours7 sections

Quick answer

A model on a GPU is limited by memory for its weights and cache. Quantization stores weights in fewer bits, such as 8 or 4, so a model fits and runs cheaper with little loss.

Why it matters

Self-hosting decisions are GPU decisions, and GPUs are expensive and scarce in Indian cloud regions. Platform engineers who can size a deployment on the back of an envelope save clients lakhs a month and avoid buying the wrong hardware.

Sign up free to read the whole lesson

The rest of this lesson: 7 parts, video lessons, the best course, practice and a quick check. It is free, takes a few seconds and needs no card.

Accounts are for learners aged 18 and over. Under 18? The quick answer and the first part of every lesson stay open: browse lessons by topic.

On the roadmap: MLOps Engineer

Deployment and opsPro 1: Quality engineerVideo lessonsGPUKV cacheQuantizationvLLM

Track

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.