GPUs and quantization
Understand what limits a model on a GPU (memory for weights and cache, compute and bandwidth), choose GPU types and counts, and use quantization to fit and speed up models.
Pro 1 · Quality engineerabout 3 hours7 sections
Quick answer
A model on a GPU is limited by memory for its weights and cache. Quantization stores weights in fewer bits, such as 8 or 4, so a model fits and runs cheaper with little loss.
Why it matters
Self-hosting decisions are GPU decisions, and GPUs are expensive and scarce in Indian cloud regions. Platform engineers who can size a deployment on the back of an envelope save clients lakhs a month and avoid buying the wrong hardware.
Sign up free to read the whole lesson
The rest of this lesson: 7 parts, video lessons, the best course, practice and a quick check. It is free, takes a few seconds and needs no card.
Accounts are for learners aged 18 and over. Under 18? The quick answer and the first part of every lesson stay open: browse lessons by topic.
Lessons on MLOps Engineer0 of 37 done
- 1Packages and virtual environments
- 2The terminal and Git
- 3Call an LLM API from code
- 4Python for production
- 5Linux, the shell and SSH
- 6Docker and Compose
- 7Kubernetes and Helm basics
- 8One cloud in depth
- 9Experiment tracking and model registry
- 10Data versioning
- 11Serving a model
- 12Monitoring and drift
- 13Serving an AI app
- 14Deploy and monitor an LLM app
- 15LLM gateways
- 16Open-weight models with vLLM
- 17GPUs and quantization
- 18LoRA, QLoRA and the GPU memory maths
- 19Inside LLM inference: prefill, decode and the KV cache
- 20Running models on many GPUs: replicas, tensor and pipeline parallelism
- 21CI/CD with GitHub Actions
- 22Environments, secrets and Terraform
- 23Regression tests for prompts
- 24Versioning prompts and models
- 25Tracing LLM apps
- 26Logs and traces for agents
- 27Test sets and evals
- 28Online evals and feedback
- 29Caching and speed
- 30Cost control
- 31Batching and model tiering
- 32Latency budgets and load testing
- 33Security and privacy
- 34Audit logs
- 35Compliance for AI systems
- 36AI risk frameworks
- 37Incident response for AI
Related
On the roadmap: MLOps Engineer
Deployment and opsPro 1: Quality engineerVideo lessonsGPUKV cacheQuantizationvLLM