GPUs-um quantization-um
Oru model-a GPU-la enna limit pannudhu-nu (weights-kum cache-kum memory, compute, bandwidth) purinjukkalaam, GPU types-um counts-um select pannalaam, models-a fit pannavum vegamaakkavum quantization use pannalaam.
Pro 1 · Quality engineerabout 3 hours7 sections
Short-aa sonna
GPU-la irukkura oru model-a, adhoda weights-kum cache-kum irukkura memory thaan limit pannudhu. Quantization weights-a 8 illa 4 maadhiri kammiyaana bits-la store pannum, appo model fit aagum, konjam loss-oda sellavu kammiyaa odum.
Idhu yen mukkiyam
Self-hosting decisions-ae GPU decisions thaan, India cloud regions-la GPUs sellavum adhigam, kidaikkuradhum kammi. Oru deployment-a kai-kanakkula size panna theriyura platform engineers, clients-ku maadhathukku lakhs migaikkuraanga, thappaana hardware vaangaama thavirkkuraanga.
Sign up free to read the whole lesson
The rest of this lesson: 7 parts, video lessons, the best course, practice and a quick check. It is free, takes a few seconds and needs no card.
Accounts are for learners aged 18 and over. Under 18? The quick answer and the first part of every lesson stay open: browse lessons by topic.
Lessons on MLOps Engineer0 of 37 done
- 1Packages and virtual environments
- 2The terminal and Git
- 3Call an LLM API from code
- 4Python for production
- 5Linux, the shell and SSH
- 6Docker and Compose
- 7Kubernetes and Helm basics
- 8One cloud in depth
- 9Experiment tracking and model registry
- 10Data versioning
- 11Serving a model
- 12Monitoring and drift
- 13Serving an AI app
- 14Deploy and monitor an LLM app
- 15LLM gateways
- 16Open-weight models with vLLM
- 17GPUs and quantization
- 18LoRA, QLoRA and the GPU memory maths
- 19Inside LLM inference: prefill, decode and the KV cache
- 20Running models on many GPUs: replicas, tensor and pipeline parallelism
- 21CI/CD with GitHub Actions
- 22Environments, secrets and Terraform
- 23Regression tests for prompts
- 24Versioning prompts and models
- 25Tracing LLM apps
- 26Logs and traces for agents
- 27Test sets and evals
- 28Online evals and feedback
- 29Caching and speed
- 30Cost control
- 31Batching and model tiering
- 32Latency budgets and load testing
- 33Security and privacy
- 34Audit logs
- 35Compliance for AI systems
- 36AI risk frameworks
- 37Incident response for AI
Related
On the roadmap: MLOps Engineer
Deployment and opsPro 1: Quality engineerVideo lessonsGPUKV cacheQuantizationvLLM