Intermediate
What is Quantization?
Shrinking models by using fewer bits per number — with minimal quality loss.
Model weights are numbers. Training typically uses 16-bit floats; quantization converts them to 8-bit, 4-bit, or even fewer bits, cutting memory use dramatically — a 70B model at 16-bit needs ~140 GB of VRAM, but ~40 GB at 4-bit.
Lower precision introduces small rounding errors. Modern quantization methods (AWQ, GPTQ, GGUF quants) choose precision intelligently so quality loss is small for most uses.
Quantization is what makes running capable models on consumer GPUs, laptops, and phones possible. It's the foundation of the local-AI ecosystem.
Key points
- Fewer bits per weight = less memory
- 70B model: ~140GB → ~40GB at 4-bit
- Smart methods keep quality loss small
- Enables local AI on consumer hardware