Quantization

A technique that represents weights and, in some designs, activations at lower numerical precision than originally used. The mapping commonly uses scales, zero-points, and per-tensor, per-channel, or group-wise granularity. Fewer bits reduce storage and memory traffic; compatible kernels and hardware can turn that reduction into higher throughput and lower latency.

Post-training quantization applies the conversion after training, while quantization-aware training simulates its effects during training. The trade-off covers quantization error, task quality, memory, energy, and actual runtime support. A smaller file does not guarantee faster inference when hardware must dequantize values or lacks efficient kernels. Distillation compresses knowledge through training; quantization changes the numerical representation.