Applying quantization to neural networks: PTQ and QAT, second-order weight quantization, activation-aware scaling, fake quantization, and hardware-supported low-precision formats.
The previous notes developed the basic mechanics of quantization. We now ask how those ideas are applied to LLMs and other neural networks. There are two broad workflows.
Start with a trained model and quantize it without retraining or fine-tuning.
Simulate quantization during training so the model can adapt to the errors introduced by low precision.
| Scheme | Examples | What happens |
|---|---|---|
| Weight-only | W8, W4 | Weights are stored at low precision; activations remain in a floating-point format. |
| Weight + activation | W8A8, W4A4 | Both operands of a matrix multiplication use low precision. |
In a weight-only kernel, low-bit weights may be dequantized inside the kernel and combined with BF16/FP16 activations. For an integer weight-and-activation kernel, the data path is different:
GPTQ applies ideas related to Optimal Brain Surgeon to post-training weight quantization. Instead of rounding every weight independently, it quantizes one coordinate and adjusts the remaining coordinates to compensate for the induced error.
Let:
The ideal discrete problem is
Searching the complete grid is intractable, and repeatedly evaluating the full model is expensive. GPTQ therefore uses a local layer-reconstruction objective:
For $X\in\mathbb R^{n\times d_{\mathrm{in}}}$ and $W\in\mathbb R^{d_{\mathrm{in}}\times d_{\mathrm{out}}}$, this separates over output dimensions—that is, over columns of $W$:
The input-side Hessian is, up to an optional factor of two,
Define the perturbation using the convention $\delta W=\widehat W-W$. If coordinate $i$ is fixed to its quantized value, then
Under the local quadratic model, the minimum loss increase caused by fixing this coordinate is
$S_i$ is the coordinate’s saliency. It measures the smallest quadratic loss increase after optimally compensating with all other coordinates. A greedy method prefers coordinates with lower saliency.
Factoring a dense $d\times d$ Hessian costs $O(d^3)$. Repeating that factorization after each of $d$ decisions would lead to an $O(d^4)$ procedure. GPTQ avoids this naive cost through several distinct optimizations:
Across the coordinates, the factorization and inverse-update work is $O(d^3)$. SparseGPT adapts related second-order machinery to pruning; GPTQ uses it to quantize weights.
GPTQ compensates after a weight is moved to a fixed grid. AWQ takes a complementary view: activation statistics reveal which weight channels are sensitive, and an exact rescaling can protect those channels before quantization.
Consider a diagonal approximation to the local quadratic loss. For a perturbation $\delta w_i$,
For squared layer-reconstruction error, $H_{ii}\propto\|X_{:,i}\|_2^2$. Hence
One option is to keep salient channels at higher precision. That creates an irregular mixed-precision matrix and is difficult to execute efficiently with standard GEMM kernels. AWQ instead preserves a uniform bit width and rescales channels.
For a positive scale vector $s$, rewrite the exact product as
Before quantization this transformation is lossless. The benefit comes from changing how quantization error is distributed.
Consider a scalar contribution $xw$. Scale $w$ to $sw$, quantize it, and compensate by multiplying with $x/s$.
If $\Delta'=s\Delta$, then
No reduction in output error.
If other values set the range and $\Delta'=\Delta$, then
Error is reduced by a factor of $s$.
The scale can be selected through layer reconstruction:
At lecture level, AWQ parameterizes the scales as
and searches over the migration parameter $\alpha$.
AWQ uses activation statistics to improve weight-only quantization. SmoothQuant targets weight-and-activation quantization, where activation outliers are often the main difficulty.
Per-channel activation quantization along the inner dimension would isolate those outliers, but that granularity does not map efficiently to standard matrix-multiplication kernels. SmoothQuant instead moves part of the dynamic range from activations into weights:
A commonly used channel scale has the form
where $\alpha$ controls the migration strength: larger $\alpha$ moves more quantization difficulty from activations to weights.
Conceptually, we would like to choose
| Method | Main signal | Main intervention |
|---|---|---|
| GPTQ | Second-order curvature | Compensate other weights after quantizing a coordinate |
| AWQ | Activation magnitude | Protect salient weight channels through scaling |
| SmoothQuant | Activation and weight ranges | Move dynamic range from activations into weights |
PTQ leaves the trained weights fixed, so any quality loss must be controlled entirely through calibration and the quantization method. QAT lets the model adapt while seeing simulated quantization noise.
Insert fake-quantization operators wherever weights or activations will be quantized:
$Q$ maps a floating-point value to a quantized code, while $Q^{-1}$ dequantizes that code back to a representable floating-point value. The surrounding computation graph therefore remains floating-point, but its forward values lie on the intended quantization grid.
Rounding creates a staircase function whose true derivative is zero almost everywhere and undefined at thresholds. QAT replaces that derivative with a useful surrogate. For a clipped fake quantizer, a common STE is
Ignoring clipping gives the simpler identity STE, which treats the derivative as $1$ everywhere.
A low-precision representation is useful only when the hardware can consume it efficiently. Modern accelerators support several floating and microscaled formats.
Two common FP8 encodings are:
Four exponent bits and three explicit mantissa bits.
Five exponent bits and two explicit mantissa bits.
| Format | Data | Scale granularity | Scale format |
|---|---|---|---|
| MXFP8 | FP8 values | One shared scale per group of 32 values | E8M0 |
| NVFP4 | FP4 E2M1 values | One shared scale per group of 16 values | FP8 E4M3, plus a tensor-level FP32 scale |
Hardware support means that the accelerator exposes primitives which understand both the low-precision tensors and their scale metadata, conceptually:
Fusing scale handling into the low-precision matrix-multiplication primitive minimizes or absorbs most conversion and scaling overhead. It does not mean that scale metadata and scale arithmetic have literally zero cost.
Scale layout matters during training. A forward linear layer uses $W$, while backpropagation also requires products involving $W^\top$:
A one-dimensional scale layout optimized for $W$ may become awkward after transposition. Two-dimensional scale tiles can be more symmetric with respect to transposition and are therefore useful in training. Exact tile shapes and transpose support remain hardware- and implementation-specific.
Previous: Quantization basics. Source: handwritten lecture notes.