Quantization Basics
Aditya Desai
Quantization maps a continuous (or large) set of values to a smaller discrete set.
There are two maps to keep track of:
If the only goal is to store and recover data, then a natural metric is reconstruction error.
For values drawn from a distribution $P$, this is the mean squared error:
Q. Should we always care about reconstruction loss?
A. No — it depends on how the quantized data is used.
For example, if we quantize weights $W$, we may care more about the downstream loss or about inner products than about $\|W-\hat W\|_2$.
This distinction is important in ML: a quantizer that looks good by reconstruction error may not be the one that preserves model quality.
Computers already quantize numbers: a dtype determines which values are representable.
Before designing our own low-bit maps, recall the two basic families of number formats.
Example: signed INT32.
Formats such as FP16, FP32, FP64 use a sign / exponent / mantissa layout (“$1+e+m$ bits”):
What does the bit string represent?
Here $n(E)$ is the integer encoded by the exponent field, and $n(0.M)$ is the fractional value of the mantissa bits.
FP16 encoding cases (normals, subnormals, specials).
Common ML dtypes (FP32, BF16, FP16, INT8, …) trade precision (spacing between representable points) against range (max − min representable value).
FP32 and FP16 are the common “go-to” formats, but efficient ML often pushes further toward smaller bit widths:
The objects we quantize are usually tensors or matrices: weights, activations, KV cache entries, and similar model states / optimizer states.
Either way, once we have a tensor or a calibration distribution, the core design question is how to place a small number of integer codes on the real line.
Q. Given a set of values, how do we design $Q$?
Target of quantization: signed $b$-bit integer codes: there are $2^b$ representable values, with range
(e.g. $b=8$ gives $-128,\ldots,127$). How we place those levels on the real line really depends on the data.
Simplest function that can map $\{-2^{b-1},\ldots,2^{b-1}-1 \}$ to $[-r_{\max}, r_{\max}]$ is a linear function.
The simplest answer is to use a uniformly spaced grid.
Linear quantization: real value $r$ is mapped to integer code $q$ using the scale $\lambda$: $r = \lambda q$.
\[ Q(r) = \left[ \frac{r}{\lambda} \right]_Q \] \[ Q^{-1}(q) = \lambda q \]
Choose $\lambda$ so the largest magnitude maps to the largest positive code.
Q. When is a pure scale map bad?
When the distribution is shifted / asymmetric: a zero-centered integer grid wastes codes on an empty region of the real line.
To move the grid left or right, introduce a zero-point $z_q$ as well as a scale:
Q. When can linear quantization still fail?
Outliers. A few extreme values stretch the scale, so most of the mass gets too few effective bits.
Outliers consume dynamic range that the bulk of the data needs.
E.g. Activation tensors are being quantised.
One approach: Separately store a sparse mask.
Alternatively "Rotate the Vector" to redistribute the energy
* we can use linear quantisation for \(Q\).
If \(\vec{x}\) was fixed / unknown?
Say, \(\vec{x}\) was fixed.
A. Determine \(R\) which is optimal.
\(\Rightarrow\) But now you have to store \(R\) or \(R^{-1}\in\mathbb{R}^{d\times d}\)
\(R\): needs to be structured to reduce the memory footprint.
A given structured matrix may or may not work for a specific \(\vec{x}\).
Side Note: In fact for any chosen \(R\), there is a \(\vec{x}\) for which \(R\vec{x}\) is catastrophic w.r.t quantisation!!
\(\rightarrow\) Example?
[Idea] Randomisation
So that with high probability, \(Rx\) is not adversarial.
It can be stored efficiently.
Randomised Hadamard
Let \(n=2^k\).
\(\vec{s}\): Random sign vector
Computational Complexity of transforming nxn matrix
Note that this is additional computation that we are introducing.
'HadaCore': Kernel for Hadamard transform. https://pytorch.org/blog/hadacore/
1 sign bit per element.
* Note: we can avoid this \(O(n)\) memory by using Universal Hash functions. (More on that later)
Given \(x\), what is the probability that some coordinate of \(Rx\) is greater than \(M\).
Which tail bound do we use?
Each term in \(y_i\) is bounded, so Hoeffding applies. Let \(X_1,\ldots,X_n\) be independent with \(X_j \in [a_j,b_j]\) almost surely, and \(S=\sum_j X_j\). For \(t>0\),
Exercise: Derive the Chernoff bound for the same sum.
Random signs \(s_j h_{ij}\in\{\pm 1\}\), so \(X_j \in [a_j,b_j]\) with
\(\mathbb{E}[y_i]=0\). Plug \(t=M\) into two-sided Hoeffding:
Q. Where else does linear quantization fail?
Multimodal distributions. A uniform grid spends codes evenly across an interval, but the data may live in separated clusters with empty space between them.
Linear levels miss multimodal structure.
Weight clustering: float weights mapped to low-bit cluster indices plus centroids.
| Aspect | Linear quantization | Codebook (non-linear) |
|---|---|---|
| Mapping | Affine: $q=\mathrm{round}(x/s)+z$ | Lookup: $q=\arg\min_i |x-c_i|$ |
| Reconstruction | $x\approx s(q-z)$ | $x\approx c_q$ |
| Best for | Roughly uniform / unimodal mass | Skewed or multimodal mass |
| Hardware | Excellent (integer ops) | Moderate / poor |
| Compute | Often stays in quantized space | Usually needs dequantization |
Reconstruction error is only a proxy — quantized values are used in downstream computation. The simplest case: an inner product.
Q. What is $\langle \hat{w},\, x\rangle$, and how well does it approximate $\langle w,\, x\rangle$?
Assume a grid of precision $\delta$ (spacing between representable values). Round-to-nearest guarantees
The quantized inner product expands as
If the signs align, every term $x_i\Delta_i$ can push in the same direction.
For example, take $x_i \ge 0$ and $\Delta_i=\delta/2$ for all $i$. Then
Squaring gives a worst-case squared error of order
Round-to-nearest minimizes per-coordinate error, but the inner-product error can still accumulate coherently across coordinates.
Suppose $w_i$ lies between consecutive grid points $a \lt w_i \lt b$ with $b-a=\delta$. Stochastic rounding sets
Equivalently, $\hat{w}=w+\Delta$ with a random per-coordinate error $\Delta_i$.
The quantized inner product is unbiased.
Assuming independent rounding noise across coordinates,
With fractional part $\alpha_i=(w_i-a)/\delta \in [0,1]$,
Therefore
The bound can be smaller still, depending on the actual fractional parts $\{\alpha_i\}$ and on $x$.
Since $\|x\|_2 \le \|x\|_1 \le \sqrt{d}\,\|x\|_2$, the $\ell_1$ worst case can be up to a factor $d$ larger in squared error than the stochastic $\ell_2$ MSE, especially for dense $x$.
To define a quantization scheme, we need to decide which weights to group together for quantization. This choice determines the granularity of quantization.
Finer granularity (quantizing smaller groups separately) can improve accuracy by adapting quantization to the distribution of each group, but may require storing more quantization parameters.