Baseten unpacks the math behind TurboQuant
Baseten's research team has published a walkthrough of TurboQuant, Google Research's approach to KV-cache quantization, explaining how random preconditioning could remove the per-block scaling overhead that limits low-bit compression.
Baseten's research team published a deep dive into the math behind TurboQuant, a quantization method from Google Research aimed at shrinking the key-value (KV) cache in transformer inference. The post starts with PolarQuant, which it calls a prerequisite for understanding TurboQuant.
The post explains why the problem matters. The KV cache grows linearly with sequence length. For Llama-3.1-8B (32 layers, 8 KV heads, head dimension 128) at 128K context in FP16, the cache alone takes about 16 GB per user session. With several sessions running on one GPU, the KV cache becomes the main memory bottleneck.
According to Baseten, current aggressive schemes such as NVFP4 use two levels of scaling: a global maximum plus a local maximum for each 16-element block. Those scale factors and zero-points are kept in 16-bit precision. That adds overhead, and a single outlier can ruin precision for a whole block.
The Google Research authors say this can be avoided by pushing data into a predictable distribution ahead of time. They rely on two properties. First, multiplying any fixed vector by a random Gaussian matrix gives a zero-centered multivariate Gaussian. Second, the norm of a high-dimensional Gaussian vector concentrates tightly around √d. Baseten checks both properties in code. One check uses a 64-dimensional key vector from GPT-2 (layer 5, head 0, token 5), multiplied by 10,000 random matrices, and each resulting coordinate follows a normal distribution.
The excerpt ends partway through the explanation of preconditioning. It does not cover TurboQuant's full algorithm, its compression ratios, or any benchmark results.