KIVI Quantization Secures Llama-2 Accuracy at Two Bits
Researchers have demonstrated that quantizing the key-value cache of large language models like Llama-2 along different geometric axes prevents catastrophic accuracy loss at ultra-low precision.
Applying 2-bit quantization to the key-value (KV) cache of the Llama-2-13B model with a group size of 32 reveals that compression quality depends heavily on the chosen geometric axis. By quantizing keys along the channel dimension and values along the token dimension, researchers achieved a CoQA accuracy score of 63.53, close to the full-precision score of 66.37. Reversing these geometric choices caused the model to fail completely, plunging the benchmark score to a mere 2.88. This finding shifts the focus of KV cache compression from simple bit-reduction to structural optimization.
Managing KV cache memory is critical for serving large models. In a LLaMA-7B model, weights consume 98 percent of memory at a sequence length of 512, but at a 128K context length, the KV cache dominates at 84 percent of memory. For the OPT-175B model, a batch size of 512 with a 512-token prompt creates a massive 1.2TB KV cache. To compress this data without losing accuracy, practitioners must treat keys and values differently. Keys contain extreme outlier activations in fixed channels—with magnitudes up to 7000 compared to a median of 0.3 in Mixtral 8x7B. Quantizing keys per-channel isolates these outliers, reducing Llama-2-13B key reconstruction error from 13.67 to 4.55.
Conversely, the value cache lacks these outliers but must be quantized per-token due to how it is consumed. Although per-channel value quantization yields a slightly better raw tensor reconstruction error of 3.73 compared to 4.57 for per-token, it destroys model performance. This is because attention is highly sparse, measured at 84.3 percent. Measuring error at the attention output reveals that per-token quantization yields a relative error of only 3.55, whereas per-channel quantization spikes to 49.89.
For practitioners, this geometric approach yields massive system improvements. Using the KIVI method on Llama-2-7B reduces peak memory usage by 2.6 times, supports four times larger batch sizes, and increases throughput by up to 3.47 times. To implement this successfully, developers should quantize keys before applying Rotary Position Embeddings (RoPE), which otherwise increases quantization errors by 145 percent. Additionally, keeping a 128-token full-precision residual buffer helps recover accuracy on complex tasks, boosting Llama-2-7B GSM8K scores from 5.76 to 12.74. Finally, teams must validate compression using attention output or end-task performance rather than raw reconstruction error.
This is our own summary of reporting by Unite.AI
