Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.
FDC: Fast KV Dimensionality Compression for Efficient LLM Inference
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In large-language models, memory constraints in the Key-Value Cache (KVC) pose a challenge during inference. In this work, we propose FDC, a fast KV dimensionality compression system that eliminates the decompression overhead incurred in the existing KV dimensionality compression system, Palu, and reduces attention time. Moreover, FDC employs adaptive compression, tailoring KV compression rates across heads and layers based on their contributions to inference to maximize overall compression while maintaining an accuracy loss constraint. Additionally, FDC enhances the attention kernel to balance the uneven workloads caused by the adaptive compression approach to further reduce attention computation latency. Comprehensive experiments demonstrate that compared to Palu, FDC can reduce Job Completion Time (JCT) by up to 64%, and delivers up to 1.97X throughput under the same latency, while maintaining 99% of the accuracy without compression. When state-of-the-art eviction and quantization methods are combined with FDC, they exhibit similar improvements compared to those combined with Palu. We open-sourced the code.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.