Pith. sign in

REVIEW 5 cited by

IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.01241 v2 pith:C2FKGQK7 submitted 2024-03-02 cs.CL cs.AI

Ruikang Liu , Haoli Bai , Haokun Lin , Yuening Li , Han Gao , Zhengzhuo Xu , Lu Hou , Jun Yao
show 1 more author
Chun Yuan
This is my paper · ORCID
classification cs.CLcs.AI
keywords intactkvquantizationllmstokenslanguagepivotlargemethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outliers in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which are crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the KV cache of pivot tokens losslessly from the full-precision model. The approach is simple and easy to combine with existing quantization solutions with no extra inference overhead. Besides, IntactKV can be calibrated as additional LLM parameters to boost the quantized LLMs further with minimal training costs. Mathematical analysis also proves that IntactKV effectively reduces the upper bound of quantization error. Empirical results show that IntactKV brings consistent improvement over various quantization methods across different LLMs and downstream tasks, leading to the new state-of-the-art for LLM quantization. The codes are available at https://github.com/ruikangliu/IntactKV.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Causal Mask Attention for Vision-Language Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Relaxing causal masking so image tokens can preview future image and text context during prefill improves several vision-language benchmarks, and pooling future attention into a single prefix token preserves most of the gain.

  2. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.

  3. AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AKVQ-VL quantizes VLM KV caches to mostly 2 bits with attention-aware token protection and Walsh-Hadamard outlier removal, staying near FP16 accuracy on MileBench.

  4. RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations

    cs.LG 2025-01 conditional novelty 6.0 of 10

    RotateKV combines outlier-aware, pre-RoPE grouped-head Hadamard rotation with attention-sink-aware retention to make 2-bit KV cache quantization accurate on LLaMA-2, LLaMA-3, and Mistral models.

  5. Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse

    cs.CL 2026-02 unverdicted novelty 5.0 of 10

    Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...

Pith tools