Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PM-KVQ claims 2-bit KV cache quantization can match 16-bit accuracy within a few points on long-CoT LLMs by storing 16-bit first, then shrinking precision only when memory fills.

desk verdict Progressive KV-cache quantization is a genuinely new idea with credible accuracy gains, but the abstract's throughput claim outruns the evidence. read the letter →

arxiv 2505.18610 v2 pith:T7YOGEZX submitted 2025-05-24 cs.CL

Tengxuan Liu , Shiyao Li , Jiayi Yang , Tianchen Zhao , Feng Zhou , Xiaohui Song , Guohao Dai , Shengen Yan
show 2 more authors
Huazhong Yang Yu Wang
This is my paper · ORCID
classification cs.CL
keywords KVcachequantizationprogressivemixed-precisionlongchain-of-thoughtpositionalinterpolationRoPEcalibrationLLMinferencememory-efficientserving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes PM-KVQ, a post-training quantization scheme for the key-value cache of long chain-of-thought LLMs. It argues that directly quantizing the KV cache at every decoding step accumulates errors over tens of thousands of generated tokens, and that calibration on short sequences misreads RoPE-affected key channels. PM-KVQ instead starts each block at 16 bits and shrinks the cache to its target bit-width only when the memory budget fills, allocates different target bit-widths to different transformer blocks according to measured sensitivity, and calibrates using short sequences with positional interpolation to imitate long contexts. On 7B-70B reasoning models, the paper reports matching 16-bit accuracy within a few points at 2-bit KV cache and gains up to 8% over baselines under equal memory.

What carries the argument

The load-bearing object is the progressive quantization schedule with the 'Equivalent Right Shift' identity, $X_b = ((2^{2b}-2^b+1)(X_{2b}+2^{b-1})) \gg 3b$, which is claimed to equal dequantizing a $2b$-bit cache to FP16 and requantizing at $b$-bit while keeping the zero point and scaling the step by $(2^b+1)$. At each power-of-two transition (16 to 8, 8 to 4, 4 to 2) this operation lets the method shrink the existing cache in place. Two secondary mechanisms carry the rest: a first-order Taylor sensitivity score per block feeding an integer-programming solver that picks each block's final bit-width under a memory cap, and RoPE positional interpolation during calibration that stretches short sequences to approximate long-context key-channel distributions.

What would settle it

Run PM-KVQ side by side with a variant that literally dequantizes each block's KV cache to FP16 and re-quantizes at the lower bit-width at every shrinking step; if Eq. (3) is exact the two runs must produce identical integer caches, and any numerical divergence, or an AIME pass@1 gap between the two, would show the equivalence claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the two failure modes of short-context KV cache quantization on long-CoT LLMs, cumulative error and short-horizon calibration, can both be removed without changing the memory budget. Progressive quantization begins every block at 16-bit and applies an integer 'Equivalent Right Shift' (Eq. 3) to shrink 16 to 8 to 4 to 2 only when the block's memory allocation is exhausted, so early tokens keep high precision throughout. Block-wise memory allocation solves an integer program that minimizes a first-order Taylor sensitivity term over bit-width choices under a memory constraint, assigning 4-bit to sensitive blocks while others run at 2-bit. Calibration with positional interpolation multiplies the RoPE position index by a factor s, so 2,048-token calibration sequences stand in for 8,192-token contexts. The experiments on DeepSeek-R1-Distill and QwQ models claim that at 2-bit the method stays within a few points of the 16-bit model and beats KIVI by up to 8%.

Load-bearing premise

The method's in-place shrinking rests on Eq. (3) being exactly equivalent to dequantizing the 2b-bit KV cache and requantizing it at b-bit, but the paper asserts this without proof; if the formula's dyadic rounding introduces a systematic bias for some attention distributions, progressive quantization could accumulate errors the experiments do not isolate.

Editorial extensions

If this is right

  • A 2-bit KV cache becomes usable for long-CoT reasoning models, with reported pass@1 within a few points of the uncompressed 16-bit model on AIME and LiveCodeBench.
  • Under a fixed memory budget the same GPU can hold a larger batch, giving reported throughput improvements of 2.73-5.18x over 16-bit inference.
  • Calibration can stay on short sequences: with s=4, 2,048-token calibration data approximates 8,192-token contexts, avoiding the high cost of long-sequence calibration.
  • Non-uniform block-wise bit-widths under a memory cap beat any uniform bit-width, because sensitive blocks (often deep blocks, and the first block in Qwen-based models) get 4-bit while others run at 2-bit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A formal proof or counterexample for Eq. (3)'s exactness would settle whether the method can be extended to more aggressive schedules, such as non-power-of-two scales or 3-bit targets.
  • The sensitivity-based allocation measures only first-order loss change; second-order or attention-map-based sensitivity might change which blocks receive high bit-width, especially for models with MLA-style attention.
  • Because the paper reports that s=16 degrades performance, the scaling factor needs per-model tuning; a testable extension is to select s by the longest RoPE period in the model's key channels rather than by grid search.
  • The same 'start high, shrink when full' schedule could be applied to weight or activation quantization in long-context serving, where cumulative error is less studied.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes PM-KVQ, a post-training KV cache quantization method for long-CoT LLMs. It introduces three techniques: (i) progressive quantization, which initially stores KV caches in FP16 and gradually shrinks them through 8/4/2-bit using an 'Equivalent Right Shift' formula; (ii) block-wise memory allocation, cast as an integer program with sensitivity estimates; and (iii) calibration with positional interpolation on short sequences to approximate long-context key distributions. Experiments on 7B-70B models (DeepSeek-R1-Distill series and QwQ-32B) across AIME-2024/2025, CMIMC-2024, and LiveCodeBench show accuracy improvements over KIVI, MiKV, and RotateKV under equal memory budgets. The abstract also claims 2.73-5.18x throughput over 16-bit baselines.

Significance. If validated, PM-KVQ addresses a key serving bottleneck for long-CoT models, and the empirical matrix is a strength: five model sizes, multiple benchmarks, three baselines, plus ablations for the shrinking strategies and calibration length. The release of code is a plus. However, the advertised throughput gain currently has no experimental support, and a central algebraic equivalence is stated without proof, so the paper's practical and theoretical claims are not yet fully established.

major comments (3)
  1. [Abstract, Section 4.1.3] The abstract's final sentence claims 2.73-5.18x throughput over 16-bit LLMs, but the paper contains no end-to-end throughput or latency experiments. Section 4.1.3 states that all results are obtained with fake quantization, and Tables 2-4 report only accuracy metrics. The claimed speedup cannot be inferred from the 8x nominal memory reduction of 2-bit over 16-bit KV cache because progressive quantization adds requantization/bit-shift overhead, the integer program is solved at runtime setup, and weight/activation memory is not accounted for. Please either remove the throughput claim or add real kernel-level throughput measurements on the target GPUs.
  2. [Section 3.1, Eq. (3)] The 'Equivalent Right Shift' formula is asserted to be exactly equivalent to dequantizing the 2b-bit cache and re-quantizing to b-bit, but no derivation is provided. The formula is a dyadic approximation of round(X_{2b}/(2^b+1)); although it appears to match round for the integer ranges used in this paper (denominators 5, 17, and 257 are odd, so exact half-integer boundaries are avoided), the equivalence should be proven or at least stated with conditions. Because the progressive quantization benefit and the overhead argument rely on this mapping, a rigorous proof (or a test over the full integer range for b=8,4,2) is needed.
  3. [Table 2 / Section 4.2] Several headline improvements are within one standard deviation of the baseline (e.g., Qwen-7B AIME-2024: KIVI 32.08±5.25 vs. PM-KVQ at BS=40 40.00±5.40; LLaMA-8B AIME-2024 BS=6: 41.25±6.65 vs. 47.71±6.84). The paper reports means and standard deviations but no statistical tests or confidence intervals. Given that 'up to 8% improvement' is a central claim, please clarify how many independent runs/seeds the standard deviations reflect and whether the differences are stable across seeds.
minor comments (6)
  1. [Section 3.1, sentence after Eq. (3)] The sentence 'X_b and X_{2b} represent the b-bit ant 2b-bit tensor respectively' contains a typo: 'ant' should be 'and'.
  2. [Table 4] The benchmark name 'AIME-2024-I' is not defined in Section 4.1.1; please clarify which subset of AIME-2024 it refers to.
  3. [Table 2 header] The column header 'V oting' should be 'Voting'.
  4. [Section 3.1] The phrase 'we formulize the bit-width shrinking strategy' should read 'we formulate the bit-width shrinking strategy'.
  5. [Conclusion vs. Abstract] The conclusion restates the accuracy claim but omits the throughput claim from the abstract; please make the summary of contributions consistent.
  6. [Section 4.2, 70B evaluation] For the 70B model, only AIME-2024 is reported; adding a second benchmark would strengthen the claim at that scale.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: all fitted or selected quantities are calibrated on proxies or on a validation subset, and the central claims are judged against external benchmarks and baselines.

full rationale

Walking the derivation chain, no load-bearing step is defined by the reported evaluation. The three components are: progressive quantization (Sec. 3.1) with the 'Equivalent Right Shift' formula (Eq. 3); block-wise allocation via first-order sensitivity (Eqs. 4-8) solved as an Integer Program; and calibration with positional interpolation (Sec. 3.3). The only tuned scalar, alpha in Eq. (10), is grid-searched to minimize self-attention reconstruction loss (Sec. 4.1.1), not to optimize pass@1, and the calibration data (RedPajama arXiv) are disjoint from the AIME/CMIMC/LiveCodeBench test problems. The block-wise allocation optimizes a sensitivity proxy rather than the benchmark objective, and the resulting allocations are validated against external baselines KIVI, RotateKV, and MiKV. Self-citations [11,27] are cited only as inspiration for the sensitivity/IP formulation and are not load-bearing evidence for the long-CoT gains. The selection of the 'Equivalent Right Shift' among three strategies in Table 3 uses an AIME-2024 subset; this is a model-selection and overfitting risk, not circularity, because the formula's outputs are not constructed from that benchmark and Table 2 includes additional independent benchmarks (AIME-2025, CMIMC-2024, LiveCodeBench). Eq. (3) is asserted without proof and is a correctness risk, but an unproved approximation is not an input-output identity. The abstract's 2.73-5.18x throughput claim is unsupported: Sec. 4.1.3 states 'All of the performance results are conducted with fake quantization,' and no wall-clock experiment is reported; however, an unmeasured claim is an evidence gap, not a reduction of the prediction to its inputs. Section C's limitation that MLA and system-level inference engines are not considered further confirms the evaluation is narrow, but not circular. Overall, no prediction in the paper is equivalent by construction to a fitted input or to a self-citation, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method relies on two tuned hyperparameters (alpha and s) and several domain assumptions about the reliability of sensitivity estimates, the approximation quality of bit-shift shrinking, and the transfer of short-context calibration statistics to long-context inference.

free parameters (2)
  • alpha (outlier transfer strength) = not reported, grid-searched over [0,1] with 20 steps
    Used in Eq. (10) to calibrate the channel-wise reparameterization factor. Chosen on the RedPajama arXiv calibration set to minimize attention reconstruction loss, so it is fitted per model.
  • position scaling factor s = 4
    Used in Eq. (12) for positional interpolation during calibration. Set to 4 based on ablation (Table 4); s=16 degrades performance, showing the choice is data-dependent.
assumptions (4)
  • domain assumption First-order Taylor approximation of the loss with respect to KV cache quantization errors (Eq. 4) is accurate enough to rank block sensitivity.
    The block-wise allocation relies on this approximation; no second-order terms are evaluated.
  • domain assumption RoPE channel frequencies are as described, and positional interpolation with s=4 on 2K-token calibration data approximates the distribution of sequences up to 32K tokens.
    The paper gives the 54,410-token period example for the lowest frequency but does not verify coverage of all channels at 8,192 effective length.
  • ad hoc to paper The 'Equivalent Right Shift' formula (Eq. 3) exactly implements dequantize-then-quantize for bit-width transitions.
    The formula is presented as an exact equivalence but is only a dyadic approximation; for b=2 it matches round(x/5) on the tested integer range but no proof is given.
  • standard math The memory budget M is a fixed hardware constraint, and the integer program is solvable in negligible latency with CVXPY.
    Integer programming solver existence is standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs." pith.science (2026). https://pith.science/paper/T7YOGEZX

@misc{pith2026250518610,
  author       = {Pith},
  title        = {Pith review of: PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T7YOGEZX}},
  note         = {Machine review of arXiv:2505.18610}
}
read the original abstract

Recently, significant progress has been made in developing reasoning-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) techniques. However, this long-CoT reasoning process imposes substantial memory overhead due to the large Key-Value (KV) Cache memory overhead. Post-training KV Cache quantization has emerged as a promising compression technique and has been extensively studied in short-context scenarios. However, directly applying existing methods to long-CoT LLMs causes significant performance degradation due to the following two reasons: (1) Large cumulative error: Existing methods fail to adequately leverage available memory, and they directly quantize the KV Cache during each decoding step, leading to large cumulative quantization error. (2) Short-context calibration: Due to Rotary Positional Embedding (RoPE), the use of short-context data during calibration fails to account for the distribution of less frequent channels in the Key Cache, resulting in performance loss. We propose Progressive Mixed-Precision KV Cache Quantization (PM-KVQ) for long-CoT LLMs to address the above issues in two folds: (1) To reduce cumulative error, we design a progressive quantization strategy to gradually lower the bit-width of KV Cache in each block. Then, we propose block-wise memory allocation to assign a higher bit-width to more sensitive transformer blocks. (2) To increase the calibration length without additional overhead, we propose a new calibration strategy with positional interpolation that leverages short calibration data with positional interpolation to approximate the data distribution of long-context data. Extensive experiments on 7B-70B long-CoT LLMs show that PM-KVQ improves reasoning benchmark performance by up to 8% over SOTA baselines under the same memory budget and achieves 2.73-5.18x throughput over the original 16-bit LLMs.

Figures

Figures reproduced from arXiv: 2505.18610 by the authors.

Figure 1
Figure 1. Method Overview. (a) The Progressive quantization technique: we progressively shrink the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Different bit-width shrinking strategies when the bit-width is reduced from 4-bit to 2-bit. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Sensitivity to quantization of KV Cache in different transformer blocks. Different colors [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Sensitivity to quantization of KV Cache in different transformer blocks. Different colors [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A learned top-K distribution-restoration layer recovers near-FP16 long-context accuracy under 1-bit KV-cache quantization.

  2. PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    PAROAttention permutes tokens along frame, height, and width axes to make visual attention block-wise, enabling sparse and INT8/INT4 quantized attention with near-baseline generation quality.

Reference graph

Works this paper leans on

27 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    American invitational mathematics examination, 2025

    AIME. American invitational mathematics examination, 2025

    work page 2025

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  3. [3]

    Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

    arXiv 2023

  4. [4]

    Carnegie mellon informatics and mathematics competition, 2025

    CMIMC. Carnegie mellon informatics and mathematics competition, 2025

    work page 2025

  5. [5]

    CVXPY: A Python-embedded modeling language for convex optimization.Journal of Machine Learning Research, 17(83):1–5, 2016

    Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization.Journal of Machine Learning Research, 17(83):1–5, 2016

    2016

  6. [6]

    Skvq: Sliding-window key and value cache quantization for large language models.arXiv preprint arXiv:2405.06219, 2024

    Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skvq: Sliding-window key and value cache quantization for large language models.arXiv preprint arXiv:2405.06219, 2024

    arXiv 2024

  7. [7]

    Moa: Mixture of sparse attention for automatic large language model compression.arXiv preprint arXiv:2406.14909, 2024

    Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression.arXiv preprint arXiv:2406.14909, 2024

    arXiv 2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 27 references
  1. [9]

    More for keys, less for values: Adaptive kv cache quantization.arXiv preprint arXiv:2502.15075, 2025

    Mohsen Hariri, Lam Nguyen, Sixu Chen, Shaochen Zhong, Qifan Wang, Xia Hu, Xiaotian Han, and Vipin Chaudhary. More for keys, less for values: Adaptive kv cache quantization.arXiv preprint arXiv:2502.15075, 2025

    2025 arXiv

  2. [10]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  3. [11]

    Llm-mq: Mixed-precision quantization for efficient llm deployment

    Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. Llm-mq: Mixed-precision quantization for efficient llm deployment. InNeurIPS 2023 Efficient Natural Language and Speech Processing Workshop, pages 1–5, 2023

    2023

  4. [12]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024

    2024

  5. [13]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving.arXiv preprint arXiv:2405.04532, 2024

    Yujun Lin*, Haotian Tang*, Shang Yang*, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving.arXiv preprint arXiv:2405.04532, 2024

    2024 arXiv

  6. [14]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024. 10

    2024 arXiv

  7. [15]

    Intactkv: Improving large language model quantization by keeping pivot tokens intact

    Ruikang Liu, Haoli Bai, LIN Haokun, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. InThe 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024

    2024

  8. [16]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024

    2024 arXiv

  9. [17]

    Introducing openai o1, September 2024

    OpenAI. Introducing openai o1, September 2024

    2024

  10. [18]

    Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Noam Shazeer. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  11. [19]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  12. [20]

    Rotatekv: Accurate and robust 2-bit kv cache quantization for llms via outlier-aware adaptive rotations.arXiv preprint arXiv:2501.16383, 2025

    Zunhai Su, Zhe Chen, Wang Shen, Hanyu Wei, Linge Li, Huangqi Yu, and Kehong Yuan. Rotatekv: Accurate and robust 2-bit kv cache quantization for llms via outlier-aware adaptive rotations.arXiv preprint arXiv:2501.16383, 2025

    2025 arXiv

  13. [21]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  14. [22]

    Maurice Weber, Daniel Y . Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. R...

    2024

  15. [23]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  16. [24]

    No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization.arXiv preprint arXiv:2402.18096, 2024

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization.arXiv preprint arXiv:2402.18096, 2024

    2024 arXiv

  17. [25]

    Wkvquant: Quantizing weight and key/value cache for large language models gains more.arXiv preprint arXiv:2402.12065, 2024

    Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more.arXiv preprint arXiv:2402.12065, 2024

    2024 arXiv

  18. [26]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing System...

    2023

  19. [27]

    Mixdq: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization

    Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. InEuropean Conference on Computer Vision, pa...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.