{"id":"9a0bd951-1136-4bfb-821f-d8a8ae3538c0","arxiv_id":"2505.18610","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PM-KVQ introduces progressive bit-width shrinking, per-block memory allocation, and positional-interpolation calibration to make 2-bit KV cache quantization nearly lossless on long-CoT LLMs.","lead":"This paper presents PM-KVQ, a method that compresses the key-value cache of long chain-of-thought language models by progressively lowering the bit-width during generation, from 16-bit down to 2-bit, while allocating extra precision to the most sensitive transformer blocks. It reports accuracy gains of up to 8 percentage points over leading KV cache quantization baselines on reasoning benchmarks at the same memory budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract claims 2.73-5.18x throughput over 16-bit LLMs, but no throughput or latency measurement appears anywhere in the paper; all reported results use fake quantization, so this headline practical claim is unsupported.","rationale":"I read the reader's verdict and the full manuscript. The central accuracy claim is credible: Table 2 shows consistent gains over KIVI across 7B-70B models, the comparisons use the same batch-size/memory budgets, and the released code provides reproducibility. The reader's weakest assumption about Eq. (3) is less severe than it appears: the formula is a fixed-point implementation of round(X2b/(2^b+1)); with Z_b=Z_2b and S_b=(2^b+1)S_2b the zero point cancels, and the coefficient (2^2b-2^b+1)/2^3b approximates 1/(2^b+1) with the +2^(b-1) term supplying rounding. Over the relevant unsigned integer ranges, including 4-bit to 2-bit, the mapping matches round(X/5) and the approximation error is below half an LSB, so Eq. (3) does not threaten the accuracy results. The genuinely load-bearing concern is the abstract's throughput claim, which the reader also flags in the rationale but not as the weakest assumption. Since the body contains no timing experiments and explicitly uses fake quantization, the 2.73-5.18x throughput figure is unsupported. This does not invalidate the accuracy findings, but it requires a correction: either remove the claim or add a real end-to-end benchmark. The reader's CONDITIONAL verdict therefore stands, with the condition being substantiation or retraction of the throughput claim.","tokens_in":13599,"tokens_out":14149,"duration_ms":116867,"concrete_test":"Run the released PM-KVQ implementation in real end-to-end batched decoding on the same A100/4090 configurations as Table 2, measuring tokens/s for the original FP16 model and for PM-KVQ at each batch size and bit-width. If the measured speedup falls outside the 2.73-5.18x range, or if no inference kernel is provided and the speedup cannot be measured, the abstract claim must be removed or revised to a statement supported by actual measurements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's final sentence states that PM-KVQ 'achieves 2.73-5.18x throughput over the original 16-bit LLMs,' but the body contains no end-to-end timing experiment. Section 4.1.3 says 'All of the performance results are conducted with fake quantization,' and the tables report only accuracy metrics. The progressive scheme itself requires periodic requantization of the cached tensor (16->8->4->2) plus bit-shift operations, and the block-wise allocation requires solving an Integer Program; whether these operations yield a net speedup depends on kernel efficiency, memory bandwidth, requantization overhead, and batch-size effects, none of which are measured. The 2.73-5.18x range also does not follow from the 8x nominal memory reduction of 2-bit versus 16-bit KV cache, because the requantization cost is nonzero and the memory footprint of the model weights and activations is not accounted for. This is a quantitative headline claim with zero supporting evidence, so the paper's advertised practical benefit is currently unsubstantiated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PM-KVQ, a post-training KV cache quantization method for long-CoT LLMs. It introduces three techniques: (i) progressive quantization, which initially stores KV caches in FP16 and gradually shrinks them through 8/4/2-bit using an 'Equivalent Right Shift' formula; (ii) block-wise memory allocation, cast as an integer program with sensitivity estimates; and (iii) calibration with positional interpolation on short sequences to approximate long-context key distributions. Experiments on 7B-70B models (DeepSeek-R1-Distill series and QwQ-32B) across AIME-2024/2025, CMIMC-2024, and LiveCodeBench show accuracy improvements over KIVI, MiKV, and RotateKV under equal memory budgets. The abstract also claims 2.73-5.18x throughput over 16-bit baselines.","tokens_in":13809,"tokens_out":10824,"duration_ms":84104,"significance":"If validated, PM-KVQ addresses a key serving bottleneck for long-CoT models, and the empirical matrix is a strength: five model sizes, multiple benchmarks, three baselines, plus ablations for the shrinking strategies and calibration length. The release of code is a plus. However, the advertised throughput gain currently has no experimental support, and a central algebraic equivalence is stated without proof, so the paper's practical and theoretical claims are not yet fully established.","major_comments":[{"comment":"The abstract's final sentence claims 2.73-5.18x throughput over 16-bit LLMs, but the paper contains no end-to-end throughput or latency experiments. Section 4.1.3 states that all results are obtained with fake quantization, and Tables 2-4 report only accuracy metrics. The claimed speedup cannot be inferred from the 8x nominal memory reduction of 2-bit over 16-bit KV cache because progressive quantization adds requantization/bit-shift overhead, the integer program is solved at runtime setup, and weight/activation memory is not accounted for. Please either remove the throughput claim or add real kernel-level throughput measurements on the target GPUs.","section":"Abstract, Section 4.1.3"},{"comment":"The 'Equivalent Right Shift' formula is asserted to be exactly equivalent to dequantizing the 2b-bit cache and re-quantizing to b-bit, but no derivation is provided. The formula is a dyadic approximation of round(X_{2b}/(2^b+1)); although it appears to match round for the integer ranges used in this paper (denominators 5, 17, and 257 are odd, so exact half-integer boundaries are avoided), the equivalence should be proven or at least stated with conditions. Because the progressive quantization benefit and the overhead argument rely on this mapping, a rigorous proof (or a test over the full integer range for b=8,4,2) is needed.","section":"Section 3.1, Eq. (3)"},{"comment":"Several headline improvements are within one standard deviation of the baseline (e.g., Qwen-7B AIME-2024: KIVI 32.08±5.25 vs. PM-KVQ at BS=40 40.00±5.40; LLaMA-8B AIME-2024 BS=6: 41.25±6.65 vs. 47.71±6.84). The paper reports means and standard deviations but no statistical tests or confidence intervals. Given that 'up to 8% improvement' is a central claim, please clarify how many independent runs/seeds the standard deviations reflect and whether the differences are stable across seeds.","section":"Table 2 / Section 4.2"}],"minor_comments":[{"comment":"The sentence 'X_b and X_{2b} represent the b-bit ant 2b-bit tensor respectively' contains a typo: 'ant' should be 'and'.","section":"Section 3.1, sentence after Eq. (3)"},{"comment":"The benchmark name 'AIME-2024-I' is not defined in Section 4.1.1; please clarify which subset of AIME-2024 it refers to.","section":"Table 4"},{"comment":"The column header 'V oting' should be 'Voting'.","section":"Table 2 header"},{"comment":"The phrase 'we formulize the bit-width shrinking strategy' should read 'we formulate the bit-width shrinking strategy'.","section":"Section 3.1"},{"comment":"The conclusion restates the accuracy claim but omits the throughput claim from the abstract; please make the summary of contributions consistent.","section":"Conclusion vs. Abstract"},{"comment":"For the 70B model, only AIME-2024 is reported; adding a second benchmark would strengthen the claim at that scale.","section":"Section 4.2, 70B evaluation"}],"recommendation":"major_revision","confidential_remarks":"The throughput claim in the abstract is the main risk for acceptance; if the authors cannot provide measurements, they should remove it. The Eq. (3) proof is straightforward to add. The statistical significance issue may be addressable by reporting per-response bootstrap confidence intervals or paired tests. The paper otherwise presents a broad and useful empirical study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth your time: instead of quantizing the KV cache to a fixed low bit-width from the first token, PM-KVQ starts at 16-bit and progressively shrinks to the target width as memory fills, then allocates different final widths per transformer block via an integer program. The progressive schedule and the positional-interpolation calibration trick are both new in the KV-cache quantization literature as far as I can tell, and they are sensible responses to the long-CoT failure modes the paper identifies. The block-wise allocation is borrowed from weight quantization, but the combination is cohesive.\n\nThe experimental work is solid. They evaluate on 7B-70B models, five benchmarks, three baselines, and the gains are consistent: at 2-bit, PM-KVQ often recovers most of the 16-bit accuracy where KIVI degrades sharply. The ablations on the shrinking strategy and the positional interpolation are informative. Code is released, and the fake-quantization protocol is stated plainly. On the accuracy claims, I trust the results.\n\nThe soft spots are real but localized. Most importantly, the abstract claims 2.73-5.18x throughput over 16-bit, but there is no end-to-end timing anywhere in the paper. All results use fake quantization. The progressive scheme adds requantization passes and the allocation step requires solving an IP; whether that nets a speedup depends on kernel efficiency and batch size, and none of that is measured. That claim should be removed or substantiated before publication. Second, the \"Equivalent Right Shift\" formula in Eq. (3) is asserted to be equivalent to dequantize-then-requantize, but no proof is given. It looks like a reasonable dyadic approximation and the ablation shows it beats the alternatives, but the exactness claim is stronger than the evidence. Third, alpha is grid-searched per model and s is fixed at 4; that is standard practice in this area, so I would not call it a flaw, only a limit on generality. The limitations section honestly notes that MLA is not covered.\n\nWho is this for? Anyone working on KV-cache compression or long-context inference. It deserves a serious referee: the central method is new, the accuracy evidence is substantial, and the fixable overclaim in the abstract should not sink the whole paper. I would engage with it, and I would ask for the throughput claim to be either deleted or supported with actual latency measurements.","headline":"Progressive KV-cache quantization is a genuinely new idea with credible accuracy gains, but the abstract's throughput claim outruns the evidence.","tokens_in":14369,"tokens_out":888,"would_cite":true,"duration_ms":9625,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PM-KVQ claims 2-bit KV cache quantization can match 16-bit accuracy within a few points on long-CoT LLMs by storing 16-bit first, then shrinking precision only when memory fills.","keywords":["KV cache quantization","progressive quantization","mixed-precision quantization","long chain-of-thought","positional interpolation","RoPE calibration","LLM inference","memory-efficient serving"],"falsifier":"Run PM-KVQ side by side with a variant that literally dequantizes each block's KV cache to FP16 and re-quantizes at the lower bit-width at every shrinking step; if Eq. (3) is exact the two runs must produce identical integer caches, and any numerical divergence, or an AIME pass@1 gap between the two, would show the equivalence claim fails.","tokens_in":13387,"feed_emoji":"🧠","tokens_out":5149,"duration_ms":41208,"temperature":0.7,"pith_summary":"The paper proposes PM-KVQ, a post-training quantization scheme for the key-value cache of long chain-of-thought LLMs. It argues that directly quantizing the KV cache at every decoding step accumulates errors over tens of thousands of generated tokens, and that calibration on short sequences misreads RoPE-affected key channels. PM-KVQ instead starts each block at 16 bits and shrinks the cache to its target bit-width only when the memory budget fills, allocates different target bit-widths to different transformer blocks according to measured sensitivity, and calibrates using short sequences with positional interpolation to imitate long contexts. On 7B-70B reasoning models, the paper reports matching 16-bit accuracy within a few points at 2-bit KV cache and gains up to 8% over baselines under equal memory.","feed_headline":"2-bit KV cache scores within a few points of 16-bit accuracy","feed_subtitle":"By shrinking precision only when memory fills, PM-KVQ cuts KV cache memory while preserving long-reasoning benchmarks.","key_machinery":"The load-bearing object is the progressive quantization schedule with the 'Equivalent Right Shift' identity, $X_b = ((2^{2b}-2^b+1)(X_{2b}+2^{b-1})) \\gg 3b$, which is claimed to equal dequantizing a $2b$-bit cache to FP16 and requantizing at $b$-bit while keeping the zero point and scaling the step by $(2^b+1)$. At each power-of-two transition (16 to 8, 8 to 4, 4 to 2) this operation lets the method shrink the existing cache in place. Two secondary mechanisms carry the rest: a first-order Taylor sensitivity score per block feeding an integer-programming solver that picks each block's final bit-width under a memory cap, and RoPE positional interpolation during calibration that stretches short sequences to approximate long-context key-channel distributions.","core_discovery":"The central claim is that the two failure modes of short-context KV cache quantization on long-CoT LLMs, cumulative error and short-horizon calibration, can both be removed without changing the memory budget. Progressive quantization begins every block at 16-bit and applies an integer 'Equivalent Right Shift' (Eq. 3) to shrink 16 to 8 to 4 to 2 only when the block's memory allocation is exhausted, so early tokens keep high precision throughout. Block-wise memory allocation solves an integer program that minimizes a first-order Taylor sensitivity term over bit-width choices under a memory constraint, assigning 4-bit to sensitive blocks while others run at 2-bit. Calibration with positional interpolation multiplies the RoPE position index by a factor s, so 2,048-token calibration sequences stand in for 8,192-token contexts. The experiments on DeepSeek-R1-Distill and QwQ models claim that at 2-bit the method stays within a few points of the 16-bit model and beats KIVI by up to 8%.","pith_inferences":["A formal proof or counterexample for Eq. (3)'s exactness would settle whether the method can be extended to more aggressive schedules, such as non-power-of-two scales or 3-bit targets.","The sensitivity-based allocation measures only first-order loss change; second-order or attention-map-based sensitivity might change which blocks receive high bit-width, especially for models with MLA-style attention.","Because the paper reports that s=16 degrades performance, the scaling factor needs per-model tuning; a testable extension is to select s by the longest RoPE period in the model's key channels rather than by grid search.","The same 'start high, shrink when full' schedule could be applied to weight or activation quantization in long-context serving, where cumulative error is less studied."],"forward_implications":["A 2-bit KV cache becomes usable for long-CoT reasoning models, with reported pass@1 within a few points of the uncompressed 16-bit model on AIME and LiveCodeBench.","Under a fixed memory budget the same GPU can hold a larger batch, giving reported throughput improvements of 2.73-5.18x over 16-bit inference.","Calibration can stay on short sequences: with s=4, 2,048-token calibration data approximates 8,192-token contexts, avoiding the high cost of long-sequence calibration.","Non-uniform block-wise bit-widths under a memory cap beat any uniform bit-width, because sensitive blocks (often deep blocks, and the first block in Qwen-based models) get 4-bit while others run at 2-bit."],"supporting_citations":[{"why":"KIVI is the main baseline and supplies the per-channel key, per-token value quantization plus the INT16 pivot tokens that PM-KVQ adopts.","marker":"[16]"},{"why":"QServe provides the channel-wise outlier reparameterization from Key to Query that PM-KVQ uses in calibration.","marker":"[13]"},{"why":"Positional interpolation is the mechanism PM-KVQ uses to embed long-context positional information into short calibration sequences.","marker":"[3]"},{"why":"MiKV is an importance-aware mixed-precision baseline that PM-KVQ compares against and outperforms in the 2-bit regime.","marker":"[24]"},{"why":"RotateKV is a rotation-based outlier equalization baseline used for comparison in the 2-bit and 4-bit experiments.","marker":"[20]"},{"why":"CVXPY is the solver used to solve the integer programming problem for block-wise bit-width allocation.","marker":"[5]"},{"why":"RoFormer defines RoPE and its per-channel rotary frequencies, which is the basis for the short-context calibration failure and the positional-interpolation fix.","marker":"[19]"},{"why":"DeepSeek-R1 supplies the distilled long-CoT model families used in the evaluation.","marker":"[8]"}],"fun_headline_variants":["Progressive KV cache quantization cuts memory for long-CoT LLMs","Mixed-precision KV cache: near-16-bit accuracy at 2-bit cost","PM-KVQ: Smarter KV cache for long reasoning models","Long-CoT LLMs get memory-efficient KV cache with PM-KVQ","Progressive precision KV cache: up to 5x throughput, same accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's in-place shrinking rests on Eq. (3) being exactly equivalent to dequantizing the 2b-bit KV cache and requantizing it at b-bit, but the paper asserts this without proof; if the formula's dyadic rounding introduces a systematic bias for some attention distributions, progressive quantization could accumulate errors the experiments do not isolate.","fun_headline_variants_meta":{"raw":{"variants":["Progressive KV cache quantization cuts memory for long-CoT LLMs","Mixed-precision KV cache: near-16-bit accuracy at 2-bit cost","PM-KVQ: Smarter KV cache for long reasoning models","Long-CoT LLMs get memory-efficient KV cache with PM-KVQ","Progressive precision KV cache: up to 5x throughput, same accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1688,"prompt_tokens":1107,"completion_tokens":581,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":723,"tokens_out":581,"duration_ms":4414,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:29:18.827351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PM-KVQ side by side with a variant that literally dequantizes each block's KV cache to FP16 and re-quantizes at the lower bit-width at every shrinking step; if Eq. (3) is exact the two runs must produce identical integer caches, and any numerical divergence, or an AIME pass@1 gap between the two, would show the equivalence claim fails.","supporting_citations":[],"review_version":1}