{"id":"d1f39d80-6fc4-4a97-805e-97e3b3d20973","arxiv_id":"2411.18077","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MiniKV combines 2-bit KV quantization with heavy-hitter and recent-window token selection plus pyramid layer budgets, achieving 86% KV cache compression with about 98.5% of full-model LongBench accuracy.","lead":"MiniKV shrinks the KV cache that stores a language model's past tokens to roughly one seventh of its usual size, using 2-bit numbers and keeping only the half of tokens attention scores call important. This lets long-context models run faster and fit more work on a single GPU while keeping about 98.5% of full-model accuracy on LongBench.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Persistent heavy-hitter selection is validated on one sample; need multi-sample evidence that 60–80% persistence holds across LongBench tasks and models before the 2-bit layout assumption is load-bearing.","rationale":"The reader correctly identified the persistent heavy-hitter assumption as the weakest link. I agree that the evidence is a single Lcc sample, and I want to make the dependence of the 2-bit quantization scheme on this assumption explicit: the sub-channel quantization groups are frozen at prefill, so any token eviction or replacement during decoding would break the group encoding. The paper does cite prior work that heavy hitters persist at larger budgets, but the actual measurement is anecdotal, and the paper itself notes that the persistence claim is regime-dependent. This is an internal-evidence weakness, not an outside-consensus disagreement. A concrete multi-sample persistence study across datasets and models would settle it. The system results are less load-bearing because even if throughput were lower, the algorithmic claim would still stand; conversely, if persistence fails, the whole 2-bit+selection composition collapses. The reader's other concerns (no code, tuned hyperparameters, figure-only speedups) also support CONDITIONAL, and I do not see an internal inconsistency that would force REJECT. I would keep the verdict CONDITIONAL rather than UNVERDICTED because the central accuracy numbers are reported in tables and are internally coherent; the missing piece is a specific validation of the persistence mechanism.","tokens_in":18945,"tokens_out":1389,"duration_ms":12509,"concrete_test":"Run the Appendix E persistence measurement on the same H2O setting (k=150) for at least 20 random prompts per dataset across the 13 LongBench datasets and all three models, and additionally measure MiniKV's own selected HH fraction under MiniKV's 25%+25% budget. If median persistence falls below 60% on any task category, or if the MiniKV-specific persistence is materially lower than the H2O-derived estimate, the fixed-selection 2-bit layout must be revisited.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The core claim is that MiniKV can fix the KV token subset at the end of prefill and quantize only those tokens to 2-bit without updating selection, because 60–80% of prefill-selected heavy hitters persist through generation. This assumption underlies the fixed 2-bit sub-channel quantization layout; if token importance shifts substantially during decoding, the retained set is wrong and accuracy recovery collapses. The supporting evidence in Appendix E is a single sample prompt from Lcc (one dataset, unspecified model, k=150). H2O's own heavy-hitter persistence is known to degrade at larger cache budgets and across tasks, and the paper's Section 3.1.1 explicitly acknowledges that prior work disputes persistence at tiny budgets but only asserts a large-budget regime. No evidence is provided across LongBench's six task categories, across Llama2-7B/13B and Mistral-7B, or with MiniKV's own 25% HH + 25% RW selection. The persistence claim is therefore not merely a missing detail; it is the load-bearing condition that permits the entire algorithm to avoid re-selecting and re-quantizing during decoding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MiniKV combines 2-bit KV-cache quantization with adaptive token eviction (heavy hitters plus a recent window) and a pyramid-style layer-wise budget allocation, together with purpose-built Triton kernels that make the selective attention compatible with FlashAttention-style linear-memory execution. The paper evaluates accuracy on LongBench (with prompts truncated to 4096 tokens), InfiniteBench, and GSM8K, and reports latency, throughput, and peak-memory measurements on A100 and GH200 GPUs. The headline claims are more than 80% (86% in one configuration) KV-cache compression, roughly 98.5% accuracy recovery for LLaMA2-7B-chat on LongBench, Pareto-optimality across all six LongBench task categories, and a 48% throughput improvement over the strongest baseline.","tokens_in":19193,"tokens_out":14913,"duration_ms":129625,"significance":"If the claims hold, MiniKV is a genuinely useful algorithm/system co-design contribution: it is one of the few works to combine ultra-low-bit KV quantization with eviction in a FlashAttention-compatible kernel, and the matched-cache-size accuracy table plus the Pareto-curve analysis provide a reasonable evaluation template. The two-pass selective-attention kernel with linear memory is a concrete systems contribution, and the manuscript is candid about limitations, including the failure of a SnapKV+KIVI combination and the prefill-phase slowdown. The significance is contingent, however, on fixing the load-bearing issues listed below: the headline numbers are configuration- and model-specific, the matched-size comparison is internally inconsistent for one baseline, and the persistent-heavy-hitter assumption that the fixed 2-bit layout depends on is supported by only a single sample.","major_comments":[{"comment":"The persistent-context-selection assumption is load-bearing: it justifies fixing the 2-bit sub-channel quantization layout and never re-selecting heavy hitters during decoding. The only supporting evidence is one sample from the Lcc dataset (Figure 11) with k=150, with no model or layer specification and no quantitative persistence curve; the '60-80%' claim in §3.1.1 cannot be read off that figure as presented. Moreover, the measurement uses H2O's retained set, not MiniKV's own (25% HH + 25% RW) selection. Please provide per-head persistence curves across the three evaluated models, the six LongBench task categories, and the actual MiniKV budgets, and show that the accuracy loss from a frozen selection is bounded.","section":"§3.1.1 / Appendix E"},{"comment":"The headline accuracy recovery is measured on inputs truncated to 4096 tokens: 'The maximum prompt length is 4096 for all models with the first and last 2048 tokens taken.' Table 1 therefore reports 4K-context behavior, not LongBench's long-context regime, and the 86% compression figure in Appendix I is computed for lprompt=4096 and lgen=512. The abstract's 'long context tasks' claim and the 98.5% recovery figure should be explicitly scoped to this truncated setting; results on untruncated LongBench prompts (or another long-context suite with matched cache-size controls) are needed to support the broader claim.","section":"§4.1 / Table 1"},{"comment":"The main accuracy comparison is not at matched cache sizes under the paper's own formulas. With lprompt=4096 and lgen=512, MiniKV uses (H·d)·(0.5·4096+512) ≈ 0.33 GB. H2O(15%) is about 0.34 GB only if generated tokens are excluded from the budget, but the Appendix I formula for SnapKV includes them, giving SnapKV(15%) ≈ 2·H·d·2·(0.15·4096+512) ≈ 0.55 GB. Since Table 1 is introduced as a comparison 'under a similar KV cache size', the SnapKV row is not a matched-size comparison. Please recompute the comparison using a SnapKV budget that matches MiniKV's actual cache size, or plot accuracy versus cache size for each method across several budgets.","section":"§4.2 / Appendix I"},{"comment":"The abstract's claim of 'recovering over 98.5% of accuracy' is supported only by the LLaMA2-7B-chat MiniKV-Pyramid row (34.65/35.19 = 98.5%). The other evaluated models recover less: LLaMA2-13B-chat achieves 32.37/33.32 = 97.1% and Mistral-7B-Instruct achieves 39.97/41.20 = 97.0%. Please qualify the headline number with the model-specific range and avoid presenting a single best configuration as the general result.","section":"Abstract / Table 1"},{"comment":"The default configuration (50% budget, 25% HH + 25% RW, group size 16, nr=128, pyramid depth d=7) is fixed using the design observations in Figures 2 and 3, but no sensitivity analysis is reported for d, the HH/RW split, group size, or nr. The GSM8K result (§4.5) shows that on a reasoning task the required adaptive budget is about 90%, which illustrates that the 50%-budget configuration is not universally applicable. To support the claim that the 86%-compression accuracy point is general rather than a single tuned configuration, please include ablations around these hyperparameters and state which choices were made on a validation split versus the reported test sets.","section":"§4.1 / Appendix H / §4.5"}],"minor_comments":[{"comment":"The table refers to 'rows marked in brown', but this color cue is not visible in standard arXiv rendering; please use a symbol, boldface, or an explicit marker column to indicate the matched-size rows.","section":"Table 1"},{"comment":"The first-pass paragraph contains an incomplete sentence: 'we follow FlashAttention's online softmax reduction to compute the weighted V block write it back.' Please split and rephrase this sentence.","section":"§3.2"},{"comment":"The column headers in Table 2 are concatenated without separators (e.g., 'LongBook ChoiceLongDialogue QACode Debug'), making the table hard to read; please use clearly separated column names.","section":"Table 2"},{"comment":"The margins between MiniKV and the full model are often within one LongBench point, and no variance or number of runs is reported; please state whether these are single greedy-decoding runs and, ideally, report standard deviations across seeds.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The revision is feasible: the kernel contribution and the experimental infrastructure are real, and the main overclaims are localizable. The most important request is the multi-sample persistence evidence, because the fixed-layout kernel design depends on it; without that evidence, the accuracy claims would need to be substantially downgraded. The matched-size inconsistency for SnapKV and the 4K-truncated LongBench setup are also straightforward to address by rerunning or by carefully rescoping the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, MiniKV is a real engineering result: it is the first system I know of that makes 2-bit KV quantization work together with adaptive eviction rather than just bolting them together, and it ships kernels that keep the combination FlashAttention-compatible. The LongBench table is the heart of the paper and it is fairly convincing: at matched cache sizes MiniKV-Pyramid gets 34.65 vs the full model's 35.19, which is the claimed 98.5% recovery, and it beats H2O, SnapKV, and Q-Hitter at comparable budgets. The 86% compression figure is derived directly from the cache-size analysis, not inflated.\n\nWhat is genuinely new is the composition: KIVI alone does 2-bit without selection; H2O and SnapKV select without 2-bit; Q-Hitter does 4-bit plus H2O. The specific combination of persistent heavy hitters, recent window, sub-channel key quantization, and pyramid layer budget is the paper's contribution, plus the two-pass Triton kernel that returns cumulative attention scores with linear memory. The kernel is a legit system piece; the microbenchmarks show quadratic standard attention going OOM while MiniKV stays linear.\n\nThe soft spots are real but not fatal. The load-bearing assumption is that prefill-selected heavy hitters persist throughout generation, because the 2-bit sub-channel layout is fixed at prefill. Appendix E supports this with one sample from Lcc and a visual, not a number, and not across the six LongBench task types or the three models. That is a genuine gap. The paper itself acknowledges prior work showing non-persistence at tiny budgets and only claims the large-budget regime, but it does not give the multi-sample evidence needed to make that regime claim robust. Also, LongBench prompts are truncated to 4096 tokens, so \"long context\" here is moderate; the InfiniteBench results at longer lengths are thinner and show a few tasks where MiniKV loses to KIVI, such as StringKV and Math Find. Hyperparameters like pyramid depth d=7 and the HH/RW split are tuned on the same benchmark used for final reporting, which is standard for this area but limits the strength of the \"generalizes\" claim. No code is released.\n\nOn balance the central claim holds up: co-design is real, the accuracy table is legible, and the system wins are plausible. The citations are appropriate and the limitations section is honest, including the SnapKV+KIVI failure that drops from 35 to 32. This deserves a serious referee. My recommendation: accept into the review process; insist the authors release code and add a real multi-sample persistence analysis plus per-task tables for the longer benchmarks. It is a solid paper, not a breakthrough, and the review should treat it as a systems contribution that needs reproducibility work.","headline":"A solid systems paper that makes 2-bit quantization plus eviction work together, held back mainly by one-sample evidence for its load-bearing persistence assumption.","tokens_in":19723,"tokens_out":2089,"would_cite":true,"duration_ms":18863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MiniKV combines 2-bit KV quantization with token eviction to cut KV-cache memory by 86% while keeping 98.5% of full-model accuracy on LongBench.","keywords":["KV cache quantization","2-bit quantization","adaptive KV cache","heavy hitter selection","long-context LLM inference","FlashAttention compatibility","layer-wise KV allocation","Triton kernels"],"falsifier":"Track the overlap between the heavy-hitter set chosen at the end of prefill and the set that would be chosen at each later decoding step across all 13 LongBench datasets; if average overlap drops below the reported 60–80% on datasets with topic shifts, or if re-selecting tokens every 32 steps improves the average score by more than about one point, the persistence assumption fails.","tokens_in":18767,"feed_emoji":"⚡","tokens_out":6655,"duration_ms":57623,"temperature":0.7,"pith_summary":"MiniKV claims that the two main KV-cache compression strategies—ultra-low-bit quantization and token eviction—can be combined to far exceed either alone, provided the combination is co-designed with the attention kernel. On LongBench, with LLaMA2-7B-chat, it reports a KV cache of 0.33 GB versus 2.4 GB for the full model (an 86% reduction) while recovering 98.5% of full-model accuracy (34.65 vs 35.19 with its pyramid variant). The same recipe generalizes to LLaMA2-13B, Mistral-7B, and Llama-3 models on InfiniteBench, and translates into lower latency, higher throughput, and longer processable prompts on an A100 GPU. The practical stake is that long-context LLM serving, currently bottlenecked by KV-cache memory, could run dramatically more requests per GPU.","feed_headline":"MiniKV cuts KV-cache memory 86% while keeping 98.5% accuracy","feed_subtitle":"Heavy-hitter selection plus FlashAttention-friendly kernels lets long-context LLMs run faster with more memory headroom.","key_machinery":"Persistent context selection is the load-bearing mechanism: heavy-hitter tokens are chosen once from cumulative attention scores at the end of prefill and are never re-selected during generation, which makes sub-channel 2-bit key quantization well-defined even when eviction would otherwise change group membership. Around this sit a pyramid layer budget that allocates the heavy-hitter cache unequally across layers, a KIVI-style packing scheme that stores 16 INT2 scalars per INT32 word with FP16 scale and zero-point metadata, and a two-pass selective flash-attention kernel that produces both the attention output and column-wise cumulative attention scores using memory that grows linearly with sequence length rather than quadratically.","core_discovery":"The paper's central claim is that a 2-bit quantized KV cache remains accurate in long-context tasks only when quantization is paired with adaptive selection of which tokens to keep, and that this pairing is most accurate when the token budget is distributed pyramid-style across layers, with more capacity in lower layers and less in higher ones. The authors show the composition is not trivial: per-token INT2 quantization of H2O-style selected tokens collapses accuracy, and combining SnapKV-style selection with KIVI-style quantization drops LongBench scores from 35 to 32 points. MiniKV instead selects persistent heavy hitters at the end of prefill, retains a critical recent-window fraction, applies sub-channel quantization to keys and per-token quantization to values, and processes attention with a two-pass kernel that returns cumulative attention scores without materializing the attention matrix. With a 50% cache budget it reports 86% total KV compression while keeping at least 98.5% of full-model accuracy on LLaMA2-7B-chat, and it lies on the Pareto frontier of accuracy versus cache size across all six LongBench task categories.","pith_inferences":["If the persistent-heavy-hitter assumption holds across workloads, the prefill-phase selection could be reused across many queries that share a prompt prefix, turning the fixed layout into a reusable prompt-cache block rather than a per-request cost.","The paper's own failed SnapKV-plus-KIVI combination suggests a testable rule: eviction methods whose retained tokens are quantization-sensitive fail under 2-bit compression, so a robust co-design should score tokens by post-quantization error, not by attention alone.","The pyramid result implies that lower layers carry more tokens; a natural untested extension is to give lower layers higher bit-widths and upper layers lower bit-widths within a fixed memory budget.","Because the persistence evidence in Appendix E is a single sample prompt, a multi-dataset persistence audit would determine whether fixed prefill selection survives topic shifts, multi-turn dialogue, and streaming inputs."],"forward_implications":["At matched KV-cache size, MiniKV-Pyramid outperforms H2O, SnapKV, and Q-Hitter on LongBench, staying within 0.54 points of the full model on LLaMA2-7B-chat while using 86% less cache memory.","Because selection happens once at prefill and never changes, the retained KV set has a fixed layout, enabling 2-bit sub-channel quantization and fused dequantization kernels during decoding.","The two-pass kernel removes the need to materialize the quadratic attention matrix, making adaptive KV eviction compatible with FlashAttention-style memory-efficient attention and longer prompts.","On a single A100 GPU, MiniKV reports a 48% throughput improvement over its strongest baseline and supports prompts of 44K tokens, with peak memory low enough to avoid out-of-memory failures that H2O and KIVI encounter at larger batch sizes.","MiniKV's gains are concentrated in attention and KV loading, so the benefit grows with sequence length, making the method most valuable exactly in the long-context regime where KV-cache memory is the bottleneck."],"supporting_citations":[{"why":"Supplies the heavy-hitter eviction policy and the persistence hypothesis that MiniKV's persistent context selection builds on.","marker":"(Zhang et al., 2023)"},{"why":"Supplies the 2-bit asymmetric quantization, INT32 packing, and streaming-buffer decoding scheme that MiniKV adapts.","marker":"(Liu et al., 2024b)"},{"why":"FlashAttention is the memory-efficient attention implementation that MiniKV's selective attention kernel must remain compatible with.","marker":"(Dao et al., 2022)"},{"why":"Introduces the pyramid layer-wise budget allocation that MiniKV-Pyramid adopts for its best accuracy.","marker":"(Cai et al., 2024)"},{"why":"SnapKV is a baseline and also the source of a negative result: combining SnapKV selection with KIVI quantization fails.","marker":"(Li et al., 2024)"},{"why":"Q-Hitter is the strongest adaptive-plus-quantized baseline that MiniKV compares against for accuracy and system performance.","marker":"(Zhang et al., 2024b)"},{"why":"LongBench is the benchmark on which MiniKV's accuracy and compression claims are evaluated.","marker":"(Bai et al., 2023)"},{"why":"Documents channel-dimension outliers in keys that motivate sub-channel key quantization.","marker":"(Hooper et al., 2024)"}],"fun_headline_variants":["MiniKV: 86% KV cache cut, 98.5% accuracy kept","Layer-aware 2-bit KV cache: 86% smaller, 98.5% accuracy","MiniKV compresses KV cache 86% via layer-aware 2-bit selection","Pyramid KV budget: 2-bit cache, 86% smaller, 98.5% accuracy","2-bit KV cache with layer-wise budget: 86% compression, 98.5% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The selection of which tokens to keep and quantize is made once at the end of prefill and never revised, based on the claim that 60–80% of heavy hitters persist through generation, a claim the paper supports with a single sample prompt.","fun_headline_variants_meta":{"raw":{"variants":["MiniKV: 86% KV cache cut, 98.5% accuracy kept","Layer-aware 2-bit KV cache: 86% smaller, 98.5% accuracy","MiniKV compresses KV cache 86% via layer-aware 2-bit selection","Pyramid KV budget: 2-bit cache, 86% smaller, 98.5% accuracy","2-bit KV cache with layer-wise budget: 86% compression, 98.5% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001828,"raw_usage":{"total_tokens":7176,"prompt_tokens":920,"completion_tokens":6256,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":6151}},"tokens_in":536,"tokens_out":6256,"duration_ms":39653,"temperature":1.0,"reasoning_tokens":6151,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:31:44.923936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track the overlap between the heavy-hitter set chosen at the end of prefill and the set that would be chosen at each later decoding step across all 13 LongBench datasets; if average overlap drops below the reported 60–80% on datasets with topic shifts, or if re-selecting tokens every 32 steps improves the average score by more than about one point, the persistence assumption fails.","supporting_citations":[],"review_version":1}