{"id":"4faad18e-9bf2-42b9-b050-3c45b0ff7b0c","arxiv_id":"2412.05693","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Compressing the KV cache during prefilling as well as decoding enables larger batch sizes and roughly 44% higher LLM inference throughput at near-full accuracy in long-context settings.","lead":"Batch-Max compresses the KV cache during both prompt processing and token generation, allowing larger batch sizes on limited GPU memory. In long-input, short-output tasks, it reports about 44% higher throughput than a decoding-only compression upper bound while keeping accuracy close to the full model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ED is asserted, not proven, to be an upper bound; a single-pair or quantized decoding-only baseline could invalidate the headline comparison.","rationale":"The reader's weakest assumption correctly identifies the load-bearing step: the throughput comparison is only as strong as ED's status as an upper bound. I agree with that identification. The concern is not manufactured: the paper states the ED bound in a single sentence and does not test the natural extreme variants that would falsify it. Within the eviction family, |kv|=1 is the most direct challenge because it reduces the number of KV pairs present during a decode forward pass below ED's |kv|=2; the paper never justifies why 'non-empty' requires two pairs rather than one. Quantization is a second challenge if 'compression' is interpreted as the title suggests, though the paper's formal claim is scoped to eviction methods. Even if |kv|=1 turns out to be only slightly faster than ED and still far slower than BM, the paper's proof-by-assertion of the upper bound would need revision. However, the empirical direction of the paper is probably sound: BM's larger batch sizes are a real mechanism, and the reported throughput gains are large. The appropriate remedy is to add the missing baselines and explicitly scope the upper-bound claim, which is exactly what the reader's conditional verdict asks. I therefore do not recommend changing the verdict; I recommend keeping it conditional on the additional experiments.","tokens_in":7026,"tokens_out":21477,"duration_ms":228722,"concrete_test":"On the Llama-2 CNN/DM setup, implement and measure two decoding-only variants under the same memory budget and 512-generation protocol: (a) Algorithm 2 with |kv|=1, so the cache retains only the current token's KV pair, and (b) ED's |kv|=2 policy with 4-bit quantized KV storage, optionally combined with a full-cache 4-bit baseline. Record tokens/s and rouge-2 for each. If variant (a) or (b) exceeds ED's reported 42.0 tokens/s, the 'fastest possible' upper-bound claim is refuted; if a quantized decoding-only variant reaches BM's 73.8 tokens/s at rouge-2 at or above FKV's 0.145, the central claim of superiority over all decoding-only compression fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparison depends on Section 4's claim that ED (Algorithm 2, |kv|=2) is 'the fastest possible decoding-only KV cache eviction algorithm.' This is an assertion, not a demonstration. Within the eviction family, |kv|=1 is a legitimate policy: it keeps only the current token's KV pair during each decode step and discards all previous context, so its per-step KV memory traffic is strictly smaller than ED's |kv|=2. The paper never tests |kv|=1, nor argues that such a policy is outside the scope of 'decoding-only eviction.' Additionally, the title and abstract use 'KV cache compression,' not just eviction; quantizing the KV cache (e.g., 4-bit storage) is a decoding-only compression method that reduces bytes per KV pair and can increase the feasible batch size. ED does not upper-bound such methods. Since BM's throughput is compared directly to ED, if any untested decoding-only variant (|kv|=1, quantized cache, or lower-overhead bookkeeping) exceeds ED, the claimed 'higher than an upper bound' conclusion is not established by the paper's data. The absence of any real decoding-only baseline at matched accuracy leaves this as an empirical gap, not a proven result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper claims that, in GPU-memory-limited settings where input context is longer than the generated sequence, compressing the KV cache during both prefilling and decoding (P&D) yields higher throughput than compressing only during decoding. The proposed method, Batch-Max (BM), uses a block-wise prefilling eviction step (Algorithm 1) with an average-attention eviction rule that the authors describe as hyperparameter-free. Experiments on CNN/DM, NarrativeQA, and GSM8K with Llama-2-13b-chat and Phi-3.5-mini-instruct report that BM achieves 44.0% higher average throughput than an 'extreme' decoding-only eviction baseline (ED, Algorithm 2), considered an upper bound on decoding-only throughput, while keeping accuracy within 2.2% of the full-KV-cache model on average.","tokens_in":7264,"tokens_out":7140,"duration_ms":68100,"significance":"Strengths: the throughput numbers are measured end-to-end rather than modeled; the experiments cover two architectures and three tasks; the average-attention eviction rule is simple and removes a tuning hyperparameter; and the paper explicitly separates prefilling and decoding memory costs, which is the right frame for the setting. If the central claim were fully supported, the result would be practically valuable for long-context summarization and in-context-learning workloads on memory-limited GPUs. However, the advertised comparison rests on an unproven upper-bound assertion, and the accuracy-preservation claim is constructed by selecting |kv| per experiment rather than predicted or tested at a fixed budget; for these reasons the significance is currently contingent.","major_comments":[{"comment":"The statement that ED with |kv|=2 is 'an upper bound on the potential throughput decoding-only compression can produce' and 'the fastest possible decoding-only KV cache eviction algorithm' is asserted, not demonstrated. A decoding-only eviction policy that keeps only the most recent KV pair after every decode step, with the eviction trigger adjusted so that eviction actually fires, would load no more KV data per step than ED, and KV quantization (e.g., 4-bit storage) reduces bytes per pair without being an eviction rule at all; the paper's title includes 'KV cache compression', so quantized decoding-only compression is in scope. Since no |kv|=1 variant, no quantized cache, and no other low-overhead decoding-only baseline is tested or ruled out, the headline '44.0% higher than an upper bound' is not established by the measurements in Tables 1-2.","section":"Section 4, Algorithm 2"},{"comment":"The accuracy-preservation claim is a per-task selected constraint rather than a tested property of BM. The procedure states that |kv| was chosen 'sufficiently large' to keep accuracy near FKV, with values searched over multiples of 128 (and 64 in two cases); this means the reported 2.2% average accuracy gap is a consequence of hand-picking a memory/accuracy operating point for each experiment, not evidence that BM maintains FKV accuracy at a fixed compression budget. The paper also reports no confidence intervals or repeated-seed variation, and the 960 same-seed samples provide a single draw, so rouge-2 differences of 0.001-0.010 in Tables 1-2 are not distinguishable from noise. Please report accuracy across a fixed |kv| sweep and include error bars or multiple seeds.","section":"Section 4.1"},{"comment":"The throughput comparison is not at matched accuracy. ED with |kv|=2 obtains rouge-2 0.000 or accuracy 0.000 on every task, whereas BM is run at the accuracy of the full-KV model. Thus the paper compares a high-accuracy method against a zero-accuracy throughput ceiling. To support the claim of a better throughput/accuracy trade-off than 'any decoding-only KV cache eviction method', the authors should include decoding-only baselines (e.g., H2O, SnapKV, or FastGen) with |kv| chosen to match BM's accuracy, or otherwise report throughput at matched accuracy levels.","section":"Section 4, Tables 1-2"}],"minor_comments":[{"comment":"The condition 'if |kv| = |kv|' uses the same symbol for the maximum cache size and the current number of pairs; please introduce k_max and k_cur and state explicitly how many KV pairs are resident at each decode step, since this is the basis of the upper-bound argument.","section":"Section 4, Algorithm 2"},{"comment":"'H20' should be 'H2O', and 'TOV A' has an irregular spacing; please standardize method names across the text and references.","section":"Section 2"},{"comment":"The expression '|kv| ≥ |kv| − p' is confusing as written; presumably it means the eviction step p keeps the cache size within p of the maximum, but the notation should be made explicit.","section":"Section 4.1"},{"comment":"The set D is described with an ellipsis after 48; either give a closed-form definition (divisors of 960, evidently) or list the full set, since batch sizes are restricted to D for the fairness argument.","section":"Section 4.2"},{"comment":"The word 'significantly' is used for throughput and accuracy differences that are based on single runs; please qualify these statements with the variance estimates requested in the major comments.","section":"Introduction and Section 4"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's assessment that the ED upper-bound assertion is the load-bearing weakness; it is correctable by re-framing the comparison or by adding the missing baselines. The per-task |kv| tuning is a second issue that should be addressed with fixed-budget sweeps. The paper's idea is promising and the direct throughput measurements are useful, so I do not think rejection is warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper makes a simple, useful move—compress the KV cache during prefilling as well as decoding—and shows it raises throughput by roughly 44% on average in the long-context, memory-limited regime, because larger batches fit in memory. That result is new relative to the cited decoding-only methods, and it is measured, not derived. I buy the core claim.\n\nThe method, Batch-Max, is refreshingly straightforward: block-wise eviction during prefilling, using a hyperparameter-free average-attention rule. The average-attention correction for the bias toward early tokens is a nice touch. The experiments cover two architectures, three tasks, and two GPU classes, and the throughput numbers are direct and clearly reported.\n\nWhere I'd push back: the paper compares against ED, which it calls an upper bound on decoding-only compression. That bound holds only for eviction methods, not for quantization or other compression. And within eviction, ED keeps a single previous KV pair; a policy that keeps none would be faster but degenerate, so ED is a reasonable speed bound for non-degenerate eviction. Still, the claim \"higher than any decoding-only compression\" is stronger than what is shown.\n\nThe bigger soft spot is the accuracy story. |kv| is chosen per task on the same test set to keep accuracy near FKV, which makes 'maintaining accuracy' a hand-picked constraint rather than a prediction. There are also no error bars, and no comparison with H2O or SnapKV at matched accuracy. These are fixable, and they don't sink the main finding that P&D eviction enables larger batches.\n\nI'd send this to a referee. With validation-based cache-size selection, variance reporting, and one real decoding-only baseline, it would be a solid systems paper. As it stands, it's a useful empirical contribution for anyone working on LLM serving.\n\nRecommendation: accept with major revisions, conditional on those additions.","headline":"Batch-Max shows a real throughput gain from P&D KV compression in the long-context, memory-limited regime, but the headline comparison rests on an eviction-only upper bound and test-set-tuned |kv|.","tokens_in":7785,"tokens_out":4779,"would_cite":true,"duration_ms":43888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Compressing the KV cache during both prefilling and decoding raises LLM throughput by 44% on average while holding accuracy near the full-cache model.","keywords":["KV cache compression","prefill-and-decode eviction","LLM inference throughput","average attention eviction","batch size scaling","limited GPU memory","long-context inference"],"falsifier":"Measure the token-generation throughput of a decoding-only eviction method that moves less data per step than ED under the same GPU memory budget, for example a KV cache quantized to 8-bit values or a cache that stores a single KV pair in a lower-overhead layout; if such a method matches or exceeds ED's tokens per second at a larger batch size, the claimed upper bound on decoding-only throughput fails.","tokens_in":6810,"feed_emoji":"⚡","tokens_out":4425,"duration_ms":40328,"temperature":0.7,"pith_summary":"This paper tries to establish that, when GPU memory is fixed and the input context is longer than the text to be generated, LLM inference throughput is best served by compressing the KV cache during both prefilling and decoding, rather than only during decoding. Restricting the cache during prefilling frees memory that would otherwise sit idle during generation, so the batch size can be increased. The proposed method, Batch-Max, uses a hyperparameter-free average-attention eviction rule with fixed-size block eviction. Across CNN/DM, NarrativeQA, and GSM8K on two model architectures, it reports 44.0% higher average throughput than an upper-bound decoding-only scheme, while keeping accuracy on average 2.2% below the full-KV-cache model.","feed_headline":"Compress KV caches during prefill to get 44% more LLM tokens","feed_subtitle":"Also compressing during decoding lets bigger batches fit in GPU memory while keeping near-full-cache accuracy.","key_machinery":"The load-bearing mechanism is the P&D eviction loop of Algorithm 1 paired with the average-attention ranking rule. During prefilling, tokens are processed in blocks of up to |kv|, then p KV pairs are evicted every p tokens; the same block eviction continues during decoding, so the KV cache never exceeds |kv| pairs per head and sample. The eviction score is ave weights = sum weights / (curr id + 1 - kv ids), which corrects the bias toward older tokens present in sum-attention ranking. The throughput upper bound is Algorithm 2, ED, which keeps only the most recent KV pair with |kv|=2 and thereby loads the smallest non-empty cache possible in a decoding-only scheme.","core_discovery":"The central claim is that P&D eviction can beat the best possible decoding-only eviction on throughput without sacrificing accuracy, because it converts otherwise idle prefilling memory into additional batch capacity. Batch-Max achieves this by processing the prompt in blocks, evicting p KV pairs every p tokens, and ranking KV pairs by average past attention weight rather than summed weight, which removes the recency bias of methods like H2O. The comparison target, Extreme Decoding-only eviction (ED), keeps only the most recent KV pair at all times and is claimed to be an upper bound on the throughput of any decoding-only KV-cache eviction algorithm. In the reported experiments, Batch-Max delivers roughly 38% higher throughput than ED while slightly exceeding full-cache accuracy, and up to 50% higher when allowed to sit about three percent below full-cache accuracy.","pith_inferences":["A natural extension the paper leaves implicit is tuning the eviction block size p: the experiments fix p=64, but larger blocks lower eviction overhead and smaller blocks keep fresher context, so a per-GPU grid over p could push throughput and accuracy higher.","The average-attention rule could be combined with protected sets (attention sinks, high-variance tokens, or layer-wise budgets) without altering the P&D structure, potentially recovering accuracy on tasks where simple average attention degrades.","The 44% margin is measured against ED as the decoding-only upper bound; a decoding-only method that moves less data per step, for example by quantizing the KV cache or skipping cache loads for certain heads, could narrow or close the gap.","The claim is scope-limited to memory-bound decoding on fixed GPUs; on systems where interconnect or compute is the bottleneck, the batch-size advantage of P&D eviction may not translate directly into throughput."],"forward_implications":["In the long-context, memory-constrained regime, the practical route to higher throughput is increasing batch size, not speeding up per-token decoding.","The average-attention eviction rule is hyperparameter-free, so Batch-Max can be adopted without tuning eviction windows or protecting recent-token windows.","When input length exceeds generation length, decoding-only compression will leave GPU memory idle during prefilling, whereas P&D eviction puts that memory to work.","Accuracy stays within a few percent of the full-cache model, while the extreme decoding-only baseline collapses to near-zero accuracy, so the accuracy/throughput frontier is materially better with P&D eviction.","For tasks where generation length exceeds input length, or where GPU memory is abundant, the paper's conditions do not apply and decoding-only compression may remain the better choice."],"supporting_citations":[{"why":"Provides the premise that prefilling is compute-bound while decoding is memory-bandwidth-bound, motivating the whole batch-size mechanism.","marker":"(Shazeer, 2019)"},{"why":"Defines the decoding-only sum-attention eviction approach whose recency bias the paper's average-attention rule is designed to correct.","marker":"(Zhang et al., 2023)"},{"why":"Introduced the RoCo eviction rule, of which the average-attention component is a part, giving precedent for the paper's ranking score.","marker":"(Ren & Zhu, 2024)"},{"why":"Supplies the Llama-2-13b-chat architecture used for the primary experiments, including its 4096-token context limit.","marker":"(Touvron et al., 2023)"},{"why":"Supplies the Phi-3.5-mini model and its long-context support used in the second set of experiments.","marker":"(Abdin et al., 2024)"},{"why":"Provides the CNN/DM summarization dataset, one of the three long-input tasks tested.","marker":"(Nallapati et al., 2016)"},{"why":"Provides the NarrativeQA reading-comprehension dataset used to evaluate the method.","marker":"(Kočiský et al., 2018)"},{"why":"Provides the GSM8K math reasoning dataset and its few-shot accuracy metric used in the experiments.","marker":"(Cobbe et al., 2021)"}],"fun_headline_variants":["Prefill KV compression boosts LLM throughput by up to 50%","Evict KV pairs during prefill for 38% faster token generation","Batch-Max: Compress KV cache during prefill for larger batches","KV cache eviction in prefill enables larger batch sizes","Compress KV cache during prefill to fit bigger batches, faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on the claim that ED, which always keeps only the single most recent KV pair, is the fastest possible decoding-only KV-cache eviction algorithm, so no decoding-only method could transfer less data per generation step.","fun_headline_variants_meta":{"raw":{"variants":["Prefill KV compression boosts LLM throughput by up to 50%","Evict KV pairs during prefill for 38% faster token generation","Batch-Max: Compress KV cache during prefill for larger batches","KV cache eviction in prefill enables larger batch sizes","Compress KV cache during prefill to fit bigger batches, faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1240,"prompt_tokens":796,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":412,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":412,"tokens_out":444,"duration_ms":4038,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:26:58.475566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the token-generation throughput of a decoding-only eviction method that moves less data per step than ED under the same GPU memory budget, for example a KV cache quantized to 8-bit values or a cache that stores a single KV pair in a lower-overhead layout; if such a method matches or exceeds ED's tokens per second at a larger batch size, the claimed upper bound on decoding-only throughput fails.","supporting_citations":[{"cited_title":"H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models","cited_arxiv_id":null,"evidence_quote":"Defines the decoding-only sum-attention eviction approach whose recency bias the paper's average-attention rule is designed to correct."},{"cited_title":"N., Çaglar G \\\"u lçehre, and Xiang, B","cited_arxiv_id":null,"evidence_quote":"Provides the CNN/DM summarization dataset, one of the three long-input tasks tested."}],"review_version":1}