REVIEW 3 major objections 5 minor 15 references
Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Compressing the KV cache during both prefilling and decoding raises LLM throughput by 44% on average while holding accuracy near the full-cache model.
desk verdict Batch-Max shows a real throughput gain from P&D KV compression in the long-context, memory-limited regime, but the headline comparison rests on an eviction-only upper bound and test-set-tuned |kv|. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the P&D eviction loop of Algorithm 1 paired with the average-attention ranking rule. During prefilling, tokens are processed in blocks of up to |kv|, then p KV pairs are evicted every p tokens; the same block eviction continues during decoding, so the KV cache never exceeds |kv| pairs per head and sample. The eviction score is ave weights = sum weights / (curr id + 1 - kv ids), which corrects the bias toward older tokens present in sum-attention ranking. The throughput upper bound is Algorithm 2, ED, which keeps only the most recent KV pair with |kv|=2 and thereby loads the smallest non-empty cache possible in a decoding-only scheme.
What would settle it
Measure the token-generation throughput of a decoding-only eviction method that moves less data per step than ED under the same GPU memory budget, for example a KV cache quantized to 8-bit values or a cache that stores a single KV pair in a lower-overhead layout; if such a method matches or exceeds ED's tokens per second at a larger batch size, the claimed upper bound on decoding-only throughput fails.
Extended reading notes
Core claim
The central claim is that P&D eviction can beat the best possible decoding-only eviction on throughput without sacrificing accuracy, because it converts otherwise idle prefilling memory into additional batch capacity. Batch-Max achieves this by processing the prompt in blocks, evicting p KV pairs every p tokens, and ranking KV pairs by average past attention weight rather than summed weight, which removes the recency bias of methods like H2O. The comparison target, Extreme Decoding-only eviction (ED), keeps only the most recent KV pair at all times and is claimed to be an upper bound on the throughput of any decoding-only KV-cache eviction algorithm. In the reported experiments, Batch-Max delivers roughly 38% higher throughput than ED while slightly exceeding full-cache accuracy, and up to 50% higher when allowed to sit about three percent below full-cache accuracy.
Load-bearing premise
The headline comparison rests on the claim that ED, which always keeps only the single most recent KV pair, is the fastest possible decoding-only KV-cache eviction algorithm, so no decoding-only method could transfer less data per generation step.
Editorial extensions
If this is right
- In the long-context, memory-constrained regime, the practical route to higher throughput is increasing batch size, not speeding up per-token decoding.
- The average-attention eviction rule is hyperparameter-free, so Batch-Max can be adopted without tuning eviction windows or protecting recent-token windows.
- When input length exceeds generation length, decoding-only compression will leave GPU memory idle during prefilling, whereas P&D eviction puts that memory to work.
- Accuracy stays within a few percent of the full-cache model, while the extreme decoding-only baseline collapses to near-zero accuracy, so the accuracy/throughput frontier is materially better with P&D eviction.
- For tasks where generation length exceeds input length, or where GPU memory is abundant, the paper's conditions do not apply and decoding-only compression may remain the better choice.
Reading between the lines
- A natural extension the paper leaves implicit is tuning the eviction block size p: the experiments fix p=64, but larger blocks lower eviction overhead and smaller blocks keep fresher context, so a per-GPU grid over p could push throughput and accuracy higher.
- The average-attention rule could be combined with protected sets (attention sinks, high-variance tokens, or layer-wise budgets) without altering the P&D structure, potentially recovering accuracy on tasks where simple average attention degrades.
- The 44% margin is measured against ED as the decoding-only upper bound; a decoding-only method that moves less data per step, for example by quantizing the KV cache or skipping cache loads for certain heads, could narrow or close the gap.
- The claim is scope-limited to memory-bound decoding on fixed GPUs; on systems where interconnect or compute is the bottleneck, the batch-size advantage of P&D eviction may not translate directly into throughput.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper claims that, in GPU-memory-limited settings where input context is longer than the generated sequence, compressing the KV cache during both prefilling and decoding (P&D) yields higher throughput than compressing only during decoding. The proposed method, Batch-Max (BM), uses a block-wise prefilling eviction step (Algorithm 1) with an average-attention eviction rule that the authors describe as hyperparameter-free. Experiments on CNN/DM, NarrativeQA, and GSM8K with Llama-2-13b-chat and Phi-3.5-mini-instruct report that BM achieves 44.0% higher average throughput than an 'extreme' decoding-only eviction baseline (ED, Algorithm 2), considered an upper bound on decoding-only throughput, while keeping accuracy within 2.2% of the full-KV-cache model on average.
Significance. Strengths: the throughput numbers are measured end-to-end rather than modeled; the experiments cover two architectures and three tasks; the average-attention eviction rule is simple and removes a tuning hyperparameter; and the paper explicitly separates prefilling and decoding memory costs, which is the right frame for the setting. If the central claim were fully supported, the result would be practically valuable for long-context summarization and in-context-learning workloads on memory-limited GPUs. However, the advertised comparison rests on an unproven upper-bound assertion, and the accuracy-preservation claim is constructed by selecting |kv| per experiment rather than predicted or tested at a fixed budget; for these reasons the significance is currently contingent.
major comments (3)
- [Section 4, Algorithm 2] The statement that ED with |kv|=2 is 'an upper bound on the potential throughput decoding-only compression can produce' and 'the fastest possible decoding-only KV cache eviction algorithm' is asserted, not demonstrated. A decoding-only eviction policy that keeps only the most recent KV pair after every decode step, with the eviction trigger adjusted so that eviction actually fires, would load no more KV data per step than ED, and KV quantization (e.g., 4-bit storage) reduces bytes per pair without being an eviction rule at all; the paper's title includes 'KV cache compression', so quantized decoding-only compression is in scope. Since no |kv|=1 variant, no quantized cache, and no other low-overhead decoding-only baseline is tested or ruled out, the headline '44.0% higher than an upper bound' is not established by the measurements in Tables 1-2.
- [Section 4.1] The accuracy-preservation claim is a per-task selected constraint rather than a tested property of BM. The procedure states that |kv| was chosen 'sufficiently large' to keep accuracy near FKV, with values searched over multiples of 128 (and 64 in two cases); this means the reported 2.2% average accuracy gap is a consequence of hand-picking a memory/accuracy operating point for each experiment, not evidence that BM maintains FKV accuracy at a fixed compression budget. The paper also reports no confidence intervals or repeated-seed variation, and the 960 same-seed samples provide a single draw, so rouge-2 differences of 0.001-0.010 in Tables 1-2 are not distinguishable from noise. Please report accuracy across a fixed |kv| sweep and include error bars or multiple seeds.
- [Section 4, Tables 1-2] The throughput comparison is not at matched accuracy. ED with |kv|=2 obtains rouge-2 0.000 or accuracy 0.000 on every task, whereas BM is run at the accuracy of the full-KV model. Thus the paper compares a high-accuracy method against a zero-accuracy throughput ceiling. To support the claim of a better throughput/accuracy trade-off than 'any decoding-only KV cache eviction method', the authors should include decoding-only baselines (e.g., H2O, SnapKV, or FastGen) with |kv| chosen to match BM's accuracy, or otherwise report throughput at matched accuracy levels.
minor comments (5)
- [Section 4, Algorithm 2] The condition 'if |kv| = |kv|' uses the same symbol for the maximum cache size and the current number of pairs; please introduce k_max and k_cur and state explicitly how many KV pairs are resident at each decode step, since this is the basis of the upper-bound argument.
- [Section 2] 'H20' should be 'H2O', and 'TOV A' has an irregular spacing; please standardize method names across the text and references.
- [Section 4.1] The expression '|kv| ≥ |kv| − p' is confusing as written; presumably it means the eviction step p keeps the cache size within p of the maximum, but the notation should be made explicit.
- [Section 4.2] The set D is described with an ellipsis after 48; either give a closed-form definition (divisors of 960, evidently) or list the full set, since batch sizes are restricted to D for the fairness argument.
- [Introduction and Section 4] The word 'significantly' is used for throughput and accuracy differences that are based on single runs; please qualify these statements with the variance estimates requested in the major comments.
Circularity Check
Accuracy-maintenance claim is enforced by tuning |kv| to the FKV target rather than predicted; throughput comparison is an independent measurement but rests on an asserted ED upper bound.
-
fitted input called prediction
[Section 4.1, Experimental Procedure & Analysis; echoed in the Abstract]
"Using a batch size of b = b0 − 1, the highest possible throughput using ED was computed, as well as the accuracy of FKV . We then tried to maximize the throughput of BM by increasing b, while maintaining the same level of accuracy as FKV by keeping |kv| sufficiently large."
The paper's stated result that BM 'maintain[s] the accuracy of the full KV cache model' is not an independent prediction or derived property of the algorithm; it is the selection criterion for the |kv| hyperparameter. The procedure tries multiples of 128/64 until BM accuracy is reported as slightly greater than or near FKV, so the accuracy-preservation claim is enforced by construction rather than tested. Since taking |kv| equal to the full cache trivially reproduces FKV, finding a sufficiently large |kv| that preserves accuracy is guaranteed. The genuinely independent contribution is the measured throughput gain at those selected configurations, not the ability to maintain accuracy.
full rationale
The throughput comparison is a direct measurement: BM is run at larger batch sizes and its tokens/s are compared with ED at b0-1; no fitted parameter is renamed as the throughput number, and the 44% improvement is not derived from the inputs. The only step that reduces by construction is the accuracy claim: Section 4.1 explicitly tunes |kv| to keep BM accuracy at the FKV level, and then the Abstract and Conclusion present accuracy maintenance as a demonstrated property. That makes the accuracy preservation a hand-selected constraint rather than a prediction, but it is a secondary claim and the throughput finding retains independent content. The assertion that ED with |kv|=2 is 'the fastest possible decoding-only KV cache eviction algorithm' is load-bearing for the 'upper bound' language but is an unproven premise, not a circular reduction; alternative decoding-only compression such as |kv|=1 or quantized caches is not considered. Self-citations to H2O and StreamingLLM appear only as background literature and do not carry the argument. Overall, the circularity is partial and localized to the accuracy-maintenance wording, so a score of 4 is appropriate.
Assumptions & free parameters
free parameters (4)
- max KV cache size |kv| =
per task/model: 896, 1024, 1408, 1536, 1664, 1728, 1792, 1984, 2176
- eviction step p =
64
- generation length =
512
- input length cap =
3584
assumptions (3)
- domain assumption The GPU memory bottleneck during inference is dominated by the KV cache size
- domain assumption Average past attention weight is a sufficient proxy for KV pair importance
- ad hoc to paper ED with a single retained KV pair is an upper bound on decoding-only throughput
Cite this review
Pith. "Pith review of Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression." pith.science (2026). https://pith.science/paper/RZOZJWGL
@misc{pith2026241205693,
author = {Pith},
title = {Pith review of: Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression},
year = {2026},
howpublished = {\url{https://pith.science/paper/RZOZJWGL}},
note = {Machine review of arXiv:2412.05693}
}
read the original abstract
Several works have developed eviction policies to remove key-value (KV) pairs from the KV cache for more efficient inference. The focus has been on compressing the KV cache after the input prompt has been processed for faster token generation. In settings with limited GPU memory, and when the input context is longer than the generation length, we show that by also compressing the KV cache during the input processing phase, larger batch sizes can be used resulting in significantly higher throughput while still maintaining the original model's accuracy.
Reference graph
Works this paper leans on
-
[1]
Abdin, M., Ade Jacobs, S., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett,...
arXiv 2024
-
[2]
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Łukasz Kaiser, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems . arXiv:2110.14168, 2021
arXiv 2021
-
[3]
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs . In ICLR, 2024
work page 2024
-
[4]
M., Melis, G., and Grefenstette, E
Ko c isk \'y , T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. The NarrativeQA Reading Comprehension Challenge . TACL, 6: 0 317--328, 2018
work page 2018
-
[5]
SnapKV: LLM Knows What You are Looking for Before Generation
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. SnapKV: LLM Knows What You are Looking for Before Generation . arXiv:2404.14469, 2024
arXiv 2024
-
[6]
Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., and Shrivastava, A. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time . In NeurIPS, pp.\ 52342--52364, 2023
work page 2023
-
[7]
N., Çaglar G \"u lçehre, and Xiang, B
Nallapati, R., Zhou, B., dos Santos, C. N., Çaglar G \"u lçehre, and Xiang, B. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond . In CoNLL, pp.\ 280--290, 2016
work page 2016
-
[8]
Transformers are Multi-State RNNs
Oren, M., Hassid, M., Yarden, N., Adi, Y., and Schwartz, R. Transformers are Multi-State RNNs . arXiv:2401.06104, 2024
arXiv 2024
Show all 15 references
-
[9]
and Zhu, K
Ren, S. and Zhu, K. Q. On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference . arXiv:2402.06262, 2024
2024 arXiv
-
[10]
Fast Transformer Decoding: One Write-Head is All You Need
Shazeer, N. Fast Transformer Decoding: One Write-Head is All You Need . arXiv:1911.02150, 2019
1911 arXiv
-
[11]
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....
2023 arXiv
-
[12]
and Gan, S
Wang, Z. and Gan, S. SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget . arXiv:2404.04793, 2024
2024 arXiv
-
[13]
Efficient Streaming Language Models with Attention Sinks
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient Streaming Language Models with Attention Sinks . In ICLR, 2024
2024
-
[14]
SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction
Zhang, X., Du, C., Du, C., Pang, T., Gao, W., and Lin, M. SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction . arXiv:2410.13846, 2024
2024 arXiv
-
[15]
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., R \'e , C., Barrett, C., Wang, Z., and Chen, B. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models . In NeurIPS, pp.\ 34661--34710, 2023
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.