Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache

T0 review · 5 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MiniKV combines 2-bit KV quantization with token eviction to cut KV-cache memory by 86% while keeping 98.5% of full-model accuracy on LongBench.

desk verdict A solid systems paper that makes 2-bit quantization plus eviction work together, held back mainly by one-sample evidence for its load-bearing persistence assumption. read the letter →

arxiv 2411.18077 v3 pith:54PQIH33 submitted 2024-11-27 cs.CL cs.LG

Akshat Sharma , Hangliang Ding , Jianping Li , Neel Dani , Minjia Zhang This is my paper · ORCID
classification cs.CLcs.LG
keywords KVcachequantization2-bitadaptiveheavyhitterselectionlong-contextLLMinferenceFlashAttentioncompatibilitylayer-wiseallocationTritonkernels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MiniKV claims that the two main KV-cache compression strategies—ultra-low-bit quantization and token eviction—can be combined to far exceed either alone, provided the combination is co-designed with the attention kernel. On LongBench, with LLaMA2-7B-chat, it reports a KV cache of 0.33 GB versus 2.4 GB for the full model (an 86% reduction) while recovering 98.5% of full-model accuracy (34.65 vs 35.19 with its pyramid variant). The same recipe generalizes to LLaMA2-13B, Mistral-7B, and Llama-3 models on InfiniteBench, and translates into lower latency, higher throughput, and longer processable prompts on an A100 GPU. The practical stake is that long-context LLM serving, currently bottlenecked by KV-cache memory, could run dramatically more requests per GPU.

What carries the argument

Persistent context selection is the load-bearing mechanism: heavy-hitter tokens are chosen once from cumulative attention scores at the end of prefill and are never re-selected during generation, which makes sub-channel 2-bit key quantization well-defined even when eviction would otherwise change group membership. Around this sit a pyramid layer budget that allocates the heavy-hitter cache unequally across layers, a KIVI-style packing scheme that stores 16 INT2 scalars per INT32 word with FP16 scale and zero-point metadata, and a two-pass selective flash-attention kernel that produces both the attention output and column-wise cumulative attention scores using memory that grows linearly with sequence length rather than quadratically.

What would settle it

Track the overlap between the heavy-hitter set chosen at the end of prefill and the set that would be chosen at each later decoding step across all 13 LongBench datasets; if average overlap drops below the reported 60–80% on datasets with topic shifts, or if re-selecting tokens every 32 steps improves the average score by more than about one point, the persistence assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 2-bit quantized KV cache remains accurate in long-context tasks only when quantization is paired with adaptive selection of which tokens to keep, and that this pairing is most accurate when the token budget is distributed pyramid-style across layers, with more capacity in lower layers and less in higher ones. The authors show the composition is not trivial: per-token INT2 quantization of H2O-style selected tokens collapses accuracy, and combining SnapKV-style selection with KIVI-style quantization drops LongBench scores from 35 to 32 points. MiniKV instead selects persistent heavy hitters at the end of prefill, retains a critical recent-window fraction, applies sub-channel quantization to keys and per-token quantization to values, and processes attention with a two-pass kernel that returns cumulative attention scores without materializing the attention matrix. With a 50% cache budget it reports 86% total KV compression while keeping at least 98.5% of full-model accuracy on LLaMA2-7B-chat, and it lies on the Pareto frontier of accuracy versus cache size across all six LongBench task categories.

Load-bearing premise

The selection of which tokens to keep and quantize is made once at the end of prefill and never revised, based on the claim that 60–80% of heavy hitters persist through generation, a claim the paper supports with a single sample prompt.

Editorial extensions

If this is right

  • At matched KV-cache size, MiniKV-Pyramid outperforms H2O, SnapKV, and Q-Hitter on LongBench, staying within 0.54 points of the full model on LLaMA2-7B-chat while using 86% less cache memory.
  • Because selection happens once at prefill and never changes, the retained KV set has a fixed layout, enabling 2-bit sub-channel quantization and fused dequantization kernels during decoding.
  • The two-pass kernel removes the need to materialize the quadratic attention matrix, making adaptive KV eviction compatible with FlashAttention-style memory-efficient attention and longer prompts.
  • On a single A100 GPU, MiniKV reports a 48% throughput improvement over its strongest baseline and supports prompts of 44K tokens, with peak memory low enough to avoid out-of-memory failures that H2O and KIVI encounter at larger batch sizes.
  • MiniKV's gains are concentrated in attention and KV loading, so the benefit grows with sequence length, making the method most valuable exactly in the long-context regime where KV-cache memory is the bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the persistent-heavy-hitter assumption holds across workloads, the prefill-phase selection could be reused across many queries that share a prompt prefix, turning the fixed layout into a reusable prompt-cache block rather than a per-request cost.
  • The paper's own failed SnapKV-plus-KIVI combination suggests a testable rule: eviction methods whose retained tokens are quantization-sensitive fail under 2-bit compression, so a robust co-design should score tokens by post-quantization error, not by attention alone.
  • The pyramid result implies that lower layers carry more tokens; a natural untested extension is to give lower layers higher bit-widths and upper layers lower bit-widths within a fixed memory budget.
  • Because the persistence evidence in Appendix E is a single sample prompt, a multi-dataset persistence audit would determine whether fixed prefill selection survives topic shifts, multi-turn dialogue, and streaming inputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. MiniKV combines 2-bit KV-cache quantization with adaptive token eviction (heavy hitters plus a recent window) and a pyramid-style layer-wise budget allocation, together with purpose-built Triton kernels that make the selective attention compatible with FlashAttention-style linear-memory execution. The paper evaluates accuracy on LongBench (with prompts truncated to 4096 tokens), InfiniteBench, and GSM8K, and reports latency, throughput, and peak-memory measurements on A100 and GH200 GPUs. The headline claims are more than 80% (86% in one configuration) KV-cache compression, roughly 98.5% accuracy recovery for LLaMA2-7B-chat on LongBench, Pareto-optimality across all six LongBench task categories, and a 48% throughput improvement over the strongest baseline.

Significance. If the claims hold, MiniKV is a genuinely useful algorithm/system co-design contribution: it is one of the few works to combine ultra-low-bit KV quantization with eviction in a FlashAttention-compatible kernel, and the matched-cache-size accuracy table plus the Pareto-curve analysis provide a reasonable evaluation template. The two-pass selective-attention kernel with linear memory is a concrete systems contribution, and the manuscript is candid about limitations, including the failure of a SnapKV+KIVI combination and the prefill-phase slowdown. The significance is contingent, however, on fixing the load-bearing issues listed below: the headline numbers are configuration- and model-specific, the matched-size comparison is internally inconsistent for one baseline, and the persistent-heavy-hitter assumption that the fixed 2-bit layout depends on is supported by only a single sample.

major comments (5)
  1. [§3.1.1 / Appendix E] The persistent-context-selection assumption is load-bearing: it justifies fixing the 2-bit sub-channel quantization layout and never re-selecting heavy hitters during decoding. The only supporting evidence is one sample from the Lcc dataset (Figure 11) with k=150, with no model or layer specification and no quantitative persistence curve; the '60-80%' claim in §3.1.1 cannot be read off that figure as presented. Moreover, the measurement uses H2O's retained set, not MiniKV's own (25% HH + 25% RW) selection. Please provide per-head persistence curves across the three evaluated models, the six LongBench task categories, and the actual MiniKV budgets, and show that the accuracy loss from a frozen selection is bounded.
  2. [§4.1 / Table 1] The headline accuracy recovery is measured on inputs truncated to 4096 tokens: 'The maximum prompt length is 4096 for all models with the first and last 2048 tokens taken.' Table 1 therefore reports 4K-context behavior, not LongBench's long-context regime, and the 86% compression figure in Appendix I is computed for lprompt=4096 and lgen=512. The abstract's 'long context tasks' claim and the 98.5% recovery figure should be explicitly scoped to this truncated setting; results on untruncated LongBench prompts (or another long-context suite with matched cache-size controls) are needed to support the broader claim.
  3. [§4.2 / Appendix I] The main accuracy comparison is not at matched cache sizes under the paper's own formulas. With lprompt=4096 and lgen=512, MiniKV uses (H·d)·(0.5·4096+512) ≈ 0.33 GB. H2O(15%) is about 0.34 GB only if generated tokens are excluded from the budget, but the Appendix I formula for SnapKV includes them, giving SnapKV(15%) ≈ 2·H·d·2·(0.15·4096+512) ≈ 0.55 GB. Since Table 1 is introduced as a comparison 'under a similar KV cache size', the SnapKV row is not a matched-size comparison. Please recompute the comparison using a SnapKV budget that matches MiniKV's actual cache size, or plot accuracy versus cache size for each method across several budgets.
  4. [Abstract / Table 1] The abstract's claim of 'recovering over 98.5% of accuracy' is supported only by the LLaMA2-7B-chat MiniKV-Pyramid row (34.65/35.19 = 98.5%). The other evaluated models recover less: LLaMA2-13B-chat achieves 32.37/33.32 = 97.1% and Mistral-7B-Instruct achieves 39.97/41.20 = 97.0%. Please qualify the headline number with the model-specific range and avoid presenting a single best configuration as the general result.
  5. [§4.1 / Appendix H / §4.5] The default configuration (50% budget, 25% HH + 25% RW, group size 16, nr=128, pyramid depth d=7) is fixed using the design observations in Figures 2 and 3, but no sensitivity analysis is reported for d, the HH/RW split, group size, or nr. The GSM8K result (§4.5) shows that on a reasoning task the required adaptive budget is about 90%, which illustrates that the 50%-budget configuration is not universally applicable. To support the claim that the 86%-compression accuracy point is general rather than a single tuned configuration, please include ablations around these hyperparameters and state which choices were made on a validation split versus the reported test sets.
minor comments (4)
  1. [Table 1] The table refers to 'rows marked in brown', but this color cue is not visible in standard arXiv rendering; please use a symbol, boldface, or an explicit marker column to indicate the matched-size rows.
  2. [§3.2] The first-pass paragraph contains an incomplete sentence: 'we follow FlashAttention's online softmax reduction to compute the weighted V block write it back.' Please split and rephrase this sentence.
  3. [Table 2] The column headers in Table 2 are concatenated without separators (e.g., 'LongBook ChoiceLongDialogue QACode Debug'), making the table hard to read; please use clearly separated column names.
  4. [Table 1] The margins between MiniKV and the full model are often within one LongBench point, and no variance or number of runs is reported; please state whether these are single greedy-decoding runs and, ideally, report standard deviations across seeds.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: MiniKV's accuracy, compression, and speedup claims are empirical measurements, with only a minor related-work self-citation and mild in-benchmark hyperparameter tuning.

full rationale

MiniKV's central claims—accuracy recovery, compression ratio, and system performance—are empirical measurements reported against external baselines (H2O, SnapKV, Q-Hitter, KIVI, full model) on LongBench, InfiniteBench, and GSM8K. No core result is derived by definition from a fitted parameter, and no equation in the paper reduces a prediction to its own input. The persistent heavy-hitter assumption is load-bearing, but it is an empirical premise supported only by a single Lcc sample in Appendix E; that is a fragility and evidence-strength concern, not circularity. The only self-citation is Ge et al. (2023), cited in related-work enumerations in Sections 1 and 2; it is not used to justify MiniKV's design or results and is therefore not load-bearing. The 50% cache budget, 25%/25% HH/RW split, and pyramid depth d=7 were evidently chosen from LongBench experiments (Figures 2 and 3) and then evaluated on the same LongBench benchmark, so the headline 98.5% accuracy recovery is in-sample rather than a strict out-of-sample prediction. This is a benchmark-validity caveat, but it does not make the result equivalent to the input by construction: the reported accuracy is a measured outcome, not a quantity forced by the tuned hyperparameters. Score 2 reflects the minor self-citation and mild in-benchmark tuning, with no significant circular derivation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

MiniKV introduces no new physical or architectural entities. Its central claim rests on four domain assumptions about attention structure, plus four hand-tuned hyperparameters that were selected using the same LongBench benchmark used for final evaluation. The persistence assumption is the least supported because it relies on a single sample.

free parameters (4)
  • KV cache budget = 50% of prompt tokens (25% heavy hitters + 25% recent window per layer)
    Selected from LongBench exploratory sweeps in Section 3.1.2 and used in all main experiments.
  • Pyramid depth d = 7
    Controls the linear interpolation of heavy-hitter budgets across layers; chosen by hand and validated on LongBench in Section 3.1.3 and Appendix H.
  • Quantization group size = 16 scalars
    Group size for sub-channel key and per-token value quantization; balances quantization error against metadata overhead.
  • Residual buffer length nr = 128
    Number of decoding steps before newly generated KV states are quantized; inherited from KIVI and kept for MiniKV.
assumptions (4)
  • domain assumption Accumulated attention scores identify the tokens that matter for generation (heavy-hitter assumption)
    Borrowed from H2O and used to select Kprefill_HH at the end of prefill in Section 3.1.1.
  • domain assumption Heavy hitters selected at prefill persist through the whole generation
    Required by persistent context selection; supported only by a single-sample analysis in Appendix E.
  • domain assumption Layer-wise pyramid allocation of heavy-hitter budgets preserves accuracy
    Taken from PyramidKV and verified empirically on LongBench in Figure 3, not derived from first principles.
  • domain assumption 2-bit sub-channel key and per-token value quantization with group size 16 introduces acceptable error
    Based on KIVI and KVQuant observations and checked empirically on LongBench in Appendix F.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache." pith.science (2026). https://pith.science/paper/54PQIH33

@misc{pith2026241118077,
  author       = {Pith},
  title        = {Pith review of: MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/54PQIH33}},
  note         = {Machine review of arXiv:2411.18077}
}
read the original abstract

How to efficiently serve LLMs in practice has become exceptionally challenging due to their prohibitive memory and computation requirements. In this study, we investigate optimizing the KV cache, whose memory footprint poses a critical bottleneck in LLM inference, especially when dealing with long context tasks. To tackle the challenge, we introduce MiniKV, a KV cache optimization method that simultaneously preserves long context task accuracy while significantly reducing KV cache size via a novel 2-bit layer-discriminative KV cache. More importantly, we develop specialized CUDA kernels to make MiniKV compatible with FlashAttention. Experiments on a wide range of long context tasks show that MiniKV effectively achieves 86% KV cache compression ratio while recovering over 98.5% of accuracy, outperforming state-of-the-art methods while achieving excellent measured system performance improvements.

Figures

Figures reproduced from arXiv: 2411.18077 by the authors.

Figure 1
Figure 1. An overview of MiniKV. Tensors colored red/blue indicate 16-bit/2-bit representation, and shaded tokens are evicted during inference. During the prefill phase, we employ pyramid KV with rectified token selection policy across layers to identify a sparse set of important tokens. For all the important tokens, we employ sub-channel Key quantization and per-token Value quantization to minimize the quantization errors wh… view at source ↗
Figure 2
Figure 2. (left) reveals an interesting aspect of the KV cache selectivity: The model performs bet￾ter on some datasets with more HH (on Passage Count) and on some with a longer RW (on Trivi￾aQA). More importantly, using solely RW or HH leads to a catastrophic accuracy drop in certain tasks (in Lcc and TriviaQA). This indicates that to have a robustly optimized KV cache selection policy, the model needs to maintain at least a… view at source ↗
Figure 3
Figure 3. Performance of layer-wise KV cache alloca [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Two-pass kernel parallelism: In the first pass, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Performance on GSM8K: Since GSM8K is a reasoning-intensive task, MiniKV requires a significant adaptive KV cache budget (∼ 90%) to match the perfor￾mance of the full model. reduces latency through a compression and system co-design approach, which reduces the number of…
Figure 7
Figure 7. Figure 7: Left: Latency (s) for batch size = 1 and gener [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Left: Peak memory usage (GB) vs batch size [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: shows that with 50% KV cache size, the LLM can still obtain comparable accuracy (e.g., <1 point) as the full KV cache. However, high levels of KV eviction (e.g., 80-95%) hurts LLM’s performance on long context tasks significantly [PITH_FULL_IMAGE:figures/full_fig_p013…
Figure 10
Figure 10. Figure 10: The attention distribution of LLaMA2-7B over the HotpotQA dataset in LongBench. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Top-k tokens with the highest cumulative attention score on the Lcc dataset from LongBench. Green tokens mark the heavy hitters retained by the H2O algorithm. Here, we choose k = 150. 20% 40% 60% 80% Cache Budget 29 30 31 32 33 34 35 LongBench Accuracy Different Quant…
Figure 12
Figure 12. Figure 12: Performance of per-token quantized H2O on the LongBench dataset. INT8/4 quantization can maintain performance across cache budgets. However, INT2 quantization suffers from a catastrophic drop in performance. test the model’s tendency to forgot information over a long …
Figure 15
Figure 15. Figure 15: Per token latency breakdown for the decod [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 13
Figure 13. Figure 13: Performance Versus KV Cache Size: MiniKV offers the best performance for the smallest KV cache size [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Performance Versus KV Cache Size: MiniKV offers the best performance for the smallest KV cache size [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Reference graph

Works this paper leans on

41 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Meta AI. 2024. Introducing Meta LLaMA-3 . https://ai.meta.com/blog/meta-llama-3/

    work page 2024

  2. [2]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. CoRR, abs/2308.14508

    arXiv 2023

  3. [3]

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kelly. 2024. Reducing transformer key-value cache size with cross-layer attention. CoRR, abs/2405.12981

    arXiv 2024

  4. [4]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. Pyramidkv: Dynamic KV cache compression based on pyramidal information funneling. CoRR, abs/2406.02069

    arXiv 2024

  5. [5]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \' e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \' e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022

    2022

  6. [6]

    Daya Guo DeepSeek-AI, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

  7. [7]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: accurate post-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323

    arXiv 2022

  8. [8]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive KV cache compression for llms. CoRR, abs/2310.01801

    arXiv 2023

Show all 41 references
  1. [9]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length LLM inference with KV cache quantization. CoRR, abs/2401.18079

    2024 arXiv

  2. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  3. [11]

    Jure Leskovec and Rok Sosic. 2016. SNAP: A General-Purpose Network Analysis and Graph-Mining Library . ACM TIST , 8(1):1:1--1:20

    2016

  4. [12]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469

    2024 arXiv

  5. [13]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

    2024

  6. [14]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024 a . Minicache: Kv cache compression in depth dimension for large language models. CoRR, abs/2405.14366

    2024 arXiv

  7. [15]

    Liyuan Liu, Jialu Liu, and Jiawei Han. 2021. Multi-head or single-head? an empirical comparison for transformer training. CoRR, abs/2106.09650

    2021 arXiv

  8. [16]

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023 a . LLM-QAT: data-free quantization aware training for large language models. CoRR, abs/2305.17888

    2023 arXiv

  9. [17]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023 b . Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Advances in Neural Informa...

    2023

  10. [18]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 b . KIVI: A tuning-free asymmetric 2bit quantization for KV cache. CoRR, abs/2402.02750

    2024 arXiv

  11. [19]

    Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, and Edoardo M. Ponti. 2024. Dynamic memory compression: Retrofitting llms for accelerated inference. CoRR, 2403.09636

    2024 arXiv

  12. [20]

    NVidia. 2025. Introducing New KV Cache Reuse Optimizations in NVIDIA TensorRT-LLM . https://tinyurl.com/4zbvwpcz. Accessed: 14-Feburary-2025

    2025

  13. [21]

    OpenAI. 2024. Introducing OpenAI o1 . https://openai.com/o1/

    2024

  14. [22]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R \' e , Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single GPU . In International Conference on Machine Le...

    2023

  15. [23]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774

    2024 arXiv

  16. [24]

    Philippe Tillet, Hsiang - Tsung Kung, and David D. Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2019, Phoeni...

    2019

  17. [25]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 arXiv

  18. [26]

    Raushan Turganbay. 2024. Unlocking Longer Generation with Key-Value Cache Quantization . https://huggingface.co/blog/kv-cache-quantization. Accessed: 14-Feburary-2025

    2024

  19. [27]

    Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. 2025. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461

    2025 arXiv

  20. [28]

    vLLM. 2025. Quantized KV Cache . https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache.html. Accessed: 14-Feburary-2025

    2025

  21. [29]

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019,...

    2019

  22. [30]

    Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. 2024. D2o: Dynamic discriminative operations for efficient generative inference of large language models. CoRR, abs/2406.13035

    2024 arXiv

  23. [31]

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024. Retrieval head mechanistically explains long-context factuality. CoRR, abs/2404.15574

    2024 arXiv

  24. [32]

    Guangxuan Xiao, Ji Lin, Micka \" e l Seznec, Hao Wu, Julien Demouth, and Song Han. 2023 a . Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, ...

    2023

  25. [33]

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2024. Duoattention: Efficient long-context LLM inference with retrieval and streaming heads. CoRR, abs/2410.10819

    2024 arXiv

  26. [34]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023 b . Efficient streaming language models with attention sinks. CoRR, abs/2309.17453

    2023 arXiv

  27. [35]

    Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024 a . Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference. CoRR, abs/2405.12532

    2024 arXiv

  28. [36]

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024 b . No token left behind: Reliable KV cache compression via importance-aware mixed precision quantization. CoRR, abs/2402.18096

    2024 arXiv

  29. [37]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024 a . bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718

    2024 arXiv

  30. [38]

    Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. 2024 b . Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6:381--394

    2024

  31. [39]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \' e , Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advan...

    2023

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.