Pith. sign in

REVIEW 4 major objections 5 minor 32 references

ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ZigZagKV cuts KV cache memory to about 20% of full inference while keeping accuracy nearly intact by giving each layer its own token budget instead of a uniform one.

desk verdict Data-derived per-layer KV budgets are a genuine but narrow extension of PyramidKV; the paper's own evidence is thinner than its prose. read the letter →

arxiv 2412.09036 v1 pith:Y3EJWHYL submitted 2024-12-12 cs.CL

Meizhi Zhong , Xikai Liu , Chen Zhang , Yikun Lei , Yan Gao , Yao Hu , Kehai Chen , Min Zhang This is my paper · ORCID
classification cs.CL
keywords KVcachecompressionlayeruncertaintylong-contextmodelingattentionscorebudgetallocationinferenceefficiencypartial
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that partial KV cache compression loses unnecessary information because it gives every layer the same token budget. It claims the minimum budget needed to preserve 90% of attention and hidden-state information varies sharply across layers, so a fixed uniform split is suboptimal. ZigZagKV uses a per-layer uncertainty measure, computed once on a small calibration set, to redistribute a fixed total budget across layers with a floor for every layer. The result is a training-free, inference-time compression method that reportedly uses about one-fifth of full KV memory while keeping performance close to full-KV inference on LongBench and needle-in-a-haystack retrieval.

What carries the argument

The carrying object is the Layer Minimum Budget to maintain Attention (LMBA), defined as the average across heads of the smallest token subset that keeps 90% of attention mass. ZigZagKV measures LMBA once per model on 200 samples of the 2WikiQA dataset, normalizes it across layers into a relative uncertainty, and uses the bounded allocation formula to convert one average budget $B$ into per-layer budgets. The floor $B_{\text{bound}}$ protects low-uncertainty layers from being starved when another layer has a very high LMBA. Token selection then reuses cumulative attention scores from the last $w$ tokens, so the mechanism adds a layer-level allocation rule on top of an existing per-head eviction policy.

What would settle it

Recompute the optimal per-layer budget on a different dataset and a different input length; if the dataset-specific profile clearly beats the fixed 2WikiQA profile on LongBench or a 100K-token needle test, the transferability assumption fails. Concretely, compare ZigZagKV with profiles calibrated on 2WikiQA, on NarrativeQA, and on a long-document corpus, all evaluated at the same average budgets.

Watch

Extended reading notes

Core claim

The central claim is that the optimal KV cache budget is layer-dependent, and that a zigzag allocation, with larger budgets in low-uncertainty layers and smaller budgets where attention is concentrated, preserves more information than uniform allocation at the same average budget. The paper formalizes layer uncertainty as LMBA, the average across heads of the smallest number of prefix tokens needed to retain 90% of total attention weight, and also measures LMBO, the analogous minimum budget for 90% hidden-state output similarity. ZigZagKV sets each layer's budget to $B_{\text{bound}} + (B - B_{\text{bound}}) \cdot L \cdot \text{uncertainty}_l$, then selects tokens by cumulative attention scores from the last $w$ instruction tokens. Experiments on Mistral-7B-Instruct-v0.3 and LLaMA-3.1-8B-Instruct show the method outperforming StreamingLLM, H2O, SnapKV, and PyramidKV under matched budgets, with KV memory around 20% of full-KV inference.

Load-bearing premise

The fixed per-layer budget profile, estimated once from 200 question-answering samples, is assumed to transfer across tasks, prompt lengths, and contexts without needing recalibration.

Editorial extensions

If this is right

  • At the same total KV budget, ZigZagKV reports lower attention loss and hidden-state output loss than uniform-budget baselines, meaning fewer important tokens are discarded.
  • KV cache memory can be reduced to roughly 20% of full-KV usage while retaining near-full accuracy on LongBench, which enables longer contexts on fixed hardware.
  • The method is training-free and operates purely at inference time, so it can be added to existing pipelines that already compute cumulative attention scores for token eviction.
  • The layer-uncertainty profile is model-specific but task-independent, so one calibration pass per model is claimed to suffice across the 16 LongBench tasks and needle-in-a-haystack tests.
  • The bounded budget floor prevents degenerate allocations, so extreme LMBA imbalances across layers do not cause any single layer to lose all its cache.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed profile derived from 2WikiQA may not be optimal for tasks whose attention patterns differ sharply, so a testable extension is per-domain calibration and measuring whether it improves scores further.
  • Layer uncertainty could be updated online as tokens stream in, making the zigzag shape adaptive to input length and content rather than frozen after calibration.
  • The same uncertainty principle might transfer to encoder-decoder or mixture-of-experts architectures, which the paper explicitly leaves unvalidated.
  • If LMBA is stable across languages and domains, ZigZagKV could be combined with quantization or sparse attention to push memory savings beyond the reported 80% reduction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ZigZagKV, a training-free KV-cache compression method that allocates per-layer cache budgets according to a layer-uncertainty profile (LMBA) estimated from attention scores on a calibration set, with a fixed minimum budget per layer. The method is evaluated on Needle-in-a-Haystack and LongBench with two decoder-only LLMs, comparing against uniform-budget and pyramid-budget baselines. The main claims are that different layers require different minimum budgets to preserve information, that the LMBA profile captures this variation, and that ZigZagKV reduces memory usage to about 20% of full KV inference while remaining nearly lossless on average.

Significance. If the transferability of the LMBA profile were established, ZigZagKV would be a simple and practical contribution: it is inference-only, has negligible overhead, and the layer-wise budget analysis is an interesting empirical observation that goes beyond the depth-only heuristic of PyramidKV. The method ships with an ablation of the bounded-budget mechanism and a latency comparison, which are useful. However, the strength of the empirical support is well below the level of the prose claims: the central hyperparameter Bbound is never specified, LongBench gains over PyramidKV are typically fractions of a point with no error bars or significance tests, and the attention-loss analysis is evaluated on the same dataset used to fit the profile. These issues need to be addressed before the central claim can be considered supported.

major comments (4)
  1. [§4.1, Eq. (6)] The parameter Bbound in Eq. (6) is never given a value anywhere in the manuscript, so the exact per-layer allocation used in the experiments is not reproducible. Since Eq. (6) defines the actual budget for every layer, a reader cannot re-implement ZigZagKV without guessing this parameter. The ablation in Figure 6 only compares the method with and without the bound; it does not examine sensitivity to the magnitude of Bbound, which is necessary because the bound can materially change which layers receive extra tokens.
  2. [§3, §4.1, Tables 1-2] The LMBA profile is computed from 200 samples of the 2WikiMQA dataset and then applied unchanged to all 16 LongBench datasets and to Needle-in-a-Haystack contexts up to 30K tokens, without any evidence that the profile transfers across tasks and context lengths. The LongBench average gains over PyramidKV are typically 0.1 to 0.5 points (e.g., Table 1, Mistral: 43.3 vs 43.2 at B=128; 45.3 vs 45.1 at B=256; 46.5 vs 46.3 at B=512; 47.1 vs 47.0 at B=1024; 47.7 vs 47.6 at B=2048), and no error bars, seeds, or significance tests are reported. These differences are within run-to-run noise for such benchmarks, so the claim that a single fixed profile is near-optimal across all tasks and lengths is not supported.
  3. [§5.5, Table 3] The attention-loss evaluation in Table 3 is performed on the 2WikiMQA dataset, which is the same dataset used to compute the LMBA profile in Section 3. Minimizing attention loss on that dataset is essentially the objective the budget was fitted to optimize, so the lower attention loss of ZigZagKV in Table 3 is partly circular. The independent evidence for the method is the task accuracy in Tables 1-2, but those gains are small and not statistically tested. The authors should report attention loss on held-out tasks or on the longer Needle-in-a-Haystack documents to demonstrate that the profile generalizes.
  4. [§5.4, Tables 1-2, Figure 4] The text states that ZigZagKV 'consistently outperforms' prior methods, but in several LongBench entries it is equal to or worse than PyramidKV or SnapKV. For example, Table 2 (LLaMA, B=256) SAMSum shows PyramidKV 41.7 and ZigZagKV 41.6; Table 1 (Mistral, B=1024) 2WikiMQA shows ZigZagKV 38.9, the same as SnapKV and PyramidKV. The average-score advantage should be described as such, not as consistent per-task superiority, especially because no significance testing is provided.
minor comments (5)
  1. [Figure 4 caption] The caption lists budget sizes '96, 128, 256, 512 and 1024', while the x-axis of the plots runs from 200 to 1000; Section 5.2 states the budgets are {128, 256, 512, 1024}. These should be aligned.
  2. [Eq. (4)] Equation (4) uses LMBAl in the numerator, but Eq. (3) defines LMBA without a layer index; use a consistent notation such as LMBA_l throughout Section 4.1.
  3. [Eq. (7)] In Eq. (7), the notation A_h_ij is used for attention weights, but the head index h is not defined in that equation; the reader must infer it from Eq. (1) in Section 2.1. Please define it explicitly.
  4. [Abstract and Section 5.4] The '~20%' memory reduction claim is not derived anywhere; the paper should state the full cache size and context length used for this calculation, since the experiments report mean cache budgets rather than a memory ratio.
  5. [Section 5.4, Table 5] Table 5 reports latency only for StreamingLLM, PyramidKV, and ZigZagKV; the other baselines from the main comparison are omitted without explanation, making the computational-overhead comparison incomplete.

Circularity Check

1 steps flagged · score 4.0 of 10

Attention-retention analysis is fitted on the metric it reports; LongBench/NIAH results provide independent support.

  1. fitted input called prediction [Section 3 (Eq. 3, LMBA calibration) and Section 5.5 (Table 3)]
    "Empirically, we analyze the LMBA on two widely-used large language models ... using 200 samples from the 2WikiQA dataset ... To investigate whether the proposed method achieves more attention information, as described in Section 3, we calculated the average deviation from achieving 0.9 attention across various cache settings, which we refer to as attention loss. The results are presented in Table 3."

    ZigZagKV's per-layer budgets are set by Eq. 4-6 as proportional to LMBA, and LMBA (Eq. 3) is, by definition, the minimum budget needed to keep attention loss below 0.1 on the calibration attention scores. Table 3 then measures 'attention loss' on 2WikiMQA, a dataset directly in the same multi-hop QA family as the 2WikiQA calibration set, and reports that ZigZagKV 'substantially reduces attention loss.' This is not an independent prediction: the allocation was constructed to minimize the very quantity being reported. Uniform-budget baselines cannot win this metric because they were not tuned on it.

full rationale

The paper's main contribution is a per-layer budget profile derived from LMBA on 200 2WikiQA samples and then applied at inference. The headline results on LongBench (16 datasets) and Needle-in-a-Haystack are external benchmarks, not constructed from the LMBA fit, so the claim that ZigZagKV outperforms uniform-budget baselines on task accuracy has independent empirical content. The circularity is localized to the attention-loss analysis in Section 5.5/Table 3: the budget is fitted to minimize attention loss, and the table reports attention loss on a closely related corpus, so that specific supporting claim reduces to a fitted-input re-measurement. No load-bearing self-citation or imported uniqueness theorem is present; the cited prior work is used only for context or standard baselines. The unspecified value of Bbound in Eq. 6 is a reproducibility concern rather than a circularity concern. Overall, partial circularity in a secondary analysis, while the central task-based evaluation remains self-contained.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method rests on a calibration profile (LMBA), one unspecified floor parameter (Bbound), and an inherited window size (w). No new physical or architectural entities are postulated; the main risk is that the calibration profile is treated as model-level ground truth without validated transfer.

free parameters (3)
  • Bbound (bounded budget floor) = not stated
    Introduced in Eq. 6 as a per-layer minimum; no value, search range, or selection criterion is reported. The ablation in Figure 6 only shows that a floor helps.
  • w (window size for importance scores) = not stated
    Eq. 7 sums attention over the last w tokens, following SnapKV, but the exact w used in experiments is not reported.
  • LMBA uncertainty profile = per-layer proportions from 200 2WikiQA samples
    Eq. 4 converts LMBA into uncertainty; this profile is estimated once per model on 200 2WikiQA samples and then applied to all evaluation contexts. No evidence of stability across tasks is given.
assumptions (4)
  • domain assumption The per-layer LMBA measured on 200 2WikiQA samples is a stable property of the model that transfers to all tasks and context lengths.
    Section 3 computes LMBA on a small sample of 2WikiQA; Section 4 uses it to allocate budgets on Needle-in-a-Haystack and all 16 LongBench datasets.
  • domain assumption Attention scores from the final w tokens are a sufficient signal for token-level eviction.
    Inherited from SnapKV and applied in Eq. 7; if this signal fails, token selection fails regardless of the layer budgets.
  • domain assumption Reallocating a fixed total budget from low-uncertainty to high-uncertainty layers cannot increase information loss more than the uniform allocation.
    This is the core premise of the reallocation in Eq. 5-6; it is supported only by empirical tables, with no bound or analysis.
  • ad hoc to paper A fixed minimum budget Bbound is necessary to prevent starvation in low-uncertainty layers.
    Bbound is added in Eq. 6 as a patch to the proportional allocation; the paper provides no principled derivation of its value.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty." pith.science (2026). https://pith.science/paper/Y3EJWHYL

@misc{pith2026241209036,
  author       = {Pith},
  title        = {Pith review of: ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3EJWHYL}},
  note         = {Machine review of arXiv:2412.09036}
}
abstract

Large Language models (LLMs) have become a research hotspot. To accelerate the inference of LLMs, storing computed caches in memory has become the standard technique. However, as the inference length increases, growing KV caches might lead to out-of-memory issues. Many existing methods address this issue through KV cache compression, primarily by preserving key tokens throughout all layers to reduce information loss. Most of them allocate a uniform budget size for each layer to retain. However, we observe that the minimum budget sizes needed to retain essential information vary across layers and models based on the perspectives of attention and hidden state output. Building on this observation, this paper proposes a simple yet effective KV cache compression method that leverages layer uncertainty to allocate budget size for each layer. Experimental results show that the proposed method can reduce memory usage of the KV caches to only $\sim$20\% when compared to Full KV inference while achieving nearly lossless performance.

Figures

Figures reproduced from arXiv: 2412.09036 by the authors.

Figure 1
Figure 1. Comparison of the proposed method (ZigZagKV, Right) with previous PartialKV inference methods (Left). Most existing PartialKV methods allocate a uniform cache size per layer, whereas ZigZagKV dynamically adjusts the cache size based on layer uncertainty. “L0" and “L31" refer to Layer 0 and Layer 31, respectively. The numbers in brackets represent importance scores from the current token to prefix tokens, sorted from… view at source ↗
Figure 2
Figure 2. LMBA across various Layers of Mistral and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. The evaluation results from Needle-in￾a-HayStack testing across 96, 128, 256, 512 and 1024 budget sizes on Mistral-7B-Instruct-v0.3 and LLaMa-3.1-8B-Instruct. Proposed method ZigZagKV outperforms H2O, SnapKV, PyramidKV and StreamLM, especially in limited budget sizes. layer. PyramidKV (PyramidKV) (Zhang et al., 2024b) proposes a layer-wise retention strategy that reduces cache size per layer based on depth. For KV c… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Performance comparison for Needle-in-a-Haystack testing with a budget size of 256 on Mistral-7B [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies between ZigZagKV w/ Bounded Budget and w/o Bounded Budget on both Mistral and LLaMa on 2WikiMQA Dataset. cache budgets to higher layers, resulting in a significant deviation in attention scores compared to FullKV. In contrast, ZigZagKV substantially re…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 5 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. https://arxiv.org/abs/2308.14508 Longbench: A bilingual, multitask benchmark for long context understanding . ArXiv preprint, abs/2308.14508

    arXiv 2023

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  5. [5]

    Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. 2024. https://arxiv.org/abs/2408.03675 Nacl: A general and effective kv cache eviction framework for llms at inference time . ArXiv preprint, abs/2408.03675

    arXiv 2024

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . ArXiv preprint, abs/2407.21783

    arXiv 2024

  7. [7]

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. https://arxiv.org/abs/2402.10171 Data engineering for scaling language models to 128k context . ArXiv preprint, abs/2402.10171

    arXiv 2024

  8. [8]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3991--4008

    2024

Show all 32 references
  1. [9]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  2. [10]

    Hui Huang, Bing Xu, Xinnian Liang, Kehai Chen, Muyun Yang, Tiejun Zhao, and Conghui Zhu. 2024. Multi-view fusion for instruction mining of large language model. Information Fusion, 110:102480

    2024

  3. [11]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . ArXiv preprint, abs/2310.06825

    2023 arXiv

  4. [12]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. https://doi.org/10.1162/tacl_a_00407 How can we know when language models know? on the calibration of language models for question answering . Transactions of the Association for Computational Linguistics, 9:962--977

    2021 doi

  5. [13]

    Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. 2023. https://arxiv.org/abs/2305.06984 Evaluating open-domain question answering in the era of large language models . ArXiv preprint, abs/2305.06984

    2023 arXiv

  6. [14]

    Greg Kamradt. 2023. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/LLMTest_NeedleInAHaystack

    2023

  7. [15]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. https://arxiv.org/abs/2404.14469 Snapkv: Llm knows what you are looking for before generation . ArXiv preprint, abs/2404.14469

    2024 arXiv

  8. [16]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2024. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information P...

    2024

  9. [17]

    Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. 2024. https://arxiv.org/abs/2401.06104 Transformers are multi-state rnns . ArXiv preprint, abs/2401.06104

    2024 arXiv

  10. [18]

    Bowen Peng and Jeffrey Quesnelle. 2023. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have

    2023

  11. [19]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. https://arxiv.org/abs/2309.00071 Yarn: Efficient context window extension of large language models . ArXiv preprint, abs/2309.00071

    2023 arXiv

  12. [20]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training

    2018

  13. [21]

    Siyu Ren and Kenny Q Zhu. 2024. https://arxiv.org/abs/2402.06262 On the efficacy of eviction policy for key-value constrained generative language model inference . ArXiv preprint, abs/2402.06262

    2024 arXiv

  14. [22]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défoss...

    2023 arXiv

  15. [23]

    Dan Su, Yan Xu, Genta Indra Winata, Peng Xu, Hyeondey Kim, Zihan Liu, and Pascale Fung. 2019. https://doi.org/10.18653/v1/D19-5827 Generalizing question answering system with pre-trained language model fine-tuning . In Proceedings of the 2nd Workshop on Machine Reading for Que...

    2019 doi

  16. [24]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . ArXiv preprint, abs/2...

    2023 arXiv

  17. [25]

    Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. 2024. https://arxiv.org/abs/2406.13035 D2o: Dynamic discriminative operations for efficient generative inference of large language models . ArXiv preprint, abs/2...

    2024 arXiv

  18. [26]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. https://arxiv.org/abs/2309.17453 Efficient streaming language models with attention sinks . ArXiv preprint, abs/2309.17453

    2023 arXiv

  19. [27]

    Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024. https://arxiv.org/abs/2405.12532 Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference . ArXiv preprint, abs/2405.12532

    2024 arXiv

  20. [28]

    Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. 2023 a . https://arxiv.org/abs/2311.07052 Towards the law of capacity gap in distilling language models . ArXiv preprint, abs/2311.07052

    2023 arXiv

  21. [29]

    Chen Zhang, Meizhi Zhong, Qimeng Wang, Xuantao Lu, Zheyu Ye, Chengqiang Lu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang, et al. 2024 a . Modification: Mixture of depths made easy. arXiv preprint arXiv:2410.14268

    2024 arXiv

  22. [30]

    Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. 2024 b . https://arxiv.org/abs/2406.02069 Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling . ArXiv preprint, abs/2406.02069

    2024 arXiv

  23. [31]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. 2023 b . H 2 o: Heavy-hitter oracle for efficient generative inference of large language models.(2023). arXiv preprint cs.LG/2306.14048

    2023 arXiv

  24. [32]

    Meizhi Zhong, Chen Zhang, Yikun Lei, Xikai Liu, Yan Gao, Yao Hu, Kehai Chen, and Min Zhang. 2024. https://arxiv.org/abs/2406.13282 Understanding the rope extensions of long-context llms: An attention perspective . ArXiv preprint, abs/2406.13282

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.