Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DAM claims attention patterns captured from a 512-token window can be extrapolated to arbitrarily long sequences with near-full-attention accuracy.

desk verdict Promising retrieval result, but the efficiency claim collapses under its own table and the method is less dynamic than advertised. read the letter →

arxiv 2506.11104 v1 pith:KMPNV6GS submitted 2025-06-06 cs.CL cs.AI

Hanzhi Zhang , Heng Fan , Kewei Sha , Yan Huang , Yunhe Feng This is my paper
classification cs.CLcs.AI
keywords dynamicsparseattentionlong-contextinferencemasksBox-CoxtransformationretrievalaccuracyLongEvalLV-EvalLLMacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a frozen large language model can run long-context retrieval and question answering at nearly full-attention accuracy while spending far less compute and memory, without fine-tuning and without hand-designing one mask per sequence length. The method, DAM, captures each attention head's structural pattern once from a 512-token window using a contrast-enhancing transformation, then extrapolates the matched diagonal and vertical bands to inputs of up to 104K tokens. The reported numbers put the method within about half a percent of full attention on the LongEval retrieval benchmark (0.7966 versus 0.8011) and within about 0.7 points on LV-Eval at 64K tokens (18.61 versus 19.29), while lowering attention cost from $O(L^2)$ to $O(sL)$. If correct, this makes long-context deployment of existing LLMs feasible on modest hardware without retraining.

What carries the argument

The machinery is the two-stage Dynamic Attention Mask. In stage one, a frozen LLaMA 3.2 model reads sequences truncated to a Pattern Capture Length of $L=512$ tokens; average attention maps are accumulated over the Multi-News dataset, and a Box-Cox transform with $\lambda=0.5$ amplifies small values so thresholding at $\tau=0.3$ yields binary 'true masks' $M_{\ell,h}$. A pattern pool of diagonal patterns ($j=i-r$) and vertical patterns ($j=c$, $i\ge c$) is scored against each true mask via $\gamma_k=\sum_{i,j} M_{\ell,h,i,j}P_{k,i,j} / \sum_{i,j} P_{k,i,j}$; patterns scoring at least $\mu=0.8$ are kept. In stage two, for inputs longer than 512 the kept patterns are pasted into an extended mask $\tilde M_{\ell,h}$ of size $S\times S$, and that mask is applied to the attention logits before softmax, zeroing out all other token pairs so inference computes only the retained connections.

What would settle it

A concrete test is a long-context needle task where the search target sits beyond position 512 and full attention reaches it through an off-diagonal or content-specific connection that is neither diagonal nor vertical; if DAM's retrieval accuracy drops well below full attention in that setting, the transfer premise is falsified. A second, cheaper check is to capture masks from the test domain itself instead of from Multi-News and compare DAM's accuracy; a large gap would show the captured patterns are dataset-dependent rather than universal.

Watch

Extended reading notes

Core claim

The central claim is that pretrained transformer attention maps contain reusable geometric structure—chiefly diagonal and vertical bands—that is stable enough to be captured once at a short length and then extrapolated to thousands of tokens. DAM builds 'true masks' by averaging attention over the Multi-News corpus at a Pattern Capture Length of $L=512$, amplifying small values with a Box-Cox transform, and thresholding at $\tau=0.3$. A pattern pool of diagonal and vertical binary patterns is matched against these true masks with threshold $\mu=0.8$, and the matched patterns are pasted into an extended mask for positions beyond 512. Applied before softmax, the extended mask restricts attention to roughly $s$ keys per query, dropping the per-layer cost from $O(L^2)$ to $O(sL)$; empirically, DAM keeps LongEval average retrieval at 0.7966 versus 0.8011 for full attention.

Load-bearing premise

The load-bearing premise is that the diagonal and vertical attention patterns observed once on 512-token Multi-News documents remain valid for unrelated inputs up to 104K tokens, so pasting those patterns into positions beyond 512 does not discard the long-range connections a task needs.

Editorial extensions

If this is right

  • DAM runs LLaMA 3.2 3B at 8K tokens and Vicuna 7B at 2K tokens where the dense versions run out of memory, while holding retrieval accuracy close to the dense model.
  • On LongEval line retrieval, DAM's average accuracy is 0.7966 versus 0.8011 for full attention, and it tracks full attention across retrieval positions instead of degrading past 20K tokens like MoA, H2O, and StreamingLLM.
  • On LV-Eval QA at 64K tokens, DAM scores 18.61 versus 19.29 for full attention, outperforming MoA (7.56), H2O (7.59), and StreamingLLM (7.47).
  • Because masks are generated once and applied before softmax, the complexity becomes $O(sL)$ with $s\ll L$, and the sparse layout is compatible with tile-based GPU kernels such as FlashAttention for further fusion.
  • No fine-tuning and no per-length predefined masks are required: the same pattern pool serves all tested sequence lengths and model sizes (1B, 3B, 7B).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the capture pass runs once on Multi-News, DAM's advantage grows with how often the same frozen model serves long sequences; for one-shot short queries the preprocessing cost may exceed the savings.
  • Editorial inference: the pattern pool contains only diagonal and vertical structures, so attention that is genuinely content-addressed and off-diagonal at long distance cannot be represented; extending the pool with data-driven motifs or retrieval-based sparsity is a natural next step.
  • Editorial inference: a testable extension is to vary the Pattern Capture Length (e.g., 256 or 1024) and the capture corpus; if accuracy on long inputs is insensitive to these choices, the extrapolation claim is stronger than if it is tightly tied to the chosen 512-token window.
  • Editorial inference: the paper's efficiency table shows FlashAttention's 8K throughput spike is tied to GPU tiling alignment; an independent benchmark controlling tiling would clarify whether DAM's predictable scaling generalizes beyond the reported configuration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DAM, a fine-tuning-free sparse-attention method for long-context LLM inference. DAM first runs a frozen model on Multi-News inputs truncated to a Pattern Capture Length (PCL=512), averages attention maps across batches, applies a Box-Cox transform, thresholds the result to obtain per-layer/per-head binary 'true masks,' and then matches diagonal and vertical templates from a predefined pattern pool. For sequences longer than the PCL, the matched patterns are extrapolated to construct an S×S mask, which is applied before softmax as A'=(QK^T/√d_k)⊙M. The paper reports LongEval retrieval accuracy close to full attention (0.7966 vs 0.8011), LV-Eval scores at 64K (18.61 vs 19.29), and claims a complexity reduction from O(S²) to O(sL), with memory and latency comparisons in Table 1.

Significance. If the accuracy results are reliable, DAM is a noteworthy contribution: it assigns heterogeneous, head- and layer-specific sparse masks without fine-tuning, it provides public code, and it compares against several sparse-attention baselines on long retrieval tasks. The idea of capturing mask structure on short sequences and extrapolating it to longer ones is interesting and testable. However, the significance is currently limited by three issues: the efficiency claim is not supported by the reported implementation or measurements, the accuracy claim lacks error bars, and the cross-domain transfer of a fixed diagonal/vertical pattern pool is not validated. As submitted, the paper demonstrates a plausible accuracy-preserving sparsity pattern but does not demonstrate the acceleration that motivates the method.

major comments (4)
  1. [Section 4.3, Section 5.3, Table 1] The efficiency half of the central claim is not supported. The masked score is computed as A'_{ℓ,h}=(Q_{ℓ,h}K_{ℓ,h}^T/√d_k)⊙M_{ℓ,h}, which is a full S×S QK^T product followed by elementwise masking; without a sparse kernel this is O(S²), so the O(sL) statement in Section 5.3 is asserted rather than derived or measured. Table 1 directly contradicts the abstract's 'reducing memory and compute overhead' when compared with FlashAttention: for LLaMA-3.2-3B at 8K, DAM uses 31.71 GB and 238 tokens/s, while FlashAttention uses 21.15 GB and 25,796 tokens/s; the 1B row shows the same pattern (10.64 GB/640 tok/s vs 10.21 GB/69,823 tok/s). The text explains the FlashAttention throughput as a 'GPU tiling effect,' but this is not a controlled comparison. The authors should either implement and benchmark an actual sparse attention kernel, reporting prefill vs decode, batch size, and kernel details, or substantially weaken the efficiency claims.
  2. [Sections 4.2.1, 4.2.4, 4.3 Case 2] The central extrapolation assumption is not tested. The pattern pool is derived from Multi-News attention maps at PCL=512, and Case 2 pastes the matched diagonal and vertical patterns onto all positions beyond the first 512 tokens. The paper never measures how well these extrapolated masks agree with the target model's full attention on LongEval or LV-Eval inputs, nor does it ablate the capture corpus or the PCL. Appendix A itself reports transient, length-dependent structures (e.g., sliding windows that fade by 256 tokens), so the sufficiency of the fixed diagonal/vertical pool is not established. Please add mask-agreement metrics or oracle-mask comparisons on long inputs, and vary the capture corpus and PCL to show the extrapolation is robust.
  3. [Section 5.2, Figures 4 and 6] The statistical support for 'comparable' performance is thin. The LongEval gap is about 0.005 (0.7966 vs 0.8011), but no confidence intervals, multiple seeds, or per-length variance are reported, so it is impossible to tell whether the gap is within noise. In addition, Figure 4 and the text disagree on the maximum tested length (3.1k–38.7k tokens in the caption versus 'up to 104K tokens' in the text), and LV-Eval results are described for 16K–256K but reported only up to 64K. The authors should report the full length range with error bars and align the stated and plotted ranges.
  4. [Section 4.2 and DAM Configuration] The method depends on four manually set hyperparameters—PCL=512, Box-Cox λ=0.5, true-mask threshold τ=0.3, and pattern-match threshold μ=0.8—but no sensitivity analysis is provided. The claim that μ is robust between 0.7 and 1.0 is stated without supporting experiments. Since the sparsity level and the extrapolated masks are fully determined by these values, the paper should report how retrieval accuracy and mask sparsity vary with each hyperparameter, preferably across multiple datasets.
minor comments (6)
  1. [Sections 4.2.2 and Appendix B] The numerical stabilizer ε is given as 10⁻⁸ in Section 4.2.2 and as 10⁻¹⁰ in Appendix B; the two values should be unified.
  2. [References] The LongEval citation (Krishna et al., 2023) appears to be the summarization-faithfulness benchmark rather than the line-retrieval benchmark used in this paper; please verify and cite the correct source.
  3. [Table 1 and Section 4.2.1] Table 1 shows the original LLaMA-3.2-3B model running out of memory at 4K tokens, while Section 4.2.1 states that OOM occurs only 'when processing sequences longer than 8k tokens'; these statements should be reconciled.
  4. [Figures 4 and 9] The maximum reported length is inconsistent: Figure 4's caption says 3.1k–38.7k tokens, the main text says up to 104K tokens, and Figure 9 shows results only to 40K tokens. Please align the text, captions, and plots.
  5. [References] The Ainslie et al. citation is dated 2004 in the bibliography; the ETC paper appeared in 2019/2020. Please correct the year and venue.
  6. [Section 5.3 and Table 1] The efficiency benchmark would benefit from reporting batch size and whether the timings correspond to prefill or decode; the unusually high FlashAttention throughput values (e.g., 69,823 tokens/s for the 1B model at 8K) are difficult to interpret without this context.

Circularity Check

2 steps flagged · score 4.0 of 10

Retrieval transfer is independent, but 'dynamic' masks are defined as a union of two predefined patterns and O(sL) efficiency is asserted, not derived.

  1. renaming known result [Section 4.2.4, 'Dynamic Mask Generation via Structural Pattern Matching' (pattern pool definition; also relates to Section 4.2 and abstract claim of no predefined masks)]
    "The pattern pool, in this work, includes diagonal and vertical patterns, reflecting common attention structures observed in Transformer models. ... The complete pattern pool is the union of these sets: P={P diag,r} ∪ {Pvert,c}."

    The extended mask is defined as the sum of matched patterns from this pool, so every DAM mask is by construction a union of diagonal and vertical patterns. Diagonal/vertical are the same shapes as sliding-window and global-token attention that the related work lists as static predefined masks. Thus the paper's central novelty ('dynamic, adaptive, no predefined mask structures') is self-definitional: the method can only output the two predefined shapes, so 'heterogeneous pattern discovery' is equivalent to the pool definition, not an independent finding.

  2. other [Section 4.3 (Applying Dynamic Attention Masks) and Section 5.3 (Efficiency)]
    "The attention mask applies before softmax. The modified attention score matrix is: A′ℓ,h = Qℓ,hKTℓ,h√dk ⊙ M̃ℓ,h. ... This reduces FLOPs complexity from O(L2) to O(sL), where s is the average number of retained keys per query (s≪L)."

    The formula specifies a full S×S QK^T product followed by masking; no sparse kernel is defined anywhere, and §5.3 only says sparse layout is 'compatible' with future fusion. The claimed O(sL) reduction is not implied by the dense equation—it is an additional assumption. The efficiency half of the central claim ('reducing memory and compute overhead') is therefore an asserted target rather than a derived consequence of the method as specified, and Table 1's FlashAttention comparisons do not support it (e.g., 3B at 8K: DAM 31.71 GB/238 tok/s vs FlashAttention 21.15 GB/25,796 tok/s).

full rationale

The retrieval-accuracy claim is not circular in the strict sense: DAM's masks are built from Multi-News attention maps of the base model, while LongEval/LV-Eval results are produced on unseen inputs, and the benchmark full-attention scores are not used to choose the masks or thresholds. The agreement with full attention is an empirical transfer result, not a tautology. However, the 'dynamic/adaptive' contribution is partly self-definitional: the method selects from a hand-built pattern pool containing only diagonal and vertical shapes, so the claim that predefined mask structures are eliminated is essentially a renaming of the known sliding-window plus global-token sparse-attention family. The O(sL) complexity claim is also not derived from the paper's own dense masked-score equation, and Table 1 shows DAM slower and more memory-hungry than FlashAttention at 8K for the 3B model; the paper attributes this to 'GPU tiling effects,' but that does not establish the claimed efficiency improvement. These problems affect the novelty and efficiency claims, but they do not reduce the LongEval comparison itself to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on several untested design choices: a calibration dataset (Multi-News), a capture length of 512, two thresholds tau and mu, a Box-Cox parameter lambda, and a predefined pattern pool limited to diagonal and vertical structures. None of these is derived from first principles, and their interaction determines the final sparsity patterns.

free parameters (4)
  • PCL (Pattern Capture Length) = 512
    Selected as the longest sequence length the A100 40GB GPU supports for full attention on the LLaMA 3.2 3B model; it defines the window from which all true masks are extracted and is therefore load-bearing for the extrapolation step.
  • Box-Cox lambda = 0.5
    Chosen because it improves visualization of attention maps; the paper claims it does not need future tuning, but it directly controls the amplified attention values used for thresholding.
  • True mask threshold tau = 0.3
    Determined through attention sparsity analysis on Multi-News; controls which connections survive into the true masks and affects all downstream pattern matching.
  • Pattern match threshold mu = 0.8
    Hyperparameter controlling pattern matching sensitivity; authors note robustness over 0.7 to 1.0, but it is still a design choice that determines extended mask density.
assumptions (4)
  • domain assumption Attention patterns observed in Multi-News at length no more than 512 generalize to LongEval and LV-Eval inputs at lengths up to 104K tokens.
    Masks are generated from Multi-News averages and applied to other datasets without per-input adaptation, as specified in Section 5.1 DAM Configuration.
  • ad hoc to paper The predefined pattern pool of diagonal and vertical patterns is sufficient to capture the attention structure needed for retrieval.
    Section 4.2.4 defines P as a union of diagonal and vertical patterns; no evidence is given that other pattern families are unnecessary.
  • domain assumption Structural patterns within the first L tokens can be extrapolated to all positions beyond L.
    Section 4.3 Case 2 extends masks by replicating matched patterns to all i,j beyond L; Appendix A provides only qualitative evidence for this extrapolation.
  • domain assumption Mean attention scores across batches are a stable proxy for individual-sequence attention structure.
    Section 4.2.2 averages attention over batches; per-sequence dynamic masking is not actually performed, so the average must be representative for the method to work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration." pith.science (2026). https://pith.science/paper/KMPNV6GS

@misc{pith2026250611104,
  author       = {Pith},
  title        = {Pith review of: DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KMPNV6GS}},
  note         = {Machine review of arXiv:2506.11104}
}
read the original abstract

Long-context understanding is crucial for many NLP applications, yet transformers struggle with efficiency due to the quadratic complexity of self-attention. Sparse attention methods alleviate this cost but often impose static, predefined masks, failing to capture heterogeneous attention patterns. This results in suboptimal token interactions, limiting adaptability and retrieval accuracy in long-sequence tasks. This work introduces a dynamic sparse attention mechanism that assigns adaptive masks at the attention-map level, preserving heterogeneous patterns across layers and heads. Unlike existing approaches, our method eliminates the need for fine-tuning and predefined mask structures while maintaining computational efficiency. By learning context-aware attention structures, it achieves high alignment with full-attention models, ensuring minimal performance degradation while reducing memory and compute overhead. This approach provides a scalable alternative to full attention, enabling the practical deployment of large-scale Large Language Models (LLMs) without sacrificing retrieval performance. DAM is available at: https://github.com/HanzhiZhang-Ulrica/DAM.

Figures

Figures reproduced from arXiv: 2506.11104 by the authors.

Figure 1
Figure 1. Attention patterns from queries × keys. The short input sequence "paper boats sailed across a puddle of stardust" is a subset of the longer input sequence "paper boats sailed across a puddle of stardust drifting toward the moon". (a) Each query attends to all keys, and longer attention patterns are extensions of short patterns. (b) Static attention map captures same classical global and sliding-window patterns to at… view at source ↗
Figure 2
Figure 2. Visualization of dynamic attention patterns [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Two-stage DAM overview. The first stage extracts full attention patterns from sequences of varying lengths, applies a Box-Cox transformation, and generates masks that capture essential dependencies. The second stage applies these masks to a sparse model, enabling efficient inference while preserving key attention structures. observed attention patterns and extrapolating struc￾tural regularities, DAM achieves a balan… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Line-level retrieval accuracy on the LongEval [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Retrieval accuracy on LongEval for LLaMA 3.2 3B and 1B models. Even at fixed token lengths, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: LV-Eval retrieval score across long-context QA tasks. DAM closely matches full attention, achieving [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Attention patterns from the LLaMA 3.2 3B [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Feature amplification examples across six attention maps from the LLaMA 3.2 3B model under nine [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: LongEval retrieval accuracy for LLaMA 3.2 3B and 1B models across input lengths to 40K tokens. DAM [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Overview of Algorithms for Contactless Cardiac Feature Extraction from Radar Signals: Advances and Challenges

    eess.SP 2025-08 unverdicted novelty 4.0 of 10

    The paper claims to be the first review of signal-processing algorithms for radar-based cardiac monitoring, with a new taxonomy and public dataset listings.

Reference graph

Works this paper leans on

41 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114--127

    work page 2024

  2. [2]

    Meta AI. 2024. https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md Llama 3.2 model card

    work page 2024

  3. [3]

    Joshua Ainslie, Santiago Onta \ n \'o n, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. 2004. Etc: encoding long and structured data in transformers. CoRR, abs

    work page 2004

  4. [5]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 b . https://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . Preprint, arXiv:2004.05150

    arXiv 2020

  5. [6]

    G. E. P. Box and D. R. Cox. 1964. https://doi.org/10.1111/j.2517-6161.1964.tb00553.x An analysis of transformations . Journal of the Royal Statistical Society: Series B (Methodological), 26(2):211--252

    arXiv 1964

  6. [7]

    Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. 2024. Nacl: A general and effective kv cache eviction framework for llms at inference time. arXiv preprint arXiv:2408.03675

    arXiv 2024

  7. [8]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509

    arXiv 2019

  8. [9]

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, et al. 2020. Masked language modeling for proteins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03555

    arXiv 2020

Show all 41 references
  1. [10]

    Correia, Vlad Niculae, and André F

    Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. 2019. https://arxiv.org/abs/1909.00015 Adaptively sparse transformers . Preprint, arXiv:1909.00015

    2019 arXiv

  2. [11]

    Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691

    2023 arXiv

  3. [12]

    Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R

    Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019. https://arxiv.org/abs/1906.01749 Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model . CoRR, abs/1906.01749

    2019 arXiv

  4. [13]

    Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. 2024. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406.14909

    2024

  5. [14]

    Semsa: Semantic sparse attention is hidden in large language models

    Tianyu Fu, Xuefei Ning, Boju Chen, Tianqi Wu, Genghan Zhang, Guohao Dai, Huazhong Yang, and Yu Wang. Semsa: Semantic sparse attention is hidden in large language models

  6. [15]

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. \ Cost-Efficient \ large language model serving for multi-turn conversations with \ CachedAttention \ . In 2024 USENIX Annual Technical Conference (USENI...

    2024

  7. [16]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801

    2023 arXiv

  8. [17]

    Akshay Goindani and Manish Shrivastava. 2021. https://arxiv.org/abs/2108.01377 A dynamic head importance computation mechanism for neural machine translation . Preprint, arXiv:2108.01377

    2021 arXiv

  9. [18]

    Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180

    2019 arXiv

  10. [19]

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, et al. 2024. Memserve: Context caching for disaggregated llm serving with elastic memory pool. arXiv preprint arXiv:2406.17565

    2024 arXiv

  11. [20]

    Nikita Kitaev, ukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451

    2020 arXiv

  12. [21]

    Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. arXiv preprint arXiv:2301.13298

    2023 arXiv

  13. [22]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469

    2024 arXiv

  14. [23]

    Yichao Liu, Zongru Shao, and Nico Hoffmann. 2021. https://arxiv.org/abs/2112.05561 Global attention mechanism: Retain information to enhance channel-spatial interactions . Preprint, arXiv:2112.05561

    2021 arXiv

  15. [24]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2024. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information P...

    2024

  16. [25]

    Ramesh Pingili. 2025. Ai-driven intelligent document processing for banking and finance

    2025

  17. [26]

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53--68

    2021

  18. [27]

    Yi Tay, Aston Zhang, Luu Anh Tuan, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. 2019. Lightweight and efficient neural natural language processing with quaternion networks. arXiv preprint arXiv:1906.04393

    2019 arXiv

  19. [28]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  20. [29]

    Dongsheng Wang, Casper Hansen, Lucas Chaves Lima, Christian Hansen, Maria Maistro, Jakob Grue Simonsen, and Christina Lioma. 2020. https://arxiv.org/abs/2012.12366 Multi-head self-attention with role-guided masks . Preprint, arXiv:2012.12366

    2020 arXiv

  21. [30]

    Da Xiao, Qingye Meng, Shengping Li, and Xingyuan Yuan. 2024 a . https://arxiv.org/abs/2405.08553 Improving transformers with dynamically composable multi-head attention . Preprint, arXiv:2405.08553

    2024 arXiv

  22. [31]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024 b . https://arxiv.org/abs/2309.17453 Efficient streaming language models with attention sinks . Preprint, arXiv:2309.17453

    2024 arXiv

  23. [32]

    Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang, Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, and Zhenxuan Pan. 2024. Layerkv: Optimizing large language model serving with layer-wise kv cache management. arXiv preprint arXiv:2410.00428

    2024 arXiv

  24. [33]

    Lu Ye, Ze Tao, Yong Huang, and Yang Li. 2024. Chunkattention: Efficient self-attention with prefix-aware kv cache and two-phase partition. arXiv preprint arXiv:2402.15220

    2024 arXiv

  25. [34]

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. https://arxiv.org/abs/2402.05136 Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k . Prepr...

    2024

  26. [35]

    Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. 2020. O (n) connections are expressive enough: Universal approximability of sparse transformers. Advances in Neural Information Processing Systems, 33:13783--13794

    2020

  27. [36]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

  28. [37]

    Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John Lui, and Haibo Chen. 2024 a . Unifying kv cache compression for large language models with leankv. arXiv preprint arXiv:2412.03131

    2024 arXiv

  29. [38]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\' e , Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f...

    2023

  30. [39]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. 2024 b . H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Pro...

    2024

  31. [40]

    Junqi Zhao, Zhijin Fang, Shu Li, Shaohui Yang, and Shichao He. 2024. Buzz: Beehive-structured sparse kv cache with segmented heavy hitters for efficient llm inference. arXiv preprint arXiv:2410.23079

    2024 arXiv

  32. [41]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024 a . Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104

    2024 arXiv

  33. [42]

    Zhen Zheng, Xin Ji, Taosong Fang, Fanghao Zhou, Chuanjie Liu, and Gang Peng. 2024 b . Batchllm: Optimizing large batched llm inference with global prefix sharing and throughput-oriented token batching. arXiv preprint arXiv:2412.03594

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.