Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Training with full and sparse attention aligned in both directions lets a sparse-attention model match full-attention quality while running at a fraction of the cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:20 UTC pith:CIC3KOK5

load-bearing objection Useful training recipe for switchable sparse/full attention, but the theory is circular and the RF=1024 SOTA is an uncontrolled extrapolation. the 3 major comments →

arxiv 2511.20102 v4 pith:CIC3KOK5 submitted 2025-11-25 cs.CL

SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space

Zhenyi Shen , Junru Lu , Lin Gui , Jiazheng Li , Yulan He , Di Yin , Xing Sun This is my paper
classification cs.CL
keywords sparse attentionattention gapcapability gapattention-output alignmentdual-stream trainingblock-sparse attentionlong-context language modelsattention sparsity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's method, SSA (Sparse Sparse Attention), is a training framework meant to settle a two-sided failure of sparse attention. A model trained on full attention loses quality when switched to sparse inference, because training and inference distributions no longer match; a model trained only on sparse attention never gets gradients from dropped tokens and has no reference that defines how far it has drifted from full attention. SSA trains a single model with both streams, randomly picking full or block-sparse attention at each step, and adds a bidirectional alignment loss between the two streams' hidden states at every layer. The paper proves that the difference between full and sparse attention outputs is bounded linearly by the attention mass the sparse selection drops, so encouraging inherently sparser attention tightens the bound. Experiments on 1B-scale models show SSA matches or beats full-attention training in full-inference mode and beats sparse-only training in sparse-inference mode, while degrading smoothly as the sparsity budget shrinks. This matters because it offers a route to long-context language models that keep dense-model quality without paying quadratic inference cost.

Core claim

The paper's central claim is that the attention gap and the capability gap are two halves of one fixable problem. Switching a full-attention-trained model to sparse inference adds a KL penalty; sparse-only training gives dropped tokens zero gradient and no reference to full attention. SSA trains with both modes and, at every layer, aligns the full and sparse hidden states bidirectionally using SmoothL1 with stop-gradient. The quantitative core is Theorem 1: the output difference between full and sparse attention is at most the dropped attention mass times a value-vector term, so concentrating attention mass inside selected blocks directly shrinks the approximation error. The objective is bui

What carries the argument

The central mechanism is bidirectional attention-output alignment at every layer. Training randomly picks full or sparse attention per step; the chosen stream propagates, the counterpart is computed only for alignment. A sparsity loss pulls full outputs toward stop-gradiented sparse outputs; a commitment loss pulls sparse outputs toward stop-gradiented full outputs, using SmoothL1 on hidden states. This aligns value-space outputs, not attention maps, so dense weights are never materialized. The supporting identity, Theorem 1, bounds the full–sparse output gap by δ(t)(max_{j∈S^c(t)}||v(j)|| + ||h_sparse||), where δ(t) is the dropped attention mass; sparser attention shrinks this bound.

Load-bearing premise

The load-bearing premise is that the per-layer bidirectional hidden-state alignment loss is a stable and effective training signal; the paper's own ablations show that using only one alignment direction, or sampling the stream randomly per layer, drives training loss to NaN, so the joint training stability is an empirical property rather than a guaranteed one.

What would settle it

Train SSA at a substantially larger scale (say, a 7B or 13B model on several hundred billion tokens) under the same protocol and compare sparse-inference perplexity and commonsense accuracy against the best full-only and sparse-only baselines; if the alignment loss destabilizes training or the sparse model fails to match full-attention quality on both benchmark families, the central claim fails. A more direct check: measure the dropped attention mass δ(t) and the bound in Theorem 1 layer by layer; if SSA's attention is not sparser than baselines on held-out data, the mechanism's stated route t

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A single model trained with SSA can be run in either full or sparse mode and retain near-full quality in both, giving practitioners a compute-quality switch.
  • The sparse mode degrades gracefully as the receptive field is reduced or expanded, so one checkpoint can serve many sparsity budgets without retraining.
  • Because SSA lowers dropped attention mass, the theoretical bound implies the sparse/full output gap shrinks under inference, which the perplexity results support.
  • Sparse attention trained this way is more resilient to RoPE length scaling than full attention, because sparser distributions depend less on distorted positional frequencies.
  • Inference cost falls roughly 2× at 128k context under sparse mode while training cost increases only about 17% over sparse-only training, making the trade attractive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not pursue: the same bidirectional hidden-state alignment should transfer to other sparse schemes—token-level top-k selection, sliding windows, or mixtures—since the bound only depends on dropped attention mass, not on how blocks are chosen.
  • The result that sparser attention improves full-attention benchmarks suggests attention-sparsity regularization could help dense-only training as well, independent of any plan to run sparsely at inference.
  • If the alignment loss scales past 1B parameters—the paper's ablations show stability is delicate—SSA-type training could replace dense pretraining for long-context models, with the sparse stream as a built-in efficiency mode after training.
  • The two-stream setup resembles a distillation loop with no separate teacher: full attention and sparse attention regularize each other, which might make SSA useful as a compression mechanism where a small model is trained from scratch under this self-alignment rather than distilled post hoc.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SSA (Sparse Sparse Attention), a training framework that alternates between full-attention and block-sparse-attention streams with equal probability and, at every layer, aligns the two streams' hidden representations using a bidirectional SmoothL1 loss with stop-gradient. The authors claim a theoretical framework in which pure sparse training causes a gradient/capability gap, pure full training causes an attention-gap KL penalty, and the approximation error of sparse attention scales linearly with dropped attention mass. They report experiments with 1B and 300M models pre-trained on 100B/50B tokens, evaluating perplexity, commonsense reasoning, sparsity extrapolation, LongBench, NIAH, and PG-19, and claim state-of-the-art performance under both full and sparse inference modes.

Significance. If the empirical results hold, SSA would be a useful contribution: it offers a single training recipe that supports both efficient sparse inference and accurate full inference, with flexible sparsity budgets. The paper has clear strengths: it releases code, reports 5-run statistics for the main commonsense results, performs extensive ablations (sampling ratio, alignment direction, alpha, loss type, sparsity level), and measures training/inference efficiency. The attention-sparsity analysis in Appendix H and L is informative. However, the headline RF=1024 comparison is not a controlled experiment, the theoretical Proposition 4.2 is a definitional identity rather than a substantive bound, and the long-context results are single-run. These issues materially reduce confidence in the paper's central 'state-of-the-art under both inference modes' claim, although the core method remains plausible and worth further investigation.

major comments (3)
  1. [§6, Table 1, Table 3, §7.5] The SSA RF=1024 results are not a controlled comparison. The paper states in §6 that 'we report the 1024-receptive-field results obtained by extrapolating the 256-receptive-field model, as that model consistently performs better,' while MoBA and NSA RF=1024 baselines are natively trained at that receptive field. Table 1's SSA 1024 row and Table 3's sparse-inference 1024 rows therefore evaluate an extrapolated model against natively trained baselines. The ablation in Table 2 (train16×64, Comm. Avg. 49.16, vs baseline inf16×64, 49.96) suggests a natively trained 1024 SSA underperforms the extrapolated baseline, so the advantage in the headline RF=1024 rows may be an artifact of model selection rather than the training method. Please report natively trained RF=1024 SSA results, or explicitly relabel the comparison as extrapolated and rephrase the SOTA claims accordingly.
  2. [§4.2, Proposition 4.2, Appendix A] Equation (2) is a tautology, not a derived penalty. Appendix A defines D_KL(p_full_θ || p_sparse_θ) = E_FA[-log p_sparse_θ] - E_FA[-log p_full_θ] = L_sparse_FA - L_full_FA. Substituting this definition into Eq. (2) proves nothing beyond the definition of KL divergence. The proposition does not establish that full-only training incurs an attention gap; it presupposes the distribution mismatch and asserts the identity. The empirical observation that MoBA has lower sparse-inference PPL than FullAttn is evidence, but the claimed proof is vacuous. Please reframe this as a definitional identity or provide a substantive bound relating the KL divergence to a mechanism that full-only training would create.
  3. [§8, Table 3, Appendix E] LongBench and NIAH results, which support the paper's 'superior long-context capabilities' claim, are from a single run. Appendix E states: 'we use greedy decoding and report results from a single run due to computational constraints—our sparse attention kernel does not currently support KV caching... Note that NIAH results may vary slightly across runs as the evaluation data is synthesized dynamically.' Without variance estimates or at least 2-3 runs, the small differences in Table 3 (e.g., LongBench averages 18.58 vs 18.23, or NIAH 34.8 vs 27.6) cannot be assessed for reliability. For a paper making SOTA claims in long-context settings, single-run results are insufficient; please provide multi-seed results or clearly mark these as preliminary.
minor comments (5)
  1. [§4.2] Typo/grammar: 'we didn't specify what kind of training attention well choose' should be 'we do not specify what kind of training attention we choose'.
  2. [Author list] The author name 'Y ulan He' has an erroneous space; it should be 'Yulan He'.
  3. [Appendix A] The notation E_FA[·] is not explicitly defined. It is used to denote expectation under full-attention inference of the full-trained model, but the subscript 'FA' is ambiguous because the same letters are used for training and inference modes.
  4. [Table 3] The table formatting is difficult to read: several entries appear without separators (e.g., '100 1004835.470.9' and '48 35 4 70.9'). Please use consistent spacing and avoid merging numbers.
  5. [§6, Appendix C] The use of Gated Attention for all models is a nonstandard modification to the FullAttn baseline. It is applied uniformly for fairness, which is reasonable, but the paper should prominently state that FullAttn here includes a gating mechanism, since readers may otherwise compare against standard Llama-style attention.

Circularity Check

1 steps flagged

Proposition 4.2's 'attention-gap penalty' is the definition of KL divergence rearranged; the empirical SSA results remain independent.

specific steps
  1. self definitional [Section 4.2 (Proposition 4.2, Eq. 2) and Appendix A]
    "Proposition 4.2 ... For a model trained only with full attention, sparse inference incurs an additional KL-divergence penalty: L_sparse_FA(t) = L_full_FA(t) + D_KL(p_full_θ(t) ∥ p_sparse_θ(t)). (2) ... Appendix A: 'D_KL(p_full_θ(t)∥p_sparse_θ(t)) = E_FA[−log p_sparse_θ(t)] − E_FA[−log p_full_θ(t)] = L_sparse_FA(t) − L_full_FA(t). Rearranging terms yields: L_sparse_FA(t) = L_full_FA(t) + D_KL(...).'"

    The appendix defines D_KL as L_sparse_FA − L_full_FA, so Eq. (2) is exactly the definition of KL divergence rearranged; the 'penalty' is not derived from any property of attention. As a result, the proposition cannot by itself establish that full-only training suffers a distribution-mismatch penalty—any two distributions would satisfy the identity. The later claim that sparse training 'eliminates the KL-divergence penalty' inherits this definitional status. The empirical PPL gap in Table 1 is independent evidence, so the circularity is partial rather than total.

full rationale

The only identifiable reduction-by-construction is Proposition 4.2: Eq. (2) restates the definition of KL divergence in terms of the two losses, so the 'attention-gap penalty' is definitionally equal to the loss difference. This is a genuine circular step in the theoretical framework and is load-bearing for the paper's claim that full-only training incurs a formal penalty. However, the core SSA contribution (dual-stream training with bidirectional alignment) and its validation are grounded in experiments: Table 1 independently shows the FullAttn sparse/full PPL gap, Theorem 1 is an algebraic bound derived from attention renormalization, and the ablations do not rely on Proposition 4.2. The RF=1024 extrapolation choice in Section 6 is a controlled-comparison concern rather than circularity. Overall score 6 reflects one central theoretical identity reducing by construction while the empirical claims remain largely independent.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The only new entity is the SSA training procedure itself; no new model components or physical entities are posited. Free parameters are standard hyperparameters. The main assumptions are about the sufficiency of block-sparse attention and the stability of the alignment objective.

free parameters (3)
  • alpha (alignment weight) = 10 for SmoothL1; 5 for L2
    Chosen by hand after ablations (Table A14); balances the cross-entropy and alignment objectives.
  • FullRatio (sampling probability of full stream) = 0.5
    Default mixing ratio for dual-stream sampling; ablations show sensitivity but no principled derivation.
  • Receptive field / block pattern = RF=256 (block size 16, top-16 blocks) for main 1B model
    Selected based on 300M ablations as a sweet spot; smaller/larger RFs hurt performance (Table 2).
axioms (4)
  • domain assumption Block-sparse attention with top-k block selection via mean-pooled block keys is the sparse pattern under study.
    Section 3; the entire method is built on this specific sparsity pattern, so results may not transfer to other sparse patterns.
  • domain assumption The dropped attention mass δ(t) is the dominant quantity controlling the full-vs-sparse approximation error.
    Theorem 1 provides an upper bound, but the bound is not tight; the paper assumes that reducing δ(t) is sufficient for reducing error (Eq. 5).
  • domain assumption Aligning hidden representations (not attention maps) with a SmoothL1 loss reduces the output distribution mismatch between full and sparse inference.
    Section 5; this is an empirical design choice validated only by experiments, not by derivation.
  • ad hoc to paper Proposition 4.2's identity (Eq. 2) holds for the actual training loss.
    The proof in Appendix A defines the expectation under the model's own full-attention distribution, making the claim a definitional identity rather than a property of standard NLL training; the paper relies on it to formalize the attention gap.

pith-pipeline@v1.3.0-alltime-deepseek · 22433 in / 19604 out tokens · 187990 ms · 2026-08-03T20:20:57.726301+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space." pith.science (2026). https://pith.science/paper/CIC3KOK5

@misc{pith2026251120102,
  author       = {Pith},
  title        = {Pith review of: SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CIC3KOK5}},
  note         = {Machine review of arXiv:2511.20102}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a training framework that integrates both sparse and full attention with bidirectional attention-output alignment. We prove that the approximation error scales linearly with the attention mass dropped under sparse attention, and show that SSA's alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities.

Figures

Figures reproduced from arXiv: 2511.20102 by Di Yin, Jiazheng Li, Junru Lu, Lin Gui, Xing Sun, Yulan He, Zhenyi Shen.

Figure 1
Figure 1. Figure 1: Performance and attention characteristics of 1B models. Under sparse inference, Sparse Attention Model (SA) achieves lower perplexity than Full Attention Model (FA) due to distribution mismatch, yet FA outperforms SA on benchmarks due to weaker training signals in SA. SSA resolves both issues, achieving the highest attention sparsity and best performance under both inference modes. is not involved during t… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the SSA training framework. At each iteration, the model has an equal probability of following either the Sparse Attention (SA) stream or the Full Attention (FA) stream. In the SA stream, the model learns sparse attention while aligning its output with a full-attention counterpart computed on the fly. Conversely, in the FA stream, the model learns full attention constrained by alignment wit… view at source ↗
Figure 3
Figure 3. Figure 3: Performance versus receptive-field size. SSA and FullAttn extrapolate well as more tokens become visible, whereas MoBA exhibits poor extrapolation. We scale the receptive field by fixing block size and varying top-k. Detailed scores are in Appendix I. 7.4 Long Context Evaluation Language Modeling. On longer context tasks, [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Perplexity across context lengths under different inference modes tested on PG-19. SSA achieves the lowest perplexity under sparse attention inference and exhibits the strongest robustness to RoPE scaling under full attention. Needle-in-a-Haystack (NIAH). In [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention

    cs.CL 2026-07 conditional novelty 6.0

    CoSA couples an order-aware sparse-mask proxy with a skip-capable attention kernel, achieving higher long-context accuracy at lower compute budgets than existing sparse-attention baselines.

Reference graph

Works this paper leans on

39 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [1]

    RepoCoder: Repository-level code completion through iterative retrieval and generation

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471–2484...

    doi:10.18653/v1/2023.emnlp-main.151 2023

  2. [2]

    SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= VTF8yNQM66

    2024

  3. [3]

    Openai o1 system card, 2024

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, and et al. Openai o1 system card, 2024. URL https://arxiv.org/ abs/2412.16720

    Pith/arXiv arXiv 2024

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, and et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URLhttps://arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  5. [5]

    DeepResearcher: Scaling deep research via reinforcement learning in real-world environments

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. DeepResearcher: Scaling deep research via reinforcement learning in real-world environments. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025. doi: 10.18653/v1/2025.emnlp-main.22. URLhttps://aclanthology.org/2025.em...

    doi:10.18653/v1/2025.emnlp-main.22 2025

  6. [6]

    Qwen2.5-1m technical report, 2025

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1m technical re...

    Pith/arXiv arXiv 2025

  7. [7]

    Efficient attention mechanisms for large language models: A survey, 2025

    Yutao Sun, Zhenyu Li, Yike Zhang, Tengyu Pan, Bowen Dong, Yuyi Guo, and Jianyong Wang. Efficient attention mechanisms for large language models: A survey, 2025. URL https://arxiv.org/abs/2507. 19595

    2025

  8. [8]

    InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=bTHFrqhASY

    2024

  9. [9]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=NG7sS51zVF

    2024

  10. [10]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerat- ing pre-filling for long-context LLMs via dynamic sparse attention. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:/...

    2024

  11. [11]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehva...

    2025

  12. [12]

    MoBA: Mixture of block attention for long-context LLMs

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. MoBA: Mixture of block attention for long-contex...

    2025

  13. [13]

    Generating long sequences with sparse transformers, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019. URLhttps://arxiv.org/abs/1904.10509

    Pith/arXiv arXiv 2019

  14. [14]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URLhttps://arxiv.org/abs/2004.05150

    Pith/arXiv arXiv 2020

  15. [15]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott ...

    2020

  16. [16]

    XAttention: Block sparse attention with antidiagonal scoring

    Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. XAttention: Block sparse attention with antidiagonal scoring. InForty-second International Conference on Machine Learning, 2025. URLhttps://openreview.net/forum?id=KG6aBfGi6e. 13 SSA

    2025

  17. [17]

    Quest: Query- aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query- aware sparsity for efficient long-context llm inference. InICML, 2024. URL https://openreview.net/ forum?id=KzACYw0MTV

    2024

  18. [18]

    Spargeattention: Accurate and training-free sparse attention accelerating any model inference

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattention: Accurate and training-free sparse attention accelerating any model inference. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum? id=74c3Wwk8Tc

    2025

  19. [19]

    Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free

    Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview...

    2025

  20. [20]

    Infllm-v2: Dense-sparse switchable attention for seamless short-to-long adaptation, 2025

    Weilin Zhao, Zihan Zhou, Zhou Su, Chaojun Xiao, Yuxuan Li, Yanghao Li, Yudi Zhang, Weilun Zhao, Zhen Li, Yuxiang Huang, Ao Sun, Xu Han, and Zhiyuan Liu. Infllm-v2: Dense-sparse switchable attention for seamless short-to-long adaptation, 2025. URLhttps://arxiv.org/abs/2509.24663

    arXiv 2025

  21. [21]

    Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention, 2025

    DeepSeek-AI. Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention, 2025

    2025

  22. [22]

    Neural discrete representation learning

    Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/fil...

    2017

  23. [23]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedbac...

    2022

  24. [24]

    Fast r-cnn, 2015

    Ross Girshick. Fast r-cnn, 2015. URLhttps://arxiv.org/abs/1504.08083

    Pith/arXiv arXiv 2015

  25. [25]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= mZn2Xyh9Ec

    2024

  26. [26]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  27. [27]

    SmolLM2: When smol goes big — data-centric training of a fully open small language model

    Loubna Ben allal, Anton Lozhkov, Elie Bakouch, Gabriel Martin Blazquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Agustín Piqueres Lajarín, Hynek Kydlíˇ cek, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan Son NGUYEN, Ben Burtenshaw, Clémentine Fourrier, Haojun Zhao, Hugo Larcher, Mathieu Morlon, Cyril Zakka, Colin Raffel, Leandro Von We...

    2025

  28. [28]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Kamal Mohamed Abbas, Cheng-Yu H...

    2024

  29. [29]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URLhttps://arxiv.org/abs/1911.11641

    Pith/arXiv arXiv 2019

  30. [30]

    HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/P19-1472. URLhttps://aclanthology.org/P19-1472/

    doi:10.18653/v1/p19-1472 2019

  31. [31]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URLhttps://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  32. [32]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. InInternational Conference on Learning Representations, 2017. URL https://openreview.net/forum?id= Byj72udxe

    2017

  33. [33]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for ...

    2024

  34. [34]

    RULER: What’s the real context size of your long-context language models? InFirst Conference on Language Modeling, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=kIoBbc76Sy

    2024

  35. [35]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P . Lillicrap. Com- pressive transformers for long-range sequence modelling. InInternational Conference on Learning Representations, 2020. URLhttps://openreview.net/forum?id=SylKikSYDH

    2020

  36. [36]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)...

    doi:10.18653/v1/2021.acl-long.427 2021

  37. [37]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. InInternational Conference on Learning Representations, 2022. URL https: //openreview.net/forum?id=R8sQPpGCv0

    2022

  38. [38]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

    arXiv 2024

  39. [39]

    Liger-kernel: Efficient triton kernels for LLM training

    Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, Yanning Chen, and Zhipeng Wang. Liger-kernel: Efficient triton kernels for LLM training. InChampioning Open-source DEvelopment in ML Workshop @ ICML25, 2025. URL https://openreview.net/forum?id=36SjAIT42G. 15 SSA A Proof of Proposition ...

    2025