Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

MSWA: Refining Local Attention with Multi-ScaleWindow Attention

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-Scale Window Attention (MSWA) splits attention heads and layers into four geometrically growing window sizes and reports better language modeling than uniform sliding-window attention at roughly seven-eighths the cost.

desk verdict MSWA is a clearly specified window reallocation with correct cost arithmetic, but the effectiveness gains are confounded with up to 4x larger per-head windows, so the central claim is premature. read the letter →

arxiv 2501.01039 v1 pith:M6IRBUQU submitted 2025-01-02 cs.CL cs.AI

Yixing Xu , Shivank Nag , Dong Li , Lu Tian , Emad Barsoum This is my paper · ORCID
classification cs.CLcs.AI
keywords multi-scalewindowattentionslidinglocallanguagemodelingTransformerefficiencyKVcachelinearcommon-sensereasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Multi-Scale Window Attention (MSWA), a drop-in change to sliding-window attention: instead of giving every head in every layer the same window size $w$, it assigns four groups of heads windows $\frac{w_i}{4}$, $\frac{w_i}{2}$, $w_i$, and $2w_i$, and lets the layer base size grow from $\frac{w}{4}$ in shallow layers to $2w$ in deep layers. The total window budget comes to about $\frac{7}{8}$ of uniform SWA. The paper claims this makes language models both better and cheaper: on Wikitext-103 perplexity drops from 30.70 to 29.56, and on enwik8 bits per character from 1.22 to 1.11, at relative cost 1.00 versus 1.14. A reader should care because it tests whether local-attention models lose quality mainly because uniform windows waste capacity on contexts of different lengths.

What carries the argument

The machinery is a window-size allocation matrix indexed by layer quartile and head quartile (Table 1). Starting from a base window size $w$, the four layer quartiles receive base sizes $\frac{w}{4}$, $\frac{w}{2}$, $w$, $2w$, and within each layer the four head quartiles receive $\frac{w_i}{4}$, $\frac{w_i}{2}$, $w_i$, $2w_i$, so that the deepest, widest quarter gets windows up to $4w$. Equation (11) computes the total window budget as approximately $\frac{7}{8}\, w\, h\, l$, meaning the whole model attends to fewer positions than uniform SWA. The schedule is what lets a single layer capture short and long contexts at the same time while shallow layers build local detail and deep layers integrate it into broader dependencies.

What would settle it

Train identical 12-layer models on Wikitext-103 where SWA is given the same total window budget as MSWA, or the same maximum window per layer, while MSWA keeps its fixed multi-scale schedule; if SWA matches or beats MSWA's perplexity under either equal-resource comparison, the paper's efficiency-effectiveness claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that the scale of context a model can absorb is a resource that should be allocated unevenly. The authors show that a fixed geometric schedule—four head groups per layer with window sizes $\frac{w_i}{4}$, $\frac{w_i}{2}$, $w_i$, $2w_i$, and four layer groups with base sizes $\frac{w}{4}$, $\frac{w}{2}$, $w$, $2w$—improves language modeling and common-sense reasoning over uniform sliding-window attention while using less compute and memory. The same schedule, applied on top of linear attention layers, narrows the gap to standard attention and outperforms SWA combined with linear attention. Fine-tuning a 7B language model with MSWA yields higher average few-shot accuracy on eight reasoning tasks than fine-tuning it with SWA. The paper's stated conclusion is that MSWA outperforms traditional local attention in both effectiveness and efficiency.

Load-bearing premise

The load-bearing premise is that the sum of window sizes is the right measure of cost and that MSWA's advantage does not reduce to the fact that its deepest heads see up to $4w$ while the SWA baseline is capped at $w$.

Editorial extensions

If this is right

  • On trained-from-scratch models, MSWA beats SWA on Wikitext-103 and enwik8 while spending about one-eighth less window budget.
  • MSWA can be stacked with linear attention layers, and the combination lands close to standard Transformer perplexity while remaining cheaper than SWA plus linear attention.
  • A pre-trained 7B model fine-tuned with MSWA holds or improves its average score on eight common-sense benchmarks relative to SWA fine-tuning, with a much smaller drop when the number of shots grows.
  • The window schedule runs on standard attention acceleration libraries, so adopting it does not require writing new attention kernels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the schedule is the true cause of the gains, the specific factor-of-two scaling is probably not essential: any monotone increase of window budget with depth that keeps the total budget fixed may capture most of the benefit.
  • The paper's cost measure counts attended positions; the real wall-clock advantage will depend on hardware and batching, so a direct latency study at longer sequences and larger batches would show where the one-eighth saving translates into speed.
  • Because only per-head window sizes change and the parameter count stays identical, existing checkpoints can in principle be fine-tuned to the multi-scale pattern without any architectural parameter increase.
  • A natural next test is to let the model learn its own per-layer and per-head window budgets instead of fixing them to a geometric schedule, which would show whether the allocation, rather than the schedule form, is what matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces Multi-Scale Window Attention (MSWA), a modification of sliding-window attention in which window sizes vary across both heads and layers: each layer is assigned a base window from a four-group schedule (w/4, w/2, w, 2w), and within each layer heads are split into four groups with windows (w_i/4, w_i/2, w_i, 2w_i). The paper shows that the total window budget is 15/16 of SWA within a layer and about 7/8 of SWA overall (Eqs. 8-11). It reports from-scratch language modeling results on Wikitext-103 and enwik8 (Table 2), a combination with linear attention (Table 3), fine-tuning of Llama-7B on RedPajama followed by commonsense reasoning evaluation (Table 4), an efficiency benchmark with FlashAttention (Fig. 3), and ablations of base window size and scheduling strategy (Tables 5 and 6). The central claim is that MSWA outperforms traditional local attention in both effectiveness and efficiency.

Significance. The proposed schedule is simple, easy to implement on top of standard sparse-attention kernels, and the complexity arithmetic in Eqs. (8)-(11) is correct: under the sum-of-window-sizes metric, MSWA is cheaper than SWA at the same base window. The main tables show consistent average improvements over SWA, and the layer/head ablation in Table 6 gives some evidence that the specific ordering of window sizes matters rather than mere budget reallocation. If the effectiveness gains survive a controlled comparison, this is a practical drop-in improvement for local-attention LLMs. The paper does not ship code or checkpoints, and all headline numbers are single runs, so reproducibility is currently not verifiable.

major comments (4)
  1. [Sec. 4.3, Table 1; Tables 2 and 5] The effectiveness comparison does not control for maximum receptive field. In Table 1, MSWA gives the deepest layer quarter and widest head quarter a window of 4w, while the SWA baseline uses w everywhere; with w=128 this is 512 vs 128, and in Table 5 the MSWA rows run up to 2048 while the SWA rows reach at most 512. In the fine-tuning setup (Sec. B.2.2) the discrepancy is even larger: SWA is capped at 1024 tokens while MSWA heads can cover the full 4096-token context. The reported PPL/bpc gains could therefore be due to a few heads receiving much longer context rather than to the multi-scale schedule. The authors should add controls that hold both total budget and maximum window fixed, for example a SWA baseline with window 4w and an alternative allocation of the same 7/8 budget that includes a 4w head but uses a non-geometric schedule. Without such a control, the 'effectiveness' part of the central claim is not established.
  2. [Sec. 5.1, Tables 2 and 3] All language-modeling results are reported as single runs with no error bars or seed variance. Some of the headline differences are small enough to be within run-to-run noise: MSWA's enwik8 bpc of 1.11 is 0.01 away from standard attention at sequence length 1024, and in Table 3 Linear+MSWA (1.13 bpc) differs from Linear+SWA (1.16 bpc) by 0.03. Please report multiple seeds with means and standard deviations, or otherwise justify that the differences are significant. This is needed because the central claim rests on small margins in Tables 2 and 3.
  3. [Sec. 5.2, Table 4, Sec. B.2.2] The downstream evaluation is not reliable as reported. The SWA baseline is highly unstable: its average accuracy drops from 47.12 to 42.66 when moving from 3-shot to 5-shot, driven by a 12.2-point drop on BoolQ and a 13-point drop on COPA, while the MSWA numbers remain roughly flat. The claimed +7.23 advantage in the 5-shot row is therefore more plausibly attributed to baseline variance than to the attention pattern. In addition, the fine-tuning setup gives MSWA a maximum window equal to the full 4096-token sequence while SWA sees only 1024 tokens, reintroducing the receptive-field confound from Tables 1 and 5. Please provide repeated fine-tuning runs and a controlled window-size setting before drawing conclusions about downstream compatibility.
  4. [Sec. 5.4.2, Table 6] The ablation of window-variation strategy does not isolate the effect of the geometric schedule. 'Decreasing for Deeper Layer' retains the same total budget and actually still has a 4w head in the shallow layers, while 'Arithmetic Progression' changes both the schedule and the maximum window size (max 320 instead of 512 at base 128). A controlled comparison would keep the total window budget and the maximum per-head window identical and vary only the intermediate schedule, so that the performance difference can be attributed to multi-scale allocation rather than budget or receptive field. As it stands, Table 6 cannot rule out the confound identified above.
minor comments (6)
  1. [Throughout] The method is inconsistently typeset as 'MSWA' and 'MSW A'; please unify the notation.
  2. [Sec. 5.4.2] There are typos: 'demenstrate' should be 'demonstrate', and 'decresing' should be 'decreasing'.
  3. [Sec. B.2.1] The phrase 'casual language modeling' should be 'causal language modeling'.
  4. [Table 2 caption] The caption states that relative cost measures cost as a proportion of the cost of MSWA, but MSWA's cost is 1.00 by construction; consider stating the total window budget explicitly to avoid confusion.
  5. [Sec. 5.1.2] The dataset is referred to as 'enwik-8' in this section but as 'enwik8' elsewhere; please make the notation consistent.
  6. [Fig. 3] The efficiency figure would benefit from reporting the number of runs and the variance of the measured times, since the curves appear to cross in the low-batch-size regime.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MSWA's window schedule is specified a priori, its cost savings follow from summation identities, and its performance claims are evaluated against external benchmarks.

full rationale

The paper's central derivation is the window-size schedule in Sec. 4 and Table 1, from which the relative-cost claims follow by straightforward arithmetic (Eqs. 8-11). The window sizes are set by a fixed design rule (groups of 1/4, 1/2, 1, 2 times a base window, applied across heads and layers) and are not fitted to the test data or to the reported outcomes. The language-modeling and downstream accuracies are measured on external benchmarks (Wikitext-103, enwik8, common-sense reasoning tasks) against SWA and standard attention, so the 'prediction' that MSWA outperforms SWA is an empirical result, not an artifact of construction. No load-bearing self-citation appears in the derivation: the cited works are external baselines, implementations, or unrelated prior work, and no 'uniqueness theorem' or equivalent is invoked to force the design. The main weakness identified by a skeptical reader—that MSWA's effectiveness gains might be confounded with its larger maximum window (up to 4w) rather than the multi-scale schedule—is a legitimate experimental-design concern about controlling maximum receptive field, but it is not circularity: the paper does not define MSWA's superiority in terms of its inputs, and the cost metric is consistent with the stated summation of window sizes. Accordingly, the circularity burden is minimal and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a hand-chosen window schedule and on the domain assumptions of locality and layer-wise scale growth. No new physical or mathematical entities are introduced.

free parameters (3)
  • window schedule multiplier = 2x per group (w/4, w/2, w, 2w)
    Hand-chosen progression inspired by Swin Transformer (Liu et al., 2021). Not fitted to test data, but defines the MSWA mechanism and is the main design choice; only two alternative schedules are ablated.
  • base window size w = 128 for main LM experiments; 256 to 512 in ablations; 1024 for Llama fine-tuning
    Set per experiment to control total cost. Results in Tables 2, 3, and 5 depend on this value, though the paper shows MSWA beats SWA at multiple w values.
  • number of head/layer groups = 4 groups
    Fixed at four groups in Sec 4.3; no ablation varies the number of groups, so the method is only evaluated for this one group count.
assumptions (3)
  • domain assumption Locality of reference: most information about a token comes from nearby tokens
    Used in Sec 1 and Sec 4.1 to justify local window attention and the decreasing attention resource distribution.
  • domain assumption Layer depth corresponds to increasing context scale; shallow layers should model local information and deep layers long-range dependencies
    Stated in Sec 4.2 and supported by a reversed-allocation ablation in Table 6, but not by external evidence.
  • standard math Total cost of local attention is proportional to the sum of all window sizes across heads and layers
    Derived from Sec 3.2 complexity O(d n w) and used in Equations (8) to (11); treated as the resource metric for all cost comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MSWA: Refining Local Attention with Multi-ScaleWindow Attention." pith.science (2026). https://pith.science/paper/M6IRBUQU

@misc{pith2026250101039,
  author       = {Pith},
  title        = {Pith review of: MSWA: Refining Local Attention with Multi-ScaleWindow Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M6IRBUQU}},
  note         = {Machine review of arXiv:2501.01039}
}
read the original abstract

Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention (SWA) solves this problem by restricting the attention range to a fixed-size local context window. Nevertheless, SWA employs a uniform window size for each head in each layer, making it inefficient in capturing context of varying scales. To mitigate this limitation, we propose Multi-Scale Window Attention (MSWA) which applies diverse window sizes across heads and layers in the Transformer. It not only allows for different window sizes among heads within the same layer but also progressively increases window size allocation from shallow to deep layers, thus enabling the model to capture contextual information with different lengths and distances. Experimental results on language modeling and common-sense reasoning tasks substantiate that MSWA outperforms traditional local attention in both effectiveness and efficiency.

Figures

Figures reproduced from arXiv: 2501.01039 by the authors.

Figure 1
Figure 1. Illustration of Multi-Scale Window Attention mechanism. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Combination of MSWA and linear attention. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Computational time required by each attention mechanism to predict the next token. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MOSAIC: Adaptive Inter-layer Composition for Efficient Heterogeneous Vision-Language Models

    cs.CV 2026-07 accept novelty 6.0 of 10

    MOSAIC uses multi-objective MIP search over linear/sparse/low-rank operators plus two-stage distillation to convert a homogeneous VLM into a hardware-aware heterogeneous model that matches teacher performance at 2.5× ...

  2. Customizing the Inductive Biases of Softmax Attention using Structured Matrices

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Structured-matrix scoring functions, BTT and MLR, let attention escape the low-rank bottleneck and add a distance-dependent compute bias, improving accuracy for fixed compute on regression, language modeling, and forecasting.

  3. GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

    cs.AI 2026-06 reject novelty 4.0 of 10

    GLIDE allocates softmax attention only to early/middle transformer layers and linearizes later layers, cutting KV-cache I/O by up to ~62× while retaining 92–96% of baseline accuracy.

Reference graph

Works this paper leans on

52 extracted references · 8 canonical work pages · cited by 3 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  5. [5]

    AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card

    2024

  6. [6]

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher R \'e . 2024. Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668

    arXiv 2024

  7. [7]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  8. [8]

    Yoshua Bengio, R \'e jean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in neural information processing systems, 13

    2000

Show all 52 references
  1. [9]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432--7439

    2020

  2. [10]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023. Longlora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations

    2023

  3. [11]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509

    2019 arXiv

  4. [12]

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. In International Conference on Learning Representations

    2020

  5. [13]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com...

    2019

  6. [14]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  7. [15]

    Together Computer. 2023. https://github.com/togethercomputer/RedPajama-Data Redpajama: An open source recipe to reproduce llama training dataset

    2023

  8. [16]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978--2988

    2019

  9. [17]

    Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations

    2023

  10. [18]

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. 2024. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv pre...

    2024 arXiv

  11. [19]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  12. [20]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752

    2023 arXiv

  13. [21]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

    2021

  14. [22]

    Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. 2022. Transformer quality in linear time. In International conference on machine learning, pages 9099--9117. PMLR

    2022

  15. [23]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  16. [24]

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR

    2020

  17. [25]

    Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. 2022. xformers: A modular and hackable transf...

    2022

  18. [26]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991--52008

    2023

  19. [27]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012--10022

    2021

  20. [28]

    Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations

    2018

  21. [29]

    Matt Mahoney. 2009. Large text compression benchmark

    2009

  22. [30]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. In International Conference on Learning Representations

    2016

  23. [31]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381--2391

    2018

  24. [32]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. NAACL HLT 2019, page 48

    2019

  25. [33]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  26. [34]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. 2023. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP 2023...

    2023

  27. [35]

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. 2021. Random feature attention. arXiv preprint arXiv:2103.02143

    2021 arXiv

  28. [36]

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R \'e . 2023. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043--28078. PMLR

    2023

  29. [37]

    Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. 2022. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025--7041

    2022

  30. [38]

    Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. 2019. Blockwise self-attention for long document understanding. arXiv preprint arXiv:1911.02972

    2019 arXiv

  31. [39]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p...

    2020

  32. [40]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series

    2011

  33. [41]

    Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023. The programmer’s assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Int...

    2023

  34. [42]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  35. [43]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv e-prints, pages arXiv--2104

    2021

  36. [44]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  37. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  38. [46]

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768

    2020 arXiv

  39. [47]

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. 2023. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635

    2023 arXiv

  40. [48]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

  41. [49]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  42. [50]

    Biao Zhang, Barry Haddow, and Alexandra Birch. 2023 a . Prompting large language model for machine translation: A case study. In International Conference on Machine Learning, pages 41092--41110. PMLR

    2023

  43. [51]

    Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Re. 2023 b . The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations

    2023

  44. [52]

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39--57

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.