Pith. sign in

REVIEW 56 references

Linear attention's decay works best when its median value sits near 0.8, across the model sizes tested.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A controlled study of decay in linear attention finds median decay near 0.8 works best, vector decay generally beats scalar decay, and RoPE/TPE give little benefit for models with sub-unity decay.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful controlled map of the decay design space, but the 'median ~0.8' rule is read off confounded comparisons and their own Simple Decay sweep favors p=0.99.

arxiv 2509.05282 v1 pith:HRH24NRU submitted 2025-09-05 cs.CL

Elucidating the Design Space of Decay in Linear Attention

Zhen Qin , Xuyang Shen , Yiran Zhong This is my paper
classification cs.CL
keywords linear attentiondecay mechanismstate space modelsparameterization strategyscalar vs vector decayrelative positional encodinglanguage modelingdesign space
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper maps the design space of the decay coefficient in linear-complexity attention and asks which design choices actually determine language-modeling quality. It finds that the value range of the decay matters more than the sophistication of the formula: across 160M, 410M, and 1.45B models, the strongest decays end up with median values near 0.8, while variants stuck near 0 (too much forgetting) or near 1 (attention dilution) lag behind. The paper also reports that vector decay beats scalar decay when the parameterization is held fixed, that sharing decay parameters with the key can push decay too far toward 0 or 1 and hurt, and that RoPE/TPE add little once decay is below 1. These results ground a simpler recipe, Simple Decay, initialized so the median decay is near p=0.99, which matches or beats the Mamba2 decay at every tested scale.

Core claim

On the paper's own terms, the central discovery is that the diagonal transition term λ in the linear-attention recurrence acts as a tunable memory-decay knob, and its effective operating point, more than its parameterization or data-dependence, predicts performance. Training the same Decay Linear Transformer with decay schemes drawn from Mamba2, GLA, Hgrn2, LightNet, and TNL shows that good methods converge to per-layer median decay values around 0.8; methods with medians near 0 forget too fast, and LightNet's near-1 values reproduce attention dilution. Removing Mamba2's A term does not hurt, and a minimal scheme λ = sigmoid(f + Δ), with Δ initialized to argsigmoid(p), reaches the same opera

What carries the argument

The decay coefficient λ_jt inside the diagonal state transition diag(λ_jt) of the recurrence s_jt = diag(λ_jt) s_j(t-1) + k_jt (v_jt)^⊤. It is the object whose parameterization, sharing, granularity, and RPE compatibility the paper varies. The diagnostic that carries the argument is the per-layer median decay value measured on a fixed 2048-token corpus after training; all conclusions are drawn by correlating that median with downstream loss, and the proposed Simple Decay controls it directly through the initialization of Δ.

Load-bearing premise

Single-run training without reported seeds means the small loss gaps (about 0.002) separating top methods could be noise, so the exact ranking and the precise 'around 0.8' optimum may not be stable.

What would settle it

Train the 1.45B model several times with different seeds for Mamba2, Simple Decay p=0.8, and Simple Decay p=0.99, then check whether the loss ordering and the per-layer median decay values near 0.8 reproduce; if the p=0.8 init lands at a different median or the 0.002 losses flip, the guideline needs revision.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can initialize decay so the trained median lands near 0.8 instead of relying on data-dependent formulas alone, since Simple Decay with p=0.99 reaches or beats the Mamba2 scheme at 160M, 410M, and 1.45B.
  • Adding RoPE or TPE to linear attention whose decay is mostly below 1 is unlikely to improve language modeling, so those encodings can be omitted without loss.
  • When choosing between scalar and vector decay with a fixed parameterization, vector decay is the safer default; scalar decay only wins when its parameterization pushes the median higher.
  • Mamba-style decay can drop its A parameter without degradation, meaning simpler parameterizations can carry the same performance.
  • For DPLR state transitions, adding decay at all is essential, and vector decay outperforms scalar decay, with larger initialization p giving lower loss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • We infer that the 'median near 0.8' result may generalize beyond the tested models as an initialization heuristic for other linear RNNs and SSMs, but that is an extrapolation the paper does not make.
  • If the range is what matters, then data-dependence itself is not load-bearing; a controlled experiment fixing the final median decay at 0.8 across static and input-dependent schemes would test this directly.
  • Because the reported comparisons are single runs, we infer that a small band of p values (roughly 0.95–0.99) is a safer design recommendation than the exact best p; the paper's own numbers may not distinguish them.
  • The fluctuating zero-shot numbers hint that decay tuning may matter more for long-context fidelity than for short-context benchmarks; a targeted long-range dependency probe would be a natural next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical design-space comparison; the 'median ≈ 0.8' guideline is a post-hoc observation, not a fitted prediction, and self-citations are background components rather than load-bearing.

full rationale

This paper is an empirical taxonomy and ablation study, not a formal derivation. No equation in the manuscript defines decay values in terms of the performance metric, and no fitted parameter is renamed as a prediction. The central guideline ('decay values should be neither too small nor too large, with median values around 0.8') is an inductive summary of correlations observed in Sections 5.1 and 5.3 when comparing Mamba2, GLA, Hgrn2, LightNet, TNL, and TNL-L. That comparison is confounded—methods differ simultaneously in parameterization, sharing, granularity, and input dependence—and the paper's own controlled Simple Decay sweep (Section 5.5, Table 3) finds p=0.99 best at all three scales, which tensions with the 0.8 rule. However, this is an internal-consistency / statistical-inference issue, not circularity: the p-sweep is an independent test, not an input to the rule, and the rule is not derived by construction from the data. The self-citations (TNL, Hgrn2, LightNet, LRPE, Xmixers) are prior published methods used as baselines or implementation infrastructure; none of the takeaways is justified solely by citing the authors' own prior work, and no uniqueness theorem is imported. The only mathematical derivation (Appendix A.3, RoPE compatibility) is a straightforward identity and is not circular. Single-run training without reported seeds is a reproducibility limitation, but it does not make the claims circular. Under the 'quote and exhibit the specific reduction' standard, no circular step can be identified.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The main quantitative claims are empirical, not derived. The framework rests on a unification of linear attention, linear RNN, and SSM recurrences (Section 3) and on the choice of one testbed architecture (Section 4). The proposed Simple Decay introduces one tuned hyperparameter p that is swept on the same benchmark used for evaluation. No new entities are invented.

free parameters (1)
  • Simple Decay initialization p = p=0.99 best; sweep over {0.8, 0.9, 0.95, 0.99}
    In Eq. 4, Delta is initialized with argsigmoid(p), so p sets the initial median decay. The paper sweeps p on the same benchmarks and selects 0.99, so the proposed method's reported advantage depends on a tuned hyperparameter.
axioms (4)
  • domain assumption All linear RNN, SSM, and linear attention variants can be represented by the recurrence s_t = M_t s_{t-1} + k_t v_t^T with diagonal M_t, and the diagonal defines the decay mechanism.
    Section 3 states this unification and then restricts to the diagonal component Eq. 2; the whole design-space study inherits this reduction.
  • domain assumption The fixed Decay Linear Transformer architecture (silu kernel, low-rank projections, TNL-style output gate) is a fair testbed such that conclusions transfer across linear attention variants.
    Section 4 says 'To ensure a fair comparison, all methods use the same network architecture'; transferability to other architectures, kernels, and larger models is assumed.
  • domain assumption Decay values below 1 provide a locality prior that suppresses the benefit of RoPE and TPE.
    Section 5.4 postulates this explanation for the observed negligible RPE effect; it is an after-the-fact hypothesis, not derived or tested independently.
  • domain assumption Single-seed training runs with 20k steps are sufficient to rank configurations on fineweb-edu-10B.
    Section 5 reports one training setup with no seeds; the authors themselves attribute DPLR zero-shot fluctuation to 'limited number of training tokens' in Section 5.6, undercutting this assumption.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Elucidating the Design Space of Decay in Linear Attention." pith.science (2026). https://pith.science/paper/HRH24NRU

@misc{pith2026250905282,
  author       = {Pith},
  title        = {Pith review of: Elucidating the Design Space of Decay in Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HRH24NRU}},
  note         = {Machine review of arXiv:2509.05282}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper presents a comprehensive investigation into the decay mechanisms inherent in linear complexity sequence models. We systematically delineate the design space of decay mechanisms across four pivotal dimensions: parameterization strategy, which refers to the computational methodology for decay; parameter sharing, which involves the utilization of supplementary parameters for decay computation; decay granularity, comparing scalar versus vector-based decay; and compatibility with relative positional encoding methods, such as Rotary Position Embedding (RoPE). Through an extensive series of experiments conducted on diverse language modeling tasks, we uncovered several critical insights. Firstly, the design of the parameterization strategy for decay requires meticulous consideration. Our findings indicate that effective configurations are typically confined to a specific range of parameters. Secondly, parameter sharing cannot be used arbitrarily, as it may cause decay values to be too large or too small, thereby significantly impacting performance. Thirdly, under identical parameterization strategies, scalar decay generally underperforms compared to its vector-based counterpart. However, in certain scenarios with alternative parameterization strategies, scalar decay may unexpectedly surpass vector decay in efficacy. Lastly, our analysis reveals that RoPE, a commonly employed relative positional encoding method, typically fails to provide tangible benefits to the majority of linear attention mechanisms.

Figures

Figures reproduced from arXiv: 2509.05282 by Xuyang Shen, Yiran Zhong, Zhen Qin.

Figure 1
Figure 1. Figure 1: Distribution of median decay values for each layer across different methods, with [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of median decay values for each layer across different methods, with [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Model Architecture. Model architecture diagram of Decay Linear Transformer: Each Decay Linear Transformer consists of multiple Decay Linear Transformer Layers, with each Layer comprising Decay Linear Attention and GLU; for Decay Linear Attention, its computational logic is shown in the right figure. A.2 Decay computation For decay, we first obtain activation F j through linear layers, then calculate λ j t … view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of median decay values for each layer across different methods, with [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of median decay values for each layer across different methods, with [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of median decay values for each layer across different methods, with [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of median decay values for each layer across different methods, with [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of median decay values for each layer in Simple Decay with different [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of median decay values for each layer in Simple Decay with different [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 28 canonical work pages

  1. [1]

    Simple linear attention language models balance the recall-throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher R \' e . Simple linear attention language models balance the recall-throughput tradeoff. CoRR, abs/2402.18668, 2024. doi:10.48550/ARXIV.2402.18668. URL https://doi.org/10.48550/arXiv.2402.18668

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.18668 2024

  2. [2]

    xlstm: Extended long short-term memory

    Maximilian Beck, Korbinian Poppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael K Kopp, G \"u nter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. ArXiv, abs/2405.04517, 2024. URL https://api.semanticscholar.org/CorpusID:269614336

    Pith/arXiv arXiv 2024

  3. [3]

    Rethinking attention with performers

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021. URL https://openrevie...

    work page 2021

  4. [4]

    Meta LA : Unified optimal linear approximation to softmax attention map

    Yuhong Chou, Man Yao, Kexin Wang, Yuqi Pan, Rui-Jie Zhu, Jibin Wu, Yiran Zhong, Yu Qiao, Bo XU, and Guoqi Li. Meta LA : Unified optimal linear approximation to softmax attention map. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=Y8YVCOMEpz

    work page 2024

  5. [5]

    Empirical evaluation of gated recurrent neural networks on sequence modeling

    Junyoung Chung, C aglar G \" u l c ehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014. URL http://arxiv.org/abs/1412.3555

    Pith/arXiv arXiv 2014

  6. [6]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. ArXiv, abs/2405.21060, 2024. URL https://api.semanticscholar.org/CorpusID:270199762

    Pith/arXiv arXiv 2024

  7. [7]

    Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando de Freitas, and Caglar Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficien...

    Pith/arXiv arXiv 2024

  8. [8]

    A framework for few-shot language model evaluation

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 2021

    work page 2021

  9. [9]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. 2023

    2023

  10. [10]

    Hippo: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R\' e . Hippo: Recurrent memory with optimal polynomial projections. arXiv preprint arXiv:2008.07669, 2020

    Pith/arXiv arXiv 2008

  11. [11]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \' e . Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022 a

    work page 2022

  12. [12]

    Efficiently modeling long sequences with structured state spaces, 2022 b

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022 b

    work page 2022

  13. [13]

    How to train your HIPPO : State space models with generalized orthogonal basis projections

    Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Re. How to train your HIPPO : State space models with generalized orthogonal basis projections. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=klK17OQ3KB

    work page 2023

  14. [14]

    Diagonal state spaces are as effective as structured state spaces

    Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=RjS0j6tsSrf

    work page 2022

  15. [15]

    Long short-term memory

    Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  16. [16]

    Mini CPM : Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu, and Maosong Sun. Mini CPM : Unveiling the potential of small language models wi...

    work page 2024

  17. [17]

    karpathy/char-rnn

    Andrej Karpathy. karpathy/char-rnn. 10 2023. URL https://github.com/karpathy/char-rnn

    work page 2023

  18. [18]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  19. [19]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  20. [20]

    Mega: Moving average equipped gated attention

    Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. CoRR, abs/2209.10655, 2022. doi:10.48550/arXiv.2209.10655. URL https://doi.org/10.48550/arXiv.2209.10655

    Pith/arXiv arXiv doi:10.48550/arxiv.2209.10655 2022

  21. [21]

    Megalodon: Efficient LLM pretraining and inference with unlimited context length

    Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, LILI YU, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient LLM pretraining and inference with unlimited context length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=XlAbMZu4Bo

    work page 2024

  22. [22]

    Parallelizing linear recurrent neural nets over sequence length

    Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=HyUNwulC-

    work page 2018

  23. [23]

    Smith, Albert Gu, Anushan Fernando, C aglar G \" u l c ehre, Razvan Pascanu, and Soham De

    Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, C aglar G \" u l c ehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 Jul...

    work page 2023

  24. [24]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  25. [25]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?...

    2024

  26. [26]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran G. V., Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wo...

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.13048 2023

  27. [27]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Gregory Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Nguyen Chung, Leon Derczynski, et al. Rwkv: Reinventing rnns for the transformer era. 2023 b

    work page 2023

  28. [28]

    Eagle and finch: RWKV with matrix-valued states and dynamic recurrence

    Bo Peng, Daniel Goldstein, Quentin Gregory Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Kranthi Kiran GV, Haowen Hou, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Jian Zhu, and Rui-Jie Zhu. Eagle and fin...

    work page 2024

  29. [29]

    Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. Rwkv-7 "goose" with expressive dynamic state evolution, 2025

    work page 2025

  30. [30]

    Random feature attention

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.02143, 2021

    Pith/arXiv arXiv 2021

  31. [31]

    Xmixers: A collection of SOTA efficient token/channel mixers , August 2025

    Zhen Qin. Xmixers: A collection of SOTA efficient token/channel mixers , August 2025. URL https://github.com/Doraemonzzz/xmixers

    work page 2025

  32. [32]

    cosformer: Rethinking softmax in attention

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. 2021

    work page 2021

  33. [33]

    The devil in linear transformer

    Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. arXiv preprint arXiv:2210.10340, 2022

    Pith/arXiv arXiv 2022

  34. [34]

    Toeplitz neural network for sequence modeling

    Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations (ICLR), 2023 a . URL https://openreview.net/forum?id=IxmWsm4xrua

    work page 2023

  35. [35]

    Scaling transnormer to 175 billion parameters

    Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Fei Yuan, Xiao Luo, et al. Scaling transnormer to 175 billion parameters. arXiv preprint arXiv:2307.14995, 2023 b

    Pith/arXiv arXiv 2023

  36. [36]

    Linearized relative positional encoding

    Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng, Dongxu Li, Xiaodong Han, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Linearized relative positional encoding. Transactions on Machine Learning Research, 2023 c

    work page 2023

  37. [37]

    Hierarchically gated recurrent neural network for sequence modeling

    Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA...

    work page 2023

  38. [38]

    Various lengths, constant speed: Efficient language modeling with lightning attention

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In Forty-first International Conference on Machine Learning, 2024 a . URL https://openreview.net/forum?id=Lwm6TiUP4X

    work page 2024

  39. [39]

    Hgrn2: Gated linear rnns with state expansion

    Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024 b

    Pith/arXiv arXiv 2024

  40. [40]

    You only scan once: Efficient multi-dimension sequential modeling with lightnet, 2025

    Zhen Qin, Yuxin Mao, Xuyang Shen, Dong Li, Jing Zhang, Yuchao Dai, and Yiran Zhong. You only scan once: Efficient multi-dimension sequential modeling with lightnet, 2025. URL https://openreview.net/forum?id=qK3XElJUbq

    work page 2025

  41. [41]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  42. [42]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  43. [43]

    Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023

    work page 2023

  44. [44]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021

    Pith/arXiv arXiv 2021

  45. [45]

    Retentive network: A successor to transformer for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism , January 2024 a

    Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism , January 2024 a . URL https://github.com/sustcsonglin/flash-linear-attention

    work page 2024

  47. [47]

    Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024 b

    Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024 b . URL https://github.com/fla-org/flash-linear-attention

    work page 2024

  48. [48]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. CoRR, abs/2312.06635, 2023. doi:10.48550/ARXIV.2312.06635. URL https://doi.org/10.48550/arXiv.2312.06635

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.06635 2023

  49. [49]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=r8H7xhYPwz

    work page 2025

  51. [51]

    Gated slot attention for efficient linear-time sequence modeling

    Yu Zhang, Songlin Yang, Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. Gated slot attention for efficient linear-time sequence modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=jY4PhQibmg

    work page 2024

  52. [52]

    fla-org/flame

    Yu Zhang, Songlin Yang, Han Guo, rakkit, and Junlin Han. fla-org/flame. 3 2025. URL https://github.com/fla-org/flame

    work page 2025

  53. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  54. [54]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  55. [55]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  56. [56]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.