Pith. sign in

REVIEW 5 major objections 6 minor 42 references

MixFormer: Linear Transformer with Mixture of Memory Experts

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MixFormer claims that replacing a single state-space memory with many time-aware memory experts yields the best long-range accuracy among efficient Transformers at linear cost.

desk verdict A plausible memory-expert idea undermined by an invalid parallel/recurrent equivalence and thin experiments; not ready for referees. read the letter →

arxiv 2608.09468 v1 pith:O6ASHHI5 submitted 2026-08-10 cs.LG cs.AI

Yu Guo , Lei Duan This is my paper · ORCID
classification cs.LGcs.AI MSC 68T07
keywords MixFormerlinearattentionstatespacemodelsmixtureofexpertslong-rangedependencytime-awaredecayArenaimagegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that a linear Transformer can keep pace with or beat softmax attention on long-range tasks if it replaces the usual single memory state with many specialized memory states gated by time. Existing state space models store all history in one fixed-size matrix, so early information gets diluted; MixFormer instead keeps sixty-four memory experts, each with its own exponential decay and learned positional bias, and routes inputs to a subset of them. On the Long-Range Arena benchmark it reports an average accuracy of 56.72 percent, above every efficient-Transformer baseline it compares against, and it shows image completions on MNIST and CIFAR-10. The reason this matters is that the model retains linear-time training and constant-memory autoregressive inference, so long contexts could be processed at lower cost than with quadratic softmax attention.

What carries the argument

The load-bearing object is the TALA recurrence and its parallel matrix form. In parallel, attention is computed as $Q \odot (K^T \odot D)V$, where $D$ is a causal mask with entries $\omega^{n-m} G_{nm}$; in recurrent form, the same rule becomes $S_n = \omega G \odot S_{n-1} + K_n^T V_n$. The MoME layer then aggregates multiple such states with router weights $E_i = \text{softmax}(K_n V_n W_g)$. This combination is what lets the model keep constant-memory inference while having more than one place to store history.

What would settle it

Train MixFormer with an explicitly absolute-position-dependent $G$ and measure the memory footprint of the recurrent state as sequence length grows; if memory grows with $L$, the $O(1)$ claim fails. Alternatively, compute the outputs of Eqs. (14) and (15) on the same input; any mismatch for a fixed-weight model shows the two modes are not computing the same function.

Watch

Extended reading notes

Core claim

The central claim, stated by the authors, is that memory dilution, not kernel approximation, is the main bottleneck for linear Transformers on ultra-long sequences. MixFormer addresses it with Time-Aware Linear Attention (TALA), where the recurrent state update is $S_n = \omega G \odot S_{n-1} + K_n^T V_n$, with $\omega$ an exponential decay weight and $G$ a learnable pairwise positional bias, and with a Mixture-of-Memory-Experts layer that routes queries to $K_n V_n$ through a softmax router. Each expert keeps an independent memory state, so different experts can preserve short-term and long-term patterns. The paper reports that this design achieves the best average score on LRA and produces high-quality image generations, and it claims these gains come with linear complexity and constant memory during inference.

Load-bearing premise

The constant-memory recurrent form works only if the learned pairwise positional bias $G$ depends on relative position or is a per-head scalar; the paper does not prove that the parallel form with absolute-position $G$ is equivalent to the recurrent update.

Editorial extensions

If this is right

  • Long-context language modeling could be done with linear instead of quadratic attention without the accuracy drop usually reported for kernel-based linear Transformers.
  • Because MixFormer supports both parallel training and recurrent inference, the same weights can switch between GPU-parallel training and constant-memory generation.
  • Routing to multiple memory experts gives a mechanistic handle on what the model remembers: the paper's visualizations show long-term experts holding distant entities and short-term experts handling recent tokens.
  • The reported LRA average of 56.72 percent would put MixFormer ahead of every baseline in its comparison table, including BigBird and Performer.
  • Image completion on CIFAR-10 suggests the architecture can be applied to pixel-sequence generation, where long-range dependencies across channels matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not show that the parallel and recurrent forms in Eqs. (14) and (15) are mathematically equivalent; if $G$ is truly pairwise and absolute-position dependent, the recurrent state would have to grow with sequence length, so the constant-memory claim needs an extra argument. That is an editorial caution, not a result in the paper.
  • The MoME router cost is $O(Nk)$; at 64 experts this is negligible next to $N d^2$, but if $k$ were scaled with $d$, the linear-complexity claim could break unless the router is made sparse.
  • A natural extension would be to make the decay $\omega$ or the bias $G$ input-dependent per token, turning TALA into a selective state space model; that would test whether the gains come from time-awareness or from the multi-expert memory alone.
  • The 'sustainable web infrastructure' conclusion is an extrapolation: the paper reports accuracy and complexity, not wall-clock time or energy per token, so that claim remains untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes MixFormer, a linear Transformer that combines a Time-Aware Linear Attention (TALA) mechanism with a Mixture-of-Memory-Experts (MoME) module. TALA is intended to add exponential decay and learnable positional biases to the SSM memory state, and MoME is designed to maintain several memory states selected by a router. The authors claim that this yields state-of-the-art results on LRA and qualitative improvements on image generation while retaining linear complexity in the sequence length.

Significance. If the claims were correct, the paper would offer a useful architectural addition to the linear-attention/SSM literature, and the idea of routing to multiple memory states is worth exploring. The manuscript, however, contains dimensional inconsistencies in the core equations, an unproven equivalence between parallel and recurrent formulations, an underspecified router, and uncontrolled experimental comparisons. These issues prevent the results from being validated, so the significance cannot be assessed on the current evidence. The paper does not provide code or detailed experimental logs, and the complexity analysis contains errors.

major comments (5)
  1. [4.3, Eqs. (14)-(15)] The recurrent form in Eq. (15) is not a valid constant-memory reformulation of the parallel form in Eq. (14). In Eq. (14), D_{nm}=ω^{n−m}G_{nm} with G a T×T pair-wise positional bias, while in Eq. (15) the same G is used in an elementwise product with the d×d state S_{n-1}; these dimensions are incompatible. If G is instead meant to be a scalar or per-head decay, then it is no longer 'the same as in Equation (14)', and the pair-wise positional dependence of the parallel form cannot be reproduced by a fixed-size recurrent state without additional approximations. No proof of equivalence is given, so the claimed O(1)-memory autoregressive inference is unsupported.
  2. [4.3, Eq. (12) vs. Eqs. (14)-(16)] The denominator in Eq. (12) is dropped in the parallel representation (Eq. 14) and in the recurrent and chunkwise forms (Eqs. 15-16). As written, Eq. (14) computes Q⊙(K^T⊙D)V without the normalization by Σ φ(K_i)(ωG_i)^{t-i}, so the parallel output scale differs from the recurrent output scale even if G is interpreted as a scalar decay. This discrepancy means the training and inference modes do not compute the same function, and it also changes the interpretation of the state S_n.
  3. [4.4, Eqs. (17)-(18)] The router is underspecified: E_i=Softmax(K_n V_n W_g) is undefined because K_n V_n is d×d while W_g is declared as 1×k_e, so the matrix product has incompatible shapes. In addition, the sum in Eq. (17) runs over n, which is the sequence length, rather than over the number of memory experts k_e. Without a consistent definition of the expert state M S_i and the routing weights, the MoME mechanism cannot be implemented or evaluated.
  4. [4.6, Eq. (21)] The complexity calculation is incorrect: the FFN term is written as 2dd′ without the factor N, although the FFN is applied to every token; it should be 2Ndd′. The MoME cost Nk also conflicts with the d×d router computation of Eq. (18), which would add a per-token O(d^2) cost. Consequently, Eq. (21) does not establish the claimed O(N) total complexity.
  5. [5.2-5.4, Table 3] The empirical evaluation is not adequately controlled. MixFormer is a large model (2B or 7B total parameters) trained on 15B-30B tokens, while the LRA baselines in Table 3 are taken from the original benchmark setup and are orders of magnitude smaller in parameter count and training data. The image-generation experiments (Sections 5.3-5.4) report only qualitative results with no quantitative metrics. The claim of 'consistently achieves significant performance gains' is therefore not supported by the evidence presented.
minor comments (6)
  1. [4.3, Eq. (11)] The notation φ(Q)=e^{Q/||Q||} should specify that the exponential is applied elementwise and that ||·|| is the row-wise L2 norm; the current notation is ambiguous for matrices.
  2. [4.3, Eq. (13)] The definition ω=1−e^{−h} with h∈[1,...,d/d_head] is unclear: is h a head index and are these decay values learnable or fixed? The relationship to the per-head decay parameters used in Eqs. (14)-(15) should be stated.
  3. [4.3, Eq. (16)] The index in R_{n−1} should be R_{i−1} to refer to the previous chunk; as written, it refers to the full sequence length.
  4. [5, Table 3] Several entries have concatenated numbers (e.g., '53.8242.77 77.0551.41' for Performer), which obscures the reported values.
  5. [5.5] The phrase 'reflection attention' appears to be a typo for 'linear attention'.
  6. [References] The reference list cites Han et al. twice for RetNet, but the correct authors are Sun et al.; the MoE-related reference [35] (Narayan et al.) is on summarization and does not support the MoE background.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's empirical claims are benchmark outcomes, not derivations from their own conclusions.

full rationale

The central claims of MixFormer are empirical: it reports LRA accuracy and image-generation results from trained models compared against external baselines. These numbers are fitted training outcomes, not quantities derived by construction from the model definition, so there is no self-definitional or fitted-input-called-prediction loop. The architecture equations (parallel TALA in Eq. 14, recurrent TALA in Eq. 15, and chunkwise TALA in Eq. 16) define the proposed mechanism; they do not presuppose the claimed performance gains. The paper's citations are to external prior work (RetNet, GLA, Mamba, linear attention literature) and are not used as a self-citation chain to force a conclusion. The reader-identified issue that Eq. 15 may not be equivalent to Eq. 14, due to the dimensionality of G and the missing denominator, is a technical correctness or consistency concern, not circularity: an invalid equivalence is not a proof step that reduces to its own input. No load-bearing step in the derivation is justified by citing the authors' own prior results or by renaming a known result. Therefore, no circularity is present, and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim rests on the assumption that a bank of memory states with a positional bias can be updated recurrently at constant memory cost. No formal proof or implementation is provided, and the free parameters are set by hand without sensitivity analysis.

free parameters (3)
  • Head-dependent decay schedule omega_h = 1 - e^{-h}
    Chosen by hand in Eq. (13) rather than learned; the abstract's 'learnable exponential decay' is not reflected in the formulation.
  • Number of memory experts k_e = 64
    Hyperparameter listed in Table 4; central to the MoE claim but no sensitivity analysis is provided.
  • Chunk length B
    Mentioned in the chunkwise recurrence in Eq. (16) but never specified in the experimental setup.
assumptions (3)
  • standard math Matrix multiplication associativity allows reordering of QKV products in linear attention.
    Used in Eq. (6) to switch from an L-by-L attention matrix to a d-by-d state, which is the basis of linear complexity.
  • domain assumption The kernel feature map phi(x) = e^{x/||x||} is non-negative and thus a valid attention similarity function.
    Invoked in Eq. (11) to ensure non-negative attention weights; non-negativity is required for the linear attention interpretation.
  • ad hoc to paper The learned positional bias G can be folded into the recurrent state update without violating the fixed-size state assumption.
    Assumed implicitly in Eq. (15) when G is used inside the recurrence; the paper does not prove this is possible for a general pair-wise bias matrix.
invented entities (1)
  • Memory expert (Mixture-of-Memory-Experts)
    purpose: Multiple differentiated memory states that collaboratively store long-range context and are fused via a router.
    The paper introduces multiple memory state matrices as a new architectural entity, but provides no external falsifiable prediction; its behavior is only illustrated qualitatively in Figure 9.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MixFormer: Linear Transformer with Mixture of Memory Experts." pith.science (2026). https://pith.science/paper/O6ASHHI5

@misc{pith2026260809468,
  author       = {Pith},
  title        = {Pith review of: MixFormer: Linear Transformer with Mixture of Memory Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O6ASHHI5}},
  note         = {Machine review of arXiv:2608.09468}
}
read the original abstract

State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling. However, existing SSMs suffer from limited input adaptivity and constrained memory capacity, leading to information loss when modeling ultra-long sequences. To address these limitations, we propose MixFormer, a novel linear Transformer that integrates a Mixture-of-Memory-Experts (MoE) mechanism. Specifically, the model maintains differentiated memory states through multiple collaborating memory experts and employs a novel Time-Aware Linear Attention (TALA) mechanism, which leverages learnable exponential decay functions and positional biases to dynamically update memory. This design enables the model to selectively reinforce important historical information while effectively mitigating memory dilution, substantially improving long-range dependency modeling. Experiments on long-sequence text and image generation tasks demonstrate that MixFormer not only achieves significant performance gains but also provides a more sustainable computational backbone for the next generation of web infrastructure.

Figures

Figures reproduced from arXiv: 2608.09468 by the authors.

Figure 1
Figure 1. Linear Attention vs SSMs. (a) is the workflow of the Linear Attention, and (b) is the State Space Models. (a) (b) Linear Linear Linear GN Linear Matmul MoME Matmul Memory State Memory State1 Memory StateN Router (c) GN TALA GN FFN N* Embedding Memory State2 .... TALA layer MoME layer TopK [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (a) is the overall architecture of our proposed MixFormer, (b) is the Time-Aware Linear Attention (TALA) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Dot-product Attention vs Linear Attention. (a) is the illustration of the Dot-product Attention, and (b) is the Linear Attention. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The recurrence of the Time-Aware Linear Attention. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Experimental results on image generation tasks. The top row is the result of unconditional generation, while [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Experimental results on CIFAR 10. (a) is the occluded input image, and (b) is the output generated by [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Visualized intermediate outputs of (a) standard dot-product attention and (b) reflection attention. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the causal mask used by MixFormer. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Visualization of memory-expert activations. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 32 canonical work pages

  1. [1]

    Attention is all you need.Proceedings of the 2017 Advances in Neural Information Processing Systems, NeuraIPS, pages 5998–6008, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Proceedings of the 2017 Advances in Neural Information Processing Systems, NeuraIPS, pages 5998–6008, 2017

    work page 2017

  2. [2]

    Llama: open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [3]

    Llama 2: open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  4. [4]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    2024

  5. [5]

    Molmo and pixmo: open weights and open data for state-of-the-art multimodal models.arXiv e-prints, pages arXiv–2409, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: open weights and open data for state-of-the-art multimodal models.arXiv e-prints, pages arXiv–2409, 2024

    work page 2024

  6. [6]

    Nvlm: open frontier-class multimodal llms.arXiv preprint arXiv:2409.11402, 2024

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Moham- mad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: open frontier-class multimodal llms.arXiv preprint arXiv:2409.11402, 2024

    arXiv 2024

  7. [7]

    Efficient transformers: a survey.ACM Computing Survey, 55(6), 2022

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: a survey.ACM Computing Survey, 55(6), 2022

    work page 2022

  8. [8]

    A survey on efficient training of transformers

    Bohan Zhuang, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, and Chunhua Shen. A survey on efficient training of transformers. InProceedings of the 2023 International Joint Conference on Artificial Intelligence, IJCAI, pages 6823–6831, 2023

    work page 2023

Show all 42 references
  1. [9]

    Lorenzo Papa, Paolo Russo, Irene Amerini, and Luping Zhou. A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):7682–7700, 2024

    2024

  2. [10]

    A survey on vision transformer.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):87–110, 2022

    Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):87–110, 2022

    2022

  3. [11]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In Proceedings of the 2022 International Conference on Learning Representations, ICLR, 2022

    2022

  4. [12]

    Diagonal state spaces are as effective as structured state spaces

    Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Proceedings of the 2022 Advances in Neural Information Processing Systems, NeuraIPS, 35:22982–22994, 2022

    2022

  5. [13]

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R´e. Combining recurrent, convolutional, and continuous-time models with linear state space layers.Proceedings of the 2021 Advances in Neural Information Processing Systems, NeuraIPS, 34:572...

    2021

  6. [14]

    Liquid structural state-space models

    Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. InProceedings of the 2023 International Conference on Learning Representations, ICLR, 2023. 15 MixFormer: Linear Transformer with Mixture of ...

    2023

  7. [15]

    Simplified state space layers for sequence modeling

    Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. InProceedings of the 2023 International Conference on Learning Representations, ICLR, 2023

    2023

  8. [16]

    Retentive network: a successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: a successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  9. [17]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. InProceedings of the 2024 International Conference on Machine Learning, ICML, pages 56501–56523, 2024

    2024

  10. [18]

    Transformers are rnns: fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ¸ois Fleuret. Transformers are rnns: fast autoregressive transformers with linear attention. InProceedings of the 2020 International Conference on Machine Learning, ICML, pages 5156–5165, 2020

    2020

  11. [19]

    Efficient attention: attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: attention with linear complexities. InProceedings of the 2021 International Conference on Computer Vision, ICCV, pages 3531–3539, 2021

    2021

  12. [20]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and J¨urgen Schmidhuber. Linear transformers are secretly fast weight programmers. InProceedings of the 2021 International Conference on Machine Learning, ICML, pages 9355–9366, 2021

    2021

  13. [21]

    Flatten transformer: vision transformer using focused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: vision transformer using focused linear attention. InProceedings of the 2023 International Conference on Computer Vision, ICCV, pages 5961–5971, 2023

    2023

  14. [22]

    Polaformer: polarity-aware linear attention for vision transformers

    Weikang Meng, Yadan Luo, Xin Li, Dongmei Jiang, and Zheng Zhang. Polaformer: polarity-aware linear attention for vision transformers. InProceedings of the 2025 International Conference on Learning Representations, ICLR, 2025

    2025

  15. [23]

    Linear attention mechanism: an efficient attention for semantic segmentation

    R Li, J Su, C Duan, and S Zheng. Linear attention mechanism: an efficient attention for semantic segmentation. arXiv preprint arXiv:2007.14902, 2020

    2007 arXiv

  16. [24]

    Random feature attention

    H Peng, N Pappas, D Yogatama, R Schwartz, N Smith, and L Kong. Random feature attention. InProceedings of the 2021 International Conference on Learning Representations, ICLR, 2021

    2021

  17. [25]

    Rethinking attention with performers

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. InProceedings of the 2021 International Conference on Lea...

    2021

  18. [26]

    Mamba: linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: linear-time sequence modeling with selective state spaces. InProceedings of the 2024 International First Conference on Language Modeling, CoLM, 2024

    2024

  19. [27]

    Transformers are ssms: generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: generalized models and efficient algorithms through structured state space duality. InProceedings of the 2024 International Conference on Machine Learning, ICML, pages 10041–10071, 2024

    2024

  20. [28]

    An attention free transformer.arXiv preprint arXiv:2105.14103, 2021

    Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer.arXiv preprint arXiv:2105.14103, 2021

    2021 arXiv

  21. [29]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. Rwkv: Reinventing rnns for the transformer era. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages...

    2023

  22. [30]

    Group normalization

    Yuxin Wu and Kaiming He. Group normalization. InProceedings of the European conference on computer vision (ECCV), pages 3–19, 2018

    2018

  23. [31]

    Efficientvit: enhanced linear attention for high-resolution low-computation visual recognition.arXiv preprint arXiv:2205.14756, 3(1), 2022

    Han Cai, Chuang Gan, and Song Han. Efficientvit: enhanced linear attention for high-resolution low-computation visual recognition.arXiv preprint arXiv:2205.14756, 3(1), 2022

    2022 arXiv

  24. [32]

    Soft: softmax-free transformer with linear complexity.Proceedings of the 2021 Advances in Neural Information Processing Systems, NeuraIPS, 34:21297–21309, 2021

    Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: softmax-free transformer with linear complexity.Proceedings of the 2021 Advances in Neural Information Processing Systems, NeuraIPS, 34:21297–21309, 2021

    2021

  25. [33]

    Nystr¨omformer: a nystr ¨om-based algorithm for approximating self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: a nystr ¨om-based algorithm for approximating self-attention. InProceedings of the 2021 Confer- ence on Artificial Intelligence, AAAI, number 16, pages 14138–...

    2021

  26. [34]

    Outrageously large neural networks: the sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. InProceedings of the 2017 International Conference on Learning Representations, ICLR, 2017....

    2017

  27. [35]

    Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization

    Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 1797–1807, 2018

    2018

  28. [36]

    Long range arena: a benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: a benchmark for efficient transformers. InProceedings of the 2020 International Conference on Learning Representations, I...

    2020

  29. [37]

    Listops: a diagnostic dataset for latent tree learning

    Nikita Nangia and Samuel Bowman. Listops: a diagnostic dataset for latent tree learning. InProceedings of the 2018 Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, ACL, pages 92–99, 2018

    2018

  30. [38]

    Learning word vectors for sentiment analysis

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. InProceedings of the 2011 Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, ACL, pages 142–150, 2011

    2011

  31. [39]

    The acl anthology network corpus.Language Resources and Evaluation, 47(4):919–944, 2013

    Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The acl anthology network corpus.Language Resources and Evaluation, 47(4):919–944, 2013

    2013

  32. [40]

    Learning long-range spatial dependencies with horizontal gated recurrent units.Proceedings of the 2018 Advances in Neural Information Processing Systems, NeuraIPS, 31, 2018

    Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units.Proceedings of the 2018 Advances in Neural Information Processing Systems, NeuraIPS, 31, 2018

    2018

  33. [41]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  34. [42]

    Emnist: Extending mnist to handwritten letters

    Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In2017 international joint conference on neural networks (IJCNN), pages 2921–2926. IEEE, 2017. 17

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.