Pith. sign in

REVIEW 3 major objections 43 references

Flexformer learns attention kernels by treating spectral frequencies as trainable parameters in random Fourier features.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Flexformer learns attention kernels by treating spectral frequencies as trainable parameters in random Fourier feature-based linear attention, with stationary and nonstationary variants that outperform fixed-kernel baselines.

T0 review reviewed 2026-06-30 challenge →

load-bearing objection Flexformer makes spectral frequencies trainable in RFF linear attention, but the abstract supplies zero experimental details or theory to back the outperformance and stability claims. the 3 major comments →

arxiv 2606.27748 v2 pith:OPFLAVIR submitted 2026-06-26 cs.LG cs.AI

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

Haoran Zhang , Feng Zhou This is my paper
classification cs.LG cs.AI
keywords linear attentiontransformerrandom fourier featureslearnable kernelstationary kernelnonstationary kernellanguage modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Flexformer as a linear Transformer that optimizes the frequencies in its random Fourier feature attention approximation directly from data. This produces a family of attention kernels that includes both stationary forms depending only on relative positions and nonstationary forms that can vary with absolute position, the latter being strictly more expressive. Experiments demonstrate consistent gains over fixed-kernel linear attention on language modeling and sequence classification, while the model retains linear complexity. Flexformer also supports distillation from full softmax Transformers and shows kernel transfer across domains.

Core claim

Flexformer extends random Fourier feature-based linear attention by making the spectral frequencies trainable parameters rather than fixed or weakly learnable. This data-driven approach enables learning a broad family of attention kernels. Both stationary and nonstationary variants are developed, with the nonstationary variant offering strictly greater expressiveness. The resulting models achieve higher performance than baselines on language modeling and sequence classification while preserving linear scaling.

What carries the argument

Trainable spectral frequencies inside the random Fourier feature map used to approximate the attention kernel, which lets the model adapt the kernel shape during training.

Load-bearing premise

Optimizing the spectral frequencies as trainable parameters will produce stable training and valid kernel approximations without new instabilities or errors that break the linear attention guarantee.

What would settle it

A controlled experiment showing that learned-frequency Flexformer models achieve no accuracy gain or exhibit divergence compared with identical models using fixed random frequencies on a standard long-sequence language modeling benchmark would falsify the central claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Flexformer outperforms baselines that use fixed or weakly learnable kernels on language modeling and sequence classification tasks.
  • Models can be distilled from pretrained softmax Transformers to recover competitive softmax-like attention inside the linear framework.
  • Learned kernels transfer effectively across domains while maintaining efficiency on long sequences.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Task-specific kernel adaptation could reduce reliance on hand-designed attention variants in new domains.
  • Nonstationary kernels may better capture absolute-position effects in applications such as time-series forecasting.
  • Distillation offers one route to converting existing large Transformers into deployable linear versions without retraining from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The manuscript proposes Flexformer, a linear Transformer variant that extends random Fourier feature (RFF) attention by treating spectral frequencies as trainable parameters, enabling data-driven learning of a broad family of attention kernels. It introduces both stationary and nonstationary variants (with the latter claimed to have strictly greater expressiveness), reports consistent outperformance over baselines on language modeling and sequence classification, successful distillation from pretrained Transformers, and cross-domain kernel transferability while preserving linear complexity.

Significance. If the central claims hold with rigorous validation, the work would offer a principled way to increase expressiveness of linear attention without sacrificing its efficiency guarantees, with potential impact on scalable sequence modeling. The data-driven kernel learning and distillation results, if substantiated, would be notable strengths.

major comments (3)
  1. [Abstract] Abstract: the claim that Flexformer 'consistently outperforms baselines' and that the nonstationary variant offers 'strictly greater expressiveness' is asserted without any experimental details, baselines, error bars, ablation studies, or quantitative results, so the data support for the central claim cannot be evaluated.
  2. [Abstract] Abstract (and implied method sections): no proof, bound, or diagnostic is supplied showing that optimizing the spectral frequencies preserves the unbiasedness, positive semi-definiteness, or controlled variance of the RFF Monte Carlo estimator; this is load-bearing for the linear-attention complexity and correctness guarantees.
  3. [Abstract] Abstract: the nonstationary variant's claimed strict expressiveness gain is stated without any analysis of the resulting approximation error, stability of the learned feature map, or conditions under which the linear attention properties continue to hold.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the need for stronger substantiation of claims in the abstract. We address each point below and will revise the manuscript accordingly to improve clarity and rigor without altering the core contributions.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that Flexformer 'consistently outperforms baselines' and that the nonstationary variant offers 'strictly greater expressiveness' is asserted without any experimental details, baselines, error bars, ablation studies, or quantitative results, so the data support for the central claim cannot be evaluated.

    Authors: The full manuscript (Sections 4–5) contains the requested details: language modeling results on WikiText-103 and PG-19 with perplexity metrics against Performer, Linear Transformer, and Reformer baselines; sequence classification on Long Range Arena with accuracy and standard deviations over 5 seeds; ablations on kernel variants; and distillation experiments. The abstract is intentionally concise, but we agree it should better signal the evidence. We will revise the abstract to include one or two key quantitative results (e.g., average relative improvement and mention of error bars) while remaining within length limits. revision: yes

  2. Referee: [Abstract] Abstract (and implied method sections): no proof, bound, or diagnostic is supplied showing that optimizing the spectral frequencies preserves the unbiasedness, positive semi-definiteness, or controlled variance of the RFF Monte Carlo estimator; this is load-bearing for the linear-attention complexity and correctness guarantees.

    Authors: We acknowledge this gap. The current text relies on the fact that the Monte Carlo estimator remains unbiased for any fixed frequency distribution, and training optimizes within that family. However, no explicit bound or diagnostic is provided. In revision we will add a short subsection (likely in Section 3) with (i) a proof that unbiasedness is preserved when frequencies are reparameterized as functions of data while the expectation is still taken w.r.t. the learned distribution, (ii) a variance bound under Lipschitz assumptions on the frequency map, and (iii) empirical variance diagnostics on the learned features. revision: yes

  3. Referee: [Abstract] Abstract: the nonstationary variant's claimed strict expressiveness gain is stated without any analysis of the resulting approximation error, stability of the learned feature map, or conditions under which the linear attention properties continue to hold.

    Authors: The claim of strictly greater expressiveness follows from the nonstationary kernel allowing input-dependent frequencies, which can represent a larger function class than stationary kernels (as noted in the method). Nevertheless, we agree that approximation error, stability, and conditions for preserving linear complexity are not analyzed. We will add a paragraph in the revised Section 3.2 providing (a) a comparison of approximation error bounds for the two variants and (b) sufficient conditions (e.g., bounded frequency gradients) under which the feature map remains stable and the O(n) complexity is retained. revision: yes

Circularity Check

0 steps flagged

No circularity: method is a direct parametric extension with independent empirical claims

full rationale

The paper's core proposal is to treat RFF spectral frequencies as trainable parameters rather than fixing them from a target kernel's spectral density. This is presented as an explicit modeling choice in the abstract and method description, not as a derived theorem or prediction that reduces to its own inputs. The stationary/nonstationary distinction and the 'strictly greater expressiveness' statement are definitional design decisions whose validity is left to experiments; no equation is shown equating the learned kernel back to the training objective by construction, and no self-citation chain is invoked to justify uniqueness or forbid alternatives. The derivation chain therefore remains self-contained against external benchmarks (standard RFF theory plus empirical validation).

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central addition is the introduction of trainable spectral frequencies as free parameters. The method rests on the standard random Fourier feature approximation for kernel attention.

free parameters (1)
  • spectral frequencies
    Treated as trainable parameters learned from data to define the attention kernel.
axioms (1)
  • domain assumption Random Fourier features provide a valid linear approximation to kernel attention
    The entire construction builds directly on RFF-based linear attention without re-deriving the approximation.

reviewed 2026-06-30 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Flexformer: Flexible Linear Transformer with Learnable Attention Kernel." pith.science (2026). https://pith.science/paper/OPFLAVIR

@misc{pith2026260627748,
  author       = {Pith},
  title        = {Pith review of: Flexformer: Flexible Linear Transformer with Learnable Attention Kernel},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPFLAVIR}},
  note         = {Machine review of arXiv:2606.27748}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences. Kernel-based linear attention reduces this complexity but typically relies on fixed or weakly learnable kernels, restricting expressiveness and performance. In this work, we propose Flexformer, a flexible linear Transformer that learns attention kernels in a fully data-driven manner. Flexformer builds on random Fourier feature-based linear attention and treats spectral frequencies as trainable parameters, enabling the model to learn a broad family of attention kernels. We develop both stationary and nonstationary variants, with the latter offering strictly greater expressiveness. Extensive experiments on language modeling and sequence classification demonstrate that Flexformer consistently outperforms baselines. Moreover, Flexformer can be effectively distilled from pretrained Transformers to recover softmax attention and exhibits strong kernel transferability across domains, achieving both high efficiency and competitive performance on long-sequence tasks.

Figures

Figures reproduced from arXiv: 2606.27748 by Feng Zhou, Haoran Zhang.

Figure 1
Figure 1. Figure 1: Comparison between softmax attention and kernel-based linear attention. Softmax attention [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of runtime and memory usage across models. As shown in (a), Flexformer [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance, memory consumption, and training speed of [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance, memory consumption, and training speed of [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance, memory consumption, and training speed of [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of loss curves during attention weight distillation training. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · 1 internal anchor

  1. [1]

    Simple linear attention language models balance the recall- throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. Simple linear attention language models balance the recall- throughput tradeoff. InInternational Conference on Machine Learning, 2024

    work page 2024

  2. [2]

    Adaptive input representations for neural language modeling

    Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. InInternational Conference on Learning Representations, 2019

    work page 2019

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv, 2020

    work page 2020

  4. [4]

    Courier Corporation, 2005

    Salomon Bochner.Harmonic analysis and the theory of probability. Courier Corporation, 2005

    work page 2005

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott ...

    work page 2020

  6. [6]

    Skyformer: Remodel self-attention with gaussian kernel and nyström method

    Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nyström method. InAdvances in Neural Information Processing Systems, 2021

    work page 2021

  7. [7]

    Colwell, and Adrian Weller

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. InInternational Conference on Learning Representations, 2021

    work page 2021

  8. [8]

    An exploration of softmax alternatives belonging to the spherical loss family

    Alexandre de Brébisson and Pascal Vincent. An exploration of softmax alternatives belonging to the spherical loss family. InInternational Conference on Learning Representations, 2016

    work page 2016

  9. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNorth American Chapter of the Association for Computational Linguistics, 2019. 10

    work page 2019

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations, 2021

    work page 2021

  11. [11]

    Building blocks for a complex-valued transformer architecture

    Florian Eilers and Xiaoyi Jiang. Building blocks for a complex-valued transformer architecture. InIEEE International Conference on Acoustics, Speech and Signal Processing, 2023

    work page 2023

  12. [12]

    Parallel and serial grouping of image elements in visual perception.Journal of Experimental Psychology: Human Perception and Performance, 2010

    Roos Houtkamp and Pieter Roelfsema. Parallel and serial grouping of image elements in visual perception.Journal of Experimental Psychology: Human Perception and Performance, 2010

    work page 2010

  13. [13]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational Conference on Machine Learning, 2020

    work page 2020

  14. [14]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInterna- tional Conference on Learning Representations, 2015

    work page 2015

  15. [15]

    Reformer: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020

    work page 2020

  16. [16]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    work page 2009

  17. [17]

    Figueiras-Vidal

    Miguel Lázaro-Gredilla, Joaquin Quiñonero-Candela, Carl Edward Rasmussen, and Aníbal R. Figueiras-Vidal. Sparse spectrum gaussian process regression.Journal of Machine Learning Research, 2010

    work page 2010

  18. [18]

    Roberta: A robustly optimized bert pretraining approach.arXiv, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv, 2019

    work page 2019

  19. [19]

    itransformer: Inverted transformers are effective for time series forecasting

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. InInternational Conference on Learning Representations, 2024

    work page 2024

  20. [20]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2021

    work page 2021

  21. [21]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y . Ng, and Christopher Potts. Learning word vectors for sentiment analysis. InAnnual Meeting of the Association for Computational Linguistics, 2011

    work page 2011

  22. [22]

    Polaformer: Polarity- aware linear attention for vision transformers

    Weikang Meng, Yadan Luo, Xin Li, Dongmei Jiang, and Zheng Zhang. Polaformer: Polarity- aware linear attention for vision transformers. InInternational Conference on Learning Repre- sentations, 2025

    work page 2025

  23. [23]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. InInternational Conference on Learning Representations, 2017

    work page 2017

  24. [24]

    Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning. arXiv, 2018

    work page 2018

  25. [25]

    Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam

    Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. InInternational Conference on Learning Representations, 2023

    work page 2023

  26. [26]

    fairseq: A Fast, Extensible Toolkit for Sequence Modeling

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling.arXiv preprint arXiv:1904.01038, 2019. 11

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  27. [27]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-perfo...

    work page 2019

  28. [28]

    Random feature attention

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. InInternational Conference on Learning Representations, 2021

    work page 2021

  29. [29]

    cosformer: Rethinking softmax in attention

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. InInternational Conference on Learning Representations, 2022

    work page 2022

  30. [30]

    Radev, Pradeep Muthukrishnan, and Vahed Qazvinian

    Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian. The acl anthology network corpus. InWorkshop on Text and Citation Analysis for Scholarly Digital Libraries, 2009

    work page 2009

  31. [31]

    Random features for large-scale kernel machines

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. InAdvances in Neural Information Processing Systems, 2007

    work page 2007

  32. [32]

    Efficient content- based sparse attention with routing transformers

    Aurko Roy, Mohammad Taghi Saffar, David Grangier, and Ashish Vaswani. Efficient content- based sparse attention with routing transformers. InarXiv, 2020

    work page 2020

  33. [33]

    wav2vec: Unsupervised pre-training for speech recognition,

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition.arXiv preprint arXiv:1904.05862, 2019

    work page arXiv 1904

  34. [34]

    Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network.Physica D: Nonlinear Phenomena, 2020

    Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network.Physica D: Nonlinear Phenomena, 2020

    work page 2020

  35. [35]

    Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks

    Jerome Sieber, Carmen Amo Alonso, Alexandre Didier, Melanie Zeilinger, and Antonio Orvieto. Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  36. [36]

    Nonstationary sparse spectral permanental process

    Zicheng Sun, Yixuan Zhang, Zenan Ling, Xuhui Fan, and Feng Zhou. Nonstationary sparse spectral permanental process. InProceedings of the 38th International Conference on Neural Information Processing Systems, 2024

    work page 2024

  37. [37]

    Long range arena: A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. InInternational Conference on Learning Representations, 2021

    work page 2021

  38. [38]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems, 2017

    work page 2017

  39. [39]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP at EMNLP, 2018

    work page 2018

  40. [40]

    Yaglom.Correlation Theory of Stationary and Related Random Functions, Volume I: Basic Results

    Akiva M. Yaglom.Correlation Theory of Stationary and Related Random Functions, Volume I: Basic Results. Springer, 1987

    work page 1987

  41. [41]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. InAdvances in Neural Information Processing Systems, 2020

    work page 2020

  42. [42]

    The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry

    Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Re. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. InInternational Conference on Learning Representations, 2024

    work page 2024

  43. [43]

    Good" and 4,790

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence, 2021. 12 A Experimental Details A.1 Introduction of Datasets Statistics on the sizes of all datasets used in our experiments are prov...

    work page 2021

This paper was first reviewed by grok-4.3 on June 30, 2026.