Pith. sign in

REVIEW 4 major objections 5 minor 54 references

HyenaND brings global, input-dependent convolution to multi-dimensional data at subquadratic cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 02:00 UTC pith:RT2HMGXA

load-bearing objection A genuinely useful ND subquadratic operator with an impressive empirical sweep, but a headline The Well claim is contradicted by its own Table 12, and the fast implementation is still closed — worth a serious referee, not a desk reject. the 4 major comments →

arxiv 2607.19378 v2 pith:RT2HMGXA submitted 2026-07-01 cs.LG cs.CVstat.ML

Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions

David R. Wessels , Farhad Ramezanghorbani , David W. Romero , Alireza Moradzadeh , Olivia Viessmann , Maksim Zhdanov , John St. John , Ken Janik
show 4 more authors
David M Knigge Yucheng Tang Erik J Bekkers Saee Gopal Paliwal
This is my paper
classification cs.LG cs.CVstat.ML
keywords HyenaNDlong convolutioninput-dependent kernelsmulti-dimensional datasubquadratic operatorsstate-space modelsFiLM conditioningFFT convolution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

HyenaND is a sequence-mixing operator that applies a global, input-dependent convolution directly on the native grid of an N-dimensional input, avoiding the ad-hoc 1D rasterization that recurrent subquadratic models require and the quadratic cost that attention pays. The paper argues that for approximately linear-time-invariant signals—images, volumes, physical fields, and much of natural data—sample-level input dependence (one kernel synthesized per input, used at every position) is the right trade, and it preserves the fast FFT path. On its controlled spatial-recall probes, pure HyenaND solves 1D/2D/3D copy and associative-recall tasks that attention fails at native resolution and that bidirectional Mamba fails on 3D transport. In downstream experiments, pure HyenaND stacks match strong attention baselines, and hybrids that interleave HyenaND and attention outperform both pure attention and recurrence-based hybrids. If right, this dissolves the patchification-vs-accuracy tradeoff for ND data and makes subquadratic global mixing a drop-in replacement for windowed attention.

Core claim

On the paper's own terms, HyenaND's central claim is that a long-convolution operator can be global, subquadratic, and input-dependent in the native geometry of N-dimensional data. The kernel K(x) is synthesized once per input: register tokens prepended to the grid summarize the input into a control variable z(x); a FiLM-conditioned SIREN MLP maps coordinates to filter values; a Gaussian window bounds the receptive field; and a single ND FFT convolution applies the kernel at every position in O(∏ₙ Lₙ log ∏ₙ Lₙ). Empirically, pure HyenaND stacks match strong attention baselines on ImageNet-1K, PanTS 3D segmentation, OpenGenome2 genomics, and the PDE suites of The Well, and striped HyenaND-att

What carries the argument

The operator is y = RMSNorm(Sigmoid(v) ⊙ (K(x) * RMSNorm(ℓ2(q) ⊙ SiLU(k)))) computed via ND FFT. Kernel synthesis is the load-bearing device: K_i(x) = w(c_i) ⊙ f_θ(c_i; z(x)), where z(x) is a softmax-weighted average over learnable register tokens, f_θ is a joint N-dimensional SIREN with block-diagonal multi-frequency initialization, and w is a per-axis learnable Gaussian window. Because z(x) is computed once per input, the same kernel is applied at every position, preserving the single-FFT path while still giving the operator input dependence. The paper also proves that a gated Hyena layer computes exactly separable attention matrices—a Toeplitz kernel sandwiched between diagonal gates—whic

Load-bearing premise

The load-bearing premise is that conditioning the kernel on the whole input once per example—not per token—is enough input dependence for multi-dimensional data; the paper's own color-cond probe shows a case where per-token bidirectional Mamba beats HyenaND, so if tasks need per-position content retrieval, this operator cannot express it.

What would settle it

A direct test: on the 2D color-cond spatial recall task, scale the number of colored distractors or the distance between key and readout and compare bidirectional Mamba and HyenaND at matched compute. If Mamba's advantage over HyenaND grows as retrieval becomes more selective and content-dependent, the sample-level conditioning premise fails in exactly the regime the paper claims to serve.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Patchification stops being a necessary preprocessing step: HyenaND can ingest images and volumes at native resolution and follow the accuracy gains of smaller patches without attention's quadratic cost; at patch 2×2 the paper reports roughly 6× fewer FLOPs than matched attention.
  • Hybrids that interleave HyenaND with attention layers outperform both pure attention and bidirectional-Mamba hybrids on ImageNet and 3D segmentation, suggesting a productive default architecture for ND data.
  • Native-geometry mixing preserves long-range transport that rasterized recurrence loses, as shown by the 3D copy task where bidirectional Mamba barely improves on the mean-predictor floor while HyenaND solves the task.
  • Long-context genomics becomes tractable: a 1B striped-hyena model trains to convergence with stable throughput through 16M-token contexts and shows decreasing per-token NLL with depth, indicating it uses distant context rather than merely tolerating it.
  • 3D medical segmentation memory drops about 11% versus windowed attention while matching or improving Dice, because the FFT path avoids materializing attention matrices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Sample-level conditioning is a deliberate bound: since the kernel is shared across all positions, any task that needs content-based retrieval at individual positions—the paper's own 2D color-cond probe at patch 4 shows bidirectional Mamba winning—may require a token-level mechanism on top of HyenaND.
  • The length-generalization guarantee (Theorem A.7) is only tight for small deviations from the training length, and the 16M-context runs use staged doubling. A testable extension is whether a model trained at 8K can jump directly to 1M without the staged schedule and whether its kernel remains stable outside the [−1,1]^N coordinate domain.
  • HyenaND's kernel synthesis could be reused as a learned tokenizer: a few layers on raw pixels or voxels could aggregate fine-grained structure into information-dense tokens for any downstream backbone, turning patch size into a learned quantity. The paper gestures at this direction but does not implement it.
  • The separable-attention characterization suggests gated Hyena's inductive bias is complementary to attention rather than simply weaker; one can test whether increasing the number of gates or the order of the operator extends its expressiveness toward fuller attention patterns.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces HyenaND, an N-dimensional long-convolution operator with implicit SIREN-parameterized kernels, Gaussian masking, and sample-level input dependence implemented through learned register tokens and FiLM conditioning. It also presents nSubQ, a fused CUDA library for ND FFT convolutions, and positions HyenaND as a native-geometry, subquadratic alternative to attention and rasterized SSMs. Experiments span controlled spatial-recall probes, 1B-parameter genomics language modeling, ImageNet classification, PDE surrogate modeling on The Well, 3D medical segmentation, and kernel-level throughput benchmarks. The central claims are that pure HyenaND stacks match strong attention baselines, hybrid HyenaND–attention configurations outperform recurrent hybrids, and nSubQ converts asymptotic advantages into wall-clock speedups.

Significance. If the claims hold, HyenaND offers a principled path around the patchification/quadratic-attention trade-off for multi-dimensional data and gives the convolutional family a form of input dependence without per-token selectivity. The theoretical characterization of gated Hyena as separable attention (Thm A.5) and the length-generalization bound (Thm A.7) are useful, though technically standard. The empirical scope is commendably broad across 1D/2D/3D tasks. However, several headline empirical statements are internally inconsistent, and the custom CUDA implementation is not made available, so the reported gains are currently not independently checkable.

major comments (4)
  1. [§5.4, Table 12, §6] The text claims HyenaND 'achieves the lowest VRMSE on all eight datasets' and 'beats the full-resolution CNextU-net baseline on every dataset'. Table 12's Shear Flow row contradicts this: CNextU-net achieves 0.0262 while the best HyenaND result is 0.0268 (p=8). The conclusion (§6) repeats the false universal. Please correct the claim and reconcile the numbers.
  2. [§5.1, Tables 5/6/8] The sentence 'HyenaND outperforms both Attention and Mamba across all tasks' is not supported by the reported best-patch results. In 1D color cond (Table 5), Mamba-bidir reaches 2.38e-4 while HyenaND's best is 5.0e-3; in 2D color cond (Tables 6/8), Mamba-bidir reaches 4.06e-4 while HyenaND's best is 2.09e-3. The valid claim appears to be limited to the native-resolution regime; the text should be revised accordingly.
  3. [Appendix B.2] The nSubQ CUDA implementation is the basis for the wall-clock speedup claims (Figures 8–12 and Table 10). The footnote states that source code and documentation are omitted for double-blind review. Without the implementation or a detailed kernel-level description, the reported speedups (6×, 40×, memory reductions, and the 16M-context genomics throughput) cannot be independently verified. For a journal publication, the artifacts or complete pseudocode should be released, or the speedup claims should be presented as preliminary.
  4. [§3.2.2, Eq. (5), Fig. 4] The computation order for the control variable z(x) is ambiguous. If the 'register activations of the present layer' are the register rows of the input to that layer, then at the first layer those registers are learned constants shared across all inputs, so the kernel K(x) is not actually sample-dependent at that layer. If they are the outputs of the same layer, the definition is circular because K(x) is needed to compute those outputs. Please specify the order of computation and clarify the precise sense in which a single HyenaND layer is input-dependent.
minor comments (5)
  1. [Appendix C.4] The appendix states 'five selected datasets' from The Well, but Table 12 reports eight datasets. The main text also says 'eight PDE datasets'. Please align the dataset count and describe the selection criterion.
  2. [§5.1] The reference 'As shown in Table 5' appears to refer to the main-text summary Table 1, not the appendix Table 5. Please correct the cross-reference.
  3. [§3.2.2] The phrase 'too crude to for kernel conditioning' contains a typo; should be 'too crude for kernel conditioning'.
  4. [§5.4] The sentence 'For both shear flow, euler multi quadrants and helmholtz staircase' is grammatically awkward; use 'For shear flow, euler multi quadrants, and helmholtz staircase'. Also, the 'both' is incorrect for a list of three.
  5. [§3.2.2] The symbol B is used both for the number of frequency blocks in §3.2.1 and for the batch size in the sentence 'one for each of the inputs in a batch of size B'. Please disambiguate the notation.

Circularity Check

0 steps flagged

No significant circularity: the operator is defined by construction, theoretical claims are proven from stated definitions, and empirical claims rest on external benchmarks.

full rationale

HyenaND's derivation chain is self-contained. Eq. 3 defines the operator with K_i(x) = w(c_i) ⊙ f_theta(c_i; z(x)), where z(x) is a softmax-weighted register summary; 'sample-level input dependence' is therefore an architectural definition rather than a derived prediction. Theorem A.5 is a direct algebraic expansion of the gated Hyena mixer into A_ij = T_{i-j} r_i c_j and verifies the claimed factorization without importing the conclusion. Theorem A.7 derives a length-generalization bound from the SIREN Lipschitz estimate and displacement normalization; the bound is an output, not an input. The SIREN variance correction W0 ~ U(-1/sqrt(N), 1/sqrt(N)) follows from computing Var(w_0^T r) and matching the 1D reference variance. Empirical claims are benchmarked against external datasets (OpenGenome2, ImageNet-1K, The Well, PanTS) and external/independent baselines, with no test-set quantity used to set a parameter that is then reported as a prediction. The only self-citations appear in the systems context: [2,14,16] are the authors' Evo2/BioNeMo codebase and prior convolution-systems work, invoked to situate nSubQ (Appx. B) rather than to justify accuracy claims; Appendix B.2 notes source code is omitted for double-blind review, which is a reproducibility limitation, not a circular step. I therefore find no load-bearing reduction of any result to its own input.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No new physical or ontological entities are introduced; registers, FiLM, and the control variable are architectural mechanisms borrowed from prior work. The central design rests on FFT convolution, SIREN expressivity, the LTI inductive bias, and the empirical assumption that register summaries can carry enough input information for kernel conditioning.

free parameters (5)
  • Gaussian window per-channel/per-axis variance σ² = learned during training; initial log-spaced ramp
    Controls kernel receptive field; learned from data rather than derived.
  • SIREN frequency block schedule (ω_min, ω_max, B, s_min, s_max) = not reported as exact numbers
    Hand-chosen initialization for multi-frequency kernel coverage; empirically motivated in Appx A.3.
  • off-block initial scale ρ = 0.1
    Hand-set to allow off-diagonal SIREN entries to evolve; arbitrary.
  • number of registers R = not specified
    Controls capacity of the control variable z(x); absence hampers reproducibility.
  • fixed frequency parameter for The Well experiments = not specified
    Kept constant across PDE settings; authors attribute mixed patch-size results to it (§5.4), so it is a hand-set knob influencing reported VRMSE.
axioms (5)
  • standard math ND FFT-based convolution with proper zero-padding computes the exact linear convolution.
    Used throughout §3.2.1 and §4; relies on the convolution theorem.
  • domain assumption A SIREN MLP over [−1,1]^N can parameterize sufficiently expressive global ND kernels.
    Eq. (4); expressivity is empirical, with no universality guarantee for ND long-range kernels.
  • domain assumption Natural and multi-dimensional data are approximately LTI, so a single per-sample kernel is appropriate.
    §2.3-2.4 motivates the design; if false for a task, per-token selectivity is needed.
  • ad hoc to paper Register tokens accumulate global information sufficient to condition the kernel after one pass through the operator.
    §3.2.2, Fig. 4 assert this; it is an optimization/representation assumption, not proven.
  • standard math Length generalization bound assumes bounded inputs and predictable SIREN behavior outside the training displacement range.
    Theorem A.7 and Remark A.9; the bound is valid but practically meaningful only for |L−L0|≪L0.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions." pith.science (2026). https://pith.science/paper/RT2HMGXA

@misc{pith2026260719378,
  author       = {Pith},
  title        = {Pith review of: Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RT2HMGXA}},
  note         = {Machine review of arXiv:2607.19378}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Subquadratic alternatives to attention require compromises when applied to multi-dimensional data: standard convolutions lack global receptive fields and input dependency, while recurrent models require rasterizing data such as images, volumes, and partial differential equation (PDE) into an ad-hoc $1\rm D$ scan order that violates their spatial structure. We introduce \textit{HyenaND}, a subquadratic, global, input-dependent operator that acts directly on the native geometry of multidimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional convolutional kernels. Our CUDA implementation, \texttt{nSubQ}, fuses the FFT-convolution path to turn HyenaND's $\mathcal{O}(L \log L)$ scaling into wall-clock speedups. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, while hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids.

Figures

Figures reproduced from arXiv: 2607.19378 by Alireza Moradzadeh, David M Knigge, David R. Wessels, David W. Romero, Erik J Bekkers, Farhad Ramezanghorbani, John St. John, Ken Janik, Maksim Zhdanov, Olivia Viessmann, Saee Gopal Paliwal, Yucheng Tang.

Figure 1
Figure 1. Figure 1: (Left): Complexity of global multi-dimensional operators by token count L. Attention is natively multi-dimensional but scales quadratically. Mamba is subquadratic but inherently 1D, requiring ad-hoc 1D scan orders to process multi-dimensional data. HyenaND (ours) is global, natively multi-dimensional and subquadratic. (Right): Forward-pass time vs. sequence length using flash-attention, the official mamba … view at source ↗
Figure 2
Figure 2. Figure 2: The subquadratic landscape. SSMs are simultaneously recurrent and convolutional. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The HyenaND operator. The input is an ND token grid with R register tokens prepended along the first axis. Top (kernel synthesis): patch coordinates feed a SIREN MLP fθ, are masked by a learned Gaussian window w, and FiLM-conditioned on the a control variable z(x), yielding the input-dependent kernel K(x) (§3.2.1–3.2.2). Bottom (data path): the input is projected into q, k, v; the inner gate ℓ2(q) ⊙ SiLU(k… view at source ↗
Figure 3
Figure 3. Figure 3: The HyenaND operator. The input is an ND token grid with R register tokens prepended along the first axis. Top (kernel synthesis): patch coordinates feed a SIREN MLP fθ, are masked by a learned Gaussian window w, and FiLM-conditioned on the a control variable z(x), yielding the input-dependent kernel K(x) (§3.2.1–3.2.2). Bottom (data path): the input is projected into q, k, v; the inner gate ℓ2(q) ⊙ SiLU(k… view at source ↗
Figure 4
Figure 4. Figure 4: ImageNet Top-1 vs. compute. Each curve sweeps patch size p ∈ {16, 8, 4, 2}; smaller patches give more tokens and more FLOPs. 5.4 PDE Surrogate Modeling on The Well (2D/3D) We assess HyenaND on five PDE datasets from The Well [33], spanning fluid dynamics, MHD, astro￾physics, acoustics, and biological pattern formation. The datasets were chosen to cover resolutions of 64–1024-pixel on 2D and 3D Cartesian gr… view at source ↗
Figure 4
Figure 4. Figure 4: FiLM-conditioned kernel synthesis in HyenaND. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: VRMSE vs. sequence length on The Well. Sequence length is controlled by patch size: smaller patches p yield more tokens. Each panel shows one dataset (lower is better). HyenaND (red) improves as sequence length grows on most datasets. CNextU-net (green dashed) is a fixed full-resolution baseline with no patch-size axis. All three Hyena-containing variants consume ∼58.7 GB of peak allocated memory per GPU d… view at source ↗
Figure 5
Figure 5. Figure 5: Spatial-recall tasks across dimensionalities (real training samples). Every example is a [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance comparison of the implicit filter generation. Profiling results demonstrate [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: ImageNet Top-1 vs. compute. Each curve sweeps patch size p∈{16, 8, 4, 2}; smaller patches yield more tokens and more FLOPs. 256 1K 4K 1 2 ·10−1 VRMSE Gray-Scott React.-Diff. 1K 4K 16K 2 4 6 8 ·10−2 Active Matter 1K 4K 16K 3 6 9 ·10−2 Acoustic Scattering 512 4K 32K 1 2 3 ·10−1 MHD Attention HyenaND CNextU-net 512 4K 32K 2 4 6 ·10−1 Seq. length VRMSE Supernova Explosion 2K 8K 32K 0.5 1 1.5 ·10−1 Seq. length … view at source ↗
Figure 7
Figure 7. Figure 7: Performance evaluation of the three-kernel causal FFTConv1D approach based on Cooley [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 7
Figure 7. Figure 7: VRMSE vs. sequence length on The Well. Sequence length is controlled by patch size: smaller patches p yield more tokens. Each panel shows one dataset (lower is better). HyenaND (red) improves as sequence length grows on most datasets. CNextU-net (green dashed) is a fixed full-resolution baseline with no patch-size axis. For Shear Flow, the Attention p=2 result is omitted (undertrained run). HyenaND achieve… view at source ↗
Figure 8
Figure 8. Figure 8: Performance comparison of our custom causal-conv1d kernels against native PyTorch implementations. Our kernels deliver substantial efficiency gains while extending support for filter sizes up to 256 (for channel-first layouts) and 128 (for channel-last layouts), surpassing the filter length constraints of existing optimized packages. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance comparison of the implicit filter generation. Profiling results demonstrate [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Performance evaluation of the fused b2b causal-conv1d kernel for the SE and MR layers. By integrating the projection convolution, pre-gating, mixer convolution, and post-gating into a single operation, the fused kernel achieves more than a 7.5× runtime speedup and lowers the memory footprint. It also halves the necessary communication points in context-parallel configurations for the hyena operator [PITH_… view at source ↗
Figure 9
Figure 9. Figure 9: Performance evaluation of the three-kernel causal FFTConv1D approach based on Cooley [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Forward-pass performance evaluation of our 2D FFT convolution kernels for image [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance comparison of our custom causal-conv1d kernels against native PyTorch implementations. Our kernels deliver substantial efficiency gains while extending support for filter sizes up to 256 (for channel-first layouts) and 128 (for channel-last layouts), surpassing the filter length constraints of existing optimized packages. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Spatial-recall task exemplars. Each cell shows a sample input canvas (left) and the asso￾ciated regression target (right) for a given task and dimensionality. Top row (simple copy): one EM￾NIST digit is placed on an otherwise blank canvas; the model must regress the digit at a fixed readout location (2D: black square bottom-right; 3D: dashed red plane). Bottom row (color conditioning): four digits scatter… view at source ↗
Figure 11
Figure 11. Figure 11: Performance evaluation of the fused b2b causal-conv1d kernel for the SE and MR layers. By integrating the projection convolution, pre-gating, mixer convolution, and post-gating into a single operation, the fused kernel achieves more than a 7.5× runtime speedup and lowers the memory footprint. It also halves the necessary communication points in context-parallel configurations for the hyena operator [PITH… view at source ↗
Figure 12
Figure 12. Figure 12: Forward-pass performance evaluation of our 2D FFT convolution kernels for image [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Motion dataset construction (consecutive depth ( [PITH_FULL_IMAGE:figures/full_fig_p027_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: 16M long-convergence validation NLL (5-eval running average): training converges, [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Position-binned per-token NLL of the converged 16M model on held-out stitched-genome [PITH_FULL_IMAGE:figures/full_fig_p032_15.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 7 linked inside Pith

  1. [1]

    Blelloch

    Guy E. Blelloch. Prefix sums and their applications. Technical Report CMU-CS-90-190, School of Computer Science, Carnegie Mellon University, 1990

    1990

  2. [2]

    Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A

    Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A. Gonzalez, Samuel H. King, David B. Li, Aditi T. Merchant, Mohsen Naghipourfar, Eric Nguyen, Chiara Ricci-Tam, David W. Romero, Gwanggyu Sun, Ali Taghibakshi, Anton V orontsov, Brandon Yang, Myra Deng, Liv Gorton, Nam Nguyen, Nicholas K. Wang, Etowah Adams, St...

    2025

  3. [3]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InInternational Conference on Machine Learning (ICML), 2024

    2024

  4. [4]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neu- ral Information Processing Systems, volume 35, pages 16344–16359. Curran Associates, Inc., 2022. URL https://proceedi...

    2022

  5. [5]

    Vision transformers need registers, 2023

    Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023

    2023

  6. [6]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations, 2021

    2021

  8. [8]

    Hungry hungry hippos: Towards language modeling with state space models

    Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher R ´e. Hungry hungry hippos: Towards language modeling with state space models. InInternational Conference on Learning Representations, 2023

    2023

  9. [9]

    Fu, Hermann Kumbong, Eric Nguyen, and Christopher R´e

    Daniel Y . Fu, Hermann Kumbong, Eric Nguyen, and Christopher R´e. FlashFFTConv: Efficient convolutions for long sequences with tensor cores. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  10. [10]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 12

    Pith/arXiv arXiv 2023

  11. [11]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations, 2022

    2022

  12. [12]

    Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images

    Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. InInternational MICCAI Brainlesion Workshop, pages 272–284. Springer, 2022

    2022

  13. [13]

    Squeeze-and-excitation networks

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018

    2018

  14. [14]

    John, Dejun Lin, Polina Binder, Malcolm Greaves, Vega Shah, John St

    Peter St. John, Dejun Lin, Polina Binder, Malcolm Greaves, Vega Shah, John St. John, Adrian Lange, Patrick Hsu, Rajesh Illango, Arvind Ramanathan, Anima Anandkumar, David H Brookes, Akosua Busia, Abhishaike Mahajan, Stephen Malina, Neha Prasad, Sam Sinai, Lindsay Edwards, Thomas Gaudelet, Cristian Regep, Martin Steinegger, Burkhard Rost, Alexander Brace, ...

    arXiv 2025

  15. [15]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational Conference on Machine Learning, pages 5156–5165. PMLR, 2020

    2020

  16. [16]

    Romero, Garyk Brixi, Brandon Yang, Anton V orontsov, Ali Taghibakhshi, Amy X

    Jerome Ku, Eric Nguyen, David W. Romero, Garyk Brixi, Brandon Yang, Anton V orontsov, Ali Taghibakhshi, Amy X. Lu, Dave P. Burke, Greg Brockman, Stefano Massaroli, Christopher R´e, Patrick D. Hsu, Brian L. Hie, Stefano Ermon, and Michael Poli. Systems and algorithms for convolutional multi-hybrid language models at scale, 2025. URL https://arxiv.org/abs/ ...

    Pith/arXiv arXiv 2025

  17. [17]

    Mamba-ND: Selective state space modeling for multi-dimensional data

    Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-ND: Selective state space modeling for multi-dimensional data. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  18. [18]

    Pants: The pancreatic tumor segmentation dataset

    Wenxuan Li, Xinze Zhou, Qi Chen, Tianyu Lin, Pedro RAS Bassi, Xiaoxi Chen, Chen Ye, Zheren Zhu, Kai Ding, Heng Li, et al. Pants: The pancreatic tumor segmentation dataset. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025

    2025

  19. [19]

    VMamba: Visual state space model.arXiv preprint arXiv:2401.10166, 2024

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. VMamba: Visual state space model.arXiv preprint arXiv:2401.10166, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021

    2021

  21. [21]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022

    2022

  22. [22]

    Downs, Preey Shah, Tri Dao, Stephen A

    Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen A. Baccus, and Christopher R´e. S4ND: Modeling images and videos as multidimensional signals with state spaces. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. 13

    2022

  23. [23]

    Durrant, Brian Kang, Dhruva Katrekar, David B

    Eric Nguyen, Michael Poli, Matthew G. Durrant, Brian Kang, Dhruva Katrekar, David B. Li, Liam J. Bartie, Armin W. Thomas, Samuel H. King, Garyk Brixi, Jeremy Sullivan, Madelena Y . Ng, Ashley Lewis, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hernandez-Boussard, Christopher R´e, Patrick D. Hsu, and Brian L. Hie. Sequence modeling and design from mol...

    doi:10.1126/science.ado9336 2024

  24. [24]

    NVIDIA Tesla V100 GPU Architecture

    NVIDIA Corporation. NVIDIA Tesla V100 GPU Architecture. Technical Report WP- 08608-001 v1.1, NVIDIA Corporation, August 2017. URL https://images.nvidia. com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf . Whitepaper

    2017

  25. [25]

    NVIDIA A100 Tensor Core GPU Architecture

    NVIDIA Corporation. NVIDIA A100 Tensor Core GPU Architecture. Technical Report V1.0, NVIDIA Corporation, 2020. URL https://images.nvidia.com/aem-dam/en-zz/ Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf . Whitepa- per

    2020

  26. [26]

    NVIDIA H100 Tensor Core GPU Architecture

    NVIDIA Corporation. NVIDIA H100 Tensor Core GPU Architecture. Technical Report V1.03, NVIDIA Corporation, 2022. URL https://nvdam.widen.net/s/95bdhpsgrs/nvidia_ h100_tensor_core_gpu_architecture_whitepaper_v1.03. Whitepaper

    2022

  27. [27]

    NVIDIA Blackwell Architecture Technical Overview

    NVIDIA Corporation. NVIDIA Blackwell Architecture Technical Overview. Tech- nical report, NVIDIA Corporation, 2024. URL https://resources.nvidia.com/ en-us-blackwell-architecture . Technical overview; covers GB200 Grace Blackwell Superchip, HGX B200, and HGX B100

    2024

  28. [28]

    NVIDIA Corporation, cuda toolkit 13.2; cufft 12.2.0.46 edition, 2026

    NVIDIA Corporation.cuFFT API Reference. NVIDIA Corporation, cuda toolkit 13.2; cufft 12.2.0.46 edition, 2026. URL https://docs.nvidia.com/cuda/cufft/index.html. Last updated 2026-04-02; accessed 2026-04-30

    2026

  29. [29]

    NVIDIA Corporation, mathdx 26.03.0 (cufftdx 1.7.0) edition, 2026

    NVIDIA Corporation.cuFFTDx Documentation. NVIDIA Corporation, mathdx 26.03.0 (cufftdx 1.7.0) edition, 2026. URL https://docs.nvidia.com/cuda/cufftdx/index. html. Accessed: 2026-04-30

    2026

  30. [30]

    CUTLASS: CUDA Templates for Linear Algebra Subroutines

    NVIDIA Corporation. CUTLASS: CUDA Templates for Linear Algebra Subroutines. https: //github.com/NVIDIA/cutlass, 2026. Version 4.3.5; released 2026-01-09; accessed 2026- 04-30

    2026

  31. [31]

    Agocs, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B

    Ruben Ohana, Michael McCabe, Lucas Meyer, Rudy Morel, Fruzsina J. Agocs, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B. Dalziel, Drummond B. Fielding, Daniel Fortunato, Jared A. Goldberg, Keiya Hirashima, Yan-Fei Jiang, Rich R. Kerswell, Suryanarayana Maddu, Jonah Miller, Payel Mukhopadhyay, Stefan S. Nixon, Jeff Shen, Romain Watteaux, Brun...

    2024

  32. [32]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Cao, Xuan Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. InEmpirical Methods in Natural Language Processing, 2023

    2023

  33. [33]

    FiLM: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. InAAAI Conference on Artificial Intelligence, 2018

    2018

  34. [34]

    Hyena hierarchy: Towards larger convolutional language models

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R´e. Hyena hierarchy: Towards larger convolutional language models. InInternational Conference on Machine Learning, pages 28043–28078. PMLR, 2023

    2023

  35. [35]

    Romero, Robert-Jan Bruintjes, Jakub M

    David W. Romero, Robert-Jan Bruintjes, Jakub M. Tomczak, Erik J. Bekkers, Mark Hoogen- doorn, and Jan C. van Gemert. FlexConv: Continuous kernel convolutions with differentiable kernel sizes. InInternational Conference on Learning Representations (ICLR), 2022. 14

    2022

  36. [36]

    Romero, Anna Kuzina, Erik J

    David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub M. Tomczak, and Mark Hoogendoorn. CKConv: Continuous kernel convolution for sequential data. InInternational Conference on Learning Representations (ICLR), 2022

    2022

  37. [37]

    Implicit neural representations with periodic activation functions

    Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. InAdvances in Neural Information Processing Systems, volume 33, pages 7462–7473, 2020

    2020

  38. [38]

    Smith, Andrew Warrington, and Scott W

    Jimmy T.H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. InInternational Conference on Learning Representations, 2023

    2023

  39. [39]

    HyenaPixel: Global image context with convolutions, 2024

    Julian Spravil, Sebastian Houben, and Sven Behnke. HyenaPixel: Global image context with convolutions, 2024. URLhttps://arxiv.org/abs/2402.19305

    arXiv 2024

  40. [40]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Tengchao Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Spike no more: Stabilizing the pre-training of large language models, 2025

    Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabilizing the pre-training of large language models, 2025. URLhttps://arxiv.org/abs/2312.16903

    Pith/arXiv arXiv 2025

  42. [42]

    Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv´e J´egou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021

    2021

  44. [44]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in neural information processing systems, volume 30, 2017

    2017

  45. [45]

    Scaling laws in patchification: An image is worth 50,176 tokens and more.arXiv preprint arXiv:2502.03738, 2025

    Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Scaling laws in patchification: An image is worth 50,176 tokens and more.arXiv preprint arXiv:2502.03738, 2025

    arXiv 2025

  46. [46]

    Vit-5: Vision transformers for the mid-2020s.arXiv preprint arXiv:2602.08071, 2026

    Feng Wang, Sucheng Ren, Tiezheng Zhang, Predrag Neskovic, Anand Bhattad, Cihang Xie, and Alan Yuille. Vit-5: Vision transformers for the mid-2020s.arXiv preprint arXiv:2602.08071, 2026

    arXiv 2026

  47. [47]

    Resnet strikes back: An improved baseline in 2.5 lines of code, 2021

    Ross Wightman, Hugo Touvron, and Herv´e J´egou. Resnet strikes back: An improved baseline in 2.5 lines of code, 2021

    2021

  48. [48]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. InInternational Conference on Learning Representations, volume 2025, pages 29687–29707, 2025

    2025

  49. [49]

    Flashattention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling,

    Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. Flashattention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling,

  50. [50]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  51. [51]

    Squeeze-and-attention networks for semantic segmentation

    Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei- Shi Zheng, Jonathan Li, and Alexander Wong. Squeeze-and-attention networks for semantic segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  52. [52]

    Vision Mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision Mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale,

    Itamar Zimerman and Lior Wolf. Multi-dimensional Hyena for spatial inductive bias. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024. 15 A Appendix: Theory A.1 Gated Hyena Computes Separable Attention We show that a gated HyenaND layer computes exactly the class ofseparable attention matrices— attention patterns that fa...

    arXiv 2024

  54. [2026]

    URLhttps://arxiv.org/abs/2603.05451

    arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.