Pith. sign in

REVIEW 4 major objections 4 minor 18 references

ENA: Efficient N-dimensional Attention

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hybrid of linear recurrence and tiled local attention matches Transformer accuracy on images and video using only about 30 percent of the sequence per window.

desk verdict Sensible hybrid architecture, but the preprint is a skeleton—most evidence tables are placeholder digits, so the central claims are not yet checkable. read the letter →

arxiv 2508.11921 v1 pith:7P7PRZIO submitted 2025-08-16 cs.LG cs.AIcs.CV

Yibo Zhong This is my paper
classification cs.LGcs.AIcs.CV
keywords linearrecurrencehybridarchitectureslidingtileattentionhigh-orderdatasparsityimageclassificationvideogeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to carry linear recurrent sequence models, built for language, into N-dimensional data such as images and video. It argues that the standard bridge, permuting the input into derived sequences (scanning), costs more than it returns, and that the effective bridge is hybridization: alternate layers of a linear recurrent state with layers of attention, and replace full attention with tiled high-order sliding window attention once sequences grow long. The central empirical claim is that each token needs to attend to only about 30% of the sequence — roughly 70% sparsity — to match full attention, because the recurrent state already compresses the remaining global context. If true, the result is a simple, practical recipe for ultra-long high-order modeling at Transformer-level accuracy with faster training and inference.

What carries the argument

The load-bearing object is the block-wise hybrid token mixer: within each block the mixer is chosen as $M_{\text{block}} = \text{linear}$ for odd block indices and $M_{\text{block}} = \text{local attention}$ for even indices (Eqs. 4–5), so global compression and local refinement alternate through the network. The linear recurrence — DeltaNet in the main experiments — is the global state: it folds the whole sequence into a fixed-size state in linear time. The attention side is Sliding Tile Attention (STA), a coarser-grained high-order sliding window attention in which tokens within one tile share a window; by matching the tile size to the attention kernel's block size it avoids generating mix

What would settle it

Two concrete tests would settle the claim. First, take a trained ENA full-attention model and measure the total attention probability mass that falls outside the local window kept at 70% sparsity; if a large share of attention mass lies outside that window, the claim that 30% of tokens suffice is false. Second, apply a fixed global permutation to image patches at inference time on a model trained at 70% sparsity: if accuracy falls far below the full-attention hybrid on the same permuted input, the locality assumption that underlies the paper's efficiency gain is refuted.

Watch

Extended reading notes

Core claim

The paper's core claim is that the dimensional gap for linear recurrent models is best bridged not by scanning but by hybridization. In the proposed architecture, Efficient N-dimensional Attention (ENA), alternating layers run a linear recurrent model (primarily DeltaNet) that compresses global information into a fixed-size state, and attention layers that provide strictly local refinement; for long sequences the attention is Sliding Tile Attention (STA), a hardware-efficient sliding window attention that shifts the window tile-by-tile so the attention map contains no mixed blocks and real speedup over FlashAttention is achieved. Systematic comparisons on ImageNet show scanning gives margina

Load-bearing premise

The load-bearing premise is stated in Section 4.5.4: images and videos have strong locality, so a window covering about 30% of the tokens captures nearly all the interactions that matter, with the recurrent state carrying the rest; this premise is validated only on ImageNet and Kinetics-400, both strongly local, and if a high-order task requires long-range non-local interactions, the 70%-sparsity rule collapses and ENA's efficiency advantage over full-attention hybrids is los

Editorial extensions

If this is right

  • A Transformer with half its layers replaced by linear recurrence — ENA with full attention — matches or slightly exceeds the pure Transformer on ImageNet classification (78.28% vs 78.51% top-1 at 30M parameters), so attention can be halved without losing accuracy.
  • At about 70% sparsity, where each token attends to a local window covering 30% of the sequence, ENA with STA matches the full-attention hybrid on ImageNet at sequence lengths 1K, 4K, and 16K, and on Kinetics-400 at 8K tokens; larger windows add compute without meaningful gains.
  • Because STA eliminates mixed attention blocks, the sparsity translates into real wall-clock savings: at 4096 tokens the ENA-STA model trains in about 10h55m versus 14h42m for the full-attention hybrid on the same hardware.
  • Scanning methods — including multi-pass cross-scan and single-pass permuted scans — yield little or no gain over the plain no-permutation baseline, so ENA drops scanning entirely and keeps a simple stacked architecture.
  • The N-dimensional locality must be respected: 2D/3D sliding windows that use locality along every data dimension clearly outperform 1D windows and block attention on images and video, and on generation ENA-STA reaches FID 4.87 (ImageNet 512x512), better than the full-attention hybrid (5.02) and the SiT baseline (5.88).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper validates the 70%-sparsity rule only on strongly local datasets; a test it does not run is to break locality — for instance, a fixed global permutation of image patches at inference — and show where the sparsity ceiling actually sits. Its own reasoning predicts accuracy will collapse once dependencies become long-range.
  • The paper treats linear recurrence and local attention as a division of labor — state carries the global, window carries the local — and notes the framework is compatible with any sub-quadratic token mixer. That implies the performance driver is the hybrid structure itself, not DeltaNet specifically, a claim the paper's own cross-model scans (HGRN, RetNet, Gated DeltaNet) partially support.
  • A testable extension the paper leaves implicit: the sparsity level, not the window size, is the portable hyperparameter. That predicts a single ~70% sparsity transfers across resolutions, frame counts, and even data modalities, and that fixing window size instead of sparsity should underperform — which the 16K-token finetuning results already gesture toward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ENA, a hybrid architecture combining linear recurrent models (primarily DeltaNet) with high-order sliding window attention implemented via Sliding Tile Attention (STA), for modeling high-dimensional data such as images and videos. The authors first compare scanning strategies and attention hybrids in pilot experiments, concluding that multi-pass scanning is not worth its cost and that attention hybrids are more promising. They then evaluate design choices on ImageNet classification, propose STA as a hardware-efficient local attention, and report experiments on video classification, image generation, video generation, distillation, and hardware efficiency. The main claims are that (i) a hybrid of half linear recurrence and half attention matches or exceeds Transformer accuracy, and (ii) roughly 70% attention sparsity is sufficient for a good efficiency/accuracy trade-off on high-order data.

Significance. If supported by complete evidence, the paper's contribution is practically useful: it provides a simple, dimension-agnostic recipe for replacing full attention in vision and video models with a linear recurrence plus local STA, potentially yielding substantial speedups with negligible accuracy loss. The systematic evaluation of scanning variants is also a useful reference point. The paper explicitly builds on machine-checked kernel implementations (FLA, Flex Attention), and the ImageNet/Fig. 6–9 curves suggest the direction is promising. However, the empirical core is incomplete: many main tables are unpopulated, the K400 video results are blank, the hardware benchmarks lack measurement details, and no direct hybrid baseline (e.g., MambaVision) is compared. These gaps currently prevent verification of the central claims.

major comments (4)
  1. [Tables 4–8, 11–14] Most of the central quantitative results are placeholder digits. Table 7, which is the key ImageNet-4096 comparison for the STA speed/accuracy claim, reports accuracies as '�����' and training time only for two rows; Table 8 (K400 video classification) has no Top-1 values in any row; Tables 11–13 are entirely blank; Table 14 is blank. The text nevertheless draws conclusions from these tables, e.g., §3.3 says 'STA hybrid achieves performance comparable to the full attention hybrid' and §4.5.4 asserts a ~70% sparsity rule. The only populated support for that rule is Fig. 9. This is a load-bearing omission: the reader cannot verify the abstract's accuracy and efficiency claims. Please populate all result tables or explicitly mark them as unavailable, and remove or qualify conclusions that depend on missing data.
  2. [§4.5.4 with Tables 7–8 and §4.3] The 'strong locality' / ~70% sparsity recommendation is validated only on ImageNet classification. The K400 results that would test the rule on video are blank, and the 2D generation experiment (§4.3) deliberately uses a window covering most tokens (sparsity roughly 44%, not 70%). The text states this rule as a general guideline for high-order data, but no experiment demonstrates that non-local high-order tasks (e.g., temporal dynamics in Something-Something v2, or long-range spatial relations in high-resolution imagery) tolerate 70% sparsity. The stress-test concern therefore lands: the generality of the 70% rule is unsupported. Please add an experiment on a non-locally-dominated task, or explicitly limit the recommendation to datasets with strong local structure.
  3. [Fig. 2 and §4.8] The hardware efficiency comparison lacks the measurement detail needed to assess the speedup claim. The caption says 'single GPU' and 'batch size 4', but does not state the GPU model, software versions, input shapes, or whether times include warmup/compile. Moreover the ENA components are Triton-based while FA2 is CUDA-optimized; the text concedes that CUDA/ThunderKittens kernels could improve ENA further. Thus the reported speedups are implementation-bounded and may not reflect the architecture's potential. Please report a precise measurement protocol and, ideally, a kernel-level comparison when both implementations use the same language/optimization level.
  4. [§5 (Related Works) and experimental baselines] MambaVision, TTT-MLP, and LaCT are cited as existing linear-recurrence + block-attention hybrids, but no experimental comparison with any of these is included. Since the paper's central practical claim is that ENA's high-order STA hybrid is better/faster than existing hybrids, the absence of a direct baseline makes the incremental contribution unquantified. Please add a comparison to at least one block-attention hybrid under the same training setup, or clearly state why such a comparison is infeasible.
minor comments (4)
  1. [Eq. (4) and general formatting] Several equations and table captions contain placeholder glyphs (e.g., '�') that obscure the definitions. Please fix the math rendering and replace all placeholder symbols with concrete numbers or variables.
  2. [Table 1] The caption mentions 'marked in light blue and light red', but the table appears monochrome. Use visible labels or patterns to distinguish the newly introduced scan types.
  3. [§2.1] The sentence 'multi-head scanning methods ... yield no observable performance gains over a simple uni-scan in our experiments..' has a doubled period and is slightly redundant; rewrite for clarity.
  4. [Reproducibility] No code or checkpoints are provided in the manuscript, despite the top-of-paper links. For a journal submission, please include code release or, at minimum, complete hyperparameter/configurations for all experiments (seeds, schedule details, augmentation, etc.).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ENA's claims are empirical and rely on external components; the K400 evidence gap is a correctness risk, not a circular derivation.

full rationale

The paper's derivation chain is empirical and modular rather than definitional. ENA is defined as an alternating hybrid of a linear recurrent mixer and local attention (Eqs. 4–5), and its performance is measured against Transformers on ImageNet (Tables 3, 6, 7), generation (Tables 9–10), and video (Table 8). No prediction is obtained by construction: the 70% sparsity recommendation in §4.5.4 is a hyperparameter choice read from ImageNet sweeps (Figs. 6–9), not a fitted parameter renamed as a prediction. Each sparsity-versus-accuracy curve is independently measured, so the conclusion that ~70% sparsity matches full attention is an empirical observation, not a numerical tautology. The paper adopts DeltaNet (Yang et al., 2024b) and STA (Zhang et al., 2025a) as external components; because the author has no overlapping prior work, these citations are not load-bearing self-citations. There is no imported uniqueness theorem, no ansatz smuggled through citation, and no renaming of a known result as a new derivation. The main defect is evidential rather than circular: Table 8's K400 accuracy values are unpopulated, so the 3D conclusions in §4.2 are unsupported, and the locality assumption underpinning the 70% sparsity rule is validated only on ImageNet. However, unsupported generalization is a correctness risk, not a circular derivation. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on domain assumptions about locality in images and videos, fairness of the uni-scan baseline, and the adequacy of a Flex Attention-based STA proxy. The sparsity level (70%), attention-layer fraction (0.5), and tile sizes are tuned on ImageNet and then presented as general design guidance; no new theoretical entities are introduced.

free parameters (3)
  • attention sparsity level = ~70% (window covers ~30% of tokens; e.g., 48x24 at sequence length 4096, 12x12 at 1024)
    Chosen on ImageNet after observing diminishing returns; generalized to a design rule in Section 4.5.4.
  • fraction of attention layers in hybrid = 0.5 (6 of 12 layers)
    Selected from ImageNet sweeps (Table 3: interleaved layers 0,2,4,6,8,10 gives 78.28, close to Transformer 78.51); presented as the default ENA configuration.
  • STA tile size = 16x8 (and 4x4 in some experiments)
    Set to match Flex Attention block size to avoid mixed blocks (Section 3.3, Section 4.2).
assumptions (3)
  • domain assumption High-order data (images, videos) have strong locality such that local attention over ~30% of tokens preserves accuracy comparable to full attention.
    Invoked in Section 4.5.4 to justify the 70% sparsity recommendation and ENA's local-attention design.
  • domain assumption A causal linear model applied in raster order (uni-scan, no permutation) is a fair and representative baseline for adapting linear recurrence to ND data.
    Used in Section 3.1 to conclude scanning offers no benefit; if this baseline is weak, the scanning comparison is unfair.
  • domain assumption Flex Attention with tile-aligned windows faithfully implements high-order sliding window attention (STA) and preserves its performance.
    Section 3.3 footnote: authors implement STA using Flex Attention while preserving the high-level design; hardware conclusions rely on this proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ENA: Efficient N-dimensional Attention." pith.science (2026). https://pith.science/paper/7P7PRZIO

@misc{pith2026250811921,
  author       = {Pith},
  title        = {Pith review of: ENA: Efficient N-dimensional Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7P7PRZIO}},
  note         = {Machine review of arXiv:2508.11921}
}
read the original abstract

Efficient modeling of long sequences of high-order data requires a more efficient architecture than Transformer. In this paper, we investigate two key aspects of extending linear recurrent models, especially those originally designed for language modeling, to high-order data (1D to ND): scanning strategies and attention-hybrid architectures. Empirical results suggest that scanning provides limited benefits, while attention-hybrid models yield promising results. Focusing on the latter, we further evaluate types of attention and find that tiled high-order sliding window attention (SWA) is efficient in both theory and practice. We term the resulting hybrid architecture of linear recurrence and high-order SWA as Efficient N-dimensional Attention (ENA). We then conduct several experiments to demonstrate its effectiveness. The intuition behind ENA is that linear recurrence compresses global information into a state, while SWA complements it by enforcing strict local modeling. Together, they form a simple framework that offers a promising and practical solution for ultra-long high-order data modeling.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 16 linked inside Pith

  1. [1]

    Simple linear attention language models balance the recall-throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher R ´e. Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668,

    arXiv

  2. [5]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv

  3. [6]

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi

    URL https://arxiv.org/abs/2506.04761. Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention trans- former. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6185–6194,

    arXiv

  4. [7]

    Mambavision: A hybrid mamba-transformer vision backbone

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083,

    arXiv

  5. [8]

    Cogvideo: Large-scale pre- training for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868,

    arXiv

  6. [10]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048,

    arXiv

  7. [11]

    Rwkv-7” goose” with expressive dynamic state evolution

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, et al. Rwkv-7” goose” with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456,

    arXiv

  8. [12]

    Benjamin F

    URL https://arxiv.org/abs/2502.10297. Benjamin F. Spector, Simran Arora, Aaryan Singhal, Daniel Y . Fu, and Christopher R´e. Thunderkit- tens: Simple, fast, and adorable ai kernels,

    arXiv

Show all 18 references
  1. [13]

    URL https://arxiv.org/abs/2410. 20399. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621,

    arXiv

  2. [14]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, an...

    arXiv

  3. [15]

    Feng Wang, Timing Yang, Yaodong Yu, Sucheng Ren, Guoyizhe Wei, Angtian Wang, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie

    URL https://arxiv.org/abs/2506.05233. Feng Wang, Timing Yang, Yaodong Yu, Sucheng Ren, Guoyizhe Wei, Angtian Wang, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Causal image modeling for efficient visual under- standing. arXiv preprint arXiv:2410.07599,

    arXiv

  4. [16]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024c. Sihyun Yu, Sangkyung Kwak, Hu...

    arXiv

  5. [17]

    Fast video generation with sliding tile attention

    Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507, 2025a. Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkava...

    arXiv

  6. [18]

    Dig: Scalable and efficient diffusion models with gated linear attention

    Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xing- gang Wang. Dig: Scalable and efficient diffusion models with gated linear attention. arXiv preprint arXiv:2405.18428,

    arXiv

  7. [2017]

    Songhua Liu, Zhenxiong Tan, and Xinchao Wang

    URL https://arxiv.org/ abs/1705.06950. Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Clear: Conv-like linearization revs pre-trained diffusion transformers up. arXiv preprint arXiv:2412.16112,

    arXiv

  8. [2023]

    Flex attention: A programming model for generating optimized attention kernels.arXiv preprint arXiv:2412.05496,

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels.arXiv preprint arXiv:2412.05496,

    arXiv

  9. [2024]

    One-minute video generation with test-time training

    Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. arXiv preprint arXiv:2504.05298,

    arXiv

  10. [2025]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.