Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that linear-complexity autoregressive image generation can match or beat quadratic softmax attention if the attention decay is computed from true 2D positions rather than 1D sequence positions.

desk verdict Promising spatial-decay mask for linear-attention AR image generation, but the central mechanism is specified contradictorily (carry vs reset) and the paper cannot be validated without code. read the letter →

arxiv 2507.01652 v1 pith:FD4YC32N submitted 2025-07-02 cs.CV cs.AIcs.MM

Yuxin Mao , Zhen Qin , Jinxing Zhou , Hui Deng , Xuyang Shen , Bin Fan , Jing Zhang , Yiran Zhong
show 1 more author
Yuchao Dai
This is my paper
classification cs.CVcs.AIcs.MM
keywords autoregressiveimagegenerationlinearattentionspatial-awaredecayrecurrentstateNetclass-conditionalefficientLlamaGen2Dspatialcontext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LASADGen is an autoregressive image generator that replaces quadratic softmax attention with linear-complexity gated linear attention plus a Spatial-Aware Decay (SAD) term. The paper argues that naive linear attention and existing decay-based linear attention from language modeling fail on images because raster-scan flattening destroys the true 2D layout: two patches far apart in the image can sit side by side in the token sequence. SAD makes the decay factor a function of real spatial location and special-cases row boundaries, so the recurrent state respects image geometry at linear cost. On 256x256 ImageNet, LASADGen-B reaches FID 4.86 versus LlamaGen-B's 5.46 and LASADGen-L reaches FID 2.90 versus LlamaGen-L's 3.80, with inference time that scales linearly in sequence length. A sympathetic reader would take this as evidence that linear attention's efficiency can be combined with the spatial understanding quadratic attention once provided.

What carries the argument

The central object is the Spatial-Aware Decay (SAD) mask applied inside the recurrent linear-attention update $s_t = \mathrm{diag}(\lambda^{\mathrm{spatial}}_t) s_{t-1} + k_t v_t^\top$, with output $o_t^\top = q_t^\top s_t$. The base decay comes from the key through $\lambda_t = 1 - k_t$ (an HGRN2-style parameter sharing), and SAD modulates it by the indicator in Eq. (12): inside a row the decay is the learned value, while at row endpoints $t \bmod w = 0$ the decay factor is set to 1. Because the state is $d \times d$, the whole recurrence has linear complexity in the token count; the mask is the only new machinery, and it is what preserves 2D adjacency information that a 1D decay would distort.

What would settle it

Train or rerun LASADGen-B with the boundary rule flipped so the state is truly reset ($\lambda^{\mathrm{spatial}}_t = 0$ at row ends) instead of carried ($\lambda^{\mathrm{spatial}}_t = 1$), and compare FID with the reported 4.86; a match would show the headline gain is insensitive to the exact boundary rule, while a large gap would identify which of the two contradictory descriptions actually delivered the result.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that flattened-image linear attention fails not because it is linear but because its decay is one-dimensional: the same exponential discounting that works for text treats a token's horizontal neighbor and the next row's first token identically, so genuine spatial adjacency is lost. The proposed fix, Spatial-Aware Decay, computes the decay from the actual 2D coordinate and intervenes at the seam where one raster row ends and the next begins; in the recurrence written as Eq. (10), this changes $\lambda_t$ to $\lambda^{\mathrm{spatial}}_t$ at those positions. With that single modification, the same generator that scored FID 7.42 without SAD and 50.86 with plain linear attention reaches 4.86 FID at base scale, outperforming the quadratic softmax baseline at every model size tested, and the mask also improves a GLA baseline from 6.45 to 5.84 FID.

Load-bearing premise

The load-bearing premise is that the reported gains come from the row-boundary decay rule, but the paper describes that rule two ways—as carrying the state fully and as resetting it—so without a corrected specification the mechanism behind the numbers is ambiguous.

Editorial extensions

If this is right

  • Autoregressive image generation no longer needs quadratic attention to reach competitive FID: at 111M/112M parameters LASADGen-B's FID 4.86 beats LlamaGen-B's 5.46, and at 343M/346M LASADGen-L's 2.90 beats LlamaGen-L's 3.80.
  • Existing decay-based linear attention mechanisms (TNL, GLA) transfer poorly from language to images, but equipping them with spatial awareness helps, as shown by GLA-B improving from 6.45 to 5.84 FID when SAD is added.
  • Because the model keeps the autoregressive next-token framework and only swaps the attention, scaling behavior is retained: larger LASADGen models give better FID and lower training loss from 111M to 1.4B parameters.
  • The linear-complexity recurrence makes inference faster than quadratic transformers of comparable size at the same FID budget, which is the practical payoff for image generation at higher resolutions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I read Eq. (12) and Algorithm 1 as carrying the recurrent state unchanged across row seams ($\lambda^{\mathrm{spatial}}_t = 1$), which is the opposite of the paper's prose about resetting accumulated state; if the released implementation carries the state, the gains may come from letting horizontal context flow into the next row, not from blocking cross-row leakage.
  • The boundary-mask idea should transfer to any rasterized or flattened structured data, such as video frames, feature maps, or multi-scale token grids, where a 1D ordering artificially separates adjacent cells; testing SAD there would show whether the benefit is generic geometry rather than ImageNet-specific.
  • A natural variant would make the boundary rule learned rather than fixed, letting the model decide at each row seam how much state to carry; the paper's ablations do not compare against learned boundary gates, so it is unknown whether the fixed rule is optimal or merely sufficient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LASAD, a linear-attention mechanism with a spatial-aware decay factor for autoregressive image generation. The mechanism modifies HGRN2-style linear attention by changing the decay value at row boundaries of a raster-scanned token sequence, with the stated goal of preserving genuine 2D spatial relationships after flattening. The authors instantiate LASADGen on the LlamaGen framework and report strong ImageNet 256×256 results (e.g., LASADGen-B FID 4.86 vs. LlamaGen-B 5.46; LASADGen-L FID 2.90 vs. LlamaGen-L 3.80) at linear computational complexity. The paper includes ablation studies (Table 4) attributing consistent FID gains to the Spatial-Aware Decay mask, scaling experiments across model sizes, and an inference-efficiency comparison.

Significance. If the mechanism were specified correctly and the results reproduced, this would be a valuable step toward efficient autoregressive image generation: it identifies a concrete failure mode of 1D decay in linear attention for 2D data and proposes a simple, cheap fix with measured FID gains across model scales. The diagnostic comparison of standard linear attention, hybrid attention, TNL, and GLA (Table 2) is useful and clearly motivates the problem. However, the central definition of SAD is internally contradictory: Eq. (12) and Algorithm 1 describe a different operation from the one explained in Sec. 3.4, and no code is provided to resolve the ambiguity. As a result, the reader cannot currently determine which mechanism produced the reported FID numbers, which undermines the paper's main claim as written.

major comments (3)
  1. [Sec. 3.4, Eq. (12), Algorithm 1] The formal specification contradicts the stated objective. In Eq. (12), I(t mod w) is 0 when t mod w = 0, so log λ_spatial_t = 0 and hence λ_spatial_t = 1 at row ends; Algorithm 1 line 11 explicitly sets λ_spatial_t ← 1 at exactly those positions. In the recurrence s_t = diag(λ_spatial_t)s_{t−1} + k_t v_t^T, this carries the accumulated state forward with no decay at the end of each row. The text in Sec. 3.4 states that this 'resets accumulated state between rows' and 'prevents inappropriate information propagation' between non-adjacent tokens; a true reset would require λ_spatial = 0 at the first token of the next row (or at the boundary), not λ_spatial = 1 at the last token of the current row. Because Table 4 attributes substantial FID gains to this mechanism, the reader cannot tell which behavior was actually run. Please correct Eq. (12), Algorithm 1, and Figure 1 so that they match the verbal description, state precisely which implementation produced the reported numbers, and provide code or re-run the ablations if the implemented behavior differs from the corrected specification.
  2. [Sec. 3.4, recurrence (10)–(12)] Even setting aside the 0-versus-1 issue, the position at which the mask is applied is inconsistent with the stated goal. The problematic transition is between token t=iw (end of row i) and token t=iw+1 (start of row i+1). In the recurrence, λ_spatial_t multiplies the state when forming s_t, so modifying λ_spatial at t=iw affects the carry within row i, not the carry across the boundary between rows. Setting λ_spatial_{iw}=0 would still allow token iw to contribute to s_{iw+1} through the learned decay at t=iw+1; preventing cross-row propagation requires controlling the decay at t=iw+1 (or resetting the state before that step). Please clarify whether the intended mask position is the first token of each row (t mod w == 1) rather than the last token, and update the equations, algorithm, and experiments accordingly.
  3. [Sec. 4.5, Table 4] The ablation baseline is inconsistently reported. In Table 4, 'LASADGen ✗' has FID 7.42 at size B, which is worse than both TNL (6.75) and GLA (6.45) in Table 2, even though LASADGen is presented as an improved linear-attention architecture. If the 'without SAD' baseline is a specific choice of base recurrence (e.g., HGRN2), the paper should say so explicitly and explain why it underperforms GLA; otherwise, the large apparent gain from SAD (7.42 → 4.86) may partly reflect a weak base model rather than the SAD mask alone. This is needed to fairly interpret the central ablation.
minor comments (6)
  1. [Abstract and Sec. 1] The phrase 'large-scale language models (LLM)' should be pluralized as 'LLMs' for grammatical consistency.
  2. [Sec. 3.4, Eq. (12)] The notation I(t mod w) is ambiguous; please write the condition explicitly, e.g., I(t mod w = 0) or I(t mod w ≠ 0), depending on the intended behavior after resolving the major contradiction.
  3. [Figure 1] The figure legend says 'Set the decay factor (log scale) here to 0,' which corresponds to λ=1, while the caption says the mask 'prevents inappropriate information flow.' The visual explanation should be made consistent with the corrected definition of the mask.
  4. [Table 2] In the SoftmaxAttn row, the IS and precision values run together as '193.610.83'; please fix the spacing/formatting.
  5. [Figure 4] The labels '-0.17' and '-0.18' in the upper-left corner appear to be stray axis annotations; please remove or explain them.
  6. [Algorithm 1] Line 17 returns O = [o_1, o_2, ..., o_n] using a lowercase n, while the input length is denoted N elsewhere; please use a consistent symbol.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: LASAD is benchmarked externally on ImageNet FID, and the Spatial-Aware Decay mask is a hand-set design choice rather than a parameter fitted to the test set or to the reported FID.

full rationale

The derivation chain is: Eq. (7) defines ordinary causal linear attention; Eq. (8) adds a learned decay; Eq. (9) adopts the HGRN2 recurrence; Eqs. (10)-(12) augment that recurrence with a hand-set spatially-aware decay mask at row boundaries. The claimed contribution is the mask, and its effect is measured by the FID difference between otherwise identical models with and without SAD (Table 4). The mask is not a fitted parameter: the boundary rule (t mod w = 0) is specified a priori, and no quantities are calibrated to ImageNet validation FID. The reported numbers come from the standard external ADM evaluation protocol on ImageNet. Self-citations to HGRN2, TNL, and related linear-attention work provide the base recurrence, but those prior results were evaluated independently in language modeling and do not by themselves imply the image-generation FID improvements; the ablation isolates the new mask from that base. Note that the paper contains an internal inconsistency between the stated intent of 'resetting accumulated state' and the literal Eq. (12)/Algorithm 1, which set the decay factor to 1 (log-scale 0) at row boundaries, meaning full state carry rather than reset; this is a correctness/ambiguity concern, not a circularity, and it does not make the FID claim equivalent to the input by construction. No circular step of the enumerated kinds is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method introduces no fitted constants beyond standard trained weights. The only hand-chosen knob is the row-boundary mask (log decay set to 0 at t mod w = 0), whose value 1 (full carry) contradicts the text's 'reset' claim. The paper relies on standard autoregressive modeling, the HGRN2 recurrence, a pretrained LlamaGen tokenizer, and the Flash Linear Attention library as external inputs.

free parameters (1)
  • Row-boundary decay reset value (log lambda = 0 at t mod w = 0) = decay factor 1.0 (log decay 0)
    Hand-set rule applied at every row end; no derivation is given, and the prose describes it as a reset (which would be decay 0) while the equations set it to 1, so the chosen value is ambiguous.
assumptions (3)
  • domain assumption Raster-scan flattening and the token width w define the 2D structure; the mask is applied at positions t mod w = 0
    Sec. 3.1 defines the autoregressive token sequence via raster scan; the method assumes this ordering and the boundary positions.
  • standard math The HGRN2 recurrence (Eq. 9) is a valid base for gated linear attention with decay
    The paper builds directly on Eq. (9) from HGRN2 [25], a previously published gated linear RNN/attention formulation.
  • domain assumption The pretrained LlamaGen tokenizer (downsampling factor 16) and the Flash Linear Attention library are used as fixed external components
    Sec. 4.1 states the tokenizer leverages pre-trained LlamaGen weights and the implementation uses the Flash Linear Attention library [55].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective." pith.science (2026). https://pith.science/paper/FD4YC32N

@misc{pith2026250701652,
  author       = {Pith},
  title        = {Pith review of: Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FD4YC32N}},
  note         = {Machine review of arXiv:2507.01652}
}
read the original abstract

Autoregressive (AR) models have garnered significant attention in image generation for their ability to effectively capture both local and global structures within visual data. However, prevalent AR models predominantly rely on the transformer architectures, which are beset by quadratic computational complexity concerning input sequence length and substantial memory overhead due to the necessity of maintaining key-value caches. Although linear attention mechanisms have successfully reduced this burden in language models, our initial experiments reveal that they significantly degrade image generation quality because of their inability to capture critical long-range dependencies in visual data. We propose Linear Attention with Spatial-Aware Decay (LASAD), a novel attention mechanism that explicitly preserves genuine 2D spatial relationships within the flattened image sequences by computing position-dependent decay factors based on true 2D spatial location rather than 1D sequence positions. Based on this mechanism, we present LASADGen, an autoregressive image generator that enables selective attention to relevant spatial contexts with linear complexity. Experiments on ImageNet show LASADGen achieves state-of-the-art image generation performance and computational efficiency, bridging the gap between linear attention's efficiency and spatial understanding needed for high-quality generation.

Figures

Figures reproduced from arXiv: 2507.01652 by the authors.

Figure 1
Figure 1. Visualization of the Spatial-Aware Decay mechanism. Each color block represents the decay factor of the token at the cor￾responding position. Our method applies a mask (black in the fig￾ure) at row boundaries when raster scan flattening, preventing in￾appropriate information flow between non-adjacent image tokens. utilizes an outer product for expansion: s0 = 0, st = diag(λt)st−1 + (1 − λt)v ⊤ t , o ⊤ t = q ⊤ t st, … view at source ↗
Figure 2
Figure 2. Autoregressive Image Generation with LASADGen. We show the samples from our class-conditional LASADGen-XL model trained on ImageNet at 256 × 256 resolution. 50 100 150 200 250 300 Epochs 2 3 4 5 6 7 8 FID LASADGen-B LASADGen-L LASADGen-XL LASADGen-XXL 0 50 100 150 200 250 300 Epochs 7.0 7.5 8.0 8.5 9.0 Train Loss Initial Loss: 9.6 LASADGen-B LASADGen-L LASADGen-XL LASADGen-XXL [PITH_FULL_IMAGE:figures/full_fig_p006… view at source ↗
Figure 3
Figure 3. Scaling properties of LASADGen. Left: FID scores and Right: Training loss vs. model size. sic linear attention by directly substituting kernel functions for the softmax operation [20], while the latter employs a hybrid architecture [18] that intersperses only one standard attention layer among linear attention layers. Our results indicate that basic linear attention fails entirely to adapt to this task. The hybrid a… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training loss comparison between the model with or without the proposed Spatial-Aware Decay. We visualize the training process of the two groups of models in Tab. 4. The blue curve is used to represent the model without using SAD, and the red curve is used to represent…
Figure 5
Figure 5. Figure 5: Comparative Analysis of inference Speed. All tests are performed on 8 NVIDIA H800 GPUs. 5. Conclusion In this paper, we proposed LASADGen, a novel autoregres￾sive image generation model incorporating Linear Atten￾tion with Spatial-Aware Decay (LASAD). Our method ef￾fec…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PhysEDA: Physics-Aware Learning Framework for Efficient EDA With Manhattan Distance Decay

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    PhysEDA folds separable Manhattan-distance exponential decay into linear attention and potential-based rewards, cutting complexity to linear while improving zero-shot transfer and sparse-reward performance on decoupli...

  2. Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

    cs.MM 2025-08 conditional novelty 5.0 of 10

    A reasoning-based agent (multimodal LLM, then Grounding-DINO, then SAM2) achieves state-of-the-art Referring Audio-Visual Segmentation without pixel-level supervision.

  3. Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    An agentic video-reasoning framework, VITAL, uses tool-based frame sampling, multimodal chain-of-thought, new datasets, and a difficulty-aware RL algorithm to improve long-video QA and temporal grounding.

Reference graph

Works this paper leans on

59 extracted references · 34 canonical work pages · cited by 3 Pith papers

  1. [1]

    Diffusion models beat gans on image synthesis.Conference on Neural Infor- mation Processing Systems (NeurIPS), 34:8780–8794, 2021

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Conference on Neural Infor- mation Processing Systems (NeurIPS), 34:8780–8794, 2021. 1

    work page 2021

  2. [2]

    Cascaded diffusion models for high fidelity image generation.Journal of Ma- chine Learning Research (JMLR), 23(1):2249–2281, 2022

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation.Journal of Ma- chine Learning Research (JMLR), 23(1):2249–2281, 2022

    work page 2022

  3. [3]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InConference on Com- puter Vision and Pattern Recognition (CVPR), pages 10684– 10695, 2022. 2

    work page 2022

  4. [4]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InConference on Computer Vision and Pattern Recognition (CVPR), pages 4195–4205, 2023. 7

    work page 2023

  5. [5]

    Tavg- bench: Benchmarking text to audible-video generation

    Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Tavg- bench: Benchmarking text to audible-video generation. In ACM Multimedia Conference (ACM MM), pages 6607–6616, 2024

    work page 2024

  6. [6]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024. 2, 3, 5, 8

    arXiv 2024

  7. [7]

    You only scan once: Efficient multi-dimension sequential modeling with lightnet.arXiv preprint arXiv:2405.21022, 2024

    Zhen Qin, Yuxin Mao, Xuyang Shen, Dong Li, Jing Zhang, Yuchao Dai, and Yiran Zhong. You only scan once: Efficient multi-dimension sequential modeling with lightnet.arXiv preprint arXiv:2405.21022, 2024. 1

    arXiv 2024

  8. [8]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018. 1

    arXiv 2018

Show all 59 references
  1. [9]

    Scaling up gans for text-to-image synthesis

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. InConference on Com- puter Vision and Pattern Recognition (CVPR), pages 10124– 10134, 2023

    2023

  2. [10]

    Stylegan- xl: Scaling stylegan to large diverse datasets

    Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan- xl: Scaling stylegan to large diverse datasets. InACM SIG- GRAPH, pages 1–10, 2022. 1

    2022

  3. [11]

    Denoising diffu- sion probabilistic models.Conference on Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models.Conference on Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020. 1, 7

    2020

  4. [12]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 2

    2023 arXiv

  5. [13]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 2

    2018

  6. [14]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 1

    2024 arXiv

  7. [15]

    Attention is all you need.Conference on Neural Information Processing Systems (NeurIPS), 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Conference on Neural Information Processing Systems (NeurIPS), 30, 2017. 1, 3

    2017

  8. [16]

    Scaling laws for linear complexity language models.arXiv preprint arXiv:2406.16690, 2024

    Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models.arXiv preprint arXiv:2406.16690, 2024. 1

    2024 arXiv

  9. [17]

    Various lengths, constant speed: Ef- ficient language modeling with lightning attention

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Ef- ficient language modeling with lightning attention. InInter- national Conference on Machine Learning (ICML), 2024. 1, 2, 6, 7

    2024

  10. [18]

    Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025. 1, 6

    2025 arXiv

  11. [19]

    Jamba: A hybrid transformer-mamba language model.arXiv preprint arXiv:2403.19887, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model.arXiv preprint arXiv:2403.19887, 2024. 1

    2024 arXiv

  12. [20]

    Transformers are rnns: Fast autoregres- sive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregres- sive transformers with linear attention. InInternational Con- ference on Machine Learning (ICML), pages 5156–5165. PMLR, 2020. 1, 2, 3, 4, 6, 7

    2020

  13. [21]

    cosformer: Rethinking softmax in attention

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. InIn- ternational Conference on Learning Representations (ICLR),

  14. [22]

    Gated linear attention trans- formers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2023

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2023. 1, 2, 4, 6, 7, 8

    2023 arXiv

  15. [23]

    Retentive 9 network: A successor to transformer for large language mod- els.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive 9 network: A successor to transformer for large language mod- els.arXiv preprint arXiv:2307.08621, 2023. 2, 4

    2023 arXiv

  16. [24]

    Hierarchi- cally gated recurrent neural network for sequence model- ing.Conference on Neural Information Processing Systems (NeurIPS), 36, 2024

    Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchi- cally gated recurrent neural network for sequence model- ing.Conference on Neural Information Processing Systems (NeurIPS), 36, 2024. 2, 4

    2024

  17. [25]

    Hgrn2: Gated linear rnns with state expansion.arXiv preprint arXiv:2404.07904,

    Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion.arXiv preprint arXiv:2404.07904,

    arXiv

  18. [26]

    Minimax-m1: Scaling test-time com- pute efficiently with lightning attention.arXiv preprint arXiv:2506.13585, 2025

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time com- pute efficiently with lightning attention.arXiv preprint arXiv:2506.13585, 2025. 1

    2025 arXiv

  19. [27]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InConference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. Ieee, 2009. 2, 5

    2009

  20. [28]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. InConference on Computer Vision and Pattern Recognition (CVPR), pages 11315–11325, 2022. 2

    2022

  21. [29]

    Mage: Masked generative encoder to unify representation learning and image synthe- sis

    Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthe- sis. InConference on Computer Vision and Pattern Recogni- tion (CVPR), pages 2142–2152, 2023

    2023

  22. [30]

    Autoregressive image generation without vec- tor quantization.Conference on Neural Information Process- ing Systems (NeurIPS), 37:56424–56445, 2025

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization.Conference on Neural Information Process- ing Systems (NeurIPS), 37:56424–56445, 2025

    2025

  23. [31]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. InConference on Com- puter Vision and Pattern Recognition (CVPR), pages 10459– 10469, 2023. 2

    2023

  24. [32]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InAssociation for Com- putational Linguistics (ACL), pages 4171–4186, 2019. 2

    2019

  25. [33]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InConfer- ence on Computer Vision and Pattern Recognition (CVPR), pages 12873–12883, 2021. 2

    2021

  26. [34]

    Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627, 2021

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627, 2021. 2

    2021 arXiv

  27. [35]

    Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013. 2

    2013 arXiv

  28. [36]

    Neural discrete representation learning.Conference on Neural Information Processing Systems (NeurIPS), 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Conference on Neural Information Processing Systems (NeurIPS), 30, 2017. 2

    2017

  29. [37]

    Visual autoregressive modeling: Scalable im- age generation via next-scale prediction.arXiv preprint arXiv:2404.02905, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable im- age generation via next-scale prediction.arXiv preprint arXiv:2404.02905, 2024. 2

    2024 arXiv

  30. [38]

    Parallelized autoregressive visual generation.arXiv preprint arXiv:2412.15119, 2024

    Yuqing Wang, Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo, Zhenheng Yang, Difan Zou, Jiashi Feng, and Xihui Liu. Parallelized autoregressive visual generation.arXiv preprint arXiv:2412.15119, 2024. 2

    2024 arXiv

  31. [39]

    Next patch prediction for autoregressive visual generation.arXiv preprint arXiv:2412.15321, 2024

    Yatian Pang, Peng Jin, Shuo Yang, Bin Lin, Bin Zhu, Zhenyu Tang, Liuhan Chen, Francis EH Tay, Ser-Nam Lim, Harry Yang, et al. Next patch prediction for autoregressive visual generation.arXiv preprint arXiv:2412.15321, 2024

    2024 arXiv

  32. [40]

    Zipar: Accelerating au- toregressive image generation through spatial locality.arXiv preprint arXiv:2412.04062, 2024

    Yefei He, Feng Chen, Yuanyu He, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Accelerating au- toregressive image generation through spatial locality.arXiv preprint arXiv:2412.04062, 2024. 2

    2024 arXiv

  33. [41]

    Imagefolder: Autoregres- sive image generation with folded tokens.arXiv preprint arXiv:2410.01756, 2024

    Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregres- sive image generation with folded tokens.arXiv preprint arXiv:2410.01756, 2024. 2

    2024 arXiv

  34. [42]

    Scalable autoregressive image generation with mamba.arXiv preprint arXiv:2408.12245,

    Haopeng Li, Jinyue Yang, Kexin Wang, Xuerui Qiu, Yuhong Chou, Xin Li, and Guoqi Li. Scalable autoregressive image generation with mamba.arXiv preprint arXiv:2408.12245,

  35. [43]

    Elucidating the de- sign space of language models for image generation.arXiv preprint arXiv:2410.16257, 2024

    Xuantong Liu, Shaozhe Hao, Xianbiao Qi, Tianyang Hu, Jun Wang, Rong Xiao, and Yuan Yao. Elucidating the de- sign space of language models for image generation.arXiv preprint arXiv:2410.16257, 2024. 2

    2024 arXiv

  36. [44]

    Random feature atten- tion

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature atten- tion. InInternational Conference on Learning Representa- tions (ICLR), 2020. 2

    2020

  37. [45]

    Rethinking attention with perform- ers

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with perform- ers. InInternational Conference on Learning Representa- ti...

    2020

  38. [46]

    The devil in linear transformer

    Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Ling- peng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. InConference on Empirical Methods in Natural Language Processing (EMNLP), pages 7025–7041,

  39. [47]

    Transnormerllm: A faster and better large language model with improved transnormer

    Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, and Yiran Zhong. Transnormerllm: A faster and better large language model with improved transnormer. arXiv preprint arXiv:2307.14995, 2023. 4

    2023 arXiv

  40. [48]

    Improving the gating mechanism of re- current neural networks, 2020

    Albert Gua, Caglar Gulcehre, Tom le Paine, Razvan Pascanu, and Matt Hoffman. Improving the gating mechanism of re- current neural networks, 2020. 4

    2020

  41. [49]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. 4

    2023 arXiv

  42. [50]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Conference on Neural Information Processing Systems (NeurIPS), 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Conference on Neural Information Processing Systems (NeurIPS), 30, 2017. 5 10

    2017

  43. [51]

    Improved techniques for training gans.Conference on Neural Information Pro- cessing Systems (NeurIPS), 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Conference on Neural Information Pro- cessing Systems (NeurIPS), 29, 2016. 5

    2016

  44. [52]

    Improved precision and recall met- ric for assessing generative models.Conference on Neural Information Processing Systems (NeurIPS), 32, 2019

    Tuomas Kynk ¨a¨anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall met- ric for assessing generative models.Conference on Neural Information Processing Systems (NeurIPS), 32, 2019. 5

    2019

  45. [53]

    Diffusion models beat gans on image synthesis.Conference on Neural Infor- mation Processing Systems (NeurIPS), 34:8780–8794, 2021

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Conference on Neural Infor- mation Processing Systems (NeurIPS), 34:8780–8794, 2021. 5

    2021

  46. [54]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 5

    2022 arXiv

  47. [55]

    Fla: A triton-based library for hardware-efficient implementations of linear attention mech- anism, January 2024

    Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mech- anism, January 2024. 5

    2024

  48. [56]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

  49. [57]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Taming transformers for high-resolution image synthesis. InConfer- ence on Computer Vision and Pattern Recognition (CVPR), pages 12873–12883, 2021. 8

    2021

  50. [58]

    Vector-quantized image modeling with im- proved VQGAN

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with im- proved VQGAN. InInternational Conference on Learning Representations (ICLR), 2022. 8

    2022

  51. [59]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. InConference on Computer Vision and Pattern Recognition (CVPR), pages 11513–11522, 2022. 8 11

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.