Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes Adaptive Dynamic Sparse Attention (ADSA), a training-free method that halves the context and roughly the KV-cache memory used by autoregressive image generation models like LlamaGen, with CLIP and FID scores…

desk verdict ADSA is a plausible training-free KV-cache memory saver for LlamaGen-class autoregressive image generation, with honestly reported quality preservation, but the paper's computational-efficiency claim is unverified until it accounts for the O(L^2) per-step selection cost and reports wall-clock time. read the letter →

arxiv 2506.18226 v1 pith:4HUUYFRZ submitted 2025-06-23 cs.CV cs.AI

Xunzhi Xiang , Qi Fan This is my paper
classification cs.CVcs.AI
keywords adaptivedynamicsparseattentionautoregressiveimagegenerationKVcachecompressiontraining-freeinferenceLlamaGentext-to-imageclass-conditional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autoregressive image generation is expensive because every new image token attends to the entire past, and the key-value cache that stores that past consumes GPU memory. This paper argues that most of that past is redundant: early tokens fix the global style and color palette, nearby tokens carry local textures, and only a small set of middle tokens needs to be kept for semantic coherence. The proposed training-free method, Adaptive Dynamic Sparse Attention (ADSA), keeps a prefix, a local window, and a dynamically selected diverse subset of previous tokens, shrinking the context from 1024 to 512 tokens on MS-COCO and from 576 to 256 on ImageNet-256 with CLIP and FID scores essentially unchanged. A companion dynamic KV-cache update evicts the most redundant stored token when the cache is full, cutting GPU memory by about 50%. If the claim holds, existing autoregressive image models gain a substantial efficiency upgrade without retraining.

What carries the argument

The load-bearing object is the three-part context decomposition plus a diversity filter over value vectors. ADSA splits the KV-cache into the first $n$ prefix tokens, the most recent $m$ local tokens, and the remaining previous tokens. Before computing attention it scores each previous token by the average cosine similarity of its value vector $v_i$ to all other previous values, $S_i = \frac{1}{t-1}\sum_{j\neq i} \frac{v_i \cdot v_j}{\lVert v_i \rVert \lVert v_j \rVert}$, then keeps the $K$ tokens with the lowest $S_i$ as the selected set, so the retained previous tokens are the most semantically diverse ones. Because RoPE already encodes position in the query and key features, the value features are treated as carriers of semantic content, which is why diversity is measured in value space. The dynamic KV-cache update applies the same similarity score to evict the single most redundant token whenever the cache reaches capacity, and offloads completed tokens to CPU memory, returning them only at final decoding.

What would settle it

Measure wall-clock latency per image for LlamaGen-T2I-XL at context 512 with ADSA versus full attention at context 1024 on the same GPU, including the TopK-V scoring time; if per-token latency does not drop or total time rises, the computational-efficiency claim is falsified. Independently, craft a prompt where a small object appears only in a middle region whose value vectors resemble the background; if ADSA prunes those tokens and the object disappears, the diversity-preserves-semantics assumption is falsified.

Watch

Extended reading notes

Core claim

ADSA's central claim is that the effective context needed for high-quality autoregressive image generation can be halved by attending to three carefully chosen token groups instead of the full history. On LlamaGen-XL, reducing the context from 1024 to 512 on MS-COCO leaves the CLIP score at 0.286 versus 0.287 for full attention, and reducing the context from 576 to 256 on ImageNet-256 raises FID only from 2.62 to 2.64, while the intermediate ADSA-384 configuration actually improves FID to 2.58. The accompanying cache update halves the maximum KV-cache length, and the paper reports roughly 50% GPU memory savings across batch sizes. The ablation study isolates the three components: removing the selected middle tokens costs FID 2.70 versus 2.58, removing the prefix costs 7.41, and removing the local window collapses quality to 51.07.

Load-bearing premise

The central bet is that the $K$ most value-diverse older tokens preserve everything later tokens need from the past, and that identifying them costs less than the attention they replace; if either half fails, the paper's efficiency-quality tradeoff no longer holds.

Editorial extensions

If this is right

  • Half the context means roughly half the KV-cache memory and a smaller attention footprint in existing LlamaGen-class models, so larger batches or higher resolutions fit on the same GPU.
  • Because ADSA is training-free, it can be dropped into already-trained autoregressive image models without finetuning or architectural changes.
  • The ablation ordering, with the local window essential, the prefix important, and the selected middle tokens useful, gives a clear priority list for future efficient autoregressive image architectures.
  • If the cache is capped at half its original length and redundant tokens are evicted, GPU memory stops growing with sequence length once the cap is reached, supporting much longer generation runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports memory savings, not wall-clock speedups; the per-step $\mathcal{O}(T^2)$ pairwise similarity computation for TopK-V must be amortized or approximated for ADSA to beat dense attention on latency. This is an editorial caution, not a paper claim.
  • A natural test is to compare ADSA against random selection of $K$ middle tokens at the same context length; if random selection matches ADSA's FID and CLIP scores, then the diversity scoring adds little beyond the prefix-plus-window structure.
  • The prefix/local/selected decomposition assumes raster-order generation where early tokens set global style; random-order autoregressive models such as RandAR would need a different grouping, so the method's transfer to those models is not automatic.
  • The observed improvement in high-frequency detail at shorter contexts hints that restricting attention could be used as an adjustable inference-time sharpness knob, separate from any efficiency motivation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Adaptive Dynamic Sparse Attention (ADSA), a training-free inference-time method for autoregressive image generation that replaces dense self-attention with a combination of a fixed prefix, a local window, and a dynamically selected set of K previous tokens chosen for semantic diversity. It also proposes a fixed-capacity KV-cache (initialized to half the baseline length) that evicts the most redundant token before inserting a new one, with generated tokens offloaded to CPU memory. Experiments on LlamaGen for ImageNet class-conditional and MS-COCO text-to-image generation show that context lengths can be reduced by 33–55% with essentially unchanged FID/CLIP scores and roughly 50% reduction in GPU memory, as shown in Figure 9.

Significance. If the efficiency claim holds, ADSA is a practically useful, architecture-agnostic inference optimization: it preserves generation quality while reducing the KV-cache memory footprint, and it requires no retraining. The memory savings in Figure 9 are clearly demonstrated, and Table 1/2 show FID and CLIP scores close to the full-context baseline. The method is framed by interesting observations about how early tokens set global style and local tokens determine texture. However, the paper's central claim is computational efficiency, and that claim is not established: the per-step selection mechanism in Eqs. (2)–(4) adds an O(L^2) cost that is never measured or amortized, and the paper reports no wall-clock time, throughput, or FLOPs. The contribution is therefore currently a memory-reduction technique with an unverified speed benefit.

major comments (3)
  1. [§4.1, Eqs. (2)–(4); §5] The central efficiency claim is unsupported because the selection step adds a per-step O(L^2) cost that the paper never accounts for. Before every attention pass, ADSA computes pairwise cosine similarities among the previous-token V features (Eq. (2)) and averages them (Eq. (3)); with a cache length L this is O(L^2) vector operations per step, while the sparse attention it replaces is O(n+m+K) per step. Over T generation steps this selection overhead is O(T L^2), which for the paper's own settings (T=1024, L=512, or T=576, L=256) is orders of magnitude larger than the attention FLOPs saved unless the similarities are updated incrementally or computed only occasionally—neither of which is stated. The experiments report context-length reduction and GPU memory (Tables 1–2, Figure 9) but no wall-clock time, throughput, or FLOP measurements. I ask the authors to provide end-to-end generation time at matched quality, with and without ADSA, and either an incremental update scheme for S_ij or an explicit analysis showing when the selection overhead is amortized.
  2. [§4.1 and §5.1] The hyperparameters n (prefix length), m (local window size), and K (number of selected tokens) are never specified in the experiments. The paper defines the three cache regions in Eq. (1) but does not state what n, m, and K were used for ADSA-384, ADSA-256, ADSA-768, ADSA-640, and ADSA-512. This makes the method irreproducible and leaves open the question of how sensitive the reported FID/CLIP scores are to these choices. Please report the exact values for each configuration and add an ablation over K (and ideally n and m) to justify the selected settings.
  3. [§4.1, Eqs. (2)–(3)] The index sets in Eqs. (2) and (3) are inconsistent: Eq. (2) restricts v_i, v_j to V_previous, while Eq. (3) sums over j=1..t (i.e., all tokens in the cache). If the similarity is computed only within the previous region, the normalization and the cost differ from a full-cache computation. Please clarify the exact index sets and the resulting asymptotic cost, and state whether the similarity matrix is recomputed from scratch at every step or maintained incrementally. This matters because the selection cost is central to the paper's efficiency claim.
minor comments (5)
  1. [Abstract] There is a missing space in 'approximately50%' in the abstract, and similar typographical issues appear in Figure 1 ('Cachelength:384((-62%)') and Figure 9 captions.
  2. [Affiliation] The affiliation reads 'School of Intelliger Science and Technology'; 'Intelliger' appears to be a typo for 'Intelligence'.
  3. [Table 1] The FID improvement of ADSA-384 over the baseline (2.58 vs. 2.62) and the IS differences are reported without confidence intervals or multiple-seed variability, so it is unclear whether the 'even surpassing baseline' claim is significant.
  4. [§5.2 User Study] The user study reports only aggregate ratings from ten users on 48 prompts; no inter-rater agreement, statistical test, or error bars are provided, so the claim that ADSA variants 'performed well' is not quantitatively substantiated.
  5. [References] Reference [34] has a formatting error: the author list ends with 'Zhenhua Han and.' with a dangling 'and'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ADSA's reported gains are empirical measurements, and its design is motivated by analyses rather than derived from its own conclusions.

full rationale

The paper's central claims are empirical: ADSA compresses the context by keeping a prefix, a local window, and a set of tokens selected for low pairwise V-feature similarity, and the paper then measures FID, IS, CLIP, and GPU memory against LlamaGen. The selection rule in Eqs. (2)-(4) is a stated heuristic rather than a quantity fitted to the evaluation benchmarks, and the reported context lengths are configurations, not predictions derived from the selection rule. The Section 3 analyses (early tokens determine global style, local attention preserves texture, and full KV-cache is unnecessary) motivate the design but are independently probed by the ablation in Table 3. There are no load-bearing self-citations, no imported uniqueness theorem, and no equation-level reduction. The reader's concern about the unmeasured O(T^2) pairwise similarity overhead for selection is a correctness/evaluation gap (no wall-clock, throughput, or FLOP measurements), not a circularity, so it does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several unquantified design choices. The prefix size n, local window size m, and selected count K are never given, and the reported context lengths are selected configurations. The method also assumes that V-feature cosine similarity is a valid proxy for semantic importance, and that early tokens exclusively control global style. No new physical or conceptual entities are introduced.

free parameters (4)
  • prefix length n = not disclosed
    Number of initial tokens kept as prefix; never specified; determines global style retention.
  • local window size m = not disclosed
    Number of most recent tokens in local region; never specified.
  • selected count K = not disclosed
    Number of previous tokens retained after TopK-V selection; never specified.
  • compression/context length = 384, 256 for ImageNet; 768, 640, 512 for COCO
    The reported context lengths are selected configurations; no ablation over this choice is shown beyond the reported points, suggesting selection for favorable results.
assumptions (4)
  • domain assumption Image tokens are high entropy and exhibit spatial locality, with attention concentrated on local neighbors.
    Motivated by Section 3.2 and Figure 2 anecdotal example; the central design of windowed local attention depends on this.
  • domain assumption Early tokens define global style and color palette.
    Section 3.1 experiment fixing first 5 percent of tokens; used to justify fixing prefix.
  • ad hoc to paper V-feature cosine similarity measures semantic redundancy, so keeping least-similar V tokens preserves diversity.
    No theoretical or empirical justification that V-feature similarity corresponds to semantic information needed for generation; this is the core selection heuristic.
  • ad hoc to paper Pairwise similarity computation (Eq. 2) is affordable relative to attention.
    Never analyzed; naive implementation is O(T^2) per step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generation." pith.science (2026). https://pith.science/paper/4HUUYFRZ

@misc{pith2026250618226,
  author       = {Pith},
  title        = {Pith review of: Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4HUUYFRZ}},
  note         = {Machine review of arXiv:2506.18226}
}
abstract

Autoregressive conditional image generation models have emerged as a dominant paradigm in text-to-image synthesis. These methods typically convert images into one-dimensional token sequences and leverage the self-attention mechanism, which has achieved remarkable success in natural language processing, to capture long-range dependencies, model global context, and ensure semantic coherence. However, excessively long contexts during inference lead to significant memory overhead caused by KV-cache and computational delays. To alleviate these challenges, we systematically analyze how global semantics, spatial layouts, and fine-grained textures are formed during inference, and propose a novel training-free context optimization method called Adaptive Dynamic Sparse Attention (ADSA). Conceptually, ADSA dynamically identifies historical tokens crucial for maintaining local texture consistency and those essential for ensuring global semantic coherence, thereby efficiently streamlining attention computation. Additionally, we introduce a dynamic KV-cache update mechanism tailored for ADSA, reducing GPU memory consumption during inference by approximately $50\%$. Extensive qualitative and quantitative experiments demonstrate the effectiveness and superiority of our approach in terms of both generation quality and resource efficiency.

Figures

Figures reproduced from arXiv: 2506.18226 by the authors.

Figure 1
Figure 1. Achieving up to a 50% reduction in maximum context length during inference with our [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The attention scores of visual tokens in the LlamaGen-XL. Despite their inherently high entropy, image tokens exhibit strong spatial locality [22], with neighboring pixels frequently sharing similar visual characteristics. Empirical evidence, as il￾lustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Dense Attention exhibits a time complexity of O(T 2 ), with computational overhead increasing rapidly as the sequence length grows. Window Attention mitigates memory overhead by calculating key-value pairs for only the most recent L tokens, providing efficient inference. However, its performance sharply degrades once the earliest tokens’ keys and values are discarded. Window Attention with Prefix partially alleviate… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Early tokens define the global visual style and color palette. We conceptualize autoregressive continuous probabilistic modeling as a path exploration problem. We hypothesize that the tokens generated during the initial stages, despite their inherent high uncertainty, …
Figure 5
Figure 5. Figure 5: Comparison of Information Density Between Text Tokens and Image Tokens in Window [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Unlike conventional static sparse atten [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: The Overview of Our Proposed Dynamic Sparse KV-Cache Updating Strategy. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: Shorter KV-cache lengths consistently reduce GPU memory usage across various datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Samples generated by the LlamaGen-C2I-XL model using a next-token prediction [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 8
Figure 8. Figure 8: User study results. Class-conditional Image Generation. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 11
Figure 11. Figure 11: Samples generated by the LlamaGen-T2I-XL model using a next-token prediction [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Text-conditional 512×512 image generation on ChatGPT-prompt. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Text-conditional 512×512 image generation on ChatGPT-prompt. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages

  1. [1]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    2024

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    work page 2023

  3. [3]

    Llama: Open and efficient foundation language models.CoRR, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.CoRR, 2023

    work page 2023

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models.CoRR, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hart...

    work page 2023

  5. [5]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y . K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Li...

    work page 2024

  6. [6]

    GPT-4 technical report.CoRR, 2023

    OpenAI. GPT-4 technical report.CoRR, 2023

    work page 2023

  7. [7]

    Autore- gressive model beats diffusion: Llama for scalable image generation.CoRR, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation.CoRR, 2024

    work page 2024

  8. [8]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. InNeurIPS, 2024

    2024

Show all 39 references
  1. [9]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. InCVPR, 2022

    2022

  2. [10]

    Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan

    Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. InICML, 2023

    2023

  3. [11]

    Neural discrete representation learning

    Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017

    2017

  4. [12]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. InNeurIPS, 2024

    2024

  5. [13]

    Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation.CoRR, 2025

    Zisheng Chen, Chunwei Wang, Xiuwei Chen, Hang Xu, Jianhua Han, and Xiaodan Liang. Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation.CoRR, 2025. 10

    2025

  6. [14]

    Robust latent matters: Boosting image generation with sampling error synthesis

    Kai Qiu, Xiang Li, Jason Kuen, Hao Chen, Xiaohao Xu, Jiuxiang Gu, Yinyi Luo, Bhiksha Raj, Zhe Lin, and Marios Savvides. Robust latent matters: Boosting image generation with sampling error synthesis. CoRR, 2025

    2025

  7. [15]

    Subobject-level image tokenization.CoRR, 2024

    Delong Chen, Samuel Cahyawijaya, Jianfeng Liu, Baoyuan Wang, and Pascale Fung. Subobject-level image tokenization.CoRR, 2024

    2024

  8. [16]

    Unitok: A unified tokenizer for visual generation and understanding.CoRR, 2025

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding.CoRR, 2025

    2025

  9. [17]

    Imagefolder: Autoregressive image generation with folded tokens.CoRR, 2024

    Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens.CoRR, 2024

    2024

  10. [18]

    Infllm: Training-free long-context extrapolation for llms with an efficient context memory

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. InNeurIPS, 2024

    2024

  11. [19]

    Reattention: Training-free infinite context with finite attention scope

    Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Qipeng Guo, Yuerong Song, Kai Lv, Hang Yan, Linlin Li, Qun Liu, and Xipeng Qiu. Reattention: Training-free infinite context with finite attention scope. InICLR, 2025

    2025

  12. [20]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InICLR, 2024

    2024

  13. [21]

    Generating long sequences with sparse transformers.CoRR, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.CoRR, 2019

    2019

  14. [22]

    Zipar: Accelerating auto-regressive image generation through spatial locality.CoRR, 2024

    Yefei He, Feng Chen, Yuanyu He, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Accelerating auto-regressive image generation through spatial locality.CoRR, 2024

    2024

  15. [23]

    Freeman, and Yu-Xiong Wang

    Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T. Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders.CoRR, 2024

    2024

  16. [24]

    Neighboring autoregressive modeling for efficient visual generation.CoRR, 2025

    Yefei He, Yuanyu He, Shaoxuan He, Feng Chen, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Neighboring autoregressive modeling for efficient visual generation.CoRR, 2025

    2025

  17. [25]

    Frequency autoregressive image generation with continuous tokens.CoRR, 2025

    Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, and Feng Zhao. Frequency autoregressive image generation with continuous tokens.CoRR, 2025

    2025

  18. [26]

    Fluid: Scaling autoregressive text-to-image generative models with continuous tokens.CoRR, 2024

    Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens.CoRR, 2024

    2024

  19. [27]

    Vector-quantized image modeling with improved VQGAN

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. InICLR, 2022

    2022

  20. [28]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InICML, 2021

    2021

  21. [29]

    Autoregressive image generation with randomized parallel decoding.CoRR, 2025

    Haopeng Li, Jinyue Yang, Guoqi Li, and Huan Wang. Autoregressive image generation with randomized parallel decoding.CoRR, 2025

    2025

  22. [30]

    Focus directions make your language models pay more attention to relevant contexts.CoRR, 2025

    Youxiang Zhu, Ruochen Li, Danqing Wang, Daniel Haehn, and Xiaohui Liang. Focus directions make your language models pay more attention to relevant contexts.CoRR, 2025

    2025

  23. [31]

    When attention sink emerges in language models: An empirical view.ICLR, 2025

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view.ICLR, 2025

    2025

  24. [32]

    Longheads: Multi-head attention is secretly a long context processor.CoRR, 2024

    Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. Longheads: Multi-head attention is secretly a long context processor.CoRR, 2024

    2024

  25. [33]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. InNeurIPS, 2024

    2024

  26. [34]

    Retrievalattention: Accelerating long-context LLM inference via vector retrieval.CoRR, 2024

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, and Zhenhua Han and. Retrievalattention: Accelerating long-context LLM inference via vector retrieval.CoRR, 2024. 11

    2024

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InICML, 2021

    2021

  28. [36]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InNeurIPS, 2017

    2017

  29. [37]

    Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen

    Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. InNeurIPS, 2016

    2016

  30. [38]

    Scaling up gans for text-to-image synthesis

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. InCVPR, 2023

    2023

  31. [39]

    High-resolution image synthesis with latent diffusion models.CVPR, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models.CVPR, 2022. 12 A Technical Appendices and Supplementary Material Figure 12: Text-conditional 512×512 image generation on ChatGPT-promp...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.