Pith. sign in

REVIEW 5 major objections 5 minor 54 references

MARch\'e: Fast Masked Autoregressive Image Generation with Cache-Aware Attention

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MARché caches stable key/value projections and refreshes only attention-relevant tokens, claiming up to 1.72x faster masked autoregressive image generation with small FID cost.

desk verdict A solid, incremental training-free speedup for MAR image generation, but the quality-preservation claim is overstated and the refresh-selection premise needs direct validation before this is publishable. read the letter →

arxiv 2506.12035 v1 pith:5OYAESIR submitted 2025-05-22 cs.LG cs.AI

Chaoyi Jiang , Sungwoo Kim , Lei Gao , Hossein Entezari Zarch , Won Woo Ro , Murali Annavaram This is my paper
classification cs.LGcs.AI
keywords maskedautoregressiveimagegenerationKVcachereusecache-awareattentionselectiverefreshinferenceaccelerationtransformerdecodingNet256x256
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked autoregressive (MAR) image transformers recompute attention and feed-forward outputs for every token at every decoding step, even though only a few tokens are newly generated. This paper claims that most key/value projections change very little across steps, so most of that recomputation is redundant. MARché is a training-free inference strategy that caches stable KV projections, recomputes only active tokens, and refreshes the top-K contextually relevant tokens chosen by attention scores from newly generated tokens. On ImageNet 256x256 it reports 1.57x, 1.68x, and 1.72x speedups over MAR-B/L/H while keeping FID increases to roughly 0.2-0.4. If right, MAR models can be served markedly faster without retraining or architecture changes.

What carries the argument

The key mechanism is cache-aware attention, which partitions tokens into active and cached sets at each decoding step and runs two separate computation paths. Active tokens compute fresh queries, keys, and values and attend over both sets, while cached tokens contribute only their stored key/value projections; the two attention sums are merged with the safe online softmax formulation, making the split exactly equivalent to standard full attention whenever cache values are current. Selective KV refresh identifies the active set's refreshing tokens by ranking attention scores from newly generated tokens at decoder layer 2, and a periodic full refresh every three steps prevents stale cached values from accumulating error.

What would settle it

A direct test is to compare MARché against full MAR across many images and several random generation-order permutations: if the FID gap varies strongly with the permutation, or if some image shows large FID degradation while its top-K attention-selected tokens all have near-0.95 cosine KV stability, then the temporal-locality and attention-selection assumptions fail.

Watch

Extended reading notes

Core claim

The central claim is that MAR models exhibit strong temporal locality: between adjacent decoding steps, most token key projections stay above roughly 0.95 cosine similarity, and value projections are only slightly more dynamic. Exploiting this, cache-aware attention divides tokens into an active set and a cached set, computing fresh key/value projections and feed-forward outputs only for active tokens while reusing stored KV projections for the rest. Selective KV refresh then picks a fixed budget of refreshing tokens by aggregating attention scores from generating tokens across heads in an early decoder layer, plus periodic full refresh every three steps to control drift. The paper reports up to 1.72x lower latency with FID penalties around 0.2-0.4 across model scales, with no change to the underlying architecture or training.

Load-bearing premise

The load-bearing premise is empirical: most key/value projections in a MAR decoder stay nearly unchanged across decoding steps, so stale cached values can be reused without error accumulating, and the top-K tokens chosen by attention scores from newly generated tokens are the ones that must be recomputed; the paper provides no bound or failure-case analysis for this premise.

Editorial extensions

If this is right

  • Masked autoregressive models can be served 1.57-1.72x faster on existing hardware without retraining or changing the transformer architecture.
  • The reported speedup grows with model size, so larger MAR models stand to gain more from cache-aware decoding.
  • Because the active/cached split is exact when cache values are fresh, the only quality loss comes from stale KV entries, not from the attention formulation.
  • Attention scores from an early decoder layer suffice to decide which tokens to refresh, avoiding the cost of running all layers to make that decision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An adaptive refresh budget that grows when attention scores are diffuse could improve the quality-latency trade-off beyond the fixed 64-token budget tested here.
  • The same KV-stability argument could transfer to other bidirectional iterative generators, such as multi-modal masked models, but the paper's evidence is limited to class-conditional ImageNet at 256 resolution.
  • A drift-triggered full refresh, recomputing all KV values when measured similarity drops below a threshold, might beat the fixed every-three-steps schedule at the cost of an extra similarity computation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MARché, a training-free inference-time framework to accelerate masked autoregressive (MAR) image generation by caching key/value projections and recomputing them only for a small active set of tokens. The active set is formed from newly generated tokens and a fixed budget of tokens selected by attention scores in an early decoder layer, with a periodic full refresh every three steps. The authors report speedups up to 1.72x on ImageNet 256x256 with FID and IS close to the original MAR models, and they provide a mathematical equivalence argument for a cache-aware attention kernel based on online softmax. The central claim is that most token representations are stable across decoding steps and that attention-based selection identifies exactly the tokens whose recomputation matters for quality.

Significance. If the speedup and quality-preservation claims are established, MARché would be a practically useful, architecture-agnostic inference optimization for a prominent class of generative transformers, since it requires no retraining or architectural changes. The motivation is well grounded in prior KV-cache work for LLMs, and the kernel-level latency measurements in Appendix B provide concrete evidence that the partitioned attention formulation can be implemented efficiently. The online-softmax equivalence is correct under the assumption of fresh cache values. However, the paper's quality claim rests on an empirical link between attention scores and token drift that is motivated only indirectly, and the reported FID/IS degradations are larger than the phrase 'negligible impact' suggests. The lack of error bars further weakens the quantitative claims.

major comments (5)
  1. [§5.2, Table 1] The abstract and Section 5.2 characterize the image-quality impact as 'negligible,' but Table 1 shows FID increases of 0.21 (MAR-B), 0.32 (MAR-L), and 0.40 (MAR-H), while IS decreases by 10.8, 17.7, and 17.2 points respectively. These are not obviously negligible, and no error bars or multiple-seed results are reported for FID, IS, or latency. The authors should either temper the claim or provide statistical evidence (e.g., confidence intervals over seeds) that the degradation is within noise.
  2. [§4.2, Algorithm 1, Appendix F] The central premise that attention-score-selected tokens are the ones whose K/V representations actually drift is never directly tested. Figure 1 and Appendix F show that most tokens are stable, but they do not identify which tokens drift or whether the drift set overlaps with the attention-selected refresh set. Tables 2 and 3 compare MARché against random and low-attention baselines, which is indirect evidence; a heuristic can beat those baselines while still failing to refresh the truly drifting tokens. The authors should add a direct measurement, e.g., the overlap between the top-K attention tokens and the top-K tokens by K/V change, along with a failure-case analysis for images where FID degrades most.
  3. [§4.3, Algorithm 1, Appendix A] The equivalence proof in Appendix A assumes that cached K/V values equal the values that would be produced by a fresh forward pass. In the actual algorithm, non-active tokens' K/V values are stale for up to three steps, and active tokens' outputs are computed using these stale contexts. Therefore the proof does not cover the approximation error of the deployed method. The paper provides no bound or empirical characterization of the cumulative drift introduced by stale caches, which is load-bearing for the quality-preservation claim.
  4. [Table 1 vs. §5.3/Figures 5–6] The latency numbers are inconsistent across the paper. Table 1 reports MARché-B at 0.064 s/image, but Figure 5 reports the Layer-2 default configuration at about 0.159 s/image and Figure 6 reports around 0.12–0.18 s/image for similar settings. The figure captions do not specify which model scale is used, and the same issue affects Table 4, where the text quotes 0.158 s/im while the table lists 0.155 s/im. The authors should unify the latency reporting and clearly state the model scale and hardware configuration for every ablation.
  5. [§5.3, Tables 2 and 3] The two random-selection baselines differ by a factor of nearly 200 in FID (564.61 in Table 2 vs. 3.01 in Table 3), which is confusing because the captions are similar. If these are different experiments (random construction of the entire active set vs. random selection of refreshing tokens only), the distinction should be explicit in the text and captions; otherwise the results appear mutually inconsistent and undermine confidence in the ablations.
minor comments (5)
  1. [§4.3] The sentence 'We then use the attention scores of the first decode layer to select the top-K tokens' appears to contradict Section 4.2, which says Layer 2 is used for selection. Please clarify which layer is used.
  2. [Appendix D, Table 4] The text says 'Full at layer 1 only yields the fastest inference (0.158 s/im)', but Table 4 lists 0.155 s/im for that entry. Please align the numbers.
  3. [Section 5.3, Figure 5 caption] The caption says 'Deeper layers improves FID' (subject-verb agreement) and the text contains 'but but' in the same paragraph. Please proofread.
  4. [Abstract and Section 1] The name is typeset as 'MARch\'e' in the abstract and introduction, but the rest of the paper uses 'MARché' or 'MARche'. Please use a consistent notation throughout, including in the title.
  5. [Appendix E] The claim that outputs are 'virtually indistinguishable' is based on visual inspection of a single figure; please either show quantitative perceptual metrics or soften the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central speedup and quality results are externally benchmarked.

full rationale

MARché's central claims are evaluated against external measures: wall-clock latency on an H100 and FID/IS computed from generated ImageNet images. The approximate cache reuse is justified by empirical stability measurements (Figures 1, 10, 11) and by ablations comparing selection heuristics (Tables 2, 3); these are data-driven validations, not definitional equivalences. Appendix A proves that the online-softmax split is exact under the stated assumption of fresh K/V for all tokens; it explicitly does not assert that the full stale-cache algorithm is exact, so there is no circular reduction of the main claim to the proof. The refresh-policy hyperparameters (top-K budget, 3-step full refresh, Layer 2 selection) are tuned on the same benchmark, which raises a potential overfitting/evaluation concern but is not circularity: the reported FID and latency are measured outcomes, not consequences implied by the choice of hyperparameters. No load-bearing self-citation appears: the MAR baseline and KV-cache prior works are external to this paper, and no known result is repackaged under new names. The skeptical gap, namely that attention-selected tokens may not coincide with the tokens whose K/V drift most, is an unvalidated empirical premise rather than a derivation that reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two empirical regularities, KV stability and attention-score relevance, plus three tuned scheduling choices: active budget 64, refresh period 3, and refresh layer 2. None of these are derived from first principles; they are validated by ablations on the same ImageNet task used for the headline results. No new physical entities are postulated.

free parameters (4)
  • Active token budget = 64
    Fixed per-step batch of active tokens; ablations in Figure 6 show 64 refreshing tokens balances FID and latency. It is a tuned hyperparameter, not derived.
  • Periodic full refresh period = every 3 decoding steps
    Chosen from Figure 8 ablation; shorter periods improve FID but cost latency, longer periods degrade quality.
  • Refresh selection layer = Layer 2
    Selected from Figures 4 and 5 as a compromise between semantic overlap and latency; the paper notes it is configurable.
  • Full-attention layers per step = Layers 1 and 2
    These layers always run standard full attention to provide relevance scores; Table 4 shows refresh placement materially changes FID, with layers 7-12 producing FID 25.35.
assumptions (4)
  • domain assumption Most token key/value projections remain stable across decoding steps, so stale cached K/V values can be reused without error accumulation.
    Motivates the whole method in Section 1 and Figure 1; measured only as cosine similarity on a single image, with no theoretical error bound or failure-case analysis.
  • domain assumption Tokens receiving high attention from newly generated tokens are the tokens that must be recomputed to preserve quality.
    Core selection heuristic in Section 4.2; supported only by ablations in Table 3, not derived from the model or task.
  • ad hoc to paper A periodic full refresh every 3 steps plus full attention in layers 1-2 is sufficient to prevent value drift and maintain generation quality.
    Introduced in Section 4.3; frequency and placement are tuned by ablation in Figure 8 and Table 4, with no convergence or stability argument.
  • standard math The safe online softmax decomposition over two disjoint KV subsets equals full softmax attention.
    Used to justify the separate active/cached computation paths; this is a known FlashAttention result and is proven in Appendix A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MARch\'e: Fast Masked Autoregressive Image Generation with Cache-Aware Attention." pith.science (2026). https://pith.science/paper/5OYAESIR

@misc{pith2026250612035,
  author       = {Pith},
  title        = {Pith review of: MARch\'e: Fast Masked Autoregressive Image Generation with Cache-Aware Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OYAESIR}},
  note         = {Machine review of arXiv:2506.12035}
}
read the original abstract

Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation. While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step, despite most tokens remaining semantically stable across steps. We propose a training-free generation framework MARch\'e to address this inefficiency through two key components: cache-aware attention and selective KV refresh. Cache-aware attention partitions tokens into active and cached sets, enabling separate computation paths that allow efficient reuse of previously computed key/value projections without compromising full-context modeling. But a cached token cannot be used indefinitely without recomputation due to the changing contextual information over multiple steps. MARch\'e recognizes this challenge and applies a technique called selective KV refresh. Selective KV refresh identifies contextually relevant tokens based on attention scores from newly generated tokens and updates only those tokens that require recomputation, while preserving image generation quality. MARch\'e significantly reduces redundant computation in MAR without modifying the underlying architecture. Empirically, MARch\'e achieves up to 1.7x speedup with negligible impact on image quality, offering a scalable and broadly applicable solution for efficient masked transformer generation.

Figures

Figures reproduced from arXiv: 2506.12035 by the authors.

Figure 2
Figure 2. Comparison of standard attention and cache-aware attention in MAR generation. (a) In standard attention, all tokens (masked or generated) are processed at every step, leading to redundant computation. (b) In cache-aware attention, only active tokens (generating, refreshing, or caching) are recomputed, while cached tokens reuse previously computed key/value projections. 0.8 0.9 Similarity [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figure 1
Figure 1. Cosine similarity of key projections between step 4 and 5 (in layer 2) during a single image generation. Motivation. Although MAR models follow an autoregressive de￾coding paradigm, they use bidirectional attention at each decoding step, which allows all tokens, including both generated and yet to be generated ones, to update their contextual representations. This design enables global context exchange, but also imp… view at source ↗
Figure 3
Figure 3. KV cache refresh via attention score. Layers 1 and 2 perform standard full attention. In Layer 2, attention scores are computed from generating tokens to all others, and aggregated across attention heads to produce global relevance scores. The top-K most contextually relevant tokens are selected based on these scores and marked as refreshing tokens. From Layer 3 onward, cache-aware attention is applied. 4.2 Selectiv… view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Latency–FID trade-off for differ￾ent refreshing token selection layers. Deeper layers improves FID with higher latency, while earlier layers decode faster but degrade quality. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Trade-off between latency and FID as a function of the number of re￾freshing tokens. Using more refreshing tokens improves FID but increases latency. Effectiveness of the number of refreshing tokens. We analyze how the number of refreshing tokens affects the trade-off …
Figure 7
Figure 7. Figure 7: Runtime comparison of standard attention vs. cache-aware attention implementation. Both methods process the same total number of attention operations over two input segments of different lengths. Standard attention concatenates the inputs and processes them jointly, wh…
Figure 8
Figure 8. Figure 8: Trade-off between refresh frequency and generation performance. Shorter refresh cycles (e.g., every 2–3 steps) improve FID at the cost of increased latency, while longer cycles reduce latency but degrade image quality. A 3- step refresh strikes a good balance, achievin…
Figure 9
Figure 9. Figure 9: Qualitative comparison of generated images from MAR-H (top) and MARché-H (bottom). MARché-H achieves similar visual quality while running 1.7× faster. We conduct a qualitative comparison between standard MAR and our proposed MARché, using the MAR-H and MARché-H, respec…
Figure 10
Figure 10. Figure 10: Cosine similarity of key projections across decoding steps and layers. Most tokens maintain high similarity across steps, especially in lower layers, indicating that their key projections change very little during decoding. Step 1 vs 2 Layer 1 Layer 4 Layer 7 Layer 10…
Figure 11
Figure 11. Figure 11: Cosine similarity of value projections across decoding steps and layers. Value projections are slightly more dynamic than key projections, particularly in deeper layers, but remain stable for the majority of tokens. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Layer-wise top-K attention overlap across decoding steps. Deeper layers show higher and more consistent alignment. To further analyze the stability of refreshing token selections across layers, we visualize the overlap of top-K attended tokens across decoder layers at…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 13 canonical work pages

  1. [1]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  3. [3]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  4. [4]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  5. [5]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  6. [6]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  7. [7]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744, 2022

    2022

  8. [8]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020

    2020

Show all 54 references
  1. [9]

    Con- ditional image generation with pixelcnn decoders

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelcnn decoders. Advances in neural information processing systems, 29, 2016

    2016

  2. [10]

    Parallel multiscale autoregressive density estimation

    Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In International conference on machine learning , pages 2912–2921. PMLR, 2017

    2017

  3. [11]

    Image transformer

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning , pages 4055–4064. PMLR, 2018

    2018

  4. [12]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839–84865, 2024

    2024

  5. [13]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  6. [14]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821–8831. Pmlr, 2021

    2021

  7. [15]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2(3):5, 2022

    2022 arXiv

  8. [16]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022

    2022

  9. [17]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  10. [18]

    Cogview: Mastering text-to-image generation via transformers

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems , 34:19822–19835, 2021

    2021

  11. [19]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315–11325, 2022

    2022

  12. [20]

    Muse: Text-to-image generation via masked generative transformers

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023

    2023 arXiv

  13. [21]

    Mage: Masked generative encoder to unify representation learning and image synthesis

    Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2142–2152, 2023

    2023

  14. [22]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems , 37:56424–56445, 2024

    2024

  15. [23]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  16. [24]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019

    2019

  17. [25]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  18. [26]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  19. [27]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Syste...

    2023

  20. [28]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  21. [29]

    Keyformer: Kv cache reduction through key tokens selection for efficient generative inference

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems , 6:114–127, 2024

    2024

  22. [30]

    Arkvale: Efficient generative llm inference with recallable key-value eviction

    Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative llm inference with recallable key-value eviction. Advances in Neural Information Processing Systems , 37:113134– 113155, 2024

    2024

  23. [31]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024

    2024 arXiv

  24. [32]

    Maskbit: Embedding-free image generation via bit tokens

    Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chieh Chen. Maskbit: Embedding-free image generation via bit tokens. arXiv preprint arXiv:2409.16211, 2024

    2024 arXiv

  25. [33]

    Improved masked image generation with token-critic

    José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In European Conference on Computer Vision, pages 70–86. Springer, 2022

    2022

  26. [34]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022

    2022

  27. [35]

    Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022. 11

    2022 arXiv

  28. [36]

    Oms-dpm: Optimizing the model schedule for diffusion probabilistic models

    Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang, and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. In International Conference on Machine Learning, pages 21915–21936. PMLR, 2023

    2023

  29. [37]

    Mixdq: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization

    Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. In European Conference on Computer Vision, p...

    2024

  30. [38]

    Vidit-q: Efficient and accu- rate quantization of diffusion transformers for image and video generation

    Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Enshu Liu, Rui Wan, Widyadewi Soedar- madji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, et al. Vidit-q: Efficient and accu- rate quantization of diffusion transformers for image and video generation. arXiv preprint arXiv:2406.0...

    2024 arXiv

  31. [39]

    Ditfastattn: Attention compression for diffusion transformer models

    Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems , 37:1196–1219, 2024

    2024

  32. [40]

    Deepcache: Accelerating diffusion models for free

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15762–15772, 2024

    2024

  33. [41]

    Cache me if you can: Accelerating diffusion models through block caching

    Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Visio...

    2024

  34. [42]

    Approximate caching for efficiently serving {Text-to-Image} diffusion models

    Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. Approximate caching for efficiently serving {Text-to-Image} diffusion models. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), page...

    2024

  35. [43]

    Lantern: Accelerating visual autoregressive models with relaxed speculative decoding

    Doohyuk Jang, Sihwan Park, June Yong Yang, Yeonsung Jung, Jihun Yun, Souvik Kundu, Sung-Yub Kim, and Eunho Yang. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding. arXiv preprint arXiv:2410.03355, 2024

    arXiv

  36. [44]

    Accelerating auto-regressive text-to-image generation with training-free speculative jacobi decoding

    Yao Teng, Han Shi, Xian Liu, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Accelerating auto-regressive text-to-image generation with training-free speculative jacobi decoding. arXiv preprint arXiv:2410.01699, 2024

    arXiv

  37. [45]

    Enat: Rethinking spatial-temporal interactions in token-based image synthesis

    Zanlin Ni, Yulin Wang, Renping Zhou, Yizeng Han, Jiayi Guo, Zhiyuan Liu, Yuan Yao, and Gao Huang. Enat: Rethinking spatial-temporal interactions in token-based image synthesis. Advances in Neural Information Processing Systems , 37:90431–90455, 2024

    2024

  38. [46]

    Lazymar: Accelerating masked autoregressive models via feature caching

    Feihong Yan, Qingyan Wei, Jiayi Tang, Jiajun Li, Yulin Wang, Xuming Hu, Huiqi Li, and Linfeng Zhang. Lazymar: Accelerating masked autoregressive models via feature caching. arXiv preprint arXiv:2503.12450, 2025

    2025 arXiv

  39. [47]

    Masked generative nested transformers with decode time scaling

    Sahil Goyal, Debapriya Tula, Gagan Jain, Pradeep Shenoy, Prateek Jain, and Sujoy Paul. Masked generative nested transformers with decode time scaling. arXiv preprint arXiv:2502.00382 , 2025

    2025 arXiv

  40. [48]

    Efficient attention: Attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 3531–3539, 2021

    2021

  41. [49]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  42. [50]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  43. [51]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658–68685, 2024

    2024

  44. [52]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 12

    2017

  45. [53]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016

    2016

  46. [54]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4195–4205, 2023. 13 A Equivalence of Cache-Aware Attention to Standard Attention We show that our cache-aware attent...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.