Pith. sign in

REVIEW 3 major objections 7 minor 4 cited by

Masked Autoencoders Are Effective Tokenizers for Diffusion Models

T0 review · 3 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that what makes a latent space good for diffusion models is its structure, not the variational regularization of the tokenizer, and that a masked-trained plain autoencoder can deliver state-of-the-art image generation.

desk verdict Strong empirical tokenizer paper that deserves a referee; the theory is the weak link and should not gate acceptance. read the letter →

arxiv 2502.03444 v2 pith:7RZYR5LD submitted 2025-02-05 cs.CV cs.AIcs.LG

Hao Chen , Yujin Han , Fangyi Chen , Xiang Li , Yidong Wang , Jindong Wang , Ze Wang , Zicheng Liu
show 2 more authors
Difan Zou Bhiksha Raj
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords latentdiffusionmaskedautoencoderstokenizerGaussianmixturemodelimagegenerationvariationalautoencoderspacestructureNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that what makes a latent space good for diffusion models is its structure, not the variational regularization of the tokenizer. The authors argue, with theory and experiments, that latent spaces with fewer Gaussian-mixture modes and more class-discriminative clusters are easier for a diffusion denoiser to learn, and that a plain autoencoder can produce such a space if it is trained with masked modeling and auxiliary semantic targets. They build MAETok, an autoencoder that reconstructs masked image tokens and predicts HOG, DINOv2, and CLIP features, then fine-tunes the decoder with the encoder frozen. With only 128 tokens, MAETok plus a 675M-parameter diffusion transformer reaches 1.69 gFID and 304.2 IS on 512x512 ImageNet with classifier-free guidance, surpassing a 2B-parameter baseline while training about 76 times faster and inferring about 31 times faster. The paper's takeaway is that variational constraints are not necessary; a discriminative latent space is the crucial ingredient.

What carries the argument

MAETok is a plain 1D autoencoder whose ViT encoder takes image patch tokens plus $L=128$ learnable latent tokens, randomly masks 40-60% of patch tokens, and reconstructs them through a ViT decoder and auxiliary shallow decoders that predict HOG, DINOv2, SigCLIP, and (on the LAION model) BPE text-caption targets. The theoretical engine is the GMM analysis: Theorem 2.1 (Appendix Theorem A.7), built on prior score-learning bounds, shows DDPM needs $n \geq \Theta(K^4 d^5 B^6 / \varepsilon^2)$ samples to reach $O(T \varepsilon^2)$ KL error when the latent distribution is a $K$-mode Gaussian mixture. A final decoder fine-tuning stage freezes the encoder to keep the organized latent space while recovering pixel-level fidelity.

What would settle it

Train two tokenizers that match in measured GMM mode count and rFID but differ by whether the encoder is trained with or without the variational KL objective, then train identical SiT-XL diffusion models on both; the claim that latent structure alone matters fails if the variational tokenizer still yields a gFID gap larger than evaluation noise.

Watch

Extended reading notes

Core claim

The central claim is that the number of modes in the latent distribution controls how well diffusion models learn and generate. The paper proves a sample-complexity bound for DDPM on a K-Gaussian mixture, $n \geq \Theta(K^4 d^5 B^6 / \varepsilon^2)$, and pairs it with an empirical correlation between GMM negative log-likelihood, linear-probing accuracy, and final gFID. On that basis it claims that the variational form of autoencoders, the KL constraint of VAEs, is unnecessary; what matters is an organized, discriminative latent space. MAETok delivers that space by training a plain autoencoder with masked modeling and auxiliary target predictors, and the paper reports state-of-the-art ImageNet generation at 256 and 512 resolution with 128 latent tokens, including gFID 1.69 at 512x512 with a 675M SiT-XL.

Load-bearing premise

The load-bearing premise is that a tokenizer's latent distribution can be faithfully summarized as a mixture of a few well-separated Gaussians whose mode count K controls the diffusion model's true sample complexity; if the real, entangled latent codes do not fit that summary, the theoretical explanation for why MAETok works loses its foundation.

Editorial extensions

If this is right

  • The KL loss in latent diffusion tokenizers can be dropped; a masked-trained plain AE is sufficient for state-of-the-art generation.
  • Fewer, better-separated latent modes yield lower denoising loss, faster convergence, and better gFID at fixed compute.
  • 128 tokens suffice for 256x256 and 512x512 ImageNet generation, converting directly into 31 times higher inference throughput and 76 times faster training relative to the compared pipeline.
  • Reconstruction fidelity and latent discriminativeness decouple: after masked training, fine-tuning only the decoder recovers rFID from 0.85 to 0.48 without hurting gFID.
  • On MAETok latents, unconditional and CFG-based generation are close in quality, and the paper shows that replacing CFG with autoguidance or guidance-free training pushes SiT-XL gFID from 1.67 to 1.54 and 1.51.
  • vThe theory predicts a scaling law: under a fixed dataset, tokenizers with fewer latent modes should make larger diffusion models pay off faster; this can be checked by comparing SiT-XL and SiT-L on the same MAETok latent space.
  • The paper's finding also implies that strong unconditional generation is a cheap diagnostic for latent structure: models on MAETok already reach IS 47.33 unconditionally, so a small CFG gap signals a well-clustered latent space rather than a weak generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: deliberately manipulating mode count without changing reconstruction fidelity, for example by adding a clustering regularizer to a standard AE, should reproduce the gFID ordering predicted by the paper's bound.
  • The theory predicts a scaling law: under a fixed dataset, tokenizers with fewer latent modes should make larger diffusion models pay off faster; this can be checked by comparing SiT-XL and SiT-L on the same MAETok latent space.
  • The paper's finding also implies that strong unconditional generation is a cheap diagnostic for latent structure: models on MAETok already reach IS 47.33 unconditionally, so a small CFG gap signals a well-clustered latent space rather than a weak generator.
  • The GMM-mode lens suggests a broader design rule: any training objective that clusters semantically related latents, such as contrastive alignment or clustering, should raise downstream diffusion quality even without masking; MAE is one instance, not the only one.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies how the latent space of image tokenizers affects diffusion model training and generation. It argues, both empirically and theoretically, that latent distributions with fewer Gaussian mixture modes and more discriminative features lead to better diffusion learning and sampling. To instantiate this idea, it proposes MAETok, a plain autoencoder trained with masked modeling and auxiliary shallow decoders that predict HOG, DINOv2, CLIP, and (for the LAION variant) BPE text targets, followed by a decoder-only fine-tuning stage. Using only 128 latent tokens, diffusion models such as SiT-XL and LightningDiT trained on MAETok achieve state-of-the-art ImageNet generation (e.g., gFID 1.69 and IS 304.2 on 512x512 with CFG, outperforming a 2B-parameter USiT). The paper concludes that variational regularization is not necessary for effective latent diffusion, provided the latent space is sufficiently discriminative.

Significance. If the empirical results hold, MAETok is a practically important tokenizer: it decouples semantic richness from reconstruction fidelity, uses only 128 tokens, and greatly improves training and inference throughput. The ablations are extensive (mask ratio, reconstruction target, auxiliary decoder depth, component ablation, token count), and the latent-space analyses (linear probing accuracy, UMAP separation, GMM NLL, and convergence speed) consistently correlate with generation quality. The paper also releases code and trained models, which is a strength. The theoretical argument, however, is currently not rigorous enough to establish the causal claim that 'fewer modes cause better diffusion' for the actual system; it functions as a motivating analogy rather than a proof for the transformer denoisers used in the experiments. The empirical contribution is significant and publishable once the theory section is either repaired or explicitly reframed as idealized motivation.

major comments (3)
  1. [Appendix A, Eq. (18)] Equation (18) does not follow from Theorem A.5 as stated. Theorem A.5 guarantees only min_i ||mu~_i - mu*_i|| <= epsilon, i.e., that at least one estimated center is close, whereas Eq. (18) bounds a sum over all K centers and also involves the weight mismatch (w_i,t - w*_i,t), which requires simultaneous closeness of all centers. The proof sketch hides constants and an implicit K dependence; without an all-centers guarantee, the displayed O(K^4 d^5 B^6 / eps^2) sample complexity in Theorem A.7 is not established. This is load-bearing for the Section 2 claim that more modes require more training samples.
  2. [Section 2 / Appendix B.3] The GMM analysis does not actually measure the K appearing in Theorem 2.1. The latent codes are first flattened, PCA-reduced to 90% explained variance, and then standardized before fitting (Appendix B.3). These preprocessing steps alter cluster geometry, and the NLL-versus-component-count curves in Fig. 2a are not estimators of the number of modes of the original latent distribution. Consequently, the connection between the empirically observed 'fewer modes' for MAETok and the theoretical K in Eq. (1) is not established, and the paper's explanation of why MAETok works is supported only by correlation and analogy.
  3. [Appendix A, Eq. (13)] The theoretical result is proved only for a softmax-linear score network of the form s_theta_t(x) = sum_i w_i,t(x) mu_i,t - x, whose parametric form exactly matches the oracle score of the Gaussian mixture. The denoisers actually used in all experiments (SiT and LightningDiT transformers) are not of this form, and the paper gives no argument that they can approximate this score with the same sample complexity. Thus even a corrected GMM bound would not directly transfer to the systems whose performance is reported.
minor comments (7)
  1. [Theorem 2.1] There is a typo: 'distirbution' should be 'distribution'.
  2. [Appendix A, Remark A.4] 'second movement bound' should be 'second moment bound'.
  3. [Related Work] 'Imgae tokenization' should be 'Image tokenization'.
  4. [Fig. 2] The caption uses 'GMM Loss' while the text and axis label use 'GMM NLL'; please make the terminology consistent.
  5. [Table 12] Table 12 reports gFID and IS computed on 10K generated samples while the main tables use 50K samples; please state this clearly in the caption or use a consistent evaluation protocol to avoid the appearance of cherry-picking.
  6. [Section 4.5] The claim of '76x faster training to perform similarly to REPA' is imprecise; please define the baseline, compute budget, and performance threshold used for this comparison.
  7. [Appendix C.3] 'differente reconstruction targets' should be 'different reconstruction targets'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central method is benchmarked externally, the latent-space analyses are post-hoc measurements, and the cited theorems are prior independent results.

full rationale

The paper's derivation chain has two parts: an empirical/theoretical link between latent-space structure and diffusion performance (Section 2, Appendix A), and the MAETok method with ImageNet benchmarks (Sections 3-4). Neither part reduces to its own inputs by construction. The theoretical bound in Theorem 2.1 is assembled from external prior theorems, Theorem 16 of Shah et al. (2023) and Theorem 2.2 of Chen et al. (2023), under explicit assumptions (A.1-A.3). The paper does not fit parameters to force the K^4 scaling; the bound is quoted and combined, not derived from the authors' own unpublished claims. The empirical GMM analysis in Fig. 2a is a post-hoc measurement of learned latent codes, and Fig. 2b reports independently trained diffusion losses; the alignment is correlational evidence, not a fitted quantity renamed as a prediction. The latent-space probes (UMAP, linear probing accuracy, GMM NLL) are diagnostic evaluations of trained tokenizers, not losses used to train the tokenizer or the diffusion model, so they cannot be circular in the sense of 'input equals output by construction.' MAETok's design choices are ablated (Table 1, Table 6) against external metrics (rFID, gFID), and the main results compare SiT-XL and LightningDiT trained on MAETok against a broad set of published baselines, including DiT, SiT, REPA, MAR-H, and USiT. Self-citations appear (XQ-GAN, ImageFolder, SoftVQ), but they are used as implementation scaffolds or baseline references, not as load-bearing justifications for the central claim that variational regularization is unnecessary. There are genuine technical gaps: the score network in Eq. 13 is a softmax-linear mean estimator rather than the transformer denoiser used in experiments; Theorem A.5 guarantees only min_i ||mu_i - mu*_i|| while Eq. 18 uses a sum over all centers and weight mismatches; and the GMM mode counts are obtained after PCA and standardization, so the K in Theorem 2.1 is not exactly the empirically plotted component count. These are correctness and transfer-relevance concerns, not cases where a prediction is equivalent to its input by construction. No passage in the manuscript asserts a circular step or a missing proof that would change this assessment. Overall, the paper's central claims are supported by external benchmarks and independent measurements, and no specific reduction from a derived result back to its own fitted input or self-citation chain can be exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method's practical recipe introduces several hand-tuned hyperparameters, and the theory relies on standard mixture-learning assumptions plus a strong structural assumption about the score network. No new physical or conceptual entities are posited.

free parameters (4)
  • mask ratio range = 40-60%
    Chosen by ablation (Table 1c); higher ratios trade reconstruction fidelity for latent discriminativeness, and the final recipe uses a schedule from 60% to 0% during decoder fine-tuning.
  • loss weights (lambda1, lambda2) = lambda1=1.0, lambda2=0.4
    Chosen for the tokenizer objective in Eq. (6); these are standard hyperparameters and are not ablated in the paper.
  • auxiliary decoder depth = 3 layers
    Selected by ablation (Table 1d); too shallow or too deep hurts generation or reconstruction.
  • number of latent tokens = 128
    Ablated in Table 13a; 128 tokens is enough to match 256-token performance and is central to the efficiency claims.
assumptions (5)
  • domain assumption Assumption A.1: Gaussian mixture components are separated by at least C sqrt(log(min(K,d))).
    Imported from Shah et al. (2023); needed for Theorem A.5. It may not hold for real latent clusters.
  • domain assumption Assumption A.2: the algorithm is warm-started with each center estimate within C' sqrt(log(min(K,d))) of the true center.
    Also from Shah et al. (2023); a strong initialization requirement that is not checked on real tokenizer latents.
  • domain assumption Assumption A.3: max_i ||mu_i|| <= B, bounding the mean norm of each GMM component.
    Used to control second moments and bound the score error in Eq. (18); the paper argues empirically that compared tokenizers have similar B (Appendix A, Fig. 8).
  • ad hoc to paper The learned denoising network is assumed to have the softmax-linear form of Eq. (13), matching the oracle score's structure.
    The real transformer denoiser is not of this form; the theory holds for this restricted network class only.
  • ad hoc to paper Real latent spaces are treated as Gaussian mixtures with well-separated modes, so that fitted GMM mode counts correspond to the theorem's K.
    The empirical GMM fitting (Appendix B.3) uses PCA and normalization, so the extracted K is not the theorem's exact K; this is the main theory-to-practice bridge.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Masked Autoencoders Are Effective Tokenizers for Diffusion Models." pith.science (2026). https://pith.science/paper/7RZYR5LD

@misc{pith2026250203444,
  author       = {Pith},
  title        = {Pith review of: Masked Autoencoders Are Effective Tokenizers for Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RZYR5LD}},
  note         = {Machine review of arXiv:2502.03444}
}
read the original abstract

Recent advances in latent diffusion models have demonstrated their effectiveness for high-resolution image synthesis. However, the properties of the latent space from tokenizer for better learning and generation of diffusion models remain under-explored. Theoretically and empirically, we find that improved generation quality is closely tied to the latent distributions with better structure, such as the ones with fewer Gaussian Mixture modes and more discriminative features. Motivated by these insights, we propose MAETok, an autoencoder (AE) leveraging mask modeling to learn semantically rich latent space while maintaining reconstruction fidelity. Extensive experiments validate our analysis, demonstrating that the variational form of autoencoders is not necessary, and a discriminative latent space from AE alone enables state-of-the-art performance on ImageNet generation using only 128 tokens. MAETok achieves significant practical improvements, enabling a gFID of 1.69 with 76x faster training and 31x higher inference throughput for 512x512 generation. Our findings show that the structure of the latent space, rather than variational constraints, is crucial for effective diffusion models. Code and trained models are released.

Figures

Figures reproduced from arXiv: 2502.03444 by the authors.

Figure 1
Figure 1. Diffusion models with MAETok achieves state-of-the-art image generation on ImageNet of 512×512 and 256×256 resolution. models and thus to better sampling during inference. Motivated by these insights, we demonstrate that diffusion models trained on AEs with discriminative latent space are enough to achieve SOTA performance. We propose to train AEs as Masked Autoencoders (MAE) (He et al., 2022; Xie et al., 2022; Wei … view at source ↗
Figure 2
Figure 2. GMM fitting on latent space of AE, VAE, VAVAE, and MAETok. Fewer GMM modes in latent space usually corresponds to lower diffusion losses and better generation performance. 2. On the Latent Space and Diffusion Models To study the relationship of latent space for diffusion models, we start with popular tokenizers, including AE (Hinton & Salakhutdinov, 2006), VAE (Kingma, 2013), representation aligned VAE, i.e., VAVAE … view at source ↗
Figure 3
Figure 3. Model architecture of MAETok. We adopt the plain 1D autoencoder (AE) as tokenizer, with a vision transformer (ViT) en￾coder E and decoder D. MAETok is trained using mask modeling at encoder, with a mask ratio of 40-60%, and predict multiple tar￾get features, e.g., HOG, DINO-v2, and CLIP features, of masked tokens from the unmasked ones using auxiliary shallow decoders. why, under a finite number of training samples,… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: UMAP visualization on ImageNet of the learned latent space from (a) AE; (b) VAE; (c) MAETok. Colors indicate differ￾ent classes. MAETok presents a more discriminative latent space. as highlighted in previous works (He et al., 2022; Wei et al., 2022; Xie et al., 2022). …
Figure 6
Figure 6. Figure 6: Additional selected samples from 512×512 SiT-XL model on MAETok. We use a classifier-free guidance scale of 2.0. A. Theoretical Analysis Preliminary. We begin the theoretical analysis by introducing the preliminaries of the problem and the necessary notation. Following…
Figure 7
Figure 7. Figure 7: Additional selected samples from 256×256 diffusion models on MAETok. We use a classifier-free guidance scale of 2.0. Assumptions. To ensure the denoising network approximates the score function with sufficient accuracy, we consider the following three common assumption…
Figure 8
Figure 8. Figure 8: We compare the maximum mean norm across different numbers of components and observe that AE, VAE, VAVAE, and our method MAETok exhibit similar maximum mean norms. This suggests that these latent spaces share a comparable prior upper bound B, supporting the rationale fo…
Figure 9
Figure 9. Figure 9: UMAP visualization on ImageNet of the learned latent space from (a) MAETok with raw pixel target; (b) MAETok with HOG target; (c) MAETok with DINOv2 target; (d) MAETok with CLIP target. MAETok presents a more discriminative latent space. C.4. More Ablation Results We p…
Figure 10
Figure 10. Figure 10: Uncurated generation results of 256×256 MAETok + SiT-XL. We use CFG of 3.0. Class label = “Loggerhead” (33). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Uncurated generation results of 256×256 MAETok + SiT-XL. We use CFG of 3.0. Class label = “Macaw” (88) [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Uncurated generation results of 256×256 MAETok + SiT-XL. We use CFG of 3.0. Class label = “Cacatua galerita” (89). 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Uncurated generation results of 256×256 MAETok + SiT-XL. We use CFG of 3.0. Class label = “Flamingo” (130) [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Siberian husky” (250). 23 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Arctic fox” (279) [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Otter” (360). 24 [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Guitar” (402) [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Fire Truck” (555). 25 [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Go-kart” (573) [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Laptop” (620). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Carriage” (705) [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: Uncurated generation results of 512×512 MAETok + SiT-XL. We use CFG of 2.0. Class label = “Sports Car” (402). 27 [PITH_FULL_IMAGE:figures/full_fig_p027_22.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Physics-Informed Distillation of Diffusion Models for PDE-Constrained Generation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Post-hoc distillation with a PDE-residual loss on final samples avoids the Jensen gap and yields one-step physics-constrained generation.

  2. BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation

    cs.AI 2026-06 unverdicted novelty 5.0 of 10

    A volumetric MAE tokenizer decouples clinical embedding from reconstruction to support both 23-task linear probing and conditional 3D brain MRI generation via DiT.

  3. DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer

    cs.CV 2025-07 conditional novelty 5.0 of 10

    DC-AR generates 512x512 images in 12 masked autoregressive steps plus 20 diffusion refinement steps, using a 32x compressed 2D tokenizer, and reports gFID 5.49 on MJHQ-30K.

  4. Adaptive Mask-guided K-space Diffusion for Accelerated MRI Reconstruction

    eess.IV 2025-06 reject novelty 4.0 of 10

    AMDM reconstructs undersampled MRI by masking k-space frequency components with adaptive masks inside a diffusion model, and reports large PSNR gains over baseline methods.

Reference graph

Works this paper leans on

45 extracted references · 9 canonical work pages · cited by 4 Pith papers

  1. [4]

    Y ., Nam, H., and Ye, J

    Chung, H., Kim, J., Park, G. Y ., Nam, H., and Ye, J. C. Cfg++: Manifold-constrained classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08070,

    arXiv

  2. [6]

    We use a classifier-free guidance scale of 2.0

    Additional selected samples from 512×512 SiT-XL model on MAETok. We use a classifier-free guidance scale of 2.0. A. Theoretical Analysis Preliminary. We begin the theoretical analysis by introducing the preliminaries of the problem and the necessary notation. Following the empirical analysis setting, we first consider the latent data distribution is the G...

    work page 2023

  3. [7]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,

    2009

  4. [9]

    Mdtv2: Masked diffusion transformer is a strong image synthe- sizer

    Gao, S., Zhou, P., Cheng, M.-M., and Yan, S. Mdtv2: Masked diffusion transformer is a strong image synthe- sizer. arXiv preprint arXiv:2303.14389,

    arXiv

  5. [10]

    Learning mixtures of gaussians using diffusion models

    Gatmiry, K., Kelner, J., and Lee, H. Learning mixtures of gaussians using diffusion models. arXiv preprint arXiv:2404.18869,

    arXiv

  6. [11]

    Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis

    URL https: //arxiv.org/abs/2212.03185. He, K., Chen, X., Xie, S., Li, Y ., Doll´ar, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000–16009,

    work page Pith review arXiv

  7. [15]

    Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,

    arXiv

  8. [16]

    Applying guidance in a limited interval improves sample and distribution quality in diffusion models

    Kynk¨a¨anniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724,

    arXiv

Show all 45 references
  1. [17]

    Scalable autoregressive image generation with mamba

    Li, H., Yang, J., Wang, K., Qiu, X., Chou, Y ., Li, X., and Li, G. Scalable autoregressive image generation with mamba. arXiv preprint arXiv:2408.12245, 2024a. Li, T., Chang, H., Mishra, S. K., Zhang, H., Katabi, D., and Krishnan, D. Mage: Masked generative encoder to unify re...

  2. [18]

    Li, T., Tian, Y ., Li, H., Deng, M., and He, K

    URL https://arxiv.org/abs/2211.09117. Li, T., Tian, Y ., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization, 2024b. URL https://arxiv.org/abs/2406.11838. Li, X., Chen, H., Qiu, K., Kuen, J., Gu, J., Raj, B., and Lin, Z. Imagefolder: Auto...

    arXiv

  3. [20]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Loshchilov, I. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    arXiv

  4. [22]

    Umap: Uniform manifold approximation and projection for dimension reduction

    McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426,

    arXiv

  5. [23]

    org/abs/2102.09672

    URL https://arxiv. org/abs/2102.09672. Oquab, M., Darcet, T., Moutakanni, T., V o, H. V ., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.-Y ., Xu, H., Sharma, V ., Li, S.-W., Galuba, W., Rabbat, M., As- sran, M., Ballas,...

    arXiv

  6. [24]

    Qiu, K., Li, X., Kuen, J., Chen, H., Xu, X., Gu, J., Luo, Y ., Raj, B., Lin, Z., and Savvides, M

    URL https://arxiv.org/abs/ 2212.09748. Qiu, K., Li, X., Kuen, J., Chen, H., Xu, X., Gu, J., Luo, Y ., Raj, B., Lin, Z., and Savvides, M. Robust latent matters: Boosting image generation with sampling error synthesis. arXiv preprint arXiv:2503.08354,

    arXiv

  7. [25]

    K., Yuan, Z., and Wu, X

    Qu, L., Zhang, H., Liu, Y ., Wang, X., Jiang, Y ., Gao, Y ., Ye, H., Du, D. K., Yuan, Z., and Wu, X. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069,

    arXiv

  8. [26]

    Generating diverse high-fidelity images with vq-vae-2

    Razavi, A., Van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019a. Razavi, A., van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2, 2019b. URL ht...

    1906 arXiv

  9. [27]

    Deep unsupervised learning using nonequi- librium thermodynamics

    Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256–2265. PMLR, 2015a. Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli...

    arXiv

  10. [28]

    Sun, P., Jiang, Y ., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z

    URL https://arxiv.org/ abs/2010.02502. Sun, P., Jiang, Y ., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525,

    2010 arXiv

  11. [29]

    org/abs/2404.02905

    URL https://arxiv. org/abs/2404.02905. Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Gen- erative infinite-vocabulary transformers. In European Conference on Computer Vision, pp. 292–309. Springer,

    arXiv

  12. [30]

    Vahdat, A., Kreis, K., and Kautz, J

    URL https://arxiv.org/abs/ 2104.03310. Vahdat, A., Kreis, K., and Kautz, J. Score-based generative modeling in latent space,

    arXiv

  13. [31]

    org/abs/2106.05931

    URLhttps://arxiv. org/abs/2106.05931. 12 Masked Autoencoders Are Effective Tokenizers for Diffusion Models Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. Conditional image generation with pixelcnn decoders. Advances in neural information pro-...

    arXiv

  14. [32]

    Vincent, P., Larochelle, H., Bengio, Y ., and Manzagol, P.-A

    URL https://arxiv.org/ abs/1706.03762. Vincent, P., Larochelle, H., Bengio, Y ., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096–1103,

    arXiv

  15. [34]

    Vila-u: a unified foundation model integrating visual understanding and generation

    Wu, Y ., Zhang, Z., Chen, J., Tang, H., Li, D., Fang, Y ., Zhu, L., Xie, E., Yin, H., Yi, L., et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429,

    arXiv

  16. [35]

    and Wang, X

    Yao, J. and Wang, X. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501.01423,

    arXiv

  17. [36]

    Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J., and Wu, Y

    Yu, J., Li, X., Koh, J. Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J., and Wu, Y . Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627,

    arXiv

  18. [37]

    B., Versari, L., Sohn, K., Minnen, D., Cheng, Y ., Gupta, A., Gu, X., Haupt- mann, A

    Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y ., Gupta, A., Gu, X., Haupt- mann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion - tokenizer is key to visual generation. In The Twelfth Inte...

    arXiv

  19. [38]

    URLhttps://arxiv.org/ abs/1801.03924. Zhao, X. and Schwing, A. G. Studying classifier (-free) guid- ance from a classifier-centric perspective. arXiv preprint arXiv:2503.10638,

    arXiv

  20. [39]

    Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%

    Zhu, L., Wei, F., Lu, Y ., and Chen, D. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406.11837,

    arXiv

  21. [41]

    We use a classifier-free guidance scale of 2.0

    Additional selected samples from 256×256 diffusion models on MAETok. We use a classifier-free guidance scale of 2.0. Assumptions. To ensure the denoising network approximates the score function with sufficient accuracy, we consider the following three common assumptions, which...

    2022

  22. [42]

    The maximum mean norm of the GMM in GMM 9 is bounded as: maxi ∥µi∥ ≤B

    Assumption A.3. The maximum mean norm of the GMM in GMM 9 is bounded as: maxi ∥µi∥ ≤B. Remark A.4. By Assumption A.3, we could derive the second movement bound of p0 as Ex∼p0 [∥x∥2] = Z p0(x)∥x∥2dx ≤ d + B2 (17) Then, we can have the following analysis, Step 1: From K Modes to...

    2023

  23. [43]

    In the practical sampling process, we adopt an early stopping strategy to improve the generation quality

    Therefore, consider a step size hk ≤ γ, we can have the learned score function sθt (x) satisfies 1 T NX k=1 hkE[∥sθtk (xtk ) − ∇xtk log pt(xtk )∥2] ≲ N γ T (ϵ2 + B2) (19) Step 2: From Training Loss to Samlping Error. In the practical sampling process, we adopt an early stoppin...

    2023

  24. [44]

    Training Details of Diffusion Models We present the training details of SiT-XL and LightningDiT in Tables 8 and 9, which mainly follows their original setup

    B.2. Training Details of Diffusion Models We present the training details of SiT-XL and LightningDiT in Tables 8 and 9, which mainly follows their original setup. B.3. Training Details of GMM Models In Fig. 2, we train our own AE, KL-V AE, and MAETok under exactly the same set...

    2025

  25. [45]

    However, it is still extremely difficult to tune the guidance scale

    to cutout the high timesteps with CFG can mitigate this issue. However, it is still extremely difficult to tune the guidance scale. We use a guidance scale of 1.9 and an interval of [0, 0.75] for 256×256 SiT-XL and a guidance scale of 1.8 and an interval of [0, 0.75] for 256×2...

    2024

  26. [2005]

    Causal diffusion transformers for generative modeling

    Deng, C., Zh, D., Li, K., Guan, S., and Fan, H. Causal diffusion transformers for generative modeling. arXiv preprint arXiv:2412.12095,

    arXiv

  27. [2006]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

    arXiv

  28. [2008]

    Maskbit: Embedding-free image gen- eration via bit tokens

    Weber, M., Yu, L., Yu, Q., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. Maskbit: Embedding-free image gen- eration via bit tokens. arXiv preprint arXiv:2409.16211,

    arXiv

  29. [2014]

    Customize your visual autoregressive recipe with set au- toregressive modeling

    Liu, W., Zhuo, L., Xin, Y ., Xia, S., Gao, P., and Yue, X. Customize your visual autoregressive recipe with set au- toregressive modeling. arXiv preprint arXiv:2410.10511,

    arXiv

  30. [2017]

    S., Boffi, N

    11 Masked Autoencoders Are Effective Tokenizers for Diffusion Models Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. arXiv preprint arXiv:...

    arXiv

  31. [2018]

    Johnson, J., Alahi, A., and Fei-Fei, L

    URL https://arxiv.org/abs/ 1611.07004. Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Com- puter Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part II...

    2016 arXiv

  32. [2019]

    Guiding a diffusion model with a bad version of itself

    Karras, T., Aittala, M., Kynk¨a¨anniemi, T., Lehtinen, J., Aila, T., and Laine, S. Guiding a diffusion model with a bad version of itself. arXiv preprint arXiv:2406.02507,

    arXiv

  33. [2021]

    Esser, P., Rombach, R., and Ommer, B

    URL https: //arxiv.org/abs/2010.11929. Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873–12883,

    2010 arXiv

  34. [2022]

    Chen, H., Lee, H., and Lu, J

    URL https://arxiv.org/abs/2202.04200. Chen, H., Lee, H., and Lu, J. Improved analysis of score- based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Con- ference on Machine Learning , pp. 4735–4763. PMLR,

    arXiv

  35. [2023]

    Softvq- vae: Efficient 1-dimensional continuous tokenizer

    Chen, H., Wang, Z., Li, X., Sun, X., Chen, F., Liu, J., Wang, J., Raj, B., Liu, Z., and Barsoum, E. Softvq- vae: Efficient 1-dimensional continuous tokenizer. arXiv preprint arXiv:2412.10958, 2024a. Chen, H., Jiang, K., Zheng, K., Chen, J., Su, H., and Zhu, J. Visual generatio...

    arXiv

  36. [2024]

    and Triggs, B

    Dalal, N. and Triggs, B. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), volume 1, pp. 886–893. Ieee,

    2005

  37. [2025]

    Deep compression autoen- coder for efficient high-resolution diffusion models.arXiv preprint arXiv:2410.10733, 2024b

    Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y ., and Han, S. Deep compression autoen- coder for efficient high-resolution diffusion models.arXiv preprint arXiv:2410.10733, 2024b. Chen, S., Chewi, S., Li, J., Li, Y ., Salim, A., and Zhang, A. R. Sampli...

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.