Pith. sign in

REVIEW 3 major objections 8 minor 1 cited by

Nested Diffusion Models Using Hierarchical Latent Priors

T0 review · 3 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Nested diffusion models use a frozen encoder's semantic hierarchy to cut ImageNet FID from 45.19 to 11.05 unconditionally.

desk verdict A simple, effective recipe for stacking diffusion models on frozen encoder features, with big FID gains; the main caveat is heavy tuning on the reported metric and an unaddressed exposure-bias gap. read the letter →

arxiv 2412.05984 v1 pith:V5QC2PIY submitted 2024-12-08 cs.CV

Xiao Zhang , Ruoxi Jiang , Rebecca Willett , Michael Maire This is my paper · ORCID
classification cs.CV
keywords nesteddiffusionmodelshierarchicallatentpriorssemantichierarchyimagegenerationfrozenpretrainedencoderfeaturecompressionNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces nested diffusion models, a hierarchy of diffusion models that generate an image by first synthesizing abstract semantic latents and then progressively refining them into finer details. The key claim is that conditioning each level on compressed, noise-corrupted features from a frozen pretrained visual encoder dramatically improves generation quality. On ImageNet-1K, the five-level model reduces FID from 45.19 to 11.05 for unconditional generation and from 31.13 to 9.87 for class-conditional generation, with only about a 25% increase in compute, and the unconditional system outperforms the class-conditional baseline (19.74 FID). The central insight is that a semantic ladder of intermediate targets prevents the generator from degenerating into an autoencoder and guides it along meaningful structure.

What carries the argument

The nested diffusion architecture: a sequence of L diffusion models Dθl, where level l generates latent zl conditioned on higher-level latents z>l and z1 is the image itself. The latent hierarchy is extracted from a frozen encoder (MoCo-v3 for ImageNet, CLIP for COCO) via patchification, SVD channel reduction, and Gaussian noise injection (σl), which together control the information each level transmits. The training loss (Eqn. 3) sums per-level denoising objectives, each conditioned on noisy higher-level latents, with a separate noising process for the conditional signal.

What would settle it

Train the same five-level architecture but replace the encoder features with random noise of identical shape (or features from an untrained encoder): if the FID improvement persists, the semantic content is not the cause. Alternatively, during sampling, feed oracle encoder features as the top-level latent rather than the model's own generated latent and measure whether FID degrades sharply, which would reveal a teacher-forcing distribution shift.

Watch

Extended reading notes

Core claim

The paper claims that a chain of diffusion models, each generating a latent variable at a different semantic level and conditioned on the outputs of higher levels, produces substantially better images than a single-level model at similar computational cost. The hierarchy is built from a frozen pretrained visual encoder applied to image patches of decreasing size, with singular value decomposition to trim feature channels and Gaussian noise injection to limit information capacity. With five levels, the unconditional generator surpasses the class-conditional baseline, suggesting that hierarchical semantic priors—not just model size or class labels—are what drive the quality gain.

Load-bearing premise

The load-bearing premise is that a frozen pretrained encoder's features, after SVD truncation and Gaussian noise, form a generable semantic ladder, and that training on ground-truth encoder latents transfers to sampling with model-generated latents.

Editorial extensions

If this is right

  • Adding more hierarchical levels consistently improves FID on ImageNet-1K, both with and without classifier-free guidance, so depth itself is a scaling axis for generation quality.
  • Unconditional generation can outperform class-conditional generation when anchored to a rich semantic hierarchy, implying that semantic structure can substitute for explicit labels.
  • Gaussian noise injection is essential for deep hierarchies: without it, the generator reverts to an autoencoder that bypasses higher levels, so information-capacity control is what makes the approach scale.
  • The extra compute from deeper hierarchies is modest because higher levels operate on low-dimensional representations, making the quality gain nearly compute-free.
  • On COCO text-to-image generation, a 2-level model trained only on COCO beats much larger models trained on extra data, suggesting the hierarchy transfers to complex scene synthesis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The semantic ladder could be extended beyond static images to video or 3D scenes, where the same frozen-encoder hierarchy might guide temporal or geometric structure.
  • Because the paper shows generation quality tracks the KNN accuracy of the chosen encoder, swapping in a better representation learner should yield further FID gains without architectural changes.
  • The paper trains each level on ground-truth encoder latents (teacher forcing); the authors implicitly assume generated latents stay close to those, and a testable extension is to measure FID when the top-level latent is drawn from the model's own prior versus an oracle encoder feature.
  • The hierarchy offers a natural control handle: resampling only low levels edits fine detail while preserving global semantics, which could enable semantic editing and style transfer with no extra training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes a hierarchical generative framework called nested diffusion models. It trains a chain of L diffusion models: the top-level model generates a low-dimensional semantic latent z_L from noise; each lower-level model generates z_l conditioned on previously generated higher-level latents z_{>l}; the bottom level generates the image. The latent targets z_l are extracted from a frozen pretrained visual encoder (MoCo-v3 for ImageNet, CLIP for COCO) via patchification, SVD-based channel reduction, and Gaussian noise injection, with the noise level sigma_l controlling information capacity. The training loss (Eq. 3) is a denoising objective in which lower-level models are conditioned on noisy ground-truth encoder latents. On ImageNet-1K 256x256, a five-level model reduces FID from 45.19 to 11.05 for unconditional generation and from 31.13 to 9.87 for class-conditional generation relative to a single-level baseline, at an inference cost of roughly 34 GFlops versus 26.8 GFlops. On COCO-2014 text-to-image generation, the hierarchical model outperforms the U-ViT baseline. The paper also claims that the unconditional five-level model surpasses the class-conditional single-level baseline. The paper includes ablations over depth, noise level, encoder choice, and the gamma schedule for generation-time noise.

Significance. If the results are robust, the contribution is significant: it offers a conceptually simple and computationally lightweight way to inject semantic structure into diffusion generation using frozen visual encoders, with large FID improvements over single-level baselines. The controlled comparison to a GFlops-matched baseline (Table 6), the consistent improvements with depth for nonzero noise levels in Table 1, the encoder ablation in Table 5, and the cross-dataset COCO experiments are strengths. The paper is also transparent about the role of noise injection and reports the key hyperparameter choices. However, three issues currently prevent full confidence: (i) the teacher-forcing mismatch between training on noisy encoder latents and sampling from model-generated latents is never measured; (ii) the headline FIDs are selected over sigma_l and gamma without seed variance or a held-out validation split; and (iii) the ImageNet encoder is pretrained on the same distribution as the evaluation set, which may inflate the unconditional-generation gains. These concerns are addressable with additional experiments and analysis, so the work merits a major revision rather than rejection.

major comments (3)
  1. [Sec. 3.3, Eq. (3), Sec. 4.2] The training objective in Eq. (3) conditions every lower-level denoiser on noisy ground-truth encoder latents, hat z_{>l} ~ N(z_{>l}(x), sigma_l^2 I), but at generation time the conditioning latents are ancestral samples from the upper-level diffusion models. The paper never trains on model-generated latents, so the lower-level models only see encoder-derived conditioning inputs during training. This distribution shift is never quantified. Moreover, for the no-CFG case the best FID is obtained with gamma = infinity (Table 3), i.e., no noise is added to the conditioning latents at generation, while all training runs use nonzero sigma_l. The reported gains therefore depend on the untested assumption that model-generated latents fall in a region where the lower-level denoisers, trained on noisier encoder latents, remain well-behaved. I request a direct experiment comparing FID when conditioning on encoder latents (oracle) versus model-generated latents, together with statistics of the marginal distributions of the latents under q and p_theta, or an experiment with scheduled sampling / self-conditioning during training. Without this, the headline FIDs (11.05 and 9.87) cannot be confidently attributed to a usable hierarchical latent prior.
  2. [Table 1, Sec. 4.1, Sec. 4.2] For each depth L, the reported FID in Table 1 is the minimum over sigma_l in {0, 0.5, 1.0}, and the gamma value in Table 3 is likewise chosen on the reported FID. No seed variance or error bars are given. Under a fixed noise level sigma^2 = 0 in Table 1(a), the FIDs for L = 2, 3, 4, 5 are 19.32, 20.34, 17.67, and 19.04, which are not monotonic in L; the claim that generation quality consistently improves with L therefore rests on selecting a different noise level for each depth. The same selection issue applies to the CFG weights in Sec. 4.3. I request at least three seeds for the reported configurations, with mean and standard deviation, and a validation split for all hyperparameter choices (sigma_l, gamma, CFG weights) so that the reported test FIDs are not the result of selection on the evaluation set.
  3. [Sec. 4.1, Table 5] For the ImageNet experiments, the hierarchy is built with MoCo-v3 features from a model pretrained on the same ImageNet-1K distribution. The unconditional system is therefore conditioned, at every level, on dataset-specific statistics learned from the very distribution used to compute FID. This is not circular in the derivation, but it weakens the benchmark claim: the FID gains may partly reflect that the conditioning latents provide a compressed encoding of the training distribution, rather than a generable semantic hierarchy. The COCO encoder ablation in Table 5 shows that feature quality matters, but it does not control for pretraining-domain overlap. A concrete test would be to swap MoCo-v3 for a ViT-B/16 encoder pretrained on data that excludes ImageNet (e.g., ImageNet-21K or another large corpus) and report ImageNet FID. If the FID degrades sharply, the claims that the method learns a general semantic prior, and that the unconditional system surpasses the conditional baseline, would need to be qualified.
minor comments (8)
  1. [Eq. (2)] In the top-level term of Eq. (2), the notation p_theta_l should be p_theta_L, since the KL divergence is for the top-level latent z_L.
  2. [Sec. 3.2] The text says 'the latent variable z_l in R' and should read 'z_l in R^{d_l}' to be consistent with the dimensions used later.
  3. [Eq. (4)] Eq. (4) uses a discrete sum over z_l even though z_l is a continuous latent; the expression should be written as an integral, or the discrete notation should be justified.
  4. [Sec. 3.3] The sentence 'p_theta_L(z_0 | z_L) is approximately 1' appears to be a typo: the reconstruction path at the bottom level is through D_theta_1, not D_theta_L. Please correct the subscript or clarify the intended level.
  5. [Table 1] The column header 'sigma^2 = 0.0 0.5 1.0' is ambiguous: the text alternates between sigma and sigma^2 for the noise level. Please state clearly whether the table reports sigma or sigma^2.
  6. [Sec. 4.3 and Table 6] The abstract says the computational overhead is about 25%, but Sec. 4.3 says 27.00% and Table 6 implies roughly 26.7% ((33.98-26.82)/26.82). Please harmonize these numbers.
  7. [Appendix A.2, Eqs. (12)-(15)] The derivation drops the outer expectation over z_{>l} that appears in Eq. (12) when passing to Eq. (15); the loss in Eq. (3) conditions on z_{>l}, so this expectation should be retained or the notational shortcut should be explained.
  8. [Table 4] The explanation that L=3 underperforms L=2 with CFG 'likely due to a suboptimal CFG weight' is speculative; please either provide the CFG-tuned result or remove the conjecture.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ELBO derivation and external FID benchmarks keep the central claims independent of their inputs.

full rationale

The derivation chain is self-contained: Eqn. 3 is the standard hierarchical ELBO (Eqn. 2) with frozen encoder latents; Appendix A.2 derives it via Jensen's inequality and the usual diffusion bound, without inserting the target FID or the final samples as assumptions. The headline numbers are measured against external precomputed ImageNet/COCO validation statistics (Dhariwal and Nichol), so no fitted parameter is renamed as a prediction. The noise levels sigma_l and gamma are validation-set hyperparameters selected by greedy search; this is model selection, not a definitional reduction. The pretrained encoders (MoCo-v3, CLIP, DINO) are external models; Table 5's correlation between KNN accuracy and FID is an empirical dependence, not a definitional equivalence. The paper explicitly acknowledges the only real weakness, the train/test distribution shift from teacher-forced noisy latents to ancestrally sampled latents (Sec. 4.2), and proposes a gamma decay schedule; this is a correctness/robustness risk, not circularity. The two self-citations ([78], [79]) are related-work mentions and carry none of the argument. No uniqueness theorem or ansatz is imported from the authors' prior work.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method's free parameters are mostly noise variances and CFG weights tuned on the evaluation metric. The key assumptions are about the usefulness of frozen encoder latents and the transfer from teacher-forced training to ancestral sampling. No new physical entities are introduced.

free parameters (5)
  • sigma_l (per-level Gaussian noise variance) = sigma_2 in {0.0, 0.5, 1.0}; deeper sigma_l retained from previous levels
    Controls information capacity of conditional latents. Selected by top-down greedy search on FID (Sec. 4.1, Table 1).
  • CFG weights {w_i} = [0.5, 0.4, 0.3, 0.2, 0.1] for conditional; constant 0.8 for unconditional
    Selected via limited hyperparameter search on FID (Sec. 4.3).
  • gamma (noise decay exponent) = 0.3 with CFG; infinity without CFG
    Controls (t/T)^gamma sigma_l scheduling during generation; chosen from Table 3 to minimize FID.
  • SVD dimension budgets = z_L in R^256; z_{L-1} in R^{4x128}; z_{L-2} in R^{16x64}
    Hand-chosen architecture choices for channel reduction; not fitted but part of the method's manual design.
  • sigma_l for COCO = 0.5 fixed
    Fixed without sweep in COCO; still a hand-set constant.
assumptions (4)
  • domain assumption Pre-trained visual encoder features, after patchification and compression, form a semantic hierarchy that improves generation.
    Central premise of Sec. 3.3; validated only by FID results, not by an independent measure of hierarchy quality.
  • domain assumption Training each level conditioned on ground-truth noisy latents (teacher forcing) transfers to sampling the full hierarchy from noise.
    Loss Eqn. 3 uses E over z_{>l}; error compounding across levels is not analyzed or mitigated.
  • domain assumption Gaussian noise injection and SVD truncation prevent the lower-level model from degenerating into an autoencoder.
    Shown by Figure 4 for one oracle condition and by the sigma_2=0 vs sigma_2>0 gap in Table 1, but not theoretically justified.
  • standard math The optimal denoiser is a weighted average over data points (mean-shift interpretation, Eqn. 4).
    Borrowed from cited analyses [36, 63] and used to motivate neighbor structure; not central to the loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nested Diffusion Models Using Hierarchical Latent Priors." pith.science (2026). https://pith.science/paper/V5QC2PIY

@misc{pith2026241205984,
  author       = {Pith},
  title        = {Pith review of: Nested Diffusion Models Using Hierarchical Latent Priors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5QC2PIY}},
  note         = {Machine review of arXiv:2412.05984}
}
read the original abstract

We introduce nested diffusion models, an efficient and powerful hierarchical generative framework that substantially enhances the generation quality of diffusion models, particularly for images of complex scenes. Our approach employs a series of diffusion models to progressively generate latent variables at different semantic levels. Each model in this series is conditioned on the output of the preceding higher-level models, culminating in image generation. Hierarchical latent variables guide the generation process along predefined semantic pathways, allowing our approach to capture intricate structural details while significantly improving image quality. To construct these latent variables, we leverage a pre-trained visual encoder, which learns strong semantic visual representations, and modulate its capacity via dimensionality reduction and noise injection. Across multiple datasets, our system demonstrates significant enhancements in image quality for both unconditional and class/text conditional generation. Moreover, our unconditional generation system substantially outperforms the baseline conditional system. These advancements incur minimal computational overhead as the more abstract levels of our hierarchy work with lower-dimensional representations.

Figures

Figures reproduced from arXiv: 2412.05984 by the authors.

Figure 1
Figure 1. Image generation via diffusion models nested along a hierarchical semantic chain. We synthesize images using a sequence of diffusion models to generate a hierarchy of latent representations, starting from a low-dimensional semantic feature embedding and refining to a detailed image. At each hierarchical level, synthesis of a higher-dimensional latent from noise is conditioned on the more abstract latents generated a… view at source ↗
Figure 2
Figure 2. Image generation quality when scaling our nested dif￾fusion models on ImageNet-1K dataset. The deeper hierarchies we build lead to a slight increase in computational overhead (par￾ticularly when L ≤ 4), as measured by GFlops, while significantly improving the generation quality. Compared to the single-level baseline model using comparable GFlops, our 5-level unconditional system significantly improves the performanc… view at source ↗
Figure 3
Figure 3. Nested diffusion architecture. Left: We train a sequence of diffusion models to generate a hierarchical collection of latent representations {z3, z2, z1 = x} of increasing dimensionality up to an image z1 = x. Generated latents serve as conditional inputs (dotted lines) to diffusion models at subsequent levels, with separately parameterized noising processes, zˆl ∼ N (zl, σ2 l I), controlling the information capacit… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Feature compression via Gaussian noise. For a two￾level hierarchical generator (L = 2), we generate images condi￾tioned on an oracle CLIP feature z2, inferred from input images, with feature channels reduced from 512 to 256 dimensions via SVD. Without noise (σ2 = 0) ad…
Figure 5
Figure 5. Figure 5: Visualization of K-Nearest Neighbors (KNN) with different sources of latent features. For each input image, we display neighboring images, based on features extracted from two types of visual representations: CLIP representations, and VAE bottlenecks. Unlike the VAE, w…
Figure 6
Figure 6. Figure 6: Visualization of unconditional image generation on ImageNet-1K. We present visualizations of images generated by hierarchical diffusion models containing from 2 to 5 levels, demonstrating that image quality improves as the depth of the hierarchy increases. models due t…
Figure 7
Figure 7. Figure 7: Visualization of text-to-image generation on COCO2014. We present visualizations of images generated by hierarchical diffusion models of 2 and 3 levels. Following the derivation in Sohl-Dickstein et al.1 , the loss at each level l can be further reduced as Ll ≤ P t R d…
Figure 8
Figure 8. Figure 8: Visualization of conditional image generation on ImageNet-1K. We present visualizations of images generated by hierarchical diffusion models containing from 2 to 5 levels [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Visualization of unconditional image generation on ImageNet-1K. More visualizations of images generated by hierarchical diffusion models containing from 2 to 5 levels [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Implicit Neural Emulators

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Feeding a hierarchy of predicted coarse-grained future states into an autoregressive neural emulator greatly improves long-term stability for 2D turbulent flow forecasting.

Reference graph

Works this paper leans on

81 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Styleflow: Attribute-conditioned exploration of stylegan- generated images using conditional continuous normalizing flows

    Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Attribute-conditioned exploration of stylegan- generated images using conditional continuous normalizing flows. TOG, 2021. 2

    2021

  2. [2]

    Diffusion-based representa- tion learning

    Korbinian Abstreiter, Sarthak Mittal, Stefan Bauer, Bernhard Schölkopf, and Arash Mehrjou. Diffusion-based representa- tion learning. arXiv:2105.14257, 2021. 3

    arXiv 2021

  3. [3]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023. 2, 5, 6, 8

    work page 2023

  4. [4]

    Label-efficient semantic seg- mentation with diffusion models

    Dmitry Baranchuk, Ivan Rubachev, Andrey V oynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic seg- mentation with diffusion models. arXiv:2112.03126, 2021. 2, 3

    arXiv 2021

  5. [5]

    End- to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-end object detection with transformers. In ECCV, 2020. 1

    work page 2020

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InICCV,

  7. [7]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geof- frey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2

    2020

  8. [8]

    Re-imagen: Retrieval-augmented text-to-image gen- erator

    Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmented text-to-image gen- erator. arXiv:2209.14491, 2022. 8

    arXiv 2022

Show all 81 references
  1. [9]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. InICCV,

  2. [10]

    Mean shift, mode seeking, and clustering

    Yizong Cheng. Mean shift, mode seeking, and clustering. PAMI, 1995. 5

    1995

  3. [11]

    Very deep vaes generalize autoregressive mod- els and can outperform them on images

    Rewon Child. Very deep vaes generalize autoregressive mod- els and can outperform them on images. arXiv:2011.10650,

    2011 arXiv

  4. [12]

    Mean shift analysis and applications

    Dorin Comaniciu and Peter Meer. Mean shift analysis and applications. In ICCV, 1999. 5

    1999

  5. [13]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021. 6

    2021

  6. [14]

    Generative models: What do they know? do they know things? let’s find out! arXiv:2311.17137, 2023

    Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, and Anand Bhattad. Generative models: What do they know? do they know things? let’s find out! arXiv:2311.17137, 2023. 2

    2023 arXiv

  7. [15]

    Frido: Feature pyramid diffusion for complex scene image synthesis

    Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In AAAI, 2023. 8

    2023

  8. [16]

    Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik

    Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Region-based convolutional networks for accurate object detection and segmentation. PAMI, 2016. 1

    2016

  9. [17]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS,

  10. [18]

    Matryoshka diffusion models

    Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In ICLR,

  11. [19]

    Vector quan- tized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quan- tized diffusion model for text-to-image synthesis. In CVPR,

  12. [20]

    Girshick, and Jitendra Malik

    Bharath Hariharan, Pablo Andrés Arbeláez, Ross B. Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-grained localization. In CVPR, 2015. 1

    2015

  13. [21]

    Efficientvdvae: Less is more

    Louay Hazami, Rayhane Mama, and Ragavan Thurairatnam. Efficientvdvae: Less is more. arXiv:2203.13751, 2022. 3

    2022 arXiv

  14. [22]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 1

    2016

  15. [23]

    Girshick

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017. 1

    2017

  16. [24]

    Momentum contrast for unsupervised visual repre- sentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual repre- sentation learning. In CVPR, 2020. 2

    2020

  17. [25]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 2, 8

    2022

  18. [26]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022. 5

    2022 arXiv

  19. [27]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 3

    2020

  20. [28]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 4

    2020

  21. [29]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 2022. 3

    2022

  22. [30]

    Self-guided diffusion mod- els

    Vincent Tao Hu, David W Zhang, Yuki M Asano, Gertjan J Burghouts, and Cees GM Snoek. Self-guided diffusion mod- els. In CVPR, 2023. 2, 3

    2023

  23. [31]

    Soda: Bottle- neck diffusion models for representation learning

    Drew A Hudson, Daniel Zoran, Mateusz Malinowski, An- drew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottle- neck diffusion models for representation learning. In CVPR,

  24. [32]

    Training neural operators to preserve invariant measures of chaotic attractors

    Ruoxi Jiang, Peter Y Lu, Elena Orlova, and Rebecca Willett. Training neural operators to preserve invariant measures of chaotic attractors. In NeurIPS, 2024. 3

    2024

  25. [33]

    How far is video generation from world model: A physical law perspective

    Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. arXiv:2411.02385, 2024. 2

    2024 arXiv

  26. [34]

    Scaling up gans for text-to-image synthesis

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, 2023. 3

    2023

  27. [35]

    Diffusion models for zero-shot open-vocabulary segmentation

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. arXiv:2306.09316, 2023. 3

    2023 arXiv

  28. [36]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022. 5

    2022

  29. [37]

    Variational diffusion models

    Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurIPS, 2021. 4

    2021

  30. [38]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv:1312.6114, 2013. 2, 3

    2013 arXiv

  31. [39]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Im- agenet classification with deep convolutional neural networks. In NeurIPS, 2012. 1

    2012

  32. [40]

    Your diffusion model is secretly a zero-shot classifier

    Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In CVPR, 2023. 2

    2023

  33. [41]

    Mage: Masked generative en- coder to unify representation learning and image synthesis

    Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative en- coder to unify representation learning and image synthesis. In CVPR, 2023. 2, 3

    2023

  34. [42]

    Self- conditioned image generation via generating representations

    Tianhong Li, Dina Katabi, and Kaiming He. Self- conditioned image generation via generating representations. arXiv:2312.03701, 2023. 2, 3

    2023 arXiv

  35. [43]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 5

    2014

  36. [44]

    Alleviating distortion in image genera- tion via multi-resolution diffusion models

    Qihao Liu, Zhanpeng Zeng, Ju He, Qihang Yu, Xiaohui Shen, and Liang-Chieh Chen. Alleviating distortion in image genera- tion via multi-resolution diffusion models. arXiv:2406.09416,

    arXiv

  37. [45]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR,

  38. [46]

    Optimizing hierarchical image vaes for sample quality

    Eric Luhman and Troy Luhman. Optimizing hierarchical image vaes for sample quality. arXiv:2210.10205, 2022. 2, 3

    2022 arXiv

  39. [47]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv:2112.10741, 2021. 2, 8

    2021 arXiv

  40. [48]

    Normalizing flows for probabilistic modeling and inference

    George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. JMLR, 2021. 2

    2021

  41. [49]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 6, 8

    2023

  42. [50]

    Variance reduction in hierarchical variational autoencoders

    Adeel Pervez and Efstratios Gavves. Variance reduction in hierarchical variational autoencoders. 2020. 2, 4, 12

    2020

  43. [51]

    Diffusion autoencoders: Toward a meaningful and decodable representation

    Konpat Preechakul, Nattanat Chatthee, Suttisak Wizad- wongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In CVPR,

  44. [52]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 6, 8

    2021

  45. [53]

    Hierarchical text-conditional image genera- tion with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image genera- tion with clip latents. arXiv:2204.06125, 2022. 8

    2022 arXiv

  46. [54]

    Generative adver- sarial text to image synthesis

    Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee. Generative adver- sarial text to image synthesis. In ICML, 2016. 3

    2016

  47. [55]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 2, 3

    2022

  48. [56]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 5

    2015

  49. [57]

    Cads: Unleashing the diver- sity of diffusion models through condition-annealed sampling

    Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Cads: Unleashing the diver- sity of diffusion models through condition-annealed sampling. arXiv:2310.17347, 2023. 7

    2023 arXiv

  50. [58]

    Pho- torealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Pho- torealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022. 8

    2022

  51. [59]

    Shadows don’t lie and lines can’t bend! generative models don’t know projective geometry

    Ayush Sarkar, Hanlin Mai, Amitabh Mahapatra, Svetlana Lazebnik, David A Forsyth, and Anand Bhattad. Shadows don’t lie and lines can’t bend! generative models don’t know projective geometry... for now. In CVPR, 2024. 2

    2024

  52. [60]

    Very deep convolu- tional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolu- tional networks for large-scale image recognition. In ICLR,

  53. [61]

    Ladder variational autoencoders

    Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NeurIPS, 2016. 2

    2016

  54. [62]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020. 2, 3

    2010 arXiv

  55. [63]

    Improved techniques for training score-based generative models

    Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In NeurIPS, 2020. 5

    2020

  56. [64]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv:2011.13456, 2020. 3

    2011 arXiv

  57. [65]

    Hq-vae: Hierarchical discrete representation learning with variational bayes

    Yuhta Takida, Yukara Ikemiya, Takashi Shibuya, Kazuki Shimada, Woosung Choi, Chieh-Hsin Lai, Naoki Murata, Toshimitsu Uesaka, Kengo Uchida, Wei-Hsiang Liao, et al. Hq-vae: Hierarchical discrete representation learning with variational bayes. arXiv:2401.00365, 2023. 3, 12

    2023 arXiv

  58. [66]

    Emergent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. NeurIPS, 2023. 3

    2023

  59. [67]

    Visual autoregressive modeling: Scalable image gen- eration via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image gen- eration via next-scale prediction. arXiv:2404.02905, 2024. 3

    2024 arXiv

  60. [68]

    Nvae: A deep hierarchical variational autoencoder

    Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. In NeurIPS, 2020. 2, 3, 4, 12

    2020

  61. [69]

    Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation

    Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR,

  62. [70]

    Instancediffusion: Instance-level control for image generation

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In CVPR, 2024. 3

    2024

  63. [71]

    Low-light image enhancement with normalizing flow

    Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap- Pui Chau, and Alex Kot. Low-light image enhancement with normalizing flow. In AAAI, 2022. 2

    2022

  64. [72]

    Open-vocabulary panoptic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR,

  65. [73]

    Diffusion model as repre- sentation learner

    Xingyi Yang and Xinchao Wang. Diffusion model as repre- sentation learner. In ICCV, 2023. 2, 3

    2023

  66. [74]

    Scaling autore- gressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autore- gressive models for content-rich text-to-image generation. arXiv:2206.10789, 2022. 8

    2022 arXiv

  67. [75]

    Scaling autoregres- sive multi-modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregres- sive multi-modal models: Pretraining and instruction tuning. arXiv:2309.02591, 2023. 8

    2023 arXiv

  68. [76]

    Representa- tion alignment for generation: Training diffusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representa- tion alignment for generation: Training diffusion transformers is easier than you think. arXiv:2410.06940, 2024. 2, 6, 8

    arXiv

  69. [77]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 2, 3

    2023

  70. [78]

    Structural adversar- ial objectives for self-supervised representation learning

    Xiao Zhang and Michael Maire. Structural adversar- ial objectives for self-supervised representation learning. arXiv:2310.00357, 2023. 3

    2023 arXiv

  71. [79]

    Decipher- ing’what’and’where’visual pathways from spectral clustering of layer-distributed neural representations

    Xiao Zhang, David Yunis, and Michael Maire. Decipher- ing’what’and’where’visual pathways from spectral clustering of layer-distributed neural representations. In CVPR, 2024. 3

    2024

  72. [80]

    Learning hierarchical features from generative models

    Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. arXiv:1702.08396, 2017. 3

    2017 arXiv

  73. [81]

    dz(1:T ) l pθ(z(0:T ) l |z>l) q(z(1:T ) l |z(0), x) q(z(1:T ) l |z(0), x) # ≥ Z dz(0:T ) l q(z(0:T ) l |x) (18) log

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv:2111.07832, 2021. 2 A. Appendix A.1. Further comparison L 1 2 3 4 5 1∗ GFlops 26.82 27.13 28.17 29.61 33.98 35.11 Param(M) 104 208 31...

    2021 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.