REVIEW 4 major objections 5 minor 102 references
Canonical Latent Representations in Conditional Diffusion Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper identifies canonical latent codes in conditional diffusion models — class-defining representations with context removed — and uses them to train robust classifiers from a tenth of the data.
desk verdict A plausible but unproven central assumption is the weak spot; the distillation study is the strong part — worth a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Jacobian of the CDM feature extractor at the inversion time step $t_e$ and layer $l$, restricted to a first-order expansion around the inverted latent $x_{t_e}$. The top-$k$ right singular vectors of this Jacobian are treated as extraneous directions: they produce the largest change in the CDM's internal features, which the paper interprets as appearance changes that leave class identity intact. CLARID removes these directions by orthogonal projection and decodes the result, adapting $k$ per sample through the explained variance ratio's elbow; CaDistill then transfers the resulting canonical features into a student using Centered Kernel Alignment plus contrastive losses and cross-entropy.
What would settle it
Take a synthetic dataset in which the class label is defined entirely by the background, with identical foreground objects across classes. If CLARID's removed directions are truly class-irrelevant, it must preserve the background and change only the foreground; if canonical samples become class-indistinguishable or a classifier trained on them loses accuracy, the assumption fails. A simpler quantitative check: train a linear probe on canonical features; if its class accuracy is not at least as high as a probe on ordinary features for held-out classes, the claim that canonical features preserve essential categorical information is contradicted.
Extended reading notes
Core claim
The paper's central claim is that canonical latent representations exist in pre-trained conditional diffusion models and can be found without extra supervision or training. Concretely, for a sample $x_0$ inverted to $x_{t_e}$, the paper takes the Jacobian $J$ of the CDM's feature extractor at a chosen layer and time step; its top-$k$ right singular vectors are declared extraneous directions because moving along them changes the decoded appearance but preserves class identity. Projecting them out, $\tilde{x}_{t_e} = (I - V_k V_k^T) x_{t_e}$, and decoding gives a canonical sample; features extracted from such samples cluster more cleanly by class, as measured by normalized mutual information. The paper further claims that a student trained with CKA-based distillation on these canonical features and contrastive alignment to canonical samples, using only 10 percent of the data as teacher signal, gains adversarial robustness and generalization while relying less on background cues.
Load-bearing premise
The whole construction assumes that the directions in which the diffusion model's internal features change most quickly are exactly the class-irrelevant directions, so deleting them from the latent leaves class identity untouched while removing context; the paper validates this by visual inspection and downstream results, not by deriving it from the model's training objective.
Editorial extensions
If this is right
- If CLAReps carry core class semantics, a conditional diffusion model can supply human-interpretable prototypes of what each class means to the model.
- Because only about 10 percent of teacher data is needed for effective distillation, the class manifold in CDM latent space must be low-dimensional and largely redundant.
- Students trained with CaDistill should depend less on spurious background cues, as measured on the Backgrounds Challenge splits.
- The same canonical representations improve white-box and black-box adversarial robustness and generalization benchmarks over vanilla and full-data diffusion distillation baselines.
- CLARID extends to text-conditioned diffusion models and different samplers and architectures, so the canonical-latent idea is not tied to a single diffusion model.
Reading between the lines
- Extension: if the Jacobian's top directions indeed encode context, the same projection could be used to detect and remove harmful spurious correlations in a trained classifier, not only to improve robustness.
- Extension: applying CLARID to datasets where labels are defined solely by background would directly test whether the removed directions are truly class-irrelevant, since the method would have to preserve the background to preserve the label.
- Extension: because text prompts already give finer control over where CLAReps lie, a natural next step is concept-level rather than class-level canonical representations, which could clarify what a diffusion model associates with a given phrase.
- Extension: the paper fixes one projection time step globally and projects once; a per-sample or multi-step removal schedule could discard more context, and measuring whether NMI and downstream robustness improve would show whether the single projection is sufficient.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CLARID, a method that extracts 'canonical latent representations' (CLAReps) from pretrained conditional diffusion models by inverting a training image to a noisy latent code, computing the Jacobian of a CDM feature map at that latent code, and projecting out the top-k right singular vectors, which the paper calls 'extraneous directions.' The number k is chosen per-sample via an explained-variance-ratio elbow rule, and the time step te is chosen by a two-stage generation-accuracy saturation criterion. The CLAReps are then decoded into Canonical Samples and their CDM features are used as Canonical Features. The paper also introduces CaDistill, a distillation framework in which a student network is trained on the full training set while the CDM teacher transfers knowledge only through CLAReps (about 10% of the training data), using contrastive losses and a CKA-based feature distillation loss. Experiments on CIFAR10 and ImageNet report improved adversarial robustness and generalization over baselines, and a toy experiment is used to argue that CLARID recovers a known low-dimensional class manifold.
Significance. If the central identification is correct, the paper offers a genuinely new way to use conditional diffusion models as compact, interpretable teachers, and the reported robustness gains would be valuable. The manuscript is well organized, with an unusually detailed appendix covering hyperparameter selection, ablations, failure cases, and generalization across samplers and architectures. The toy experiment with a known ground-truth generative process is a useful proof of concept, and the Backgrounds Challenge result is a good attempt to show that the student relies less on spurious cues. The authors are also transparent about limitations, including computational cost, occasional suboptimal te/k choices, and the fact that NMI is not valid for all comparisons. However, the paper's central claim rests on an unproven assumption about Jacobian singular directions, and part of the validation is self-referential; the downstream gains, while suggestive, do not by themselves establish that CLAReps are the mechanism. The significance is therefore conditional on additional direct evidence.
major comments (4)
- [Section 3.2, Eq. (1)-(2)] The central identification of the top-k right singular vectors of the Jacobian Jte with 'extraneous directions' is not derived and is load-bearing. The Jacobian is computed with the class condition c fixed, so its top singular directions are simply the most input-sensitive perturbations of the feature map; nothing in the diffusion training objective implies that those directions are class-irrelevant. They could equally align with class-defining texture or shape cues. Appendix D provides only qualitative image-editing examples, and Section 3.3 is a constructed toy model where the class manifold is known and aligned with a coordinate axis. Because the definition of CLARep and the interpretation of the CaDistill gains both depend on this assumption, I ask for a direct quantitative test: for example, measure a pretrained classifier's accuracy or agreement on decoded images after removing top singular directions versus after removing random directions or directions from an unconditional/unrelated-condition Jacobian, and report whether class identity is actually preserved while background changes. Without such a test, the method may be discarding discriminative content and the improvements in Table 1 could be attributable to the CaDistill losses rather than to the discovery of canonical semantics.
- [Section 3.2.1, Figure 3, Appendices G.1, G.2] The NMI metric is used both to select the hyperparameters (feature extraction layer, time step tr, total direction count n, and the adaptive-k rule) and to validate that CLARID achieves the highest NMI. This makes the reported NMI improvement partly self-referential: the method is tuned to maximize the same cluster-separability score on which it is then evaluated. Additionally, Appendix G.2 concedes that NMI is not valid for comparing CLARID with CFG, and that on CIFAR10 the raw features already achieve NMI 1.0. The paper should validate the feature-quality claim with a metric that was not used in the selection loop, or at least report a held-out class or sample split for the NMI-based selection. This is important because the 'Empirical validation' paragraph in Section 3.2.1 is one of the main supports for the claim that Canonical Features are more class-pure.
- [Table 1, Appendix J, Appendix G.2] The main quantitative claims are based on single training runs without error bars, and the CIFAR10 hyperparameters were obtained by brute-force search (Appendix G.2: 'we simply brute-force search the best hyperparameters in CLARID on CIFAR10'). Because the same benchmark table is used to report the final gains, the risk of overfitting to the evaluation suite is real. I ask the authors to report results over at least three training seeds for the main comparisons in Table 1 (or state clearly which hyperparameters were locked on a validation split), and to make explicit which hyperparameters were selected on the test benchmarks. This is standard practice for robustness claims and would materially strengthen the paper.
- [Section 4.1, Table 1, Figure 34] The claim that 10% of CLAReps are sufficient is not yet isolated from the choice of which 10% of data is used. Figure 34 ablates the number of CLAReps, but no baseline trains the same CaDistill loss on a randomly selected 10% of the original training images (or of CFG-generated samples at the same budget). CFGDistill is a helpful control, but it changes the teacher signal, not only the data selection. A random-10% control would directly test the paper's assertion that the value of CLAReps comes from their canonical semantics rather than from the distillation framework or from having fewer, less noisy teacher examples.
minor comments (5)
- [Section 3.2, Eq. (1)] The word 'Jacobian' is misspelled as 'Jabobian' in the text following Eq. (1).
- [Section 3.3] The phrase 'Reliably recovering the exact generative model is interactable' should read 'is intractable.'
- [Section 3.2 and Appendix D] The editing strengths E1, E2, E3 in Appendix D are not defined quantitatively, which makes it hard to assess how far one can move along an extraneous direction before class identity is lost.
- [Section G.2] The admission that NMI ignores low-dimensional manifold structure is important; consider moving a short version of that caveat into the main text near Figure 3, since the figure is presented as validation of feature quality.
- [Appendix J] The statement that adversarial attack runs have standard deviations smaller than 0.05 is useful, but it covers the attack evaluation, not the variance of the trained student models; the main-table missing error bars should be addressed as in Major Comment 3.
Circularity Check
Two self-referential validation loops: NMI is used both to select the CLARID configuration and to report its quality, and CIFAR10 hyperparameters are brute-force searched on the same benchmark that is later reported as evidence; the core Jacobian claim is an unproven postulate rather than a circular step.
-
fitted input called prediction
[Section 3.2.1; Appendix G.1-G.2, H.2; Figures 3, 17, 18, 23]
"NMI on ImageNet20 v.s. total number of extraneous directions n on DiT. A large n can diminish the discriminative power of the input images, whereas a small one cannot change the inputs too much. Neither case is desired. Hence, we choose n=10 for DiT in our experiment. Note that this is a self-evaluation within the CLARID framework, hence NMI is valid in this case."
The same ImageNet20 NMI metric both selects the CLARID configuration (n in Fig. 23, tr and layer in Figs. 17-18) and is then reported as the validation that 'CLARID achieves the highest NMI' (Fig. 3). Because the configuration is chosen to maximize this very metric on the same 1000-sample set, the highest-NMI claim is the selection objective restated as a result rather than an independent test. The paper itself labels this a 'self-evaluation' in G.2, confirming the metric is not a held-out prediction.
-
fitted input called prediction
[Appendix G.2 (CIFAR10 setup; Table 1)]
"Therefore, we simply brute-force search the best hyperparameters in CLARID on CIFAR10. The hyperparameters on CIFAR10 are: te = 0.8, n= 10, tr = 0.13T,layer =up.0. We also only use a 10% subset for obtaining CLAReps, as shown in Table 1."
The CIFAR10 CLARID hyperparameters are selected by brute-force search 'on CIFAR10' (no validation/holdout split is described), and the same CIFAR10 benchmark is then used for the CaDistill results in Table 1 (PGD, CW, APGD, CIFAR10-C). The reported CIFAR10 improvements are therefore fitted on the evaluation task, so they are not an unbiased prediction of the method's benefit on that dataset. This is the paper's own description; no held-out confirmation is provided.
full rationale
The paper contains two genuine fitted-input-called-prediction loops. First, NMI is used as the objective for choosing te, n, tr, and the Jacobian layer on ImageNet20, and then NMI is reported as the quantitative evidence that CLARID identifies better features; Appendix G.2 explicitly calls this a self-evaluation. Second, the CIFAR10 configuration is obtained by brute-force searching hyperparameters on CIFAR10 itself, and the same CIFAR10 evaluation is then reported in Table 1. These loops inflate the apparent support for CLARID and for the CIFAR10 half of the CaDistill results. The central assumption that top right singular vectors of the CDM feature Jacobian are class-irrelevant (Eq. 2) is a strong unproven postulate, but an unproven postulate is not circularity under the rules here; it is a correctness risk. The paper does not rely on load-bearing self-citations: references to the authors' prior work [81, 92] are background citations and do not carry the derivation. The ImageNet robustness and generalization results, the CFGDistill control, and the qualitative comparisons provide independent content, so the overall circularity is partial rather than total. Score 4 reflects two self-referential validation loops while the central claim still has externally benchmarked components.
Assumptions & free parameters
free parameters (6)
- te (inversion/projection time step) =
0.8T (DiT), T (Stable Diffusion), 0.8 (CIFAR10)
- n (total candidate extraneous directions) =
10 (DiT), 11 (Stable Diffusion), 10 (CIFAR10)
- tr (feature extraction time step) =
0.1T (DiT), 0.13T (Stable Diffusion), 0.13T (CIFAR10)
- layer index for Jacobian and feature extraction =
27 (DiT), up_blocks.1 (SD), up.0 (CIFAR10)
- CFG magnitude after projection =
3.0 (ImageNet CaDistill and CFGDistill)
- CaDistill loss weights (lambda_cs, lambda_dist, lambda_cka, lambda_cf) =
(0.4, 1.0, 0.5, 0.5) for ImageNet; (0.2, 0.25, 0.5) for CIFAR10
assumptions (4)
- domain assumption The essential semantics of each class lie on a low-dimensional manifold inside the CDM latent space.
- ad hoc to paper The top-k right singular vectors of the feature Jacobian at time te are exactly the class-irrelevant (extraneous) directions.
- domain assumption DDIM inversion of a sample produces a latent xt that preserves the semantic content of x0.
- ad hoc to paper The elbow of the explained variance ratio sequence gives the correct number k of directions to remove.
invented entities (3)
-
CLARep (Canonical Latent Representation)
-
Canonical Sample
-
Canonical Feature
Cite this review
Pith. "Pith review of Canonical Latent Representations in Conditional Diffusion Models." pith.science (2026). https://pith.science/paper/HSPOW35Z
@misc{pith2026250609955,
author = {Pith},
title = {Pith review of: Canonical Latent Representations in Conditional Diffusion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/HSPOW35Z}},
note = {Machine review of arXiv:2506.09955}
}
read the original abstract
Conditional diffusion models (CDMs) have shown impressive performance across a range of generative tasks. Their ability to model the full data distribution has opened new avenues for analysis-by-synthesis in downstream discriminative learning. However, this same modeling capacity causes CDMs to entangle the class-defining features with irrelevant context, posing challenges to extracting robust and interpretable representations. To this end, we identify Canonical LAtent Representations (CLAReps), latent codes whose internal CDM features preserve essential categorical information while discarding non-discriminative signals. When decoded, CLAReps produce representative samples for each class, offering an interpretable and compact summary of the core class semantics with minimal irrelevant details. Exploiting CLAReps, we develop a novel diffusion-based feature-distillation paradigm, CaDistill. While the student has full access to the training set, the CDM as teacher transfers core class knowledge only via CLAReps, which amounts to merely 10 % of the training data in size. After training, the student achieves strong adversarial robustness and generalization ability, focusing more on the class signals instead of spurious background cues. Our findings suggest that CDMs can serve not just as image generators but also as compact, interpretable teachers that can drive robust representation learning.
Figures
Figures from the paper (33 more)
Reference graph
Works this paper leans on
-
[1]
Square at- tack: a query-efficient black-box adversarial attack via random search
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. InEuropean Conference on Computer Vision, pages 484–501. Springer, 2020
2020
-
[2]
Synthetic data from diffusion models improves imagenet classification.Transactions on Machine Learning Research
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification.Transactions on Machine Learning Research
-
[3]
Leaving reality to imagination: Robust classification via generated datasets
Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. InICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models
2023
-
[4]
All are worth words: A vit backbone for diffusion models
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22669–22679, 2023
2023
-
[5]
Label-efficient semantic segmentation with diffusion models
Dmitry Baranchuk, Andrey V oynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. InInternational Conference on Learning Representations, 2022
2022
-
[6]
Are we done with imagenet?arXiv preprint arXiv:2006.07159, 2020
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet?arXiv preprint arXiv:2006.07159, 2020
arXiv 2006
-
[7]
Towards evaluating the robustness of neural networks
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017
2017
-
[8]
Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.ACM Transactions on Graphics (TOG), 42(4):1–10, 2023
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.ACM Transactions on Graphics (TOG), 42(4):1–10, 2023
2023
Show all 102 references
-
[9]
Exploring low- dimensional subspace in diffusion models for controllable image editing
Siyi Chen, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu. Exploring low- dimensional subspace in diffusion models for controllable image editing. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
2024
-
[10]
Deconstructing denoising diffusion models for self-supervised learning.arXiv preprint arXiv:2401.14404, 2024
Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning.arXiv preprint arXiv:2401.14404, 2024
2024 arXiv
-
[11]
Multilinear operator networks
Yixin Cheng, Grigorios Chrysos, Markos Georgopoulos, and V olkan Cevher. Multilinear operator networks. InThe Twelfth International Conference on Learning Representations, 2023
2023
-
[12]
Deep feature factorization for concept discovery
Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorization for concept discovery. InProceedings of the European Conference on Computer Vision (ECCV), pages 336–352, 2018
2018
-
[13]
Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. InInternational Conference on Machine Learning, pages 2206–2216. PMLR, 2020
2020
-
[14]
Aligning model and macaque inferior temporal cortex representations improves model-to-human behavioral alignment and adversarial robust- ness
Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Baldwin Geary, Michael Ferguson, David Daniel Cox, and James J DiCarlo. Aligning model and macaque inferior temporal cortex representations improves model-to-human behavioral alignment and adversarial robust- ness. InThe Eleve...
2023
-
[15]
ImageNet: A large- scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009
2009
-
[16]
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021. 10
2021
-
[17]
On robustness and transferability of convolutional neural networks
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. InProceedings of the IEEE/CVF Con...
2021
-
[18]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conf...
2020
-
[19]
DyTox: Trans- formers for continual learning with dynamic token expansion
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. DyTox: Trans- formers for continual learning with dynamic token expansion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285–9295, 2022
2022
-
[20]
Dreamda: Generative data augmentation with diffusion models.arXiv preprint arXiv:2403.12803, 2024
Yunxiang Fu, Chaoqi Chen, Yu Qiao, and Yizhou Yu. Dreamda: Generative data augmentation with diffusion models.arXiv preprint arXiv:2403.12803, 2024
2024 arXiv
-
[21]
Improving robustness using generated data.Advances in Neural Information Processing Systems, 34:4218–4233, 2021
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data.Advances in Neural Information Processing Systems, 34:4218–4233, 2021
2021
-
[22]
Discovering interpretable directions in the semantic latent space of diffusion models
René Haas, Inbar Huberman-Spiegelglas, Rotem Mulayoff, Stella Graßhof, Sami S Brandt, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models. In2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition ...
2024
-
[23]
Deep residual learning for im- age recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016
2016
-
[24]
Is synthetic data from generative models ready for image recognition? InThe Eleventh International Conference on Learning Representations, 2023
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? InThe Eleventh International Conference on Learning Representations, 2023
2023
-
[25]
Benchmarking neural network robustness to common corruptions and perturbations
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. InInternational Conference on Learning Representations, 2018
2018
-
[26]
Prompt-to-prompt image editing with cross-attention control
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. InThe Eleventh International Conference on Learning Representations
-
[27]
Classifier-free diffusion guidance
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. InNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications
2021
-
[28]
Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 33:6840–6851, 2020
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 33:6840–6851, 2020
2020
-
[29]
Dif- fusemix: Label-preserving data augmentation with diffusion models
Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, and Karthik Nandakumar. Dif- fusemix: Label-preserving data augmentation with diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27621–27630, 2024
2024
-
[30]
Training-free content injection using h-space in diffusion models
Jaeseok Jeong, Mingi Kwon, and Youngjung Uh. Training-free content injection using h-space in diffusion models. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5151–5161, 2024
2024
-
[31]
Elucidating the design space of diffusion-based generative models.Advances in Neural Information Processing Systems, 35: 26565–26577, 2022
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models.Advances in Neural Information Processing Systems, 35: 26565–26577, 2022
2022
-
[32]
Guiding a diffusion model with a bad version of itself.Advances in Neural Information Processing Systems, 37:52996–53021, 2024
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself.Advances in Neural Information Processing Systems, 37:52996–53021, 2024. 11
2024
-
[33]
Analyzing and improving the training dynamics of diffusion models
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24174–24184, 2024
2024
-
[34]
Supervised contrastive learning.Advances in Neural Information Processing Systems, 33:18661–18673, 2020
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning.Advances in Neural Information Processing Systems, 33:18661–18673, 2020
2020
-
[35]
Torchattacks: A pytorch repository for adversarial attacks.arXiv preprint arXiv:2010.01950, 2020
Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks.arXiv preprint arXiv:2010.01950, 2020
2010 arXiv
-
[36]
Dense text-to-image generation with attention modulation
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7701–7711, 2023
2023
-
[37]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015
2015
-
[38]
Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2014
Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2014
2014 arXiv
-
[39]
Similarity of neural network representations revisited
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. InInternational Conference on Machine Learning, pages 3519–3529. PMLR, 2019
2019
-
[40]
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009
2009
-
[41]
Diffusion models already have a semantic latent space
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. InInternational Conference on Learning Representations, 2023
2023
-
[42]
Applying guidance in a limited interval improves sample and distribution quality in diffusion models
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. InConference on Neural Information Processing Systems, 2024
2024
-
[43]
Your diffusion model is secretly a zero-shot classifier
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2206–2217, 2023
2023
-
[44]
Dreamteacher: Pretraining image backbones with deep generative models
Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Dreamteacher: Pretraining image backbones with deep generative models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 16698–16708, 2023
2023
-
[45]
Towards understanding cross and self-attention in stable diffusion for text-guided image editing
Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pttern Recognition, pages 7817–7826, 2024
2024
-
[46]
Swin Transformer: Hierarchical vision transformer using shifted windows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021
2021
-
[47]
Swin Transformer v2: Scaling up capacity and resolution
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin Transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12009–12019, 2022
2022
-
[48]
Challenging common assumptions in the unsupervised learning of disentangled representations
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. InInternational Conference on Machine Learning, pages 4114–4124. ...
2019
-
[49]
Towards deep learning models resistant to adversarial attacks
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. InInternational Conference on Learning Representations, 2018. 12
2018
-
[50]
Umap: Uniform manifold approximation and projection.Journal of Open Source Software, 3(29):861, 2018
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection.Journal of Open Source Software, 3(29):861, 2018
2018
-
[51]
Not all diffusion model activations have been evaluated as discriminative features.Advances in Neural Information Processing Systems, 37:55141–55177, 2024
Benyuan Meng, Qianqian Xu, Zitai Wang, Xiaochun Cao, and Qingming Huang. Not all diffusion model activations have been evaluated as discriminative features.Advances in Neural Information Processing Systems, 37:55141–55177, 2024
2024
-
[52]
Diffusion models beat gans on image classification.arXiv preprint arXiv:2307.08702, 2023
Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion models beat gans on image classification.arXiv preprint arXiv:2307.08702, 2023
2023 arXiv
-
[53]
Do text-free diffusion models learn discriminative visual representations? InEuropean Conference on Computer Vision, pages 253–272
Soumik Mukhopadhyay, Matthew Gwilliam, Yosuke Yamaguchi, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Tianyi Zhou, Jun Ohya, and Abhinav Shrivastava. Do text-free diffusion models learn discriminative visual representations? InEuropean Conference on Computer Visio...
2024
-
[54]
Relational knowledge distillation
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019
2019
-
[55]
Understanding the latent space of diffusion models through the lens of riemannian geometry.Advances in Neural Information Processing Systems, 36:24129–24142, 2023
Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry.Advances in Neural Information Processing Systems, 36:24129–24142, 2023
2023
-
[56]
Scalable diffusion models with transformers
William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023
2023
-
[57]
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InInternational Conference on Learning Representations, 2024
2024
-
[58]
High- resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022
2022
-
[59]
Fitnets: Hints for thin deep nets
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. 2015
2015
-
[60]
Distilling representational similarity using centered kernel alignment (cka)
Aninda Saha, Alina Bialkowski, and Sara Khalifa. Distilling representational similarity using centered kernel alignment (cka). InProceedings of the the 33rd British Machine Vision Conference (BMVC 2022). British Machine Vision Association, 2022
2022
-
[61]
Fake it till you make it: Learning transferable representations from synthetic imagenet clones
Mert Bülent Sarıyıldız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8011–8021, 2023
2023
-
[62]
Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Information Processing Systems, 35:25278–25294, 2022
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Informa...
2022
-
[63]
Robust learning meets generative models: Can proxy distributions improve adversarial robustness? InInternational Conference on Learning Representations, 2022
Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? InInternational Conference on Learning Representations, 2022
2022
-
[64]
Diffaug: A diffuse-and- denoise augmentation for training robust classifiers.Advances in Neural Information Processing Systems, 37:20745–20785, 2024
Chandramouli Shama Sastry, Sri Harsha Dumpala, and Sageev Oore. Diffaug: A diffuse-and- denoise augmentation for training robust classifiers.Advances in Neural Information Processing Systems, 37:20745–20785, 2024
2024
-
[65]
Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 13
2021
-
[66]
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021
2021
-
[67]
Latent traversals in generative models as potential flows
Yue Song, Andy Keller, Nicu Sebe, and Max Welling. Latent traversals in generative models as potential flows. InProceedings of the 40th International Conference on Machine Learning, pages 32288–32303, 2023
2023
-
[68]
Training data-efficient image transformers & distillation through attention
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021
2021
-
[69]
Deit iii: Revenge of the vit
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. InEuropean Conference on Computer Vision, pages 516–533. Springer, 2022
2022
-
[70]
Effective data augmentation with diffusion models
Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. InInternational Conference on Learning Representations, 2024
2024
-
[71]
Attention is all you need.Advances in Neural Information Processing Systems, 30, 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in Neural Information Processing Systems, 30, 2017
2017
-
[72]
Diffusion model learns low-dimensional distributions via subspace clustering
Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion model learns low-dimensional distributions via subspace clustering. InNeurIPS 2024 Workshop on Mathematics of Modern Machine Learning
2024
-
[73]
Better diffusion models further improve adversarial training
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. InInternational Conference on Machine Learning, pages 36246–36263. PMLR, 2023
2023
-
[74]
Pytorch image models
Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models, 2019
2019
-
[75]
Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine Learning, 8:229–256, 1992
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine Learning, 8:229–256, 1992
1992
-
[76]
Convnext v2: Co-designing and scaling convnets with masked autoencoders
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133–16142, 2023
2023
-
[77]
Verbs semantics and lexical selection
Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. InProceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 133–138, 1994
1994
-
[78]
Denoising diffusion autoencoders are unified self-supervised learners
Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15802–15812, 2023
2023
-
[79]
Noise or signal: The role of image backgrounds in object recognition
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. InInternational Conference on Learning Representations
-
[80]
Open-vocabulary panoptic segmentation with text-to-image diffusion models
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966, 2023
2023
-
[81]
Adanca: Neural cellular automata as adaptors for more robust vision transformer
Yitao Xu, Tong Zhang, and Sabine Süsstrunk. Adanca: Neural cellular automata as adaptors for more robust vision transformer. InThe Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024
2024
-
[82]
DER: Dynamically expandable representation for class incremental learning
Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2021. 14
2021
-
[83]
Diffusion model as representation learner
Xingyi Yang and Xinchao Wang. Diffusion model as representation learner. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 18938–18949, 2023
2023
-
[84]
Vitkd: Feature-based knowledge distillation for vision transformers
Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Feature-based knowledge distillation for vision transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1379–1388, 2024
2024
-
[85]
Distill vision transformers to cnns via low-rank representation approximation
Xufeng Yao, Yuechen ZHANG, Zuyao Chen, Jiaya Jia, and Bei Yu. Distill vision transformers to cnns via low-rank representation approximation. 2022
2022
-
[86]
Coca: Contrastive captioners are image-text foundation models.Transactions on Machine Learning Research
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.Transactions on Machine Learning Research
-
[87]
S2-mlp: Spatial-shift mlp architecture for vision
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mlp architecture for vision. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 297–306, 2022
2022
-
[88]
Cutmix: Regularization strategy to train strong classifiers with localizable features
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019
2019
-
[89]
Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. InInternational Conference on Learning Representations, 2017
2017
-
[90]
Mixup: Beyond empirical risk minimization
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. InInternational Conference on Learning Representations, 2018
2018
-
[91]
Three things we need to know about transferring stable diffusion to visual dense prediction tasks
Manyuan Zhang, Guanglu Song, Xiaoyu Shi, Yu Liu, and Hongsheng Li. Three things we need to know about transferring stable diffusion to visual dense prediction tasks. InEuropean Conference on Computer Vision, pages 128–145. Springer, 2024
2024
-
[92]
Scalable deep k-subspace clustering
Tong Zhang, Pan Ji, Mehrtash Harandi, Richard Hartley, and Ian Reid. Scalable deep k-subspace clustering. InComputer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part V 14, pages 466–481. Springer, 2019
2018
-
[93]
Unsupervised representation learning from pre- trained diffusion probabilistic models.Advances in Neural Information Processing Systems, 35: 22117–22130, 2022
Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised representation learning from pre- trained diffusion probabilistic models.Advances in Neural Information Processing Systems, 35: 22117–22130, 2022
2022
-
[95]
Unleashing text- to-image diffusion models for visual perception
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5729–5739, 2023
2023
-
[96]
Understanding the robustness in vision transformers
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. InInternational Conference on Machine Learning, pages 27378–27394. PMLR, 2022
2022
-
[97]
Golden noise for diffusion models: A learning framework.arXiv preprint arXiv:2411.09502, 2024
Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework.arXiv preprint arXiv:2411.09502, 2024
2024 arXiv
-
[98]
Rethinking centered kernel alignment in knowledge distillation
Zikai Zhou, Yunhang Shen, Shitong Shao, Linrui Gong, and Shaohui Lin. Rethinking centered kernel alignment in knowledge distillation. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 5680–5688, 2024. 15 Appendix: Table of Conte...
2024
-
[99]
converging
to improve the visual quality, to provide a more straightforward insight into the effectiveness of CLARID on EDM. Class Original CLARID Class Original CLARID EDM [31, 33] Robin Goldfish Jeep Wall Clock Figure 15: Preliminary results on Canonical Samples generated by a ImageNet...
-
[100]
The input size is 224
A ViT-Large pre-trained on ImageNet12K [15]. The input size is 224
-
[101]
The input size is 256
An ImageNet22k-pre-trained Swin V2 [47]. The input size is 256
-
[102]
The input size is 384
An ImageNet22k-pre-trained ConvNeXt V2 [76]. The input size is 384. All model weights are downloaded from the PyTorch Image Model library (timm) [74]. The accuracy curve for DiT is shown in Figure 20. The maximum time step T is 1000 for both DiT [56] and Stable Diffusion [58]....
-
[103]
pull-away
to all spatial tokens, collecting the discriminative signals inside the feature map for classification. We adopt a challenging baseline network, DeiT-III-Huge [69], which is a ViT model solely trained on ImageNet with the number of parameters (DeiT: 632.1M; DiT: 675M) and FLOP...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.