REVIEW 5 major objections 6 minor 21 references
Diffusion Classifier Guidance for Non-robust Classifiers
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that classifier guidance can work with non-robust classifiers by scoring one-step denoised predictions and stabilizing guidance gradients with a deliberately biased exponential moving average.
desk verdict A simple, useful trick for using non-robust classifiers in diffusion guidance, but the headline comparison against robust guidance is confounded by a missing control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the stabilized guidance gradient $\hat{g}_t = \mathrm{EMA}_t(\nabla_{x_t}\log p_{cl}(y \mid \hat{x}_0(x_t)))$ computed from the one-step denoised estimate $\hat{x}_0(x_t) = x_t/\sqrt{\bar\alpha_t} - (\sqrt{1-\bar\alpha_t}/\sqrt{\bar\alpha_t})\epsilon_\theta(x_t,t)$, with the EMA update $\nu_t = \beta \nu_{t-1} + (1-\beta) g$ and no de-biasing. This replaces the raw classifier gradient in the DDPM reverse mean shift $x'_{t-1} = x_{t-1} + s \Sigma_t \hat{g}_t$. The denoised estimate turns the classifier input into a clean-image prediction so a noise-unaware classifier can score it; the biased EMA suppresses guidance in the early high-noise steps and enforces smooth feature changes over time.
What would settle it
Compute, for a non-robust classifier and a diffusion model, the alignment between the one-step-denoised guidance gradient $\nabla_{x_t}\log p_{cl}(y\mid\hat{x}_0(x_t))$ at large $t$ and the clean-image classifier gradient $\nabla_{x_0}\log p_{cl}(y\mid x_0)$. If those early gradients carry strong class-aligned signal, then a de-biased EMA that keeps them should improve class-conditional FID, contradicting the paper's assumption that they should be suppressed.
Extended reading notes
Core claim
The central claim is that the failure of non-robust classifier guidance is caused by unstable guidance gradients, not by the classifier being useless on clean images, and that this instability can be measured and smoothed away. The authors define sensitivity scores $S_l$ and $S_g$ that compare logit and gradient changes between adjacent steps on the same diffusion trajectory; non-robust classifiers show far higher sensitivity than robust ones. A one-step denoised prediction $\hat{x}_0(x_t)$ restores accuracy and lowers gradient sensitivity, and an exponential moving average over guidance gradients with $\beta=0.99$, deliberately zero-initialized and without de-biasing, closes the remaining gap. In the reverse process this stabilized guidance achieves the best class-conditional FID (13.9) on CelebA while keeping unconditional FID at 29.37, and the paper reports it outperforms the unmodified robust classifier. The same recipe transfers to SportBalls and to an off-the-shelf CelebA-HQ diffusion model.
Load-bearing premise
The load-bearing premise is that early high-noise reverse-sampling steps contain no useful class information, so deliberately biasing guidance toward zero there (by omitting EMA de-biasing) cannot hurt conditioning; if informative class signal appears only early, this design would weaken conditioning.
Editorial extensions
If this is right
- A classifier trained on clean data can be plugged into classifier guidance without robust retraining; only the sampling procedure changes.
- On CelebA, the proposed EMA-stabilized non-robust guidance reaches cFID 13.9 with FID 29.37, beating the robust-classifier baseline on class-conditional quality.
- The method works on a third-party pre-trained DDPM (CelebA-HQ) and on the synthetic multi-class SportBalls set, so it transfers off the shelf.
- ADAM-style second-moment rescaling amplifies gradient differences and is the wrong stabilizer; EMA with a large window ($\beta=0.99$) is the recommended choice.
- Omitting EMA de-biasing is deliberate: it biases guidance toward zero early, preventing unreliable high-noise gradients from breaking the reverse process.
Reading between the lines
- If the early-step suppression is the active ingredient, a time-varying EMA schedule (small $\beta$ early, larger $\beta$ later) could adapt to per-step noise and may improve conditioning further; the paper does not test this.
- The sensitivity metrics $S_l$ and $S_g$ double as a cheap pre-sampling diagnostic: a classifier whose gradient sensitivity stays high after $\hat{x}_0$-prediction is likely to need stronger stabilization.
- The same stabilization recipe should transfer to other iterative samplers such as flow-matching or consistency models, since it only modifies the guidance vector, not the denoiser; that transfer is untested in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a method for using non-robust classifiers (trained on clean data, without diffusion noise) for classifier guidance in DDPM sampling. The method applies the classifier to the one-step denoised estimate xhat0(xt) and stabilizes the resulting guidance gradients with a zero-initialized, de-biasing-free exponential moving average (or, alternatively, an ADAM-style update). The authors analyze classifier accuracy, logit sensitivity, and gradient sensitivity along the diffusion forward process, then evaluate conditional sampling on CelebA, SportBalls, and CelebA-HQ. They report that the non-robust classifier with xhat0 and a 0.99 EMA achieves the best class-conditional FID on CelebA (13.9) and state that this setup outperforms the unmodified robust classifier.
Significance. If the central claim holds, the paper removes a practical obstacle: off-the-shelf non-robust classifiers could steer pre-trained diffusion models without robust retraining. The paper ships code and trained weights, reports consistent qualitative trends across three datasets, and includes a useful negative result for ADAM-style gradient rescaling. The forward-process sensitivity metrics are a reasonable diagnostic tool. However, the headline comparison is confounded by an asymmetric baseline, the target-class accuracy metric is partly self-confirming, and the quantitative results lack error bars or multiple seeds, so the strongest claims are not yet established.
major comments (5)
- [§3.3, Table 1, Figures 5–7] The claim that the non-robust setup 'outperforms even the unmodified robust classifier' is confounded. The robust baseline is run without xhat0 and without EMA stabilization (standard guidance on xt), while the proposed EMA is zero-initialized and de-biasing-free (Section 2.5), so it acts as a time-varying ramp that suppresses guidance in early reverse steps. The comparison therefore mixes classifier robustness with the EMA ramp; a robust classifier given the same ramp might achieve equal or better cFID at comparable FID, which would invalidate the attribution. Please add a robust + xhat0 + EMA arm (and ideally a non-robust + xhat0 arm with an equivalent ramp) before making the outperformance claim.
- [§3.2, Figures 5–7, Table 1] Target-class accuracy is measured with the same classifier that performed the guidance. Since guidance pushes samples toward that classifier's decision boundary, high accuracy is partly self-confirming and does not establish that the samples are recognizable to an independent observer. The cFID metric, which uses Inception features, is less circular, but the paper should also report accuracy with an independently trained classifier (e.g., a different architecture or training seed) or a human evaluation to support the claim of successful conditional generation.
- [Algorithm 1, lines 5–6] The pseudocode as written is not the guidance rule from Eq. (1). Line 5 assigns xt−1 as a sample from N(µθ(xt), Σt(xt)), and line 6 adds s Σt(xt) ∇xt g, where g already equals ∇xt log pcl(y | xhat0(xt)). Taken literally, line 6 computes a second-order gradient (∇xt g) and the shift is applied after sampling rather than to the mean. The text and Eq. (1) indicate the intended update is x′t−1 = xt−1 + s Σt(xt) g with sampling performed after shifting the mean; the pseudocode must be corrected for reproducibility.
- [§2.3, Eq. (4)] The metric S_g measures smoothness of the guidance gradients along the diffusion trajectory, not their informativeness. A classifier with constant output would have S_g ≈ 0 but no class signal, so the inference 'low S_g ⇒ informative gradients' does not logically follow. The design choice to suppress early-step gradients (Section 2.5) is therefore supported only indirectly by downstream FID/cFID results. Please provide an analysis of the class-direction content of early gradients (e.g., cosine similarity to the class-conditional mean shift) or temper the claim.
- [§3.3, Tables 1–3, Figures 5–8] All FID, cFID, and target-class accuracy values are point estimates from single runs, and the guidance scales reported as 'best' are selected from the same curves used for the headline numbers. The claim that the proposed setup achieves the best cFID (13.9) may reflect selection noise. Please report multiple seeds (at least for the compared setups) with means and standard errors, or explicitly state the single-run nature of the evaluation in the main text.
minor comments (6)
- [Algorithm 1, line 5] Replace the assignment 'xt−1 = N(xt−1; µθ(xt), Σt(xt))' with an explicit sampling statement such as 'sample xt−1 ∼ N(µθ(xt), Σt(xt))'.
- [§3.3 and Tables 2–3] The text refers to 'figure 2' and 'figure 3' for SportBalls and CelebA-HQ sample images, but the actual sample images appear in Tables 2 and 3; please fix the cross-references.
- [Eq. (8)] The notation g^2 in νema_t(g^2, β = 0.999) is not defined; please state that squaring is elementwise.
- [§2 and §3.2] The hardware is described as 'NVIDIA A80' multiple times; the commonly used GPU is the A100, so please verify the hardware name.
- [Throughout] The data set name is spelled inconsistently as 'Celeba-HQ' and 'CelebA-HQ'; please unify.
- [Limitations] The Limitations paragraph already acknowledges hyperparameter sensitivity and the lack of FID optimization; this is good self-assessment, but the main text should also connect this to the absence of error bars in the headline comparisons.
Circularity Check
No significant circularity: the proposed xhat0-plus-EMA guidance is an empirical stabilization recipe validated by ImageNet-based FID/cFID; the reported caveats are experimental-design confounds rather than circular reductions.
full rationale
The paper's derivation chain is short and self-contained. The two main ingredients are Eq. 5, the standard DDPM one-step denoising estimate xhat0(xt) = xt/sqrt(alpha_t) - sqrt(1-alpha_t)/sqrt(alpha_t) * eps_theta(xt,t), and Eq. 7, a standard exponential moving average of guidance gradients. Neither quantity is fitted to the evaluation targets, and neither is justified by a self-citation: references [2,1,20] are cited only as prior uses of the one-step denoising trick, while the analysis in Sec. 2 is supported by the paper's own trained classifiers and validation-set measurements in Figs. 1-4. No uniqueness theorem or forced-choice argument is imported from prior work by the same authors. The target-class accuracy metric in Sec. 3.2 does apply the same classifier that supplies the guidance gradients in Eq. 1, so high accuracy is partly self-confirming; however, the paper's central quality claims rest on FID and cFID, which use an ImageNet-pretrained feature extractor, i.e., an external benchmark independent of the guided classifier. The accuracy metric is therefore a measurement caveat, not a load-bearing circular step. The skeptic's missing-control observation is also a correctness risk rather than circularity: the robust baseline is run without xhat0 and without EMA, so the comparative claim 'outperforms even the unmodified robust classifier' is not cleanly attributed, but this confound does not make the proposed method's derivation equivalent to its inputs. Overall, no circularity is present in the claimed derivation chain; score 0.
Assumptions & free parameters
free parameters (3)
- guidance scale s =
s=150 (CelebA), s=15 (SportBalls), s=10 (CelebA-HQ)
- EMA momentum beta =
beta=0.99 (0.9 also tested)
- ADAM stabilization hyperparameters =
beta1=0.9, beta2=0.999, epsilon unspecified
assumptions (6)
- standard math Standard DDPM forward noising process q(xt|x0) = N(xt; sqrt(alpha_bar_t) x0, (1 - alpha_bar_t) I) with a linear beta schedule.
- domain assumption The learned noise predictor epsilon_theta is accurate enough that the one-step denoised estimate xhat0(xt) in Eq. 5 is a valid differentiable surrogate for x0.
- ad hoc to paper Low gradient-sensitivity S_g is the right proxy for guidance quality.
- ad hoc to paper Zero-initialized, de-biasing-free EMA suppresses only unreliable early-step gradients, not informative ones.
- domain assumption The same-noise adjacent-step construction in S_l and S_g faithfully represents reverse-sampling conditions.
- domain assumption Observations with one classifier architecture (MobileNetV3) and one diffusion setup (DDPM) transfer to other setups.
Cite this review
Pith. "Pith review of Diffusion Classifier Guidance for Non-robust Classifiers." pith.science (2026). https://pith.science/paper/SDJVZURO
@misc{pith2026250700687,
author = {Pith},
title = {Pith review of: Diffusion Classifier Guidance for Non-robust Classifiers},
year = {2026},
howpublished = {\url{https://pith.science/paper/SDJVZURO}},
note = {Machine review of arXiv:2507.00687}
}
read the original abstract
Classifier guidance is intended to steer a diffusion process such that a given classifier reliably recognizes the generated data point as a certain class. However, most classifier guidance approaches are restricted to robust classifiers, which were specifically trained on the noise of the diffusion forward process. We extend classifier guidance to work with general, non-robust, classifiers that were trained without noise. We analyze the sensitivity of both non-robust and robust classifiers to noise of the diffusion process on the standard CelebA data set, the specialized SportBalls data set and the high-dimensional real-world CelebA-HQ data set. Our findings reveal that non-robust classifiers exhibit significant accuracy degradation under noisy conditions, leading to unstable guidance gradients. To mitigate these issues, we propose a method that utilizes one-step denoised image predictions and implements stabilization techniques inspired by stochastic optimization methods, such as exponential moving averages. Experimental results demonstrate that our approach improves the stability of classifier guidance while maintaining sample diversity and visual quality. This work contributes to advancing conditional sampling techniques in generative models, enabling a broader range of classifiers to be used as guidance classifiers.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Augustin, M., Boreiko, V., Croce, F., Hein, M.: Diffusion Visual Counterfactual Explanations. NeurIPS (2022)
work page 2022
-
[2]
Avrahami, O., Lischinski, D., Fried, O.: Blended Diffusion for Text-driven Editing of Natural Images. CVPR (2022)
work page 2022
-
[3]
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A Large-scale Hierarchical Image Database. CVPR (2009)
work page 2009
-
[4]
Dhariwal, P., Nichol, A.: Diffusion Models Beat Gans on Image Synthesis. NeurIPS (2021)
work page 2021
-
[5]
Dinh, A.D., Liu, D., Xu, C.: Pixelasparam: A Gradient View on Diffusion Sampling with Guidance. ICML (2023)
work page 2023
-
[6]
Dinh, A.D., Liu, D., Xu, C.: Rethinking Conditional Diffusion Sampling with Pro- gressive Guidance. NeurIPS (2024)
work page 2024
-
[7]
Gruver, N., Stanton, S., Frey, N., Rudner, T.G., Hotzel, I., Lafrance-Vanasse, J., Rajpal, A., Cho, K., Wilson, A.G.: Protein Design with Guided Discrete Diffusion. NeurIPS (2024)
work page 2024
-
[8]
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans Trained by a two Time-scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS (2017)
work page 2017
Show all 21 references
-
[9]
NeurIPS (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. NeurIPS (2020)
2020
-
[10]
NeurIPS Workshop on Deep Generative Models and Downstream Applications (2021)
Ho, J., Salimans, T.: Classifier-free Diffusion Guidance. NeurIPS Workshop on Deep Generative Models and Downstream Applications (2021)
2021
-
[11]
ICCV (2019)
Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for Mobilenetv3. ICCV (2019)
2019
-
[12]
ICLR (2018)
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive Growing of Gans for Im- proved Quality, Stability, and Variation. ICLR (2018)
2018
-
[13]
ICLR (2015)
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. ICLR (2015)
2015
-
[14]
ICCV (2015)
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep Learning Face Attributes in the Wild. ICCV (2015)
2015
-
[15]
https: //github.com/huggingface/diffusers (2022)
von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., Wolf, T.: Diffusers: State-of-the-art Diffusion Models. https: //github.com/huggingface/diffusers (2022)
2022
-
[16]
MICCAI (2015)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional Networks for Biomed- ical Image Segmentation. MICCAI (2015)
2015
-
[17]
Nature (1986)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Representations by Back- propagating Errors. Nature (1986)
1986
-
[18]
ICML (2015)
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep Unsuper- vised Learning using Nonequilibrium Thermodynamics. ICML (2015)
2015
-
[19]
arXiv:2410.20890 (2024)
Vaeth, P., Fruehwald, A.M., Paassen, B., Gregorova, M.: Generative Example- based Explanations: Bridging the Gap Between Generative Modeling and Explain- ability. arXiv:2410.20890 (2024)
2024 arXiv
-
[20]
arXiv:2406.17399 (2024)
Vaeth, P., Fruehwald, A.M., Paassen, B., Gregorova, M.: Gradcheck: An- alyzing Classifier Guidance Gradients for Conditional Diffusion Sampling. arXiv:2406.17399 (2024)
2024 arXiv
-
[21]
Weiss, T., Mayo Yanes, E., Chakraborty, S., Cosmo, L., Bronstein, A.M., Gershoni- Poranne,R.:GuidedDiffusionforInverseMolecularDesign.NatureComputational Science (2023) 18 Vaeth et al. A Supplementary material (compact version) (a) Unconditional CelebA samples (b) Conditional ...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.