Pith. sign in

REVIEW 5 major objections 6 minor 21 references

Diffusion Classifier Guidance for Non-robust Classifiers

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper claims that classifier guidance can work with non-robust classifiers by scoring one-step denoised predictions and stabilizing guidance gradients with a deliberately biased exponential moving average.

desk verdict A simple, useful trick for using non-robust classifiers in diffusion guidance, but the headline comparison against robust guidance is confounded by a missing control. read the letter →

arxiv 2507.00687 v1 pith:SDJVZURO submitted 2025-07-01 cs.LG cs.CV

classification cs.LGcs.CV
keywords diffusionmodelsclassifierguidancenon-robustclassifiersexponentialmovingaveragegradientstabilizationclass-conditionalsamplingDDPMFID
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to remove a standard restriction on classifier guidance for diffusion models: the classifier must be trained on diffusion-noise-corrupted data (robust) for its gradients to steer sampling. It claims that any ordinary, non-robust classifier can be used instead if two changes are made: score the one-step denoised estimate $\hat{x}_0(x_t)$ rather than the noisy sample, and pass the guidance gradients through an exponential moving average with no de-biasing. On CelebA this combination reaches a class-conditional FID of 13.9 while holding an unconditional FID of 29.37, beating the robust classifier. The point is not a new diffusion model, but a drop-in sampling-side fix that widens which classifiers can be used for conditional generation.

What carries the argument

The central object is the stabilized guidance gradient $\hat{g}_t = \mathrm{EMA}_t(\nabla_{x_t}\log p_{cl}(y \mid \hat{x}_0(x_t)))$ computed from the one-step denoised estimate $\hat{x}_0(x_t) = x_t/\sqrt{\bar\alpha_t} - (\sqrt{1-\bar\alpha_t}/\sqrt{\bar\alpha_t})\epsilon_\theta(x_t,t)$, with the EMA update $\nu_t = \beta \nu_{t-1} + (1-\beta) g$ and no de-biasing. This replaces the raw classifier gradient in the DDPM reverse mean shift $x'_{t-1} = x_{t-1} + s \Sigma_t \hat{g}_t$. The denoised estimate turns the classifier input into a clean-image prediction so a noise-unaware classifier can score it; the biased EMA suppresses guidance in the early high-noise steps and enforces smooth feature changes over time.

What would settle it

Compute, for a non-robust classifier and a diffusion model, the alignment between the one-step-denoised guidance gradient $\nabla_{x_t}\log p_{cl}(y\mid\hat{x}_0(x_t))$ at large $t$ and the clean-image classifier gradient $\nabla_{x_0}\log p_{cl}(y\mid x_0)$. If those early gradients carry strong class-aligned signal, then a de-biased EMA that keeps them should improve class-conditional FID, contradicting the paper's assumption that they should be suppressed.

Watch

Extended reading notes

Core claim

The central claim is that the failure of non-robust classifier guidance is caused by unstable guidance gradients, not by the classifier being useless on clean images, and that this instability can be measured and smoothed away. The authors define sensitivity scores $S_l$ and $S_g$ that compare logit and gradient changes between adjacent steps on the same diffusion trajectory; non-robust classifiers show far higher sensitivity than robust ones. A one-step denoised prediction $\hat{x}_0(x_t)$ restores accuracy and lowers gradient sensitivity, and an exponential moving average over guidance gradients with $\beta=0.99$, deliberately zero-initialized and without de-biasing, closes the remaining gap. In the reverse process this stabilized guidance achieves the best class-conditional FID (13.9) on CelebA while keeping unconditional FID at 29.37, and the paper reports it outperforms the unmodified robust classifier. The same recipe transfers to SportBalls and to an off-the-shelf CelebA-HQ diffusion model.

Load-bearing premise

The load-bearing premise is that early high-noise reverse-sampling steps contain no useful class information, so deliberately biasing guidance toward zero there (by omitting EMA de-biasing) cannot hurt conditioning; if informative class signal appears only early, this design would weaken conditioning.

Editorial extensions

If this is right

  • A classifier trained on clean data can be plugged into classifier guidance without robust retraining; only the sampling procedure changes.
  • On CelebA, the proposed EMA-stabilized non-robust guidance reaches cFID 13.9 with FID 29.37, beating the robust-classifier baseline on class-conditional quality.
  • The method works on a third-party pre-trained DDPM (CelebA-HQ) and on the synthetic multi-class SportBalls set, so it transfers off the shelf.
  • ADAM-style second-moment rescaling amplifies gradient differences and is the wrong stabilizer; EMA with a large window ($\beta=0.99$) is the recommended choice.
  • Omitting EMA de-biasing is deliberate: it biases guidance toward zero early, preventing unreliable high-noise gradients from breaking the reverse process.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the early-step suppression is the active ingredient, a time-varying EMA schedule (small $\beta$ early, larger $\beta$ later) could adapt to per-step noise and may improve conditioning further; the paper does not test this.
  • The sensitivity metrics $S_l$ and $S_g$ double as a cheap pre-sampling diagnostic: a classifier whose gradient sensitivity stays high after $\hat{x}_0$-prediction is likely to need stronger stabilization.
  • The same stabilization recipe should transfer to other iterative samplers such as flow-matching or consistency models, since it only modifies the guidance vector, not the denoiser; that transfer is untested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a method for using non-robust classifiers (trained on clean data, without diffusion noise) for classifier guidance in DDPM sampling. The method applies the classifier to the one-step denoised estimate xhat0(xt) and stabilizes the resulting guidance gradients with a zero-initialized, de-biasing-free exponential moving average (or, alternatively, an ADAM-style update). The authors analyze classifier accuracy, logit sensitivity, and gradient sensitivity along the diffusion forward process, then evaluate conditional sampling on CelebA, SportBalls, and CelebA-HQ. They report that the non-robust classifier with xhat0 and a 0.99 EMA achieves the best class-conditional FID on CelebA (13.9) and state that this setup outperforms the unmodified robust classifier.

Significance. If the central claim holds, the paper removes a practical obstacle: off-the-shelf non-robust classifiers could steer pre-trained diffusion models without robust retraining. The paper ships code and trained weights, reports consistent qualitative trends across three datasets, and includes a useful negative result for ADAM-style gradient rescaling. The forward-process sensitivity metrics are a reasonable diagnostic tool. However, the headline comparison is confounded by an asymmetric baseline, the target-class accuracy metric is partly self-confirming, and the quantitative results lack error bars or multiple seeds, so the strongest claims are not yet established.

major comments (5)
  1. [§3.3, Table 1, Figures 5–7] The claim that the non-robust setup 'outperforms even the unmodified robust classifier' is confounded. The robust baseline is run without xhat0 and without EMA stabilization (standard guidance on xt), while the proposed EMA is zero-initialized and de-biasing-free (Section 2.5), so it acts as a time-varying ramp that suppresses guidance in early reverse steps. The comparison therefore mixes classifier robustness with the EMA ramp; a robust classifier given the same ramp might achieve equal or better cFID at comparable FID, which would invalidate the attribution. Please add a robust + xhat0 + EMA arm (and ideally a non-robust + xhat0 arm with an equivalent ramp) before making the outperformance claim.
  2. [§3.2, Figures 5–7, Table 1] Target-class accuracy is measured with the same classifier that performed the guidance. Since guidance pushes samples toward that classifier's decision boundary, high accuracy is partly self-confirming and does not establish that the samples are recognizable to an independent observer. The cFID metric, which uses Inception features, is less circular, but the paper should also report accuracy with an independently trained classifier (e.g., a different architecture or training seed) or a human evaluation to support the claim of successful conditional generation.
  3. [Algorithm 1, lines 5–6] The pseudocode as written is not the guidance rule from Eq. (1). Line 5 assigns xt−1 as a sample from N(µθ(xt), Σt(xt)), and line 6 adds s Σt(xt) ∇xt g, where g already equals ∇xt log pcl(y | xhat0(xt)). Taken literally, line 6 computes a second-order gradient (∇xt g) and the shift is applied after sampling rather than to the mean. The text and Eq. (1) indicate the intended update is x′t−1 = xt−1 + s Σt(xt) g with sampling performed after shifting the mean; the pseudocode must be corrected for reproducibility.
  4. [§2.3, Eq. (4)] The metric S_g measures smoothness of the guidance gradients along the diffusion trajectory, not their informativeness. A classifier with constant output would have S_g ≈ 0 but no class signal, so the inference 'low S_g ⇒ informative gradients' does not logically follow. The design choice to suppress early-step gradients (Section 2.5) is therefore supported only indirectly by downstream FID/cFID results. Please provide an analysis of the class-direction content of early gradients (e.g., cosine similarity to the class-conditional mean shift) or temper the claim.
  5. [§3.3, Tables 1–3, Figures 5–8] All FID, cFID, and target-class accuracy values are point estimates from single runs, and the guidance scales reported as 'best' are selected from the same curves used for the headline numbers. The claim that the proposed setup achieves the best cFID (13.9) may reflect selection noise. Please report multiple seeds (at least for the compared setups) with means and standard errors, or explicitly state the single-run nature of the evaluation in the main text.
minor comments (6)
  1. [Algorithm 1, line 5] Replace the assignment 'xt−1 = N(xt−1; µθ(xt), Σt(xt))' with an explicit sampling statement such as 'sample xt−1 ∼ N(µθ(xt), Σt(xt))'.
  2. [§3.3 and Tables 2–3] The text refers to 'figure 2' and 'figure 3' for SportBalls and CelebA-HQ sample images, but the actual sample images appear in Tables 2 and 3; please fix the cross-references.
  3. [Eq. (8)] The notation g^2 in νema_t(g^2, β = 0.999) is not defined; please state that squaring is elementwise.
  4. [§2 and §3.2] The hardware is described as 'NVIDIA A80' multiple times; the commonly used GPU is the A100, so please verify the hardware name.
  5. [Throughout] The data set name is spelled inconsistently as 'Celeba-HQ' and 'CelebA-HQ'; please unify.
  6. [Limitations] The Limitations paragraph already acknowledges hyperparameter sensitivity and the lack of FID optimization; this is good self-assessment, but the main text should also connect this to the absence of error bars in the headline comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed xhat0-plus-EMA guidance is an empirical stabilization recipe validated by ImageNet-based FID/cFID; the reported caveats are experimental-design confounds rather than circular reductions.

full rationale

The paper's derivation chain is short and self-contained. The two main ingredients are Eq. 5, the standard DDPM one-step denoising estimate xhat0(xt) = xt/sqrt(alpha_t) - sqrt(1-alpha_t)/sqrt(alpha_t) * eps_theta(xt,t), and Eq. 7, a standard exponential moving average of guidance gradients. Neither quantity is fitted to the evaluation targets, and neither is justified by a self-citation: references [2,1,20] are cited only as prior uses of the one-step denoising trick, while the analysis in Sec. 2 is supported by the paper's own trained classifiers and validation-set measurements in Figs. 1-4. No uniqueness theorem or forced-choice argument is imported from prior work by the same authors. The target-class accuracy metric in Sec. 3.2 does apply the same classifier that supplies the guidance gradients in Eq. 1, so high accuracy is partly self-confirming; however, the paper's central quality claims rest on FID and cFID, which use an ImageNet-pretrained feature extractor, i.e., an external benchmark independent of the guided classifier. The accuracy metric is therefore a measurement caveat, not a load-bearing circular step. The skeptic's missing-control observation is also a correctness risk rather than circularity: the robust baseline is run without xhat0 and without EMA, so the comparative claim 'outperforms even the unmodified robust classifier' is not cleanly attributed, but this confound does not make the proposed method's derivation equivalent to its inputs. Overall, no circularity is present in the claimed derivation chain; score 0.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The method depends on several empirical assumptions about xhat0 accuracy, the value of gradient smoothness, and the safety of suppressing early guidance, all of which are validated only indirectly through final image-quality metrics.

free parameters (3)
  • guidance scale s = s=150 (CelebA), s=15 (SportBalls), s=10 (CelebA-HQ)
    Swept over a wide range; 'best' setups are selected from the resulting accuracy/FID/cFID curves (Section 3.3, Tables 1-3).
  • EMA momentum beta = beta=0.99 (0.9 also tested)
    Chosen by hand as the stabilization window; larger beta gives more gradient smoothing in Figure 4, not derived from a principle.
  • ADAM stabilization hyperparameters = beta1=0.9, beta2=0.999, epsilon unspecified
    Used only for the ADAM-stabilized variant in Equation 8; no tuning reported.
assumptions (6)
  • standard math Standard DDPM forward noising process q(xt|x0) = N(xt; sqrt(alpha_bar_t) x0, (1 - alpha_bar_t) I) with a linear beta schedule.
    All sensitivity analysis and guidance steps rely on this process (Section 2, Eq. 2).
  • domain assumption The learned noise predictor epsilon_theta is accurate enough that the one-step denoised estimate xhat0(xt) in Eq. 5 is a valid differentiable surrogate for x0.
    xhat0 is applied to the classifier at every reverse step (Algorithm 1, line 3); poor denoising would invalidate the gradients.
  • ad hoc to paper Low gradient-sensitivity S_g is the right proxy for guidance quality.
    The paper asserts and empirically supports that unstable gradients are unsuitable for guidance (Sections 2.3-2.5), but does not derive this equivalence.
  • ad hoc to paper Zero-initialized, de-biasing-free EMA suppresses only unreliable early-step gradients, not informative ones.
    Section 2.5 deliberately omits de-biasing to bias guidance toward zero for early high-noise steps; this assumes early gradients carry no useful class signal.
  • domain assumption The same-noise adjacent-step construction in S_l and S_g faithfully represents reverse-sampling conditions.
    Metrics in Section 2.2 compare xt and xt-1 produced with identical noise; actual reverse sampling does not pair steps this way.
  • domain assumption Observations with one classifier architecture (MobileNetV3) and one diffusion setup (DDPM) transfer to other setups.
    The study uses only MobileNetV3 and standard DDPM; the limitations section notes other architectures may need different hyperparameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffusion Classifier Guidance for Non-robust Classifiers." pith.science (2026). https://pith.science/paper/SDJVZURO

@misc{pith2026250700687,
  author       = {Pith},
  title        = {Pith review of: Diffusion Classifier Guidance for Non-robust Classifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDJVZURO}},
  note         = {Machine review of arXiv:2507.00687}
}
read the original abstract

Classifier guidance is intended to steer a diffusion process such that a given classifier reliably recognizes the generated data point as a certain class. However, most classifier guidance approaches are restricted to robust classifiers, which were specifically trained on the noise of the diffusion forward process. We extend classifier guidance to work with general, non-robust, classifiers that were trained without noise. We analyze the sensitivity of both non-robust and robust classifiers to noise of the diffusion process on the standard CelebA data set, the specialized SportBalls data set and the high-dimensional real-world CelebA-HQ data set. Our findings reveal that non-robust classifiers exhibit significant accuracy degradation under noisy conditions, leading to unstable guidance gradients. To mitigate these issues, we propose a method that utilizes one-step denoised image predictions and implements stabilization techniques inspired by stochastic optimization methods, such as exponential moving averages. Experimental results demonstrate that our approach improves the stability of classifier guidance while maintaining sample diversity and visual quality. This work contributes to advancing conditional sampling techniques in generative models, enabling a broader range of classifiers to be used as guidance classifiers.

Figures

Figures reproduced from arXiv: 2507.00687 by the authors.

Figure 1
Figure 1. Classification accuracy comparison of the robust, non-robust, and non [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Logit sensitivity Sl (log scale) as defined in eq. 3 over time t for the robust, non-robust, and non-robust with xˆ (xt) 0 -prediction (eq. 5) classifiers on CelebA. The metric is reported as the average (and std) over the validation data set. 2.3 Stability of the classifier gradients Going a step further beyond logits, we can directly compute gradients just as they would be used in the sampling process of the diffu… view at source ↗
Figure 3
Figure 3. Gradient sensitivity Sg (log scale) as defined in eq. 4 over time t for the robust, non-robust, and non-robust with xˆ (xt) 0 -prediction (eq. 5) classifiers on CelebA. The metric is reported as the average (and std) over the validation set. 2.4 Informative classifier gradients through xˆ (xt) 0 -prediction To summarize, we have shown that classifying noisy data points with a classifier not trained over the same noi… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Gradient sensitivity Sg (log scale) over time t for the robust, non-robust, and non-robust with xˆ (xt) 0 -prediction (eq. 5) classifiers, as well as multiple sta￾bilization techniques (eq. 7,8). The metric is reported as the average (and std) over the CelebA validatio…
Figure 5
Figure 5. Figure 5: Accuracy comparison for conditional sampling on CelebA with various [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Unconditional FID comparison for conditional sampling on CelebA with [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Target class FID comparison for conditional sampling on CelebA with [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Accuracy, FID and cFID metrics for conditional sampling on SportBalls [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: More generations for the different data sets. All conditional samples are [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 canonical work pages

  1. [1]

    NeurIPS (2022)

    Augustin, M., Boreiko, V., Croce, F., Hein, M.: Diffusion Visual Counterfactual Explanations. NeurIPS (2022)

  2. [2]

    CVPR (2022)

    Avrahami, O., Lischinski, D., Fried, O.: Blended Diffusion for Text-driven Editing of Natural Images. CVPR (2022)

  3. [3]

    CVPR (2009)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A Large-scale Hierarchical Image Database. CVPR (2009)

  4. [4]

    NeurIPS (2021)

    Dhariwal, P., Nichol, A.: Diffusion Models Beat Gans on Image Synthesis. NeurIPS (2021)

  5. [5]

    ICML (2023)

    Dinh, A.D., Liu, D., Xu, C.: Pixelasparam: A Gradient View on Diffusion Sampling with Guidance. ICML (2023)

  6. [6]

    NeurIPS (2024)

    Dinh, A.D., Liu, D., Xu, C.: Rethinking Conditional Diffusion Sampling with Pro- gressive Guidance. NeurIPS (2024)

  7. [7]

    NeurIPS (2024)

    Gruver, N., Stanton, S., Frey, N., Rudner, T.G., Hotzel, I., Lafrance-Vanasse, J., Rajpal, A., Cho, K., Wilson, A.G.: Protein Design with Guided Discrete Diffusion. NeurIPS (2024)

  8. [8]

    NeurIPS (2017)

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans Trained by a two Time-scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS (2017)

Show all 21 references
  1. [9]

    NeurIPS (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. NeurIPS (2020)

  2. [10]

    NeurIPS Workshop on Deep Generative Models and Downstream Applications (2021)

    Ho, J., Salimans, T.: Classifier-free Diffusion Guidance. NeurIPS Workshop on Deep Generative Models and Downstream Applications (2021)

  3. [11]

    ICCV (2019)

    Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for Mobilenetv3. ICCV (2019)

  4. [12]

    ICLR (2018)

    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive Growing of Gans for Im- proved Quality, Stability, and Variation. ICLR (2018)

  5. [13]

    ICLR (2015)

    Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. ICLR (2015)

  6. [14]

    ICCV (2015)

    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep Learning Face Attributes in the Wild. ICCV (2015)

  7. [15]

    https: //github.com/huggingface/diffusers (2022)

    von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., Wolf, T.: Diffusers: State-of-the-art Diffusion Models. https: //github.com/huggingface/diffusers (2022)

  8. [16]

    MICCAI (2015)

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional Networks for Biomed- ical Image Segmentation. MICCAI (2015)

  9. [17]

    Nature (1986)

    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Representations by Back- propagating Errors. Nature (1986)

  10. [18]

    ICML (2015)

    Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep Unsuper- vised Learning using Nonequilibrium Thermodynamics. ICML (2015)

  11. [19]

    arXiv:2410.20890 (2024)

    Vaeth, P., Fruehwald, A.M., Paassen, B., Gregorova, M.: Generative Example- based Explanations: Bridging the Gap Between Generative Modeling and Explain- ability. arXiv:2410.20890 (2024)

  12. [20]

    arXiv:2406.17399 (2024)

    Vaeth, P., Fruehwald, A.M., Paassen, B., Gregorova, M.: Gradcheck: An- alyzing Classifier Guidance Gradients for Conditional Diffusion Sampling. arXiv:2406.17399 (2024)

  13. [21]

    Weiss, T., Mayo Yanes, E., Chakraborty, S., Cosmo, L., Bronstein, A.M., Gershoni- Poranne,R.:GuidedDiffusionforInverseMolecularDesign.NatureComputational Science (2023) 18 Vaeth et al. A Supplementary material (compact version) (a) Unconditional CelebA samples (b) Conditional ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.