Pith. sign in

REVIEW 3 major objections 4 minor 16 references

Comparing SGLD and a fixed-noise Predictor-Corrector adaptation in canonical Joint Energy-Based Models on CIFAR-10

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fixed-noise Predictor-Corrector sampler gives canonical JEM no detectable edge over SGLD on CIFAR-10.

desk verdict Honest, well-scoped replication with a fragile central null: the training-protocol 'no advantage' result flips sign under the paper's own fixed-epoch analysis. read the letter →

arxiv 2608.05025 v2 pith:2BSY6SCO submitted 2026-08-05 cs.LG stat.ML

Dmytro Knopov This is my paper · ORCID
classification cs.LGstat.ML
keywords energy-basedmodelsJointModelSGLDPredictor-CorrectorsamplerOODdetectionCIFAR-10replicationstudyhierarchicalbootstrap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a theoretically motivated alternative to stochastic gradient Langevin dynamics (SGLD) — the Predictor-Corrector (PC) sampler from score-based generative modelling — delivers any practical gain when dropped into the canonical Joint Energy-Based Model (JEM) on CIFAR-10. The author reproduces canonical JEM on WideResNet-28-10 (92.88% test accuracy vs the canonical 92.90%) and then tests a fixed-noise PC adaptation across three protocols: full training-trajectory replacement, cold-start generation, and refinement-style OOD detection. The result is a consistent negative: no method-level advantage of PC over SGLD appears on any protocol, the training-protocol 95% confidence interval on the macro-averaged AUROC difference contains zero, and seeded cold-start generation favours SGLD by about five FID points. This is theoretically consistent with the fact that the PC predictor step degenerates under the fixed noise $\sigma=0.01$ of canonical JEM, so the annealed-noise guarantees do not transfer. A sympathetic reader would take the paper as establishing that sampler-level swaps within the SGLD family are unlikely to fix JEM's instability, and that structural changes are the more promising route.

What carries the argument

The load-bearing object is the fixed-noise PC adaptation (Algorithm 2): a deterministic gradient step replaces the degenerate annealed-noise predictor, followed by $K_c$ Langevin corrector steps with constant $\sigma=0.01$ and step size $\alpha=1.0$ inherited from canonical SGLD. The argument turns on the degeneration of the predictor step under constant noise: in the variance-exploding SDE, the predictor step is $x' \leftarrow x + (\sigma_{i+1}^2 - \sigma_i^2) s_\theta(x, \sigma_{i+1})$, so with $\sigma_i = \sigma_{i+1}$ the step vanishes and the theoretical coupling between predictor and corrector is broken. The statistical machinery that carries the comparison is the hierarchical seed-by-image bootstrap, which resamples runs at the seed level before paired-resampling images, and a seed-level Welch two one-sided tests procedure; the paper uses these to separate per-image scoring noise from the between-run variability that dominates the training comparison.

What would settle it

Run Protocol 1 with a differently configured PC variant—for instance annealed noise $\sigma_t$ or a training-time $K_c=3$—and observe a hierarchical 95% CI on the macro-5 AUROC shift whose lower bound exceeds $+0.016$; alternatively, find a paired cold-start FID difference near $-5$ in PC's favour across three seeds. Either result would overturn the paper's no-advantage claim on the training or generation protocol.

Watch

Extended reading notes

Core claim

The central claim is that, within the unchanged canonical JEM setting, a fixed-noise PC adaptation shows no detectable advantage over SGLD on any of three protocols, and on cold-start generation it is detectably worse. Across ten checkpoint–OOD pairs, refinement AUROC differences stay below 0.007; over three paired seeds, cold-start FID is $57.76 \pm 0.26$ for PC versus $52.89 \pm 0.12$ for SGLD ($\Delta = +4.87 \pm 0.31$); on the training protocol, a hierarchical seed-by-image bootstrap gives a 95% confidence interval on the macro-5 AUROC shift of $[-0.005, +0.016]$, which contains zero, while a seed-level equivalence test with two runs per method cannot establish formal equivalence. The paper also documents two failure modes of canonical JEM: catastrophic late-training divergence with the signature of the canonical outlier-buffer mechanism in all four runs, and run-dependent SVHN OOD-discrimination dynamics. The explanation offered is that the theoretical guarantees of the annealed-noise PC framework do not transfer: with fixed noise, the VE predictor step length is proportional to the difference between adjacent noise levels, which is zero, so the adaptation reduces to an SGLD-like recurrence with half the stochastic noise budget.

Load-bearing premise

The load-bearing premise is that the fixed-noise PC adaptation, configured with untuned hyperparameters inherited from canonical SGLD ($\alpha=1.0$, $\sigma=0.01$, $K_c=1$), fairly represents the PC sampler family for this comparison.

Editorial extensions

If this is right

  • Replacing SGLD with the fixed-noise PC adaptation in canonical JEM training yields no detectable method-level change in OOD discrimination: the macro-5 AUROC shift is $+0.006$ with an exploratory 95% CI $[-0.005,+0.016]$.
  • At an equal gradient budget, cold-start generation is worse with PC: FID rises by about five points across three paired seeds, and the effect is consistent in sign.
  • At inference, refinement-style OOD detection does not benefit from the PC sampler: all ten checkpoint–dataset AUROC differences stay below 0.007 AUROC, and the static energy score outperforms every refinement variant.
  • Catastrophic late-training divergence occurs in all four runs regardless of sampler, so the divergence mode is structural to the canonical configuration rather than caused by the choice between SGLD and PC.
  • Any practical improvement in stability or generation for this model class is more likely to come from structural changes—bounded deterministic samplers, diffusion-like noise schedules, or cooperative training—than from swapping samplers within the static-noise SGLD family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the predictor-degeneration argument generalises, the no-advantage result should extend to any constant-noise JEM variant, not just CIFAR-10; a cheap check is to repeat Protocol 1 on a different dataset with the same canonical settings.
  • The seed-dependent SVHN dynamics suggest that published single-seed OOD evaluations of JEM may be reporting seed artefacts; the natural extension is a multi-seed benchmark of OOD detectors in which stability across seeds is a first-class metric.
  • The untested alternative—an annealed-noise PC with a native noise schedule—would sit outside canonical JEM and inside the EBM–diffusion hybrid space, where the PC guarantees could reappear; the present result should not be read as evidence against that family.
  • A tuned PC parameterisation (e.g., larger $K_c$ at training or annealed $\sigma$) has not been ruled out by these data, so the practical takeaway for practitioners is that the specific fixed-noise adaptation is not worth adopting, not that the predictor-corrector idea is exhausted.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript reproduces canonical JEM (Grathwohl et al.) on WideResNet-28-10 without normalization on two independent runs and compares SGLD with a fixed-noise Predictor-Corrector (PC) adaptation across three protocols: full training-trajectory replacement, cold-start FID generation, and refinement-style multi-OOD AUROC. The reconstruction reaches 92.88% test accuracy versus the canonical 92.90%, with a buffer-FID of 44.46 versus 38.40. The paper documents two failure modes: catastrophic late-training divergence in all four runs and run-dependent SVHN OOD-discrimination dynamics. The main empirical claim is that no consistent method-level advantage of the PC adaptation is detected; cold-start generation favors SGLD by about 5 FID points; and on the training protocol the hierarchical seed-by-image bootstrap interval at the margin-10 checkpoints contains zero, while a seed-level TOST with two runs per method is underpowered. The paper explicitly labels exploratory intervals, reports a sign-flipping fixed-epoch analysis, and identifies the restricted PC parameterization as a limitation.

Significance. If the result holds, it is a useful negative result: sampler-level interventions within the SGLD family are unlikely to resolve canonical JEM's known instability, and the annealed-noise guarantees of the Predictor-Corrector framework do not transfer to the fixed-noise setting. The manuscript is unusually honest about its own limitations: it reports the fixed-epoch sign flip, labels the hierarchical bootstrap as exploratory, interprets the TOST as underpowered, and provides reproducible code and per-image data. The two-seed documentation of run-dependent OOD dynamics is a substantive contribution to the replication literature on JEM. The theoretical explanation in §6.1 is coherent, though post-hoc. The main weaknesses are the sensitivity of the training-protocol conclusion to checkpoint alignment and a potential deviation from the canonical training schedule in the reconstruction.

major comments (3)
  1. [§5.2, Table 3, and Abstract] The training-protocol null is tied to the margin-10 checkpoint rule, and the paper's own fixed-epoch analysis does not support the same conclusion. At epoch 90 the exploratory hierarchical 95% CI on Δmacro-5 is [−0.016,−0.002] (SGLD ahead), and at epoch 105 it is [+0.003,+0.033] (PC ahead, zero excluded); only the margin-10 alignment gives a CI containing zero. Because the margin-10 rule requires post-hoc knowledge of each run's future divergence epoch and is sensitive to the ±1–2 epoch identification error acknowledged in §6.3, the abstract's unqualified statement that a hierarchical bootstrap gives a confidence interval that contains zero is misleading. The main claim should state explicitly that the zero-containing interval depends on the margin-10 selection and that a prospective fixed-epoch comparison gives sign-flipping results.
  2. [§3, Algorithm 1] The reconstruction applies K=40 SGLD steps 'from the first epoch onward' and describes this as 'a pure canonical reconstruction without the two-phase 20→40 switch.' If the canonical JEM procedure includes the two-phase switch, then this is a deliberate departure from canonical training, and Table 1's comparison against the canonical reference values conflates sampler-configuration differences with reconstruction fidelity. The manuscript should either verify that the official repository uses K=40 from epoch 1 or re-scope the 'high-fidelity canonical reconstruction' claim and discuss the potential impact on the FID gap and divergence epochs.
  3. [§5.3 and Abstract] The cold-start FID conclusion (ΔFID = +4.87±0.31, SGLD better) is based on a single training checkpoint from one SGLD run; the dispersion covers generation seeds only and does not sample between-run variability, as the paper itself notes in §6.3. The abstract nevertheless presents 'seeded cold-start generation favours SGLD' as a method-level result. This should be qualified as 'on the single evaluated checkpoint' or supported with additional training runs, since the central claim of a detectable difference rests on one model.
minor comments (4)
  1. [§4.2] The sentence 'the canonical reading of JEM as a model that consistently assigns higher likelihoods ... turns out to be a seed-dependent property' is based on two runs; suggest rewording to 'is seed-dependent in the two runs observed here' to avoid overgeneralization.
  2. [§5.5] The statement that the seed-level percentile interval 'coincides with the full support of the nine possible cross-method pairings' is clear but deserves one additional sentence noting that this support-size argument formally precludes coverage calibration, not just in practice but by construction.
  3. [§6.3] The prospective power calculation assumes a pooled per-run standard deviation of approximately 0.008 macro-5 AUROC; please state explicitly which checkpoints and epochs this estimate is pooled over, since the fixed-epoch analysis suggests the dispersion depends on the alignment rule.
  4. [Throughout] The text contains minor encoding and formatting issues, including 'Fr´echet' in the abstract and 'T able 1' in the full text; these should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical replication against external benchmarks, with the null result read from measurements rather than derived from fitted parameters or self-citations.

full rationale

This paper is an empirical replication and comparison study, not a derivation. The central claims—no consistent method-level advantage of the fixed-noise PC adaptation over SGLD across three protocols—are read off measured quantities (test accuracy, FID, AUROC) evaluated against external canonical references from Grathwohl et al. The PC adaptation is explicitly defined as an algorithm (Algorithm 2) with fixed, untuned hyperparameters inherited from canonical SGLD; the no-advantage outcome is an experimental result, not a consequence of the definition. The theoretical discussion in §6.1 (predictor degeneracy under fixed noise) is presented post hoc as an explanation of the observed null result and is not used to construct the samplers or to select evaluation checkpoints. The margin-10 checkpoint alignment is post-hoc, but the paper explicitly acknowledges this in §6.3 and separately reports the fixed-epoch analysis, including the sign flip between epoch 90 and epoch 105; this is a methodological limitation openly documented, not a circular reduction of the conclusion to its inputs. No load-bearing self-citation occurs: the paper cites external work for JEM, SGLD, PC samplers, and OOD baselines, and the only arbitrary hand-chosen quantity (the TOST margin of ±0.01 AUROC) is not fitted to the data and is not renamed as a prediction. The paper is self-contained against external benchmarks and honestly reports the limited power of the n=2 design; there is no step in which a predicted quantity equals an input by construction or in which a fitted parameter is relabeled as a finding.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on (i) the fidelity of the canonical JEM reproduction, (ii) the contrastive-divergence approximation inherited from Grathwohl et al., (iii) the attribution of Mode 1 to the Appendix H.3 outlier-buffer mechanism, which the authors explicitly label an association rather than a demonstrated cause, and (iv) the theoretical argument that PC guarantees require an annealed noise schedule. The analysis also depends on two hand-chosen quantities: the PC corrector count K_c=1 and the TOST equivalence margin ±0.01 AUROC. No new entities are introduced.

free parameters (2)
  • PC corrector steps K_c for training = 1
    Chosen by hand for Protocol 1 to match the 40-gradient budget (K_outer=20). The training-protocol conclusion applies only to this configuration; at inference K_c values of 1, 3, and 5 are tested and the AUROC difference shrinks toward zero as K_c grows.
  • TOST equivalence margin = ±0.01 AUROC
    Hand-chosen margin for the formal equivalence test. The paper notes equivalence would only be declared at a ±0.043 margin, so the choice affects the TOST outcome, though the main non-detection claim is independent of it.
assumptions (4)
  • domain assumption Faithful canonical JEM reproduction (WideResNet-28-10, no normalization, K=40 SGLD, α=1.0, σ=0.01, buffer 10000 with 5% reinit)
    The reconstruction's validity rests on matching Grathwohl et al. [1]; accuracy within 0.02 pp supports fidelity, but the buffer-FID gap of +6.06 and earlier divergence (epochs 116-122 vs 150) indicate small differences from the canonical trajectory.
  • domain assumption Contrastive divergence with SGLD negative samples approximates the JEM maximum-likelihood gradient (Eq. 1)
    Inherited from Grathwohl et al. without re-derivation; the entire training protocol relies on this approximation.
  • domain assumption Mode 1 divergence is triggered by the replay-buffer outlier mechanism of Grathwohl et al. Appendix H.3
    The authors report a consistent cascade signature across all four runs but explicitly state that per-batch diagnostics were not retained, so the cause is an association, not a demonstrated mechanism.
  • standard math PC theoretical guarantees (Song et al. [4]) apply only under an annealed noise schedule; the VE predictor step length is proportional to σ² differences, which vanish at fixed σ
    Used in §6.1 to explain why the fixed-noise adaptation is not expected to improve on SGLD; the degeneracy follows from the form of the predictor update in [4].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparing SGLD and a fixed-noise Predictor-Corrector adaptation in canonical Joint Energy-Based Models on CIFAR-10." pith.science (2026). https://pith.science/paper/2BSY6SCO

@misc{pith2026260805025,
  author       = {Pith},
  title        = {Pith review of: Comparing SGLD and a fixed-noise Predictor-Corrector adaptation in canonical Joint Energy-Based Models on CIFAR-10},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2BSY6SCO}},
  note         = {Machine review of arXiv:2608.05025}
}
read the original abstract

Joint Energy-Based Models (JEM) unify classification and generation within a single network and support out-of-distribution (OOD) detection. Canonical JEM training relies on stochastic gradient Langevin dynamics (SGLD); a theoretically motivated alternative, the Predictor-Corrector (PC) sampler, has not previously undergone a systematic replication test on the canonical model. We reproduce canonical JEM on WideResNet-28-10 without normalisation layers on two independent runs and test a fixed-noise PC adaptation - with the degenerate annealed-noise predictor replaced by a deterministic gradient step - across three protocols: the adapted sampler replacing SGLD throughout the full training trajectories (115-132 epochs); cold-start generation (FID); and refinement-style multi-OOD detection (AUROC). The reconstruction reaches 92.88% test accuracy and buffer-FID 44.46 (canonical: 92.9% and 38.40). We document two failure modes: catastrophic late-training divergence with the signature of the canonical outlier-buffer mechanism (all four runs), and run-dependent SVHN OOD-discrimination dynamics. No consistent method-level advantage of the adaptation over SGLD is observed on any protocol: refinement AUROC differences stay below 0.007 across ten checkpoint-OOD pairs; seeded cold-start generation favours SGLD by about five FID points; on the training protocol a hierarchical seed-by-image bootstrap gives a 95% confidence interval on the macro-averaged AUROC difference that contains zero, while a seed-level equivalence test with two runs per method cannot establish formal equivalence. The training-protocol data are consistent both with equivalence and with a small directional effect. This outcome is consistent with theory: the guarantees of the annealed-noise PC framework do not transfer to the constant-noise regime of canonical JEM.

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages

  1. [1]

    Your classifier is secretly an energy based model and you should treat it like one

    Grathwohl W, Wang KC, Jacobsen JH, Duvenaud D, Norouzi M, Swersky K. Your classifier is secretly an energy based model and you should treat it like one. In: International Conference on Learning Representations; 2020. Available from: https://openreview.net/forum?id=Hkxzx0NtDB

    2020

  2. [2]

    Bayesian learning via stochastic gradient Langevin dynam- ics

    Welling M, Teh YW. Bayesian learning via stochastic gradient Langevin dynam- ics. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11); 2011. p. 681–688

    2011

  3. [3]

    Generative Modeling by Estimating Gradients of the Data Distribution

    Song Y, Ermon S. Generative Modeling by Estimating Gradients of the Data Distribution. In: Wallach H, Larochelle H, Beygelzimer A, d’Alch´ e Buc F, Fox E, Garnett R, editors. Advances in Neural Informa- tion Processing Systems. vol. 32. Curran Associates, Inc.; 2019. p. 11895– 11907. Available from: https://proceedings.neurips.cc/paper files/paper/2019/ f...

    2019

  4. [4]

    Score-Based Generative Modeling through Stochastic Differential Equations

    Song Y, Sohl-Dickstein J, Kingma DP, Kumar A, Ermon S, Poole B. Score-Based Generative Modeling through Stochastic Differential Equations. In: Interna- tional Conference on Learning Representations; 2021. Available from: https: //openreview.net/forum?id=PxTIG12RRHS

    2021

  5. [5]

    Implicit Generation and Modeling with Energy Based Models

    Du Y, Mordatch I. Implicit Generation and Modeling with Energy Based Models. In: Wallach H, Larochelle H, Beygelzimer A, d’Alch´ e Buc F, Fox E, Garnett 19 R, editors. Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc.; 2019. p. 3603–3613. Available from: https://proceedings.neurips. cc/paper files/paper/2019/file/378a063b...

    work page 2019

  6. [6]

    Classifier-Free Guidance is a Predictor-Corrector

    Bradley A, Nakkiran P. Classifier-Free Guidance is a Predictor-Corrector. Trans- actions on Machine Learning Research. 2025;https://doi.org/10.48550/arXiv. 2408.09000

    arXiv doi:10.48550/arxiv.2408.09000 2025

  7. [7]

    Towards Bridging the Performance Gaps of Joint Energy- Based Models

    Yang X, Su Q, Ji S. Towards Bridging the Performance Gaps of Joint Energy- Based Models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023. p. 15732–15741

    work page 2023

  8. [8]

    Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generation

    Jiang K, Wang H, Hao X, Yang X, Liu A, Chu Q, et al. Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2026. p. 42310–42320. arXiv:2505.19459. Available from: https://arxiv.org/abs/2505.19459

    arXiv 2026

Show all 16 references
  1. [9]

    Scalable Energy-Based Models via Adversarial Training: Unifying Discrimination and Generation

    Yin X, Zhang C, Steele J, Shavit N, Wang TT. Scalable Energy-Based Models via Adversarial Training: Unifying Discrimination and Generation. In: International Conference on Learning Representations; 2026. arXiv:2510.13872. Available from: https://openreview.net/forum?id=I9iai932rK

    2026

  2. [10]

    Improving Ad- versarial Energy-Based Model via Diffusion Process

    Geng C, Han T, Jiang PT, Zhang H, Chen J, Hauberg S, et al. Improving Ad- versarial Energy-Based Model via Diffusion Process. In: Forty-first International Conference on Machine Learning; 2024. Available from: https://openreview.net/ forum?id=QXEx16jWdN

    2024

  3. [11]

    EGC: Image Generation and Clas- sification via a Diffusion Energy-Based Model

    Guo Q, Ma C, Jiang Y, Yuan Z, Yu Y, Luo P. EGC: Image Generation and Clas- sification via a Diffusion Energy-Based Model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); 2023. p. 22952–22962

    2023

  4. [12]

    Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood

    Zhu Y, Xie J, Wu YN, Gao R. Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood. In: International Conference on Learning Repre- sentations; 2024. Available from: https://openreview.net/forum?id=Cb1jdy4XfL

    2024

  5. [13]

    Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo

    Thornton J, B´ ethune L, Zhang R, Bradley A, Nakkiran P, Zhai S. Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo. In: Proceedings of the 28th International Conference on Artificial Intelligence and Statistics. vol. 258 of Proceedings o...

    2025

  6. [14]

    Do Deep Generative Models Know What They Don’t Know? In: International Confer- ence on Learning Representations; 2019

    Nalisnick E, Matsukawa A, Teh YW, G¨ or¨ ur D, Lakshminarayanan B. Do Deep Generative Models Know What They Don’t Know? In: International Confer- ence on Learning Representations; 2019. Available from: https://openreview.net/ forum?id=H1xwNhCcYm. 20

    2019

  7. [15]

    Diffusion for Out-of-Distribution Detec- tion on Road Scenes and Beyond

    Galesso S, Schr¨ oppel P, Driss H, Brox T. Diffusion for Out-of-Distribution Detec- tion on Road Scenes and Beyond. In: Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editors. Computer Vision — ECCV 2024. Cham: Springer Nature Switzerland; 2025. p. 110–126

    2024

  8. [16]

    Energy-based Out-of-distribution Detection

    Liu W, Wang X, Owens J, Li Y. Energy-based Out-of-distribution Detection. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, editors. Advances in Neural Information Processing Systems. vol. 33. Curran Associates, Inc.; 2020. p. 21464–21475. Available from: https://proce...

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.