Pith. sign in

REVIEW 5 major objections 6 minor 89 references

Exploring bidirectional bounds for minimax-training of Energy-based models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training energy-based models by sandwiching the negative log-likelihood between lower and upper bounds prevents the collapse of standard minimax EBM training and yields one-step sample generation competitive with diffusion models.

desk verdict Solid empirical extension with honest limitations, but the implemented 'bounds' are not actual bounds, so the central theoretical claim needs a rewrite before the paper can be taken as a bounded training method. read the letter →

arxiv 2506.04609 v1 pith:PCXIQQCA submitted 2025-06-05 cs.LG cs.CV

Cong Geng , Jia Wang , Li Chen , Zhiyong Gao , Jes Frellsen , S{o}ren Hauberg This is my paper
classification cs.LGcs.CV
keywords Energy-basedmodelsBidirectionalboundsMinimaxtrainingDensityestimationSingular-valuelowerboundGradientpenaltyDiffusion-basedupperOne-stepgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make energy-based models (EBMs), which specify an unnormalized density through an energy function, trainable without the divergence that plagues standard minimax training. The proposal is to bound the negative log-likelihood from both sides, maximizing a lower bound over a variational generator and minimizing an upper bound over the energy, a "sandwich" that keeps the objective from drifting to negative infinity. The paper derives two lower bounds (from singular values of the generator Jacobian and from mutual information) and two upper bounds (from a gradient penalty and from a diffusion process), with practical algorithms for each. It reports that the sandwich stabilizes training, gives useful density estimates and sample quality, and that a UNet-scaled version reaches FID 5.47 on CIFAR-10 and 6.57 on ImageNet with one-step generation. The authors are explicit that some of the bounds are only approximately valid: the diffusion bound needs the energy to satisfy the Fokker-Planck equation, and the singular-value bound is computed with premature stopping.

What carries the argument

The central object is the sandwich of the negative data log-likelihood: a lower bound from the variational/Jensen bound with a generator entropy estimate, and an upper bound equal to the lower bound plus an upper bound on the KL divergence between the generator and the energy distribution. The lower bound is realized either by a singular-value estimate of the generator Jacobian, with the smallest singular value found by an iterative eigensolver, or by a mutual-information estimator based on the Jensen-Shannon divergence. The upper bound is realized either by a gradient-penalty term or by a score-based diffusion KL bound derived from a variance-exploding stochastic differential equation. Alternating optimization, maximizing the lower bound over the generator and minimizing the upper bound over the energy, is what prevents the energy from diverging, and the paper describes efficient evaluation via Jacobian-vector products, stochastic trace estimators, and Monte Carlo sampling over diffusion time.

What would settle it

On a trained model, compute the exact negative log-likelihood (for example with a normalizing-flow energy) and evaluate all four proposed bound expressions; if any lower bound exceeds the true value or any upper bound falls below it, the sandwich is not a valid bound. A quicker check targets the weakest link: for a neural Eθ(x,t) that does not satisfy the Fokker-Planck equation, compare the right-hand side of Eq. (27) with the true KL(pg∥pθ) estimated by Monte Carlo; a violation would confirm the authors' stated suspicion that the diffusion bound can fail.

Watch

Extended reading notes

Core claim

The central claim is that minimax training of EBMs fails because it optimizes a single variational lower bound; when the bound is loose, minimizing it drives the energy of training data to negative infinity. Training instead with a lower bound and an upper bound sandwiched around the negative log-likelihood, maximizing the lower bound over the generator and minimizing the upper bound over the energy, avoids this collapse. The paper derives four bounds, provides evaluation algorithms for each, and shows empirically that combinations such as EBMSV+GP and EBMMI+diff give stable training curves and competitive generation, while the diffusion upper bound with a large UNet gives FID 5.47 on CIFAR-10 and 6.57 on ImageNet using one-step generation, comparable to diffusion models that require far more score evaluations. The paper also claims that this is the first demonstration that gradient penalties provide a bound on the log-likelihood, giving a probabilistic justification for a common GAN stabilization heuristic.

Load-bearing premise

The four quantities are genuine upper and lower bounds on the negative log-likelihood, and in particular the diffusion bound requires the time-dependent energy to satisfy the Fokker-Planck equation while the singular-value bound is computed with premature stopping that technically breaks the bound.

Editorial extensions

If this is right

  • Standard minimax EBM training based only on the variational lower bound diverges in practice; the bidirectional sandwich keeps training curves stable.
  • Gradient penalties, already used heuristically in WGANs, are justified as an upper bound on the negative log-likelihood, so a common regularizer gains a maximum-likelihood interpretation.
  • With a UNet-scale network, EBMMI+diff achieves FID 5.47 on CIFAR-10 and 6.57 on ImageNet with one-step generation, approaching diffusion models that use a thousand or more sampling steps.
  • The singular-value lower bound is memory-efficient because it needs no extra network, while the mutual-information lower bound is time-efficient, so users can choose a pairing based on whether memory or speed matters more.
  • Bidirectional training increases generator entropy and capacity usage compared to a zero-centered gradient penalty, as measured by an anisotropy index.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sandwich bounds are valid, the gap between the upper and lower bound becomes a computable training diagnostic: monitoring the gap could reveal when the proposal generator is far from the energy distribution and when the energy is overfitting.
  • The diffusion upper bound might be made a true bound by parameterizing the time-dependent energy to satisfy the Fokker-Planck equation, for instance through a score network, which would remove the paper's main theoretical weakness; the empirical results already suggest the approximation is benign in practice.
  • Because the method needs only one generator forward pass at test time, a successful bidirectional EBM offers a cheap alternative to diffusion sampling, and one could test whether the same sandwich idea transfers to other divergences or to conditional generation.
  • The ImageNet result suggests that the perceived superiority of diffusion models on image generation may be due largely to UNet architectures and tuning rather than to the likelihood objective itself; this is testable by scaling bidirectional EBMs further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes training energy-based models (EBMs) by 'bidirectional bounds': maximize a lower bound and minimize an upper bound of the negative data log-likelihood L(θ). Four bounds are introduced: a singular-value-based lower bound, a mutual-information-based lower bound, a gradient-penalty upper bound, and a diffusion-based upper bound. The authors combine these bounds into four training schemes (SV+GP, SV+diff, MI+GP, MI+diff) and compare them against GANs, other EBMs, and diffusion models on toy data, MNIST, CIFAR-10, AnimeFace, CelebA, ImageNet, and LSUN Church. The reported empirical results are competitive, including FID 5.47 on CIFAR-10 and 6.57 on ImageNet with one-step generation using a scaled-up MI+diff model.

Significance. If the four expressions were genuine bounds on the negative log-likelihood, the bidirectional-bounds idea would be a valuable conceptual bridge between minimax EBM training and maximum likelihood, with a clean explanation for the stabilizing effect of gradient penalties. The paper also deserves credit for an unusually explicit limitations section, for comparing many bound combinations, and for including large-scale experiments with competitive FID scores. However, the central theoretical claim is not supported as stated: the manuscript itself concedes that none of the four implemented losses is a guaranteed bound during training. The Fokker-Planck condition for the diffusion bound is not satisfied, the LOPCG solver is stopped prematurely, the constants M and m of Theorem 1 are never evaluated, and the mutual-information estimator bounds a Jensen-Shannon divergence rather than a KL mutual information. The empirical contribution remains interesting as an evaluation of regularizers, but the paper currently overclaims by presenting these as bounds on the log-likelihood.

major comments (5)
  1. [Section 2, Eq. (8)] The central sandwich condition ⌊L(θ)⌋ ≤ L(θ) ≤ ⌈L(θ)⌉ is not maintained by any of the four implemented losses. The authors' own Limitations section (§5) states that the diffusion bound requires a Fokker-Planck condition that is not ensured for neural energy functions, that premature stopping of the LOPCG solver 'technically violates the bound', that M/s1^2 is treated as a tunable parameter, and that the entropy weight λ is reduced below its theoretical value in large-scale runs. Furthermore, Eq. (18) bounds a Jensen-Shannon divergence, not the KL-based mutual information used in Eq. (17). Consequently, the optimizers are not maximizing and minimizing valid bounds on the negative log-likelihood. The abstract's claim that the method 'uses bidirectional bounds' is therefore not supported. The authors should either implement bounds that provably satisfy Eq. (8) during training, or explicitly reframe the method as bidirectional regularizers and remove the bound claims.
  2. [Section 2.3, Theorem 1 and Eq. (23)] Theorem 1 is only stated, with its proof deferred to the conference paper [17]; a journal submission should include the proof or at least a complete statement of assumptions with verification in the application. More importantly, the theorem assumes g has finite support and f is L-Lipschitz, but the application uses g = pg, which has infinite support because of the Gaussian noise in Eq. (9), and f = exp(-Eθ)/pg, whose Lipschitz regularity is not established. The constants M and m are never computed, and the passage from Eq. (21) to Eq. (22) requires a condition that is only asserted to hold 'throughout most of training'. The hinge modification in Eq. (33) and the scheduled M/s1^2 values in Appendix B are further departures from the stated bound. Thus Eq. (23) is not a guaranteed upper bound on KL(pg∥pθ) as implemented.
  3. [Section 2.4, Eqs. (27)–(30)] The diffusion upper bound is not valid as stated. Eq. (28) relies on pθt(x) satisfying the Fokker-Planck equation (29), which the authors explicitly say is not guaranteed when Eθ(x,t) is a neural network with Eθ(x,t) = Eθ(x)/σ(t). The term KL(pgT∥pθT) is discarded in Eq. (30) and the constant C is uncomputed, so the expression is not a computable upper bound. The tightness discussion when pg = pθ does not repair the missing guarantee during training, and the authors' own 'Limitations' paragraph acknowledges that 'this part remains heuristic'.
  4. [Section 2.1, Eq. (14) and §5] The singular-value lower bound is not guaranteed during training. A prematurely stopped LOPCG iteration yields an overestimate of s1; the authors note in Fig. 6 that the low-precision estimator upper-bounds the high-precision one, so d log s1_est can exceed the true entropy lower bound, breaking Eq. (14). Setting the inner-loop iterations to zero in the large-scale ImageNet experiments (Appendix D.2) replaces the bound with a Hutchinson estimator that the authors elsewhere state is an upper bound on entropy. The use of λ < 1 in Table 8 also departs from the theoretical value λ = 1. These are not minor implementation details; they invalidate the claimed lower bound.
  5. [Section 2.2, Eqs. (17)–(18)] The mutual-information lower bound uses IJSD, a variational lower bound on the Jensen-Shannon divergence, not on the KL mutual information I[x,z] used in Eq. (17). Since H[x] = I[x,z] + constant only when I is the KL mutual information, Eq. (18) does not provide a guaranteed entropy lower bound. The text acknowledges that the JSD choice is based on empirical observations by Kumar et al., but the surrounding presentation still presents it as a bound. At minimum, the paper should consistently label this objective as a heuristic estimator rather than a bound.
minor comments (6)
  1. [Abstract] There is a stray comma in 'we compare the different bounds to investigate, the pros and cons'; it should read 'to investigate the pros and cons'.
  2. [References] Reference [66] is incomplete: 'International conference on learning representations. 2021.' lacks a title; the entry should be completed.
  3. [Table 1] The row 'EBMSV+GP (conv)' reports '- -' without explanation; the authors should state why convolutional results are omitted for that model.
  4. [Section 2.1, Eqs. (10)–(12)] The comparison H[pg] ≥ H[G(z)] mixes differential entropies defined with respect to different reference measures (RD vs. a d-dimensional manifold). A clarifying statement about the reference measure and the role of the additive Gaussian noise would help.
  5. [Section 4.3, Fig. 5] The Figure 5 caption appears to contain duplicated axis labels and subfigure labels; please clean up the caption.
  6. [Appendix B] The dynamic decay schedule for M/s1^2 is described as 'consistent with the approximation of KL (pg∥pθ) from the Taylor series point of view', but no derivation is provided; either supply the derivation or label the schedule as purely empirical.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the empirical claims are benchmarked externally; the self-admitted departures from true bounds are correctness limitations, not circular reductions.

full rationale

The derivation chain starts from standard inequalities: Jensen's inequality for the variational lower bound, change-of-variables and smallest-singular-value inequalities for Eq. (14), the D_JS <= D_KL relation for the mutual-information lower bound, the Lipschitz/gradient inequality of Theorem 1 (cited from the authors' prior work [17]) for the gradient-penalty upper bound, and the diffusion-process KL decomposition of Song et al. [69] for the diffusion upper bound. None of these defines the target quantity in terms of the final FID, IS, or test-likelihood numbers, and the experimental section compares against external baselines (DCGAN, SNGAN, WGAN-0GP, MEG, VERA, DDPM, NCSN) on standard datasets. Thus the central claim that bidirectional bounds stabilize training and yield high-quality generation is not forced by construction. The self-citation [17] supplies the proof of Theorem 1; a proved inequality is independent support rather than circularity, and the implemented GP objective (hinge, scheduled M/s1^2, ignored Mm) in fact departs from that theorem, so the empirical results are not a restatement of the theorem. The paper itself explicitly lists limitations in Sec. 5: p_theta(x,t) need not satisfy the Fokker-Planck equation, the constants M and m are not computed, the LOPCG solver is stopped prematurely, the entropy weight lambda is reduced in large-scale runs, and KL(pgT || p_thetaT) is discarded. These show that the implemented losses are not guaranteed bounds; that is a validity or correctness concern, not a circularity concern. No exhibited equation shows a 'prediction' reducing to a fitted parameter or to the paper's own prior claim by construction. Therefore a low circularity score is appropriate.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The method rests on several hand-tuned quantities (lambda, M/s1^2, zeta, sigma_noise, diffusion schedule, LOPCG iterations) and on assumptions the authors concede are violated in practice, such as the Fokker-Planck condition and premature stopping. No fundamentally new physical or conceptual entity is postulated.

free parameters (6)
  • Entropy regularizer weight lambda = 1 in theory; 0.0001 to 0.01 in large-scale experiments
    Scales the entropy term in the lower bound; authors reduce it on CIFAR-10, AnimeFace, and CelebA to improve FID, so the optimized objective is not exactly the theoretical bound.
  • M/s1^2 ratio for gradient penalty upper bound = 0.1/zdim, with dynamic decay from 0.01 to 0.0001 over training
    Volume of support M is unknown; the ratio is treated as a tunable parameter and must be adjusted together with lambda.
  • Zeta margin in hinge upper bound = 1 in most experiments
    Hinge offset added to the gradient penalty bound; authors recommend starting at 0 and searching.
  • Generator noise scale sigma_noise = small, not specified exactly
    Gaussian noise added to the generator output; determines entropy and makes the mutual information bound finite, but the exact value is not reported.
  • LOPCG inner iterations = stopped prematurely, no fixed count
    Number of iterations trades accuracy and cost; early stopping technically violates the singular-value lower bound.
  • Diffusion SDE schedule sigma_min, sigma_max, T = sigma_min=0.01, sigma_max=0.1 (adjusted), T=1
    Hyperparameters of the VE-SDE used in the diffusion upper bound; sigma_max is adjusted per experiment.
assumptions (7)
  • standard math Jensen's inequality for the lower bound in Eqs (3)-(5)
    Used to derive log Z_theta >= E_pg[-E_theta] + H[pg].
  • domain assumption G maps R^d to an immersed d-dimensional manifold in R^D with full-rank Jacobian
    Needed for the change-of-variables formula in Eq (10); authors restrict the generator architecture but do not guarantee full rank.
  • ad hoc to paper H[pg] >= H[G(z)] for x = G(z) + epsilon
    Stated in Sec 2.1 without proof; questionable because H[pg] is a Lebesgue entropy on R^D while H[G(z)] is defined on the manifold.
  • ad hoc to paper There exist constants M, m, p such that inequality (21) holds
    Proof only in the authors' prior conference paper [17]; the constants are not constructed here and M/s1^2 is treated as a tunable parameter.
  • domain assumption p_theta(x,t) satisfies the Fokker-Planck equation (29)
    Required for the diffusion upper bound to be valid; authors admit a neural E_theta(x,t) generally violates it.
  • domain assumption KL(pgT || p_thetaT) is close to 0 when enough noise is added
    This terminal KL term is ignored in Eq (30); validity depends on the SDE schedule.
  • ad hoc to paper A few LOPCG iterations turn a Hutchinson upper entropy estimate into a lower bound
    Authors note in Sec 5 that premature stopping technically violates the bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring bidirectional bounds for minimax-training of Energy-based models." pith.science (2026). https://pith.science/paper/PCXIQQCA

@misc{pith2026250604609,
  author       = {Pith},
  title        = {Pith review of: Exploring bidirectional bounds for minimax-training of Energy-based models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCXIQQCA}},
  note         = {Machine review of arXiv:2506.04609}
}
read the original abstract

Energy-based models (EBMs) estimate unnormalized densities in an elegant framework, but they are generally difficult to train. Recent work has linked EBMs to generative adversarial networks, by noting that they can be trained through a minimax game using a variational lower bound. To avoid the instabilities caused by minimizing a lower bound, we propose to instead work with bidirectional bounds, meaning that we maximize a lower bound and minimize an upper bound when training the EBM. We investigate four different bounds on the log-likelihood derived from different perspectives. We derive lower bounds based on the singular values of the generator Jacobian and on mutual information. To upper bound the negative log-likelihood, we consider a gradient penalty-like bound, as well as one based on diffusion processes. In all cases, we provide algorithms for evaluating the bounds. We compare the different bounds to investigate, the pros and cons of the different approaches. Finally, we demonstrate that the use of bidirectional bounds stabilizes EBM training and yields high-quality density estimation and sample generation.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 69 canonical work pages

  1. [17]

    Bounds all around: training energy-based models with bidirectional bounds.Advances in Neural Information Processing Systems, 34:19808– 19821, 2021

    Cong Geng, Jia Wang, Zhiyong Gao, Jes Frellsen, and Søren Hauberg. Bounds all around: training energy-based models with bidirectional bounds.Advances in Neural Information Processing Systems, 34:19808– 19821, 2021

    work page 2021

  2. [1]

    A gen- erative adversarial density estimator

    M Ehsan Abbasnejad, Qinfeng Shi, Anton van den Hengel, and Lingqiao Liu. A gen- erative adversarial density estimator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10782–10791, 2019

    2019

  3. [2]

    Gade: A generative adversarial approach to density estimation and its applications.Interna- tional Journal of Computer Vision, 128(10): 2731–2743, 2020

    M Ehsan Abbasnejad, Javen Shi, Anton van den Hengel, and Lingqiao Liu. Gade: A generative adversarial approach to density estimation and its applications.Interna- tional Journal of Computer Vision, 128(10): 2731–2743, 2020

    2020

  4. [3]

    Ackley, Geoffrey E

    David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm Springer Nature 2021 LATEX template Article Title21 (a) EBMSV+GP (b) EBMMI+GP (c) EBMMI+diff (Large) Fig. D3Visual results of 32×32 ImageNet dataset with different settings. for boltzmann machines.Cognitive Science, 9 (1):147–169, 1985

    2021

  5. [4]

    Uncertainty in the variational informa- tion bottleneck

    Alexander A Alemi, Ian Fischer, and Joshua V Dillon. Uncertainty in the variational informa- tion bottleneck. InUncertainty in Artificial Intelligence Workshop, 2018

    2018

  6. [5]

    Wasserstein generative adversar- ial networks

    Martin Arjovsky, Soumith Chintala, and L´ eon Bottou. Wasserstein generative adversar- ial networks. InInternational conference on machine learning, pages 214–223. PMLR, 2017

    2017

  7. [6]

    Variational inference: A review for statisticians.Journal of the American statis- tical Association, 112(518):859–877, 2017

    David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians.Journal of the American statis- tical Association, 112(518):859–877, 2017

    2017

  8. [7]

    Accurate and conservative estimates of MRF log-likelihood using reverse annealing

    Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Accurate and conservative estimates of MRF log-likelihood using reverse annealing. InProceedings of the Eighteenth International Conference on Artificial Intelli- gence and Statistics, volume 38, pages 102–110. PMLR, 09–12 May 2015

Show all 89 references
  1. [8]

    Your GAN is secretly an energy-based model and you should use discriminator driven latent sam- pling

    Tong Che, Ruixiang Zhang, Jascha Sohl- Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an energy-based model and you should use discriminator driven latent sam- pling. InAdvances in Neural Information Processing Systems, volume 33, pag...

    2020

  2. [9]

    W AIC, but why? Generative ensembles for robust anomaly detection.arXiv preprint arXiv:1810.01392, 2018

    Hyunsun Choi, Eric Jang, and Alexander A Alemi. W AIC, but why? Generative ensembles for robust anomaly detection.arXiv preprint arXiv:1810.01392, 2018

    2018 arXiv

  3. [10]

    Generative modeling through the semi-dual formulation of unbalanced optimal transport.Advances in Neural Information Processing Systems, 36, 2024

    Jaemoo Choi, Jaewoong Choi, and Myungjoo Kang. Generative modeling through the semi-dual formulation of unbalanced optimal transport.Advances in Neural Information Processing Systems, 36, 2024

    2024

  4. [11]

    Calibrating energy-based generative adversar- ial networks

    Zihang Dai, Amjad Almahairi, Philip Bach- man, Eduard Hovy, and Aaron Courville. Calibrating energy-based generative adversar- ial networks. InInternational Conference on Learning Representations, 2017

    2017

  5. [12]

    Prescribed gen- erative adversarial networks.arXiv preprint arXiv:1910.04302, 2019

    Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed gen- erative adversarial networks.arXiv preprint arXiv:1910.04302, 2019

    1910 arXiv

  6. [13]

    Nice: Non-linear independent compo- nents estimation

    Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent compo- nents estimation. InInternational Conference on Learning Representations Workshop, 2015

    2015

  7. [14]

    Implicit genera- tion and modeling with energy based models

    Yilun Du and Igor Mordatch. Implicit genera- tion and modeling with energy based models. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  8. [15]

    Bayesian generalised ensemble markov chain monte Springer Nature 2021 LATEX template 22Article Title carlo

    Jes Frellsen, Ole Winther, Zoubin Ghahra- mani, and Jesper Ferkinghoff-Borg. Bayesian generalised ensemble markov chain monte Springer Nature 2021 LATEX template 22Article Title carlo. InProceedings of the 19th Interna- tional Conference on Artificial Intelligence and Statisti...

    2021

  9. [16]

    Learning energy-based models by diffusion recovery like- lihood.International Conference on Learning Representations, 2021

    Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery like- lihood.International Conference on Learning Representations, 2021

    2021

  10. [18]

    Improving adversarial energy-based model via diffusion process.Proceedings of the 41th International Conference on Machine Learning, 2024

    Cong Geng, Tian Han, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, Søren Hauberg, and Bo Li. Improving adversarial energy-based model via diffusion process.Proceedings of the 41th International Conference on Machine Learning, 2024

    2024

  11. [19]

    Generative adversarial networks.Communi- cations of the ACM, 63(11):139–144, 2020

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communi- cations of the ACM, 63(11):139–144, 2020

    2020

  12. [20]

    No MCMC for me: Amortized sampling for fast and stable training of energy-based models

    Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No MCMC for me: Amortized sampling for fast and stable training of energy-based models. In International Conference on Learning Repre- sentations, 2021

    2021

  13. [21]

    Annealing between dis- tributions by averaging moments.Advances in Neural Information Processing Systems, 26, 2013

    Roger B Grosse, Chris J Maddison, and Russ R Salakhutdinov. Annealing between dis- tributions by averaging moments.Advances in Neural Information Processing Systems, 26, 2013

    2013

  14. [22]

    Improved training of wasserstein gans.Advances in neural information process- ing systems, 30, 2017

    Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans.Advances in neural information process- ing systems, 30, 2017

    2017

  15. [23]

    Noise-contrastive estimation: A new estima- tion principle for unnormalized statistical models

    Michael Gutmann and Aapo Hyv¨ arinen. Noise-contrastive estimation: A new estima- tion principle for unnormalized statistical models. InProceedings of the Thirteenth International Conference on Artificial Intel- ligence and Statistics, pages 297–304. JMLR Workshop and Conferen...

    2010

  16. [24]

    Divergence triangle for joint training of genera- tor model, energy-based model, and inferential model

    Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training of genera- tor model, energy-based model, and inferential model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2019

  17. [25]

    Joint training of variational auto-encoder and latent energy-based model

    Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7978–7987, 2020

    2020

  18. [26]

    Hierarchical vaes know what they don’t know

    Jakob D Havtorn, Jes Frellsen, Søren Hauberg, and Lars Maaløe. Hierarchical vaes know what they don’t know. InInternational Confer- ence on Machine Learning, pages 4117–4128. PMLR, 2021

    2021

  19. [27]

    A base- line for detecting misclassified and out-of- distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A base- line for detecting misclassified and out-of- distribution examples in neural networks. In International Conference on Learning Repre- sentations, 2017

    2017

  20. [28]

    Deep anomaly detection with outlier exposure

    Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. InInternational Con- ference on Learning Representations, 2019

    2019

  21. [29]

    Training products of experts by minimizing contrastive divergence

    Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771–1800, 2002

    2002

  22. [30]

    Optimal perceptual inference

    Geoffrey E Hinton and Terrence J Sejnowski. Optimal perceptual inference. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition, volume 448, 1983

    1983

  23. [31]

    A fast learning algorithm for deep Springer Nature 2021 LATEX template Article Title23 belief nets.Neural computation, 18(7):1527– 1554, 2006

    Geoffrey E Hinton, Simon Osindero, and Yee- Whye Teh. A fast learning algorithm for deep Springer Nature 2021 LATEX template Article Title23 belief nets.Neural computation, 18(7):1527– 1554, 2006

    2021

  24. [32]

    Learning deep representations by mutual information estimation and maximization

    R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. InInterna- tional Conference on Learning Representa- tions, 2019

    2019

  25. [33]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020

    2020

  26. [34]

    Neural networks and physical systems with emergent collective computa- tional abilities.Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982

    J J Hopfield. Neural networks and physical systems with emergent collective computa- tional abilities.Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982. ISSN 0027-8424

    1982

  27. [35]

    A stochastic estimator of the trace of the influence matrix for lapla- cian smoothing splines.Communications in Statistics-Simulation and Computation, 18(3): 1059–1076, 1989

    Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for lapla- cian smoothing splines.Communications in Statistics-Simulation and Computation, 18(3): 1059–1076, 1989

    1989

  28. [36]

    Estimation of non- normalized statistical models by score match- ing.Journal of Machine Learning Research, 6 (4), 2005

    Aapo Hyv¨ arinen. Estimation of non- normalized statistical models by score match- ing.Journal of Machine Learning Research, 6 (4), 2005

    2005

  29. [37]

    Bi-level doubly varia- tional learning for energy-based latent variable models

    Ge Kan, Jinhu L¨ u, Tian Wang, Baochang Zhang, Aichun Zhu, Lei Huang, Guodong Guo, and Hichem Snoussi. Bi-level doubly varia- tional learning for energy-based latent variable models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 184...

    2022

  30. [38]

    ContraGAN: Contrastive learning for conditional image generation

    Minguk Kang and Jaesik Park. ContraGAN: Contrastive learning for conditional image generation. InAdvances in Neural Informa- tion Processing Systems, volume 33, pages 21357–21369, 2020

    2020

  31. [39]

    Soft truncation: A universal training technique of score-based diffusion model for high preci- sion score estimation

    Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high preci- sion score estimation. InProceedings of the 38th International Conference on Machine Learning, 2021

    2021

  32. [40]

    Deep directed generative models with energy- based probability estimation.arXiv preprint arXiv:1606.03439, 2016

    Taesup Kim and Yoshua Bengio. Deep directed generative models with energy- based probability estimation.arXiv preprint arXiv:1606.03439, 2016

    2016 arXiv

  33. [41]

    Toward the optimal pre- conditioned eigensolver: Locally optimal block preconditioned conjugate gradient method

    Andrew V Knyazev. Toward the optimal pre- conditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM journal on scientific computing, 23(2): 517–541, 2001

    2001

  34. [42]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  35. [43]

    Regularized autoencoders via relaxed injective probability flow

    Abhishek Kumar, Ben Poole, and Kevin Mur- phy. Regularized autoencoders via relaxed injective probability flow. InInternational Conference on Artificial Intelligence and Statistics, pages 4292–4301. PMLR, 2020

    2020

  36. [44]

    Max- imum entropy generators for energy-based models.arXiv preprint arXiv:1901.08508, 2019

    Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Max- imum entropy generators for energy-based models.arXiv preprint arXiv:1901.08508, 2019

    1901 arXiv

  37. [45]

    A tutorial on energy-based learning.Predicting structured data, 1(0), 2006

    Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning.Predicting structured data, 1(0), 2006

    2006

  38. [46]

    Guiding energy-based models via contrastive latent variables

    Hankook Lee, Jongheon Jeong, Sejun Park, and Jinwoo Shin. Guiding energy-based models via contrastive latent variables. In International Conference on Learning Repre- sentations, 2023

    2023

  39. [47]

    Yingzhen Li and Richard E. Turner. Gra- dient estimators for implicit models. In International Conference on Learning Repre- sentations, 2018

    2018

  40. [48]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. InProceedings of the IEEE inter- national conference on computer vision, pages 3730–3738, 2015. Springer Nature 2021 LATEX template 24Article Title

    2015

  41. [49]

    Knowledge distillation in iterative generative models for improved sampling speed.arXiv preprint arXiv:2101.02388, 2021

    Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed.arXiv preprint arXiv:2101.02388, 2021

    2021 arXiv

  42. [50]

    Equation of state calculations by fast computing machines.The journal of chemical physics, 21(6):1087–1092, 1953

    Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. Equation of state calculations by fast computing machines.The journal of chemical physics, 21(6):1087–1092, 1953

    1953

  43. [51]

    Spectral nor- malization for generative adversarial networks

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral nor- malization for generative adversarial networks. InInternational Conference on Learning Rep- resentations, 2018

    2018

  44. [52]

    MCMC using hamil- tonian dynamics.Handbook of markov chain monte carlo, 2(11):2, 2011

    Radford M Neal et al. MCMC using hamil- tonian dynamics.Handbook of markov chain monte carlo, 2(11):2, 2011

    2011

  45. [53]

    Learning non-convergent non-persistent short-run mcmc toward energy- based model.Advances in Neural Information Processing Systems, 32, 2019

    Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy- based model.Advances in Neural Information Processing Systems, 32, 2019

    2019

  46. [54]

    On the anatomy of MCMC-based maximum likeli- hood learning of energy-based models

    Erik Nijkamp, Mitch Hill, Tian Han, Song- Chun Zhu, and Ying Nian Wu. On the anatomy of MCMC-based maximum likeli- hood learning of energy-based models. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 34, pages 5272–5280, 2020

    2020

  47. [55]

    Boltzmann machines and energy-based models.arXiv preprint arXiv:1708.06008, 2017

    Takayuki Osogami. Boltzmann machines and energy-based models.arXiv preprint arXiv:1708.06008, 2017

    2017 arXiv

  48. [56]

    Automatic differ- entiation in pytorch.NIPS 2017 Workshop Autodiff, 2017

    Adam Paszke, Sam Gross, Soumith Chin- tala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differ- entiation in pytorch.NIPS 2017 Workshop Autodiff, 2017

    2017

  49. [57]

    Unsupervised representation learning with deep convolutional generative adversar- ial networks

    Alec Radford, Luke Metz, and Soumith Chin- tala. Unsupervised representation learning with deep convolutional generative adversar- ial networks. InInternational Conference on Learning Representations, 2016

    2016

  50. [58]

    Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan

    Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likeli- hood ratios for out-of-distribution detection. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  51. [59]

    Assessing generative models via precision and recall.Advances in neural information processing systems, 31, 2018

    Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall.Advances in neural information processing systems, 31, 2018

    2018

  52. [60]

    Deep boltzmann machines

    Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. InProceedings of the Twelth International Conference on Arti- ficial Intelligence and Statistics, volume 5, pages 448–455. PMLR, 16–18 Apr 2009

    2009

  53. [61]

    On the quantitative analysis of deep belief net- works

    Ruslan Salakhutdinov and Iain Murray. On the quantitative analysis of deep belief net- works. InProceedings of the 25th Interna- tional Conference on Machine Learning, pages 872–879, 2008

    2008

  54. [62]

    Tim Salimans, Andrej Karpathy, Xi Chen, and DiederikP. Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mix- ture likelihood and other modifications. In International Conference on Learning Repre- sentations, Jan 2017

    2017

  55. [63]

    PhD the- sis, Universit´ e de Montr´ eal, Quebec, Canada, 2020

    Benjamin Scellier.A deep learning theory for neural networks grounded in physics. PhD the- sis, Universit´ e de Montr´ eal, Quebec, Canada, 2020

    2020

  56. [64]

    A spectral approach to gradient estimation for implicit distributions

    Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit distributions. InInternational Confer- ence on Machine Learning, pages 4644–4653. PMLR, 2018

    2018

  57. [65]

    Smolensky.Information Processing in Dynamical Systems: Foundations of Harmony Theory, page 194–281

    P. Smolensky.Information Processing in Dynamical Systems: Foundations of Harmony Theory, page 194–281. MIT Press, Cambridge, MA, USA, 1986

    1986

  58. [66]

    International conference on learning representations

    Jiaming Song, Chenlin Meng, and Stefano Ermon. International conference on learning representations. 2021. Springer Nature 2021 LATEX template Article Title25

    2021

  59. [67]

    Generative modeling by estimating gradients of the data distribution.Advances in Neural Information Processing Systems, 32, 2019

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.Advances in Neural Information Processing Systems, 32, 2019

    2019

  60. [68]

    Improved techniques for training score-based genera- tive models.Advances in neural information processing systems, 33:12438–12448, 2020

    Yang Song and Stefano Ermon. Improved techniques for training score-based genera- tive models.Advances in neural information processing systems, 33:12438–12448, 2020

    2020

  61. [69]

    Maximum likelihood training of score-based diffusion models.Advances in Neural Information Processing Systems, 34: 1415–1428, 2021

    Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models.Advances in Neural Information Processing Systems, 34: 1415–1428, 2021

    2021

  62. [70]

    Score-based generative mod- eling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative mod- eling through stochastic differential equations. InInternational Conference on Learning Rep- resentations, 2021

    2021

  63. [71]

    Consistency models.Pro- ceedings of the 40th International Conference on Machine Learning, 2023

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models.Pro- ceedings of the 40th International Conference on Machine Learning, 2023

    2023

  64. [72]

    Cambridge Univer- sity Press, Cambridge, UK, 2019

    Simo S¨ arkk¨ a and Arno Solin.Applied Stochas- tic Differential Equations. Cambridge Univer- sity Press, Cambridge, UK, 2019

    2019

  65. [73]

    Improving generalization and sta- bility of generative adversarial networks

    Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. Improving generalization and sta- bility of generative adversarial networks. In International Conference on Learning Repre- sentations, 2019

    2019

  66. [74]

    A connection between score matching and denoising autoencoders.Neural computation, 23(7):1661–1674, 2011

    Pascal Vincent. A connection between score matching and denoising autoencoders.Neural computation, 23(7):1661–1674, 2011

    2011

  67. [75]

    The geom- etry of deep generative image models and its applications

    Binxu Wang and Carlos R Ponce. The geom- etry of deep generative image models and its applications. InInternational Conference on Learning Representations, 2021

    2021

  68. [76]

    Sparse and deep gener- alizations of the frame model.Annals of Mathematical Sciences and Applications, 3(1): 211–254, 2018

    Ying Nian Wu, Jianwen Xie, Yang Lu, and Song-Chun Zhu. Sparse and deep gener- alizations of the frame model.Annals of Mathematical Sciences and Applications, 3(1): 211–254, 2018

    2018

  69. [77]

    Tackling the generative learning trilemma with denoising diffusion gans.Inter- national Conference on Learning Representa- tions, 2022

    Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans.Inter- national Conference on Learning Representa- tions, 2022

    2022

  70. [78]

    Learning sparse FRAME mod- els for natural image patterns.International Journal of Computer Vision, 114(2):91–112, 2015

    Jianwen Xie, Wenze Hu, Song-Chun Zhu, and Ying Nian Wu. Learning sparse FRAME mod- els for natural image patterns.International Journal of Computer Vision, 114(2):91–112, 2015

    2015

  71. [79]

    Inducing wavelets into random fields via generative boosting.Applied and Computational Harmonic Analysis, 41(1):4– 25, 2016

    Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Inducing wavelets into random fields via generative boosting.Applied and Computational Harmonic Analysis, 41(1):4– 25, 2016

    2016

  72. [80]

    Synthesizing dynamic patterns by spatial-temporal generative convnet

    Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial-temporal generative convnet. In Proceedings of the ieee conference on com- puter vision and pattern recognition, pages 7093–7101, 2017

    2017

  73. [81]

    Learning descriptor networks for 3d shape synthesis and analysis

    Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wen- guan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 8629–8638, 2018

    2018

  74. [82]

    Cooperative learning of energy-based model and latent variable model via MCMC teaching

    Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via MCMC teaching. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018a

  75. [83]

    Cooperative training of descriptor and generator networks.IEEE transactions on pattern analysis and machine intelligence, 42(1):27–45, 2018b

    Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks.IEEE transactions on pattern analysis and machine intelligence, 42(1):27–45, 2018b

  76. [84]

    Cooper- ative training of fast thinking initializer and slow thinking solver for conditional learning

    Jianwen Xie, Zilong Zheng, Xiaolin Fang, Song-Chun Zhu, and Ying Nian Wu. Cooper- ative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. Springer Nature 2021 LATEX tem...

    2021

  77. [85]

    Learning energy-based model with variational auto-encoder as amortized sampler

    Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. InThe Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), volume 2, 2021

    2021

  78. [86]

    Generative adversarial net- works as variational training of energy based models.arXiv preprint arXiv:1611.01799, 2016

    Shuangfei Zhai, Yu Cheng, Rogerio Feris, and Zhongfei Zhang. Generative adversarial net- works as variational training of energy based models.arXiv preprint arXiv:1611.01799, 2016

    2016 arXiv

  79. [87]

    Grade: Gibbs reaction and diffusion equations

    Song Chun Zhu and David Mumford. Grade: Gibbs reaction and diffusion equations. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pages 847–854. IEEE, 1998

    1998

  80. [88]

    Filters, random fields and maxi- mum entropy (FRAME): Towards a unified theory for texture modeling.International Journal of Computer Vision, 27(2):107–126, 1998

    Song Chun Zhu, Yingnian Wu, and David Mumford. Filters, random fields and maxi- mum entropy (FRAME): Towards a unified theory for texture modeling.International Journal of Computer Vision, 27(2):107–126, 1998

    1998

  81. [89]

    Learning energy-based models by cooperative diffusion recovery likelihood

    Yaxuan Zhu, Jianwen Xie, Yingnian Wu, and Ruiqi Gao. Learning energy-based models by cooperative diffusion recovery likelihood. In International Conference on Learning Repre- sentations, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.