SCOREBED isolates EIG double intractability in a policy-independent score-matching stage, then trains design policies with a singly intractable gradient estimator, enabling cheap multi-policy selection.
hub
How to train your energy-based models
21 Pith papers cite this work, alongside 81 external citations. Polarity classification is still indexing.
21
Pith papers citing it
81
external citations · Pith
abstract
Energy-Based Models (EBMs), also known as non-normalized probabilistic models, specify probability density or mass functions up to an unknown normalizing constant. Unlike most other probabilistic models, EBMs do not place a restriction on the tractability of the normalizing constant, thus are more flexible to parameterize and can model a more expressive family of probability distributions. However, the unknown normalizing constant of EBMs makes training particularly difficult. Our goal is to provide a friendly introduction to modern approaches for EBM training. We start by explaining maximum likelihood training with Markov chain Monte Carlo (MCMC), and proceed to elaborate on MCMC-free approaches, including Score Matching (SM) and Noise Constrastive Estimation (NCE). We highlight theoretical connections among these three approaches, and end with a brief survey on alternative training methods, which are still under active research. Our tutorial is targeted at an audience with basic understanding of generative models who want to apply EBMs or start a research project in this direction.
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2representative citing papers
Model merging is cast as PoE inference with EBM experts, revealing Gaussian assumptions in prior work and proposing convergent Cauchy experts that improve empirical performance.
Energy-navigated trajectory shaping during training produces 8-step discrete flow matching students that achieve 32% lower perplexity than 1024-step teachers on 170M language models with unchanged inference cost.
Langevin sampling on the modern Hopfield energy produces training-free stochastic attention that transitions from exact retrieval to generation as temperature rises, with an entropy inflection condition marking the shift.
CreTTA reformulates test-time adaptation of marginal distributions as residual energy learning, producing a contrastive objective that cancels the partition function and uses relative energy differences for adaptive gradient reweighting to avoid overfitting.
Derives Õ(d β² A² / ε⁴) oracle complexity for AIS estimating normalizing constant Z to relative error ε and introduces reverse diffusion sampler for geometric paths with large action.
Stochastic interpolants unify flow-based and diffusion-based generative models by bridging target densities exactly via latent-variable processes whose drifts minimize quadratic objectives.
SDEdit performs guided image synthesis and editing by adding noise to inputs and refining them via denoising with a diffusion model's SDE prior, outperforming GAN methods in human studies without task-specific training.
STREAM decouples text (via AdaLN) from music (via energy-based BEAM attention) to generate editable, musically aligned dance motions with a new annotated dataset and editability metric.
Helmholtz Distillation plus Negative Caching amortizes Energy Matching to 3D CT volumes, producing a conservative prior that improves FID and sparse-view reconstruction over pure flow baselines.
The generalization advantage of SGD over random sampling diminishes with growing training set size in binary networks, as measured by joint density of states over train and test accuracy.
Explicit MSE bounds derived for time-average estimators in adaptive increasingly rare MCMC under simultaneous Wasserstein contraction.
Decentralized diffusion policies trained with importance sampling score matching enhance exploration and performance in cooperative MARL over Gaussian policy baselines.
Training and sampling in static scalar energy generative models are two instances of the same Lyapunov-driven density transport dynamics on Wasserstein space, differing only by initial condition, which yields a finite stopping criterion for Langevin sampling and additive composition rules that keep
A baseline-trained energy manifold in MRI intensity space showed progressive drift toward the tumour regime in a recurrence case but not in a stable case, suggesting a segmentation-free longitudinal monitoring signal.
CLIC uses set-valued action targets from interactive human corrections instead of pointwise labels to train more robust imitation learning policies.
ERAlign aligns GNN and LLM embeddings on text-attributed graphs via energy-based models and an Energy Discrepancy objective, reporting state-of-the-art results on eight datasets under varying supervision.
Edwin integrates dynamic maximum entropy dimensionality reduction with symbolic regression to recover physically interpretable low-dimensional dynamics from high-dimensional observations that generalize to unseen conditions.
Under the chain rule of probability, autoregressive models and energy-based models are in exact bijection in function space, making the global optimum of teacher forcing equivalent to an energy-based model with implicit lookahead.
Score-difference flow reduces KL divergence between distributions and is formally equivalent to denoising diffusion models and a hidden subproblem in optimal GAN training under stated conditions.
The tutorial synthesizes diffusion model techniques for generative semantic communications to achieve high compression while preserving meaning in wireless transmission.
citing papers explorer
-
Bayesian Experimental Design via Score Matching
SCOREBED isolates EIG double intractability in a policy-independent score-matching stage, then trains design policies with a singly intractable gradient estimator, enabling cheap multi-policy selection.
-
Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space
Model merging is cast as PoE inference with EBM experts, revealing Gaussian assumptions in prior work and proposing convergent Cauchy experts that improve empirical performance.
-
Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation
Energy-navigated trajectory shaping during training produces 8-step discrete flow matching students that achieve 32% lower perplexity than 1024-step teachers on 170M language models with unchanged inference cost.
-
Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy
Langevin sampling on the modern Hopfield energy produces training-free stochastic attention that transitions from exact retrieval to generation as temperature rises, with an entropy inflection condition marking the shift.
-
Contrastive Residual Energy Test-time Adaptation
CreTTA reformulates test-time adaptation of marginal distributions as residual energy learning, producing a contrastive objective that cancels the partition function and uses relative energy differences for adaptive gradient reweighting to avoid overfitting.
-
Complexity Analysis of Normalizing Constant Estimation: from Jarzynski Equality to Annealed Importance Sampling and beyond
Derives Õ(d β² A² / ε⁴) oracle complexity for AIS estimating normalizing constant Z to relative error ε and introduces reverse diffusion sampler for geometric paths with large action.
-
Stochastic Interpolants: A Unifying Framework for Flows and Diffusions
Stochastic interpolants unify flow-based and diffusion-based generative models by bridging target densities exactly via latent-variable processes whose drifts minimize quadratic objectives.
-
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
SDEdit performs guided image synthesis and editing by adding noise to inputs and refining them via denoising with a diffusion model's SDE prior, outperforming GAN methods in human studies without task-specific training.
-
Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation
STREAM decouples text (via AdaLN) from music (via energy-based BEAM attention) to generate editable, musically aligned dance motions with a new annotated dataset and editability metric.
-
Projected Energy Matching for Generative 3D Priors
Helmholtz Distillation plus Negative Caching amortizes Energy Matching to 3D CT volumes, producing a conservative prior that improves FID and sparse-view reconstruction over pure flow baselines.
-
Revisiting the Volume Hypothesis
The generalization advantage of SGD over random sampling diminishes with growing training set size in binary networks, as measured by joint density of states over train and test accuracy.
-
Error bounds for simultaneous Wasserstein contractive adaptive increasingly rare MCMC
Explicit MSE bounds derived for time-average estimators in adaptive increasingly rare MCMC under simultaneous Wasserstein contraction.
-
Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning
Decentralized diffusion policies trained with importance sampling score matching enhance exploration and performance in cooperative MARL over Gaussian policy baselines.
-
Energy Generative Modeling: A Lyapunov-based Energy Matching Perspective
Training and sampling in static scalar energy generative models are two instances of the same Lyapunov-driven density transport dynamics on Wasserstein space, differing only by initial condition, which yields a finite stopping criterion for Langevin sampling and additive composition rules that keep
-
Energy-based Tissue Manifolds for Longitudinal Multiparametric MRI Analysis
A baseline-trained energy manifold in MRI intensity space showed progressive drift toward the tumour regime in a recurrence case but not in a stable case, suggesting a segmentation-free longitudinal monitoring signal.
-
From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback
CLIC uses set-valued action targets from interactive human corrections instead of pointwise labels to train more robust imitation learning policies.
-
ERAlign: Energy-based Representation Alignment of GNNs and LLMs on Text-attributed Graphs
ERAlign aligns GNN and LLM embeddings on text-attributed graphs via energy-based models and an Energy Discrepancy objective, reporting state-of-the-art results on eight datasets under varying supervision.
-
Discovering interpretable low-dimensional dynamics using maximum entropy
Edwin integrates dynamic maximum entropy dimensionality reduction with symbolic regression to recover physically interpretable low-dimensional dynamics from high-dimensional observations that generalize to unseen conditions.
-
Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
Under the chain rule of probability, autoregressive models and energy-based models are in exact bijection in function space, making the global optimum of teacher forcing equivalent to an energy-based model with implicit lookahead.
-
The Score-Difference Flow for Implicit Generative Modeling
Score-difference flow reduces KL divergence between distributions and is formally equivalent to denoising diffusion models and a hidden subproblem in optimal GAN training under stated conditions.
-
Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
The tutorial synthesizes diffusion model techniques for generative semantic communications to achieve high compression while preserving meaning in wireless transmission.