FMRG reformulates guidance as deterministic optimal control, deriving a single-trajectory method using the flow map that matches or exceeds baselines on reward-guided generation and inverse problems with 3 NFEs at text-to-image scale.
hub
Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review
22 Pith papers cite this work. Polarity classification is still indexing.
22
Pith papers citing it
abstract
This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. While diffusion models are widely known to provide excellent generative modeling capability, practical applications in domains such as biology require generating samples that maximize some desired metric (e.g., translation efficiency in RNA, docking score in molecules, stability in protein). In these cases, the diffusion model can be optimized not only to generate realistic samples but also to explicitly maximize the measure of interest. Such methods are based on concepts from reinforcement learning (RL). We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning, tailored specifically for fine-tuning diffusion models. We aim to explore fundamental aspects such as the strengths and limitations of different RL-based fine-tuning algorithms across various scenarios, the benefits of RL-based fine-tuning compared to non-RL-based approaches, and the formal objectives of RL-based fine-tuning (target distributions). Additionally, we aim to examine their connections with related topics such as classifier guidance, Gflownets, flow-based diffusion models, path integral control theory, and sampling from unnormalized distributions such as MCMC. The code of this tutorial is available at https://github.com/masa-ue/RLfinetuning_Diffusion_Bioseq
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
Many reward-based fine-tuning methods for diffusion and flow models are special cases of a common score-matching objective whose differences reduce to value-guidance estimator design, temporal weighting, and trust-region realization.
A Gaussian process surrogate gate inserted between generative crystal models and property oracles matches or exceeds ungated fine-tuning while using roughly one-fifth the oracle calls for heat capacity and bulk modulus.
MCHF defines a Markov kernel from pairwise utilities U and proves geometric convergence to its stationary distribution at a rate set by the seminorm measuring non-transitivity of U, with first-order equivalence to RLHF and NLHF solutions.
ActFlow expands the generable set of pre-trained flow models for out-of-distribution molecular and sequence design via active synthetic data generation and verifier feedback, with new statistical guarantees.
Minimum-flow GFlowNets on graphs encode optimal transport plans, with the learned policy recovering the optimal coupling between source and target distributions.
FAV aligns few-step generative models by amortizing SVGD updates from reward-tilted sampling into generator parameters via fixed-point regression, requiring only sample access, and shows outperformance on robotics tasks plus scaling on image generators.
CDM amortizes SMC inference for reward-tilted discrete diffusion by training a parameterized twist function on contrastive samples with closed-form kernels.
A new adjoint matching framework formulates flow model alignment as optimal control, enabling direct regression training and terminal-trajectory truncation for efficiency gains on models like SiT-XL and FLUX.
MSDDA derives a closed-form optimal reverse denoising distribution for multi-objective diffusion alignment that is exactly equivalent to step-level RL fine-tuning with no approximation error.
RMMD simultaneously distills diffusion models and optimizes rewards, yielding better FID-reward trade-offs on ImageNet than DI++, DRaFT and HyperNoise, and a 7.5x faster GenCast model that beats its teacher on 93% of weather variables while improving calibration.
BayesFP provides a unified retraining-free sampler for diffusion and flow policies by casting constrained trajectory generation as posterior sampling via an extended Feynman-Kac corrector.
A2D2 derives the Radon-Nikodym derivative for joint insertion-unmasking paths in discrete diffusion to enable reward-tilted fine-tuning and introduces the Adaptive Joint Decoding loss.
GILC guides discrete diffusion at inference time by adding reward gradients to the prediction logits, matching or beating fine-tuned baselines on DNA, protein, and molecule generation tasks without retraining.
MaskDiff-AD uses reconstruction difficulty of masked coordinates in a diffusion model trained only on nominal data to detect anomalies, with a non-parametric variant and theoretical error guarantees, achieving the best average rank on 18 datasets.
AdvantageFlow proposes an advantage-weighted forward-process least-squares loss for RL in rectified flow models, stabilized by rollout policy regularization, and reports better image generation performance than Flow-GRPO on Stable Diffusion 3.5.
AdaScope adaptively selects optimal RL intervention points during diffusion denoising by monitoring structural and semantic changes, delivering 66% higher performance at 59% lower cost than full-trajectory RL baselines.
Policy entropy remains constant in flow-matching models during RLHF due to fixed noise schedules while perceptual diversity collapses from mode-seeking policy gradients, so perceptual entropy constraints are introduced to preserve diversity and improve quality.
ZeNO frames noise optimization as a path-integral control problem solvable from zeroth-order reward evaluations, connecting to implicit Langevin dynamics for reward-tilted distributions.
Error in approximating the tangent conditional score by the unconditional score in diffusion models is bounded by dimension-free conditional mutual information, with a projected-Langevin method outperforming baselines in inpainting and super-resolution.
ALGD augments the Lagrangian to locally convexify the energy landscape in diffusion models, stabilizing safe RL training and generation without changing optimal policies.
An offline-trained controller augments autoregressive diffusion models to perform fast, feed-forward data assimilation in chaotic spatiotemporal PDEs with order-of-magnitude speedups and improved accuracy over baselines.
citing papers explorer
-
How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance
FMRG reformulates guidance as deterministic optimal control, deriving a single-trajectory method using the flow map that matches or exceeds baselines on reward-guided generation and inverse problems with 3 NFEs at text-to-image scale.
-
Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models
Many reward-based fine-tuning methods for diffusion and flow models are special cases of a common score-matching objective whose differences reduce to value-guidance estimator design, temporal weighting, and trust-region realization.
-
Surrogate-Gated Generation and Foundation-Model Embeddings for Bayesian Materials Design
A Gaussian process surrogate gate inserted between generative crystal models and property oracles matches or exceeds ungated fine-tuning while using roughly one-fifth the oracle calls for heat capacity and bulk modulus.
-
A Markov Chain Approach to Preference Alignment
MCHF defines a Markov kernel from pairwise utilities U and proves geometric convergence to its stationary distribution at a rate set by the seminorm measuring non-transitivity of U, with first-order equivalence to RLHF and NLHF solutions.
-
Active Flow Expansion for Out-of-Distribution Discovery: from Theory to Molecules
ActFlow expands the generable set of pre-trained flow models for out-of-distribution molecular and sequence design via active synthetic data generation and verifier feedback, with new statistical guarantees.
-
Your GFlowNet Secretly Learns an Optimal Transport Plan
Minimum-flow GFlowNets on graphs encode optimal transport plans, with the learned policy recovering the optimal coupling between source and target distributions.
-
Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference
FAV aligns few-step generative models by amortizing SVGD updates from reward-tilted sampling into generator parameters via fixed-point regression, requiring only sample access, and shows outperformance on robotics tasks plus scaling on image generators.
-
Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion
CDM amortizes SMC inference for reward-tilted discrete diffusion by training a parameterized twist function on contrastive samples with closed-form kernels.
-
Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline
A new adjoint matching framework formulates flow model alignment as optimal control, enabling direct regression training and terminal-trajectory truncation for efficiency gains on models like SiT-XL and FLUX.
-
Step-level Denoising-time Diffusion Alignment with Multiple Objectives
MSDDA derives a closed-form optimal reverse denoising distribution for multi-objective diffusion alignment that is exactly equivalent to step-level RL fine-tuning with no approximation error.
-
Diffusion Fine-tuning with Rewarded Moment Matching Distillation
RMMD simultaneously distills diffusion models and optimizes rewards, yielding better FID-reward trade-offs on ImageNet than DI++, DRaFT and HyperNoise, and a 7.5x faster GenCast model that beats its teacher on 93% of weather variables while improving calibration.
-
BayesFP: Posterior Estimation for Flow-Based Policies via Feynman-Kac Sampling
BayesFP provides a unified retraining-free sampler for diffusion and flow policies by casting constrained trajectory generation as posterior sampling via an extended Feynman-Kac corrector.
-
A2D2: Fine-Tuning Any-Length Discrete Diffusion for Adaptive Decoding
A2D2 derives the Radon-Nikodym derivative for joint insertion-unmasking paths in discrete diffusion to enable reward-tilted fine-tuning and introduces the Adaptive Joint Decoding loss.
-
Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction
GILC guides discrete diffusion at inference time by adding reward gradients to the prediction logits, matching or beating fine-tuned baselines on DNA, protein, and molecule generation tasks without retraining.
-
Masked Diffusion Modeling for Anomaly Detection
MaskDiff-AD uses reconstruction difficulty of masked coordinates in a diffusion model trained only on nominal data to detect anomalies, with a non-parametric variant and theoretical error guarantees, achieving the best average rank on 18 datasets.
-
AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Models
AdvantageFlow proposes an advantage-weighted forward-process least-squares loss for RL in rectified flow models, stabilized by rollout policy regularization, and reports better image generation performance than Flow-GRPO on Stable Diffusion 3.5.
-
Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?
AdaScope adaptively selects optimal RL intervention points during diffusion denoising by monitoring structural and semantic changes, delivering 66% higher performance at 59% lower cost than full-trajectory RL baselines.
-
When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
Policy entropy remains constant in flow-matching models during RLHF due to fixed noise schedules while perceptual diversity collapses from mode-seeking policy gradients, so perceptual entropy constraints are introduced to preserve diversity and improve quality.
-
Gradient-Free Noise Optimization for Reward Alignment in Generative Models
ZeNO frames noise optimization as a path-integral control problem solvable from zeroth-order reward evaluations, connecting to implicit Langevin dynamics for reward-tilted distributions.
-
Conditional Diffusion Under Linear Constraints: Langevin Mixing and Information-Theoretic Guarantees
Error in approximating the tangent conditional score by the unconditional score in diffusion models is bounded by dimension-free conditional mutual information, with a projected-Langevin method outperforming baselines in inpainting and super-resolution.
-
How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?
ALGD augments the Lagrangian to locally convexify the energy landscape in diffusion models, stabilizing safe RL training and generation without changing optimal policies.
-
Control-Augmented Autoregressive Diffusion for Data Assimilation
An offline-trained controller augments autoregressive diffusion models to perform fast, feed-forward data assimilation in chaotic spatiotemporal PDEs with order-of-magnitude speedups and improved accuracy over baselines.