REVIEW 5 cited by
Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
read the original abstract
Fine-tuning foundation models via reinforcement learning (RL) has proven promising for aligning to downstream objectives. In the case of diffusion models (DMs), though RL training improves alignment from early timesteps, critical issues such as training instability and mode collapse arise. We address these drawbacks by exploiting the hierarchical nature of DMs: we train them dynamically at each epoch with a tailored RL method, allowing for continual evaluation and step-by-step refinement of the model performance (or alignment). Furthermore, we find that not every denoising step needs to be fine-tuned to align DMs to downstream tasks. Consequently, in addition to clipping, we regularise model parameters at distinct learning phases via a sliding-window approach. Our approach, termed Hierarchical Reward Fine-tuning (HRF), is validated on the Denoising Diffusion Policy Optimisation method, where we show that models trained with HRF achieve better preservation of diversity in downstream tasks, thus enhancing the fine-tuning robustness and at uncompromising mean rewards.
Forward citations
Cited by 5 Pith papers
-
An Analysis of Posterior Collapse, Parameterization and Initialization in Variational Deep Gaussian Processes
Analysis of posterior collapse in variational DGPs shows the linear prior mean benefit comes from better optimization conditioning at initialization; a new zero-mean initialization prevents collapse and matches linear...
-
Primitive-based Truncated Diffusion for Efficient Trajectory Generation of Differential Drive Mobile Manipulators
A primitive-based truncated diffusion model with keypoint attention encoding generates more efficient and diverse trajectories for mobile manipulators than vanilla diffusion in cluttered 3D simulations.
-
When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models
MedCausalX adds adaptive causal reflection tokens and trajectory-level RL on a new CRMed dataset to cut medical VLM hallucinations and raise diagnostic consistency.
-
Statistical Properties of Training & Generalization
Neural scaling laws in deep learning interact with physics constraints and inductive biases beyond classical statistics.
-
Statistical Properties of Training & Generalization
Review of neural scaling laws and their relation to constraints and inductive biases when applying machine learning to physics problems.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.