Pith. sign in

REVIEW 3 major objections 5 minor 65 references

Diffusion RL can drop softmax for any monotonic reweighting—including negative weights—by matching policies to signed target measures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 23:46 UTC pith:Y7SNPJCS

load-bearing objection Clean unification of reweighted diffusion RL under f-divergences and signed measures; negative-weight gains are real but rest on practical stabilizers the theory does not fully cover. the 3 major comments →

arxiv 2603.10250 v2 pith:Y7SNPJCS submitted 2026-03-10 cs.LG

GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning

Haitong Ma , Chenxiao Gao , Tianyi Chen , Na Li , Bo Dai This is my paper
classification cs.LG
keywords diffusion reinforcement learningreweighted flow matchingf-divergencesigned measuresnegative reweightingpolicy optimizationmeasure matchingonline RL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard diffusion-policy RL reweights behavior samples with a softmax of advantages. That choice is often over-greedy and simply ignores bad samples. This paper shows the same pipeline works for any monotonic weight function once the problem is split in two: first build a (possibly signed) target measure that solves a regularized policy objective, then pull the current diffusion policy toward that measure with reweighted flow matching. Negative weights reverse the conditional velocity of low-value actions, actively repelling the generative path from them and aiding exploration, while a short theorem still guarantees policy improvement. Bandit, MuJoCo, and DNA-sequence experiments confirm that linear, square, and carefully truncated negative schemes match or beat prior methods, and supply simple rules for matching the weight shape to the reward landscape.

Core claim

By constructing a virtual target policy as the solution of an f-divergence-regularized objective and then aligning a diffusion policy to that target via reweighted flow matching, one obtains a single framework that recovers exponential and Q-weighted algorithms as special cases, extends them to power-function weights, and, once non-negativity of the target is relaxed, supplies a principled justification for negative reweighting that repels the policy from suboptimal actions while still improving expected return.

What carries the argument

The two-stage measure-matching construction: an optimal (signed) target measure π* ∝ φ(Q/α)·π_old obtained from the f-divergence-regularized problem, followed by reweighted flow matching whose closed-form velocity field is a weighted average of conditional velocities; negative weights reverse those conditional velocities and thereby repel the generative trajectory.

Load-bearing premise

A batch estimate of the normalizer, a hand-chosen floor on how negative the weights may become, and an auto-tuned temperature that only constrains KL to uniform stay stable enough for the improvement guarantee to survive in practice.

What would settle it

On a known multi-modal bandit whose global optimum is outside the initial policy support, every GeMPO run that uses negative weights still locks onto the local mode while a non-negative baseline reaches the global mode, or increasing the allowed magnitude of negative weights systematically raises rather than lowers simple regret past the claimed sweet spot.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Linear weights suit sharp reward peaks and square weights suit broad plateaus; both can outperform pure exponential reweighting.
  • Modest negative floors (τ ≈ −0.1 to −0.3) improve exploration on locomotion and DNA tasks without collapsing training.
  • DPMD, QVPO and WD1 become special cases of one f-divergence / signed-measure template.
  • The same reweighted-flow-matching pipeline applies unchanged to discrete diffusion models.
  • Automatic temperature control via a KL-to-uniform budget removes the need for a fixed temperature schedule.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The signed-measure view of reweighting may transfer directly to post-training of other generative families (flows, energy-based models) that already produce many negative samples.
  • Making the generator f itself a function of local reward curvature could eliminate the need to hand-select linear versus square weights.
  • When the empirical weight sum becomes negative the loss is theoretically unbounded; online monitoring of that sign is a practical safeguard the paper leaves open.
  • Population normalizers may scale poorly in high-dimensional action spaces, suggesting particle-free or amortized normalizers as a natural next step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GeMPO, a measure-matching view of online RL for diffusion/flow policies. A (possibly signed) target measure is obtained as the solution of an f-divergence-regularized policy objective; the current diffusion policy is then aligned to that target by reweighted conditional flow matching. Instantiating the generator recovers exponential (DPMD/AWR), χ²/QVPO, power-function, and negative (WD1-style) reweighting as special cases. Relaxing non-negativity yields a formal justification for negative weights, with a covariance argument (Theorem 3.3) that the normalized signed target still improves expected return, and an interpretation that negative weights reverse conditional velocities and repel trajectories from low-value actions. Empirically, Linear/Square/Exp/Neg variants are competitive or better than diffusion and classic baselines on bandits, MuJoCo locomotion, and DNA enhancer fine-tuning, with qualitative guidelines linking reweighting curvature to reward landscape.

Significance. If the unification and the signed-measure extension hold under the stated assumptions, the work usefully organizes a growing family of reweighted flow-matching RL methods and supplies a clean design axis (choice of f / monotonic weight) that practitioners can match to reward curvature. The closed-form reweighted CFM solution (Appendix D), the policy-improvement guarantee for normalized signed targets (Theorem 3.3), and multi-domain experiments with means ± SE over seeds and ablations on τ and KL budget are concrete strengths. The practical value of negative reweighting for exploration is of clear interest to diffusion RL and discrete diffusion post-training; even if stabilizers remain necessary, a coherent derivation that recovers existing algorithms and motivates new ones is a worthwhile contribution.

major comments (3)
  1. [§3.3, Eq. (11), App. I] Section 3.3, Proposition 3.2, Equation (11), and Appendix I (Case II): the signed-measure theory assumes a normalized target and yields a well-defined reweighted CFM optimum only when the empirical weight mass E[w] > 0 (Theorem D.2). When E[w] ≤ 0 the quadratic CFM loss is unbounded below; the paper therefore inserts a hand-chosen truncation floor τ and a batch population estimator for Z(s) (Appendix E). These devices lie outside Propositions 3.1–3.2 and Theorem 3.3. The manuscript should either (i) state the practical algorithm as an approximation of the signed target and quantify bias/stability of the population normalizer and of τ, or (ii) temper claims of a fully “principled justification” for negative reweighting to the regime where the empirical denominator stays positive. Empirics already show collapse for aggressive τ (Figure 1 right; Table 3 Neg(-0.5); Figure 4), so this is load
  2. [App. E, Alg. 1] Appendix E and Algorithm 1: the population-based normalizer for linear/square weights is computed from a finite batch of old-policy samples (and, offline, from replay mixtures). No error bound or consistency argument is given for the estimated Z(s) relative to the true partition function of the target measure. Because the reweighted velocity (Equation 9) depends on these weights, uncontrolled normalizer bias can re-introduce the over-greediness the framework aims to avoid. A short analysis (e.g., concentration under i.i.d. sampling, or ablation of batch size N beyond the fixed N=64) would substantially strengthen the bridge from theory to the reported MuJoCo/DNA results.
  3. [§5.1–5.2, Fig. 2–3] Section 5.1–5.2 and the “practical guidelines” claim: the recommendation to use Linear for steep rewards and Square for flat rewards is supported by handcrafted bandit and DMC reward shapes, but remains qualitative. There is no quantitative diagnostic (e.g., a curvature or advantage-spread statistic) that would let a practitioner choose the generator without a sweep. Given that temperature/KL budget ε and τ are already free parameters (Appendix F, Figure 6), the paper should either supply a more operational selection rule or clearly present the guidelines as empirical heuristics rather than a systematic design principle.
minor comments (5)
  1. [Table 1] Table 1 and Examples 3.1–3.3: the generator and inverse-derivative columns are helpful but several entries are hard to parse (missing parentheses, ambiguous α ranges). A short expanded table with f, f', (f')^{-1}, and the resulting w(Q) side by side would improve readability.
  2. [Fig. 1–2] Figure 1–2 captions and axis labels are dense; several symbols (τ, regret scale) are hard to read in the rendered text. Consider larger fonts and explicit legends for Linear/Square/Exp/Neg.
  3. [App. A] Related work (Appendix A) cites reverse-KL and mirror-descent lines but could more explicitly position GeMPO against recent reverse-flow-matching and energy-weighted flow-matching works already listed in the bibliography (e.g., Li et al. 2026; Zhang et al. 2025) so that novelty of the signed-measure step is clearer.
  4. [§2–3] Notation: ϖ vs π*, signed measures M±(A), and the clipped inverse φ_c appear with slight inconsistencies between main text and appendices; a single notation paragraph would help.
  5. [§6] Section 6 limitation paragraph correctly notes unbounded loss for large negative τ; cross-referencing Appendix I Case I vs Case II there would make the limitation self-contained for readers who skip the appendix.

Circularity Check

0 steps flagged

No significant circularity: target measures and reweighted flow matching are derived from standard Lagrange and continuity arguments; special cases are intentional subsumption, not tautological predictions.

full rationale

The load-bearing chain is self-contained. Prop. 3.1 solves the f-divergence-regularized objective (Eq. 6) by Lagrange multipliers with non-negativity and normalization (App. C.1), yielding the clipped inverse-derivative weights; Prop. 3.2 drops non-negativity to obtain signed measures with monotonic weights (App. C.2); Thm. 3.3 is a covariance argument showing policy improvement for any strictly increasing reweighting under unit mass (App. C.3). Reweighted conditional flow matching is shown to recover the velocity of the reweighted path by expanding the quadratic loss and setting the gradient to zero (Thm. D.2 / App. D), with the continuity-equation check that the field generates the reweighted density. Instantiating f as forward KL, χ², or α-divergences recovers DPMD, QVPO, and power weights by construction of the generator—this is deliberate unification, not a fitted input renamed as a prediction. Negative reweighting is justified by the signed-measure optimum (Eq. 10), not by a self-citation uniqueness theorem; practical truncation τ (Eq. 11), population normalizers (App. E), and KL-to-uniform temperature tuning (App. F) are openly presented as implementation devices outside the pure theory, and the paper itself flags unbounded loss when E[w] ≤ 0 (App. I Case II, Sec. 6). Self-citations (e.g., Ma et al. DPMD) appear only as recovered special cases and related work, not as load-bearing premises. Empirical claims are comparative against external baselines (TD3, SAC, QSM, QVPO, DACER, DIPO, DRAKES, etc.), not restatements of fitted constants. No self-definitional loop, no fitted-parameter-as-prediction, no uniqueness imported from the authors, and no ansatz smuggled via citation for the central derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The framework rests on standard MDP and f-divergence machinery plus the usual diffusion/flow matching identities; the only non-standard modeling choice is the relaxation of the policy family to normalized signed measures, which is justified mathematically but requires extra practical controls (truncation, temperature). Free parameters are the usual RL and diffusion knobs plus the new truncation floor and KL budget.

free parameters (5)
  • regularization strength / temperature β
    Controls sharpness of reweighting; auto-tuned via a KL-to-uniform dual but still requires a target KL budget that is swept per task.
  • negative-weight truncation floor τ
    Hand-chosen lower bound (Eq. 11) that keeps the loss bounded; performance is sensitive to its value (Figs. 1, 4; Table 3).
  • target KL budget ε
    Hyper-parameter of the dual temperature update; different values are used for Hopper, Ant and other tasks (Table 4).
  • number of sampled particles N
    Fixed to 64 for the population normalizer; affects both weight estimation and exploration.
  • diffusion steps and noise schedule
    Standard diffusion hyper-parameters (20 steps, cosine); ablated but still chosen by the authors.
axioms (4)
  • standard math The optimal policy under an f-divergence regularized objective admits the form of a (clipped) inverse-derivative reweighting of the behavior policy (Prop. 3.1).
    Standard Lagrange-multiplier derivation for convex f-divergences; appears in Sec. 3.1 and App. C.1.
  • standard math Reweighted conditional flow matching recovers the marginal velocity field of the reweighted data distribution (Thm. D.2).
    Extension of the usual CFM identity of Lipman et al.; used to justify the training objective (8).
  • ad hoc to paper Policies may be relaxed from probability measures to normalized signed measures while still guaranteeing policy improvement when the weight function is strictly increasing (Thm. 3.3).
    The signed-measure relaxation is the paper’s key modeling choice; the improvement proof is a covariance argument (App. C.3).
  • domain assumption Empirical population normalizers computed from a finite batch of particles adequately approximate the true partition function for linear and square weights.
    Required for practical Algorithm 1; detailed in App. E but not accompanied by concentration bounds.
invented entities (1)
  • virtual target policy measure (possibly signed) no independent evidence
    purpose: Serves as the intermediate object that the diffusion policy is matched to; allows negative reweighting.
    Constructed as the solution of the regularized objective over signed measures; no independent physical existence outside the optimization problem.

pith-pipeline@v1.1.0-grok45 · 24943 in / 3105 out tokens · 32446 ms · 2026-07-14T23:46:20.789577+00:00 · methodology

0 comments
read the original abstract

A commonly used family of RL algorithms for diffusion policies conducts softmax reweighting over samples from the behavior policy, which often induces an overgreedy policy and fails to utilize feedback from negative samples. In this work, we introduce GeMPO, a simple and unified framework that generalizes reweighting scheme in diffusion RL from softmax to general monotonic functions. GeMPO revisits diffusion RL via a measure matching perspective: First, we construct a virtual target policy measure via solving a regularized policy optimization objective; Second, we minimize the divergence between the current policy and this target measure through reweighted flow matching. This formulation offers two key advantages: i) It extends weight design beyond traditional exponential reweighting, allowing it to be tailored to diverse reward landscapes; and ii) by relaxing the non-negativity constraint on the target measure, our framework provides a principled justification for negative reweighting. We provide interpretations of how negative reweighting actively repels the policy from suboptimal actions and thus facilitates exploration. Extensive empirical evaluations demonstrate that GeMPO achieves competitive or superior performance by leveraging these flexible weighting schemes, and we provide practical guidelines for selecting reweighting methods in practice.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 34 linked inside Pith

  1. [1]

    Iterated denoising energy matching for sampling from boltzmann densities.arXiv preprint arXiv:2402.06121, 2024

    Tara Akhound-Sadegh, Jarrid Rector-Brooks, Avishek Joey Bose, Sarthak Mittal, Pablo Lemos, Cheng-Hao Liu, Marcin Sendera, Siamak Ravanbakhsh, Gauthier Gidel, Yoshua Bengio, et al. Iterated denoising energy matching for sampling from boltzmann densities.arXiv preprint arXiv:2402.06121, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    A general class of coefficients of divergence of one distribution from another.Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131–142, 1966

    Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another.Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131–142, 1966

    1966

  3. [3]

    Effective gene expression prediction from sequence by integrating long-range interactions

    Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska- Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10):1196–1203, 2021

    2021

  4. [4]

    Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301, 2023

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Minimization of � -divergences on sets of signed measures.Studia Scientiarum Mathematicarum Hungarica, 43(4):403–442, 2006

    Michel Broniatowski and Amor Keziou. Minimization of � -divergences on sets of signed measures.Studia Scientiarum Mathematicarum Hungarica, 43(4):403–442, 2006

    2006

  6. [6]

    Dime: Diffusion-based maximum entropy reinforcement learning

    Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palanicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion-based maximum entropy reinforcement learning. arXiv preprint arXiv:2502.02316, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Dime: Diffusion-based maximum entropy reinforcement learning

    Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion-based maximum entropy reinforcement learning. In International Conference on Machine Learning, pages 6958–6977. PMLR, 2025

    2025

  8. [8]

    Directly fine-tuning diffusion models on differentiable rewards.arXiv preprint arXiv:2309.17400, 2023

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards.arXiv preprint arXiv:2309.17400, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Information theory and statistics: A tutorial

    Imre Csiszár and Paul C Shields. Information theory and statistics: A tutorial. 2004

    2004

  10. [10]

    Mem pixel correlated solutions for generalized moment and interpolation problems.IEEE Transactions on Information Theory, 45(7):2253– 2270, 2002

    Imre Csiszár, F Gamgoa, and Elisabeth Gassiat. Mem pixel correlated solutions for generalized moment and interpolation problems.IEEE Transactions on Information Theory, 45(7):2253– 2270, 2002

    2002

  11. [11]

    Diffusion-based reinforcement learning via q-weighted variational policy optimization

    Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv:2405.16173, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Genpo: Generative diffusion models meet on-policy reinforcement learning.arXiv preprint arXiv:2505.18763, 2025

    Shutong Ding, Ke Hu, Shan Zhong, Haoyang Luo, Weinan Zhang, Jingya Wang, Jun Wang, and Ye Shi. Genpo: Generative diffusion models meet on-policy reinforcement learning.arXiv preprint arXiv:2505.18763, 2025

    arXiv 2025

  13. [13]

    Maximum entropy reinforcement learning with diffusion policy.arXiv preprint arXiv:2502.11612, 2025

    Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy.arXiv preprint arXiv:2502.11612, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023

    2023

  15. [15]

    Diffusion guidance is a controllable policy improvement operator.arXiv preprint arXiv:2505.23458, 2025

    Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a controllable policy improvement operator.arXiv preprint arXiv:2505.23458, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InInternational conference on machine learning, pages 1587–1596. PMLR, 2018

    2018

  17. [17]

    Behavior-regularized diffusion policy optimization for offline reinforcement learning

    Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Chenjun Xiao, Yang Yu, and Zongzhang Zhang. Behavior-regularized diffusion policy optimization for offline reinforcement learning. InForty- second International Conference on Machine Learning, 2025. 11

    2025

  18. [18]

    Flowrl: A taxonomy and modular framework for reinforcement learning with diffusion policies.arXiv preprint arXiv:2603.27450, 2026

    Chenxiao Gao, Edward Chen, Tianyi Chen, and Bo Dai. Flowrl: A taxonomy and modular framework for reinforcement learning with diffusion policies.arXiv preprint arXiv:2603.27450, 2026

    arXiv 2026

  19. [19]

    Machine-guided design of synthetic cell type-specific cis-regulatory elements.bioRxiv, 2023

    Sager J Gosai, Rodrigo I Castro, Natalia Fuentes, John C Butts, Susan Kales, Ramil R Noche, Kousuke Mouri, Pardis C Sabeti, Steven K Reilly, and Ryan Tewhey. Machine-guided design of synthetic cell type-specific cis-regulatory elements.bioRxiv, 2023

    2023

  20. [20]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. PMLR, 2018

    2018

  21. [21]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Pith/arXiv arXiv 2022

  22. [22]

    Denoising Diffusion Probabilistic Models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020

    2020

  23. [23]

    Rl-100: Performant robotic manipulation with real-world reinforcement learning.arXiv preprint arXiv:2510.14830, 2025

    Kun Lei, Huanyu Li, Dongjie Yu, Zhenyu Wei, Lingxiao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, and Huazhe Xu. Rl-100: Performant robotic manipulation with real-world reinforcement learning.arXiv preprint arXiv:2510.14830, 2025

    arXiv 2025

  24. [24]

    Back to basics: Let denoising generative models denoise.arXiv preprint arXiv:2511.13720, 2025

    Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise.arXiv preprint arXiv:2511.13720, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Reverse flow matching: A unified frame- work for online reinforcement learning with diffusion and flow policies.arXiv preprint arXiv:2601.08136, 2026

    Zeyang Li, Sunbochen Tang, and Navid Azizan. Reverse flow matching: A unified frame- work for online reinforcement learning with diffusion and flow policies.arXiv preprint arXiv:2601.08136, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    On divergences and informations in statistics and information theory.IEEE Transactions on Information Theory, 52(10):4394–4412, 2006

    Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory.IEEE Transactions on Information Theory, 52(10):4394–4412, 2006

    2006

  27. [27]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  28. [28]

    Flow matching guide and code.arXiv preprint arXiv:2412.06264, 2024

    Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code.arXiv preprint arXiv:2412.06264, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834, 2023

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. InInternational Conference on Machine Learning, pages 22825–22855. PMLR, 2023

    2023

  33. [33]

    Efficient online reinforcement learning for diffusion policy.arXiv preprint arXiv:2502.00361, 2025

    Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy.arXiv preprint arXiv:2502.00361, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Reinforcement learning with discrete diffusion policies for combinatorial action spaces.arXiv preprint arXiv:2509.22963, 2025

    Haitong Ma, Ofir Nabati, Aviv Rosenberg, Bo Dai, Oran Lang, Idan Szpektor, Craig Boutilier, Na Li, Shie Mannor, Lior Shani, et al. Reinforcement learning with discrete diffusion policies for combinatorial action spaces.arXiv preprint arXiv:2509.22963, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Escaping the gravitational pull of softmax.Advances in Neural Information Processing Systems, 33:21130–21140, 2020

    Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Escaping the gravitational pull of softmax.Advances in Neural Information Processing Systems, 33:21130–21140, 2020. 12

    2020

  36. [36]

    Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020

    Pith/arXiv arXiv 2006

  37. [37]

    Unlocking guidance for discrete state-space diffusion and flow models.arXiv preprint arXiv:2406.01572, 2024

    Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models.arXiv preprint arXiv:2406.01572, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    Pith/arXiv arXiv 1910

  39. [39]

    � ∗ ���: a vla that learns from experience.arXiv preprint arXiv:2511.14759, 2025

    Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, et al. � ∗ ���: a vla that learns from experience.arXiv preprint arXiv:2511.14759, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Video diffusion alignment via reward gradients.arXiv preprint arXiv:2407.08737, 2024

    Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients.arXiv preprint arXiv:2407.08737, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Learning a diffusion model policy from rewards via q-score matching.arXiv preprint arXiv:2312.11752, 2023

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching.arXiv preprint arXiv:2312.11752, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    John Wiley & Sons, 2014

    Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  43. [43]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  44. [44]

    Diffusion policy policy optimization.arXiv preprint arXiv:2409.00588, 2024

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization.arXiv preprint arXiv:2409.00588, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Trust region policy optimization.arXiv preprint arXiv:1502.05477, 2015

    John Schulman. Trust region policy optimization.arXiv preprint arXiv:1502.05477, 2015

    Pith/arXiv arXiv 2015

  46. [46]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  47. [47]

    Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps

    Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 5668–5675, 2020

    2020

  48. [48]

    Simplified and generalized masked diffusion for discrete data.Advances in neural information processing systems, 37:103131–103167, 2024

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data.Advances in neural information processing systems, 37:103131–103167, 2024

    2024

  49. [49]

    Generative modeling by estimating gradients of the data distribution.Advances in neural information processing systems, 32, 2019

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.Advances in neural information processing systems, 32, 2019

    2019

  50. [50]

    wd1: Weighted policy optimization for reasoning in diffusion language models.arXiv preprint arXiv:2507.08838, 2025

    Xiaohang Tang, Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. wd1: Weighted policy optimization for reasoning in diffusion language models.arXiv preprint arXiv:2507.08838, 2025

    arXiv 2025

  51. [51]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    Pith/arXiv arXiv 2018

  52. [52]

    Mirror descent policy optimization, 2021

    Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization, 2021. URL��������������������������������

    2021

  53. [53]

    Leverage the average: an analysis of kl regularization in rl.arXiv preprint arXiv:2003.14089, 2020

    Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in rl.arXiv preprint arXiv:2003.14089, 2020. 13

    Pith/arXiv arXiv 2003

  54. [54]

    Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design.arXiv preprint arXiv:2410.13643, 2024

    Chenyu Wang, Masatoshi Uehara, Yichun He, Amy Wang, Tommaso Biancalani, Avantika Lal, Tommi Jaakkola, Sergey Levine, Hanchen Wang, and Aviv Regev. Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design.arXiv preprint arXiv:2410.13643, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    Diffusion Actor-Critic with Entropy Regulator, December 2024

    Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, and Shengbo Eben Li. Diffusion Actor-Critic with Entropy Regulator, December 2024

    2024

  56. [56]

    Enhanced dacer algorithm with high diffusion efficiency.arXiv preprint arXiv:2505.23426, 2025

    Yinuo Wang, Likun Wang, Mining Tan, Wenjun Zou, Xujie Song, Wenxuan Wang, Tong Liu, Guojian Zhan, Tianze Zhu, Shiqi Liu, et al. Enhanced dacer algorithm with high diffusion efficiency.arXiv preprint arXiv:2505.23426, 2025

    arXiv 2025

  57. [57]

    Diffusion policies as an expressive policy class for offline reinforcement learning.arXiv preprint arXiv:2208.06193, 2022

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning.arXiv preprint arXiv:2208.06193, 2022

    Pith/arXiv arXiv 2022

  58. [58]

    Practi- cal and asymptotically exact conditional sampling in diffusion models.Advances in Neural Information Processing Systems, 36:31372–31403, 2023

    Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practi- cal and asymptotically exact conditional sampling in diffusion models.Advances in Neural Information Processing Systems, 36:31372–31403, 2023

    2023

  59. [59]

    Offline rl with no ood actions: In-sample learning via implicit value regularization.arXiv preprint arXiv:2303.15810, 2023

    Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization.arXiv preprint arXiv:2303.15810, 2023

    Pith/arXiv arXiv 2023

  60. [60]

    Uni-rl: Unifying online and offline rl via implicit value regularization

    Haoran Xu, Liyuan Mao, Hui Jin, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Uni-rl: Unifying online and offline rl via implicit value regularization. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  61. [61]

    Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    Energy-weighted flow matching for offline reinforcement learning.arXiv preprint arXiv:2503.04975, 2025

    Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning.arXiv preprint arXiv:2503.04975, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    Reinflow: Fine-tuning flow matching policy with online reinforcement learning.arXiv preprint arXiv:2505.22094, 2025

    Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching policy with online reinforcement learning.arXiv preprint arXiv:2505.22094, 2025

    arXiv 2025

  64. [64]

    Sac flow: Sample-efficient reinforcement learning of flow-based policies via velocity-reparameterized sequential modeling.arXiv preprint arXiv:2509.25756, 2025

    Yixian Zhang, Shu’ang Yu, Tonghe Zhang, Mo Guang, Haojia Hui, Kaiwen Long, Yu Wang, Chao Yu, and Wenbo Ding. Sac flow: Sample-efficient reinforcement learning of flow-based policies via velocity-reparameterized sequential modeling.arXiv preprint arXiv:2509.25756, 2025

    arXiv 2025

  65. [65]

    flat” landscape to “steep

    Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. Diffusionnft: Online diffusion reinforcement with forward process.arXiv preprint arXiv:2509.16117, 2025. 14 A Related works RL for diffusion and flow models.RL for diffusion or flow models can be classified into two categories:...

    Pith/arXiv arXiv 2025