REVIEW 5 major objections 5 minor 96 references
Decision Flow Policy Optimization
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Decision Flow treats every intermediate velocity of a flow policy as a decision, so offline RL optimizes each generation step directly.
desk verdict Impressive D4RL results and a sensible flow-MDP idea, but the theoretical core is internally inconsistent and needs a major rewrite before the claims can be believed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the flow MDP, an inner decision process built on the flow model's ODE rollout, together with the flow value functions $Q^f$ and $V^f$ that score intermediate velocity decisions. The consistency identity $Q^{f*}(s, a_t, u_t) = V^{f*}(s, a_t, \hat{u}_t) = Q^*(s, a_1)$ connects every intermediate flow state to the final action value, and the policy objective maximizes $Q^f$ while matching the behavior flow policy through a divergence term. This identity is what lets the method propagate final-action value signals back through all $T$ generation steps.
What would settle it
One concrete check: build a two-mode offline dataset in which the optimal action is reachable only by an intermediate direction that the behavior flow policy never takes, train Decision Flow with the regression losses of Lemma 4.1, and test whether the policy discovers that action; Theorem 4.1 predicts it should, while the training objective only ever evaluates directions seen under the current policy.
Extended reading notes
Core claim
The paper's central claim is that the generation process of a flow policy can be reorganized as a flow MDP, and that optimizing flow value functions on intermediate velocities yields an optimal flow policy. Concretely, it defines the flow state as (s, a_t), the flow action as the velocity field u_theta(s, a_t, t), and the flow reward as zero on intermediate steps and Q(s, a_1) at the final step. Two implementations follow: Direction-Oriented Decision Flow aligns every intermediate flow value with the final action value, and Divergence-Oriented Decision Flow uses divergence from a behavior flow policy as intermediate reward in a nested MDP. Theorems 4.1 and 4.2 assert convergence to optimal actions and optimal flow policy under Q convergence, sufficient capacity, a unique Q maximum, and small generation step size.
Load-bearing premise
The argument depends on the flow value functions $Q^f$ and $V^f$ correctly evaluating the return of arbitrary intermediate velocity directions, even though they are trained by regression onto $Q(s, a_1)$ using only data from the current policy's generation path.
Editorial extensions
If this is right
- Flow policies can be trained with one objective that fits multi-modal action distributions and improves returns simultaneously, removing the separate policy optimizer used by prior flow and diffusion RL methods.
- Because flow value functions assign values to intermediate velocities, a policy can be corrected at any point of its ODE rollout, not only at the final action.
- The two variants give a concrete tradeoff: DF-dir propagates final Q directly to every step, while DF-div keeps the policy near the behavior flow through divergence rewards, which is useful for suboptimal or narrow datasets.
- If the convergence results transfer to practice, flow policies should match or beat diffusion and transformer baselines on benchmarks with multi-modal action distributions, which the paper demonstrates on D4RL.
Reading between the lines
- A natural extension is to train flow value functions with off-policy or counterfactual intermediate directions, so they can evaluate velocity choices the current policy never generates; this would test whether the regression objective alone is enough for the theory's assumptions.
- The flow MDP formulation likely transfers to other iterative samplers, such as diffusion denoising or rectified flow, where each denoising step could be scored by a stepwise value function.
- A concrete diagnostic is whether performance gains concentrate on tasks where optimal actions differ from behavior actions by more than one ODE step; if gains appear even when a single step suffices, the intermediate-value mechanism may not be the cause.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Decision Flow (arXiv:2505.20350) proposes to model the iterative action-generation process of a flow-based policy as a Markov decision process over flow time steps. The paper defines two variants: DF-dir, in which flow value functions are regressed onto the conventional action-value function, and DF-div, in which intermediate flow rewards penalize divergence from a behavior flow policy. The main theoretical claim is that iterated flow-policy evaluation and improvement converge to an optimal policy (Theorem 4.1, Theorem 4.2), and the paper reports D4RL experiments on Gym-MuJoCo, Adroit, and Pointmaze against 30+ baselines.
Significance. If the central convergence results were valid, Decision Flow would be a meaningful advance: it would let a flow policy be optimized at intermediate generation steps rather than treating the generator as a fixed behavior model, and the reported empirical comparison is broad. The two variants are clearly separated, the ablations in Section 5.5 isolate the contribution of the flow value functions, and Table 1 reports variance for the proposed methods. However, the theoretical core is the paper's central claim, and it contains load-bearing gaps and internal inconsistencies (detailed below). Without a valid proof of Theorems 4.1 and 4.2, the paper is an empirical algorithm description whose formal claims do not follow; no code or machine-checked proof is provided to compensate. For these reasons I do not regard the main result as established.
major comments (5)
- [4.2, Lemma 4.1 (Eqs. 10-12)] The critic-consistency lemma cannot follow from the stated regression objectives. Equations (10) and (11) regress Qf and Vf onto the scalar Q(s,a1) using data drawn from the current or behavior flow policy. An MSE fit only determines these functions on the sampled support; it provides no supervision for a different intermediate state (s,at) or velocity u_t, so the 'for all t, at, ut' equality in Eq. (12) is an extrapolation, not a consequence of the loss. Moreover, Eq. (12) is functionally inconsistent for a fixed final action a1: a1 in Q*(s,a1) is itself the result of integrating the velocity u_t from at, so the value of an intermediate decision should depend on at and on the magnitude of u_t; requiring Qf*(s,at,lambda*uhat)=Vf*(s,at,uhat)=Qf*(s,at,lambda'*uhat) whenever lambda differs from lambda' forces Q*(s,a1) to be independent of the first Euler step, which is generically false. Lemma 4.1 is therefore not a valid premise for Lemma 4.2 or Theorem 4.1.
- [Appendix E, Lemma E.2 (Eqs. 39-46)] The direction-optimality proof does not establish the 'if and only if' claim. The Lagrangian stationarity condition grad F = eta*uhat only says that stationary points occur when grad F is parallel to uhat; it does not say that the direction is aligned with grad Q*, nor that such a point is a maximum. The identification of grad F with h_lambda * (grad_{at} Q*(s, Gamma(at)))^top J_Gamma(at) requires J_Gamma(at)=I+O(h_lambda), which is assumed via a Peano-Baker estimate rather than derived under the actual training distribution, and the gradient of Vf* with respect to uhat is not supervised by Eq. (11) because Vf* is regressed only on the scalar Q(s,a1) and never evaluated at alternative directions. Thus the key gradient connection between the flow value function and Q* is missing.
- [Appendix E, Theorem E.2 (Eqs. 48-52)] The proof of the practical DF-dir objective contradicts Lemma 4.1. Lemma 4.1 asserts Qf* = Vf* = Q*(s,a1) for every intermediate at and ut, while Eqs. (50)-(51) assert Vf* is approximately Q*(s,at) and Qf* is approximately Q*(s,at)+h_lambda*grad Q* . uhat, so the surrogate Qf*-Vf* used in Eq. (14) is first order in h_lambda and not identically zero. These two expansions are mutually inconsistent unless at = a1 or h_lambda = 0. Since Eq. (14) and Theorem E.2 are built on the difference Qf*-Vf*, the gradient signal used in the algorithm is not justified by the stated theory, and the claim that ascending the surrogate performs stochastic gradient ascent on Q* does not follow.
- [Appendix F, Lemma F.3 / Theorem F.1] The DF-div policy-improvement proof uses Vf_chi^old inside a Bellman recursion under the new policy, which is invalid. In the displayed chain of inequalities, the term Vf_chi^old(a_{t+Delta}) is replaced by -D(unew||uv)+Vf_chi^old(a_{t+2Delta}); but Vf_chi^old was trained to satisfy the Bellman equation for the old policy uold, not for unew, so the replacement does not follow from Eq. (21). The final inequality J_old <= J_new is therefore not established. Since Theorem 4.2 rests entirely on Lemmas F.2 and F.3, the convergence claim for DF-div is unproven as stated.
- [Theorem 4.1] Even if Lemmas 4.1-4.3 were accepted, Theorem 4.1 does not show convergence to a*_1. Monotone improvement of Q*(s,at) along the ODE gives a nondecreasing sequence bounded by Q*(s,a*_1), but the proof does not establish that the limit is the unique maximum, nor that the policy parameters theta converge independently of the sampled initial a0. In addition, the assumption 'Q -> Q*' is not a consequence of the IQL-style training in Eqs. (23)-(25); offline value learning only approximates Q under distributional constraints, so the optimality conclusion inherits an unverified premise.
minor comments (5)
- [4.2, Eq. (14)] The expression D(u_theta||u_v) = ||u_theta-u_v||_2 is called a divergence, but it is an L2 distance rather than a statistical divergence; the terminology should be corrected or replaced.
- [Appendix D] The presentation in Appendix D concludes equality of the flow-matching and conditional flow-matching losses, but the derivation omits additive constants and should instead state the standard result that the two objectives provide the same gradient.
- [Appendix B.1, Table 3] Table 3 reports results only for DF-div, while the abstract and Section 5.4 claim the method generally; the paper should either include DF-dir on Pointmaze or qualify the claim.
- [Table 2] Table 2 lacks standard deviations and contains many missing entries, which makes the claimed approximate 16% gain over the best flow-based baselines difficult to verify.
- [Algorithm 1, line 12] In the DF-div branch, the algorithm trains Vf_Omega, but the loss in Eq. (21) and the surrounding text concern Vf_chi; this notational mismatch should be fixed.
Circularity Check
DF-dir's claimed policy-improvement signal is defined as a regression to Q and collapses to zero under the paper's own Lemma 4.1; it is then revived only by an assumed Taylor expansion, making Theorem 4.1 reduce by construction to behavior cloning.
-
self definitional
[Section 4.2, Lemma 4.1 and Eqs. (10)-(12); Appendix E, Lemma E.1]
"If Q → Q∗, where Q∗ is the optimal conventional critic and Qf and V f with sufficient model capacity, and the objectives LQf and LV f is defined as LQf = E[||QfΨ − Q||2], LV f = E[||V fΩ − Q||2]. Then, we will conclude that Qf∗(s, at, ut) = V f∗(s, at, ût) = Q∗(s, a1), ∀t, at."
The 'convergence' of the flow critics is not obtained from Bellman backups over intermediate velocities; Qf and Vf are defined as MSE regressions onto the terminal value Q(s,a1). Lemma 4.1/E.1 therefore restates the regression target: with perfect optimization the outputs equal Q(s,a1) by construction. The appendix proof says exactly this: 'With perfect optimization, the MSE loss forces the outputs to equal the target Q(s,a1).' This lemma is then used as the foundation of Lemmas 4.2-4.3 and Theorem 4.1, so the flow value functions' 'consistency' is the input definition, not a derived result about intermediate actions.
-
other
[Appendix E, Theorem E.2 and Lemma E.2]
"Recall from Lemma E.1 that the difference between the flow Qf function and the flow V f function is zero at the optimum Q∗. ... V f ∗ = Q∗(s, at) + O((hλ)2), Qf ∗ = Q∗(s, at) + hλ∇aQ∗(s, at)ˆu + O((hλ)2), which indicates that ∇θJ f∗ θ = ∇θE[Qf ∗ − V f ∗] ≈ E[hλ∇at Q∗(s, at) ∂at ∂θ]."
Immediately after Lemma E.1 declared Qf*=Vf*=Q*(s,a1) for every velocity, Theorem E.2 gives Qf*-Vf* a nonzero first-order term hλ∇aQ*(s,at)·û. That directional dependence is exactly the property needed to conclude that intermediate velocities point toward higher-return regions; it is never implied by the regression losses Eqs. (10)-(11), whose target Q(s,a1) is constant with respect to the direction û for a matched sample. Lemma E.2 makes the same move by Taylor-expanding Vf* in ût and differentiating it, i.e., it assumes the conclusion it is supposed to prove.
1 more flagged steps
-
fitted input called prediction
[Section 4.2, Eq. (14)]
"min LDF −dir uθ = −Es,a,t∼U (0,1),at=ϕt(at|a)[Qf Ψ(s, at, uθ(s, at, t)) − V f Ω(s, at, ûθ(s, at, t))] + ρ ∗ D(uθ(s, at, t)||uv(s, at, t))"
This is the method's policy-improvement objective. Under the paper's own Lemma 4.1, at convergence Qf*=Vf*=Q*(s,a1), so the advantage term Qf-Vf is identically zero for every velocity and the objective reduces to ρD(uθ||uv), i.e., behavior cloning of the dataset flow policy. The paper's abstract claims the framework 'seamlessly optimizes the flow policy while capturing multi-modal action distributions'; by its own equations, the optimization signal is either zero (if Lemma 4.1 is accepted) or reintroduced by the contradictory Taylor expansion in Theorem E.2. The 'prediction' that the flow policy is improved is therefore fitted into the definition rather than derived.
full rationale
The central claim of the paper is Theorem 4.1: 'the generated actions by flow policy converge to the optimal actions a∗1 and the flow policy uθ is optimal.' For the direction-oriented variant this claim reduces to the definitions. Qf and Vf are not learned by any Bellman evaluation over alternative intermediate velocities; they are regressed directly onto the scalar Q(s,a1) (Eqs. 10-11), so Lemma 4.1's equality is a restatement of the fit. The proof then needs a nonzero Qf-Vf advantage to drive improvement, but Lemma 4.1 makes that advantage zero; the only way the proof obtains a nonzero signal is the Taylor expansion in Lemma E.2 and Theorem E.2, which silently assumes that Vf* varies with direction in the way Q* does. That variation is precisely what the regression objective never supervises, so the derivation assumes its own conclusion. Equation (14) confirms the collapse: with exact critics the first term vanishes and DF-dir is behavior cloning. This is central, not marginal, because DF-dir is the method whose proofs are advertised as 'rigorous proofs of Decision Flow' and whose results include the headline optimality theorem. No load-bearing self-citation issue is present; the circularity is internal to the equations. The empirical D4RL comparisons are independent benchmarks, but they do not repair the derivation. DF-div's policy-iteration proof is closer to a standard Bellman recursion with divergence rewards and is less affected, so the score reflects partial (central-claim) circularity rather than complete vacuity.
Assumptions & free parameters
free parameters (3)
- behavior tradeoff rho =
1.0
- flow time steps T =
10
- expectile weight tau =
0.5
assumptions (5)
- standard math The flow matching loss L_fm equals the conditional flow matching loss L_cfm in gradient.
- domain assumption The flow map Gamma is smooth, invertible, and L-Lipschitz, so its Jacobian J_Gamma(a_t) = I + O(h_lambda).
- ad hoc to paper The conventional Q function converges to the optimal Q* in the offline setting.
- ad hoc to paper Flow value functions have sufficient model capacity and perfectly minimize the MSE regression objectives.
- domain assumption The generation step size h_lambda is small enough for the Taylor expansions to be accurate.
invented entities (1)
-
Flow MDP
Cite this review
Pith. "Pith review of Decision Flow Policy Optimization." pith.science (2026). https://pith.science/paper/LBUE6WJT
@misc{pith2026250520350,
author = {Pith},
title = {Pith review of: Decision Flow Policy Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/LBUE6WJT}},
note = {Machine review of arXiv:2505.20350}
}
read the original abstract
In recent years, generative models have shown remarkable capabilities across diverse fields, including images, videos, language, and decision-making. By applying powerful generative models such as flow-based models to reinforcement learning, we can effectively model complex multi-modal action distributions and achieve superior robotic control in continuous action spaces, surpassing the limitations of single-modal action distributions with traditional Gaussian-based policies. Previous methods usually adopt the generative models as behavior models to fit state-conditioned action distributions from datasets, with policy optimization conducted separately through additional policies using value-based sample weighting or gradient-based updates. However, this separation prevents the simultaneous optimization of multi-modal distribution fitting and policy improvement, ultimately hindering the training of models and degrading the performance. To address this issue, we propose Decision Flow, a unified framework that integrates multi-modal action distribution modeling and policy optimization. Specifically, our method formulates the action generation procedure of flow-based models as a flow decision-making process, where each action generation step corresponds to one flow decision. Consequently, our method seamlessly optimizes the flow policy while capturing multi-modal action distributions. We provide rigorous proofs of Decision Flow and validate the effectiveness through extensive experiments across dozens of offline RL environments. Compared with established offline RL baselines, the results demonstrate that our method achieves or matches the SOTA performance.
Figures
Reference graph
Works this paper leans on
-
[1]
Diffusion policies for out-of-distribution generalization in offline reinforcement learning
Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement learning. IEEE Robotics and Automation Letters, 9(4): 3116–3123, 2024
2024
-
[2]
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022
arXiv 2022
-
[3]
Let offline rl flow: Training conservative agents in the latent space of normalizing flows
Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline rl flow: Training conservative agents in the latent space of normalizing flows. arXiv preprint arXiv:2211.11096, 2022
arXiv 2022
-
[4]
Uncertainty-based offline reinforcement learning with diversified q-ensemble
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436–7447, 2021
2021
-
[5]
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. arXiv preprint arXiv:2008.05556, 2020
arXiv 2008
-
[6]
The peano-baker series
Michael Baake and Ulrike Schlaegel. The peano-baker series. Proceedings of the Steklov Institute of Mathematics, 275(1):155–159, 2011
2011
-
[7]
Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11566, 2022
arXiv 2022
-
[8]
Training diffusion models with reinforcement learning
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023
arXiv 2023
Show all 96 references
-
[9]
π0: A vision-language-action flow model for general robot control
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024
-
[10]
Reinforcement learning for generative ai: A survey
Yuanjiang Cao, Quan Z Sheng, Julian McAuley, and Lina Yao. Reinforcement learning for generative ai: A survey. arXiv preprint arXiv:2308.14328, 2023
2023 arXiv
-
[11]
Simple hierarchi- cal planning with diffusion
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchi- cal planning with diffusion. arXiv preprint arXiv:2401.02644, 2024
2024 arXiv
-
[12]
Offline reinforcement learning via high-fidelity generative behavior modeling
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548, 2022
2022 arXiv
-
[13]
Deep generative models for offline policy learning: Tutorial, survey, and perspectives on future directions
Jiayu Chen, Bhargav Ganguly, Yang Xu, Yongsheng Mei, Tian Lan, and Vaneet Aggarwal. Deep generative models for offline policy learning: Tutorial, survey, and perspectives on future directions. arXiv preprint arXiv:2402.13777, 2024
2024 arXiv
-
[14]
Decision transformer: Reinforcement learning via sequence modeling
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021
2021
-
[15]
Diffusion policy: Visuomotor policy learning via action diffusion
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023
2023 arXiv
-
[16]
Flow matching in latent space
Quan Dao, Hao Phung, Binh Nguyen, and Anh Tran. Flow matching in latent space. arXiv preprint arXiv:2307.08698, 2023
2023 arXiv
-
[17]
Fisher flow matching for generative modeling over discrete data
Oscar Davis, Samuel Kessler, Mircea Petrache, Ismail Ceylan, Michael Bronstein, and Joey Bose. Fisher flow matching for generative modeling over discrete data. Advances in Neural Information Processing Systems, 37:139054–139084, 2024. 10
2024
-
[18]
Diffusion-based reinforcement learning via q-weighted variational policy optimization
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv:2405.16173, 2024
2024 arXiv
-
[19]
Diffuserlite: Towards real-time diffusion planning
Zibin Dong, Jianye Hao, Yifu Yuan, Fei Ni, Yitian Wang, Pengyi Li, and Yan Zheng. Diffuserlite: Towards real-time diffusion planning. arXiv preprint arXiv:2401.15443, 2024
2024 arXiv
-
[20]
Probabilistic number theory I: Mean-value theorems, volume 239
Peter DTA Elliott. Probabilistic number theory I: Mean-value theorems, volume 239. Springer Science & Business Media, 2012
2012
-
[21]
Scaling rectified flow trans- formers for high-resolution image synthesis
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machin...
2024
-
[22]
A reinforcement learning diffusion decision model for value-based decisions
Laura Fontanesi, Sebastian Gluth, Mikhail S Spektor, and Jörg Rieskamp. A reinforcement learning diffusion decision model for value-based decisions. Psychonomic bulletin & review, 26(4):1099–1121, 2019
2019
-
[23]
Reinforcement learning for generative ai: State of the art, opportunities and open research challenges
Giorgio Franceschelli and Mirco Musolesi. Reinforcement learning for generative ai: State of the art, opportunities and open research challenges. Journal of Artificial Intelligence Research, 79:417–446, 2024
2024
-
[24]
D4rl: Datasets for deep data-driven reinforcement learning
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020
2004 arXiv
-
[25]
A minimalist approach to offline reinforcement learning
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021
2021
-
[26]
Off-policy deep reinforcement learning without exploration
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052–2062. PMLR, 2019
2019
-
[27]
Generalized decision transformer for offline hindsight information matching
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10364, 2021
2021 arXiv
-
[28]
Behavior-regularized diffusion policy optimization for offline reinforcement learning
Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Chenjun Xiao, Yang Yu, and Zongzhang Zhang. Behavior-regularized diffusion policy optimization for offline reinforcement learning. arXiv preprint arXiv:2502.04778, 2025
2025 arXiv
-
[29]
Discrete flow matching
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37:133345–133385, 2024
2024
-
[30]
Offline rl policies should be trained to be adaptive
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pages 7513–7530. PMLR, 2022
2022
-
[31]
Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr, 2018
2018
-
[32]
Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023
2023 arXiv
-
[33]
Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems, 36:64896–64917, 2023
2023
-
[34]
Lectures on Lipschitz analysis
Juha Heinonen. Lectures on Lipschitz analysis. Number 100. University of Jyväskylä, 2005. 11
2005
-
[35]
Flow++: Improving flow-based generative models with variational dequantization and architecture design
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International conference on machine learning, pages 2722–2730. PMLR, 2019
2019
-
[36]
Instructed diffuser with temporal condition guidance for offline reinforcement learning
Jifeng Hu, Yanchao Sun, Sili Huang, SiYuan Guo, Hechang Chen, Li Shen, Lichao Sun, Yi Chang, and Dacheng Tao. Instructed diffuser with temporal condition guidance for offline reinforcement learning. arXiv preprint arXiv:2306.04875, 2023
2023 arXiv
-
[37]
On transforming reinforce- ment learning by transformer: The development trajectory
Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao. On transforming reinforce- ment learning by transformer: The development trajectory. arXiv preprint arXiv:2212.14164, 2022
2022 arXiv
-
[38]
Graph decision transformer
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. arXiv preprint arXiv:2303.03747, 2023
2023 arXiv
-
[39]
Adaflow: Imitation learning with variance- adaptive flow-based policies
Xixi Hu, Qiang Liu, Xingchao Liu, and Bo Liu. Adaflow: Imitation learning with variance- adaptive flow-based policies. Advances in Neural Information Processing Systems, 37:138836– 138858, 2024
2024
-
[40]
Diffusion models as optimizers for efficient planning in offline rl
Renming Huang, Yunqiang Pei, Guoqing Wang, Yangming Zhang, Yang Yang, Peng Wang, and Hengtao Shen. Diffusion models as optimizers for efficient planning in offline rl. In European Conference on Computer Vision, pages 1–17. Springer, 2024
2024
-
[41]
Offline reinforcement learning as one big sequence modeling problem
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273– 1286, 2021
2021
-
[42]
Planning with diffusion for flexible behavior synthesis
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022
2022 arXiv
-
[43]
Efficient planning in a compact latent action space
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in a compact latent action space. arXiv preprint arXiv:2208.10291, 2022
2022 arXiv
-
[44]
Pyramidal flow matching for efficient video generative modeling
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954, 2024
2024
-
[45]
Efficient diffusion policies for offline reinforcement learning
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 67195–67212, 2023
2023
-
[46]
Morel: Model-based offline reinforcement learning
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810–21823, 2020
2020
-
[47]
Offline reinforcement learning with implicit q-learning
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021
2021 arXiv
-
[48]
Stabilizing off- policy q-learning via bootstrapping error reduction
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019
2019
-
[49]
Conservative q-learning for offline reinforcement learning
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179–1191, 2020
2020
-
[50]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020
2005 arXiv
-
[51]
Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching
Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, and Weinan Zhang. Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching. arXiv preprint arXiv:2402.02439, 2024. 12
2024 arXiv
-
[52]
Learning multimodal behaviors from scratch with diffusion policy gradient
Steven Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neural Information Processing Systems, 37:38456–38479, 2024
2024
-
[53]
Efficient planning with latent diffusion
Wenhao Li. Efficient planning with latent diffusion. arXiv preprint arXiv:2310.00311, 2023
2023 arXiv
-
[54]
Hierarchical diffusion for offline decision making
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning , pages 20035–20064. PMLR, 2023
2023
-
[55]
Generative models in decision making: A survey
Yinchuan Li, Xinyu Shao, Jianping Zhang, Haozhi Wang, Leo Maxime Brunswic, Kaiwen Zhou, Jiqian Dong, Kaiyang Guo, Xiu Li, Zhitang Chen, et al. Generative models in decision making: A survey. arXiv preprint arXiv:2502.17100, 2025
2025
-
[56]
Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis
Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis. In Proceedings of the IEEE/CVF Conference on Computer Visio...
2024
-
[57]
Adaptdif- fuser: Diffusion models as adaptive self-evolving planners
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdif- fuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877, 2023
2023 arXiv
-
[58]
Dataset distillation for offline reinforcement learning
Jonathan Light, Yuanzhe Liu, and Ziniu Hu. Dataset distillation for offline reinforcement learning. arXiv preprint arXiv:2407.20299, 2024
2024
-
[59]
Flow matching for generative modeling
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022
2022 arXiv
-
[60]
Flow matching guide and code
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.06264, 2024
2024 arXiv
-
[61]
Generative pre-training for speech with flow matching
Alexander H Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Generative pre-training for speech with flow matching. arXiv preprint arXiv:2310.16338, 2023
2023 arXiv
-
[62]
Selfbc: Self behavior cloning for offline reinforcement learning
Shirong Liu, Chenjia Bai, Zixian Guo, Hao Zhang, Gaurav Sharma, and Yang Liu. Selfbc: Self behavior cloning for offline reinforcement learning. arXiv preprint arXiv:2408.02165, 2024
2024 arXiv
-
[63]
Uncertainty- aware reward model: Teaching reward models to know what is unknown
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty- aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847, 2024
2024 arXiv
-
[64]
Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825–22855. PMLR, 2023
2023
-
[65]
Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning. arXiv preprint arXiv:2407.20109, 2024
2024 arXiv
-
[66]
Awac: Accelerating online reinforcement learning with offline datasets
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020
2006 arXiv
-
[67]
Normalizing flows for probabilistic modeling and inference
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1–64, 2021
2021
-
[68]
Flow q-learning
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arXiv:2502.02538, 2025
2025 arXiv
-
[69]
Advantage-weighted regression: Simple and scalable off-policy reinforcement learning
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. 13
1910 arXiv
-
[70]
Learning complex dexterous manipulation with deep reinforcement learning and demonstrations
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017
2017 arXiv
-
[71]
Flowar: Scale-wise autoregressive image generation meets flow matching
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Flowar: Scale-wise autoregressive image generation meets flow matching. arXiv preprint arXiv:2412.15205, 2024
2024 arXiv
-
[72]
Offline reinforcement learning as anti-exploration
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8106–8114, 2022
2022
-
[73]
Flow matching imitation learning for multi-support manipulation
Quentin Rouxel, Andrea Ferrari, Serena Ivaldi, and Jean-Baptiste Mouret. Flow matching imitation learning for multi-support manipulation. In 2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids), pages 528–535. IEEE, 2024
2024
-
[74]
Universal value density estimation for imitation learning and goal-conditioned reinforcement learning
Yannick Schroecker and Charles Isbell. Universal value density estimation for imitation learning and goal-conditioned reinforcement learning. arXiv preprint arXiv:2002.06473, 2020
2002 arXiv
-
[75]
Video prediction by modeling videos as continu- ous multi-dimensional processes
Gaurav Shrivastava and Abhinav Shrivastava. Video prediction by modeling videos as continu- ous multi-dimensional processes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7236–7245, 2024
2024
-
[76]
Ensemble reinforcement learning: A survey
Yanjie Song, Ponnuthurai Nagaratnam Suganthan, Witold Pedrycz, Junwei Ou, Yongming He, Yingwu Chen, and Yutong Wu. Ensemble reinforcement learning: A survey. Applied Soft Computing, 149:110975, 2023
2023
-
[77]
Flowllm: Flow matching for material generation with large language models as base distributions
Anuroop Sriram, Benjamin Miller, Ricky TQ Chen, and Brandon Wood. Flowllm: Flow matching for material generation with large language models as base distributions. Advances in Neural Information Processing Systems, 37:46025–46046, 2024
2024
-
[78]
Reinforcement learning: An introduction, volume 1
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998
1998
-
[79]
Imitationflow: Learning deep stable stochastic dynamic systems by normalizing flows
Julen Urain, Michele Ginesi, Davide Tateo, and Jan Peters. Imitationflow: Learning deep stable stochastic dynamic systems by normalizing flows. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5231–5237. IEEE, 2020
2020
-
[80]
Deep reinforcement learning with double q-learning
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016
2016
-
[81]
Matrix calculus operations and taylor expansions
William J Vetter. Matrix calculus operations and taylor expansions. SIAM review, 15(2): 352–369, 1973
1973
-
[82]
Boot- strapped transformer for offline reinforcement learning
Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Boot- strapped transformer for offline reinforcement learning. arXiv preprint arXiv:2206.08569 , 2022
2022 arXiv
-
[83]
Prioritized generative replay
Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized generative replay. arXiv preprint arXiv:2410.18082, 2024
2024 arXiv
-
[84]
Diffusion policies as an expressive policy class for offline reinforcement learning
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022
2022 arXiv
-
[85]
Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In Interna- tional Conference on Machine Learning, pages 38989–39007. PMLR, 2023
2023
-
[86]
A behavior regularized implicit policy for offline reinforcement learning
Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.09673, 2022
2022 arXiv
-
[87]
Policy-to-language: Train llms to explain decisions with flow-matching generated rewards
Xinyi Yang, Liang Zeng, Heng Dong, Chao Yu, Xiaoran Wu, Huazhong Yang, Yu Wang, Milind Tambe, and Tonghan Wang. Policy-to-language: Train llms to explain decisions with flow-matching generated rewards. arXiv preprint arXiv:2502.12530, 2025. 14
2025
-
[88]
Flow to control: Offline reinforcement learning with lossless primitive discovery
Yiqin Yang, Hao Hu, Wenzhe Li, Siyuan Li, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Flow to control: Offline reinforcement learning with lossless primitive discovery. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10843–10851, 2023
2023
-
[89]
Mopo: Model-based offline policy optimization
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129–14142, 2020
2020
-
[90]
Combo: Conservative offline model-based policy optimization
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954–28967, 2021
2021
-
[91]
Affordance-based robot manipulation with flow matching
Fan Zhang and Michael Gienger. Affordance-based robot manipulation with flow matching. arXiv preprint arXiv:2409.01083, 2024
2024
-
[92]
Saformer: A conditional sequence modeling approach to offline safe reinforcement learning
Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: A conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv:2301.12203, 2023
2023 arXiv
-
[93]
Energy-weighted flow matching for offline reinforcement learning
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. arXiv preprint arXiv:2503.04975, 2025
2025 arXiv
-
[94]
Preferred-action-optimized diffusion policies for offline reinforcement learning
Tianle Zhang, Jiayi Guan, Lin Zhao, Yihang Li, Dongjiang Li, Zecui Zeng, Lei Sun, Yue Chen, Xuelong Wei, Lusong Li, et al. Preferred-action-optimized diffusion policies for offline reinforcement learning. arXiv preprint arXiv:2405.18729, 2024
2024 arXiv
-
[95]
Guided flows for generative modeling and decision making
Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, and Ricky TQ Chen. Guided flows for generative modeling and decision making. arXiv preprint arXiv:2311.13443, 2023
2023 arXiv
-
[96]
uθ(xt, t) ∗ Z pt(x1|xt) ut(xt|x1) pt(xt|x1)p(x1) pt(xt) dx1 # pt(xt)dxtdt (30) = Z t,pt(xt)
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404, 2024. 15 A Pseudocode of Decision Flow Algorithm 1: Decision Flo...
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.