Pith. sign in

REVIEW 3 major objections 5 minor 69 references

Training instability of flow policies under value-gradient RL comes from the behavior-cloning sampler, not from iterative generation; VINE fixes the sampler so full end-to-end backpropagation through ten denoising steps becomes stable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 12:15 UTC pith:WNIQ6ZKL

load-bearing objection Sampler redesign, not iteration count, is the real fix for value-gradient BPTT on flow policies; the empirical case is strong even if the appendix theorem is idealized. the 3 major comments →

arxiv 2607.10369 v1 pith:WNIQ6ZKL submitted 2026-07-11 cs.RO cs.AI

VINE: Taming Generative Control Policies for Reinforcement Learning

Rushuai Yang , Zhuo Han , Houlin Li , Hecheng Wang , Zhichao Wu , Rui Zhang , Zhaowei Zhang , Zihong Chen
show 5 more authors
Xiaohan Yan Chiming Liu Yi Chen Wei Shan Maoqing Yao
This is my paper
classification cs.RO cs.AI
keywords flow matchinggenerative control policiesvalue-gradient RLoffline reinforcement learningrobot learningdenoising samplerBPTT stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Flow-matching policies generate robot actions by iteratively refining noise, which gives them the power to model complex multimodal action distributions. When people try to improve those policies with value-gradient reinforcement learning, training usually becomes unstable, and prior work blamed the multi-step generation process itself. This paper shows the real culprit is the ordinary single-trajectory sampler that was designed for behavior cloning: under value gradients it produces brittle paths that cannot support stable backpropagation. VINE redesigns only the sampler. At every denoising step it rebuilds a fresh noisy interpolation state around the current action estimate and then applies the same velocity field, creating a stable differentiable path while remaining compatible with the original flow-matching training objective. The result is that full end-to-end value-gradient optimization through all ten steps becomes practical, expressiveness and iterative refinement are preserved, and the method outperforms strong baselines on the OGBench offline suite and on a real-robot socket-insertion task.

Core claim

The instability of value-gradient RL on flow-matching policies does not arise from iterative generation itself, but from the vanilla single-noise Euler sampler inherited from behavior cloning. By reconstructing a new interpolation state and injecting fresh noise at every denoising step, VINE creates a stable path for backpropagation of the critic’s action gradient through the entire chain while remaining compatible with the pretrained flow-matching velocity field, thereby allowing end-to-end value-gradient optimization without sacrificing expressiveness or multi-step refinement.

What carries the argument

VINE sampling: at each step k reconstruct a noisy interpolation state x̂_k = t_k â_k + (1−t_k) z_k and recover the next action estimate by â_{k+1} = x̂_k + (1−t_k) v_θ(x̂_k, t_k; s). This supplies a stable differentiable path for value-gradient BPTT while keeping every network query inside the original flow-matching training semantics.

Load-bearing premise

The velocity field trained under ordinary flow matching still gives a valid endpoint prediction when its inputs are the reconstructed noisy states that VINE invents rather than the usual single Euler trajectory, even after value gradients have shifted the policy away from the original data.

What would settle it

Train the identical velocity network with ordinary Euler BPTT versus VINE BPTT on the same offline OGBench tasks and measure both gradient-norm trajectories and final success rates; if Euler remains as stable and successful as VINE, or if VINE’s reconstructed states produce large prediction error under the pretrained velocity field after distribution shift, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that training instability when applying value-gradient RL to flow-matching policies arises not from iterative generation itself, but from the single-trajectory Euler sampler inherited from behavior cloning. It proposes VINE, a sampler that at each denoising step reconstructs a fresh noisy interpolation state around the current action estimate and applies the same velocity field for endpoint prediction, thereby creating a stable path for end-to-end BPTT through all K denoising steps while remaining formally compatible with conditional flow matching. Compatibility is justified by Theorem 1 (optimal velocity yields the posterior mean for any query state) and Corollary 1. Empirically, VINE with K=10 is evaluated on 40 OGBench offline tasks (12 seeds, bootstrap CIs) and a real-world plug-insertion task under human-in-the-loop online RL, reporting aggregate gains over Gaussian, around-the-chain, outside-the-chain, and through-the-chain baselines, plus gradient-norm and iterative-refinement ablations.

Significance. If the sampler redesign is the primary source of stability, the work removes a long-standing practical barrier to using highly expressive multi-step generative policies with standard actor-critic value gradients, without sacrificing iterative refinement or requiring external guidance, distillation, or scalar reweighting. The contribution is concrete and immediately usable: only the sampler changes, pretrained flow policies can be reused, and the method is demonstrated both on a large offline suite and on a contact-rich real robot with wall-clock and intervention metrics. Strengths include a clear algorithmic statement, an explicit compatibility theorem for the optimal field, multi-seed OGBench evaluation with confidence intervals, and ablations that link gradient stability (Fig. 4) and multi-step refinement (Fig. 5) to the claimed mechanism.

major comments (3)
  1. [Appendix A.1, Theorem 1 / Corollary 1] Theorem 1 and Corollary 1 (Appendix A.1–A.3) establish that the endpoint map equals the posterior mean only for the pointwise minimizer v* of the CFM loss, and Corollary 1 further assumes a Gaussian action posterior and deterministic VINE (zk=0). Algorithm 1 and the actual experiments use a finite-capacity network that is itself updated by BPTT through the reconstructed states, with fresh noise every step and multimodal actions. Under value-driven mode shift the approximation gap can grow, so the formal justification does not yet cover the trained, non-stationary network that carries the central claim. A discussion of this gap, and ideally an empirical check that the network remains a faithful endpoint predictor after the policy has moved (e.g., reconstruction error or posterior-mean residual vs. training step), is needed to make the ‘why VINE stabilizes BPTT’ argument load-bearing rathe
  2. [Algorithm 1, lines 5–8; Eqs. (7)–(8)] Algorithm 1 GENERATE is inconsistent with the method defined by Eqs. (7)–(8). Line 5 performs an Euler-style update on an undefined/uninitialized xk (‘xk ← xk + (1/K)vθ…’) with the comment ‘Replace Euler Method’, while the subsequent lines correctly implement the reconstruction ˆxk = tk âk + (1−tk)zk and endpoint prediction. As written, the procedure is not reproducible from the pseudocode alone and conflicts with the prose claim that VINE does not follow a single Euler trajectory. The algorithm should be rewritten to match Eqs. (7)–(8) exactly (initialize â0, reconstruct ˆxk, predict âk+1, no residual Euler state).
  3. [Abstract; Table 1; §5.2] The abstract and §5.2 claim that VINE ‘consistently outperforms’ SOTA methods, yet Table 1 / Table 5 show near-floor performance on cube-double ([1,6]4) and cube-triple ([0,1]1), well below several through- and outside-the-chain baselines (e.g., DSRL, QAM, CGQL). Aggregate leadership is driven by large gains on antmaze-giant and humanoidmaze-large. The paper should qualify the consistency claim, analyze why the sampler fails or underperforms on these sparse manipulation domains, and state whether the failure is expected under the method’s assumptions.
minor comments (5)
  1. [Figure 1; §2.3] Fig. 1 caption and body use ‘value BPTT’ / ‘BPTT path’ without defining BPTT on first use in the main text; a one-line expansion would help non-RL readers.
  2. [§2.3, Eq. (6)] In §2.3 Eq. (6) the chain-rule expansion writes ∂xK/∂xk+1 without specifying how that Jacobian is obtained under the Euler discretization; a short note that it is the product of (I + (1/K)∂v/∂x) terms would clarify the contrast with VINE.
  3. [Table 2; §5.1] Table 2 reports human-intervention percentages for methods that use BC init from π0.5 versus methods trained from scratch; a brief note on whether intervention protocol and budget were matched across the two groups would strengthen the real-robot comparison.
  4. [Appendix A.4, Table 4] Hyperparameter Table 4 lists large, task-specific BC coefficients α (up to 1000). Sensitivity of VINE to α, or a default selection rule, would improve reproducibility.
  5. [§3; Algorithm 1] Minor notation: the manuscript alternates between âk, ˆak, and a for the action estimate; standardizing on one hat convention in the main text would reduce friction.

Circularity Check

0 steps flagged

No circularity: VINE is a free sampler redesign justified by a standard CFM property; success rates are external empirical measurements, not quantities forced by definition or self-citation.

full rationale

The paper's load-bearing chain is: (i) standard conditional flow-matching trains a velocity field on noisy interpolants xt = t a + (1-t)z; (ii) the Euler ODE sampler is only one of many paths consistent with that training; (iii) VINE rebuilds a fresh interpolant at each step and applies the same endpoint map â = x + (1-t)v(x,t;s); (iv) Theorem 1 (Appendix A.1) shows that for the pointwise minimizer v* this map equals the posterior mean for any query x, so the sampler change does not require a new generative objective; (v) actor-critic BPTT is then run through the VINE chain and evaluated on OGBench and a physical insertion task. None of these steps reduces a claimed prediction to its own inputs. Theorem 1 is a short derivation from the CFM squared-error objective (standard, externally cited), not a self-definition of the reported success rates. Algorithm 1 and Eqs. (7)–(8) are design choices whose merit is measured against fixed offline datasets and real-robot trials, not against quantities fitted from the same targets. Self-citations among the authors' other arXiv notes are confined to related-work context and do not underwrite uniqueness or forbid alternatives. The skeptic concern that Theorem 1 covers only optimal v* (not finite networks under value-driven shift) is a correctness/assumption risk, not circularity: the paper does not redefine validity as whatever the trained network happens to do. Score 0 is therefore the honest finding.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The central claim rests on standard flow-matching and actor-critic machinery plus one new sampling construction. The only free parameters that materially affect reported rankings are the per-task behavior-cloning coefficients α; everything else is either a conventional hyper-parameter or a fixed design choice (K=10). No new physical entities are postulated.

free parameters (3)
  • BC coefficient α = 10–1000 per domain (Table 4)
    Task-specific scalar that balances the Q-maximization term against the behavior-cloning penalty; values range from 10 (antmaze) to 1000 (puzzle) and are chosen by hand for each of the eight domains.
  • number of denoising steps K = 10
    Fixed to 10 for all experiments; controls both expressiveness and the length of the BPTT chain.
  • MIP time grid = uniform 0.1 increments
    Uniform schedule {0.0, 0.1, …, 0.9} used for the reconstructed interpolants; not learned.
axioms (3)
  • domain assumption The conditional flow-matching objective trains the velocity field so that x+(1-t)v*(x,t;s) equals the posterior mean of the data action (Theorem 1).
    Taken as given from the flow-matching literature and used to justify that VINE queries remain on-distribution for a pretrained velocity network.
  • domain assumption Behavior-regularized actor-critic (Eqs. 1–2) with a twin Q-network and target network is a valid offline RL objective.
    Standard assumption shared with ReBRAC, FQL and related baselines; not re-derived.
  • ad hoc to paper Fresh Gaussian noise injected at each reconstruction step remains compatible with the linear interpolation paths used in training.
    Stated without a full variance analysis under the non-stationary distribution induced by value-gradient updates.
invented entities (1)
  • VINE sampler (per-step reconstructed interpolation state ˆx_k) no independent evidence
    purpose: Provides a stable, noise-injected differentiable path for value-gradient BPTT while preserving flow-matching semantics.
    Defined by Eqs. 7–8 and Algorithm 1; no independent existence outside this paper.

pith-pipeline@v1.1.0-grok45 · 26998 in / 2807 out tokens · 53171 ms · 2026-07-14T12:15:37.931144+00:00 · methodology

0 comments
read the original abstract

Flow-matching policies have emerged as an effective policy parameterization for robot learning. They iteratively generate actions from noise, enabling highly expressive modeling of complex and multimodal action distributions. However, prior works observed that scaling these policies with value-gradient reinforcement learning (RL) often leads to training instability. Existing methods attribute this instability to iterative generation and therefore avoid end-to-end value-gradient optimization by sacrificing iterative generation, high expressiveness, or value-gradient optimization. Contrary to prior belief, we show the instability does not stem from iterative generation itself, but from the vanilla sampling strategy originally designed for behavior cloning, which becomes brittle under value-gradient RL. Motivated by this insight, we propose VINE, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies. Instead of following a single flow trajectory, VINE reconstructs a new interpolation state at every denoising step, creating a stable differentiable path for value-gradient propagation while remaining compatible with the original flow-matching denoising process. As a result, VINE preserves the expressiveness and iterative generation of flow-matching without sacrificing end-to-end value-gradient optimization. Despite performing end-to-end backpropagation through all ten denoising steps, VINE achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task. Videos are available on our website: https://agibottech.github.io/vine.

Figures

Figures reproduced from arXiv: 2607.10369 by Chiming Liu, Hecheng Wang, Houlin Li, Maoqing Yao, Rui Zhang, Rushuai Yang, Wei Shan, Xiaohan Yan, Yi Chen, Zhaowei Zhang, Zhichao Wu, Zhuo Han, Zihong Chen.

Figure 1
Figure 1. Figure 1: Left: A flow policy trained by behavior cloning fits the multi-modal data distribution. Middle: Directly backpropagating the critic gradient through the denoising steps (value BPTT) destabilizes the trajectory. Right: VINE produces a stable denoising trajectory that supports value￾gradient BPTT toward a1. arXiv:2607.10369v1 [cs.RO] 11 Jul 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Toy simulation of probability paths induced by different samplers under behavior cloning and offline value-gradient fine-tuning. Left: Under behavior cloning, all samplers re￾cover the multimodal data distribution, but Euler trajectories form isolated petal-like paths, DDPM explores broadly with noisy endpoints, and VINE provides structured exploration and broader state coverage. Right: After assigning dif… view at source ↗
Figure 3
Figure 3. Figure 3: Real-world online RL setting. VINE learns an policy that insert successfully from any location. SAC-Flow fails to learn a successful policy during online RL [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sampling ablation. We compare VINE against the vanilla Euler solver for flow-matching on OGBench. VINE consistently achieves higher success rates and more stable backpropagated gradients. 1 Iteration steps: 0/10 2 Iteration steps: 0/10 4 Iteration steps: 0/10 6 Iteration steps: 2/10 8 Iteration steps: 7/10 10 Iteration steps: 9/10 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Preserving the Iterative Refinement. We generate the final action using different num￾bers of denoising steps K and evaluate the resulting policies on the AntMaze task. As the number of refinement iterations increases, the generated action quality improves, leading to higher task success rates. insufficient for solving the long-horizon AntMaze task. With only 1 or 4 denoising steps, the agent largely remai… view at source ↗
Figure 6
Figure 6. Figure 6: VINE training curves (success rate vs. training steps), maze navigation domains. Each [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: VINE training curves (success rate vs. training steps), manipulation domains. Each row [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 24 linked inside Pith

  1. [1]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InRobotics: Science and Systems, 2023

    2023

  2. [2]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2023

    2023

  3. [3]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Physical Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, et al.π ∗ 0.6: a vla that learns from experience.arXiv preprint, arXiv:2511.14759, 2025. arXiv preprint arXiv:2511.14759

    Pith/arXiv arXiv 2025

  5. [5]

    K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, Q. Zhang, Z. Yu, et al. πrl: Online rl fine-tuning for flow-based vision-language-action models.arXiv preprint, arXiv:2510.25889, 2025. arXiv preprint arXiv:2510.25889

    arXiv 2025

  6. [6]

    Zhang, C

    T. Zhang, C. Yu, S. Su, and Y . Wang. Reinflow: Fine-tuning flow matching pol- icy with online reinforcement learning. In D. Belgrave, C. Zhang, H. Lin, R. Pas- canu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances in Neural Informa- tion Processing Systems, volume 38, pages 106282–106319. Curran Associates, Inc.,

  7. [7]

    URLhttps://proceedings.neurips.cc/paper_files/paper/2025/file/ 98d6f928497d9eaad9f320fb2db3040d-Paper-Conference.pdf

    2025

  8. [8]

    J. Luo, C. Xu, J. Wu, and S. Levine. Precise and dexterous robotic manipulation via human- in-the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    2025

  9. [9]

    R. Yang, H. Wang, Z. Wu, C. Liu, X. Yan, X. Du, S. Yue, C. Zhang, Y . Wang, Y . Liu, L. Qi, Y . Chen, W. Shan, and M. Yao. ALOE: Action-level off-policy evaluation for vision-language- action model post-training, 2026. URLhttps://arxiv.org/abs/2602.12691

    Pith/arXiv arXiv 2026

  10. [10]

    R. Yang, Z. Feng, T. Zhang, K. Wang, C. Zhang, L. Zhao, X. Su, Y . Chen, and J. Bian. Discover, learn, and reinforce: Scaling vision-language-action pretraining with diverse RL- generated trajectories, 2025. URLhttps://arxiv.org/abs/2511.19528

    arXiv 2025

  11. [11]

    R. Yang, H. Wei, R. Zhang, Z. Feng, X. Chen, T. Li, C. Zhang, L. Zhao, J. Bian, X. Su, and Y . Chen. Beyond human demonstrations: Diffusion-based reinforcement learning to generate data for VLA training, 2025. URLhttps://arxiv.org/abs/2509.19752

    arXiv 2025

  12. [12]

    Zhang, S

    Y . Zhang, S. Yu, T. Zhang, M. Guang, H. Hui, K. Long, Y . Wang, C. Yu, and W. Ding. Sac flow: Sample-efficient reinforcement learning of flow-based policies via velocity-reparameterized sequential modeling, 2026. URLhttps://arxiv.org/abs/2509.25756

    arXiv 2026

  13. [13]

    Li and S

    Q. Li and S. Levine. Q-learning with adjoint matching.International Conference on Learning Representations, 2026

    2026

  14. [14]

    S. Park, Q. Li, and S. Levine. Flow q-learning. In A. Singh, M. Fazel, D. Hsu, S. Lacoste- Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, editors,Proceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Ma- chine Learning Research, pages 48104–48127. PMLR, 13–19 Jul 2025. URLhttps: //proceedings.mlr.pr...

    2025

  15. [15]

    Psenka, A

    M. Psenka, A. Escontrela, P. Abbeel, and Y . Ma. Learning a diffusion model policy from rewards via Q-score matching. InInternational Conference on Machine Learning, 2024. 10

    2024

  16. [16]

    Dhariwal and A

    P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In M. Ran- zato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 8780–8794. Curran As- sociates, Inc., 2021. URLhttps://proceedings.neurips.cc/paper_files/paper/ 2021/file/49ad23d1ec9fa4bd8d77d0268...

    2021

  17. [17]

    H. Xu, K. Hu, S. Sojoudi, and A. Zhang. Reinforcement learning via value gradient flow. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps: //openreview.net/forum?id=JLL4VNVhM9

    2026

  18. [18]

    Z. Liu, T. Z. Xiao, C. Domingo-Enrich, W. Liu, and D. Zhang. Value gradient guidance for flow matching alignment. InNeurIPS 2025 Workshop on Structured Probabilistic Inference & Generative Modeling, 2025. URLhttps://openreview.net/forum?id=3oCOEL13OT

    2025

  19. [19]

    Wagenmaker, M

    A. Wagenmaker, M. Nakamoto, Y . Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine. Steering your diffusion policy with latent space reinforcement learning.Conference on Robot Learning, 2025

    2025

  20. [20]

    L. Fang, R. Liu, J. Zhang, W. Wang, and B. Jing. Diffusion actor-critic: Formulating con- strained policy iteration as diffusion noise regression for offline reinforcement learning. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  21. [21]

    P. Dong, Q. Li, D. Sadigh, and C. Finn. EXPO: Stable reinforcement learning with expressive policies.arXiv preprint arXiv:2507.07986, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage weighted regression: Simple and scalable off-policy reinforcement learning, 2020. URLhttps://openreview.net/forum? id=H1gdF34FvS

    2020

  23. [23]

    Hansen-Estruch, I

    P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. IDQL: Implicit Q- learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    P. Dong, C. Zheng, C. Finn, D. Sadigh, and B. Eysenbach. Value flows. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview. net/forum?id=2VyNYUVF2k

    2026

  25. [25]

    C. Xu, J. T. Springenberg, M. Equi, A. Amin, A. Esmail, S. Levine, and L. Ke. Rl token: Bootstrapping online rl with vision-language-action models.arXiv preprint arXiv:2604.23073, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Zhang, Z

    C. Zhang, Z. Wan, F. Chen, X. Yu, I. Tsang, and B. An. Gorl: An algorithm-agnostic framework for online reinforcement learning with generative policies.arXiv preprint arXiv:2512.02581, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Zhang, Y

    S. Zhang, Y . Lou, H. Cheng, Y . Guo, C. Fu, Y . Lyu, X. Zhang, H. Li, P. Wang, Z. Wang, et al. Force: Efficient vla reinforcement fine-tuning via value-calibrated warm-up and self- distillation.arXiv preprint arXiv:2606.26006, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Y . Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning.arXiv preprint arXiv:1911.11361, 2019

    Pith/arXiv arXiv 1911

  29. [29]

    Fujimoto and S

    S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning.Advances in Neural Information Processing Systems, 2021

    2021

  30. [30]

    Tarasov, A

    D. Tarasov, A. Nikulin, D. Akimov, V . Kurenkov, and S. Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library.Advances in Neural Information Processing Sys- tems, 2023. 11

    2023

  31. [31]

    R. Yang, C. Bai, H. Guo, S. Li, B. Zhao, Z. Wang, P. Liu, and X. Li. Behavior contrastive learning for unsupervised skill discovery. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors,Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 ofProceedings of Machine Learning Research, pages 3918...

    2023

  32. [32]

    C. Bai, R. Yang, Q. Zhang, K. Xu, Y . Chen, T. Xiao, and X. Li. Constrained ensem- ble exploration for unsupervised skill discovery. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors,Proceedings of the 41st Inter- national Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Re...

    2024

  33. [33]

    Lipman, R

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, and M. Nickel. Flow matching for generative mod- eling.International Conference on Learning Representations, 2023

    2023

  34. [34]

    X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.International Conference on Learning Representations, 2023

    2023

  35. [35]

    M. S. Albergo and E. Vanden-Eijnden. Building normalizing flows with stochastic interpolants. International Conference on Learning Representations, 2023

    2023

  36. [36]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 2020

    2020

  37. [37]

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations.International Conference on Learning Representations, 2021

    2021

  38. [38]

    D. P. Kingma and R. Gao. Understanding diffusion objectives as the elbo with simple data augmentation.Advances in Neural Information Processing Systems, 2024

    2024

  39. [39]

    K. Shi, J. Shi, P. Hebbar, Z. Zhao, T. Amarnath, Y . Su, S. Bahl, and D. Pathak. Flowdpg: Deterministic policy gradient on flow matching policies for real-world manipulation, 2026. URLhttps://arxiv.org/abs/2606.22303

    Pith/arXiv arXiv 2026

  40. [40]

    Silver, G

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In E. P. Xing and T. Jebara, editors,Proceedings of the 31st International Conference on Machine Learning, volume 32 ofProceedings of Machine Learning Research, pages 387–395, Bejing, China, 22–24 Jun 2014. PMLR. URLhttps://proceedings.mlr. ...

    2014

  41. [41]

    Fujimoto, D

    S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In K. Chaudhuri and R. Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 2052–2062. PMLR, 09–15 Jun 2019. URLhttps://proceedings.mlr.press/v97/ fujimoto19a.html

    2052

  42. [42]

    Kumar, J

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URLhttps://proceedings.neurips.cc/paper_ files/...

    2019

  43. [43]

    Ding and C

    Z. Ding and C. Jin. Consistency models as a rich and efficient policy class for reinforce- ment learning. In B. Kim, Y . Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y . Sun, editors,International Conference on Learning Representations, volume 2024, pages 53047– 53066, 2024. URLhttps://proceedings.iclr.cc/paper_files/paper/2024/file/ e9e1a0abc1a5b19a4a...

    2024

  44. [44]

    T. Chen, Z. Wang, and M. Zhou. Diffusion policies creating a trust region for offline reinforce- ment learning.Advances in Neural Information Processing Systems, 2024

    2024

  45. [45]

    Espinosa-Dice, Y

    N. Espinosa-Dice, Y . Zhang, Y . Chen, B. Guo, O. Oertell, G. Swamy, K. Brantley, and W. Sun. Scaling offline RL via efficient and expressive shortcut models.arXiv preprint arXiv:2505.22866, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    T. Chen, H. Ma, N. Li, K. Wang, and B. Dai. One-step flow policy mirror descent.arXiv preprint arXiv:2507.23675, 2025

    arXiv 2025

  47. [47]

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In2nd Workshop on Dexterous Manipulation: Design, Perception and Control (RSS), 2024. URLhttps://openreview. net/forum?id=KwwJuZIBXH

    2024

  48. [48]

    Barreiros, A

    J. Barreiros, A. Beaulieu, A. Bhat, R. Cory, E. Cousineau, H. Dai, C.-H. Fang, K. Hashimoto, M. Z. Irshad, M. Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation.Science Robotics, 11(113):eaea6201, 2026. doi:10.1126/scirobotics. aea6201. URLhttps://www.science.org/doi/abs/10.1126/scirobotics.aea6201

    doi:10.1126/scirobotics 2026

  49. [49]

    Z. Zhu, H. Zhao, H. He, Y . Zhong, S. Zhang, H. Guo, T. Chen, and W. Zhang. Diffusion models for reinforcement learning: A survey, 2024. URLhttps://arxiv.org/abs/2311.01223

    Pith/arXiv arXiv 2024

  50. [50]

    Uehara, Y

    M. Uehara, Y . Zhao, T. Biancalani, and S. Levine. Understanding reinforcement learning- based fine-tuning of diffusion models: A tutorial and review, 2024. URLhttps://arxiv. org/abs/2407.13734

    Pith/arXiv arXiv 2024

  51. [51]

    Zhang, W

    S. Zhang, W. Zhang, and Q. Gu. Energy-weighted flow matching for offline reinforcement learning. InInternational Conference on Learning Representations, 2025

    2025

  52. [52]

    B. Kang, X. Ma, C. Du, T. Pang, and S. Yan. Efficient diffusion policies for offline reinforce- ment learning. InNeural Information Processing Systems, 2023

    2023

  53. [53]

    S. Ding, K. Hu, Z. Zhang, K. Ren, W. Zhang, J. Yu, J. Wang, and Y . Shi. Diffusion-based reinforcement learning via Q-weighted variational policy optimization. InNeural Information Processing Systems, 2024

    2024

  54. [54]

    C. Lu, H. Chen, J. Chen, H. Su, C. Li, and J. Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. InInternational Confer- ence on Machine Learning, 2023

    2023

  55. [55]

    Frans, S

    K. Frans, S. Park, P. Abbeel, and S. Levine. Diffusion guidance is a controllable policy im- provement operator.arXiv preprint arXiv:2505.23458, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    Domingo-Enrich, W

    C. Domingo-Enrich, W. Chen, and B. Amos. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control.arXiv preprint arXiv:2409.03698, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Uehara, Y

    M. Uehara, Y . Zhao, K. Black, E. Hajiramezanali, G. Scalia, N. L. Diamant, A. M. Tseng, T. Biancalani, and S. Levine. Fine-tuning of continuous-time diffusion models as entropy- regularized control.arXiv preprint arXiv:2402.15194, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    Bergmeister, S

    A. Bergmeister, S. Jegelka, N. N ¨usken, C. Domingo-Enrich, and J. Pidstrigach. Reinforce ad- joint matching: Scaling rl post-training of diffusion and flow-matching models.arXiv preprint, 2025

    2025

  59. [59]

    Z. Guo, J. Sheng, D. D. Yao, and W. Tang. Improved techniques for fine-tuning flow models via adjoint matching: A deterministic control pipeline.arXiv preprint arXiv:2605.06583, 2026. 13

    Pith/arXiv arXiv 2026

  60. [60]

    J. Shin, D. Shin, J. Lee, J. Choi, and J. Choi. Efficient adjoint matching for fine-tuning diffusion models.arXiv preprint arXiv:2605.11480, 2026

    Pith/arXiv arXiv 2026

  61. [61]

    Ankile, A

    L. Ankile, A. Simeonov, I. Shenfeld, M. Torne, and P. Agrawal. From imitation to refinement– residual rl for precise assembly.arXiv preprint arXiv:2407.16677, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    X. Yuan, T. Mu, S. Tao, Y . Fang, M. Zhang, and H. Su. Policy decorator: Model-agnostic online refinement for large policy model.arXiv preprint arXiv:2412.13630, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    M. S. Mark, T. Gao, G. G. Sampaio, M. K. Srirama, A. Sharma, C. Finn, and A. Kumar. Policy-agnostic RL: Offline RL and online RL fine-tuning of any class and backbone. InRobot Learning Workshop, 2025

    2025

  64. [64]

    Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InInternational Conference on Learning Representations, 2023

    2023

  65. [65]

    L. He, L. Shen, L. Zhang, J. Tan, and X. Wang. DiffCPS: Diffusion model based constrained policy search for offline reinforcement learning.arXiv preprint arXiv:2310.05333, 2023

    Pith/arXiv arXiv 2023

  66. [66]

    Zhang, Z

    R. Zhang, Z. Luo, J. Sj ¨olund, T. B. Sch ¨on, and P. Mattsson. Entropy-regularized diffusion policy with Q-ensembles for offline reinforcement learning. InNeural Information Processing Systems, 2024

    2024

  67. [67]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus), 2016. URLhttps:// arxiv.org/abs/1606.08415

    Pith/arXiv arXiv 2016

  68. [68]

    J. L. Ba. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  69. [69]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations (ICLR), 2015. URLhttps://arxiv.org/abs/ 1412.6980. 14 A Appendix A.1 Compatibility with Pretrained Flow Policies A natural question is whether VINE can leverage an existing flow-matching pretrained policy. The following theorem justif...

    Pith/arXiv arXiv 2015