Pith. sign in

REVIEW 5 major objections 4 minor 9 cited by

Reinforcement Learning for Flow-Matching Policies

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that flow-matching policies fine-tuned with reinforcement learning, particularly the GRPO approach with a learned reward surrogate, incur between 50% and 85% less cost than naive imitation learning on six simulated…

desk verdict A clean, useful empirical contribution on RL fine-tuning of flow-matching policies; the main quantitative claims need error bars and a cleaner treatment of the reward surrogate's out-of-distribution scoring. read the letter →

arxiv 2507.15073 v1 pith:O2HCJ6MQ submitted 2025-07-20 cs.LG

Samuel Pfrommer , Yixiao Huang , Somayeh Sojoudi This is my paper
classification cs.LG
keywords flowmatchingpoliciesreinforcementlearningGRPOreward-weightedimitationvariable-horizonplanningactionexplorationunicyclecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that flow-matching policies, the action-chunk generators behind modern vision-language-action robot models, can be fine-tuned with reinforcement learning to outperform the demonstrations they were trained on. It presents two methods, RWFM and GRPO with a learned reward surrogate, plus an action explorer that lets the policy try action trajectories no demonstrator ever produced. The central quantitative claim is that on a suite of six simulated unicycle tasks, the GRPO approach generally incurs between 50% and 85% less cost than naive imitation learning flow matching. If that holds, RL fine-tuning could make real robot policies faster and more skillful than their human demonstrators without hand-crafting new policy structure.

What carries the argument

The central object is the learned velocity field $v_\theta(\hat{A}_\tau, \tilde{o}, \tau)$ of a flow-matching model that generates augmented action chunks $\hat{A} \in \mathbb{R}^{(d_a+1) \times H'}$, where the extra channel encodes the intended planning horizon. The machinery has four moving parts: the RWFM loss, a weighted conditional flow-matching loss with weight $w = \exp(\alpha R)$ that shapes the learned density toward high-reward trajectories; the GRPO loss, which samples $G$ action chunks, explores each with a bump-based action explorer, computes advantages from the reward surrogate $R_\phi$, and applies the same weighted flow-matching loss with weights $\exp(\alpha a_i)$; the reward surrogate, a time-series regressor trained to predict true rollout reward without running rollouts; and the action explorer, which adds random Gaussian bumps to action trajectories so the policy can reach actions outside the demonstration support. The horizon channel makes variable-duration planning possible by interpolating chunks to a fixed length and adding a constant channel that the model learns to decode back into a horizon.

What would settle it

Run GRPO with a fixed exploration magnitude on a task whose reward is known exactly, and log the surrogate prediction $R_\phi(\tilde{o}, A')$ against the true rollout reward $R(\tilde{o}, A', \mathrm{Rollout}(s, A'))$ for every explored action. If high-advantage explored actions consistently show true reward below the surrogate prediction, so that validation reward stops improving with more collection rounds, then the reward-surrogate assumption is broken and the central claim fails.

Watch

Extended reading notes

Core claim

The paper claims that a flow-matching policy pretrained by imitation on suboptimal demonstrations can be improved by reinforcement learning, and that the improvement comes in two complementary forms. First, variation suboptimality, where demonstrators are inconsistent, is addressed by reweighting the flow-matching loss toward high-reward action chunks, either through the exponential reward weight in RWFM or through group-relative advantages in GRPO. Second, support suboptimality, where the best actions lie outside anything the demonstrator ever produced, is addressed by an action explorer that adds random Gaussian bumps to action trajectories before scoring them. On a simulated unicycle, both methods surpass the demonstrator and the naive ILFM baseline, with GRPO generally incurring 50% to 85% less cost than ILFM across the six reward functions. The paper also shows a horizon-channel scheme that lets a single flow-matching model generate action chunks of variable duration, which is what makes minimum-time behavior learnable.

Load-bearing premise

The reward surrogate scores explored, out-of-distribution action trajectories accurately enough that the GRPO advantages it produces improve the policy rather than mislead it, with periodic true-reward collection correcting bias only indirectly through the policy distribution.

Editorial extensions

If this is right

  • GRPO-style fine-tuning can be applied to any flow-matching policy without computing the policy likelihood or the divergence trace of the flow, keeping training tractable for high-dimensional action chunks.
  • The variable-horizon channel lets one model generate trajectories of different durations, so time-optimal behavior is representable and learnable rather than being clipped by a fixed planning horizon.
  • Because the reward surrogate needs no simulator reset per candidate action, the GRPO loop is real-world compatible in principle: only the current policy's on-policy actions need to be rolled out for periodic data collection.
  • Learning a braking behavior for the Position and Velocity task shows the method can discover control strategies outside the demonstrator's support, not merely sharpen imitation.
  • Performance improves with the number of collected trajectories across all six rewards, suggesting the gains come from RL data rather than from reweighting the original dataset alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same surrogate-scored GRPO loop could in principle fine-tune a vision-language-action model on a physical robot using only onboard rollouts and a VLM-based reward, since the paper places no constraints on the reward function's form; this extension is ours, not demonstrated by the paper.
  • A natural stress test is whether the plateau-triggered data collection closes the loop tightly enough: if the surrogate overvalues out-of-distribution actions, performance would be expected to dip between collection rounds, which the paper's aggregate curves do not isolate.
  • The bump-based explorer is a domain-agnostic prior; smoother basis perturbations, such as frequency-domain or latent-noise perturbations, might scale better to higher-dimensional action spaces than i.i.d. Gaussian bumps.
  • The RWFM alpha trade-off the paper describes suggests that an observation-conditioned weighting, or a per-sample temperature, could avoid the collapse to a single action chunk at large reward scaling factors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. Flow-matching visuomotor policies are typically pretrained by imitation on suboptimal demonstrations. This paper proposes to fine-tune such policies with reward feedback. It formalizes two sources of demonstrator suboptimality (variation and support), introduces a horizon-channel representation that lets a fixed-horizon flow-matching U-Net emit action chunks of variable duration, and presents two fine-tuning algorithms: RWFM, which reweights collected trajectories by exponentiated reward and uses Gaussian-bump action exploration during rollouts, and a GRPO-style method that computes group-relative advantages from a learned reward surrogate and uses them as flow-matching weights. Experiments on simulated unicycle tasks with six reward functions compare ILFM, RWFM, and GRPO and report that both methods exceed the demonstrator, with GRPO generally incurring 50-85% less cost than ILFM. Code is provided.

Significance. The paper addresses a timely problem: improving suboptimal imitation data for action-chunk policies without expensive likelihoods or differentiable rewards. Its variable-horizon encoding is simple and plausible, and the distinction between variation and support suboptimality gives a useful vocabulary for why reward-weighted imitation alone is insufficient. The authors also release code and test on multiple reward functions. That said, the evidence for the headline claim is currently incomplete: the GRPO reward surrogate is trained on a distribution that excludes the explored actions it scores; there are no error bars or seed counts; hyperparameters are selected per task on a validation set; and no true-reward RL baseline is included to isolate the surrogate's contribution. These are fixable with additional experiments, but they are load-bearing for the paper's central quantitative conclusions.

major comments (5)
  1. [Section 3.2 (Definition 3, Eq. (9); Algorithm 2)] The learned reward surrogate used in the GRPO loss is never trained on the explored actions whose rewards it scores. In Eq. (9), the advantages ai are computed from ri = Rφ(˜o, A′ i) with A′ i ∼ E(A′ i | A_i), and the weights w(˜o, A′ i) = exp(α ai) enter the flow-matching loss for those A′ i. However, Algorithm 2 collects the appended dataset Dπ by sampling A ∼ πθ(A | ˜o) and O ← Rollout(s, A), and the text explicitly states that 'we do not employ E when collecting rollouts for GRPO.' The surrogate Rφ (Definition 4) is therefore trained only on demonstrations and on unperturbed policy actions, never on the specific A′ samples whose rewards drive the update. The paper's safeguard—that OOD overvaluation by the surrogate triggers a data-collection round and reward feedback for 'these learned policy actions'—obtains true rewards for A ∼ πθ, not for A′. If the surrogate overvalues certain exploratory perturbations, GRPO will be trained toward phantom rewards, and the reported gap between GRPO and ILFM in Figure 1 may partly reflect surrogate error rather than true cost improvement. The authors should train Rφ on rollouts of explored actions (even at a reduced frequency or for a subset of the group), or report surrogate error measured on held-out A′ samples, or otherwise demonstrate that the advantage estimates are reliable on the explored-action distribution.
  2. [Section 4.2, Figure 1] The headline quantitative claim—'between 50% and 85% less cost'—is reported without any uncertainty quantification. No seed counts, standard deviations, confidence intervals, or significance tests accompany the learning curves in Figure 1 or the supporting figures. Because the environment initial states, demonstrator parameters, and stochastic training all contribute variance, single-curve comparisons are insufficient to support a quantitative range. The authors should report averages over multiple independent runs with error bars and specify how the 50–85% figure was computed from the curves.
  3. [Section 4.2, hyperparameter selection] The statement that 'Figure 1 uses the best performing hyperparameters for each reward function on a validation set' raises a model-selection concern. With six reward functions and sweeps over α ∈ {0,5,10,20,40} and M ∈ {0.0,0.05,0.1,0.2}, choosing the best validation configuration per task and then reporting test curves can exaggerate the improvement and makes the RWFM/GRPO comparison difficult to interpret (GRPO also fixes α = 2.0, so the two methods are not compared under the same selection protocol). The authors should report results for a pre-specified hyperparameter choice, use nested validation, or show that the ranking is robust across the sweep.
  4. [Section 3.2, Algorithm 2] Algorithm 2's data-collection line is internally inconsistent. The pseudocode writes the collected tuple as (˜o(i), (A′)(i), O(i)) but the sampling line specifies A ∼ πθ(A | ˜o) and O ← Rollout(s, A), with no definition of A′ in that line. If the intended tuple is (˜o, A, O), the A′ notation is misleading; if the intended tuple is (˜o, A′, O) with A′ ∼ E(A′ | A), it contradicts the text and the method's stated sample-efficiency motivation. This ambiguity is directly related to the first major comment and must be resolved in the revised manuscript.
  5. [Section 3.2 vs. Section 4.2] The comparison between RWFM and GRPO does not isolate the learned reward surrogate. The two methods differ in the reward signal (rollout-based true reward vs. surrogate-based advantages), the use of group-relative normalization, and the role of the action explorer (rollout-time vs. scoring-time). To support the claim that the learned reward surrogate improves sample efficiency, the authors should include a GRPO variant that computes advantages from true rollouts for the G sampled actions (or an otherwise matched ablation). Without such a baseline, the faster improvement attributed to GRPO could come from the group-relative weighting or from the different exploration scheme rather than from the surrogate.
minor comments (4)
  1. [Section 3, variable-horizon construction] The construction of the augmented action chunk is dimensionally unclear: after interpolating A ∈ R^{da×H} to A′ ∈ R^{da×H′}, the text says 'concatenating A with H·1_da' to obtain  ∈ R^{da+1×H′}; this should presumably concatenate A′ with a constant channel of length H′ (e.g., H·1_{H′}). Please clarify.
  2. [Section 3.1, Eq. (7)] In the informal weighted-flow-matching statement, Eq. (7) writes p_θ*^E(x) ∝ w(x)E_q(x)/Z_E, but E_q(x) is not defined. The notation should be corrected (likely E_q[w] or an explicit expectation over q).
  3. [Appendix A.2, Position & Velocity reward] The formula for 'Final position and final velocity' is missing the leading negative sign: it reads R(˜o,A,O) = ∥pH+1 − g∥2 − vH+1, but the accompanying text and the rest of the appendix indicate that the position penalty should be −∥pH+1 − g∥2 − vH+1.
  4. [Section 3.2, Definition 3] The loss in Eq. (9) is a reward-weighted flow-matching objective with normalized exponential advantages; it does not include the clipped importance ratio usually associated with GRPO. If this is intentional, the text should say so to avoid confusion with the standard GRPO update.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GRPO surrogate risk is an acknowledged generalization issue, not a definitional reduction.

full rationale

After walking the derivation chain, I find no step that reduces to its own inputs by construction or via load-bearing self-citation. The RWFM loss (Def. 2) is a weighted ILFM objective justified by an external weighted-flow-matching result [Fan et al. 2025]; the weight is the true reward, and collected rollouts label the explored actions A' with Rollout(s,A'), so the training signal is grounded. The GRPO loss (Def. 3) uses advantages r_i = R_phi(˜o,A'_i) from a surrogate trained by Def. 4 on D. The paper is explicit that the explorer E is not used when collecting GRPO rollouts: 'unlike RWFM, we do not employ E when collecting rollouts for GRPO,' so A' is never directly labeled with true reward; it also warns that 'the policy may produce out-of-distribution action trajectories which are overvalued by the reward surrogate.' This is an acknowledged distribution-shift / reward-model generalization risk, not a definitional circularity: no equation forces R_phi(A') to equal R(˜o,A',Rollout(s,A')), and the headline results in Fig. 1 are evaluated with the true reward on held-out validation rollouts, so the reported improvement is externally grounded. There are no self-citations by the current authors, and the variable-horizon scheme is a representation change (append an H channel, interpolate, read out H), not a circular prediction. The paper is self-contained against its own true-reward benchmark; the surrogate limitation should be weighed as a correctness/robustness risk, not circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The central claims depend on several hand-chosen or validation-selected hyperparameters (alpha, M, G, plateau patience, collection schedule), two domain assumptions about the reward surrogate and the weighted flow matching theorem, and one new model input (the horizon channel). No new physical entities are postulated.

free parameters (6)
  • RWFM reward scaling alpha = best in {5,10,20,40} per reward, validation-selected
    Controls how strongly high-reward trajectories are weighted in the RWFM loss; final curves use the best validation setting (Section 4.2, Appendix B).
  • GRPO reward scaling alpha = 2.0
    Fixed for GRPO in Appendix A.1, not swept.
  • Action exploration magnitude M = best in {0.0,0.05,0.1,0.2} per reward, validation-selected
    Amplitude of random Gaussian bumps added to action trajectories; central to the support-suboptimality claim.
  • GRPO group size G = 10
    Number of actions sampled per observation to compute relative advantages.
  • Plateau patience = 500 epochs (RWFM), 50 epochs (GRPO)
    Hand-chosen termination criterion for deciding when to collect new trajectories.
  • Collection iterations C and sampling fraction gamma = C=10, gamma=0.2
    Control how much new data is added in each online round.
assumptions (4)
  • domain assumption Weighted flow matching with reward weights produces a distribution proportional to the weighting, and iterative reweighting concentrates on reward-maximizing actions.
    Used to justify the RWFM and GRPO losses in Section 3.1 (Eq. 5-7) and Definition 3; cited as an informal result from Fan et al. 2025, not proved or checked in this paper.
  • domain assumption The learned reward surrogate R_phi generalizes well enough to score out-of-distribution explored actions for computing advantages.
    Load-bearing for GRPO; the paper only indirectly corrects surrogate bias via periodic true-reward validation (Section 3.2, Algorithm 2).
  • ad hoc to paper Representing variable horizons by interpolating action chunks to fixed length and concatenating a constant H channel is an adequate generative representation, and the model can decode H from the time channel.
    Proposed in Section 3 without theoretical justification; validated only empirically on the simulated unicycle.
  • domain assumption The reward function used for validation and data collection is an exact oracle, while the surrogate is used only inside GRPO updates.
    The paper assumes access to R(o, A, Rollout(s, A)) for validation and collection (Section 2, Algorithm 2).
invented entities (1)
  • Horizon-conditioned action channel (H * 1_da)
    purpose: Allows the flow-matching U-Net to generate action chunks of variable duration by encoding the original horizon as an extra constant channel after interpolation to fixed length.
    This is a new model input introduced by the paper; its only evidence is the simulated unicycle experiments, and it has no falsifiable handle outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning for Flow-Matching Policies." pith.science (2026). https://pith.science/paper/O2HCJ6MQ

@misc{pith2026250715073,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning for Flow-Matching Policies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2HCJ6MQ}},
  note         = {Machine review of arXiv:2507.15073}
}
abstract

Flow-matching policies have emerged as a powerful paradigm for generalist robotics. These models are trained to imitate an action chunk, conditioned on sensor observations and textual instructions. Often, training demonstrations are generated by a suboptimal policy, such as a human operator. This work explores training flow-matching policies via reinforcement learning to surpass the original demonstration policy performance. We particularly note minimum-time control as a key application and present a simple scheme for variable-horizon flow-matching planning. We then introduce two families of approaches: a simple Reward-Weighted Flow Matching (RWFM) scheme and a Group Relative Policy Optimization (GRPO) approach with a learned reward surrogate. Our policies are trained on an illustrative suite of simulated unicycle dynamics tasks, and we show that both approaches dramatically improve upon the suboptimal demonstrator performance, with the GRPO approach in particular generally incurring between $50\%$ and $85\%$ less cost than a naive Imitation Learning Flow Matching (ILFM) approach.

Figures

Figures reproduced from arXiv: 2507.15073 by the authors.

Figure 1
Figure 1. Performance comparison of ILFM, RWFM, and GRPO policies across different reward functions, with the constant demonstration policy πD performance shown as a gray dotted horizontal line. The x-axis represents the number of training trajectories, and the y-axis shows the average test-time reward. We then alternate between training the reward surrogate and the policy, as described in Algorithm 1. Note that the demonstra… view at source ↗
Figure 2
Figure 2. Performance comparison of exploration magnitudes [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of ILFM, RWFM, and GRPO policies for the Position & Time reward function, with the constant demonstration policy πD performance shown as dotted horizontal lines. The x-axis represents the number of training trajectories, and the y-axis shows the average test-time reward. The reward is decomposed as a stacked chart into the time reward component (dark shading) and the position plus time reward … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: RWFM policy performance comparison with different reward scaling factors α [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: RWFM policy performance comparison with different action exploration magnitudes M. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: GRPO policy performance comparison with different action exploration magnitudes M. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    QGF performs test-time policy optimization for flow models in RL by guiding a behavior-cloned reference policy with value-function gradients, achieving strong results on high-dimensional offline RL benchmarks without ...

  2. Reinforcement Learning for Flow-Matching Policies with Density Transport

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    RLDT fine-tunes pretrained flow-matching policies for continuous control by aligning them to a max-entropy RL transport field constructed via SVGD, using expected-target estimation for stable multi-step updates.

  3. Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT

    cs.RO 2026-05 unverdicted novelty 7.0 of 10

    ConSFT prevents catastrophic forgetting in fine-tuning flow-matching VLAs by dynamically scaling gradients based on model confidence, retaining over 20% more pre-trained capability than standard SFT without prior data...

  4. Contrastive Conceptor Activation Steering (COAST): Unlocking Vision-Language-Action Models through Hidden States

    cs.RO 2026-05 conditional novelty 6.0 of 10

    COAST applies contrastive conceptors to steer VLA hidden states into task-specific success subspaces, yielding over 20% simulation and 40% real-robot success rate gains across three distinct policies.

  5. From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning

    cs.RO 2026-03 accept novelty 6.0 of 10

    Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...

  6. Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    FPO fine-tunes flow-matching vision-language-action policies with a PPO-style objective that replaces intractable policy ratios with per-sample conditional flow-matching loss differences, reaching 87.2% average succes...

  7. RLMM-Flow: A Flow-based Mobile Manipulation Framework with Latent-Space Reinforcement Learning

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Steering a frozen flow-based whole-body motion policy in latent space with a warmed-up critic and coarse-to-fine residuals improves mobile manipulation success and safety.

  8. Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT

    cs.RO 2026-05 unverdicted novelty 5.0 of 10

    ConSFT is a gradient-scaling fine-tuning objective for flow-matching VLAs that bounds parameter disruption via model-confidence weighting, yielding over 20% better capability retention than vanilla SFT on LIBERO and RoboTwin.

  9. HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing

    cs.RO 2026-03 conditional novelty 5.0 of 10

    A vision-language-action robot policy that claims tactile-aware manipulation without tactile sensors at inference via reward-weighted flow matching and action distillation.

Reference graph

Works this paper leans on

27 extracted references · 2 canonical work pages · cited by 8 Pith papers

  1. [1]

    Training diffusion models with reinforcement learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301,

    arXiv

  2. [3]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818,

    arXiv

  3. [5]

    Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control

    Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861,

    arXiv

  4. [6]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767,

    arXiv

  5. [7]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied ...

    arXiv

  6. [8]

    Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu

    URL https://doi.org/ 10.48550/arXiv.2303.03378. Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu. Online reward- weighted fine-tuning of flow matching with wasserstein regularization. International Conference on Learning Representations,

    arXiv doi:10.48550/arxiv.2303.03378

  7. [9]

    Chats: Combining human-aligned optimization and test-time sampling for text-to-image generation

    Minghao Fu, Guo-Hua Wang, Liangfu Cao, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Chats: Combining human-aligned optimization and test-time sampling for text-to-image generation. arXiv preprint arXiv:2502.12579,

    arXiv

  8. [12]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991,

    arXiv

Show all 27 references
  1. [14]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024a. Sungyoon Kim, Yunseon Choi,...

    arXiv

  2. [15]

    A self-correcting vision-language- action model for fast and slow system manipulation

    Chenxuan Li, Jiaming Liu, Guanqun Wang, Xiaoqi Li, Sixiang Chen, Liang Heng, Chuyan Xiong, Jiaxin Ge, Renrui Zhang, Kaichen Zhou, and Shanghang Zhang. A self-correcting vision-language- action model for fast and slow system manipulation. arXiv preprint arXiv:2405.17418,

    arXiv

  3. [16]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747,

    arXiv

  4. [17]

    Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al

    URL https://arxiv.org/abs/2412.06264. Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint arXiv:2501.13918,

    arXiv

  5. [18]

    Generative trajectory stitching through diffusion composition

    Yunhao Luo, Utkarsh A Mishra, Yilun Du, and Danfei Xu. Generative trajectory stitching through diffusion composition. arXiv preprint arXiv:2503.05153,

    arXiv

  6. [19]

    Grounding multimodal llms to embodied agents that ask for help with reinforcement learning

    12 Ram Ramrakhya, Matthew Chang, Xavier Puig, Ruta Desai, Zsolt Kira, and Roozbeh Mottaghi. Grounding multimodal llms to embodied agents that ask for help with reinforcement learning. arXiv preprint arXiv:2504.00907,

  7. [20]

    Diffusion policy policy optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588,

    arXiv

  8. [21]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  9. [23]

    Smolvla: A vision- language-action model for affordable and efficient robotics

    Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zoui- tine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision- language-action model for affordable and efficient robotics. arXiv preprint arXiv:...

    arXiv

  10. [24]

    Understanding the perfor- mance gap between online and offline alignment algorithms

    Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.08448,

    arXiv

  11. [25]

    Dancegrpo: Unleashing grpo on visual generation

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818,

    arXiv

  12. [26]

    We start by collecting 30, 000 demonstration trajectories from πD

    13 A Experimental details A.1 Architectures and hyperparameters Data collection. We start by collecting 30, 000 demonstration trajectories from πD. These are held fixed for all remaining experiments, and are split 95% for training, 1% for validation, and 4% for testing. We def...

    2024

  13. [128]

    To generate samples, we use Euler integration with 4 steps

    Our GRPO implementation uses a group size G = 10 and α = 2.0. To generate samples, we use Euler integration with 4 steps. Reward surrogate. Our reward surrogate Rϕ is derived from the TimesNet architecture for time- series classification [Wu et al., 2023]. The input sequence c...

    2023

  14. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  15. [2020]

    Sequence-augmented se (3)-flow matching for conditional protein backbone generation

    Guillaume Huguet, James Vuckovic, Kilian Fatras, Eric Thibodeau-Laufer, Pablo Lemos, Riashat Islam, Cheng-Hao Liu, Jarrid Rector-Brooks, Tara Akhound-Sadegh, Michael Bronstein, et al. Sequence-augmented se (3)-flow matching for conditional protein backbone generation. arXiv pr...

    arXiv

  16. [2022]

    Refined policy distillation: From vla generalists to rl experts

    Tobias Jülg, Wolfram Burgard, and Florian Walter. Refined policy distillation: From vla generalists to rl experts. arXiv preprint arXiv:2503.05833,

    arXiv

  17. [2023]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024a. Chang Chen, Fei Deng, Kenji Kawaguchi...

    arXiv

  18. [2024]

    arXiv preprint arXiv:2410.24164,

    arXiv

  19. [2025]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025a. Yanjiang Guo, Jianke Zhang,...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.