Pith. sign in

REVIEW 4 major objections 5 minor 36 references

RedFlow claims failed rollouts, not just successes, can be turned into dense action-level corrective supervision for flow-matching VLA policies, lifting LIBERO success from 56.2% to 68.2% and real-world success from 56.7% to 74.7%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:27 UTC pith:XZRI52YH

load-bearing objection A worthwhile idea with plausible gains, but the evidence is unanchored and the visual-free matching is a real weak point. the 4 major comments →

arxiv 2607.27782 v1 pith:XZRI52YH submitted 2026-07-30 cs.RO cs.AI

RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy

Zhengyang Yan , Junhao Li , Fangqi Zhu , Zijun Wang , Quanxin Shou , Yikun Miao , Xiaoyi Pang , Zicong Hong
show 1 more author
Song Guo
This is my paper
classification cs.RO cs.AI
keywords flow-matching VLA policyoffline reinforcement learningfailure reuseaction-level corrective supervisionrobotic manipulationprogress-state clusteringsample efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a flow-matching vision-language-action policy can be improved offline by treating failed rollouts as a source of corrective labels, not as data to discard or rank. It proposes RedFlow, which estimates an action-level quality score per chunk from smoothed task-progress signals and trajectory outcomes, clusters similar progress–state contexts, and uses quality-weighted positive centroids as corrective targets. An adaptive objective then reinforces good actions, suppresses failure-inducing ones, and redirects recoverable failures toward those targets. If true, this means offline buffers containing failures can substitute for large amounts of online interaction: RedFlow lifts LIBERO average success from 56.2% to 68.2% and real-world success from 56.7% to 74.7%, and matches on-policy RL baselines with roughly an order of magnitude fewer rollouts.

Core claim

RedFlow's central claim is that fine-grained, action-level corrective supervision can be extracted from mixed-quality offline rollouts without human intervention. The paper identifies failure-inducing action chunks via a signed proxy advantage built from local progress changes and outcome bias, then assigns each correctable failure a corrective target from successful actions in the same execution-context cluster. The Adaptive Redirection Objective realizes these signals in the flow-matching velocity field—attracting to high-quality chunks, suppressing low-quality chunks within a finite margin, and redirecting correctable failures toward the cluster centroid—so that the update is an asymmetri

What carries the argument

The load-bearing mechanism is progress–state context clustering: a pretrained general reward model produces smoothed progress estimates, combined with normalized proprioception into a feature [q̃_t; β p̄_t], clustered by HDBSCAN; within each cluster, positive chunks are averaged with softmax advantage weights to form a corrective target, and the objective's suppression and redirection terms act as a finite-range obstacle plus attraction to that target. This transforms trajectory-level success/failure labels into dense, geometrically meaningful action-level transport directions.

Load-bearing premise

Everything rests on the pretrained progress estimator and the clustering over its outputs being accurate enough that the sign of the proxy advantage identifies genuinely failure-inducing chunks and the centroid from co-clustered positive chunks is a valid local correction; the paper explicitly lists these dependencies as limitations in Appendix F.

What would settle it

A controlled experiment would corrupt the progress estimates, for example by randomizing the sign of the proxy advantage while keeping the redirection objective unchanged; if RedFlow's gains persist, the claimed mechanism is not the source of improvement. More directly, if ground-truth failure-inducing chunks are known and the cluster-centroid targets do not match actions that actually recover the state, the reported success-rate gains should disappear.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Failed rollouts become useful training signal: LIBERO average success rises from 56.2% to 68.2%, and real-world average success from 56.7% to 74.7%.
  • The policy learns recovery behaviors not present in the base policy, such as using the opposite arm to pull a garment back before retrying.
  • RedFlow matches strong on-policy RL baselines while requiring roughly an order of magnitude fewer training trajectories, suggesting failure reuse can substitute for a large share of online interaction.
  • Ablations show both successes and failures are necessary: removing failures drops average success from 72.5% to 68.0%, removing successes to 62.4%.
  • Uncorrectable failures (those without positive support in their cluster) should be suppressed rather than corrected; indiscriminately assigning targets hurts performance, dropping from 72.5% to 61.0%.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable prediction is that RedFlow's gains concentrate on frequent, cluster-supported failure modes, while rare or novel failures benefit mostly from suppression; stratifying results by failure-mode frequency would check this.
  • Using uncertainty or confidence estimates from the progress model as gates for target assignment could extend correction beyond the paper's current positive-support requirement, which the paper itself lists as a limitation in Appendix F.
  • The centroid-based corrective target suggests a natural extension: on non-Euclidean action manifolds, the squared-distance barycenter should be replaced by a manifold Fréchet mean, which could matter for contact-rich, high-DoF manipulation.
  • The sample-efficiency result implies that logged robot deployment failures are a reusable asset for post-training generalist VLA policies without human relabeling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. RedFlow proposes an offline RL post-training method for flow-matching VLA policies. Given a frozen buffer of successful and failed rollouts collected by a base policy, it computes a proxy advantage per action chunk from smoothed GRM progress estimates plus a trajectory-outcome bias (Eq. 5), clusters chunks in a progress–proprioception feature space (Eq. 6), and assigns each negative chunk a corrective target equal to an advantage-weighted centroid of positive chunks in its cluster (Eq. 7). The training objective (Eq. 14) combines quality-weighted flow-matching attraction, a hinge-based suppression loss for negative chunks, and a correction loss pulling predicted endpoints toward the cluster target. Experiments on four LIBERO suites and three real-robot tasks report consistent gains over AWR and DPO and sample-efficiency comparisons to PPO/GRPO/DDPO, with real-world success rising from 56.7% to 74.7%. The appendix supplies a Wasserstein transport interpretation and convergence/dissipation arguments.

Significance. If the mechanism works as claimed, the contribution is significant: it offers a way to turn failure rollouts — usually discarded or used only as negative trajectories — into dense action-level supervision for a modern VLA family, and the reported real-robot improvement is practically meaningful. The paper also provides a clear algorithm, a formal transport view in Appendix A, and an ablation sequence that separates data composition, target assignment, and loss terms. The evaluation on fresh rollouts mitigates the concern that the method only overfits the training buffer. However, the strength of the contribution rests on the validity of the corrective targets and on the statistical reliability of the reported margins.

major comments (4)
  1. [§3.2, Eq. (6)–(7), Eq. (13)] The execution context f_t in Eq. (6) contains only normalized proprioception and smoothed GRM progress. Yet the policy is vision-conditioned: π_θ(a_t | I_t, q_t, l). A scalar p̄_t cannot distinguish visually different scenes with equal progress and identical q_t, so a negative chunk from one visual configuration can be paired in HDBSCAN with positive chunks from another configuration; the centroid a*_t in Eq. (7) then averages incompatible actions. Minimizing L_cor in Eq. (13) pulls the velocity field toward an endpoint that may be wrong for the actual observation. The sentence in §3.2 that ‘GRM-predicted progress implicitly incorporates task-relevant visual information’ mitigates only partially, because R(o_t,l) is a scalar and Eq. (4) additionally smooths it. Since this matching is the central mechanism, the paper needs either to use visual features in the matching space or provide a d
  2. [Tables 1, 2, Fig. 6] All success rates are single-run point estimates. Table 1 reports 500 evaluation episodes per suite and Table 2/Fig. 6 similar single evaluations, but no standard errors, confidence intervals, or multiple seeds are given. Many ablated differences are small (e.g., 69.1 vs 68.8 vs 72.5 in Table 2; 70.4 vs 68.4 in Table 2). Without across-seed variance, the claim of ‘consistent’ improvement and the ordering of ablations is not statistically supported. Reporting mean±s.e. over at least 3 seeds (or bootstrap CIs) is required to support the headline claims.
  3. [Table 3, Table 2] The LIBERO-Lo results use a different base initialization and changed W, b, λ_sup, λ_cor (Table 3), so the average in Table 1 is not controlled across suites. Moreover, all ablations (Table 2) cover only Spatial/Object/Goal, not Long. This weakens the cross-suite claim of consistency. If the central claim is general, the same hyperparameter configuration should be used across suites, or per-suite tuning must be justified with sensitivity analyses, and Long should be included in ablations.
  4. [Eq. (5), (7), Table 7] The corrective targets used in L_cor are derived from positive chunks labeled by the same proxy score \hat{A}_t (Eq. 5) that gates the attraction term. Because no independent validation of the proxy score's accuracy is provided, the redirection signal may reinforce bias in the score rather than actual task progress. Table 7 also shows that only 10.5–13.7% of negative chunks receive correction, so the corrective-matching component is active on a small subset. The authors should add a validation of \hat{A}_t (e.g., correlation with chunk-level human labels or success of target actions in sim) and report how the low correction coverage interacts with the small L_cor gain.
minor comments (5)
  1. [§3.1, Eq. (1)] The reward r(o_t,a_t,l) is never specified; clarify whether it is a sparse terminal reward or has intermediate components. In Eq. (5), the offsets t±W are used as integer indices; state explicitly that W is an integer.
  2. [Appendix A, Eq. (26)] The coefficient λ_bc appears in Eq. (26) without definition; in Eq. (40) the corresponding first term has no coefficient. Align the notation between the transport energy and the practical objective.
  3. [Fig. 5] The x-axis labels appear garbled in the rendered text and the caption should state that each curve is a single run. Also specify the exact number of rollouts collected per update step for each on-policy baseline.
  4. [Table 2 caption] The table reports averages over three suites (Spatial/Object/Goal), while Table 1 averages over four suites. State this explicitly in the caption to avoid confusion.
  5. [Eq. (12)] The margin m is described as ‘adaptive’ via a stop-gradient running average, but the initial value and update frequency are not given. Please report these details.

Circularity Check

0 steps flagged

No significant circularity: corrective targets are explicit constructions and the claimed gains are externally evaluated on held-out episodes.

full rationale

RedFlow's pipeline is a transparent construction rather than a hidden identification. Eq. (5) defines a proxy action-level advantage from smoothed GRM progress plus trajectory outcome; Eq. (6) defines the clustering feature; Eq. (7) defines each corrective target as the advantage-weighted centroid of positive chunks in the same HDBSCAN cluster; Eqs. (9)-(13) define the attraction, suppression, and correction losses directly in terms of these quantities. Theorems 1-7 are internal consistency statements about the defined objective—e.g., Theorem 6 shows that the endpoint minimizer is the projection of the corrective target outside the obstacle ball—rather than independent predictions. The central empirical claim, higher success on held-out LIBERO and real-robot episodes, is not an algebraic consequence of these definitions; it is measured against external success criteria. The only self-citations ([19] Halo, [20] WMPO) appear in background enumerations and are not load-bearing. The acknowledged dependence on GRM accuracy and clustering quality (Appendix F) is an external assumption, not a circular reduction, and the paper's own limitations section identifies this dependency explicitly.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 0 invented entities

The central claim rests on the reliability of GRM progress estimates, the validity of progress-state clustering, and the assumption that a centroid of successful actions is a useful correction for nearby failed actions. The paper itself lists these dependencies in Appendix F. Numerous hyperparameters (W, b, T_w, λ_sup, λ_cor, κ, β, margin, HDBSCAN settings) are hand-set or suite-specific and directly affect the reported results.

free parameters (9)
  • Progress smoothing window W = 10 (Spatial/Object/Goal), 20 (Long)
    Controls temporal smoothing of GRM progress estimates in Eq. (4); changed per suite, affecting which chunks are labeled positive/negative.
  • Outcome-bias coefficient b = 0.15 (Spatial/Object/Goal), 0.25 (Long)
    Mixes episode outcome into the proxy advantage Eq. (5); suite-specific tuning affects the action-level labels.
  • Soft-weight temperature T_w = 3.0
    Converts the proxy advantage into training weight w_t in Eq. (9), controlling how strongly positive and negative chunks are separated.
  • Suppression strength λ_sup = 0.3 (Spatial/Object/Goal), 0.1 (Long)
    Scales the hinge suppression loss Eq. (12); changed per suite to balance suppression against correction.
  • Correction strength λ_cor = 0.3 (Spatial/Object/Goal), 0.1 (Long)
    Scales the corrective-target pull Eq. (13); suite-specific, directly affects how strongly failures are redirected.
  • Corrective-target temperature κ = 1.0
    Controls concentration of the corrective target toward high-advantage positive chunks in Eq. (7).
  • Progress weight β = 5.0
    Relative scale of task progress vs. proprioceptive state in the clustering feature Eq. (6), affecting cluster assignment and hence which failures get corrective targets.
  • Margin m = Adaptive stop-gradient running average; scale 1.0 in real-robot
    Defines the finite-radius suppression ball in Eq. (12); the adaptive margin influences when suppression becomes active.
  • HDBSCAN min_cluster_size / min_samples = 15/5 (LIBERO), 50/10 (real)
    Controls granularity of execution-context clusters and therefore the availability and reliability of corrective targets.
axioms (6)
  • domain assumption The pretrained GRM R(o_t,l) yields progress estimates accurate enough to sign action-level advantages.
    Eqs. (4)-(5) use R to compute p̄_t and Â_t; Appendix F explicitly states that inaccurate progress signals may lead to imperfect failure-to-success matching.
  • domain assumption HDBSCAN clusters in [q̃_t; β p̄_t] correspond to semantically similar execution contexts.
    Section 3.2 defines execution contexts this way; the method assumes that chunks close in this space are interchangeable enough for corrective-target retrieval.
  • domain assumption The advantage-weighted positive centroid a*_t is a valid corrective direction for co-clustered negative chunks.
    Eq. (7) constructs a*_t as a barycenter of positive chunks; the paper acknowledges in Appendix F that failures lacking nearby positive support are only suppressed, not corrected.
  • standard math Endpoint-space transport with squared Euclidean action distance faithfully implements the flow-matching policy update.
    Appendix A.4-A.5 derives the push-pull velocity field and endpoint regression equivalence; this is standard geometry for Euclidean action spaces.
  • domain assumption The Wasserstein gradient-flow characterization transfers to finite-sample deep-network optimization.
    Appendix A.6 proves energy dissipation for an idealized particle flow, while the paper acknowledges that global convergence is intractable for non-linear deep networks.
  • standard math Linear flow-matching interpolation x0 = x_n - n·v_θ is the correct denoising parameterization.
    Used throughout Appendix A.5 to translate endpoint losses into velocity regression; consistent with flow-matching theory.

pith-pipeline@v1.3.0-daily-deepseek · 19757 in / 11371 out tokens · 112722 ms · 2026-08-01T01:27:11.587114+00:00 · methodology

0 comments
read the original abstract

Flow-matching Vision-Language-Action (VLA) policies have shown strong potential for robotic manipulation but often suffer from compounding errors caused by distribution shifts during deployment. While offline reinforcement learning (RL) provides a practical way to improve deployed policies using rollout data, existing methods either ignore failure data or exploit it only at the trajectory level, resulting in low learning efficiency and persistent errors. We propose **RedFlow**, a fine-grained offline RL framework that redirects failure experiences into action-level corrective supervision for flow-matching VLA policies. RedFlow consists of two key components: (1) a **Context-Aware Corrective Matching** mechanism that identifies failure-inducing actions and retrieves successful alternatives from similar contexts as corrective targets, and (2) an **Adaptive Redirection Objective** that jointly reinforces successful actions, suppresses undesirable ones, and redirects recoverable failures toward corrective targets. By converting both successful and failed experiences into dense supervision, RedFlow enables robust recovery learning from mixed-quality data. Experiments on the LIBERO benchmark and three real-world manipulation tasks show that RedFlow consistently outperforms state-of-the-art offline RL baselines, improving the real-world success rate from 56.7% to 74.7%. It also matches strong on-policy methods (PPO, GRPO, and DDPO) while requiring roughly an order of magnitude fewer training samples.

Figures

Figures reproduced from arXiv: 2607.27782 by Fangqi Zhu, Junhao Li, Quanxin Shou, Song Guo, Xiaoyi Pang, Yikun Miao, Zhengyang Yan, Zicong Hong, Zijun Wang.

Figure 1
Figure 1. Figure 1: RedFlow combines two properties that prior methods lack: learning from failures and action-level guidance. (a) Preference methods learn from failures but only at the trajectory level, signaling what to avoid without action-level corrections. (b) RedFlow identifies failure-inducing actions within failed rollouts and redirects them toward corrective targets derived from successful experiences, providing dens… view at source ↗
Figure 2
Figure 2. Figure 2: RedFlow pipeline. RedFlow consists of three stages: (I) Data Collection, where success￾ful and failed rollouts are stored in an offline buffer; (II) Context-Aware Corrective Matching, which leverages a pre-trained GRM to estimate task progress, combines progress estimates with proprioceptive states to define execution contexts, clusters similar contexts with HDBSCAN, and derives corrective targets from pos… view at source ↗
Figure 3
Figure 3. Figure 3: Context-aware corrective matching in execution-context space. RedFlow represents each action chunk by its execution context (˜qt, p¯t), combining the robot’s proprioceptive state and task progress. Action chunks with similar contexts are grouped into execution-context clusters. Given a candidate failure chunk (red cross), RedFlow retrieves matched successful chunks from the corresponding cluster and uses t… view at source ↗
Figure 4
Figure 4. Figure 4: Real-robot experiment setups. We evaluate RedFlow on a dual-arm Agilex Cobot Magic [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Sample efficiency comparison on LIBERO-Spatial against on-policy RL baselines. Clothes Folding Object Sweeping Table Cleaning Avg 0 10 20 30 40 50 60 70 80 90 100 Success Rate (%) 36.0 63.0 71.0 56.7 48.0 66.0 78.0 64.0 41.0 69.0 76.0 62.0 67.0 73.0 84.0 74.7 Base Policy DPO AWR RedFlow [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative correction behavior on clothes folding. The base policy fails when the T-shirt [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Successful trajectory of cloth folding [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Successful trajectory of object sweeping. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Successful trajectory of table cleaning. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Failed trajectory of cloth folding. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Failed trajectory of object sweeping [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Failed trajectory of table cleaning. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 16 linked inside Pith

  1. [1]

    π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsc...

    Pith/arXiv arXiv 2025

  3. [3]

    Sanketi, Grecia Salazar, Michael S

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski...

    2023

  4. [4]

    Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018

    Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018. URLhttps://arxiv.org/abs/1805.00909

    Pith/arXiv arXiv 2018

  5. [5]

    Simplevla-rl: Scaling vla training via reinforcement learning, 2025

    Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, Dehui Wang, Dingxiang Luo, Yuchen Fan, Youbang Sun, Jia Zeng, Jiangmiao Pang, Shanghang Zhang, Yu Wang, Yao Mu, Bowen Zhou, and Ning Ding. Simplevla-rl: Scaling vla training via reinforcement learning, 2025. URL https: //arxiv.org/...

    Pith/arXiv arXiv 2025

  6. [6]

    Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning, 2025

    Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning, 2025. URLhttps://arxiv.org/abs/2505.18719

    Pith/arXiv arXiv 2025

  7. [7]

    Reinflow: Fine-tuning flow matching policy with online reinforcement learning

    Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching policy with online reinforcement learning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  8. [8]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019. URL https://arxiv.org/ abs/1910.00177

    Pith/arXiv arXiv 2019

  9. [9]

    Reinforcement learning by reward-weighted regression for operational space control

    Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. InProceedings of the 24th international conference on Machine learning, pages 745–750, 2007

    2007

  10. [10]

    Lundell, and Dongdong Chen

    Yiye Chen, Yanan Jian, Xiaoyi Dong, Shuxin Cao, Jing Wu, Patricio Vela, Benjamin E. Lundell, and Dongdong Chen. Vista: Enhancing visual conditioning via track-following preference optimization in vision-language-action models, 2026. URL https://arxiv.org/abs/2602. 05049

    2026

  11. [11]

    Grape: Generalizing robot policy via preference alignment.arXiv preprint arXiv:2411.19309, 2024

    Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment.arXiv preprint arXiv:2411.19309, 2024. 10

    Pith/arXiv arXiv 2024

  12. [12]

    Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szym...

    Pith/arXiv arXiv 2025

  13. [13]

    Hg- dagger: Interactive imitation learning with human experts

    Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In2019 International Conference on Robotics and Automation (ICRA), pages 8077–8083. IEEE, 2019

    2019

  14. [14]

    Precise and dexterous robotic manip- ulation via human-in-the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manip- ulation via human-in-the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    2025

  15. [15]

    Human-assisted robotic policy refinement via action preference optimization

    Wenke Xia, Yichu Yang, Hongtao Wu, Xiao Ma, Tao Kong, and Di Hu. Human-assisted robotic policy refinement via action preference optimization. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  16. [16]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. InConference on Robot Learning, pages 2679–2713. PMLR, 2025

    2025

  17. [17]

    Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024

    2024

  18. [18]

    Octo: An open-source generalist robot policy, 2024

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy, 2024. URL https://arxiv.org/abs/2405.12213

    Pith/arXiv arXiv 2024

  19. [19]

    Halo: A unified vision-language-action model for embodied multimodal chain-of-thought reasoning, 2026

    Quanxin Shou, Fangqi Zhu, Shawn Chen, Puxin Yan, Zhengyang Yan, Yikun Miao, Xiaoyi Pang, Zicong Hong, Ruikai Shi, Hao Huang, Jie Zhang, and Song Guo. Halo: A unified vision-language-action model for embodied multimodal chain-of-thought reasoning, 2026. URL https://arxiv.org/abs/2602.21157

    arXiv 2026

  20. [20]

    Wmpo: World model-based policy optimization for vision-language-action models.arXiv preprint arXiv:2511.09515, 2025

    Fangqi Zhu, Zhengyang Yan, Zicong Hong, Quanxin Shou, Xiao Ma, and Song Guo. Wmpo: World model-based policy optimization for vision-language-action models.arXiv preprint arXiv:2511.09515, 2025

    arXiv 2025

  21. [21]

    Rlinf: Flexible and efficient large-scale reinforcement learning via macro-to-micro flow transformation.arXiv preprint arXiv:2509.15965, 2025

    Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, Chunyang Zhu, Junhao Hu, et al. Rlinf: Flexible and efficient large-scale reinforcement learning via macro-to-micro flow transformation.arXiv preprint arXiv:2509.15965, 2025

    arXiv 2025

  22. [22]

    Co-rft: Efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning.arXiv preprint arXiv:2508.02219, 2025

    Dongchi Huang, Zhirui Fang, Tianle Zhang, Yihang Li, Lin Zhao, and Chunhe Xia. Co-rft: Efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning.arXiv preprint arXiv:2508.02219, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Rl-100: Performant robotic manipulation with real-world reinforcement learning.arXiv preprint arXiv:2510.14830, 2025

    Kun Lei, Huanyu Li, Dongjie Yu, Zhenyu Wei, Lingxiao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, and Huazhe Xu. Rl-100: Performant robotic manipulation with real-world reinforcement learning.arXiv preprint arXiv:2510.14830, 2025. 11

    arXiv 2025

  24. [24]

    Diffusion guidance is a controllable policy improvement operator.arXiv preprint arXiv:2505.23458, 2025

    Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a controllable policy improvement operator.arXiv preprint arXiv:2505.23458, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Roboreward: General-purpose vision-language reward models for robotics, 2026

    Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn. Roboreward: General-purpose vision-language reward models for robotics, 2026. URL https://arxiv.org/abs/2601.00675

    arXiv 2026

  26. [26]

    Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, and Jesse Zhang

    Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, and Jesse Zhang. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons, 2026. URL https: //arxiv.org/abs/...

    Pith/arXiv arXiv 2026

  27. [27]

    Robo-dopamine: General process reward modeling for high- precision robotic manipulation, 2025

    Huajie Tan, Sixiang Chen, Yijie Xu, Zixiao Wang, Yuheng Ji, Cheng Chi, Yaoxu Lyu, Zhongxia Zhao, Xiansheng Chen, Peterson Co, Shaoxuan Xie, Guocai Yao, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Robo-dopamine: General process reward modeling for high- precision robotic manipulation, 2025. URLhttps://arxiv.org/abs/2512.23703

    arXiv 2025

  28. [28]

    Kto: Model alignment as prospect theoretic optimization, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL https://arxiv.org/abs/ 2402.01306

    Pith/arXiv arXiv 2024

  29. [29]

    Flow match- ing for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow match- ing for generative modeling. In11th International Conference on Learning Representations, ICLR 2023, 2023

    2023

  30. [30]

    Accelerated hierarchical density based clustering

    Leland McInnes and John Healy. Accelerated hierarchical density based clustering. In2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 33–42, 2017. doi: 10.1109/ICDMW.2017.12

    doi:10.1109/icdmw.2017.12 2017

  31. [31]

    Libero: Benchmarking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neu- ral Information Processing Systems, volume 36, pages 44776–44791. Curran Associates, Inc., 2023. URL http...

    2023

  32. [32]

    URL https://arxiv.org/ abs/2510.25889

    Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, and Chao Yu.πRL: Online rl fine-tuning for flow-based vision-language-action models, 2026. URL https://arxiv.org/ abs/2510.25889

    arXiv 2026

  33. [33]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associ...

    2023

  34. [34]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URLhttps://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  35. [35]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    Pith/arXiv arXiv 2024

  36. [36]

    Training diffusion models with reinforcement learning, 2024

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning, 2024. URLhttps://arxiv.org/abs/2305.13301. 12 Appendix A Bounded Corrective Redirection as Constrained Wasserstein Policy Transport This appendix formalizes the endpoint-redirection principle stated in Theorem 1 and gives a broa...

    Pith/arXiv arXiv 2024