Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Failed robot rollouts can train VLAs if a VLM relabels them as successes at the tasks they actually did.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 00:45 UTC pith:6EQULQPS

load-bearing objection Solid online hindsight for VLA RL with real multi-backbone and Franka gains; the 5 imes claim is empirically supported but the off-policy ratio is under-checked. the 3 major comments →

arxiv 2607.09042 v1 pith:6EQULQPS submitted 2026-07-10 cs.LG

Learning More from Less: Reinforcement Learning from Hindsight

Iris Xu , Sunshine Jiang , John Marangola , Nitish Dashora , Richard Li , Thomas Liu , Zexue He , Yuheng Zhi
show 3 more authors
Alex Pentland Pulkit Agrawal Zhang-Wei Hong
This is my paper
classification cs.LG
keywords hindsight relabelingvision-language-action modelsreinforcement learningsample efficiencysparse rewardsrobot manipulationGRPOVLM reward models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Robot fine-tuning with reinforcement learning is starved for data: each update needs slow physical rollouts, and sparse success rewards make early failures look useless even when the robot executed coherent skills. This paper argues that those failures are successes under different language instructions, and that a vision-language model can recover that signal. Learning from Hindsight (LfH) takes groups of failed rollouts, has one VLM invent a shared hindsight instruction describing what the robot actually did, scores every trajectory against that instruction, and jointly trains the policy on original and relabeled data with an importance-corrected group-relative update. Because VLAs already generalize across language, credit for nearby skills transfers back to the commanded tasks. On hard out-of-distribution LIBERO-PRO suites the method reaches standard RL's final success in roughly one-fifth the steps (about 5 imes sample efficiency), beats a dense progress-reward baseline, and the gains transfer across VLA backbones and to a real Franka arm.

Core claim

On out-of-distribution LIBERO-PRO tasks where standard sparse-reward GRPO improves only slowly from a weak initial policy, Learning from Hindsight recovers usable training signal from most failed rollouts by language-and-reward relabeling, reaches the final success rate of standard RL in roughly one-fifth the training steps (approximately 5 imes sample-efficiency gain), and outperforms a dense progress-reward baseline; the same gains hold across multiple VLA backbones and on a physical Franka robot.

What carries the argument

Learning from Hindsight (LfH): for low-mean-reward GRPO groups, a single VLM proposes one shared hindsight instruction from an anchor failed trajectory and assigns 0/0.5/1 rewards to the whole group under that instruction; the policy is then updated with both the original GRPO loss and an importance-corrected hindsight GRPO loss that evaluates actions under the new instruction while correcting for having been sampled under the old one.

Load-bearing premise

The whole method rests on one off-the-shelf vision-language model being able to invent a shared, accurate language description of what a group of failed robot videos actually achieved and to score them consistently enough that the resulting rewards give useful advantages rather than noise.

What would settle it

Run the same OOD LIBERO-PRO protocol with a deliberately degraded or blinded VLM relabeler (or with random hindsight instructions) and check whether the 5 imes sample-efficiency gap over standard GRPO and the dense-progress baseline disappears; if the gap remains, the VLM-quality premise is not load-bearing.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Each costly robot trajectory can supervise multiple language goals consistent with what actually happened, raising bits extracted per physical sample.
  • In low-success sparse-reward regimes, changing which task a failure is credited for can be more valuable than densifying progress rewards for the original command alone.
  • The same relabeling loop works across different VLA architectures and transfers from simulation to a physical Franka without changing the human binary reward.
  • Failed attempts cease to be pure negative examples and become reusable experience for nearby object interactions and grounding skills.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If VLM relabel quality continues to improve, the same recipe could turn large offline failure corpora into free multi-task post-training data without new robot collection.
  • Hindsight language credit may also help counteract behavioral collapse during RL, because alternative skills keep receiving positive updates instead of being uniformly treated as failures.
  • The method suggests a practical division of labor: let the VLA explore physical contact while a frozen VLM decides which language goals those contacts support.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Learning from Hindsight (LfH) for sample-efficient RL post-training of vision-language-action (VLA) policies under sparse rewards. Failed GRPO groups (near-zero mean reward) are converted into training signal by a single VLM that (i) proposes a shared hindsight instruction describing what an anchor failed rollout actually did and (ii) scores every trajectory in the group under that instruction with ternary rewards {0, 0.5, 1}. The policy is trained jointly on original and hindsight-relabeled groups via an importance-corrected GRPO objective (Eqs. 6–9, Algorithm 1). On out-of-distribution LIBERO-PRO task-perturbation suites, LfH is reported to match standard GRPO’s final success in roughly one-fifth the training steps (~5× sample efficiency), to outperform a dense progress-reward baseline (RoboMETER), and to transfer across Pi-0.5, GR00T, and OpenVLA-OFT as well as to a physical Franka task.

Significance. If the reported gains hold under scrutiny, LfH is a practical and timely contribution: robot rollouts are expensive, sparse-reward cold starts are common for VLA fine-tuning, and language-space hindsight is a natural fit for pretrained VLAs. Strengths include a clear algorithmic construction, careful same-budget comparisons with standard error over seeds, ablations that isolate rephrase-only, reward-only, and random-reward alternatives (all weaker than full LfH), group-selection and VLM-prompt ablations, multi-backbone transfer, and a real-robot result with human binary labels. The work is complementary to dense progress rewards and to offline instruction-augmentation methods, and it makes a concrete case that failed trajectories can be mined as multi-task supervision rather than discarded.

major comments (3)
  1. [Section 4.2, Eq. (7)] Section 4.2, Eq. (7): the hindsight importance ratio ˜r_{i,t}=π_θ(a|o,g′)/π_θold(a|o,g) is load-bearing for the claimed mechanism (language-mediated transfer from off-policy hindsight groups). When g′ is semantically distant from g (as in the microwave→mug examples of Fig. 2b), the two conditionals can differ substantially, producing high-variance or biased weights even after clipping. The manuscript never reports the distribution of these ratios, the fraction of mass affected by the hindsight-loss clip c (Appendix B.1), or an ablation that disables the correction / replaces it with on-policy re-rollout under g′. Without that check, it remains unclear how much of the ~5× gain in Fig. 3a is genuine language transfer versus noisy off-policy updates. A short diagnostic (ratio histograms, clip rates, and one ablation) would substantially strengthen the central efficiency claim’s interpretati
  2. [Section 5.1, Fig. 2b] Section 5.1 and Fig. 2b: the paper observes that hindsight instructions need not be semantically related to the commanded task, yet still improve target-task success, and offers a brief “contrastive grounding” hypothesis. This is central to the narrative that language generalization bridges hindsight successes back to Pg. The evidence is currently qualitative (one microwave example) plus the aggregate gain curves. A minimal quantitative check—e.g., success on the original instruction as a function of semantic distance between g and g′, or a controlled set of related vs. unrelated hindsight labels—would make the transfer story falsifiable rather than post-hoc. Absent that, the 5× claim remains empirically supported but mechanistically under-specified.
  3. [Section 5.4, Fig. 4b] Section 5.4 / Fig. 4b: real-world evaluation is limited to a single held-out task (“put the green container into the bowl”) with human binary rewards and a modest rollout budget. The sim results are multi-suite and multi-backbone; the physical claim in the abstract (“gains hold … on a physical Franka robot”) currently rests on one curve. At least one additional held-out task, or a clear statement that real-world evidence is preliminary and single-task, is needed so the abstract claim matches the evidence.
minor comments (5)
  1. [Section 4.1 / Algorithm 1] Eq. (1) vs. Algorithm 1 / Appendix B.1: the main text uses threshold η on mean group reward, while the appendix describes skipping groups with mean reward in [0.8, 1.0]. State η explicitly in the main method section and keep notation consistent.
  2. [Section 5.2, Fig. 3a] Figure 3a caption and §5.2: “Gain(t)=SRt/SR0−1” is undefined or unstable when SR0≈0 (as in the microwave illustrative example). Clarify how Gain is computed when the initial success rate is zero, or report absolute success for those cases.
  3. [Section 2] Related Work: concurrent open-ended instruction relabeling [16] is acknowledged; a one-sentence contrast on online GRPO integration and robotics evaluation would help readers place the contribution more sharply.
  4. [Appendix B.1] Appendix B.1: hyperparameters λ and clip c for the hindsight loss are described procedurally (“85th percentile”) but not given numerical values in the main text or a single table. A compact hyperparameter table would aid reproducibility.
  5. [Figures 3–4] Typos / polish: “Robometer” vs. “RoboMETER” inconsistency in Fig. 3/4 captions; “π 0.5” spacing in Fig. 3c caption; arXiv-style citation formatting is fine for preprint but should be cleaned for journal production.

Circularity Check

0 steps flagged

No significant circularity: LfH is an algorithmic construction whose sample-efficiency claims are measured by external environment/human success rates, not quantities defined from its own inputs.

full rationale

The paper defines Learning from Hindsight as a concrete procedure (VLM instruction/reward relabeling of low-reward GRPO groups, followed by the importance-corrected objective in Eqs. 6–9 and Algorithm 1) and evaluates it empirically. The central 5× sample-efficiency claim, the RoboMETER comparison, the multi-backbone transfer, and the Franka results are all reported as measured success rates (or Gain = SR_t/SR_0 − 1) under the original commanded instructions and environment/human binary rewards. Those metrics are independent of the VLM-generated hindsight labels; the labels affect only the training signal, not the evaluation definition. There is no fitted parameter that is later re-presented as a prediction, no self-definitional loop (X defined via Y and then used to derive Y), and no load-bearing uniqueness theorem or ansatz imported solely via overlapping-author citation. Related-work citations (HER, GRPO, RoboMETER, VLA backbones) are standard external references. The importance-ratio correction (Eq. 7) is a conventional off-policy adjustment, not a circular reduction. Consequently the derivation chain is self-contained against external benchmarks and exhibits none of the six circularity patterns.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central claim rests on standard RL and VLA machinery plus a small set of algorithmic choices (when to relabel, how to weight the hindsight loss, how to prompt the VLM). No new physical entities are postulated. The free parameters are ordinary hyper-parameters whose values are stated; the domain assumptions are the usual ones for language-conditioned robot RL plus the practical assumption that a frontier VLM can serve as a reliable hindsight oracle from RGB video.

free parameters (4)
  • hindsight group threshold η (mean reward) = 0.8
    Groups with mean original reward below η are selected for relabeling; default 0.8 chosen by ablation (Figure 5).
  • hindsight loss weight λ and clip c = c ≈ 85th percentile; λ set for loss balance
    L_LfH = L_GRPO + λ min(L_H-GRPO, c); c set near 85th percentile of early hindsight losses, λ chosen so weighted hindsight loss roughly matches original batch loss.
  • GRPO group size K / N = 8 (sim), 4 (real)
    Number of trajectories per instruction group used for advantage normalization and shared hindsight instruction; set to 8 (sim) / 4 (real).
  • VLM sampling temperature = 0.8
    Temperature for instruction and reward generation; fixed at 0.8.
axioms (5)
  • domain assumption A trajectory that fails under commanded instruction g can be a success under a different language instruction g' that describes the achieved behavior.
    Core premise of hindsight relabeling, stated in Section 4 and Figure 1; inherited from HER but applied in language space.
  • domain assumption A pretrained VLM can generate a shared, meaningful hindsight instruction from RGB video of an anchor failed trajectory and assign comparable 0/0.5/1 rewards to the whole group.
    Invoked in Eqs. 4–5 and Algorithm 1; quality is partially mitigated by Nothing/unsure options but not independently verified at scale.
  • domain assumption VLAs generalize across related language instructions, so training on hindsight tasks transfers to the original commanded distribution Pg.
    Stated in the abstract and Introduction; required for hindsight successes to improve original-task success rate.
  • standard math Importance ratio π_ heta(a|o,g') / π_ heta old(a|o,g) correctly corrects for the distribution shift between commanded and hindsight instructions inside GRPO.
    Eq. 7, following hindsight policy gradients; standard off-policy correction under the usual Markov assumptions.
  • standard math Groups with zero reward variance under the original sparse reward provide no learning signal and may be discarded or replaced.
    Section 3, GRPO advantage normalization; motivates activating LfH only on low-signal groups.
invented entities (1)
  • Learning from Hindsight (LfH) joint instruction-and-reward relabeling procedure independent evidence
    purpose: Convert all-zero GRPO groups into usable training signal by VLM-proposed hindsight instructions and scores, then train jointly with original groups.
    The algorithmic object introduced by the paper (Section 4, Algorithm 1). It is a procedure, not a physical entity; independent evidence is the empirical transfer to held-out tasks and a real robot.

pith-pipeline@v1.1.0-grok45 · 21387 in / 3641 out tokens · 29589 ms · 2026-07-13T00:45:19.036202+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior. Such a failure, however, is a success at a different task. We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training of VLAs by scoring failed rollouts against the tasks they actually achieved. A single vision-language model relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories. On out-of-distribution LIBERO-PRO tasks, where standard RL improves only slowly, LfH achieves $5\times$ improvement in sample efficiency, and outperforms a dense progress-reward baseline. The gains hold across VLA backbones and on a physical Franka robot.

Figures

Figures reproduced from arXiv: 2607.09042 by Alex Pentland, Iris Xu, John Marangola, Nitish Dashora, Pulkit Agrawal, Richard Li, Sunshine Jiang, Thomas Liu, Yuheng Zhi, Zexue He, Zhang-Wei Hong.

Figure 1
Figure 1. Figure 1: Learning from Hindsight. A trajectory that fails the commanded instruction can still exhibit meaningful behavior. LfH uses a VLM to relabel the achieved behavior in language, turning otherwise unused failures into training signal for VLA fine-tuning. such relabeled rollouts provide useful supervision from data that standard RL would ignore. The remaining challenge is to infer, in language, what the robot a… view at source ↗
Figure 2
Figure 2. Figure 2: (a) Fraction of groups kept in GRPO versus LfH during training. LfH keeps more trajectory groups for training through relabeling, while GRPO discards most groups. LfH also escapes the weak initial policy with zero success rate and reaches 60% success. (b) Example hindsight relabeled trajectories with their respective VLM-relabled instructions. Although the relabeled instructions are not the target task of … view at source ↗
Figure 3
Figure 3. Figure 3: (a) Gain during training for vanilla GRPO, Robometer, and LfH. LfH improves sample efficiency over GRPO and RoboMETER, achieving higher gain with fewer training steps. (b) The fraction of groups kept during training. LfH keeps a larger fraction of trajectory groups usable for training, making better use of collected data. (c) Final gain for other VLA models. The benefit of LfH transfers beyond π0.5, improv… view at source ↗
Figure 4
Figure 4. Figure 4: (a) Comparison of full LfH with ablations that remove/modify parts of the hindsight relabeling mechanism. LfH’s gains come from jointly relabeling instructions and rewards; instruction rephrasing or reward relabeling alone, or additional reward variation does not substantially improve performance. (b) Real-world physical-robot fine-tuning results. LfH improves success rate over GRPO across rollout budgets.… view at source ↗
Figure 5
Figure 5. Figure 5: (a) Selecting groups with mean reward below 0.8 outperforms random selection and relabeling all groups, showing that targeted relabeling is important. (b) Among the tested thresholds, 0.8 performs best, suggesting that LfH benefits from relabeling a broad set of non-saturated groups while avoiding indiscriminate relabeling of all groups. As shown in Figure 5a, threshold-based group selection is important. … view at source ↗
Figure 6
Figure 6. Figure 6: (a) Increasing the number of GRPO update epochs does not match LfH, showing that the main bottleneck is not insufficient optimization on the same rollout data. (b) Increasing PPO update epochs also fails to consistently match LfH, further indicating that LfH improves sample efficiency by making each collected trajectory more informative rather than by simply reusing data more aggressively. How does data di… view at source ↗
Figure 7
Figure 7. Figure 7: Ablation on the noise level used for rollout sampling. As the rollout sampling noise level [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: VLM prompt ablation. Removing the scene object list, the [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Random mid-training rollouts on the microwave task for [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Full training curves for Pi-0.5 0 100 200 Training Steps 0.05 0.10 0.15 Success Rate LIBERO-Pro-Goal 0 100 200 Training Steps 0.025 0.050 0.075 0.100 LIBERO-Pro-Object 0 100 200 Training Steps 0.1 0.2 0.3 0.4 LIBERO-Pro-Spatial 0 100 200 Training Steps 0.05 0.10 0.15 Max Success Rate 0 100 200 Training Steps 0.06 0.07 0.08 0.09 0.10 0 100 200 Training Steps 0.1 0.2 0.3 0.4 GRPO +LfH (Ours) [PITH_FULL_IMA… view at source ↗
Figure 11
Figure 11. Figure 11: Full training curves for GR00T 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Full training curves for OpenVLA-OFT 0 20 40 60 Training Steps 0.20 0.22 0.24 0.26 0.28 Success Rate LIBERO-Pro-Goal × position 0 20 40 60 Training Steps 0.000 0.025 0.050 0.075 0.100 LIBERO-Pro-Object × position 0 20 40 60 Training Steps 0.3 0.4 0.5 0.6 LIBERO-Pro-Spatial × position 0 20 40 60 Training Steps 0.20 0.22 0.24 0.26 0.28 Max Success Rate 0 20 40 60 Training Steps 0.00 0.02 0.04 0.06 0.08 0 20… view at source ↗
Figure 13
Figure 13. Figure 13: Full training curves for Pi-0.5 on position perturbation [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Realworld experiment setup [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Qualitative examples of LfH relabeling. For each rollout, we show the original instruction and the VLM-generated hindsight instruction describing what the robot actually did. Meaningful failures are relabeled as alternative successful behaviors, uninformative rollouts are filtered with a “nothing” label, and rare VLM hallucinations remain possible. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 15 linked inside Pith

  1. [1]

    L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey.Journal of Artificial Intelligence Research, 4:237–285, 1996

    1996

  2. [2]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems, volume...

    2022

  3. [3]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Schneider, A. Stone, Q. Tan, J. Tanner, M. Toussaint, Q. Vuong, and A. Zeng. π0: A vision-language-action flow model for general robot control.arXiv preprint a...

    Pith/arXiv arXiv 2024

  5. [5]

    Bellemare, S

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation.Advances in neural information processing systems, 29, 2016

    2016

  6. [6]

    Pathak, P

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. InInternational conference on machine learning, pages 2778–2787. PMLR, 2017

    2017

  7. [7]

    Andrychowicz, F

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. InAdvances in Neural Information Processing Systems, 2017

    2017

  8. [8]

    Rauber, A

    P. Rauber, A. Ummadisingu, F. Mutz, and J. Schmidhuber. Hindsight policy gradients.arXiv preprint arXiv:1711.06006, 2017

    Pith/arXiv arXiv 2017

  9. [9]

    Pathak, P

    D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y . Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. InProceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 2050–2053, 2018

    2050

  10. [10]

    Eysenbach, T

    B. Eysenbach, T. Zhang, R. Salakhutdinov, and S. Levine. Contrastive learning as goal- conditioned reinforcement learning. InAdvances in Neural Information Processing Systems, 2022

    2022

  11. [11]

    Sahni, T

    H. Sahni, T. Buckley, P. Abbeel, and I. Kuzovkin. Visual hindsight experience replay.Proc. of NeurIPS, 2019

    2019

  12. [12]

    Liang, Y

    A. Liang, Y . Korkmaz, J. Zhang, M. Hwang, A. Anwar, S. Kaushik, A. Shah, A. S. Huang, L. Zettlemoyer, D. Fox, et al. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons.arXiv preprint arXiv:2603.02115, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    X. Zhou, Y . Xu, G. Tie, Y . Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun. Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization.[arXiv preprint arXiv:2510.03827], 2025. 9

    Pith/arXiv arXiv 2025

  15. [15]

    Z. Zhou, P. Atreya, A. Lee, H. Walke, O. Mees, and S. Levine. Autonomous improvement of instruction following skills via foundation models, 2024. URL https://arxiv.org/ abs/2407.20635

    Pith/arXiv arXiv 2024

  16. [16]

    Zhang, Z

    Z. Zhang, Z. Wang, Y . Du, and F. Fang. Learning instruction-following policies through open- ended instruction relabeling with large language models.arXiv preprint arXiv:2506.20061, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    A. Y . Ng, D. Harada, and S. J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. InProceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, page 278–287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122

    1999

  18. [18]

    Y . J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V . Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training.arXiv preprint arXiv:2210.00030, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Y . J. Ma, V . Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representa- tions and rewards for robotic control. InInternational Conference on Machine Learning, pages 23301–23320. PMLR, 2023

    2023

  20. [20]

    Y . J. Ma, J. Hejna, C. Fu, D. Shah, J. Liang, Z. Xu, S. Kirmani, P. Xu, D. Driess, T. Xiao, et al. Vision language models are in-context value learners. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  21. [21]

    S. Zhai, Q. Zhang, T. Zhang, F. Huang, H. Zhang, M. Zhou, S. Zhang, L. Liu, S. Lin, and J. Pang. A vision-language-action-critic model for robotic real-world reinforcement learning. arXiv preprint arXiv:2509.15937, 2025

    arXiv 2025

  22. [22]

    Y . Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson. Rl-vlm-f: Reinforcement learning from vision language foundation model feedback.arXiv preprint arXiv:2402.03681, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. InRobotics: Science and Systems, 2023. URLhttps://arxiv.org/abs/2211.11736

    Pith/arXiv arXiv 2023

  24. [24]

    Zhang, K

    J. Zhang, K. Pertsch, J. Zhang, and J. J. Lim. Sprint: Scalable policy pre-training via language instruction relabeling. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 9168–9175. IEEE, 2024

    2024

  25. [25]

    Glossop, W

    C. Glossop, W. Chen, A. Bhorkar, D. Shah, and S. Levine. Cast: Counterfactual labels improve instruction following in vision-language-action models.arXiv preprint arXiv:2508.13446, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    S. Yang, H. Li, B. Wang, Y . Chen, Y . Tian, T. Wang, H. Wang, F. Zhao, Y . Liao, and J. Pang. Instructvla: Vision-language-action instruction tuning from understanding to manipulation. arXiv preprint arXiv:2507.17520, 2025

    arXiv 2025

  27. [27]

    Y . Wu, A. Li, T. Hermans, F. Ramos, A. Bajcsy, and C. P˜AˇSrez-D’Arpino. Do what you say: Steering vision-language-action models via runtime reasoning-action alignment verification. 2026 IEEE International Conference on Robotics & Automation (ICRA), 2026

    2026

  28. [28]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 10

    Pith/arXiv arXiv 2024

  30. [30]

    K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, Q. Zhang, Z. Yu, G. Fan, T. Huang, Y . Wang, and C. Yu.πRL: Online rl fine-tuning for flow-based vision-language-action models, 2025. URLhttps://arxiv.org/abs/2510.25889

    arXiv 2025

  31. [31]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  32. [32]

    Bjorck, F

    J. Bjorck, F. Casta˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Nothing” option that allows the relabeler to identify uninteresting trajectories, which are then filtered out during training, and (3) an “unsure

    Y . LeCun. A path towards autonomous machine intelligence.OpenReview, 2022. 11 A Additional experimental results A.1 Additional ablation studies What is the best way to select groups for hindsight relabeling?We ablate which rollout groups should be selected for hindsight relabeling. Relabeling every trajectory is not necessarily beneficial: VLM relabeling...

    2022

  34. [34]

    pick up the mug and place it in the microwave

    Which object(s) did the gripper physically contact? 2. Where did the object(s) move to? 3. What is the final spatial arrangement? Rules: - Write ONE imperative instruction (e.g., "pick up the mug and place it in the microwave"). - Use SHORT, GENERIC names for objects: strip any numeric suffixes ( 1, 2) and brand/material prefixes (e.g. "akita black bowl 1...

  35. [35]

    pick up the tape

    FIRST FRAME: Describe the positions of all relevant objects at the start. 2. LAST FRAME: Describe the positions of all relevant objects at the end. 3. CHANGES: What specifically changed between the first and last frames? List each object that moved and where it moved to. 4. VERIFICATION: For each requirement in the instruction, state whether it is satisfi...