Pith. sign in

REVIEW 3 major objections 6 minor 76 references

A video-language model can be the sole reward that teaches robots new tasks from scratch.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 16:09 UTC pith:KEN26KA6

load-bearing objection A real, usable sole-reward video reasoner for from-scratch online robot RL, with honest failure analysis and stronger anti-hacking behavior than GPT/Gemini baselines. the 3 major comments →

arxiv 2603.28730 v2 pith:KEN26KA6 submitted 2026-03-30 cs.RO cs.CLcs.CV

SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

Philip Schroeder , Thomas Weng , Karl Schmeckpeper , Eric Rosen , Stephen Hart , Ondrej Biza This is my paper
classification cs.RO cs.CLcs.CV
keywords video-language modelsreinforcement learningrobot learningreward modelschain-of-thoughtprogress predictionzero-shot online RLmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

General vision-language models look strong at describing scenes, yet when used as rewards for robot reinforcement learning they often get fooled: policies learn to look successful without finishing the job. This paper argues that the missing piece is video-native spatiotemporal reasoning trained for dense progress estimation. The authors build SOLE-R1, a model that watches raw video, reasons step by step about what changed and whether it advances a natural-language goal, and outputs a continuous progress score usable as the only reward. Training combines a large synthesis pipeline of expert and deliberately non-expert trajectories with foundational spatial and multi-frame data, then refines the progress numbers with reinforcement learning on verifiable targets. In four simulators and on a real robot, policies that start random can learn previously unseen manipulation tasks without ground-truth rewards, demonstrations, or task-specific engineering, succeeding on more tasks and resisting reward hacking better than strong baselines.

Core claim

SOLE-R1 can serve as the sole reward signal for zero-shot online reinforcement learning from random initialization: given only raw video and a natural-language goal, its per-timestep chain-of-thought progress estimates let robots learn 24 previously unseen manipulation tasks across four simulators and a real-robot setting, without ground-truth rewards, success detectors, demonstrations, or task-specific tuning, while outperforming specialized and general vision-language rewarders and resisting reward hacking more often.

What carries the argument

SOLE-R1: a video-language model that, at every timestep, produces spatiotemporal chain-of-thought reasoning over raw frames plus a dense task-progress score; trained by a synthesis pipeline of temporally grounded CoT and continuous progress labels (including structured non-expert failures), mixed with foundational spatial and multi-frame data, then refined by supervised fine-tuning followed by reinforcement learning from verifiable rewards.

Load-bearing premise

The synthesized progress labels and chain-of-thought traces remain accurate and hard to game when the robot explores new viewpoints, embodiments, and partial views that never appeared in the training videos.

What would settle it

Run the same zero-shot online RL protocol on held-out tasks and a real robot while logging both SOLE-R1’s predicted progress and true task success: if policies routinely reach high predicted progress without true completion (reward hacking) or fail to improve under the dense signal on most of the 24 tasks, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SOLE-R1, a video-language model that produces per-timestep spatiotemporal chain-of-thought reasoning and dense task-progress estimates from raw video and a natural-language goal, intended to serve as the sole reward for online robot RL. Training combines a large-scale synthesis pipeline (structured non-expert trajectory levels, simulator geometry progress, real-video temporal reversals, and grounded CoT) with foundational spatial/multi-frame reasoning data, using hybrid SFT plus RL from verifiable rewards. The central empirical claim is zero-shot online RL from random initialization on 24 previously unseen manipulation tasks across four simulators and a real-robot setting, without ground-truth rewards, success detectors, demonstrations, or task-specific tuning, with substantial gains over Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro and reduced reward hacking (supported by perceived-vs-true success analysis, ablations, task-diversity scaling, OpenX VOC, and limited real-robot runs).

Significance. If the zero-shot sole-reward results hold under clean task holdout, this is a meaningful step toward reusable, language-conditioned reward models that reduce hand-engineered rewards for on-robot RL. Strengths include: (i) a clear failure taxonomy separating reward hacking from signal-limited failures (Figure 4, Appendix C); (ii) ablations that isolate CoT, non-expert coverage, and foundational spatial/temporal data (Section 5.5, Figure 5); (iii) scaling with training-task diversity (Section 5.6, Figure 6); (iv) OpenX VOC comparison to GVL (Table 7); and (v) planned full release of models, data, and RL code. These elements make the contribution more than another VLM-as-reward baseline and provide reusable methodology for evaluating reward exploitability.

major comments (3)
  1. [Section 5 / Table 5 / Appendix J] Central claim of 24 'unseen' tasks (Abstract, Section 5): training synthesis is dominated by RoboCasa families (PnP, drawers/doors, appliances; Table 5 and Appendix J) that match the stated evaluation families (pick-and-place, articulated objects, buttons/levers/knobs). The manuscript does not provide an explicit task-level holdout table enumerating, for each of the 24 tasks, environment, scene/layout/camera, embodiment, and whether any related RoboCasa/OXE source was in the progress-synthesis mixture. Without that ledger, the zero-shot claim cannot be audited for instance-level vs family-level novelty, which is load-bearing for the paper's main result.
  2. [Appendix I / Section 5] Real-robot portion of the central claim (Abstract; Appendix I): the real setup is described (Franka FR3, asynchronous actor/learner/reward server, ~1.5k actions/hour), but quantitative success rates, episode budgets, number of tasks beyond open/close drawer, and head-to-head baseline comparisons on hardware are not reported at the same standard as the simulation suites. Given that the title and abstract advertise on-robot RL with SOLE-R1 as sole reward, the real-robot evidence needs comparable metrics (success curves, perceived-vs-true analysis, and baseline failures) or the claim should be narrowed to 'sim + limited real validation.'
  3. [Section 5.4 / Figure 4 / Figure 8 / Appendix C] Sole-reward premise under online shift (Section 5.4, Figure 4, Appendix C, Limitations): the paper correctly shows SOLE-R1 fails more often as signal-limited (flat/noisy progress, low predicted–GT correlation) than via classic hacking. That supports robustness relative to GPT-5/Gemini, but it also means the 'sole reward' claim depends on online rollouts remaining in regions where progress is sufficiently shaped. A load-bearing addition would be a quantitative coverage check: e.g., fraction of online states with brief events/occlusions/goal-adjacent shortcuts (Table 1 modes) and how often predicted progress correlates with GT on failed vs successful tasks (Figure 8). Without this, residual calibration risk under distribution shift remains asserted rather than bounded.
minor comments (6)
  1. [Section 5] Main-text numerical results for the 24-task suite (per-environment success rates, sample budgets, seeds) are hard to reconstruct from the provided narrative and figures alone; a compact results table in Section 5 would strengthen reproducibility.
  2. [Section 1 / Figure 2] Figure 2 is referenced for video-native spatial/temporal integration but is not described in the manuscript body; ensure all early figures are present and captioned in the camera-ready version.
  3. [Appendix E] RLVR stage (Section 3 / Appendix E, Figure 9): specify the exact verifiable reward for progress (tolerance, discretization, format checks) so the hybrid recipe can be reproduced without reverse-engineering.
  4. [Appendix H] Baseline prompting fairness (Appendix H, Figure 10) is only shown for Meta-World; briefly state whether GPT-5/Gemini comparisons elsewhere used the same multi-frame protocol and temperature settings.
  5. [Section 7] Limitations and Broader impact are appropriately cautious; consider cross-linking them to the signal-limited failure quadrant so readers see residual risk is measured, not only stated.
  6. [Section 5.4 / Table 1] Minor polish: consistent naming of SOLE-R1 vs SOLE-R1, and ensure Table 1 error-mode frequencies (if claimed) appear with counts, not only qualitative labels.

Circularity Check

0 steps flagged

No significant circularity: progress labels and RL success metrics are independent of the model being evaluated.

full rationale

SOLE-R1 is an empirical systems paper, not a first-principles derivation. Continuous progress supervision in simulation is taken from ground-truth simulator geometry and normalized per trajectory (Appendix J.3); real-world progress inherits expert temporal order with explicit reversal-as-regression labels (J.4). The hybrid SFT+RLVR stage optimizes against those fixed labels (verifiable progress correctness), not against the online RL success metric. Zero-shot online RL claims are scored by independent ground-truth success / max ground-truth reward (Figure 4, Section 5.4), explicitly separated from the model’s own perceived progress. Held-out tasks, embodiments, and viewpoints are outside the training task set by construction of the evaluation suites. Self-citations and related-work citations are not load-bearing uniqueness claims that force the result. Use of foundation VLMs only for CoT paraphrasing does not define the progress targets or the evaluation success criteria. No step reduces a claimed prediction to its own fitted input or to a self-definitional identity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central claim rests on empirical engineering choices (how non-expert trajectories and progress are defined, how CoT is generated, hybrid training) rather than free physical constants. Invented entities are the model and pipeline themselves; domain assumptions are standard for dense-reward RL and video progress modeling.

free parameters (3)
  • non-expert trajectory level sampling fractions
    Uniform per-level mass within each task family (Table 4) is a design choice that shapes the progress distribution the model sees; ablations show expert-only mixtures increase hacking.
  • SFT category-level data mixture weights
    Category balancing across progress, embodied, and general spatial/temporal data (Table 5 / Section K) is chosen rather than proportional sampling and affects hacking vs signal trade-offs.
  • RLVR progress-verification reward design
    How scalar progress correctness is scored during RLVR is a free training objective choice that concentrates learning on the answer channel after SFT.
axioms (3)
  • domain assumption Continuous task progress can be defined from simulator geometry (or inherited expert ordering plus temporal reversal) and used as a dense reward that is policy-invariant enough for online RL.
    Standard reward-shaping premise (cf. Ng et al.) invoked throughout Sections 3 and J; if progress is misaligned with true completion under partial views, learning fails or hacks.
  • ad hoc to paper Per-timestep spatiotemporal CoT improves calibration of progress estimates relative to answer-only training.
    Supported by No-CoT ablation (Section 5.5 / Appendix D) but remains an empirical modeling assumption of the method.
  • domain assumption Foundation VLMs can paraphrase templated or progress-conditioned explanations without destroying temporal grounding.
    Used in CoT generation for both sim and real video (J.5); errors here would inject hallucinated success narratives.
invented entities (2)
  • SOLE-R1 video-language progress reasoner independent evidence
    purpose: Emit per-timestep CoT and dense progress used as sole online RL reward.
    The model is the paper's primary artifact; independent evidence is the released checkpoints and multi-environment RL results, not an external physical prediction.
  • Video trajectory and reasoning synthesis pipeline (non-expert levels + CoT + progress) independent evidence
    purpose: Create >1M temporally grounded training examples from real and simulated videos.
    Core methodological invention enabling the reward model; falsifiable via ablations that remove non-expert or foundational data.

pith-pipeline@v1.1.0-grok45 · 20458 in / 2864 out tokens · 30137 ms · 2026-07-13T16:09:10.928182+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: https://philip-mit.github.io/sole-r1/

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 36 linked inside Pith

  1. [1]

    Alakuijala, M., McLean, R., Woungang, I., Farsad, N., Kaski, S., Marttinen, P., and Yuan, K. (2024). Video-language critic: Transferable reward functions for language-conditioned robotics. arXiv preprint arXiv:2405.19988

    arXiv 2024

  2. [2]

    Ankile, L., Jiang, Z., Duan, R., Shi, G., Abbeel, P., and Nagabandi, A. (2025a). Residual off-policy rl for finetuning behavior cloning policies.arXiv preprint arXiv:2509.19301

    arXiv

  3. [3]

    Ankile, L., Simeonov, A., Shenfeld, I., Torne, M., and Agrawal, P. (2025b). From imitation to refinement—residual rl for precise assembly. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 01–08. IEEE

  4. [4]

    Baumli, K., Baveja, S., Behbahani, F., Chan, H., Comanici, G., Flennerhag, S., Gazeau, M., Holsheimer, K., Horgan, D., Laskin, M., et al. (2023). Vision-language models as a source of rewards. arXiv preprint arXiv:2312.09187

    Pith/arXiv arXiv 2023

  5. [5]

    J., Platt, R., van de Meent, J.-W., and Wong, L

    Biza, O., Weng, T., Sun, L., Schmeckpeper, K., Kelestemur, T., Ma, Y . J., Platt, R., van de Meent, J.-W., and Wong, L. L. (2025). On-robot reinforcement learning with goal-contrastive rewards. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 4797–4805. IEEE

    2025

  6. [6]

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. (2024). pi 0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164

    Pith/arXiv arXiv 2024

  7. [7]

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y ., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. (2022). Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817

    Pith/arXiv arXiv 2022

  8. [8]

    Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., and Xia, F. (2024). Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401.12168

    Pith/arXiv arXiv 2024

  9. [9]

    R., and Goldberg, K

    Chen, K., Xie, S., Ma, Z., Sanketi, P. R., and Goldberg, K. (2025a). Robo2vlm: Visual question answering from large-scale in-the-wild robot manipulation datasets.arXiv preprint arXiv:2505.15517

    Pith/arXiv arXiv

  10. [10]

    J., Ren, Z., Ratliff, L

    Chen, S., Harrison, C., Lee, Y .-C., Yang, A. J., Ren, Z., Ratliff, L. J., Duan, J., Fox, D., and Krishna, R. (2026). Topreward: Token probabilities as hidden zero-shot rewards for robotics. arXiv preprint arXiv:2602.19313

    arXiv 2026

  11. [11]

    Chen, Y ., Tian, S., Liu, S., Zhou, Y ., Li, H., and Zhao, D. (2025b). Conrft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.05450

    Pith/arXiv arXiv

  12. [12]

    Dalal, M., Chiruvolu, T., Chaplot, D., and Salakhutdinov, R. (2024). Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks.arXiv preprint arXiv:2405.01534

    Pith/arXiv arXiv 2024

  13. [13]

    Du, Y ., Konyushkova, K., Denil, M., Raju, A., Landon, J., Hill, F., de Freitas, N., and Cabi, S. (2023). Vision-language models as success detectors. arXiv preprint arXiv:2303.07280

    Pith/arXiv arXiv 2023

  14. [14]

    Gemini 3 Pro Model Card

    Google DeepMind (2025). Gemini 3 Pro Model Card. Technical report, Google DeepMind. Accessed: 2025-12-23

    2025

  15. [15]

    Gu, S., Holly, E., Lillicrap, T., and Levine, S. (2017). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In2017 IEEE international conference on robotics and automation (ICRA), pages 3389–3396. IEEE

    2017

  16. [16]

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. 10

    Pith/arXiv arXiv 2025

  17. [17]

    Hu, J., Hendrix, R., Farhadi, A., Kembhavi, A., Martín-Martín, R., Stone, P., Zeng, K.-H., and Ehsani, K. (2025). Flare: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 3617–3624. IEEE

    2025

  18. [18]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. (2024). Gpt-4o system card.arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  19. [19]

    Intelligence, P., Amin, A., Aniceto, R., Balakrishna, A., Black, K., Conley, K., Connors, G., Darpinian, J., Dhabalia, K., DiCarlo, J., et al. (2025). pi0.6: a vla that learns from experience. arXiv preprint arXiv:2511.14759

    Pith/arXiv arXiv 2025

  20. [20]

    and Berg-Kirkpatrick, T

    Jhamtani, H. and Berg-Kirkpatrick, T. (2018). Learning to describe differences between pairs of similar images. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2018

  21. [21]

    J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al

    Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024). Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246

    Pith/arXiv arXiv 2024

  22. [22]

    Lee, T., Wagenmaker, A., Pertsch, K., Liang, P., Levine, S., and Finn, C. (2026). Roboreward: General-purpose vision-language reward models for robotics.arXiv preprint arXiv:2601.00675

    arXiv 2026

  23. [23]

    Lei, K., Li, H., Yu, D., Wei, Z., Guo, L., Jiang, Z., Wang, Z., Liang, S., and Xu, H. (2025). Rl-100: Performant robotic manipulation with real-world reinforcement learning.arXiv preprint arXiv:2510.14830

    arXiv 2025

  24. [24]

    Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2016). End-to-end training of deep visuomotor policies.Journal of Machine Learning Research, 17(1):1334–1373

    2016

  25. [25]

    Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., and Quillen, D. (2018). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.International Journal of Robotics Research, 37(4-5):421–436

    2018

  26. [26]

    S., Zettlemoyer, L., Fox, D., et al

    Liang, A., Korkmaz, Y ., Zhang, J., Hwang, M., Anwar, A., Kaushik, S., Shah, A., Huang, A. S., Zettlemoyer, L., Fox, D., et al. (2026). Robometer: Scaling general-purpose robotic reward models via trajectory comparisons.arXiv preprint arXiv:2603.02115

    Pith/arXiv arXiv 2026

  27. [27]

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). Let’s verify step by step. arXiv preprint arXiv:2305.20050

    Pith/arXiv arXiv 2023

  28. [28]

    Liu, Y ., Ma, M., Yu, X., Ding, P., Zhao, H., Sun, M., Huang, S., and Wang, D. (2025). Ssr: Enhancing depth perception in vision-language models via rationale-guided spatial reasoning. arXiv preprint arXiv:2505.12448

    arXiv 2025

  29. [29]

    Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., Tang, Y ., and Zhang, D. (2025a). Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583

    Pith/arXiv arXiv

  30. [30]

    L., Berg, J., Sharma, A., Schaal, S., Finn, C., Gupta, A., and Levine, S

    Luo, J., Hu, Z., Xu, C., Tan, Y . L., Berg, J., Sharma, A., Schaal, S., Finn, C., Gupta, A., and Levine, S. (2024). Serl: A software suite for sample-efficient robotic reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16961–16969. IEEE

    2024

  31. [31]

    Luo, J., Xu, C., Wu, J., and Levine, S. (2025b). Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning.Science Robotics, 10(105):eads5033

  32. [32]

    Luo, R., Zheng, Z., Wang, L., Wang, Y ., Ni, X., Lin, Z., Jiang, S., Yu, Y ., Shi, C., Chu, R., et al. (2025c). Unlocking multimodal mathematical reasoning via process reward model. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

  33. [33]

    M., Lee, Y ., Lee, D., Kim, S., Kim, M

    Luu, T. M., Lee, Y ., Lee, D., Kim, S., Kim, M. J., and Yoo, C. D. (2025). Erlvlm: Enhancing rating-based reinforcement learning to effectively leverage feedback from large vision–language models. arXiv preprint arXiv:2506.12822. 11

    Pith/arXiv arXiv 2025

  34. [34]

    J., Hejna, J., Fu, C., Shah, D., Liang, J., Xu, Z., Kirmani, S., Xu, P., Driess, D., Xiao, T., et al

    Ma, Y . J., Hejna, J., Fu, C., Shah, D., Liang, J., Xu, Z., Kirmani, S., Xu, P., Driess, D., Xiao, T., et al. (2024a). Vision language models are in-context value learners. InThe Thirteenth International Conference on Learning Representations

  35. [35]

    Ma, Y . J., Hejna, J., Wahid, A., Fu, C., Shah, D., Liang, J., Xu, Z., Kirmani, S., Xu, P., Driess, D., Xiao, T., Tompson, J., Bastani, O., Jayaraman, D., Yu, W., Zhang, T., Sadigh, D., and Xia, F. (2024b). Vision language models are in-context value learners. arXiv preprint arXiv:2411.04549

    Pith/arXiv arXiv

  36. [36]

    J., Kumar, V ., Zhang, A., Bastani, O., and Jayaraman, D

    Ma, Y . J., Kumar, V ., Zhang, A., Bastani, O., and Jayaraman, D. (2023). Liv: Language-image representations and rewards for robotic control. InInternational Conference on Machine Learning, pages 23301–23320. PMLR

    2023

  37. [37]

    S., Gao, T., Sampaio, G

    Mark, M. S., Gao, T., Sampaio, G. G., Srirama, M. K., Sharma, A., Finn, C., and Kumar, A. (2024). Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685

    Pith/arXiv arXiv 2024

  38. [38]

    Mendonca, R., Panov, E., Bucher, B., Wang, J., and Pathak, D. (2024). Continuously improving mobile manipulation with autonomous real-world rl. arXiv preprint arXiv:2409.20568

    Pith/arXiv arXiv 2024

  39. [39]

    Nakamoto, M., Mees, O., Kumar, A., and Levine, S. (2024). Steering your generalists: Improv- ing robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816

    Pith/arXiv arXiv 2024

  40. [40]

    Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y . (2024). Robocasa: Large-scale simulation of everyday tasks for generalist robots.arXiv preprint arXiv:2406.02523

    Pith/arXiv arXiv 2024

  41. [41]

    Y ., Harada, D., and Russell, S

    Ng, A. Y ., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Bratko, I. and Dzeroski, S., editors,Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 - 30, 1999, pages 278–287. Morgan Kaufmann

    1999

  42. [42]

    L., Sanketi, P., Vuong, Q., Xiao, T., Sadigh, D., Finn, C., and Levine, S

    Octo Model Team, Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., Kreiman, T., Tan, Y . L., Sanketi, P., Vuong, Q., Xiao, T., Sadigh, D., Finn, C., and Levine, S. (2024). Octo: An open-source generalist robot policy. InProceedings of Robotics: Science and Systems (RSS), Delft, Netherlands

    2024

  43. [43]

    O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. (2024). Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE

    2024

  44. [44]

    Spacethinker dataset

    remyxai (2024). Spacethinker dataset. https://huggingface.co/datasets/remyxai/ SpaceThinker. Hugging Face Datasets

    2024

  45. [45]

    Riedmiller, M., Gabel, T., Hafner, R., and Lange, S. (2009). Reinforcement learning for robot soccer.Autonomous Robots, 27(1):55–73

    2009

  46. [46]

    Rocamonde, J., Montesinos, V ., Nava, E., Perez, E., and Lindner, D. (2023). Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310.12921

    Pith/arXiv arXiv 2023

  47. [47]

    J., et al

    Sermanet, P., Ding, T., Zhao, J., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac- Arnold, G., Maddineni, S., Joshi, N. J., et al. (2024). Robovqa: Multimodal long-horizon reasoning for robotics. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 645–652. IEEE

    2024

  48. [48]

    K., Wu, Y ., and Guo, D

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  49. [49]

    Singh, A., Bhaskar, A., Yu, P., Chakraborty, S., Dasyam, R., Bedi, A., and Tokekar, P. (2025a). Varp: Reinforcement learning from vision–language model feedback with agent-regularized preferences. arXiv preprint arXiv:2503.13817. 12

    Pith/arXiv arXiv

  50. [50]

    Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., et al. (2025b). Openai gpt-5 system card.arXiv preprint arXiv:2601.03267

    Pith/arXiv arXiv

  51. [51]

    Singh, A., Yang, L., Hartikainen, K., Finn, C., and Levine, S. (2019). End-to-end robotic reinforcement learning without reward engineering.Robotics: Science and Systems

    2019

  52. [53]

    Tan, H., Chen, S., Xu, Y ., Wang, Z., Ji, Y ., Chi, C., Lyu, Y ., Zhao, Z., Chen, X., Co, P., Xie, S., Yao, G., Wang, P., Wang, Z., and Zhang, S. (2025b). Robo-dopamine: General process reward modeling for high-precision robotic manipulation. arXiv preprint arXiv:2512.23703

    arXiv

  53. [54]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530

    Pith/arXiv arXiv 2024

  54. [55]

    Venkataraman, S., Wang, Y ., Wang, Z., Erickson, Z., and Held, D. (2024). Real-world offline reinforcement learning from vision language model feedback. arXiv preprint arXiv:2411.05273

    Pith/arXiv arXiv 2024

  55. [56]

    N., Klissarov, M., Precup, D., Yang, S., and Anand, A

    Venuto, D., Islam, S. N., Klissarov, M., Precup, D., Yang, S., and Anand, A. (2024). Code as reward: Empowering reinforcement learning with vlms. arXiv preprint arXiv:2402.04764

    Pith/arXiv arXiv 2024

  56. [57]

    Wagenmaker, A., Nakamoto, M., Zhang, Y ., Park, S., Yagoub, W., Nagabandi, A., Gupta, A., and Levine, S. (2025). Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.15799

    Pith/arXiv arXiv 2025

  57. [58]

    Wang, D., Jia, M., Zhu, X., Walters, R., and Platt, R. (2022). On-robot learning with equivariant models. In Liu, K., Kulic, D., and Ichnowski, J., editors,Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 ofProceedings of Machine Learning Research, pages 1345–1354. PMLR

    2022

  58. [59]

    Wang, Y ., Sun, Z., Zhang, J., Xian, Z., Biyik, E., Held, D., and Erickson, Z. (2024). Rl-vlm- f: Reinforcement learning from vision language foundation model feedback.arXiv preprint arXiv:2402.03681

    Pith/arXiv arXiv 2024

  59. [60]

    Weng, J., Wang, X., Liu, M., Chen, Y ., Yang, Y ., Zhang, Y ., Li, H., and Wang, J. (2022). Tianshou: A highly modularized deep reinforcement learning library.Journal of Machine Learning Research, 23(267):1–6

    2022

  60. [61]

    Wu, Y ., Yuan, W., Qi, A., Guizilini, V ., Mao, J., and Wang, Y . (2026). Large reward models: Generalizable online robot reward generation with vision-language models.arXiv preprint arXiv:2603.16065

    arXiv 2026

  61. [62]

    Xu, P., Wang, S., Zhu, Y ., Li, J., and Zhang, Y . (2025). Spatialbench: Benchmarking multimodal large language models for spatial cognition.arXiv preprint arXiv:2511.21471

    Pith/arXiv arXiv 2025

  62. [63]

    Yang, D., Tjia, D., Berg, J., Damen, D., Agrawal, P., and Gupta, A. (2024a). Rank2reward: Learning shaped reward functions from passive video. arXiv preprint arXiv:2404.14735

    Pith/arXiv arXiv

  63. [64]

    Yang, Y ., Chen, M., Qiu, Q., Wu, J., Wang, W., Lin, B., Guan, Z., and He, X. (2024b). Adapt2reward: Adapting video–language models to generalizable robotic rewards via failure prompts. arXiv preprint arXiv:2407.14872

    Pith/arXiv arXiv

  64. [65]

    Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. (2022). Mastering visual continuous control: Improved data-augmented reinforcement learning. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net

    2022

  65. [66]

    Yuan, Y ., Cui, H., Chen, Y ., Dong, Z., Ni, F., Kou, L., Liu, J., Li, P., Zheng, Y ., and Hao, J. (2025). From seeing to doing: Bridging reasoning and decision for robotic manipulation.arXiv preprint arXiv:2505.08548. 13

    Pith/arXiv arXiv 2025

  66. [67]

    Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., and Levine, S. (2024). Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693

    Pith/arXiv arXiv 2024

  67. [68]

    Zeng, A., Song, S., Lee, J., Rodriguez, A., and Funkhouser, T. (2020). Tossingbot: Learning to throw arbitrary objects with residual physics.IEEE Transactions on Robotics, 36(4):1307–1319

    2020

  68. [69]

    Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., and Funkhouser, T. A. (2018). Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1-5, 2018, pages 4238–4245. IEEE

    2018

  69. [70]

    A., Lim, J

    Zhang, J., Luo, Y ., Anwar, A., Sontakke, S. A., Lim, J. J., Thomason, J., Biyik, E., and Zhang, J. (2025a). RewiND: Language-guided rewards teach robot policies without new demonstrations. In 9th Annual Conference on Robot Learning

  70. [71]

    A., Lim, J

    Zhang, J., Luo, Y ., Anwar, A., Sontakke, S. A., Lim, J. J., Thomason, J., Bıyık, E., and Zhang, J. (2025b). Rewind: Language-guided rewards teach robot policies without new demonstrations. arXiv preprint arXiv:2505.10911

    arXiv

  71. [72]

    Zhang, Q., Zhai, S., Zhang, S., Liu, L., Huang, F., HaoranECNU, Z., Zhou, M., Pang, J., et al. (2025c). Vlac: A generalist action-critic model via pair-wise progress understanding.arXiv preprint arXiv:2509.15937

    arXiv

  72. [73]

    Zhang, Z., Zheng, K., Chen, Z., Jang, J., Li, Y ., Han, S., Wang, C., Ding, M., Fox, D., and Yao, H. (2024). Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.19309

    Pith/arXiv arXiv 2024

  73. [74]

    close drawer

    Zhu, N., Dong, Y ., Wang, T., Li, X., Deng, S., Wang, Y ., Hong, Z., Geng, T., Niu, G., Huang, H., et al. (2025). Cvbench: Evaluating cross-video synergies for complex multimodal understanding and reasoning.arXiv e-prints, pages arXiv–2508. 14 A Visual overview of online RL evaluation suites 15 B Reasoning examples Figure 7: Extended reasoning examples ac...

    2025

  74. [75]

    To prevent optimistic extrapolation and reward hacking, training data must include authentic non-expert behaviors spanning varying degrees of task completion

    Explicit coverage of partial success and failure states.Online RL policies frequently visit intermediate, incorrect, or regressive states that are underrepresented in expert demonstrations. To prevent optimistic extrapolation and reward hacking, training data must include authentic non-expert behaviors spanning varying degrees of task completion

  75. [76]

    Temporal locality of supervision.Progress supervision and reasoning are provided at the granularity of individual timesteps, forcing the model to reason aboutwhat changedbetween consecutive frames rather than relying on static appearance or final outcomes

  76. [77]

    near the handle

    Task-structure-aware decomposition.Manipulation tasks admit natural decompositions into subgoals (approach, contact, grasp, transport, release, articulation). We explicitly encode this structure in the non-expert trajectory design so that the model learns reusable progress primitives rather than task-specific heuristics. J.2 Non-expert trajectory level ta...