Pith. sign in

REVIEW 5 major objections 6 minor 66 references

Adding a world-prediction objective to the critic—predicting the next latent state while estimating value—gives vision-language-action RL a temporally aware state representation and improves manipulation success and out-of-distribution gene

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:28 UTC pith:NUBENC25

load-bearing objection WCM is a solid empirical method paper: adding next-latent prediction to a history-conditioned critic helps VLA-RL, but the 'world dynamics' mechanism story is not established by the current ablations. the 5 major comments →

arxiv 2607.29613 v1 pith:NUBENC25 submitted 2026-07-31 cs.RO cs.CLcs.CV

WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning

Senyu Fei , Xiaopeng Yu , Siyin Wang , Xianzhong Zhao , Jingjing Gong , Xipeng Qiu This is my paper
classification cs.RO cs.CLcs.CV
keywords vision-language-action modelsreinforcement learningvalue estimationworld modellatent predictionpartial observabilityrobotic manipulationout-of-distribution generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Critic-based reinforcement learning for vision-language-action (VLA) robot policies, the paper argues, is limited by the critic's state representation: a single frame (or weakly supervised frame-stacking) cannot capture the motion, contact progress, and near-future evolution that determine a manipulation's value, and pure scalar-return regression does not push the representation to encode dynamics. WCM is a critic that consumes a short history of observation latents, predicts the next latent state, and estimates the value from the same representation, all trained end-to-end. The next-latent prediction acts as a world-model objective, giving dense temporal supervision that scalar value regression lacks. If the claim holds, replacing the standard critic with WCM—leaving the policy algorithm untouched—raises success rates, smoothness, and out-of-distribution robustness across many simulated and real manipulation tasks.

Core claim

On the paper's own terms, the discovery is a representation bottleneck and its fix: under partial observability, value estimation from a single frame or from history without a prediction objective cannot recover the temporally informative state, so the critic overfits and the policy learns from stale values. WCM fixes this by jointly predicting the future latent state and estimating value, so the critic's representation is explicitly trained to encode environment dynamics. In the reported experiments, WCM achieves the best results among compared critics across four simulation benchmarks (149 tasks), with especially large out-of-distribution gains; on one pick-and-place suite it takes an auto

What carries the argument

The load-bearing component is the WCM critic itself: a per-frame observation encoder (a vision transformer or the VLA backbone) produces latents for the last K frames; a language-conditioned causal transformer trunk summarizes them; and two heads are attached—one value head that regresses the return and one world head that predicts the next latent using an action-conditioned residual update with gated feature modulation. Training minimizes the value loss plus a weighted next-latent loss (teacher-forced against the observed next latent) plus a latent-collapse regularizer, all end-to-end. The world head's role is to force the shared representation to be a predictive state—a compact, updateable

Load-bearing premise

The load-bearing assumption is that predicting the next observation's latent representation from the current history is a genuinely informative learning signal; if that prediction is easy, uninformative, or does not force the critic's representation to encode dynamics, WCM reduces to a larger history-conditioned value estimator.

What would settle it

Train WCM and an equally sized history-conditioned critic without the prediction head (the paper's λ=0 ablation) on the same tasks with matched compute, data, and steps; if the no-prediction critic matches or beats WCM on both in-distribution and out-of-distribution success, the central claim fails. A complementary check: shuffle the order of the K input frames during WCM training; if success rates do not drop, the critic is not actually using temporal structure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • WCM can be swapped into existing on-policy and off-policy RL pipelines by replacing only the critic; the policy and its loss stay unchanged.
  • The improvement is not just from seeing more frames: a temporal critic with the same architecture but no prediction loss does not match WCM, so the predictive objective is the claimed driver.
  • Out-of-distribution generalization improves along with in-distribution success, indicating that critic overfitting under distribution shift is partly a state-representation problem.
  • The optimal history is short—three frames in these tasks—so more history is not automatically better; the window appears to need only enough frames to express first- and second-order dynamics.
  • On physical robots, the predictive critic yields smooth trajectories and fewer collisions and stalls, and value curves remain discriminative between successful and failed rollouts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the predictive-state objective is the mechanism, the same critic design should transfer to other partially observable robot or autonomous control settings where single-frame value estimation fails; an easy test is to take an existing POMDP RL benchmark and swap in a WCM-style critic.
  • The paper's choice of an L2 teacher-forced next-latent loss is one of several possible world-model objectives; contrastive, flow-matching, or discrete-token alternatives might yield different representation quality and training stability on the same WCM skeleton.
  • The K=3 result suggests an adaptive design rule: set history length to cover the number of frames needed to express the task's relevant temporal derivatives (for example, velocity at K=2, acceleration at K=3) rather than using a fixed window or ever-larger stacks.
  • The observed smoother out-of-distribution value landscapes hint that WCM could double as a learned dense reward shaper or a safety filter for policy rollouts, though the paper does not test that use.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. WCM proposes a critic for VLA-RL that consumes a K-frame observation history, predicts the next VLM latent with an action-conditioned world predictor, and estimates a scalar value, trained jointly by a value loss, a next-latent prediction loss, and (in the off-policy setting) SIGReg. The critic is inserted into on-policy PPO/Flow-SDE and off-policy AWR/RECAP pipelines and evaluated on ManiSkill, MetaWorld, CALVIN, and LIBERO-Plus (149 tasks total), plus seven real-world WidowX-250S tasks with OpenVLA-OFT and pi0.5. The paper reports consistent IND/OOD improvements over single-frame critic baselines, larger gains from a near-zero-shot initialization, and advantages in real-world RL training.

Significance. If the empirical claims survive scrutiny, WCM is a practical and timely contribution: it targets a genuine POMDP limitation of current VLA critics, is lightweight, integrates with multiple backbones and both on-policy and off-policy RL, and the authors release code and checkpoints. The benchmark breadth and the lambda=0 architecture ablation are strengths. The main risk is that the mechanistic interpretation currently outruns the evidence: the self-supervised prediction target in Eq. (6) is vulnerable to shortcut solutions, hyperparameters are selected on the evaluation benchmarks, and the real-world and LIBERO-Plus evidence is thinner than the SOTA claims require. These concerns are fixable within the manuscript's scope, so a major revision is appropriate.

major comments (5)
  1. [§3.2, Eq. (6); App. D.1] The central mechanistic claim—that the critic is explicitly trained to encode temporal dynamics—is not yet established. The target z_{t+1}=enc(o_{t+1}) is produced by the same trainable encoder, and SIGReg (Eq. 7) is disabled in the on-policy pipeline where the main results are obtained. L_pred can therefore be minimized by a representational shortcut (making consecutive latents similar) rather than by predictive world modeling. The lambda>0 vs lambda=0 comparison does not separate 'better dynamics' from 'regularization' or 'extra capacity'. Please provide (a) collapse diagnostics (e.g., singular-value spectrum, temporal similarity) for the learned latents, (b) a frozen/EMA target or stop-gradient variant, or (c) on-policy runs with SIGReg enabled, to show that L_pred remains non-trivial.
  2. [§5.1, §5.2, App. A (Figs. 5–6)] The hyperparameters lambda and K are selected on the same ManiSkill/MetaWorld evaluation curves: Figure 6 reports best IND/OOD for lambda in [0.3,0.5] and Figure 5 selects K=3. When the reported SOTA numbers use hyperparameters tuned on the test benchmark, the comparison is biased. Please select lambda/K on a held-out task suite, or report the complete grid for every benchmark, and state clearly whether the reported results are the best over the grid.
  3. [Table 2 (LIBERO-Plus)] This table only compares WCM initialized from One-SFT with Full-SFT; it contains no RL baseline with a standard critic (PPO, Flow-SDE, or FlowNoise) under the same training budget. Consequently it does not support the statement in §4.3 that WCM reaches SOTA generalization on LIBERO-Plus. Add the corresponding single-frame critic RL baseline trained for the same number of steps.
  4. [Table 3 (real-world)] Each entry is a single 50-trial count without error bars or seeds; several improvements are only 2–7 successes (e.g., Carrot 32 vs 29, Pepper 26 vs 24). The claim that WCM 'outperforms baselines across all tasks' is not supported at that sample size. Report multiple independent runs, bootstrap confidence intervals, or exact binomial tests; also state whether the Gemma 270M baseline critic and WCM (107.2M) are matched for parameter count and training data.
  5. [§1 and §5.1] The 'root cause' claim that scalar-return regression is insufficient for cross-temporal dynamics is asserted rather than derived. The only direct evidence is the ViT lambda=0 ablation, which may be undertrained or capacity-mismatched. Please add training curves for the ViT baseline, a capacity-matched MLP history critic, and an analysis of the learned representations; otherwise the diagnosis remains a hypothesis rather than a demonstrated cause.
minor comments (6)
  1. [App. D.4] Heading typo: 'Traning Curve' should be 'Training Curve'.
  2. [Algorithm 1] Line 6 computes the total critic loss from Eq. (6) and Eq. (9) (Eq. (12)); make explicit in the caption that SIGReg is not used in the on-policy pipeline, consistent with App. D.1.
  3. [§5.2] The explanation that K=3 captures second-order dynamics (acceleration) and K=2 captures velocity is speculative. Mark it as intuition, or test it by analyzing the learned latent differences across K.
  4. [Table 3 caption] Define what 'after test starts' means and why only the first 50 trajectories are reported. As written, the evaluation protocol is ambiguous.
  5. [D.4 / Table 3] The baseline critic is a SigLip 400M + Gemma 270M model while WCM has 107.2M learnable parameters. Parameter-count mismatch could partially explain differences; please report a matched-capacity baseline or discuss the effect.
  6. [Eq. (7)] The empirical characteristic function estimator in SIGReg is described only briefly; cite the exact estimator used in the code and state the number of random projections a drawn per batch.

Circularity Check

0 steps flagged

No circularity: the world-prediction target is the next encoder latent rather than the return label, and the lambda=0 ablation provides the relevant control.

full rationale

WCM's core auxiliary objective, Eq. (6), minimizes ||z_hat_{t+1} - z_{t+1}||_2^2, where z_{t+1} = enc(o_{t+1}) is the next observation's latent produced by the same encoder; this is a self-supervised prediction target, not the value label G_t. The paper does not define the value estimate in terms of this prediction, nor does it claim the prediction loss itself is the return. The lambda=0 ablation in Sec. 5.1 is the correct control for whether adding the prediction term changes the critic, and the full objective in Eq. (10) includes SIGReg to discourage collapse. The fact that Appendix D.1 drops SIGReg in the on-policy pipeline raises a real risk of representational shortcuts and weakens the mechanistic 'captures temporal dynamics' interpretation, but that is a robustness/correctness concern, not circularity: the equations do not force the prediction to reduce to an identity or to a renamed fit of the value. Self-citations such as LIBERO-Plus [66] are benchmarks or related-work context, not load-bearing justifications for the WCM derivation. Hyperparameter choices (lambda, K) tuned on evaluation benchmarks are selection bias, not circularity. No circular step can be exhibited from the paper's own equations.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The central claim rests on a domain POMDP assumption, the empirical sufficiency of K=3, and the premise that next-latent prediction is useful supervision. Four hand-chosen hyperparameters affect the value targets and loss balance.

free parameters (4)
  • λ = 0.3–0.5 (best range)
    Weight on the world-prediction loss in Eq. (10); tuned in Fig. 6, with OOD varying 10.6 percentage points across the grid.
  • K = 3
    Observation history length; selected after ablation in Fig. 5, with the paper noting length 3 performed best on average.
  • η = not reported
    Weight on SIGReg in Eq. (10); used in the off-policy pipeline, but its value is not given in the paper.
  • C_fail = 300
    Failure penalty in the reward definition Eq. (8); inherited from the RECAP/π0.6 style but still a chosen constant that shapes all value targets.
axioms (4)
  • domain assumption VLA-RL is a POMDP and history is needed for value estimation
    Motivates the entire approach; standard for robotics but not proven for these tasks.
  • domain assumption A truncated K-frame history is a sufficient statistic for the value function
    K=3 is chosen empirically in §5.2; no guarantee of sufficiency for all tasks.
  • ad hoc to paper Minimizing L2 next-latent prediction in VLM latent space improves critic state representation
    Core premise of Eq. (6); supported only by ablation, not theory.
  • standard math SIGReg enforces an isotropic Gaussian latent distribution and prevents collapse
    Characteristic-function argument from LeJEPA [59]; mathematically sound as an auxiliary regularization.

pith-pipeline@v1.3.0-daily-deepseek · 20645 in / 11318 out tokens · 112434 ms · 2026-08-03T03:28:39.798550+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 35 linked inside Pith

  1. [1]

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.𝜋0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    𝜋0.5: Avision-language-actionmodelwithopen-worldgeneralization

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, MichaelEqui,ChelseaFinn,NiccoloFusai,etal. 𝜋0.5: Avision-language-actionmodelwithopen-worldgeneralization. 9th Annual Conference on Robot Learning, 2025

    2025

  3. [3]

    Physical Intelligence, Bo Ai, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Greg Balke, Kevin Black, George Bokinsky, Shihao Cao, Thomas Charbonnier, et al.𝜋0.7: A steerable generalist robotic foundation model with emergent capabilities.arXiv preprint arXiv:2604.15483, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    What can rl bring to vla generalization? an empirical study.arXiv preprint arXiv:2505.19789, 2025

    Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, Yi Wu, Chao Yu, and Yu Wang. What can rl bring to vla generalization? an empirical study.arXiv preprint arXiv:2505.19789, 2025

    arXiv 2025

  7. [7]

    Srpo: Self-referential policy optimization for vision-language-action models.arXiv preprint arXiv:2511.15605, 2025

    Senyu Fei, Siyin Wang, Li Ji, Ao Li, Shiduo Zhang, Liming Liu, Jinlong Hou, Jingjing Gong, Xianzhong Zhao, and Xipeng Qiu. Srpo: Self-referential policy optimization for vision-language-action models.arXiv preprint arXiv:2511.15605, 2025

    arXiv 2025

  8. [8]

    Simplevla-rl: Scaling vla training via reinforcement learning.arXiv preprint arXiv:2509.09674, 2025

    Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, et al. Simplevla-rl: Scaling vla training via reinforcement learning.arXiv preprint arXiv:2509.09674, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Rlinf-vla: A unified and efficient framework for vla+ rl training.arXiv preprint arXiv:2510.06710, 2025

    Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo, Yuanqing Wang, Hao Lin, Liangzhi Shi, Yuqing Xie, Zhexuan Xu, et al. Rlinf-vla: A unified and efficient framework for vla+ rl training.arXiv preprint arXiv:2510.06710, 2025

    arXiv 2025

  10. [10]

    Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, et al.𝜋0.6: A vla that learns from experience.arXiv preprint arXiv:2511.14759, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, et al.𝜋rl: Online rl fine-tuning for flow-based vision-language-action models.arXiv preprint arXiv:2510.25889, 2025

    arXiv 2025

  12. [12]

    Rlinf-user: A unified and extensible system for real-world online policy learning in embodied ai

    Hongzhi Zang, Shu’ang Yu, Hao Lin, Tianxing Zhou, Zefang Huang, Zhen Guo, Xin Xu, Jiakai Zhou, Yuze Sheng, Shizhe Zhang, et al. Rlinf-user: A unified and extensible system for real-world online policy learning in embodied ai. arXiv preprint arXiv:2602.07837, 2026

    arXiv 2026

  13. [13]

    Predictive representations of state.Advances in neural information processing systems, 14, 2001

    Michael Littman and Richard S Sutton. Predictive representations of state.Advances in neural information processing systems, 14, 2001

    2001

  14. [14]

    Learning predictive state representations

    Satinder P Singh, Michael L Littman, Nicholas K Jong, David Pardoe, and Peter Stone. Learning predictive state representations. InProceedings of the 20th International Conference on Machine Learning (ICML-03), pages 712–719, 2003

    2003

  15. [15]

    Predictive state representations: A new theory for modeling dynamical systems.arXiv preprint arXiv:1207.4167, 2012

    Satinder Singh, Michael James, and Matthew Rudary. Predictive state representations: A new theory for modeling dynamical systems.arXiv preprint arXiv:1207.4167, 2012

    Pith/arXiv arXiv 2012

  16. [16]

    When is partially observable reinforcement learning not scary? InConference on Learning Theory, pages 5175–5220

    Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforcement learning not scary? InConference on Learning Theory, pages 5175–5220. PMLR, 2022

    2022

  17. [17]

    Approximate information state for approximateplanningandreinforcementlearninginpartiallyobservedsystems.JournalofMachineLearningResearch, 23(12):1–83, 2022

    Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximateplanningandreinforcementlearninginpartiallyobservedsystems.JournalofMachineLearningResearch, 23(12):1–83, 2022. 10

    2022

  18. [18]

    Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning.arXiv preprint arXiv:2505.18719, 2025

    Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning.arXiv preprint arXiv:2505.18719, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Reinforcement learning with latent flow.Advances in Neural Information Processing Systems, 34:22171–22183, 2021

    Wenling Shang, Xiaofei Wang, Aravind Srinivas, Aravind Rajeswaran, Yang Gao, Pieter Abbeel, and Misha Laskin. Reinforcement learning with latent flow.Advances in Neural Information Processing Systems, 34:22171–22183, 2021

    2021

  20. [20]

    Provable reinforcement learning with a short-term memory

    Yonathan Efroni, Chi Jin, Akshay Krishnamurthy, and Sobhan Miryoosefi. Provable reinforcement learning with a short-term memory. InInternational Conference on Machine Learning, pages 5832–5850. PMLR, 2022

    2022

  21. [21]

    Improving sample efficiencyinmodel-freereinforcementlearningfromimages

    Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiencyinmodel-freereinforcementlearningfromimages. InProceedingsoftheaaaiconferenceonartificialintelligence, volume 35, pages 10674–10681, 2021

    2021

  22. [22]

    Weakly supervised representation learning with sparse perturbations.Advances in Neural Information Processing Systems, 35:15516–15528, 2022

    Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations.Advances in Neural Information Processing Systems, 35:15516–15528, 2022

    2022

  23. [23]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Data-efficient reinforcement learning with self-predictive representations.International Conference on Learning Representations, 2020

    Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations.International Conference on Learning Representations, 2020

    2020

  26. [26]

    Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    BriannaZitkovich,TianheYu,SichunXu,PengXu,TedXiao,FeiXia,JialinWu,PaulWohlhart,StefanWelker,Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  28. [28]

    Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Beyond imitation: Reinforcement learning-based sim-real co-training for vla models.arXiv preprint arXiv:2602.12628, 2026

    Liangzhi Shi, Shuaihang Chen, Feng Gao, Yinuo Chen, Kang Chen, Tonghe Zhang, Hongzhi Zang, Weinan Zhang, Chao Yu, and Yu Wang. Beyond imitation: Reinforcement learning-based sim-real co-training for vla models.arXiv preprint arXiv:2602.12628, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    Interactive post-training for vision-language-action models.arXiv preprint arXiv:2505.17016, 2025

    Shuhan Tan, Kairan Dou, Yue Zhao, and Philipp Krähenbühl. Interactive post-training for vision-language-action models.arXiv preprint arXiv:2505.17016, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  32. [32]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905, 2018

    TuomasHaarnoja,AurickZhou,KristianHartikainen,GeorgeTucker,SehoonHa,JieTan,VikashKumar,HenryZhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905, 2018

    Pith/arXiv arXiv 2018

  34. [34]

    Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    Pith/arXiv arXiv 2021

  35. [35]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019. 11

    Pith/arXiv arXiv 1910

  36. [36]

    Reinflow: Fine-tuning flow matching policy with online reinforcement learning.arXiv preprint arXiv:2505.22094, 2025

    Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching policy with online reinforcement learning.arXiv preprint arXiv:2505.22094, 2025

    arXiv 2025

  37. [37]

    Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Diffusion policy policy optimization.arXiv preprint arXiv:2409.00588, 2024

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization.arXiv preprint arXiv:2409.00588, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Steering your diffusion policy with latent space reinforcement learning.arXiv preprint arXiv:2506.15799, 2025

    Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning.arXiv preprint arXiv:2506.15799, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Precise and dexterous robotic manipulation via human-in- the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in- the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    2025

  41. [41]

    Gigabrain-0.5 m*: a vla that learns from world model-based reinforcement learning.arXiv preprint arXiv:2602.12099, 2026

    GigaBrain Team, Boyuan Wang, Bohan Li, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Jie Li, Jindi Lv, Jingyu Liu, et al. Gigabrain-0.5 m*: a vla that learns from world model-based reinforcement learning.arXiv preprint arXiv:2602.12099, 2026

    arXiv 2026

  42. [42]

    Optimal control of markov decision processes with incomplete state estimation.J

    Karl J Astrom. Optimal control of markov decision processes with incomplete state estimation.J. Math. Anal. Applic., 10:174–205, 1965

    1965

  43. [43]

    The optimal control of partially observable markov processes over a finite horizon.Operations research, 21(5):1071–1088, 1973

    Richard D Smallwood and Edward J Sondik. The optimal control of partially observable markov processes over a finite horizon.Operations research, 21(5):1071–1088, 1973

    1973

  44. [44]

    Reinforcement learning with augmented data.Advances in neural information processing systems, 33:19884–19895, 2020

    Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data.Advances in neural information processing systems, 33:19884–19895, 2020

    2020

  45. [45]

    Contextual decision processes with low bellman rank are pac-learnable

    Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable. InInternational Conference on Machine Learning, pages 1704–1713. PMLR, 2017

    2017

  46. [46]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  47. [47]

    Deep recurrent q-learning for partially observable mdps

    Matthew J Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. InAAAI fall symposia, volume 45, page 141, 2015

    2015

  48. [48]

    Robust reinforcement learning in pomdps with incomplete and noisy observations.arXiv preprint arXiv:1902.05795, 2019

    Yuhui Wang, Hao He, and Xiaoyang Tan. Robust reinforcement learning in pomdps with incomplete and noisy observations.arXiv preprint arXiv:1902.05795, 2019

    Pith/arXiv arXiv 1902

  49. [49]

    Robust finite-memory policy gradients for hidden-model pomdps.arXiv preprint arXiv:2505.09518, 2025

    Maris FL Galesloot, Roman Andriushchenko, Milan Češka, Sebastian Junges, and Nils Jansen. Robust finite-memory policy gradients for hidden-model pomdps.arXiv preprint arXiv:2505.09518, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation

    Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation. arXiv preprint arXiv:2508.19236, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Hamlet: Switch your vision-language-action model into a history-aware policy.arXiv preprint arXiv:2510.00695, 2025

    Myungkyu Koo, Daewon Choi, Taeyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, and Jinwoo Shin. Hamlet: Switch your vision-language-action model into a history-aware policy.arXiv preprint arXiv:2510.00695, 2025

    Pith/arXiv arXiv 2025

  52. [52]

    Cronusvla: Transferring latent motion across time for multi-frame prediction in manipulation.arXiv e-prints, pages arXiv–2506, 2025

    Hao Li, Shuai Yang, Yilun Chen, Yang Tian, Xiaoda Yang, Xinyi Chen, Hanqing Wang, Tai Wang, Feng Zhao, Dahua Lin, et al. Cronusvla: Transferring latent motion across time for multi-frame prediction in manipulation.arXiv e-prints, pages arXiv–2506, 2025

    2025

  53. [53]

    Decision transformer: Reinforcement learning via sequence modeling.Advances in neural information processing systems, 34:15084–15097, 2021

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling.Advances in neural information processing systems, 34:15084–15097, 2021. 12

    2021

  54. [54]

    World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

    SeonghyeonYe, YunhaoGe,KaiyuanZheng, ShenyuanGao, SihyunYu, GeorgeKurian, SuneelIndupuru, YouLiang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

    Pith/arXiv arXiv 2026

  55. [55]

    Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163, 2026

    Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163, 2026

    Pith/arXiv arXiv 2026

  56. [56]

    Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030, 2025

    Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, et al. Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

    Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

    Pith/arXiv arXiv 2026

  58. [58]

    Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    Pith/arXiv arXiv 2026

  59. [59]

    Lejepa: Provable and scalable self-supervised learning without the heuristics

    Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025

    Pith/arXiv arXiv 2025

  60. [60]

    Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels.arXiv preprint arXiv:2603.19312, 2026

    Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels.arXiv preprint arXiv:2603.19312, 2026

    Pith/arXiv arXiv 2026

  61. [61]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  62. [62]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  63. [63]

    Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.arXiv preprint arXiv:2107.14483, 2021

    Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.arXiv preprint arXiv:2107.14483, 2021

    Pith/arXiv arXiv 2021

  64. [64]

    Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. InConference on robot learning, pages 1094–1100. PMLR, 2020

    2020

  65. [65]

    OierMees,LukasHermann,ErickRosete-Beas,andWolframBurgard.Calvin: Abenchmarkforlanguage-conditioned policy learning for long-horizon robot manipulation tasks.IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022

    2022

  66. [66]

    Libero-plus: In-depth robustness analysis of vision-language-action models.arXiv preprint arXiv:2510.13626, 2025

    Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, et al. Libero-plus: In-depth robustness analysis of vision-language-action models.arXiv preprint arXiv:2510.13626, 2025

    Pith/arXiv arXiv 2025