pith. sign in

arxiv: 2605.12334 · v2 · pith:TQVTQAHAnew · submitted 2026-05-12 · 💻 cs.AI

Reinforcing VLAs in Task-Agnostic World Models

Yucen Wang , Rui Yu , Fengming Zhang , Junjie Lu , Xinyao Qin , Tianxiang Zhang , Kaixin Wang , Li Zhao This is my paper

Pith reviewed 2026-05-21 08:08 UTC · model grok-4.3

classification 💻 cs.AI
keywords Vision-Language-Action modelstask-agnostic world modelsreinforcement learningzero-shot adaptationimagination-based trainingvision-language modelspolicy finetuningrobot learning
0
0 comments X

The pith

A task-agnostic world model pre-trained on diverse behaviors plus an off-the-shelf VLM lets VLAs finetune for new tasks entirely inside imagined rollouts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes RAW-Dream to adapt vision-language-action models to new tasks without task-specific data collection or real-world trials. It pre-trains a world model on varied task-free behaviors to generate future trajectories and relies on a general vision-language model to supply rewards during reinforcement learning inside those imagined sequences. A dual-noise check removes unreliable predictions to limit hallucinations. This setup claims to work because the pre-trained components already hold transferable physical knowledge that supports zero-shot inference for unseen tasks. A reader would care if the approach truly scales adaptation across many tasks by removing repeated data needs.

Core claim

The authors claim that world and reward models should encode transferable physical priors enabling zero-shot inference. RAW-Dream therefore pre-trains the world model solely on diverse task-free behaviors for rollout prediction and uses an off-the-shelf vision-language model for reward signals. Because both components remain task-agnostic, vision-language-action policies can be reinforced for any new task inside this zero-shot imagination. A dual-noise verification mechanism filters unreliable imagined trajectories. Experiments in simulation and real-world environments show consistent gains, supporting the substitution of generalized priors for task-dependent fine-tuning data.

What carries the argument

RAW-Dream, the paradigm that disentangles world-model learning from any downstream task by using a pre-trained task-agnostic world model for rollouts and an off-the-shelf VLM for rewards.

If this is right

  • VLAs become finetunable for any new task without collecting or using task-specific data.
  • World and reward model updates are no longer required when switching tasks.
  • Consistent performance gains appear across both simulated and physical robot settings.
  • Generalized physical priors can replace costly task-dependent training data.
  • A scalable route opens for repeated VLA adaptation at low additional cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the priors prove broadly transferable, large-scale behavior pre-training could become a reusable foundation step before any task-specific policy work.
  • The approach might lower overall data and interaction budgets in robotics by amortizing one broad pre-training run across many downstream applications.
  • Similar disentanglement could apply to other imagination-based planning or control methods that currently rely on task-specific models.
  • Success would encourage testing whether the same zero-shot imagination pipeline works when the VLM or world model is swapped for newer foundation models.

Load-bearing premise

Pre-training on diverse task-free behaviors produces world and reward models whose physical priors transfer to arbitrary new tasks for reliable zero-shot use.

What would settle it

Performance on a new task whose dynamics or interactions never appeared in the task-free pre-training set falls below that of methods using task-specific world-model fine-tuning.

Figures

Figures reproduced from arXiv: 2605.12334 by Fengming Zhang, Junjie Lu, Kaixin Wang, Li Zhao, Rui Yu, Tianxiang Zhang, Xinyao Qin, Yucen Wang.

Figure 1
Figure 1. Figure 1: Left: Previous WM-based RL pipelines for VLA post-training tightly couple the WM and reward models to known target tasks, requiring thousands of in-domain rollouts, precluding unseen adaptation. Right: RAW-Dream decouples dynamics learning from task semantics. A general￾purpose WM pre-trained on diverse task-free behaviors captures transferable physical priors, while a foundation VLM provides zero-shot rew… view at source ↗
Figure 2
Figure 2. Figure 2: (a) Sample scenes from our collected play data spanning diverse object arrangements and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative examples of first-frame ghosting and its mitigation via progressive first￾frame timestep noise. For each task, we show two world-model rollouts produced from the same initial observation and the same action sequence, differing only in whether progressive first-frame timestep noise is applied at inference. Top row of each subfigure: rollout without progressive first-frame timestep noise. The mod… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative examples of Dual-Noise Verification (DNV). For each task, we show two world-model rollouts produced under the same action sequence but with independently re-sampled initial diffusion noise at every autoregressive step. Top row of each subfigure: the original imagined rollout, on which the VLM reward returns a success verdict. Bottom row: the second-pass rollout using the same action sequence, u… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative real-world rollouts of our task-agnostic world model. Top row of each subfigure: the ground-truth real-world video executed on the AgileX Piper arm. Bottom row: the corresponding autoregressive prediction from our WM, conditioned on the same initial observation o0 and the same teleoperated action sequence. These results are evaluated on entirely unseen scene layouts absent from the WM’s play-da… view at source ↗
read the original abstract

Post-training Vision-Language-Action (VLA) models via reinforcement learning (RL) in learned world models has emerged as an effective strategy to adapt to new tasks without costly real-world interactions. However, while using imagined trajectories reduces the sample complexity of policy training, existing methods still heavily rely on task-specific data to fine-tune both the world and reward models, fundamentally limiting their scalability to unseen tasks. To overcome this, we argue that world and reward models should capture transferable physical priors that enable zero-shot inference. We propose RAW-Dream (Reinforcing VLAs in task-Agnostic World Dreams), a new paradigm that completely disentangles world model learning from downstream task dependencies. RAW-Dream utilizes a world model pre-trained on diverse task-free behaviors for predicting future rollouts, and an off-the-shelf Vision-Language Model (VLM) for reward generation. Because both components are task-agnostic, VLAs can be readily finetuned for any new task entirely within this zero-shot imagination. Furthermore, to mitigate world model hallucinations, we introduce a dual-noise verification mechanism to filter out unreliable rollouts. Extensive experiments across simulation and real-world settings demonstrate consistent performance gains, proving that generalized physical priors can effectively substitute for costly task-dependent data, offering a highly scalable roadmap for VLA adaptation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces RAW-Dream, a paradigm for post-training Vision-Language-Action (VLA) models via reinforcement learning inside task-agnostic world models. A world model is pre-trained on diverse task-free behaviors to generate imagined rollouts, while an off-the-shelf VLM supplies rewards; a dual-noise verification filter removes unreliable trajectories. The central claim is that these components enable zero-shot finetuning of VLAs for arbitrary new tasks entirely inside the learned imagination, with experiments in simulation and real-world settings reported to show consistent performance gains over prior approaches that rely on task-specific data.

Significance. If the empirical results and the claimed generalization of the task-free priors hold, the work would offer a concrete route to scalable VLA adaptation that removes the need for per-task data collection and model fine-tuning. This could materially lower the barrier to deploying VLAs on novel tasks and would strengthen the case for investing in large-scale task-agnostic world-model pre-training.

major comments (3)
  1. [Abstract] Abstract: the assertion that 'generalized physical priors can effectively substitute for costly task-dependent data' is load-bearing for the zero-shot claim, yet the abstract supplies no quantitative support such as forward-prediction error on held-out task dynamics, coverage statistics of the pre-training data over relevant state transitions, or direct comparison against task-specific world-model baselines.
  2. [Abstract] Abstract: the dual-noise verification mechanism is presented as the safeguard against hallucinations, but no description is given of the two noise sources, the filtering threshold, or any ablation showing that it improves rollout reliability on tasks whose dynamics differ from the pre-training distribution; without this, it is unclear whether the filter addresses systematic extrapolation failure.
  3. [Experiments] Experiments section: the claim of 'consistent performance gains' across simulation and real-world settings is central, yet the manuscript provides neither the concrete metrics (success rate, cumulative reward, etc.), the set of baselines (task-specific world models, standard RL fine-tuning, etc.), nor ablation results isolating the contribution of the task-agnostic pre-training; this prevents verification that the priors are sufficient for arbitrary downstream tasks.
minor comments (2)
  1. The acronym expansion 'Reinforcing VLAs in task-Agnostic World Dreams' is slightly inconsistent with the title's use of 'World Models'; a single consistent phrasing would reduce reader confusion.
  2. [Abstract] The term 'zero-shot imagination' is used without explicit definition of whether the world model itself remains frozen or receives any task-specific adaptation during the imagined rollouts.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed comments. We have revised the manuscript to provide additional quantitative details, clarifications, and ablations as requested. Below we respond to each major comment.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the assertion that 'generalized physical priors can effectively substitute for costly task-dependent data' is load-bearing for the zero-shot claim, yet the abstract supplies no quantitative support such as forward-prediction error on held-out task dynamics, coverage statistics of the pre-training data over relevant state transitions, or direct comparison against task-specific world-model baselines.

    Authors: We agree that the abstract would be strengthened by including quantitative support for the central claim. The full manuscript reports zero-shot adaptation results that indirectly demonstrate the value of the priors, but we have revised the abstract to reference key performance metrics (e.g., success-rate improvements) from the experiments. We have also added forward-prediction error and coverage statistics on held-out transitions, along with explicit comparisons to task-specific world-model baselines, in a new paragraph in the Experiments section. revision: yes

  2. Referee: [Abstract] Abstract: the dual-noise verification mechanism is presented as the safeguard against hallucinations, but no description is given of the two noise sources, the filtering threshold, or any ablation showing that it improves rollout reliability on tasks whose dynamics differ from the pre-training distribution; without this, it is unclear whether the filter addresses systematic extrapolation failure.

    Authors: The full manuscript describes the dual-noise mechanism in Section 3.3, but we acknowledge the abstract lacks sufficient detail. The two noise sources are ensemble disagreement (epistemic) and VLM output variance (aleatoric); the threshold is set via validation on a small set of trajectories. We have expanded the abstract and Methods to include these specifics and added an ablation study showing improved rollout reliability and reduced extrapolation errors on out-of-distribution tasks. revision: yes

  3. Referee: [Experiments] Experiments section: the claim of 'consistent performance gains' across simulation and real-world settings is central, yet the manuscript provides neither the concrete metrics (success rate, cumulative reward, etc.), the set of baselines (task-specific world models, standard RL fine-tuning, etc.), nor ablation results isolating the contribution of the task-agnostic pre-training; this prevents verification that the priors are sufficient for arbitrary downstream tasks.

    Authors: The Experiments section reports success rates and comparisons, but we agree that a consolidated presentation would improve verifiability. We have added a summary table listing all concrete metrics, the full set of baselines (including task-specific world models and standard RL fine-tuning), and ablations that isolate the task-agnostic pre-training contribution. These revisions directly support the claim that the priors enable adaptation to arbitrary tasks. revision: partial

Circularity Check

0 steps flagged

No circularity: methodological choice presented as enabling assumption, not derived by construction

full rationale

The paper's derivation chain consists of an argument that task-specific fine-tuning of world/reward models limits scalability, followed by the proposal to use a pre-trained task-free world model plus off-the-shelf VLM, with the statement that 'Because both components are task-agnostic, VLAs can be readily finetuned for any new task entirely within this zero-shot imagination.' This is a definitional consequence of the chosen components rather than a mathematical reduction, fitted parameter, or self-citation that forces the zero-shot claim. No equations, uniqueness theorems, ansatzes, or renamed empirical patterns appear; the dual-noise filter is an independent mitigation step. The core assumption that task-free pretraining yields sufficiently general priors is asserted and tested experimentally rather than shown equivalent to the inputs by construction, making the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the untested premise that a world model trained only on task-free behaviors will encode general physical priors sufficient for arbitrary new tasks; the dual-noise verification is introduced ad hoc to handle hallucinations without independent validation.

axioms (1)
  • domain assumption A world model pre-trained on diverse task-free behaviors captures transferable physical priors enabling zero-shot inference on new tasks.
    Explicitly stated as the foundational argument in the abstract.
invented entities (1)
  • dual-noise verification mechanism no independent evidence
    purpose: Filter unreliable imagined rollouts to mitigate world-model hallucinations.
    New component introduced in the paper; no external evidence or prior citation provided in the abstract.

pith-pipeline@v0.9.0 · 5777 in / 1293 out tokens · 43323 ms · 2026-05-21T08:08:38.649154+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · 21 internal anchors

  1. [1]

    Ali, A. et al. World simulation with video foundation models for physical ai.arXiv preprint arXiv:2511.00062, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Bai, S. et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    Black, K. et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Chandra, A.L. et al. Diwa: Diffusion policy adaptation with world models.arXiv preprint arXiv:2508.03645, 2025

    work page arXiv 2025

  5. [5]

    Chen, K. et al. πRL: Online rl fine-tuning for flow-based vision-language-action models.arXiv preprint arXiv: 2510.25889, 2025

    work page arXiv 2025

  6. [6]

    Chen, X. et al. Villa-x: enhancing latent action modeling in vision-language-action models. arXiv preprint arXiv:2507.23682, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Collaboration, O.X.E. et al. Open X-Embodiment: Robotic learning datasets and RT-X models. https://arxiv.org/abs/2310.08864, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Guo, Y . et al. Vlaw: Iterative co-improvement of vision-language-action policy and world model.arXiv preprint arXiv:2602.12063, 2026

    work page arXiv 2026

  9. [9]

    He, H. et al. Pre-trained video generative models as world simulators. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 4645–4653, 2026

    work page 2026

  10. [10]

    Hung, C.Y . et al. Nora-1.5: A vision-language-action model trained using world model-and action-based preference rewards.arXiv preprint arXiv:2511.14659, 2025

    work page arXiv 2025

  11. [11]

    Intelligence, P. et al. π∗ 0.6: a vla that learns from experience.arXiv preprint arXiv: 2511.14759, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Intelligence, P. et al. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Jiang, Z. et al. Wovr: World models as reliable simulators for post-training vla policies with rl. arXiv preprint arXiv:2602.13977, 2026

    work page arXiv 2026

  14. [14]

    Kidambi, R. et al. Morel: Model-based offline reinforcement learning.Advances in neural information processing systems, 33:21810–21823, 2020

    work page 2020

  15. [15]

    Kim, M.J. et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  16. [16]

    Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success

    Kim, M.J., Finn, C. and Liang, P. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Li, H. et al. Simplevla-rl: Scaling vla training via reinforcement learning.arXiv preprint arXiv:2509.09674, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Li, H. et al. Vla-rft: Vision-language-action reinforcement fine-tuning with verified rewards in world simulators.arXiv preprint arXiv:2510.00406, 2025

    work page arXiv 2025

  19. [19]

    Liang, A. et al. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons.arXiv preprint arXiv:2603.02115, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  20. [20]

    Liu, B. et al. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36:44776–44791, 2023

    work page 2023

  21. [21]

    Liu, X. et al. World-vla-loop: Closed-loop learning of video world model and vla policy.arXiv preprint arXiv:2602.06508, 2026

    work page internal anchor Pith review arXiv 2026

  22. [22]

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

    Liu, X., Gong, C. and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  23. [23]

    Lu, C. et al. Challenges and opportunities in offline reinforcement learning from visual observations.arXiv preprint arXiv:2206.04779, 2022

    work page arXiv 2022

  24. [24]

    Lu, G. et al. Vla-rl: Towards masterful and general robotic manipulation with scalable rein- forcement learning.arXiv preprint arXiv:2505.18719, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    Mazzaglia, P. et al. Genrl: Multimodal-foundation world models for generalization in embodied agents.Advances in neural information processing systems, 37:27529–27555, 2024. 10

    work page 2024

  26. [26]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    work page 2023

  27. [27]

    Quevedo, J. et al. Worldgym: World model as an environment for policy evaluation.arXiv preprint arXiv: 2506.00613, 2025

    work page arXiv 2025

  28. [28]

    Sekar, R. et al. Planning to explore via self-supervised world models. InInternational conference on machine learning, pages 8583–8592. PMLR, 2020

    work page 2020

  29. [29]

    Shao, Z. et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Sharma, A.K. et al. World-gymnast: Training robots with reinforcement learning in a world model.arXiv preprint arXiv:2602.02454, 2026

    work page arXiv 2026

  31. [31]

    Team, G.R. et al. Evaluating gemini robotics policies in a veo world simulator.arXiv preprint arXiv: 2512.10675, 2025

    work page arXiv 2025

  32. [32]

    Tong, Z. et al. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078–10093, 2022

    work page 2022

  33. [33]

    Tseng, W.C. et al. Scalable policy evaluation with video world models.arXiv preprint arXiv: 2511.11520, 2025

    work page arXiv 2025

  34. [34]

    Wan, T. et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Wang, Y . et al. Founder: Grounding foundation models in world models for open-ended embodied decision making.arXiv preprint arXiv:2507.12496, 2025

    work page arXiv 2025

  36. [36]

    Wang, Y . et al. Co-evolving latent action world models.arXiv preprint arXiv:2510.26433, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    Xiao, J. et al. World-env: Leveraging world model as a virtual environment for vla post-training. arXiv preprint arXiv:2509.24948, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    Xu, C. et al. Rl token: Bootstrapping online rl with vision-language-action models.arXiv preprint arXiv:2604.23073, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  39. [39]

    Yang, J. et al. Rise: Self-improving robot policy with compositional world model.arXiv preprint arXiv:2602.11075, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  40. [40]

    Yin, T. et al. Playworld: Learning robot world models from autonomous play.arXiv preprint arXiv:2603.09030, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  41. [41]

    Yu, C. et al. Rlinf: Flexible and efficient large-scale reinforcement learning via macro-to-micro flow transformation.arXiv preprint arXiv:2509.15965, 2025

    work page arXiv 2025

  42. [42]

    Yu, T. et al. Mopo: Model-based offline policy optimization.Advances in neural information processing systems, 33:14129–14142, 2020

    work page 2020

  43. [43]

    Zhang, J. et al. Reinforcing action policies by prophesying.arXiv preprint arXiv:2511.20633, 2025

    work page arXiv 2025

  44. [44]

    Zhang, Z. et al. Towards practical world model-based reinforcement learning for vision- language-action models.arXiv preprint arXiv:2603.20607, 2026

    work page arXiv 2026

  45. [45]

    Zhu, F. et al. Irasim: A fine-grained world model for robot manipulation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9834–9844, 2025

    work page 2025

  46. [46]

    Wmpo: World model-based policy optimization for vision-language-action models.arXiv preprint arXiv:2511.09515, 2025

    Zhu, F. et al. Wmpo: World model-based policy optimization for vision-language-action models. arXiv preprint arXiv:2511.09515, 2025. 11 A Implementation Details A.1 Action-Conditioned World Model Architecture.We build on the WAN 2.1 T2V-1.3B DiT backbone with a paired V AE (latent dim C=16, stride (4,8,8) ), yielding a 32×32 spatial latent grid from 256×2...

    work page arXiv 2025