Robotics · arXiv:2608.26103 v2 · submitted 26 Aug 2026
Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
The paper says
“Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning.”
more of the abstract
In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Aug 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Real gains on unseen RoboTwin tasks and a sensible IFP objective, but the zero-shot claim rests on an unquantified synthetic-video pipeline and an evaluation protocol that may allow low-level visual copying. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 2.2, Figure 2
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The HumanGen construction pipeline is load-bearing for the central claim, but its output fidelity is asserted rather than quantified. The text says the VLM filter accepts videos that preserve 'task semantic preservation and physical plausibility', yet no pass rates, score distributions, rejection examples, or quality metrics are reported, and HumanGen is not released.
more of objection 1
If accepted videos contain semantic errors or generation artifacts, the ICL training signal could teach the model to exploit artifacts or distributional regularities rather than task understanding. Please provide quantitative validation of the filter (pass rates per data source, human or VLM agreement on a sample, and examples of accepted and rejected videos), and add a control that uses deliberately mismatched human videos (wrong task, or same task with different object placement) to confirm that the model's behavior depends on task semantics rather than low-level artifacts.
The paper says, at Section 4.2
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The simulation evaluation protocol is underspecified in a way that directly affects the headline number. The paper states that for every RoboTwin trajectory a corresponding human video instruction is generated, and that human-video-conditioned evaluation uses a generated video, but it does not state whether the test-time prompt video is generated from the exact trajectory or initial state being rolled out.
more of objection 2
If the prompt video is episode-matched, the 46.95% success rate could be explained by copying low-level visual state changes from a synthetic oracle demonstration rather than by open-ended task generalization. Please state the matching protocol precisely, and add a control in which the prompt video is taken from a different trajectory of the same task with different object placement or initial state, or from a semantically related but not identical task, and report the resulting success rates.
The paper says, at Section 4.2, Table 2 footnote
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The success criteria for two of the seven unseen tasks ('stamp seal' and 'move stapler to pad') were adjusted after the fact, as stated in the footnote: 'we slightly adjust the success criteria to make them more appropriate for cross-task generalization evaluation'.
more of objection 3
Because these are two of only seven tasks and the adjusted criteria are used for all methods, this is a potential confound for the headline average of 46.95%. Please justify the adjustments a priori, report the results under the original criteria, and provide a sensitivity analysis showing how the ranking and average change when the original criteria are used.
4 minor objections
The paper says, at Section 4.3, Table 3
Pith says: minor objection 1 of 4
The statement 'we report task success rates over 30 real-robot trials' is ambiguous: please clarify whether 30 trials were collected per task family or across all three families, and report the number of trials per configuration together with per-configuration breakdowns.
The paper says, at Figure 2 caption
Pith says: minor objection 2 of 4
The figure contains placeholder text 'xxx tasks, Xxx samples' in the Task-diverse VA panel; this should be filled in with the actual numbers or removed.
show 2 more
The paper says, at Section 4.2 footnote
Pith says: minor objection 3 of 4
The footnote says the modified task versions will be released, but no link or repository identifier is provided; please include the release location for the modified RoboTwin tasks.
The paper says, at Section 4.4
Pith says: minor objection 4 of 4
The naming 'Zero-WAM w/o pretrain' is confusing because the variant is initialized from Wan-2.2 pretrained weights and is only missing the large-scale ICL and task-balanced pre-training; consider renaming it to something like 'w/o large-scale pre-training' to avoid ambiguity.
What would settle it
On the seven RoboTwin unseen tasks, keep all rollouts and seeds fixed and evaluate Zero-WAM under three human-video conditions: (A) the current protocol; (B) a human video of a different unseen task (mismatched semantics); (C) a human video generated from the same task but a different trajectory and initial state, with the trajectory-pairing rule explicitly recorded. If condition (B) stays within about 5 percentage points of (A), the policy is not using semantic task content. If condition (C) drops substantially from (A), the headline result is inflated by episode-matched demonstrations. Also report the VLM filter acceptance rate and a random sample of accepted and rejected videos to quantify HumanGen fidelity.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Aug 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Quick read of Zero-WAM. The headline result is real: 46.95% average on seven unseen RoboTwin tasks versus 17.45% for LingBot-VA, and the ablations show each piece (ICL, IFP, task-balanced data) pulls weight. IFP is a clever training-only auxiliary loss, and the idea of generating paired human videos from robot trajectories at scale is worth taking seriously. This is a legitimate engineering contribution.
But the central claim—that the policy follows human video instructions semantically—is only as strong as the pipeline producing the videos and the protocol evaluating them. The paper gives no numbers on HumanGen fidelity: no pass rate for the VLM filter, no quality scores, no rejected examples, no release. That matters because ICL training on artifact-ridden synthetic videos can teach the model to exploit generation artifacts. The ablations do not settle it, since the same pipeline supplies both training and test prompts.
The bigger problem is the simulation protocol. For every RoboTwin trajectory a corresponding human video is generated, and the unseen task is specified by such a video. The paper does not state whether the test-time video is generated from the exact trajectory or initial state being rolled out. If it is episode-matched, the model may be copying low-level visual state changes from a synthetic oracle demonstration, which is closer to one-shot imitation than to open-ended generalization. The stress-test note lands here. A simple control—using a mismatched human video for the same task, or a video generated from a different initial state—would separate semantic following from visual copying. Without it, the 46.95% number is not yet evidence for the zero-shot cross-task framing.
Minor but worth saying: two success criteria were adjusted post hoc (Section 4.2 footnote). The authors say they will release the modified tasks and apply the same criteria to all methods, which is fair, but it needs to happen before the claim can be checked. Real-world results have no error bars and only 30 trials per cell; useful as existence proofs, not as quantitative comparisons.
On citations: the LingBot-VA foundation is credited, and the related work is honestly positioned. No red flags there. Code/data absence is the main practical obstacle.
Who is this for? People working on video-conditioned policies and world action models. It deserves a serious referee: the system is coherent, the gains are large, and IFP is worth discussing. I would send it to review with a request for (a) HumanGen quality numbers, (b) a mismatched-video control, and (c) clarity on test-time video construction. If those hold, this is a solid paper.
The strongest objection
The central claim—that Zero-WAM executes unseen tasks by following in-context human video guidance—requires both that the human video prompt carries task semantics and that the policy actually extracts those semantics. Both legs are unverified. First, HumanGen is built entirely from synthetic videos produced by a VLM-authored editing/generation pipeline followed by a VLM filter (Section 2.2). The paper reports no pass rates, no quality scores, no rejected examples, and no data release; 'task semantic preservation and physical plausibility' is asserted but not quantified. If accepted videos are semantically wrong or contain generation artifacts, the ICL training signal can teach the model to exploit artifacts or distributional regularities rather than task understanding. Second, even granting fidelity, the simulation evaluation may give the policy more than task-level guidance. The paper states that for every RoboTwin trajectory a corresponding human video is generated (Section 4.2), and the test task is specified by such a video, but it does not state whether the test-time video is generated from the exact trajectory or initial state being rolled out. If the human video is episode-matched, 46.95% success could come from copying low-level visual state changes from a synthetic oracle demonstration rather than from open-ended task generalization. The load-bearing assumption is therefore twofold: generated videos preserve semantics, and the evaluation protocol requires semantic reasoning rather than low-level visual correspondence. Neither is currently supported by the reported experiments.
The reader's weakest assumption is the fidelity of the automatic human-video generation pipeline, which is indeed load-bearing: if the synthetic human videos do not preserve task semantics, the model may learn generation artifacts rather than task understanding, and the main comparison against language-conditioned baselines would no longer support the claim that human video guidance is the cause of the gains. I add a second, closely related threat that the reader noted but did not elevate: the test-time human video in simulation is generated from the unseen task's own robot trajectories, and the paper does not specify whether that video is matched to the exact rollout episode. If it is episode-matched, the 46.95% result could reflect low-level visual copying of a synthetic oracle rather than cross-task generalization. Both threats are testable with a single controlled evaluation that varies the semantic and trajectory correspondence of the human video prompt. Since the reader already issued a conditional verdict and my concern strengthens rather than overturns that verdict, I recommend keeping the verdict unchanged rather than escalating to rejection.
The referee's report
If the central claims hold, the paper makes a valuable contribution: it offers a scalable automatic pipeline for creating human-robot in-context pairs, a concrete training objective that appears to reduce shortcut learning, and a reasonably thorough simulation protocol with three seeds, 100 rollouts per task per seed, and per-task standard deviations. The real-world evaluation across three task families adds evidence of transfer to unseen configurations. The principal weakness is that the synthetic human videos are not independently validated and the evaluation protocol does not rule out the policy exploiting low-level alignment between the prompt video and the rollout, so the semantic-following claim is not yet fully established. The paper reports detailed hyperparameters and ablation setups, which is a strength for reproducibility.
Pith's reading of the paper (a machine's summary, not the authors')
The paper tries to establish that cross-task generalization in robot manipulation can be framed as in-context learning, where a human demonstration video serves as the task specification and no parameter update is needed at test time. It argues that language underspecifies manipulation because it cannot convey spatial constraints and temporal structure as directly as video can. To train such a policy, the authors build an automatic pipeline that converts robot trajectories into semantically matched human videos, yielding 74.2K human-robot pairs across 8.6K tasks, and they also re-balance public robot data by task. They then introduce an auxiliary objective that forces the model to draw task information from the human video rather than from shortcuts in the robot history. If the results hold, this represents a scalable route toward open-ended task generalization without task-specific robot data collection.
What carries the argument. The load-bearing mechanism is the in-context future chunk prediction (IFP) objective: during training, after the model predicts the immediate next video chunk, lightweight auxiliary modules must predict several strided future video chunks from the fused current robot-video representation. These modules are conditioned only on that representation and not directly on the human video, so the main video Transformer must encode the human video's task information into the representation or the auxiliary loss cannot be reduced. At test time the IFP modules are removed, leaving a policy that has been shaped to use the in-context video. Supporting machinery includes a pipeline that turns each sampled robot trajectory into a semantically matched synthetic human video, a height-axis rotary position offset that keeps human-video latents distinct from robot-video latents, and a task-balanced re-sampling of public robot trajectories.
Core claim. The central claim is that a causal video-action model, Zero-WAM, can execute unseen manipulation tasks by treating a human video as an in-context instruction: at deployment the video is prepended to the robot's history, and the model autoregressively predicts future robot video chunks and the temporally aligned executable action chunks. The paper demonstrates this on seven unseen simulated tasks, achieving a 46.95% average success rate against 17.45% for the strongest video-action baseline, and shows transfer to unseen real-world configurations involving multi-object scenes, long-horizon sequential manipulation, and fine-grained insertion, all without collecting task-specific robot data or updating model parameters. The paper attributes the gain to three components working together: human video instructions generated at scale, task-balanced robotic pre-training data, and the in-context future chunk prediction objective that suppresses shortcut learning from seen tasks.
The premise it rests on. The load-bearing premise is that synthetic human videos generated from robot trajectories preserve task semantics faithfully enough that the model learns to understand tasks rather than to exploit generation artifacts; the paper reports no quantitative pass rates for its filtering and does not release the generated data, so this fidelity is unverified.
If the paper is right (Pith's inferences, not the authors')
If the claim is correct, any task that can be demonstrated by a human video becomes executable without new robot data, sharply lowering the cost of deploying robots to novel manipulation tasks.
Language-conditioned policies could be upgraded by adding a video demonstration channel, since video carries visual evidence of task evolution that text omits.
The synthetic pairing pipeline suggests a way to create large human-robot instruction corpora from existing robot datasets, potentially enabling ICL-style scaling for manipulation.
The IFP training objective is a transferable technique that could reduce shortcut learning in other world-action models and enforce stronger use of whatever prompt modality is available.
If the synthetic human videos are faithful enough, the same pipeline could be applied to much larger in-the-wild human video corpora, pairing them retroactively with robot trajectories to expand ICL data without new collection.
The architecture separates task specification (human video) from action decoding (inverse dynamics), suggesting that the action branch is embodiment-specific while the video branch carries transferable task knowledge; one might swap action decoders across robots without retraining the video side.
A decisive test would be to compare performance when the human video prompt is replaced by a real, non-generated human demonstration of the same task; if success drops sharply, the model may be exploiting generation artifacts rather than task understanding.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.44
“On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.23 · human participants 0.31 · controlled comparison 0.93 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.05
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. | background 0.97 | 0.05 | 0.03 | asserted 1.00 |
| In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. | background 1.00 | 0.05 | 0.04 | asserted 0.99 |
| This form of in-context learning (ICL) turns generalization into a problem of task specification. | background 0.90 | 0.06 | 0.06 | asserted 1.00 |
| To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. | contribution 0.64 | 0.22 | 0.12 | asserted 0.97 |
| We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. | contribution 0.93 | 0.90 | 0.31 | asserted 1.00 |
| To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. | method 0.53 | 0.96 | 0.05 | asserted 1.00 |
| For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. | method 0.93 | 0.92 | 0.05 | asserted 0.98 |
| On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. | result 1.00 | 0.96 | 0.36 | asserted 1.00 |
| In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion. | result 1.00 | 0.93 | 0.27 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.82
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,087 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 15 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- World In-Context Learning for Agentic Robot Manipulation cs.CV · Nov 2026
- Open-HumanWAM: An Open Recipe for Scaling Human and Robot Experience for Embodied Foundation Models cs.CV · Nov 2026
- Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution cs.RO · Oct 2026
- WAM-Cache: Staleness-Bounded KV Reuse for Efficient World Action Models cs.RO · Oct 2026
- GeoMind: A Benchmark for Spatial Understanding in Robotic Manipulation cs.RO · Nov 2026
- Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent cs.RO · Oct 2026
- Contact Trajectory Prompting: In-Context Transfer of Contact-Rich Behaviors from a Single Demonstration cs.RO · Oct 2026
- Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence cs.AI · Oct 2026
- OpenWAM: An Open Framework for Composable World-Action Models cs.RO · Oct 2026
- EVO-WAM: Evolving World Action Models through Video-Action Verification cs.CV · Sep 2026
- Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer cs.RO · Sep 2026
- Cycle WAM: Coupling Action and Video through Cycle Consistency cs.CV · Nov 2026
- When the Past Shapes Action: Understanding Embodied Memory cs.RO · Oct 2026
- In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks cs.RO · Sep 2026
- RoboICL: Embodied In-Context Learning with GPT-6 Astra cs.RO · Sep 2026
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 31 sentences without a confident label
Uses the method or tool of 5 papers
-
Following LingBot-VA [ref] , we instantiate from Wan-2.2-TI2V-5B [ref] and convert the image-to-video generation backbone into the causal video-action model described in <Ref>.
Causal World Modeling for Robot Control · Implementation · 0.99
-
Following LingBot-VA [ref] , we instantiate from Wan-2.2-TI2V-5B [ref] and convert the image-to-video generation backbone into the causal video-action model described in <Ref>.
Wan: Open and Advanced Large-Scale Video Generative Models · Implementation · 0.99
-
We follow the causal video-action framework of the LingBot-VA series [ref] , in which each control step is a causal prediction problem: given the task condition and history observation, the model predicts the next robot video chunk together with its temporally aligned executable action chunk.
Native Video-Action Pretraining for Generalizable Robot Control · Preliminaries · 0.97
Uses data or values from 3 papers
-
This subset is built from the same five public robotic video-action datasets used for data: AgiBot [ref] , InternData-A1 [ref] , Open-X-Embodiment [ref] , RoboCOIN [ref] , and RoboMIND [ref] .
AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems · In-Context HumanGen Dataset · 1.00
-
This subset is built from the same five public robotic video-action datasets used for data: AgiBot [ref] , InternData-A1 [ref] , Open-X-Embodiment [ref] , RoboCOIN [ref] , and RoboMIND [ref] .
2511.16651 · In-Context HumanGen Dataset · 1.00
-
This subset is built from the same five public robotic video-action datasets used for data: AgiBot [ref] , InternData-A1 [ref] , Open-X-Embodiment [ref] , RoboCOIN [ref] , and RoboMIND [ref] .
RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation · In-Context HumanGen Dataset · 1.00
Compares with 1 paper
-
In RoboTwin 2.0, achieves a $46.95%$ average success rate across seven unseen tasks, outperforming LingBot-VA [ref] by an absolute margin of $29.50$ percentage points.
Causal World Modeling for Robot Control · Introduction · 0.99
Cites as background 34 papers
-
World action models for robotics [ref] jointly predict future visual states, such as videos or latent representations, and executable robot actions from an instruction and the current observation.
2603.10448 · World Action Models · 0.98
-
World action models for robotics [ref] jointly predict future visual states, such as videos or latent representations, and executable robot actions from an instruction and the current observation.
2602.22010 · World Action Models · 0.98
-
World action models for robotics [ref] jointly predict future visual states, such as videos or latent representations, and executable robot actions from an instruction and the current observation.
WALL-WM: Carving World Action Modeling at the Event Joints · World Action Models · 0.98
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment