Pith.

Computer Vision and Pattern Recognition · arXiv:2611.12870 v2 · submitted 9 Nov 2026

SWAM: Scaling World Action Models with Cross-Modal Alignment

Yu Zhou, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen, Bowen Liu, Jianhan Ma, Weicheng Duan

show 3 more authors

Jun Lv, Cewu Lu, Yanwei Li

The paper says

“World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment.”

more of the abstract

Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 10 Nov 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

the referee pass did not run

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

What would settle it

Re-run the RoboTwin Hard evaluation for the final model (Table 10) with 5 random seeds instead of 1, and report mean and standard deviation of SR across seeds. If the 61.56% SR is within 2 standard deviations of the OpenWAM baseline (48.70%), the headline claim of superiority under Hard conditions would weaken. If the gap remains significant, the central claim is robust.

from the skeptic's reading

How the review went

2 readings by deepseek-v4-flash · 10 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic disagrees 15 s · 27,192 tokens read, 629 written

Not run on this paper: the referee, the editor, the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The strongest objection

The paper's central claim is that preserving language-video capabilities and extending them to language-video-action modeling jointly support control generalization in world action models. The evidence for this claim is extensive and internally consistent. The reader's weakest_assumption—that the video branch is not conditioned on proprioceptive state (Eq. 2)—is a design choice that the paper explicitly states and does not ablate. However, this is not a load-bearing concern for the central claim. The paper's claim is about the joint effect of data mixing, language annotation quality, and architecture design; the state-conditioning choice is a specific architectural detail that could be suboptimal but does not undermine the overall conclusion that the proposed training recipe improves generalization. The paper acknowledges its limitations (complex reasoning, long-horizon tasks) and provides open-source data, code, and weights. The results are reported without confidence intervals, but the trends are consistent across multiple ablations and two evaluation domains (simulation and real-world). The scaling study shows monotonic improvement in real-world Hard SR/PSR, and the simulation results show a clear advantage over baselines. No single assumption or missing control is likely to reverse the central finding.

The reader identified the video branch not conditioning on proprioceptive state as the weakest assumption. While this is a design choice that could be suboptimal, it does not threaten the central claim. The paper's claim is that the overall training recipe (data mixing, language annotation quality, architecture) improves control generalization. The state-conditioning choice is one architectural detail among many; even if it were suboptimal, the paper's main results would still demonstrate that the recipe works. The paper provides extensive ablations on data composition, language annotation, and architecture, and the results are consistent across simulation and real-world. The lack of confidence intervals is a limitation but not a fatal flaw. The reader's concern is valid but not load-bearing. I disagree with the reader's assessment that this is the weakest assumption; the weakest assumption is more likely the reliance on a single random seed for the main results, but even that does not invalidate the central claim given the consistency across ablations. The paper is a solid empirical contribution and the verdict should remain CONDITIONAL (or UNCHANGED from the reader's CONDITIONAL). I recommend UNCHANGED because the reader's verdict already accounts for the limitations.

Pith's reading of the paper (a machine's summary, not the authors')

The paper investigates how to preserve the language–video capabilities acquired during large-scale video pretraining when extending a model to also predict robot actions. The authors curate a large multisource dataset (47,000 hours) and systematically study how language annotation quality, action supervision from human manipulation data, data mixing ratios, and model architecture affect control generalization. Their central claim is that maintaining language–video alignment while adding language–video–action alignment jointly supports better generalization, and they provide a concrete training recipe that achieves 88.10% success on RoboTwin Easy, 61.56% on RoboTwin Hard, and 40.00% across three real-world tasks (vs. 23.33% for the strongest baseline). The key insight is that detailed language descriptions, subtask-level captions, and mixing egocentric video with human manipulation demonstrations all contribute to preserving the pretrained video priors while learning action capabilities.

What carries the argument. The model uses a dual-stream transformer architecture (Mixture-of-Transformers) with a video branch initialized from a pretrained video generation model (Wan2.2) and a randomly initialized action branch. The video branch receives language conditioning through cross-attention, while the action branch accesses language only through attention interactions with the video branch (no direct text cross-attention). Video and action maintain separate residual streams with unidirectional attention from video to action. The training objective is flow matching on both video latents and action sequences, with equal weighting. The key design choice is that the video branch is not conditioned on the proprioceptive state, which the authors state explicitly.

Core claim. The paper establishes that preserving the language–video alignment acquired during video pretraining and extending it to language–video–action modeling through appropriate data mixing, language annotation quality, and architecture design jointly support control generalization in world action models. The specific quantitative manifestation is that the resulting training recipe achieves 88.10% success rate on RoboTwin Easy, 61.56% on RoboTwin Hard, and 40.00% SR across three real-world tasks under the Hard setting, compared to 23.33% for the strongest baseline OpenWAM.

The premise it rests on. The video branch is not conditioned on the proprioceptive state (Eq. 2), meaning the model predicts visual changes without knowing the robot's arm position, which may be suboptimal for tasks requiring tight visual-state coupling.

If the paper is right (Pith's inferences, not the authors')

If the recipe generalizes beyond the evaluated tasks, it provides a scalable path from video pretraining to robot control using human manipulation demonstrations collected with handheld grippers (UMI).

The finding that detailed language descriptions and subtask-level captions improve control generalization provides concrete guidance for annotation pipeline design in robot learning.

The result that mixing egocentric video with UMI demonstrations under equal sampling outperforms either source alone suggests that language–video supervision and language–video–action supervision play complementary roles.

Scaling pretraining from 0 to 100k video hours shows continued improvements in real-world success rates, supporting further investment in multisource data collection.

The architecture with language conditioning only in the video branch outperforms variants with direct text cross-attention in the action branch, providing a design principle for future world action models.

The paper's emphasis on language annotation quality implies that current robot datasets with generic task instructions (e.g., 'pick up the snack') may be systematically underutilizing the language–video priors from pretrained video models. Investing in richer annotations could yield significant gains without additional robot data.

The modeling choice to not condition the video branch on proprioceptive state (stated in Eq. 2) is not ablated. If video prediction depends on knowing arm position, this choice could limit the model's ability to learn state-dependent dynamics. A comparison against a variant where the video branch also receives state information would clarify this.

The paper's limited support for complex reasoning and multi-stage planning (acknowledged by the authors) suggests that the current recipe is primarily effective for short-horizon manipulation tasks. Extending to long-horizon tasks may require memory mechanisms or hierarchical planning, as the authors note.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-11

The authors' main claim, as the classifier picked it out 0.90

“Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions.”

How the main claim is stated: asserted 0.61

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.31 · human participants 0.11 · controlled comparison 0.93 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
World action models jointly predict video and actions conditioned on language. background 1.00 0.08 0.03 asserted 1.00
Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. background 0.99 0.04 0.02 asserted 1.00
However, introducing action learning may impair this alignment. background 0.66 0.12 0.04 hedged possibility 0.99
Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. gap 1.00 0.06 0.04 asserted 0.98
To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. method 0.91 0.97 0.02 asserted 0.99
Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. contribution 0.74 0.93 0.05 asserted 0.85
In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. contribution 0.54 0.80 0.05 asserted 0.84
To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. method 0.81 0.90 0.03 asserted 0.76
Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. result 1.00 0.96 0.13 asserted 1.00
Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. result 0.81 0.91 0.10 asserted 1.00
The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. result 1.00 0.97 0.25 asserted 1.00
Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. result 0.98 0.96 0.25 asserted 0.61
We release the curated data, training code, and model weights. contribution 0.99 0.92 0.04 asserted 0.99

What kind of paper the classifier reports this is: empirical study 0.55 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-11: 1 request, 9,442 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (16)
Figure 1 of the paper
Figure 1. Overview of SWAM. We study how multisource supervision preserves the language–video capabilities acquired during video pretraining and extends them to robot control. We curate a multisource data pool of approximately 47,000 hours to support this study. Experiments show that additional pretraining further improves real-world performance, and that the resulting recipe generalizes to unseen scene conditions in RoboTwin and on real robots. ‡Work completed during internships at Shanghai Jiao Tong University. arXiv:2611.12870v1 [cs.CV] 9 Nov 2026
Figure 2 of the paper
Figure 2. Multisource data and model overview. (a) Composition of the 47k-hour pretraining data pool. (b) Our world action model architecture, with video and action branches connected through shared masked attention. To study these questions, we first construct a multisource data pool with approximately 47,000 hours, comprising approximately 11,000 hours of general videos, 11,000 hours of egocentric videos, 15,000 hours of in-house UMI demonstrations, and 10,000 hours of robot demonstrations, as shown in Fig￾ure 2(a). These sources provide different forms of cross-modal supervision. General…
Figure 3 of the paper
Figure 3. Pretraining data composition and video dura￾tion across sources. (a) Shares of total video hours. (b) Shares of total episodes. (c) Video-duration distribution within each source. 3.1. World Action Modeling 3.1.1. Problem Formulation We consider language-conditioned robot manipulation. Given the visual observations acquired so far, a language instruction, and its current proprioceptive state, the robot predicts actions over a future interval. Let 𝑛 be a prediction start point sampled from a trajectory, 𝐾 > 0 a fixed prediction horizon, 𝑉0:𝑛 the visual history including the current…
Figure 4 of the paper
Figure 4. World action model architecture and attention masks. (a) The video and action DiT branches are con￾nected through shared masked attention, with text cross-attention in the video branch only. Attention masks for (b) training with clean video history under teacher forcing and (c) generating video and actions chunk by chunk at inference. 3.1.3. Training Objectives For chunk 𝑡, 𝑥𝑡 and 𝐴𝑡 denote the clean video latents and action targets, respectively. Let 𝜏 ∈ [0, 1] denote flow matching time. We sample independent standard Gaussian noise 𝜖𝑣 and 𝜖𝑎 with the same shapes as their…
Figure 5 of the paper
Figure 5. Multisource training data and curation pipeline. The upper panel shows representative examples from four data sources. The lower panel illustrates visual checks, action checks, format conversion, subtask segmentation, and caption generation. Captions include action descriptions at three levels of detail, along with scene and lighting information. General videos. We use OpenVid [32] and Panda70M [33], totaling 10,909.6 hours. These open-source video generation datasets contain videos and text descriptions covering diverse scenes, objects, and visual dynamics. We retain their videos…
Figure 6 of the paper
Figure 6. Scaling pretraining improves control generalization. We vary training video hours on a fixed corpus of multisource experience, using the same downstream post-training protocol within each domain. (a) RoboTwin SR under the Easy (C2C) and Hard (C2R) settings. (b) Real-world SR and PSR under the Hard setting, averaged across three tasks. The 0-hour setting uses the video-model initialization without additional pretraining and achieves no success in real-world evaluation. As shown in
Figure 7 of the paper
Figure 7. Our method achieves the best performance under the Hard setting in real-world evaluation. Results are averaged equally across three real-world tasks. a fuller view of our model’s execution,
Figure 8 of the paper
Figure 8. Real-world execution comparisons. Our model completes cup stacking and towel folding under the Easy setting and gift-in-box under the Hard setting, while the corresponding baseline executions fail. Light red dashed boxes highlight the baselines’ main execution errors. Fruit Placement External Head Rainbow Stack External Head
Figure 9 of the paper
Figure 9. Generalization examples from head and external camera views. Fruit placement and rainbow stack demonstrate generalization to new objects, containers, and scene appearances. 18
Figure 10 of the paper
Figure 10. Real-world evaluation layouts under Easy and Hard settings. All models use the same ten layouts for each task and setting, with three trials per layout. Observations and actions. We use a bimanual Flexiv Rizon 4 platform. Observations comprise two fisheye wrist-camera views and one fixed head-camera view. Actions encode the end-effector position, orientation, and continuous gripper width for each arm. For OpenWAM, we follow its official default configuration and use a 6D rotation representation. All other models represent orientation with a wxyz quaternion, yielding a…
Figure 11 of the paper
Figure 11. Subtask illustrations for the three real-world tasks. C.2. Evaluation Protocols and Metrics Evaluation protocols. We evaluate under Easy (C2C) and Hard (C2R) settings. Easy settings retain the scene conditions and physical objects used in post-training, including their shapes and colors, while varying their initial configurations across layouts. Hard settings vary across scene and object attributes, including background, tabletop material, object instance, shape, color, size, and texture. The evaluation layouts are illustrated in
Figure 12 of the paper
Figure 12. reports SR and PSR for each real-world task and their average under the Easy and Hard settings. StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours 0 20 40 60 80 100 Easy setting Cup stacking StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours Towel folding StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours Gift-in-box StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours Average StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours 0 20 40 60 80 100 Hard setting StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours StarVLA π0.5 Fast￾WAM LingBot￾VA Open WAM Ours SR PSR
Figure 13 of the paper
Figure 13. compares the 0-hour setting with models pretrained for 25k, 50k, 75k, and 100k steps, corre￾sponding to approximately 22.5k, 45k, 67.5k, and 90k training video hours, respectively. At 0 hours, SR and PSR are both 0% for every task under both settings. 0 20K 40K 60K 80K 100K 0 20 40 60 80 100 Easy setting 10.00 30.00 23.33 10.00 35.71 62.38 52.38 40.48 0 Cup stacking 0 20K 40K 60K 80K 100K 63.33 26.67 50.00 43.33 71.67 30.00 56.67 45.00 0 Towel folding 0 20K 40K 60K 80K 100K 100.00 100.00 96.67 93.33 100.00 100.00 96.67 93.33 0 Gift-in-box 0 20K 40K 60K 80K 100K 57.78 52.22 56.67…
Figure 14 of the paper
Figure 14. Cup stacking. The robot builds a three-tier pyramid from nested cups. Towel folding (Hard) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Figure 15 of the paper
Figure 15. Towel folding. The robot spreads the towel and folds it twice. Gift-in-box (Hard) 1 2 3 4 5 6 7 8 9 10
Figure 16 of the paper
Figure 16. Gift-in-box. The robot places the gift in the box and releases it. 30

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment