REVIEW 3 major objections 4 minor 22 references
Vid2WAM: Distilling Video Diffusion Priors into World Action Models
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Vid2WAM claims that future supervision for World Action Models can come from generated video rollouts rather than target-task expert demonstrations, and that this improves novel-task success and data efficiency without adding…
desk verdict Vid2WAM is a genuinely new distillation recipe with consistent gains, but the novel-task claim rests on an unverified pretraining-leakage assumption the authors need to close. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is dual-channel distillation from a frozen video teacher into a compact student, with the student's VAE re-encoding teacher-generated frames into the same latent space used for its future-prediction targets. Action supervision comes from an inverse dynamics model trained on embodiment-compatible, but not target-task, trajectories, and the two supervision sources are combined through source-aware residual adapters: zero-initialized bottleneck projections around a shared action backbone, one for real actions and one for pseudo-actions, with only the real adapter retained at inference. Flow matching supervises both future-video latents and action chunks. This machinery lets generated futures serve as direct world-model supervision rather than being compressed through an imperfect action bottleneck, while residual adapters contain the noise from synthetic actions.
What would settle it
Search the teacher's pretraining data (the LVP / Wan2.1-14B corpus) for overlap with the 15 RoboTwin novel tasks, the 8 LIBERO novel tasks, and the three real-world novel tasks; if any held-out task is present, re-run the novel regime with a teacher pretrained only on seen-task data. The claim collapses if the gains concentrate in the overlapped tasks and vanish after retraining on a disjoint corpus.
Extended reading notes
Core claim
Vid2WAM establishes that the future supervision used to train a World Action Model need not be recorded target-task trajectories. An embodiment-adapted video diffusion teacher generates task-conditioned rollouts, and the teacher's predictive prior is transferred into a compact student WAM by supervising both the future-prediction branch, through re-encoded teacher latents, and the action branch, through IDM-recovered pseudo-actions. The student trains jointly on real demonstrations and teacher rollouts with source-aware residual adapters, so noisy pseudo-actions get source-specific corrections instead of corrupting a shared action backbone. After training, the video teacher and the inverse dynamics model are discarded, leaving only the compact student for deployment. The paper reports consistent gains over the strongest WAM baseline on novel-task and low-data regimes in simulation and on real bimanual tasks, with ablation evidence that future-latent supervision and pseudo-action supervision are complementary.
Load-bearing premise
The novel-task gains assume that the large video teacher's pretraining corpus does not already contain the held-out evaluation tasks, so the teacher's rollouts for those tasks are genuine generalization rather than memorized replay.
Editorial extensions
If this is right
- Robot policies can be adapted to novel tasks with no target-task expert trajectories: only an initial observation, a language instruction, and offline teacher rollouts are needed.
- The video teacher and IDM add zero latency at deployment; the student's chunk-level inference time is comparable to Fast-WAM and far faster than online video-generation pipelines.
- Data efficiency improves: with only 10 expert trajectories per task, Vid2WAM outperforms VLA and WAM baselines on RoboTwin, LIBERO, and LIBERO-Plus.
- Future-latent supervision is not redundant with action supervision: the ablation shows Future Latent Only beats Pseudo Action Only, and the full dual-channel model is best.
- Offline distillation beats an online teacher-plus-IDM controller, so the gains come from training-time transfer rather than from a stronger runtime policy.
- The benefits persist across different held-out task splits, indicating they are not an artifact of one task selection.
Reading between the lines
- The same dual-channel distillation could be driven by other generative sources, such as VLMs, subgoal proposers, or action-conditioned video models, so the specific 14B video teacher may be replaceable with smaller or cheaper generators.
- The 8-second rollout cutoff used for simulation pseudo-data suggests long-horizon hallucination is the current bottleneck, so improving teacher temporal consistency may yield further gains without changing the student architecture.
- If confirmed, novel-task evaluation protocols should audit the teacher's pretraining corpus for held-out tasks, because otherwise gains attributed to generalization could be memorized replay.
- Source-aware residual adaptation is a general recipe for mixing real labels with noisy synthetic labels, potentially applicable beyond robotics to any distillation setting with unreliable teacher outputs.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents Vid2WAM, an offline distillation framework that transfers the predictive priors of a large video diffusion model into a compact World Action Model (WAM) student. Given an observation and language instruction, the video teacher generates future rollouts, which are decoded and re-encoded into student latent space to supervise the future-prediction branch, while an inverse dynamics model recovers pseudo-actions to supervise action learning. Source-aware residual adapters separate real and pseudo domains during training, and at inference only the compact student is deployed. Experiments on RoboTwin 2.0, LIBERO, LIBERO-Plus, and a real dual-arm platform report improvements over Fast-WAM, Motus, and π0.5 in low-data and novel-task regimes, with particular gains on the RoboTwin novel subset (54.7–55.3% vs. 45.0–42.8%).
Significance. The paper is a carefully executed empirical system contribution. Its main idea—decoupling future supervision for WAMs from target-task expert trajectories by distilling a pretrained video teacher into a compact student—is timely and practically attractive, since it preserves low-latency deployment while borrowing broad visual priors. The paper includes a thorough set of ablations (dual action DiT, no adapter, pseudo-action only, future-latent only, teacher policy), sensitivity analyses over different novel-task splits, and real-robot validation, all of which strengthen the causal interpretation of the components. If the results hold, the framework demonstrates a scalable route for injecting video-foundation-model knowledge into efficient robot policies. However, the headline novel-task generalization claim rests on the unverified assumption that the teacher's large-scale pretraining corpus does not already contain the held-out evaluation tasks; the paper audits only fine-tuning exposure, not pretraining exposure.
major comments (3)
- [Implementation Details / Table S5] The novel-task claims (e.g., RoboTwin novel subset 54.7–55.3% vs. Fast-WAM 45.0–42.8% in Table 1) are interpreted as evidence that the video teacher provides generalizable world knowledge. This interpretation requires that the teacher's initialization, the LVP checkpoint (Chen et al. 2025a), which is described as pretrained on robot manipulation data, does not already contain the held-out RoboTwin 2.0 or LIBERO tasks. Table S5 only audits whether the teacher and IDM were fine-tuned on held-out-task trajectories ('No'), which is not the same as absence from the pretraining corpus. If any held-out task appears in LVP's pretraining data, the teacher's future predictions (and hence the IDM pseudo-actions distilled from them) could be near-memorized solutions, and the novel-regime comparisons would no longer measure generalization. Please provide verifiable evidence about the LVP pretraining corpus, or add an experiment with a teacher checkpoint known to be task-exclusive, or at minimum (c) explicitly re-frame the novel-regime results as conditional on this assumption and discuss the contamination risk.
- [Experimental Setup / Evaluation Regimes] In the novel-task regime, baselines are directly transferred from the remaining tasks with no exposure to target-task data, whereas Vid2WAM is allowed to use target-task instructions and initial observations to generate pseudo supervision. This is an asymmetric comparison: the observed gains could stem in part from Vid2WAM having seen the target-task language instruction and initial visual context during training, rather than from distillation of a generalizable prior. The headline wording 'improves novel-task generalization' should make this information asymmetry explicit (e.g., 'zero-shot transfer vs. task-conditioned offline distillation'). A useful control would be a baseline that is also given the target-task instructions and initial observations (without the video teacher) to isolate the teacher's contribution.
- [Tables 1–5 and S10–S15] All success rates are point estimates from a single training run, with no confidence intervals, seed variance, or binomial intervals. Several headline improvements are modest (e.g., LIBERO novel average 78.3 vs. 76.7 in Table 3; LIBERO-Plus overall novel 48.1 vs. 45.8 in Table 2), and the real-world table reports only 20 trials per task, so differences of 10–15 points could be within sampling noise. Please report at least three seeds (mean ± std) for the simulation benchmarks, or binomial confidence intervals for task-level rates, and state explicitly whether the 20 real-world trials per task are the entire evaluation. This is essential for assessing the reliability of the central empirical claims.
minor comments (4)
- [General] The manuscript does not mention code or data release; please add a statement on availability to support reproducibility.
- [Figure 3] The caption says 'Fast-WAM (top) and Vid2WAM (bottom)', but the sub-figures are not labeled. Please label each row inside the figure for clarity.
- [Eq. (3) / Table S6] The adapter residual scale α is set to 1.0 in Table S6 but its effect is not ablated; please clarify whether α was tuned and, if not, state why a fixed value was used.
- [Table S5] The 'Teacher exposure' and 'IDM exposure' columns use '/' for low-data regimes; consider using 'N/A' to avoid implying a binary value.
Circularity Check
No circularity: gains are externally evaluated; pretraining-corpus caveat is a correctness risk, not a circular step.
full rationale
Vid2WAM is an empirical system paper rather than a derivation, and no step reduces to its inputs by construction. The student's future-latent and action losses (Eqs. 5-7) supervise the student with teacher-generated rollouts and IDM pseudo-actions, but the reported success rates are measured by executing the distilled student on held-out tasks against external baselines; no parameter is fitted to the novel-task outcomes, and the paper does not rename a fitted quantity as a prediction. The same-group citations (Wang et al. 2026; Zhou et al. 2026) are peripheral and not load-bearing. The legitimate caveat is a contamination risk, not circularity: the video teacher is initialized from LVP, a checkpoint 'pretrained on robot manipulation data' (Implementation Details), and Table S5 audits only fine-tuning exposure to held-out trajectories, marking 'Teacher IDM Held-out-task exposure No No'. Thus novel-task generalization could in principle reflect pretraining memorization rather than genuine synthesis. This is an external-validity concern about the benchmark protocol, not a derivation-level circularity, because the evaluation itself is still an independent measurement of the distilled student's task success.
Assumptions & free parameters
free parameters (4)
- lambda_pseudo =
0.25
- beta =
1.0
- adapter residual scale alpha =
1.0
- simulation pseudo-rollout cutoff =
8 seconds
assumptions (4)
- domain assumption The LVP/Wan2.1 teacher checkpoint was pretrained on data that does not include the held-out evaluation tasks.
- domain assumption The IDM trained on seen-task action-labeled trajectories can recover executable actions from teacher-generated rollouts for novel tasks.
- domain assumption Student VAE re-encoding of teacher-generated RGB frames preserves the task-relevant signal needed for future-prediction supervision.
- domain assumption Generated futures are sufficiently physically consistent to serve as training targets rather than noise.
Cite this review
Pith. "Pith review of Vid2WAM: Distilling Video Diffusion Priors into World Action Models." pith.science (2026). https://pith.science/paper/JZNBN5OU
@misc{pith2026260808558,
author = {Pith},
title = {Pith review of: Vid2WAM: Distilling Video Diffusion Priors into World Action Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/JZNBN5OU}},
note = {Machine review of arXiv:2608.08558}
}
read the original abstract
World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[6]
Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models
Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models. arXiv:2604.25859. Fei,S.;Wang,S.;Shi,J.;Dai,Z.;Cai,J.;Qian,P.;Ji,L.;He, X.; Zhang, S.; Fei, Z.; Fu, J.; Gong, J.; and Qiu, X
-
[7]
Ha, H.; Florence, P.; and Song, S
LIBERO-Plus: In-depth Robustness Analysis of Vision- Language-ActionModels.arXivpreprintarXiv:2510.13626. Ha, H.; Florence, P.; and Song, S
-
[8]
InProceedings of the 2023 Conference on Robot Learning
Scaling Up and DistillingDown:Language-GuidedRobotSkillAcquisition. InProceedings of the 2023 Conference on Robot Learning. He,K.;Zhang,X.;Ren,S.;andSun,J.2016. DeepResidual LearningforImageRecognition. InProceedingsoftheIEEE Conference on Computer Vision and Pattern Recognition, 770–778. Hinton, G.; Vinyals, O.; and Dean, J
work page 2023
-
[10]
Video Predic- tionPolicy:AGeneralistRobotPolicywithPredictiveVisual Representations. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceed- ings of Machine Learning Research, 24328–24346. PMLR. Jang,J.;Ye,S.;Lin,Z.;Xiang,J.;Bjorck,J.;Fang,Y.;Hu,F.; Huang, S.; Kundalia, K.; Lin, Y.-C.; Magne, L.; Mandlekar, A.; Naraya...
work page 2025
-
[11]
CKT-WAM: Parameter-Efficient Context Knowledge Transfer Between World Action Models. arXiv:2605.06247. Kim, M. J.; Gao, Y.; Lin, T.-Y.; Lin, Y.-C.; Ge, Y.; Lam, G.; Liang, P.; Song, S.; Liu, M.-Y.; Finn, C.; and Gu, J. 2026a. Cosmos Policy: Fine-Tuning Video Models for Visuomotor ControlandPlanning. InTheFourteenthInternationalCon- ference on Learning Rep...
-
[12]
OpenVLA: An Open-Source Vision-Language-Action Model. InProceed- ings of the 8th Conference on Robot Learning, volume 270 ofProceedings of Machine Learning Research, 2679–2713. PMLR. Kim, S.; Jang, S.; Yoon, B.; Kim, D.; Won, J.; and Shin, J. 2026b. RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning. arXiv:2602.1874...
-
[13]
Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination. arXiv:2606.10040. Li,S.;Gao,Y.;Sadigh,D.;andSong,S.2025. UnifiedVideo Action Model. arXiv:2503.00200. Liang, J.; Liu, R.; Ozguroglu, E.; Sudhakar, S.; Dave, A.; Tokmakov,P.;Song,S.;andVondrick,C.2025. Dreamitate: Real-World Visuomotor Policy Learning via Video Genera- tion....
arXiv 2025
-
[14]
Orca: Progres- sive Learning from Complex Explanation Traces of GPT-4. arXiv:2306.02707. Physical Intelligence; Black, K.; Brown, N.; Darpinian, J.; Dhabalia, K.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; Galliker, M. Y.; Ghosh, D.; Groom, L.; Hausman, K.; Ichter, B.; Jakubczak, S.; Jones, T.; Ke, L.; LeBlanc, D.; Levine, S.; Li-Bell, A.; Mot...
arXiv 2025
Show all 22 references
-
[15]
In8th Annual Conference on Robot Learning
Theia: Dis- tilling Diverse Vision Foundation Models for Robot Learn- ing. In8th Annual Conference on Robot Learning. Wan, T.; Wang, A.; Ai, B.; Wen, B.; Mao, C.; Xie, C.-W.; Chen,D.;Yu,F.;Zhao,H.;Yang,J.;etal.2025. Wan:Open and advanced large-scale video generative models.arX...
2025 arXiv
-
[16]
arXiv:2603.16195
S-VAM: Shortcut Video-Action ModelbySelf-DistillingGeometricandSemanticForesight. arXiv:2603.16195. Ye, S.; Ge, Y.; Zheng, K.; Gao, S.; Yu, S.; Kurian, G.; Indupuru, S.; Tan, Y. L.; Zhu, C.; Xiang, J.; Malik, A.; Lee, K.; Liang, W.; Ranawaka, N.; Gu, J.; Xu, Y.; Wang, G.; Hu, ...
-
[17]
arXiv:2602.15922
World Action Models are Zero-shot Policies. arXiv:2602.15922. Ye, S.; Jang, J.; Jeon, B.; Joo, S.; Yang, J.; Peng, B.; Man- dlekar, A.; Tan, R.; Chao, Y.-W.; Lin, B. Y.; Liden, L.; Lee, K.; Gao, J.; Zettlemoyer, L.; Fox, D.; and Seo, M
-
[18]
InThe Thirteenth International Conference on Learning Representations
Latent Action Pretraining from Videos. InThe Thirteenth International Conference on Learning Representations. Yu, T.; Xiao, T.; Stone, A.; Tompson, J.; Brohan, A.; Wang, S.;Singh,J.;Tan,C.;M,D.;Peralta,J.;Ichter,B.;Hausman, K.;andXia,F.2023. ScalingRobotLearningwithSemanti- ca...
2023
-
[19]
Zhou,P.;Chen,S.;Chen,D.;Wang,J.;Jin,R.;Zhu,B.;Pan, Y.; Gu, S.; Wang, K.; Nan, S.; Qiu, X.; Qiu, C.; Yang, P.; Cai, Y.; Gao, J.; Li, Y.; Fu, Y.; Yue, X.; Chen, Z.; and Luo, J
Fast-WAM: Do World Action Models Need Test-Time Future Imagina- tion? arXiv:2603.16666. Zhou,P.;Chen,S.;Chen,D.;Wang,J.;Jin,R.;Zhu,B.;Pan, Y.; Gu, S.; Wang, K.; Nan, S.; Qiu, X.; Qiu, C.; Yang, P.; Cai, Y.; Gao, J.; Li, Y.; Fu, Y.; Yue, X.; Chen, Z.; and Luo, J. 2026.τ 0-WM: A...
2026 arXiv
-
[20]
arXiv:2504.02792
Unified World Models: Coupling Video and Ac- tion Diffusion for Pretraining on Large Robotic Datasets. arXiv:2504.02792. Zitkovich,B.;Yu,T.;Xu,S.;Xu,P.;Xiao,T.;Xia,F.;Wu,J.; Wohlhart,P.; Welker,S.; Wahid,A.; Vuong,Q.;Vanhoucke, V.; Tran, H.; Soricut, R.; Singh, A.; Singh, J.; ...
-
[21]
InConference on Robot Learning, 2165–2183
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. InConference on Robot Learning, 2165–2183. A Implementation Details Forallmodelsincludingvideoteacher,IDMandVid2WAM, on LIBERO and LIBERO-Plus, the head and wrist camera views are resized to224×224a...
2025
-
[2015]
arXiv:1503.02531
Distilling the Knowledge in a Neural Network. arXiv:1503.02531. Hsieh, C.-Y.; Li, C.-L.; Yeh, C.-k.; Nakhost, H.; Fujii, Y.; Ratner,A.;Krishna,R.;Lee,C.-Y.;andPfister,T.2023. Dis- tilling Step-by-Step! Outperforming Larger Language Mod- els with Less Training Data and Smaller ...
2023 arXiv
-
[2016]
Teacher exposure
whose first convolution is expanded from three to nine input channels to jointly process the three RGB frames. The pre- trainedconvolutionalfiltersarerepeatedacrossthetemporal dimension and divided by three. The final ResNet feature map is passed through a learnable spatial-so...
-
[2023]
arXiv:2302.06671
GenAug: Retargeting Behaviors to Unseen Situations via Generative Augmentation. arXiv:2302.06671. Collins, J. A.; Cheng, L.; Aneja, K.; Wilcox, A.; Joffe, B.; andGarg,A.2025. AMPLIFY:ActionlessMotionPriorsfor Robot Learning from Videos. arXiv:2506.14198. Du,Y.;Yang,M.;Dai,B.;D...
2025 arXiv
-
[2024]
Chen, B.; Zhang, T.; Geng, H.; Zhang, C.; Li, P.; Song, K.; Freeman, W
Diffusion forcing: Next-token predictionmeetsfull-sequencediffusion.AdvancesinNeural Information Processing Systems, 37: 24081–24125. Chen, B.; Zhang, T.; Geng, H.; Zhang, C.; Li, P.; Song, K.; Freeman, W. T.; Malik, J.; Abbeel, P.; Tedrake, R.; et al. 2025a. Largevideoplanner...
-
[2025]
arXiv:2512.13030
Motus: A Unified Latent Action World Model. arXiv:2512.13030. Black,K.;Brown,N.;Driess,D.;Esmail,A.;Equi,M.;Finn, C.;Fusai,N.;Groom,L.;Hausman,K.;Ichter,B.;Jakubczak, S.; Jones, T.; Ke, L.; Levine, S.; Li-Bell, A.; Mothukuri, M.; Nair, S.; Pertsch, K.; Shi, L. X.; Tanner, J.; ...
2024 arXiv
-
[2026]
arXiv:2606.24448
Supervise What Survives: Geometry- Guided VLA Adaptation from Synthetic Robot Videos. arXiv:2606.24448. Chen, T.; Chen, Z.; Chen, B.; Cai, Z.; Liu, Y.; Li, Z.; Liang, Q.;Lin,X.;Ge,Y.;Gu,Z.;Deng,W.;Guo,Y.;Nian,T.;Xie, X.; Chen, Q.; Su, K.; Xu, T.; Liu, G.; Hu, M.; Gao, H.-a.; W...
-
[9172]
InThe Twelfth International Conference on Learning Representations
Du,Y.;Yang,M.;Florence,P.;Xia,F.;Wahid,A.;Ichter,B.; Sermanet,P.;Yu,T.;Abbeel,P.;Tenenbaum,J.B.;Kaelbling, L.P.;Zeng,A.;andTompson,J.2024.VideoLanguagePlan- ning. InThe Twelfth International Conference on Learning Representations. Fang, P.; Chen, H.; and Cai, X
2024
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.