Native Video-Action Pretraining for Generalizable Robot Control

Chaojian Li; Fangyi Xu; Gangwei Xu; Guanxing Lu; Jiahao Shao; Jiaming Zhou; Jiapeng Zhu; Jiaqi Liao; Junke Wang; Ka Leong Cheng

Native video-action pretraining lets robots learn control from web video

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 03:52 UTC pith:RNHARHC7

load-bearing objection Real-world claims lack statistical support; simulation gains over prior generation are marginal. But the system integration is genuine and the engineering is serious. the 3 major comments →

arxiv 2607.08639 v2 pith:RNHARHC7 submitted 2026-07-09 cs.RO cs.CV

Native Video-Action Pretraining for Generalizable Robot Control

Qihang Zhang , Lin Li , Luyao Zhang , Shuai Yang , Yiming Luo , Shuaiting Li , Ruilin Wang , Junke Wang

show 21 more authors

Jiahao Shao Gangwei Xu Jiaming Zhou Yishu Shen Yudong Jin Fangyi Xu Shuailei Ma Jiaqi Liao Guanxing Lu Zifan Shi Yongkun Wen Yujie Zhao Weixuan Tang Xinyang Wang Chaojian Li Jiapeng Zhu Ka Leong Cheng Nan Xue Xing Zhu Yujun Shen Yinghao Xu

This is my paper

classification cs.RO cs.CV

keywords modelcontrollingbot-vavideo-actionactioncausaldynamicsfoundation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that robot control models should not be built by bolting an action module onto a video generator designed for digital content creation. Such generators use reconstruction-oriented tokenizers and bidirectional attention, both of which mismatch the demands of physical control: the representation preserves appearance rather than dynamics, and the architecture sees the future during training but cannot at deployment. Instead, the authors pretrain the entire stack from scratch for embodiment. The central object is a semantic visual-action tokenizer that aligns visual latents to a frozen foundation model's features and simultaneously learns compact latent actions through self-supervised inverse and forward dynamics on unlabeled video. This places world states and actions in a single shared latent space, so even web video without action labels provides control-relevant supervision. On top of this space, a causal diffusion transformer with a sparse mixture-of-experts video stream is pretrained natively, learning to predict future visual latents and the latent actions connecting them. The paper claims this native route yields a representation that generalizes from 10-15 demonstrations, transfers across embodiments, and in some cases executes new tasks zero-shot, while an asynchronous inference scheme called Foresight Reasoning enables real-time closed-loop control at up to 225 Hz on physical robots.

Core claim

The paper's central claim is that pretraining a video-action model natively for embodiment—using a semantic visual-action tokenizer that jointly learns visual latents and latent actions from unlabeled web video, combined with a causal architecture trained from scratch—produces a control representation that generalizes better and faster than repurposing generic video generators. The semantic tokenizer is the load-bearing piece: by aligning visual latents to foundation-model semantics and extracting latent actions through self-supervised transition prediction, it converts passive web video into supervision for physical control without requiring robot demonstrations at pretraining scale. The ab

What carries the argument

The semantic visual-action tokenizer (Sec. 2.2) is the central mechanism. It augments a ViT autoencoder with two objectives: (1) semantic alignment, which pulls visual latents toward a frozen Perception Encoder's features via temporal average pooling (Eq. 10), and (2) latent action tokenization, which trains an inverse dynamics model q_phi to predict a compact transition variable l_t between consecutive latents (Eq. 12) and a forward dynamics model f_psi to reconstruct the next latent via a transport map and residual (Eq. 13). The resulting paired visual-action latents serve as training targets for a causal DiT with a sparse MoE video stream (128 routed experts, top-8 routing, loss-free load

Load-bearing premise

The entire architecture rests on the semantic visual-action tokenizer: if the latent actions extracted by self-supervised inverse and forward dynamics on passive web video do not capture control-relevant structure for physical manipulation, then the shared latent space that the causal DiT and downstream policy depend on is misaligned from the start.

What would settle it

Train the semantic visual-action tokenizer on web video alone, then freeze it and train the downstream policy on real-robot tasks. If the resulting policy fails to generalize from 10-15 demonstrations or underperforms a model using a reconstruction-only VAE on real-world manipulation (not just simulation), the claim that self-supervised latent actions from passive video provide control-relevant supervision is not supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If self-supervised latent actions from passive web video genuinely capture control-relevant dynamics, then the bottleneck of robot data collection shifts: web-scale video becomes the primary training signal, and robot demonstrations are needed only for fine-tuning rather than for learning dynamics.
The causal-from-scratch pretraining route implies that bidirectional video generators pretrained on web data may be the wrong starting point for control, and that the field's common practice of adapting off-the-shelf generators carries a structural tax that native pretraining avoids.
The Foresight Reasoning scheme, which predicts future latents in parallel with action execution and re-grounds on real observations, suggests that the serial bottleneck between model inference and robot execution can be broken without losing closed-loop feedback, potentially changing how real-time robot control systems are architected.
The human-robot co-training approach, which maps hand poses into a shared action space with robot grippers, implies that egocentric human video can directly improve robot policies during pretraining rather than only during adaptation, if the action-space mismatch is handled at the representation level.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The semantic tokenizer's latent actions are learned from passive video where no physical embodiment is present; whether these abstract transition variables capture the same control structure that a physical gripper's actions impose on the world is an empirical question the paper does not fully resolve, since the tokenizer ablation (Table 2) is conducted only in simulation.
If the native pretraining route is correct, then scaling the web-video corpus further should monotonically improve downstream robot performance without additional robot data—a testable prediction that the paper's scaling experiments do not yet push to the regime where this would be the dominant factor.
The Foresight Reasoning re-grounding step assumes the learned forward dynamics can correct imagined latents fast enough to prevent drift; the paper does not report how prediction error accumulates over long horizons without real observations, which would determine the scheme's limits in tasks with infrequent visual feedback.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents LingBot-VA 2.0, a video-action foundation model for robot manipulation that is pretrained from scratch rather than adapted from generic video generators. The architecture comprises four components: (1) a semantic visual-action tokenizer that aligns visual latents with a frozen foundation model and extracts latent actions via self-supervised inverse/forward dynamics on unlabeled video; (2) a causal diffusion transformer (DiT) with a sparse Mixture-of-Experts (MoE) video stream, trained natively in a causal regime; (3) multi-chunk prediction (MCP), in-context learning (ICL), and human-robot co-training as auxiliary objectives; and (4) Foresight Reasoning, an asynchronous inference scheme that predicts future latents in parallel with action execution and re-grounds on real observations. The system is evaluated on RoboTwin simulation and four real-world manipulation tasks, reporting improvements over π0.5 and the prior LingBot-VA, with a peak asynchronous execution frequency of 225 Hz.

Significance. The paper addresses a well-motivated problem: the mismatch between bidirectional video generators (designed for content creation) and the causal, closed-loop demands of robot control. The native pretraining route—building a semantic tokenizer and causal DiT from scratch rather than retrofitting an off-the-shelf backbone—is a coherent response to this mismatch. The engineering scope is substantial: a 15.3B-parameter model with sparse MoE, consistency distillation, FP8 TensorRT compilation, and paged KV-cache attention optimization, all contributing to a measurable 6.5× end-to-end speedup (Table 3). The Foresight Reasoning scheme (Sec. 2.3.7), which overlaps prediction with execution while re-grounding on real observations, is a concrete and falsifiable contribution to closed-loop control latency. The in-context learning demonstration (Fig. 9) and the human-robot co-training recipe (Sec. 2.3.5) add practical value. However, the central empirical claim—that native pretraining outperforms adapting generic video generators—rests on evaluation evidence that is not yet sufficient in rigor or scale, as detailed below.

major comments (3)

§4.2.1, Figure 8: The real-world evaluation reports success rates and task progress for four tasks but provides no number of evaluation rollouts, no confidence intervals, and no statistical significance tests. Several reported differences between VA 2.0 and VA 1.0 are small (e.g., Pen Collection success rate: 77% vs. 75%; Plate Handover progress: 90% vs. 89%). Without variance estimates or episode counts, it is impossible to determine whether these differences reflect genuine improvement or sampling noise. This is load-bearing because the paper's central claim is that native pretraining outperforms the prior adapted approach. At minimum, the number of rollouts per task and bootstrap or binomial confidence intervals should be reported for all conditions in Figure 8.
Table 1: The simulation improvement over LingBot-VA (the most direct comparison for the native-vs-adapted thesis) is 93.6% vs. 92.2% average, a 1.4 percentage-point difference. The paper does not report the number of evaluation episodes or variance for RoboTwin. A 1.4 pp difference on a benchmark with 50 tasks could be within noise. The paper should either (a) report per-task variance and episode counts to establish that this difference is statistically meaningful, or (b) acknowledge that the simulation gain over the prior generation is marginal and reframe the contribution accordingly. The claim of 'substantial margin' improvement (end of Sec. 1) is not supported by this table alone.
§4.4, Table 2: The tokenizer ablation—the most direct test of the semantic visual-action tokenizer, which is the foundation of the entire architecture—is conducted only in simulation (RoboTwin) and only on a 1.3B model, not the 15.3B deployed system. The MCP ablation (Fig. 10) uses a 5B model. The few-shot and real-world generalization claims (Sec. 4.2) are made for the full-scale system in the real-world domain, but no ablation bridges the gap between the ablated small-model/simulation setting and the full-scale/real-world setting. The paper should either run the tokenizer ablation at or near the deployed scale, or on at least one real-world task, or explicitly discuss the risk that the tokenizer's benefit does not transfer to the full-scale real-world setting.

minor comments (7)

§2.2.2, Eq. (13): The transport map K_t is described as moving information across spatial latent tokens, but its dimensionality and parameterization are not specified. Is K_t a full attention matrix, a per-token linear map, or something else? A brief clarification would help reproducibility.
§2.3.2, Eq. (19): The load-balancing bias update uses a sign function on (c_i - c̄), which zeroes out the mean-centered component. The choice of η_lb and its interaction with training stability is not discussed. A sentence on the range of η_lb used and whether it was tuned would help.
§2.3.7, Eq. (29): The notation v^vid_θ is used both for the video expert's velocity field (Eq. 6) and for the integrated latent obtained from it (Eq. 8, with the noted abuse of notation). In Eq. (29), it is unclear whether FDM_θ denotes the velocity field or the integrated latent. The authors should clarify which quantity is meant, as the reader may confuse the two.
§4.1.1: The model has approximately 15.3B trained parameters with 2.5B active per token. The ratio of MCP heads (1.7B) to the total is notable. It would help to state whether MCP heads are included in the 15.3B count and whether they are discarded at inference (stated in Sec. 2.3.3 but not cross-referenced in the parameter table).
Figure 8: The y-axis labels ('Success Rate' and 'Progress Rate') appear twice with different ranges, and the legend is small. Consider consolidating into a single panel per metric with clearer labeling.
§2.3.5: The hand-to-gripper mapping φ is described qualitatively but the quantile normalization per dataset is not formalized. A brief equation or reference to the normalization procedure would improve reproducibility.
The paper builds on several closely related prior works by the same group (LingBot-VA [50], RepWAM [98], Next Forcing [111], Zero-WAM [60], LingBot-Video [70]). While each is cited, the paper would benefit from a clearer statement of which components are novel vs. inherited, to help readers assess the incremental contribution.

Circularity Check

0 steps flagged

Multiple self-citations for methodological components, but none are load-bearing for the central claim in a circular way; the thesis is tested through experiments against external baselines.

full rationale

The paper builds on several prior works by overlapping authors: the semantic visual-action tokenizer follows RepWAM [98] (Sec. 2.2), multi-chunk prediction follows Next Forcing [111] (Sec. 2.3.3), and in-context learning follows Zero-WAM [60] (Sec. 2.3.4). However, these are methodological adoptions, not circular derivations. Each component is described in sufficient technical detail within this paper (with explicit equations: Eq. 9–14 for the tokenizer, Eq. 21–23 for MCP, Eq. 24 for ICL) and is independently ablated (Table 2 for the tokenizer, Figure 10 for MCP). The central thesis — that native from-scratch causal video-action pretraining outperforms adapting generic bidirectional video generators — is tested through direct experimental comparison against external baselines (π0.5 [77]) and the prior generation (LingBot-VA [50]) in both simulation (Table 1) and real-world deployment (Figure 8). No 'prediction' or 'result' reduces by construction to its inputs. The latent action tokenizer (Eq. 12–14) defines latent actions as compact transition variables learned via self-supervised forward/backward consistency — this is a standard autoencoder-style objective, not a self-definitional loop. The Foresight Reasoning scheme (Eq. 29–30) trains the video expert as a forward-dynamics predictor with an explicit loss; it does not define the prediction as the input. The self-citations are normal methodological lineage, not a chain where the central claim is justified only by citing the authors' own unverified prior work. The derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 3 invented entities

The paper introduces several design choices as free parameters without always specifying values, and relies on domain assumptions about latent action transfer and causal pretraining that are not fully ablated. The invented entities (tokenizer, Foresight Reasoning, latent actions) have partial independent evidence through simulation ablations but limited isolation on real-world tasks.

free parameters (7)

λ_align = Not specified
Controls strength of semantic alignment loss in visual tokenizer (Eq. 11); value not stated in paper.
λ_act = Not specified
Weight of action loss in video-action objective L_VA = L_vid + λ_act * L_act (Sec. 2.1.2); value not stated.
MCP loss weights (w1, w2, w3) = (0.5, 0.2, 0.1)
Weights for multi-chunk prediction horizons (Eq. 23), adopted from Next Forcing [111].
Task mixing distribution π(τ) = Coarse-to-fine schedule
Controls relative weight of T2I, T2V, TI2VA, ICL, HCT tasks during training (Eq. 28); schedule described qualitatively but not parameterized.
MoE routing: Ne, k, γ = Ne=128, k=8, γ not specified
Number of experts, top-k routing, and scaling factor for sparse MoE (Sec. 2.3.2).
Context noise augmentation probability = 0.5
Probability of replacing clean history frames with noised versions during training (Sec. 4.1.1).
Timestep shifts = image=2, web/video=2, robot=5, MCP=10, action=1
Modality-specific timestep shift parameters for flow-matching training (Sec. 4.1.2).

axioms (4)

domain assumption Latent actions extracted from passive video via inverse/forward dynamics capture control-relevant information transferable to robot manipulation.
Foundational to the entire approach (Sec. 2.2.2); the tokenizer learns latent actions from unlabeled video and the downstream policy operates on these latents. Not independently verified on real-world tasks.
domain assumption Causal pretraining from scratch avoids catastrophic forgetting better than bidirectional-to-causal finetuning.
Stated in Sec. 1 and Sec. 2.3.6 as motivation for from-scratch training. No direct ablation comparing from-scratch causal vs. bidirectional-pretrained-then-causal-finetuned is provided.
domain assumption The frozen Perception Encoder [10] provides semantic features suitable for aligning robot control representations.
Used as teacher model G in Eq. 10 for semantic alignment of the visual tokenizer.
domain assumption Human hand motion retargeted to gripper apertures provides useful action supervision for robot policies.
Underlies the human-robot co-training of Sec. 2.3.5; the functional gripper opening map φ is assumed to transfer meaningful grasp timing.

invented entities (3)

Semantic visual-action tokenizer independent evidence
purpose: Jointly encodes visual states and latent actions in a shared semantic latent space aligned to a foundation model
Ablated against reconstruction VAE in Table 2 on RoboTwin simulation, showing improvement. Not ablated on real-world tasks.
Foresight Reasoning no independent evidence
purpose: Asynchronous inference scheme that predicts future latents in parallel with execution and re-grounds on real observations
Described in Sec. 2.3.7 with forward-dynamics grounding loss (Eq. 30). No ablation isolating Foresight Reasoning from the rest of the system is provided; the 225 Hz claim includes all acceleration techniques (Table 3) but does not separate the contribution of asynchronous re-grounding.
Latent action token (a_t ≡ ℓ_t) independent evidence
purpose: Compact transition variable between visual latents serving as action representation
Learned via self-supervised inverse/forward dynamics (Eq. 12-14) with bottleneck. The approach follows RepWAM [98] and is validated indirectly through downstream task performance.

pith-pipeline@v1.1.0-glm · 34301 in / 3489 out tokens · 599421 ms · 2026-07-10T03:52:24.423418+00:00 · methodology

0 comments

read the original abstract

The advent of video-action models offers a promising path for robot control. Nevertheless, we argue that repurposing video generative models designed for digital content creation is inherently inadequate for physical environments. To bridge this gap, we present LingBot-VA 2.0, a video-action foundation model built from the ground up for embodiment. Four core design principles showcase its evolution from LingBot-VA. (1) Departing from traditional reconstruction-focused VAEs, we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actions, improving instruction following and action precision in subsequent policy learning. (2) Given the strictly causal nature of temporal dynamics, we adopt a causal pretraining paradigm, training from scratch to circumvent the catastrophic forgetting that frequently occurs when adapting bidirectional architectures. (3) To meet the demands of high-frequency inference, our model employs a sparse MoE backbone, expanding model capacity without compromising efficiency. (4) Real-time closed-loop control is realized through an enhanced asynchronous inference scheme, which predicts future latents in parallel with action execution while re-grounding each rollout on the latest observation via learned forward dynamics. Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.

Figures

Figures reproduced from arXiv: 2607.08639 by Chaojian Li, Fangyi Xu, Gangwei Xu, Guanxing Lu, Jiahao Shao, Jiaming Zhou, Jiapeng Zhu, Jiaqi Liao, Junke Wang, Ka Leong Cheng, Lin Li, Luyao Zhang, Nan Xue, Qihang Zhang, Ruilin Wang, Shuailei Ma, Shuaiting Li, Shuai Yang, Weixuan Tang, Xing Zhu, Xinyang Wang, Yiming Luo, Yinghao Xu, Yishu Shen, Yongkun Wen, Yudong Jin, Yujie Zhao, Yujun Shen, Zifan Shi.