Pith. sign in

REVIEW 3 major objections 5 minor 56 references

RxBrain claims embodied plans are best represented as a single interleaved sequence of text and imagined visual states, and shows a 6.2B model can learn and act on that representation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-02 03:12 UTC pith:IVXPFY5O

load-bearing objection Substantial system paper with a real idea, but the central evaluation is self-referential: the same MLLM that generates training labels also generates the benchmark ground truth, so the headline planning scores are unverified. the 3 major comments →

arxiv 2607.14187 v1 pith:IVXPFY5O submitted 2026-07-15 cs.AI cs.RO

RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination

classification cs.AI cs.RO
keywords embodied cognitionjoint language-visual planningworld state predictionMixture-of-Transformersflow matchingvision-language-actionsubgoal imaginationautomatic video annotation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RxBrain sets out to prove that embodied planning should be a coupled process: instead of a vision-language model producing text plans and a separate world model predicting images, one model should emit both in a single planning sequence, where language carries the task structure and visual imagination carries the physical goal states. To train for this, the paper builds an automatic pipeline that turns 50,177 hours of robot, simulation, and egocentric human video into 21.5M verified step-level text-visual supervision samples. It introduces RxBrain-Bench to test whether models can actually plan in this interleaved way. Experiments indicate that a 6.2B model can do embodied question answering, future-frame prediction, and joint subgoal planning without losing general image generation, and that the same model can be extended to real robotic control with no large-scale action pretraining. A sympathetic reader would take the central claim as: coupling language and visual imagination in one generative trajectory is a viable path to embodied cognition.

Core claim

The central claim is that an embodied plan is a trajectory in which each step carries both a textual component (action name, description, constraint, decision) and a visual component (the world state that step is meant to produce), and that a single model can generate this trajectory autoregressively: text tokens via next-token prediction, image latents via flow matching, with generated images re-encoded and fed back into the context. The paper argues that neither text-only reasoning nor video-only prediction captures an executable intention, and that the coupling itself—what it calls joint language-visual reasoning and imagination—is the capability to build. Evidence comes from the model's

What carries the argument

The carrier of the argument is a modality-aware Mixture-of-Transformers (MoT) backbone with three specialized feed-forward experts—text, visual understanding, and visual generation—that share attention projections so perception and imagination interact inside the same vision tower. Text generation uses causal attention; visual tokens use intra-image bidirectional attention; world-state prediction is done by flow matching in a frozen VAE latent space; interleaved planning is implemented by a learned <Image> transition token that pauses text decoding, synthesizes the goal frame, re-encodes it, and resumes decoding. On the data side, the load-bearing machinery is the automatic annotation pipeli

Load-bearing premise

The load-bearing premise is that the single multimodal LLM driving the annotation pipeline detects genuine visual state changes correctly, because if its segment boundaries, step names, and text-visual consistency judgments are systematically wrong, the model is trained on labels that encode the annotator's imagination rather than physical states.

What would settle it

Take a random sample of the 21.5M retained segments and have independent human annotators judge, from the start/end frames and the video, whether a visible state change occurred and whether the step description matches it. If human agreement with the pipeline's multimodal LLM is no better than chance, the joint supervision is circular and the model's planning ability cannot be attributed to genuine text-visual grounding.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single unified model can hold embodied understanding, world-state prediction, and joint subgoal planning, so separate VLM and world-model stacks are not necessary for these capabilities.
  • Embodied video without action labels can be converted into joint text-visual planning supervision at scale (21.5M trainable segments from 50,177 hours at a 75.18% verified pass rate), lowering the annotation bottleneck for embodied cognition training.
  • Real-robot policies can be initialized from an embodied cognition model and fine-tuned with the same joint-planning labels, achieving 97%, 95%, and 68% success on three tasks without large-scale action pretraining.
  • Free-running interleaved planning degrades with horizon (0.69 at two steps to 0.55 at eight), so error accumulation in visual imagination is the current practical limit of the approach.
  • Language reasoning in generated plans outpaces visual goal-image correctness (0.78 vs 0.52 on the JointPlan criteria), identifying visual imagination as the bottleneck to address next.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the coupling claim is right, the next test is whether the model's imagined goal frames can serve as a self-checking loop: a plan could be revised when the generated image does not match the text, turning the learned transition token into an internal simulator.
  • The gated reuse of the generation expert for action generation suggests a transfer principle—world-model features may substitute for large action datasets—that would be worth testing directly by ablating the gate in a controlled study.
  • The paper's benchmark protocol (autoregressive free-running evaluation of interleaved text-image planning) could become a standard evaluation for embodied foundation models, since it measures closed-loop behavior rather than isolated understanding or generation.
  • A cautious extension: if the automatic annotation pipeline's verifier is itself the annotation source, future work should test whether human-validated subsets give the same planning gains, or whether the model has partly learned the annotator's imagination habits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Hy-Embodied-RxBrain, a 6.2B unified multimodal model built on a modality-aware Mixture-of-Transformers architecture that supports language, image, and video understanding and generation. The central idea is to represent embodied plans as an interleaved sequence of textual reasoning and imagined visual goal states, trained with an automatic pipeline that converts embodied videos into joint text-visual planning supervision. The pipeline decomposes videos into planning steps, refines start/end anchors, quality-verifies each segment, and organizes the results into L0–L3 tasks. The paper also introduces RxBrain-Bench with EVQA, WorldPred, and JointPlan tracks, reports results on 19 standard benchmarks, and extends the model to continuous robot action generation with three real-world manipulation tasks. The claimed contribution is a single model that jointly performs embodied reasoning, world-state prediction, and subgoal planning, while preserving general multimodal abilities.

Significance. If the joint-planning evaluation is trustworthy, this is a meaningful step toward unified embodied cognition models: the architecture is clearly specified, the loss formulations (Eqs. 2–11) are standard and internally consistent, and the model is evaluated across a broad set of public understanding/generation benchmarks as well as held-out subsets of the new benchmark. Strengths include the released model links, the detailed baseline-composition protocol in Appendix F, the balanced held-out evaluation subsets, and the real-robot comparisons on 100 trials per task. The central risk is that the key joint-planning evidence is potentially circular: the same MLLM (Phi) that generates the training supervision also defines the benchmark ground truth, and no human validation or external reference is reported. This concern is load-bearing for the paper's main claim, not a stylistic issue.

major comments (3)
  1. [§3.1, §3.2, §5.2, App. C.1] The data-construction pipeline and the key benchmark share the same MLLM. Section 3.1 and C.1 state that 'a single multimodal large language model, written Φ, makes every semantic decision': it proposes candidate steps, refines boundaries, corrects descriptions, and verifies text–visual consistency. RxBrain-Bench-JointPlan and WorldPred then use ground-truth frames and texts derived from this same pipeline. Thus the headline Splan = 0.68 (vs. 0.43–0.52 for baselines) may largely measure how well RxBrain imitates Phi's annotation distribution rather than whether the plans are grounded in physical world states. The paper reports no human-annotation validation of segment boundaries, step descriptions, or state-transition judgments, and Appendix B explicitly notes that several sources have no verified external reference. This is an unverified key condition for the central claim. Please add a
  2. [§5.3, Table 4] The Splan rubric compounds the circularity. Goal-Image Correctness (GI, weight 25%) is rated by the GPT-5.5 judge against ground-truth goal images produced by the same Phi-based pipeline, and Image Similarity (IS, weight 10%) is DINO cosine to those same ground-truth frames. Together GI+IS are 35% of Splan. Using a different MLLM as judge does not break the circularity because the reference labels themselves are pipeline-derived. The per-criterion breakdown in Table 4 is informative, but the 'GoalImg' column (0.52) cannot be interpreted as physical goal-state correctness without external ground truth. Please provide at least a human-verified subset for this track and report the correlation between the MLLM judge and human judgments.
  3. [§7.2] The real-robot action extension is presented as evidence that the joint planning representation transfers to control, but the setup does not isolate the contribution of that representation. Section 7.2 says baselines and the model are trained on the same real-world robot data and that the joint planning pipeline is used to annotate additional subgoal labels incorporated into training. The 87% average success rate vs. 68%/82% for π0/π0.5 is encouraging, yet there is no ablation that removes the joint-planning labels or the interleaved supervision. The claim of 'promising real-robot performance without large-scale action-data pretraining' is therefore not attributable to the paper's core mechanism. Please include an ablation and report confidence intervals or per-task statistical significance for the three tasks.
minor comments (5)
  1. [App. C.1] The phrase 'written Φ' should be 'denoted Φ'; the same typo appears in the stress-test summary but should be corrected in the manuscript.
  2. [Tables 3–4] The captions state that n differs across methods due to judge content-filtering. Please report the exact n for each method and explain how filtering could differentially affect methods; as written, the comparison is not guaranteed to be on identical records.
  3. [Fig. 10] Add confidence intervals or trial counts per bar. The text mentions 100 trials per policy per task, but the figure would benefit from error bars and a note clarifying whether values are percentages.
  4. [§2.2] Minor typo: 'within a autoregressive process' should be 'within an autoregressive process.'
  5. [§9] The Limitations paragraph says 'shifting ground-truth VAE latents during training,' but Eq. (9) describes reconstruction through the VAE encoder/decoder, not a shift. Rephrase to match the formulation.

Circularity Check

1 steps flagged

RxBrain-Bench ground truth is produced by the same unvalidated MLLM that generates training labels, so the headline joint-planning scores partly measure imitation of Phi rather than physical grounding.

specific steps
  1. fitted input called prediction [Section 3.1/3.2/3.3 (Eq. 1, L1 target) -> Section 5.3 and Appendix F (Splan/JointPlan)]
    "Throughout, a single multimodal large language model, written Φ, makes every semantic decision, so the module is one annotator queried with different prompts rather than a stack of specialized detectors."

    Φ proposes candidate steps, refines start/end anchors, corrects descriptions, and verifies text-visual consistency (Sections 3.1-3.2). These verified segments are then used both as L1-L3 training targets (Section 3.3: '[C1:m, g_seg] -> [T_1, I_end_1, ...]') and as the ground-truth goal images/subtask texts for JointPlan scoring (Section 5.3; Appendix F: 'per step the GT subtask text, the model subtask text, the GT goal image, and the model-generated image'). Thus the headline Splan = 0.68 is evaluated against labels generated by the same automatic pipeline that created the training supervision, and no human-annotation or external physical reference is reported; Appendix B explicitly flags CyberOrigin, EgoVerse, Xperience, and AI Data raw as having no verified external reference. The benchm

full rationale

The central novel claim—joint language-visual planning grounded in physical states—rests on two pillars that share the same origin. The data-construction pipeline uses a single MLLM (Φ) for every semantic decision, producing the very text-visual pairs used for training; RxBrain-Bench then scores the joint-planning output against ground-truth subtask texts and goal images drawn from that same pipeline. This creates a partial circularity: the model is trained to imitate Φ's annotations and is evaluated on how well it reproduces Φ-derived labels, with no human validation of segment boundaries, step descriptions, or state-transition judgments. The paper is transparent about the pipeline and even notes that some sources have no verified external reference, which supports the concern rather than resolving it. However, the circularity is not complete. The model is also evaluated on independent external benchmarks (GenEval, CV-Bench, EmbSpatial, DA-2k, Share-Robot-Trajectory, 3DRSBench, etc.) and on real-robot success rates against π0/π0.5, which provide outside evidence for general understanding, generation, and action competence. Those results do not depend on Φ's annotations and therefore prevent the entire derivation from reducing to its inputs. The self-referential benchmark affects the specific claim of joint embodied planning but not the whole paper, so a score of 4 is appropriate rather than a higher score.

Axiom & Free-Parameter Ledger

7 free parameters · 7 axioms · 0 invented entities

No fundamentally new physical or ontological entities are postulated. The architectural components (MoT branches, flow-matching heads, gated expert fusion) are engineering constructs. The main non-verifiable inputs are the MLLM-generated supervision and the author-defined benchmark, both captured as domain assumptions above. The free parameters are hand-chosen training and evaluation constants, not quantities fitted to external ground truth.

free parameters (7)
  • Keyframe sampler constants (plo, phi, beta, delta_min, gmax, Amax, n) = 0.8, 0.9, 0.2, 0.2s, 0.5s, 50, 30
    Appendix C.1 states these are empirical constants applied uniformly to every source rather than values from an optimality criterion.
  • Stage 1 and Stage 2 loss weights (lambda_CE, lambda_FM) = Stage1: 0.25/1.0; Stage2: 1.0/1.0
    Appendix E Table 7; hand-chosen to balance understanding and generation, and they directly affect the trained behavior.
  • Stage data-mixture ratios = Stage1 6:4; Stage2 100:12:150:70:90
    Appendix E Tables 6 and 8; hand-specified bucket ratios determine per-step shares of text-to-image, understanding, world-model, and planning data.
  • Quality verification retention policy = semantic score >= 4; near-duplicate filter 99.5%; retained 75.18%
    Section 3.2; the threshold determines which video segments become the joint-planning supervision and therefore shapes the model's learned planning behavior.
  • Action flow-time distribution and ODE steps = s ~ Beta(1.5,1); K=2 at deployment
    Section 7.1 and Appendix G; per-chunk flow time biased to high noise and 2-step inference are choices, not fitted to data.
  • Benchmark rubric weights = Sgen: 0.15/0.30/0.20/0.20/0.15; Splan: 0.10/0.25/0.25/0.20/0.10/0.10
    Section 5.3; hand-chosen weights define the headline scores used to compare against baselines.
  • Flow-matching timestep shift, CFG text-condition dropout, generation resolution = shift 1.0, dropout 0.1, <=256px
    Appendix E; training choices that affect generation quality and text-image alignment.
axioms (7)
  • domain assumption Atomic planning steps are identifiable from visible object-state changes and can be described in language.
    Section 3.1 and C.1 define the annotation target; if segments do not correspond to real state changes, all downstream text-visual supervision is invalid.
  • domain assumption A single MLLM Phi can reliably propose, refine, and verify video segments without human correction.
    Sections 3.1-3.2; all semantic decisions are made by one annotator, and the paper reports no inter-annotator agreement or human validation of the pipeline.
  • domain assumption Interleaved text-plus-image generation is an adequate representation of embodied planning.
    Introduction and Section 2.2; the central design choice that language structure plus visual goals is sufficient for task reasoning.
  • standard math Flow-matching on VAE latents with the linear path and logit-normal timestep is a valid generative objective.
    Equations 3-4 and Section 4.1; standard flow-matching formulation.
  • domain assumption HY-Embodied-0.5 provides a reusable vision encoder and base multimodal capability.
    Section 2.1 and Appendix E; initialization from a same-team paper (Team et al., 2026a) is taken as valid, with no external reproduction.
  • ad hoc to paper GPT-5.5 as MLLM judge plus DINO similarity measures planning and world-prediction quality.
    Section 5.3 and Appendix F; the claimed capabilities are scored with a rubric defined and judged by a general-purpose VLM, with no external ground-truth benchmark.
  • domain assumption Real-robot, UMI, simulation, and egocentric video are representative of embodied tasks.
    Section 3 and Appendix B; the corpus mixes public and internal sources, and some sources have no verified external reference.

pith-pipeline@v1.3.0-alltime-deepseek · 44007 in / 13910 out tokens · 138909 ms · 2026-08-02T03:12:37.017269+00:00 · methodology

0 comments
read the original abstract

Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved. We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination. Unlike vision-language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents embodied plans in a single planning sequence where language and visual imagination play complementary roles. Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts embodied videos into joint text-visual planning supervision by decomposing videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains embodied understanding and generation abilities, and produces plans with coupled textual reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous robot action generation, where it shows promising real-robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for embodied cognition.

Figures

Figures reproduced from arXiv: 2607.14187 by Bohan Ma, Han Hu, Haotian Liang, Kaixuan Wang, Ling Chen, Lingzhu Xiang, Minghui Wang, Mingkang Chen, Ping Luo, Shirong Zeng, Tianshuo Yang, Wanjia He, Weijie Zhou, Xiangli Shi, Xiaomeng Zhu, Xingyu Zhou, Xin Yang, Xiran Huang, Xuantang Xiong, Yajuan Zhu, Yao Mu, Yuchen Si, Yuchun Guo, Yueyu Long, Yufei Huang, Yunxuan Mao, Zhengyou Zhang, Zhiheng Liu, Zhiqing Liu, Zisheng Lu.

Figure 1
Figure 1. Figure 1: Capability overview of RxBrain. RxBrain supports multiple embodied tasks within a unified model, including embodied visual question answering, multi-frame visual generation, and joint textual reasoning with visual goal imagination. Given human instructions and visual observations, the model can reason about the current scene, imagine future or goal states, and generate joint language and visual planning st… view at source ↗
Figure 2
Figure 2. Figure 2: Architecture overview of RxBrain. The model adopts a modality-aware MoT architecture to jointly process visual input tokens, text tokens, and generated vision tokens. Visual tokens are handled by a shared vision backbone with full attention, where understanding and generation tokens share attention projections but are routed to separate FFN experts. Text tokens are processed by the Language Transformer wit… view at source ↗
Figure 3
Figure 3. Figure 3: Hybrid attention patterns in RxBrain. RxBrain adopts hybrid attention masks for unified multimodal understanding and generation. For text generation (a), input images are encoded with bidirectional visual attention, while text tokens follow causal attention for autoregressive decoding. For multi frames generation (b), input images and text prompts follow the same pattern, while generated frames use bidirec… view at source ↗
Figure 4
Figure 4. Figure 4: Data construction overview and statistics. This figure summarizes the verified post-training corpus by scale, source composition, and scene coverage. Panel A reports total video hours across four data categories: Real-Robot Data, UMI Data, Simulation Data, and Egocentric Human Data. Panel B shows the source composition, with source categories in the inner ring and individual datasets in the outer ring. Pan… view at source ↗
Figure 5
Figure 5. Figure 5: Data construction pipeline. Raw embodied task videos are converted into joint text-visual planning samples through three stages: Temporal Segment Annotation, Quality Verification, and Segment Structuring. The final samples are organized into L0–L3 tasks, covering intra-step world state prediction, step-level planning, subgoal planning, and final-state imagination. detailed description, and metadata. Qualit… view at source ↗
Figure 6
Figure 6. Figure 6: Hierarchical data construction pipeline. Verified planning segments are organized into four levels of supervision. L0 learns the visual state change inside a single planning step, while L1 links consecutive steps for step-level joint planning. L2 groups neighboring steps into subgoals, and L3 connects the initial observation with the final task state. to learn intra-step visual state changes, while L1 link… view at source ↗
Figure 7
Figure 7. Figure 7: RxBrain-Bench-WorldPred (UMI). Four-frame future prediction for handheld manipulation. Rows compare the ground truth (GT), RxBrain (HY-Unified), the general-purpose I2V model Wan2.2-TI2V, and the embodied world model Cosmos3. RxBrain preserves the scene layout and reproduces the intended grasping behavior [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: RxBrain-Bench-JointPlan (BridgeV2), task “move to the right edge of the blue cloth, grasp it, pull it leftward, and release it.” rows are models, columns are planning steps, and each cell pairs the imagined goal frame with the model’s generated sub-step text. RxBrain reproduces the grasp–pull–release sequence and the cloth’s changing shape in step with the ground-truth plan (GT goal). 7 Extending RxBrain t… view at source ↗
Figure 9
Figure 9. Figure 9: Action model architecture. A modality-specialized Action MoT branch uses dedicated attention and feed-forward parameters while sharing global self-attention with language, vision, and state tokens. A zero-initialized channel-wise gate transfers features from the pretrained generation expert to the action expert, whose outputs are decoded into action chunks. Gated Expert Fusion. In RxBrain, the generation b… view at source ↗
Figure 10
Figure 10. Figure 10: Real-world robot evaluation on three manipulation tasks. We compare our method with π0 and π0.5 on three manipulation tasks. Our method achieves the highest success rate on average. Experimental Setup and Tasks. We further evaluate our method on two real-world robot embodiments: the DOBOT X-Trainer and an ARX dual-arm platform. The first two tasks are conducted on the DOBOT X-Trainer, while the third task… view at source ↗
Figure 11
Figure 11. Figure 11: Per-scene taxonomy sunburst charts (14 scene categories). Each panel is one self-contained chart for a single scene category, flowing left-to-right and top-to-bottom across three columns (and across pages as needed). The colored center disc names the scene category; the ring around it is divided into wedges, one per action subtype observed in that scene, with wedge angle proportional to the subtype’s shar… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 45 linked inside Pith

  1. [1]

    Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800,

    Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800,

  2. [4]

    Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966,

  3. [6]

    Revisiting feature prediction for learning visual representations from video.arXiv:2404.08471,

    Adrien Bardes, Quentin Garrido, Jean Ponce, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nico- las Ballas. Revisiting feature prediction for learning visual representations from video.arXiv:2404.08471,

  4. [9]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127,

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127,

  5. [10]

    AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669,

    Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669,

  6. [11]

    Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, et al

    Hugging Face dataset, approximately 10,000 hours of egocentric factory video. Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, et al. Internvla-a1: Unifying understanding, generation and action for robotic manipulation.arXiv preprint arXiv:2601.02456,

  7. [13]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

  8. [14]

    Internvla-m1: A spatially guided vision-language-action framework for generalist robot policy.arXiv preprint arXiv:2510.13778,

    Xinyi Chen, Yilun Chen, Yanwei Fu, Ning Gao, Jiaya Jia, Weiyang Jin, Hao Li, Yao Mu, Jiangmiao Pang, Yu Qiao, et al. Internvla-m1: A spatially guided vision-language-action framework for generalist robot policy.arXiv preprint arXiv:2510.13778,

  9. [15]

    Rynnec: Bringing mllms into embodied world.arXiv preprint arXiv:2508.14160,

    Ronghao Dang, Yuqian Yuan, Yunxuan Mao, Kehan Li, Jiangpin Liu, Zhikai Wang, Xin Li, Fan Wang, and Deli Zhao. Rynnec: Bringing mllms into embodied world.arXiv preprint arXiv:2508.14160,

  10. [16]

    Rynnbrain: Open embodied foundation models.arXiv preprint arXiv:2602.14979,

    Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, et al. Rynnbrain: Open embodied foundation models.arXiv preprint arXiv:2602.14979,

  11. [17]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

  12. [18]

    Rethinking video generation model for the embodied world.arXiv preprint arXiv:2601.15282,

    Yufan Deng, Zilin Pan, Hongyu Zhang, Xiaojie Li, Ruoqing Hu, Yufei Ding, Yiming Zou, Yan Zeng, and Daquan Zhou. Rethinking video generation model for the embodied world.arXiv preprint arXiv:2601.15282,

  13. [19]

    URL https: //arxiv.org/abs/2406.05756. 37 Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synt...

  14. [20]

    Molmoact2: Action reasoning models for real-world deployment

    Haoquan Fang, Jiafei Duan, Donovan Clay, Sam Wang, Shuo Liu, Weikai Huang, Xiang Fan, Wei-Chuan Tsai, Shirui Chen, Yi Ru Wang, et al. Molmoact2: Action reasoning models for real-world deployment. arXiv preprint arXiv:2605.02881,

  15. [21]

    Galaxea open-world dataset and G0 dual-system VLA model.arXiv preprint arXiv:2509.00576,

    Galaxea Team. Galaxea open-world dataset and G0 dual-system VLA model.arXiv preprint arXiv:2509.00576,

  16. [22]

    Dreamdojo: A generalist robot world model from large-scale human videos.arXiv preprint arXiv:2602.06949,

    Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, et al. Dreamdojo: A generalist robot world model from large-scale human videos.arXiv preprint arXiv:2602.06949,

  17. [23]

    Google DeepMind

    URLhttps://arxiv.org/abs/2310.11513. Google DeepMind. Genie 2: A large-scale foundation world model. https://deepmind.google/discov er/blog/genie-2-a-large-scale-foundation-world-model/,

  18. [24]

    Ctrl-world: A controllable generative world model for robot manipulation.arXiv preprint arXiv:2510.10125,

    Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-world: A controllable generative world model for robot manipulation.arXiv preprint arXiv:2510.10125,

  19. [25]

    Yoon, Mouli Sivapurapu, and Jian Zhang

    Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709,

  20. [26]

    Video prediction policy: A generalist robot policy with predictive visual representations.arXiv preprint arXiv:2412.14803,

    Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations.arXiv preprint arXiv:2412.14803,

  21. [27]

    URL https://arxiv.org/abs/ 2602.09849. Physical Intelligence, Bo Ai, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Greg Balke, Kevin Black, George Bokinsky, Shihao Cao, Thomas Charbonnier, Vedant Choudhary, Foster Collins, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Maitrayee Dhaka, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmai...

  22. [28]

    InternRobotics

    URLhttps://arxiv.org/abs/2604.15483. InternRobotics. InternData-A1. https://huggingface.co/datasets/InternRobotics/InternData-A1,

  23. [29]

    RoboBrain: A unified brain model for robotic manipulation from abstract to concrete.arXiv preprint arXiv:2502.21257,

    Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, Xinda Huang, Jiaming Wang, Shengjie Zhang, Zhongyuan Wang, et al. RoboBrain: A unified brain model for robotic manipulation from abstract to concrete.arXiv preprint arXiv:2502.21257,

  24. [30]

    π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164,

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164,

  25. [31]

    Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163,

    Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163,

  26. [32]

    Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125,

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125,

  27. [33]

    MolmoAct: Action reasoning models that can reason in space.arXiv preprint arXiv:2508.07917,

    Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, Winson Han, Wilbert Pumacay, Angelica Wu, Rose Hendrix, Karen Farley, Eli VanderBilt, Ali Farhadi, Dieter Fox, and Ranjay Krishna. MolmoAct: Action reasoning models that can reason in space.arXiv preprint arXiv:2508.07917,

  28. [34]

    Viewspatial-bench: Evaluating multi-perspective spatial localization in vision-language models.arXiv preprint arXiv:2505.21500,

    Dingming Li, Hongxing Li, Zixuan Wang, Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, et al. Viewspatial-bench: Evaluating multi-perspective spatial localization in vision-language models.arXiv preprint arXiv:2505.21500,

  29. [35]

    Causal world modeling for robot control.arXiv preprint arXiv:2601.21998,

    Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, Yujun Shen, and Yinghao Xu. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998,

  30. [36]

    Mm-act: Learn from multimodal parallel generation to act

    Haotian Liang, Xinyi Chen, Bin Wang, Mingkang Chen, Yitian Liu, Yuhao Zhang, Zanxin Chen, Tianshuo Yang, Yilun Chen, Jiangmiao Pang, et al. Mm-act: Learn from multimodal parallel generation to act. arXiv preprint arXiv:2512.00975,

  31. [37]

    Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996,

    39 Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996,

  32. [38]

    URL https://www.biorxiv.org/content/10.64898/2026.05.01.722168v1

    doi: 10.64898/2026.05.01.722168. URL https://www.biorxiv.org/content/10.64898/2026.05.01.722168v1. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems (NeurIPS),

  33. [39]

    3dsrbench: A comprehensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825,

    Wufei Ma, Haoyu Chen, Guofeng Zhang, Celso M de Melo, Alan Yuille, and Jieneng Chen. 3dsrbench: A comprehensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825,

  34. [40]

    GPT-4o system card.arXiv preprint arXiv:2410.21276,

    OpenAI. GPT-4o system card.arXiv preprint arXiv:2410.21276,

  35. [41]

    π0.5: A vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054,

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Chelsea Finn, Sergey Levine, et al. π0.5: A vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054,

  36. [42]

    Heqian Qiu, Zhaofeng Shi, Lanxiao Wang, Huiyu Xiong, Xiang Li, and Hongliang Li

    Hugging Face dataset, real-world human manipulation captured with an exoskeleton-glove system. Heqian Qiu, Zhaofeng Shi, Lanxiao Wang, Huiyu Xiong, Xiang Li, and Hongliang Li. EgoMe: A new dataset and challenge for following me via egocentric view in real world.arXiv preprint arXiv:2501.19061,

  37. [43]

    RoboCOIN: An open-sourced bimanual robotic data collection for integrated ma- nipulation.arXiv preprint arXiv:2511.17441,

    RoboCOIN Team. RoboCOIN: An open-sourced bimanual robotic data collection for integrated ma- nipulation.arXiv preprint arXiv:2511.17441,

  38. [44]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaugh- lin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

  39. [45]

    BAAI RoboBrain Team

    URLhttps://arxiv.org/abs/2601.14352. BAAI RoboBrain Team. RoboBrain 2.0 technical report.arXiv preprint,

  40. [46]

    Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020,

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020,

  41. [47]

    Hy-embodied-0.5: Embodied foundation models for real-world agents.arXiv preprint arXiv:2604.07430, 2026a

    HY Team, Xumin Yu, Zuyan Liu, Ziyi Wang, He Zhang, Yongming Rao, Fangfu Liu, Yani Zhang, Ruowen Zhao, Oran Wang, et al. Hy-embodied-0.5: Embodied foundation models for real-world agents.arXiv preprint arXiv:2604.07430, 2026a. Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, e...

  42. [48]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, T...

  43. [49]

    HO-Cap: A capture system and dataset for 3d reconstruction and pose tracking of hand-object interaction

    Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, and Yu Xiang. HO-Cap: A capture system and dataset for 3d reconstruction and pose tracking of hand-object interaction. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025a. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Ch...

  44. [50]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848,

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848,

  45. [51]

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    URL https://arxiv.org/abs/2601.18692. Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024a. Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything...

  46. [52]

    Gigaworld- policy: An efficient action-centered world–action model, 2026a

    Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jie Li, Jindi Lv, Jingyu Liu, Min Cao, Peng Li, Qiuping Deng, Wenjun Mei, Xiaofeng Wang, Xinze Chen, Xinyu Zhou, Yang Wang, Yifan Chang, Yifan Li, Yukun Zhou, Yun Ye, Zhichao Liu, and Zheng Zhu. Gigaworld- policy: An efficient action-centered world–action model, 2026a. URL h...

  47. [53]

    Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models.arXiv preprint arXiv:2606.17846,

    Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, Jiazhao Zhang, et al. Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models.arXiv preprint arXiv:2606.17846,

  48. [54]

    Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721,

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721,

  49. [55]

    Hy-embodied-0.5-vla: From vision-language-action models to a real-world robot learning stack.arXiv preprint arXiv:2606.14409,

    He Zhang, Lingzhu Xiang, Haitao Lin, Zeyu Huang, Minghui Wang, Dingyan Zhong, Yubo Dong, Yihao Wu, Yongming Rao, Dongsheng Zhang, Wanjia He, Ling Chen, Kai Huang, Jiahao Chen, Sichang Su, Xumin Yu, Ziyi Wang, Chengwei Zhu, Xiao Teng, Yuchun Guo, Yufeng Zhang, Yuandong Liu, Rui Wang, Zisheng Lu, Han Hu, and Zhengyou Zhang. Hy-embodied-0.5-vla: From vision-...

  50. [56]

    Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han

    URL https://arxiv.org/abs/2501 .18867. Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han. TASTE- Rob: Advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025a. Zhenyu Zhao, Hongyi Jing, Xiawei ...

  51. [2021]

    Sat: Spatial aptitude training for multimodal language models.arXiv preprint arXiv:2412.07755, 3,

    Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat: Spatial aptitude training for multimodal language models.arXiv preprint arXiv:2412.07755, 3,

  52. [2022]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

    Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, X...

  53. [2023]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,

  54. [2024]

    FLUX.1 kontext: Flow matching for in-context image generation and editing in latent space.arXiv preprint arXiv:2506.15742,

    Black Forest Labs. FLUX.1 kontext: Flow matching for in-context image generation and editing in latent space.arXiv preprint arXiv:2506.15742,

  55. [2025]

    Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558,

    Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558,

  56. [2026]

    Rynnvla-002: A unified vision-language-action and world model.arXiv preprint arXiv:2511.17502,

    Jun Cen, Siteng Huang, Yuqian Yuan, Kehan Li, Hangjie Yuan, Chaohui Yu, Yuming Jiang, Jiayan Guo, Xin Li, Hao Luo, Fan Wang, Fan Wang, and Deli Zhao. Rynnvla-002: A unified vision-language-action and world model.arXiv preprint arXiv:2511.17502,