REVIEW 3 major objections 5 minor 56 references
RxBrain claims embodied plans are best represented as a single interleaved sequence of text and imagined visual states, and shows a 6.2B model can learn and act on that representation.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 03:12 UTC pith:IVXPFY5O
load-bearing objection Substantial system paper with a real idea, but the central evaluation is self-referential: the same MLLM that generates training labels also generates the benchmark ground truth, so the headline planning scores are unverified. the 3 major comments →
RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that an embodied plan is a trajectory in which each step carries both a textual component (action name, description, constraint, decision) and a visual component (the world state that step is meant to produce), and that a single model can generate this trajectory autoregressively: text tokens via next-token prediction, image latents via flow matching, with generated images re-encoded and fed back into the context. The paper argues that neither text-only reasoning nor video-only prediction captures an executable intention, and that the coupling itself—what it calls joint language-visual reasoning and imagination—is the capability to build. Evidence comes from the model's
What carries the argument
The carrier of the argument is a modality-aware Mixture-of-Transformers (MoT) backbone with three specialized feed-forward experts—text, visual understanding, and visual generation—that share attention projections so perception and imagination interact inside the same vision tower. Text generation uses causal attention; visual tokens use intra-image bidirectional attention; world-state prediction is done by flow matching in a frozen VAE latent space; interleaved planning is implemented by a learned <Image> transition token that pauses text decoding, synthesizes the goal frame, re-encodes it, and resumes decoding. On the data side, the load-bearing machinery is the automatic annotation pipeli
Load-bearing premise
The load-bearing premise is that the single multimodal LLM driving the annotation pipeline detects genuine visual state changes correctly, because if its segment boundaries, step names, and text-visual consistency judgments are systematically wrong, the model is trained on labels that encode the annotator's imagination rather than physical states.
What would settle it
Take a random sample of the 21.5M retained segments and have independent human annotators judge, from the start/end frames and the video, whether a visible state change occurred and whether the step description matches it. If human agreement with the pipeline's multimodal LLM is no better than chance, the joint supervision is circular and the model's planning ability cannot be attributed to genuine text-visual grounding.
If this is right
- A single unified model can hold embodied understanding, world-state prediction, and joint subgoal planning, so separate VLM and world-model stacks are not necessary for these capabilities.
- Embodied video without action labels can be converted into joint text-visual planning supervision at scale (21.5M trainable segments from 50,177 hours at a 75.18% verified pass rate), lowering the annotation bottleneck for embodied cognition training.
- Real-robot policies can be initialized from an embodied cognition model and fine-tuned with the same joint-planning labels, achieving 97%, 95%, and 68% success on three tasks without large-scale action pretraining.
- Free-running interleaved planning degrades with horizon (0.69 at two steps to 0.55 at eight), so error accumulation in visual imagination is the current practical limit of the approach.
- Language reasoning in generated plans outpaces visual goal-image correctness (0.78 vs 0.52 on the JointPlan criteria), identifying visual imagination as the bottleneck to address next.
Where Pith is reading between the lines
- If the coupling claim is right, the next test is whether the model's imagined goal frames can serve as a self-checking loop: a plan could be revised when the generated image does not match the text, turning the learned transition token into an internal simulator.
- The gated reuse of the generation expert for action generation suggests a transfer principle—world-model features may substitute for large action datasets—that would be worth testing directly by ablating the gate in a controlled study.
- The paper's benchmark protocol (autoregressive free-running evaluation of interleaved text-image planning) could become a standard evaluation for embodied foundation models, since it measures closed-loop behavior rather than isolated understanding or generation.
- A cautious extension: if the automatic annotation pipeline's verifier is itself the annotation source, future work should test whether human-validated subsets give the same planning gains, or whether the model has partly learned the annotator's imagination habits.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hy-Embodied-RxBrain, a 6.2B unified multimodal model built on a modality-aware Mixture-of-Transformers architecture that supports language, image, and video understanding and generation. The central idea is to represent embodied plans as an interleaved sequence of textual reasoning and imagined visual goal states, trained with an automatic pipeline that converts embodied videos into joint text-visual planning supervision. The pipeline decomposes videos into planning steps, refines start/end anchors, quality-verifies each segment, and organizes the results into L0–L3 tasks. The paper also introduces RxBrain-Bench with EVQA, WorldPred, and JointPlan tracks, reports results on 19 standard benchmarks, and extends the model to continuous robot action generation with three real-world manipulation tasks. The claimed contribution is a single model that jointly performs embodied reasoning, world-state prediction, and subgoal planning, while preserving general multimodal abilities.
Significance. If the joint-planning evaluation is trustworthy, this is a meaningful step toward unified embodied cognition models: the architecture is clearly specified, the loss formulations (Eqs. 2–11) are standard and internally consistent, and the model is evaluated across a broad set of public understanding/generation benchmarks as well as held-out subsets of the new benchmark. Strengths include the released model links, the detailed baseline-composition protocol in Appendix F, the balanced held-out evaluation subsets, and the real-robot comparisons on 100 trials per task. The central risk is that the key joint-planning evidence is potentially circular: the same MLLM (Phi) that generates the training supervision also defines the benchmark ground truth, and no human validation or external reference is reported. This concern is load-bearing for the paper's main claim, not a stylistic issue.
major comments (3)
- [§3.1, §3.2, §5.2, App. C.1] The data-construction pipeline and the key benchmark share the same MLLM. Section 3.1 and C.1 state that 'a single multimodal large language model, written Φ, makes every semantic decision': it proposes candidate steps, refines boundaries, corrects descriptions, and verifies text–visual consistency. RxBrain-Bench-JointPlan and WorldPred then use ground-truth frames and texts derived from this same pipeline. Thus the headline Splan = 0.68 (vs. 0.43–0.52 for baselines) may largely measure how well RxBrain imitates Phi's annotation distribution rather than whether the plans are grounded in physical world states. The paper reports no human-annotation validation of segment boundaries, step descriptions, or state-transition judgments, and Appendix B explicitly notes that several sources have no verified external reference. This is an unverified key condition for the central claim. Please add a
- [§5.3, Table 4] The Splan rubric compounds the circularity. Goal-Image Correctness (GI, weight 25%) is rated by the GPT-5.5 judge against ground-truth goal images produced by the same Phi-based pipeline, and Image Similarity (IS, weight 10%) is DINO cosine to those same ground-truth frames. Together GI+IS are 35% of Splan. Using a different MLLM as judge does not break the circularity because the reference labels themselves are pipeline-derived. The per-criterion breakdown in Table 4 is informative, but the 'GoalImg' column (0.52) cannot be interpreted as physical goal-state correctness without external ground truth. Please provide at least a human-verified subset for this track and report the correlation between the MLLM judge and human judgments.
- [§7.2] The real-robot action extension is presented as evidence that the joint planning representation transfers to control, but the setup does not isolate the contribution of that representation. Section 7.2 says baselines and the model are trained on the same real-world robot data and that the joint planning pipeline is used to annotate additional subgoal labels incorporated into training. The 87% average success rate vs. 68%/82% for π0/π0.5 is encouraging, yet there is no ablation that removes the joint-planning labels or the interleaved supervision. The claim of 'promising real-robot performance without large-scale action-data pretraining' is therefore not attributable to the paper's core mechanism. Please include an ablation and report confidence intervals or per-task statistical significance for the three tasks.
minor comments (5)
- [App. C.1] The phrase 'written Φ' should be 'denoted Φ'; the same typo appears in the stress-test summary but should be corrected in the manuscript.
- [Tables 3–4] The captions state that n differs across methods due to judge content-filtering. Please report the exact n for each method and explain how filtering could differentially affect methods; as written, the comparison is not guaranteed to be on identical records.
- [Fig. 10] Add confidence intervals or trial counts per bar. The text mentions 100 trials per policy per task, but the figure would benefit from error bars and a note clarifying whether values are percentages.
- [§2.2] Minor typo: 'within a autoregressive process' should be 'within an autoregressive process.'
- [§9] The Limitations paragraph says 'shifting ground-truth VAE latents during training,' but Eq. (9) describes reconstruction through the VAE encoder/decoder, not a shift. Rephrase to match the formulation.
Circularity Check
RxBrain-Bench ground truth is produced by the same unvalidated MLLM that generates training labels, so the headline joint-planning scores partly measure imitation of Phi rather than physical grounding.
specific steps
-
fitted input called prediction
[Section 3.1/3.2/3.3 (Eq. 1, L1 target) -> Section 5.3 and Appendix F (Splan/JointPlan)]
"Throughout, a single multimodal large language model, written Φ, makes every semantic decision, so the module is one annotator queried with different prompts rather than a stack of specialized detectors."
Φ proposes candidate steps, refines start/end anchors, corrects descriptions, and verifies text-visual consistency (Sections 3.1-3.2). These verified segments are then used both as L1-L3 training targets (Section 3.3: '[C1:m, g_seg] -> [T_1, I_end_1, ...]') and as the ground-truth goal images/subtask texts for JointPlan scoring (Section 5.3; Appendix F: 'per step the GT subtask text, the model subtask text, the GT goal image, and the model-generated image'). Thus the headline Splan = 0.68 is evaluated against labels generated by the same automatic pipeline that created the training supervision, and no human-annotation or external physical reference is reported; Appendix B explicitly flags CyberOrigin, EgoVerse, Xperience, and AI Data raw as having no verified external reference. The benchm
full rationale
The central novel claim—joint language-visual planning grounded in physical states—rests on two pillars that share the same origin. The data-construction pipeline uses a single MLLM (Φ) for every semantic decision, producing the very text-visual pairs used for training; RxBrain-Bench then scores the joint-planning output against ground-truth subtask texts and goal images drawn from that same pipeline. This creates a partial circularity: the model is trained to imitate Φ's annotations and is evaluated on how well it reproduces Φ-derived labels, with no human validation of segment boundaries, step descriptions, or state-transition judgments. The paper is transparent about the pipeline and even notes that some sources have no verified external reference, which supports the concern rather than resolving it. However, the circularity is not complete. The model is also evaluated on independent external benchmarks (GenEval, CV-Bench, EmbSpatial, DA-2k, Share-Robot-Trajectory, 3DRSBench, etc.) and on real-robot success rates against π0/π0.5, which provide outside evidence for general understanding, generation, and action competence. Those results do not depend on Φ's annotations and therefore prevent the entire derivation from reducing to its inputs. The self-referential benchmark affects the specific claim of joint embodied planning but not the whole paper, so a score of 4 is appropriate rather than a higher score.
Axiom & Free-Parameter Ledger
free parameters (7)
- Keyframe sampler constants (plo, phi, beta, delta_min, gmax, Amax, n) =
0.8, 0.9, 0.2, 0.2s, 0.5s, 50, 30
- Stage 1 and Stage 2 loss weights (lambda_CE, lambda_FM) =
Stage1: 0.25/1.0; Stage2: 1.0/1.0
- Stage data-mixture ratios =
Stage1 6:4; Stage2 100:12:150:70:90
- Quality verification retention policy =
semantic score >= 4; near-duplicate filter 99.5%; retained 75.18%
- Action flow-time distribution and ODE steps =
s ~ Beta(1.5,1); K=2 at deployment
- Benchmark rubric weights =
Sgen: 0.15/0.30/0.20/0.20/0.15; Splan: 0.10/0.25/0.25/0.20/0.10/0.10
- Flow-matching timestep shift, CFG text-condition dropout, generation resolution =
shift 1.0, dropout 0.1, <=256px
axioms (7)
- domain assumption Atomic planning steps are identifiable from visible object-state changes and can be described in language.
- domain assumption A single MLLM Phi can reliably propose, refine, and verify video segments without human correction.
- domain assumption Interleaved text-plus-image generation is an adequate representation of embodied planning.
- standard math Flow-matching on VAE latents with the linear path and logit-normal timestep is a valid generative objective.
- domain assumption HY-Embodied-0.5 provides a reusable vision encoder and base multimodal capability.
- ad hoc to paper GPT-5.5 as MLLM judge plus DINO similarity measures planning and world-prediction quality.
- domain assumption Real-robot, UMI, simulation, and egocentric video are representative of embodied tasks.
read the original abstract
Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved. We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination. Unlike vision-language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents embodied plans in a single planning sequence where language and visual imagination play complementary roles. Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts embodied videos into joint text-visual planning supervision by decomposing videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains embodied understanding and generation abilities, and produces plans with coupled textual reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous robot action generation, where it shows promising real-robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for embodied cognition.
Figures
Reference graph
Works this paper leans on
-
[1]
Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800,
Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800,
-
[4]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966,
-
[6]
Revisiting feature prediction for learning visual representations from video.arXiv:2404.08471,
Adrien Bardes, Quentin Garrido, Jean Ponce, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nico- las Ballas. Revisiting feature prediction for learning visual representations from video.arXiv:2404.08471,
-
[9]
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127,
-
[10]
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669,
-
[11]
Hugging Face dataset, approximately 10,000 hours of egocentric factory video. Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, et al. Internvla-a1: Unifying understanding, generation and action for robotic manipulation.arXiv preprint arXiv:2601.02456,
-
[13]
Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,
-
[14]
Xinyi Chen, Yilun Chen, Yanwei Fu, Ning Gao, Jiaya Jia, Weiyang Jin, Hao Li, Yao Mu, Jiangmiao Pang, Yu Qiao, et al. Internvla-m1: A spatially guided vision-language-action framework for generalist robot policy.arXiv preprint arXiv:2510.13778,
-
[15]
Rynnec: Bringing mllms into embodied world.arXiv preprint arXiv:2508.14160,
Ronghao Dang, Yuqian Yuan, Yunxuan Mao, Kehan Li, Jiangpin Liu, Zhikai Wang, Xin Li, Fan Wang, and Deli Zhao. Rynnec: Bringing mllms into embodied world.arXiv preprint arXiv:2508.14160,
-
[16]
Rynnbrain: Open embodied foundation models.arXiv preprint arXiv:2602.14979,
Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, et al. Rynnbrain: Open embodied foundation models.arXiv preprint arXiv:2602.14979,
-
[17]
Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,
-
[18]
Rethinking video generation model for the embodied world.arXiv preprint arXiv:2601.15282,
Yufan Deng, Zilin Pan, Hongyu Zhang, Xiaojie Li, Ruoqing Hu, Yufei Ding, Yiming Zou, Yan Zeng, and Daquan Zhou. Rethinking video generation model for the embodied world.arXiv preprint arXiv:2601.15282,
-
[19]
URL https: //arxiv.org/abs/2406.05756. 37 Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synt...
-
[20]
Molmoact2: Action reasoning models for real-world deployment
Haoquan Fang, Jiafei Duan, Donovan Clay, Sam Wang, Shuo Liu, Weikai Huang, Xiang Fan, Wei-Chuan Tsai, Shirui Chen, Yi Ru Wang, et al. Molmoact2: Action reasoning models for real-world deployment. arXiv preprint arXiv:2605.02881,
-
[21]
Galaxea open-world dataset and G0 dual-system VLA model.arXiv preprint arXiv:2509.00576,
Galaxea Team. Galaxea open-world dataset and G0 dual-system VLA model.arXiv preprint arXiv:2509.00576,
-
[22]
Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, et al. Dreamdojo: A generalist robot world model from large-scale human videos.arXiv preprint arXiv:2602.06949,
-
[23]
URLhttps://arxiv.org/abs/2310.11513. Google DeepMind. Genie 2: A large-scale foundation world model. https://deepmind.google/discov er/blog/genie-2-a-large-scale-foundation-world-model/,
-
[24]
Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-world: A controllable generative world model for robot manipulation.arXiv preprint arXiv:2510.10125,
-
[25]
Yoon, Mouli Sivapurapu, and Jian Zhang
Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709,
-
[26]
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations.arXiv preprint arXiv:2412.14803,
-
[27]
URL https://arxiv.org/abs/ 2602.09849. Physical Intelligence, Bo Ai, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Greg Balke, Kevin Black, George Bokinsky, Shihao Cao, Thomas Charbonnier, Vedant Choudhary, Foster Collins, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Maitrayee Dhaka, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmai...
-
[28]
URLhttps://arxiv.org/abs/2604.15483. InternRobotics. InternData-A1. https://huggingface.co/datasets/InternRobotics/InternData-A1,
-
[29]
Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, Xinda Huang, Jiaming Wang, Shengjie Zhang, Zhongyuan Wang, et al. RoboBrain: A unified brain model for robotic manipulation from abstract to concrete.arXiv preprint arXiv:2502.21257,
-
[30]
π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164,
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164,
-
[31]
Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163,
-
[32]
Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125,
Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125,
-
[33]
MolmoAct: Action reasoning models that can reason in space.arXiv preprint arXiv:2508.07917,
Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, Winson Han, Wilbert Pumacay, Angelica Wu, Rose Hendrix, Karen Farley, Eli VanderBilt, Ali Farhadi, Dieter Fox, and Ranjay Krishna. MolmoAct: Action reasoning models that can reason in space.arXiv preprint arXiv:2508.07917,
-
[34]
Dingming Li, Hongxing Li, Zixuan Wang, Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, et al. Viewspatial-bench: Evaluating multi-perspective spatial localization in vision-language models.arXiv preprint arXiv:2505.21500,
-
[35]
Causal world modeling for robot control.arXiv preprint arXiv:2601.21998,
Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, Yujun Shen, and Yinghao Xu. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998,
-
[36]
Mm-act: Learn from multimodal parallel generation to act
Haotian Liang, Xinyi Chen, Bin Wang, Mingkang Chen, Yitian Liu, Yuhao Zhang, Zanxin Chen, Tianshuo Yang, Yilun Chen, Jiangmiao Pang, et al. Mm-act: Learn from multimodal parallel generation to act. arXiv preprint arXiv:2512.00975,
-
[37]
39 Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996,
-
[38]
URL https://www.biorxiv.org/content/10.64898/2026.05.01.722168v1
doi: 10.64898/2026.05.01.722168. URL https://www.biorxiv.org/content/10.64898/2026.05.01.722168v1. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems (NeurIPS),
-
[39]
3dsrbench: A comprehensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825,
Wufei Ma, Haoyu Chen, Guofeng Zhang, Celso M de Melo, Alan Yuille, and Jieneng Chen. 3dsrbench: A comprehensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825,
-
[40]
GPT-4o system card.arXiv preprint arXiv:2410.21276,
OpenAI. GPT-4o system card.arXiv preprint arXiv:2410.21276,
-
[41]
π0.5: A vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054,
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Chelsea Finn, Sergey Levine, et al. π0.5: A vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054,
-
[42]
Heqian Qiu, Zhaofeng Shi, Lanxiao Wang, Huiyu Xiong, Xiang Li, and Hongliang Li
Hugging Face dataset, real-world human manipulation captured with an exoskeleton-glove system. Heqian Qiu, Zhaofeng Shi, Lanxiao Wang, Huiyu Xiong, Xiang Li, and Hongliang Li. EgoMe: A new dataset and challenge for following me via egocentric view in real world.arXiv preprint arXiv:2501.19061,
-
[43]
RoboCOIN Team. RoboCOIN: An open-sourced bimanual robotic data collection for integrated ma- nipulation.arXiv preprint arXiv:2511.17441,
-
[44]
Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaugh- lin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,
-
[45]
URLhttps://arxiv.org/abs/2601.14352. BAAI RoboBrain Team. RoboBrain 2.0 technical report.arXiv preprint,
-
[46]
Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020,
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020,
-
[47]
HY Team, Xumin Yu, Zuyan Liu, Ziyi Wang, He Zhang, Yongming Rao, Fangfu Liu, Yani Zhang, Ruowen Zhao, Oran Wang, et al. Hy-embodied-0.5: Embodied foundation models for real-world agents.arXiv preprint arXiv:2604.07430, 2026a. Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, e...
-
[48]
Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, T...
-
[49]
Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, and Yu Xiang. HO-Cap: A capture system and dataset for 3d reconstruction and pose tracking of hand-object interaction. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025a. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Ch...
-
[50]
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848,
-
[51]
Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie
URL https://arxiv.org/abs/2601.18692. Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024a. Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything...
-
[52]
Gigaworld- policy: An efficient action-centered world–action model, 2026a
Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jie Li, Jindi Lv, Jingyu Liu, Min Cao, Peng Li, Qiuping Deng, Wenjun Mei, Xiaofeng Wang, Xinze Chen, Xinyu Zhou, Yang Wang, Yifan Chang, Yifan Li, Yukun Zhou, Yun Ye, Zhichao Liu, and Zheng Zhu. Gigaworld- policy: An efficient action-centered world–action model, 2026a. URL h...
-
[53]
Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, Jiazhao Zhang, et al. Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models.arXiv preprint arXiv:2606.17846,
-
[54]
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721,
-
[55]
He Zhang, Lingzhu Xiang, Haitao Lin, Zeyu Huang, Minghui Wang, Dingyan Zhong, Yubo Dong, Yihao Wu, Yongming Rao, Dongsheng Zhang, Wanjia He, Ling Chen, Kai Huang, Jiahao Chen, Sichang Su, Xumin Yu, Ziyi Wang, Chengwei Zhu, Xiao Teng, Yuchun Guo, Yufeng Zhang, Yuandong Liu, Rui Wang, Zisheng Lu, Han Hu, and Zhengyou Zhang. Hy-embodied-0.5-vla: From vision-...
-
[56]
Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han
URL https://arxiv.org/abs/2501 .18867. Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han. TASTE- Rob: Advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025a. Zhenyu Zhao, Hongyi Jing, Xiawei ...
Pith/arXiv arXiv 2025
-
[2021]
Sat: Spatial aptitude training for multimodal language models.arXiv preprint arXiv:2412.07755, 3,
Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat: Spatial aptitude training for multimodal language models.arXiv preprint arXiv:2412.07755, 3,
-
[2022]
Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, X...
-
[2023]
Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,
-
[2024]
Black Forest Labs. FLUX.1 kontext: Flow matching for in-context image generation and editing in latent space.arXiv preprint arXiv:2506.15742,
-
[2025]
Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558,
Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558,
-
[2026]
Rynnvla-002: A unified vision-language-action and world model.arXiv preprint arXiv:2511.17502,
Jun Cen, Siteng Huang, Yuqian Yuan, Kehan Li, Hangjie Yuan, Chaohui Yu, Yuming Jiang, Jiayan Guo, Xin Li, Hao Luo, Fan Wang, Fan Wang, and Deli Zhao. Rynnvla-002: A unified vision-language-action and world model.arXiv preprint arXiv:2511.17502,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.