Pith. sign in

REVIEW 3 major objections 5 minor 152 references

Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Post-training a robot vision-language model by execution capability, not by dataset, improves most benchmarks and transfers to simulated robot execution.

desk verdict State Verification results rest on a benchmark that mirrors the training reward, but the consolidation recipe and closed-loop results are solid enough to warrant a serious look. read the letter →

arxiv 2608.06756 v1 pith:DQYAGKYI submitted 2026-08-07 cs.AI

Ying Chen , Weizhen Li , Zhe Hu , Zhenjiang Li , Rui Jiang , Zhifeng Gu , Lihuang Fang , Jiangping Liu
show 2 more authors
Lei Yi Jie Chen
This is my paper · ORCID
classification cs.AI
keywords embodiedvision-languagemodelcapabilitytaxonomyreinforcementlearningGRPOmergingTIESon-policydistillationstateverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models for robots typically pile post-training data from many tasks into one model, making it hard to tell which capabilities were acquired or kept. Capek 0.5 instead organizes post-training around four capabilities that every robot execution repeatedly needs—spatial reasoning, temporal understanding, action guidance, and state verification—trains a separate specialist for each from the same backbone with reinforcement learning and verifiable rewards, then merges and distills the specialists into one checkpoint. The paper reports that this recipe improves the large majority of matched benchmarks over its Qwen start at both 2B and 35B-A3B scales, keeps all four capabilities in a single model, and raises closed-loop simulated task success. If the recipe is right, it offers a practical template for building unified embodied vision-language models without task-specific heads or routing at inference.

What carries the argument

The load-bearing machinery is the execution-centric capability taxonomy plus the specialist-to-unified consolidation pipeline. Four specialists (Spatial Reasoning, Temporal Understanding, Action Guidance, State Verification) are post-trained from one shared backbone with token-level GRPO and capability-specific verifiable rewards; their parameter updates relative to that backbone are task vectors. A TIES merge resolves conflicting updates by trimming low-magnitude entries, electing a per-coordinate consensus sign, and averaging sign-consistent updates, producing the unified student's initialization. Routed MOPD then distills each specialist into that student on the student's own sampled prefixes, with the responsible teacher queried per sample and all teachers frozen; this is what makes the four behaviors compatible in a single autoregressive checkpoint.

What would settle it

Check the 500 Capek-StateBench samples against the State Verification training corpus for shared episode identifiers, checkpoint timestamps, or evidence frames; if any overlap exists, measure StateBench-P and StateBench-T after retraining the State specialist on a provably disjoint split, and see whether the +11.0 and +2.45 point gains survive.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that capability acquisition and capability consolidation can be separated cleanly: four same-origin specialists—trained with GRPO on capability-aligned data under verifiable rewards—define what the unified model should preserve, and a TIES weight-space merge followed by routed multi-teacher on-policy distillation defines how to consolidate them. The paper reports that this two-stage recipe improves 28 of 34 matched benchmark rows (35B-A3B track) and 30 of 34 (2B track) over the shared Qwen initialization, retains all four specialized capabilities in one checkpoint with quantified losses, and transfers to closed-loop simulation, with EB-HAB success rising from 46.0 to 63.0 and VIGIL benchmark success from 28.8 to 32.2. The claim is that execution-centric organization—grouping supervision by functional role rather than dataset—is what carries these gains, not any single benchmark or task.

Load-bearing premise

The load-bearing premise is that Capek-StateBench, the new state-verification benchmark, is genuinely held out from training: both it and the State Verification training data are built from BEHAVIOR-1K trajectories, and the paper does not demonstrate that benchmark episodes, checkpoints, or frames are disjoint from the training runs, so if they overlap the reported state-verification gains would be partly memorization.

Editorial extensions

If this is right

  • At both 2B and 35B-A3B scales, the unified model improves the majority of matched benchmark rows over its Qwen initialization, including every Action Guidance row at both scales.
  • Consolidating via TIES plus MOPD retains all four specialist capabilities with smaller losses than either merging alone or mixing all data into a single RL run.
  • The gains carry into closed-loop simulated execution, with higher average success than the backbone on EmbodiedBench and higher benchmark success on VIGIL.
  • Inference uses a single autoregressive checkpoint with no task-specific heads and no routing, so the taxonomy and its specialist routes are training-time apparatus only.
  • The unified output interface—points, boxes, intervals, trajectories, verdicts, progress JSON—lets all four capability families share one backbone and one tokenizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that the same specialist-to-unified pipeline should extend to new capability families, such as tool use or skill orchestration, by adding new task vectors and new MOPD routes without changing the consolidation machinery.
  • A testable extension would be to swap the verifiable reward functions for open-vocabulary or 3D action outputs; the taxonomy and merge-then-distill structure would remain, but the gains would hinge on whether the new rewards capture the capability as cleanly.
  • The large closed-loop gains on long-horizon and complex-instruction subsets suggest the binding constraint in embodied VLMs may be coordinating multiple capabilities over many steps rather than any single perception or reasoning skill.
  • The paper's comparison treats the recipe as the cause of improvement, but a cleaner attribution would require ablating the taxonomy itself (e.g., grouping the same data by dataset rather than capability) while holding the consolidation method fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Capek 0.5, a vision-language model for embodied reasoning whose post-training is organized around four execution-centric capability families: Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification. For each capability, a specialist is trained from a shared Qwen backbone using GRPO with verifiable rewards; the specialists are then consolidated into a single inference-time model by TIES weight-space merging followed by routed multi-teacher on-policy distillation. The authors instantiate the recipe at 2B and 35B-A3B scales and evaluate it with three complementary protocols: a large public-benchmark suite augmented by the new Capek-StateBench, a controlled consolidation ablation comparing specialists, Mix-RL, TIES, MOPD, and TIES+MOPD, and closed-loop simulated evaluation in EmbodiedBench and VIGIL. The main empirical claims are that Capek 0.5 improves 28/34 and 30/34 matched benchmark rows over its Qwen initialization, retains all four specialized capabilities in one checkpoint, and transfers to closed-loop execution, with the largest gains on action-guidance and state-verification tasks.

Significance. If the benchmark evidence holds, the paper provides a practical and unusually auditable template for building unified embodied VLMs: same-origin specialists trained under capability-aligned verifiable rewards, followed by weight-space and policy-space consolidation. The evaluation design has genuine strengths: matched Capek–Qwen pairs share decoding settings, all external baseline entries are DeepInsight reruns rather than quoted numbers, the consolidation ablation compares all variants from the same backbone with identical prompts and scorers, and the closed-loop experiments are held out from specialist training. The strongest positive evidence is the consistent improvement across action-guidance and temporal benchmarks at both scales. However, the State Verification claim currently rests on a benchmark that may overlap with the training data, and the headline row-count claim is made without any uncertainty quantification. These issues are local in the sense that they can be addressed with additional analysis, but they are load-bearing for the paper's central contribution, so the appropriate outcome is a major revision.

major comments (3)
  1. [§3.4, §5.1.1, Appendix A.2] The paper does not demonstrate that Capek-StateBench is a clean held-out evaluation. Section 3.4 states that all State Verification post-training data are constructed from BEHAVIOR-1K trajectories, and Section 5.1.1/Appendix A.2 build Capek-StateBench from the same BEHAVIOR-1K goal-condition and primitive-skill decompositions, with progress defined by Eq. (11) and scored by Eq. (9) using the same 25-point truncated proximity as the training reward r_value in Section 4.1.2. The manuscript never shows that the benchmark episodes, checkpoints, annotated frames, next-action targets, or progress labels are disjoint from the training trajectories; the general deduplication statement in Section 3.5 is about training-data construction and does not explicitly cover the new benchmark. This is consequential because the VIGIL state-verification diagnostic in Table 6 shows no improvement over the Qwen backbone (96.0/84.8 vs 96.0/85.6 W/B), leaving Capek-StateBench as the sole positive evidence for the State Verification capability family. The authors should provide a formal disjointness analysis (episode-, checkpoint-, and frame-level overlap counts, plus label provenance for next-action and progress targets) or replace/reinforce Capek-StateBench with an independently constructed external state-verification benchmark.
  2. [§5.1.3, §5.1.4, Tables 2 and 3] The headline claim that Capek 0.5 improves 28 of 34 and 30 of 34 matched rows is based on single evaluation runs without error bars, significance tests, or multiple decoding seeds, even though the inference configuration uses temperature 0.7 sampling. Several decisive row-wise differences are small (e.g., CV-Bench 88.56→89.99, MMBench-EN 92.35→92.40, RealWorldQA 82.09→83.40 at 35B-A3B), so the row-count claim is not robustly established for close rows. At minimum, the paper should report confidence intervals or bootstrap estimates for the matched Capek–Qwen pairs and should rerun the closest rows with multiple seeds; ideally this should also cover the EmbodiedBench and VIGIL success rates in Tables 5 and 6, where the reported gains are based on 300- and 1000-episode sets respectively but still lack any variance estimate.
  3. [§4.1.2 vs §5.1.1, Eq. (10)] The task-track benchmark score s_T = (5s_p + 4s_a)/9 in Eq. (10) is a near re-test of the training reward R_prog+act = 0.5 r_value + 0.4 r_action + 0.1 r_fmt in Section 4.1.2: both use the same 25-percentage-point truncated proximity for progress and the same semantic next-action matching. Metric alignment is acceptable when labels are truly held out, but the paper should state this explicitly and explain why the measured StateBench-T gain (+2.45 at 35B-A3B) reflects acquired capability rather than reward overfitting to the training objective. The absence of a corresponding gain on the VIGIL state-verification probe makes this concern concrete rather than hypothetical.
minor comments (5)
  1. [§5.1.1 and Appendix A.2] Please clarify the size of Capek-StateBench: Appendix A.2 says the benchmark 'contains two 500-example tracks,' while Section 5.1.1 says 'The released 500-example split combines both schemes, with 213 task-condition records and 287 primitive-skill records.' The reader cannot tell whether each track has 500 examples or whether the 213/287 split is the entire task track.
  2. [Table 4] The Mix-RL baseline is described only as 'data-mixing reinforcement learning.' Please specify what is mixed (all four specialist corpora in one GRPO run, the same total number of samples, and the same KL and reward settings) and give its training budget, so the comparison against TIES/MOPD is interpretable.
  3. [Appendix A.3, Eq. (15)] The symbol M is used both for the total number of compared waypoints and as an index in the sum; please use a separate symbol for the index or restate the sum over waypoint pairs to avoid notational collision.
  4. [Figure 1] The bar chart uses many closely related colors that are hard to distinguish in print; adding hatched patterns or marker labels would make the five-domain comparison readable.
  5. [§5.1.1] The text says next-step prediction is evaluated by 'semantic exact matching,' but Appendix A.2 says a 'fixed semantic judge' is used; please state whether this judge was frozen before the evaluation and whether the same judge was used during training rewards, since a learned judge could introduce a shared error surface.

Circularity Check

2 steps flagged · score 6.0 of 10

Capek-StateBench's progress score is identical to the PVE training reward and both are built from BEHAVIOR-1K, making the State Verification results self-referential; the rest of the benchmark evidence is external and independent.

  1. self definitional [Section 5.1.1 (Eq. 9) and Section 4.1.2 (State Verification reward)]
    "For the task track, progress is scored using sp = max(0, 1 − |ˆp−p|/25), (9) ... the progress reward is defined as a 25-percentage-point truncated proximity score: rvalue = max(0, 1 − |ˆp−p|/25)."

    The Capek-StateBench-T progress metric is the same truncated-proximity function used as the PVE training reward. Because the benchmark is described as 'aligned with the State Verification training objectives' and its progress labels use the same BEHAVIOR-1K goal-condition/child-skill decompositions as the training labels, the benchmark's primary score is a re-evaluation of the optimized reward. The paper provides no disjointness guarantee for the 500-example split; Section 5.4's 'strictly held out' statement covers only EmbodiedBench and VIGIL. Thus the StateBench-T improvement is at least partly a by-construction re-test of the training objective, not an independent capability measurement.

  2. self definitional [Section 3.4 and Appendix A.2 / Section 5.1.1]
    "We construct all State Verification post-training data from BEHAVIOR-1K household-manipulation trajectories. ... Capek-StateBench contains two 500-example tracks aligned with the State Verification training objectives."

    Both the PVE training corpus and the Capek-StateBench-T benchmark are derived from BEHAVIOR-1K with the same progress definitions (fraction of satisfied goal conditions or completed child skills). The paper never states that the benchmark episodes, checkpoints, or frames are disjoint from the training trajectories. Combined with the identical scoring function (Eq. 9 = r_value), the benchmark cannot provide independent confirmation that State Verification was acquired; it can only confirm that the model optimizes the training reward on data from the same source.

full rationale

The bulk of the paper's benchmark evidence is external and protocol-matched: 28/34 and 30/34 row improvements are measured on public spatial, temporal, action-guidance, and general-retention benchmarks with native metrics, and the closed-loop EmbodiedBench gains are independent of the authors' own constructions. The circularity is concentrated in the State Verification pillar, which is one of the four central capability families. Its only supporting benchmark, Capek-StateBench, is built from BEHAVIOR-1K, the same source as all State Verification training data, and its task-track progress score is literally the training reward r_value from Section 4.1.2. Appendix A.2 explicitly calls the benchmark 'aligned with the State Verification training objectives,' and Section 5.1.1 says the Task track 'instantiates the execution-level progress-and-next-step readout used during State Verification training.' No disjointness guarantee is given for the 500-example split; the 'strictly held out' statement in Section 5.4 applies only to EmbodiedBench and VIGIL. The VIGIL state-verification diagnostic shows no gain over the Qwen backbone (96.0/84.8 vs 96.0/85.6), so Capek-StateBench is the sole evidence for the claimed State Verification improvement. This makes the State Verification component partially circular, while the overall multi-capability recipe retains substantial independent support. VIGIL is a self-citation of the authors' prior work, but it is not load-bearing for the main claim because EmbodiedBench independently supports closed-loop transfer.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim rests on benchmark validity, specifically disjointness of Capek-StateBench from training, the efficacy of the TIES plus MOPD consolidation recipe, and the correctness of cited RL and distillation machinery. There are no newly postulated physical entities. Free parameters are hand-set training and reward constants, not fitted to the reported benchmark results, with the caveat that the progress margin of 25 is used both as a training reward threshold and as the primary evaluation metric for StateBench-T.

free parameters (6)
  • TIES trim fraction (tau) = 0.8
    Masks the 80% lowest-magnitude entries in each specialist task vector before sign-consensus averaging in Eq. 6; chosen by hand and directly shapes the consolidated initialization.
  • TIES merge scale (lambda_TIES) = 1.0
    Scales the merged task-vector update in Eq. 6; hand-set.
  • Trajectory reward kernel scale (lambda_traj) = 10
    Converts raw discrete Frechet distance to an exponential reward in Eq. 5; hand-set, affects Guidance specialist optimization.
  • Progress proximity margin = 25 percentage points
    Truncated proximity score gives full credit within 25 points in Eq. 9 and the PVE reward; hand-set and reused as the StateBench-T evaluation metric.
  • GRPO KL coefficient (beta) = 0.01 (off for trajectory)
    KL regularization in Eq. 2; set per branch in Table 7, affects stability of specialist RL.
  • PVE reward weights = 0.5 progress / 0.4 next-action / 0.1 format
    Hand-set combination in Section 4.1.2 for progress-and-next-action samples; changes the State Verification specialist's optimization target.
assumptions (3)
  • domain assumption Capek-StateBench is disjoint from the BEHAVIOR-1K trajectories used to train the State Verification specialist, despite sharing the same source and being described as aligned with training objectives.
    The paper does not state or show this disjointness; the primary StateBench gains depend on it.
  • domain assumption Four capability families have sufficiently separated reward geometries that independent specialist training followed by TIES plus routed MOPD avoids catastrophic interference and preserves each capability.
    This is the premise of the two-stage recipe in Section 2.2; Table 4 provides empirical support but no guarantee, and the State specialist's evaluation is partly self-referential.
  • domain assumption GRPO, TIES, and MOPD behave as described when applied to multimodal VLMs at 2B and 35B scale.
    The paper relies on the correctness and scalability of these cited algorithms as background technology.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence." pith.science (2026). https://pith.science/paper/DQYAGKYI

@misc{pith2026260806756,
  author       = {Pith},
  title        = {Pith review of: Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQYAGKYI}},
  note         = {Machine review of arXiv:2608.06756}
}
read the original abstract

Vision-language models are increasingly serving as the reasoning core of embodied agents. Robot execution is inherently iterative: each action reshapes the scene and physical state, continually renewing what must be perceived, reasoned about, and verified. Meeting these demands requires complementary capabilities that differ in supervision signals, prediction formats, and verification criteria. Existing approaches typically develop these capabilities against isolated, task-specific objectives, leaving open how they should be organized and integrated around execution as a whole. We present Capek 0.5, an embodied vision-language model built around an execution-centric capability taxonomy. Rather than organizing training by datasets or tasks, the taxonomy groups embodied capabilities according to their functional roles throughout execution and comprises four capability families: Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification. Each capability is first acquired by a dedicated specialist through reinforcement learning with verifiable rewards from a shared backbone, and the specialists are then consolidated into a single inference-time model through weight-space merging followed by routed policy-space distillation. We instantiate Capek 0.5 at the 2B and 35B-A3B scales and evaluate it from three complementary perspectives: comprehensive benchmark suites including Capek-StateBench, a new benchmark for state verification; a controlled study of capability retention from specialists to the unified model; and closed-loop evaluation in simulated embodied environments. Capek 0.5 improves the large majority of matched benchmark rows over its initialization, retains all four specialized capabilities in one checkpoint with quantified losses, and transfers to closed-loop embodied task execution.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

152 extracted references · 22 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (ICML), 2021

    2021

  2. [2]

    Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems (NeurIPS), 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  3. [3]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning (ICML), 2023

    2023

  4. [4]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  5. [5]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  6. [6]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  7. [7]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  8. [8]

    Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

Show all 152 references
  1. [9]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  2. [10]

    Code as policies: Language model programs for embodied control.arXiv preprint arXiv:2209.07753, 2022

    Jacky Liang, Wenlong Huang, Fei Xia, et al. Code as policies: Language model programs for embodied control.arXiv preprint arXiv:2209.07753, 2022

    2022 arXiv

  3. [11]

    V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

    Wenlong Huang, Chen Wang, Ruohan Zhang, et al. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  4. [12]

    Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020, 2025

    Gemini Robotics Team. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020, 2025

    2025 arXiv

  5. [13]

    Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    Wentao Yuan, Jiafei Duan, Valts Blukis, et al. Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    2024 arXiv

  6. [14]

    Robobrain: A unified brain model for robotic manipulation from abstract to concrete.arXiv preprint arXiv:2502.21257, 2025

    Yuheng Ji, Huajie Tan, Jiayu Shi, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete.arXiv preprint arXiv:2502.21257, 2025

    2025 arXiv

  7. [15]

    Robobrain 2.5: Depth in sight, time in mind.arXiv preprint arXiv:2601.14352, 2026

    Huajie Tan, Enshen Zhou, Zhiyu Li, et al. Robobrain 2.5: Depth in sight, time in mind.arXiv preprint arXiv:2601.14352, 2026

    2026

  8. [16]

    Rynnbrain: Open embodied foundation models

    Ronghao Dang, Jiayan Guo, Bohan Hou, et al. Rynnbrain: Open embodied foundation models. arXiv preprint arXiv:2602.14979, 2026

    2026

  9. [17]

    Hy-embodied-vlm-1.0: Efficient physical-world agents.arXiv preprint arXiv:2607.12894, 2026

    Ziyi Wang, Xumin Yu, Yongming Rao, Yonggen Ling, Yunheng Li, et al. Hy-embodied-vlm-1.0: Efficient physical-world agents.arXiv preprint arXiv:2607.12894, 2026. 20

    2026 arXiv

  10. [18]

    Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026

    NVIDIA. Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026. URL https://research.nvidia.com/labs/cosmos-lab/cosmos3/ technical-report.pdf

    2026 arXiv

  11. [19]

    Mimo-embodied: X-embodied foundation model technical report.arXiv preprint arXiv:2511.16518, 2025

    Xiaoshuai Hao, Lei Zhou, Zhijian Huang, et al. Mimo-embodied: X-embodied foundation model technical report.arXiv preprint arXiv:2511.16518, 2025

    2025 arXiv

  12. [20]

    Embodied-r1.5: Evolving physical intelligence via embodied foundation models.arXiv preprint arXiv:2606.11324, 2026

    Yifu Yuan, Yaoting Huang, Xianze Yao, et al. Embodied-r1.5: Evolving physical intelligence via embodied foundation models.arXiv preprint arXiv:2606.11324, 2026

    2026 arXiv

  13. [21]

    Vesta: A generalist embodied reasoning model.arXiv preprint arXiv:2606.20905, 2026

    Johan Bjorck, Zhiqi Li, Yunze Man, et al. Vesta: A generalist embodied reasoning model.arXiv preprint arXiv:2606.20905, 2026

    2026 arXiv

  14. [22]

    Ace-brain-0.5: A unified embodied foundational model for physical agentic ai.arXiv preprint arXiv:2607.04426, 2026

    ACE-Brain Team, Ziyang Gong, Haoming Gu, Zehang Luo, et al. Ace-brain-0.5: A unified embodied foundational model for physical agentic ai.arXiv preprint arXiv:2607.04426, 2026

    2026 arXiv

  15. [23]

    Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents.arXiv preprint arXiv:2502.09560, 2025

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents.arXiv preprint arXi...

    2025 arXiv

  16. [24]

    Towards embodied agentic ai: Review and classification of llm-and vlm-driven robot autonomy and interaction.arXiv preprint arXiv:2508.05294, 2025

    Sahar Salimpour, Lei Fu, Kajetan Rachwał, Pascal Bertrand, Kevin O’Sullivan, Robert Jakob, Farhad Keramat, Leonardo Militano, Giovanni Toffetti, Harry Edelman, et al. Towards embodied agentic ai: Review and classification of llm-and vlm-driven robot autonomy and interaction.ar...

    2025

  17. [25]

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation.Conference on Robot Learning (CoRL), 2023

    Chengshu Li, Ruohan Zhang, Josiah Wong, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation.Conference on Robot Learning (CoRL), 2023

    2023

  18. [26]

    Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training.arXiv preprint arXiv:2606.30406, 2026

    Wenhan Ma, Jianyu Wei, Liang Zhao, et al. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training.arXiv preprint arXiv:2606.30406, 2026

    2026 arXiv

  19. [27]

    To mix or to merge: Toward multi-domain reinforcement learning for large language models.arXiv preprint arXiv:2602.12566, 2026

    Haoqing Wang, Xiang Long, Ziheng Li, Yilong Xu, Tingguang Li, and Yehui Tang. To mix or to merge: Toward multi-domain reinforcement learning for large language models.arXiv preprint arXiv:2602.12566, 2026

    2026

  20. [28]

    Ties-merging: Resolving interference when merging models.arXiv preprint arXiv:2306.01708, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, et al. Ties-merging: Resolving interference when merging models.arXiv preprint arXiv:2306.01708, 2023

    2023 arXiv

  21. [29]

    Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026

    Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026. URL https://qwen.ai/blog?id=qwen3.6-35b-a3b

    2026

  22. [30]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen. ai/blog?id=qwen3.5

    2026

  23. [31]

    Done, but not sure: Disentangling world completion from self-termination in embodied agents.arXiv preprint arXiv:2605.08747, 2026

    Ying Chen, Lihuang Fang, Rui Jiang, Mingxu Wang, Zhifeng Gu, Lei Yi, and Jie Chen. Done, but not sure: Disentangling world completion from self-termination in embodied agents.arXiv preprint arXiv:2605.08747, 2026

    2026 arXiv

  24. [32]

    Spatial intelligence in vision-language models: A comprehensive survey

    Disheng Liu, Tuo Liang, Zhe Hu, Jierui Peng, Yiren Lu, Yi Xu, Yun Fu, Yu Yin, et al. Spatial intelligence in vision-language models: A comprehensive survey. 2025

    2025

  25. [33]

    Scaling spatial intelligence with multimodal foundation models

    Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Tongxi Zhou, et al. Scaling spatial intelligence with multimodal foundation models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2026

  26. [34]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10632...

    2025

  27. [35]

    Mindcube: Spatial mental modeling from limited views.arXiv preprint arXiv:2506.21458, 2025

    Qineng Wang, Baiqiao Yin, Pingyue Zhang, et al. Mindcube: Spatial mental modeling from limited views.arXiv preprint arXiv:2506.21458, 2025

    2025

  28. [36]

    Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models.arXiv preprint arXiv:2406.05756, 2024

    Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models.arXiv preprint arXiv:2406.05756, 2024

    2024 arXiv

  29. [37]

    Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019

    1910 arXiv

  30. [38]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  31. [39]

    Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems, 36: 42748–42761, 2023

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Syste...

    2023

  32. [40]

    Star: A benchmark for situated reasoning in real-world videos.arXiv preprint arXiv:2405.09711, 2024

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos.arXiv preprint arXiv:2405.09711, 2024

    2024 arXiv

  33. [41]

    Llava-video: Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, et al. Llava-video: Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  34. [42]

    Scaling rl to long videos

    Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, Sifei Liu, Hongxu Yin, Yao Lu, and Song Han. Scaling rl to long videos. 2025

    2025

  35. [43]

    Sigurdsson, Gul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta

    Gunnar A. Sigurdsson, Gul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. InProceedings of the European Conference on Computer Vision (ECCV), 2016

    2016

  36. [44]

    Localizing moments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. InProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017

    2017

  37. [45]

    Hierarchical video-moment retrieval and step-captioning

    Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23056–23065, 2023

    2023

  38. [46]

    Queryd: A video dataset with high-quality text and audio narrations

    Andreea-Maria Oncescu, Joao F Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. Queryd: A video dataset with high-quality text and audio narrations. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2265–2269. ...

    2021

  39. [47]

    Llava-st: A multimodal large language model for fine- grained spatial-temporal understanding.arXiv preprint arXiv:2501.08282, 2025

    Hongyu Li, Jinyu Chen, Ziyu Wei, et al. Llava-st: A multimodal large language model for fine- grained spatial-temporal understanding.arXiv preprint arXiv:2501.08282, 2025

    2025 arXiv

  40. [48]

    Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    Lin Chen, Xilin Wei, Jinsong Li, et al. Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024. 22

    2024 arXiv

  41. [49]

    Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. InProceedings of the Computer Vision and Pattern Re...

    2025

  42. [50]

    Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    NVIDIA, Alisson Azzolini, Junjie Bai, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    2025 arXiv

  43. [51]

    Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  44. [52]

    Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12945, 2024

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, et al. Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12945, 2024

    2024 arXiv

  45. [53]

    Agibot world colosseo: A large-scale ma- nipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025

    AgiBot-World Contributors, Qingwen Bu, Jisong Cai, et al. Agibot world colosseo: A large-scale ma- nipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025

    2025 arXiv

  46. [54]

    Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence.arXiv preprint arXiv:2512.24653, 2025

    Chengkai Hou, Kun Wu, Jiaming Liu, et al. Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence.arXiv preprint arXiv:2512.24653, 2025

    2025

  47. [55]

    From seeing to doing: Bridging reasoning and decision for robotic manipulation.arXiv preprint arXiv:2505.08548, 2025

    Yifu Yuan, Haiqin Cui, Yibin Chen, Zibin Dong, Fei Ni, Longxin Kou, Jinyi Liu, Pengyi Li, Yan Zheng, and Jianye Hao. From seeing to doing: Bridging reasoning and decision for robotic manipulation.arXiv preprint arXiv:2505.08548, 2025

    2025 arXiv

  48. [56]

    Robovqa: Multimodal long-horizon reasoning for robotics.arXiv preprint arXiv:2311.00899, 2023

    Pierre Sermanet, Tianli Ding, Jeffrey Zhao, et al. Robovqa: Multimodal long-horizon reasoning for robotics.arXiv preprint arXiv:2311.00899, 2023

    2023 arXiv

  49. [57]

    Sanketi, and Ken Goldberg

    Kaiyuan Chen, Shuangyu Xie, Zehan Ma, Pannag R. Sanketi, and Ken Goldberg. Robo2vlm: Visual question answering from large-scale in-the-wild robot manipulation datasets.arXiv preprint arXiv:2505.15517, 2025

    2025 arXiv

  50. [58]

    Roboafford++: A generative ai-enhanced dataset for multimodal affordance learning in robotic manipulation and navigation.arXiv preprint arXiv:2511.12436, 2025

    Xiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, et al. Roboafford++: A generative ai-enhanced dataset for multimodal affordance learning in robotic manipulation and navigation.arXiv preprint arXiv:2511.12436, 2025

    2025

  51. [59]

    From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025

    Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yujie Yuan, Xinyue Cai, Guowei Huang, Xingyue Quan, Hang Xu, and Li Zhang. From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025

    2025

  52. [60]

    Onethinker: All-in-one reasoning model for image and video.arXiv preprint arXiv:2512.03043, 2025

    Kaituo Feng, Manyuan Zhang, Hongyu Li, et al. Onethinker: All-in-one reasoning model for image and video.arXiv preprint arXiv:2512.03043, 2025

    2025 arXiv

  53. [61]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  54. [62]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  55. [63]

    Thinking with visual primitives.Technical report, 2026

    Ruijie Lu, Yiyang Ma, Xiaokang Chen, et al. Thinking with visual primitives.Technical report, 2026

    2026

  56. [64]

    Embodied-r1: Reinforced embodied reasoning for general robotic manipulation.arXiv preprint arXiv:2508.13998, 2025

    Yifu Yuan, Haiqin Cui, Yaoting Huang, et al. Embodied-r1: Reinforced embodied reasoning for general robotic manipulation.arXiv preprint arXiv:2508.13998, 2025

    2025 arXiv

  57. [65]

    Computing discrete fréchet distance

    Thomas Eiter, Heikki Mannila, et al. Computing discrete fréchet distance. 1994. 23

    1994

  58. [66]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, et al. Editing models with task arithmetic. International Conference on Learning Representations (ICLR), 2023

    2023

  59. [67]

    On-policy distillation of language models: Learning from self-generated mistakes.arXiv preprint arXiv:2306.13649, 2023

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al. On-policy distillation of language models: Learning from self-generated mistakes.arXiv preprint arXiv:2306.13649, 2023

    2023 arXiv

  60. [68]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, Adithya Jairam Iyer, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. InThe Thirty-eighth Annual Conference o...

    2024

  61. [69]

    Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models.arXiv preprint arXiv:2506.03135, 2025

    Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models.arXiv preprint arXiv:2506.03135, 2025

    2025

  62. [70]

    Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics.arXiv preprint arXiv:2411.16537, 2024

    Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics.arXiv preprint arXiv:2411.16537, 2024

    2024

  63. [71]

    Seeing from another perspective: Evaluating multi-view understanding in mllms.arXiv preprint arXiv:2504.15280, 2025

    Chun-Hsiao Yeh, Chenyu Wang, Shengbang Tong, Ta-Ying Cheng, Rouyu Wang, Tianzhe Chu, Yuexiang Zhai, Yubei Chen, Shenghua Gao, and Yi Ma. Seeing from another perspective: Evaluating multi-view understanding in mllms.arXiv preprint arXiv:2504.15280, 2025

    2025 arXiv

  64. [72]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra,...

    2024

  65. [73]

    Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos

    Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24129–24138, 2025

    2025

  66. [74]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the IEEE/CVF conference on ...

    2025

  67. [75]

    Mvbench: A comprehensive multi-modal video under- standing benchmark.arXiv preprint arXiv:2311.17005, 2024

    Kunchang Li, Yali Wang, Yinan He, et al. Mvbench: A comprehensive multi-modal video under- standing benchmark.arXiv preprint arXiv:2311.17005, 2024

    2024 arXiv

  68. [76]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    Haoning Wu, Dongxu Li, Bei Chen, et al. Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    2024 arXiv

  69. [77]

    Timelens: Rethinking video temporal grounding with multimodal llms.arXiv preprint arXiv:2512.14698, 2025

    Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, and Limin Wang. Timelens: Rethinking video temporal grounding with multimodal llms.arXiv preprint arXiv:2512.14698, 2025

    2025

  70. [78]

    VL-Grasp: a 6-Dof interactive grasp policy for language-oriented objects in cluttered indoor scenes.arXiv preprint arXiv:2308.00640, 2023

    Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. VL-Grasp: a 6-Dof interactive grasp policy for language-oriented objects in cluttered indoor scenes.arXiv preprint arXiv:2308.00640, 2023. URLhttps://arxiv.org/abs/2308.00640. IROS 2023

    2023 arXiv

  71. [79]

    Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer.arXiv preprint arXiv:2510.03342, 2025

    Abbas Abdolmaleki, Saminda Abeyruwan, Joshua Ainslie, et al. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer.arXiv preprint arXiv:2510.03342, 2025. 24

    arXiv

  72. [80]

    Point-it-out: Benchmarking embodied reasoning for vision language models in multi-stage visual grounding.arXiv preprint arXiv:2509.25794, 2025

    Haotian Xue, Yunhao Ge, Yu Zeng, Zhaoshuo Li, Ming-Yu Liu, Yongxin Chen, and Jiaojiao Fan. Point-it-out: Benchmarking embodied reasoning for vision language models in multi-stage visual grounding.arXiv preprint arXiv:2509.25794, 2025

    2025

  73. [81]

    Navitrace: Evaluating embodied navigation of vision- language models.arXiv preprint arXiv:2510.26909, 2025

    Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Li- outikov, Marco Hutter, and Jonas Frey. Navitrace: Evaluating embodied navigation of vision- language models.arXiv preprint arXiv:2510.26909, 2025

  74. [82]

    Xiang Yue, Yuansheng Ni, Kai Zhang, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  75. [83]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, et al. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  76. [84]

    Grok-1.5 vision preview and realworldqa

    xAI. Grok-1.5 vision preview and realworldqa. https://x.ai/news/grok-1.5v, 2024. Accessed 2026-07-30

    2024

  77. [85]

    Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, et al. Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  78. [86]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  79. [87]

    MMLU-Pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language unders...

    2024

  80. [88]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InProceedings of the 42nd International Conferenc...

    2025

  81. [89]

    LiveCodeBench: Holistic and contamination-free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination-free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  82. [90]

    Deepinsight: A unified evaluation infrastructure across the physical ai stack.arXiv preprint arXiv:2606.17574, 2026

    Siyi Li, Chunyu Sun, Jiahao Zhang, Yuchen Kang, Wuliang Wang, Yu Qiu, Rui Jiang, Haitao Cui, and Jie Chen. Deepinsight: A unified evaluation infrastructure across the physical ai stack.arXiv preprint arXiv:2606.17574, 2026

    2026 arXiv

  83. [91]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  84. [92]

    Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474, 2017

    Eric Kolve, Roozbeh Mottaghi, Winson Han, et al. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474, 2017

    2017 arXiv

  85. [93]

    Procthor: Large-scale embodied ai using procedural generation.Advances in Neural Information Processing Systems (NeurIPS), 2022

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, et al. Procthor: Large-scale embodied ai using procedural generation.Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  86. [94]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024. 25

    2024 arXiv

  87. [95]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  88. [96]

    bbox_2d": [x1, y1, x2, y2],

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems ...

    2023

  89. [97]

    The left hand is holding a green lighter

    **Analyze the first image:** I see a first-person perspective of a character holding two items. The left hand is holding a green lighter. The top of the lighter is emitting a flame. The right hand is holding a gun. There is also a fire in the fireplace

  90. [98]

    The top of the lighter is now glowing red/orange, indicating it’s hot or just used, but crucially, there is no flame coming out of it

    **Analyze the second image:** The character is still holding the green lighter in the left hand. The top of the lighter is now glowing red/orange, indicating it’s hot or just used, but crucially, there is no flame coming out of it. The hand holding the lighter is moving away/d...

  91. [99]

    Based on observations, is the lighter on or off?

    **Synthesize observations:** In the first image, the lighter is clearly on (flame visible). In the second image, the flame is gone. The question asks "Based on observations, is the lighter on or off?". Looking at the progression, the lighter was lit and is now extinguished. Th...

  92. [100]

    * The club sandwich must be inside the packing box

    **Identify the goal conditions:** There are six conditions that must be satisfied: * The apples must be inside the packing box. * The club sandwich must be inside the packing box. * The chocolate chip cookie must be inside the packing box. * The bottle of tea must be inside th...

  93. [101]

    However, the bottle of tea has not yet been placed inside the packing box

    **Analyze the current state:** Both apple halves, the club sandwich, and the chocolate chip cookie are already inside the packing box. However, the bottle of tea has not yet been placed inside the packing box

  94. [102]

    The only remaining unsatisfied condition is that the bottle of tea must be placed inside the packing box

    **Evaluate the progress:** This represents a progress of 3 out of 4, corresponding to 83.33% completion. The only remaining unsatisfied condition is that the bottle of tea must be placed inside the packing box

  95. [103]

    progress

    **Determine the next action:** The bottle of tea is inside the refrigerator, which is currently closed. Therefore, the immediate next step is to open the refrigerator door so the bottle can be retrieved. After placing the bottle of tea inside the packing box, the refrigerator ...

  96. [104]

    * Images 6-9 show a shower curtain and a shower head

    **Scan the images for a shower area:** * Images 1-5 show the toilet and part of the bathtub. * Images 6-9 show a shower curtain and a shower head. * Image 8 specifically shows a small glass shelf in the shower area. * Image 9 shows the shower head and the shelf again. * Image ...

  97. [105]

    * In Image 10, there are actually small bottles sitting on this shelf, confirming it’s used for toiletries like shampoo

    **Identify the specific location:** * Looking closely at Image 8, 9, and 10, there is a small, clear glass shelf attached to the wall inside the shower/tub area. * In Image 10, there are actually small bottles sitting on this shelf, confirming it’s used for toiletries like shampoo

  98. [106]

    * Describe its location (in the shower/tub area, on the wall)

    **Formulate the answer:** * State that there is a shelf. * Describe its location (in the shower/tub area, on the wall). * Mention that it can hold the bottle

  99. [107]

    You can place a bottle of shampoo on the small glass shelf located inside the shower area, attached to the wall

    **Refine the answer:** "You can place a bottle of shampoo on the small glass shelf located inside the shower area, attached to the wall." This is accurate based on the visual evidence. </think> You can place a bottle of shampoo on the small glass shelf located on the wall insi...

  100. [108]

    * **Silver pot:** This is the metal bowl/pot to the left of the plate

    **Identify the objects:** * **Yellow object:** This is the corn on the cob on the red plate. * **Silver pot:** This is the metal bowl/pot to the left of the plate

  101. [109]

    * It needs to grasp the yellow object

    **Determine the trajectory:** * The robot needs to approach the yellow object (corn). * It needs to grasp the yellow object. * It needs to lift the yellow object. * It needs to move the yellow object over the silver pot. * It needs to place the yellow object into the silver pot

  102. [110]

    Let’s say, near the corn

    **Select waypoints:** * **Point 1:** Initial approach to the yellow object. Let’s say, near the corn. [508, 682] * **Point 2:** Grasping the yellow object. [500, 690] * **Point 3:** Lifting the object slightly. [510, 600] * **Point 4:** Moving towards the pot. [400, 550] * **P...

  103. [111]

    Approach corn: [508, 682]

  104. [112]

    Grasp corn: [500, 690]

  105. [113]

    Lift corn: [505, 620]

  106. [114]

    Move towards pot (mid-air): [380, 500]

  107. [115]

    Position over pot: [300, 520]

  108. [116]

    Lower into pot: [285, 560]

  109. [117]

    Place inside: [285, 580]

  110. [118]

    point_2d

    Final placement: [290, 575] This looks like a reasonable 8-point trajectory. </think> ‘‘‘json [ {"point_2d": [508, 682], "label": "waypoint 1"}, {"point_2d": [505, 672], "label": "waypoint 2"}, {"point_2d": [502, 600], "label": "waypoint 3"}, {"point_2d": [425, 520], "label": ...

  111. [119]

    It looks like a small, rectangular component, possibly a stick of RAM or a similar computer part

    **Identify the device:** The device is the red object being held by the left gripper. It looks like a small, rectangular component, possibly a stick of RAM or a similar computer part

  112. [120]

    **Identify the left gripper:** This is the robotic arm on the left side of the image, holding the red device

  113. [121]

    **Identify the right gripper:** This is the robotic arm on the right side of the image

  114. [122]

    The most logical point for the right gripper to grasp is the center or a stable part of the red device itself

    **Determine the grasp point:** The right gripper needs to grasp the red device to take it from the left gripper. The most logical point for the right gripper to grasp is the center or a stable part of the red device itself

  115. [123]

    **Locate the red device:** It’s positioned roughly in the center, slightly below the horizontal midline, being held by the left gripper

  116. [124]

    point_2d

    **Select a point:** A point on the red device, specifically near where the right gripper would approach, is appropriate. The red object is centrally located between the two grippers. A point on the red object itself, perhaps slightly towards the right side of it to align with ...

  117. [125]

    The starting point should be near the bottom center

    **Identify the current position:** The image shows a gravel path leading towards a rock face with two tunnels. The starting point should be near the bottom center

  118. [126]

    Take the right tunnel

    **Identify the goal:** The task is to "Take the right tunnel". Looking at the rock face, there are two openings. The one on the right is the target

  119. [127]

    point_2d

    **Determine the path:** * The path starts on the gravel road. * The road curves slightly or leads straight towards the tunnels. * To enter the *right* tunnel, the person needs to aim for the opening on the right side of the rock face. * The path seems to split or widen slightl...

  120. [128]

    * At 00:06, the scale shows the weight

    **Identify the grape weighing scene:** * At 00:05, the person puts green grapes on the scale. * At 00:06, the scale shows the weight. * At 00:09, the person takes the grapes off the scale

  121. [129]

    * At 00:10, they put the grapes into the plastic bag

    **Observe the immediate next action:** * Right after taking the grapes off the scale (00:09), the person is seen holding a clear plastic bag. * At 00:10, they put the grapes into the plastic bag. * At 00:13, the person is holding the bag with the grapes inside. * At 00:14, the...

  122. [130]

    * They put them into a plastic bag

    **Synthesize the answer:** * The person takes the grapes off the scale. * They put them into a plastic bag. * They place the bag into their shopping basket

  123. [131]

    </think> The person puts the grapes into a plastic bag and places it in the shopping basket

    **Formulate the final response:** The person puts the grapes into a plastic bag and then places the bag into a shopping basket. </think> The person puts the grapes into a plastic bag and places it in the shopping basket. General:InfoVQA_VAL / 95160 The model locates the 2010–2...

    2010

  124. [132]

    GLOBAL INSTABILITY CAUSES UNPREDICTABLE PRICE SPIKES

    **Scan the infographic for sections related to price spikes.** I see a section titled "GLOBAL INSTABILITY CAUSES UNPREDICTABLE PRICE SPIKES". This looks very promising

  125. [133]

    Quarterly retail price per gallon of regular grade gas

    **Look at the chart in that section.** It’s a bar/area chart showing "Quarterly retail price per gallon of regular grade gas" from 1976 to 2012

    1976

  126. [134]

    **Identify the specific time period.** The user is asking about 2010-2012

    2010

  127. [135]

    **Locate the year 2010-2012 on the x-axis.** I see labels for 2010, 2011, 2012

    2010

  128. [136]

    Arab Spring

    **Look for labels pointing to spikes in that area.** There is an arrow pointing to a spike around 2011/2012 labeled "Arab Spring"

    2011

  129. [137]

    reason". The section title is

    **Verify the context.** The question asks for the "reason". The section title is "GLOBAL INSTABILITY CAUSES UNPREDICTABLE PRICE SPIKES". The specific label pointing to the 2010-2012 spike is "Arab Spring"

    2010

  130. [138]

    Arab Spring

    **Formulate the answer.** The user requested a single word or phrase. "Arab Spring" fits perfectly. </think> Arab Spring 35 D.1 Static QA Failure Cases The examples below show representative failures under the same static-QA evaluation interface. In each case, the rollout rema...

  131. [139]

    **Identify the oven door:** It’s the large black rectangular area on the left side, below the stovetop

  132. [140]

    Looking at the oven, there is a horizontal silver bar near the bottom edge of the door

    **Identify the handle:** The oven door typically has a handle used for opening. Looking at the oven, there is a horizontal silver bar near the bottom edge of the door

  133. [141]

    The handle is located roughly in the middle of the lower part of the door

    **Determine the force application point:** To open a standard oven door like this, one would pull the handle. The handle is located roughly in the middle of the lower part of the door

  134. [142]

    Where should I apply force to open the oven door?

    **Select a point:** A point on the handle or just below it on the door panel would be appropriate. The handle is clearly visible as a horizontal bar. Let’s pick a point on the handle itself or slightly below it on the door frame where the handle is attached. The handle is roug...

  135. [143]

    **Identify the object:** The object is the grey toy rabbit with white ears and a pink nose located near the bottom center of the image

  136. [144]

    on top of the drawer

    **Identify the target location:** The instruction says "on top of the drawer". Looking at the image, there is a transparent plastic surface in the bottom left corner which looks like the top of a drawer or a shelf. The rabbit is currently on the wooden table. I need to move it...

  137. [145]

    drawer" might refer to the white box-like structure on the left, but usually

    **Determine the point:** I need to pick a point on the transparent surface where the rabbit should be placed. A good spot would be roughly in the center of that visible transparent area. Looking at the bottom left, the transparent surface is visible. Let’s aim for a point on t...

  138. [146]

    on top of the drawer

    **Refine the point:** The rabbit is at [570, 680]. The transparent surface is below and to the left. A point like [200, 850] seems appropriate for "on top of the drawer". Let’s look again. Is there another drawer? The white box on the left has a red stripe. It looks like a con...

  139. [147]

    on top of the drawer

    **Final Point Selection:** I will select a point on the transparent surface visible in the bottom left corner. Let’s say around x=200, y=850. Wait, looking at the coordinates, y increases downwards. So y=850 is near the bottom edge. x=200 is towards the left. This seems like a...

  140. [148]

    Phase One

    **Locate "Phase One"**: I need to scan the image for the text "PHASE ONE". I see it in the upper- left quadrant of the circular timeline

  141. [149]

    PHASE ONE

    **Understand the layout**: The timeline is circular, but there’s a specific section labeled "PHASE ONE". It seems to flow clockwise starting from the top

  142. [150]

    IRON MAN

    **Identify the first movie**: Looking at the start of the Phase One section, there’s a circle labeled "IRON MAN" with the Roman numeral "I" below it. Next to it, there’s text saying "2008 Iron Man"

    2008

  143. [151]

    MOVIES CHRONOLOGICAL ORDER (roman numbers)

    **Verify chronological order**: The text explicitly mentions "MOVIES CHRONOLOGICAL ORDER (roman numbers)". The first movie has the Roman numeral "I". This confirms "Iron Man" is the first one

  144. [152]

    Iron Man

    **Formulate the answer**: The question asks for a single word or phrase. "Iron Man" is the correct answer. </think> Iron Man 37

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.