Pith. sign in

REVIEW 5 major objections 6 minor 272 references

RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation

T0 review · 5 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that dense, per-frame intermediate representations—subtasks, object and gripper boxes, affordances, grasp poses, motion traces—produced at scale with human review, improve embodied reasoning, action execution, and world-mod

desk verdict A large, genuinely useful dataset and a plausible world-model conditioning recipe; the headline claims run ahead of the evidence because annotation accuracy is unaudited and the strongest world-model numbers are oracle-conditioned. read the letter →

arxiv 2607.18709 v2 pith:QU7QVVS7 submitted 2026-07-21 cs.RO

Ziqin Wang , Hao Li , Weijun Wang , Junhao Cai , Jia Zeng , Yilun Chen , Jiangmiao Pang , Si Liu This is my paper · ORCID
classification cs.RO
keywords intermediaterepresentationsroboticmanipulationworldmodelsvision-language-actionembodiedVQAdatasetannotationplan-then-executevideodiffusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RoboInter1.5 is a bid to make intermediate representations—structured labels that sit between raw video and raw actions—the load-bearing interface for robot learning. The paper assembles 230k manipulation episodes annotated per frame with more than ten label types, then trains three families of models on them: a VLM planner for spatial and temporal question answering, a plan-then-execute VLA for control, and a video-diffusion world model for future-frame prediction. Its central claim is that these labels are not just interpretable side information but actively regularize low-level actions and constrain world-model rollouts. Evidence includes a world-model PSNR jump from 18.26 to 21.05 when action conditioning is replaced by intermediate representations, and closed-loop out-of-distribution success rising from 38.3% to 58.3% when the executor is initialized from the planner. A sympathetic reader would say the work is trying to establish that one shared annotation schema can simultaneously strengthen reasoning, control, and simulation.

What carries the argument

The central object is RoboInter-Data, a per-frame annotated corpus of 230k manipulation episodes with ten-plus label types—subtasks, primitive skills, object and gripper boxes, segmentation masks, affordances, grasp poses, contact points, motion traces—all synchronized with actions and two camera views. Around it sit three constructions that carry the argument: RoboInter-VQA, which converts those labels into roughly 2.2 million spatial and temporal QA pairs; F-CoT, a flexible chain-of-thought that feeds planner outputs into the executor in textual or visual form; and RoboInter-CV, which renders object-point trajectories and gripper traces onto a blank canvas to serve as visually encoded cont

What would settle it

Take a random sample of, say, 1,000 episodes and have two independent annotators re-label all ten annotation types with the same tool; if agreement on grasp poses, affordances, and contact frames is near chance, the benchmark gains are likely inflated by label bias. Similarly, a held-out set labeled entirely by a second independent team would settle whether the VQA and world-model gains replicate.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a unified suite of dense intermediate representations, synchronized with executed actions across 230k episodes and 571 scenes, makes embodied models better at all three things they are asked to do: understand manipulation scenes, execute manipulation policies, and simulate future world states. The strongest evidence is in the world model: conditioning a 14B video-diffusion model on rendered segmentation masks and gripper traces instead of raw action sequences raises PSNR from 18.26 to 21.05 and cuts LPIPS from 0.171 to 0.102, and the gain grows with prediction horizon. On the control side, decoupling planning from execution and feeding the exec

Load-bearing premise

The result depends on the automatically generated, human-reviewed labels being accurate enough to serve as ground truth; the paper reports no inter-annotator agreement or audited error rates, so every downstream gain inherits whatever noise and bias the annotation pipeline has.

Editorial extensions

If this is right

  • If the central claim holds, future manipulation datasets should include dense per-frame labels as a standard component rather than just instructions and actions.
  • Intermediate-conditioned world models should replace action-only conditioning for long-horizon simulation; the fidelity gap widens as prediction horizon grows.
  • Explicit, decoupled plan-then-execute architectures are likely to beat implicit end-to-end designs on out-of-distribution tasks, because they give the executor actionable geometric priors.
  • The planner trained on this corpus transfers to external embodied reasoning benchmarks, suggesting the learned representations generalize beyond the original scenes.
  • World-model-predicted latent features can serve as actionable inputs to a VLA policy, narrowing the gap to ground-truth latent features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same annotation schema could serve as a shared token vocabulary across embodiments; if traces, boxes, and affordances are defined in image space, a policy trained on one robot may transfer to another with only the low-level executor retrained.
  • Editorial inference: the paper's noise-injection training on intermediate controls hints that planner-generated imperfect controls are enough; a testable extension is to measure how much annotation noise the system tolerates before world-model gains disappear.
  • Editorial inference: because the world model accepts rendered control videos, it may be possible to optimize plans by back-propagating through the world model over the control video, effectively turning the planner into a differentiable simulator—this is not explored in the paper.
  • Editorial inference: if annotation errors are systematic (for example, biased grasp poses), the reported gains partly measure the annotation pipeline rather than the representations themselves; independent audits would separate those effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents RoboInter1.5, a suite of dense, per-frame intermediate representations for robotic manipulation, built on a 230k-episode dataset with ten-plus annotation types, a VQA benchmark (RoboInter-VQA), a VLM planner (RoboInter-VLM), plan-then-execute VLA variants (RoboInter-VLA), and an intermediate-representation-conditioned world model (RoboInter-World). The central claim is that these dense, human-verified intermediate representations improve embodied reasoning, action execution, and world-model fidelity, with the strongest reported results being large VQA gains over zero-shot generalists, an OOD closed-loop success improvement from 38.3% to 58.3% on a Franka arm, and a world-model PSNR improvement from 18.26 to 21.05 when conditioning on rendered traces and masks instead of raw actions.

Significance. If the annotation quality and the empirical gains hold, this is a valuable resource and a useful conceptual contribution: it unifies reasoning, control, and world modeling around a single intermediate-representation interface, and it reports results on third-party benchmarks, a real-robot closed-loop study, and oracle-versus-planner control protocols. Strengths include the breadth of annotation types, the explicit construction of control videos, multiple VLA paradigms, and evaluation across model scales. However, the load-bearing premise—that the automatically generated, human-reviewed labels are accurate enough to serve as ground truth for 230k episodes—is not directly measured, and several headline numerical claims lack error bars or significance tests. The resource is likely to be useful regardless, but the strength of the as-stated claims exceeds what the evidence currently supports.

major comments (5)
  1. [§3.1, Fig. 2, A.2] The dataset-quality claim rests on an unmeasured annotation error rate. The pipeline combines SAM2 tracking, estimated calibration matrices, gripper detection, and ChatGPT pre-annotations, with human review described only as 'inspection'. No inter-annotator agreement, audit subset, or per-type error statistics are reported; calibration details are deferred to RoboInter1.0. Since grasp boxes, contact points, traces, and placement proposals all derive from the 2D end-effector at the contact frame, calibration or frame-offset errors propagate into every downstream annotation. All VQA, OLS, PSNR, and closed-loop results are evaluated on splits from the same annotated corpora, so unknown label noise is confounded with the claimed benefits. Please provide an independent audit: a random sample of episodes re-annotated by multiple annotators, with per-type agreement/error rates and a calibration
  2. [§5.3, Fig. 7] The closed-loop claim that RoboInter-IC-E2E improves OOD success from 38.3% to 58.3% is reported without error bars, confidence intervals, trial-level logs, or statistical tests. The caption says results come from 15 ID and 15 OOD trials per task; with four tasks and binary outcomes, the standard error of a 58.3% success proportion from 15 trials is about 12.7 percentage points, so the 20-point gap is not obviously significant. Please report exact trial counts, per-task Wilson intervals, and a significance test (e.g., permutation or Fisher's exact test) over the full trial set.
  3. [§5.5, Table 10] The claim that RoboInter-World latent features yield 'consistent and substantial gains' is not supported at the 55K training step: OLS@0.03 improves from 22.09 to 22.17 (+0.08 percentage points) and OLS@0.05 from 35.74 to 35.97 (+0.23). These differences are within plausible noise, especially since no repeated seeds or confidence intervals are reported, and the I2V-baseline comparison itself changes sign across thresholds. Please report seed variance, confidence intervals, or additional checkpoints, and soften the conclusion accordingly.
  4. [§3.2, §5.1, Table 3] The RoboInter-VQA benchmark is constructed from the same annotation pipeline that trains RoboInter-VLM, so the large margins over zero-shot generalists (e.g., 76.1% vs 46.6% on object grounding) are partly by construction: the model is fine-tuned on the exact annotation schema used to generate the test questions. This does not invalidate the benchmark, but it means Table 3 does not independently validate annotation quality or generalization to a different annotation distribution. A human ceiling, an external annotation benchmark, or a cross-corpus test split would strengthen the claim that the representations themselves are accurate and transferable.
  5. [§5.4, Tables 6–9] World-model results are reported as single numbers without error bars or repeated-seed variance. Many comparisons are small (e.g., Table 8: Seg 20.38 vs Seg+Trace 20.43 at 1.3B; Table 9: planner-control 20.17 vs action 18.26 at 14B but with no spread). The large PSNR gains from Inter over Action are encouraging, but the manuscript should state the number of seeds, standard deviations, or confidence intervals so the reader can distinguish real improvements from checkpoint or tuning noise.
minor comments (6)
  1. [Table 2] The column header 'CO-CO' appears to be a typo for 'COCO'; please check the benchmark name and caption.
  2. [Table 3 caption] The caption says 'ACC@IOU>0.1' for spatial generation, but several entries are '–' and the threshold is not applied consistently across model rows. Clarify the evaluation protocol for missing entries.
  3. [Figure 5] The x-axis is labeled 'Training Steps (log scale)' with ticks at 1k, 5k, 10k, and 20k, but the text says curves run to 40k steps. Add the 40k tick or adjust the caption.
  4. [A.2] The appendix states 'For detailed appendix content, please refer to RoboInter1.0.' Since the present paper is presented as a standalone suite, essential details such as calibration estimation, annotation tool workflows, and prompt templates should either be included or the dependency on the previous report should be made explicit in the main text.
  5. [§4.3, Eq. (2)] In Eq. (2), the flow-matching target v_t and the clean future latent y are used without explicit definitions. Please define them and state the noise schedule used.
  6. [§3.3] RoboInter-CV is described as containing 65k clip-level samples from 16.9k episodes, but RoboInter-Data contains 230k episodes. Clarify what fraction of episodes survives the filtering and why; this affects the representativeness of the world-model training set.

Circularity Check

1 steps flagged · score 4.0 of 10

Oracle-control world-model evaluation partially answers itself; core claims still have independent support.

  1. self definitional [Section 4.3 (Control Protocol; Eqs. 1-2), Table 6]
    "In the oracle-control protocol, control videos are rendered from ground-truth annotations, measuring the upper-bound utility of high-quality intermediate representations."

    Under Eqs. (1)-(2), the control video u is rendered from future-annotation traces and masks, while the optimization target y is the future video latent of the same episode. The oracle-conditioned 'prediction' therefore has the future's spatial structure already in its input; the reported 18.26→21.05 PSNR gain over action-only conditioning partly reflects this leak. The paper transparently labels this an upper bound and supplies a planner-control variant (Table 9) that also beats action-only, so the circularity is partial and not the sole basis of the world-model claim.

full rationale

The suite's central claims are not globally circular. RoboInter-Data is a dataset resource; its annotations are produced by the described semi-automatic pipeline, not derived from the downstream models. The VLM is evaluated both on RoboInter-VQA (a held-out split of the same annotation pool, which is a standard supervised evaluation rather than a circular reduction) and on third-party benchmarks (Table 2: Where2Place, RoboRefIt, RoboVQA, RefCOCO), which provide external signal. The VLA closed-loop gains are measured on real-world ID/OOD tasks against OpenVLA/Pi-0 baselines, and the world-model improvement is reproduced under planner-control (Table 9), where the controls are predicted rather than taken from ground truth. The main caveats are auditability, not circularity: §3.1 reports no inter-annotator agreement or per-type error rates, and A.2 defers substantial methodology to the self-cited RoboInter1.0 report; these undermine verification of the 'high fidelity' claim but do not equate the derivations to their inputs. The one by-construction element is the oracle-control protocol, scored above.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on one big empirical bet — that the annotation pipeline yields accurate ground truth at 230k-episode scale — plus standard ML training assumptions. There are no derived physical parameters; free parameters are hand-chosen filtering thresholds, sampling densities, evaluation thresholds, and an unspecified perturbation schedule. No new physical entities are postulated; the 'control video' is a rendering of existing annotations, not a new theoretical entity.

free parameters (4)
  • RoboInter-CV clip-filtering thresholds
    Frame-level validity masks and discard rules (non-trivial object mask, non-empty trace, no valid anchor frame, overly short sequences) are hand-chosen; they determine which 65k clips enter world-model training.
  • Control-video point sampling density
    'Uniformly sampling points within object bounding boxes or segmentation masks' — the density is unspecified and directly affects conditioning strength.
  • OLS error thresholds (@0.1, @0.05, @0.03, @0.01)
    Open-loop score thresholds are arbitrary discretizations; conclusions shift across thresholds (e.g., @0.01 flips some rankings), and the mean over them is the headline.
  • Stochastic perturbation schedule for planner-control robustness
    Introduced to close the train/test gap between teacher-forced and planner-generated controls; strength and timing are unspecified and not ablated.
assumptions (4)
  • domain assumption Human-reviewed automated annotations (SAM2 tracking, estimated calibration, ChatGPT pre-annotation) are accurate at 230k-episode scale
    §3.1 'Annotations & Check with RoboInter-Tool'; no inter-annotator agreement, audit subset, or error-rate statistics are provided.
  • domain assumption OpenX/RH20T/DROID teleoperated episodes provide sufficient diversity for the generalization claims
    The dataset is assembled from these corpora; 'zero-shot' claims are evaluated on held-out episodes from the same sources (§5.4, Figure 8).
  • standard math Standard VLM, VAE, and diffusion training assumptions hold (cross-entropy, flow-matching objective)
    Eq. (1)–(2), §4.3; no new mathematical machinery is introduced.
  • domain assumption Open-loop score correlates with closed-loop success
    The paper itself documents an inversion: EC-E2E beats IC-E2E open-loop but loses in-distribution closed-loop (§5.2 vs §5.3), weakening this proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation." pith.science (2026). https://pith.science/paper/QU7QVVS7

@misc{pith2026260718709,
  author       = {Pith},
  title        = {Pith review of: RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QU7QVVS7}},
  note         = {Machine review of arXiv:2607.18709}
}
read the original abstract

Existing robot datasets remain expensive to curate, embodiment-specific, and insufficiently annotated with the fine-grained structure required for generalizable reasoning, execution, or long-horizon environment dynamics simulation. Building on our prior work, RoboInter1.0, we present RoboInter1.5, an extended and holistic suite of intermediate representations for both robotic manipulation and embodied world modeling. RoboInter1.5 provides a unified resource of data, benchmarks, and models centered on dense manipulation-oriented intermediate representations. Specifically, RoboInter-Data contains over 230k manipulation episodes across 571 scenes with dense per-frame annotations covering more than ten types of intermediate representations, including subtasks, primitive skills, object and gripper grounding, segmentation, affordance, grasp poses, contact points, motion traces, etc. Built upon these annotations, RoboInter-VQA introduces spatial and temporal embodied VQA tasks to benchmark and improve the intermediate-representation reasoning capabilities of our RoboInter-VLM. RoboInter-VLA further studies how such representations benefit action execution through implicit, explicit, and modular plan-then-execute paradigms. To better model the physical world, we further introduce RoboInter-World, which leverages intermediate representations as structured conditioning signals for controllable prediction of future world states. Extensive evaluations demonstrate that RoboInter1.5 provides a unified spatiotemporal scaffolding for intermediate representations. Rather than treating intermediate representations merely as interpretable signals, RoboInter1.5 conceptualizes them as a bidirectional interface that both regularizes low-level action spaces and constrains the latent rollouts of open-world physical simulators.

Figures

Figures reproduced from arXiv: 2607.18709 by the authors.

Figure 1
Figure 1. RoboInter1.5 manipulation suite includes annotation tools, annotated data, a curated VQA dataset, and their applications in VLMs, VLAs, and World Models. RoboInter provides a dataset with over 230k episodes and 10+ types of intermediate representation annotations, named RoboInter￾Data; VLMs trained on the curated embodied VQA dataset covering 29 spatial- and temporal-level categories, named RoboInter-VLM; an integra… view at source ↗
Figure 2
Figure 2. Overview of RoboInter-Data and RoboInter-VQA. We annotate 230k manipulation episodes with 10 types of intermediate representation annotations through Data Collection and Annotation & Check. A large-scale, diverse set of VQA spanning spatial and temporal dimensions is further constructed. Statistics of raw episodes and curated VQA are also provided. world settings. In addition, we study world modeling based on interm… view at source ↗
Figure 3
Figure 3. Framework of RoboInter-VLA. Our model follows a plan-then-execute paradigm with a VLM-based Planner and an Executor. The Planner exhibits enhanced understanding and generation for manipulation, strong general grounding abilities, and robust perception across diverse scenes. The Executor shares the VLM backbone with the Planner. Three variants are supported, and intermediate representations in Flexible Chain-of-Thoug… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Overview of RoboInter-World. The mask latent is employed to encode temporal information, explicitly distinguishing between historical/current states and future contents. The control video is encoded into control latents, which are subsequently concatenated with the inp…
Figure 5
Figure 5. Figure 5: Open-loop evaluation in TableTop setting. We show the curve of OLS@0.05 from 1k to 40k training steps. We mainly report the five variances of RoboInter-VLA. any pretrained VLM from Planner, performing action learning only; (b-e).RoboInter-IC-E2E, EC-E2E, Te-Modular and…
Figure 6
Figure 6. Figure 6: Inference cases. For the Visual Motion Planning, RoboInter-VLM accurately infers the ac￾tion direction of the gripper, demonstrating a strong understanding of spatial relations. For the Subtask Planning, given a video input, RoboInter-VLM can reason about the underlyin…
Figure 7
Figure 7. Figure 7: Real-World experiments. The top charts present results from 15 in-distribution (ID) and 15 out-of-distribution (OOD) trials. The bottom panel illustrates the OOD test setup. Notably, the performance drop from ID to OOD reflects each model’s generalization under distrib…
Figure 8
Figure 8. Figure 8: Qualitative comparison in two representative zero-shot, long-horizon scenarios from the [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Additional real-world ID and OOD validation. (Left) Tool Inserting: a precision ID task requiring accurate contact handling and slot alignment. (Right) Object Sorting: an OOD task that tests language-guided generalization using novel objects and containers. To further …
Figure 10
Figure 10. Figure 10: Chain of thought of RoboInter-VLA. Two representative examples (used in RoboInter-EC￾E2E and RoboInter-Modular) are shown. The upper example involves a sequence of manipulation behaviors, including picking, placing, and pushing, while the lower one focuses on a pick-a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

272 extracted references · 134 linked inside Pith

  1. [5]

    Q. Bu, J. Cai, L. Chen, X. Cui, Y. Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025

    arXiv 2025

  2. [6]

    J. Cen, C. Yu, H. Yuan, Y. Jiang, S. Huang, J. Guo, X. Li, Y. Song, H. Luo, F. Wang, D. Zhao, and H. Chen. Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.21539, 2025. URL https://arxiv.org/abs/2506.21539

    arXiv 2025

  3. [7]

    X. Chen, Y. Chen, Y. Fu, N. Gao, J. Jia, W. Jin, H. Li, Y. Mu, J. Pang, Y. Qiao, et al. Internvla-m1: A spatially guided vision-language-action framework for generalist robot policy. arXiv preprint arXiv:2510.13778, 2025

    arXiv 2025

  4. [8]

    Z. Chen, Z. Shi, X. Lu, L. He, S. Qian, Z. Yin, W. Ouyang, J. Shao, Y. Qiao, C. Lu, et al. Rh20t-p: A primitive-level robotic dataset towards composable generalization agents. arXiv preprint arXiv:2403.19622, 2024

    arXiv 2024

  5. [9]

    R. Chu, Y. He, Z. Chen, S. Zhang, X. Xu, B. Xia, D. Wang, H. Yi, X. Liu, H. Zhao, et al. Wan-move: Motion-controllable video generation via latent trajectory guidance. arXiv preprint arXiv:2512.08765, 2025

    arXiv 2025

  6. [10]

    R. Chu, Y. He, Z. Chen, S. Zhang, X. Xu, D. WANG, H. Yi, X. Liu, H. Zhao, Y. Liu, et al. Wan-move: Motion-controllable video generation via latent trajectory guidance. Advances in Neural Information Processing Systems, 38: 0 404--432, 2026

    2026

  7. [11]

    R. Dang, J. Guo, B. Hou, S. Leng, K. Li, X. Li, J. Liu, Y. Mao, Z. Wang, Y. Yuan, et al. Rynnbrain: Open embodied foundation models. arXiv preprint arXiv:2602.14979, 2026

    arXiv 2026

  8. [12]

    S. Deng, M. Yan, S. Wei, H. Ma, Y. Yang, J. Chen, Z. Zhang, T. Yang, X. Zhang, H. Cui, et al. Graspvla: a grasping foundation model pre-trained on billion-scale synthetic action data. arXiv preprint arXiv:2505.03233, 2025

    arXiv 2025

Show all 272 references
  1. [13]

    Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text‐guided video generation. arXiv preprint arXiv:2302.00111, 2023. URL https://arxiv.org/abs/2302.00111

    2023 arXiv

  2. [14]

    A. O. et al. Open X-E mbodiment: Robotic learning datasets and RT-X models. https://arxiv.org/abs/2310.08864, 2023

    2023 arXiv

  3. [15]

    H. Fang, M. Zhang, H. Dong, W. Li, Z. Wang, Q. Zhang, X. Tian, Y. Hu, and H. Li. Robix: A unified model for robot interaction, reasoning and planning. arXiv preprint arXiv:2509.01106, 2025

    2025 arXiv

  4. [18]

    C. Gao, Z. Liu, Z. Chi, J. Huang, X. Fei, Y. Hou, Y. Zhang, Y. Lin, Z. Fang, Z. Jiang, et al. Vla-os: Structuring and dissecting planning representations and paradigms in vision-language-action models. arXiv preprint arXiv:2506.17561, 2025 a

    2025 arXiv

  5. [19]

    N. Gao, Y. Chen, S. Yang, X. Chen, Y. Tian, H. Li, H. Huang, H. Wang, T. Wang, and J. Pang. Genmanip: Llm-driven simulation for generalizable instruction-following manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12187--...

    2025

  6. [20]

    S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W.-C. Tseng, Y. Dong, K. Mo, C.-H. Lin, et al. Dreamdojo: A generalist robot world model from large-scale human videos. arXiv preprint arXiv:2602.06949, 2026

    2026 arXiv

  7. [22]

    P. Guo, X. Wei, Y. Hu, B. Yang, D. Liu, F. Huang, et al. Emma-x: an em-like multilingual pre-training algorithm for cross-lingual representation learning. Advances in Neural Information Processing Systems, 36: 0 10116--10144, 2023

    2023

  8. [23]

    Y. Guo, L. X. Shi, J. Chen, and C. Finn. Ctrl-world: A controllable generative world model for robot manipulation. arXiv preprint arXiv:2510.10125, 2025

    2025 arXiv

  9. [24]

    C. Hou, K. Wu, J. Liu, Z. Che, D. Wu, F. Liao, G. Li, J. He, Q. Feng, Z. Jin, et al. Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence. arXiv preprint arXiv:2512.24653, 2025

    2025

  10. [26]

    Huang, X

    H. Huang, X. Chen, Y. Chen, H. Li, X. Han, Z. Wang, T. Wang, J. Pang, and Z. Zhao. Roboground: Robotic manipulation with grounded vision-language priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22540--22550, 2025

    2025

  11. [27]

    Huang, I

    S. Huang, I. Ponomarenko, Z. Jiang, X. Li, X. Hu, P. Gao, H. Li, and H. Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), ...

    2024

  12. [30]

    C.-Y. Hung, N. Majumder, H. Deng, L. Renhang, Y. Ang, A. Zadeh, C. Li, D. Herremans, Z. Wang, and S. Poria. Nora-1.5: A vision-language-action model trained using world model-and action-based preference rewards. arXiv preprint arXiv:2511.14659, 2025

    2025

  13. [31]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. pi-0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025

    2025 arXiv

  14. [32]

    Intelligence, B

    P. Intelligence, B. Ai, A. Amin, R. Aniceto, A. Balakrishna, G. Balke, K. Black, G. Bokinsky, S. Cao, T. Charbonnier, et al. _ 0.7 : a steerable generalist robotic foundation model with emergent capabilities. arXiv preprint arXiv:2604.15483, 2026

    2026 arXiv

  15. [33]

    Y. Ji, H. Tan, J. Shi, X. Hao, Y. Zhang, H. Zhang, P. Wang, M. Zhao, Y. Mu, P. An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1724--1734, 2025

    2025

  16. [36]

    M. J. Kim, Y. Gao, T.-Y. Lin, Y.-C. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M.-Y. Liu, C. Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163, 2026

    2026 arXiv

  17. [37]

    L. Kou, F. Ni, Y. Zheng, P. Han, J. Liu, H. Cui, R. Liu, and J. Hao. Roboannotatorx: A comprehensive and universal annotation framework for accurate understanding of long-horizon robot demonstration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p...

    2025

  18. [38]

    Kuang, S

    Z. Kuang, S. Cai, H. He, Y. Xu, H. Li, L. J. Guibas, and G. Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. Advances in Neural Information Processing Systems, 37: 0 16240--16271, 2024

    2024

  19. [40]

    H. Li, Y. Li, Y. Yang, J. Cao, Z. Zhu, X. Cheng, and L. Chen. Dispose: Disentangling pose guidance for controllable human image animation. arXiv preprint arXiv:2412.09349, 2024 b

    2024 arXiv

  20. [43]

    L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, et al. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998, 2026 b

    2026 arXiv

  21. [45]

    Q. Li, Z. Xing, R. Wang, H. Zhang, Q. Dai, and Z. Wu. Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12112--12123, 2025 b

    2025

  22. [47]

    X. Li, X. He, L. Zhang, M. Wu, X. Li, and Y. Liu. A comprehensive survey on world models for embodied ai. arXiv preprint arXiv:2510.16732, 2025 c

    2025 arXiv

  23. [48]

    Y. Li, Y. Deng, J. Zhang, J. Jang, M. Memmel, R. Yu, C. R. Garrett, F. Ramos, D. Fox, A. Li, et al. Hamster: Hierarchical action models for open-world robot manipulation. arXiv preprint arXiv:2502.05485, 2025 d

    2025 arXiv

  24. [49]

    Liang, Y

    J. Liang, Y. Fan, K. Zhang, R. Timofte, L. Van Gool, and R. Ranjan. Movideo: Motion-aware video generation with diffusion model. In European conference on computer vision, pages 56--74. Springer, 2024

    2024

  25. [50]

    Y. Liao, P. Zhou, S. Huang, D. Yang, S. Chen, Y. Jiang, Y. Hu, J. Cai, S. Liu, J. Luo, et al. Genie envisioner: A unified world foundation platform for robotic manipulation. arXiv preprint arXiv:2508.05635, 2025

    2025 arXiv

  26. [51]

    F. Lin, R. Nai, Y. Hu, J. You, J. Zhao, and Y. Gao. Onetwovla: A unified vision-language-action model with adaptive reasoning. arXiv preprint arXiv:2505.11917, 2025. URL https://arxiv.org/abs/2505.11917

    2025

  27. [52]

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll \'a r, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740--755. Springer, 2014

    2014

  28. [53]

    F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024

    2024

  29. [54]

    Y. Lou, Y. Ye, Y. Fu, J. Cen, X. Chi, Y. Lyu, P. Jia, S. Han, Z. Lu, and S. Zhang. Dream-tac: A unified tactile world action model for contact-rich robot manipulation. arXiv preprint arXiv:2606.08737, 2026

    2026 arXiv

  30. [55]

    G. Lu, W. Guo, C. Zhang, Y. Zhou, H. Jiang, Z. Gao, Y. Tang, and Z. Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719, 2025 a

    2025 arXiv

  31. [56]

    G. Lu, B. Jia, P. Li, Y. Chen, Z. Wang, Y. Tang, and S. Huang. Gwm: Towards scalable gaussian world models for robotic manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9263--9274, 2025 b

    2025

  32. [57]

    Y. Lu, Y. Fan, B. Deng, F. Liu, Y. Li, and S. Wang. Vl-grasp: a 6-dof interactive grasp policy for language-oriented objects in cluttered indoor scenes. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 976--983. IEEE, 2023

    2023

  33. [58]

    G. Luo, G. Yang, Z. Gong, G. Chen, H. Duan, E. Cui, R. Tong, Z. Hou, T. Zhang, Z. Chen, et al. Visual embodied brain: Let multimodal large language models see, think, and control in spaces. arXiv preprint arXiv:2506.00123, 2025 a

    2025 arXiv

  34. [59]

    Luo, C.-K

    Y. Luo, C.-K. Fan, M. Dong, J. Shi, M. Zhao, B.-W. Zhang, C. Chi, J. Liu, G. Dai, R. Zhang, et al. Robobench: A comprehensive evaluation benchmark for multimodal large language models as embodied brain. arXiv preprint arXiv:2510.17801, 2025 b

    2025 arXiv

  35. [60]

    Q. Lv, W. Kong, H. Li, J. Zeng, Z. Qiu, D. Qu, H. Song, Q. Chen, X. Deng, and J. Pang. F1: A vision-language-action model bridging understanding and generation to actions. arXiv preprint arXiv:2509.06951, 2025

    2025 arXiv

  36. [61]

    H. Ma, J. Cai, X. Xu, H. Li, Y. Yang, Y. Tian, J. Cao, H. Zhu, Z. Qiu, Y. Yang, et al. Internvla-a1. 5: Unifying understanding, latent foresight, and action for compositional generalization. arXiv preprint arXiv:2607.04988, 2026

    2026 arXiv

  37. [62]

    Morimitsu, X

    H. Morimitsu, X. Zhu, R. M. Cesar, X. Ji, and X.-C. Yin. Dpflow: Adaptive optical flow estimation with a dual-pyramid framework. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17810--17820, 2025

    2025

  38. [64]

    D. Niu, Y. Sharma, G. Biamby, J. Quenum, Y. Bai, B. Shi, T. Darrell, and R. Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406.11815, 2024

    2024 arXiv

  39. [67]

    J. Pai, L. Achenbach, V. Montesinos, B. Forrai, O. Mees, and E. Nava. mimic-video: Video-action models for generalizable robot control beyond vlas. arXiv preprint arXiv:2512.15692, 2025

    2025 arXiv

  40. [68]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195--4205, 2023

    2023

  41. [69]

    Psiris, V

    A. Psiris, V. Argyriou, E. K. Markakis, P. Sarigiannidis, E. Gavves, K. Bekris, et al. Foundation models in robotics: A comprehensive review of methods, models, datasets, challenges and future research directions. arXiv preprint arXiv:2604.15395, 2026

    2026 arXiv

  42. [71]

    Sermanet, T

    P. Sermanet, T. Ding, J. Zhao, F. Xia, D. Dwibedi, K. Gopalakrishnan, C. Chan, G. Dulac-Arnold, S. Maddineni, N. J. Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 645--652. I...

    2024

  43. [74]

    Sundaresan, S

    P. Sundaresan, S. Belkhale, D. Sadigh, and J. Bohg. Kite: Keypoint-conditioned policies for semantic manipulation. arXiv preprint arXiv:2306.16605, 2023

    2023 arXiv

  44. [75]

    H. Tan, E. Zhou, Z. Li, Y. Xu, Y. Ji, X. Chen, C. Chi, P. Wang, H. Jia, Y. Ao, et al. Robobrain 2.5: Depth in sight, time in mind. arXiv preprint arXiv:2601.14352, 2026

    2026

  45. [76]

    B. R. Team, M. Cao, H. Tan, Y. Ji, M. Lin, Z. Li, Z. Cao, P. Wang, E. Zhou, Y. Han, et al. Robobrain 2.0 technical report. arXiv preprint arXiv:2507.02029, 2025 a

    2025 arXiv

  46. [77]

    G. Team, A. Ye, B. Wang, C. Ni, G. Huang, G. Zhao, H. Li, J. Zhu, K. Li, M. Xu, et al. Gigaworld-0: World models as data engine to empower embodied ai. arXiv preprint arXiv:2511.19861, 2025 b

    2025

  47. [78]

    G. Team, B. Wang, B. Li, C. Ni, G. Huang, G. Zhao, H. Li, J. Li, J. Lv, J. Liu, et al. Gigabrain-0.5 m*: a vla that learns from world model-based reinforcement learning. arXiv preprint arXiv:2602.12099, 2026

    2026

  48. [79]

    G. D. Team. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. URL https://arxiv.org/abs/2312.11805

    2023 arXiv

  49. [80]

    T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  50. [81]

    L. Wang, X. Chen, J. Zhao, and K. He. Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers. Advances in Neural Information Processing Systems, 37: 0 124420--124450, 2024 a

    2024

  51. [83]

    K. Wu, C. Hou, J. Liu, Z. Che, X. Ju, Z. Yang, M. Li, Y. Zhao, Z. Xu, G. Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024

    2024 arXiv

  52. [84]

    Z. Wu, Y. Zhou, X. Xu, Z. Wang, and H. Yan. Momanipvla: Transferring vision-language-action models for general mobile manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1714--1723, 2025

    2025

  53. [85]

    M. Xu, Z. Xu, Y. Xu, C. Chi, G. Wetzstein, M. Veloso, and S. Song. Flow as the cross-domain manipulation interface. In 8th Annual Conference on Robot Learning, 2024

    2024

  54. [87]

    H. Xue, Y. Ge, Y. Zeng, Z. Li, M.-Y. Liu, Y. Chen, and J. Fan. Point-it-out: Benchmarking embodied reasoning for vision language models in multi-stage visual grounding. arXiv preprint arXiv:2509.25794, 2025

    2025

  55. [89]

    S. Yang, L. Xu, H. Li, J. Mu, J. Zeng, D. Lin, and J. Pang. Robo3r: Enhancing robotic manipulation with accurate feed-forward 3d reconstruction. arXiv preprint arXiv:2602.10101, 2026

    2026 arXiv

  56. [91]

    S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026 b

    2026 arXiv

  57. [92]

    Y. Ye, J. Ma, J. Cen, and Z. Lu. Token expand-merge: Training-free token compression for vision-language-action models. IEEE Robotics and Automation Letters, 11 0 (8): 0 9279--9286, 2026 c . doi:10.1109/LRA.2026.3703265

    2026

  58. [93]

    W. Yuan, J. Duan, V. Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024

    2024 arXiv

  59. [95]

    Zhang, K

    J. Zhang, K. Pertsch, J. Zhang, and J. J. Lim. Sprint: Scalable policy pre-training via language instruction relabeling. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 9168--9175. IEEE, 2024

    2024

  60. [96]

    Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1702--1713, 2025

    2025

  61. [97]

    E. Zhou, J. An, C. Chi, Y. Han, S. Rong, C. Zhang, P. Wang, Z. Wang, T. Huang, L. Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. arXiv preprint arXiv:2506.04308, 2025 a

    2025

  62. [98]

    Z. Zhou, Y. Zhu, M. Zhu, J. Wen, N. Liu, Z. Xu, W. Meng, R. Cheng, Y. Peng, C. Shen, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-action model. arXiv preprint arXiv:2502.14420, 2025 b

    2025 arXiv

  63. [99]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  64. [100]

    arXiv preprint arXiv:2406.08545 , year=

    RVT-2: Learning Precise Manipulation from Few Demonstrations , author=. arXiv preprint arXiv:2406.08545 , year=

    arXiv

  65. [101]

    Conference on Robot Learning , pages=

    Transporter networks: Rearranging the visual world for robotic manipulation , author=. Conference on Robot Learning , pages=. 2021 , organization=

    2021

  66. [102]

    arXiv preprint arXiv:2210.03094 , volume=

    Vima: General robot manipulation with multimodal prompts , author=. arXiv preprint arXiv:2210.03094 , volume=

    arXiv

  67. [103]

    arXiv preprint arXiv:2603.10712 , year=

    FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model , author=. arXiv preprint arXiv:2603.10712 , year=

  68. [104]

    arXiv preprint arXiv:2506.19816 , year=

    CronusVLA: Transferring Latent Motion Across Time for Multi-Frame Prediction in Manipulation , author=. arXiv preprint arXiv:2506.19816 , year=

  69. [105]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    RoboGround: Robotic Manipulation with Grounded Vision-Language Priors , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  70. [106]

    IEEE Robotics and Automation Letters , volume=

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks , author=. IEEE Robotics and Automation Letters , volume=. 2022 , publisher=

    2022

  71. [107]

    Advances in Neural Information Processing Systems , volume=

    Vlmbench: A compositional benchmark for vision-and-language manipulation , author=. Advances in Neural Information Processing Systems , volume=

  72. [108]

    arXiv preprint arXiv:2305.10355 , year=

    Evaluating object hallucination in large vision-language models , author=. arXiv preprint arXiv:2305.10355 , year=

    arXiv

  73. [109]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    ARNOLD: A benchmark for language-grounded task learning with continuous states in realistic 3D scenes , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  74. [110]

    arXiv preprint arXiv:2602.09973 , year=

    Robointer: A holistic intermediate representation suite towards robotic manipulation , author=. arXiv preprint arXiv:2602.09973 , year=

  75. [111]

    arXiv preprint arXiv:2402.08191 , year=

    The colosseum: A benchmark for evaluating generalization for robotic manipulation , author=. arXiv preprint arXiv:2402.08191 , year=

    arXiv

  76. [112]

    arXiv preprint arXiv:2410.01345 , year=

    Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy , author=. arXiv preprint arXiv:2410.01345 , year=

    arXiv

  77. [113]

    arXiv preprint arXiv:2410.07408 , year=

    Automated Creation of Digital Cousins for Robust Policy Learning , author=. arXiv preprint arXiv:2410.07408 , year=

    arXiv

  78. [114]

    arXiv preprint arXiv:2412.14058 , year=

    Towards generalist robot policies: What matters in building vision-language-action models , author=. arXiv preprint arXiv:2412.14058 , year=

    arXiv

  79. [115]

    2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Sprint: Scalable policy pre-training via language instruction relabeling , author=. 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2024 , organization=

    2024

  80. [116]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  81. [117]

    arXiv preprint arXiv:2312.13303 , year=

    Realgen: Retrieval augmented generation for controllable traffic scenarios , author=. arXiv preprint arXiv:2312.13303 , year=

    arXiv

  82. [118]

    IEEE Transactions on Intelligent Transportation Systems , volume=

    A survey on safety-critical driving scenario generation—A methodological perspective , author=. IEEE Transactions on Intelligent Transportation Systems , volume=. 2023 , publisher=

    2023

  83. [119]

    Conference on Robot Learning , pages=

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation , author=. Conference on Robot Learning , pages=. 2023 , organization=

    2023

  84. [120]

    arXiv preprint arXiv:2406.02523 , year=

    RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots , author=. arXiv preprint arXiv:2406.02523 , year=

    arXiv

  85. [121]

    First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024 , year=

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting , author=. First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024 , year=

    2024

  86. [122]

    arXiv preprint arXiv:2406.18915 , year=

    Manipulate-anything: Automating real-world robots using vision-language models , author=. arXiv preprint arXiv:2406.18915 , year=

    arXiv

  87. [123]

    arXiv preprint arXiv:2403.08248 , year=

    Copa: General robotic manipulation through spatial constraints of parts with foundation models , author=. arXiv preprint arXiv:2403.08248 , year=

    arXiv

  88. [124]

    arXiv preprint arXiv:2307.05973 , year=

    Voxposer: Composable 3d value maps for robotic manipulation with language models , author=. arXiv preprint arXiv:2307.05973 , year=

    arXiv

  89. [125]

    arXiv preprint arXiv:2405.05941 , year=

    Evaluating Real-World Robot Manipulation Policies in Simulation , author=. arXiv preprint arXiv:2405.05941 , year=

    arXiv

  90. [126]

    arXiv preprint arXiv:2505.08243 , year=

    Training Strategies for Efficient Embodied Reasoning , author=. arXiv preprint arXiv:2505.08243 , year=

    arXiv

  91. [127]

    arXiv preprint arXiv:2004.10964 , year=

    Don't stop pretraining: Adapt language models to domains and tasks , author=. arXiv preprint arXiv:2004.10964 , year=

    2004 arXiv

  92. [128]

    International journal of computer vision , volume=

    Visual genome: Connecting language and vision using crowdsourced dense image annotations , author=. International journal of computer vision , volume=. 2017 , publisher=

    2017

  93. [129]

    European Conference on Computer Vision , pages=

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding , author=. European Conference on Computer Vision , pages=. 2025 , organization=

    2025

  94. [130]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Memflow: Optical flow estimation and prediction with memory , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  95. [131]

    arXiv preprint arXiv:2407.10943 , year=

    Grutopia: Dream general robots in a city at scale , author=. arXiv preprint arXiv:2407.10943 , year=

    arXiv

  96. [132]

    IEEE Robotics and Automation Letters , volume=

    Rlbench: The robot learning benchmark & learning environment , author=. IEEE Robotics and Automation Letters , volume=. 2020 , publisher=

    2020

  97. [133]

    arXiv preprint arXiv:1712.05474 , year=

    Ai2-thor: An interactive 3d environment for visual ai , author=. arXiv preprint arXiv:1712.05474 , year=

    arXiv

  98. [134]

    arXiv preprint arXiv:2108.10470 , year=

    Isaac gym: High performance gpu-based physics simulation for robot learning , author=. arXiv preprint arXiv:2108.10470 , year=

    arXiv

  99. [135]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Objaverse: A universe of annotated 3d objects , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  100. [136]

    Advances in Neural Information Processing Systems , volume=

    Objaverse-xl: A universe of 10m+ 3d objects , author=. Advances in Neural Information Processing Systems , volume=

  101. [137]

    arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv:2303.08774 , year=

    arXiv

  102. [138]

    arXiv preprint arXiv:2311.01455 , year=

    Robogen: Towards unleashing infinite data for automated robot learning via generative simulation , author=. arXiv preprint arXiv:2311.01455 , year=

    arXiv

  103. [139]

    arXiv preprint arXiv:2305.12821 , year=

    Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation , author=. arXiv preprint arXiv:2305.12821 , year=

    arXiv

  104. [140]

    arXiv preprint arXiv:2310.17596 , year=

    Mimicgen: A data generation system for scalable robot learning using human demonstrations , author=. arXiv preprint arXiv:2310.17596 , year=

    arXiv

  105. [141]

    Technical Report, Tech

    Pddl| the planning domain definition language , author=. Technical Report, Tech. Rep. , year=

  106. [142]

    IEEE Transactions on Robotics , year=

    Anygrasp: Robust and efficient grasp perception in spatial and temporal domains , author=. IEEE Transactions on Robotics , year=

  107. [143]

    Advances in neural information processing systems , volume=

    Alvinn: An autonomous land vehicle in a neural network , author=. Advances in neural information processing systems , volume=

  108. [144]

    2018 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Deep imitation learning for complex manipulation tasks from virtual reality teleoperation , author=. 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2018 , organization=

    2018

  109. [145]

    arXiv preprint arXiv:2003.06085 , year=

    Learning to generalize across long-horizon tasks from human demonstrations , author=. arXiv preprint arXiv:2003.06085 , year=

    2003 arXiv

  110. [146]

    arXiv preprint arXiv:2212.06817 , year=

    Rt-1: Robotics transformer for real-world control at scale , author=. arXiv preprint arXiv:2212.06817 , year=

    arXiv

  111. [147]

    arXiv preprint arXiv:2307.15818 , year=

    Rt-2: Vision-language-action models transfer web knowledge to robotic control , author=. arXiv preprint arXiv:2307.15818 , year=

    arXiv

  112. [148]

    Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation , pages=

    Scenic: a language for scenario specification and scene generation , author=. Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation , pages=

  113. [149]

    arXiv preprint arXiv:2310.01361 , year=

    Gensim: Generating robotic simulation tasks via large language models , author=. arXiv preprint arXiv:2310.01361 , year=

    arXiv

  114. [150]

    8th Annual Conference on Robot Learning , year=

    SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment , author=. 8th Annual Conference on Robot Learning , year=

  115. [151]

    arXiv preprint arXiv:2402.07872 , year=

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms , author=. arXiv preprint arXiv:2402.07872 , year=

    arXiv

  116. [152]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Zero-1-to-3: Zero-shot one image to 3d object , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  117. [153]

    Advances in Neural Information Processing Systems , volume=

    ProcTHOR: Large-Scale Embodied AI Using Procedural Generation , author=. Advances in Neural Information Processing Systems , volume=

  118. [154]

    CoRR , year=

    Active task randomization: Learning visuomotor skills for sequential manipulation by proposing feasible and novel tasks , author=. CoRR , year=

  119. [155]

    2018 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Sim-to-real transfer of robotic control with dynamics randomization , author=. 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2018 , organization=

    2018

  120. [156]

    Proceedings of Robotics: Science and Systems , year =

    Octo: An Open-Source Generalist Robot Policy , author =. Proceedings of Robotics: Science and Systems , year =

  121. [157]

    2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=

    Domain randomization for transferring deep neural networks from simulation to the real world , author=. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=. 2017 , organization=

    2017

  122. [158]

    The International journal of robotics research , volume=

    Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , author=. The International journal of robotics research , volume=. 2018 , publisher=

    2018

  123. [159]

    Advances in Neural Information Processing Systems , volume=

    Libero: Benchmarking knowledge transfer for lifelong robot learning , author=. Advances in Neural Information Processing Systems , volume=

  124. [160]

    arXiv preprint arXiv:2109.13396 , year=

    Bridge data: Boosting generalization of robotic skills with cross-domain datasets , author=. arXiv preprint arXiv:2109.13396 , year=

    arXiv

  125. [161]

    arXiv preprint arXiv:2104.08212 , year=

    Mt-opt: Continuous multi-task robotic reinforcement learning at scale , author=. arXiv preprint arXiv:2104.08212 , year=

    arXiv

  126. [162]

    a high-fidelity experimental dataset of planar pushing , author=

    More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing , author=. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=. 2016 , organization=

    2016

  127. [163]

    arXiv preprint arXiv:1910.11215 , year=

    Robonet: Large-scale multi-robot learning , author=. arXiv preprint arXiv:1910.11215 , year=

    1910 arXiv

  128. [164]

    arXiv preprint arXiv:2403.12945 , year=

    Droid: A large-scale in-the-wild robot manipulation dataset , author=. arXiv preprint arXiv:2403.12945 , year=

    arXiv

  129. [165]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Sigmoid loss for language image pre-training , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  130. [166]

    arXiv preprint arXiv:2304.07193 , year=

    Dinov2: Learning robust visual features without supervision , author=. arXiv preprint arXiv:2304.07193 , year=

    arXiv

  131. [167]

    , howpublished =

    Abby O’Neill et al. , howpublished =. Open

  132. [168]

    arXiv preprint arXiv:2311.12871 , year=

    An embodied generalist agent in 3d world , author=. arXiv preprint arXiv:2311.12871 , year=

    arXiv

  133. [169]

    The Conference on Robot Learning (CoRL) , year =

    Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, and Dieter Fox , title =. The Conference on Robot Learning (CoRL) , year =

  134. [170]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  135. [171]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Lisa: Reasoning segmentation via large language model , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  136. [172]

    arXiv preprint arXiv:2401.00025 , year=

    Any-point trajectory modeling for policy learning , author=. arXiv preprint arXiv:2401.00025 , year=

    arXiv

  137. [173]

    arXiv preprint arXiv:2311.01977 , year=

    Rt-trajectory: Robotic task generalization via hindsight trajectory sketches , author=. arXiv preprint arXiv:2311.01977 , year=

    arXiv

  138. [174]

    2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Robotap: Tracking arbitrary points for few-shot visual imitation , author=. 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2024 , organization=

    2024

  139. [175]

    arXiv preprint arXiv:2205.04382 , year=

    Flowbot3d: Learning 3d articulation flow to manipulate articulated objects , author=. arXiv preprint arXiv:2205.04382 , year=

    arXiv

  140. [176]

    8th Annual Conference on Robot Learning , year=

    Flow as the Cross-domain Manipulation Interface , author=. 8th Annual Conference on Robot Learning , year=

  141. [177]

    arXiv preprint arXiv:2401.11439 , year=

    General flow as foundation affordance for scalable robot learning , author=. arXiv preprint arXiv:2401.11439 , year=

    arXiv

  142. [178]

    arXiv preprint arXiv:2403.03954 , year=

    3d diffusion policy , author=. arXiv preprint arXiv:2403.03954 , year=

    arXiv

  143. [179]

    2020 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Keto: Learning keypoint representations for tool manipulation , author=. 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2020 , organization=

    2020

  144. [180]

    arXiv:2304.02643 , year=

    Segment Anything , author=. arXiv:2304.02643 , year=

    arXiv

  145. [181]

    arXiv preprint arXiv:2311.17842 , year=

    Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning , author=. arXiv preprint arXiv:2311.17842 , year=

    arXiv

  146. [182]

    arXiv preprint arXiv:2204.02311 , year=

    Palm: Scaling language modeling with pathways , author=. arXiv preprint arXiv:2204.02311 , year=

    arXiv

  147. [183]

    arXiv preprint arXiv:2303.03378 , year=

    Palm-e: An embodied multimodal language model , author=. arXiv preprint arXiv:2303.03378 , year=

    arXiv

  148. [184]

    Conference on Robot Learning (CoRL) , year=

    Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning , author=. Conference on Robot Learning (CoRL) , year=

  149. [185]

    2023 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Code as policies: Language model programs for embodied control , author=. 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2023 , organization=

    2023

  150. [186]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  151. [187]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    arXiv

  152. [188]

    arXiv preprint arXiv:2401.14159 , year=

    Grounded sam: Assembling open-world models for diverse visual tasks , author=. arXiv preprint arXiv:2401.14159 , year=

    arXiv

  153. [189]

    arXiv preprint arXiv:2412.09858 , year=

    RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning , author=. arXiv preprint arXiv:2412.09858 , year=

    arXiv

  154. [190]

    arXiv preprint arXiv:2307.00595 , year=

    Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot , author=. arXiv preprint arXiv:2307.00595 , year=

    arXiv

  155. [191]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Sapien: A simulated part-based interactive environment , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  156. [192]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  157. [193]

    IEEE Access , year=

    Chatgpt for robotics: Design principles and model abilities , author=. IEEE Access , year=

  158. [194]

    IEEE Robotics and Automation Letters , year=

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model , author=. IEEE Robotics and Automation Letters , year=

  159. [195]

    arXiv preprint arXiv:2409.01652 , year=

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation , author=. arXiv preprint arXiv:2409.01652 , year=

    arXiv

  160. [196]

    arXiv preprint arXiv:2308.12966 , year=

    Qwen-vl: A frontier large vision-language model with versatile abilities , author=. arXiv preprint arXiv:2308.12966 , year=

    arXiv

  161. [197]

    European Conference on Computer Vision , pages=

    Llava-grounding: Grounded visual chat with large multimodal models , author=. European Conference on Computer Vision , pages=. 2025 , organization=

    2025

  162. [198]

    arXiv preprint arXiv:2405.03685 , year=

    Language-Image Models with 3D Understanding , author=. arXiv preprint arXiv:2405.03685 , year=

    arXiv

  163. [199]

    2023 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Visual language maps for robot navigation , author=. 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2023 , organization=

    2023

  164. [200]

    International Conference on Machine Learning , pages=

    Learning transferable visual models from natural language supervision , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  165. [201]

    European Conference on Computer Vision , pages=

    Simple open-vocabulary object detection , author=. European Conference on Computer Vision , pages=. 2022 , organization=

    2022

  166. [202]

    Conference on Robot Learning , pages=

    Goal-auxiliary actor-critic for 6d robotic grasping with point clouds , author=. Conference on Robot Learning , pages=. 2022 , organization=

    2022

  167. [203]

    Conference on Robot Learning , pages=

    Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds , author=. Conference on Robot Learning , pages=. 2023 , organization=

    2023

  168. [204]

    2024 , eprint=

    Track2Act: Predicting Point Tracks from Internet Videos enables Diverse Zero-shot Robot Manipulation , author=. 2024 , eprint=

    2024

  169. [205]

    International Conference on Learning Representations , year=

    ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills , author=. International Conference on Learning Representations , year=

  170. [206]

    arXiv preprint arXiv:2310.11441 , year=

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V , author=. arXiv preprint arXiv:2310.11441 , year=

    arXiv

  171. [207]

    arXiv preprint arXiv:2311.01378 , year=

    Vision-Language Foundation Models as Effective Robot Imitators , author =. arXiv preprint arXiv:2311.01378 , year=

    arXiv

  172. [208]

    Conference on Robot Learning , pages=

    Perceiver-actor: A multi-task transformer for robotic manipulation , author=. Conference on Robot Learning , pages=. 2023 , organization=

    2023

  173. [209]

    Conference on robot learning , pages=

    Cliport: What and where pathways for robotic manipulation , author=. Conference on robot learning , pages=. 2022 , organization=

    2022

  174. [210]

    arXiv preprint arXiv:2410.24164 , year=

    _0 : A Vision-Language-Action Flow Model for General Robot Control , author=. arXiv preprint arXiv:2410.24164 , year=

    arXiv

  175. [211]

    arXiv preprint arXiv:2303.00905 , year=

    Open-world object manipulation using pre-trained vision-language models , author=. arXiv preprint arXiv:2303.00905 , year=

    arXiv

  176. [212]

    arXiv preprint arXiv:2304.13705 , year=

    Learning fine-grained bimanual manipulation with low-cost hardware , author=. arXiv preprint arXiv:2304.13705 , year=

    arXiv

  177. [213]

    arXiv preprint arXiv:2410.06158 , year=

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation , author=. arXiv preprint arXiv:2410.06158 , year=

    arXiv

  178. [214]

    arXiv e-prints , pages=

    Robomamba: Multimodal state space model for efficient robot reasoning and manipulation , author=. arXiv e-prints , pages=

  179. [215]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Ego4d: Around the world in 3,000 hours of egocentric video , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  180. [216]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  181. [217]

    arXiv preprint arXiv:2401.02117 , year=

    Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation , author=. arXiv preprint arXiv:2401.02117 , year=

    arXiv

  182. [218]

    arXiv preprint arXiv:2406.09246 , year=

    OpenVLA: An Open-Source Vision-Language-Action Model , author=. arXiv preprint arXiv:2406.09246 , year=

    arXiv

  183. [219]

    arXiv preprint arXiv:1807.06757 , year=

    On evaluation of embodied navigation agents , author=. arXiv preprint arXiv:1807.06757 , year=

    arXiv

  184. [220]

    arXiv preprint arXiv:2408.00714 , url=

    SAM 2: Segment Anything in Images and Videos , author=. arXiv preprint arXiv:2408.00714 , url=

    arXiv

  185. [221]

    arXiv preprint arXiv:2405.17398 , year=

    Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability , author=. arXiv preprint arXiv:2405.17398 , year=

    arXiv

  186. [222]

    Foundations and Trends

    Model-based reinforcement learning: A survey , author=. Foundations and Trends. 2023 , publisher=

    2023

  187. [223]

    Benjamin Ellenberger , title =

  188. [224]

    IEEE International Symposium on System Integrations (SII) , year=

    The Pinocchio C++ library -- A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives , author=. IEEE International Symposium on System Integrations (SII) , year=

  189. [225]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    2001 arXiv

  190. [226]

    arXiv preprint arXiv:2410.18647 , year=

    Data scaling laws in imitation learning for robotic manipulation , author=. arXiv preprint arXiv:2410.18647 , year=

    arXiv

  191. [227]

    IEEE Robotics & Automation Magazine , volume=

    The franka emika robot: A reference platform for robotics research and education , author=. IEEE Robotics & Automation Magazine , volume=. 2022 , publisher=

    2022

  192. [228]

    Proceedings of the 2019 Conference of the North

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 Conference of the North

    2019

  193. [229]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. Journal of Machine Learning Research , year =

  194. [230]

    arXiv preprint arXiv:2009.12293 , year=

    robosuite: A Modular Simulation Framework and Benchmark for Robot Learning , author=. arXiv preprint arXiv:2009.12293 , year=

    2009 arXiv

  195. [231]

    European Conference on Computer Vision , pages=

    Cotracker: It is better to track together , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  196. [232]

    Guo, Runlin (Kolin) and Lin, Xinsong and Liu, Minghua and Gu, Jiayuan and Su, Hao , license =

  197. [233]

    Llava-next: Improved reasoning, ocr, and world knowledge , author=

  198. [234]

    arXiv preprint arXiv:2312.13139 , year=

    Unleashing large-scale video generative pre-training for visual robot manipulation , author=. arXiv preprint arXiv:2312.13139 , year=

    arXiv

  199. [235]

    2025 , url=

    RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning , author=. 2025 , url=

    2025

  200. [236]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  201. [237]

    arXiv preprint arXiv:2412.10345 , year=

    Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies , author=. arXiv preprint arXiv:2412.10345 , year=

    arXiv

  202. [238]

    arXiv preprint arXiv:2411.19650 , year=

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation , author=. arXiv preprint arXiv:2411.19650 , year=

    arXiv

  203. [239]

    arXiv preprint arXiv:2204.01691 , year=

    Do as i can, not as i say: Grounding language in robotic affordances , author=. arXiv preprint arXiv:2204.01691 , year=

    arXiv

  204. [240]

    arXiv preprint arXiv:2403.01823 , year=

    Rt-h: Action hierarchies using language , author=. arXiv preprint arXiv:2403.01823 , year=

    arXiv

  205. [241]

    arXiv preprint arXiv:2502.19417 , year=

    Hi robot: Open-ended instruction following with hierarchical vision-language-action models , author=. arXiv preprint arXiv:2502.19417 , year=

    arXiv

  206. [242]

    Advances in Neural Information Processing Systems , volume=

    Sharegpt4video: Improving video understanding and generation with better captions , author=. Advances in Neural Information Processing Systems , volume=

  207. [243]

    The International Journal of Robotics Research , pages=

    Diffusion policy: Visuomotor policy learning via action diffusion , author=. The International Journal of Robotics Research , pages=. 2023 , publisher=

    2023

  208. [244]

    arXiv preprint arXiv:2210.02747 , year=

    Flow matching for generative modeling , author=. arXiv preprint arXiv:2210.02747 , year=

    arXiv

  209. [245]

    Advances in neural information processing systems , volume=

    Diffusion models beat gans on image synthesis , author=. Advances in neural information processing systems , volume=

  210. [246]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. ICLR , volume=

  211. [247]

    arXiv preprint arXiv:2502.14786 , year=

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features , author=. arXiv preprint arXiv:2502.14786 , year=

    arXiv

  212. [248]

    arXiv preprint arXiv:2501.14818 , year=

    Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models , author=. arXiv preprint arXiv:2501.14818 , year=

    arXiv

  213. [249]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  214. [250]

    Advances in Neural Information Processing Systems , volume=

    Mixture-of-experts with expert choice routing , author=. Advances in Neural Information Processing Systems , volume=

  215. [251]

    , year =

    X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Design , author =. , year =

  216. [252]

    arXiv preprint arXiv:2407.08693 , year=

    Robotic control via embodied chain-of-thought reasoning , author=. arXiv preprint arXiv:2407.08693 , year=

    arXiv

  217. [253]

    European Conference on Computer Vision , pages=

    Pointllm: Empowering large language models to understand point clouds , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  218. [254]

    arXiv preprint arXiv:2411.15594 , year=

    A survey on llm-as-a-judge , author=. arXiv preprint arXiv:2411.15594 , year=

    arXiv

  219. [255]

    2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Robovqa: Multimodal long-horizon reasoning for robotics , author=. 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2024 , organization=

    2024

  220. [256]

    arXiv preprint arXiv:2307.07864 , year=

    CIDER: Context sensitive sentiment analysis for short-form text , author=. arXiv preprint arXiv:2307.07864 , year=

    arXiv

  221. [257]

    arXiv preprint arXiv:2402.10329 , year=

    Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots , author=. arXiv preprint arXiv:2402.10329 , year=

    arXiv

  222. [258]

    arXiv preprint arXiv:2412.04445 , year=

    Moto: Latent motion token as the bridging language for robot manipulation , author=. arXiv preprint arXiv:2412.04445 , year=

  223. [259]

    arXiv preprint arXiv:2408.06072 , year=

    Cogvideox: Text-to-video diffusion models with an expert transformer , author=. arXiv preprint arXiv:2408.06072 , year=

    arXiv

  224. [260]

    arXiv preprint arXiv:2502.02175 , year=

    VLA-Cache: Towards Efficient Vision-Language-Action Model via Adaptive Token Caching in Robotic Manipulation , author=. arXiv preprint arXiv:2502.02175 , year=

  225. [261]

    arXiv preprint arXiv:2403.20330 , year=

    Are We on the Right Way for Evaluating Large Vision-Language Models? , author=. arXiv preprint arXiv:2403.20330 , year=

    arXiv

  226. [262]

    Proceedings of CVPR , year=

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI , author=. Proceedings of CVPR , year=

  227. [263]

    OCRBench: on the hidden mystery of OCR in large multimodal models , volume=

    Liu, Yuliang and Li, Zhang and Huang, Mingxin and Yang, Biao and Yu, Wenwen and Li, Chunyuan and Yin, Xu-Cheng and Liu, Cheng-Lin and Jin, Lianwen and Bai, Xiang , year=. OCRBench: on the hidden mystery of OCR in large multimodal models , volume=. Science China Information Sci...

    doi

  228. [264]

    arXiv preprint arXiv:2411.03409 , year=

    STEER: Flexible Robotic Manipulation via Dense Language Grounding , author=. arXiv preprint arXiv:2411.03409 , year=

    arXiv

  229. [265]

    arXiv preprint arXiv:2501.09747 , year=

    Fast: Efficient action tokenization for vision-language-action models , author=. arXiv preprint arXiv:2501.09747 , year=

    arXiv

  230. [266]

    arXiv preprint arXiv:2501.15830 , year=

    SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model , author=. arXiv preprint arXiv:2501.15830 , year=

    arXiv

  231. [267]

    arXiv preprint arXiv:2503.14734 , year=

    GR00T N1: An Open Foundation Model for Generalist Humanoid Robots , author=. arXiv preprint arXiv:2503.14734 , year=

    arXiv

  232. [268]

    arXiv preprint arXiv:2410.11758 , year=

    Latent action pretraining from videos , author=. arXiv preprint arXiv:2410.11758 , year=

    arXiv

  233. [269]

    arXiv preprint arXiv:2507.17520 , year=

    InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation , author=. arXiv preprint arXiv:2507.17520 , year=

  234. [270]

    2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=

    From LLMs to Actions: latent codes as bridges in hierarchical robot control , author=. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=. 2024 , organization=

    2024

  235. [271]

    arXiv preprint arXiv:2412.11974 , year=

    Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning , author=. arXiv preprint arXiv:2412.11974 , year=

    arXiv

  236. [272]

    arXiv preprint arXiv:2405.09818 , year=

    Chameleon: Mixed-modal early-fusion foundation models , author=. arXiv preprint arXiv:2405.09818 , year=

    arXiv

  237. [273]

    arXiv preprint arXiv:2408.11039 , year=

    Transfusion: Predict the next token and diffuse images with one multi-modal model , author=. arXiv preprint arXiv:2408.11039 , year=

    arXiv

  238. [274]

    arXiv preprint arXiv:2504.06256 , year=

    Transfer between Modalities with MetaQueries , author=. arXiv preprint arXiv:2504.06256 , year=

    arXiv

  239. [275]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  240. [276]

    arXiv preprint arXiv:2410.07864 , year=

    Rdt-1b: a diffusion foundation model for bimanual manipulation , author=. arXiv preprint arXiv:2410.07864 , year=

    arXiv

  241. [277]

    arXiv preprint arXiv:2402.11684 , year=

    Allava: Harnessing gpt4v-synthesized data for lite vision-language models , author=. arXiv preprint arXiv:2402.11684 , year=

    arXiv

  242. [278]

    arXiv preprint arXiv:2402.11530 , year=

    Efficient multimodal learning from data-centric perspective , author=. arXiv preprint arXiv:2402.11530 , year=

    arXiv

  243. [279]

    arXiv preprint arXiv:2502.19645 , year=

    Fine-tuning vision-language-action models: Optimizing speed and success , author=. arXiv preprint arXiv:2502.19645 , year=

    arXiv

  244. [280]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Film: Visual reasoning with a general conditioning layer , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  245. [281]

    arXiv preprint arXiv:2502.13130 , year=

    Magma: A foundation model for multimodal ai agents , author=. arXiv preprint arXiv:2502.13130 , year=

    arXiv

  246. [282]

    Advances in Neural Information Processing Systems , volume=

    Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers , author=. Advances in Neural Information Processing Systems , volume=

  247. [283]

    arXiv preprint arXiv:2410.08001 , year=

    Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation , author=. arXiv preprint arXiv:2410.08001 , year=

    arXiv

  248. [284]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Scalable diffusion models with transformers , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  249. [285]

    2023 , journal=

    V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs , author=. 2023 , journal=

    2023

  250. [286]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  251. [287]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

  252. [288]

    2024 , eprint=

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models , author=. 2024 , eprint=

    2024

  253. [289]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  254. [290]

    European Conference on Computer Vision , pages=

    Mmbench: Is your multi-modal model an all-around player? , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  255. [291]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Towards VQA Models That Can Read , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  256. [292]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    Docvqa: A dataset for vqa on document images , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  257. [293]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    Infographicvqa , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  258. [294]

    arXiv preprint arXiv:2408.03326 , year=

    Llava-onevision: Easy visual task transfer , author=. arXiv preprint arXiv:2408.03326 , year=

    arXiv

  259. [295]

    arXiv preprint arXiv:2409.12191 , year=

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , author=. arXiv preprint arXiv:2409.12191 , year=

    arXiv

  260. [296]

    European Conference on Computer Vision , pages=

    A diagram is worth a dozen images , author=. European Conference on Computer Vision , pages=. 2016 , organization=

    2016

  261. [297]

    arXiv preprint arXiv:2203.10244 , year=

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning , author=. arXiv preprint arXiv:2203.10244 , year=

    arXiv

  262. [298]

    2024 , url=

    RealWorldQA , author=. 2024 , url=

    2024

  263. [299]

    2024 , url=

    Helix , author=. 2024 , url=

    2024

  264. [300]

    Forty-first International Conference on Machine Learning , year=

    Prismatic vlms: Investigating the design space of visually-conditioned language models , author=. Forty-first International Conference on Machine Learning , year=

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.