Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

EmbRACE-3K: Embodied Reasoning and Action in Complex Environments

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EmbRACE-3K, a 3,000-task photorealistic embodied benchmark, reports zero-shot VLM success below 20% and improves a 7B model beyond larger baselines after SFT and RL.

desk verdict Useful photorealistic embodied VLM benchmark with a plausible SFT+RL recipe, but the zero-shot claim is contradicted by its own tables and the Gemini-generated rationales are post-hoc, not online reasoning. read the letter →

arxiv 2507.10548 v1 pith:DLC5GAFB submitted 2025-07-14 cs.CV cs.AIcs.CL

Mingxian Lin , Wei Huang , Yitang Li , Chengjie Jiang , Kui Wu , Fangwei Zhong , Shengju Qian , Xin Wang
show 1 more author
Xiaojuan Qi
This is my paper
classification cs.CVcs.AIcs.CL
keywords embodiedreasoningvision-languagemodelsbenchmarkdatasetspatialreinforcementlearningchain-of-thoughtnavigationmulti-stageplanning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the gap between vision-language models' strong static-image performance and their weak embodied performance can be quantified and narrowed. It introduces EmbRACE-3K, a benchmark of more than 3,000 language-guided tasks and 26,000 decision steps in photorealistic indoor and outdoor environments, with each step annotated by an egocentric view, a discrete action, and a natural-language rationale. Zero-shot, current models such as GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL-7B all score below 20% success on the benchmark, exposing short-sighted exploration, spatial-semantic drift, and target forgetting. Fine-tuning Qwen2.5-VL-7B on the dataset with supervised learning and then reinforcement learning raises success rates in all categories and improves out-of-domain robustness, which the paper attributes to the step-wise reasoning annotations plus trajectory-level reward shaping. If the result holds, the dataset provides both a diagnostic test for embodied reasoning and a training resource for building embodied agents.

What carries the argument

The central machinery is the per-step annotation unit: each decision in a trajectory pairs an egocentric RGB frame with the executed discrete action and a natural-language rationale expressing the agent's intent, so perception, reasoning, and action are aligned at the same time step. On top of this, training uses a two-stage recipe: supervised fine-tuning that teaches the model to emit <think> rationale and <action> tokens, then GRPO reinforcement learning with a rule-based reward that checks format and action accuracy and compares a group of candidate rollouts. The step-wise rationale is what lets the model learn perception-conditioned decision-making instead of end-to-end action mapping; the RL stage is what generalizes it to unseen scenes.

What would settle it

Rebuild EmbRACE-3K's supervision with human-written instructions and rationales on the same captured trajectories, then rerun the zero-shot evaluation and the SFT+RL training; if the below-20% zero-shot numbers and the training gains largely disappear or change direction, the benchmark's difficulty and its utility are artifacts of Gemini's annotations rather than of embodied interaction itself.

Watch

Extended reading notes

Core claim

The paper claims that current vision-language models, however strong on static image and video understanding, lack the embodied capabilities that online egocentric interaction demands, and that this gap can be measured and partly closed. EmbRACE-3K provides 3,000+ language-guided tasks with 26,000 decision steps in photorealistic indoor and outdoor environments, each step carrying an egocentric frame, a discrete action, and a natural-language rationale; on this benchmark, GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL-7B all score below 20% success zero-shot. Fine-tuning Qwen2.5-VL-7B with supervised learning on the rationales followed by GRPO reinforcement learning raises success rates across Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage Goal Execution, in some out-of-domain categories surpassing the larger proprietary models. The authors take the SFT-only versus SFT+RL comparison as evidence that explicit step-wise reasoning supervision and trajectory-level reward signals both contribute, with RL supplying the out-of-domain robustness that pure imitation lacks.

Load-bearing premise

The load-bearing assumption is that Gemini-generated task instructions and step-by-step rationales, after human filtering, are genuinely grounded in the environment rather than reflecting the language model's own priors about what should be seen and done.

Editorial extensions

If this is right

  • Current VLMs, fine-tuned only for passive visual tasks, fail at information-seeking behavior: they fixate on immediate cues, lose targets that leave view, and let spatial references drift as the camera moves.
  • Step-wise rationale supervision is doing real work: removing <think> annotations lowers success on spatial-semantic and multi-stage tasks compared with SFT that keeps them.
  • RL is the generalization lever: SFT-only models drop sharply out-of-domain, while SFT+RL retains more success and lower goal-distance error in new scenes.
  • The EmbRACE-3K-trained 7B model outperforms GPT-4o and Gemini 2.5 Pro on several out-of-domain task categories, so smaller open models can surpass larger closed models once trained on interaction-centric data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Gemini's stepwise rationales carry model priors rather than environmental grounding, the benchmark's difficulty and the SFT gains could both be inflated; a control version with human-authored rationales on identical trajectories would separate these.
  • The same annotation pipeline should transfer to continuous-control and real-robot settings; if it does, trajectory-level RL plus rationale supervision could become a general recipe rather than a simulator-specific one.
  • An independent test set with object layouts and instructions sampled from a different generator would clarify whether the out-of-domain robustness comes from learned exploration strategies or from similarity to training maps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces EmbRACE-3K, a dataset of over 3,000 language-guided embodied tasks in photorealistic Unreal Engine environments, with egocentric observations, grounded actions, and step-wise natural language rationales. The authors benchmark GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL in zero-shot settings and fine-tune Qwen2.5-VL-7B with supervised learning followed by GRPO reinforcement learning. They report that the fine-tuned model outperforms the proprietary baselines and that step-wise reasoning annotations improve decision quality. The central claims are that current VLMs are weak at embodied interaction (with all zero-shot models supposedly below 20% success) and that training on EmbRACE-3K yields substantial embodied reasoning improvements.

Significance. If these claims are validated, EmbRACE-3K would be a valuable resource: it provides large-scale, photorealistic, closed-loop embodied tasks with human demonstrations and step-level annotation, filling a gap relative to passive video benchmarks and code-generation-only embodied datasets. The SFT+RL pipeline and the no-thinking versus sft-only comparison are informative for the community. However, the current manuscript contains an internal contradiction in the headline zero-shot claim, incomplete reporting of evaluation sizes and error bars, and a potentially circular annotation process: Gemini generates both the instructions and the stepwise rationales and is also evaluated as a baseline. The paper's central contribution depends on these rationales being genuinely grounded in online perception, but no validation is provided. These issues need to be resolved before the benchmark's validity and the reported improvements can be fully credited.

major comments (4)
  1. [Abstract / §1 / Table 2] The claim that "in zero-shot settings, all models achieve success rates below 20%" is contradicted by Table 2: in-domain, GPT-4o reaches 53.6% SR on Basic and Gemini 2.5 Pro reaches 76.4% on Basic and 39.3% on Exploration. Even if the claim is intended to apply only to the three named challenge dimensions or to out-of-domain tasks, the current wording is false and must be qualified precisely, with the table rows cited.
  2. [§6.2 / Tables 2-3] Tables 2 and 3 report success rate, GDE, SSPL, Steps, and Timeout Rate without any indication of test-set size, number of evaluation episodes, or variance across runs. This makes it impossible to assess whether differences such as the sft-rl versus GPT-4o comparisons are significant. Additionally, §6.2 defines success as reaching "within 300 meters of the target," which is implausibly large for an embodied indoor/outdoor benchmark and appears to be a unit error; the success thresholds and on-policy termination criteria must be stated explicitly and per task type.
  3. [§4.2 Stage 4 / §6.3] The stepwise rationales are generated by Gemini after receiving the complete egocentric video and the entire action trajectory, so they may be post-hoc rationalizations rather than online, perception-conditioned reasoning. The paper reports manual verification for instructions (Stage 2) but not for Stage-4 rationales. Because Gemini is also an evaluated baseline, the dataset may partly distill Gemini's priors, and the claimed reasoning-annotation gains (no-thinking vs. sft-only) could reflect imitation of fluent hindsight language rather than acquisition of embodied online reasoning. Please provide evidence that each rationale is inferable from the observations available at the corresponding step, for example via human ratings or by withholding future frames during rationale generation.
  4. [§5.2 / §6.1] The GRPO training description is incomplete: the accuracy reward is not defined per task type, and the phrase "trajectory-level reward shaping" in §6.1 is never specified. The reader cannot tell how success is judged during RL, how the distance threshold is applied, how multi-stage subgoal completion is scored, or how interaction actions (open, pick, drop) are rewarded. Please specify the reward functions and hyperparameters so that the RL results are reproducible.
minor comments (6)
  1. [Abstract] The dataset name is written "EmRACE-3K" in the abstract but "EmbRACE-3K" in the title and body; please unify.
  2. [§6.2] The units of GDE are stated as centimeters but the success threshold is "300 meters"; please check the unit consistency and define a per-task spatial threshold (and what constitutes a "Finish action").
  3. [Figure 4] The task type distribution percentages appear to sum to more than 100% (50+12+13+8+5+12+17 = 117); please verify the pie chart labels or the legend.
  4. [Table 1] The checkmark/cross notation in the table is not defined in the caption; please add a legend or spell out "yes/no".
  5. [§5.2] The GRPO objective is typeset with a broken line break after "JGRP O(θ)"; fix the equation formatting.
  6. [General] The manuscript does not state whether the dataset, code, and evaluation scripts will be publicly released beyond a project page; please add an availability statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and fine-tuning results are empirically measured and do not reduce to dataset construction by definition.

full rationale

The paper's central chain—construct EmbRACE-3K, report zero-shot failure of existing VLMs, fine-tune Qwen2.5-VL-7B with SFT+RL, and measure improvements—is empirical rather than definitional. The zero-shot evaluation uses environment success rate, goal-distance error, SSPL, steps, and timeout rate on held-out tasks, not the dataset's own annotations. The fine-tuning improvements are likewise measured by environment outcomes, so no fitted parameter is renamed as a prediction; the SFT/RL objective optimizes action correctness, and the reported SR/GDE gains are external outcomes. A possible validity concern is that Stage-4 rationales are generated by Gemini with access to the full trajectory, so the 'thinking' traces may encode hindsight rather than online reasoning; however, this affects the interpretation of why CoT supervision helps, not whether the measured improvements are logically forced. The comparison of sft-only vs no-thinking is an ablation, not an identity. Self-citations (e.g., UnrealZoo [33], Visual CoT [21], V-IRL [27]) appear only as platform/tooling or related-work context and are not load-bearing for the main conclusion. Under the rubric requiring an explicit reduction (Eq. X = Eq. Y by construction, fitted input called prediction, or a load-bearing self-citation chain), no significant circularity is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on evaluation thresholds and on assumptions about annotation validity rather than on mathematical axioms. There are no invented physical entities; the main burden is that Gemini-generated instructions and rationales are treated as ground truth, and that success thresholds are chosen by hand.

free parameters (2)
  • success distance threshold = 300 cm, stated as '300 meters' in §6.2
    Hand-chosen spatial radius for counting a task as successful; the textual typo makes the threshold ambiguous, and all success-rate numbers depend directly on this value.
  • maximum episode length = 32 steps
    Trajectories with more than 32 steps are filtered during curation (§4.3) and used as the timeout threshold, shaping task difficulty and the reported Timeout Rate.
assumptions (3)
  • domain assumption Human demonstrations recorded through UnrealCV-Zoo via keyboard and mouse are valid proxies for embodied decision-making under partial observability.
    The dataset's behavioral supervision is human gameplay in simulation; the paper does not validate transfer to real-world robots or to alternative input devices.
  • domain assumption Gemini 2.5 Pro generated instructions and rationales, after manual filtering, are semantically grounded and solvable in the local environment.
    Stage 2 and Stage 4 of §4.2 rely on model generation plus manual verification; no inter-annotator agreement or independent grounding is reported.
  • domain assumption Success metrics (SR, GDE, SSPL, Steps, TR) capture embodied reasoning ability.
    Evaluation treats distance errors and step efficiency as proxies for reasoning; no human baseline or validation of metric adequacy is given in §6.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmbRACE-3K: Embodied Reasoning and Action in Complex Environments." pith.science (2026). https://pith.science/paper/DLC5GAFB

@misc{pith2026250710548,
  author       = {Pith},
  title        = {Pith review of: EmbRACE-3K: Embodied Reasoning and Action in Complex Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLC5GAFB}},
  note         = {Machine review of arXiv:2507.10548}
}
read the original abstract

Recent advanced vision-language models(VLMs) have demonstrated strong performance on passive, offline image and video understanding tasks. However, their effectiveness in embodied settings, which require online interaction and active scene understanding remains limited. In such scenarios, an agent perceives the environment from a first-person perspective, with each action dynamically shaping subsequent observations. Even state-of-the-art models such as GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro struggle in open-environment interactions, exhibiting clear limitations in spatial reasoning and long-horizon planning. To address this gap, we introduce EmRACE-3K, a dataset of over 3,000 language-guided tasks situated in diverse, photorealistic environments constructed using Unreal Engine and the UnrealCV-Zoo framework. The tasks encompass a wide range of embodied challenges, including navigation, object manipulation, and multi-stage goal execution. Each task unfolds as a multi-step trajectory, pairing first-person visual observations with high-level instructions, grounded actions, and natural language rationales that express the agent's intent at every step. Using EmRACE-3K, we establish a benchmark to evaluate the embodied reasoning capabilities of VLMs across three key dimensions: Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage Goal Execution. In zero-shot settings, all models achieve success rates below 20%, underscoring the challenge posed by our benchmark and the current limitations of VLMs in interactive environments. To demonstrate the utility of EmRACE-3K, we further fine-tune Qwen2.5-VL-7B using supervised learning followed by reinforcement learning. This approach yields substantial improvements across all three challenge categories, highlighting the dataset's effectiveness in enabling the development of embodied reasoning capabilities.

Figures

Figures reproduced from arXiv: 2507.10548 by the authors.

Figure 1
Figure 1. EmbRACE-3K dataset. The dataset contains over 3k language-guided tasks and 26k deci￾sion steps set in diverse, photorealistic environments. Each task involves high-level natural language instructions, grounded actions (e.g., move, turn, look, interact), egocentric visual observations, and step-wise reasoning. Agents interpret visual inputs and follow instructions to execute multi-step decision trajectories, with eac… view at source ↗
Figure 2
Figure 2. Multi-stage Embodied Task Data Collection Pipeline. The [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The Token Number and Word-Cloud Distribution of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Two-stage Training framework for Embodied Agent on [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 4
Figure 4. Figure 4: Distribution of task types. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    The manuscript body introduces PokeGym, a vision-only automated 3D-game benchmark, while the abstract claims a G-EvoMAC method and 60.18% success rate absent from the body.

Reference graph

Works this paper leans on

32 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

  2. [2]

    Introducing claude 3.5 sonnet

    Anthropic. Introducing claude 3.5 sonnet. https://www.anthropic.com/news/ introducing-claude-3-5-sonnet , 2024. Accessed: 2025-05-16

    work page 2024

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    R1-v: Reinforcing super generalization ability in vision-language models with less than $3

    Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. https://github.com/Deep-Agent/R1-V , 2025. Accessed: 2025-02-02

    work page 2025

  5. [5]

    Navila: Legged robot vision-language-action model for navigation

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453, 2024

    arXiv 2024

  6. [6]

    Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025

    arXiv 2025

  7. [8]

    Video-r1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025

    arXiv 2025

  8. [9]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

Show all 32 references
  1. [10]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision ...

    2022

  2. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [12]

    Free-form language-based robotic reasoning and grasping

    Runyu Jiao, Alice Fasoli, Francesco Giuliari, Matteo Bortolon, Sergio Povoli, Guofeng Mei, Yiming Wang, and Fabio Poiesi. Free-form language-based robotic reasoning and grasping. arXiv preprint arXiv:2503.13082, 2025

    2025 arXiv

  4. [13]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  5. [14]

    Mcu: A task-centric framework for open-ended agent evaluation in minecraft

    Haowei Lin, Zihao Wang, Jianzhu Ma, and Yitao Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft. arXiv preprint arXiv:2310.08367, 2023

    2023 arXiv

  6. [15]

    Codemind: A framework to challenge large language models for code reasoning

    Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. Codemind: A framework to challenge large language models for code reasoning. arXiv preprint arXiv:2402.09664, 2024

    2024 arXiv

  7. [17]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  8. [18]

    OpenAI. Gpt-4o. 2025

    2025

  9. [19]

    Teach: Task-driven embodied agents that chat

    Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence , ...

    2017

  10. [20]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  11. [21]

    Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems, ...

    2024

  12. [22]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  13. [23]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern ...

    2020

  14. [24]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  15. [25]

    Habitat challenge 2023

    Karmesh Yadav, Jacob Krantz, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Jimmy Yang, Austin Wang, John Turner, Aaron Gokaslan, Vincent-Pierre Berges, Roozbeh Mootaghi, Oleksandr Maksymets, Angel X Chang, Manolis Savva, Alexander Clegg, Devendra Singh Chaplot, and Dhruv Batra. ...

    2023

  16. [26]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  17. [27]

    V-irl: Grounding virtual intelligence in real life

    Jihan Yang, Runyu Ding, Ellis Brown, Xiaojuan Qi, and Saining Xie. V-irl: Grounding virtual intelligence in real life. In European conference on computer vision, pages 36–55. Springer, 2024

    2024

  18. [28]

    Octopus: Embodied vision-language programmer from environmental feedback

    Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. In European Conference on Computer Vision, pages 20–38. Springer, 2024

    2024

  19. [29]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025

    2025

  20. [30]

    Spatial mental modeling from limited views

    Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. Spatial mental modeling from limited views. arXiv preprint arXiv:2506.21458, 2025

    2025

  21. [31]

    Octopi: Object property reasoning with large tactile-language models

    Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property reasoning with large tactile-language models. arXiv preprint arXiv:2405.02794, 2024

    2024 arXiv

  22. [32]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  23. [33]

    Unrealzoo: Enriching photo-realistic virtual worlds for embodied ai

    Fangwei Zhong, Kui Wu, Churan Wang, Hao Chen, Hai Ci, Zhoujun Li, and Yizhou Wang. Unrealzoo: Enriching photo-realistic virtual worlds for embodied ai. arXiv preprint arXiv:2412.20977, 2024

    2024 arXiv

  24. [34]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 12

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.