Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MTU3D claims that visual grounding and spatial exploration can be a single scoring problem over object and frontier queries, and that this union beats RL and modular navigators on four benchmarks.

desk verdict A sensible unified grounding-and-exploration architecture with a genuinely nice frontier-query idea, but the headline gains come from seen splits and the evaluation lacks error bars; still deserves a serious referee. read the letter →

arxiv 2507.04047 v2 pith:N4TVG4RG submitted 2025-07-05 cs.CV

Ziyu Zhu , Xilin Wang , Yixuan Li , Zhuofan Zhang , Xiaojian Ma , Yixin Chen , Baoxiong Jia , Wei Liang
show 4 more authors
Qian Yu Zhidong Deng Siyuan Huang Qing Li
This is my paper
classification cs.CV
keywords embodiednavigation3Dvision-languagevisualgroundingfrontierexplorationspatialmemoryRGB-Dperceptionvision-language-actionopen-vocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Embodied scene understanding has two halves: reading the space already seen, and deciding where to look next. The paper argues that these halves belong in one model, and builds MTU3D to prove it: a single end-to-end navigator that turns streaming RGB-D frames into object queries stored in a spatial memory bank, and treats unexplored regions as frontier queries scored against the same goal. Because both choices come out of one decision layer, the agent can ground a named object, an image, a description, a task plan, or a question, and can explore when the target is not yet visible. On four embodied benchmarks, the same model beats modular and reinforcement-learning baselines by 14, 23, 9, and 2 percentage points in success rate, and it transfers to a real robot without real-world fine-tuning. The point that would matter if true: active exploration and visual grounding reinforce each other, so jointly optimizing them works better than either a static-scene grounding model or a reward-trained explorer.

What carries the argument

The load-bearing object is the unified query token. Each object is one token carrying a 3D box, an open-vocabulary semantic vector, a mask, a confidence score, and a feature vector; each frontier is a token carrying a 3D coordinate. Both kinds of tokens are stored and updated in a dynamic spatial memory bank fused by bounding-box IoU, and both are scored by one spatial-reasoning transformer against the same goal embedding from a contrastive language-image encoder. The frontier tokens make exploration a grounding problem: choosing where to explore is the same operation as choosing which object to go to.

What would settle it

On a fixed set of HM3D-OVON episodes, add increasing Gaussian noise to the camera poses before memory fusion, from zero up to errors that break bounding-box IoU matches between frames; if success rate stays flat as pose error grows, the online-memory claim is robust, and if it collapses, the method depends on pose accuracy that the real-robot section does not report.

Watch

Extended reading notes

Core claim

MTU3D claims that object grounding and spatial exploration can be reduced to one scoring problem. Every visible object is represented as a query with a 3D box, a mask, a semantic embedding, a feature vector, and a confidence score; every unexplored region boundary is represented as a frontier query with a 3D coordinate. A spatial-reasoning transformer takes the goal, encoded as text or an image, and scores both kinds of queries in a shared space, so the highest-scoring query either grounds an object or sends the agent to a frontier. Local queries from each RGB-D frame are merged into a global memory by bounding-box IoU matching with exponentially weighted fusion, so the scene representation is built online and never requires an explicit mesh. With vision-language-exploration pretraining on over a million trajectories and task-specific fine-tuning, the model reports success-rate gains over the strongest baselines on open-vocabulary navigation, multimodal lifelong navigation, sequential task navigation, and embodied question answering.

Load-bearing premise

The spatial memory bank places every object query and frontier in a shared world frame using the camera pose that comes with each RGB-D frame; if that pose is wrong or drifts, objects get fused into the wrong locations and both grounding and exploration degrade.

Editorial extensions

If this is right

  • With the same decision layer for both object queries and frontier queries, a goal can be specified as a category, a sentence, a reference image, a step in a task plan, or a question, without changing the architecture.
  • No explicit 3D reconstruction is needed at inference: the memory bank is assembled online from RGB-D frames and camera poses, so the approach can operate in partially observed and changing environments.
  • Joint grounding-plus-exploration training yields more efficient search than blind frontier exploration: at exploration step six, MTU3D reaches 50.0% success rate versus 33.3% for frontier-only search, with higher SPL.
  • Lifelong spatial memory is what makes multimodal navigation work: resetting memory per sub-episode on GOAT-Bench drops success rate from 52.6% to 10.5% for object goals and from 71.4% to 28.6% for description goals.
  • Vision-language-exploration pretraining on mixed expert and noisy trajectories improves success rate on all three navigation benchmarks, and the same checkpoints transfer to a real robot without real-world fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the unified-query formulation holds, frontier selection and object grounding may be trainable from demonstrations alone, so large-scale trajectory collection could replace reward design for other embodied search tasks.
  • Beyond the paper: the spatial memory bank doubles as a queryable scene representation, so the same trajectory could feed a language model's answers; the A-EQA configuration already suggests that exploration quality, not just answerer capability, limits embodied question answering.
  • Beyond the paper: a concrete testable prediction is that MTU3D's margin over video-based navigators grows with episode length, because its memory advantage compounds, while short-horizon episodes favor direct goal recognition.
  • Beyond the paper: the frontier-as-query idea could transfer to manipulation planning, where 'where to reach next' is scored against the same goal embedding instead of 'where to move next'.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MTU3D, a unified end-to-end model for embodied 3D navigation that replaces offline mesh/point-cloud reconstruction with an online query-based representation built from RGB-D frames plus camera poses, a dynamic spatial memory bank, and frontier queries for exploration. Training is done in three stages: low-level perception, vision-language-exploration pre-training on more than a million simulated and real trajectories, and task-specific fine-tuning. The authors evaluate on HM3D-OVON, GOAT-Bench, SG3D, and A-EQA, reporting improvements in SR and SPL, and also deploy the model on a real robot. The central claim is that a single model jointly optimizing grounding and frontier selection outperforms prior RL and modular navigation methods on all four benchmarks.

Significance. If the results were fully supported, MTU3D would be a meaningful step: it removes the static-reconstruction assumption, integrates exploration and grounding in one objective, and demonstrates cross-benchmark transfer from large-scale pretraining. The paper's strengths are its clear architectural integration of PQ3D-style queries and EmbodiedSAM-style memory, the large trajectory pretraining pipeline with an automatic mixing strategy, and the explicit real-robot deployment. The paper also includes ablations for VLE pretraining and spatial memory that show consistent benefits. However, the headline quantitative claims currently rest on favorable split selection and small, unquantified subsamples; the contribution is valuable but needs stronger evidence before being accepted as stated.

major comments (4)
  1. [Abstract; Table 3; Table 5; Appendix B] The abstract's 'outperforms ... by 14%, 23%, 9%, and 2%' is not supported by the reported evaluation. For HM3D-OVON, the 14% figure is taken from the Val Seen split (55.0 vs. 41.3, Table 3); on the Val Unseen split, which the paper itself emphasizes for generalization, the margin over Uni-NaVid is 40.8 vs. 39.5, a 1.3-point difference. Appendix B states that only 360 episodes were randomly sampled for HM3D-OVON and 90 tasks for GOAT-Bench, with no confidence intervals, multiple seeds, or significance tests. For a success rate near 0.4, the standard error of a 360-episode proportion is about 2.6 points, so the Val Unseen margin is within sampling noise. The GOAT-Bench Val Unseen margin (47.2 vs. 32.1) is more substantial, but it is also computed on a 90-task sample with no uncertainty quantification. The authors should report full-split or bootstrapped results and should not cherry-pick the most favorable split in the abstract.
  2. [Table 6; Appendix B (A-EQA)] The A-EQA comparison is under-specified. The number of questions used is not reported, and the LLM-SR gain of GPT-4V with MTU3D over GPT-4V (44.2 vs. 41.8) is 2.4 points, plausibly within sampling noise. The LLM-SPL gain (37.0 vs. 7.5) is very large and suggests that the baseline and MTU3D trajectories differ substantially in path efficiency, but no details are given about how baseline trajectories are generated, how many episodes are scored, or whether the same episodes are used. The claim in Appendix B that 'we use the same prompts and the same number of video frames as the baseline methods' does not establish comparability of exploration policies. At minimum, report the question count, per-question standard errors or bootstrap intervals, and a matched comparison on identical episode sets.
  3. [Table 4; Section 4.2; references [21] and [87]] The SG3D evaluation is too thin to support a state-of-the-art claim. Table 4 compares MTU3D against only two baselines: Embodied Video Agent (reference [21], from the same research group as this paper) and SenseAct-NN Monolithic. In addition, the SG3D benchmark itself was introduced by the authors' prior work (reference [87]). The 9.1-point s-SR improvement over Embodied Video Agent may be real, but with only two baselines and no independent leaderboard numbers, the 'state-of-the-art' phrasing is not justified. I request comparisons with at least one independent modular navigation baseline (e.g., VLFM or GOAT) on SG3D, or a clear statement of why those baselines cannot run on SG3D.
  4. [Section 3.1; Section 4.5; Appendix A (Eqs. 3-4)] The entire spatial memory and mapping pipeline assumes an accurate camera pose P_t in SE(3) at every frame. Bounding boxes, memory fusion via IoU matching (Eq. 3), occupancy maps, and frontier coordinates are all expressed in a world frame derived from these poses. The real-robot section (Section 4.5) reports no pose-error statistics and no robustness experiments under odometry drift. If pose errors exceed the IoU threshold epsilon, the memory bank will misassociate or duplicate objects, degrading both grounding and exploration. Please report pose accuracy (e.g., ATE/RPE) on the real-robot runs and include a sensitivity analysis of success rate as a function of pose noise, or at least discuss why the architecture is robust to realistic pose errors.
minor comments (4)
  1. [Throughout] There are several typos: 'impratical' in Section 1, 'Vision-Langauge' and 'Pe-training' in the Section 4.3 heading, and 'Monotholic' in Appendix B. These should be corrected.
  2. [Figure 4] The caption order is inconsistent with the text: the caption labels (b) as 'exploration efficiency' and (c) as 'contribution of spatial memory', while Section 4.3 refers to Fig. 4b for memory and Fig. 4c for exploration efficiency. Please fix the correspondence.
  3. [Table 3] The column header 'Val Seen Val Seen UnseenSynonyms' is malformed; it should be three distinct split columns with clear subheadings.
  4. [Section 4.5] The real-world deployment section is qualitative only; a small table reporting success rates, path lengths, and pose errors for the three scenes would strengthen the deployment claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the headline results are empirical evaluations on external benchmarks, not consequences of the model's assumptions.

full rationale

The paper's central claims are empirical: MTU3D is trained end-to-end and evaluated on HM3D-OVON, GOAT-Bench, SG3D, and A-EQA using standard success metrics. These metrics are not defined in terms of the model's own outputs or fitted parameters, so the reported numbers are not forced by construction. The main caveats are statistical and experimental rather than circular: the abstract's margins are taken from the most favorable 'Val Seen' splits, HM3D-OVON and GOAT-Bench evaluations use small random subsamples (360 episodes and 90 tasks) without confidence intervals, and the VLE pre-training consumes the training splits of these same benchmarks, which weakens the 'unseen environment' generalization claim but does not make the evaluation a restatement of the training objective. The paper does cite the authors' own prior work (PQ3D [94], SG3D [87], Embodied Video Agent [21]) as architectural inspiration, benchmark source, and baseline, respectively, but none of these citations supplies a load-bearing premise that reduces the result to itself. The trajectory collection oracle in Algorithm 1 uses visibility and reachability to label decisions, and the learned policy is later scored by success in reaching goals, but test-time inputs do not include the oracle's privileged goal/map state, so the model's predictions are not the fitted labels themselves. No equation-level identity or definitional equivalence between input and claimed output can be exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on a set of domain assumptions common in embodied AI, plus several hand-chosen hyperparameters. The most load-bearing invented entity is the frontier query, which unifies grounding and exploration in one decision space. No new physical entities or forces are postulated; this is a systems paper where the novelty is architectural and in the training data mix.

free parameters (5)
  • IoU matching threshold epsilon = not reported
    Used in query matching and fusion (Section 3.1 and Appendix A); hand-set threshold controlling when local queries are merged into global memory. Central to memory quality, yet its value and sensitivity are not reported.
  • Stage 1 loss weights (lambda_b, lambda_m, lambda_v, lambda_s) = 1.0, 1.0, 1.0, 0.5
    Chosen by hand in Section 3.4 for perception training; these weights shape the query proposal quality that the rest of the pipeline depends on.
  • Number of frames subsampled per trajectory segment = 18
    Set in Section 4.1; affects how much history the spatial reasoning transformer sees during training and inference.
  • Random vs optimal frontier mixing ratio = not specified
    The hybrid strategy in Section 3.3 mixes random and optimal frontier selection, but the mixing ratio is not given, making the exploration pretraining data distribution non-reproducible.
  • Number of decoder layers in query refinement and spatial reasoning = 4
    Architecture choice stated in Section 3.4; affects capacity and runtime.
assumptions (5)
  • domain assumption Camera pose P_t is accurate and drift-free
    All world-coordinate fusion, occupancy mapping, and frontier detection rely on pose from Section 3.1; no robustness analysis for pose error is provided, especially in real-world deployment.
  • domain assumption FastSAM segments correspond to semantically meaningful objects
    Query proposal pools per-segment features (Section 3.1); if objects are undersegmented or merged, grounding and exploration decisions will be based on corrupted queries.
  • domain assumption DINOv2 and CLIP embeddings provide sufficient cross-modal alignment for open-vocabulary goals
    The model matches CLIP text/image embeddings to DINO segment features via training (Section 3.2); if the domains shift, open-vocabulary performance degrades.
  • domain assumption Frontier-based exploration will eventually make goals visible in test environments
    Standard frontier exploration is complete for reachable free space, but it does not handle goals behind occlusions or in dynamic spaces; the paper does not analyze failure modes.
  • domain assumption The benchmark training splits used in VLE pretraining are representative of the test episodes
    Stage 2 and 3 training uses HM3D-OVON, GOAT, and SG3D training data (Table 2); test sets come from the same distributions, which helps performance but may not transfer to truly novel layouts.
invented entities (2)
  • Frontier queries (Q_F)
    purpose: Encodes unexplored regions as query tokens so a single transformer can choose between grounding a known object and moving to a frontier.
    A new representational device introduced in this paper; its value is measured only through benchmark performance and it has no external falsifiable prediction outside the model.
  • Dynamic Spatial Memory Bank
    purpose: Stores and fuses object queries over time to support lifelong grounding and navigation.
    A new component whose benefit is shown in ablation Figure 4b, but the ablation is on the authors' own evaluation setup and no separate external validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation." pith.science (2026). https://pith.science/paper/N4TVG4RG

@misc{pith2026250704047,
  author       = {Pith},
  title        = {Pith review of: Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4TVG4RG}},
  note         = {Machine review of arXiv:2507.04047}
}
read the original abstract

Embodied scene understanding requires not only comprehending visual-spatial information that has been observed but also determining where to explore next in the 3D physical world. Existing 3D Vision-Language (3D-VL) models primarily focus on grounding objects in static observations from 3D reconstruction, such as meshes and point clouds, but lack the ability to actively perceive and explore their environment. To address this limitation, we introduce \underline{\textbf{M}}ove \underline{\textbf{t}}o \underline{\textbf{U}}nderstand (\textbf{\model}), a unified framework that integrates active perception with \underline{\textbf{3D}} vision-language learning, enabling embodied agents to effectively explore and understand their environment. This is achieved by three key innovations: 1) Online query-based representation learning, enabling direct spatial memory construction from RGB-D frames, eliminating the need for explicit 3D reconstruction. 2) A unified objective for grounding and exploring, which represents unexplored locations as frontier queries and jointly optimizes object grounding and frontier selection. 3) End-to-end trajectory learning that combines \textbf{V}ision-\textbf{L}anguage-\textbf{E}xploration pre-training over a million diverse trajectories collected from both simulated and real-world RGB-D sequences. Extensive evaluations across various embodied navigation and question-answering benchmarks show that MTU3D outperforms state-of-the-art reinforcement learning and modular navigation approaches by 14\%, 23\%, 9\%, and 2\% in success rate on HM3D-OVON, GOAT-Bench, SG3D, and A-EQA, respectively. \model's versatility enables navigation using diverse input modalities, including categories, language descriptions, and reference images. These findings highlight the importance of bridging visual grounding and exploration for embodied intelligence.

Figures

Figures reproduced from arXiv: 2507.04047 by the authors.

Figure 1
Figure 1. MTU3D is a versatile embodied navigation model capable of processing diverse inputs, including object categories, image snapshots, natural language descriptions, task plan sequences, and questions. It iteratively explores the environment and performs visual grounding to reach the target location. Through large-scale Vision-Language-Exploration pre-training, it achieves state-of-the-art perfor￾mance across various be… view at source ↗
Figure 2
Figure 2. Our approach bridges online exploration with dynami [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Our proposed model processes RGB-D sequences to generate object queries, which are stored in a memory bank. The spatial [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Ablation studies showing (a) the impact of vision-language-exploration pretraining, (b) exploration efficiency on seen environ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of results in Habitat-Sim. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Real world trajectory. Here, we assume that the j-th current local query is matched with the i-th previous global query. The variable n denotes the number of queries that have been merged into QG t−1 [i] so far. This weighted average approach ensures that the global re…
Figure 8
Figure 8. Figure 8: Real Device. Q l+1 t = FFN  Norm  SpatialSelfAttn(Q l ′′ t )  (8) Through the process, the Spatial Reasoning Transformer effectively fuses spatial and linguistic information for fur￾ther exploration decision. B. Benchmarks and baseline HM3D-OVON. HM3D-OVON is an op…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SPG: Style-Prompting Guidance for Style-Specific Content Creation

    cs.GR 2025-08 unverdicted novelty 4.0 of 10

    SPG is not described anywhere in the supplied text; the body is a different paper (OVSegDT) about robot navigation.

Reference graph

Works this paper leans on

96 extracted references · 61 canonical work pages · cited by 1 Pith paper

  1. [21]

    Embodied videoagent: Persis- tent memory from egocentric videos and embodied sen- sors enables dynamic scene understanding

    Yue Fan, Xiaojian Ma, Rongpeng Su, Jun Guo, Rujie Wu, Xi Chen, and Qing Li. Embodied videoagent: Persis- tent memory from egocentric videos and embodied sen- sors enables dynamic scene understanding. arXiv preprint arXiv:2501.00358, 2024. 6

    arXiv 2024

  2. [87]

    Task-oriented sequen- tial grounding and navigation in 3d scenes

    Zhuofan Zhang, Ziyu Zhu, Junhao Li, Pengxiang Li, Tianxu Wang, Tengyu Liu, Xiaojian Ma, Yixin Chen, Baoxiong Jia, Siyuan Huang, and Qing Li. Task-oriented sequen- tial grounding and navigation in 3d scenes. arXiv preprint arXiv:2408.04034, 2024. 2, 3, 5, 6

    arXiv 2024

  3. [1]

    Scanents3d: Exploit- ing phrase-to-3d-object correspondences for improved visio- linguistic models in 3d scenes

    Ahmed Abdelreheem, Kyle Olszewski, Hsin-Ying Lee, Pe- ter Wonka, and Panos Achlioptas. Scanents3d: Exploit- ing phrase-to-3d-object correspondences for improved visio- linguistic models in 3d scenes. In Proceedings of Winter Conference on Applications of Computer Vision (WACV) , pages 3524–3534, 2024. 2, 3

    2024

  4. [2]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision (ECCV), 2020. 3, 5

    2020

  5. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 5

    2022

  6. [4]

    Do as i can, not as i say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL), 2022. 3

    2022

  7. [5]

    3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds

    Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  8. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 2, 4

    2021

Show all 96 references
  1. [7]

    Object goal naviga- tion using goal-oriented semantic exploration

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Ab- hinav Gupta, and Russ R Salakhutdinov. Object goal naviga- tion using goal-oriented semantic exploration. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3

    2020

  2. [8]

    Object goal navi- gation using goal-oriented semantic exploration

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Ab- hinav Gupta, and Russ R Salakhutdinov. Object goal navi- gation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems (NeurIPS) , 33: 4247–4258, 2020. 2, 5

    2020

  3. [9]

    Scanrefer: 3d object localization in rgb-d scans using natu- ral language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natu- ral language. In European Conference on Computer Vision (ECCV), 2020. 2, 3, 5

    2020

  4. [10]

    Language conditioned spatial relation reasoning for 3d object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 3

    2022

  5. [11]

    Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVP...

    2024

  6. [12]

    Scan2cap: Context-aware dense captioning in rgb- d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb- d scans. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2, 3

    2021

  7. [13]

    Unit3d: A unified trans- former for 3d dense captioning and visual grounding

    Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, and Angel X Chang. Unit3d: A unified trans- former for 3d dense captioning and visual grounding. In In- ternational Conference on Computer Vision (ICCV) , pages 18109–18119, 2023. 3

    2023

  8. [14]

    Schwing, Alexan- der Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 4

    2022

  9. [15]

    4d spatio-temporal convnets: Minkowski convolutional neural networks

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 4

    2019

  10. [16]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5828–5839, 2017. 5

    2017

  11. [17]

    Embodied question answer- ing

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3

    2018

  12. [18]

    A survey of embodied ai: From simulators to research tasks

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230–244, 2022. 2, 3

    2022

  13. [19]

    The one ring: a robotic indoor navigation generalist

    Ainaz Eftekhar, Luca Weihs, Rose Hendrix, Ege Caglar, Jordi Salvador, Alvaro Herrasti, Winson Han, Eli Vander- Bil, Aniruddha Kembhavi, Ali Farhadi, et al. The one ring: a robotic indoor navigation generalist. arXiv preprint arXiv:2412.14401, 2024. 3

    2024 arXiv

  14. [20]

    Spoc: Imitating short- est paths in simulation enables effective navigation and ma- nipulation in the real world

    Kiana Ehsani, Tanmay Gupta, Rose Hendrix, Jordi Salvador, Luca Weihs, Kuo-Hao Zeng, Kunal Pratap Singh, Yejin Kim, Winson Han, Alvaro Herrasti, et al. Spoc: Imitating short- est paths in simulation enables effective navigation and ma- nipulation in the real world. In The IEEE/...

    2024

  15. [22]

    Efficient graph-based image segmentation

    Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. International Journal of Computer Vision (IJCV), 59:167–181, 2004. 4

    2004

  16. [23]

    Cows on pasture: Base- lines and benchmarks for language-driven zero-shot object navigation

    Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Base- lines and benchmarks for language-driven zero-shot object navigation. In The IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2023. 3

    2023

  17. [24]

    Scaling open-vocabulary image segmentation with image- level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image- level labels. In European Conference on Computer Vision (ECCV), pages 540–557. Springer, 2022. 3

    2022

  18. [25]

    Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning. In 2024 IEEE International Confer-...

    2024

  19. [26]

    Viewrefer: Grasp the multi-view knowledge for 3d visual grounding

    Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang, Zhi- gang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. In In- ternational Conference on Computer Vision (ICCV) , pages 15372–15383, 2023. 2, 3

    2023

  20. [27]

    Transrefer3d: Entity-and- relation aware transformer for fine-grained 3d visual ground- ing

    Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transrefer3d: Entity-and- relation aware transformer for fine-grained 3d visual ground- ing. In Proceedings of the 29th ACM International Confer- ence on Multimedia, 2021. 3

    2021

  21. [28]

    Vln bert: A recurrent vision- and-language bert for navigation

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez- Opazo, and Stephen Gould. Vln bert: A recurrent vision- and-language bert for navigation. In The IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  22. [29]

    3d-llm: In- jecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: In- jecting the 3d world into large language models. Advances in Neural Information Processing Systems (NeurIPS) , 36: 20482–20494, 2023. 2, 3

    2023

  23. [30]

    A real-time occupancy map from multiple video streams

    Adam Hoover and Bent David Olsen. A real-time occupancy map from multiple video streams. InProceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No. 99CH36288C), pages 2261–2266. IEEE, 1999. 2

    1999

  24. [31]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In International Conference on Machine Learning (ICML), 2024. 2, 3

    2024

  25. [32]

    Language models as zero-shot planners: Ex- tracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Ex- tracting actionable knowledge for embodied agents. In In- ternational Conference on Machine Learning (ICML), pages 9118–9147. PMLR, 2022. 2, 3

    2022

  26. [33]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Ji- ajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning (CoRL), 2023. 3

    2023

  27. [34]

    Bottom up top down detection transform- ers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Kate- rina Fragkiadaki. Bottom up top down detection transform- ers for language grounding in images and point clouds. In European Conference on Computer Vision (ECCV), 2022. 3

    2022

  28. [35]

    Conceptfusion: Open-set multi- modal 3d mapping

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Subramanian Iyer, Soroush Saryazdi, Nikhil Varma Keetha, et al. Conceptfusion: Open-set multi- modal 3d mapping. In ICRA2023 Workshop on Pretraining for Robotic...

    2023

  29. [36]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. European Conference on Computer Vision (ECCV), 2024. 3, 5

    2024

  30. [37]

    Goat-bench: A benchmark for multi-modal lifelong navigation

    Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mot- taghi. Goat-bench: A benchmark for multi-modal lifelong navigation. In Proceedings of the IEEE/CVF Conference on...

    2024

  31. [38]

    Realfred: An em- bodied instruction following benchmark in photo-realistic environments

    Taewoong Kim, Cheolhong Min, Byeonghwi Kim, Jinyeon Kim, Wonje Jeung, and Jonghyun Choi. Realfred: An em- bodied instruction following benchmark in photo-realistic environments. In European Conference on Computer Vision, pages 346–364. Springer, 2024. 3

    2024

  32. [39]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In International Conference on Computer Vision (ICCV), pages 4015–4026, 2023. 2, 4

    2023

  33. [40]

    Uni- clip: Unified framework for contrastive language-image pre- training

    Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uni- clip: Unified framework for contrastive language-image pre- training. arXiv preprint arXiv:2209.13430, 2022. 3

    2022 arXiv

  34. [41]

    Less is more: Clipbert for video-and-language learning via sparse sampling

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 5

    2021

  35. [42]

    Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation.arXiv preprint arXiv:2403.09227, 2024

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gok- men, Sanjana Srivastava, Roberto Mart ´ın-Mart´ın, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, et al. Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation.arXiv...

    2024 arXiv

  36. [43]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 3

    2023 arXiv

  37. [44]

    Panoptic segformer: Delving deeper into panoptic segmen- tation with transformers

    Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmen- tation with transformers. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1280–12...

    2022

  38. [45]

    Code as policies: Language model programs for embodied con- trol

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied con- trol. In International Conference on Robotics and Automa- tion (ICRA), 2023. 3

    2023

  39. [46]

    Navcot: Boosting llm-based vision-and- language navigation via learning disentangled reasoning

    Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Navcot: Boosting llm-based vision-and- language navigation via learning disentangled reasoning. arXiv preprint arXiv:2403.07376, 2024. 3

    2024 arXiv

  40. [47]

    Aligning cyber space with physical world: A comprehensive survey on embodied ai

    Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.06886, 2024. 2

    2024 arXiv

  41. [48]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. International Conference on Learning Representations (ICLR), 2023. 2, 3

    2023

  42. [49]

    Zson: Zero-shot object-goal navigation using multimodal goal embeddings

    Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. Advances in Neural Information Processing Systems, 35:32340–32352,

  43. [50]

    Openeqa: Embodied question answering in the era of foun- dation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foun- dation models. In The IEEE/CVF Conference on Computer Visio...

    2024

  44. [51]

    Openeqa: Embodied question answering in the era of foun- dation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foun- dation models. In The IEEE/CVF Conference on Com- puter Vis...

    2024

  45. [52]

    Spatial memory

    David S Olton. Spatial memory. Scientific American, 236 (6):82–99, 1977. 4

    1977

  46. [53]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 4

    2023 arXiv

  47. [54]

    Teach: Task-driven embodied agents that chat

    Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivas- tava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. InAAAI Con- ference on Artificial Intelligence (AAAI), 2022. 2, 3

    2022

  48. [55]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In The IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 815–824, 2023. 2

    2023

  49. [56]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  50. [57]

    Pirlnav: Pretraining with imitation and rl finetuning for objectnav

    Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Abhishek Das. Pirlnav: Pretraining with imitation and rl finetuning for objectnav. In The IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 17896–17906,

  51. [58]

    Sayplan: Ground- ing large language models using 3d scene graphs for scalable task planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable task planning. In 7th Annual Conference on Robot Learning,

  52. [59]

    Explore until confi- dent: Efficient exploration for embodied question answering

    Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until confi- dent: Efficient exploration for embodied question answering. arXiv preprint arXiv:2403.15941, 2024. 3

    2024 arXiv

  53. [60]

    Language- grounded indoor 3d semantic segmentation in the wild

    David Rozenberszki, Or Litany, and Angela Dai. Language- grounded indoor 3d semantic segmentation in the wild. In European Conference on Computer Vision (ECCV) , pages 125–141. Springer, 2022. 5

    2022

  54. [61]

    Habitat: A plat- form for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A plat- form for embodied ai research. In The IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (C...

    2019

  55. [62]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 2

    2017 arXiv

  56. [63]

    Mask3d: Mask trans- former for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask trans- former for 3d semantic instance segmentation. In Inter- national Conference on Robotics and Automation (ICRA) , pages 8216–8223. IEEE, 2023. 2, 3, 4

    2023

  57. [64]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern ...

    2020

  58. [65]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In International Conference on Computer Vision (ICCV), 2023. 3

    2023

  59. [66]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir V ondrus, Sameer Dharur, Franziska Meier, Woj- ciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun,...

  60. [67]

    Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann

    Ayc ¸a Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. OpenMask3D: Open-V ocabulary 3D Instance Segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 2, 3

    2023

  61. [68]

    Rio: 3d object instance re- localization in changing indoor environments

    Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re- localization in changing indoor environments. In Interna- tional Conference on Computer Vision (ICCV), 2019. 3

    2019

  62. [69]

    Embodiedscan: A holistic multi- modal 3d perception suite towards embodied ai

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi- modal 3d perception suite towards embodied ai. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  63. [70]

    Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames

    Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv:1911.00357, 2019. 5

    1911 arXiv

  64. [71]

    Ver: Scaling on- policy rl leads to the emergence of navigation in embodied rearrangement

    Erik Wijmans, Irfan Essa, and Dhruv Batra. Ver: Scaling on- policy rl leads to the emergence of navigation in embodied rearrangement. Advances in Neural Information Processing Systems (NeurIPS), 35:7727–7740, 2022. 2, 3

    2022

  65. [72]

    Scenegraphfusion: Incre- mental 3d scene graph prediction from rgb-d sequences

    Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incre- mental 3d scene graph prediction from rgb-d sequences. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3

    2021

  66. [73]

    Embodiedsam: Online segment any 3d thing in real time

    Xiuwei Xu, Huangxing Chen, Linqing Zhao, Ziwei Wang, Jie Zhou, and Jiwen Lu. Embodiedsam: Online segment any 3d thing in real time. arXiv preprint arXiv:2408.11811,

    arXiv

  67. [74]

    Habitat-matterport 3d semantics dataset

    Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakr- ishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, et al. Habitat-matterport 3d semantics dataset. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and...

    2023

  68. [75]

    Frontier-based exploration using multiple robots

    Brian Yamauchi. Frontier-based exploration using multiple robots. In Proceedings of the second international confer- ence on Autonomous agents, pages 47–53, 1998. 2, 5

    1998

  69. [76]

    Thinking in space: How mul- timodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024. 2

    2024 arXiv

  70. [77]

    3d-mem: 3d scene memory for embodied exploration and reasoning

    Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17294–17303, 2025. 3

    2025

  71. [78]

    Vlfm: Vision-language frontier maps for zero-shot semantic navigation

    Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision-language frontier maps for zero-shot semantic navigation. In International Conference on Robotics and Automation (ICRA) , 2024. 3, 6

    2024

  72. [79]

    Hm3d-ovon: A dataset and bench- mark for open-vocabulary object goal navigation

    Naoki Yokoyama, Ram Ramrakhya, Abhishek Das, Dhruv Batra, and Sehoon Ha. Hm3d-ovon: A dataset and bench- mark for open-vocabulary object goal navigation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5543–5550. IEEE, 2024. 2, 3, 5, 6

    2024

  73. [80]

    Frontier semantic exploration for visual target navigation

    Bangguo Yu, Hamidreza Kasaei, and Ming Cao. Frontier semantic exploration for visual target navigation. In Inter- national Conference on Robotics and Automation (ICRA) ,

  74. [81]

    Instancere- fer: Cooperative holistic understanding for visual ground- ing on point clouds through instance multi-level contextual referring

    Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancere- fer: Cooperative holistic understanding for visual ground- ing on point clouds through instance multi-level contextual referring. In International Conference on Computer Vision...

    2021

  75. [82]

    Poliformer: Scaling on-policy rl with transformers results in masterful navigators

    Kuo-Hao Zeng, Zichen Zhang, Kiana Ehsani, Rose Hendrix, Jordi Salvador, Alvaro Herrasti, Ross Girshick, Aniruddha Kembhavi, and Luca Weihs. Poliformer: Scaling on-policy rl with transformers results in masterful navigators. arXiv preprint arXiv:2406.20083, 2024. 3

    2024 arXiv

  76. [83]

    Uni-navid: A video-based vision- language-action model for unifying embodied navigation tasks

    Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision- language-action model for unifying embodied navigation tasks. arXiv preprint arXiv:2412.06224, 2024. 3, 6

    2024 arXiv

  77. [84]

    Vision-language pre-training with object con- trastive learning for 3d scene understanding

    Taolin Zhang, Sunan He, Dai Tao, Bin Chen, Zhi Wang, and Shu-Tao Xia. Vision-language pre-training with object con- trastive learning for 3d scene understanding. arXiv preprint arXiv:2305.10714, 2023. 2, 3

    2023 arXiv

  78. [85]

    Multi3drefer: Grounding text description to multiple 3d objects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In International Conference on Computer Vision (ICCV), pages 15225–15236, 2023. 2, 3, 5

    2023

  79. [86]

    Microsoft kinect sensor and its effect

    Zhengyou Zhang. Microsoft kinect sensor and its effect. IEEE multimedia, 19(2):4–10, 2012. 8

    2012

  80. [88]

    To- wards explainable 3d grounded visual question answering: A new benchmark and strong baseline

    Lichen Zhao, Daigang Cai, Jing Zhang, Lu Sheng, Dong Xu, Rui Zheng, Yinjie Zhao, Lipeng Wang, and Xibo Fan. To- wards explainable 3d grounded visual question answering: A new benchmark and strong baseline. IEEE Transactions on Circuits and Systems for Video Technology, 2022. 2, 3

    2022

  81. [89]

    Fast segment any- thing

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing. arXiv preprint arXiv:2306.12156, 2023. 4

    2023 arXiv

  82. [90]

    3d-vla: A 3d vision-language-action generative world model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631, 2024. 3

    2024 arXiv

  83. [91]

    Dual memory units with uncertainty regulation for weakly supervised video anomaly detection

    Hang Zhou, Junqing Yu, and Wei Yang. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection. In AAAI Conference on Artificial Intelli- gence (AAAI), pages 3769–3777, 2023. 4

    2023

  84. [92]

    Scanreason: Empowering 3d visual grounding with reasoning capabilities

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities. In European Conference on Computer Vision (ECCV) , pages 151–168. Springer, 2024. 2

    2024

  85. [93]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In International Conference on Computer Vision (ICCV), pages 2911–2921, 2023. 2, 3

    2023

  86. [94]

    Unifying 3d vision-language understanding via prompt- able queries

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via prompt- able queries. In European Conference on Computer Vision, pages 188–206. Springer, 2024. 2, 3, 4

    2024

  87. [95]

    Tango: Training-free embodied ai agents for open-world tasks

    Filippo Ziliotto, Tommaso Campari, Luciano Serafini, and Lamberto Ballan. Tango: Training-free embodied ai agents for open-world tasks. arXiv preprint arXiv:2412.10402 ,

    arXiv

  88. [96]

    Generalized decoding for pixel, image, and language

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15116–15127,...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.