Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dynam3D replaces video frames with dynamically updated, language-aligned patch-instance-zone 3D tokens and claims new state-of-the-art monocular results on R2R-CE, REVERIE-CE, and NavRAG-CE.

desk verdict Dynam3D is a credible dynamic 3D-token system with a real R2R-CE gain, but the NavRAG-CE portion of the SOTA claim is compromised by train/test overlap. read the letter →

arxiv 2505.11383 v1 pith:3EKOQTDC submitted 2025-05-16 cs.CV cs.RO

Zihan Wang , Seungjun Lee , Gim Hee Lee This is my paper · ORCID
classification cs.CVcs.RO
keywords vision-and-languagenavigation3Dvision-languagemodelhierarchicaltokensdynamicsceneupdatesmonocularcontinuousenvironmentsCLIPfeaturealignmentembodiedagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a navigation agent perceives better when its visual memory is a structured 3D hierarchy of patch, instance, and zone tokens rather than a stack of video frames. It argues that CLIP features projected into 3D space and aligned with language give a 3.8-billion-parameter vision-language model enough geometry and long-term memory to outperform larger video-based agents in continuous indoor environments, even when objects move. If true, this would mean structured, language-grounded 3D memory is a stronger bottleneck for embodied navigation than raw frames, and that smaller models can win by changing the representation rather than the parameter count. The paper backs the claim with state-of-the-art numbers on R2R-CE, REVERIE-CE, and NavRAG-CE, plus real-robot tests with pre-exploration and moved targets.

What carries the argument

The key object is the Dynam3D patch-instance-zone representation, a three-level 3D token hierarchy. Patch features come from CLIP-ViT-L/14@336px projected into world coordinates using depth and pose; instance tokens are formed by aggregating FastSAM 2D mask features through a transformer-based instance encoder; zone tokens aggregate instance features inside cubic cells spanning several meters. The mechanism that makes it dynamic has two parts: frustum culling removes patch points that lie between the camera and a newly observed surface, and the merging discriminator, an MLP trained with binary classification on multi-room 3D instance segmentation data, decides whether a fresh 2D segment should merge into an existing 3D instance or start a new one. A generalizable feature field renders agent-centric panoramic patch tokens from the stored feature points. All three levels are aligned with language through 3D-language contrastive learning and CLIP feature distillation, using subspace contrastive learning to subtract each view's semantic center and reduce view-dependent bias, which is what allows a pretrained 2D VLM to reason about the 3D tokens.

What would settle it

Measure generalization on NavRAG-CE with test instructions generated by a different LLM and prompt template than the training instructions, while keeping the same simulator and environments; if the success-rate margin over video baselines collapses to the R2R-CE margin, the benchmark does not isolate generalization to unseen user demands.

Watch

Extended reading notes

Core claim

Dynam3D's central claim is that a navigation agent should perceive through a dynamically updated, language-aligned 3D token hierarchy rather than a sequence of video frames. The hierarchy has three levels: patch feature points from CLIP projected into world coordinates, instance tokens formed by merging FastSAM 2D masks into consistent 3D instances, and zone tokens that summarize several-meter spatial cubes. The dynamic part is a frustum-culling update that deletes stale patch points and a merging discriminator that decides whether a newly observed segment belongs to an existing 3D instance, with updates propagating so zone summaries stay current. After large-scale 3D-language pretraining aligns all three levels to CLIP's semantic space, the representation feeds a 3.8B LLaVA-Phi-3-mini that outputs atomic actions such as turning, moving forward, or stopping. On R2R-CE Val Unseen the agent reaches 52.9% success rate versus 47.2% for g3D-LF and 47.0% for the larger Uni-NaVid, and the paper reports similar advantages on REVERIE-CE and NavRAG-CE plus real-robot success after the target is moved.

Load-bearing premise

The large NavRAG-CE gain stands on the assumption that those test episodes are genuinely unseen rather than distributionally overlapping with the NavRAG-generated training instructions; the R2R-CE and real-robot results do not depend on this assumption.

Editorial extensions

If this is right

  • Pre-exploration and lifelong memory become usable: keeping stored 3D representations across episodes raises success rate on R2R-CE from 52.9% to 58.4% and on REVERIE-CE from 40.1% to 49.5%.
  • Dynamic scenes are tractable without retraining: when the target object is moved during a real-robot run, Dynam3D still reaches 45% success, against 20% for NaVid and 10% for g3D-LF.
  • Video frames are not necessary for strong monocular VLN: a 3.8B model with layered 3D tokens beats the 7B video-based Uni-NaVid on R2R-CE Val Unseen (52.9 vs 47.0 SR), so representation quality can substitute for model scale.
  • Instance-zone tokens carry the long-horizon signal: ablations show dropping them cuts REVERIE-CE success rate by about 15 points, meaning local patch-level rendering alone is insufficient for target localization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A clean stress test would regenerate NavRAG-CE test instructions from a fresh prompt distribution or LLM; if the margin over video baselines then shrinks to the R2R-CE margin, part of the claimed NavRAG-CE gain reflects distributional memory rather than generalization to unseen user demands.
  • Because the 3D tokens live in CLIP's semantic space, the same representation could be plugged into other CLIP-based VLMs or next-step predictors as a drop-in memory module, allowing tests of whether the gains transfer without retraining the representation model.
  • The paper's Limitations section explicitly says the model does not output the target instance's coordinates and cannot answer questions or update tasks; decoding those coordinates from the instance tokens and adding a dialogue head are the natural next steps toward mobile manipulation and interactive replanning.
  • The subspace-contrastive alignment is a local fix with wider scope: any online 3D learner that distills CLIP features from multiple views could subtract the view's semantic center before contrastive learning to suppress per-view bias.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Dynam3D, a monocular vision-and-language navigation (VLN) system that replaces raw video frames with a dynamically updated hierarchical 3D representation. Posed RGB-D frames are encoded by CLIP, projected into 3D patch feature points, aggregated into FastSAM-based 3D instance tokens, and further aggregated into zone-level tokens, with a frustum-culling mechanism for removing outdated features and a merging discriminator for online instance updates. These tokens are fed into a 3.8B LLaVA-Phi-3-mini model that predicts atomic navigation actions. The authors report state-of-the-art results on R2R-CE, REVERIE-CE, and NavRAG-CE, and additional experiments on pre-exploration, lifelong memory, and real-world static and dynamic environments.

Significance. If the results hold, Dynam3D would be a meaningful advance over video-frame VLN systems: it demonstrates that language-aligned, dynamically updated 3D tokens can provide better spatial understanding and long-term memory for embodied navigation, with a smaller LLM than video-based counterparts. The core idea is coherent and well motivated, and the paper includes several strengths: the frustum-culling and merging mechanism is a sensible approach to dynamic scene updates, the subspace contrastive distillation is a thoughtful remedy for view-dependent CLIP feature bias, the code is released, and the real-robot experiments, including moving targets, address practical deployment. However, the headline SOTA claim is currently under-supported by the evaluation protocol, most importantly because the NavRAG-CE benchmark shares its data source with the training set and because the R2R-CE comparison is not controlled for training data.

major comments (3)
  1. [§4.1, Table 2; Supplementary A.1] The NavRAG-CE evaluation does not currently establish generalization to unseen instructions. The training set is described as "model-generated data (ScaleVLN, NavRAG; 4M+)" and the NavRAG-CE test set is created "by transferring NavRAG datasets to continuous environments." The paper neither reports how NavRAG episodes were partitioned between the 4M+ training samples and the NavRAG-CE test split nor provides any overlap analysis. Since NavRAG is also the authors' own dataset, the reported 24.7 versus 21.4 SR advantage over g3D-LF on NavRAG-CE could reflect memorization of the training distribution rather than generalization to new user demands. The authors should disclose the split, remove any overlapping episodes, and re-report NavRAG-CE, or alternatively drop the NavRAG-CE clause from the SOTA claim.
  2. [§4.1, Table 1] The R2R-CE SOTA comparison is not controlled for training data. All baseline numbers in Table 1 are taken from prior publications and are not retrained under the paper's protocol, while Dynam3D is trained on 4M+ model-generated instruction-trajectory pairs in addition to human-annotated R2R-CE and REVERIE-CE data (Supplementary A.1). The +5.7 SR gain over g3D-LF on Val Unseen may therefore be due to the much larger training set rather than to the proposed patch-instance-zone representation. A controlled comparison with g3D-LF and at least one video-based VLM retrained on the same navigation data is needed to support the R2R-CE SOTA claim.
  3. [§4.1, Tables 1–2] The claim of "state-of-the-art performance on VLN benchmarks including R2R-CE, REVERIE-CE and NavRAG-CE" is stronger than the reported evidence. Only R2R-CE is evaluated on a Test split; REVERIE-CE and NavRAG-CE are reported on Val Unseen only, and NavRAG-CE is a self-constructed benchmark without an independent test set. The abstract and Section 4.1 should be revised to say "validation" for the latter two benchmarks unless test-split results are provided.
minor comments (6)
  1. [§3.1, Eq. (2)] The notation in Eq. (2) is inconsistent: the depth map is indexed by row h and column w, but the pixel condition is written with u and v, and the inequalities 0<u<H and 0<v<W appear to swap width and height. Please correct the subscripts and coordinate ranges.
  2. [§4.1, Table 2 caption] The statement in Section 4.1 that NaVid and g3D-LF were retrained on the paper's training dataset should be repeated in the table caption, together with the exact training split and number of optimization steps used for these baselines.
  3. [Tables 1, 2, 3, 4, 5] All metrics are reported from a single run without error bars or multiple seeds; given the stochasticity of navigation evaluation, reporting variance over at least three seeds would make the SOTA comparisons substantially more convincing.
  4. [§4.2, Table 3] The Pre-exploration setting uses navigable viewpoints annotated in Matterport3D; the paper should clarify whether this provides the agent with ground-truth navigability information and how the procedure would be realized in an unseen environment without such annotations.
  5. [§4.3, Tables 4–5] The real-world dynamic experiments are based on only 20 test cases per setting; please state whether the target-move procedure and the success criterion were fixed in advance and report per-case statistics, since the 45% SR in Table 5 rests on a small sample.
  6. [§3.2] The efficiency comparison with Uni-NaVid refers to the LLM parameter count (3.8B versus 7B), but the full Dynam3D system also includes CLIP, FastSAM, the generalizable feature field, and the instance and zone encoders; the efficiency claim should be scoped accordingly.

Circularity Check

1 steps flagged · score 6.0 of 10

NavRAG-CE evaluation uses the same self-authored NavRAG data source as the training pool; without a disclosed split, that benchmark's SOTA claim is not a held-out generalization result.

  1. fitted input called prediction [Supplementary A.1, 'Navigation Datasets and Training Details'; claim in Abstract and Sec. 4.1]
    "To train our 3D-VLM with sufficient navigation data, we transfer datasets generated by ScaleVLN [56] and NavRAG [4] from discrete environments to the continuous Habitat simulator [22]. After removing samples with impassable paths, we obtain 4M+ instruction-trajectory pairs in continuous settings. For a comprehensive and fair evaluation, we evaluate our model on R2R-CE [3], REVERIE-CE and NavRAG-CE by transferring REVERIE [2] and NavRAG [4] datasets to continuous environments."

    The manuscript's model-generated training pool is sampled from '(ScaleVLN, NavRAG; 4M+)', while the NavRAG-CE evaluation set is created 'by transferring ... NavRAG [4] datasets to continuous environments.' No train/test split, episode de-duplication, or overlap analysis is reported between these two uses of the same NavRAG data. Consequently, for the NavRAG-CE benchmark, the test episodes may be drawn from the same self-authored data distribution on which the 3D-VLM was fitted. If the same instruction-trajectory pairs or scenes appear in both pools, the reported NavRAG-CE gains (SR 24.7 vs 17.4-21.4) partly measure memorization of the training distribution, not generalization to unseen user demands.

full rationale

The central R2R-CE and REVERIE-CE results are anchored to external, standard benchmarks, and the representation model is pretrained on external 3D datasets (ScanNet, HM3D, Matterport3D, 3RScan, ARKitScenes, Structured3D) plus SceneVerse/g3D-LF; those components do not reduce to their inputs. The dynamic 3D patch-instance-zone mechanism is not defined in terms of the benchmark scores, so it is not self-definitional. The only load-bearing circularity is the NavRAG-CE portion of the SOTA claim. Supplementary A.1 builds the 4M+ training set from ScaleVLN and NavRAG and also creates NavRAG-CE by transferring NavRAG, with no disclosed split between training episodes and evaluation episodes. Since the NavRAG benchmark is a self-citation by the same first author and senior author, the training-data-to-evaluation chain is closed inside the authors' own dataset. Without evidence of disjointness, the claimed NavRAG-CE generalization result is not a held-out prediction. This does not invalidate the strong R2R-CE (SR 52.9) or REVERIE-CE results, so the paper is only partially circular rather than wholly so.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several pretrained components (CLIP, FastSAM, g3D-LF feature fields, LLaVA-Phi-3-mini) and on hand-chosen thresholds that are not reported or ablated. No new physical entities are introduced; the patch-instance-zone representation is a methodological construct. The weakest support is the assumption that CLIP feature distillation transfers to unseen 3D objects in dynamic scenes.

free parameters (4)
  • Frustum culling thresholds delta and Delta = not reported
    Equation (2) removes feature points based on depth margin delta and farthest distance Delta. No sensitivity analysis or chosen values are provided, and results are likely to depend on these thresholds.
  • 3D zone size = several cubic meters (unspecified)
    Section 3.1 partitions world space into uniform cubic zones. The exact size is not given and is a hand-chosen hyperparameter that controls the granularity of zone tokens.
  • Contrastive temperature tau = not reported
    Used in Equations (5)-(7) for instance and zone language alignment and distillation. The value is not specified in the paper.
  • Merging Top-K = not reported
    Section 3.1: for each new 2D instance, the Top-K nearest existing 3D instances are retrieved. K is not specified and affects merging accuracy.
assumptions (6)
  • domain assumption CLIP image features are sufficiently language-aligned and transferable into 3D to support navigation.
    The whole method projects CLIP patch features into 3D and aligns instance and zone tokens to CLIP space (Section 3.1), assuming that this alignment preserves the semantic information needed for navigation.
  • domain assumption Posed RGB-D inputs provide accurate depth and camera poses.
    All 3D projections and frustum culling in Equations (1)-(2) rely on calibrated depth and pose, both in the Habitat simulator and on the Stretch robot. Errors in pose or depth would corrupt the 3D representation.
  • domain assumption FastSAM 2D instance masks are reliable enough for merging into 3D instances.
    Section 3.1: instance-level representations are constructed by aggregating patch features within FastSAM masks. Mask errors propagate to 3D instances and affect target localization.
  • domain assumption The generalizable feature field (g3D-LF) renders accurate agent-centric patch tokens.
    Section 3.2 adopts g3D-LF's feature field for panoramic 3D patch tokens. Any rendering bias affects the inputs to the VLM and therefore the predicted actions.
  • domain assumption The pretrained LLaVA-Phi-3-mini can reason over CLIP-aligned 3D tokens without additional modality-specific training.
    Section 3.2 claims that the 2D VLM's reasoning transfers to 3D tokens by alignment to CLIP space. This is an empirical assumption and is not proven beyond the reported navigation results.
  • domain assumption Merging discriminator trained on 5K static annotated rooms generalizes to unseen and dynamic scenes.
    Section 3.1 trains the discriminator on ScanNet, HM3D, Matterport3D, and 3RScan. No domain-gap analysis is provided for real-world dynamic environments, where object motion and novel objects are common.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation." pith.science (2026). https://pith.science/paper/3EKOQTDC

@misc{pith2026250511383,
  author       = {Pith},
  title        = {Pith review of: Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3EKOQTDC}},
  note         = {Machine review of arXiv:2505.11383}
}
read the original abstract

Vision-and-Language Navigation (VLN) is a core task where embodied agents leverage their spatial mobility to navigate in 3D environments toward designated destinations based on natural language instructions. Recently, video-language large models (Video-VLMs) with strong generalization capabilities and rich commonsense knowledge have shown remarkable performance when applied to VLN tasks. However, these models still encounter the following challenges when applied to real-world 3D navigation: 1) Insufficient understanding of 3D geometry and spatial semantics; 2) Limited capacity for large-scale exploration and long-term environmental memory; 3) Poor adaptability to dynamic and changing environments.To address these limitations, we propose Dynam3D, a dynamic layered 3D representation model that leverages language-aligned, generalizable, and hierarchical 3D representations as visual input to train 3D-VLM in navigation action prediction. Given posed RGB-D images, our Dynam3D projects 2D CLIP features into 3D space and constructs multi-level 3D patch-instance-zone representations for 3D geometric and semantic understanding with a dynamic and layer-wise update strategy. Our Dynam3D is capable of online encoding and localization of 3D instances, and dynamically updates them in changing environments to provide large-scale exploration and long-term memory capabilities for navigation. By leveraging large-scale 3D-language pretraining and task-specific adaptation, our Dynam3D sets new state-of-the-art performance on VLN benchmarks including R2R-CE, REVERIE-CE and NavRAG-CE under monocular settings. Furthermore, experiments for pre-exploration, lifelong memory, and real-world robot validate the effectiveness of practical deployment.

Figures

Figures reproduced from arXiv: 2505.11383 by the authors.

Figure 1
Figure 1. Different vision-language large models for monocular VLN tasks. Compared to previous [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our Dynam3D framework. Our Dynam3D takes posed monocular RGB [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Left: Illustration of the feature points update and frustum culling strategy. Right: The [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A demonstration of navigation in a dynamic real-world environment. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation

    cs.RO 2026-07 conditional novelty 7.0 of 10

    A new physics-enabled benchmark with 10,000 indoor, outdoor, and indoor-to-outdoor navigation episodes shows current zero-shot agents fail most when crossing the indoor-outdoor boundary, especially on the new PlaceNav task.

Reference graph

Works this paper leans on

60 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683, 2018

    2018

  2. [2]

    Reverie: Remote embodied visual referring expression in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9982–9991, 2020

    2020

  3. [3]

    Beyond the nav- graph: Vision-and-language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and-language navigation in continuous environments. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16, pages 104–120. Springer, 2020

    2020

  4. [4]

    Navrag: Generating user de- mand instructions for embodied navigation through retrieval-augmented llm

    Zihan Wang, Yaohui Zhu, Gim Hee Lee, and Yachun Fan. Navrag: Generating user de- mand instructions for embodied navigation through retrieval-augmented llm. arXiv preprint arXiv:2502.11142, 2025

    arXiv 2025

  5. [5]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation. In Proceedings of Robotics: Science and Systems (RSS), 2024

    work page 2024

  6. [6]

    Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks

    Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks. arXiv preprint arXiv:2412.06224, 2024

    arXiv 2024

  7. [7]

    Navila: Legged robot vision-language-action model for navigation

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Xueyan Zou, Jan Kautz, Erdem Biyik, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation. In RSS, 2025

    work page 2025

  8. [8]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    arXiv 2023

Show all 60 references
  1. [9]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26689–26699, 2024

    2024

  2. [10]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023...

    2023

  3. [11]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  4. [12]

    Fast segment anything

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306.12156, 2023

    2023 arXiv

  5. [13]

    Embodied- sam: Online segment any 3d thing in real time

    Xiuwei Xu, Huangxing Chen, Linqing Zhao, Ziwei Wang, Jie Zhou, and Jiwen Lu. Embodied- sam: Online segment any 3d thing in real time. In The Thirteenth International Conference on Learning Representations

  6. [14]

    g3d-lf: Generalizable 3d-language feature fields for embodied tasks

    Zihan Wang and Gim Hee Lee. g3d-lf: Generalizable 3d-language feature fields for embodied tasks. arXiv preprint arXiv:2411.17030, 2024

    2024 arXiv

  7. [15]

    Vln bert: A recurrent vision-and-language bert for navigation

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. Vln bert: A recurrent vision-and-language bert for navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 1643–1653, 2021. 10

    2021

  8. [16]

    History aware multimodal transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. Advances in neural information processing systems, 34:5834–5847, 2021

    2021

  9. [17]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16537–16547, 2022

    2022

  10. [18]

    Vision- and-language navigation via causal learning

    Liuyi Wang, Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision- and-language navigation via causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13139–13150, 2024

    2024

  11. [19]

    Hop+: History- enhanced and order-aware pre-training for vision-and-language navigation

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History- enhanced and order-aware pre-training for vision-and-language navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8524–8537, 2023

    2023

  12. [20]

    Bird’s-eye-view scene graph for vision- language navigation

    Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird’s-eye-view scene graph for vision- language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10968–10980, 2023

    2023

  13. [21]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2017

    2017

  14. [22]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, ...

    2019

  15. [23]

    Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation

    Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022

    2022

  16. [24]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15625–15636, 2023

    2023

  17. [25]

    Etpnav: Evolving topological planning for vision-language navigation in continuous environments

    Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  18. [26]

    Bevbert: Multimodal map pre-training for language-guided navigation

    Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Multimodal map pre-training for language-guided navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2737–2748, 2023

    2023

  19. [27]

    Sim-to-real transfer via 3d feature fields for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Sim-to-real transfer via 3d feature fields for vision-and-language navigation. In 8th Annual Conference on Robot Learning, 2024

    2024

  20. [28]

    Open-nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms

    Yanyuan Qiao, Wenqi Lyu, Hui Wang, Zixu Wang, Zerui Li, Yuan Zhang, Mingkui Tan, and Qi Wu. Open-nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms. In Proceedings of the IEEE International Conference on Robotics and Automat...

    2025

  21. [29]

    Navgpt-2: Un- leashing navigational reasoning capability for large vision-language models

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Un- leashing navigational reasoning capability for large vision-language models. arXiv preprint arXiv:2407.12366, 2024

    2024 arXiv

  22. [30]

    Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K. Wong. Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. 11

    2024

  23. [31]

    Jiaqi Chen, Bingqian Lin, Xinmin Liu, Lin Ma, Xiaodan Liang, and Kwan-Yee K. Wong. Affordances-oriented planning using foundation models for continuous vision-language naviga- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025

    2025

  24. [32]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  25. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  26. [34]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  27. [35]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168, 2023

    2023 arXiv

  28. [36]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024

    2024

  29. [37]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  30. [38]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023

    2023

  31. [39]

    Video-3d llm: Learning position-aware video representation for 3d scene understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. arXiv preprint arXiv:2412.00493, 2024

    2024 arXiv

  32. [40]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    2024 arXiv

  33. [41]

    Lookahead exploration with neural radiance representation for continuous vision-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Junjie Hu, Ming Jiang, and Shuqiang Jiang. Lookahead exploration with neural radiance representation for continuous vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  34. [42]

    Learning generalizable feature fields for mobile manipulation

    Ri-Zhao Qiu, Yafei Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye, Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, et al. Learning generalizable feature fields for mobile manipulation. arXiv preprint arXiv:2403.07563, 2024

    2024 arXiv

  35. [43]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017

    2017

  36. [44]

    Habitat- matterport 3d semantics dataset

    Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, et al. Habitat- matterport 3d semantics dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2023

  37. [45]

    Rio: 3d object instance re-localization in changing indoor environments

    Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re-localization in changing indoor environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7658–7667, 2019

    2019

  38. [46]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision (ECCV), 2024. 12

    2024

  39. [47]

    Feature splatting: Language-driven physics-based scene synthesis and editing

    Ri-Zhao Qiu, Ge Yang, Weijia Zeng, and Xiaolong Wang. Feature splatting: Language-driven physics-based scene synthesis and editing. arXiv preprint arXiv:2404.01223, 2024

    2024 arXiv

  40. [48]

    Xtuner: A toolkit for efficiently fine-tuning llm

    XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/ InternLM/xtuner, 2023

    2023

  41. [49]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  42. [50]

    Cross-modal map learning for vision and language navigation

    Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15460–15...

    2022

  43. [51]

    Weakly-supervised multi-granularity map learning for vision-and-language navigation

    Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas Li, Mingkui Tan, and Chuang Gan. Weakly-supervised multi-granularity map learning for vision-and-language navigation. Advances in Neural Information Processing Systems, 35:38149–38161, 2022

    2022

  44. [52]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. In 8th Annual Conference on Robot Learning, 2024

    2024

  45. [53]

    Affordances-oriented planning using foundation models for continuous vision-language nav- igation

    Jiaqi Chen, Bingqian Lin, Xinmin Liu, Lin Ma, Xiaodan Liang, and Kwan-Yee K Wong. Affordances-oriented planning using foundation models for continuous vision-language nav- igation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23568–23576, 2025

    2025

  46. [54]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. In Thirty-fifth Conference on N...

  47. [55]

    Structured3d: A large photo-realistic dataset for structured 3d modeling

    Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Proceedings of The European Conference on Computer Vision (ECCV), 2020

    2020

  48. [56]

    Scaling data generation in vision-and-language navigation

    Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12009–12020, 2023

    2023

  49. [57]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and C...

    2011

  50. [58]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018

    2018

  51. [59]

    Training deep nets with sublinear memory cost

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016

    2016 arXiv

  52. [60]

    Dynamem: Online dynamic spatio-semantic memory for open world mobile manipulation

    Peiqi Liu, Zhanqiu Guo, Mohit Warke, Soumith Chintala, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Dynamem: Online dynamic spatio-semantic memory for open world mobile manipulation. In CoRL 2024 Workshop on Mastering Robot Manipulation in a World of Abundant ...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.