Pith. sign in

REVIEW 4 major objections 5 minor 74 references

g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a 3D-language feature field pre-trained on ~1M indoor captions yields volume-rendered views, panoramas, and bird's-eye maps that transfer to unseen rooms and lift navigation, object-goal, and situated-QA baselines to…

desk verdict Solid incremental contribution; the generalization claim needs explicit split-disjointness verification and error bars, but the ablations and method merit a serious referee. read the letter →

arxiv 2411.17030 v1 pith:JV6A2XAX submitted 2024-11-26 cs.CV cs.AIcs.RO

Zihan Wang , Gim Hee Lee This is my paper · ORCID
classification cs.CVcs.AIcs.RO
keywords 3DfeaturefieldslanguagegroundingembodiedAIvision-and-languagenavigationzero-shotobjectsituatedquestionansweringvolumerenderingcontrastivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

g3D-LF is a pre-trained 3D representation for embodied agents: from posed RGB-D images it builds a feature field and volume-renders novel views, 12-view panoramas, and 16.8-meter bird's-eye maps at interactive frame rates. The paper's claim is that these representations, aligned to language through multi-level contrastive learning over about 5,000 indoor scenes and nearly one million captions, generalize to environments the model has not seen during training. Plugged into existing baselines, the representations raise monocular and panoramic vision-and-language navigation, zero-shot object navigation, and situated question answering to state-of-the-art or competitive scores. The reason a sympathetic reader should care: it suggests that one reusable, language-queryable spatial memory can replace task-specific perception stacks across several embodied tasks.

What carries the argument

The central object is a generalizable 3D-language feature field: a set of CLIP feature points stored at world coordinates derived from posed RGB-D frames, queried by k-nearest-neighbor aggregation, and rendered along rays by volume density weighting. The mechanism that carries the argument is multi-level contrastive pre-training, which aligns ray-rendered features at three scales (novel view, panorama, bird's-eye map) with language at three granularities (object names over a 1,883-category vocabulary, object descriptions, and long spatial texts matched word-by-word through an affinity matrix). The balanced object loss, the fine-grained top-k text-matching loss, and CLIP knowledge distillation together are what make the field transferable to unseen scenes and queryable by language.

What would settle it

Compare scene identifiers: if any SQA3D validation or test scene shares a ScanNet scan ID with the ScanNet train split used in pre-training, the reported SQA3D localization numbers (e.g., Acc@0.5m rising from 14.6 to 23.4) would be inflated by the model having memorized test rooms. The clean check is to re-run g3D-LF pre-training with all downstream evaluation scenes held out entirely and confirm the gains in Tables 1 through 4 persist.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that scalable 3D-language supervision can make generalizable feature fields useful for embodied tasks. g3D-LF encodes each posed RGB-D frame with a CLIP image encoder, places the per-patch features at 3D world coordinates, and stores a continuously updated field of feature points. Two MLPs — one for view rays and one for top-down bird's-eye rays — predict volume density and latent features at query points from the k nearest stored features, and volume rendering composites these into a 12x12 novel-view feature map, a 12-view panorama, and a 168x168 bird's-eye map covering a 16.8 m x 16.8 m area centered on the agent. Three contrastive objectives tie these renderings to language at increasing granularity: a balanced object-vocabulary loss over 1,883 indoor categories, a fine-grained word-level contrastive loss that matches bird's-eye windows and panorama views to tokens of long spatial text via a top-k affinity matrix, and CLIP distillation that keeps the features grounded in a web-scale visual embedding. Pre-trained on about 5,000 scenes and nearly one million captions, the model is then used to produce waypoint representations for vision-and-language navigation, value maps for zero-shot object navigation, and localization plus answer features for situated question answering, where it improves several baselines to state of the art.

Load-bearing premise

The pre-training scenes (ScanNet train, HM3D train, Structured3D) are actually disjoint from the scenes used in evaluation, especially SQA3D's validation and test scenes which live in ScanNet buildings, so the model has not already seen the test rooms; the paper states it removed val and test splits but does not specify the exact split convention or verify scene-by-scene overlap.

Editorial extensions

If this is right

  • Monocular agents can achieve panoramic perception: g3D-LF predicts 12 novel views around the agent from forward-only observations, and using them in monocular vision-and-language navigation raises val-unseen Success Rate from 44.9 to 47.2 and SPL from 30.4 to 34.6 (Table 1, rows 2 and 4 of Table 5).
  • The same volume-rendered bird's-eye and view features, without any 2D vision-language model at inference, replace the BLIP-2 value map in VLFM and improve zero-shot object navigation on HM3D (SR 52.5 to 55.6, SPL 30.4 to 31.8) and MP3D (SR 36.4 to 39.0, SPL 17.5 to 18.8).
  • Fine-grained word-level contrastive learning, rather than a single text-vector loss, is what enables the long-text understanding needed for vision-and-language navigation (ablated in Table 6, row 5 versus row 6).
  • Every module earns its keep: ablating the novel views or the bird's-eye map drops vision-and-language navigation and object navigation scores (Table 5), and pre-training without language supervision drops both tasks substantially (Table 6, row 1 versus row 6).
  • The representation is usable in real time: novel views render at 73.6 FPS, bird's-eye maps at 6.3 FPS, and full panoramas at 5.9 FPS (Table 7).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the paper never tests dynamic scenes, so the strongest untested extension is whether the same field can be incrementally updated when objects move; the authors list this as a limitation.
  • Editorial: because the object vocabulary is fixed at 1,883 indoor categories, the open-vocabulary claim in zero-shot navigation ultimately leans on CLIP; a natural benchmark is evaluating targets entirely absent from the vocabulary and measuring how much the top-k text matching compensates.
  • Editorial: the generalization claim rests on scene split hygiene; a scene-disjoint stress test would also reveal which pre-training corpus (ScanNet, HM3D, or Structured3D) contributes the transfer.
  • Editorial: if a single pre-trained field can boost three task families, it could also serve as the spatial memory component in a larger embodied agent with a language model head, a direction the authors name as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces g3D-LF, a 3D representation model pre-trained on a large-scale 3D-language dataset comprising about 5K indoor scenes and nearly one million language descriptions. The model encodes posed RGB-D observations into feature fields and uses volume rendering plus multiscale encoders to predict novel-view, panorama, and BEV representations, which are aligned with multi-granularity language via a multi-level contrastive objective (object-level alignment, fine-grained long-text contrast, and CLIP knowledge distillation). The pre-trained representation is then integrated into existing baselines for four embodied tasks: monocular VLN, panorama VLN, zero-shot object navigation, and situated question answering. The paper reports consistent improvements over the corresponding baselines and abalations showing that each proposed loss and module contributes. The headline claim is that g3D-LF generalizes to unseen environments and improves multiple baseline methods to state-of-the-art performance.

Significance. If the claims hold, g3D-LF is a meaningful step toward reusable 3D-language representations for embodied agents: it consolidates a large-scale language-aligned training set, provides a self-contained pre-training recipe, and demonstrates broad applicability across navigation, object-goal search, and situated QA. The ablations (Tables 5 and 6) are informative and show that the proposed components each matter. The paper also makes code available and includes runtime measurements that support the real-time usability claim. The main significance is conditioned on a careful verification of the train/evaluation split disjointness, which is not currently established in the manuscript.

major comments (4)
  1. [§4.1, 'g3D-LF Pre-training'] The statement 'all training data only includes the train split, the val and test splits are removed' is ambiguous and load-bearing for the generalization claim. SQA3D is built on ScanNet scenes and the paper also pre-trains g3D-LF on ScanNet scenes; the paper never specifies which ScanNet split convention is used for the SQA3D train/val/test partitions, nor does it verify that the SQA3D val/test scene IDs are disjoint from the ScanNet train scenes used for pre-training. Similarly, for the HM3D object-navigation evaluation, the paper does not explicitly intersect the HM3D validation scene IDs with the HM3D train scenes used for pre-training. As written, the results in Table 4 (SQA3D) and Table 3 (HM3D) could reflect scene-level leakage rather than generalization to unseen environments. The authors must report the exact split rule for each corpus and provide a concrete scene-ID intersection check (e.g., a table or a statement of overlap) for the ScanNet, HM3D, and MP3D splits used in downstream evaluation.
  2. [Abstract and Table 4] The abstract concludes that g3D-LF 'improves multiple baseline methods to state-of-the-art performance across various embodied tasks,' but the SQA3D results in Table 4 do not support this for that task. The EM@1 of 47.7 is lower than several published methods (SceneVerse 49.9, 3D-VisTA 48.5, LEO 52.4, Scene-LLM 54.2). The task-specific discussion in §4.2 acknowledges this, but the abstract and the conclusion in §5 do not qualify the SOTA claim. The claim should be softened to localization-specific SOTA (Acc@0.5m, Acc@1.0m, Acc@15°, Acc@30°) and competitive accuracy, or the authors should provide evidence of SOTA on the QA metric from an equivalent image-only setting.
  3. [Tables 1–6] No error bars, confidence intervals, or multiple seeds are reported anywhere in the experimental section. This is especially concerning because several reported differences over the strongest baselines are small (e.g., Table 2: SR 61 vs 61 for val unseen HNR; Table 3: SR 55.6 vs 53.1 for HM3D), and it is impossible to tell whether the improvements are statistically significant. The authors should report results over at least three independent runs (or a comparable protocol) with means and standard deviations for at least the main comparisons in Tables 1–4, and ideally also for the ablations.
  4. [§3.3 and Supplementary Material] Several hyperparameters that are essential for reproducing the pre-training are not specified in the paper: the temperature/logit scale τ (the supplementary code uses logit_scale=10, but the main text does not state it), the balanced-loss scaling factor α in §3.3, the relative weights of L_object, L_view_clip, L_pano_clip, L_bev_clip, L_long_text, and the number of top-k words used in fine-grained contrastive learning in Eq. (5). The paper states only that 'constant coefficients' keep the losses in the same order of magnitude. These values materially change the trained model, and the results cannot be reproduced without them. A complete hyperparameter table (including the auxiliary losses used for the downstream task heads, if any) should be added.
minor comments (5)
  1. [§4.1] The word 'splited' appears in the SQA3D paragraph; it should be 'split.'
  2. [§4.2] The sentence 'Compared to methods using LLM: InstructNav and SG-Nav, our g3D-LF also offers significant advantages in response time and computational cost' is not backed by any timing or computational comparison in the paper. Either provide measurements or remove the claim.
  3. [§4.3] There are typographical issues: 'Perfromance impact' in the subsection heading, 'advanatages' in §4.2, and 'samesparse' in the runtime paragraph. These should be corrected.
  4. [§5, Limitations] The limitations paragraph says the dataset is limited in 'scale and quality,' but the paper does not state how the dataset will be released or whether the annotations are fully originated from the author's curation versus inherited from SceneVerse, ScanRefer, and VLA-3D. A short data-availability and licensing statement would help readers and prospective users.
  5. [Table 2] The caption states 'g3D-LF (Ours) × 4.53 68 61 52' but the table's SPL column under Test Unseen shows 51; please ensure the reported numbers are internally consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: g3D-LF is trained on external 3D-language annotations and evaluated on held-out embodied benchmarks.

full rationale

The derivation chain is linear: posed RGB-D observations are encoded into a feature field, volume rendering produces novel-view, panorama, and BEV representations, and multi-level contrastive losses align those representations with CLIP features, an object vocabulary, and long-text annotations. Each supervision signal is external to the downstream tasks: CLIP, SceneVerse/ScanRefer language data, and standard benchmark splits. The claimed predictions, such as novel views and BEV maps in unseen scenes, are rendered from the field rather than retrieved from a fitted target. Self-citations to HNR and VLN-3DFF are prior published baselines that are compared against, not invoked as evidence for the central generalization claim; they are externally falsifiable and do not make the derivation circular. The only load-bearing uncertainty is scene-split disjointness: Section 4.1 states 'all training data only includes the train split, the val and test splits are removed' without specifying the exact partition for SQA3D/ScanNet, which is a potential data-leakage/correctness concern, not a circularity because no equation, fit, or self-citation reduces the evaluation target to the training input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on several domain assumptions and hand-set hyperparameters; the most consequential is the unverified split overlap between pretraining and evaluation data.

free parameters (4)
  • Contrastive temperature tau / logit scale = logit_scale=10 in code (Fig. 7)
    Hand-set scaling in all contrastive losses (Eq. 3-7); value affects loss sharpness and is not derived from data.
  • Balanced loss scaling factor alpha = not specified
    Section 3.3: weights for top 10% hardest rays are increased by alpha, but alpha is not given; affects pretraining and downstream performance.
  • K-nearest search radii = 0.5 m (view), 0.4 m (BEV)
    Supplementary A: search radius for feature aggregation; chosen by hand, influences density of rendered features.
  • Relative loss weights = not specified
    Section 3.3 says 'constant coefficients' balance each loss but values are not reported; important for multi-task pretraining.
assumptions (5)
  • domain assumption Posed RGB-D images and depth maps are available and accurate.
    Sec. 3.2: feature points are mapped to 3D world coordinates using depth and camera parameters; noisy depth would corrupt the feature field.
  • domain assumption Instance-level point clouds and language annotations from SceneVerse/ScanRefer are correct enough for supervision.
    Sec. 3.1: language labels are retrieved by nearest instance point; annotation errors directly bias contrastive learning.
  • domain assumption The pre-training scenes are disjoint from evaluation scenes.
    Sec. 4.1 states val/test splits are removed, but overlap with downstream benchmark splits (especially SQA3D/ScanNet) is not verified; the claim of generalization depends on this.
  • domain assumption CLIP text and image features provide a suitable semantic space for multi-granularity language alignment.
    Sec. 3.3: CLIP encoders are used as frozen teachers/anchors; if CLIP cannot capture spatial layouts or long-text relations, the alignment signal is weak.
  • ad hoc to paper Volume rendering of latent features preserves object identity and spatial layout.
    Sec. 3.2: the MLP predicts latent r and density sigma from k-nearest features; this is a design choice not guaranteed by any theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks." pith.science (2026). https://pith.science/paper/JV6A2XAX

@misc{pith2026241117030,
  author       = {Pith},
  title        = {Pith review of: g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JV6A2XAX}},
  note         = {Machine review of arXiv:2411.17030}
}
read the original abstract

We introduce Generalizable 3D-Language Feature Fields (g3D-LF), a 3D representation model pre-trained on large-scale 3D-language dataset for embodied tasks. Our g3D-LF processes posed RGB-D images from agents to encode feature fields for: 1) Novel view representation predictions from any position in the 3D scene; 2) Generations of BEV maps centered on the agent; 3) Querying targets using multi-granularity language within the above-mentioned representations. Our representation can be generalized to unseen environments, enabling real-time construction and dynamic updates. By volume rendering latent features along sampled rays and integrating semantic and spatial relationships through multiscale encoders, our g3D-LF produces representations at different scales and perspectives, aligned with multi-granularity language, via multi-level contrastive learning. Furthermore, we prepare a large-scale 3D-language dataset to align the representations of the feature fields with language. Extensive experiments on Vision-and-Language Navigation under both Panorama and Monocular settings, Zero-shot Object Navigation, and Situated Question Answering tasks highlight the significant advantages and effectiveness of our g3D-LF for embodied tasks.

Figures

Figures reproduced from arXiv: 2411.17030 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our g3D-LF model. Our model encodes the observed RGB-D images into the feature fields (consists of many feature points). Through aggregating k-nearest features, the MLP networks predict the latent feature and volume density of sampled points along the rendered ray. The hierarchical encoders further generate representations of novel view, panorama, and BEV map, then conduct multi-level contrastive learnin… view at source ↗
Figure 3
Figure 3. Monocular VLN framework based on VLN-3DFF [58]. + The toilet CLIP Text Encoder X Cosine similarity The toilet X Cosine similarity Local value map Global value map Point-goal Navigation Detection Model Controller Action CLIP Text Encoder [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Zero-shot object navigation framework based on VLFM [62]. HNR [57], a waypoint predictor [20] is used to predict candi￾date waypoints, and our g3D-LF model generates panorama representations of these waypoints for navigation planning. Zero-shot Object Navigation. As sh…
Figure 5
Figure 5. Figure 5: The framework of situated question answering [ [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Architecture of modules in the g3D-LF model. FC de￾notes a fully connected layer, LN denotes layer normalization and LeakyReLU [38] is the activation function. A. More Details of the g3D-LF Model Model structure [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 8
Figure 8. Figure 8: PyTorch implementation of loss function for the fine-grained contrastive learning. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Demonstration of a 3D scene in the training data. Instance-level point clouds mark all instances with object categories, and some [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Visualization of querying objects on rays of the [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Visualization of querying long texts on the BEV map of our [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 57 canonical work pages

  1. [1]

    Bevbert: Multimodal map pre-training for language-guided navigation

    Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Multimodal map pre-training for language-guided navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 2737–2748, 2023. 2, 7

    work page 2023

  2. [2]

    Etpnav: Evolving topo- logical planning for vision-language navigation in continuous environments

    Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. Etpnav: Evolving topo- logical planning for vision-language navigation in continuous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 2, 5, 7, 8

    work page 2024

  3. [3]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S ¨underhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683,

  4. [4]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19129– 19139, 2022. 1, 3, 7

    work page 2022

  5. [5]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Hal- ber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2017. 6

    work page 2017

  6. [6]

    Object goal navigation using goal-oriented semantic exploration

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhi- nav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247–4258, 2020. 1, 2

    work page 2020

  7. [7]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202–221. Springer, 2020. 2, 3

    work page 2020

  8. [8]

    Weakly- supervised multi-granularity map learning for vision-and- language navigation

    Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas Li, Mingkui Tan, and Chuang Gan. Weakly- supervised multi-granularity map learning for vision-and- language navigation. Advances in Neural Information Pro- cessing Systems, 35:38149–38161, 2022. 6

    work page 2022

Show all 74 references
  1. [9]

    History aware multimodal transformer for vision- and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision- and-language navigation. Advances in neural information processing systems, 34:5834–5847, 2021. 2

    2021

  2. [10]

    Think global, act lo- cal: Dual-scale graph transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act lo- cal: Dual-scale graph transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16537– 1...

    2022

  3. [11]

    Grounded 3d-llm with referent tokens

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370,

    arXiv

  4. [12]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 3, 6

    2017

  5. [13]

    Embodied question answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018. 3

    2018

  6. [14]

    Scene-llm: Extending language model for 3d visual un- derstanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual un- derstanding and reasoning. arXiv preprint arXiv:2403.11401,

    arXiv

  7. [15]

    Cows on pasture: Base- lines and benchmarks for language-driven zero-shot object navigation

    Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Base- lines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23...

    2023

  8. [16]

    Cross-modal map learning for vision and language navigation

    Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Dani- ilidis. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15460–...

    2022

  9. [17]

    Navigating to objects in the real world

    Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 8(79):eadf6991, 2023. 2

    2023

  10. [18]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 2

    2017

  11. [19]

    Vln bert: A recurrent vision-and- language bert for navigation

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. Vln bert: A recurrent vision-and- language bert for navigation. InProceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 1643–1653, 2021. 2

    2021

  12. [20]

    Bridg- ing the gap between learning in discrete and continuous envi- ronments for vision-and-language navigation

    Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridg- ing the gap between learning in discrete and continuous envi- ronments for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 5, 7

    2022

  13. [21]

    Learning naviga- tional visual representations with semantic map supervision

    Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Dernoncourt, Trung Bui, Stephen Gould, and Hao Tan. Learning naviga- tional visual representations with semantic map supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3055–3067, 2023. 7

    2023

  14. [22]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baox- iong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024. 1, 3, 7

    2024

  15. [23]

    Sceneverse: Scaling 3d vision-language learning for grounded scene un- derstanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene un- derstanding. In European Conference on Computer Vision (ECCV), 2024. 2, 3, 7

    2024

  16. [24]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF Interna- 9 tional Conference on Computer Vision, pages 19729–19739,

  17. [25]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 4015–4026, 2023. 2

    2023

  18. [26]

    Sim-2-sim transfer for vision- and-language navigation in continuous environments

    Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision- and-language navigation in continuous environments. In Eu- ropean Conference on Computer Vision (ECCV) , 2022. 2, 7

    2022

  19. [27]

    Beyond the nav-graph: Vision-and-language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part XXVIII 16, pages 104–

    2020

  20. [28]

    Renderable neural radiance map for visual navigation

    Obin Kwon, Jeongho Park, and Songhwai Oh. Renderable neural radiance map for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9099–9108, 2023. 1

    2023

  21. [29]

    Less is more: Clipbert for video-and-language learning via sparse sampling

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7331–7341, 2021. 7

    2021

  22. [30]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

  23. [31]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2022

  24. [32]

    Bird’s- eye-view scene graph for vision-language navigation

    Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird’s- eye-view scene graph for vision-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10968–10980, 2023. 2

    2023

  25. [33]

    Vision-language navigation with energy-based policy

    Rui Liu, Wenguan Wang, and Yi Yang. Vision-language navigation with energy-based policy. In Advances in Neural Information Processing Systems, 2024. 7

    2024

  26. [34]

    V olumetric envi- ronment representation for vision-language navigation

    Rui Liu, Wenguan Wang, and Yi Yang. V olumetric envi- ronment representation for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 16317–16328, 2024. 1

    2024

  27. [35]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 2

    2023 arXiv

  28. [36]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. In 8th Annual Conference on Robot Learning, 2024. 6, 7

    2024

  29. [37]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In The Eleventh Interna- tional Conference on Learning Representations, 2023. 1, 2, 3, 6, 7

    2023

  30. [38]

    Rec- tifier nonlinearities improve neural network acoustic models

    Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rec- tifier nonlinearities improve neural network acoustic models. In Proc. icml, page 3. Atlanta, GA, 2013. 13

    2013

  31. [39]

    Zson: Zero-shot object-goal navigation using multimodal goal embeddings

    Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. Advances in Neural Information Processing Systems, 35:32340–32352,

  32. [40]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mc- vay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on...

    2024

  33. [41]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM , 65(1):99–106, 2021. 2

    2021

  34. [42]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Trans- actions on Machine Learning Research Jou...

  35. [43]

    Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 45(7): 8524–8537, 2023. 2

    2023

  36. [44]

    Learning generalizable feature fields for mobile manipulation

    Ri-Zhao Qiu, Yafei Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye, Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, et al. Learning generalizable feature fields for mobile manipulation. arXiv preprint arXiv:2403.07563,

    arXiv

  37. [45]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  38. [46]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wi- jmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew West- bury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai....

  39. [47]

    Poni: Potential functions for objectgoal navigation with interaction- free learning

    Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman. Poni: Potential functions for objectgoal navigation with interaction- free learning. In Proceedings of the IEEE/CVF Conference 10 on Computer Vision and Pattern Recognition...

    2022

  40. [48]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision ,...

    2019

  41. [49]

    Distilled feature fields enable few-shot language-guided manipulation

    William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation. In Proceedings of The 7th Conference on Robot Learning , pages 405–424. PMLR, 2023. 1

    2023

  42. [50]

    Language- enhanced rnr-map: Querying renderable neural radiance field maps with natural language

    Francesco Taioli, Federico Cunico, Federico Girella, Riccardo Bologna, Alessandro Farinelli, and Marco Cristani. Language- enhanced rnr-map: Querying renderable neural radiance field maps with natural language. In Proceedings of the IEEE/CVF International Conference on Compute...

    2023

  43. [51]

    Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes

    Suhani V ora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. Transactions on Machine Learning Research. 2

  44. [52]

    Yolov7: Trainable bag-of-freebies sets new state-of- the-art for real-time object detectors

    Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of-freebies sets new state-of- the-art for real-time object detectors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 7464–7475, 2023. 2

    2023

  45. [53]

    Dreamwalker: Mental planning for continuous vision- language navigation

    Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision- language navigation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 10873– 10883, 2023. 7

    2023

  46. [54]

    Vision-and-language naviga- tion via causal learning

    Liuyi Wang, Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision-and-language naviga- tion via causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13139–13150, 2024. 2

    2024

  47. [55]

    Scaling data generation in vision-and-language navigation

    Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 12009–12020, 2023. 7

    2023

  48. [56]

    Gridmm: Grid memory map for vision- and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision- and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15625– 15636, 2023. 2, 5, 7

    2023

  49. [57]

    Lookahead exploration with neural radiance representation for continuous vision-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Junjie Hu, Ming Jiang, and Shuqiang Jiang. Lookahead exploration with neural radiance representation for continuous vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  50. [58]

    Sim-to-real transfer via 3d feature fields for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Sim-to-real transfer via 3d feature fields for vision-and-language navigation. In 8th Annual Confer- ence on Robot Learning, 2024. 1, 2, 5, 6, 7

    2024

  51. [59]

    Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames

    Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv:1911.00357, 2019. 2

    1911 arXiv

  52. [60]

    Habitat-matterport 3d semantics dataset

    Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakr- ishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, et al. Habitat-matterport 3d semantics dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  53. [61]

    Sg-nav: Online 3d scene graph prompting for llm-based zero- shot object navigation

    Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for llm-based zero- shot object navigation. In Advances in Neural Information Processing Systems, 2024. 7

    2024

  54. [62]

    Vlfm: Vision-language frontier maps for zero-shot semantic navigation

    Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision-language frontier maps for zero-shot semantic navigation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 42–48. IEEE, 2024. 1, 2, 5, 7

    2024

  55. [63]

    Gamap: Zero-shot object goal navigation with multi-scale geometric-affordance guidance

    Shuaihang Yuan, Hao Huang, Yu Hao, Congcong Wen, An- thony Tzes, and Yi Fang. Gamap: Zero-shot object goal navigation with multi-scale geometric-affordance guidance. In Advances in Neural Information Processing Systems, 2024. 7

    2024

  56. [64]

    Gnfactor: Multi-task real robot learning with generalizable neural feature fields

    Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuy- ing Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xi- aolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Conference on Robot Learning, pages 284–301. PMLR, 2023. 1

    2023

  57. [65]

    Faster segment anything: Towards lightweight sam for mobile appli- cations

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile appli- cations. arXiv preprint arXiv:2306.14289, 2023. 2

    2023 arXiv

  58. [66]

    Vla-3d: A dataset for 3d semantic scene understanding and navigation

    Haochen Zhang, Nader Zantout, Pujith Kachana, Zongyuan Wu, Ji Zhang, and Wenshan Wang. Vla-3d: A dataset for 3d semantic scene understanding and navigation. arXiv preprint arXiv:2411.03540, 2024. 2

    2024 arXiv

  59. [67]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation. In Proceedings of Robotics: Science and Systems (RSS), 2024. 6

    2024

  60. [68]

    Hierarchical object-to-zone graph for object navigation

    Sixian Zhang, Xinhang Song, Yubing Bai, Weijie Li, Yakui Chu, and Shuqiang Jiang. Hierarchical object-to-zone graph for object navigation. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 15130–15140,

  61. [69]

    Vision-and-language navigation today and tomorrow: A survey in the era of foundation models

    Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-and-language navigation today and tomorrow: A survey in the era of foundation models. arXiv preprint arXiv:2407.07035, 2024. 2

    2024 arXiv

  62. [70]

    Structured3d: A large photo-realistic dataset for structured 3d modeling

    Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Proceedings of The European Conference on Computer Vision (ECCV), 2020. 3, 6 11

    2020

  63. [71]

    Esc: Ex- ploration with soft commonsense constraints for zero-shot object navigation

    Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Esc: Ex- ploration with soft commonsense constraints for zero-shot object navigation. In International Conference on Machine Learning, pages 42829–42842. PMLR, 2023. 2, 7

    2023

  64. [72]

    Target-driven visual navigation in indoor scenes using deep reinforcement learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. IEEE, 2017. 2

    2017

  65. [73]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911– 2921, 2023. 1, 7 12 g3D-LF: Generalizable 3D-...

    2023

  66. [120]

    Springer, 2020. 1, 2, 6

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.