Pith. sign in

REVIEW 3 major objections 5 minor 86 references

CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen 3D-text model can guide a vision-language navigation agent by sending textual spatial hypotheses, improving four navigation and two spatial-reasoning benchmarks.

desk verdict Solid framework, but the causal role of the 3D-text channel is not isolated because the ablation lacks a fine-tuned image-text-only control. read the letter →

arxiv 2505.16663 v1 pith:R4DTNHJ2 submitted 2025-05-22 cs.CV cs.MM

Haihong Hao , Mingfei Han , Changlin Li , Zhihui Li , Xiaojun Chang This is my paper
classification cs.CVcs.MM
keywords embodiednavigationvision-languagecross-modalreasoning3D-textmodelpointcloudmodel-levelfusionspatialVLN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoNav argues that a navigation agent does not need one monolithic model that fuses 2D images, 3D point clouds, and text. Instead, two separately pretrained models can collaborate: an image-text agent that sees RGB frames, and a 3D-text model that sees the point cloud reconstructed from the agent's RGB-D camera. At every step the frozen 3D model writes a textual hypothesis about the spatial layout, and that text is inserted into the navigation agent's prompt; a light fine-tuning step on a small 2D-3D-text corpus teaches the agent to use the hint. The paper reports that this model-level fusion improves four embodied navigation benchmarks (R2R, CVDN, REVERIE, SOON) and two spatial-reasoning benchmarks (ScanQA, SQA3D), often by taking shorter paths rather than merely reaching the goal. The payoff would be a way to combine strong 2D and 3D models without the large triple-modality datasets that feature-level fusion requires.

What carries the argument

The load-bearing mechanism is model-level fusion via text, implemented as the Cross-Modal Belief Alignment procedure and its Communication Interface. Rather than merging image features and point-cloud features into one transformer, CoNav lets the frozen 3D-text model emit a natural-language spatial hypothesis $\pi_{3D} = \mathrm{LLM}(\mathrm{Proj}(\mathrm{Enc}(p_t)), L)$ from the current point cloud $p_t$ and instruction $L$ (Eq. 2), and a formatting operator $\mathrm{Fmt}(\cdot)$ slots that hypothesis into the navigation agent's prompt alongside the RGB tokens, history, and instruction (Eq. 3). The navigation agent's policy becomes $\pi_{nav}(r_t, h_t, \mathrm{Fmt}(\pi_{3D}(p_t, L), L))$, trained by the supervised loss in Eq. 4 on a small triple-modality corpus $\mathcal{D}_{3MT}$, with the 3D model's parameters frozen and only a subset of the agent's LLM parameters trainable. The 3D-text model itself is pretrained with a three-stage progressive curriculum (Cap3D object descriptions, 3D-FRONT/3D-GRAND indoor scenes, then real scanned point clouds from R2R and ScanQA), so that the textual hypotheses describe volumetric structure rather than pixel colors.

What would settle it

At inference on R2R Val-Unseen, replace each textual hypothesis $\pi_{3D}$ with a blank or shuffled placeholder of similar length while keeping the trained image agent and all other inputs identical. If SR/SPL stay near CoNav's reported 69/62, the 3D-text guidance is not what drives the gain; if they collapse, the text channel is the cause. A complementary test is to flip 'left' and 'right' inside the hypothesis and check whether the agent's path choices reverse accordingly.

Watch

Extended reading notes

Core claim

The paper's central claim is that a pretrained 3D-text model can explicitly guide a pretrained image-text navigation agent through a text-only channel, and that this guidance — called Cross-Modal Belief Alignment — resolves ambiguities that a pixel-only agent cannot handle, such as deciding which bedroom is nearest to a wall map. The 3D-text model is kept frozen at guidance time and produces the spatial hypothesis $\pi_{3D}$ from the current point cloud and instruction; the image-text navigation agent receives $\pi_{3D}$ through the Communication Interface and outputs the final action. With only lightweight fine-tuning of the navigation agent's LLM parameters on a small 2D-3D-text corpus, CoNav reaches state-of-the-art SPL of 62/62 on R2R, 37.54/36.16 on REVERIE, 30.0/27.2 on SOON, CVDN test SPL 0.14 with GP 7.98, and improved Exact Match on ScanQA and SQA3D compared with the best open-source baselines it evaluates.

Load-bearing premise

The load-bearing premise is that the frozen 3D-text model's natural-language output carries enough accurate spatial structure to change the agent's decisions for the better, since every piece of 3D knowledge reaches the navigation agent through that text channel.

Editorial extensions

If this is right

  • If the CoNav results hold, navigation agents can be improved by bolting a frozen 3D-text model onto an existing image-text agent, without collecting large triple-modality datasets.
  • The reported SPL gains under similar success rates imply that spatial-text guidance makes agents pick more direct routes, not just stop more accurately.
  • Because the 3D-text model is frozen and communicates through language, the 3D side can be upgraded or swapped independently of the navigation agent, which the authors argue improves flexibility and generalizability.
  • The same Communication Interface transfers to non-navigation spatial QA: prompt templates for ScanQA and SQA3D feed the 3D model's answer into the image-text model, and the paper reports Exact Match improvements there too.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • (Editorial inference) If the text channel is the bottleneck, then replacing the frozen 3D model with a stronger 3D-LLM, without retraining the agent, would be a cheap way to test how much spatial information survives in text.
  • (Editorial inference) Feeding the same agent ground-truth spatial annotations at inference would bracket the ceiling of the text-guidance channel and show how much gain comes from the quality of the 3D model's hypotheses.
  • (Editorial inference) Robustness to hallucinated or adversarial 3D descriptions is not tested; a navigation system that trusts textual spatial hints may need a confidence or verification mechanism before deployment.
  • (Editorial inference) The short-path bias suggests CoNav could be applied to efficiency-sensitive robotics settings where route length matters more than raw success, such as delivery or inspection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CoNav, a framework for embodied navigation in which a frozen pretrained 3D-text model generates natural-language spatial hypotheses from point clouds and passes them through a text-level Communication Interface to an image-text navigation agent. The navigation agent is lightly fine-tuned on a small 2D-3D-text corpus (D3MT) to learn to use these hypotheses, in a procedure the authors call Cross-Modal Belief Alignment. The authors evaluate CoNav on R2R, CVDN, REVERIE, SOON, ScanQA, and SQA3D, reporting improved SPL/GP over their reproduced NaviLLM baseline and competitive or better results on several benchmarks. The central claims are that model-level fusion of 2D and 3D modalities through text is effective and that it can be achieved without large-scale triple-modality training data.

Significance. If the central claim were established, CoNav would be a practically valuable design: it decouples 2D and 3D models, avoids the need for large-scale aligned 2D-3D-text corpora, and uses a frozen 3D-text model at inference. The paper gives a detailed curriculum pretraining scheme for the 3D-text model (Appendix A.1), and the ablation in Table 6 shows that adding alignment fine-tuning on top of model-level fusion improves both ScanQA METEOR/ROUGE and R2R SPL/SR compared with the unaligned fusion variant. These are meaningful contributions. However, the paper's headline claims of 'significant improvements' and 'state-of-the-art performance' are not fully supported by its own tables: SQA3D Exact Match is 49.0, below SceneVerse (49.9) and LEO (50.0), and R2R Val-U SPL is 62, below BEV-BERT's 64. Moreover, the experimental design does not isolate the cross-modal text channel from the effect of multi-task fine-tuning on the navigation agent, which is the load-bearing mechanism of the paper.

major comments (3)
  1. [§4.4, Table 6 and Eq. (4)-(5)] The causal role of the 3D-text channel is not isolated. Setting i in Table 6 (Text+Image only, R2R SPL 57, SR 64) is the pretrained image-text agent, identical to NaviLLM* in Table 1, rather than an agent fine-tuned on the D3MT corpus with the π3D text omitted. CoNav (setting iv, SPL 62, SR 69) differs from setting i in two coupled ways: it is fine-tuned on D3MT, which contains navigation and QA tasks, and it receives π3D at input. The reported gain could therefore come entirely from the additional multi-task fine-tuning rather than from the text-channel fusion. The authors should add an ablation that fine-tunes πnav on D3MT without π3D (or with a non-spatial placeholder text) while keeping all other settings identical; without this control, the claims that Cross-Modal Belief Alignment and text-based model-level fusion cause the benchmark improvements are not established.
  2. [Abstract, §4.3, Table 1, and Table 3] The claim of 'significant improvements' on two spatial reasoning benchmarks and of 'state-of-the-art performance' is contradicted by the paper's own quantitative results. Table 3 reports SQA3D Exact Match of 49.0 for CoNav, below SceneVerse (49.9) and LEO (50.0); Table 1 also lists SceneVerse at 49.9. On ScanQA Val-U, CoNav's Exact Match is 23.2, below LEO's 24.5. Furthermore, Table 1 shows that BEV-BERT achieves R2R Val-U SPL 64, above CoNav's 62. The abstract and Section 4.3 should be revised to state precisely which benchmarks and which metrics improve, and the 'state-of-the-art' claim should be removed or qualified. Additionally, the word 'significant' is used without any statistical significance testing or confidence intervals; this is not appropriate for the magnitude of the differences reported.
  3. [§3.2 and Appendix B.1] The 3D-text model is the sole source of spatial guidance, yet its output quality is never quantitatively validated. Equation (4) routes all 3D knowledge through π3D, and the paper's own Appendix C.1 concedes that text descriptions lose and distort 3D geometric structure. The paper reports no accuracy measure for the 3D-text model's spatial hypotheses on the navigation or QA scenes (Table 9 is qualitative only). Since the missing no-text control already confounds the attribution, the authors should at least report, for a sample of episodes, whether π3D contains the spatial cues needed for the tested instructions (e.g., relative distances, left/right, 'nearest to'). Without such evidence, it is unclear whether the text channel carries usable signal or whether the navigation agent is effectively learning to ignore it.
minor comments (5)
  1. [§3.3, Eq. (4)] There is a typo: 'L is a is the loss function' should read 'L is the loss function'.
  2. [§3.3, first paragraph] 'To allows the 3D-text model guide the navigation agent' should be corrected to 'To allow the 3D-text model to guide the navigation agent'.
  3. [Table 14] The training cost for the image-text navigation agent is given as 'Refer to NaviLLM', which is not reproducible. Since the paper emphasizes lightweight fine-tuning, the actual GPU-hours for the navigation agent pretraining and for each stage should be reported.
  4. [§4.1, Eq. (6)] The SPL formula notation is slightly confusing: the condition 'T_pred ≥ T_i' is written inline and the definition of S_i is repeated; a cleaner display would improve readability.
  5. [Table 3] The ScanQA test results for LEO are shown as dashes, but Table 21 reports ScanQA test leaderboard entries including CoNav. Please clarify whether LEO's test results are unavailable or not submitted, to avoid an asymmetric comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the CoNav pipeline is compositional and benchmark-validated; the ablation gap is an experimental-design confound, not a self-referential reduction.

full rationale

Walking the paper's derivation chain, the core mechanism is: a separately pretrained 3D-text model produces textual hypotheses (Eq. 2), the Communication Interface formats them into the agent's prompt (Eq. 3), Cross-Modal Belief Alignment fine-tunes only the navigation agent's LLM parameters on a triple-modality corpus (Eq. 4), and inference composes the frozen 3D-text model with the fine-tuned agent (Eq. 5). No equation defines the target in terms of itself, and no fitted parameter is renamed as a prediction: the benchmark numbers (SPL, GP, EM) are measured on held-out evaluation splits and reported from external leaderboards. The strongest confound is that Table 6 setting iv differs from setting i in two coupled ways at once (addition of the 3D-text hypothesis and addition of multi-task fine-tuning on new navigation and QA data), so the causal role of the textual guidance is not isolated; but this is a missing control that weakens attribution, not circularity. The self-citations (RoomTour3D [24], NavCoT [44], SOON [83]) appear only as related work or benchmark sources and are not load-bearing. Appendix C.1 explicitly concedes that text loses 3D information and that the data-generation pipeline can inherit image-text model biases; these are limitations that reduce the strength of the causal story, but they do not make the derivation equivalent to its inputs. No uniqueness theorem is imported, and no ansatz is smuggled in via citation. The paper is therefore self-contained as an empirical claim, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger tracks what the central claim pays for. The main cost is the assumption that text is a lossless-enough channel for spatial knowledge; this is explicit in Eq. 4-5 and acknowledged in Appendix C.1. The 3D-text model is trained by the authors on a curriculum whose stage volumes and point tokenization are hand-chosen, and the small fine-tuning corpus has a hand-chosen mixing ratio with no sensitivity analysis. No new physical entities are postulated; the Communication Interface is a prompt-formatting module.

free parameters (3)
  • D3MT sampling ratio = 4:1:2:1:1:1 (R2R:CVDN:REVERIE:SOON:ScanQA:SQA3D)
    Hand-chosen composition of the fine-tuning corpus; no sensitivity analysis is reported, and it directly affects navigation-vs-QA skill balance.
  • Point cloud tokenization budget = 8192 points sampled, 512 patches per cloud
    Hand-chosen input size for Point-BERT; Appendix C.1 says downsampling can introduce noise, so this choice affects spatial hypothesis quality.
  • Curriculum stage data composition = Stage 1: 770K Cap3D; Stage 2: 51K 3D-FRONT; Stage 3: 73K R2R/ScanQA point-text pairs
    Hand-designed curriculum volumes and task types; the paper's own ablation (Table 12) shows the schedule changes R2R SPL from 57 to 62, so the choice is load-bearing.
assumptions (4)
  • domain assumption Textual descriptions preserve enough spatial-semantic information for navigation decisions.
    The entire framework transmits 3D knowledge through natural-language hypotheses (Eq. 4-5). If the text bottleneck strips geometry, the method has a hard ceiling. Appendix C.1 concedes this limitation.
  • domain assumption The frozen 3D-text model generalizes to unseen scenes at inference without adaptation.
    CoNav relies on hypotheses from the 3D-text model on Val-U and test splits (Eq. 5); if the model overfits to its curriculum data, guidance quality degrades on new environments.
  • ad hoc to paper A prompt-level communication interface is sufficient for cross-modal fusion.
    Equation (3) and Figure 2 assume shared text is enough to combine modalities; the paper contrasts this with feature-level fusion rather than proving it.
  • domain assumption RGB-D frames at each step yield point clouds of sufficient quality.
    Section 3.1 says point clouds are extracted from RGB-D frames. Noisy or partial depth, which the authors acknowledge from downsampling, would degrade the 3D hypotheses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation." pith.science (2026). https://pith.science/paper/R4DTNHJ2

@misc{pith2026250516663,
  author       = {Pith},
  title        = {Pith review of: CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R4DTNHJ2}},
  note         = {Machine review of arXiv:2505.16663}
}
read the original abstract

Embodied navigation demands comprehensive scene understanding and precise spatial reasoning. While image-text models excel at interpreting pixel-level color and lighting cues, 3D-text models capture volumetric structure and spatial relationships. However, unified fusion approaches that jointly fuse 2D images, 3D point clouds, and textual instructions face challenges in limited availability of triple-modality data and difficulty resolving conflicting beliefs among modalities. In this work, we introduce CoNav, a collaborative cross-modal reasoning framework where a pretrained 3D-text model explicitly guides an image-text navigation agent by providing structured spatial-semantic knowledge to resolve ambiguities during navigation. Specifically, we introduce Cross-Modal Belief Alignment, which operationalizes this cross-modal guidance by simply sharing textual hypotheses from the 3D-text model to the navigation agent. Through lightweight fine-tuning on a small 2D-3D-text corpus, the navigation agent learns to integrate visual cues with spatial-semantic knowledge derived from the 3D-text model, enabling effective reasoning in embodied navigation. CoNav achieves significant improvements on four standard embodied navigation benchmarks (R2R, CVDN, REVERIE, SOON) and two spatial reasoning benchmarks (ScanQA, SQA3D). Moreover, under close navigation Success Rate, CoNav often generates shorter paths compared to other methods (as measured by SPL), showcasing the potential and challenges of fusing data from different modalities in embodied navigation. Project Page: https://oceanhao.github.io/CoNav/

Figures

Figures reproduced from arXiv: 2505.16663 by the authors.

Figure 1
Figure 1. (a) Image–text navigation agents rely on only visual cues may fail when dealing with tasks involving spatial distance. Although 3D–text models excel in spatial reasoning, they are less sensitive to textures (e.g., map and painting on the wall). Embodied navigation often demands both visual cues and spatial geometry for reasoning. In the scene of Figure, the agent need to use visual cues to locate a world map and the… view at source ↗
Figure 2
Figure 2. Our CoNav collaborative framework comprises an image–text navigation agent and a 3D–text model. The pretrained 3D–text model explicitly guides the image–text agent by providing structured spatial-semantic knowledge to resolve navigation ambiguities. The core of CoNav is the Cross-Modal Belief Alignment, which employs a Communication Interface to facilitate the straightforward sharing of textual hypotheses from our 3… view at source ↗
Figure 3
Figure 3. Visualization of CoNav in R2R. In Figure (a), the baseline image-text navigation agent rely on only [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: (a) Structure of our auxiliar 3D-text model. (b) Progressive Curriculum Learning training [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: More Qualitative results of CoNav for different Embodied Navigation task. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Our Point-Text pairs generation pipline in Stage 3 for pretraining 3D-text model. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Word clouds of different Stage during training of our 3D-text model. [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Single object point cloud data category distribution. [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

86 extracted references · 28 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Bevbert: Topo-metric map pre-training for language-guided navigation.arXiv preprint arXiv:2212.04385, 2022

    Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Topo-metric map pre-training for language-guided navigation.arXiv preprint arXiv:2212.04385, 2022

    arXiv 2022

  3. [3]

    On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    arXiv 2018

  4. [4]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Snderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683, 2018

    2018

  5. [5]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674–3683, 2018

    2018

  6. [6]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022

    2022

  7. [7]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  8. [8]

    Coyo-700m: Image-text pair dataset.https://github.com/kakaobrain/coyo-dataset, 2022

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset.https://github.com/kakaobrain/coyo-dataset, 2022

    2022

Show all 86 references
  1. [9]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158, 2017

    2017 arXiv

  2. [10]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InCVPR, 2021

    2021

  3. [11]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    2024

  4. [12]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InEuropean conference on computer vision, pages 202–221. Springer, 2020

    2020

  5. [13]

    Mapgpt: Map-guided prompting for unified vision-and-language navigation.arXiv preprint arXiv:2401.07314, 2024

    Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K Wong. Mapgpt: Map-guided prompting for unified vision-and-language navigation.arXiv preprint arXiv:2401.07314, 2024

    2024 arXiv

  6. [14]

    History aware multimodal transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems...

    2021

  7. [15]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16537–16547, 2022

    2022

  8. [16]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  9. [17]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3193–3203, 2021. 10

    2021

  10. [18]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

  11. [19]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023

    2023

  12. [20]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  13. [21]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2023

  14. [22]

    3d-front: 3d furnished rooms with layouts and semantics

    Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933–10942, 2021

    2021

  15. [23]

    Adaptive zone-aware hierarchical planner for vision-language navigation

    Chen Gao, Xingyu Peng, Mi Yan, He Wang, Lirong Yang, Haibing Ren, Hongsheng Li, and Si Liu. Adaptive zone-aware hierarchical planner for vision-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14911–14920, June 2023

    2023

  16. [24]

    Roomtour3d: Geometry-aware video-instruction tuning for embodied navigation.arXiv preprint arXiv:2412.08591, 2024

    Mingfei Han, Liang Ma, Kamila Zhumakhanova, Ekaterina Radionova, Jingyi Zhang, Xiaojun Chang, Xiaodan Liang, and Ivan Laptev. Roomtour3d: Geometry-aware video-instruction tuning for embodied navigation.arXiv preprint arXiv:2412.08591, 2024

    2024 arXiv

  17. [25]

    Multimodal fusion and vision-language models: A survey for robot vision.arXiv preprint arXiv:2504.02477, 2025

    Xiaofeng Han, Shunpeng Chen, Zenghuang Fu, Zhe Feng, Lue Fan, Dong An, Changwei Wang, Li Guo, Weiliang Meng, Xiaopeng Zhang, et al. Multimodal fusion and vision-language models: A survey for robot vision.arXiv preprint arXiv:2504.02477, 2025

    2025

  18. [26]

    Towards learning a generic agent for vision-and-language navigation via pre-training

    Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-language navigation via pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137–13146, 2020

    2020

  19. [27]

    A recurrent vision- and-language bert for navigation

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. A recurrent vision- and-language bert for navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1643–1653, June 2021

    2021

  20. [28]

    3d-llm: Injecting the 3d world into large language models.NeurIPS, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.NeurIPS, 2023

    2023

  21. [29]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    2023 arXiv

  22. [30]

    Clip2point: Transfer clip to point cloud classification with image-depth pre-training

    Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In ICCV, 2023

    2023

  23. [31]

    Autonomous multi-view navigation via deep reinforcement learning

    Xueqin Huang, Wei Chen, Wei Zhang, Ran Song, Jiyu Cheng, and Yibin Li. Autonomous multi-view navigation via deep reinforcement learning. In2021 IEEE International Conference on Robotics and Automation (ICRA), page 13798–13804. IEEE Press, 2021

    2021

  24. [32]

    Meta-explore: Ex- ploratory hierarchical vision-and-language navigation using scene object spectrum grounding

    Minyoung Hwang, Jaeyeon Jeong, Minsoo Kim, Yoonseon Oh, and Songhwai Oh. Meta-explore: Ex- ploratory hierarchical vision-and-language navigation using scene object spectrum grounding. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2023

  25. [33]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. InEuropean Conference on Computer Vision, pages 289–310. Springer, 2024

  26. [34]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. InEuropean Conference on Computer Vision, pages 289–310. Springer, 2024. 11

  27. [35]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. ...

    2021

  28. [36]

    Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning.arXiv preprint arXiv:2504.09772, 2025

    Can Jin, Hongwu Peng, Qixin Zhang, Yujin Tang, Dimitris N Metaxas, and Tong Che. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning.arXiv preprint arXiv:2504.09772, 2025

    2025 arXiv

  29. [37]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023

    2023

  30. [38]

    AI2-THOR: An Interactive 3D Environment for Visual AI

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017

    2017

  31. [39]

    Room-across-room: Multi- lingual vision-and-language navigation with dense spatiotemporal grounding

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multi- lingual vision-and-language navigation with dense spatiotemporal grounding. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), page...

    2020

  32. [40]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  33. [41]

    Improving vision-and-language navigation by generating future-view image semantics

    Jialu Li and Mohit Bansal. Improving vision-and-language navigation by generating future-view image semantics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10803–10812, 2023

    2023

  34. [42]

    Robust navigation with language pretraining and stochastic sampling.arXiv preprint arXiv:1909.02244, 2019

    Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling.arXiv preprint arXiv:1909.02244, 2019

    1909 arXiv

  35. [43]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal trans- formers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal trans- formers. InEuropean conference on computer vision, pages 1–18. Springer, 2022

    2022

  36. [44]

    Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning.arXiv preprint arXiv:2403.07376, 2024

    Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning.arXiv preprint arXiv:2403.07376, 2024

    2024 arXiv

  37. [45]

    Multi-modal situated reasoning in 3d scenes.Advances in Neural Information Processing Systems, 37:140903–140936, 2024

    Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Shawn Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes.Advances in Neural Information Processing Systems, 37:140903–140936, 2024

    2024

  38. [46]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  39. [47]

    Openshape: Scaling up 3d shape representation towards open-world understanding.arXiv preprint arXiv:2305.10764, 2023

    Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding.arXiv preprint arXiv:2305.10764, 2023

    2023 arXiv

  40. [48]

    V olumetric environment representation for vision-language navigation

    Rui Liu, Wenguan Wang, and Yi Yang. V olumetric environment representation for vision-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16317–16328, 2024

    2024

  41. [49]

    Bird’s-eye-view scene graph for vision-language navigation

    Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird’s-eye-view scene graph for vision-language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10968– 10980, 2023

    2023

  42. [50]

    Discuss before moving: Visual language navigation via multi-expert discussions.arXiv preprint arXiv:2309.11382, 2023

    Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. Discuss before moving: Visual language navigation via multi-expert discussions.arXiv preprint arXiv:2309.11382, 2023

    2023 arXiv

  43. [51]

    Scalable 3d captioning with pretrained models.arXiv:2306.07279, 2023

    Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models.arXiv:2306.07279, 2023

    2023 arXiv

  44. [52]

    Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    2022 arXiv

  45. [53]

    Reverie: Remote embodied visual referring expression in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2020

  46. [54]

    Hop: history-and-order aware pre-training for vision-and-language navigation

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: history-and-order aware pre-training for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15418–15427, 2022

    2022

  47. [55]

    Vln-petl: Parameter-efficient transfer learning for vision-and- language navigation

    Yanyuan Qiao, Zheng Yu, and Qi Wu. Vln-petl: Parameter-efficient transfer learning for vision-and- language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15443–15452, 2023

    2023

  48. [56]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  49. [57]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021

    2021

  50. [58]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, ...

    2019

  51. [59]

    Semantic scene completion from a single depth image

    Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1746–1754, 2017

    2017

  52. [60]

    Learning to navigate unseen environments: Back translation with environmental dropout

    Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. InProceedings of NAACL-HLT, pages 2610–2621, 2019

    2019

  53. [61]

    Vision-and-dialog navigation

    Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. InConference on Robot Learning, pages 394–406. PMLR, 2020

    2020

  54. [62]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  55. [63]

    Automl-agent: A multi-agent llm framework for full-pipeline automl.arXiv preprint arXiv:2410.02958, 2024

    Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent llm framework for full-pipeline automl.arXiv preprint arXiv:2410.02958, 2024

    2024 arXiv

  56. [64]

    Vision-and- language navigation via causal learning

    Liuyi Wang, Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision-and- language navigation via causal learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13139–13150, 2024

    2024

  57. [65]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation

    Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. InProceedings of the IEEE/CVF Conference on Computer ...

    2019

  58. [66]

    Lavie: High-quality video generation with cascaded latent diffusion models.International Journal of Computer Vision, 133(5):3059–3078, 2025

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models.International Journal of Computer Vision, 133(5):3059–3078, 2025

    2025

  59. [67]

    Bootstrapping language-guided navigation learning with self-refining data flywheel

    Zun Wang, Jialu Li, Yicong Hong, Songze Li, Kunchang Li, Shoubin Yu, Yi Wang, Yu Qiao, Yali Wang, Mohit Bansal, et al. Bootstrapping language-guided navigation learning with self-refining data flywheel. arXiv preprint arXiv:2412.08467, 2024

    2024 arXiv

  60. [68]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4818–...

    2024

  61. [69]

    Pointllm: Empower- ing large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. InEuropean Conference on Computer Vision, pages 131–147. Springer, 2024

    2024

  62. [70]

    Ulip-2: Towards scalable multimodal pre-training for 3d understanding.arXiv:2305.08275, 2023

    Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding.arXiv:2305.08275, 2023

    2023 arXiv

  63. [71]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024. 13

    2024 arXiv

  64. [72]

    3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination

    Jianing Yang, Xuweiyi Chen, Nikhil Madaan, Madhavan Iyengar, Shengyi Qian, David F Fouhey, and Joyce Chai. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination. arXiv preprint arXiv:2406.05132, 2024

    2024 arXiv

  65. [73]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling

    Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. InCVPR, 2022

    2022

  66. [74]

    Mlink: Linking black-box models from multiple domains for collaborative inference.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12085–12097, 2023

    Mu Yuan, Lan Zhang, Zimu Zheng, Yi-Nan Zhang, and Xiang-Yang Li. Mlink: Linking black-box models from multiple domains for collaborative inference.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12085–12097, 2023

    2023

  67. [75]

    Building cooperative embodied agents modularly with large language models.arXiv preprint arXiv:2307.02485, 2023

    Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models.arXiv preprint arXiv:2307.02485, 2023

    2023 arXiv

  68. [76]

    Agent journey beyond rgb: Unveiling hybrid semantic-spatial environmental representations for vision-and-language navigation.arXiv preprint arXiv:2412.06465, 2024

    Xuesong Zhang, Yunbo Xu, Jia Li, Zhenzhen Hu, and Richnag Hong. Agent journey beyond rgb: Unveiling hybrid semantic-spatial environmental representations for vision-and-language navigation.arXiv preprint arXiv:2412.06465, 2024

    2024

  69. [77]

    Meta-transformer: A unified framework for multimodal learning.arXiv preprint arXiv:2307.10802, 2023

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning.arXiv preprint arXiv:2307.10802, 2023

    2023 arXiv

  70. [78]

    Embodied-r: Collaborative framework for activating embodied spatial reasoning in foundation models via reinforcement learning.arXiv preprint arXiv:2504.12680, 2025

    Baining Zhao, Ziyou Wang, Jianjie Fang, Chen Gao, Fanhang Man, Jinqiang Cui, Xin Wang, Xinlei Chen, Yong Li, and Wenwu Zhu. Embodied-r: Collaborative framework for activating embodied spatial reasoning in foundation models via reinforcement learning.arXiv preprint arXiv:2504.1...

    2025 arXiv

  71. [79]

    Towards learning a generalist model for embodied navigation.arXiv preprint arXiv:2312.02010, 2024

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for embodied navigation.arXiv preprint arXiv:2312.02010, 2024

    2024 arXiv

  72. [80]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. InEuropean Conference on Computer Vision, pages 260–278. Springer, 2024

    2024

  73. [81]

    Same: Learn- ing generic language-guided visual navigation with state-adaptive mixture of experts.arXiv preprint arXiv:2412.05552, 2024

    Gengze Zhou, Yicong Hong, Zun Wang, Chongyang Zhao, Mohit Bansal, and Qi Wu. Same: Learn- ing generic language-guided visual navigation with state-adaptive mixture of experts.arXiv preprint arXiv:2412.05552, 2024

    2024 arXiv

  74. [82]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  75. [83]

    Soon: Scenario oriented object navigation with graph-based exploration

    Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. Soon: Scenario oriented object navigation with graph-based exploration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12689–12699, 2021

    2021

  76. [84]

    Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning

    Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. InICCV, 2023

    2023

  77. [85]

    How would you interpret this3D point cloud?

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921, 2023. 14 Appendices and Supplementary Mat...

    2023

  78. [86]

    We express our deep respect for the contributions of the developers and researchers who have made these models and datasets available

    data, all of which are authorized for research purposes. We express our deep respect for the contributions of the developers and researchers who have made these models and datasets available. Their work has significantly advanced the field of language modeling and 3D data coll...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.