Pith. sign in

REVIEW 3 major objections 8 minor 66 references

Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigation

T0 review · 3 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A navigation agent can learn viewpoint-stable object recognition from positive pairs selected by a frozen vision-language model, improving success on three VLN benchmarks without fine-tuning the VLM.

desk verdict A cheap VLM-guided contrastive trick with consistent benchmark gains, but the mechanism is unproven: no density stats for the selection matrix and no ablations isolating the VLM signal. read the letter →

arxiv 2506.15757 v1 pith:V2NUDO3Y submitted 2025-06-18 cs.CV

Ruoyu Wang , Tong Yu , Junda Wu , Yao Liu , Julian McAuley , Lina Yao This is my paper · ORCID
classification cs.CV
keywords weakly-supervisedlearningpartialcontrastivevision-languagenavigationmodelsviewpointinvarianceembodiedAIzero-shotobjectdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a partial contrastive training objective, whose positive pairs are chosen by object-overlap lists from a frozen vision-language model (VLM), makes a vision-language navigation agent recognize the same objects across changing viewpoints and thereby navigate better. This matters because existing agents either rely on pretrained visual backbones that are sensitive to viewpoint changes, use off-the-shelf LLMs or VLMs that lack navigation-specific knowledge, or fine-tune those large models at high computational cost. WPCL aims for the middle path: it keeps the VLM frozen and uses it only to build a binary sample-selection matrix that tells the contrastive loss which pairs of historical observations should be partially invariant. If the claim holds, VLM knowledge can be injected into embodied agents cheaply and robustly across different VLM choices.

What carries the argument

The central object is the sample-selection matrix $A$ together with the partial InfoNCE loss. $A$ is an $N\times N$ symmetric binary matrix over the observation history, defined by requiring at least $m$ common VLM-listed objects between two views; it converts a per-image object list into pairwise weak labels. The loss is InfoNCE applied only to $h_{\text{obj}}\in\mathbb{R}^{\lambda d}$, the object segment of the split feature $h_{\text{visual}}=[h_{\text{obj}};h_{\text{flex}}]$, so the model is forced to make the shared-object portion invariant while leaving the remaining $h_{\text{flex}}$ free. This machinery does the work of choosing which views contain mutual information worth extracting and of avoiding the collapse that would occur if all views were treated as fully invariant.

What would settle it

Measure VLM object-list accuracy on a sample of annotated panoramas from R2R; then train WPCL with the true labels swapped for corrupted lists (e.g., shuffled or random object sets) while keeping everything else fixed. If success rate does not drop materially, then the VLM object-overlap signal is not what carries the method, and the central claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that full invariance between positive observations is the wrong target for navigation. The paper models observations as causally produced by static objects seen through a moving viewpoint; objects act as confounders between observations, so two observations share mutual information only when they contain a common object. The method therefore builds a symmetric matrix $A$ with $A_{i,j}=1$ exactly when the VLM's object lists for observations $o_i$ and $o_j$ share at least $m$ objects, treats those pairs as positive, takes any observation sharing no object with both as negative, and applies the InfoNCE loss only to a segment $h_{\text{obj}}$ of the visual feature, leaving the rest $h_{\text{flex}}$ free to encode viewpoint-specific information. Trained on top of CLIP features and the DUET architecture with this partial contrastive objective, the agent reaches 78% success rate on R2R val-unseen (versus 72% for DUET) and improves on REVERIE and SOON as well, all without fine-tuning the VLM.

Load-bearing premise

The method's load-bearing premise is that the frozen VLM's object lists are accurate enough that two views sharing at least one listed object genuinely share visual content worth being invariant, and that the prompt's instruction to ignore walls and floors keeps the selection matrix from collapsing to all-positive pairs.

Editorial extensions

If this is right

  • A frozen VLM that never sees navigation data can still provide useful perceptual supervision, so VLN performance gains do not require expensive VLM fine-tuning.
  • The choice of VLM matters little: experiments with LLaVA 1.5, LLaVA 1.6, OpenFlamingo, and BLIP-2 show similar success rates, indicating the object-list signal is robust.
  • Partial invariance is sufficient: reserving part of the feature for viewpoint-specific information does not hurt, which suggests full contrastive invariance is unnecessary for navigation.
  • The objective transfers across instruction styles, improving step-by-step (R2R), high-level (REVERIE), and object-oriented (SOON) navigation benchmarks.
  • The method runs on a single 24GB GPU for training, unlike LLM-fine-tuning baselines that require multiple or larger accelerators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the object-list signal is the true driver, then making $A$ more reliable—for example by requiring agreement across multiple VLMs or thresholding object confidence—should further improve performance; this is a testable extension the paper does not run.
  • The same recipe of VLM-chosen partial invariance could apply to other embodied tasks with viewpoint shifts, such as object-goal navigation or mobile manipulation, not just instruction-following navigation.
  • A control experiment that replaces $A$ with random binary matrices would isolate whether the gains come from the VLM's semantic content or merely from the contrastive regularizer itself.
  • If the method is correct, it points to a general design principle: rather than fine-tuning a large VLM into an agent, one can use the frozen VLM offline to define which invariances the small trainable encoder should learn.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes WPCL, a weakly-supervised partial contrastive learning method for Vision-Language Navigation. A frozen VLM (LLaVA-1.5) produces object lists for each observation in the agent's history; two observations are treated as positive pairs when their object lists share at least m objects, yielding a binary sample selection matrix A. A trainable Partial Contrast Module splits the CLIP visual feature into an object segment and a flexible segment, and an InfoNCE-style loss (Eq. 6) is applied only to the object segment. This loss is combined with the DUET training objectives (Eq. 11) without fine-tuning the VLM. Experiments on R2R, REVERIE, and SOON report improvements over several baselines, including DUET, across most metrics. The paper also reports a sensitivity experiment across four VLMs and argues that the method is computationally efficient because no VLM fine-tuning is performed.

Significance. If the reported gains are attributable to the proposed mechanism, the paper offers a lightweight and generally applicable way to inject frozen-VLM object knowledge into VLN training, avoiding the cost of LLM/VLM fine-tuning while improving viewpoint robustness. The evaluation is broad (three benchmarks) and the VLM-choice sensitivity experiment is a genuine strength, as is the explicit claim that no VLM fine-tuning is needed. However, the central attribution of the gains to VLM-guided partial contrastive learning is not yet supported: the paper lacks ablations isolating the partial contrastive loss, provides no statistics or validation of the sample selection matrix A, and selects checkpoints on the val-unseen split without error bars. These omissions leave plausible alternative explanations for the observed improvements, so the contribution is promising but not fully demonstrated.

major comments (3)
  1. [Section III-A, Eq. (6), Section IV-C3] The central claim that gains come from VLM-guided object invariance requires the sample selection matrix A to be both accurate and selective. With m=1, two observations are positive whenever their LLaVA object lists share any single token, and indoor scenes in R2R/REVERIE/SOON routinely contain generic categories such as chair, sofa, and table. The paper provides no statistics on the density of A, no ablation replacing VLM selection with random, adjacent, or all-positive masks, and no quantitative VLM object-detection evaluation; Section IV-C3 only reports case-study impressions and explicitly states that object-level labels are unavailable. If A is nearly dense, the contrastive loss in Eq. (6) approximates a uniform pull over the observation history, and the reported gains could stem from generic temporal/contrastive regularization rather than from VLM-guided object overlap. This is directly testable and should be addressed with A-density statistics and mask-ablation experiments.
  2. [Section III-B, Eq. (5), Eq. (11)] There is no ablation isolating the contribution of the partial contrastive loss Lpcon. The overall objective in Eq. (11) adds Lpcon to the existing DUET/SAP/PID losses, but the paper does not report a DUET baseline trained under the same recipe with Lpcon removed, nor does it ablate the hyperparameters lambda, alpha, gamma, and m. In addition, Eq. (5) assumes that the first lambda fraction of the projected CLIP feature can be assigned to shared-object information; this is an untested ad-hoc assumption. Without an experiment showing that (i) Lpcon is necessary, (ii) the partial (object-segment-only) formulation outperforms full-feature contrastive learning, and (iii) the learned hobjs actually captures shared objects, the paper's central architectural claims are not supported.
  3. [Section IV-A4, Tables I-III] The checkpoint is selected by the highest SPL on validation unseen data, and all key comparisons in Tables I-III are on val-seen and val-unseen. Reporting val-unseen results after selecting the checkpoint on that split makes those numbers optimistic relative to a truly held-out protocol, and the absence of error bars or multiple-seed variances means the reported gains (e.g., R2R val-unseen SR 78 vs. DUET 72) cannot be assessed for statistical significance. The authors should either report results under a selection protocol that does not use val-unseen for checkpoint selection, or provide seed-level variance and confirm that the ranking is stable.
minor comments (8)
  1. [Section IV-D (heading)] The heading 'Compuatational Cost' contains a typo; it should read 'Computational Cost'.
  2. [Section IV-B] The phrase 'we also coducted case studies' should read 'we also conducted case studies'.
  3. [Section IV-A3] The sentence 'select the checkpoint with the highest SPL on validation unseen data and evaluate on other matrices' should say 'other metrics', not 'matrices'.
  4. [Section IV-A4] The phrase 'the object portion ration λ' contains a typo; it should be 'the object portion ratio λ'.
  5. [Table I] The WPCL row is difficult to read because numbers are merged: for val-seen the row reads '11.26 1.91898280' and for val-unseen '12.83 2.6986 78 70', which obscures the OSR values. Please format all entries with consistent spacing per column.
  6. [Section III-A and Eq. (4)] The symbol N is used both for the number of historical observations and for the set of negative samples in Eq. (4); this overloaded notation is confusing and should be disambiguated.
  7. [Section IV-C3] The claim that VLM object detection is 'highly accurate, achieving near-perfect quality' is not supported by quantitative evidence and is inconsistent with the later admission that object-level labels are unavailable. Please temper this claim or provide a quantitative proxy evaluation.
  8. [Figure 6] Figure 6 reports SR across VLMs but provides no numerical values or error bars; the claim that 'the choice of VLM has minimal impact' should be backed by the actual numbers or a table.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: WPCL's VLM-selected positives are an external training signal, and the navigation results are not re-derivations of that signal.

full rationale

The paper's claimed derivation is empirical rather than formal: a frozen LLaVA produces object lists per observation, matrix A is defined by thresholding list intersections (A_ij=1 iff |{Objk}(i)∩{Objk}(j)|≥m), and the partial InfoNCE loss in Eq. 6 is applied to the hobjs segment. The VLM signal is external to the navigation objective: it is not fitted from SR/SPL targets, and the reported navigation metrics are not a re-statement of Eq. 6, so no prediction reduces to its inputs by construction. The paper does not rely on self-citations; the DUET architecture is an external prior work, and the causal/contrastive motivation cites independent ML theory. The main validity caveats—"We select the checkpoint with the highest SPL on validation unseen data and evaluate on other matrices" (Sec. IV-A4) and "due to the absence of object-level labels in the datasets, we cannot directly evaluate VLM detection accuracy" (Sec. IV-C3)—weaken attribution of the gains to VLM-guided object invariance, but they are evaluation and measurement limitations, not circular derivations. Hence score 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The claim rests on standard contrastive-learning theory, the static-environment assumption of VLN simulators, the reliability of VLM object lists, and an unverified feature-split assumption. Hyperparameters lambda, alpha, gamma and m are chosen by hand without sensitivity studies.

free parameters (4)
  • lambda (object feature ratio) = 0.2
    Section IV-A: the first 20% of the projected visual feature is designated as the object segment for partial contrastive loss; no sensitivity analysis is reported.
  • alpha (contrastive loss weight) = 0.5
    Equation 11; chosen by hand, no ablation or sensitivity study.
  • gamma (SAP loss weight) = 1
    Equation 11; set to 1, no analysis.
  • m (shared-object threshold) = 1
    Section III-A and IV-A; a pair is positive if at least one object is shared; no tuning experiment.
assumptions (4)
  • standard math InfoNCE contrastive loss extracts mutual information between positive samples
    Invoked in Section II-B and Eq 6-7; relying on Zimmermann et al. [17].
  • domain assumption Agent actions change only viewpoint, not the object layout
    Section II-A: the entire motivation for treating objects as confounders. Holds in the simulated R2R/REVERIE/SOON environments.
  • domain assumption The frozen VLM's object lists are accurate enough to define shared-object positives
    Section III-A uses LLaVA v1.5 outputs to construct matrix A; Section IV-C3 admits no quantitative verification.
  • ad hoc to paper The first lambda fraction of the projected CLIP feature can be assigned to shared-object information
    Equation 5; no mechanism forces the split and no ablations vary lambda.
invented entities (1)
  • object segment h_obj
    purpose: A designated slice of the visual feature that the partial contrastive loss trains to be viewpoint-invariant
    Defined in Eq 5; it has no falsifiable handle outside the model, and the paper does not show that it contains only shared-object information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigation." pith.science (2026). https://pith.science/paper/V2NUDO3Y

@misc{pith2026250615757,
  author       = {Pith},
  title        = {Pith review of: Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V2NUDO3Y}},
  note         = {Machine review of arXiv:2506.15757}
}
read the original abstract

Visual Language Navigation (VLN) is a fundamental task within the field of Embodied AI, focusing on the ability of agents to navigate complex environments based on natural language instructions. Despite the progress made by existing methods, these methods often present some common challenges. First, they rely on pre-trained backbone models for visual perception, which struggle with the dynamic viewpoints in VLN scenarios. Second, the performance is limited when using pre-trained LLMs or VLMs without fine-tuning, due to the absence of VLN domain knowledge. Third, while fine-tuning LLMs and VLMs can improve results, their computational costs are higher than those without fine-tuning. To address these limitations, we propose Weakly-supervised Partial Contrastive Learning (WPCL), a method that enhances an agent's ability to identify objects from dynamic viewpoints in VLN scenarios by effectively integrating pre-trained VLM knowledge into the perception process, without requiring VLM fine-tuning. Our method enhances the agent's ability to interpret and respond to environmental cues while ensuring computational efficiency. Experimental results have shown that our method outperforms the baseline methods on multiple benchmarks, which validate the effectiveness, robustness and generalizability of our method.

Figures

Figures reproduced from arXiv: 2506.15757 by the authors.

Figure 1
Figure 1. An example of Visual Language Navigation task. The [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. A generalized causal framework between objects and [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. The overall workflow of our method. The observation histories are first processed by the pre-trained CLIP model. And [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Case study on R2R with the instruction: Walk through the kitchen and into the living area with two red sofas facing each other. Stop just when you reach the black rug. (a) Our method successfully completes the task. (b) DUET deviates in the wrong direction and fails to…
Figure 6
Figure 6. Figure 6: Sensitivity to VLM choice. We report the Success Rate [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 21 canonical work pages

  1. [1]

    Vision-and- language navigation: A survey of tasks, methods, and future directions,

    J. Gu, E. Stefani, Q. Wu, J. Thomason, and X. E. Wang, “Vision-and- language navigation: A survey of tasks, methods, and future directions,” arXiv preprint arXiv:2203.12667, 2022

    arXiv 2022

  2. [2]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3674–3683

    2018

  3. [3]

    Speaker- follower models for vision-and-language navigation,

    D. Fried, R. Hu, V . Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell, “Speaker- follower models for vision-and-language navigation,”Advances in neural information processing systems, vol. 31, 2018

    2018

  4. [4]

    Learning to navigate unseen environ- ments: Back translation with environmental dropout,

    H. Tan, L. Yu, and M. Bansal, “Learning to navigate unseen environ- ments: Back translation with environmental dropout,”arXiv preprint arXiv:1904.04195, 2019

    arXiv 1904

  5. [5]

    Towards learning a generic agent for vision-and-language navigation via pre-training,

    W. Hao, C. Li, X. Li, L. Carin, and J. Gao, “Towards learning a generic agent for vision-and-language navigation via pre-training,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13 137–13 146

    2020

  6. [6]

    A recurrent vision-and-language bert for navigation,

    Y . Hong, Q. Wu, Y . Qi, C. Rodriguez-Opazo, and S. Gould, “A recurrent vision-and-language bert for navigation,”arXiv preprint arXiv:2011.13922, 2020

    arXiv 2011

  7. [7]

    Scaling data generation in vision-and-language navigation,

    Z. Wang, J. Li, Y . Hong, Y . Wang, Q. Wu, M. Bansal, S. Gould, H. Tan, and Y . Qiao, “Scaling data generation in vision-and-language navigation,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12 009–12 020

    2023

  8. [8]

    Navgpt: Explicit reasoning in vision- and-language navigation with large language models,

    G. Zhou, Y . Hong, and Q. Wu, “Navgpt: Explicit reasoning in vision- and-language navigation with large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, 2024, pp. 7641–7649

    2024

Show all 66 references
  1. [9]

    Mapgpt: Map-guided prompting for unified vision-and-language navigation,

    J. Chen, B. Lin, R. Xu, Z. Chai, X. Liang, and K.-Y . K. Wong, “Mapgpt: Map-guided prompting for unified vision-and-language navigation,” arXiv preprint arXiv:2401.07314, 2024

    2024 arXiv

  2. [10]

    Discuss before moving: Visual language navigation via multi-expert discussions,

    Y . Long, X. Li, W. Cai, and H. Dong, “Discuss before moving: Visual language navigation via multi-expert discussions,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 17 380–17 387

    2024

  3. [11]

    Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning,

    B. Lin, Y . Nie, Z. Wei, J. Chen, S. Ma, J. Han, H. Xu, X. Chang, and X. Liang, “Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning,”arXiv preprint arXiv:2403.07376, 2024

    2024 arXiv

  4. [12]

    Langnav: Language as a perceptual representation for navigation,

    B. Pan, R. Panda, S. Jin, R. Feris, A. Oliva, P. Isola, and Y . Kim, “Langnav: Language as a perceptual representation for navigation,” arXiv preprint arXiv:2310.07889, 2023

    2023 arXiv

  5. [13]

    Towards learning a generalist model for embodied navigation,

    D. Zheng, S. Huang, L. Zhao, Y . Zhong, and L. Wang, “Towards learning a generalist model for embodied navigation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 624–13 634

    2024

  6. [14]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  7. [15]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  8. [16]

    Active open-vocabulary recognition: Let intelligent moving mitigate clip limitations,

    L. Fan, J. Zhou, X. Xing, and Y . Wu, “Active open-vocabulary recognition: Let intelligent moving mitigate clip limitations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16 394–16 403

    2024

  9. [17]

    Contrastive learning inverts the data generating process,

    R. S. Zimmermann, Y . Sharma, S. Schneider, M. Bethge, and W. Brendel, “Contrastive learning inverts the data generating process,” inInternational conference on machine learning. PMLR, 2021, pp. 12 979–12 990

    2021

  10. [18]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” inInternational conference on machine learning. PMLR, 2020, pp. 1597–1607

    2020

  11. [19]

    Improved baselines with momentum contrastive learning,

    X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,”arXiv preprint arXiv:2003.04297, 2020

    2003 arXiv

  12. [20]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

    2023

  13. [21]

    Simple but effective: Clip embeddings for embodied ai,

    A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, “Simple but effective: Clip embeddings for embodied ai,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 829–14 838

    2022

  14. [22]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation,

    S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 537–16 547

    2022

  15. [23]

    History aware multimodal transformer for vision-and-language navigation,

    S. Chen, P.-L. Guhur, C. Schmid, and I. Laptev, “History aware multimodal transformer for vision-and-language navigation,”Advances in neural information processing systems, vol. 34, pp. 5834–5847, 2021

    2021

  16. [24]

    Scene-intuitive agent for remote embodied visual grounding,

    X. Lin, G. Li, and Y . Yu, “Scene-intuitive agent for remote embodied visual grounding,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7036–7045

    2021

  17. [25]

    Bevbert: Topo-metric map pre-training for language-guided navigation,

    D. An, Y . Qi, Y . Li, Y . Huang, L. Wang, T. Tan, and J. Shao, “Bevbert: Topo-metric map pre-training for language-guided navigation,”arXiv preprint arXiv:2212.04385, vol. 2, no. 7, p. 8, 2022

    2022 arXiv

  18. [26]

    Vision- and-language navigation via causal learning,

    L. Wang, Z. He, R. Dang, M. Shen, C. Liu, and Q. Chen, “Vision- and-language navigation via causal learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 139–13 150

    2024

  19. [27]

    Gridmm: Grid memory map for vision-and-language navigation,

    Z. Wang, X. Li, J. Yang, Y . Liu, and S. Jiang, “Gridmm: Grid memory map for vision-and-language navigation,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 625–15 636

    2023

  20. [28]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models,

    G. Zhou, Y . Hong, Z. Wang, X. E. Wang, and Q. Wu, “Navgpt-2: Unleashing navigational reasoning capability for large vision-language models,”arXiv preprint arXiv:2407.12366, 2024

    2024 arXiv

  21. [29]

    Reverie: Remote embodied visual referring expression in real indoor environments,

    Y . Qi, Q. Wu, P. Anderson, X. Wang, W. Y . Wang, C. Shen, and A. v. d. Hengel, “Reverie: Remote embodied visual referring expression in real indoor environments,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9982–9991

    2020

  22. [30]

    Soon: Scenario oriented object navigation with graph-based exploration,

    F. Zhu, X. Liang, Y . Zhu, Q. Yu, X. Chang, and X. Liang, “Soon: Scenario oriented object navigation with graph-based exploration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12 689–12 699

    2021

  23. [31]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,

    X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y .-F. Wang, W. Y . Wang, and L. Zhang, “Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recogni...

    2019

  24. [32]

    Minivln: Efficient vision-and-language navigation by progressive knowledge distillation,

    J. Zhu, Y . Qiao, S. Zhang, X. He, Q. Wu, and J. Liu, “Minivln: Efficient vision-and-language navigation by progressive knowledge distillation,” arXiv preprint arXiv:2409.18800, 2024

    2024 arXiv

  25. [33]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar,et al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  26. [34]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  27. [35]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma,et al., “Scaling instruction-finetuned language models,”Journal of Machine Learning Research, vol. 25, no. 70, pp. 1–53, 2024

    2024

  28. [36]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat,et al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  29. [37]

    On evalu- ation of embodied navigation agents,

    P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V . Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva,et al., “On evalu- ation of embodied navigation agents,”arXiv preprint arXiv:1807.06757, 2018

    2018 arXiv

  30. [38]

    Lxmert: Learning cross-modality encoder representations from transformers,

    H. Tan and M. Bansal, “Lxmert: Learning cross-modality encoder representations from transformers,”arXiv preprint arXiv:1908.07490, 2019

    1908 arXiv

  31. [39]

    Airbert: In-domain pretraining for vision-and-language navigation,

    P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid, “Airbert: In-domain pretraining for vision-and-language navigation,” inProceed- ings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1634–1643

    2021

  32. [40]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  33. [41]

    Openflamingo: An open-source framework for training large autoregressive vision- language models,

    A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y . Hanafy, W. Zhu, K. Marathe, Y . Bitton, S. Gadre, S. Sagawa,et al., “Openflamingo: An open-source framework for training large autoregressive vision- language models,”arXiv preprint arXiv:2308.01390, 2023

    2023 arXiv

  34. [42]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  35. [43]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  36. [44]

    Vinl: Visual navigation and locomotion over obstacles,

    S. Kareer, N. Yokoyama, D. Batra, S. Ha, and J. Truong, “Vinl: Visual navigation and locomotion over obstacles,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 2018–2024

    2023

  37. [45]

    Exploitation- guided exploration for semantic embodied navigation,

    J. Wasserman, G. Chowdhary, A. Gupta, and U. Jain, “Exploitation- guided exploration for semantic embodied navigation,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 2901–2908

    2024

  38. [46]

    The regretful agent: Heuristic-aided navigation through progress estimation,

    C.-Y . Ma, Z. Wu, G. AlRegib, C. Xiong, and Z. Kira, “The regretful agent: Heuristic-aided navigation through progress estimation,” in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2019, pp. 6732–6740

    2019

  39. [47]

    Tactical rewind: Self-correction via backtracking in vision-and-language navigation,

    L. Ke, X. Li, Y . Bisk, A. Holtzman, Z. Gan, J. Liu, J. Gao, Y . Choi, and S. Srinivasa, “Tactical rewind: Self-correction via backtracking in vision-and-language navigation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6741–6749

    2019

  40. [48]

    Bird’s-eye-view scene graph for vision-language navigation,

    R. Liu, X. Wang, W. Wang, and Y . Yang, “Bird’s-eye-view scene graph for vision-language navigation,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 10 968–10 980

    2023

  41. [49]

    Visual language maps for robot navigation,

    C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 10 608–10 615

    2023

  42. [50]

    Robot navigation in unseen envi- ronments using coarse maps,

    C. Xu, C. Amato, and L. L. Wong, “Robot navigation in unseen envi- ronments using coarse maps,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 2932–2938

    2024

  43. [51]

    Placenav: Topological navigation through place recognition,

    L. Suomela, J. Kalliola, H. Edelman, and J.-K. K ¨am¨ar¨ainen, “Placenav: Topological navigation through place recognition,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 5205–5213

    2024

  44. [52]

    Kerm: Knowledge enhanced reasoning for vision-and-language navigation,

    X. Li, Z. Wang, J. Yang, Y . Wang, and S. Jiang, “Kerm: Knowledge enhanced reasoning for vision-and-language navigation,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2583–2592

    2023

  45. [53]

    Zero-shot object goal visual navigation,

    Q. Zhao, L. Zhang, B. He, H. Qiao, and Z. Liu, “Zero-shot object goal visual navigation,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 2025–2031

    2023

  46. [54]

    Wavn: Wide area visual navigation for large-scale, gps-denied environments,

    D. M. Lyons and M. Rahouti, “Wavn: Wide area visual navigation for large-scale, gps-denied environments,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 2039–2045

    2023

  47. [55]

    Loc-zson: Language-driven object-centric zero-shot object retrieval and navigation,

    T. Guan, Y . Yang, H. Cheng, M. Lin, R. Kim, R. Madhivanan, A. Sen, and D. Manocha, “Loc-zson: Language-driven object-centric zero-shot object retrieval and navigation,”arXiv preprint arXiv:2405.05363, 2024

    2024 arXiv

  48. [56]

    Aligning knowledge graph with visual perception for object-goal navigation,

    N. Xu, W. Wang, R. Yang, M. Qin, Z. Lin, W. Song, C. Zhang, J. Gu, and C. Li, “Aligning knowledge graph with visual perception for object-goal navigation,”arXiv preprint arXiv:2402.18892, 2024

    2024 arXiv

  49. [57]

    Improving vision-and-language navigation with image-text pairs from the web,

    A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, “Improving vision-and-language navigation with image-text pairs from the web,” inComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI

    2020

  50. [58]

    Springer, 2020, pp. 259–274

    2020

  51. [59]

    Panogen: Text-conditioned panoramic envi- ronment generation for vision-and-language navigation,

    J. Li and M. Bansal, “Panogen: Text-conditioned panoramic envi- ronment generation for vision-and-language navigation,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  52. [60]

    Counterfactual vision-and-language navigation: Unravelling the unseen,

    A. Parvaneh, E. Abbasnejad, D. Teney, J. Q. Shi, and A. Van den Hengel, “Counterfactual vision-and-language navigation: Unravelling the unseen,”Advances in neural information processing systems, vol. 33, pp. 5296–5307, 2020

    2020

  53. [61]

    Counterfac- tual cycle-consistent learning for instruction following and generation in vision-language navigation,

    H. Wang, W. Liang, J. Shen, L. Van Gool, and W. Wang, “Counterfac- tual cycle-consistent learning for instruction following and generation in vision-language navigation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 15 471–15 481

    2022

  54. [62]

    Transferable representation learning in vision-and-language navigation,

    H. Huang, V . Jain, H. Mehta, A. Ku, G. Magalhaes, J. Baldridge, and E. Ie, “Transferable representation learning in vision-and-language navigation,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7404–7413

    2019

  55. [63]

    Self-monitoring navigation agent via auxiliary progress estimation,

    C.-Y . Ma, J. Lu, Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong, “Self-monitoring navigation agent via auxiliary progress estimation,” arXiv preprint arXiv:1901.03035, 2019

    1901 arXiv

  56. [64]

    Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation,

    X. Wang, W. Xiong, H. Wang, and W. Y . Wang, “Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation,” inProceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 37–53

    2018

  57. [65]

    Vision-language navigation with self-supervised auxiliary reasoning tasks,

    F. Zhu, Y . Zhu, X. Chang, and X. Liang, “Vision-language navigation with self-supervised auxiliary reasoning tasks,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 012–10 022

    2020

  58. [66]

    Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill,

    W. Cai, S. Huang, G. Cheng, Y . Long, P. Gao, C. Sun, and H. Dong, “Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 5228–5234

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.