Pith. sign in

REVIEW 4 major objections 6 minor 65 references

Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that fine-grained navigation-instruction alignments can be generated automatically at scale, and that training on the resulting FCA-R2R dataset improves vision-language navigation agents.

desk verdict Useful large-scale dual-alignment VLN dataset, but the 'precise' claim is unvalidated and the missing augmentation baselines weaken the core comparison. read the letter →

arxiv 2506.08566 v1 pith:FZPW3INE submitted 2025-06-10 cs.CV

Yibo Cui , Liang Xie , Yu Zhao , Jiawei Sun , Erwei Yin This is my paper
classification cs.CV
keywords Vision-LanguageNavigationDataAugmentationFine-GrainedCross-ModalAlignmentInstructionGenerationEntity-LandmarkSub-InstructionOFA-SpeakerFCA-R2R
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language navigation agents need to know which part of an instruction applies at which moment and which object in the view is being referenced, but existing datasets mostly provide only whole-instruction-whole-trajectory matches. This paper tries to establish that both finer levels of alignment can be produced automatically at scale, without human annotation, and that agents trained on such data navigate more accurately. The FCA-NIG pipeline chunks trajectories, detects landmarks, writes template-based draft instructions, rewrites them into R2R-style language with an OFA-based speaker, selects matching entities with CLIP, and stitches sub-pairs into full instruction-trajectory pairs. The resulting FCA-R2R dataset contains millions of sub-instruction-sub-trajectory and entity-landmark pairs, and the paper reports consistent success-rate and SPL gains across four VLN agents and in the Babywalk, AFAC, and GELA evaluation settings.

What carries the argument

The load-bearing object is the OFA-Speaker, an OFA-based sequence-to-sequence generator with an added Transformer panorama encoder that stacks spatial and temporal layers over CLIP-ViT panorama embeddings. It takes a prompt, a crafted template instruction, and panorama features, and outputs R2R-style navigation instructions; SimCTG contrastive training and decoding suppress repetition. Around it, the pipeline's other components fix the supervision: trajectory chunking splits sampled paths into straight and turn segments, GLIP detects the salient landmark in each segment, a library of 108 template instructions encodes heading changes, vertical movement, and landmark-relative position, and CLIP ranks candidate entity phrases against the detected landmark to form entity-landmark matches. Sub-pair integration stitches the pieces into complete pairs while tracking entity positions. Each component does a specific alignment job: chunking defines sub-instruction-sub-trajectory boundaries, GLIP supplies the visual referent, templates provide spatial grounding, OFA-Speaker converts templates to natural language, and CLIP resolves which word names which object.

What would settle it

Take a random sample of entity-landmark pairs from FCA-R2R and have annotators judge whether the entity phrase actually refers to the GLIP-detected landmark in the corresponding panorama view. If agreement is close to chance, the entity-level supervision is essentially noise, and the reported gains would have to be attributed to data volume or sub-instruction alignment rather than to the entity-landmark annotations.

Watch

Extended reading notes

Core claim

FCA-R2R is claimed to be the first large-scale augmentation dataset with precise sub-instruction-sub-trajectory and entity-landmark alignments. It is built by sampling 5-7-step Matterport3D trajectories, splitting them by movement direction, detecting a representative landmark per sub-trajectory with GLIP, constructing a template instruction from a 108-template library, rewriting that instruction into R2R-like language with OFA-Speaker, and choosing the entity in the text whose CLIP similarity to the detected landmark is highest. On top of this, sub-instructions are aligned to their sub-trajectories and entity phrases to landmarks; the dataset totals 178,270 trajectories, 534,810 instructions, 2,519,919 sub-pairs, and 5,240,981 entity-landmark pairs. The paper reports that replacing existing augmented training sets with FCA-R2R improves SR and SPL for SF, EnvDrop, RecBERT, and HAMT, and that using the sub-pair and entity-landmark annotations through Babywalk, AFAC, and GELA yields further gains, with best unseen validation/test SR of 71.1%/68.3% and SPL of 65.5%/63.3%.

Load-bearing premise

The automatic entity-landmark matches, made by picking the entity with the highest CLIP similarity to each GLIP-detected landmark, are accurate enough to teach the agent; the paper reports no human check or measured accuracy for these matches.

Editorial extensions

If this is right

  • Training with FCA-R2R improves success rate (SR) and success weighted by path length (SPL) across SF, EnvDrop, RecBERT, and HAMT, with the largest gains on agents that were not pretrained.
  • Adding sub-instruction-sub-trajectory supervision through Babywalk and AFAC improves state awareness and action decisions, including gains on unseen environments.
  • Adding entity-landmark supervision through GELA improves navigation performance and generalization, with SR and SPL gains on the unseen test split.
  • Because FCA-NIG is automatic, the same sampling loop can produce millions more fine-grained pairs without additional human annotation.
  • FCA-R2R is, per the paper, the first large-scale augmentation dataset to contain both sub-instruction and entity-level alignments, enabling direct fine-grained supervision that prior datasets provided only at one level or not at all.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if the automatic CLIP-based entity matching is noisy, the entity-landmark gains may partly reflect larger training volume rather than alignment precision; a controlled comparison against randomly paired entity-landmark labels of the same size would separate these effects.
  • The chunking rule that groups straight steps and isolates turns is a simplifying assumption about instruction granularity; real instructions sometimes refer to landmarks mid-straight, so a more semantic chunker could strengthen the sub-pair supervision.
  • FCA-NIG's reliance on GLIP and CLIP suggests alignment quality is bounded by the zero-shot object vocabulary and by CLIP's ability to distinguish nearby indoor objects; evaluating on out-of-domain or dynamic environments would test whether the annotations transfer.
  • The same pipeline could plausibly be applied to other instruction styles such as REVERIE or conversational navigation, since the template and speaker stages are not R2R-specific, though the paper does not demonstrate this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces FCA-NIG, an automatic pipeline that generates navigation instruction-trajectory pairs annotated with two levels of fine-grained cross-modal alignment: sub-instruction-sub-trajectory matching and entity-landmark matching. The pipeline samples trajectories, chunks them by turning angles, detects landmarks with GLIP, builds template-based 'crafted' instructions, rewrites them into R2R-style instructions with an OFA-based speaker, and selects entity-landmark matches via CLIP. The resulting FCA-R2R dataset contains 178,270 trajectories, 534,810 instructions, and over five million entity-landmark pairs. The authors evaluate the dataset by training four VLN agents (SF, EnvDrop, RecBERT, HAMT) with FCA-R2R replacing their usual augmentation data, and by plugging FCA-R2R's sub-pair and entity-landmark annotations into Babywalk, AFAC, and GELA, reporting improvements on standard navigation metrics.

Significance. If the generated alignments are accurate and if the reported gains are attributable to them, FCA-R2R would be a scalable and useful resource for the VLN community, addressing a real data-scarcity problem. The paper is commendable for evaluating across multiple agent architectures and for attempting to isolate the effects of sub-pair and entity-landmark supervision. The principal weakness is that the central causal claims -- that precise alignments, not data volume or dataset scale, drive the gains -- are not yet supported by the experimental design. The framework itself is modular and reproducible in principle, but the missing validation of the entity-landmark matching and the lack of head-to-head comparisons with existing augmentation datasets materially limit what can be concluded.

major comments (4)
  1. [5.3, Table 3] The main effectiveness experiment compares each agent trained with FCA-R2R against the same agent without any augmentation, yet §5.3 states that 'the original augmented datasets were replaced with the FCA-R2R dataset.' Because FCA-R2R is an order of magnitude larger than the original augmentation sets (Table 2) and is also distributionally different, the observed gains could stem from scale or instruction diversity rather than from the fine-grained alignment annotations. The paper should include (a) agents trained on the original augmentation datasets (SF-R2R, EnvDrop-R2R, Prevalent-R2R) under the same protocol, and (b) an ablation that augments with a volume-matched subset of FCA-R2R or adds FCA-R2R on top of the original dataset, to isolate the contribution of the alignment annotations.
  2. [4.5] The entity-landmark alignment is produced by taking, for each GLIP-detected landmark, the NLTK-extracted entity with the highest CLIP cosine similarity, and no verification or accuracy measurement is reported. The abstract and contributions describe the resulting alignments as 'precise,' and §5.5 attributes navigation gains to this supervision. Since FCA-R2R has roughly 35× more entity-landmark pairs than GEL-R2R (Table 2), the gains could be a volume effect or could even arise despite noisy matches. Please add a validation of the CLIP-based matching (for example, precision against the human-annotated GEL-R2R pairs on a shared subset, or a human evaluation of a random sample) and an ablation replacing the matched entity with a random or frequency-based entity at matched volume.
  3. [5.3-5.5, Tables 3-5] No variance or significance information is provided, and several reported differences are small, including GELA val-unseen SR (71.1 vs 71.1), HAMT val-unseen SPL (60.9 vs 61.5, i.e., +0.6), and AFAC val-seen SR (+0.7). Meanwhile, SF's val-seen SR decreases from 66.0 to 64.3 after adding FCA-R2R. The repeated wording 'significantly improves' (abstract, §5.3) is therefore not supported. Reporting multiple seeds with means and standard deviations, or at least tempering the language, is necessary before the effectiveness claim can be assessed.
  4. [5.5, Table 5] The only evidence for the benefit of entity-landmark alignment comes from GELA, a method developed by the same group that produced the GEL-R2R dataset on which the OFA-Speaker was fine-tuned. To rule out an in-group or data-distribution artifact, the comparison should be extended to an independent method (e.g., a straightforward entity-augmented training objective) and should include a control in which the same volume of GEL-R2R pairs or randomly paired entities is used.
minor comments (6)
  1. [Table 4] In the Babywalk + FCA-R2R row, the val-seen SR is printed as 0.55, which is inconsistent with the SPL of 52.6 and is presumably 55.0; please correct and double-check all numbers in the tables.
  2. [5.3] The phrase 'significant improvement in SR by 10%' should read '10 percentage points,' and the accompanying regression of SF on the seen split (66.0 to 64.3) should be mentioned.
  3. [4.4, Eq. (4)] The hyperparameter is written as '𝑎𝑙𝑝ℎ𝑎'; use the symbol 𝛼 and report its value and the top-K size in Section 5.1.3.
  4. [4.1, Table 1] The angular intervals in Table 1 partition the heading-change space but leave boundary values (e.g., exactly 45◦) undefined; a short clarification would avoid ambiguity.
  5. [5.4, 5.5] The headings of Sections 5.4 and 5.5 contain 'FAC-R2R' instead of 'FCA-R2R'.
  6. [5.2] The description of how the 178,270 trajectories are sampled from the Matterport3D environments (which R2R splits are used, how 'all 5-7 step trajectories' is defined, and whether any trajectories overlap with the test environments) should be expanded, as this is important for assessing possible data leakage.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the core effectiveness claims are tested on the external R2R benchmark, and the same-group GEL-R2R/GELA dependencies are not load-bearing in the derivation.

full rationale

The paper's central claims are that FCA-R2R provides scalable fine-grained alignment annotations and that training on it improves four VLN agents. These claims are evaluated against the external R2R val-unseen and test splits using standard SR/SPL metrics (Tables 3–5), so no predicted quantity reduces to an input by construction. The FCA-NIG pipeline generates sub-instructions from sub-trajectories and entity-landmark pairs via GLIP, templates, OFA-Speaker, and CLIP; the resulting pairs are not fitted to the navigation metrics, and the downstream agents (SF, EnvDrop, RecBERT, HAMT) are published methods with their own pretrained checkpoints. The main non-circular dependency is the authors' own prior work: Section 4.2 builds the landmark category library “drawing on the GEL-R2R dataset [12]”, Section 4.4 trains OFA-Speaker using GEL-R2R entity-landmark pairs, and Section 5.5 evaluates entity-landmark supervision with the same group's GELA method. These are dependencies and self-citations, but they do not force the reported SR/SPL gains, and the benchmark itself is external. Section 4.5 selects entity-landmark matches as the highest CLIP cosine similarity and reports no human verification or precision measurement; this is a real validation gap for the “precise alignments” claim, but it is a correctness/evidence risk rather than a circular reduction. Overall, no circular step satisfying the quoted-reduction standard was found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen thresholds in the template library and on the unverified accuracy of GLIP and CLIP based alignments. No new physical or conceptual entities are introduced.

free parameters (4)
  • horizontal turn angular bins = thresholds at 15, 45, 75, 105, 135, 165 degrees and negative counterparts
    Table 1 maps heading changes to 12 turn categories; these hand-chosen thresholds shape the crafted instructions and hence the generated dataset.
  • vertical movement threshold = 0.2 m
    Section 4.3 classifies vertical motion as horizontal, upward, or downward using +/-0.2m; this choice affects instruction content.
  • landmark bounding box heading thresholds = chi_min, chi_max from GLIP boxes
    Section 4.3 uses the agent heading relative to landmark box boundaries to decide 'towards', 'left of', 'right of'; thresholds are detector outputs rather than fitted, but the classification rule is hand-designed.
  • SimCTG hyperparameters = alpha in Eq. (4), rho in Eq. (2)
    Chosen for decoding diversity; influences instruction text, not core alignments.
assumptions (4)
  • domain assumption GLIP produces reliable landmark detections in MP3D panoramas
    Section 4.2 selects the highest-confidence GLIP detection as the representative landmark for each sub-trajectory; no verification of detection accuracy is reported.
  • domain assumption CLIP cosine similarity ranks correct entity-landmark matches
    Section 4.5 selects the highest CLIP similarity entity as the match for each landmark; the precision of these matches is never evaluated, yet they are used as training supervision.
  • standard math R2R benchmark agents and training procedures are correctly implemented
    Sections 5.3-5.5 rely on published models (SF, EnvDrop, RecBERT, HAMT, Babywalk, AFAC, GELA) as given; no code is shipped to verify.
  • domain assumption Template instruction library of 108 patterns covers navigation language
    Section 4.3 constructs crafted instructions only from the template library; if the templates miss common patterns, generated instructions are unnatural or ambiguous.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations." pith.science (2026). https://pith.science/paper/FZPW3INE

@misc{pith2026250608566,
  author       = {Pith},
  title        = {Pith review of: Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZPW3INE}},
  note         = {Machine review of arXiv:2506.08566}
}
read the original abstract

Vision-Language Navigation (VLN) enables intelligent agents to navigate environments by integrating visual perception and natural language instructions, yet faces significant challenges due to the scarcity of fine-grained cross-modal alignment annotations. Existing datasets primarily focus on global instruction-trajectory matching, neglecting sub-instruction-level and entity-level alignments critical for accurate navigation action decision-making. To address this limitation, we propose FCA-NIG, a generative framework that automatically constructs navigation instructions with dual-level fine-grained cross-modal annotations. In this framework, an augmented trajectory is first divided into sub-trajectories, which are then processed through GLIP-based landmark detection, crafted instruction construction, OFA-Speaker based R2R-like instruction generation, and CLIP-powered entity selection, generating sub-instruction-trajectory pairs with entity-landmark annotations. Finally, these sub-pairs are aggregated to form a complete instruction-trajectory pair. The framework generates the FCA-R2R dataset, the first large-scale augmentation dataset featuring precise sub-instruction-sub-trajectory and entity-landmark alignments. Extensive experiments demonstrate that training with FCA-R2R significantly improves the performance of multiple state-of-the-art VLN agents, including SF, EnvDrop, RecBERT, and HAMT. Incorporating sub-instruction-trajectory alignment enhances agents' state awareness and decision accuracy, while entity-landmark alignment further boosts navigation performance and generalization. These results highlight the effectiveness of FCA-NIG in generating high-quality, scalable training data without manual annotation, advancing fine-grained cross-modal learning in complex navigation tasks.

Figures

Figures reproduced from arXiv: 2506.08566 by the authors.

Figure 1
Figure 1. An illustration of our speaker-follower framework. The follower (blue) navigates through a 3D photorealistic environment, while the speaker (green) generates grounded descriptions of navigation trajectories along with fine-grained alignment annotations. Several prior works have explored fine-grained cross￾modal alignment in VLN. Self-Monitoring [40] located completed or ongoing sub-instructions through visual-textua… view at source ↗
Figure 2
Figure 2. Overview of the FCA-NIG framework. Initially, a randomly sampled augmented trajectory is segmented into multiple sub￾trajectories via trajectory chunking (A). Each sub-trajectory undergoes sequential processing through intermediate steps: landmark detection (B), crafted instruction construction (C), R2R-like instruction generation (D), and entity selection (E), producing corresponding sub-instructions with entity-la… view at source ↗
Figure 3
Figure 3. Architecture of the OFA-Speaker model. The OFA-Speaker introduces a panorama encoder into the standard OFA model, which includes a text encoder, a vision encoder, a cross-modal encoder, and a text decoder. Specifically, the panorama encoder features spatial and temporal layers to process context-rich panoramas. The OFA-Speaker integrates prompts, crafted instructions, and a sequence of panoramas as inputs, generatin… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 39 canonical work pages

  1. [1]

    Spatio- temporal dynamics and semantic attribute enriched visual encoding forvideocaptioning,in:CVPR,pp.12487–12496

    Aafaq, N., Akhtar, N., Liu, W., Gilani, S.Z., Mian, A., 2019. Spatio- temporal dynamics and semantic attribute enriched visual encoding forvideocaptioning,in:CVPR,pp.12487–12496. doi:10.1109/CVPR. 2019.01277

    arXiv 2019

  2. [2]

    Bottom-up and top-down attention for image captioningandvisualquestionanswering,in:CVPR,pp.6077–6086

    Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L., 2018a. Bottom-up and top-down attention for image captioningandvisualquestionanswering,in:CVPR,pp.6077–6086. doi:10.1109/CVPR.2018.00636

    arXiv 2018

  3. [3]

    Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments, in: CVPR, pp

    Anderson,P.,Wu,Q.,Teney,D.,Bruce,J.,Johnson,M.,Sünderhauf, N., Reid, I.D., Gould, S., van den Hengel, A., 2018b. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments, in: CVPR, pp. 3674–3683. doi:10. 1109/CVPR.2018.00387

    arXiv

  4. [4]

    NLTK: the natural language toolkit, in: Calzolari, N., Cardie, C., Isabelle, P

    Bird, S., 2006. NLTK: the natural language toolkit, in: Calzolari, N., Cardie, C., Isabelle, P. (Eds.), ACL. doi:10.3115/1225403.1225421

    work page arXiv 2006

  5. [5]

    Matterport3d: LearningfromRGB-Ddatainindoorenvironments,in:3DV,pp.667–

    Chang, A.X., Dai, A., Funkhouser, T.A., Halber, M., Nießner, M., Savva, M., Song, S., Zeng, A., Zhang, Y., 2017. Matterport3d: LearningfromRGB-Ddatainindoorenvironments,in:3DV,pp.667–

    work page 2017

  6. [6]

    TOUCH- DOWN: natural language navigation and spatial reasoning in visual street environments, in: CVPR, pp

    Chen, H., Suhr, A., Misra, D., Snavely, N., Artzi, Y., 2019. TOUCH- DOWN: natural language navigation and spatial reasoning in visual street environments, in: CVPR, pp. 12538–12547. doi:10.1109/CVPR. 2019.01282

    arXiv 2019

  7. [7]

    Historyawaremul- timodaltransformerforvision-and-languagenavigation,in:NeurIPS, pp

    Chen,S.,Guhur,P.,Schmid,C.,Laptev,I.,2021. Historyawaremul- timodaltransformerforvision-and-languagenavigation,in:NeurIPS, pp. 5834–5847. URL:https://proceedings.neurips.cc/paper/2021/ hash/2e5c2cb8d13e8fba78d95211440ba326-Abstract.html

    work page 2021

  8. [8]

    Learning from unlabeled 3d environments for vision-and-language navigation,in:ECCV,pp.638–655

    Chen, S., Guhur, P., Tapaswi, M., Schmid, C., Laptev, I., 2022a. Learning from unlabeled 3d environments for vision-and-language navigation,in:ECCV,pp.638–655. doi:10.1007/978-3-031-19842-7\ _37

    doi:10.1007/978-3-031-19842-7

Show all 65 references
  1. [10]

    Unifying vision-and- language tasks via text generation, in: ICML, pp

    Cho, J., Lei, J., Tan, H., Bansal, M., 2021. Unifying vision-and- language tasks via text generation, in: ICML, pp. 1931–1942. URL: http://proceedings.mlr.press/v139/cho21a.html

    2021

  2. [11]

    Auxiliary fine-grained alignment constraints for vision-and-language navigation, in: ICME, pp

    Cui,Y.,Huang,R.,Zhang,Y.,Cen,Y.,Xie,L.,Yan,Y.,Yin,E.,2023a. Auxiliary fine-grained alignment constraints for vision-and-language navigation, in: ICME, pp. 2621–2626. doi:10.1109/ICME55011.2023. 00446

    2023

  3. [13]

    Meteor universal: Language specifictranslationevaluationforanytargetlanguage,in:Proceedings of the Ninth Workshop on Statistical Machine Translation, pp

    Denkowski, M.J., Lavie, A., 2014. Meteor universal: Language specifictranslationevaluationforanytargetlanguage,in:Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 376–

    2014

  4. [14]

    Visionformobilerobotnavigation: A survey

    DeSouza,G.N.,Kak,A.C.,2002. Visionformobilerobotnavigation: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 24, 237–267. doi:10.1109/34.982903

    2002 doi

  5. [15]

    Aerial vision-and-dialog navigation, in: Findings of the Association forComputationalLinguistics,pp.3043–3061.doi:10.18653/V1/2023

    Fan, Y., Chen, W., Jiang, T., Zhou, C., Zhang, Y., Wang, X., 2023. Aerial vision-and-dialog navigation, in: Findings of the Association forComputationalLinguistics,pp.3043–3061.doi:10.18653/V1/2023. FINDINGS-ACL.190

    2023 doi

  6. [16]

    Francis, J., Kitamura, N., Labelle, F., Lu, X., Navarro, I., Oh, J., 2022.Corechallengesinembodiedvision-languageplanning.J.Artif. Intell. Res. 74, 459–515. doi:10.1613/jair.1.13646

    2022 doi

  7. [17]

    Speaker-follower models for vision-and-language navigation, in: NeurIPS, pp

    Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L., Berg-Kirkpatrick, T., Saenko, K., Klein, D., Darrell, T., 2018. Speaker-follower models for vision-and-language navigation, in: NeurIPS, pp. 3318–3329. URL:https://proceedings.neurips.cc/ paper/2018/hash/6a8...

    2018

  8. [19]

    Vision- and-language navigation: A survey of tasks, methods, and future directions, in: ACL, pp

    Gu, J., Stefani, E., Wu, Q., Thomason, J., Wang, X., 2022. Vision- and-language navigation: A survey of tasks, methods, and future directions, in: ACL, pp. 7606–7623. doi:10.18653/v1/2022.acl-long. 524

    2022 doi

  9. [22]

    Landmark-rxr: Solving vision-and-language naviga- tion with fine-grained alignment supervision, in: NeurIPS, pp

    He, K., Huang, Y., Wu, Q., Yang, J., An, D., Sima, S., Wang, L., 2021. Landmark-rxr: Solving vision-and-language naviga- tion with fine-grained alignment supervision, in: NeurIPS, pp. 652–663. URL:https://proceedings.neurips.cc/paper/2021/hash/ 0602940f23884f782058efac46f64b0f...

    2021

  10. [23]

    Lan- guage and visual entity relationship graph for agent navigation, in: NeurIPS

    Hong, Y., Opazo, C.R., Qi, Y., Wu, Q., Gould, S., 2020a. Lan- guage and visual entity relationship graph for agent navigation, in: NeurIPS. URL:https://proceedings.neurips.cc/paper/2020/hash/ 56dc0997d871e9177069bb472574eb29-Abstract.html

    2020

  11. [24]

    Sub-instruction aware vision-and-language navigation, in: EMNLP, pp

    Hong, Y., Opazo, C.R., Wu, Q., Gould, S., 2020b. Sub-instruction aware vision-and-language navigation, in: EMNLP, pp. 3360–3376. doi:10.18653/v1/2020.emnlp-main.271

    2020 doi

  12. [25]

    VLNBERT: Arecurrentvision-and-languageBERTfornavigation,in:CVPR,pp

    Hong,Y.,Wu,Q.,Qi,Y.,Opazo,C.R.,Gould,S.,2021. VLNBERT: Arecurrentvision-and-languageBERTfornavigation,in:CVPR,pp. 1643–1653. doi:10.1109/CVPR46437.2021.00169

    2021

  13. [26]

    Transferable representation learning in vision-and- language navigation, in: ICCV, pp

    Huang, H., Jain, V., Mehta, H., Ku, A., Magalhães, G., Baldridge, J., Ie, E., 2019. Transferable representation learning in vision-and- language navigation, in: ICCV, pp. 7403–7412. doi:10.1109/ICCV. 2019.00750

    2019

  14. [28]

    Kim, H., Zala, A., Burri, G., Tan, H., Bansal, M., 2020. Arramon: A joint navigation-assembly instruction interpretation task in dynamic environments,in:ProceedingsoftheAnnualMeetingoftheAssocia- tionforComputationalLinguistics,pp.3910–3927. doi:10.18653/V1/ 2020.FINDINGS-EMNLP.348

    2020 doi

  15. [29]

    Simple and effective synthesisofindoor3dscenes,in:AAAI,pp.1169–1178

    Koh, J.Y., Agrawal, H., Batra, D., Tucker, R., Waters, A., Lee, H., Yang, Y., Baldridge, J., Anderson, P., 2023. Simple and effective synthesisofindoor3dscenes,in:AAAI,pp.1169–1178. doi:10.1609/ AAAI.V37I1.25199. :Preprint submitted to Elsevier Page 13 of 15

    2023

  16. [31]

    Waypoint models for instruction-guided navigation in continuous environments, in: ICCV, pp

    Krantz, J., Gokaslan, A., Batra, D., Lee, S., Maksymets, O., 2021. Waypoint models for instruction-guided navigation in continuous environments, in: ICCV, pp. 15162–15171. doi:10.1109/ICCV48922. 2021.01488

    2021

  17. [32]

    Room- across-room:Multilingualvision-and-languagenavigationwithdense spatiotemporalgrounding,in:EMNLP,pp.4392–4412.doi:10.18653/ v1/2020.emnlp-main.356

    Ku, A., Anderson, P., Patel, R., Ie, E., Baldridge, J., 2020. Room- across-room:Multilingualvision-and-languagenavigationwithdense spatiotemporalgrounding,in:EMNLP,pp.4392–4412.doi:10.18653/ v1/2020.emnlp-main.356

    2020

  18. [33]

    Unicoder- vl: A universal encoder for vision and language by cross-modal pre- training, in: AAAI, pp

    Li, G., Duan, N., Fang, Y., Gong, M., Jiang, D., 2020. Unicoder- vl: A universal encoder for vision and language by cross-modal pre- training, in: AAAI, pp. 11336–11344. doi:10.1609/AAAI.V34I07.6795

    2020 doi

  19. [34]

    Panogen: Text-conditioned panoramic envi- ronment generation for vision-and-language navigation, in: NeurIPS

    Li, J., Bansal, M., 2023. Panogen: Text-conditioned panoramic envi- ronment generation for vision-and-language navigation, in: NeurIPS

    2023

  20. [36]

    KERM: knowledge enhanced reasoning for vision-and-language navigation, in: CVPR, pp

    Li, X., Wang, Z., Yang, J., Wang, Y., Jiang, S., 2023. KERM: knowledge enhanced reasoning for vision-and-language navigation, in: CVPR, pp. 2583–2592. doi:10.1109/CVPR52729.2023.00254

    2023

  21. [37]

    ROUGE: A package for automatic evaluation of summaries,in:TextSummarizationBranchesOut,Barcelona,Spain

    Lin, C.Y., 2004. ROUGE: A package for automatic evaluation of summaries,in:TextSummarizationBranchesOut,Barcelona,Spain. pp. 74–81. URL:https://aclanthology.org/W04-1013/

    2004

  22. [38]

    Learning vision-and-language navigation from youtube videos, in: CVPR, pp

    Lin, K., Chen, P., Huang, D., Li, T.H., Tan, M., Gan, C., 2023. Learning vision-and-language navigation from youtube videos, in: CVPR, pp. 8283–8292. doi:10.1109/ICCV51070.2023.00764

    2023

  23. [40]

    Self-monitoring navigation agent via auxiliary progress estimation, in: ICLR

    Ma, C., Lu, J., Wu, Z., AlRegib, G., Kira, Z., Socher, R., Xiong, C., 2019a. Self-monitoring navigation agent via auxiliary progress estimation, in: ICLR. URL:https://openreview.net/forum?id= r1GAsjC5Fm

  24. [41]

    Theregretful agent: Heuristic-aided navigation through progress estimation, in: CVPR, pp

    Ma,C.,Wu,Z.,AlRegib,G.,Xiong,C.,Kira,Z.,2019b. Theregretful agent: Heuristic-aided navigation through progress estimation, in: CVPR, pp. 6732–6740. doi:10.1109/CVPR.2019.00689

    2019

  25. [42]

    Improving vision-and-language navigation with image-text pairs from the web, in: ECCV, pp

    Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., Batra, D., 2020. Improving vision-and-language navigation with image-text pairs from the web, in: ECCV, pp. 259–274. doi:10.1007/ 978-3-030-58539-6\_16

    2020

  26. [43]

    SOAT: A scene- and object-aware transformer for vision-and-language navigation, in: NeurIPS, pp

    Moudgil, A., Majumdar, A., Agrawal, H., Lee, S., Batra, D., 2021. SOAT: A scene- and object-aware transformer for vision-and-language navigation, in: NeurIPS, pp. 7357–

    2021

  27. [44]

    Bridging the visual semantic gapinVLNviasemanticallyricherinstructions,in:ECCV,pp.54–69

    Ossandón, J., Earle, B., Soto, Á., 2022. Bridging the visual semantic gapinVLNviasemanticallyricherinstructions,in:ECCV,pp.54–69. doi:10.1007/978-3-031-19836-6\_4

    2022 doi

  28. [45]

    Teach: Task-driven embodied agents that chat, in: AAAI, pp

    Padmakumar, A., Thomason, J., Shrivastava, A., Lange, P., Narayan- Chen, A., Gella, S., Piramuthu, R., Tür, G., Hakkani-Tür, D., 2022. Teach: Task-driven embodied agents that chat, in: AAAI, pp. 2017–

    2022

  29. [46]

    Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Lin- guistics, pp

    Papineni, K., Roukos, S., Ward, T., Zhu, W., 2002. Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Lin- guistics, pp. 311–318. doi:10.3115/1073083.1073135

    2002

  30. [47]

    The road to know-where: An object-and-room informed sequential BERT for indoor vision-language navigation, in: ICCV, pp

    Qi,Y.,Pan,Z.,Hong,Y.,Yang,M.,vandenHengel,A.,Wu,Q.,2021. The road to know-where: An object-and-room informed sequential BERT for indoor vision-language navigation, in: ICCV, pp. 1635–

    2021

  31. [48]

    Object- and-actionawaremodelforvisuallanguagenavigation,in:ECCV,pp

    Qi,Y.,Pan,Z.,Zhang,S.,vandenHengel,A.,Wu,Q.,2020a. Object- and-actionawaremodelforvisuallanguagenavigation,in:ECCV,pp. 303–317. doi:10.1007/978-3-030-58607-2\_18

    doi

  32. [50]

    HOP+: history-enhanced and order-aware pre-training for vision- and-language navigation

    Qiao, Y., Qi, Y., Hong, Y., Yu, Z., Wang, P., Wu, Q., 2023. HOP+: history-enhanced and order-aware pre-training for vision- and-language navigation. IEEE Trans. Pattern Anal. Mach. Intell. 45, 8524–8537. URL:https://doi.org/10.1109/TPAMI.2023.3234243, doi:10.1109/TPAMI.2023.3234243

    2023

  33. [51]

    Learningtransferablevisualmodelsfromnatural language supervision, in: ICML, pp

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever,I.,2021. Learningtransferablevisualmodelsfromnatural language supervision, in: ICML, pp. 8748–8763. URL:http:// proceedings.mlr.press...

    2021

  34. [53]

    Towards long-horizon vision-language navigation: Platform, benchmark and method, in: CVPR

    Song,X.,Chen,W.,Liu,Y.,Chen,W.,Li,G.,Lin,L.,2025. Towards long-horizon vision-language navigation: Platform, benchmark and method, in: CVPR. URL:https://cvpr.thecvf.com/virtual/2025/ poster/33993

    2025

  35. [54]

    Contrastive search is what you need for neuraltextgeneration

    Su, Y., Collier, N., 2023. Contrastive search is what you need for neuraltextgeneration. TransactionsonMachineLearningResearch

    2023

  36. [55]

    Learning to navigate unseen envi- ronments:Backtranslationwithenvironmentaldropout,in:NAACL- HLT, pp

    Tan, H., Yu, L., Bansal, M., 2019. Learning to navigate unseen envi- ronments:Backtranslationwithenvironmentaldropout,in:NAACL- HLT, pp. 2610–2621. doi:10.18653/v1/n19-1268

    2019 doi

  37. [56]

    Vision-and-dialog navigation, in: CoRL, pp

    Thomason, J., Murray, M., Cakmak, M., Zettlemoyer, L., 2019. Vision-and-dialog navigation, in: CoRL, pp. 394–406. URL:http: //proceedings.mlr.press/v100/thomason20a.html

    2019

  38. [57]

    Cider: Consensus- based image description evaluation, in: CVPR, pp

    Vedantam, R., Zitnick, C.L., Parikh, D., 2015. Cider: Consensus- based image description evaluation, in: CVPR, pp. 4566–4575. doi:10.1109/CVPR.2015.7299087

    2015

  39. [58]

    Show and tell: A neural image caption generator, in: CVPR, pp

    Vinyals, O., Toshev, A., Bengio, S., Erhan, D., 2015. Show and tell: A neural image caption generator, in: CVPR, pp. 3156–3164. doi:10.1109/CVPR.2015.7298935

    2015

  40. [60]

    Soft expert reward learning for vision-and-language navigation, in: ECCV, pp

    Wang, H., Wu, Q., Shen, C., 2020. Soft expert reward learning for vision-and-language navigation, in: ECCV, pp. 126–141. doi:10. 1007/978-3-030-58545-7\_8

    2020

  41. [61]

    GIT: A generative image-to-text transformer for vision and language

    Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., Wang, L., 2022b. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research

  42. [62]

    Wang, L., Liu, C., He, Z., Li, S., Yan, Q., Chen, H., Chen, Q.,

  43. [63]

    OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning frame- work, in: ICML, pp

    Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., Yang, H., 2022c. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning frame- work, in: ICML, pp. 23318–23340. URL:https://proceedings.mlr. press/v162...

  44. [64]

    Less is more: Generating grounded navigation instructions from landmarks, in: CVPR, pp

    Wang,S.,Montgomery,C.,Orbay,J.,Birodkar,V.,Faust,A.,Gur,I., Jaques, N., Waters, A., Baldridge, J., Anderson, P., 2022d. Less is more: Generating grounded navigation instructions from landmarks, in: CVPR, pp. 15407–15417. doi:10.1109/CVPR52688.2022.01499

  45. [65]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation, in: CVPR, pp

    Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y., Wang, W.Y., Zhang, L., 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation, in: CVPR, pp. 6629–6638. doi:10.1109/CVPR.2019.00679

    2019

  46. [66]

    LANA: A language- capablenavigatorforinstructionfollowingandgeneration,in:CVPR, IEEE

    Wang, X., Wang, W., Shao, J., Yang, Y., 2023. LANA: A language- capablenavigatorforinstructionfollowingandgeneration,in:CVPR, IEEE. pp. 19048–19058. URL:https://doi.org/10.1109/CVPR52729. 2023.01826, doi:10.1109/CVPR52729.2023.01826

    2023

  47. [67]

    Show, attend and tell: Neural image :Preprint submitted to Elsevier Page 14 of 15 caption generation with visual attention, in: ICML, pp

    Xu,K.,Ba,J.,Kiros,R.,Cho,K.,Courville,A.C.,Salakhutdinov,R., Zemel, R.S., Bengio, Y., 2015. Show, attend and tell: Neural image :Preprint submitted to Elsevier Page 14 of 15 caption generation with visual attention, in: ICML, pp. 2048–2057. URL:http://proceedings.mlr.press/v37...

    2015

  48. [68]

    Unified vision-language pre-training for image captioning and VQA, in: AAAI, pp

    Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J., 2020. Unified vision-language pre-training for image captioning and VQA, in: AAAI, pp. 13041–13049. doi:10.1609/AAAI.V34I07.7005

    2020 doi

  49. [69]

    Vision-language navigation with self-supervised auxiliary reasoning tasks, in: CVPR, pp

    Zhu, F., Zhu, Y., Chang, X., Liang, X., 2020a. Vision-language navigation with self-supervised auxiliary reasoning tasks, in: CVPR, pp. 10009–10019. doi:10.1109/CVPR42600.2020.01003

    2020

  50. [70]

    Babywalk:Goingfartherinvision-and-languagenavigationbytaking babysteps,in:ACL,pp.2539–2556

    Zhu, W., Hu, H., Chen, J., Deng, Z., Jain, V., Ie, E., Sha, F., 2020b. Babywalk:Goingfartherinvision-and-languagenavigationbytaking babysteps,in:ACL,pp.2539–2556. doi:10.18653/v1/2020.acl-main. 229. :Preprint submitted to Elsevier Page 15 of 15

    2020 doi

  51. [380]

    doi:10.3115/V1/W14-3348

    doi

  52. [676]

    doi:10.1109/3DV.2017.00081

    2017

  53. [1644]

    doi:10.1109/ICCV48922.2021.00168

    2021

  54. [2022]

    URL:https://openreview.net/forum?id=b4tMhpN0JC

  55. [2024]

    PASTS: progress-aware spatio-temporal transformer speaker for vision-and-language navigation. Eng. Appl. Artif. Intell. 128, 107487. doi:10.1016/J.ENGAPPAI.2023.107487

    2023

  56. [2025]

    doi:10.1609/AAAI.V36I2.20097

    doi

  57. [7367]

    URL:https://proceedings.neurips.cc/paper/2021/hash/ 3c8a49145944fed2bbcaade178a426c4-Abstract.html

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.