Pith. sign in

REVIEW 4 major objections 5 minor 34 references

InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InstruGen claims that automatically generated path-instruction pairs from YouTube house tours, produced by ChatGPT-4V with a multi-stage verification mechanism, give vision-and-language navigation agents state-of-the-art performance on…

desk verdict A solid data-generation pipeline with an honest ablation, but the SOTA claim is confounded by unequal fine-tuning data. read the letter →

arxiv 2411.11394 v1 pith:CFBUFHUV submitted 2024-11-18 cs.RO

Yu Yan , Rongtao Xu , Jiazhao Zhang , Peiyang Li , Xiaodan Liang , Jianqin Yin This is my paper · ORCID
classification cs.RO
keywords vision-and-languagenavigationinstructiongenerationlargemultimodalmodelsChatGPT-4VYouTubehousetourvideospath-instructionpairsR2RbenchmarkRxR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InstruGen sets out to solve a data bottleneck in vision-and-language navigation: agents trained on the standard R2R/RxR pairs generalize poorly to unseen homes because the training scenes are few and the instructions are templated. The paper's claim is that a fully automatic pipeline can mine house-tour videos from YouTube, sample navigation trajectories, and use a large multimodal model (an AI system that reads images and text and writes text) to write fresh instructions for each path, with a multi-stage verification mechanism filtering out hallucinated or inconsistent outputs. If that claim is right, the expensive manual collection of path-instruction pairs can be replaced by an endlessly renewable source: any walking-through-a-home video becomes usable training data. The reported experiments support the claim by showing that agents pre-trained on InstruGen data beat prior methods on R2R and RxR, with the largest gains on unseen environments.

What carries the argument

The load-bearing object is the trajectory-as-triplets representation combined with the verification mechanism. Each sampled path becomes a sequence such as $\{..., (I_{\mathrm{room}}^t, C_t, a_t), (I_{\mathrm{trans}}, \mathrm{None}, \mathrm{None}), ...\}$, where $C_t$ is a CLIP label of the room and its key objects and $a_t$ is an action estimated by the inverse model of [12]; these triplets are what ChatGPT-4V is prompted to turn into prose. The verification mechanism then extracts $(\text{node label}, \text{action})$ pairs from the generated instruction using the proximity principle and compares them against the ground-truth triplet sequence, regenerating the instruction on any mismatch and applying regular expressions to remove artifacts. This mechanism is what converts a stochastic, sometimes hallucinating LMM into a reliable generator of aligned path-instruction pairs, and its ablation accounts for most of the reported quality gain.

What would settle it

Collect a test batch of sampled YouTube trajectories, deliberately flip a random subset of action labels (left to right, straight to turn), run the full InstruGen generation and verification pipeline, and measure how many corrupted instructions pass verification and how far a follower agent trained on them drifts on R2R unseen splits. If corrupted labels pass verification at a high rate, the consistency module is not actually checking against the path, and the claimed alignment is not the source of the gains.

Watch

Extended reading notes

Core claim

The core discovery is that the bottleneck in VLN generalization is not the navigation model itself but the scarcity and rigidity of training instructions. InstruGen demonstrates that by representing a video-sampled trajectory as a sequence of triplets—image, CLIP-derived room-and-object label, and an estimated turn-or-straight action—a multimodal LLM can be prompted to produce instructions at either coarse or fine granularity that stay aligned with the visual path. The multi-stage verification mechanism (extract the room-and-action pairs implied by the generated text, compare them against the ground-truth triplet sequence, regenerate on mismatch, then clean with regular expressions) is what makes the pipeline reliable enough to use for pre-training. Across R2R and RxR, agents trained with these pairs reach the reported state-of-the-art success rates, particularly on unseen validation splits, and the zero-shot and one-shot settings also improve.

Load-bearing premise

The entire pipeline presupposes that the external action-inverse model correctly identifies whether the camera turned left, turned right, or went straight between consecutive room nodes; a wrong action label produces a plausible-sounding instruction that describes a different route than the actual trajectory.

Editorial extensions

If this is right

  • Pre-training on InstruGen pairs raises success rate on R2R unseen validation from 70.00 (Lily) to 71.10, with larger relative gains on the harder RxR zero-shot setting (29.35 vs. 27.20), so the value concentrates exactly where VLN agents historically fail: never-seen environments.
  • Because the pipeline accepts any house-tour video and any prompt granularity, VLN training data can expand without manual annotation or simulator rendering, removing a major scaling constraint on instruction diversity.
  • The verification module makes LMM-generated instructions nearly as reliable as manually reviewed ones, as the manual-review row in Table I differs only slightly from the automatic verification row, suggesting that expensive human inspection can be largely automated.
  • Coarse-grained instructions during pre-training outperformed fine-grained ones, while mixing rewritten R2R instructions with originals (R2R*) gave the best fine-tuning, implying instruction diversity matters more than instruction length during pre-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains survive replication, the same triplet-and-verify recipe could be applied to other instruction-following domains where raw video is plentiful but labeled demonstrations are scarce, such as tabletop manipulation, driving, or service robots, provided an action estimator exists.
  • The method's ceiling is set by the action inverse model: since actions are estimated rather than ground truth, instruction quality cannot exceed label accuracy; a testable consequence is that replacing the inverse model with a stronger one should further improve R2R unseen success.
  • The paper compares against template-generated instructions (Lily) but not against another LMM-based instruction generator; a like-for-like comparison of generation cost and downstream performance would separate the contribution of the LMM itself from the verification pipeline.
  • An auxiliary metric on instruction quality itself, such as human-rated instruction validity or text-to-path consistency measured by a trained discriminator, would make the mechanism's contribution more directly observable than downstream navigation metrics alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents InstruGen, a pipeline that uses ChatGPT-4V to generate vision-and-language navigation (VLN) path-instruction pairs from trajectories sampled in YouTube house-tour videos. Trajectory nodes are labeled with CLIP, actions between nodes are estimated with an inverse action model, and the generated instructions are passed through a consistency module and regular-expression rules to reduce hallucinations. The authors pre-train and fine-tune a ViLBERT-style agent with the generated data and report state-of-the-art success rates on the R2R and RxR benchmarks, supported by ablations on the verification mechanism and on coarse- versus fine-grained instructions. The paper also acknowledges in Section V a limitation: discrete trajectory sampling may restrict applicability to continuous navigation scenes.

Significance. If the central claim were established, the paper would be a useful contribution: it replaces template-based instruction generation with LMM-based generation, proposes a multi-stage verification mechanism, and shows that YouTube house-tour videos can serve as weakly supervised navigation data that transfers to Matterport-style benchmarks. The provision of code is a positive step for reproducibility. However, the current significance is limited by three issues that are load-bearing for the SOTA claim: the headline comparison is confounded by fine-tuning on R2R* (R2R plus an equal-sized InstruGen rewrite of R2R trajectories), the reported gains are small and lack error bars or significance tests, and the generated dataset itself is not released. The self-referential consistency check is partly mitigated by the manual-review row in Table I, but that mitigation is itself only a single run and is not quantified in detail.

major comments (4)
  1. [§IV-C, Table III (with Table II)] The central SOTA claim is not currently supported because the 'Ours' row in Table III is identical to row #6 of Table II, whose fine-tuning set is R2R* = R2R ∪ R2R#. The baselines in Table III (Speaker-Follower, VLNbert, Airbert, Lily) are fine-tuned on R2R only. Row #4 of Table II, which keeps the InstruGen YouTube pretraining but fine-tunes on R2R only, achieves 69.93 SR on Val-Unseen, essentially the same as Lily's 70.00. The +1.17 SR difference between row #4 and row #6 therefore measures the effect of doubling the number of fine-tuning instructions, not the quality of InstruGen-generated instructions per se. A controlled comparison is needed: for example, fine-tune the same architecture on R2R plus a same-sized set of template-generated or speaker-generated rewrites, or compare the full InstruGen pipeline against baselines that also receive an equal-sized additional fine-tuning set, with multiple seeds. The same confound affects Table V, where the zero-shot RxR model is described as using row #6 from Table II.
  2. [§IV-B, Tables I–II] All conclusions about the verification mechanism and instruction granularity rest on single-run SR values with no error bars or significance tests. The differences at stake are small: in Table I, rows #3 and #4 differ by 0.2 SR (70.9 vs 71.10), and Table II contains several 1-point-scale differences. The text uses phrases such as 'significantly better' and 'significant effect,' but no statistical evidence is provided. Please report means and standard deviations over multiple seeds, and where feasible paired tests or confidence intervals; if the training cost makes this prohibitive, state that explicitly and provide the best-of-N or a similar protocol.
  3. [§III-A] The action inverse model borrowed from [12] is not evaluated independently. The actions produced by this model are used both as input to ChatGPT-4V for instruction generation and as ground truth in the consistency verification module, so any systematic error in the action model will propagate directly into the generated instructions and cannot be caught by the verification step. Please include a validation of the action model on a sample of manually annotated action sequences, or at least a sensitivity analysis that replaces it with alternative action estimates, so that the reliability of the generated path-instruction pairs can be assessed.
  4. [§IV-A and Data Availability] The paper lists a dataset as one of its contributions and provides a code URL, but the generated YouTube path-instruction pairs and, in particular, the R2R# and R2R* sets are not released. Because the central experiments depend on this generated data, the inability to download or inspect the data prevents independent verification of the pipeline and of the main empirical claims. Please release the dataset, or at least a representative sample, with the revised version.
minor comments (5)
  1. [Appendix A] The negative-sample construction contains unresolved placeholders: 'see Figure ??' appears twice in the description of the two methods for constructing trajectory negative samples. These references should be fixed or removed.
  2. [§IV-A, Table II] The table header has typos and inconsistent spacing, including 'Finetuing' and 'R2R #' versus 'R2R#' in the text. The column alignment should be cleaned so that the pretraining, fine-tuning, and testing columns are unambiguous.
  3. [§III-B] The displayed prompt appears to contain stray line numbers ('451', '452', etc.) at the beginning of lines in the code block. If these are not part of the actual prompt, they should be removed for clarity.
  4. [§III-B and Table I] The consistency module asks ChatGPT-4V to judge outputs that the same model generated, so it is not an independent verifier. Although Table I row #4 shows that manual review gives a similar result (71.10 vs 70.9), the paper should state this self-referential limitation explicitly and report the agreement rate between the automated consistency check and manual review.
  5. [§II-B and §IV-C] The paper does not compare with Marky [2] or other recent LMM-based instruction generation methods in the experimental section. Even if those methods require different training protocols, a discussion of why they are not included as baselines would help position the contribution.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; benchmark evaluation is external, with minor non-load-bearing self-citations and a data-quantity confound in the SOTA comparison.

full rationale

The paper's derivation chain is not circular. The central claim—that agents using InstruGen-generated path-instruction pairs achieve SOTA on R2R/RxR—is tested on external benchmark splits (R2R val-unseen, RxR val-unseen) against baselines, not on the generated data themselves. The InstruGen pipeline (CLIP node labeling, action inverse model [12], ChatGPT-4V instruction generation, consistency/regular-expression verification, and PIJ/PR pretext tasks) feeds training data into a ViLBERT-style agent; nothing in the pipeline fits to or predicts the test benchmark. The consistency module is a self-consistency filter: it compares extracted (node, action) pairs with the same trajectory triplets used in the generation prompt, so it cannot detect errors in the external action-inverse model; however, Table I row #4 shows manual review gives nearly identical val-unseen performance (71.10 vs 70.9), so this self-reference is not load-bearing. The main experimental caveat is a fairness confound, not circularity: the best model (Table III 'Ours' 71.10) is fine-tuned on R2R* = R2R ∪ R2R#, while Lily and Airbert are fine-tuned on R2R only; the paper itself attributes the R2R*-vs-R2R gain to 'seeing more instructions' (Section IV-B). This affects attribution of the SOTA margin to instruction quality rather than data quantity, but the evaluation remains external and the generated set is not used to define the metric. The self-citations ([3], [31]) are related-work references and are not load-bearing. Section V limitation (discrete trajectory sampling) is acknowledged and does not bear on circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The claim does not introduce new free parameters; it inherits pretrained models whose correctness is assumed, such as CLIP, the action inverse model, ChatGPT-4V, and ViLBERT. No new physical or conceptual entities are postulated.

assumptions (4)
  • domain assumption ChatGPT-4V interprets image sequences and produces coherent navigation instructions from the provided labels and actions.
    Used throughout Section III-B; all generated data quality depends on this ability.
  • domain assumption CLIP labels for room types and key objects are correct.
    Section III-A Node Labeling uses CLIP to create the (image, node label, action) triplets that ground the instructions.
  • domain assumption The action inverse model from [12] returns accurate transition actions between frames.
    Section III-A Action Grounding adopts this model; wrong actions propagate into wrong instructions.
  • domain assumption YouTube house tour videos are representative of real indoor navigation environments for R2R transfer.
    The pretraining data in Section IV.A is drawn from 3879 YouTube videos; transfer to Matterport-based R2R/RxR depends on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models." pith.science (2026). https://pith.science/paper/CFBUFHUV

@misc{pith2026241111394,
  author       = {Pith},
  title        = {Pith review of: InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CFBUFHUV}},
  note         = {Machine review of arXiv:2411.11394}
}
read the original abstract

Recent research on Vision-and-Language Navigation (VLN) indicates that agents suffer from poor generalization in unseen environments due to the lack of realistic training environments and high-quality path-instruction pairs. Most existing methods for constructing realistic navigation scenes have high costs, and the extension of instructions mainly relies on predefined templates or rules, lacking adaptability. To alleviate the issue, we propose InstruGen, a VLN path-instruction pairs generation paradigm. Specifically, we use YouTube house tour videos as realistic navigation scenes and leverage the powerful visual understanding and generation abilities of large multimodal models (LMMs) to automatically generate diverse and high-quality VLN path-instruction pairs. Our method generates navigation instructions with different granularities and achieves fine-grained alignment between instructions and visual observations, which was difficult to achieve with previous methods. Additionally, we design a multi-stage verification mechanism to reduce hallucinations and inconsistency of LMMs. Experimental results demonstrate that agents trained with path-instruction pairs generated by InstruGen achieves state-of-the-art performance on the R2R and RxR benchmarks, particularly in unseen environments. Code is available at https://github.com/yanyu0526/InstruGen.

Figures

Figures reproduced from arXiv: 2411.11394 by the authors.

Figure 1
Figure 1. Generate navigation instructions for trajectories sampled from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. InstruGen consists of the following stages: (a) Trajectory Generation, (b) Instruction Generation, and (c) Trajectory Judgment. consists of room nodes and transition nodes, i.e., I =  ..., Iroom t , Itrans, Iroom t+1 , ... , where I room t denotes the im￾age of the room node at time t, I trans denotes the image of the transition node. Node Labeling. We found that directly inputting trajectory images into ChatGPT-4V… view at source ↗
Figure 5
Figure 5. The impact of data quality on loss and accuracy during the pre [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: The model’s performance with instructions of different granularities [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 28 canonical work pages

  1. [12]

    Learning Navigation Subroutines from Egocentric Videos

    A. Kumar, S. Gupta, and J. Malik, “Learning navigation subroutines by watching videos,” ArXiv, vol. abs/1905.12612, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:168169685

    work page Pith review arXiv 1905

  2. [1]

    Learning from unlabeled 3d environments for vision-and-language navigation,

    S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Learning from unlabeled 3d environments for vision-and-language navigation,” in European Conference on Computer Vision . Springer, 2022, pp. 638–655

    work page 2022

  3. [2]

    A new path: Scaling vision- and-language navigation with synthetic instructions and imitation learning,

    A. Kamath, P. Anderson, S. Wang, J. Y . Koh, A. Ku, A. Waters, Y . Yang, J. Baldridge, and Z. Parekh, “A new path: Scaling vision- and-language navigation with synthetic instructions and imitation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 10 813–10 823

    work page 2023

  4. [3]

    Navid: Video-based vlm plans the next step for vision-and-language navigation,

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and W. He, “Navid: Video-based vlm plans the next step for vision-and-language navigation,” arXiv preprint arXiv:2402.15852, 2024

    arXiv 2024

  5. [4]

    Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3674–3683

    2018

  6. [5]

    Diagnosing the environment bias in vision-and-language navigation,

    Y . Zhang, H. Tan, and M. Bansal, “Diagnosing the environment bias in vision-and-language navigation,” 2020. [Online]. Available: https://openreview.net/forum?id=S1eYKlrYvr

    work page 2020

  7. [6]

    Matterport3d: Learning from RGB-D data in indoor environments,

    A. X. Chang, A. Dai, T. A. Funkhouser, M. Halber, M. Nießner, M. Savva, S. Song, A. Zeng, and Y . Zhang, “Matterport3d: Learning from RGB-D data in indoor environments,” in 3DV, 2017, pp. 667– 676

    work page 2017

  8. [7]

    Learning to navigate unseen envi- ronments: Back translation with environmental dropout,

    H. Tan, L. Yu, and M. Bansal, “Learning to navigate unseen envi- ronments: Back translation with environmental dropout,” in NAACL, J. Burstein, C. Doran, and T. Solorio, Eds., 2019, pp. 2610–2621

    work page 2019

Show all 34 references
  1. [8]

    The regretful agent: Heuristic-aided navigation through progress estimation,

    C.-Y . Ma, Z. Wu, G. AlRegib, C. Xiong, and Z. Kira, “The regretful agent: Heuristic-aided navigation through progress estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6732–6740

    2019

  2. [9]

    Speaker- follower models for vision-and-language navigation,

    D. Fried, R. Hu, V . Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell, “Speaker- follower models for vision-and-language navigation,” Advances in neural information processing systems , vol. 31, 2018

    2018

  3. [10]

    Improving vision-and-language navigation with image-text pairs from the web,

    A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, “Improving vision-and-language navigation with image-text pairs from the web,” in ECCV, 2020, pp. 259–274

    2020

  4. [11]

    Air- bert: In-domain pretraining for vision-and-language navigation,

    P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid, “Air- bert: In-domain pretraining for vision-and-language navigation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1634–1643

    2021

  5. [13]

    Semantic visual navigation by watching youtube videos,

    M. Chang, A. Gupta, and S. Gupta, “Semantic visual navigation by watching youtube videos,” Advances in Neural Information Processing Systems, vol. 33, pp. 4283–4294, 2020

    2020

  6. [14]

    Learning vision-and-language navigation from youtube videos,

    K. Lin, P. Chen, D. Huang, T. H. Li, M. Tan, and C. Gan, “Learning vision-and-language navigation from youtube videos,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 8317–8326

    2023

  7. [15]

    Self-monitoring navigation agent via auxiliary progress estimation,

    C. Ma, J. Lu, Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong, “Self-monitoring navigation agent via auxiliary progress estimation,” in ICLR, 2019

    2019

  8. [16]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision),

    Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, “The dawn of lmms: Preliminary explorations with gpt-4v (ision),” arXiv preprint arXiv:2309.17421 , vol. 9, no. 1, p. 1, 2023

    2023 arXiv

  9. [17]

    Touchdown: Natural language navigation and spatial reasoning in visual street envi- ronments,

    H. Chen, A. Suhr, D. Misra, N. Snavely, and Y . Artzi, “Touchdown: Natural language navigation and spatial reasoning in visual street envi- ronments,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 12 538–12 547

    2019

  10. [18]

    Vision- and-dialog navigation,

    J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer, “Vision- and-dialog navigation,” in Conference on Robot Learning . PMLR, 2020, pp. 394–406

    2020

  11. [19]

    Cross-lingual vision-language navigation,

    A. Yan, X. E. Wang, J. Feng, L. Li, and W. Y . Wang, “Cross-lingual vision-language navigation,” arXiv preprint arXiv:1910.11301 , 2019

    1910 arXiv

  12. [20]

    Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,

    A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge, “Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,” in EMNLP, 2020, pp. 4392–4412

    2020

  13. [21]

    Landmark-rxr: Solving vision-and-language navigation with fine- grained alignment supervision,

    K. He, Y . Huang, Q. Wu, J. Yang, D. An, S. Sima, and L. Wang, “Landmark-rxr: Solving vision-and-language navigation with fine- grained alignment supervision,” Advances in Neural Information Pro- cessing Systems, vol. 34, pp. 652–663, 2021

    2021

  14. [22]

    Reverie: Remote embodied visual referring expression in real indoor environments,

    Y . Qi, Q. Wu, P. Anderson, X. Wang, W. Y . Wang, C. Shen, and A. v. d. Hengel, “Reverie: Remote embodied visual referring expression in real indoor environments,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9982–9991

    2020

  15. [23]

    Teach: Task-driven embodied agents that chat,

    A. Padmakumar, J. Thomason, A. Shrivastava, P. Lange, A. Narayan- Chen, S. Gella, R. Piramuthu, G. Tur, and D. Hakkani-Tur, “Teach: Task-driven embodied agents that chat,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 2, 2022, pp. 2017– 2025

    2022

  16. [24]

    Vision-and- language navigation: A survey of tasks, methods, and future direc- tions,

    J. Gu, E. Stefani, Q. Wu, J. Thomason, and X. Wang, “Vision-and- language navigation: A survey of tasks, methods, and future direc- tions,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Compu...

    2022

  17. [25]

    Ndh-full: Learning and evaluating nav- igational agents on full-length dialogue,

    H. Kim, J. Li, and M. Bansal, “Ndh-full: Learning and evaluating nav- igational agents on full-length dialogue,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021

    2021

  18. [26]

    Multimodal text style transfer for outdoor vision- and-language navigation,

    W. Zhu, X. E. Wang, T.-J. Fu, A. Yan, P. Narayana, K. Sone, S. Basu, and W. Y . Wang, “Multimodal text style transfer for outdoor vision- and-language navigation,” arXiv preprint arXiv:2007.00229 , 2020

    2007 arXiv

  19. [27]

    Counterfactual vision-and-language navigation: Unravelling the unseen,

    A. Parvaneh, E. Abbasnejad, D. Teney, J. Q. Shi, and A. Van den Hengel, “Counterfactual vision-and-language navigation: Unravelling the unseen,” Advances in neural information processing systems , vol. 33, pp. 5296–5307, 2020

    2020

  20. [28]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,

    X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y .-F. Wang, W. Y . Wang, and L. Zhang, “Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recogni...

    2019

  21. [29]

    Vision- language navigation with random environmental mixup,

    C. Liu, F. Zhu, X. Chang, X. Liang, Z. Ge, and Y .-D. Shen, “Vision- language navigation with random environmental mixup,” in Proceed- ings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1644–1654

    2021

  22. [30]

    Envedit: Environment editing for vision-and-language navigation,

    J. Li, H. Tan, and M. Bansal, “Envedit: Environment editing for vision-and-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 15 407–15 417

    2022

  23. [31]

    Visual-language nav- igation pretraining via prompt-based environmental self-exploration,

    X. Liang, F. Zhu, L. Li, H. Xu, and X. Liang, “Visual-language nav- igation pretraining via prompt-based environmental self-exploration,” in ACL, 2022, pp. 4837–4851

    2022

  24. [32]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  25. [33]

    Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,” Advances in neural information processing systems , vol. 32, 2019

    2019

  26. [34]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation,

    S. Chen, P. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” in CVPR, 2022, pp. 16 516–16 526

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.