Pith. sign in

REVIEW 48 references

MemVLN reaches state-of-the-art continuous vision-and-language navigation at 14 FPS by pairing pyramidal episodic memory with single-token mid-level actions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 20:36 UTC pith:37PPN6U2

load-bearing objection Solid real-time monocular VLN system; mechanisms are real but incremental, and the R2R SOTA claim is thinner than the abstract sells.

arxiv 2607.23504 v1 pith:37PPN6U2 submitted 2026-07-26 cs.CV

MemVLN: Episodic and Procedural Memory for Vision-and-Language Navigation

Yuqi Liu , Shengju Qian , Tianyuan Qu , Mingxian Lin , Zixuan Wang , Xin Wang , Bei Yu , Jiaya Jia This is my paper
classification cs.CV
keywords vision-and-language navigationVLN-CEepisodic memoryprocedural memorypyramidal resolutionmid-level actionsreal-time inferencemonocular RGB
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Continuous vision-and-language navigation needs two things at once: long visual history so the agent does not drift off the instructed path, and low-latency actions so it can control a body in real time. Most video-based vision-language models fail one or the other—either they throw away temporal density or they drown in slow autoregressive decoding. MemVLN claims both can be solved together with two memory designs: an episodic store that keeps recent frames sharp and older frames progressively coarser (a resolution pyramid), and a procedural store that turns navigation into a one-shot choice among a handful of atomic mid-level action tokens. Using only monocular RGB and language, the resulting models set new marks on R2R-CE and RxR-CE while running at roughly 14 frames per second. A reader who wants practical embodied agents should care because the work shows that careful memory and action design, not richer sensors or heavier models alone, can unlock real-time instruction following from ordinary vision-language backbones.

Core claim

MemVLN shows that a vision-language agent can keep long-horizon visual history and still act at real-time speed by combining pyramidal-resolution episodic memory with a single-shot vocabulary of mid-level actions. With monocular RGB only, MemVLN-8B reaches 58.4% success on R2R-CE and 66.0% on RxR-CE Val-Unseen, beating prior VLM navigators and lifting the Qwen3-VL-4B baseline by several points in success rate while cutting action latency by about 7× versus autoregressive decoding.

What carries the argument

Episodic memory with pyramidal resolution: frames are split into immediate, short-term, and long-term tiers and rescaled (e.g. 512×512 / 256×256 / 128×128) so recent views stay detailed while distant history is compressed, still on a uniform spatial grid. Procedural memory: a compact set of single tokens for fixed-distance forwards, fixed-angle turns, and stop, so the policy is one classification step instead of multi-token generation.

Load-bearing premise

That a small hand-designed set of roughly ten mid-level actions with fixed step lengths and turn angles is expressive enough to stand in for continuous low-level control without costing path precision or stop accuracy.

What would settle it

Hold the visual pyramid fixed and replace the discrete mid-level action tokens with either finer continuous control or multi-token text actions; if success rate and SPL fall sharply once latency is matched, or if stop errors rise on long RxR paths, the procedural-memory claim fails. A second check is equal-token uniform high-resolution history versus the ascending pyramid on the same long trajectories—if the reported SR/SPL gains vanish, the episodic design is not doing the claimed work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Real-time (~14 FPS) monocular VLN is achievable without panoramic images, depth, or odometry.
  • Pyramidal downsampling of older frames expands look-back under a fixed token budget more usefully than uniform sampling for navigation.
  • Single-token mid-level actions cut inference latency by roughly 7× versus autoregressive text actions and can also raise success metrics.
  • Adding interactive DAgger-style data further lifts oracle success and success rate on both R2R and RxR.
  • The same recipe transfers zero-shot from R2R-only training to RxR at competitive success rates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because attention already concentrates on recent frames, ascending resolution pyramids may transfer to other long-video embodied tasks (mobile manipulation, multi-room dialogue) without changing the language backbone.
  • If the fixed mid-level vocabulary is the precision bottleneck, learned or hierarchical action tokens could restore finer control while keeping single-shot speed.
  • Doing compression in pixel space rather than by latent token merging keeps compatibility with grid-based positional schemes, so the same memory pattern is a portable drop-in for other modern vision-language backbones.
  • The paper’s own GPU-dependence note plus the already-tiny action head suggest quantization and edge ports are the next practical test for onboard robots.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Circularity Check

0 steps flagged

No circularity: empirical SFT architecture paper evaluated on external Habitat VLN-CE metrics

full rationale

MemVLN is a standard empirical VLN systems paper. Its load-bearing claims are (i) monocular-RGB SOTA on R2R-CE/RxR-CE Val-Unseen and (ii) real-time latency via two design choices (pyramidal frame resolutions; single-token mid-level action vocabulary). Neither claim is derived from a fitted constant, a self-defined identity, or a load-bearing self-citation uniqueness theorem. Training maximizes ordinary cross-entropy on public trajectory corpora (Eq. 2); success/NE/SPL are measured in Habitat against held-out episodes. Ablations (Tables 3–6) compare design variants under fixed protocols; gains are empirical deltas, not quantities forced by construction from the inputs. Bio-inspired naming (episodic/procedural memory) is framing, not a renamed known identity presented as a derivation. No step reduces a reported prediction to its own fit or definition. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Load-bearing content is almost entirely engineering design choices and standard embodied-AI assumptions, not mathematical axioms. The central performance claim rests on: Habitat/Matterport evaluation fidelity; the sufficiency of monocular RGB; a hand-chosen three-tier resolution schedule and action token set; and supervised imitation on a particular data mixture. No new physical entity is postulated. Free parameters are the usual VLN/VLM knobs that were selected rather than derived.

free parameters (5)
  • pyramidal resolutions (r_imm, r_short, r_long) = 512 / 256 / 128
    Chosen as 512×512 / 256×256 / 128×128; not derived. Ablated only coarsely via frame counts, not via a principled resolution schedule.
  • tier frame counts and temporal boundaries B_short, B_long = (4, 8, 32) preferred
    Best reported config is (#imm,#short,#long)=(4,8,32); boundaries define the pyramid but exact time cutoffs are design choices tuned on val performance.
  • augmented mid-level action set A_aug = 10 tokens (Appendix Table 8)
    Ten discrete tokens with fixed metric steps (25/50/75 cm, 15/30/45°) hand-designed; expressivity is assumed, not learned from continuous control.
  • SFT hyperparameters (lr, epochs, batch, warmup) = lr=1e-5, 1 epoch, bs=128
    Peak lr 1e-5, 1 epoch, batch 128, warmup 0.03—standard fitted training knobs that affect final SR/SPL.
  • training data mixture weights = full mixture in final model
    R2R/RxR + EnvDrop + 416K ScaleVLN + 557K human + 278K DAgger; composition strongly moves metrics (Table 6) and is chosen by authors.
axioms (5)
  • domain assumption Habitat continuous control with the standard 4-base / mid-level discrete action interface is a valid proxy for real monocular navigation performance.
    All claims are on R2R-CE/RxR-CE Val-Unseen in Habitat; sim-to-real is acknowledged as prior open problem (Intro) but not re-validated here.
  • domain assumption Qwen3-VL visual encoder + LLM backbone, after SFT, can ground language landmarks in egocentric RGB without depth or odometry.
    Architecture section 3.1 treats the LVLM as the decision module; success of monocular-only SOTA rests on this.
  • ad hoc to paper Recent frames deserve higher spatial fidelity than distant frames (attention mass concentrates on the last few observations).
    Justified by a one-step attention probe on 12 uniform frames (Fig. 3 / Sec. 3.2), then hard-coded into the pyramid; treated as general when it is an empirical observation on their model.
  • domain assumption Pixel-space downsampling preserves a uniform 2D grid and is therefore preferable to latent token merging for M-RoPE/DeepStack compatibility.
    Sec. 3.2 'Why not Token Merging?'—architectural compatibility claim used to motivate the method over Uni-NaVid/StreamVLN compression.
  • domain assumption Standard cross-entropy on expert/mid-level actions is a sufficient training objective for long-horizon VLN-CE (no RL fine-tune required for the reported claims).
    Sec. 3.4 objective is pure CE over A_aug; DAgger data is included but no online RL loop is claimed.
invented entities (2)
  • Episodic Memory pyramidal resolution tiers (immediate / short-term / long-term) no independent evidence
    purpose: Compress long visual history while keeping recent frames high-resolution inside a fixed token budget.
    Named 'episodic memory' but is a multi-resolution frame buffer with fixed cutoffs; independent evidence is only the paper's own ablations, not external memory theory validation.
  • Procedural Memory augmented action vocabulary A_aug (single-token mid-level actions) no independent evidence
    purpose: Collapse action generation to one forward-pass token to avoid autoregressive decode latency.
    Re-purposed LLM tokens bound to fixed metric primitives; listed in Appendix Table 8. Falsifiable only via navigation metrics in this paper, not via an external motor-control benchmark.

pith-pipeline@v1.2.0-grok45-kimik3 · 18841 in / 4506 out tokens · 94675 ms · 2026-07-30T20:36:32.057346+00:00 · methodology

0 comments
read the original abstract

Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to maintain long-horizon visual history for trajectory consistency while executing actions with low latency. Existing video-based VLN approaches typically struggle to satisfy both demands simultaneously. To address these challenges, we propose MemVLN, a novel VLN framework that achieves state-of-the-art performance with real-time inference efficiency (14 FPS). MemVLN utilizes a visual encoder to process continuous observations and a Large Language Model (LLM) to interpret instructions and generate actions. Central to our approach is an Episodic Memory management that applies pyramidal resolutions. This mechanism concentrates computation on immediate percepts while retaining compressed long-term history. Complementing to this design, we introduce Procedural Memory for fast action with a compact vocabulary of atomic mid-level actions to bypass auto-regressive decoding latency. Experiments on VLN-CE show that MemVLN-4B surpasses the baseline Qwen3-VL-4B architecture by 5.8\% SR in R2R and 9.7\% SR in RxR, while achieving a 7$\times$ speedup in inference latency.

Figures

Figures reproduced from arXiv: 2607.23504 by Bei Yu, Jiaya Jia, Mingxian Lin, Shengju Qian, Tianyuan Qu, Xin Wang, Yuqi Liu, Zixuan Wang.

Figure 2
Figure 2. Figure 2: Overview of the MemVLN Architecture. Given a natural language instruction and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Attention analysis across visual observations. Left: The final four frames account for over [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pyramidal Resolution vs. Token Merging. Our [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Top: Different types of pyramidal resolutions. Bottom: Attention distribution. Darker [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization on R2R-CE and RxR-CE in the Habitat simulator. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 14 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35: 23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35: 23716–23736, 2022

    2022

  3. [3]

    Etpnav: Evolving topological planning for vision-language navigation in continuous environments.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. Etpnav: Evolving topological planning for vision-language navigation in continuous environments.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  4. [4]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683, 2018

    2018

  5. [5]

    Sim-to-real transfer for vision-and-language navigation

    Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee. Sim-to-real transfer for vision-and-language navigation. InConference on Robot Learning, pages 671–681. PMLR, 2021

    2021

  6. [6]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Topological planning with transformers for vision-and-language navigation

    Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. Topological planning with transformers for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11276–11286, 2021

    2021

  8. [8]

    Weakly-supervised multi-granularity map learning for vision-and-language navigation

    Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas Li, Mingkui Tan, and Chuang Gan. Weakly-supervised multi-granularity map learning for vision-and-language navigation. Advances in Neural Information Processing Systems, 35:38149–38161, 2022

    2022

  9. [9]

    Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  11. [11]

    Speaker- follower models for vision-and-language navigation.Advances in neural information processing systems, 31, 2018

    Daniel Fried, Ronghang Hu, V olkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for vision-and-language navigation.Advances in neural information processing systems, 31, 2018

    2018

  12. [12]

    Counterfactual vision-and-language navigation via adversarial path sampler

    Tsu-Jui Fu, Xin Eric Wang, Matthew F Peterson, Scott T Grafton, Miguel P Eckstein, and William Yang Wang. Counterfactual vision-and-language navigation via adversarial path sampler. InEuropean Conference on Computer Vision, pages 71–86. Springer, 2020

    2020

  13. [13]

    Cross-modal map learning for vision and language navigation

    Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15460–15470, 2022

    2022

  14. [14]

    Language and visual entity relationship graph for agent navigation.Advances in Neural Information Processing Systems, 33:7685–7696, 2020

    Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity relationship graph for agent navigation.Advances in Neural Information Processing Systems, 33:7685–7696, 2020. 10

    2020

  15. [15]

    Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation

    Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15439–15449, 2022

    2022

  16. [16]

    Sim-2-sim transfer for vision-and-language navigation in con- tinuous environments

    Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision-and-language navigation in con- tinuous environments. InEuropean conference on computer vision, pages 588–603. Springer, 2022

    2022

  17. [17]

    Beyond the nav- graph: Vision-and-language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and-language navigation in continuous environments. InEuropean Conference on Computer Vision, pages 104–120. Springer, 2020

    2020

  18. [18]

    Waypoint models for instruction-guided navigation in continuous environments

    Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction-guided navigation in continuous environments. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15162–15171, 2021

    2021

  19. [19]

    Room-across- room: Multilingual vision-and-language navigation with dense spatiotemporal grounding

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across- room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392–4412, 2020

    2020

  20. [20]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  22. [22]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26689–26699, 2024

    2024

  23. [23]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  24. [24]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.arXiv preprint arXiv:2406.04882, 2024

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.arXiv preprint arXiv:2406.04882, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Discuss before moving: Visual language navigation via multi-expert discussions

    Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. Discuss before moving: Visual language navigation via multi-expert discussions. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 17380–17387. IEEE, 2024

    2024

  26. [26]

    Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Advances in Neural Information Processing Systems, 37:23464–23487, 2024

    2024

  27. [27]

    Reverie: Remote embodied visual referring expression in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9982–9991, 2020

    2020

  28. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  29. [29]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.arXiv preprint arXiv:2109.08238, 2021

    Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.arXiv preprint arXiv:2109.08238, 2021. 11

    Pith/arXiv arXiv 2021

  30. [30]

    Language- aligned waypoint (law) supervision for vision-and-language navigation in continuous envi- ronments

    Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel Chang. Language- aligned waypoint (law) supervision for vision-and-language navigation in continuous envi- ronments. InProceedings of the 2021 conference on empirical methods in natural language processing, pages 4018–4028, 2021

    2021

  31. [31]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings, 2011

    2011

  32. [32]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. InProceedings of the IEEE/CVF international conference on computer vision, pages 9339–9347, 2019

    2019

  33. [33]

    Learning to navigate unseen environments: Back translation with environmental dropout

    Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 2610–2621, 2019

    2019

  34. [34]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Vision-and-dialog navigation

    Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. InConference on Robot Learning, pages 394–406. PMLR, 2020

    2020

  36. [36]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Dreamwalker: Mental planning for continuous vision-language navigation

    Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation. InProceedings of the IEEE/CVF international conference on computer vision, pages 10873–10883, 2023

    2023

  38. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. InProceedings of the IEEE/CVF International conference on computer vision, pages 15625–15636, 2023

    2023

  40. [40]

    Lookahead exploration with neural radiance representation for continuous vision-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Junjie Hu, Ming Jiang, and Shuqiang Jiang. Lookahead exploration with neural radiance representation for continuous vision-language navigation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13753–13762, 2024

    2024

  41. [41]

    Sim-to-real transfer via 3d feature fields for vision-and-language navigation.arXiv preprint arXiv:2406.09798, 2024

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Sim-to-real transfer via 3d feature fields for vision-and-language navigation.arXiv preprint arXiv:2406.09798, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Scaling data generation in vision-and-language navigation

    Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation. InProceedings of the IEEE/CVF international conference on computer vision, pages 12009–12020, 2023

    2023

  43. [43]

    Streamvln: Streaming vision-and-language navigation via slowfast context modeling.arXiv preprint arXiv:2507.05240, 2025

    Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang, Yuqiang Yang, Xiaohan Mao, Chenming Zhu, Wenzhe Cai, Hanqing Wang, Yilun Chen, et al. Streamvln: Streaming vision-and-language navigation via slowfast context modeling.arXiv preprint arXiv:2507.05240, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Gibson env: Real-world perception for embodied agents

    Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 9068–9079, 2018. 12

    2018

  45. [45]

    Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks.arXiv preprint arXiv:2412.06224, 2024

    Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks.arXiv preprint arXiv:2412.06224, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Navid: Video-based vlm plans the next step for vision-and-language navigation.arXiv preprint arXiv:2402.15852, 2024

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation.arXiv preprint arXiv:2402.15852, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Lyra: An efficient and speech-centric framework for omni-cognition

    Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, et al. Lyra: An efficient and speech-centric framework for omni-cognition. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3694–3704, 2025

    2025

  48. [48]

    Navgpt: Explicit reasoning in vision-and-language navigation with large language models

    Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649, 2024. 13 Table 7: Cross-dataset performance on the RxR-CE Val Unseen split. MemVLN is trained exclusively on R2R training split in this se...

    2024