Pith. sign in

REVIEW 4 major objections 6 minor 19 cited by

StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read StreamVLN shows that a video-LLM can drive continuous navigation with a fixed-size dialogue memory by splitting context into a fast sliding-window cache and a slow pruned visual memory.

desk verdict Solid streaming VLN system with real ablations, but the bounded-context headline needs an explicit memory-eviction rule before the claim holds. read the letter →

arxiv 2507.05240 v2 pith:PGIX3K5A submitted 2025-07-07 cs.RO cs.CV

Meng Wei , Chenyang Wan , Xiqian Yu , Tai Wang , Yuqiang Yang , Xiaohan Mao , Chenming Zhu , Wenzhe Cai
show 4 more authors
Hanqing Wang Yilun Chen Xihui Liu Jiangmiao Pang
This is my paper · ORCID
classification cs.ROcs.CV
keywords vision-and-languagenavigationvideo-LLMstreamingdialogueKVcachereusevoxel-basedtokenpruningslow-fastcontextmodelinglow-latencyinferenceembodiedAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StreamVLN is a streaming framework for vision-and-language navigation that turns each navigation episode into a multi-turn dialogue between the agent and a video-based large language model. The paper's central claim is that a hybrid slow-fast context design lets such a model follow long, low-level navigation instructions while keeping the transformer's context size bounded and its per-turn latency roughly constant. On the two standard continuous-navigation benchmarks, R2R-CE and RxR-CE, the method reports new state-of-the-art results among RGB-only video-LLM approaches, with 56.9% SR and 51.9% SPL on R2R and 52.9% SR and 46.0% SPL on RxR when trained with additional data. A sympathetic reader would care because the result suggests that real-time embodied deployment of video-LLMs does not require unbounded memory or repeated recomputation of the whole history.

What carries the argument

The load-bearing mechanism is the hybrid slow-fast context: a fixed-size sliding window of recent dialogue turns with KV-cache reuse for responsiveness, plus a slowly updated visual memory compressed by voxel-based spatial pruning. The pruning back-projects image-patch tokens into a shared 3D voxel grid using depth information, keeps only the newest token for each voxel within a stride window, and drops entire frames whose kept token count falls below a threshold. This is what keeps the context bounded while retaining high-resolution spatial detail; it works because in a static indoor scene many patches from consecutive frames describe the same physical surface.

What would settle it

Run StreamVLN on R2R-CE validation-unseen with voxel pruning switched off and with depth corrupted by noise while holding all else fixed; the paper's Table 5 predicts success rate should drop by about one point and token count should rise about 20% without pruning, so if noisy depth erases or reverses that gain, the claim that 3D pruning is the mechanism fails. The authors also do not report the stride $K$ and threshold $ heta$, so a reproducibility check would need those values first.

Watch

Extended reading notes

Core claim

The paper extends the LLaVA-Video 7B model into an interleaved vision-language-action model in which observations and actions alternate as dialogue turns. A fast-streaming path keeps the most recent eight dialogue turns in a sliding-window KV cache, reusing cached key/value states so that decoding a new action only prefills the current observation, eliminating over 99% of prefill time. A slow-updating memory path compresses older visual states by uniformly sampling frames and then pruning tokens whose image patches land in the same 3D voxel, retaining only the most recent observation per voxel. This voxel-based spatial pruning cuts about 20% of input tokens and slightly improves success rate on both benchmarks. The paper reports that with extra data these components yield 56.9% SR and 51.9% SPL on R2R-CE and 52.9% SR and 46.0% SPL on RxR-CE, and that a robot deployment runs at about 0.27 seconds for four actions.

Load-bearing premise

For the pruning to deliver the reported memory savings and accuracy gain, the agent must have accurate metric depth at test time, yet depth is not part of the advertised RGB-only input and the pruning thresholds are not reported.

Editorial extensions

If this is right

  • KV-cache reuse across turns removes over 99% of prefill time, so decoding latency stays roughly flat as episodes lengthen, whereas a single-turn cache grows linearly.
  • A model trained on 16-frame clips transfers to long video streams without growing the context window, because older turns are compressed into memory tokens rather than appended.
  • Pruning about 20% of visual tokens by 3D redundancy does not hurt navigation; it improves success rate by about one point on both R2R-CE and RxR-CE.
  • With the extra-data recipe (oracle trajectories plus DAgger rollouts plus VideoQA, MMC4, and a ScaleVLN subset), the method reaches the best reported RGB-only scores on both benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: because the voxel pruning consumes depth at test time, 'RGB-only' should be read as applying to the model's visual tokens rather than to the full deployed system; an RGB-depth-free variant would be a cleaner demonstration of the stated claim.
  • Editorial: the same sliding-window-plus-pruned-memory pattern could transfer to other streaming embodied decision tasks, such as object-goal navigation or mobile manipulation, wherever consecutive observations overlap heavily in 3D.
  • Editorial: the paper's own limitations section concedes that low-level action generation is sensitive to viewpoint and occlusion and that longer-horizon reasoning remains hard; those caveats temper the deployment-ready reading of the benchmark numbers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes StreamVLN, a Video-LLM-based vision-language-action model for continuous vision-and-language navigation. The model treats each episode as a multi-turn dialogue with interleaved RGB observations, instruction tokens, and action tokens. A fast sliding-window KV cache retains a fixed number of recent dialogue turns, while a slow-updating memory compresses historical visual states using uniform temporal sampling plus a test-time voxel-based spatial pruning step that uses depth to remove redundant tokens. Training combines oracle VLN trajectories, DAgger rollouts, and general VQA/MMC4 data. On R2R-CE and RxR-CE val-unseen the paper reports state-of-the-art results among RGB-only video-LLM methods (e.g., 56.9% SR on R2R-CE with extra data), plus results on ScanQA, HM3D ObjectNav, real-robot demos, latency curves, and ablations over memory size, window size, data composition, and pruning.

Significance. If the claims hold, the paper is significant: it would show that a 7B video-LLM can produce low-latency, long-horizon VLN by reusing KV caches and compressing visual history, addressing a real deployment bottleneck. The empirical comparison against NaVILA and UniNaVid, the DAgger data-efficiency results, and the cross-task transfer to ScanQA are useful contributions. The ablations are informative and the method is evaluated on standard benchmarks. However, the central 'bounded context size and inference cost' claim is not currently supported by the formal description, the 'RGB-only' sensor label is inaccurate because depth is used for pruning, and key hyperparameters and error bars are missing. The core idea is defensible, but the manuscript needs a substantial revision before the claims can be accepted.

major comments (4)
  1. [Section 3.2, Eq. (1); Supplementary Section 3] The central claim of 'bounded context size and inference cost' (Abstract) is not established. Eq. (1) conditions decoding on memory states {M0,...,Mj} with no stated cap on j and no forgetting or merging rule for older memory blocks. Algorithm 1 prunes redundant tokens within a sampled frame set, but it does not limit the number of past windows retained. The supplementary says 'At the end of each session, we uniformly sample a fixed number of frames (8) to store in memory,' which bounds one session but not the number of sessions whose memories are preserved. If all M_j are concatenated, the KV cache and prefill cost grow with the number of past windows, contradicting Figure 5's flat latency over long episodes. If only a fixed recent subset is kept, the paper still owes an explicit eviction/merging rule and should update Eq. (1) accordingly; without this, the long-horizon gains on RxR-CE cannot be cleanly attributed to the slow-updating memory mechanism. The authors should specify the rule and report context length or KV cache size versus episode length.
  2. [Table 1 and Section 3.3] The 'RGB-only' characterization is inaccurate. Table 1 lists StreamVLN as using only a single RGB stream, but Section 3.3's voxel pruning back-projects image patches into a shared 3D space 'using depth information,' and Algorithm 1 explicitly requires a voxel map. The paper does not state the source of depth in the VLN-CE benchmarks. The authors should either mark Depth as used in Table 1 or clearly state that depth is used only to compute a test-time pruning mask and is available from the simulator. The 'state-of-the-art among RGB-only methods' claim must be revised accordingly.
  3. [Algorithm 1 and Table 5] The pruning hyperparameters (stride K and threshold theta) are never reported. Table 5 only reports that pruning removes 'approximately 20%' of tokens and gives small improvements (+1.2 SR on R2R-CE). Without K and theta, or a sensitivity study, the pruning procedure is not reproducible and the claimed benefit is hard to evaluate. The authors should report the actual values and ideally a sweep over them, and clarify whether the same values are used across R2R-CE, RxR-CE, and HM3D.
  4. [Tables 1, 3, 5 and Figure 5] No error bars or multiple seeds are reported. The pruning gains (+1.2 SR / +1.0 SPL on R2R-CE, +1.1 SR / +1.0 SPL on RxR-CE) and several data-ablation differences are small, and a single run cannot distinguish these from run-to-run variance. Please report the mean and standard deviation over at least three seeds for the main results and the key ablations, and state the number of evaluation episodes used for the latency curves in Figure 5.
minor comments (6)
  1. [Section 4.3] The comparison with 'HMAT' is unsupported: the method is not defined, does not appear in Table 1, and has no citation. Please add the baseline and reference, or remove the claim.
  2. [Throughout] The model name 'LLaVA-Video' is spelled inconsistently as 'LLaV A-Video' in multiple places (e.g., Section 3 and Section 4.2); please standardize the spelling.
  3. [Table 4] The memory-context entries '2*196', '4*196', '8*196', and 'all' are not defined. Please clarify what these numbers represent (e.g., number of frames times tokens per frame) and what 'all' means in terms of sessions or frames.
  4. [Figure 5] The caption and text should specify the exact configuration used to produce the latency curves (e.g., number of sessions, memory budget, whether pruning is applied) and define the x-axis units; the text says 'Turns 8-16 and 16-24 has similar latency growth with a fixed memory size,' but the mechanism that fixes memory size is not described anywhere.
  5. [Supplementary Section 3] The statement that applying pruning during training hurts performance is made without supporting data; either add the corresponding ablation or soften the claim.
  6. [Section 3.1 and Supplementary Section 1] The action space (four actions and their symbolic representations) is defined only in the supplementary material; it should be defined in the main text for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all load-bearing claims are evaluated against external benchmarks, not derived from fitted inputs.

full rationale

StreamVLN's central claims are empirical rather than derivational: the method is a video-LLM finetuned on collected VLN trajectories and evaluated on held-out Val-Unseen splits of R2R-CE and RxR-CE and on ScanQA. There is no equation in the paper that is defined in terms of the quantity it is said to predict; the only 'prediction' is navigation success, which is scored by external Habitat simulators against human-annotated paths. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity; the sliding-window size, memory size, and pruning fraction are swept and reported as ablations (Tables 4 and 5), not hidden as free parameters in the headline result. Citations to prior work are used as baselines or architecture backbones (e.g., LLaVA-Video as the base model), not as a self-citation chain that forbids alternatives. The one apparent self-citation with overlapping authorship, ChatScene [44] sharing author J. Pang, is only listed as a comparison method in the ScanQA table and is not load-bearing for any central claim. The DAgger rollouts are self-generated training data, but the resulting policy is tested on unseen scenes, which is standard imitation learning and does not make the evaluation circular. The skeptical concern about whether the context in Eq. (1) actually remains bounded as j grows is a legitimate correctness and reproducibility issue, but it is not circularity: the bounded-context claim is not true by construction of the inputs, nor is it equivalent to a fitted parameter. Since no circular step can be exhibited with a specific reduction from output to input, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a pretrained Video-LLM, accurate depth for pruning, benchmark validity, and several hyperparameters (pruning stride, threshold, memory size, window size) chosen by ablation without sensitivity analysis. No new physical entities are introduced; memory tokens and pruning masks are internal algorithmic constructs.

free parameters (5)
  • Voxel pruning stride K = not reported
    Algorithm 1 stride for temporal voxel grouping; not specified in the paper, affects the pruning mask.
  • Voxel pruning threshold theta = not reported
    Minimum fraction of tokens to retain per frame in Algorithm 1; not specified.
  • Memory context frames M = 8 frames (8*196 tokens)
    Number of uniformly sampled historical frames stored in slow memory; chosen by ablation (Table 4).
  • Sliding window size N = 8 dialogue turns
    Number of recent dialogue turns retained in fast context; chosen by ablation (Table 4).
  • Action token representation = 4 symbolic tokens (up, left, right, stop)
    Chosen by ablation (Table A1) balancing expressiveness and efficiency.
assumptions (6)
  • domain assumption LLaVA-Video 7B provides a strong pretrained video-language representation
    The method is built on this backbone and does not justify its choice beyond citing its capabilities (Sections 3 and 4.2).
  • domain assumption Depth information from Habitat and RealSense is sufficiently accurate for voxel back-projection
    Voxel-based spatial pruning (Section 3.3, Algorithm 1) depends on accurate depth; no sensitivity analysis is given.
  • domain assumption VLN-CE benchmarks measure navigation ability relevant to real-world deployment
    Success rate and SPL on R2R-CE/RxR-CE are used as the primary evidence for the system's value (Section 4.1).
  • domain assumption DAgger rollouts with a shortest-path expert improve generalization
    The paper assumes the expert policy provides useful corrective supervision (Section 3.4).
  • domain assumption Uniform temporal sampling of 8 frames preserves long-horizon visual information
    The slow memory compresses history to 8 frames; ablation shows memory size matters (Table 4), but no analysis of longer horizons.
  • domain assumption Token pruning at test time does not harm, and helps, the model despite train/test mismatch
    Pruning is applied only at test time (supplementary Section 3), yet improves results; the paper provides no explanation for this distribution shift benefit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling." pith.science (2026). https://pith.science/paper/PGIX3K5A

@misc{pith2026250705240,
  author       = {Pith},
  title        = {Pith review of: StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGIX3K5A}},
  note         = {Machine review of arXiv:2507.05240}
}
read the original abstract

Vision-and-Language Navigation (VLN) in real-world settings requires agents to process continuous visual streams and generate actions with low latency grounded in language instructions. While Video-based Large Language Models (Video-LLMs) have driven recent progress, current VLN methods based on Video-LLM often face trade-offs among fine-grained visual understanding, long-term context modeling and computational efficiency. We introduce StreamVLN, a streaming VLN framework that employs a hybrid slow-fast context modeling strategy to support multi-modal reasoning over interleaved vision, language and action inputs. The fast-streaming dialogue context facilitates responsive action generation through a sliding-window of multi-turn dialogues, while the slow-updating memory context compresses historical visual states using a 3D-aware token pruning strategy. With this slow-fast design, StreamVLN achieves real-time dialogues through KV cache reuse, supporting long video streams with bounded context size and inference cost. Experiments on VLN-CE benchmarks show state-of-the-art performance with low latency, ensuring robustness and efficiency in real-world deployment. The project page is: https://streamvln.github.io/.

Figures

Figures reproduced from arXiv: 2507.05240 by the authors.

Figure 1
Figure 1. Framework of StreamVLN. The input consists of a language instruction and a stream of RGB images. Each navigation episode is framed as a multi-turn dialogue, where the agent continually queries for the next actions. To support long-horizon reasoning while maintaining a manageable context size and low latency, we adopt a fixed-size sliding window to retain recent dialogue history. The context in inactive windows is up… view at source ↗
Figure 2
Figure 2. Co-Training Data Recipe of StreamVLN Vision-Language Action Data. We collect navigation-specific training data using the Habitat simulator across multiple pub￾lic VLN datasets. Specifically, we collect 450K samples (video clips) from 60 Matterport3D [25] (MP3D) environments, sourced from R2R [7], R2R-EnvDrop [26] and RxR [8]. To further improve generalization through increased scene diver￾sity, we incorporate an add… view at source ↗
Figure 3
Figure 3. StreamVLN transfers visual reasoning ability to interpreting navigation instructions. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results of StreamVLN in several representative real-world environments. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Impact of KV cache reuse in multiple turns. Sliding window size:8. Memory Window NE↓ OS↑ SR↑ SPL↑ 2*196 8 6.96 48.2 37.3 34.2 4*196 8 6.62 49.1 38.9 35.4 8*196 8 6.05 53.8 45.5 41.6 all 8 6.76 49.5 40.0 36.4 8*196 4 6.31 51.1 41.4 37.5 8*196 2 6.16 52.8 43.7 40.3 [PIT…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Think in Sets for Streaming Video Token Compression

    cs.CV 2026-08 conditional novelty 7.0 of 10

    NovaCov uses a bounded, recency-weighted historical reference bank and a dual-branch submodular coverage objective to select streaming video tokens, outperforming training-free baselines.

  2. NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation

    cs.RO 2026-07 conditional novelty 7.0 of 10

    A new physics-enabled benchmark with 10,000 indoor, outdoor, and indoor-to-outdoor navigation episodes shows current zero-shot agents fail most when crossing the indoor-outdoor boundary, especially on the new PlaceNav task.

  3. Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A feed-forward feature-Gaussian plus one-step geometry-aware pixel-flow simulator converts large image collections into 20K interactive scenes and 10M+ navigation samples that improve zero-shot Habitat and real-robot ...

  4. HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments

    cs.RO 2026-08 conditional novelty 6.0 of 10

    HumanoidVLN provides a physics-based benchmark for VLN on diverse humanoid robots, with 933 episodes and a 20-episode sim-to-real pilot showing strong alignment.

  5. Goal-oriented Navigation Instruction Generation with Tour Video Priors

    cs.CV 2026-08 conditional novelty 6.0 of 10

    VideoNIG tests whether multimodal models can turn tour videos into executable navigation instructions, and a two-stage curriculum with preference optimization improves their spatial reasoning.

  6. Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Unmodified software-agent harnesses, given only a monocular RGB camera and four discrete actions, achieve 68–78% success on zero-shot R2R-CE navigation, rivaling trained and workflow-based systems.

  7. ReferTrack: Referring Then Tracking for Embodied Visual Tracking

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A refer-then-track policy picks the target from indexed detections before planning waypoints, achieving state-of-the-art single-view results on EVT-Bench and approaching multi-camera performance.

  8. Token-Wise Latent Streaming from Slow Reasoners to Fast Planners for Dynamic Vision Language Navigation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Streaming intermediate hidden states from a slow VLM to a fast flow-matching planner, token by token, improves dynamic social VLN success and reduces observation staleness.

  9. Joint On-and-Off Policy Learning for Vision-and-Language Navigation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    JOP-VLN combines DAgger imitation learning with GRPO reinforcement learning, using high-entropy trajectory filtering and error-correction prioritization, achieving 69.9% SR on R2R Val-Unseen.

  10. ABot-N1: Toward a General Visual Language Navigation Foundation Model

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    A slow–fast VLN model that routes five navigation tasks through CoT plus image-space pixel goals reaches SOTA on established and new urban benchmarks, including 77.3% POI arrival.

  11. ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A single 8B backbone unifies spatial perception, decision making, navigation/manipulation, and progress estimation with SSR+ merging, reporting gains on most spatial benchmarks and competitive action/progress results.

  12. Structured Observation Language for Efficient and Generalizable Vision-Language Navigation

    cs.CV 2026-03 reject novelty 6.0 of 10

    SOL-Nav encodes RGB-D observations as grid-organized text and uses a 0.6B text-embedding model with four classification heads to predict navigation action blocks, reporting SOTA/comparable R2R-CE/RxR-CE results with 1...

  13. Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigation

    cs.CV 2025-12 reject novelty 6.0 of 10

    The submitted abstract, the paper body, and the conclusion report incompatible headline results (73.2/75.6, 64.2/67.0, and 62.3/64.5 SR on R2R-CE/RxR-CE), so the claimed state of the art is not internally consistent.

  14. CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Model

    cs.RO 2025-08 unverdicted novelty 6.0 of 10

    By iteratively retraining on automatically generated corrective examples derived from its own wrong paths, CorrectNav reports new state-of-the-art success rates of 65.1% (R2R-CE) and 69.3% (RxR-CE).

  15. A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation

    cs.RO 2026-07 accept novelty 5.5 of 10

    VLN methods show a large sim-to-real gap; a hierarchical system reaches 51% real-world success versus 22% for a monolithic RGB-only system across ten physical scenes.

  16. MemVLN: Episodic and Procedural Memory for Vision-and-Language Navigation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Pyramidal multi-resolution visual memory plus single-token mid-level actions let a 4B–8B VLM navigate continuous indoor environments at 14 FPS with SOTA R2R/RxR success rates.

  17. Existence of small semi-vortex solutions for the cubic nonlinear Schr\"{o}dinger system with Rashba type Spin-Orbit coupling on $\mathbb{R}^2$

    math.AP 2026-04 unverdicted novelty 5.0 of 10

    Small semi-vortex and ground-state solutions of the cubic NLS system with Rashba SOC on R² exist as energy minimizers under small mass, via concentration-compactness.

  18. Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation

    cs.CV 2026-07 conditional novelty 4.0 of 10

    FSC-VLN: a future-prediction training signal improves long-horizon vision-language navigation on R2R, with no future frames needed at test time.

  19. Nav-R1: Reasoning and Navigation in Embodied Scenes

    cs.RO 2025-09 reject novelty 4.0 of 10

    Nav-R1 uses a 110K synthetic CoT dataset, GRPO with three rewards, and a fast-in-slow system to set new SOTA on R2R-CE, RxR-CE, and HM3D-OVON.

Reference graph

Works this paper leans on

49 extracted references · 31 canonical work pages · cited by 19 Pith papers

  1. [1]

    Y . Li, C. Wang, and J. Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, 2024

    work page 2024

  2. [3]

    Zhang, K

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation.Robotics: Science and Systems, 2024

    work page 2024

  3. [4]

    Zhang, K

    J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks.Robotics: Science and Systems, 2025

    work page 2025

  4. [5]

    Cheng, Y

    A.-C. Cheng, Y . Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang. Navila: Legged robot vision-language-action model for navigation.Robotics: Science and Systems, 2025

    work page 2025

  5. [6]

    Zhang, X

    L. Zhang, X. Hao, Q. Xu, Q. Zhang, X. Zhang, P. Wang, J. Zhang, Z. Wang, S. Zhang, and R. Xu. Mapnav: A novel memory representation via annotated semantic maps for vlm-based vision-and-language navigation.arXiv preprint arXiv:2502.13451, 2025

    arXiv 2025

  6. [7]

    Anderson, Q

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference on computer vision and pattern recognition, 2018

    work page 2018

  7. [8]

    A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge. Room-across-room: Multilin- gual vision-and-language navigation with dense spatiotemporal grounding.arXiv preprint arXiv:2010.07954, 2020

    arXiv 2010

  8. [9]

    Chen, P.-L

    S. Chen, P.-L. Guhur, C. Schmid, and I. Laptev. History aware multimodal transformer for vision-and-language navigation.Advances in Neural Information Processing Systems, 2021. 9

    work page 2021

Show all 49 references
  1. [10]

    Chen, P.-L

    S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev. Think global, act local: Dual- scale graph transformer for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16537–16547, 2022

    2022

  2. [11]

    H. Wang, W. Wang, W. Liang, C. Xiong, and J. Shen. Structured scene memory for vision- language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  3. [12]

    G. Zhou, Y . Hong, and Q. Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models.arXiv preprint arXiv:2305.16986, 2023

    2023 arXiv

  4. [13]

    Raychaudhuri, S

    S. Raychaudhuri, S. Wani, S. Patel, U. Jain, and A. X. Chang. Language-aligned waypoint (law) supervision for vision-and-language navigation in continuous environments.arXiv preprint arXiv:2109.15207, 2021

    2021 arXiv

  5. [14]

    Georgakis, K

    G. Georgakis, K. Schmeckpeper, K. Wanchoo, S. Dan, E. Miltsakaki, D. Roth, and K. Daniilidis. Cross-modal map learning for vision and language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

    2022

  6. [15]

    P. Chen, D. Ji, K. Lin, R. Zeng, T. H. Li, M. Tan, and C. Gan. Weakly-supervised multi- granularity map learning for vision-and-language navigation.arXiv preprint arXiv:2210.07506, 2022

    2022 arXiv

  7. [16]

    D. An, H. Wang, W. Wang, Z. Wang, Y . Huang, K. He, and L. Wang. Etpnav: Evolving topological planning for vision-language navigation in continuous environments.arXiv preprint arXiv:2304.03047, 2023

    2023 arXiv

  8. [17]

    H. Wang, W. Liang, L. Van Gool, and W. Wang. Dreamwalker: Mental planning for continuous vision-language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  9. [18]

    Krantz, E

    J. Krantz, E. Wijmans, A. Majundar, D. Batra, and S. Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. InEuropean Conference on Computer Vision (ECCV), 2020

    2020

  10. [19]

    Z. Wang, J. Li, Y . Hong, Y . Wang, Q. Wu, M. Bansal, S. Gould, H. Tan, and Y . Qiao. Scaling data generation in vision-and-language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    2023

  11. [20]

    Z. Wang, X. Li, J. Yang, Y . Liu, J. Hu, M. Jiang, and S. Jiang. Lookahead exploration with neural radiance representation for continuous vision-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  12. [21]

    Y . Long, X. Li, W. Cai, and H. Dong. Discuss before moving: Visual language navigation via multi-expert discussions.arXiv preprint arXiv:2309.11382, 2023

    2023 arXiv

  13. [22]

    P. Chen, X. Sun, H. Zhi, R. Zeng, T. H. Li, G. Liu, M. Tan, and C. Gan.a 2 nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models. arXiv preprint arXiv:2308.07997, 2023

    2023 arXiv

  14. [23]

    Y . Long, W. Cai, H. Wang, G. Zhan, and H. Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.arXiv preprint arXiv:2406.04882, 2024

    2024 arXiv

  15. [24]

    J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024

    2024

  16. [25]

    Chang, A

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y . Zhang. Matterport3D: Learning from RGB-D data in indoor environments.International Conference on 3D Vision (3DV), 2017. 10

    2017

  17. [26]

    H. Tan, L. Yu, and M. Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. InProceedings of NAACL-HLT, 2019

    2019

  18. [27]

    S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Under- sander, W. Galuba, A. Westbury, A. X. Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.arXiv preprint arXiv:2109.08238, 2021

    2021 arXiv

  19. [28]

    S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artificial intelligence and statistics, 2011

    2011

  20. [29]

    Zhang, J

    Y . Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li. Video instruction tuning with synthetic data, 2024. URLhttps://arxiv.org/abs/2410.02713

    2024 arXiv

  21. [30]

    Azuma, T

    D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022

    2022

  22. [31]

    W. Zhu, J. Hessel, A. Awadalla, S. Y . Gadre, J. Dodge, A. Fang, Y . Yu, L. Schmidt, W. Y . Wang, and Y . Choi. Multimodal C4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023

    2023 arXiv

  23. [32]

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  24. [33]

    Krantz, A

    J. Krantz, A. Gokaslan, D. Batra, S. Lee, and O. Maksymets. Waypoint models for instruction- guided navigation in continuous environments. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2021

    2021

  25. [34]

    Y . Hong, Z. Wang, Q. Wu, and S. Gould. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

    2022

  26. [35]

    Krantz and S

    J. Krantz and S. Lee. Sim-2-sim transfer for vision-and-language navigation in continuous environments. InEuropean Conference on Computer Vision, 2022

    2022

  27. [36]

    Z. Wang, X. Li, J. Yang, Y . Liu, and S. Jiang. Gridmm: Grid memory map for vision-and- language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    2023

  28. [37]

    K. Chen, J. K. Chen, J. Chuang, M. V ´azquez, and S. Savarese. Topological planning with transformers for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021

    2021

  29. [38]

    Z. Wang, X. Li, J. Yang, Y . Liu, and S. Jiang. Sim-to-real transfer via 3d feature fields for vision-and-language navigation.arXiv preprint arXiv:2406.09798, 2024

    2024 arXiv

  30. [39]

    Krantz, E

    J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. InEuropean Conference on Computer Vision, 2020

    2020

  31. [40]

    Z. Yu, J. Yu, Y . Cui, D. Tao, and Q. Tian. Deep modular co-attention networks for visual question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019

    2019

  32. [41]

    Z. Zhu, X. Ma, Y . Chen, Z. Deng, S. Huang, and Q. Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921, 2023. 11

    2023

  33. [42]

    Y . Hong, H. Zhen, P. Chen, S. Zheng, Y . Du, Z. Chen, and C. Gan. 3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 2023

    2023

  34. [43]

    Huang, S

    J. Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y . Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    2023 arXiv

  35. [44]

    Huang, Y

    H. Huang, Y . Chen, Z. Wang, R. Huang, R. Xu, T. Wang, L. Liu, X. Cheng, Y . Zhao, J. Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers.arXiv preprint arXiv:2312.08168, 2023

    2023 arXiv

  36. [45]

    R. Fu, J. Liu, X. Chen, Y . Nie, and W. Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  37. [46]

    Zheng, S

    D. Zheng, S. Huang, L. Zhao, Y . Zhong, and L. Wang. Towards learning a generalist model for embodied navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  38. [47]

    S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  39. [48]

    K. Zhou, K. Zheng, C. Pryor, Y . Shen, H. Jin, L. Getoor, and X. E. Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation.arXiv preprint arXiv:2301.13166, 2023

    2023 arXiv

  40. [49]

    W. Cai, S. Huang, G. Cheng, Y . Long, P. Gao, C. Sun, and H. Dong. Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. In2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024

    2024

  41. [50]

    move forward 25 cm,

    P. Wu, Y . Mu, B. Wu, Y . Hou, J. Ma, S. Zhang, and C. Liu. V oronav: V oronoi-based zero-shot object navigation with large language model, 2024. 12 StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling Supplementary Material 1 Navigation Instructio...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.