Pith. sign in

REVIEW 3 major objections 5 minor 10 cited by

VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims a vision-language model trained with supervised fine-tuning and reinforcement fine-tuning can translate egocentric video plus a natural-language instruction directly into low-level navigation actions, reaching…

desk verdict A plausible empirical recipe for turning an LVLM into a continuous navigation policy via GRPO-based RFT, but the paper under-specifies the ground-truth action labels that both training stages depend on, so the headline SOTA claim is not yet verifiable. read the letter →

arxiv 2506.17221 v2 pith:LTG7HVJN submitted 2025-06-20 cs.CV

Zhangyang Qi , Zhixiong Zhang , Yizhou Yu , Jiaqi Wang , Hengshuang Zhao This is my paper · ORCID
classification cs.CV
keywords vision-languagenavigationegocentricvideocontinuouscontrolreinforcementfine-tuninggrouprelativepolicyoptimizationtime-decayedrewardVLN-CElargemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VLN-R1 sets out to show that a single large vision-language model can act as an end-to-end navigation agent, converting an egocentric RGB video stream and a natural-language instruction directly into a sequence of low-level actions (forward, turn left, turn right, stop) in a continuous 3D environment. The authors build a large training corpus of instruction-video-future-action samples, first align the model with supervised fine-tuning, then apply reinforcement fine-tuning with a time-decayed reward that weights earlier correct actions more heavily. On the VLN-CE benchmark, this pushes a 7-billion-parameter model to 30.2% success rate on unseen R2R scenes (from 24.9% after supervised fine-tuning alone), and lets a 2-billion-parameter model reach 25.6% after reinforcement fine-tuning, close to the larger model's supervised result. The broader claim is that reward-driven post-training, previously applied to math and code reasoning, transfers to embodied sequential decision-making, so the paper asks readers to see navigation as a downstream skill of general vision-language models.

What carries the argument

The load-bearing component is the Time-Decayed Reward (TDR): for a predicted block of $n$ future actions, it sums $\gamma^k$ over exactly those positions where the predicted action matches the ground-truth action, so the first action contributes weight $1$, the second $\gamma$, and so on. This turns action prediction into a softly graded, position-aware optimization target rather than an all-or-nothing text match. It is paired with group-relative advantage normalization, which compares each sampled completion against its group, and with a frame-sampling scheme (dense sampling of recent frames plus sparse sampling of older frames) that feeds both immediate context and long-horizon memory into the vision-language model.

What would settle it

A direct test is to rerun the reinforcement stage with the ground-truth action labels in the reward shifted by one time step; if success rates still improve, the reward is teaching the model to mimic label statistics rather than to time its actions correctly, and the navigation claim is weakened.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that low-level continuous navigation can be reduced to a text-generation problem over a compact action vocabulary, and that this behavior is improved by reinforcement fine-tuning with a temporally weighted reward. The model predicts the next six actions as text; supervised fine-tuning first imitates expert demonstrations, and then group-relative policy optimization is run with the Time-Decayed Reward $$R_{\text{nav}}=\sum_{k=0}^{n-1}\gamma^k\,\mathbb{1}(\alpha_{t+k}=\$\alpha$^*_{t+k}),$$ which rewards correct matches but discounts later steps geometrically. The reported numbers on the unseen validation split are the evidence: SFT alone gives 24.9 SR on R2R for the 7B model, and adding RFT raises this to 30.2 SR; on RxR, reinforcement fine-tuning with only 10K cross-domain samples outperforms training on the full dataset. The authors read these results as proof that large vision-language models can drive embodied navigation and that verifiable-reward reinforcement post-training is a data-efficient way to teach them the skill.

Load-bearing premise

The argument assumes that the six-action ground-truth sequences generated from simulator trajectories are correct, unique, and noise-free expert demonstrations, even though the paper does not describe how those action sequences are produced or validate their quality; if the labels are wrong or non-unique, both the supervised loss and the reinforcement reward inherit the error, and the reported gains measure label alignment rather than navigation ability.

Editorial extensions

If this is right

  • Egocentric RGB video plus language is enough for a vision-language model to produce low-level navigation commands, so map, depth, and odometry modules are not required at inference time.
  • Reinforcement fine-tuning with a temporally weighted reward improves navigation success beyond supervised fine-tuning alone, so RL post-training can be a data-efficient second stage for embodied tasks.
  • A 2-billion-parameter model after reinforcement fine-tuning matches a 7-billion-parameter model trained only with supervision, suggesting model scale can be traded for post-training compute.
  • On the RxR benchmark, fine-tuning on 10K cross-domain samples after R2R pretraining outperforms training on the full dataset, indicating fast cross-domain transfer through reinforcement post-training.
  • Predicting a six-step action sequence outperforms single-action prediction, so multi-step temporal supervision is itself a source of navigation performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the time-decayed reward is a generic temporal-credit-assignment trick; the same functional form could be lifted to manipulation or driving benchmarks where early action correctness dominates task success.
  • Because the six-action ground truths are simulator-derived, a natural next test is to vary the expert that produces them; if the reward's gains vanish with a different expert, the method is learning one expert's action style, not navigation generally.
  • The four-action discrete vocabulary is the main limit on fine-grained control; a natural extension, not tested here, is to let the model emit continuous parameters such as distance and angle in the same text format.
  • If the 'small model plus RFT matches large model plus SFT' pattern generalizes, it suggests the efficient frontier for embodied agents may involve spending compute at post-training time rather than at pretraining time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VLN-R1, an end-to-end vision-language navigation framework that feeds egocentric video streams to a large vision-language model (Qwen2-VL) and outputs low-level navigation actions (FORWARD, TURN-LEFT, TURN-RIGHT, STOP). Training proceeds in two stages: supervised fine-tuning (SFT) on a newly constructed VLN-Ego dataset of instruction-video-action samples, followed by reinforcement fine-tuning (RFT) with Group Relative Policy Optimization (GRPO) and a proposed Time-Decayed Reward (TDR) that weights exact matches to ground-truth future actions with an exponential temporal decay. The authors evaluate on VLN-CE R2R and RxR val-unseen, reporting that RFT improves success rate (SR) from 24.9 to 30.2 on R2R for the 7B model and from 21.2 to 25.6 for the 2B model, with similar gains on RxR. The paper claims state-of-the-art performance and that RFT enables smaller models to match larger SFT-only models.

Significance. If the reported results are reliable, the paper makes a useful empirical contribution: it demonstrates that GRPO-based reinforcement fine-tuning improves an LVLM's low-level navigation ability beyond SFT, and that the proposed TDR reward, together with long-short memory sampling, is an effective design choice. The internal ablations (action space size, history sampling, number of GRPO generations, reward type) are coherent and support the main RFT conclusion. However, the significance is tempered by two issues: the 'state-of-the-art' claim is not supported by the baselines actually compared, and the ground-truth action labels on which both SFT and RFT rely are not described or validated. The paper also does not release code, data, or checkpoints, so the central assumption about label correctness is not independently checkable. With the label-generation process clarified and comparisons against recent LVLM-based navigation agents added, the work would be a solid contribution to the VLN-CE community.

major comments (3)
  1. [Section 3.2 and Eqs. (8)-(9)] The construction of the VLN-Ego ground-truth action sequences is not described. Section 3.2 states only that annotations are 'acquired action-by-action' and that 'the ground truth will include the next 6 actions', but does not specify how R2R/RxR reference paths are converted into fixed atomic actions (FORWARD 25 cm, TURN 30 degrees). This is load-bearing because the SFT loss in Eq. (8) and the TDR reward in Eq. (9) both supervise against exactly these labels. In Habitat, a reference waypoint path does not determine a unique low-level action sequence: the conversion depends on the controller, discretization, action ordering, and tie-breaking. If the labels were produced by a shortest-path planner or a hand-tuned controller, the reported RFT gains (e.g., 30.2 vs. 24.9 SR in Table 1) could reflect better imitation of that generator rather than improved navigation ability. Please provide the generation algorithm, discuss non-uniqueness, and report a quality/agreement analysis, or the headline conclusion is uninterpretable.
  2. [Section 4.2, Tables 1 and 2] The claim of state-of-the-art performance is not supported by the presented baselines. All task-specific baselines in Table 1 and the first two tables are from 2021-2022, while the paper's own related work section (§2) identifies Navid and Uni-Navi as the most relevant LVLM-based navigation agents; neither is benchmarked. The abstract and conclusion both state that VLN-R1 'achieves state-of-the-art navigation performance', but no comparison against these recent approaches is provided. Please include results for Navid and Uni-Navi (and, if feasible, other recent VLN-CE methods) on the same val-unseen splits, or revise the claim to 'strong performance among the evaluated methods'.
  3. [Section 4.1 and Section 3.3] The inference-time control loop is unspecified. The paper says the agent outputs a sequence of n=6 future actions and that a navigation episode is successful if the agent stops within a threshold distance, but it does not describe how the re-planning frequency, the STOP action, and episode termination are handled when the model outputs a multi-action sequence. Specifically, does the agent execute all six actions before re-querying the model, and can an intermediate STOP action end the episode? These details directly affect SR and SPL numbers and are needed for reproducibility.
minor comments (5)
  1. [Figure 2] The figure contains a typo: 'Syetem Message' should be 'System Message'.
  2. [Section 4.1] The sentence 'We presents our experimental setup and results here' should read 'We present our experimental setup and results here'.
  3. [Figure 4] In the qualitative example, 'poast' should be 'past'.
  4. [Tables 1 and 2] The table headers use 'Odom.', 'Depth', and 'S.RGB' without defining these abbreviations in the table captions; please add a note explaining the observation modalities.
  5. [Section 3.2] The dataset sizes (630K R2R, 1.2M RxR) would benefit from a brief explanation of how these counts are derived from the 7,189 R2R and 42,023 RxR trajectories, since this affects the reader's understanding of the data engine.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training pipeline optimizes proxy labels, but the central SOTA claim is verified by independent VLN-CE metrics.

full rationale

The paper's derivation chain is an empirical training pipeline, not a derivation of a result from its own definition. VLN-Ego is constructed by converting R2R/RxR reference trajectories into next-6-action annotations via the Habitat simulator, then SFT (Eq. 8) supervises the model on those annotations and RFT (Eq. 9) rewards matches to the same annotations with exponential decay. This does mean that RFT is optimizing a supervised proxy rather than an independently measured environment reward, and the quality/uniqueness of the six-action labels is not validated. However, the paper's central claims, namely state-of-the-art navigation performance and the reported improvements, are evaluated with external VLN-CE metrics (SR, OS, SPL, NE, TL) on Val-Unseen splits, which are defined by stopping within a threshold distance of the goal and are independent of the training-time next-6-action labels. No equation reduces the predicted navigation outcome to the fitted label by construction: Eq. 8 and Eq. 9 both use the same ground-truth action sequence, but the reported SR/SPL numbers come from the simulator's success criterion, not from the reward function. The GRPO/DeepSeek-R1 citations are standard external algorithmic references, not self-citations bearing the central claim; the only self-citation (Ref. [63] includes co-author Jiaqi Wang) concerns hallucination-aware DPO and is not load-bearing. No uniqueness theorem or author-imported ansatz is invoked. The manuscript's own limitation statement (simulated indoor environments only, discrete action space) is an honest scope restriction, not evidence of circularity. Overall, the derivation is self-contained as an empirical method paper, and concerns about label generation quality should be treated as data-correctness risk, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

Everything in the pipeline rests on the Habitat-generated dataset and on treating action-label matching as reward. Hyperparameters for the reward decay, memory sampling, horizon, and group size are selected by ablation and partly unreported. No code, data, or formal proof is provided, so most of the contribution is asserted rather than shipped.

free parameters (5)
  • Action prediction horizon n = 6
    Selected from ablation Table 3a; six future actions performed best and is the default for all main results.
  • Time-decay factor gamma = Not reported
    Exponential decay in Eq. 9 is selected over hard, uniform, and linear rewards in Table 3d, but the actual gamma value is never stated.
  • Long-short memory sampling rates delta1, delta2, M = Not reported
    Eq. 6 defines the sampler, but the concrete rates used in training are not given; Table 3b only compares qualitative strategies.
  • GRPO group size G = 8
    Ablation Table 3c shows k=8 as convergence point; used for all main results.
  • KL penalty beta = 0.04
    Reported in Section 4.1; controls deviation from reference policy in Eq. 4.
assumptions (4)
  • domain assumption Habitat simulation with Matterport3D scenes is a valid proxy for indoor vision-language navigation.
    The entire dataset and evaluation rest on this; no real-world validation is provided (Section 3.2, Section 5).
  • domain assumption The VLN-Ego ground-truth action sequences are correct expert labels.
    Both SFT loss (Eq. 8) and TDR reward (Eq. 9) assume these six-action sequences are reliable and unambiguous; the generation procedure is not described.
  • ad hoc to paper Matching predicted discrete actions to ground-truth actions is a meaningful proxy for navigation success.
    TDR reward (Eq. 9) replaces the actual benchmark success criterion (stopping within a threshold distance) with action-label agreement, an assumption introduced by this paper.
  • standard math Standard RLVR/GRPO background from DeepSeekMath and DeepSeek-R1.
    Section 3.1 relies on GRPO and KL penalty; Eq. 5 misstates the KL formula but the intended background is standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning." pith.science (2026). https://pith.science/paper/LTG7HVJN

@misc{pith2026250617221,
  author       = {Pith},
  title        = {Pith review of: VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTG7HVJN}},
  note         = {Machine review of arXiv:2506.17221}
}
read the original abstract

Vision-Language Navigation (VLN) is a core challenge in embodied AI, requiring agents to navigate real-world environments using natural language instructions. Current language model-based navigation systems operate on discrete topological graphs, limiting path planning to predefined node connections. We propose VLN-R1, an end-to-end framework that leverages Large Vision-Language Models (LVLM) to directly translate egocentric video streams into continuous navigation actions, adopting GRPO-based training inspired by DeepSeek-R1. To enable effective training, we first construct the VLN-Ego dataset using a 3D simulator, Habitat, and propose Long-Short Memory Sampling to balance historical and current observations. While large language models can supervise complete textual instructions, they lack fine-grained action-level control. Our framework employs a two-stage training approach: a) Supervised fine-tuning (SFT) to align the model's action sequence text predictions with expert demonstrations, followed by b) Reinforcement fine-tuning (RFT) enhanced with a Time-Decayed Reward (TDR) mechanism that strategically weights multi-step future actions. Experimental results show VLN-R1 achieves strong performance on VLN-CE benchmark. VLN-R1 proves LVLMs can drive embodied navigation and enhance task-specific reasoning through data-efficient, reward-driven post-training.

Figures

Figures reproduced from arXiv: 2506.17221 by the authors.

Figure 1
Figure 1. Overview of VLN-R1. Previous LLM/LVLM models were based on discrete positions and used a third-person perspective for path planning. In contrast, VLN-R1 directly explores in a continuous environment using first-person perspective videos. We train the LVLM using Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT). and Uni-Navi constitute important advances [60, 59], their persistent reliance on modular v… view at source ↗
Figure 2
Figure 2. Data Engine: VLN-Ego. We created a dataset named VLN-Ego for LVLM-based navigation using Habitat’s virtual simulation engine. Its textual annotations primarily consist of three parts: Instruction Part, Vision Part, and Action Part. provided as input. The second part is the vision part, which contains the first-person visual informa￾tion, including all the historical frames <History Memory> and the <Current Observati… view at source ↗
Figure 3
Figure 3. Model Architecture of VLN-R1. VLN-R1 employs a Long-Short Memory approach for processing visual inputs. The training consists of two stages. During the supervised fine-tuning (SFT) stage, we only supervise the output text. In the reinforcement fine-tuning (RFT) stage, we implement supervision using a designed Time-Decayed Reward (TDR) mechanism. Prediction and loss. The supervised fine-tuning (SFT) phase aligns the … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative Results of VLN-R1. As shown, VLN-R1 accepts egocentric video input and navigates through a continuous environment to ultimately reach the target location. (a) Action Space Action Space Variant R2R Val SR↑ OS↑ Single Discrete Action 15.1 33.6 4-Discrete-Acti…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Steadily moving semi-infinite fracture in plane poroelasticity

    physics.geo-ph 2026-04 unverdicted novelty 7.0 of 10

    XEmbodied achieves SOTA on 18 embodied VQA benchmarks by fusing 3D geometric tokens and distilled physical cues into a 30B VLM with progressive curriculum training.

  2. Joint On-and-Off Policy Learning for Vision-and-Language Navigation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    JOP-VLN combines DAgger imitation learning with GRPO reinforcement learning, using high-entropy trajectory filtering and error-correction prioritization, achieving 69.9% SR on R2R Val-Unseen.

  3. IRPO: Boosting Image Restoration via Post-training GRPO

    cs.CV 2025-11 conditional novelty 6.0 of 10

    GRPO post-training on the worst 30% of samples with a mixed fidelity/perceptual reward improves AdaIR by 0.83 dB in-domain and 3.43 dB on out-of-domain benchmarks.

  4. DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigation

    cs.RO 2025-09 conditional novelty 6.0 of 10

    DreamNav achieves new zero-shot SOTA on VLN-CE with an egocentric-only pipeline that generates candidate trajectories, imagines their futures, and selects the best by language alignment.

  5. A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation

    cs.RO 2026-07 accept novelty 5.5 of 10

    VLN methods show a large sim-to-real gap; a hierarchical system reaches 51% real-world success versus 22% for a monolithic RGB-only system across ten physical scenes.

  6. RecoverFly: A Failure-Aware Reinforcement Learning Post-Training Framework for Aerial Vision-Language Navigation

    cs.CV 2026-08 conditional novelty 5.0 of 10

    RecoverFly uses failure replay, a scene curriculum, and KL regularization in a PPO-based post-training stage to raise UAV vision-language navigation success rates across seen and unseen environments.

  7. Pseudo Depth Meets Gaussian: A Feed-forward RGB SLAM Baseline

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A feed-forward recurrent pose predictor replaces test-time optimization in 3D Gaussian SLAM, claiming SplaTAM-level accuracy with over 90% less tracking time on Replica and TUM-RGBD.

  8. Towards Dual-Brain Minimal Sufficient Representation for Vision-Language Navigation

    cs.CV 2026-07 reject novelty 4.0 of 10

    A CP-decomposed, instruction-conditioned latent bottleneck (CompactNav) improves VLN-CE success rate by about 2% over prior state of the art on two benchmarks.

  9. Nav-R1: Reasoning and Navigation in Embodied Scenes

    cs.RO 2025-09 reject novelty 4.0 of 10

    Nav-R1 uses a 110K synthetic CoT dataset, GRPO with three rewards, and a fast-in-slow system to set new SOTA on R2R-CE, RxR-CE, and HM3D-OVON.

  10. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

67 extracted references · 35 canonical work pages · cited by 10 Pith papers

  1. [1]

    On evaluation of embodied navigation agents.arXiv:1807.06757, 2018

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents.arXiv:1807.06757, 2018. 1

    arXiv 2018

  2. [2]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InCVPR, 2018. 1, 2, 3, 7, 8

    work page 2018

  3. [3]

    Internlm2 technical report.arXiv:2403.17297, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report.arXiv:2403.17297, 2024. 3

    arXiv 2024

  4. [4]

    Matterport3D: Learning from RGB-D data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments. In3DV, 2017. 4

    work page 2017

  5. [5]

    Mapgpt: Map-guided prompting for unified vision-and-language navigation

    Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K Wong. Mapgpt: Map-guided prompting for unified vision-and-language navigation. InACL, 2024. 1

    work page 2024

  6. [6]

    Topological planning with transformers for vision-and-language navigation

    Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. Topological planning with transformers for vision-and-language navigation. InCVPR, 2021. 3, 8

    work page 2021

  7. [7]

    Weakly-supervised multi-granularity map learning for vision-and-language navigation

    Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas H Li, Mingkui Tan, and Chuang Gan. Weakly-supervised multi-granularity map learning for vision-and-language navigation. In NeurIPS, 2022. 3, 8

    work page 2022

  8. [8]

    Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models

    Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng, Thomas H Li, Gaowen Liu, Mingkui Tan, and Chuang Gan. Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models. InNeurIPS, 2023. 1, 3, 8

    work page 2023

Show all 67 references
  1. [9]

    History aware multimodal transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. InNeurIPS, 2021. 3

    2021

  2. [10]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. InECCV, 2020. 3

    2020

  3. [11]

    Clip-nav: Using clip for zero-shot vision-and-language navigation

    Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu, Jesse Thomason, and Gaurav S Sukhatme. Clip-nav: Using clip for zero-shot vision-and-language navigation. arXiv:2211.16649, 2022. 1, 3

    2022 arXiv

  4. [12]

    Cross-modal map learning for vision and language navigation

    Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis. Cross-modal map learning for vision and language navigation. In CVPR, 2022. 3, 8

    2022

  5. [13]

    Airbert: In-domain pretraining for vision-and-language navigation

    Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In-domain pretraining for vision-and-language navigation. InICCV, 2021. 3

    2021

  6. [14]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv:2501.12948, 2025. 2, 3, 4, 7

    2025 arXiv

  7. [15]

    Towards learning a generic agent for vision-and-language navigation via pre-training

    Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-language navigation via pre-training. InCVPR, 2020. 3

    2020

  8. [16]

    Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation

    Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. InCVPR, 2022. 1, 3, 4

    2022

  9. [17]

    Visual language maps for robot navigation

    Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. InICRA, 2023. 1, 3

    2023

  10. [18]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report.arXiv:2409.12186, 2024. 3

    2024 arXiv

  11. [19]

    Sasra: Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuous environments.arXiv:2108.11945, 2021

    Muhammad Zubair Irshad, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, Supun Samarasekera, and Rakesh Kumar. Sasra: Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuous environments.arXiv:2108.11945, 2021. 3 10

    2021 arXiv

  12. [20]

    Preference optimization for reasoning with pseudo feedback.arXiv:2411.16345, 2024

    Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F Chen, Shafiq Joty, and Furu Wei. Preference optimization for reasoning with pseudo feedback.arXiv:2411.16345, 2024. 3

    2024 arXiv

  13. [21]

    Waypoint models for instruction-guided navigation in continuous environments

    Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction-guided navigation in continuous environments. InICCV, 2021. 1, 3, 4, 5

    2021

  14. [22]

    Sim-2-sim transfer for vision-and-language navigation in continu- ous environments

    Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision-and-language navigation in continu- ous environments. InECCV, 2022. 1, 3

    2022

  15. [23]

    Beyond the nav-graph: Vision and language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majundar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. InECCV, 2020. 3, 4, 7, 8

    2020

  16. [24]

    Room-across- room: Multilingual vision-and-language navigation with dense spatiotemporal grounding

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across- room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In EMNLP, 2020. 2, 3, 4, 7, 8

    2020

  17. [25]

    Tulu 3: Pushing frontiers in open language model post-training.arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv:2411.15124, 2024. 3

    2024 arXiv

  18. [26]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv:2408.03326, 2024. 3

    2024 arXiv

  19. [27]

    Visualbert: A simple and performant baseline for vision and language.arXiv:1908.03557, 2019

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language.arXiv:1908.03557, 2019. 3

    1908 arXiv

  20. [28]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. InECCV, 2020. 3

    2020

  21. [29]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv:2412.19437, 2024. 3

    2024 arXiv

  22. [30]

    Skywork-Reward: Bag of tricks for reward modeling in llms

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-Reward: Bag of tricks for reward modeling in llms. arXiv:2410.18451, 2024. 3

    2024 arXiv

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 3

    2023

  24. [32]

    Aligning cyber space with physical world: A comprehensive survey on embodied ai

    Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv:2407.06886, 2024. 1

    2024 arXiv

  25. [33]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. InCoRL, 2024. 1, 3

    2024

  26. [34]

    Discuss before moving: Visual language navigation via multi-expert discussions.arXiv:2309.11382, 2023

    Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. Discuss before moving: Visual language navigation via multi-expert discussions.arXiv:2309.11382, 2023. 1, 3

    2023 arXiv

  27. [35]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, and al. Openeqa: Embodied question answering in the era of foundation models. InCVPR, 2024. 2

    2024

  28. [36]

    Improving vision-and-language navigation with image-text pairs from the web

    Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-language navigation with image-text pairs from the web. InECCV, 2020. 3

    2020

  29. [37]

    Core challenges of social robot navigation: A survey.ACM Transactions on Human-Robot Interaction, 2023

    Christoforos Mavrogiannis, Francesca Baldini, Allan Wang, Dapeng Zhao, Pete Trautman, Aaron Steinfeld, and Jean Oh. Core challenges of social robot navigation: A survey.ACM Transactions on Human-Robot Interaction, 2023. 1

    2023

  30. [38]

    Wmnav: Integrating vision-language models into world models for object goal navigation.arXiv:2503.02247, 2025

    Dujun Nie, Xianda Guo, Yiqun Duan, Ruijun Zhang, and Long Chen. Wmnav: Integrating vision-language models into world models for object goal navigation.arXiv:2503.02247, 2025. 4

    2025 arXiv

  31. [39]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. 3 11

    2024

  32. [40]

    Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv:2310.13724, 2023

    Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv:2310.13724, 2023. 4

    arXiv

  33. [41]

    Hop: History- and-order aware pre-training for vision-and-language navigation

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: History- and-order aware pre-training for vision-and-language navigation. InCVPR, 2022. 3

    2022

  34. [42]

    March in chat: Interactive prompting for remote embodied referring expression

    Yanyuan Qiao, Yuankai Qi, Zheng Yu, Jing Liu, and Qi Wu. March in chat: Interactive prompting for remote embodied referring expression. InICCV, 2023. 1, 3

    2023

  35. [43]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InNeurIPS, 2023. 4

    2023

  36. [44]

    Zero: Memory optimiza- tions toward training trillion parameter models.arXiv:1910.02054, 2019

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models.arXiv:1910.02054, 2019. 7

    1910 arXiv

  37. [45]

    Language- aligned waypoint (law) supervision for vision-and-language navigation in continuous environ- ments.arXiv:2109.15207, 2021

    Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel X Chang. Language- aligned waypoint (law) supervision for vision-and-language navigation in continuous environ- ments.arXiv:2109.15207, 2021. 3, 8

    2021 arXiv

  38. [46]

    Habitat: A Platform for Embodied AI Research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. InICCV, 2019. 4

    2019

  39. [47]

    Proximal policy optimization algorithms.arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv:1707.06347, 2017. 4

    2017 arXiv

  40. [48]

    Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action

    Dhruv Shah, Bła ˙zej Osi ´nski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. InCoRL, 2023. 1, 3

    2023

  41. [49]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv:2402.03300, 2024. 2, 3, 4, 6, 7

    2024 arXiv

  42. [50]

    Vl-bert: Pre-training of generic visual-linguistic representations.arXiv:1908.08530, 2019

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations.arXiv:1908.08530, 2019. 3

    1908 arXiv

  43. [52]

    Aligning large multimodal models with factually augmented rlhf.arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv:2309.14525, 2023. 3

    2023 arXiv

  44. [53]

    Lxmert: Learning cross-modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. InEMNLP, 2019. 3

    2019

  45. [54]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv:2501.12599, 2025. 3

    2025 arXiv

  46. [55]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv:2409.12191, 2024. 2, 3, 7

    2024 arXiv

  47. [56]

    Cross-modal semantic alignment pre-training for vision-and-language navigation

    Siying Wu, Xueyang Fu, Feng Wu, and Zheng-Jun Zha. Cross-modal semantic alignment pre-training for vision-and-language navigation. InACMMM, 2022. 3

    2022

  48. [57]

    RlHF-V: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RlHF-V: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. InCVPR, 2024. 3

    2024

  49. [58]

    RLAIF-V: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.arXiv:2405.17220, 2024

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. RLAIF-V: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.arXiv:2405.17220, 2024. 3

    2024

  50. [59]

    Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks

    Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks. InRSS, 2025. 2, 3 12

    2025

  51. [60]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation. InRSS, 2024. 2, 3, 5

    2024

  52. [61]

    Codedpo: Aligning code models with self generated and verified source code.arXiv:2410.05605,

    Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, and Zhi Jin. Codedpo: Aligning code models with self generated and verified source code.arXiv:2410.05605,

    arXiv

  53. [62]

    o1-coder: an o1 replication for coding.arXiv:2412.00154, 2024

    Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. o1-coder: an o1 replication for coding.arXiv:2412.00154, 2024. 3

    2024 arXiv

  54. [63]

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv:2311.16839, 2023. 3

    2023 arXiv

  55. [64]

    Towards learning a generalist model for embodied navigation

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for embodied navigation. InCVPR, 2024. 1, 3

    2024

  56. [65]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. InECCV, 2025. 1, 3

    2025

  57. [66]

    Navgpt: Explicit reasoning in vision-and-language navigation with large language models.arXiv:2305.16986, 2023

    Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models.arXiv:2305.16986, 2023. 1, 3

    2023 arXiv

  58. [67]

    Aligning modalities in vision large language models via preference fine-tuning.arXiv:2402.11411, 2024

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning.arXiv:2402.11411, 2024. 3

    2024 arXiv

  59. [68]

    Deep learning for embodied vision navigation: A survey.arXiv:2108.04097, 2021

    Fengda Zhu, Yi Zhu, Vincent Lee, Xiaodan Liang, and Xiaojun Chang. Deep learning for embodied vision navigation: A survey.arXiv:2108.04097, 2021. 1 13

    2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.