Pith. sign in

REVIEW 5 major objections 5 minor 11 cited by

A unified embodied model with CoT cold-start and GRPO RL sets new state-of-the-art results on continuous vision-language navigation and open-vocabulary object navigation while preserving dialogue, reasoning, and planning performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:27 UTC pith:P36NT5Q6

load-bearing objection The simulation results are worth a look, but the real-world SR values above 1.0 alone sink the paper as submitted. the 5 major comments →

arxiv 2509.10884 v1 pith:P36NT5Q6 submitted 2025-09-13 cs.RO cs.CV

Nav-R1: Reasoning and Navigation in Embodied Scenes

Qingxiang Liu , Ting Huang , Zeyu Zhang , Hao Tang This is my paper
classification cs.RO cs.CV
keywords embodied navigationchain-of-thoughtGRPOvision-language-action modelVLN-CEobject navigationdual-system reasoning3D scene understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Nav-R1 is an embodied foundation model that tries to show that reasoning and navigation can be unified in one policy. The authors claim that a cold-start on 110K synthetic chain-of-thought navigation traces, followed by GRPO reinforcement learning with rewards for output format, semantic understanding, and path/endpoint fidelity, transfers the reasoning-RL recipe to 3D action. They report consistent gains over prior methods, e.g. 72.5% SR and 68.8% SPL on R2R-CE Val-Unseen and 42.2% SR on HM3D-OVON Val-Unseen, with dialogue and reasoning scores close to the 3D-R1 base. If correct, this means the CoT-plus-RL pipeline that improved LLM reasoning also works for embodied decision-making, and that a slow-fast architecture can keep long-horizon semantics without sacrificing control latency.

Core claim

The paper's central claim is that the gap between a 3D vision-language model that can talk about scenes and one that can navigate them can be closed by a two-stage procedure: supervised fine-tuning on Nav-CoT-110K, a corpus of 110K step-by-step reasoning-action traces generated by Gemini 2.5 Pro from egocentric RGB-D, instructions, and action options; then GRPO RL with three complementary rewards. On top of this, a Fast-in-Slow paradigm runs a slow semantic reasoner at low frequency and a fast reactive controller at high frequency, coordinated asynchronously at a 1:3 ratio. The authors claim this yields an average improvement of 8% over strong baselines across dialogue, reasoning, planning,

What carries the argument

The load-bearing mechanism is the combination of (1) the Nav-CoT-110K dataset — synthetic chains of thought produced by prompting Gemini 2.5 Pro with egocentric views, instructions, and action sets, filtered by rule checks and trajectory verification — which provides the cold-start supervision; (2) GRPO policy optimization with three rewards: a format reward for <think>/<action> structure, an understanding reward combining exact-match answer correctness with CLIPScore image-answer alignment, and a navigation reward combining a path-fidelity exponential term with an endpoint-distance exponential term; and (3) the Fast-in-Slow dual-system design, where the slow system emits latent semantic fea

Load-bearing premise

The synthetic chain-of-thought traces in Nav-CoT-110K are assumed to be correct, diverse, and aligned with feasible actions; the paper provides no human evaluation, diversity metric, or hallucination analysis, and the whole cold-start and RL pipeline is built on them.

What would settle it

Run Nav-R1's cold-start with the same action labels but corrupted or shuffled reasoning traces (e.g., random <think> text), keep the rest of the pipeline identical, and measure Val-Unseen SR on R2R-CE. If success stays near 72.5%, the reasoning traces are not the source of the improvement; if it collapses, the CoT content is load-bearing. A direct head-to-head reproduction using only open-source generators instead of Gemini would also test the dependence on the specific VLM.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If Nav-R1's results hold, a single policy can handle instruction-following VLN, open-vocabulary object search, spatial QA, dialogue, and planning without task-specific modules.
  • The 8% average improvement over strong baselines implies CoT cold-start plus GRPO is an effective recipe for embodied action, not only for text-based LLM reasoning.
  • The Fast-in-Slow asynchronous design (1:3 ratio) shows that decoupling semantic memory from reactive control can preserve both long-horizon coherence and low-latency response.
  • Removing any of the three rewards degrades performance (ablation on HM3D-OVON), so format, understanding, and navigation rewards are complementary rather than redundant.
  • Cloud-assisted inference keeps per-frame latency at about 95 ms, and the model outperforms baselines on a real robot across three indoor scenes, supporting applicability under limited onboard compute with a server link.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the Nav-CoT traces are generated by Gemini on benchmark instructions, the method is likely to inherit any biases or hallucinations of that generator; a human-verified or adversarially corrupted CoT ablation would show how much of the gain is due to the reasoning traces themselves rather than the RL rewards.
  • The Fast-in-Slow idea suggests a general architecture for embodied agents: keep a slow semantic world model and a fast policy, with the slow features acting as a prior for the fast loop. This could be tested on manipulation or legged locomotion, where control frequency matters even more.
  • The dataset is built from existing benchmarks (R2R, RxR, SOON, OVON), so the value of the CoT engine likely depends on instruction diversity; extending it with LLM-generated instructions or cross-domain goals might further improve generalization.
  • The reported real-world deployment relies on cloud inference, so the claim about 'limited onboard resources' is about streaming latency rather than on-device inference; an on-device distillation of the fast system would be a natural next test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. Nav-R1 is an embodied vision-language model that combines a synthetic chain-of-thought dataset (Nav-CoT-110K), cold-start supervised fine-tuning from 3D-R1, GRPO-based reinforcement learning with format/understanding/navigation rewards, and a Fast-in-Slow dual-system architecture for continuous navigation. The paper reports state-of-the-art results on R2R-CE and RxR-CE, results on HM3D-OVON, comparable dialogue/reasoning/planning performance on 3D-LLM and SQA3D, and real-world experiments on a WHEELTEC R550 robot with a Jetson Orin Nano. The central claim is that Nav-R1 consistently outperforms strong baselines across embodied navigation, reasoning, dialogue, and planning, with over 8% average improvement and validated robustness under limited onboard resources.

Significance. The framework is topical and the individual components are concrete: the Nav-CoT-110K dataset construction, the three-reward GRPO formulation, and the Fast-in-Slow reasoning scheme are all substantive engineering contributions, and the authors provide code and a project website. If the results were reproducible, the paper would represent a useful step toward unified embodied navigation and reasoning. However, as written, the evidence does not support the headline claims. The real-world success-rate table contains impossible values, the reported parameter count is self-contradictory, the main VLN comparison is confounded by depth input, and the 'limited onboard resources' claim is contradicted by cloud-only deployment. I find no circularity problem in the core pipeline: rewards are computed against external ground truth and the benchmark evaluations are held out. The limitations section is honest, but the numerical inconsistencies are load-bearing and cannot be overlooked.

major comments (5)
  1. [Table V, §V.A.b] Success rate is defined as a ratio metric; values of 1.03, 1.12, and 1.02 in the Meeting Room, Lounge, and Corridor rows are mathematically impossible. This invalidates the quantitative real-world comparison and, with it, the abstract/conclusion claim that deployment 'validates its robustness under limited onboard resources.' The authors must report raw episode counts, success counts, and a properly bounded metric; without this, the real-world evidence is not trustworthy.
  2. [§V.B.d] The parameter-efficiency paragraph is internally contradictory. It first states that LoRA introduces ~12M trainable parameters, then says 'In total, about 142M parameters are updated, reducing trainable parameters by ~98% compared to full fine-tuning.' If the full model has roughly 700M parameters, 12M is ~98% fewer, while 142M is only ~20% fewer. The authors need to clarify exactly which parameters are trained in each stage and recompute the reduction. This directly affects the parameter-efficiency and edge-deployment narrative.
  3. [Table III, §V.C.d] Nav-R1 uses depth input (marked 'depth: ✓'), while the RGB-only block (NaVid, Uni-NaVid, NaVILA, VLN-R1, OctoNav, StreamVLN, CorrectNav) does not. The caption claims superiority over methods without waypoint predictors 'even when those methods leverage additional inputs such as depth, panoramic views, and odometry,' but no depth-ablated Nav-R1 is reported. Without an RGB-only variant or another controlled comparison, the observed gains cannot be attributed to CoT/GRPO/Fast-in-Slow rather than to the extra sensor modality. This confound is load-bearing for the 'consistently outperforms' claim.
  4. [§V.D.a, Table VI, Appendix IX] The real-world system does not run Nav-R1 on the Jetson Orin Nano; the model is deployed on a cloud server and only navigation commands are returned to the robot. Table VI reports no on-board latency for Nav-R1. Appendix IX itself concedes that 'current deployment depends on cloud inference, limiting real-time scalability on edge devices.' The abstract's claim of robustness 'under limited onboard resources' is therefore not demonstrated. Either report on-device execution or restate the claim as cloud-assisted real-world navigation.
  5. [§III.B, §IV.B.a, Appendix IX] Nav-CoT-110K is generated by prompting Gemini 2.5 Pro and filtered only by rule-based checks and trajectory verification. No human evaluation of the CoT quality, no diversity statistics beyond scene counts, and no hallucination-rate analysis are provided. Because the entire cold-start and RL pipeline is built on these synthetic traces, any systematic errors in the Gemini-generated reasoning will propagate into the final model. The paper's own limitations section says the data 'does not fully capture real-world complexity'; the characterization of the dataset as 'high-quality' is thus not sufficiently supported.
minor comments (5)
  1. [Abstract, §V.C] The 'over 8% average improvement' claim is not tied to a defined set of tasks or an aggregation procedure. Please specify which tables/tasks are averaged and report the aggregate metric.
  2. [Table IV] Several metrics are slightly below the 3D-R1 baseline (e.g., B-4 39.34 vs 39.45, R 55.12 vs 55.34 on dialogue; B-4 23.20 vs 23.56 and R 59.50 vs 60.02 on reasoning). The text says performance is 'on par,' but this should be quantified with confidence intervals or significance tests rather than asserted.
  3. [Fig. 1] Minor typo: 'bathtab' should be 'bathtub.'
  4. [Table I] For RxR-CE, the instruction count is listed as '-' while other datasets have counts. Please clarify whether the count is unavailable, not used, or intentionally omitted.
  5. [Table VI, §VI] Nav-R1's on-board inference entry is '-'. If the model cannot run on the edge platform, make this explicit rather than leaving the cell empty, and discuss the consequence for the 'real-time' and 'limited resources' claims.

Circularity Check

0 steps flagged

No significant circularity: benchmark rewards and evaluations are external, and the central claims do not reduce to the paper's own inputs; minor self-citation (initialization from 3D-R1) is not load-bearing. The Table V SR>1 anomaly is a correctness inconsistency, not circularity.

full rationale

The derivation chain is largely self-contained against external benchmarks. Nav-CoT-110K is synthesized from public benchmark instructions and egocentric views, then rule-filtered; the GRPO rewards in Eqs. (1)-(7) are computed against ground-truth answers, trajectories, and endpoints from those external datasets, and the R2R-CE/RxR-CE/HM3D-OVON/3D-LLM/SQA3D evaluations (Tables II-IV) use held-out benchmark splits. No equation defines its result in terms of an input of the same derivation. The only authorship self-reference is initialization from the authors' prior 3D-R1 model ([23]) and related-work citations ([24],[25]); this is normal building on prior work and does not carry the central navigation/reasoning claim, which is supported by external comparisons. The paper's own limitation section (Sec. IX) appropriately notes that Nav-CoT is synthesized from existing benchmarks and that cloud inference limits edge scalability. One in-scope anomaly must be flagged as a correctness risk, not circularity: Table V reports Nav-R1 SR values of 1.03, 1.12, and 1.02, while Section V-A.b defines SR under standard metrics [4] as a success-rate ratio bounded by 1.0. These real-world numbers are internally inconsistent with the metric definition, so the real-world deployment claim in the abstract/conclusion is not supported by that table; however, this is not a circularity of the derivation chain. Overall no load-bearing circular step was found.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

All parameters are standard ML hyperparameters; the central claim depends on the synthetic dataset and the reward design, neither of which introduces new physical entities. The key unstated assumption is the reliability of Gemini-generated CoT.

free parameters (4)
  • k (decay coefficient in navigation reward) = not stated
    Introduced in Eq. (5)-(6) to scale path and endpoint penalties; value chosen by hand, not reported.
  • n (slow-fast frequency ratio) = 3
    Chosen empirically in Section IV.E.c as the best balance between semantic fidelity and responsiveness.
  • beta (KL penalty) = 0.02
    Ablated in Table IX; selected by highest RxR-CE Val-Unseen SR.
  • LoRA rank and alpha = r=6, alpha=8
    Chosen to limit trainable parameters (Section V.B.d).
axioms (4)
  • domain assumption Gemini 2.5 Pro generates reliable embodied CoT rationales
    The entire Nav-CoT-110K dataset is synthesized by this proprietary VLM; if its reasoning is hallucinated or inconsistent, the cold-start and RL training inherit the errors (Section III.B).
  • domain assumption Ground-truth trajectories in R2R/RxR-CE are feasible and correct in continuous environments
    Used as T_hat in the navigation reward (Eq. 5); the paper does not verify that expert paths are collision-free under the simulator dynamics.
  • domain assumption CLIPScore between current image and generated answer is a valid proxy for semantic correctness
    R_sem is defined as CLIPScore(I, answer) in Eq. (3); this assumes CLIP's text-image alignment reflects grounded understanding in 3D scenes.
  • domain assumption 3D-R1 provides a strong base for embodied reasoning
    Initialization from the authors' prior 3D-R1 model; the paper does not ablate the choice of base model.

pith-pipeline@v1.3.0-alltime-deepseek · 19744 in / 12611 out tokens · 143090 ms · 2026-08-04T17:27:15.664177+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Nav-R1: Reasoning and Navigation in Embodied Scenes." pith.science (2026). https://pith.science/paper/P36NT5Q6

@misc{pith2026250910884,
  author       = {Pith},
  title        = {Pith review of: Nav-R1: Reasoning and Navigation in Embodied Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P36NT5Q6}},
  note         = {Machine review of arXiv:2509.10884}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Embodied navigation requires agents to integrate perception, reasoning, and action for robust interaction in complex 3D environments. Existing approaches often suffer from incoherent and unstable reasoning traces that hinder generalization across diverse environments, and difficulty balancing long-horizon semantic reasoning with low-latency control for real-time navigation. To address these challenges, we propose Nav-R1, an embodied foundation model that unifies reasoning in embodied environments. We first construct Nav-CoT-110K, a large-scale dataset of step-by-step Chains-of-Thought (CoT) for embodied tasks, which enables cold-start initialization with structured reasoning. Building on this foundation, we design a GRPO-based reinforcement learning framework with three complementary rewards: format, understanding, and navigation, to improve structural adherence, semantic grounding, and path fidelity. Furthermore, we introduce a Fast-in-Slow reasoning paradigm, decoupling deliberate semantic reasoning from low-latency reactive control for efficient yet coherent navigation. Extensive evaluations on embodied AI benchmarks demonstrate that Nav-R1 consistently outperforms strong baselines, with over 8% average improvement in reasoning and navigation performance. Real-world deployment on a mobile robot further validates its robustness under limited onboard resources. Code: https://github.com/AIGeeksGroup/Nav-R1. Website: https://aigeeksgroup.github.io/Nav-R1.

Figures

Figures reproduced from arXiv: 2509.10884 by Hao Tang, Qingxiang Liu, Ting Huang, Zeyu Zhang.

Figure 1
Figure 1. Figure 1: Nav-R1 is an embodied foundation model that integrates dialogue, reasoning, planning, and navigation capabilities to enable intelligent interaction and task execution in 3D environments. Abstract— Embodied navigation requires agents to integrate perception, reasoning, and action for robust interaction in complex 3D environments. Existing approaches often suffer from incoherent and unstable reasoning traces… view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of Nav-R1. Nav-R1 designs a Fast-in-Slow reasoning paradigm that processes egocentric RGB-D views, scene point cloud, and language instructions. The slow system performs long-horizon semantic reasoning, while the fast system executes real-time navigation, enabling coherent reasoning and low-latency control in embodied environments. the connective tissue between perception, grounding, and actio… view at source ↗
Figure 3
Figure 3. Figure 3: CoT Data Engine. We construct the Nav-CoT dataset by defining navigation instructions, integrating egocentric visual inputs, providing action options and specifying the output format. These components are fed into Gemini 2.5 Pro, which generates step-by-step reasoning and action decisions aligned with navigation goals. environment constraints, and instruction semantics, and to produce structured step-by-st… view at source ↗
Figure 4
Figure 4. Figure 4: The pipeline of RL Policy. The policy model generates N outputs from text-image input. Then understanding reward (answer correctness and semantic alignment), navigation reward (path fidelity and endpoint accuracy), and format reward (structure adherence) are computed, grouped, and combined with a KL term to a frozen reference model to update the policy. reinforcement learning stage, where multi-dimensional… view at source ↗
Figure 5
Figure 5. Figure 5: Real-world robot setup and deployment pipeline. (a) Hardware platform: the WHEELTEC R550 robot equipped with Jetson Orin Nano (on-board PC), M10P LiDAR for mapping, Astra Pro RGB-D camera for perception, and STM32 microcontroller for motor control. (b) Deployment process: egocentric visual inputs are transmitted to the embodied foundation model Nav-R1, which performs reasoning and navigation. The decisions… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results from the real-world deployment of Nav-R1. We evaluate the agent in three indoor scenarios: meeting room, lounge, and corridor. Each scene illustrates the BEV trajectory and ego-centric video frames, showing the model’s ability to generalize to diverse layouts and object configurations in real-world environments. system, where an STM32 microcontroller converts them into PWM signals that … view at source ↗
Figure 7
Figure 7. Figure 7: Nav-CoT-110K CoT data example [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Real-world qualitative results of Nav-R1 on VLN and ObjectNav tasks in meeting room [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Real-world qualitative results of Nav-R1 on VLN and ObjectNav tasks in lounge [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Real-world qualitative results of Nav-R1 on VLN and ObjectNav tasks in corridor [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Real-world qualitative results of Nav-R1 on embodied dialogue task [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Real-world qualitative results of Nav-R1 on embodied reasoning task [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Real-world qualitative results of Nav-R1 on embodied planning task [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Visual results of VLN on VLN-CE R2R [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Visual results of VLN on VLN-CE R2R [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Visual results of ObjectNav on HM3D [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

    cs.RO 2026-05 unverdicted novelty 7.0

    AwareVLN introduces a structural reasoning module and automatic data engine with progress division to equip VLN agents with self-awareness of agent state and task progress, outperforming prior methods on Habitat datasets.

  2. What if? Emulative Simulation with World Models for Situated Reasoning

    cs.CV 2026-03 conditional novelty 6.5

    WanderDream supplies 15.8K panoramic mental-exploration videos and 158K QA pairs showing that world-model imagination measurably improves situated spatial reasoning without active exploration.

  3. From Failures to Supervision: DynamicEnvPlan for Robust Long-Horizon Embodied Planning

    cs.RO 2026-08 conditional novelty 6.0

    Training a VLM planner on recovery traces synthesized from perturbations and privileged corrections lifts dynamic household task success from 33.3% to 76.2%.

  4. ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning

    cs.CV 2026-07 conditional novelty 6.0

    A geometry-consistent memory plus a self-supervised consistency-reinforcement stage improves video spatial reasoning by roughly 7–15 points on three benchmarks, with an advertised average gain of 12.6 points.

  5. Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views

    cs.CV 2026-06 unverdicted novelty 6.0

    DR-MV3D decomposes MV3D-VQA into global map construction, question-conditioned view planning, and egocentric grounding, supervised by global consistency and local trajectory rewards optimized via GRPO.

  6. PlatonicNav: Unveiling Semantic Correspondence in Navigation with Platonic Topological Maps

    cs.CV 2026-06 unverdicted novelty 6.0

    PlatonicNav is a training-free framework using Platonic Topological Maps from a self-supervised visual encoder to unify vision-only ObjNav, cross-modal ObjNav, and VLN via blind matching on a shared semantic manifold.

  7. Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigation

    cs.CV 2026-06 unverdicted novelty 6.0

    Goal2Pixel grounds VLN-CE goals to image pixels via VLM prediction plus keyframe memory, reaching 54.1% SR on R2R-CE Val-Unseen with 7.75 calls per episode versus 46.62 for action prediction.

  8. SpaAct: Spatially-Activated Transition Learning with Curriculum Adaptation for Vision-Language Navigation

    cs.CV 2026-04 unverdicted novelty 6.0

    SpaAct activates spatial awareness in VLMs using action retrospection, future frame prediction, and progressive curriculum learning to reach SOTA on VLN-CE benchmarks.

  9. GeoWorld: Geometric World Models

    cs.CV 2026-02 unverdicted novelty 6.0

    GeoWorld applies hyperbolic geometry to JEPA world models and introduces geometric reinforcement learning, reporting modest success-rate gains of ~3% and ~2% on 3- and 4-step planning tasks versus V-JEPA 2.

  10. Progress-Think: Semantic Progress Reasoning for Vision-Language Navigation

    cs.RO 2025-11 unverdicted novelty 6.0

    Semantic progress reasoning predicts instruction-style advancement from visual history to guide policies, yielding state-of-the-art success and efficiency on R2R-CE and RxR-CE.

  11. UniMesh: Unifying 3D Mesh Understanding and Generation

    cs.CV 2026-04 unverdicted novelty 5.0

    UniMesh unifies 3D mesh generation and understanding in one model via a Mesh Head interface, Chain of Mesh iterative editing, and an Actor-Evaluator self-reflection loop.

Reference graph

Works this paper leans on

58 extracted references · 20 linked inside Pith · cited by 11 Pith papers

  1. [1]

    Etpnav: Evolving topological planning for vision-language navigation in continuous environments,

    D. An, H. Wang, W. Wang, Z. Wang, Y . Huang, K. He, and L. Wang, “Etpnav: Evolving topological planning for vision-language navigation in continuous environments,”PAMI, 2024

    2024

  2. [2]

    1st place solutions for rxr-habitat vision-and-language navigation competition,

    D. An, Z. Wang, Y . Li, Y . Wang, Y . Hong, Y . Huang, L. Wang, and J. Shao, “1st place solutions for rxr-habitat vision-and-language navigation competition,” inCVPRW, 2022

    2022

  3. [3]

    On evaluation of embodied navigation agents,

    P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V . Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva,et al., “On evaluation of embodied navigation agents,”arXiv preprint arXiv:1807.06757, 2018

    Pith/arXiv arXiv 2018

  4. [4]

    Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language nav- igation: Interpreting visually-grounded navigation instructions in real environments,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3674–3683

    2018

  5. [5]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” inProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, J. Goldstein, A. Lavie, C.-Y . Lin, and C. V oss, Eds. Ann Arbor, Michigan: Association for Computational Li...

    2005

  6. [6]

    Touchdown: Natural language navigation and spatial reasoning in visual street environments,

    H. Chen, A. Suhr, D. Misra, N. Snavely, and Y . Artzi, “Touchdown: Natural language navigation and spatial reasoning in visual street environments,” in2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 12 530–12 539

    2019

  7. [7]

    Affordances- oriented planning using foundation models for continuous vision- language navigation,

    J. Chen, B. Lin, X. Liu, X. Liang, and K.-Y . K. Wong, “Affordances- oriented planning using foundation models for continuous vision- language navigation,”arXiv preprint arXiv:2407.05890, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Topological planning with transformers for vision-and-language nav- igation,

    K. Chen, J. K. Chen, J. Chuang, M. Vázquez, and S. Savarese, “Topological planning with transformers for vision-and-language nav- igation,” inCVPR, 2021

    2021

  9. [9]

    Weakly- supervised multi-granularity map learning for vision-and-language navigation,

    P. Chen, D. Ji, K. Lin, R. Zeng, T. Li, M. Tan, and C. Gan, “Weakly- supervised multi-granularity map learning for vision-and-language navigation,” inNeurIPS, 2022

    2022

  10. [10]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning,

    S. Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, H. Zhu, J. Fan, and T. Chen, “Ll3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning,” inCVPR, 2024, pp. 26 418– 26 428

    2024

  11. [11]

    Navila: Legged robot vision-language- action model for navigation,

    A.-C. Cheng, Y . Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang, “Navila: Legged robot vision-language- action model for navigation,”arXiv preprint arXiv:2412.04453, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Octonav: Towards generalist embodied navigation,

    C. Gao, L. Jin, X. Peng, J. Zhang, Y . Deng, A. Li, H. Wang, and S. Liu, “Octonav: Towards generalist embodied navigation,”arXiv preprint arXiv:2506.09839, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Cross-modal map learning for vision and language navigation,

    G. Georgakis, K. Schmeckpeper, K. Wanchoo, S. Dan, E. Miltsakaki, D. Roth, and K. Daniilidis, “Cross-modal map learning for vision and language navigation,” inCVPR, 2022

    2022

  15. [15]

    Gaussianvlm: Scene-centric 3d vision-language models using language-aligned gaussian splats for embodied reasoning and beyond,

    A.-M. Halacheva, J.-N. Zaech, X. Wang, D. P. Paudel, and L. V . Gool, “Gaussianvlm: Scene-centric 3d vision-language models using language-aligned gaussian splats for embodied reasoning and beyond,” arXiv preprint arXiv:2507.00886, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Bridging the gap between learning in discrete and continuous environments for vision-and- language navigation,

    Y . Hong, Z. Wang, Q. Wu, and S. Gould, “Bridging the gap between learning in discrete and continuous environments for vision-and- language navigation,” inCVPR, 2022

    2022

  17. [17]

    Learning navigational visual representations with semantic map supervision,

    Y . Hong, Y . Zhou, R. Zhang, F. Dernoncourt, T. Bui, S. Gould, and H. Tan, “Learning navigational visual representations with semantic map supervision,” inICCV, 2023

    2023

  18. [18]

    3d- LLM: Injecting the 3d world into large language models,

    Y . Hong, H. Zhen, P. Chen, S. Zheng, Y . Du, Z. Chen, and C. Gan, “3d- LLM: Injecting the 3d world into large language models,” inThirty- seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=YQA28p7qNz

    2023

  19. [19]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  20. [20]

    Chat-scene: Bridging 3d scene and large language models with object identifiers,

    H. Huang, Y . Chen, Z. Wang, R. Huang, R. Xu, T. Wang, L. Liu, X. Cheng, Y . Zhao, J. Pang, and Z. Zhao, “Chat-scene: Bridging 3d scene and large language models with object identifiers,” inThe Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  21. [21]

    An embodied generalist agent in 3d world,

    J. Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y . Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang, “An embodied generalist agent in 3d world,” inICLR 2024 Workshop: How Far Are We From AGI, 2024. [Online]. Available: https://openreview.net/forum?id=ltX3S0juSa

    2024

  22. [22]

    An embodied generalist agent in 3d world,

    ——, “An embodied generalist agent in 3d world,” inICLR 2024 Workshop: How Far Are We From AGI, 2024. [Online]. Available: https://openreview.net/forum?id=ltX3S0juSa

    2024

  23. [23]

    3d-r1: Enhancing reason- ing in 3d vlms for unified scene understanding,

    T. Huang, Z. Zhang, and H. Tang, “3d-r1: Enhancing reason- ing in 3d vlms for unified scene understanding,”arXiv preprint arXiv:2507.23478, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    3d coca: Contrastive learners are 3d captioners,

    T. Huang, Z. Zhang, Y . Wang, and H. Tang, “3d coca: Contrastive learners are 3d captioners,”arXiv preprint arXiv:2504.09518, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Dc-scene: Data- centric learning for 3d scene understanding,

    T. Huang, Z. Zhang, R. Zhang, and Y . Zhao, “Dc-scene: Data- centric learning for 3d scene understanding,”arXiv preprint arXiv:2505.15232, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    General evaluation for instruction conditioned navigation using dynamic time warping,

    G. Ilharco, V . Jain, A. Ku, E. Ie, and J. Baldridge, “General evaluation for instruction conditioned navigation using dynamic time warping,” arXiv preprint arXiv:1907.05446, 2019

    Pith/arXiv arXiv 1907

  27. [27]

    Kahneman,Thinking, Fast and Slow

    D. Kahneman,Thinking, Fast and Slow. Allen Lane, 2011

    2011

  28. [28]

    Sim-2-sim transfer for vision-and-language navigation in continuous environments,

    J. Krantz and S. Lee, “Sim-2-sim transfer for vision-and-language navigation in continuous environments,” inECCV, 2022

    2022

  29. [29]

    Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,

    J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,” inComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16. Springer, 2020, pp. 104–120

    2020

  30. [30]

    Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,

    A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge, “Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,” inEMNLP, 2020

    2020

  31. [31]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/

    2004

  32. [32]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment,

    Y . Long, W. Cai, H. Wang, G. Zhan, and H. Dong, “Instructnav: Zero-shot system for generic instruction navigation in unexplored environment,”arXiv preprint arXiv:2406.04882, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Sqa3d: Situated question answering in 3d scenes,

    X. Ma, S. Yong, Z. Zheng, Q. Li, Y . Liang, S.-C. Zhu, and S. Huang, “Sqa3d: Situated question answering in 3d scenes,” inInternational Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=IDJx97BC38

    2023

  34. [34]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ser. ACL ’02. USA: Association for Computational Linguistics, 2002, p. 311–318. [Online]. Available: https://doi.org/10.3115/1073083.1073135

    arXiv 2002

  35. [35]

    Alvinn: an autonomous land vehicle in a neural network,

    D. A. Pomerleau, “Alvinn: an autonomous land vehicle in a neural network,” inProceedings of the 2nd International Conference on Neural Information Processing Systems, ser. NIPS’88. Cambridge, MA, USA: MIT Press, 1988, p. 305–313

    1988

  36. [36]

    Vln-r1: Vision- language navigation via reinforcement fine-tuning,

    Z. Qi, Z. Zhang, Y . Yu, J. Wang, and H. Zhao, “Vln-r1: Vision- language navigation via reinforcement fine-tuning,”arXiv preprint arXiv:2506.17221, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Language- aligned waypoint (law) supervision for vision-and-language navigation in continuous environments,

    S. Raychaudhuri, S. Wani, S. Patel, U. Jain, and A. Chang, “Language- aligned waypoint (law) supervision for vision-and-language navigation in continuous environments,” inEMNLP, 2021

    2021

  38. [38]

    A reduction of imitation learning and structured prediction to no-regret online learning,

    S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, G. Gordon, D. Dunson, and M. Dudík, Eds., vol. 15. Fort Lauderdale, FL, USA: PMLR...

    2011

  39. [39]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  40. [41]

    Hazards in daily life? enabling robots to proactively detect and resolve anomalies,

    Z. Song, G. Ouyang, M. Fang, H. Na, Z. Shi, Z. Chen, F. Yujie, Z. Zhang, S. Jiang, M. Fang,et al., “Hazards in daily life? enabling robots to proactively detect and resolve anomalies,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa...

    2025

  41. [42]

    Maniplvm-r1: Reinforcement learning for reasoning in embodied manipulation with large vision- language models,

    Z. Song, G. Ouyang, M. Li, Y . Ji, C. Wang, Z. Xu, Z. Zhang, X. Zhang, Q. Jiang, Z. Chen,et al., “Maniplvm-r1: Reinforcement learning for reasoning in embodied manipulation with large vision- language models,”arXiv preprint arXiv:2505.16517, 2025

    Pith/arXiv arXiv 2025

  42. [43]

    Evaluating gemini in an arena for learning,

    L. Team, A. Modi, A. S. Veerubhotla, A. Rysbek, A. Huber, A. Anand, A. Bhoopchand, B. Wiltshire, D. Gillick, D. Kasenberg, E. Sgouritsa, G. Elidan, H. Liu, H. Winnemoeller, I. Jurenka, J. Cohan, J. She, J. Wilkowski, K. Alarakyia, K. R. McKee, K. Singh, L. Wang, M. Kunesch, M. Pîslar, N. Efron, P. Mahmoudieh, P.-A. Kamienny, S. Wiltberger, S. Mohamed, S. ...

    Pith/arXiv arXiv 2025

  43. [44]

    Cider: Consensus-based image description evaluation,

    R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4566–4575

    2015

  44. [45]

    Dreamwalker: Mental planning for continuous vision-language navigation,

    H. Wang, W. Liang, L. Van Gool, and W. Wang, “Dreamwalker: Mental planning for continuous vision-language navigation,” inICCV, 2023

    2023

  45. [46]

    Spatial 3d-llm: Exploring spatial awareness in 3d vision-language models,

    X. Wang, Z. Li, Y . Xu, J. Qi, Z. Yang, R. Ma, X. Liu, and C. Zhang, “Spatial 3d-llm: Exploring spatial awareness in 3d vision-language models,”arXiv preprint arXiv:2507.16524, 2025

    Pith/arXiv arXiv 2025

  46. [47]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,

    X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y .-F. Wang, W. Y . Wang, and L. Zhang, “Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2019, pp. 6622–6631

    2019

  47. [48]

    Looka- head exploration with neural radiance representation for continuous vision-language navigation,

    Z. Wang, X. Li, J. Yang, Y . Liu, J. Hu, M. Jiang, and S. Jiang, “Looka- head exploration with neural radiance representation for continuous vision-language navigation,” inCVPR, 2024

    2024

  48. [49]

    Gridmm: Grid memory map for vision-and-language navigation,

    Z. Wang, X. Li, J. Yang, Y . Liu, and S. Jiang, “Gridmm: Grid memory map for vision-and-language navigation,” inICCV, 2023

    2023

  49. [50]

    Streamvln: Streaming vision-and-language navigation via slowfast context modeling,

    M. Wei, C. Wan, X. Yu, T. Wang, Y . Yang, X. Mao, C. Zhu, W. Cai, H. Wang, Y . Chen, X. Liu, and J. Pang, “Streamvln: Streaming vision-and-language navigation via slowfast context modeling,”arXiv preprint arXiv:2507.05240, 2025

    Pith/arXiv arXiv 2025

  50. [51]

    Vlfm: Vision- language frontier maps for zero-shot semantic navigation,

    N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher, “Vlfm: Vision- language frontier maps for zero-shot semantic navigation,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 42–48

    2024

  51. [52]

    Hm3d- ovon: A dataset and benchmark for open-vocabulary object goal navigation,

    N. Yokoyama, R. Ramrakhya, A. Das, D. Batra, and S. Ha, “Hm3d- ovon: A dataset and benchmark for open-vocabulary object goal navigation,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 5543–5550

    2024

  52. [53]

    Correctnav: Self-correction flywheel empowers vision- language-action navigation model,

    Z. Yu, Y . Long, Z. Yang, C. Zeng, H. Fan, J. Zhang, and H. Dong, “Correctnav: Self-correction flywheel empowers vision- language-action navigation model,”arXiv preprint arXiv:2508.10416, 2025

    Pith/arXiv arXiv 2025

  53. [54]

    Uni-navid: A video-based vision-language- action model for unifying embodied navigation tasks,

    J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang, “Uni-navid: A video-based vision-language- action model for unifying embodied navigation tasks,”Robotics: Science and Systems, 2025

    2025

  54. [55]

    Navid: Video-based vlm plans the next step for vision-and-language navigation,

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and W. He, “Navid: Video-based vlm plans the next step for vision-and-language navigation,” inRSS, 2024

    2024

  55. [56]

    Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences,

    H. Zhi, P. Chen, J. Li, S. Ma, X. Sun, T. Xiang, Y . Lei, M. Tan, and C. Gan, “Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences,”arXiv preprint arXiv:2412.01292, 2024

    Pith/arXiv arXiv 2024

  56. [57]

    Soon: Scenario oriented object navigation with graph-based exploration,

    F. Zhu, X. Liang, Y . Zhu, Q. Yu, X. Chang, and X. Liang, “Soon: Scenario oriented object navigation with graph-based exploration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12 689–12 699

    2021

  57. [58]

    Move to understand a 3d scene: Bridging visual grounding and exploration for efficient and versatile embodied navigation,

    Z. Zhu, X. Wang, Y . Li, Z. Zhang, X. Ma, Y . Chen, B. Jia, W. Liang, Q. Yu, Z. Deng, S. Huang, and Q. Li, “Move to understand a 3d scene: Bridging visual grounding and exploration for efficient and versatile embodied navigation,”International Conference on Computer Vision (ICCV), 2025

    2025

  58. [2024]

    Available: https://arxiv.org/abs/2402.03300

    [Online]. Available: https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv