The paper introduces a Trajectory Waypoint paradigm with a TSDF-guided diffusion policy and trajectory-enhanced navigator that achieves better performance on VLN-CE benchmarks by ensuring waypoint reachability and planning-execution consistency.
hub Canonical reference
NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
Canonical reference. 71% of citing Pith papers cite this work as background.
38
Pith papers citing it
Background
71% of classified citations
abstract
Vision-and-language navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavor to showcase the capability of VLMs to achieve state-of-the-art level navigation performance without any maps, odometers, or depth inputs. Following human instruction, NaVid only requires an on-the-fly video stream from a monocular RGB camera equipped on the robot to output the next-step action. Our formulation mimics how humans navigate and naturally gets rid of the problems introduced by odometer noises, and the Sim2Real gaps from map or depth inputs. Moreover, our video-based approach can effectively encode the historical observations of robots as spatio-temporal contexts for decision making and instruction following. We train NaVid with 510k navigation samples collected from continuous environments, including action-planning and instruction-reasoning samples, along with 763k large-scale web data. Extensive experiments show that NaVid achieves state-of-the-art performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer. We thus believe our proposed VLM approach plans the next step for not only the navigation agents but also this research field.
hub tools
citation-role summary
background 5
baseline 2
citation-polarity summary
representative citing papers
WorldVLN proposes the first autoregressive world action model for aerial vision-language navigation that predicts short-horizon latent world states, decodes them to waypoints in closed loop, and uses two-stage training with Action-aware GRPO to achieve over 12% success-rate gains on benchmarks plus零
VLN-Cache delivers up to 1.52x faster inference in VLN models by using view-aligned remapping for geometric consistency and a task-relevance saliency filter to manage semantic changes during navigation.
GeoThinker enables active, task-conditioned geometry integration in MLLMs via spatial-grounded fusion and importance gating, reaching 72.6 on VSI-Bench.
DOMINO supplies 110K+ dynamic dual-arm trajectories across 35 tasks, and PUMA’s optical-flow history plus object-centric future queries raise dynamic success rate by 6.3 points over strong VLA baselines.
SAGE-Nav decouples LLM global planning from reactive control via hierarchical scene graphs and alignment fusion, reporting SOTA results on i-THOR and RoboTHOR with improved efficiency and zero-shot generalization.
SurveilNav integrates robot local perception with multi-view surveillance for improved collaborative object goal navigation and reports SOTA results on HM3D.
Goal2Pixel grounds VLN-CE goals to image pixels via VLM prediction plus keyframe memory, reaching 54.1% SR on R2R-CE Val-Unseen with 7.75 calls per episode versus 46.62 for action prediction.
HSGM structures 3D geometry and semantics into a multi-level map that lets VLMs perform high-level planning in zero-shot VLN, achieving SOTA on R2R-CE and RxR-CE.
GA-VLN builds a geometry-aware BEV representation from RGB-D inputs plus 3D foundation model features to deliver state-of-the-art vision-language navigation using only navigation data.
SpaAct activates spatial awareness in VLMs using action retrospection, future frame prediction, and progressive curriculum learning to reach SOTA on VLN-CE benchmarks.
FreqCache uses frequency domain properties to adaptively select, refresh, and budget token caches in VLN models, delivering 1.59x speedup with negligible overhead.
Privatar partitions VR avatar reconstruction via frequency-domain decomposition, keeping sensitive components local and offloading the rest with distribution-aware minimal perturbation noise, achieving 2.37x throughput with provable privacy.
FineCog-Nav uses fine-grained cognitive modules driven by foundation models to outperform zero-shot baselines in UAV navigation and introduces the AerialVLN-Fine benchmark with refined instructions.
A coupled world-agent framework uses 3D Gaussian reconstruction and first-person RGB-D perception with iterative planning to enable goal-directed, collision-avoiding humanoid behavior in novel reconstructed scenes.
HiRO-Nav adaptively triggers reasoning only on high-entropy actions via a hybrid training pipeline and shows better success-token trade-offs than always-reason or never-reason baselines on the CHORES-S benchmark.
A1 is a transparent VLA framework achieving state-of-the-art robot manipulation success with up to 72% lower latency via adaptive layer truncation and inter-layer flow matching.
Motion-MLLM integrates IMU egomotion data into MLLMs using cascaded filtering and asymmetric fusion to ground visual content in physical trajectories for scale-aware 3D understanding, achieving competitive accuracy at higher speed.
Semantic progress reasoning predicts instruction-style advancement from visual history to guide policies, yielding state-of-the-art success and efficiency on R2R-CE and RxR-CE.
C-Nav is a continual visual navigation framework with dual-path anti-forgetting via feature distillation and replay plus adaptive sampling that outperforms baselines on a new continual object navigation benchmark while using less memory.
R2RGen introduces a simulator-free three-stage pipeline that parses, augments, and post-processes real pointcloud observation-action pairs to improve spatial generalization in robotic manipulation policies.
GraspVLA shows that pretraining a grasping model on a billion synthetic action frames enables zero-shot open-vocabulary performance and sim-to-real transfer.
An asynchronous fast-slow dual-system with DiT action modeling and time-weighted loss doubles unseen aerial VLN success rates and halves decision latency in simulation.
FutureNav proposes a 4B-scale VLM that jointly optimizes action prediction, inverse/forward dynamics, and future state generation for VLN and reports SOTA results on multiple benchmarks.
citing papers explorer
-
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
The paper introduces a Trajectory Waypoint paradigm with a TSDF-guided diffusion policy and trajectory-enhanced navigator that achieves better performance on VLN-CE benchmarks by ensuring waypoint reachability and planning-execution consistency.
-
WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation
WorldVLN proposes the first autoregressive world action model for aerial vision-language navigation that predicts short-horizon latent world states, decodes them to waypoints in closed loop, and uses two-stage training with Action-aware GRPO to achieve over 12% success-rate gains on benchmarks plus零
-
VLN-Cache: Enabling Token Caching for VLN Models with Visual/Semantic Dynamics Awareness
VLN-Cache delivers up to 1.52x faster inference in VLN models by using view-aligned remapping for geometric consistency and a task-relevance saliency filter to manage semantic changes during navigation.
-
Thinking with Geometry: Active Geometry Integration for Spatial Reasoning
GeoThinker enables active, task-conditioned geometry integration in MLLMs via spatial-grounded fusion and importance gating, reaching 72.6 on VSI-Bench.
-
Towards Generalizable Robotic Manipulation in Dynamic Environments
DOMINO supplies 110K+ dynamic dual-arm trajectories across 35 tasks, and PUMA’s optical-flow history plus object-centric future queries raise dynamic success rate by 6.3 points over strong VLA baselines.
-
SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation
SAGE-Nav decouples LLM global planning from reactive control via hierarchical scene graphs and alignment fusion, reporting SOTA results on i-THOR and RoboTHOR with improved efficiency and zero-shot generalization.
-
SurveilNav: Collaborative Object Goal Navigation with Robot and Surveillance System
SurveilNav integrates robot local perception with multi-view surveillance for improved collaborative object goal navigation and reports SOTA results on HM3D.
-
Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigation
Goal2Pixel grounds VLN-CE goals to image pixels via VLM prediction plus keyframe memory, reaching 54.1% SR on R2R-CE Val-Unseen with 7.75 calls per episode versus 46.62 for action prediction.
-
Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
HSGM structures 3D geometry and semantics into a multi-level map that lets VLMs perform high-level planning in zero-shot VLN, achieving SOTA on R2R-CE and RxR-CE.
-
GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation
GA-VLN builds a geometry-aware BEV representation from RGB-D inputs plus 3D foundation model features to deliver state-of-the-art vision-language navigation using only navigation data.
-
SpaAct: Spatially-Activated Transition Learning with Curriculum Adaptation for Vision-Language Navigation
SpaAct activates spatial awareness in VLMs using action retrospection, future frame prediction, and progressive curriculum learning to reach SOTA on VLN-CE benchmarks.
-
FreqCache: Accelerating Embodied VLN Models with Adaptive Frequency-Guided Token Caching
FreqCache uses frequency domain properties to adaptively select, refresh, and budget token caches in VLN models, delivering 1.59x speedup with negligible overhead.
-
Dual-Anchoring: Addressing State Drift in Vision-Language Navigation
Privatar partitions VR avatar reconstruction via frequency-domain decomposition, keeping sensitive components local and offloading the rest with distribution-aware minimal perturbation noise, achieving 2.37x throughput with provable privacy.
-
FineCog-Nav: Integrating Fine-grained Cognitive Modules for Zero-shot Multimodal UAV Navigation
FineCog-Nav uses fine-grained cognitive modules driven by foundation models to outperform zero-shot baselines in UAV navigation and introduces the AerialVLN-Fine benchmark with refined instructions.
-
Visually-grounded Humanoid Agents
A coupled world-agent framework uses 3D Gaussian reconstruction and first-person RGB-D perception with iterative planning to enable goal-directed, collision-avoiding humanoid behavior in novel reconstructed scenes.
-
HiRO-Nav: Hybrid ReasOning Enables Efficient Embodied Navigation
HiRO-Nav adaptively triggers reasoning only on high-entropy actions via a hybrid training pipeline and shows better success-token trade-offs than always-reason or never-reason baselines on the CHORES-S benchmark.
-
A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Model
A1 is a transparent VLA framework achieving state-of-the-art robot manipulation success with up to 72% lower latency via adaptive layer truncation and inter-layer flow matching.
-
Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understanding
Motion-MLLM integrates IMU egomotion data into MLLMs using cascaded filtering and asymmetric fusion to ground visual content in physical trajectories for scale-aware 3D understanding, achieving competitive accuracy at higher speed.
-
Progress-Think: Semantic Progress Reasoning for Vision-Language Navigation
Semantic progress reasoning predicts instruction-style advancement from visual history to guide policies, yielding state-of-the-art success and efficiency on R2R-CE and RxR-CE.
-
C-NAV: Towards Self-Evolving Continual Object Navigation in Open World
C-Nav is a continual visual navigation framework with dual-path anti-forgetting via feature distillation and replay plus adaptive sampling that outperforms baselines on a new continual object navigation benchmark while using less memory.
-
R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation
R2RGen introduces a simulator-free three-stage pipeline that parses, augments, and post-processes real pointcloud observation-action pairs to improve spatial generalization in robotic manipulation policies.
-
GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data
GraspVLA shows that pretraining a grasping model on a billion synthetic action frames enables zero-shot open-vocabulary performance and sim-to-real transfer.
-
FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation
An asynchronous fast-slow dual-system with DiT action modeling and time-weighted loss doubles unseen aerial VLN success rates and halves decision latency in simulation.
-
FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation
FutureNav proposes a 4B-scale VLM that jointly optimizes action prediction, inverse/forward dynamics, and future state generation for VLN and reports SOTA results on multiple benchmarks.
-
Invertible Neural Network Adapter for One-Step Flow Matching in Robot Manipulation
An invertible adapter for flow matching enables one-step high-dimensional action generation in robotic manipulation, cutting inference time roughly in half while preserving performance.
-
Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation System
Qwen-RobotNav provides a parameterized navigation model trained on 15.6M samples with vision-language co-training that achieves SOTA results on benchmarks and zero-shot transfer to real robots.
-
AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning
AllDayNav encodes scene dynamics into a large model's parameters via RL and a multimodal memory, achieving near-100% success rates in lifelong navigation and outperforming map-based and VLM baselines.
-
IntentNav: Learning Spatial-Visual Object Navigation from Human Demonstrations
IntentNav is a spatial-visual imitation framework that infers human search intent via frontier labeling to train VLM policies for object navigation, reporting SOTA on MP3D and HM3D benchmarks with zero-shot transfer to wheeled, quadruped, and humanoid robots.
-
ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning
ImagineUAV is a 1.3B-parameter cascaded world-action framework that generates instruction-conditioned future observations via latent video diffusion, infers motions, and applies kinodynamic planning to outperform VLN/VLA baselines in aerial navigation.
-
SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation
SEDualVLN introduces a spatially-enhanced dual-system VLN architecture that achieves state-of-the-art results on VLN-CE benchmarks through coordinated VLM action generation and MLLM waypoint planning.
-
LCGNav: Local Candidate-Aware Geometric Enhancement for General Topological Planning in Vision-Language Navigation
LCGNav improves online topological VLN-CE by converting local depth views to physically truncated 3D point clouds and applying selective dimension-preserving fusion, yielding consistent gains on R2R-CE and RxR-CE benchmarks with open code.
-
PathPainter: Transferring the Generalization Ability of Image Generation Models to Embodied Navigation
Navigation system transfers image generation models to embodied tasks via BEV-based traversability mask generation from language and cross-view localization for odometry correction, shown on a UAV completing 160m outdoor navigation.
-
Rectified Schr\"odinger Bridge Matching for Few-Step Visual Navigation
Rectified Schrödinger Bridge Matching uses ε-invariant velocity structure and a learned prior so generative navigation policies reach ~94% cosine similarity and 92% success in three integration steps without distillation.
-
Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control
Survey organizing VLM-based social robot navigation into reasoning, planning, and bridging components with a proposed roadmap for hybrid deployable systems.
-
Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigation
Proposes cost-aware question selection for ambiguous object navigation via information-gain analysis on corpora, a cost-penalizing benchmark, and a zero-shot MLLM agent.
-
LASAR: Towards Spatio-temporal Reasoning with Latent Cognitive Map
LASAR pairs a dual-memory system with spatio-temporal contrastive learning to induce latent cognitive maps, reporting 2-3.5% zero-shot gains on VLN-CE and VSI-Bench plus high map self-consistency.
-
When control meets large language models: From words to dynamics
The paper proposes a bidirectional continuum between LLMs and control systems, covering LLM-assisted controller design, control-based LLM steering, and state-space modeling of LLMs.
- FlowDec: Temporal Conditional Flow Decorruptor for Robust Continuous Vision-Language Navigation