REVIEW 14 cited by
End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
read the original abstract
We present VLMnav, an embodied framework to transform a Vision-Language Model (VLM) into an end-to-end navigation policy. In contrast to prior work, we do not rely on a separation between perception, planning, and control; instead, we use a VLM to directly select actions in one step. Surprisingly, we find that a VLM can be used as an end-to-end policy zero-shot, i.e., without any fine-tuning or exposure to navigation data. This makes our approach open-ended and generalizable to any downstream navigation task. We run an extensive study to evaluate the performance of our approach in comparison to baseline prompting methods. In addition, we perform a design analysis to understand the most impactful design decisions. Visual examples and code for our project can be found at https://jirl-upenn.github.io/VLMnav/
Forward citations
Cited by 14 Pith papers
-
LIME: Learning Intent-aware Camera Motion from Egocentric Video
LIME formulates language-conditioned camera motion as predicting SE(3) target poses from RGB and intent text, using mined multi-intent supervision from egocentric video and a flow-matching pose head.
-
VTM-Nav: Harnessing Cross-Episode Experience for Object-Goal Navigation with Hierarchical Visual-Topological Memory
A hierarchical room-and-object memory that persists across independent ObjectNav episodes yields small success-rate gains, but most of the gain comes from within-episode memory rather than the cross-episode component.
-
EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigation
EvoMemNav builds a Visual-Semantic Memory Graph keeping raw views, applies a budgeted coarse-to-fine policy, and uses reflection-driven updates to improve zero-shot navigation on GOAT-Bench and HM3D.
-
SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation
SEDualVLN proposes a spatially-enhanced dual-system VLN framework that pairs a fast VLM action generator with a slow MLLM waypoint planner and reports state-of-the-art results on VLN-CE benchmarks.
-
Visually-grounded Humanoid Agents
A coupled world-agent framework uses 3D Gaussian reconstruction and first-person RGB-D perception with iterative planning to enable goal-directed, collision-avoiding humanoid behavior in novel reconstructed scenes.
-
Structured Observation Language for Efficient and Generalizable Vision-Language Navigation
SOL-Nav encodes RGB-D observations as grid-organized text and uses a 0.6B text-embedding model with four classification heads to predict navigation action blocks, reporting SOTA/comparable R2R-CE/RxR-CE results with 1...
-
ReMemNav: A Rethinking and Memory-Augmented Framework for Zero-Shot Object Navigation
ReMemNav improves zero-shot object navigation success and efficiency by integrating episodic memory and rethinking with VLMs, achieving SR/SPL gains of 1.7%/7.0% on HM3D v0.1, 18.2%/11.1% on HM3D v0.2, and 8.7%/7.9% on MP3D.
-
DSCD-Nav: Dual-Stance Cooperative Debate for Object Navigation
A dual-stance debate between a goal-focused and a safety-focused VLM, plus arbitration and optional micro-probing, improves zero-shot object navigation success and path efficiency on HM3Dv1, HM3Dv2, MP3D, and GOAT.
-
ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
ImagineNav++ achieves SOTA mapless visual navigation by prompting VLMs to select imagined future views generated from a human-preference-distilled module and maintained via selective foveation memory.
-
SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation
SEDualVLN introduces a spatially-enhanced dual-system VLN architecture that achieves state-of-the-art results on VLN-CE benchmarks through coordinated VLM action generation and MLLM waypoint planning.
-
GUI Agents with Reinforcement Learning: Toward Digital Inhabitants
The paper delivers the first comprehensive overview of RL for GUI agents, organizing methods into offline, online, and hybrid strategies while analyzing trends in rewards, efficiency, and deliberation to outline a fut...
-
OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
OpenFrontier treats navigation as sparse visual-frontier subgoal selection guided by vision-language priors, claiming strong zero-shot and real-robot performance without task-specific training.
-
OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
OpenFrontier formulates robot navigation as sparse subgoal reaching via visual-language-grounded frontiers, achieving zero-shot performance without fine-tuning or dense semantic maps.
-
Structured Observation Language for Efficient and Generalizable Vision-Language Navigation
A 0.6B language model navigates by reading grid-structured text descriptions of depth, object class, and color instead of images, with reported R2R-CE/RxR-CE scores near the top of the leaderboard.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.