Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

A 0.6B language model navigates unseen indoor environments at state-of-the-art levels by reading grid-structured text descriptions of its camera view instead of raw images.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:10 UTC pith:XIN4SSIK

load-bearing objection The core idea—structured grid text instead of raw pixels for a 0.6B VLN policy—is genuinely interesting, but the paper's empirical case collapses until the semantic-segmentation source is disclosed and the contradictory result claims are reconciled. the 4 major comments →

arxiv 2603.27577 v2 pith:XIN4SSIK submitted 2026-03-29 cs.CV cs.RO

Structured Observation Language for Efficient and Generalizable Vision-Language Navigation

Daojie Peng , Fulong Ma , Jun Ma This is my paper
classification cs.CV cs.RO
keywords Vision-Language NavigationStructured Observation LanguageLarge Language ModelsGeneralizable NavigationRobot Action PredictionRGB-DVLN-CESemantic Segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SOL-Nav reduces egocentric RGB-D observations to a grid of text cells—dominant semantic category, average depth, and color name per cell—and feeds that text plus the navigation instruction to a 0.6B pretrained language model. The model outputs a block of four discrete actions, and on the standard R2R-CE and RxR-CE val-unseen benchmarks it reaches state-of-the-art or comparable results without a waypoint predictor, without extra training data, and with a model 10–50x smaller than competing vision-language-action systems. The paper interprets this as evidence that structured text can carry enough spatial and semantic information for navigation, eliminating scratch visual pretraining and the modal alignment bottleneck. If right, small edge-deployable text-only policies become a practical route to generalizable robot navigation.

Core claim

SOL-Nav's central claim is that visual observations do not need to be encoded as pixels or visual tokens; they can be serialized into a structured observation language—a grid with per-cell depth, dominant semantic class, and representative color name, combined with short- and long-term histories at different grid resolutions—and concatenated with the instruction and a system description to form a pure-language prompt. A small pre-trained language model (Qwen3-Embedding, 0.6B), fine-tuned with LoRA and a weighted cross-entropy loss, reads this prompt and predicts 4-step action chunks from a vocabulary of stop, turn left, turn right, and move forward. The experiments report state-of-the-art su

What carries the argument

The central device is the structured observation language prompt: each egocentric RGB-D frame is divided into grids (6×6 for the current view, 4×4 for short-term memory, 2×2 for long-term memory), and each cell is rendered as a triple of strings—average depth, dominant semantic label, representative color name—assembled in chronological order with the instruction and a system description. This text is the sole input to a pre-trained language model with a stack of classification heads, one per step in a 4-action chunk. The grid text is doing the work of encoding spatial layout and object identity in a format a language model can reason over directly, without any visual encoder or fusion modul

Load-bearing premise

Everything hinges on where the semantic labels come from—the paper never states whether the benchmark numbers use dataset-provided ground-truth segmentation or a pretrained segmenter, and either choice undermines the paper's generalization claim as stated.

What would settle it

Inspect the released code or reproduce the R2R-CE val-unseen evaluation with the semantic maps replaced by a pretrained segmenter's predictions on unseen scenes; if the success rate drops well below the reported 53.6 SR, or if the original numbers require stored ground-truth labels, the claim that SOL-Nav generalizes through structured text rather than privileged or pretrained perception is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • State-of-the-art or comparable navigation performance on R2R-CE and RxR-CE val-unseen can be achieved with a 0.6B model, without a waypoint predictor or extra training data.
  • The dependence on large-scale visual pretraining for VLN can be removed: the policy is a pure language model over serialized observations, so no scratch visual encoder needs to be trained.
  • Because the input is text, the same model and pipeline can be deployed on edge hardware (the paper reports 0.8s inference on a Jetson Orin) with minimal compute.
  • The structured text representation abstracts away lighting and texture, so the policy should generalize to unseen environments and sim-to-real transfer better than pixel-based methods; the real-world trials across three scenes support this.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the semantic labels come from the simulator's ground truth or a pretrained segmenter; rerunning the benchmarks with a frozen segmenter on unseen scenes would settle whether the structured text alone carries the generalization, and that experiment is a natural next step.
  • The method's unusually high oracle-success rate relative to its success rate suggests it nears targets reliably but struggles to stop at the precise goal; a dedicated stopping head or learned confidence threshold would likely be a quick improvement.
  • The same grid-to-text serialization could be applied to other embodied modalities—lidar, tactile, or audio—so that a single small language model handles heterogeneous sensor streams without task-specific encoders.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SOL-Nav proposes to replace visual tokens in VLN-CE with a structured text representation: RGB-D frames are divided into N×N grids, each cell annotated with average depth, dominant semantic category, and HSV-mapped color name, and these strings are concatenated with the instruction and a system prompt into a pure-language input. A 0.6B Qwen3-Embedding model is fine-tuned with LoRA to predict four-step action chunks. Results are reported on R2R-CE and RxR-CE val-unseen, with a real-world Unitree Go2 deployment. The paper claims state-of-the-art or comparable performance with a model 10–50× smaller than existing VLA methods, no waypoint predictor, and no additional training data.

Significance. The core idea — replacing visual encoders with structured observation text — is timely and, if substantiated, would be a meaningful step toward lightweight, edge-deployable VLN policies that reuse pretrained language models. The paper has several genuine strengths: the representation is simple and interpretable; the ablation study isolates the contributions of grid resolution, history, and depth; and the real-world deployment, while preliminary, demonstrates that a text-only policy can run on an edge GPU. However, the central empirical claim currently rests on an unspecified choice about the source of semantic labels, and several internal inconsistencies make the comparison hard to verify. If the semantic labels come from the simulator's ground-truth sensor, the reported gains may be due to privileged information rather than the structured-language mechanism; if they come from a pretrained segmenter, the 'no visual pretraining' claim and the generalization story need to be re-framed. These issues are load-bearing for the paper's main claims.

major comments (4)
  1. [§3.2, §4.1, §4.3] The paper never states whether the R2R-CE/RxR-CE numbers in Tables 1–2 used semantic labels from the Habitat simulator (e.g., the ground-truth semantic sensor) or a pretrained segmentation model. Section 3.2 says the map 'can be obtained directly from the datasets or using pre-trained semantic segmentation model,' and Section 4.3 only describes the segmenter for real-world experiments. This is not a minor implementation detail. If ground-truth labels were used in simulation, SOL-Nav receives a per-cell semantic category that RGB/D-only baselines do not have, so the reported OS/SR/SPL advantages may reflect privileged information. If a pretrained segmenter was used, the 'no visual pretraining' claim collapses and the paper must provide a sensitivity analysis to segmentation errors. Please specify the exact pipeline used for every reported number and, if applicable, compare against a versi
  2. [§4.1, Table 1] The text and table contradict each other on the InternVLA-N1 comparison. The text says SOL-Nav 'performs slightly worse than InternVLA-N1 on OS, SR, and SPL' and then 'surpasses InternVLA-N1 by more than 15% on OS.' Table 1 shows OS 72.9 vs 63.3 (SOL-Nav better), but SR 53.6 vs 58.2 and SPL 49.2 vs 54.0 (SOL-Nav worse). The correct statement is that SOL-Nav is better on OS but worse on SR and SPL. The paper also says the model is 'more than 10 times smaller' and later 'more than 50 times smaller'; with 0.6B parameters versus the stated 'at least 7B' baselines, only the ~10× figure is supported unless model sizes for each baseline are listed. Please correct these statements.
  3. [§3.4, §4.1] The claim of 'no additional training data' is undermined by unspecified data augmentation. Section 3.4 says task instructions are 'augmented using LLM to get more diverse task instructions,' but no details are given about the number of generated instructions, the LLM prompt, or whether these augmented samples are used in the R2R-CE/RxR-CE results. If LLM-generated instruction paraphrases are used for training, the comparison against methods marked † in Tables 1–2 may be unfair, and the 'low data dependency' claim is overstated. Similarly, the real-world deployment fine-tunes on 50 real navigation samples and a SegFormer fine-tuned on 1000 manually annotated images; this should be stated as such rather than presented as evidence of zero-shot or near-zero-data generalization.
  4. [§1, §4.3] The paper repeatedly claims robustness to 'lighting, texture, and environmental variations,' but no controlled experiment tests this. The R2R-CE/RxR-CE val-unseen splits are from the same distribution as the training environments, and the real-world experiments involve fine-tuning on in-situ data. To support the generalization claim, the authors should include at least one simulation setting with altered lighting/texture (or a cross-dataset evaluation such as training on MP3D and testing on HM3D) and report performance without the 50-sample real-world fine-tune. Without such evidence, the robustness claim is currently an assertion rather than a measured result.
minor comments (5)
  1. [§4.1, Table 2] The text reports NE=6.95 for SOL-Nav on RxR-CE, but Table 2 lists 6.87. The text also refers to an 'SP' metric that is not defined in Section 4.1. Please harmonize all numbers and metric names.
  2. [§4.1] The phrases 'the second column of Table 1,' 'the first column,' and 'the third column' are confusing because the table columns are not labeled as such. Please describe the comparison groups explicitly (e.g., 'methods without waypoint predictor and without extra data').
  3. [§1] There is a duplicated phrase in the Introduction: 'Specifically, Specifically, we first use ...' Please remove the duplicate.
  4. [§4.2, Table 3] In the 'No His.' row, a checkmark appears under 'History' even though the row is meant to ablate history. A clearer notation (e.g., '✗') would avoid ambiguity.
  5. [General] The paper would benefit from a reproducibility statement. In particular, details on the LLM instruction augmentation prompt, the HSV-to-color lookup table, and the exact LoRA hyperparameters are missing, which would make it difficult for others to reproduce the results.

Circularity Check

0 steps flagged

No circularity: the evaluation is anchored to external VLN-CE benchmarks and the structured text is derived from observations, not from the target actions.

full rationale

No circular step is present. The derivation chain is: raw RGB-D observations -> grid-based structured text (Eq. 1) -> time-ordered prompt (Eq. 2) -> PLM -> action-block predictions (Eq. 3 loss). The structured observation is not defined in terms of the predicted actions, no fitted parameter is renamed as a prediction, and the reported success is measured against the external R2R-CE and RxR-CE val-unseen benchmarks. All cited building blocks (SegFormer, Grounded SAM, Qwen3 Embedding) are external, and I found no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via the authors' prior work. The sentence in Section 4, 'we select the model with the best performance based on validation metrics for subsequent use,' is a potential protocol ambiguity, but the paper does not state that val-unseen was the validation split, so it is not a demonstrated circular reduction. The more substantive weakness is the unresolved semantic-source ambiguity in Section 3.2: 'The semantic segmentation map ... can be obtained directly from the datasets or using pre-trained semantic segmentation model,' while Section 4.3 reveals that real-world experiments finetune a SegFormer on manually annotated images. This affects whether the simulation comparisons used privileged ground-truth semantics, but it does not make the action prediction equivalent to the input by construction. The Section 5 limitation about losing fine-grained visual information is an honest stated limitation. Because the derivation is self-contained and checked against external benchmarks, the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced; the 'structured observation language' is a representation, not an entity. The load-bearing assumptions are the availability of semantic labels, the sufficiency of the discretized text representation, the adequacy of the discrete action space, and the suitability of an embedding model as the decision backbone.

free parameters (4)
  • Grid resolutions N_curr=6, N_short=4, N_long=2 = 6/4/2
    Hand-chosen; ablation only varies current grid 6 vs 4 (OS drops from 72.9 to 43.4), so the headline result depends on this choice.
  • History lengths n_short=2, n_long=16 = 2/16
    Hand-chosen; no ablation reported for history length or resolution split.
  • Action chunk length N_a=4 = 4
    Hand-chosen; the model predicts four consecutive actions per call; no ablation.
  • HSV-to-color-name lookup table = n/a
    Hand-defined mapping from HSV to standard color names; no details or sensitivity analysis.
axioms (4)
  • domain assumption Semantic segmentation maps are available or can be predicted with sufficient accuracy in evaluation environments.
    Section 3.2: 'The semantic segmentation map ... can be obtained directly from the datasets or using pre-trained semantic segmentation model.' The R2R/RxR experiments never state which; if ground truth is used, comparison is privileged.
  • domain assumption The 6x6/4x4/2x2 grid text preserves enough spatial information for four-step action selection.
    Section 3.2/3.3; central mechanism; paper's own conclusion admits loss of shape/texture/fine geometry.
  • domain assumption The discrete action vocabulary (stop, turn +/-15 degrees, move 25 cm) is sufficient to solve VLN-CE and transfer to a real robot.
    Section 3.1 defines the action set; no analysis of discretization error is given.
  • domain assumption Qwen3-Embedding-0.6B can serve as a reasoning backbone for sequential decisions.
    Section 3.3 adopts Qwen3 Embedding and attaches classification heads; the paper repeatedly claims LLM reasoning, but an embedding model is a text encoder, not a generative reasoner.

pith-pipeline@v1.3.0-alltime-deepseek · 13328 in / 18072 out tokens · 172894 ms · 2026-08-02T17:10:57.679988+00:00 · methodology

0 comments
read the original abstract

Vision-Language Navigation (VLN) requires an embodied agent to navigate complex environments by following natural language instructions, which typically demands tight fusion of visual and language modalities. Existing VLN methods often convert raw images into visual tokens or implicit features, requiring large-scale visual pre-training and suffering from poor generalization under environmental variations (e.g., lighting, texture). To address these issues, we propose SOL-Nav (Structured Observation Language for Navigation), a novel framework that translates egocentric visual observations into compact structured language descriptions for efficient and generalizable navigation. Specifically, we divide RGB-D images into a N*N grid, extract representative semantic, color, and depth information for each grid cell to form structured text, and concatenate this with the language instruction as pure language input to a pre-trained language model (PLM). Experimental results on standard VLN benchmarks (R2R, RxR) and real-world deployments demonstrate that SOL-Nav significantly reduces the model size and training data dependency, fully leverages the reasoning and representation capabilities of PLMs, and achieves strong generalization to unseen environments.

Figures

Figures reproduced from arXiv: 2603.27577 by Daojie Peng, Fulong Ma, Jun Ma.

Figure 1
Figure 1. Figure 1: Pipeline of SOL-Nav. RGB-D observations are converted into structured tex￾tual descriptions with 2×2/4×4/6×6 multi-resolution grids (long/short-term history, current observation) encoding depth, semantic, and color information. The structured observation sequence, navigation instruction, and system description form a pure lan￾guage prompt, which is input to a LLM to predict a consecutive action block for t… view at source ↗
Figure 2
Figure 2. Figure 2: Structured Observation Language Prompt for LLM. The prompt integrates system description (Dsystem), structured observation (Ostructure), and task instruction (Itask) to provide clear system definition, structured observations, and explicit predic￾tion requirements for the language model [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Real-world Deployments. We conduct real-world navigation experiments in three distinct scenarios with varying environmental characteristics (Tea Area, Hall Stairs, Meeting Room), to comprehensively evaluate the robustness and generalization of SOL-Nav. 4.3 Real-World Deployments To validate the practical applicability of SOL-Nav, we carried out real-world deployment experiments using the Unitree Go2 robot … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Token-Wise Latent Streaming from Slow Reasoners to Fast Planners for Dynamic Vision Language Navigation

    cs.RO 2026-07 conditional novelty 6.0

    Streaming intermediate hidden states from a slow VLM to a fast flow-matching planner, token by token, improves dynamic social VLN success and reduces observation staleness.

  2. GeoSem-WAM: Geometry- and Semantic-Aware World Action Models

    cs.RO 2026-06 unverdicted novelty 5.0

    GeoSem-WAM adds geometric and semantic auxiliary prediction tasks to World Action Models during training to improve latent representations and action prediction accuracy while keeping inference efficient by avoiding e...

  3. AttenA+: Rectifying Action Inequality in Robotic Foundation Models

    cs.RO 2026-05 unverdicted novelty 5.0

    AttenA+ applies velocity-driven action attention to reweight training objectives toward kinematically critical low-velocity segments, yielding small benchmark gains on Libero and RoboTwin without added parameters.

  4. Towards Dual-Brain Minimal Sufficient Representation for Vision-Language Navigation

    cs.CV 2026-07 reject novelty 4.0

    A CP-decomposed, instruction-conditioned latent bottleneck (CompactNav) improves VLN-CE success rate by about 2% over prior state of the art on two benchmarks.

  5. AttenA+: Rectifying Action Inequality in Robotic Foundation Models

    cs.RO 2026-05 unverdicted novelty 4.0

    AttenA+ reweights action training objectives in VLA and WAM models via inverse velocity attention to prioritize kinematically critical segments, yielding small benchmark gains.

Reference graph

Works this paper leans on

61 extracted references · 22 linked inside Pith · cited by 4 Pith papers

  1. [1]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., Wang, L.: Etp- nav: Evolving topological planning for vision-language navigation in contin- uous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  2. [2]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3674–3683 (2018)

    2018

  3. [3]

    arXiv preprint arXiv:2004.05150 (2020)

    Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150 (2020)

    Pith/arXiv arXiv 2004

  4. [4]

    arXiv preprint arXiv:2505.06111 (2025)

    Bu, Q., Yang, Y., Cai, J., Gao, S., Ren, G., Yao, M., Luo, P., Li, H.: Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:1709.06158 (2017)

    Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017)

    Pith/arXiv arXiv 2017

  6. [6]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Chen, J., Lin, B., Xu, R., Chai, Z., Liang, X., Wong, K.Y.: Mapgpt: Map- guided prompting with adaptive path planning for vision-and-language nav- igation. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 9796–9810 (2024)

    2024

  7. [7]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, K., Chen, J.K., Chuang, J., Vázquez, M., Savarese, S.: Topologi- cal planning with transformers for vision-and-language navigation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11276–11286 (2021)

    2021

  8. [8]

    ArXivabs/2210.07506(2022)

    Chen, P., Ji, D., Lin, K.L.C., Zeng, R., Li, T.H., Tan, M., Gan, C.: Weakly- supervised multi-granularity map learning for vision-and-language naviga- tion. ArXivabs/2210.07506(2022)

    Pith/arXiv arXiv 2022

  9. [9]

    Advances in Neural Information Processing Systems35, 38149–38161 (2022)

    Chen, P., Ji, D., Lin, K., Zeng, R., Li, T., Tan, M., Gan, C.: Weakly- supervised multi-granularity map learning for vision-and-language naviga- tion. Advances in Neural Information Processing Systems35, 38149–38161 (2022)

    2022

  10. [10]

    arXiv preprint arXiv:2412.04453 (2024)

    Cheng, A.C., Ji, Y., Yang, Z., Gongye, Z., Zou, X., Kautz, J., Bıyık, E., Yin, H., Liu, S., Wang, X.: Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    IEEE Transactions on Emerging Topics in Computational Intelligence6(2), 230–244 (2022)

    Duan, J., Yu, S., Tan, H.L., Zhu, H., Tan, C.: A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence6(2), 230–244 (2022)

    2022

  12. [12]

    arXiv preprint arXiv:2506.09839 (2025) 16 F

    Gao, C., Jin, L., Peng, X., Zhang, J., Deng, Y., Li, A., Wang, H., Liu, S.: Octonav: Towards generalist embodied navigation. arXiv preprint arXiv:2506.09839 (2025) 16 F. Author et al

    Pith/arXiv arXiv 2025

  13. [13]

    In: Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition

    Georgakis, G., Schmeckpeper, K., Wanchoo, K., Dan, S., Miltsakaki, E., Roth, D., Daniilidis, K.: Cross-modal map learning for vision and language navigation. In: Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition. pp. 15460–15470 (2022)

    2022

  14. [14]

    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp

    Georgakis, G., Schmeckpeper, K., Wanchoo, K., Dan, S., Miltsakaki, E., Roth, D., Daniilidis, K.: Cross-modal map learning for vision and language navigation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 15439–15449 (2022)

    2022

  15. [15]

    arXiv preprint arXiv:2411.05755 (2024)

    Goetting, D., Singh, H.G., Loquercio, A.: End-to-end navigation with vision language models: Transforming spatial reasoning into question-answering. arXiv preprint arXiv:2411.05755 (2024)

    Pith/arXiv arXiv 2024

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Hong, Y., Wang, Z., Wu, Q., Gould, S.: Bridging the gap between learning in discrete and continuous environments for vision-and-language naviga- tion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15439–15449 (2022)

    2022

  17. [17]

    Iclr 1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Iclr 1(2), 3 (2022)

    2022

  18. [18]

    In: The Twelfth International Conference on Learning Representations (2023)

    Islam, M.M., Gladstone, A., Islam, R., Iqbal, T.: Eqa-mx: Embodied ques- tion answering using multimodal expression. In: The Twelfth International Conference on Learning Representations (2023)

    2023

  19. [19]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kamath, A., Anderson, P., Wang, S., Koh, J.Y., Ku, A., Waters, A., Yang, Y., Baldridge, J., Parekh, Z.: A new path: Scaling vision-and-language nav- igation with synthetic instructions and imitation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10813–10823 (2023)

    2023

  20. [20]

    In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision

    Krantz, J., Gokaslan, A., Batra, D., Lee, S., Maksymets, O.: Waypoint mod- els for instruction-guided navigation in continuous environments. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15162–15171 (2021)

    2021

  21. [21]

    In: European conference on computer vision

    Krantz, J., Lee, S.: Sim-2-sim transfer for vision-and-language navigation in continuous environments. In: European conference on computer vision. pp. 588–603. Springer (2022)

    2022

  22. [22]

    In: European Conference on Computer Vision

    Krantz, J., Wijmans, E., Majumdar, A., Batra, D., Lee, S.: Beyond the nav-graph: Vision-and-language navigation in continuous environments. In: European Conference on Computer Vision. pp. 104–120. Springer (2020)

    2020

  23. [23]

    In: European Conference on Computer Vision (2020),https : / / api

    Krantz, J., Wijmans, E., Majumdar, A., Batra, D., Lee, S.: Beyond the nav-graph: Vision-and-language navigation in continuous environments. In: European Conference on Computer Vision (2020),https : / / api . semanticscholar.org/CorpusID:214802389

    2020

  24. [24]

    arXiv preprint arXiv:2010.07954 (2020)

    Ku, A., Anderson, P., Patel, R., Ie, E., Baldridge, J.: Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954 (2020)

    Pith/arXiv arXiv 2010

  25. [25]

    Scientific Reports15(1), Abbreviated paper title 17 40854 (nov 2025).https://doi.org/10.1038/s41598- 025- 24576- 6, https://doi.org/10.1038/s41598-025-24576-6

    Li, X., Pu, X., Ling, W., et al.: Yolo-sam: an end-to-end framework for effi- cient real time object detection and segmentation. Scientific Reports15(1), Abbreviated paper title 17 40854 (nov 2025).https://doi.org/10.1038/s41598- 025- 24576- 6, https://doi.org/10.1038/s41598-025-24576-6

    doi:10.1038/s41598- 2025

  26. [26]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    Lin, B., Nie, Y., Wei, Z., Chen, J., Ma, S., Han, J., Xu, H., Chang, X., Liang, X.: Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    2025

  27. [27]

    IEEE/ASME Transactions on Mechatronics (2025)

    Liu, Y., Chen, W., Bai, Y., Liang, X., Li, G., Gao, W., Lin, L.: Aligning cyber space with physical world: A comprehensive survey on embodied ai. IEEE/ASME Transactions on Mechatronics (2025)

    2025

  28. [28]

    arXiv preprint arXiv:2406.04882 (2024)

    Long, Y., Cai, W., Wang, H., Zhan, G., Dong, H.: Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04882 (2024)

    Pith/arXiv arXiv 2024

  29. [29]

    arXiv preprint arXiv:2508.02046 (2025)

    Luo, Z., Yan, W., Gong, J., Wang, M., Zhang, Z., Wang, X., Xie, Y., Tan, X.: Navimaster: Learning a unified policy for gui and embodied navigation tasks. arXiv preprint arXiv:2508.02046 (2025)

    Pith/arXiv arXiv 2025

  30. [30]

    ACM Transactions on Human-Robot Interaction12(3), 1–39 (2023)

    Mavrogiannis, C., Baldini, F., Wang, A., Zhao, D., Trautman, P., Stein- feld, A., Oh, J.: Core challenges of social robot navigation: A survey. ACM Transactions on Human-Robot Interaction12(3), 1–39 (2023)

    2023

  31. [31]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W.Y., Shen, C., Hengel, A.v.d.: Reverie: Remote embodied visual referring expression in real indoor environments. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9982–9991 (2020)

    2020

  32. [32]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    Qiao, Y., Lyu, W., Wang, H., Wang, Z., Li, Z., Zhang, Y., Tan, M., Wu, Q.: Open-nav: Exploring zero-shot vision-and-language navigation in con- tinuous environment with open-source llms. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 6710–6717. IEEE (2025)

    2025

  33. [33]

    arXiv preprint arXiv:2109.08238 (2021)

    Ramakrishnan, S.K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A.X., et al.: Habitat-matterport3ddataset(hm3d):1000large-scale3denvironmentsfor embodied ai. arXiv preprint arXiv:2109.08238 (2021)

    Pith/arXiv arXiv 2021

  34. [34]

    In: Proceedings of the 2021 conference on empirical methods in natural language processing

    Raychaudhuri, S., Wani, S., Patel, S., Jain, U., Chang, A.: Language-aligned waypoint(law)supervisionforvision-and-languagenavigationincontinuous environments. In: Proceedings of the 2021 conference on empirical methods in natural language processing. pp. 4018–4028 (2021)

    2021

  35. [35]

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks (2024)

    2024

  36. [36]

    In: Proceedings of the IEEE/CVF international con- ference on computer vision

    Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al.: Habitat: A platform for embodied ai research. In: Proceedings of the IEEE/CVF international con- ference on computer vision. pp. 9339–9347 (2019)

    2019

  37. [37]

    IEEE Transactions on Pattern Analysis and Machine Intelligence45(10), 11948–11960 (2023) 18 F

    Tan, S., Ge, M., Guo, D., Liu, H., Sun, F.: Knowledge-based embodied question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence45(10), 11948–11960 (2023) 18 F. Author et al

    2023

  38. [38]

    arXiv preprint arXiv:2505.23189 (2025)

    Wang, S., Zhang, J., Li, M., Liu, J., Li, A., Wu, K., Zhong, F., Yu, J., Zhang, Z., Wang, H.: Trackvla: Embodied visual tracking in the wild. arXiv preprint arXiv:2505.23189 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    arXiv preprint arXiv:2508.02549 (2025)

    Wang, S., Wang, Y., Fan, Z., Wang, Y., Chen, M., Wang, K., Su, Z., Li, W., Cai, X., Jin, Y., et al.: Monodream: Monocular vision-language navigation with panoramic dreaming. arXiv preprint arXiv:2508.02549 (2025)

    arXiv 2025

  40. [40]

    In: Proceedings of the IEEE/CVF In- ternational conference on computer vision

    Wang, Z., Li, X., Yang, J., Liu, Y., Jiang, S.: Gridmm: Grid memory map for vision-and-language navigation. In: Proceedings of the IEEE/CVF In- ternational conference on computer vision. pp. 15625–15636 (2023)

    2023

  41. [41]

    arXiv preprint arXiv:2406.09798 (2024)

    Wang, Z., Li, X., Yang, J., Liu, Y., Jiang, S.: Sim-to-real transfer via 3d feature fields for vision-and-language navigation. arXiv preprint arXiv:2406.09798 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., Qiao, Y.: Scaling data generation in vision-and-language navigation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 12009–12020 (2023)

    2023

  43. [43]

    arXiv preprint arXiv:2512.08186 (2025)

    Wei, M., Wan, C., Peng, J., Yu, X., Yang, Y., Feng, D., Cai, W., Zhu, C., Wang, T., Pang, J., et al.: Ground slow, move fast: A dual-system founda- tion model for generalizable vision-and-language navigation. arXiv preprint arXiv:2512.08186 (2025)

    arXiv 2025

  44. [44]

    arXiv preprint arXiv:2507.05240 (2025)

    Wei, M., Wan, C., Yu, X., Wang, T., Yang, Y., Mao, X., Zhu, C., Cai, W., Wang, H., Chen, Y., et al.: Streamvln: Streaming vision-and-language navigation via slowfast context modeling. arXiv preprint arXiv:2507.05240 (2025)

    Pith/arXiv arXiv 2025

  45. [45]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D., Batra, D.: Embodied question answering in photorealistic en- vironments with point cloud perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6659–6668 (2019)

    2019

  46. [46]

    Neural Computing and Applications36(7), 3291– 3316 (2024)

    Wu, W., Chang, T., Li, X., Yin, Q., Hu, Y.: Vision-language navigation: a survey and taxonomy. Neural Computing and Applications36(7), 3291– 3316 (2024)

    2024

  47. [47]

    In: Neural Information Processing Systems (NeurIPS) (2021)

    Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Seg- former: Simple and efficient design for semantic segmentation with trans- formers. In: Neural Information Processing Systems (NeurIPS) (2021)

    2021

  48. [48]

    arXiv preprint arXiv:2510.18840 (2025)

    Xing, L., Wang, A.J., Yan, R., Qu, H., Li, Z., Tang, J.: See the text: From tokenization to visual reading. arXiv preprint arXiv:2510.18840 (2025)

    arXiv 2025

  49. [49]

    arXiv preprint arXiv:2505.09388 (2025)

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  50. [50]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yu, L., Chen, X., Gkioxari, G., Bansal, M., Berg, T.L., Batra, D.: Multi- target embodied question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6309–6318 (2019)

    2019

  51. [51]

    arXiv preprint arXiv:2406.10721 (2024) Abbreviated paper title 19

    Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., Fox, D.: Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721 (2024) Abbreviated paper title 19

    Pith/arXiv arXiv 2024

  52. [52]

    Advances in neural information processing systems33, 17283–17297 (2020)

    Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al.: Big bird: Transformers for longer sequences. Advances in neural information processing systems33, 17283–17297 (2020)

    2020

  53. [53]

    Zhang, J., Wang, K., Wang, S., Li, M., Liu, H., Wei, S., Wang, Z., Zhang, Z., Wang, H.: Uni-navid: A video-based vision-language-action model for unifyingembodiednavigationtasks.arXivpreprintarXiv:2412.06224(2024)

    Pith/arXiv arXiv 2024

  54. [54]

    arXiv preprint arXiv:2402.15852 (2024)

    Zhang, J., Wang, K., Xu, R., Zhou, G., Hong, Y., Fang, X., Wu, Q., Zhang, Z., Wang, H.: Navid: Video-based vlm plans the next step for vision-and- language navigation. arXiv preprint arXiv:2402.15852 (2024)

    Pith/arXiv arXiv 2024

  55. [55]

    arXiv preprint arXiv:2502.13451 (2025)

    Zhang, L., Hao, X., Xu, Q., Zhang, Q., Zhang, X., Wang, P., Zhang, J., Wang, Z., Zhang, S., Xu, R.M.: A novel memory representation via anno- tated semantic maps for vlm-based vision-and-language navigation. arXiv preprint arXiv:2502.13451 (2025)

    Pith/arXiv arXiv 2025

  56. [56]

    arXiv preprint arXiv:2508.04598 (2025)

    Zhang, L., Hao, X., Tang, Y., Fu, H., Zheng, X., Wang, P., Wang, Z., Ding, W., Zhang, S.: Nava3: Understanding any instruction, navigating anywhere, finding anything. arXiv preprint arXiv:2508.04598 (2025)

    Pith/arXiv arXiv 2025

  57. [57]

    arXiv preprint arXiv:2506.05176 (2025)

    Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., et al.: Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 (2025)

    Pith/arXiv arXiv 2025

  58. [58]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zheng, D., Huang, S., Zhao, L., Zhong, Y., Wang, L.: Towards learn- ing a generalist model for embodied navigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13624–13634 (2024)

    2024

  59. [59]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhou, G., Hong, Y., Wu, Q.: Navgpt: Explicit reasoning in vision-and- language navigation with large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 7641–7649 (2024)

    2024

  60. [60]

    arXiv preprint arXiv:2108.04097 (2021)

    Zhu, F., Zhu, Y., Lee, V., Liang, X., Chang, X.: Deep learning for embodied vision navigation: A survey. arXiv preprint arXiv:2108.04097 (2021)

    Pith/arXiv arXiv 2021

  61. [61]

    In: 2017 IEEE international conference on robotics and au- tomation (ICRA)

    Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., Farhadi, A.: Target-driven visual navigation in indoor scenes using deep reinforce- ment learning. In: 2017 IEEE international conference on robotics and au- tomation (ICRA). pp. 3357–3364. IEEE (2017)

    2017