{"id":"22231707-6557-4734-accc-d53212b83c31","arxiv_id":"2508.16654","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MSNav integrates dynamic map pruning, fine-tuned spatial reasoning (Qwen-Sp), and GPT-4o planning to improve zero-shot vision-and-language navigation on R2R and REVERIE.","lead":"MSNav is a modular system for zero-shot vision-and-language navigation that combines a pruned map memory, a fine-tuned spatial reasoning model, and an LLM planner. It reports state-of-the-art success rates on R2R and REVERIE benchmarks, alongside a new instruction-object-space dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"R2R zero-shot claim rests on Qwen-Sp fine-tuned on Matterport3D-derived I-O-S; the REVERIE switch to GPT-4o admits the risk but the headline R2R results never isolate it.","rationale":"The reader's weakest assumption is the same one I would flag: I-O-S is built from Matterport3D oracle paths and object annotations, and Qwen-Sp is used in R2R. This is not an accusation of bad faith; the paper itself says it avoids Qwen-Sp for REVERIE to prevent 'I-O-S data contamination,' which is an explicit admission that the R2R setting is not clean. Since Table 1 is the headline evidence for state-of-the-art zero-shot performance, the central claim depends on that comparison being fair. If Qwen-Sp has learned Matterport3D-specific object layouts, the Spatial Module is not purely zero-shot reasoning; it is providing privileged information from the evaluation domain, while MapGPT and NavGPT have no such training. The proposed test—swapping Qwen-Sp for GPT-4o on R2R-ZS—directly isolates whether the reported gain depends on the fine-tuned model. If the gain persists, the contamination concern is not load-bearing and the conditional verdict can be lifted. If it vanishes, the R2R zero-shot claim should be withdrawn or reframed as in-domain fine-tuned spatial reasoning. I also note, as a secondary consideration, that the numerical margins on R2R-ZS (n=216, ΔSR=5.1) and on the clean REVERIE-ZS (n=140, ΔSR=4.3) are within roughly one standard error of zero difference, so even the non-contaminated comparison is statistically fragile; this reinforces the conditional verdict.","tokens_in":13073,"tokens_out":8390,"duration_ms":93696,"concrete_test":"Rerun the R2R-ZS evaluation (Table 1) with the Spatial Module's Qwen-Sp replaced by unfine-tuned GPT-4o, keeping the Memory Module, Decision Module, YOLO-World, and all prompts identical. If SR/SPL fall from 50.9/42.6 toward MapGPT's 45.8/37.6 (or by more than ~3 points), the reported advantage is attributable to I-O-S fine-tuning on the Matterport3D domain, and the zero-shot comparison is invalid. Also inspect the I-O-S construction appendix to check whether any R2R-ZS scenes or instruction templates appear in the 25,694 training samples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MSNav is the best zero-shot LLM-based VLN system rests primarily on Table 1 (R2R-ZS: SR 50.9 vs MapGPT 45.8). In MSNav, the Spatial Module uses Qwen-Sp, fine-tuned on the I-O-S dataset. Section 3.2 states I-O-S is constructed from 'oracle paths in indoor environments' using 'the simulator's ground-truth object annotations' — i.e., Matterport3D, the same simulator used for R2R evaluation. Thus Qwen-Sp has been gradient-trained on instructions, object lists, and destination layouts drawn from the same environment distribution (and possibly the same instruction templates) as the R2R test episodes. The authors implicitly concede the risk: in 'Spatial Reasoning Model' they write that for REVERIE they 'avoid direct use of the fine-tuned model, instead leveraging its learned patterns to design prompts for commercial LLMs (e.g., GPT-4o)' to avoid 'I-O-S data contamination.' They do not take this precaution for R2R, the dataset on which they claim their largest gains. Consequently, the R2R-ZS comparison is not zero-shot in the same sense as MapGPT or NavGPT, which use no VLN-domain gradient training. If Qwen-Sp's advantage comes from memorizing Matterport3D object/layout regularities, the reported 5.1/5.0 point SR/SPL advantage is an artifact of train/test domain overlap rather than a generalizable spatial-reasoning gain. The cleaner REVERIE experiment replaces Qwen-Sp with GPT-4o and still shows a 4.3-point SR gain over MapGPT, but on only 140 samples and with a standard error of roughly 6%, so it does not by itself secure the headline R2R claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MSNav proposes a modular zero-shot vision-and-language navigation framework combining a dynamic topological Memory Module with node pruning, a Spatial Module that extracts task-relevant objects and infers destination layouts, and a Decision Module that uses GPT-4o for action selection. The authors introduce the I-O-S dataset of 28,414 instruction-object-space samples, fine-tune Qwen3-4B into Qwen-Sp, and report state-of-the-art zero-shot results on the R2R subset (SR 50.9, SPL 42.6), the full R2R validation unseen split (SR 46, SPL 40), and a REVERIE subset (SR 45.7, SPL 32.8), as well as improved object-extraction F1/NDCG over GPT-4o on the I-O-S test set. The central claim is that MSNav is the best zero-shot LLM-based VLN system on these benchmarks.","tokens_in":13531,"tokens_out":5231,"duration_ms":61918,"significance":"If the empirical claims hold, the paper makes several useful contributions: the modular decomposition of memory, spatial reasoning, and decision-making is sensible; the dynamic map pruning idea is a meaningful step toward reducing LLM context load; and the I-O-S dataset plus Qwen-Sp provides a concrete resource for spatial instruction understanding. The plug-and-play demonstration of the Spatial Module on NavGPT and MapGPT is also valuable. However, the headline zero-shot comparison is substantially weakened by the use of Qwen-Sp, which is fine-tuned on I-O-S data derived from the same Matterport3D simulation environment used to evaluate R2R. The paper itself acknowledges this contamination risk for REVERIE but not for R2R. In addition, the main numerical claims are based on small evaluation subsets without error bars or significance tests. The significance of the work is therefore conditional on resolving the domain-overlap issue and providing uncertainty quantification.","major_comments":[{"comment":"The R2R experiments are not zero-shot in the same sense as the baselines. Qwen-Sp is LoRA-fine-tuned on the I-O-S dataset, which is constructed from oracle paths and ground-truth object annotations obtained through the same simulator (Matterport3D) used to evaluate R2R. The Decision Module is conditioned on object lists and destination layouts produced by this fine-tuned model. The manuscript explicitly avoids this for REVERIE ('we used GPT-4o in the Spatial Module for a true zero-shot evaluation, avoiding I-O-S data contamination', §4.2), but does not apply the same precaution to R2R, where the largest gains are reported. As a result, the 5.1-point SR improvement over MapGPT in Table 1 may reflect memorization of test-environment regularities rather than generalizable spatial reasoning. The authors should either report the scene-level overlap between I-O-S training and the R2R evaluatio","section":"§3.2, Tables 1–2"},{"comment":"The main comparisons are based on small subsets without error bars or significance tests. R2R-ZS has 216 episodes and REVERIE-ZS has 140 episodes. On REVERIE-ZS the SR gap is 4.3 points (45.7 vs 41.4); with 140 episodes and success probabilities around 0.45, the standard error of the difference is roughly 6 percentage points, so the observed difference is not clearly significant at conventional levels. The authors should report bootstrap confidence intervals, paired tests where applicable, or evaluate on the full validation sets. As it stands, the 'state-of-the-art' claim is not supported with the reported precision.","section":"§4.2, Tables 1 and 3"},{"comment":"The Memory Module's pruning criterion depends on eight free choices: λ_t, λ_d, λ_f, λ_dist, θ_recent-visit, θ_age, N_remove, and t_start. None of these is given a concrete value in the manuscript, and no sensitivity analysis is reported. Similarly, the newly introduced ME metric uses α=0.25 without discussion of its influence. Because dynamic pruning is a core contribution and the comparisons hinge on it, the missing configuration and ablations make the results hard to reproduce and leave open the possibility that these choices were selected on the evaluation sets. Please provide the full hyperparameter setting and an ablation over the most influential parameters.","section":"§3.1, Eq. (1); §4.1, Eq. (4)"}],"minor_comments":[{"comment":"The long-distance analysis selects 25 cases 'exceeding 10 steps' from R2R-ZS. The selection criterion should be specified precisely (which instructions, how the threshold was chosen) to rule out cherry-picking, since the SR difference on this subset is much larger than on the full R2R-ZS set.","section":"§4.4"},{"comment":"The text says MSNav 'surpasses three trained and pretrained methods' but does not name them. Please identify which rows are being referred to, as some trained methods outperform MSNav on SR.","section":"Table 2"},{"comment":"The caption uses 'Nremove = 1' while Eq. (1) and the text use N_remove. Unify the notation.","section":"Figure 2"},{"comment":"The paper repeatedly refers to the Appendix for prompt designs, training details, and dataset construction. If the appendix is not part of the submitted manuscript, this limits reproducibility; please state where it is available.","section":"Appendix availability"},{"comment":"There are minor language issues, e.g., 'enhancing t he agent's ability' in Section 1 and 'we name it as R2R-ZS uniformly' in Section 4.2. A careful proofread is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The I-O-S contamination issue is the decisive technical concern. The REVERIE passage is an implicit admission that the authors know the fine-tuned spatial model is problematic for a zero-shot claim. I would require either a clean R2R experiment with a frozen spatial model or a clear re-scoping of the R2R results as a fine-tuned/domain-adapted setting before acceptance. The small subset sizes and unspecified pruning hyperparameters reinforce the need for more rigorous evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, know this: the paper's headline result—best zero-shot LLM navigation on R2R—rests on a spatial module fine-tuned on I-O-S, a dataset built from the same Matterport3D environments used for evaluation. The authors admit this for REVERIE by switching to GPT-4o, but not for R2R. So the R2R numbers aren't a clean zero-shot comparison.\n\nWhat's actually new: a dynamic map pruning rule that keeps LLM context bounded, a new dataset (I-O-S) linking instructions to objects and layouts, a fine-tuned Qwen-Sp for object extraction, and a modular integration that shows transferability into other frameworks. The pruning idea is sensible and the ablation suggests it helps on long trajectories. Qwen-Sp beats commercial LLMs on the I-O-S test set by a clear margin. That's concrete.\n\nSoft spots: the contamination issue is the big one. The I-O-S dataset is derived from oracle paths and ground-truth annotations in Matterport3D, so Qwen-Sp has learned environment-specific regularities. Using it in the Spatial Module for R2R means the agent's decisions are shaped by domain training. The REVERIE experiment avoids this but only has 140 samples, so the 4.3-point SR gain over MapGPT is within about one standard error. The R2R-ZS subset has 216 samples; a 5-point SR gain there is also within noise. No error bars anywhere. The pruning hyperparameters (thresholds, weights, N_remove, t_start) are not analyzed; we don't know how sensitive results are to choices. The ME metric's alpha=0.25 was tuned on the same data. No code or dataset released, so reproducibility is limited.\n\nIs the central argument sound? The architectural idea—separate memory, spatial reasoning, and decision—is reasonable and probably helps a bit. But the evidence that it's state-of-the-art zero-shot isn't firm. The contamination alone means the title claim is misleading.\n\nWho's this for? Researchers working on LLM-based navigation and memory management. The I-O-S dataset and the pruning formulation are worth knowing about, but treat the benchmark numbers with caution.\n\nRecommendation: send to peer review. The paper has a real contribution and the flaws are fixable: rerun R2R without the fine-tuned spatial model, add error bars, report sensitivity to pruning parameters, and release the dataset and code. A referee can help bring it to an honest state.","headline":"The headline zero-shot claim on R2R rests on a spatial module fine-tuned on the same Matterport3D data distribution used for evaluation, and the authors concede this for REVERIE but not for R2R; the architecture is plausible and the new dataset has value, but the empirical support is thin.","tokens_in":14008,"tokens_out":4415,"would_cite":false,"duration_ms":52774,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot navigator reaches 50.9% success","keywords":["zero-shot vision-and-language navigation","modular agent architecture","topological map pruning","spatial reasoning","instruction-object-space dataset","small language model fine-tuning","R2R benchmark","REVERIE benchmark"],"falsifier":"Run the R2R zero-shot evaluation with the spatial module replaced by a general-purpose LLM that has never seen the I-O-S data, keeping all other modules identical. If the success rate falls to the level of the map-based baseline, the fine-tuned model's exposure to the evaluation simulator is the source of the claimed gain. An even cleaner test would evaluate MSNav in a different indoor simulator not used to build I-O-S and check whether the 5-point SR advantage persists.","tokens_in":13017,"feed_emoji":"🧭","tokens_out":10707,"duration_ms":98197,"temperature":0.7,"pith_summary":"This paper claims that zero-shot vision-and-language navigation, normally handled by a single large language model reasoning end-to-end, becomes much more reliable when split into three cooperating modules: a dynamic map memory that prunes stale nodes, a spatial-reasoning module that extracts task-relevant objects and infers destination layouts, and an LLM decision module that chooses actions. To feed the spatial module, the authors construct an instruction-object-space dataset and fine-tune a small language model that outperforms larger commercial models at object-list extraction. On the R2R zero-shot benchmark the full system reaches 50.9% success and 42.6% SPL, ahead of the strongest prior zero-shot LLM planner; on the larger R2R validation-unseen split it reports 46% success, the best zero-shot result. If the findings hold, the modular decomposition and the fine-tuned spatial model offer a practical recipe for improving embodied instruction-following without training a navigation policy.","feed_headline":"Zero-shot navigator reaches 50.9% success","feed_subtitle":"Splitting memory, spatial reasoning, and planning beats single-LLM navigators on indoor benchmarks.","key_machinery":"The mechanisms: (1) dynamic map pruning, which scores visited nodes by staleness, connectivity, unexplored neighbors, and distance and removes the top-scored ones; (2) the Spatial Module, where a small model fine-tuned on the new Instruction-Object-Space dataset outputs ranked object lists and a destination layout, and an open-vocabulary detector highlights those objects in the best view; (3) an LLM Decision Module that reads instruction, map, layout, and enhanced view and picks the next action. The I-O-S dataset, built from oracle paths and object annotations in the same simulator used for evaluation, powers the fine-tuned spatial model.","core_discovery":"MSNav's claim: zero-shot VLN is bottlenecked not by planning but by asking one LLM to remember, ground, and reason at once. The paper separates these into three modules. A Memory Module maintains a topological graph of visited viewpoints and prunes nodes by a weighted score of staleness, connectivity, unexplored neighbors, and distance. A Spatial Module, built from a small model fine-tuned on a new instruction-object-layout dataset, outputs ranked object lists and a predicted destination layout; the object list drives an open-vocabulary detector. A Decision Module LLM reads instruction, map, layout, and observation, and picks the next viewpoint or stop. Results: R2R-ZS SR 50.9/SPL 42.6 (beat","pith_inferences":["The I-O-S training set is built from oracle paths and object annotations in the same simulator used for R2R evaluation, so the R2R gain may partly reflect memorization of the evaluation distribution; the paper itself switches to a general-purpose LLM on REVERIE to avoid I-O-S data contamination, and the analogous R2R control would replace the fine-tuned spatial model with a general-purpose LLM.","The hand-tuned pruning weights are a clear candidate for learning; the paper's own map-efficiency metric could serve as the training objective.","The modular recipe likely transfers to other embodied instruction tasks, such as outdoor delivery or household manipulation, whenever an equivalent instruction-object-layout annotation can be produced.","At $0.10-$0.14 per episode, the pruning mechanism keeps per-episode inference cost modest, but whether the accuracy gain is worth that cost at scale is an economic question the paper does not settle."],"forward_implications":["If the results hold, decomposing navigation into memory, spatial reasoning, and planning is a practical recipe that works without any learned policy or fine-tuned navigation model.","The pruning rule keeps prompt sizes manageable, and on the 25 longest R2R-ZS episodes the paper reports 44% success versus 24% for the map-based baseline with fewer steps.","The spatial module can be grafted onto other zero-shot planners: ablations show improvements when only its destination layout is appended to a non-map-based planner, and larger gains when added to a map-based one.","A small fine-tuned model at 4B parameters can beat much larger commercial LLMs on object extraction and ranking on the new dataset, suggesting a cheaper route to specialized spatial reasoning.","The new instruction-object-space dataset, with over 28,000 samples, may support orthogonal tasks such as instruction grounding and object search."],"supporting_citations":[{"why":"Supplies the R2R dataset and task definition, the primary benchmark for all navigation comparisons.","marker":"Anderson et al. 2018"},{"why":"Supplies the REVERIE dataset and task, the second benchmark used for complex long-range evaluation.","marker":"Qi et al. 2020"},{"why":"MapGPT baseline whose map-prompting design is adapted and which MSNav must beat on both benchmarks.","marker":"Chen et al. 2024"},{"why":"NavGPT, the first LLM-based zero-shot VLN agent; serves as a baseline and the starting point for spatial-module add-on experiments.","marker":"Zhou, Hong, and Wu 2023"},{"why":"DiscussNav, a multi-expert zero-shot baseline used for comparison.","marker":"Long et al. 2023"},{"why":"Qwen3-4B, the base model fine-tuned into Qwen-Sp for the spatial module.","marker":"Yang et al. 2025"},{"why":"Provides the open-vocabulary detector used to enhance visual observations with task-relevant objects.","marker":"Cheng et al. 2024"},{"why":"Gives the low-rank adaptation method used to fine-tune the spatial model efficiently.","marker":"Hu et al. 2022"},{"why":"Matterport3D simulator; source of oracle paths and object annotations for I-O-S, and the evaluation environment for R2R and REVERIE.","marker":"Chang et al. 2017"}],"fun_headline_variants":["Split reasoning beats single LLM in zero-shot navigation","Modular memory and spatial knowledge lift zero-shot navigation","Three-module agent achieves 50.9% SR in zero-shot VLN","Pruned map memory and spatial LLM boost zero-shot success","Small spatial model, LLM planner lift zero-shot VLN"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the fine-tuned spatial model does not simply memorize the test environments: its training data is extracted from oracle paths and object annotations in the same simulator used for evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Split reasoning beats single LLM in zero-shot navigation","Modular memory and spatial knowledge lift zero-shot navigation","Three-module agent achieves 50.9% SR in zero-shot VLN","Pruned map memory and spatial LLM boost zero-shot success","Small spatial model, LLM planner lift zero-shot VLN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3048,"prompt_tokens":815,"completion_tokens":2233,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2147}},"tokens_in":559,"tokens_out":2233,"duration_ms":18897,"temperature":1.0,"reasoning_tokens":2147,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:31:41.727334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the R2R zero-shot evaluation with the spatial module replaced by a general-purpose LLM that has never seen the I-O-S data, keeping all other modules identical. If the success rate falls to the level of the map-based baseline, the fine-tuned model's exposure to the evaluation simulator is the source of the claimed gain. An even cleaner test would evaluate MSNav in a different indoor simulator not used to build I-O-S and check whether the 5-point SR advantage persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Matterport3D simulator; source of oracle paths and object annotations for I-O-S, and the evaluation environment for R2R and REVERIE."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the R2R dataset and task definition, the primary benchmark for all navigation comparisons."}],"review_version":1}