{"id":"e6dc58a6-9702-4097-9f7f-a7343a569198","arxiv_id":"2506.10172","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A frozen Qwen2.5-VL model with prompt engineering and a two-frame egocentric view achieves 5% success on 20 R2R VLN-CE episodes, near the zero-movement baseline.","lead":"This paper presents a modular navigation system that pairs a frozen vision-language model with simple action planning and tests it on a standard indoor navigation benchmark. On the first 20 episodes, the system succeeds in only 5 percent of cases, barely above a do-nothing baseline, so the paper reads as an honest negative result.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'foundation' claim rests on a single successful episode out of 20 (5% SR), statistically indistinguishable from the zero-movement baseline, so the framework's core assumption is never validated.","rationale":"The reader's strongest claim correctly identifies that the empirical results contradict the abstract's positive framing. My stress-test focuses on the statistical weakness of that contradiction: with 20 episodes and one success, the result cannot even distinguish the framework from a no-op agent, so the central claim is unsupported rather than refuted. The reader's weakest_assumption—that a frozen VLM with two RGB frames, no depth, and no spatial memory can ground navigation—is related but not identical to my concern; even if that assumption were true, this evaluation could not reveal it. I agree with the REJECT verdict because a paper whose sole empirical result is at chance level does not establish a foundation for scalable navigation. However, the rejection is not based on internal inconsistency: the authors are transparent about failure and future work. The decisive defect is evidentiary: the claim 'lays a foundation' needs at least one positive, statistically meaningful signal—such as success above a matched random-action baseline on a full evaluation split—and the paper provides none. A future revision with full-split evaluation, matched baselines, and a success criterion specified in advance could make the claim testable. My recommendation is UNCHANGED because the reader's REJECT already reflects the absence of such evidence.","tokens_in":5571,"tokens_out":3642,"duration_ms":47345,"concrete_test":"Run the framework on the complete R2R VLN-CE val-unseen split (or at least a statistically powered subset of 200+ episodes) under the paper's 50-step, 256x256 settings, and include baselines under identical conditions: zero movement, random action selection, and a greedy DTG-reducing policy. Report per-episode DTG with confidence intervals and a binomial test comparing the agent's success count with the random-action baseline. If the 95% CI for SR includes the random-action SR, or if the DTG improvement over random action is not significant, the framework shows no navigation signal and the 'foundation' claim remains unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim that the modular framework 'lays a foundation for scalable and efficient navigation systems' is not supported by the only quantitative evidence, Section 4, Table 2. On the first 20 val-unseen episodes the agent achieves SR = SPL = 5% (one success) and DTG = 7.748 m, which is statistically indistinguishable from the zero-movement baseline (SR = 0%, DTG = 8.150 m). With n = 20, observing one success cannot reject the null hypothesis of no navigation skill; the 0.4 m average DTG difference is likewise within noise. The Table 1 baselines (BEVBert, ETPNav) are not evaluated under the paper's modified conditions—50-step cap and 256x256 resolution—so they provide no context for the framework's absolute numbers. Because the framework's contribution is precisely that a frozen VLM plus lightweight planner can replace learned spatial representations, the absence of any positive signal beyond chance leaves the main claim unverified. Section 4.1 further concedes that 'incorporating structured priors may still be critical,' which undercuts the claim that the modular decoupling itself is a viable foundation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a modular vision-and-language navigation framework that pairs a frozen Qwen2.5-VL-7B-Instruct model with hand-crafted prompting, a fixed-size history buffer, and a two-frame RGB visual input. The framework is evaluated on the first 20 trajectories of the R2R VLN-CE val-unseen split in Habitat-Lab, with a 50-step cap and 256x256 resolution. The authors report a success rate of 5% (one episode), an SPL of 5%, and an average distance-to-goal of 7.748 m, compared with 8.150 m for a zero-movement baseline. They conclude that although the framework struggles on unseen environments, the modular approach 'lays a foundation for scalable and efficient navigation systems.'","tokens_in":5766,"tokens_out":3894,"duration_ms":44833,"significance":"If validated, a plug-and-play navigation framework that works with a frozen vision-language model and lightweight planning would be practically valuable, since it would avoid fine-tuning and permit easy swapping of perception backbones. The manuscript has clear strengths: the modular decomposition is well described, the code is publicly released, and the authors are candid about the current limitations. However, the only quantitative evidence is too weak to support the central claim, and the comparison with published baselines is not valid under the stated experimental modifications. As a result, the paper does not currently establish its advertised contribution.","major_comments":[{"comment":"The central claim that the framework 'lays a foundation' is not supported by the reported results. With n=20 episodes, SR=5% corresponds to a single successful trajectory; the 95% binomial confidence interval for 1/20 successes extends from near zero to about 25%, so the data are compatible with a navigation success rate that is practically indistinguishable from the zero-movement baseline. The DTG reduction from 8.150 m to 7.748 m is 0.402 m, and no standard errors, confidence intervals, or per-episode variances are reported, so this difference cannot be distinguished from noise. The paper needs a proper statistical comparison or a substantially larger evaluation before any positive conclusion can be drawn.","section":"Section 4, Table 2"},{"comment":"The comparison to BEVBert and ETPNav is invalid because Table 1 reports published results on the full val-unseen split under the standard VLN-CE evaluation, while Table 2 uses only the first 20 trajectories, a 50-step cap, and a 256x256 camera resolution. These modifications are acknowledged in Section 4 but make the baselines non-comparable. Without evaluating BEVBert and ETPNav under identical conditions, Table 1 provides no context for interpreting the framework's absolute numbers and should not be used to support any relative-performance claim.","section":"Section 4, Tables 1 and 2"},{"comment":"The discussion concedes that 'incorporating structured priors may still be critical for achieving robust performance in complex environments,' which directly contradicts the abstract's claim that the modular approach 'lays a foundation for scalable and efficient navigation systems.' The single successful episode, the lack of statistical significance, and this concession together indicate that the core hypothesis—that a frozen VLM with lightweight planning suffices for navigation—is not verified by the paper's experiments.","section":"Section 4.1"}],"minor_comments":[{"comment":"The word 'Gleichzeitig' is German; replace it with 'Meanwhile' or 'At the same time.'","section":"Section 1, first paragraph"},{"comment":"The action space is described as four discrete actions even though VLN-CE is a continuous-environment benchmark; clarify whether actions are implemented as fixed turn and forward increments and specify the magnitudes.","section":"Section 3.4.2"},{"comment":"The zero-movement baseline is not defined in the text; state explicitly that it corresponds to the agent remaining at its starting position.","section":"Section 4, Table 2"},{"comment":"The GitHub URL contains spaces in the printed text; ensure the link is correctly formatted so that it is clickable and unambiguous.","section":"Section 4.1, Code Availability"},{"comment":"CLIP is described as an LVLM alongside Flamingo and Qwen; CLIP is a vision-language model but not a generative large vision-language model in the same sense, so it would be clearer to distinguish these categories.","section":"Section 2, Related Work"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a project report rather than a complete research article. The central positive claim is contradicted by the paper's own data, and the evaluation is too limited to support any conclusion beyond the observation that the current framework fails. Substantial new experiments and a fundamental reframing would be required before the findings could be considered for publication in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you follow VLM-based navigation: it is a short, honest negative result, but the abstract claims more than the data supports. The actual result, Section 4, is one success out of 20 val-unseen episodes (5% SR/SPL, DTG 7.748 m), statistically indistinguishable from the zero-movement baseline (DTG 8.150 m). The paper deserves credit for including that zero-movement baseline, releasing code, and explicitly conceding in Section 4.1 that structured priors may still be critical. There is no fitted model, no adaptation to the benchmark, no hidden circularity; it is a direct application of frozen Qwen2.5-VL with prompt engineering, history, and two-frame input.\n\nThe soft spots are real and load-bearing for the stated contribution. First, the headline claim in the abstract—a foundation for scalable and efficient systems—is not even weakly supported by the only quantitative evidence: with n=20 and one success you cannot reject the null of no navigation skill. Second, the Table 1 baselines (BEVBert, ETPNav) were evaluated under the standard VLN-CE protocol, not under this paper's modifications (50-step cap and 256x256 resolution), so those numbers provide no valid context. Third, there are no error bars, no ablations of history window or reflection, and no full-split evaluation. The framework's core assumption—that a frozen VLM plus lightweight planner can replace learned spatial representations—is tested, and it fails. The negative result is plausible and worth knowing about, but the paper does not establish a working system.\n\nWho is this for? A reader building frozen-VLM baselines might cite it as evidence that zero-shot Qwen2.5-VL is insufficient for R2R VLN-CE without structured priors. I would not bring it to a serious peer-reviewed venue now, since the evaluation is too thin and the baseline comparison is not apples-to-apples. My recommendation: do not send this to peer review in its current form. If the authors expand to the full val-unseen split, match baseline conditions, and reframe as a careful negative result with ablations, it would be worth a second look.","headline":"Honest but overclaimed negative result: one success in 20 episodes, baselines not matched, so the 'foundation' claim is unsupported.","tokens_in":6338,"tokens_out":2664,"would_cite":false,"duration_ms":33103,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tries to establish that a frozen vision-language model, prompted with two recent frames and a short history, can serve as the understanding module of a plug-and-play navigation system; on 20 unseen rooms it succeeded only 5% of…","keywords":["Vision-and-Language Navigation","frozen vision-language model","modular navigation framework","prompt engineering","VLN-CE","Room-to-Room benchmark","Matterport3D","topological priors"],"falsifier":"Take the same 20 val-unseen episodes and run the agent with the history buffer removed and one frame instead of two; if the success rate stays at 5 percent, the two-frame visual and reflection mechanism is not what carries the result. Then add a simple topological graph to the prompt; if success rises substantially, the no-structured-prior assumption is the failing link.","tokens_in":5325,"feed_emoji":"🧭","tokens_out":7317,"duration_ms":78902,"temperature":0.7,"pith_summary":"This paper proposes that vision-and-language navigation can be split into two separable parts: a large, frozen vision-language model that turns an instruction and two recent egocentric images into a suggested action plus a short reflection, and a lightweight planner that executes that suggestion without retraining. The intended payoff is a plug-and-play navigation module that can adopt better vision-language models as they appear while keeping planning fast and cheap. The authors build this system and test it on the first 20 unseen-environment episodes of the Room-to-Room benchmark in the continuous VLN-CE setting. They report a 5% success rate, an average distance to goal of 7.75 meters, and an SPL of 5%, essentially no better than the zero-movement baseline they include. Their discussion concludes that the frozen-model, no-map assumption may be too strong and that structured priors such as a topological graph might be necessary, which is the paper's own acknowledgment that the central hypothesis was not met in this evaluation.","feed_headline":"Frozen AI model navigates unseen rooms only 5 percent of the time","feed_subtitle":"A no-fine-tuning, plug-and-play navigation agent reached 5 percent success on 20 unseen episodes, suggesting maps and priors may still be…","key_machinery":"The load-bearing mechanism is the two-frame visual input $V = \\{I_{t-1}, I_t\\}$ combined with a structured history buffer whose entries are $(\\text{step}, \\text{action}, \\text{reflection})$ tuples; the frozen vision-language model consumes these plus a system prompt covering persona, action space, human common sense, and history, and outputs a JSON object containing one action and one reflection, which a lightweight planner parses and executes. The design intent is that temporal continuity across two frames and self-generated reflections provide enough grounding for action decisions without maps, depth, or learned spatial memory.","core_discovery":"The central claim is that a frozen general-purpose vision-language model (Qwen2.5-VL-7B-Instruct), prompted with a text instruction, two consecutive RGB frames, and a short history of actions and self-reflections, can act as the perception-and-language module of a navigation system while a separate lightweight module maps its output to discrete actions. The paper argues that this separation makes navigation adaptable: a better vision-language model can be swapped in, or a lighter planner substituted, without retraining the whole stack. The reported experiments are the test of that claim: on 20 val-unseen trajectories the framework reaches success only once, and the distance-to-goal reduction over doing nothing is about 0.4 meters. The authors attribute this to missing spatial structure and propose future integration of topological or geometric priors, which effectively concedes that the standalone frozen-vision-language-model formulation, as instantiated, does not yet deliver the claimed scalability.","pith_inferences":["The near-zero success on val-unseen suggests the bottleneck may be action grounding rather than instruction comprehension; a useful diagnostic would be to compare the agent's chosen actions with human or oracle actions in the first few steps of each episode.","The paper's stricter conditions (a 50-step cap and 256×256 resolution) make direct comparison with published baselines ambiguous, so a fair test would run the same baselines under identical settings.","The authors' own concession that structured priors may be critical implies that fully graph-free vision-language navigation with current frozen models may be out of reach, making hybrid designs that feed geometric priors into the prompt a natural next step.","The framework's modular interface would allow the same planner to be tested with different frozen vision-language models, providing a clean way to measure how much navigation ability comes from the model's pretraining versus the prompt."],"forward_implications":["If the modular separation works, the navigation policy can be upgraded by swapping in a newer vision-language model, with no retraining of the planner.","A lightweight planner could be exchanged for a different one, for example one with topological priors, without touching the language-understanding module.","Prompt-only adaptation would let the system take new instruction formats or new action spaces by editing the prompt rather than fine-tuning.","The two-frame temporal input and history buffer establish a reusable interface between perception and planning that later modules could build on.","If structured priors are added as the discussion suggests, they would enter as a separate module, preserving the decoupled design."],"supporting_citations":[{"why":"Defines the Room-to-Room instruction set and the continuous VLN-CE evaluation protocol, providing the 20 val-unseen episodes used.","marker":"[8]"},{"why":"Supplies the Matterport3D RGB-D indoor scenes the agent navigates.","marker":"[5]"},{"why":"Provides the Habitat simulator that executes discrete actions and reports positions for metrics.","marker":"[12]"},{"why":"The frozen vision-language model whose outputs the framework consumes.","marker":"[13]"},{"why":"Documents the Qwen2-VL line's grounding and resolution capabilities, supporting the choice of the vision-language model.","marker":"[14]"},{"why":"BEVBert is the stronger baseline whose val-unseen numbers the framework is measured against.","marker":"[2]"},{"why":"ETPNav is the second baseline, representing a coupled perception-planning architecture.","marker":"[3]"}],"fun_headline_variants":["Frozen VL model navigates only 1 in 20 unseen episodes","Modular navigation with frozen VL: 5% success, 0.4m gain","Plug-and-play VLN agent falls short: 5% on unseen rooms","No-fine-tuning navigation: 1 success out of 20 attempts","Vision-language nav framework stumbles in unfamiliar spaces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A frozen image-and-text model can decide the next navigational action from two recent camera frames, one instruction, and a short action-and-reflection history, with no map, depth, graph, or learned spatial memory.","fun_headline_variants_meta":{"raw":{"variants":["Frozen VL model navigates only 1 in 20 unseen episodes","Modular navigation with frozen VL: 5% success, 0.4m gain","Plug-and-play VLN agent falls short: 5% on unseen rooms","No-fine-tuning navigation: 1 success out of 20 attempts","Vision-language nav framework stumbles in unfamiliar spaces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000108,"raw_usage":{"total_tokens":1325,"prompt_tokens":1002,"completion_tokens":323,"prompt_tokens_details":{"cached_tokens":896},"prompt_cache_hit_tokens":896,"prompt_cache_miss_tokens":106,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":106,"tokens_out":323,"duration_ms":7193,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":896,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:32:02.341311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 20 val-unseen episodes and run the agent with the history buffer removed and one frame instead of two; if the success rate stays at 5 percent, the two-frame visual and reflection mechanism is not what carries the result. Then add a simple topological graph to the prompt; if success rises substantially, the no-structured-prior assumption is the failing link.","supporting_citations":[{"cited_title":"Beyond the nav-graph: Vision and language navigation in continuous environments","cited_arxiv_id":null,"evidence_quote":"Defines the Room-to-Room instruction set and the continuous VLN-CE evaluation protocol, providing the 20 val-unseen episodes used."},{"cited_title":"Qwen2.5-vl, 2025","cited_arxiv_id":null,"evidence_quote":"The frozen vision-language model whose outputs the framework consumes."},{"cited_title":"Bevbert: Multimodal map pre-training for language-guided navigation","cited_arxiv_id":null,"evidence_quote":"BEVBert is the stronger baseline whose val-unseen numbers the framework is measured against."},{"cited_title":"Etpnav: Evolving topo- logical planning for vision-language navigation in continu- ous environments","cited_arxiv_id":null,"evidence_quote":"ETPNav is the second baseline, representing a coupled perception-planning architecture."}],"review_version":1}