{"id":"980a47a4-61f9-4617-8240-2057389f8eea","arxiv_id":"2606.10577","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A tool-calling harness replaces learned waypoints with pixel-level action, on-demand depth, and selective memory, achieving 55% SR and 48.41% SPL zero-shot on R2R-CE with GPT-5.5.","lead":"AgenticNav lets a vision-language model drive a robot through unseen homes by pointing at pixels, asking distances, and recalling earlier views—no trained waypoint predictor. On the standard R2R-CE navigation benchmark it reports 55% success with GPT-5.5, 11 points above a same-backbone waypoint-based system.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run 100-episode evaluation leaves the 11-point SOTA gain within plausible noise; paired per-episode test required.","rationale":"The reader's weakest_assumption concerns semantic grounding of pixel selection (whether the VLM picks an instruction-relevant floor pixel). That is a plausible mechanism-level risk, but it is secondary: even if semantic grounding is imperfect, the central empirical claim could still hold if the harness outperforms baselines. The more load-bearing issue is statistical and procedural: the entire SOTA claim rests on a single 100-episode run with no variance, paired per-episode results, or determinism statement. The reader's rationale does mention 'single runs without variance' but the formal weakest_assumption does not. Since the stress-test asks for the single most load-bearing concern about the central claim, I identify the unreliability of the headline numerical comparison. The reader's verdict CONDITIONAL already accommodates this (missing variance and reproducibility), so I recommend UNCHANGED. If the paired test reveals the gain is not significant, the verdict should move to REJECT; if the test supports it, CONDITIONAL can lift to ACCEPT, but the missing code/prompts still require conditional release.","tokens_in":10582,"tokens_out":8422,"duration_ms":92417,"concrete_test":"Request from the authors the per-episode success/failure vectors for AgenticNav-GPT-5.5 and SmartWay-GPT-5.5* on the same 100 R2R-CEval episodes and run an exact McNemar paired test on discordant outcomes. Also rerun AgenticNav-GPT-5.5 and the SmartWay reproduction with five different VLM sampling temperatures/seeds, reporting mean ± std SR and SPL. If the paired difference 95% confidence interval excludes zero and the lower bound exceeds 5 points, the SOTA claim survives; otherwise the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AgenticNav establishes new SOTA zero-shot performance on R2R-CE rests entirely on a single run of 100 episodes (Section 4.1). With n=100 and success rates near 50%, the standard error is roughly 5 percentage points; the headline 55% vs 44% (Δ=11 in Table 1) and the ablation gaps (55 vs 50, 55 vs 42, 55 vs 41 in Table 3) are all within 1–2 standard errors of zero. The paper does not report variance or state whether VLM decoding is deterministic (e.g., temperature 0). If API calls are stochastic, a single run cannot substantiate the claimed superiority of the action tool over the SmartWay reproduction—which is itself an author-run baseline using a trained waypoint predictor, so mismatches in prompts, predictor thresholds, or episode bookkeeping could produce the gap. The failure analysis in Section 6 shows most failures are VLM decisions, but that does not address whether the 11-point gain is reproducible. Thus the load-bearing premise that the harness, not chance or baseline underperformance, drives the results is not yet evidenced.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AgenticNav proposes a zero-shot VLN-CE harness that replaces the learned waypoint predictor with a tool-calling interface. The VLM selects target pixels in panoramic RGB views, queries metric depth on demand, and selectively recalls past visual observations from a compact memory; the harness converts the pixel choice into a continuous motion command via deterministic depth unprojection and a geometric safety check. The paper reports 55% SR / 48.41% SPL on the R2R-CEval unseen 100-episode protocol with GPT-5.5, claiming state-of-the-art zero-shot performance under the same VLM backbone, and reports real-world results on 30 episodes. Ablations attribute gains to the action tool, the depth tool, and the selective recall mechanism.","tokens_in":10895,"tokens_out":3811,"duration_ms":46617,"significance":"If the results are reproducible, the contribution is valuable: the design is genuinely waypoint-free, keeps geometric execution deterministic, and does not fit free parameters to the benchmark. The same-backbone SmartWay comparison and the ablation design are sensible and internally consistent. The paper also ships a real-world validation, which is uncommon. However, the central SOTA claim rests on a single run of 100 simulated episodes and 30 real episodes, and several key comparisons are within plausible sampling noise; the load-bearing premise that VLM pixel selection gives semantically correct navigable targets is not independently verified. The idea is promising, and the deterministic grounding is a strength, but the current evidence is not yet sufficient for a definitive accept.","major_comments":[{"comment":"The central claim of new SOTA performance is supported by a single run on 100 episodes. With SR near 50%, the standard error of the difference between the 55% AgenticNav and 44% SmartWay reproduction is roughly 7 percentage points, so the headline 11-point gain is within 1.6 standard errors. Several ablation deltas in Table 3 (55 vs 50, 55 vs 51) are within 1 standard error, and even the larger deltas are borderline. The paper does not report whether VLM decoding is deterministic (e.g., temperature 0), nor any confidence intervals, variance, or paired per-episode statistics. Please report the number of runs, paired McNemar/Wilcoxon tests, and the decoding temperature. Without this, the claimed superiority of the harness over the SmartWay reproduction is not yet established.","section":"§4.1, Tables 1 and 3"},{"comment":"The method relies on the VLM selecting an instruction-relevant pixel whose back-projected depth corresponds to a navigable floor location. The safety check only rejects motions whose swept corridor contains body-height depth points; it does not verify that the selected pixel is semantically correct or lies on the floor. A geometrically safe but semantically wrong pixel can move the robot in the wrong direction. The failure analysis in §6 attributes 88.9% of simulation failures to VLM decisions, but this does not quantify how often the action tool receives a wrong-pixel selection, how often Reselect corrects it, or how often depth queries mislead the VLM. Please provide a breakdown of action-tool failure modes and the rate of Reselect feedback.","section":"§3.4 Action Tool, §6 Limitations"},{"comment":"The main comparison is against 'SmartWay-GPT-5.5*', which is the authors' own reproduction. The paper does not release the exact prompt, waypoint-predictor checkpoint and thresholds, stop criteria, or episode bookkeeping used in this reproduction. Because the headline gain is computed relative to this author-run baseline, the comparison cannot be independently verified. Please provide the reproduction configuration, code, or at minimum a detailed protocol describing how the baseline's waypoint predictor and stop decision were instantiated on the same 100 episodes.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"The nDTW and NE columns are concatenated as '63.415.19' in several rows; missing column separation makes the table hard to read.","section":"Tables 1 and 3"},{"comment":"The notation 's5×5_i' is introduced without a clear definition of the tuple structure. It later says 'minimum, mean, and median depth' but the notation is not explicitly tied to those quantities.","section":"§3.2"},{"comment":"The protocol is described as 'the exact same 100 R2R-CEval unseen episodes' from [3,4,5]. Please clarify whether these are a standard official split or a custom subset, and whether the episode IDs are available for reproducibility.","section":"§4.1"},{"comment":"Real-world results are based on 30 episodes with no confidence intervals or per-scene significance tests. While presented as a demonstration, the 23.3% to 46.7% overall SR difference should be interpreted with caution; please state the sample size limitation explicitly.","section":"§4.2"},{"comment":"The safety corridor condition in Eq. (1) is defined for the proposed forward motion, but it is not stated how the robot's final orientation or the turning trajectory is checked. Please clarify whether the swept volume accounts for the rotation phase before translation.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"I see this as a promising engineering contribution with a clean tool-calling formulation and deterministic grounding, but the evaluation currently does not support the strength of the SOTA claim. The single-run 100-episode protocol is the main risk; paired per-episode statistics and additional seeds would substantially raise confidence. I would also encourage the editor to request code or detailed reproduction settings for the SmartWay baseline, since the headline comparison depends on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead AgenticNav. The core idea is genuinely new: instead of feeding the VLM a set of learned waypoints, it exposes the raw RGB panorama as a pixel-selection action tool, lets the VLM query depth at specific pixels on demand, and gives it a compact map plus selective recall rather than an ever-growing history. Removing the trained waypoint predictor while improving same-backbone SR from 44% to 55% is a real result if it holds. The ablations are sensible and point the right way: replacing the action tool with a waypoint predictor costs 5 points, depth tool removal costs 13, and the map+recall combination beats the map alone by 4. The deterministic geometry in the action tool (depth unprojection, safety sweep) is clean, and the authors deserve credit for a fair same-backbone comparison against SmartWay and for making clear EvoNav is not episodic zero-shot.\n\nThe soft spots are mostly about evidence. The entire headline claim rests on one run of 100 episodes. With n=100 and success rates near 50%, the standard error is roughly 5 points, so an 11-point gain is about two standard errors. The ablation gaps of 4–13 points are similarly within the noise band. No variance, no paired episode-by-episode comparison, no statement about whether the VLM API is deterministic. The same-backbone SmartWay reproduction is author-run and not independently verifiable without code. The authors also don't release prompts, geometry constants, or the safety-check parameters, which are the actual 'harness' and are needed for reproduction. The real-world set of 30 episodes is too small to support the claim that the method doubles SmartWay's SR; it's suggestive at best.\n\nThe paper's own failure analysis is honest—it blames most failures on VLM decision errors—and that is consistent with the concern that pixel-level action selection is only as good as the VLM's ability to pick a semantically correct pixel. The safety check prevents geometric collisions but does nothing to prevent 'wrong direction but safe' motion. That's a real limitation, but the paper doesn't hide it.\n\nOverall, this is a solid, interesting design with a plausible but under-evidenced empirical payoff. It belongs in a serious peer-review process to force the authors to add runs, release the harness, or at least run paired comparisons. I'd send it to review, and I'd tell the reviewers to focus on reproducibility, not on the architecture.","headline":"Clean tool-calling redesign of zero-shot VLN-CE, but the 11-point SOTA gain is a single-run result on 100 episodes and needs more evidence before I'd trust it.","tokens_in":11383,"tokens_out":3051,"would_cite":true,"duration_ms":28411,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rethinking zero-shot navigation as a VLM tool-calling harness lifts success rate to 55% on the R2R-CE benchmark.","keywords":["zero-shot VLN-CE","vision-and-language navigation","embodied agent harness","tool calling","pixel-level action grounding","depth query","agentic memory","R2R-CE benchmark"],"falsifier":"Construct episodes where the target is visible but a geometrically safe, semantically wrong floor pixel lies along the instruction path (e.g., a glass partition or a floor region on the wrong side of a boundary). If a VLM-equipped harness repeatedly selects that safe-but-wrong pixel and fails, while waypoint-predictor baselines succeed, the claim that pixel-level action grounding is sufficient collapses. Alternatively, an offline analysis of failed episodes: if the agent's selected pixels are typically semantically wrong but geometrically safe, that confirms the bottleneck.","tokens_in":10497,"feed_emoji":"🧭","tokens_out":6293,"duration_ms":66399,"temperature":0.7,"pith_summary":"The paper claims that the bottleneck in zero-shot vision-and-language navigation is no longer just the vision-language model but the interface between the model and the environment. Rather than letting a trained waypoint predictor propose a small set of movement options, AgenticNav gives the VLM three callable tools: pick a pixel in any panorama view to move toward, query precise metric depth at any pixel on demand, and selectively recall past visual observations while a compact map image tracks the trajectory. On the R2R-CE benchmark, this design reaches 55% success rate and 48.41% SPL with the GPT-5.5 VLM—11 points and 13.37 points above the same-backbone waypoint-predictor baseline—without any trained navigation module. The ablation experiments attribute the gains to all three tools, and real-robot trials show the approach transferring without extra training. The paper also acknowledges that most failures are VLM decision mistakes, so the harness inherits the VLM's semantic limits. If right, this reframes zero-shot embodied navigation as a harness design problem rather than a modelling problem.","feed_headline":"Tool-calling harness beats waypoint predictors in zero-shot navigation","feed_subtitle":"Giving a vision-language model pixel-level action, depth queries, and recall beats the same backbone by 11 success-rate points.","key_machinery":"The Action Tool—move to(k, u, v)—is the central object. The VLM selects a pixel in one of the panoramic RGB views; the harness unprojects that pixel into 3D using depth and camera calibration, converts it to a ground-plane bearing and executable forward distance, and runs a swept-corridor safety check over body-height back-projected points. This converts the VLM's visual selection into a concrete robot motion without a trained waypoint predictor, and is what lets the model choose any visible location rather than only proposed candidates.","core_discovery":"Zero-shot VLN-CE can be solved more effectively by an agentic harness that exposes action, depth, and memory as tools than by the dominant waypoint-predictor-plus-VLM design. The action tool back-projects the VLM's chosen pixel through camera intrinsics and depth to produce a continuous motion (turn angle and forward distance), with a geometric safety check that rejects motions whose swept corridor contains body-height points. The depth tool returns metric depth and local statistics for queried pixels, giving the VLM precise spatial evidence without dumping dense depth maps into its prompt. The memory module keeps a compact map image in the prompt plus a recall tool that retrieves previously","pith_inferences":["If harness design is as decisive as model scale, the next gains in zero-shot embodied agents may come from richer tool semantics—e.g., a tool that verifies the selected pixel is floor or landmark before acting, which could attack the paper's own 88.9%-of-failures VLM-decision bottleneck.","The pixel-to-motion pattern (select a visual target, back-project with depth, safety-check geometrically) is general and could transfer to other continuous visuomotor tasks, such as pick-and-place goal selection or semantic grasping, where VLMs can point but not control.","The on-demand numeric depth tool suggests a design principle for VLMs in metric-sensitive tasks: give the model a query interface to small quantitative facts rather than a dense sensor image; this is testable on other depth-reasoning benchmarks.","A testable extension: add a semantic confirmation step that asks the VLM to double-check the chosen pixel's category before executing, then measure whether the VLM-decision failure share drops."],"forward_implications":["Removing the trained waypoint predictor and letting the VLM pick a target pixel directly improves success, from 44% to 55% SR with the same VLM backbone.","Exposing depth as an on-demand pixel query is more effective for the VLM than feeding dense depth images: the ablation drops to 42% SR with no depth, and adding depth images only partially recovers the loss.","A compact trajectory map plus selective visual recall beats accumulating long histories and beats map-only memory, raising SR from 41% (no memory) to 51% (map only) and 55% (full memory).","The tool harness is not tied to one VLM: a 27B open model still reaches 35% SR, and a mid-tier commercial VLM outperforms prior zero-shot waypoint-predictor methods under the stricter episodic zero-shot setting.","Real-robot trials with fewer panorama views (1 or 4 versus 12) double the success rate over the waypoint-predictor baseline in 30 novel episodes spanning lab, office, and outdoor scenes."],"fun_headline_variants":["Tool-calling harness lets VLMs pick pixels, not waypoints","Zero-shot navigation: Tools beat waypoints by 11 points","AgenticNav: pixel-level control for zero-shot navigation","Depth on demand, memory on recall: tools for VLN"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the VLM can reliably pick the instruction-relevant pixel in the RGB view and that the depth at that pixel is navigable floor—the geometric safety check only rejects collisions in a swept corridor, so a semantically wrong but geometrically safe pixel will move the robot in the wrong direction.","fun_headline_variants_meta":{"raw":{"variants":["Tool-calling harness lets VLMs pick pixels, not waypoints","Zero-shot navigation: Tools beat waypoints by 11 points","AgenticNav: pixel-level control for zero-shot navigation","Depth on demand, memory on recall: tools for VLN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2319,"prompt_tokens":818,"completion_tokens":1501,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":1430}},"tokens_in":562,"tokens_out":1501,"duration_ms":12804,"temperature":1.0,"reasoning_tokens":1430,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:52:04.753158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct episodes where the target is visible but a geometrically safe, semantically wrong floor pixel lies along the instruction path (e.g., a glass partition or a floor region on the wrong side of a boundary). If a VLM-equipped harness repeatedly selects that safe-but-wrong pixel and fails, while waypoint-predictor baselines succeed, the claim that pixel-level action grounding is sufficient collapses. Alternatively, an offline analysis of failed episodes: if the agent's selected pixels are typically semantically wrong but geometrically safe, that confirms the bottleneck.","supporting_citations":[],"review_version":2}