{"id":"739081c6-a62c-460a-ac01-07c0afb5ee52","arxiv_id":"2502.00114","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HAM-Nav uses pretrained vision-language models with adaptive visual prompting and missing-landmark prediction to navigate robots from hand-drawn maps without task-specific training.","lead":"This paper presents a system that lets a person sketch a map by hand and have a robot follow it using a vision-language model, even when the sketch is rough or incomplete. The authors report successful navigation on wheeled and legged robots in simulated and real environments, with better results than a voice-instruction baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on unmeasured VLM localization accuracy, yet the paper's own error analysis shows localization fails on sparse-label maps, so end-to-end success alone cannot support the generalization claim.","rationale":"The reader's weakest assumption and my concern are the same: the VLM-based position estimation from the composite visual prompt is the load-bearing mechanism, and it is never benchmarked independently. The paper's own failure-mode analysis in V-A5 explicitly identifies incorrect localization on sparse landmark labels as the common failure, which directly threatens the 'varying hand-drawing styles' part of the central claim. However, the paper does provide meaningful evidence: the ablation study shows a strong internal consistency signal (removing the Experience Manager drops success from 80% to 5%, and removing the pruned map drops it to 25%), and the real-world user study with 20 participants demonstrates the system can work in practice. These observations support CONDITIONAL rather than REJECT. My proposed oracle-localization test would settle whether the end-to-end numbers are actually explained by the mechanism the paper emphasizes, or whether other components compensate for a weak LE. Because the reader already flagged this concern and arrived at CONDITIONAL, my read does not change the verdict.","tokens_in":12366,"tokens_out":4139,"duration_ms":43052,"concrete_test":"Re-run the recorded trials and, at each timestep, compare the LE's selected node p* with the ground-truth robot position projected onto the topological map (nearest node within a defined tolerance). Report localization accuracy stratified by landmark density (low/medium/high as in Fig. 8), environment, and robot platform. Then run an oracle variant in which p* is replaced by the ground-truth node for the NPE while all other modules remain unchanged; if oracle SR exceeds the reported SR by more than 10 percentage points, LE is a demonstrated bottleneck and the central claim lacks direct support. Also include a trivial baseline (e.g., always selecting the goal-adjacent node) to confirm the LE is above chance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—zero-shot generalization across hand-drawing styles—rests on the Localization Engine (LE) correctly estimating the robot's topological node p* from the composite visual prompt (Eq. 5), because the Navigation Planning Engine's action selection (Eq. 6) is conditioned on p* and the same prompt. The paper never measures LE accuracy in isolation. Table II only reports end-to-end SR/SPL, and the V-A5 error analysis says the common failure mode is incorrect localization when hand-drawn maps have sparse landmark class labels. Sparse labeling is typical of natural freehand sketches and is exactly the variation the 'diverse hand-drawing styles' claim must cover. Without a per-step localization benchmark, the reported 80% simulated and 77% real-world success cannot be attributed to cross-modal map understanding; it could instead be carried by the NPE's coarse actions, the Experience Manager's repetition, or other modules while the LE is often wrong. If LE accuracy collapses on sparse-label maps, the headline generalization claim is overstated; if it is high, the concern is resolved. An oracle-localization control (feeding ground-truth p*) would quantify how much of the end-to-end success is actually bottlenecked by LE.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HAM-Nav, a modular architecture (TMG, SI, EM, VPG, PNPP, LE, NPE, NC) that uses pretrained vision-language models, Grounding DINO, and Grounded-SAM to convert a user's hand-drawn map into a pruned topological map and a visual prompt, from which a Localization Engine estimates the robot's position and a Navigation Planning Engine selects discrete actions. The authors claim that HAM-Nav is the first method to generalize across diverse environments and drawing styles without task-specific training, and they support this with an ablation study in two photorealistic simulated environments (indoor multi-floor workplace and outdoor construction site) using Jackal and Spot robots, reporting 80% success and SPL 0.712, plus a real-world user study reporting 77% success and SPL 0.703 against MapGPT's 64% and 0.523, along with SUS and NPS usability scores.","tokens_in":12493,"tokens_out":4183,"duration_ms":45733,"significance":"If the central claims are correct, the work is a meaningful step: it replaces metrically accurate maps with qualitative freehand sketches, works with realistic landmarks and multi-floor settings, and demonstrates that pretrained VLMs can be assembled into a zero-shot navigation pipeline. The ablation study is useful for identifying which modules matter, and the real-world user study addresses practical feasibility and usability. However, the evidence as presented does not yet establish the headline generalization claim: the localization bottleneck is not measured in isolation, no prior hand-drawn-map method is used as a baseline, and the success rates are reported without confidence intervals or significance tests.","major_comments":[{"comment":"The paper's central mechanism is the Localization Engine's cross-modal association of the robot view with the pruned topological map, yet LE accuracy is never evaluated as a standalone quantity. The NPE's action selection in Eq. (6) is conditioned on the LE's estimated p*, so end-to-end success cannot by itself attribute performance to map understanding. Section V-A5 states that incorrect localization is the common failure mode when hand-drawn maps have sparse landmark class labels, and sparse labels are exactly the kind of variation covered by the claim of robustness to diverse drawing styles. I request a per-step localization accuracy benchmark, ideally stratified by landmark density, and an oracle-localization control in which ground-truth p* is fed to the NPE; this would quantify how much of the reported 80% and 77% success is bottlenecked by the LE.","section":"Section IV-F, Eq. (5); Section V-A5"},{"comment":"The reported success rates and SPL values have no confidence intervals or significance tests, and the exact number of trials underlying each row is not stated precisely. For the full system, the procedure implies 2 environments × 2 maps × 2 robot platforms × 5 trials = 40 trials, but the denominators for each ablation row and for the Qwen variants are unclear. Differences such as 45% versus 40% when components are removed may be within sampling noise. Please report per-condition trial counts, exact binomial confidence intervals for success rate, and significance tests (or at least confidence intervals) for SPL and distance.","section":"Section V-A4, Table II"},{"comment":"The comparison against MapGPT varies two factors at once: the input modality (hand-drawn map versus verbal instruction) and the navigation algorithm itself (HAM-Nav versus MapGPT). Consequently, the reported differences in SR, SPL, NT, and D cannot be attributed to the hand-drawn-map interface, which is the paper's stated contribution. To support the abstract's claim of 'successful navigation outcomes compared against a non-hand-drawn map approach,' the experiment should isolate the map interface, for example by providing the same map-derived topological plan to the MapGPT planner or by ablating the visual prompt from HAM-Nav while keeping the user input identical.","section":"Section V-B, Table III"},{"comment":"The introduction and conclusion describe HAM-Nav as the first method to generalize across diverse hand-drawn-map styles and environments, but the experiments contain no comparison with any prior hand-drawn-map navigation method such as the probabilistic approaches in [9], [10], [15], [23]-[26]. Without such a baseline, the 'first' claim is not empirically supported. Either include one or more of these methods on the same simulated environments or temper the claim to 'first VLM-based method' and note that direct comparison with earlier methods is left for future work.","section":"Section II and Section VI"}],"minor_comments":[{"comment":"The notation in Eq. (3) is hard to parse: the comma-separated arguments are typeset inconsistently, and the variables d, δ, p', and a' are not all defined in Table I. Please add a one-sentence explanation of each symbol and clarify whether δ is a learned transition or a hand-defined function.","section":"Section IV-D, Eq. (3)"},{"comment":"The hyperparameters α, β, γ, and the 0.5 retention threshold are set by expert-guided tuning, but no sensitivity analysis is provided. Since one of the paper's selling points is zero-shot generalization, please add a short sensitivity study or at least discuss how sensitive the pruning results are to these values.","section":"Section IV-D"},{"comment":"The subsection numbering is duplicated: there are two items labeled '4) Procedure' and two '4) Results' in Section V-B. Please renumber the subsections.","section":"Section V-B, Procedure"},{"comment":"All system prompts are available only through a YouTube link. For reproducibility, the exact prompts should be included in the paper or in an archival supplementary document, because the prompts are load-bearing for the VLM behavior.","section":"Section IV-A through IV-H"},{"comment":"Reference [43] for GPT-4o is incomplete, and references [34], [35], and [44] are arXiv preprints; please include version numbers and access dates where applicable.","section":"References"},{"comment":"The phrase 'successful navigation outcomes compared against a non-hand-drawn map approach' is ambiguous given the confounding in the user study; consider rephrasing to describe the actual comparison as HAM-Nav with hand-drawn maps versus MapGPT with verbal instructions.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the real-robot results are valuable, but the evidence currently falls short of the strong generalization claim. The requested additions—localization benchmarking with oracle control, confidence intervals, prior-method baselines, and an interface-controlled comparison—are all within the scope of a revision, so I recommend major revision rather than rejection. I would also ask the editors to ensure the authors provide the exact trial counts and, if possible, release code or prompts for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, this is a real system paper: HAM-Nav is a new combination of pretrained VLMs, topological map pruning, and missing-landmark prediction, and it runs on actual robots in both simulation and real buildings. Second, the evidence for the headline 'generalizes across diverse hand-drawing styles' claim is thinner than the prose suggests, because the VLM's localization accuracy—the load-bearing step—is never measured in isolation. That said, the paper earns its place: the ablations show each module contributes, with the Experience Manager removal dropping success from 80% to 5%, which is a strong internal consistency check. The user study with 20 participants, diverse sketch styles, and real-world environments is a genuine plus.\n\nThe soft spots are real but not fatal. The stress-test concern is on point: equation (5) is the core cross-modal association, and the paper's own error analysis (Section V-A5) says incorrect localization is the common failure mode on sparse-label maps—exactly the variation the 'diverse styles' claim has to cover. An oracle-localization control would settle the bottleneck question. Trial counts are small (five per map, two maps per environment per robot) and there are no significance tests or confidence intervals. The user-study baseline MapGPT uses a different input modality (verbal instructions), so the comparison conflates interface with navigation quality—though that's arguably what they wanted to test, and they do report usability results separately.\n\nThe citation pattern looks fine; the related work on heuristic and probabilistic hand-drawn-map methods is covered, and the VLM components are standard. The main missing reference is a direct comparison to those prior methods in the same environments. Also, no code or data released, and prompts live on YouTube, which hurts reproducibility.\n\nBottom line: this is a systems contribution, not a fundamental theoretical result. It deserves a serious referee—the experiments, ablations, and user study justify a full review. The right outcome would be major revision asking for per-module localization evaluation, statistical rigor, and a baseline against prior hand-drawn-map methods. I'd bring it to a reading group for the robotics/HRI crowd; it's a useful system reference, but I wouldn't treat the generalization claim as established.","headline":"A genuinely new VLM-based hand-drawn-map navigation system with real robot experiments, but the zero-shot generalization claim rests on an unmeasured localization step that the paper's own error analysis flags as the main failure mode.","tokens_in":13130,"tokens_out":2764,"would_cite":false,"duration_ms":28545,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pretrained vision-language model can turn a person's inaccurate, incomplete hand-drawn map into successful robot navigation, zero-shot.","keywords":["hand-drawn maps","vision language models","mobile robot navigation","zero-shot navigation","topological map","landmark prediction","multi-floor navigation","prompt engineering"],"falsifier":"Run the Localization Engine alone on a collection of hand-drawn maps that vary in landmark density, with ground-truth node positions known, and compare its selected node to truth; if position accuracy collapses on maps with sparse labels, the reported 77-80% navigation success would not generalize to such sketches.","tokens_in":12063,"feed_emoji":"🗺️","tokens_out":6777,"duration_ms":60951,"temperature":0.7,"pith_summary":"This paper asks whether a mobile robot can navigate an unfamiliar environment using only a person's freehand sketch of the area, despite the sketch being distorted, out of scale, or missing landmarks. It argues yes: the HAM-Nav architecture hands the robot's live camera image and a pruned topological graph overlaid on the sketch to a pretrained vision-language model, which estimates the robot's position on the sketch and chooses the next navigation action without any task-specific training. The paper reports that the approach works for wheeled and legged robots, in indoor and outdoor settings, and on single- and multi-floor layouts. In photorealistic simulation trials the robot reaches the goal in 80% of cases, and in a real-world user study in 77% of cases, with participants rating the sketch interface far easier to use than delivering verbal navigation instructions.","feed_headline":"Robots read hand-drawn maps and navigate, with zero training","feed_subtitle":"Matching the robot's camera view to a sketch lets it follow the route even when landmarks are missing.","key_machinery":"The load-bearing mechanism is Selective Visual Association Prompting (SVAP), implemented in the Visual Prompt Generator: it places the robot's labeled camera view side-by-side with a pruned topological map $\\mathcal{M}'_{tp}$ overlaid on the hand-drawn map, so the VLM can associate live visual features with sketch elements in one image. The retained candidate nodes are selected by a logistic retention probability $\\zeta(v_i)$ that depends on distance and on a transition function from previous position and action. Around this sit the Predictive Navigation Plan Parser (PNPP), which uses the VLM to fill in missing landmarks and produce a sentence-by-sentence plan ('go straight past the desks, turn right when you see the posters'), and an Experience Manager that retrieves past observations by cosine similarity. The Localization Engine and Navigation Planning Engine both use chain-of-thought and score-based prompting to turn the composite prompt into a position estimate and an action.","core_discovery":"On its own terms, the paper's discovery is that a zero-shot vision-language pipeline can do what previously required either metric map acquisition or hand-crafted landmark templates. HAM-Nav converts a hand-drawn map $\\mathcal{M}_h$ into a topological graph $\\mathcal{M}_{tp}$ whose nodes are robot positions and landmarks, detects both object landmarks (via open-vocabulary detection) and structural landmarks (turns and junctions) in the robot's camera stream, and then builds a composite visual prompt: the labeled camera view beside a pruned topological map $\\mathcal{M}'_{tp}$ drawn over the sketch. A pretrained VLM, prompted with chain-of-thought and score-based reasoning plus a textual navigation plan, returns the estimated position node $p^*$ and the next high-level action $a$. The paper claims this is the first hand-drawn-map navigation method that generalizes across environments, drawing styles, and robot embodiments without task-specific training, and that its Predictive Navigation Plan Parser lets the system infer omitted landmarks from co-occurrence knowledge.","pith_inferences":["An explicit next step the paper only gestures at is using the VLM's own confidence scores to trigger a clarifying question to the user: when position estimates are low-confidence, ask the person to label an additional landmark on the sketch, converting the known sparse-label failure mode into a recoverable interaction.","Because the ablation shows the Experience Manager carries the largest performance share, a cheaper remedy than faster VLMs may be better retrieval (e.g., recency-weighted or action-consistent experience matching) rather than a larger model.","The architecture makes no use of metric consistency checks between the hand-drawn map and the robot's odometry; adding a weak geometric filter over the VLM's node estimates could reduce the action loops that the paper identifies as the dominant failure.","The comparative user study bundles map-drawing time into the interface, while the verbal baseline was one-shot; a fairer comparison might give MapGPT a chance to correct or repeat instructions, which would partially close the usability gap."],"forward_implications":["A robot can be given a task with a sketch drawn on a tablet in minutes, with no metric survey of the environment first.","The same visual prompting scheme transfers across robot bodies and terrain types, since the high-level actions are embodiment-agnostic.","Multi-floor buildings can be navigated using a topological graph instead of requiring a metric 2D map that distorts stacked floors.","Missing or misplaced landmarks do not have to break navigation, because the parser infers likely co-occurring landmarks from common-sense spatial knowledge.","The interface burden shifts from composing complete verbal route descriptions to sketching incrementally, which the user study reports as substantially less cognitively demanding."],"supporting_citations":[{"why":"Open-set object detector that finds landmark classes from the topological map in the robot view.","marker":"[34]"},{"why":"Segments the traversable region, feeding structural landmark detection and the navigation controller.","marker":"[35]"},{"why":"MapGPT baseline for the real-world user study; the paper compares navigation metrics and usability against it.","marker":"[27]"},{"why":"Chain-of-thought prompting technique used to decompose position estimation and action selection.","marker":"[37]"},{"why":"Score-based prompting technique used to assign probabilities to position nodes and actions.","marker":"[38]"},{"why":"GPT-4o, the vision-language model used for inference in the main experiments.","marker":"[43]"},{"why":"Qwen 2.5-VL model family used in ablations to test sensitivity to the choice of VLM.","marker":"[44]"}],"fun_headline_variants":["Robots navigate from hand-drawn maps, zero training","Hand-drawn maps steer robots without any training","Zero-shot robot navigation from hand-drawn maps","Robots infer missing landmarks from hand-drawn maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole loop depends on the vision-language model correctly linking what the robot's camera sees to a specific node on the hand-drawn map; that localization step is not benchmarked on its own, and the paper reports that on maps with few landmark labels the robot frequently localizes wrongly and then loops or reaches a dead end.","fun_headline_variants_meta":{"raw":{"variants":["Robots navigate from hand-drawn maps, zero training","Hand-drawn maps steer robots without any training","Zero-shot robot navigation from hand-drawn maps","Robots infer missing landmarks from hand-drawn maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2878,"prompt_tokens":941,"completion_tokens":1937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1875}},"tokens_in":557,"tokens_out":1937,"duration_ms":14914,"temperature":1.0,"reasoning_tokens":1875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:07:26.581136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Localization Engine alone on a collection of hand-drawn maps that vary in landmark density, with ground-truth node positions known, and compare its selected node to truth; if position accuracy collapses on maps with sparse labels, the reported 77-80% navigation success would not generalize to such sketches.","supporting_citations":[],"review_version":1}