{"id":"cfa774eb-2cab-4af3-9c4b-f5d41d6e7198","arxiv_id":"2508.01723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"OpenMap is a zero-shot visual-language map that merges 3D instances using geometric and semantic consensus, then grounds free-form instructions with LLM-assisted selection.","lead":"OpenMap builds a 3D map from an agent's camera views and uses a vision-language model plus an LLM to connect free-form instructions such as \"get the chair, I want to eat\" to the correct object in the scene. It reports better zero-shot object grounding than prior navigation maps on ScanNet200 and Matterport3D benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing step is Eq. 2: OVSAM cosine similarity decides which 2D masks become one instance, yet no quantitative evidence shows these similarities are stable across viewpoints or compatible with the CLIP features used for instruction grounding.","rationale":"The reader's weakest assumption identifies exactly the reliability of Eq. 2 for instance merging, and I concur that this is the most load-bearing condition. The entire differentiating contribution is the structural-semantic consensus constraint: if the semantic similarity term is unreliable in the operating regime of real scans, the map can over-merge or fragment, and the downstream LLM grounding is then reasoning over corrupted instances. The paper's Fig. 3 shows only that two physically adjacent but distinct objects separate in one example; it is not evidence of cross-view stability or of agreement between the OVSAM feature space used for merging and the CLIP feature space used for retrieval. Without this evidence, the headline improvements over MaskClustering and ConceptGraphs cannot be cleanly attributed to the proposed consensus mechanism. I considered the evaluation-side concerns—unspecified instruction subsets and absent error bars—but those are secondary to establishing whether the core merging signal actually works. The proposed test is concrete and would settle the question: if the within/across distributions are well separated at the chosen threshold and OVSAM/CLIP disagreements are rare, the central claim is supported; if not, the reported gains are likely artifacts of feature-extractor choice or threshold tuning.","tokens_in":13881,"tokens_out":9134,"duration_ms":113274,"concrete_test":"On ScanNet200 val, use ground-truth 3D instance associations to form matched same-instance and distinct-instance mask pairs from the 2D masks; compute OVSAM cosine similarities stratified by object size, view overlap, and occlusion, and compare the within/across distributions at theta=0.6. Then, for every mask pair that OpenMap merges, measure the CLIP cosine similarity between the two masks; if a nontrivial fraction of OVSAM-merged pairs have low CLIP similarity, or if high-CLIP-similarity pairs are rejected by OVSAM, the feature-space mismatch is a real error source. Finally, rerun the grounding pipeline using instances built from CLIP-only merging instead of OVSAM-based merging; if SR changes by more than the reported 8.7-point gap over ConceptGraphs, the OVSAM/CLIP inconsistency is load-bearing for the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires Eq. 2's semantic similarity rate to be a reliable same-instance signal when multiplied by the structural consensus rate. The paper provides only a single qualitative feature-distribution figure (Fig. 3) for adjacent objects; it does not quantify within-instance versus across-instance cosine distributions, does not stratify by object size, occlusion, or viewpoint overlap, and does not examine failure modes for small, reflective, or partially observed objects. This matters because OVSAM features drive the merge decisions in §3.2, while the instance embeddings that the instruction-to-instance module (§3.3) actually queries are aggregated from CLIP features. A merge error at the OVSAM stage—either over-merging distinct instances or fragmenting one instance—directly changes the set of candidate instances available to the LLM and therefore changes retrieval success independently of the grounding module. The ablation in Table 3 shows that removing the semantic term drops AP to 12.2, but it does not test whether the theta=0.6 operating point is robust to viewpoint/lighting variation or whether OVSAM and CLIP similarity disagree on the masks that are merged. If the two feature spaces are not aligned, the reported 14.3 AP and 49.6% SR could reflect the choice of feature extractor for merging rather than a generalizable instance-consensus principle.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OpenMap proposes a zero-shot, open-vocabulary visual-language mapping pipeline for embodied instruction grounding. It merges 2D instance masks across views using a product of structural consensus (Eq. 1) and semantic similarity (Eq. 2) against a threshold (Eq. 3), then aggregates CLIP features from the most complete masks. For grounding, an LLM first parses a free-form instruction into an unrestricted target description, retrieves top-K candidates by CLIP similarity, and then selects the final instance in a second LLM round that uses spatial context from the map. Experiments on ScanNet200 instance segmentation and Matterport3D target retrieval report improvements over zero-shot baselines (MaskClustering, OVIR-3D) and retrieval baselines (NLMap, VLMap, ConceptGraphs), with ablations supporting the two proposed components.","tokens_in":14168,"tokens_out":8848,"duration_ms":90400,"significance":"If the reported results hold, OpenMap is a useful step toward open-vocabulary instance mapping and instruction grounding: it combines off-the-shelf models without 3D training, releases code, and includes ablations and hyperparameter sensitivity studies. The main strengths are the clear pipeline, the use of strong baseline comparisons, and the public code. However, the headline claims rest on evaluation details and a load-bearing merge criterion whose quantitative reliability is not yet demonstrated; these need to be addressed before the results can be taken as definitive.","major_comments":[{"comment":"The semantic similarity rate for merging is computed with OVSAM features, while the instance embeddings queried during grounding are aggregated from CLIP features. The paper offers only a single qualitative figure (Fig. 3) and an ablation that removes the semantic term; it does not quantitatively show that OVSAM cosine similarity is a reliable same-instance signal across viewpoints, object sizes, or occlusion levels, nor that OVSAM and CLIP agree on the masks that are merged. Because merge errors directly change the candidate instances available to the grounding module, the retrieval gains in Table 2 depend on this unvalidated step. Please add distributions of same-instance vs different-instance cosine similarity from Eq. (2), stratified by relevant factors, and an analysis of OVSAM-CLIP agreement on merged masks.","section":"§3.2, Eqs. (2)–(3); §4.4, Table 3"},{"comment":"The target-retrieval evaluation is based on 20 scenes and 'subsets' of instructions from R2R-CE, VLMap, and ALFRED, but the paper does not state how many instructions of each type were used, how they were selected, or whether the identical instruction set was presented to every baseline. Since the success rate is highly sensitive to instruction difficulty, the 8.7-point gap over ConceptGraphs could be influenced by the particular subset. Please specify the instruction construction, provide per-type and per-scene results, and release the test set for reproducibility.","section":"§4.1 and §4.3, Table 2"},{"comment":"All headline results are single-run point estimates without error bars or significance statements. The grounding pipeline uses GPT-4, which is stochastic, and the 2D mask proposals may add variability, so the differences of 14.3 vs 12.0 AP and 49.6 vs 40.9 SR need variance estimates from multiple runs or at least a statement on the number of runs and seeds used before claims of consistent improvement can be supported.","section":"§4.2, Tables 1 and 2; §4.4, Tables 5 and 6"},{"comment":"The consensus threshold tau_merge and the candidate number K_n are selected by sweeping on the same ScanNet200 validation and Matterport3D subsets on which the headline results are reported, so the reported numbers are in-sample bests. Although the sensitivity ranges are small (AP 14.0±0.34, SR 47.7±1.9), the paper should state explicitly whether these hyperparameters were fixed before the evaluation or were chosen on a separate development set, and should confirm that the ranking over baselines is preserved across the swept range.","section":"§4.4, Tables 5 and 6"}],"minor_comments":[{"comment":"References [45] and [46] are duplicates of the same paper (Yuan et al., Open-Vocabulary SAM); please unify and cite consistently.","section":"References"},{"comment":"HOV-SG [38] is listed as a baseline in Section 4.1 but is missing from Table 2; if results are unavailable for comparison, please state this explicitly.","section":"§4.1 and §4.3"},{"comment":"The table rows use non-rendered symbols (e.g., checkmarks and crosses are invisible in the manuscript text), making the ablation conditions difficult to interpret; please ensure the final PDF renders these correctly.","section":"§4.4"},{"comment":"In the Introduction, 'a OpenMap-enhanced' should be 'an OpenMap-enhanced'.","section":"§1"},{"comment":"Consider introducing a consistent abbreviation for the grounding module to avoid confusion between 'OpenMap' as the full framework and 'OpenMap-assisted instance selection' as a component.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is a viable empirical contribution, but the evaluation section needs substantial strengthening before publication. The reviewer's primary concerns are the lack of quantitative validation for the merge criterion (Eq. 2), the underspecified retrieval test set, and the absence of statistical error bars. These are addressable with additional analyses and clarifications. I also note the duplicate references and the missing HOV-SG comparison; these should be fixed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OpenMap is a competent zero-shot pipeline for grounding free-form navigation instructions into 3D instances. The new bits are the structural-semantic consensus merging criterion and the two-round LLM selection with neighbor context. Both are ablation-supported: removing the semantic term drops AP from 14.3 to 12.2, and removing the second-round selection drops SR from 49.6 to 47.2. That is real, reproducible work, with code released and hyperparameter sensitivity reported (Tables 5 and 6). The headline gains are plausible.\n\nThe soft spots are mostly evaluation-side. The main SOTA comparisons in Tables 1 and 2 are single point estimates with no error bars, so the 2.3 AP and 8.7 SR gaps over the nearest baselines could be within run-to-run noise. The retrieval benchmark is only 20 scenes, mixing 11 from R2R-CE and 9 from VLMap, with the number of instructions per type left unspecified; that makes the 49.6% SR hard to reproduce externally. I also notice HOV-SG, which they cite as a graph-based baseline for grounding, is missing from the comparison table—unclear why.\n\nThe stress-test concern about Eq. 2 is real but not fatal. They merge masks using OVSAM features and then aggregate CLIP features for the instance embeddings the LLM queries. The paper explicitly says OVSAM is only used for the semantic rate, so the two feature spaces are not aligned by construction. No quantitative evidence shows that OVSAM similarity is stable across viewpoints for small, reflective, or partially occluded objects, and the single qualitative figure (Fig. 3) does not settle it. That said, the ablation shows the semantic term matters, so the mechanism is doing something; the worry is only about whether this specific operating point generalizes. A side experiment correlating OVSAM and CLIP similarities on merged pairs would strengthen the story.\n\nThe hyperparameters are swept on the same ScanNet200 and Matterport3D benchmarks where they report SOTA, which is a mild circularity. But the sensitivity tables show the results are stable across a reasonable range (threshold 0.5–0.7 gives AP 14.0 ± 0.34), so this is not a major flaw, just something the authors should acknowledge.\n\nWho is this paper for? People working on VLN or open-vocabulary map-building will get a useful, well-engineered baseline, especially the grounding module. It is not a conceptual breakthrough, and the claims are slightly ahead of the evidence, but it deserves a serious referee rather than a desk reject. I would recommend conditional accept: fix the retrieval setup description, add error bars to the main tables, include HOV-SG, and address the OVSAM-CLIP mismatch with a small consistency study.","headline":"A clean zero-shot pipeline that likely improves open-vocabulary mapping and instruction grounding, but evaluation-side gaps and a feature-space mismatch stop the SOTA claim from being fully settled.","tokens_in":14696,"tokens_out":2292,"would_cite":true,"duration_ms":22964,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenMap, a zero-shot open-vocabulary visual-language map, claims to ground free-form navigation instructions to specific 3D instances more accurately than prior open-vocabulary maps by requiring both structural and semantic consensus for…","keywords":["instruction grounding","open-vocabulary mapping","3D instance segmentation","vision-language models","large language models","embodied navigation","zero-shot learning","semantic mapping"],"falsifier":"Take a scene with known instance labels and a set of small, reflective, or partially occluded objects, vary the viewpoint, and compute the same-instance and different-instance cosine similarities of the mask features used in the merging criterion. If the two distributions overlap substantially, over-merging or fragmentation occurs and OpenMap's reported gains would not transfer to those objects; equivalently, track merge precision and recall on those objects as the consensus threshold varies over 0.5 to 0.7.","tokens_in":13699,"feed_emoji":"🗺️","tokens_out":8503,"duration_ms":88612,"temperature":0.7,"pith_summary":"OpenMap is a zero-shot open-vocabulary visual-language map that turns a stream of RGB-D observations into 3D instance-level embeddings, then uses those embeddings to ground natural-language navigation instructions. The paper's claim is that merging 2D instance masks across viewpoints is reliable only when a structural consensus rate and a vision-language feature similarity both agree, and that letting an LLM produce unconstrained instance descriptions and reason over nearby objects yields better first-attempt target retrieval than prior open-vocabulary maps that restrict output vocabularies or ignore context. On ScanNet200, OpenMap reports 14.3 average precision for zero-shot instance segmentation compared to 12.0 for the closest zero-shot baseline, and on Matterport3D it reports 49.6% first-attempt success compared to 40.9% for a leading graph-based method. A sympathetic reader would care because this is a no-training route to connecting free-form language with specific objects in incremental navigation settings.","feed_headline":"No-training map beats prior open-vocab maps at linking words to 3D","feed_subtitle":"It merges masks only when geometry and semantics agree, then lets an LLM pick the target from spatial context.","key_machinery":"The load-bearing object is the structural-semantic consensus rate: for two masks, the ratio of viewpoints that can support merging them (their point clouds are contained in a single encompassing mask) to viewpoints that observe both, multiplied by the cosine similarity of their vision-language features. The paper merges masks when this product exceeds a threshold, iterating with decreasing observer-count thresholds, then aggregates features from the most complete masks. The second mechanism is a two-round instruction-to-instance grounding loop: an LLM converts a free-form instruction into an open-vocabulary target description, the map returns top-8 candidate instances by feature similarity, and a second LLM pass chooses the target using the candidates' locations and surrounding objects.","core_discovery":"The central discovery is that instruction grounding can be decomposed into two steps that each fix a specific failure: during mapping, treating structural consensus and semantic similarity as joint gatekeepers for mask merging stops both over-merging of adjacent distinct objects and fragmentation of large objects; during grounding, letting the LLM describe the target without a predefined label set and then choosing among the top candidates using a second LLM pass with surrounding-object context resolves ambiguous instructions such as “the chair near the table.” With these two mechanisms, a pipeline built from off-the-shelf 2D segmentation and pretrained vision-language models matches or beats methods that use supervised 3D masks or predefined instance vocabularies, and in the zero-shot comparison it reports the best numbers on both benchmarks tested.","pith_inferences":["A direct extension would be to use the target instance's aggregated embedding as a goal representation for an object-goal navigation policy, allowing the grounding step and the low-level navigation controller to share the same feature space.","If the cross-view feature stability holds beyond indoor scans, the same consensus criterion could transfer to outdoor or dynamic environments, where viewpoint and lighting changes are larger; this is a testable prediction the paper does not make.","The two-stage grounding pattern—coarse retrieval by similarity, then contextual disambiguation by an LLM—is a general way to attach open-vocabulary grounding to any map that provides instance features and locations, not just the one constructed here.","A quantitative feature-stability diagnostic, measuring same-instance feature similarity across views against different-instance similarity, would tell whether the reported mapping results are limited by mask quality or by feature drift; the paper currently shows only a qualitative distribution."],"forward_implications":["The construction of 3D instance-level semantic maps no longer requires training on 3D data or supervised mask proposals; off-the-shelf 2D segmentation plus a vision-language model suffices.","Merging decisions based on both structure and semantics should reduce both over-segmentation of large objects and false merging of spatially close but semantically distinct objects, the failure modes the paper identifies in proximity-based and structure-only baselines.","LLM outputs for instruction parsing need not be restricted to a fixed instance lexicon, so instructions with implicit goals such as “I am thirsty” can be expanded into attribute-rich descriptions that match instance features.","Retrieval success on the first attempt, the metric most relevant to real navigation, improves by grounding the final selection in spatial context rather than raw similarity alone.","Because the map is built incrementally from RGB-D frames, it can be used in exploration settings where the full scene point cloud is not available."],"supporting_citations":[{"why":"Supplies the observer/supporter structural-consensus machinery that OpenMap adapts, and is the strongest zero-shot baseline in the mapping comparison.","marker":"[44]"},{"why":"CLIP provides the open-vocabulary feature space used for final instance feature aggregation and for instruction-to-instance similarity matching.","marker":"[29]"},{"why":"OVSAM produces the mask features used to compute the semantic similarity rate in the merging criterion.","marker":"[46]"},{"why":"Provides the completeness-guided top-k mask feature aggregation strategy and is a comparison baseline using supervised mask proposals.","marker":"[34]"},{"why":"ConceptGraphs is the main target-retrieval baseline, and its reliance on predefined labels motivates OpenMap's unconstrained LLM parsing.","marker":"[7]"},{"why":"VLMap is the prior visual-language map baseline whose fixed-vocabulary instruction parsing and grid aggregation OpenMap contrasts with and improves on.","marker":"[9]"},{"why":"LVIS supplies the surrounding-object labels used in the second-round LLM instance selection prompt.","marker":"[8]"},{"why":"ScanNet200 is the benchmark dataset on which the mapping performance claims are evaluated.","marker":"[30]"},{"why":"Matterport3D is the navigation dataset used for instruction-to-target retrieval evaluation.","marker":"[43]"}],"fun_headline_variants":["OpenMap: zero-shot map that grounds language in 3D","Zero-shot map grounds free-form instructions in 3D","OpenMap: LLM helps ground words to 3D instances","No-training map links language to 3D using LLM","OpenMap: structure-aware zero-shot instruction grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that a mask pair with similar vision-language features and mutually observable point clouds is almost always the same real-world object; if small, reflective, or occluded objects have unstable features across views, the merge step mislabels them and the error propagates into the LLM grounding stage.","fun_headline_variants_meta":{"raw":{"variants":["OpenMap: zero-shot map that grounds language in 3D","Zero-shot map grounds free-form instructions in 3D","OpenMap: LLM helps ground words to 3D instances","No-training map links language to 3D using LLM","OpenMap: structure-aware zero-shot instruction grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000827,"raw_usage":{"total_tokens":3589,"prompt_tokens":898,"completion_tokens":2691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2607}},"tokens_in":514,"tokens_out":2691,"duration_ms":23966,"temperature":1.0,"reasoning_tokens":2607,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:26:14.265552+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a scene with known instance labels and a set of small, reflective, or partially occluded objects, vary the viewpoint, and compute the same-instance and different-instance cosine similarities of the mask features used in the merging criterion. If the two distributions overlap substantially, over-merging or fragmentation occurs and OpenMap's reported gains would not transfer to those objects; equivalently, track merge precision and recall on those objects as the consensus threshold varies over 0.5 to 0.7.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the observer/supporter structural-consensus machinery that OpenMap adapts, and is the strongest zero-shot baseline in the mapping comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLIP provides the open-vocabulary feature space used for final instance feature aggregation and for instruction-to-instance similarity matching."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OVSAM produces the mask features used to compute the semantic similarity rate in the merging criterion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ConceptGraphs is the main target-retrieval baseline, and its reliance on predefined labels motivates OpenMap's unconstrained LLM parsing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VLMap is the prior visual-language map baseline whose fixed-vocabulary instruction parsing and grid aggregation OpenMap contrasts with and improves on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LVIS supplies the surrounding-object labels used in the second-round LLM instance selection prompt."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ScanNet200 is the benchmark dataset on which the mapping performance claims are evaluated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Matterport3D is the navigation dataset used for instruction-to-target retrieval evaluation."}],"review_version":1}