{"id":"4e9dde3f-31d4-4f92-a4a3-30f569624ebb","arxiv_id":"2607.16173","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VLMM is a 3D map representation where each object carries a fused, uncertainty-aware motion attribute (language-based movability prior + observed geometric motion) that makes motion queries such as 'what is moving' and 'what could move' answerable in natural language.","lead":"This paper gives each object in a robot's 3D map a motion attribute that combines a language-based answer to 'could it move?' with camera-based evidence for 'did it move?' plus a confidence score. A robot can then answer questions like 'what is moving?' and 'things I could move' by filtering the map.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Person-segmenter ground truth may conflate 'person' with 'moving' in the real-data validation of the uncertainty channel.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the real-data motion ground truth is a person-segmenter proxy that conflates 'moving' with 'person.' I agree this is the most fragile premise for the empirical central claim. The paper deserves credit for explicitly flagging this in Limitation 4 and for providing exact-GT simulation with injected Kinect noise, which independently supports the uncertainty channel. However, the 'consistently improves on six real sequences' claim in Table III remains conditional on a proxy, so the reader's CONDITIONAL verdict is appropriate. A corrected-GT recomputation is the concrete check that would settle whether the proxy actually biases the reported improvements. The door-query routing ambiguity and single-pinned-run reproducibility are secondary concerns that do not change the verdict.","tokens_in":9153,"tokens_out":11521,"duration_ms":110624,"concrete_test":"Recompute Table III on the same six TUM/Bonn sequences with corrected motion GT: take the person-segmenter output, manually (or via temporal verification) mark stationary people as static, and hand-label any non-person moving regions; then recompute raw-vs-Mahalanobis moving-vs-static AP and far-static false-flag rates. If every sequence still shows positive ΔAP and reduced false-flag rate, the proxy concern is settled. If any sequence flips, the real-data uncertainty claim must be re-scoped to person-only motion and supported by a real non-person-mover benchmark (e.g., a robot moving a chair/box with hand-labeled masks).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main load-bearing premise for the uncertainty channel is the real-data evaluation in Table III, whose moving/static ground truth is a person segmenter on TUM/Bonn (Sec. III.B, Limitation 4). Because those sequences contain people as the only movers, every person is labeled 'moving': static people count as false positives and non-person movers are absent. The reported AP deltas and far-static false-flag reductions could therefore reflect alignment with person masks rather than a general motion attribute. This does not defeat the paper: the exact-GT simulator with injected Kinect noise (+0.27 gap) independently supports the covariance channel, and the authors explicitly scope the real-data claim. But the 'consistently improves moving-vs-static AP ... across six real sequences' statement, the empirical anchor for the central 'uncertainty channel is the differentiator' claim, is not yet established for general non-person dynamic scenes; it rests on a proxy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VLMM, an open-vocabulary 3D map representation in which each object instance carries a semantic-feature vector, a geometric observed-motion score with a Mahalanobis uncertainty weighting, a VLM/LLM movability prior, and a fused motion class (STATIC / MOVABLE STATIC / MOVING) with per-element confidence. A rule-based parser routes natural-language queries to a single schema field, and the map answers queries such as 'what is moving', 'things I could move', and 'where it stays still'. Experiments are reported on AI2-THOR (exact ground truth, three scene types) and on TUM/Bonn RGB-D sequences (six real sequences, with a person segmenter as the moving-region ground truth). The main claims are that the schema fields are non-substitutable, that the uncertainty channel consistently improves motion detection under sensor noise, and that no prior system combines open-vocabulary, language-queryable, fused prior-and-observed motion, and per-element uncertainty.","tokens_in":9349,"tokens_out":8964,"duration_ms":77674,"significance":"If the claims hold, VLMM is a useful representation contribution: it would be the first open-vocabulary, language-queryable 3D map with a fused prior-and-observed motion attribute and per-element uncertainty, and the exact-GT AI2-THOR ablation provides a clean, controlled test of the schema. The paper is unusually honest: it reports threshold sensitivity, raw calibration failure (ECE 0.30) and post-hoc calibration (ECE 0.10), pose-robustness behavior, and an injected-noise experiment whose noise model is deliberately different from the assumed covariance. These strengths make the core proposal credible. The main weakness is that the real-data validation of the uncertainty channel relies on a person-segmenter proxy for moving regions, which limits the generality of the 'consistently improves across six real sequences' claim to person-mover scenes; the exact-GT simulator with injected noise is the only evidence for general non-person motion, and it supports the relative, not absolute, value of the uncertainty channel.","major_comments":[{"comment":"The real-data moving/static ground truth is a person segmenter on TUM and Bonn. Because these sequences contain people as the only movers, every person is labeled moving regardless of whether they are actually moving; static people are counted as false positives and non-person movers are absent. The reported AP improvements and far-static false-flag reductions in Table III could therefore reflect alignment with person-sized, near-range movers rather than a general motion attribute. The exact-GT AI2-THOR result with injected Kinect noise (+0.27 AP) independently supports the covariance channel, and Limitation 4 explicitly scopes the real-data claim. However, the abstract and Sec. III.E state that the uncertainty channel 'consistently improves moving-vs-static AP' across all six real sequences without this caveat. This is load-bearing for the paper's central 'uncertainty channel is the dif","section":"Sec. III.B, Table III, Limitation 4"}],"minor_comments":[{"comment":"The paper states that 'a single fixed fusion threshold triple (τ_o, κ_o, ρτ)' is used, but only ranges for τ_o and ρτ are given; the exact values of τ_o and κ_o, and the per-dataset frame gap k, should be reported for reproducibility.","section":"Sec. III.C"},{"comment":"The 'far-static false-flag rate' is not defined. Please specify what counts as far-static (distance threshold, pixel/instance level) so that the reductions in Table III are interpretable.","section":"Sec. III.E"},{"comment":"The class confidence for STATIC is set to 1−ρ_i, where ρ_i is a movability prior. This is a reasonable heuristic but should be justified or labeled as such; as written it appears to treat non-movability as confidence in the static observation.","section":"Sec. II.D, Eq. (11)"},{"comment":"The text says observed-motion alone 'collapses' the Q-movable and Q-static queries, yet the AP values are 0.73 and 0.71, well above random/prevalence. The word 'collapse' overstates a degradation; the conclusion of non-substitutability still holds because full fusion reaches 0.99, but the wording should be toned down.","section":"Sec. III.D, Table II"},{"comment":"The caption says 'real AI2-THOR keyframes'; AI2-THOR is a simulator, so 'real' is misleading. Rephrase to 'AI2-THOR rendered keyframes' to avoid confusion with real-world data.","section":"Fig. 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and technically sound in its core simulation study, and the authors are unusually transparent about limitations. The main issue is the real-data proxy: the abstract and contributions overstate the real-data support for the uncertainty channel. This is fixable by re-scoping the claims or adding one non-person real sequence. I do not see grounds for rejection, but the real-data anchor of the central claim needs strengthening or explicit narrowing before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate systems contribution, and the authors are more candid about its limitations than most. The combination — open-vocabulary language-queryable map plus fused observed-motion/movability prior plus per-element uncertainty — is not in any cited prior system, and the exact-GT simulator ablation makes a real point: strong region-CLIP features alone cannot answer \"what is moving\" (AP 0.10 vs random 0.09), and observed motion alone cannot answer \"what could move.\" The uncertainty channel is also not just decoration: on exact-GT sim with injected Kinect-level noise the Mahalanobis score beats raw displacement by +0.27 AP, and the direction of change is consistent across the real sequences. The paper ships no code or data and every number comes from a single pinned run, so the magnitude of the real-data gains should be treated as indicative, not certified.\n\nWeak spots, in proportion. The real-data ground truth is a person segmenter on TUM/Bonn, so \"moving\" and \"person\" are conflated. The authors flag this in Limitation 4 and scope the claim accordingly, but it means the six-sequence AP improvements in Table III are not yet demonstrated for general non-person dynamic scenes. That said, the injected-noise sim result independently supports the uncertainty channel, so I don't think the proxy sinks the paper. The \"non-substitutability\" diagonal is partly definitional — the authors admit this — but the off-diagonal and the semantic-only failure are the informative parts, and those hold. The parser is a rule-based router (80% on paraphrases, fails negation), which the paper labels honestly. Calibration: raw confidence is uncalibrated (ECE 0.30), post-hoc isotonic gets 0.10 on held-out Bonn; that is a fair, limited claim.\n\nWho this is for: people building language-queryable semantic maps for mobile robots, especially anyone who wants to move beyond static VLMaps-style representations. It deserves a serious referee; a good reviewer will push for code/data release, a non-person real benchmark, and multi-run variance, but the core representation and core ablation are sound.","headline":"Solid systems paper; the combination is genuinely new and the limitations are honestly scoped — real-data GT proxy is the main caveat.","tokens_in":9860,"tokens_out":2408,"would_cite":true,"duration_ms":19408,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3D map that stores both a semantic 'could move' prior and an observed 'has moved' score, with per-element uncertainty, can answer open-vocabulary questions about how a scene behaves.","keywords":["open-vocabulary 3D mapping","motion attribute","uncertainty-aware perception","natural-language robot query","RGB-D scene map","movability prior","observed motion detection"],"falsifier":"Hand-annotate all moving non-person objects (doors, drawers, boxes, bags) in real RGB-D sequences and rerun the pipeline; if covariance-weighted motion scoring does not beat the raw displacement score on moving-vs-static AP, or if a strong semantic feature matches the observed-motion channel on 'what is moving,' the central claims fail.","tokens_in":9013,"feed_emoji":"🤖","tokens_out":7676,"duration_ms":58672,"temperature":0.7,"pith_summary":"VLMM is a 3D scene-map representation that gives each object element a fused motion attribute: a language-derived semantic prior for whether the object could move, a geometric measurement of whether it has been observed to move across frames, and an uncertainty score on that measurement. Natural-language queries such as 'the door that opens' or 'things I could move' are routed to a single one of these fields. The paper claims these fields are non-substitutable: a strong semantic-only map fails to identify what is currently moving, the prior cannot substitute for observed motion, and observed motion cannot substitute for the prior. The uncertainty channel—the paper's main distinction from earlier fused-motion maps—consistently improves moving-vs-static ranking and reduces false motion flags on real RGB-D sequences with noisy depth and estimated poses. This matters because robots with such maps could answer behavior questions that static semantic maps cannot.","feed_headline":"3D map answers 'what moves, what could move, what stays still'","feed_subtitle":"Each object gets a movability prior, a measured motion score, and uncertainty, enabling language queries about behavior.","key_machinery":"The load-bearing machinery is a per-object schema of position, semantic feature, observed-motion score with reliability, movability prior, and fused class with confidence. Observed motion is computed by back-projecting two views into world coordinates, using forward-backward optical-flow correspondences, refining the relative pose from static inliers so that camera-tracking error does not masquerade as object motion, and scoring the residual by its Mahalanobis magnitude against a range-dependent depth covariance. The resulting per-point score is a likelihood-ratio test with a fixed static false-flag level, aggregated per object as a percentile with coherence-weighted reliability. The movabil","core_discovery":"On the paper's own terms, the central discovery is that an open-vocabulary, language-queryable motion attribute for 3D maps must combine two non-substitutable channels—observed geometric motion and a semantic movability prior—and that an explicit per-element uncertainty is what makes the fused attribute usable under real sensor noise. The authors demonstrate this with controlled exact-ground-truth scenes containing no people: ranking by the observed-motion score answers 'what is moving' perfectly while semantic-only ranking scores near random, and neither motion field alone answers all three query classes. On six real dynamic RGB-D sequences, replacing the raw world-displacement score with a","pith_inferences":["Because the real-data ground truth labeled every detected person as moving, the six-sequence gains should be read as validating the uncertainty channel under sensor noise, not as evidence about non-person movers; a hand-annotated non-person-mover benchmark is the natural next check.","If non-substitutability generalizes, the same two-channel pattern—a semantic affordance prior plus a geometric occurrence measurement—might be applied to other behavior attributes such as 'recently changed' or 'occluded,' without new supervision.","An LLM-based query front end, which the authors flag as future work, would test whether richer language understanding, including negation where the current parser fails, preserves the single-field routing that the representation relies on."],"forward_implications":["Any map that wants to answer motion-behavior queries needs both a semantic movability prior and a geometric observed-motion channel; one alone cannot supply the other's answers.","A robot can build such a map incrementally from RGB-D video and answer queries such as 'the door that opens,' 'things I could move,' and 'where it stays still' by filtering stored attributes rather than retraining per scene.","Uncertainty weighting based on range-dependent depth covariance turns sensor noise into a guard against false motion flags: distant static points are not marked moving even when depth is noisy.","The resulting confidence scores are rank-useful and post-hoc calibratable, so the same representation can support both ranking and thresholded decision tasks."],"fun_headline_variants":["3D map answers what moves, could move, stays still with uncertainty","Open-vocab map fuses semantic and seen motion for behavior queries","Uncertainty-aware 3D map answers what could move via fused motion","Two motion channels plus uncertainty make 3D maps behavior-queryable","VLMM: uncertainty-aware language queries about motion in 3D maps"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The real-data evaluation assumes that a person segmenter's output is a valid ground truth for 'moving', so static people count as false positives and non-person movers are never labeled; if that proxy is wrong, the reported uncertainty-channel gains may partly measure person detection rather than motion detection.","fun_headline_variants_meta":{"raw":{"variants":["3D map answers what moves, could move, stays still with uncertainty","Open-vocab map fuses semantic and seen motion for behavior queries","Uncertainty-aware 3D map answers what could move via fused motion","Two motion channels plus uncertainty make 3D maps behavior-queryable","VLMM: uncertainty-aware language queries about motion in 3D maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000856,"raw_usage":{"total_tokens":3600,"prompt_tokens":832,"completion_tokens":2768,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2671}},"tokens_in":576,"tokens_out":2768,"duration_ms":16495,"temperature":1.0,"reasoning_tokens":2671,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T21:08:01.882673+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-annotate all moving non-person objects (doors, drawers, boxes, bags) in real RGB-D sequences and rerun the pipeline; if covariance-weighted motion scoring does not beat the raw displacement score on moving-vs-static AP, or if a strong semantic feature matches the observed-motion channel on 'what is moving,' the central claims fail.","supporting_citations":[],"review_version":1}