{"id":"3a7bb00c-4135-40d6-80a9-31ecf7fdb6ec","arxiv_id":"2507.06719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SpatialReasoner adds LLM-based query decomposition and a visual-properties-enhanced hierarchical feature field to 3D language fields, improving instance localization for spatial-relation queries on self-constructed benchmarks.","lead":"This paper presents SpatialReasoner, a system for 3D visual grounding that locates objects described with spatial phrases like 'the book on the chair'. It combines a language model that splits the query into target, anchor, and relation with 3D scene features including color and opacity, and reports large gains over existing methods. Relevant to robots and AR assistants that must follow natural instructions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The extended LERF evaluation is confounded: SAM supplies both the training masks for the proposed method and the ground-truth boxes on rendered novel views, so the reported spatial-reasoning gains may reflect alignment to the SAM mask prior rather than genuine reasoning.","rationale":"I read the paper as an engineering systems contribution: LLM-driven query decomposition, hierarchical language and instance fields, and handcrafted spatial verification. The components are plausible, and the qualitative figures show sensible behavior in the examples presented. The claim is falsifiable and the framework is described in enough detail to be reimplemented. However, the strongest claim is empirical, and its validity depends entirely on the benchmark being independent of the method's supervision. The reader's weakest assumption identifies exactly the right soft spot: ground truth produced by SAM on rendered novel views while SAM masks are also the training supervision. I agree with that assessment. The problem is not an internal logical inconsistency or a disagreement with community consensus; it is a benchmark-design flaw that can inflate any method that uses SAM masks, and it most directly inflates the proposed method because the hierarchical field is optimized to align with SAM mask interiors. Because the paper reports no error bars, no human-annotated subset, and no standard benchmark evaluation, the reported gains are not yet independently verified. The reader's CONDITIONAL verdict is appropriate; I would not change it. A human-annotated subset or an Sr3D/Nr3D run would settle the concern: if the gains persist under independent labels, the claim should be accepted; if they collapse, the claim should be rejected or substantially weakened.","tokens_in":14467,"tokens_out":5136,"duration_ms":61293,"concrete_test":"Re-annotate a random subset of the extended LERF spatial queries, for example 50 queries per scene covering all four spatial-relation types, with human-drawn 2D boxes or masks on the same rendered novel views without using SAM. Then recompute Table 1 localization accuracy and Table 2 mIoU for LangSplat, SpatialReasoner(NeRF), and SpatialReasoner(3DGS). If the SpatialReasoner advantage over LangSplat shrinks by more than 5-10 points, or if the method fails on cases where target and anchor share semantics, the reported spatial-reasoning capability is inflated by the shared SAM distribution. A complementary external check is to run the same pipeline on Sr3D or Nr3D, where ground-truth object boxes are human-annotated and the code already fine-tunes on Sr3D.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SpatialReasoner outperforms language-field methods on spatial reasoning rests on Tables 1-4, but the main benchmark is self-referential in a specific way. Section 4.1 states that the extended LERF dataset is built by 'rendering novel views and generating ground truth labels using the Segment Anything Model (SAM).' Section 3.3 uses the same automatic SAM mask generator to create the supervision for the hierarchical feature field: CLIP features are extracted inside SAM masks, physical scales are computed by deprojecting SAM mask pixels, and the instance field is trained with a mask-consistency contrastive loss. At test time, accuracy is scored by whether the highest relevance pixel falls inside the SAM-generated box, and mIoU is computed against the SAM-generated mask. Thus the evaluation measures how well the method reproduces the same mask family that it was optimized to fit. A model that simply places confidence peaks inside SAM mask interiors can score well without resolving the actual spatial relation, such as which book is on the chair. This does not imply the method is fake; it means the reported 83-92% numbers cannot be read as independent evidence of spatial reasoning. The absence of human-annotated ground truth and the absence of any standard benchmark such as ScanRefer, Sr3D, or Nr3D makes the confound load-bearing: if the advantage over LangSplat disappears under human labels, the central claim is unsupported. An additional under-specification, that the four spatial-relation verification rules in Section 3.4 are never given as equations or pseudocode, compounds the problem, but the evaluation confound is the primary issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SpatialReasoner, a framework for open-vocabulary 3D visual grounding that augments neural representation--based language fields with an LLM fine-tuned to decompose queries into target, anchor, and spatial relation, and a hierarchical feature field (language plus instance) that incorporates opacity and color as visual properties and SAM masks as supervision. The pipeline is instantiated on NeRF, Instant-NGP, and 3DGS and evaluated on an extended LERF dataset, the Replica dataset, and a newly introduced Re3D dataset. The central claim is that adding LLM-driven spatial relation decomposition and a visual-properties-enhanced hierarchical feature field lets language-field methods outperform prior work on spatial-relation queries, with reported accuracies of 83.4%, 87.8%, and 91.7% on the extended LERF benchmark for the three backbones.","tokens_in":22,"tokens_out":4847,"duration_ms":73350,"significance":"If the claims were established, the contribution would be practically useful: it offers a modular upgrade path for existing NeRF/3DGS language fields to handle spatial queries, and the decomposition into target, anchor, and relation is a plausible design. The paper also reports experiments across three backbones and includes ablations of visual properties, the instance field, and the instance graph. However, the evaluation is currently not independent of the training signal, and the spatial-relation verification mechanism is unspecified. These issues must be addressed before the empirical claims can be taken as evidence for the advertised spatial reasoning capability.","major_comments":[{"comment":"The extended LERF evaluation is confounded with the method's training signal. Section 4.1 states that ground-truth labels for the extended LERF benchmark are generated by rendering novel views and using SAM. Section 3.3 uses the same SAM automatic mask generator to create the mask supervision for the language field (Eq. 4) and the instance field (Eq. 5), and to compute physical scales used in candidate generation. As a result, the reported 83.4--91.7% accuracies in Table 1 and 85.3--92.8% mIoU values in Table 2 largely measure how well the method reproduces the mask family it was optimized to fit, not whether it resolves spatial relations such as \"the book on the chair.\" Because no human-annotated ground truth or standard benchmark such as ScanRefer, Sr3D, or Nr3D is reported, the quantitative support for the central spatial-reasoning claim is currently undermined. Please re-evaluate with independent human-annotated boxes or masks, or on an existing benchmark, and report results separately for each of the four spatial relations.","section":"Sec. 4.1 and Sec. 3.3"},{"comment":"The core mechanism for verifying spatial relations is not described. The text states that SpatialReasoner \"considers four spatial relations (Horizontal Proximity, Vertical Proximity, Support, Allocentric)\" and that \"the specific object is determined accordingly,\" but no algorithm, thresholds, geometric tests, or decision rules are given. Equation (6) similarly defines the instance graph edge set through an unspecified threshold on affine differences, without stating how the threshold is chosen or how connected components are converted into final candidates. Since spatial relation verification is the load-bearing component for the central claim, this under-specification prevents the reader from assessing whether the method performs spatial reasoning in 3D. Please provide the full procedure, including parameters, pseudocode, and a failure analysis.","section":"Sec. 3.4"},{"comment":"The paper fine-tunes an LLM on Sr3D and Sr3D++ and mentions that the number of instructions is set to 3 for those benchmarks, but it reports no quantitative results on these human-annotated datasets or on ScanRefer/Nr3D. The newly introduced Re3D dataset is captured with the same Polycam pipeline and evaluated with the same protocol as the extended LERF benchmark, so it does not break the SAM circularity. Adding results on at least one standard human-annotated 3D visual grounding benchmark is necessary to demonstrate that the reported gains are not an artifact of the evaluation protocol. The paper should also report LLM decomposition accuracy on held-out queries and describe the fine-tuning data and instruction templates to rule out trivial overlap with the test queries.","section":"Sec. 3.2 and Sec. 4.1"}],"minor_comments":[{"comment":"There are typos in the figure text: \"mask-extraceted\" in Figure 2 and \"localozation\" in Figure 3; these should be corrected.","section":"Figures 2 and 3"},{"comment":"The notation in Eq. (6) is unclear: the dimension of V, the meaning of the tensor product, and the norm used should be defined explicitly, and the threshold for defining edges from the affine differences should be stated.","section":"Eq. (6)"},{"comment":"The phrase \"the number of instructions (n) is adapted per benchmark\" is not instantiated for LERF, Replica, or Re3D; please specify n and the actual instruction templates for each dataset.","section":"Sec. 3.4"},{"comment":"The method description refers to fine-tuning ChatGPT, while the implementation uses TinyLlama; this discrepancy should be reconciled.","section":"Sec. 3.2 and Sec. 4.1"},{"comment":"The claim that the method improves performance \"without significantly increasing the inference cost\" is not supported for the NeRF variant: SpatialReasoner(NeRF) has a single-view activation time of 1.43 s, more than 35 times slower than LangSplat's 0.04 s.","section":"Table 4"},{"comment":"The metric definitions should specify how the rendered relevance map is thresholded to compute mIoU against the annotated ground-truth mask, and the Replica head/common/tail split should be described with dataset statistics.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The SAM-based evaluation confound is serious enough that the paper should not be accepted in its current form; a revision that adds human-annotated or standard-benchmark evaluation and fully specifies the spatial-relation verification module would address my main concern. The paper's relationship to the authors' own ReasonGrounder (ref. [20]) is only cited and not discussed; the editor may wish to check novelty disclosure between these two closely related submissions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a system paper that combines three things: fine-tuning a small LLM to split a grounding query into target/anchor/relation; a hierarchical language-plus-instance field that conditions on opacity and color from the underlying neural representation; and an instance-graph merging step with simple spatial checks. That combination is not in the prior art they cite, though it is a close sibling of ReasonGrounder by the same first author, which they cite but do not position against carefully.\n\nWhat it does well: the decomposition idea is clean and easy to adopt; conditioning the field on visual properties is a reasonable way to separate instances that share semantics; and the integration with NeRF, NGP, and 3DGS shows the method is representation-agnostic. The ablations show each component adds something, and the qualitative failure cases in Figures 5 and 6 make sense. The writing is clear.\n\nThe soft spots are real and load-bearing. The main benchmark is built by SAM-generated masks on rendered novel views, and the same SAM masks are used as supervision for the instance field and as the evaluation boxes/masks. So the 83-92% numbers measure how well the method reproduces the SAM mask prior, not necessarily whether it reasons about which book is on the chair. This is not a minor detail; it is the central evidence for the claim of spatial reasoning beyond language-field methods. There is also no evaluation on ScanRefer, Sr3D, or Nr3D, and the new Re3D dataset is self-constructed without human labels. Section 3.4 says the method considers four spatial relations but never gives equations, thresholds, or pseudocode; that under-specification makes the reasoning step hard to check. No code is released, so independent confirmation is impossible right now.\n\nI do not think the method is fake: the pieces are sensible and the architecture is a plausible engineering contribution. But the current evaluation cannot support the strong claim. The fix is straightforward in principle: human-annotated boxes, or at least a standard benchmark with human ground truth, plus a concrete description of the spatial verification rules. Without that, the reported numbers should be treated as evidence that the method fits the SAM distribution, not as evidence of reasoning.\n\nWho this is for: people building language-field systems for open-vocabulary grounding and embodied AI. They will find the architecture worth studying even if the evaluation is not yet convincing. I would bring it to a reading group and would engage as a serious referee, but the decision should be conditional on the evaluation being substantially strengthened.","headline":"Solid engineering combination with a clean LLM-driven decomposition idea, but the self-referential SAM-based evaluation means the headline numbers are not yet trustworthy evidence of spatial reasoning.","tokens_in":15331,"tokens_out":2061,"would_cite":false,"duration_ms":23442,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that spatial reasoning can be bolted onto any neural 3D representation by decomposing language with an LLM and querying a hierarchical feature field.","keywords":["Open-vocabulary 3D visual grounding","Spatial reasoning","Language fields","Neural representation","LLM query decomposition","Hierarchical feature field","Visual properties","Instance field"],"falsifier":"Recompute the extended LERF spatial-reasoning accuracy using human-annotated 3D bounding boxes as ground truth while keeping the proposed method unchanged; if the gap over the language-field baselines narrows substantially or vanishes, the result rests on shared mask supervision rather than on spatial reasoning.","tokens_in":14256,"feed_emoji":"🎯","tokens_out":6984,"duration_ms":111868,"temperature":0.7,"pith_summary":"The paper proposes SpatialReasoner, a framework that gives neural 3D scene representations the ability to answer spatial-language queries such as \"the book on the chair.\" Its central claim is that language-field methods fail on such queries because they lack spatial reasoning on both sides of the problem: CLIP cannot reliably parse implicit relations in long queries, and the scene field stores only object semantics, not the geometry and appearance that distinguish one instance from another. SpatialReasoner addresses the first gap by fine-tuning an LLM to decompose a query into target, anchor, and spatial relation, and the second by building a hierarchical feature field that mixes CLIP language features with instance features and is augmented by opacity and color. On the extended LERF spatial-reasoning benchmark, integrating the framework into NeRF, Instant-NGP, and 3DGS raises overall localization accuracy to 83.4, 87.8, and 91.7 percent, respectively, roughly double the previous language-field methods. A sympathetic reader would take the contribution as evidence that spatial reasoning can be layered on top of existing neural representations rather than requiring a new representation.","feed_headline":"LLM reasoning helps 3D models find \"the book on the chair\"","feed_subtitle":"SpatialReasoner splits spatial queries into target, anchor, and relation, then locates the right instance.","key_machinery":"The load-bearing mechanism is the combination of an LLM-based query decomposer and a visual-properties-enhanced hierarchical feature field. The decomposer turns a free-form query into three instructions: target category, anchor category, and spatial relation, so that the vision side never has to infer the relation from raw CLIP text matching. The hierarchical field contains a language field and an instance field; both map 3D position, physical scale, opacity, and color to embeddings, and the instance field is trained with a margin-based contrastive loss so that objects sharing the same semantic label, such as two books, receive distinct instance embeddings. An instance graph then merges candidate regions before the spatial relation is checked, and the four supported relation types are Horizontal Proximity, Vertical Proximity, Support, and Allocentric relations.","core_discovery":"The central claim is that existing language-field methods localize objects directly from the whole query and therefore fail at spatial relations, and that the failure can be fixed by adding spatial reasoning in both the language and the scene. SpatialReasoner fine-tunes a small LLM to parse a query into target, anchor, and relation instructions, and constructs a hierarchical feature field with a language field and an instance field, both conditioned on visual properties—opacity and color—extracted from the neural reconstruction. Querying these fields hierarchically activates candidate targets and anchors, an instance graph merges related candidates, and the spatial relation is explicitly verified to pick the final instance. The paper reports that this design integrates into NeRF, Instant-NGP, and 3DGS and reaches overall localization accuracies of 83.4, 87.8, and 91.7 percent on the extended LERF spatial-reasoning benchmark, outperforming the language-field baselines it compares against.","pith_inferences":["Beyond the paper, the same decompose-then-query design could be transferred to 2D referring segmentation or video object grounding, where relational phrases such as \"left of\" or \"behind\" also break pure CLIP feature matching.","A testable extension the paper leaves implicit is a full ablation of the LLM: replacing it with a fixed rule-based parser would show how much of the gain comes from language-side decomposition versus scene-side hierarchical fields.","The shared use of the automatic segmentation model for both supervision and evaluation suggests the strongest next check is a human-annotated spatial-relation benchmark, which would separate genuine spatial understanding from segmentation bias."],"forward_implications":["Integrating SpatialReasoner into NeRF, Instant-NGP, or 3DGS yields localization accuracy of 83.4, 87.8, and 91.7 percent on the extended LERF spatial-reasoning benchmark, roughly double the best language-field baseline.","Because the framework only needs posed images plus the same neural representation already used for reconstruction, it brings spatial-relation grounding to any scene that can be reconstructed as a radiance field or Gaussian splatting.","Separating query parsing from scene reasoning means the language-side LLM can be improved or swapped without retraining the 3D fields, and vice versa.","The contrastively supervised instance field is what lets the method distinguish identical objects such as two books, so relational grounding no longer collapses to category matching."],"supporting_citations":[{"why":"LERF is the language-field baseline that integrates CLIP embeddings into NeRF, and SpatialReasoner builds its language field on this mechanism.","marker":"[12]"},{"why":"LangSplat is the 3DGS language-field baseline and main comparison point that motivates the need for spatial reasoning.","marker":"[29]"},{"why":"SAM supplies object masks both as supervision for the hierarchical fields and as the ground-truth generator for the extended LERF evaluation.","marker":"[14]"},{"why":"CLIP provides the text and image embeddings that are distilled into the language and instance fields.","marker":"[30]"},{"why":"ReferIt3D's Sr3D and Sr3D++ datasets are used to fine-tune the LLM for target-anchor-relation decomposition.","marker":"[2]"},{"why":"NeRF is one of the neural representations into which SpatialReasoner is integrated, anchoring the LERF-style language field.","marker":"[24]"},{"why":"3D Gaussian Splatting is the neural representation used in the configuration with the best reported accuracy.","marker":"[11]"}],"fun_headline_variants":["LLM parses 'book on chair' to improve 3D grounding","SpatialReasoner: LLM spatial reasoning for 3D object grounding","Split 3D queries into target, anchor, relation for accuracy","Fine-tuned LLM helps 3D models reason about spatial relations","Hierarchical fields + LLM reasoning improve 3D visual grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's ground-truth boxes on the extended LERF dataset are produced by the automatic segmentation model on rendered novel views, and the same segmentation model generates the masks that supervise the proposed fields, so the comparison assumes those automatic masks are an unbiased measure of correct localization.","fun_headline_variants_meta":{"raw":{"variants":["LLM parses 'book on chair' to improve 3D grounding","SpatialReasoner: LLM spatial reasoning for 3D object grounding","Split 3D queries into target, anchor, relation for accuracy","Fine-tuned LLM helps 3D models reason about spatial relations","Hierarchical fields + LLM reasoning improve 3D visual grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1890,"prompt_tokens":1040,"completion_tokens":850,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":753}},"tokens_in":656,"tokens_out":850,"duration_ms":9303,"temperature":1.0,"reasoning_tokens":753,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:56:40.792796+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the extended LERF spatial-reasoning accuracy using human-annotated 3D bounding boxes as ground truth while keeping the proposed method unchanged; if the gap over the language-field baselines narrows substantially or vanishes, the result rests on shared mask supervision rather than on spatial reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LERF is the language-field baseline that integrates CLIP embeddings into NeRF, and SpatialReasoner builds its language field on this mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LangSplat is the 3DGS language-field baseline and main comparison point that motivates the need for spatial reasoning."}],"review_version":1}