{"id":"4402dc90-e6fb-4266-acb2-fd270d6b32b7","arxiv_id":"2506.23120","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-stage reasoning-segmentation method plus a new LLM-generated 3D dataset improves spatial reasoning in 3D multimodal large language models on several benchmarks.","lead":"This paper presents R2S, a two-stage framework that improves spatial reasoning in 3D multimodal language models by first segmenting question-related objects, then using those object features as clues to answer and segment the true target. The authors also introduce 3D ReasonSeg, a large LLM-generated dataset of over 25,000 training examples designed to train and test such reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on LLM-generated 3D ReasonSeg labels and LLaMA-3.1 prior annotations that are filtered but never independently validated; if the noise is systematic, gains on the self-generated benchmark and prior-learning supervision could be inflated.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: LLaMA-generated annotations and QA pairs are treated as ground truth for training and evaluation after rule-based filtering, and the authors' own Limitation section concedes residual noise. This is the most central issue because 3D ReasonSeg is both a claimed contribution and the benchmark on which the reasoning-specific improvement is measured, while the R2S prior learning is supervised by the same generation pipeline. If the noise is systematic, the gIoU gains and the Stage 1 prior gains could be inflated by alignment with LLaMA's biases rather than genuine spatial reasoning. The paper has real strengths: a plausible two-stage framework, consistent directional improvements across ScanQA, ScanRefer, and 3D ReasonSeg, and a reasonable effort to filter the generated data. The absence of code, dataset release, and error bars makes the concern untestable from the submission alone, but those are secondary to the label-validity question. A secondary internal inconsistency exists around lambda_txt: Sec. 3.4 reports lambda_txt = 0.5 for the main model, while the ablation text says 1.0 is optimal even though Table 5 shows 0.5 dominating on most metrics. This is a reporting error that should be corrected, but it does not by itself overturn the central claim. The proposed human-verification test would settle whether the generated labels are trustworthy, and the verdict should remain conditional on that evidence being provided.","tokens_in":16769,"tokens_out":5977,"duration_ms":61150,"concrete_test":"Release 3D ReasonSeg and the LLaMA-generated relevant-object labels, then have two independent human annotators verify target instance IDs and relevant-object sets on a random sample of 200 validation questions, using the original SceneVerse/ScanNet object annotations as the reference. Compute inter-annotator agreement and agreement between LLaMA labels and human labels. Then re-evaluate the released R2S model only on validation samples with full human agreement; if the 3D ReasonSeg gIoU drops substantially or the improvement over baseline disappears, the self-generated benchmark is contaminated. As a second check, retrain Stage 1 using only human-corrected relevant-object labels and compare ScanRefer Acc@50 and 3D ReasonSeg gIoU with Table 2. If both checks preserve the reported gains, the central claim would be supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline improvements in Table 2 depend on two pieces of supervision both produced by LLaMA 3.1: (i) the target-relevant object sets used to train Stage 1 (\"Reasoning Prior Learning,\" Sec. 3.2, Eqs. 6-7), and (ii) the 3D ReasonSeg question-answer-target triples used for training and for the gIoU evaluation (Sec. 3.3). Appendix A describes only rule-based filtering: distance thresholds, outlier removal, phrase blacklists, and redundancy pruning. The paper claims \"manual verification processes\" in the abstract/introduction, but the appendix reports manual examination only to formulate filtering strategies; no inter-annotator agreement, spot-check rate, or correction counts are given. The Limitation section then concedes that \"some noise remains in our dataset even after data cleaning procedures.\" If LLaMA systematically misidentifies target instances or relevant objects (for example, defaulting to prototypical object categories or ignoring true geometric constraints), the model can score highly on 3D ReasonSeg by learning the generator's bias rather than human-validated spatial reasoning. Because the same LLaMA-generated labels also supervise the prior stage, the R2S-specific gains (ScanRefer Acc@50 +10.0 over baseline, 3D ReasonSeg gIoU +3.9) may partly reflect memorizing LLaMA's relevant-object patterns rather than robust spatial reasoning. The ScanQA/ScanRefer results are less exposed because their questions and answers come from human-created benchmarks, but the R2S target-relevant supervision there is still LLaMA-generated (Fig. 10), so systematic noise can inflate those gains as well. Without independent validation of the generated labels, the central claim that R2S and 3D ReasonSeg \"endow 3D point cloud perception with stronger spatial reasoning capabilities\" is not fully separable from the quirks of the data generator.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Relevant Reasoning Segmentation (R2S), a two-stage framework for 3D point-cloud segmentation guided by spatial reasoning. In Stage 1, the model predicts masks for objects relevant to a question, and these masks are used to pool visual features that are injected as reasoning priors into a revised instruction. In Stage 2, the Q-Former and LLM are run again with the prior-augmented instruction to produce the final segmentation. The paper also introduces 3D ReasonSeg, a dataset of 25,185 training and 3,966 validation samples generated by Llama 3.1 from SceneVerse object annotations and filtered with rule-based heuristics. Experiments on ScanRefer, ScanQA, and 3D ReasonSeg compare the baseline, baseline trained with 3D ReasonSeg, and the full R2S method, reporting consistent improvements across all metrics, with ablations on the prior-guided refinement, the number of latent queries, and the text-loss weight.","tokens_in":17158,"tokens_out":7084,"duration_ms":70285,"significance":"If the results are robust, R2S provides a conceptually simple and effective way to inject explicit reasoning priors into 3D multimodal LLMs, and 3D ReasonSeg could become a useful resource for evaluating spatial reasoning. The paper's strengths include a clear problem formulation, a well-motivated two-stage design, and extensive ablations. The improvements on ScanRefer Acc@50 (38.7 to 48.7) and 3D ReasonSeg gIoU (29.2 to 33.1) are substantial. However, the benchmark and the prior supervision are both generated by the same LLM, and the paper does not demonstrate that the generated labels are accurate enough to serve as reliable ground truth. The lack of independent validation of the dataset, combined with several internal specification gaps, makes the central claim of robust spatial-reasoning gains provisional pending additional evidence.","major_comments":[{"comment":"The claim that 3D ReasonSeg has 'precise annotations' and is maintained through 'manual verification processes' is not supported by the described procedure. Appendix A states that manual examination was used to formulate filtering strategies, but no spot-check rate, inter-annotator agreement, or correction counts are reported. The Limitation section concedes that 'some noise remains in our dataset even after data cleaning procedures.' Because the same Llama 3.1-generated labels supervise the R2S prior stage (Sec. 3.2, Eqs. 6-7) and also define the 3D ReasonSeg evaluation targets, the gIoU gains in Table 2 (29.2 to 33.1) and the R2S-specific improvements in Table 3 may partly reflect the model's ability to imitate the generator's biases rather than to perform human-validated spatial reasoning. Please provide an independent human-annotated subset with per-sample accuracy statistics, or evaluate on an existing human-annotated reasoning benchmark, to demonstrate that the gains transfer beyond the self-generated benchmark.","section":"Sec. 3.3, Appendix A, and Limitation"},{"comment":"The handling of multiple relevant-object masks is underspecified. Eq. (7) defines f_r as a single feature tensor via f_p × M_r, implying one mask, but the text and Fig. 4 describe n relevant objects with features f^1_r, ..., f^n_r. The paper does not state whether M_r aggregates masks by summation or averaging, whether each object is pooled separately, how many feature vectors are produced, or how these vectors are inserted into the instruction embedding w′ ⊕ f_r (e.g., as additional tokens or as a sequence concatenated to the Q-Former query). Without this detail, the architecture is not reproducible. Please specify the exact tensor operations, the number and shape of the prior features, and the exact injection mechanism for scenes with multiple relevant objects.","section":"Sec. 3.2, Eqs. (6)-(9)"},{"comment":"The prose states that 'setting λtxt to 1.0 achieves optimal balance in performance,' but Table 5 shows that λtxt=0.5 yields the best or tied-best score on all metrics except ScanRefer Acc@25 and Acc@50, where λtxt=0.1 is slightly higher. For example, ScanQA CIDEr is 75.2 for λtxt=0.5 versus 72.6 for λtxt=1.0, and 3D ReasonSeg gIoU is 29.2 versus 28.6. This is a direct contradiction between the text and the table. Please correct the text to reflect the actual optimum and state how sensitive the main conclusions are to this hyperparameter.","section":"Sec. 4.2, Table 5"},{"comment":"The paper does not disclose whether the scenes used in 3D ReasonSeg overlap with the scenes in the ScanQA and ScanRefer evaluation sets. If 3D ReasonSeg is built from ScanNet or SceneVerse scenes that also appear in ScanQA/ScanRefer, then training on 3D ReasonSeg could inflate the cross-benchmark gains (e.g., ScanRefer Acc@50 from 38.7 to 48.7) through scene-level memorization rather than transferable spatial reasoning. Please report the scene-level split, the number of unique scene IDs in each split, and verify that the evaluation scenes are disjoint from all training scenes.","section":"Sec. 3.3 and Table 2"}],"minor_comments":[{"comment":"The phrase 'manual verification processes' overstates the procedure described in Appendix A, which reports only manual examination to design filtering strategies. Consider replacing it with 'rule-based filtering informed by manual inspection.'","section":"Abstract and Sec. 1"},{"comment":"The first contribution bullet says 'Relevant Segmentation (R2S)' but the method is introduced as 'Relevant Reasoning Segmentation (R2S).' Please fix the typo.","section":"Sec. 1, Contributions"},{"comment":"The caption says 'Ablation study on the Prior-guided Refinement,' but the meaning of the 'wo PR' and 'text-based' rows is not fully explicit. Please state in the caption whether all rows include training on 3D ReasonSeg.","section":"Table 3"},{"comment":"No variance or significance tests are reported. Providing repeated-run mean and standard deviation for at least the main reported numbers would help assess the stability of the improvements, especially for metrics where the differences are a few points.","section":"Tables 2-5"},{"comment":"The sentence 'we assign equal weights of 1 to the BCE and DICE loss' is redundant with the unweighted sum in Eq. (11); clarify whether any weights are applied.","section":"Sec. 4.1 and Eq. (11)"},{"comment":"The multiplication symbol in f_r = f_p × M_r is ambiguous. Please specify whether this is elementwise multiplication between the super-point features and the mask scores, and state the dimensions of the result.","section":"Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and timely problem, and the proposed R2S mechanism is intuitively appealing. My main concern is that the self-generated 3D ReasonSeg benchmark and the LLM-generated prior supervision are co-designed with the method, and no independent validation is provided. This makes the headline gains difficult to interpret. I would encourage the editor to ask for a human-validated subset or an external benchmark evaluation. The λtxt contradiction in Sec. 4.2 is a clear internal error that should be fixed during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuinely useful paper for the 3D MLLM subfield. The R2S framework is a sensible new combination: first segment question-related objects, then inject their mask-pooled features back into the Q-Former to refine the target prediction. That emulates human reasoning in a plausible way, and the ablations show it beats both no-prior and text-only prior variants. The 3D ReasonSeg dataset is large and fills a real gap, with more complex spatial reasoning (avg 5.4 relevant objects vs 1.5 and 1.8 for ScanQA and ScanRefer). The consistent gains on ScanQA and ScanRefer are encouraging, and the qualitative examples show the prior actually helps recover targets the baseline misses.\n\nThe biggest soft spot is the data-generation circularity. Relevant-object labels for R2S training and the question-answer-target triples for 3D ReasonSeg are all produced by LLaMA 3.1, with only rule-based filtering and manual inspection. The limitation section admits noise. If the generator has systematic biases—say, always picking prototypical objects or ignoring geometric constraints—then the model can score well by learning those biases rather than genuine spatial reasoning. The gIoU benchmark is essentially self-evaluated; only the authors and one fine-tuned baseline are reported. The ScanQA/ScanRefer results are less exposed because the questions come from human benchmarks, but the prior supervision there is still LLaMA-generated, so some inflation is possible.\n\nThere is also an internal inconsistency: Sec. 4.2 says λtxt=1.0 is optimal, but Table 5 shows 0.5 is best on nearly every metric. That is sloppy and should be fixed. No error bars or significance tests appear anywhere, and code and dataset are not released. These are required before the results can be independently trusted.\n\nWho is this for? Researchers working on 3D vision-language models, reasoning segmentation, and synthetic benchmark design. If the artifacts are released, it will be a useful baseline and benchmark. I would send it to peer review: the central idea is sound, the experiments are internally consistent despite the one λtxt slip, and the dataset could be a real contribution. Reviewers should insist on data/code release and a human-validated subset of generated labels.","headline":"A clean two-stage reasoning-prior idea for 3D MLLMs with consistent gains, but the self-generated dataset and supervision make the headline results need release and human validation before they fully convince.","tokens_in":17740,"tokens_out":2871,"would_cite":true,"duration_ms":30747,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that spatial reasoning in 3D point-cloud language models is a trainable two-stage process: first segment the objects relevant to a question, then feed their mask-pooled features back through the model's connector to…","keywords":["spatial reasoning","3D point cloud","multimodal large language model","reasoning segmentation","3D ReasonSeg","visual grounding","3D question answering","referring segmentation"],"falsifier":"Take the 3D ReasonSeg validation set, have humans re-annotate every question's target and relevant objects, and recompute the reported gIoU and Acc@50 on the cleaned subset; if R2S's gains shrink to noise, the measured improvement is an artifact of noisy LLM-generated labels.","tokens_in":16567,"feed_emoji":"🧠","tokens_out":7484,"duration_ms":69232,"temperature":0.7,"pith_summary":"The paper tries to establish that the reason 3D multimodal large language models fail on complex spatial instructions is not a lack of geometric information in point clouds but a lack of explicit reasoning structure. It proposes Relevant Reasoning Segmentation (R2S), a two-stage training procedure: first the model segments target-relevant objects as visual priors, then it feeds those objects' mask-pooled features back through the Q-Former so the language model can refine its answer and final mask. Alongside the method, the authors build 3D ReasonSeg, a 25,185-sample training set and 3,966-sample validation set of questions that require object functionality, visual attributes, and spatial relations to answer. They show that training with 3D ReasonSeg and R2S improves accuracy on ScanRefer, ScanQA, and 3D ReasonSeg itself, with the largest gains on the tasks that demand spatial reasoning.","feed_headline":"Two-stage reasoning lifts 3D object grounding accuracy by 10 points","feed_subtitle":"A two-step pipeline and a new dataset help 3D point-cloud models reason about object relationships before answering.","key_machinery":"The central object is Relevant Reasoning Segmentation (R2S), a two-stage pipeline built on the [SEG]-token interface: a first instruction asks the model to segment question-related objects, giving masks $M^r$; mask-pooling aggregates super-point features $f^p$ into per-object features $f^r = f^p \\times M^r$; a second instruction embeds these features into the Q-Former along with the question ($w' \\oplus f^r$), refining the latent queries to $q'_l$ and the object features to $f'_r$; the LLM then generates the answer and the mask decoder produces the final mask from the refined [SEG] hidden state. The mechanism that carries the argument is that the first stage provides a concrete visual prior that the second stage can attend to, rather than requiring the model to infer relevant objects and target simultaneously.","core_discovery":"The central claim is that spatial reasoning in 3D point-cloud language models can be trained by emulating human two-step search: first recognize the objects that are relevant to the question, then inspect those objects closely to locate the target. Compared with direct instruction tuning, this two-step decomposition improves Acc@50 on ScanRefer from 38.7 to 48.7 and gIoU on 3D ReasonSeg from 29.2 to 33.1, with consistent gains on ScanQA's BLEU-4, CIDEr, METEOR, and Rouge-L. The authors argue the mechanism is the visual prior: mask-pooling the segmented relevant objects produces instance-level features that carry spatial detail ordinary text names cannot, and feeding these features through the Q-Former in a second pass lets the model refine its latent queries before the final mask is decoded. They also claim existing datasets are too simple (e.g., 'in front of' is ambiguous in ScanRefer) and that 3D ReasonSeg supplies the missing complex spatial reasoning supervision.","pith_inferences":["A testable extension is to iterate the two stages more than once, using the refined mask as the prior for another pass; the paper's architecture would support it without new components.","Since the priors are mask-level features rather than text, the method implicitly teaches the model to ground relational language in geometry; that suggests it could transfer to embodied instruction following where object positions matter.","Because both the dataset and the target-relevant annotations are LLM-generated, a human-annotated subset of 3D ReasonSeg would be the cleanest way to separate genuine reasoning gains from alignment with the data generator's annotation style."],"forward_implications":["R2S can be layered onto any 3D MLLM that already outputs [SEG] tokens, so the two-stage recipe is a drop-in training strategy rather than a new architecture.","3D ReasonSeg provides a reusable benchmark whose questions deliberately require object functionality, visual attributes, and spatial relations; models trained on it generalize better to ScanRefer and ScanQA.","The relevant-object augmentation (randomly dropping or adding priors during training) makes the model robust to imperfect first-stage predictions at inference.","The largest improvements appear exactly on spatial-reasoning metrics (Acc@50 and gIoU), supporting the claim that explicit prior learning addresses the identified failure mode."],"supporting_citations":[{"why":"Defines the [SEG]-token interface and training objective that R2S extends from 2D to 3D reasoning segmentation.","marker":"[26]"},{"why":"Provides the pre-trained point-cloud encoder and mask decoder that R2S uses as its visual backbone.","marker":"[25]"},{"why":"Supplies the Q-Former connector whose latent queries are the target of R2S's prior-guided refinement.","marker":"[29]"},{"why":"Establishes the 3D MLLM instruction-tuning recipe and the multi-task training mixture R2S builds on.","marker":"[6]"},{"why":"Generates the 3D ReasonSeg question-reasoning-answer pairs and the target-relevant object annotations used for training and evaluation.","marker":"[15]"},{"why":"Provides ScanQA question-answer pairs used both as a benchmark and as a source of target-relevant supervision for R2S.","marker":"[2]"},{"why":"Provides ScanRefer referring expressions used both as a benchmark and as a source of target-relevant supervision for R2S.","marker":"[4]"},{"why":"Supplies the SceneVerse object descriptions and scene annotations from which 3D ReasonSeg is constructed.","marker":"[22]"},{"why":"Serves as the strongest prior 3D MLLM baseline, fine-tuned on 3D ReasonSeg to compare against R2S fairly.","marker":"[9]"}],"fun_headline_variants":["Two-step reasoning segmentation boosts 3D grounding by 10 points","Decomposing spatial reasoning into two stages lifts 3D object grounding","Reasoning-based segmentation improves 3D point-cloud language models","New dataset and two-stage pipeline strengthen spatial reasoning in 3D","Emulating human search: two-pass pipeline sharpens 3D spatial QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the large-language-model-generated object lists and question-answer pairs, after rule-based filtering, are accurate enough to serve as ground truth for both training supervision and the 3D ReasonSeg evaluation; the authors themselves note that noise remains after cleaning.","fun_headline_variants_meta":{"raw":{"variants":["Two-step reasoning segmentation boosts 3D grounding by 10 points","Decomposing spatial reasoning into two stages lifts 3D object grounding","Reasoning-based segmentation improves 3D point-cloud language models","New dataset and two-stage pipeline strengthen spatial reasoning in 3D","Emulating human search: two-pass pipeline sharpens 3D spatial QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1290,"prompt_tokens":950,"completion_tokens":340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":566,"tokens_out":340,"duration_ms":3521,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:48:16.641255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 3D ReasonSeg validation set, have humans re-annotate every question's target and relevant objects, and recompute the reported gIoU and Acc@50 on the cleaned subset; if R2S's gains shrink to noise, the measured improvement is an artifact of noisy LLM-generated labels.","supporting_citations":[{"cited_title":"Lisa: Reasoning segmentation via large language model","cited_arxiv_id":null,"evidence_quote":"Defines the [SEG]-token interface and training objective that R2S extends from 2D to 3D reasoning segmentation."},{"cited_title":"Oneformer3d: One transformer for unified point cloud segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained point-cloud encoder and mask decoder that R2S uses as its visual backbone."},{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the Q-Former connector whose latent queries are the target of R2S's prior-guided refinement."},{"cited_title":"Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning","cited_arxiv_id":null,"evidence_quote":"Establishes the 3D MLLM instruction-tuning recipe and the multi-task training mixture R2S builds on."},{"cited_title":"The llama 3 herd of models, 2024","cited_arxiv_id":null,"evidence_quote":"Generates the 3D ReasonSeg question-reasoning-answer pairs and the target-relevant object annotations used for training and evaluation."},{"cited_title":"Scanqa: 3d question answering for spatial scene understanding","cited_arxiv_id":null,"evidence_quote":"Provides ScanQA question-answer pairs used both as a benchmark and as a source of target-relevant supervision for R2S."},{"cited_title":"Scanrefer: 3d object localization in rgb-d scans using natural language","cited_arxiv_id":null,"evidence_quote":"Provides ScanRefer referring expressions used both as a benchmark and as a source of target-relevant supervision for R2S."},{"cited_title":"SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding","cited_arxiv_id":"2401.09340","evidence_quote":"Supplies the SceneVerse object descriptions and scene annotations from which 3D ReasonSeg is constructed."}],"review_version":1}