{"id":"0b39390a-6ef9-469f-ad25-9be8875ace35","arxiv_id":"2508.08252","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ReferSplat is the first method and benchmark for referring expression segmentation in 3D Gaussian Splatting, achieving state-of-the-art results with a spatially aware feature field.","lead":"This paper introduces a new task, dataset, and method for segmenting objects in 3D Gaussian scenes based on natural language descriptions. It is relevant to anyone building language-driven 3D understanding for robotics, AR, and scene interaction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's defining capability—segmenting occluded or non-visible objects—is never tested as a distinct claim: Ref-LERF results are averaged over all test expressions with no visibility stratification, so the reported SOTA does not establish that ReferSplat handles occluded or non-visible objects.","rationale":"The reader's CONDITIONAL verdict is reasonable, and the pseudo-label quality concern in Table 3 is real: at roughly 50 percent mIoU, the supervision signal is noisy and its interaction with spatial expressions is not analyzed. However, the most load-bearing issue for the central claim is that the defining task capability—handling occluded or non-visible objects—is not operationalized or evaluated separately. Even with perfect pseudo-labels, Table 5 would not show whether the method actually exploits 3D scene knowledge for occlusion, because the average mIoU mixes all visibility conditions and the baselines are not trained under the same supervision. Both the visibility-stratified analysis and the fair-baseline comparison are concrete, addressable checks, so I would keep the verdict CONDITIONAL rather than accept or reject. The paper has credible strengths: a new task formulation, a released dataset, and consistent module ablations (Tables 1, 4) that show each component helps. But those ablations are internal; they do not test the occlusion claim or the comparison against equally-trained alternatives. My concern is therefore a request for evidence, not an allegation of error, and the reader's overall conditional assessment remains the right verdict.","tokens_in":15728,"tokens_out":12507,"duration_ms":148995,"concrete_test":"Use the reconstructed 3DGS depth and the LERF-OVS ground-truth object masks to compute, for every Ref-LERF test expression, the visible fraction of the target object in the test view (e.g., by projecting the object's 3D mask and checking depth/opacity consistency). Then stratify Table 5 by visibility level: fully visible, partially occluded, and fully occluded/invisible, reporting per-stratum mIoU for ReferSplat and at least GOI and GS-Grouping. If the fully occluded stratum has no well-defined non-empty ground-truth mask, state that explicitly and exclude it from the headline claim. Additionally, fine-tune one adapted baseline (e.g., LangSplat) with the same pseudo-mask BCE supervision on Ref-LERF training expressions and compare; if the gap largely closes, the spatial paradigm is not the cause of the reported advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty of R3DGS, stated in the Abstract and Sec. 3.2, is to 'identify newly described objects that may be occluded or not directly visible in a novel view.' However, the evaluation protocol in Sec. 4.1 computes a single average mIoU over all rendered masks and GT masks, with no visibility labels, no occlusion stratification, and no definition of the ground-truth mask for an entirely invisible object. Standard 2D segmentation masks are non-empty only where the object is visible; if the task instead expects a projected full 3D mask even when the object is occluded, that rendering procedure is not specified. The rasterized response in Eq. 3 is alpha-composited over all Gaussians, so an occluding object with low text response but high opacity would suppress the target response, making it unclear how an invisible object could yield a positive mask. The qualitative examples in Fig. 5 do not replace quantitative evidence. Consequently, the reported SOTA may be driven by fully or partially visible cases, and the paper's defining occlusion claim is unsupported. A related confound is that the main R3DGS baselines (LangSplat, GOI, GS-Grouping, SPIn-NeRF) are adapted at test time rather than trained with the same per-scene pseudo-mask supervision as ReferSplat, so the performance gap could reflect additional task-specific training rather than the proposed 3D spatial paradigm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, Referring 3D Gaussian Splatting Segmentation (R3DGS), in which a model must segment objects in a 3D Gaussian scene from natural-language expressions that often encode spatial relations and attributes. The authors construct a new dataset Ref-LERF built on the LERF scenes and propose ReferSplat, which assigns each 3D Gaussian a learnable referring feature, computes text similarity in 3D space, renders a 2D mask, and supervises it with pseudo-masks generated by Grounded SAM using a confidence-weighted IoU selection strategy. The method also includes a Position-aware Cross-Modal Interaction module and a Gaussian-Text Contrastive Learning loss, plus a two-stage self-training refinement. Experiments report state-of-the-art results on Ref-LERF (29.2 average mIoU) and on the LERF-OVS and 3D-OVS open-vocabulary benchmarks, along with a more efficient pipeline than LangSplat.","tokens_in":16009,"tokens_out":3722,"duration_ms":40865,"significance":"If the claims hold, the paper makes a useful contribution by introducing a new task and dataset that connect 3D Gaussian Splatting with referring expression segmentation. The proposed idea of directly modeling language-Gaussian similarity in 3D rather than matching text against rendered 2D features is reasonable and the reported numbers are strong. The authors also provide a practical pseudo-mask generation pipeline and an efficiency comparison. However, the central novelty—handling occluded or non-visible objects—is not quantitatively validated, the baseline comparisons are not controlled for the extra training supervision, and the pseudo-label quality is quite low. These issues currently prevent full confidence in the claimed state-of-the-art.","major_comments":[{"comment":"The defining capability of the proposed task—segmenting objects that are occluded or not directly visible in a novel view—is never evaluated as a distinct claim. The evaluation protocol in Sec. 4.1 computes a single average mIoU over all test expressions with no visibility stratification, no occlusion labels, and no definition of how the ground-truth mask is obtained for an entirely invisible object. Moreover, the rasterization in Eq. (3) alpha-composites the per-Gaussian response, so an occluding object with high opacity could suppress the response of the target object. The qualitative examples in Fig. 5 do not replace quantitative evidence. I request a visibility-stratified evaluation, with explicit definitions of masks for partially and fully occluded objects, and a discussion of how the rendering operator can produce a non-empty mask for an invisible target.","section":"Abstract, Sec. 3.2, Eq. (3), Sec. 4.1"},{"comment":"The main comparison against 3D baselines (LangSplat, SPIn-NeRF, GS-Grouping, GOI) in Table 5 is not apples-to-apples. ReferSplat is trained per scene with pseudo-mask supervision generated by Grounded SAM, whereas the baselines are only adapted at test time. The performance gap could therefore reflect the additional task-specific training rather than the proposed 3D spatial paradigm. To support the claim that the method's design is responsible for the gains, the authors should either train the baselines under the same pseudo-mask supervision, or evaluate a version of ReferSplat that does not use task-specific training (e.g., using its pretrained RGB features with only the proposed inference procedure).","section":"Sec. 4.3-4.4, Table 5"},{"comment":"The pseudo-masks used for training have only 52.9% and 49.7% mIoU against manual ground truth on ramen and kitchen scenes (Table 3). The two-stage self-training in Sec. 3.5 further uses the first-stage model's rendered masks as supervision, which can entrench errors from the noisy initial pseudo-masks. The ablation in Table 1 shows only a small gain from the second stage (35.2 to 36.9 on ramen, 24.4 to 25.2 on kitchen), but no analysis of whether the self-training improves mask quality against the manually annotated ground truth. I request an error analysis on the two scenes with manual GT, reporting how the final model's masks compare to manual GT and whether self-training reduces or propagates specific types of errors.","section":"Sec. 3.3, Sec. 3.5, Table 3"},{"comment":"The results are reported without any variance or statistical significance. With only four scenes and 59 test expressions, the reported margins (e.g., 29.2 vs. 20.5 for GOI) may be within run-to-run variation. The authors should provide standard deviations across multiple training runs or, at minimum, a per-expression breakdown, and ideally a significance test. This is particularly important because the method involves multiple random components (pseudo-mask selection, contrastive sampling, optimization).","section":"Table 5"}],"minor_comments":[{"comment":"The dataset split is only described as 236 training and 59 test expressions for 59 objects; it is not stated whether expressions for the same object appear in both splits, which affects the difficulty and interpretation of the results.","section":"Sec. 4.1"},{"comment":"The hyper-parameter sentence \"we set ��, � , �, and � to 16, 128, 0.3, and 0.02\" is unreadable because the symbols do not render; the parameters should be defined with clear names (e.g., feature dimension D, hidden dimension, confidence threshold, loss weight). The schedule for λ is also incomplete.","section":"Sec. 4.2"},{"comment":"The table reports both R3DGS mIoU results and mask quality mIoU against manual GT on the same scenes; the distinction should be clarified in the caption or text so the reader does not confuse the two columns.","section":"Table 3"},{"comment":"The sentence \"we conduct experiments comparing BERT and CLIP embeddings for language features in R3DGS in Tab. 9\" contains a minor grammar issue and the table number should be referenced more gracefully.","section":"Sec. 4.7"},{"comment":"The qualitative examples would be more informative if the input text expression were displayed for each case and if the view were accompanied by a reference frame showing the target object's visibility at test time.","section":"Fig. 5"},{"comment":"The limitation statement acknowledges the small number of scenes; I appreciate this honesty, but it should be connected to the reported performance claims, since the four-scene benchmark limits the generality of the conclusions.","section":"Sec. 6"}],"recommendation":"major_revision","confidential_remarks":"This paper introduces a timely task and a reasonable method, and the code/data release is a plus. The main risk is that the headline capability (occlusion handling) is not evaluated, and the comparisons are confounded by unequal training protocols. I believe the paper can be made sound with additional experiments and clearer definitions, so I am recommending major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the contribution that will matter is the task and the benchmark: R3DGS defines referring segmentation in 3D Gaussian Splatting, and Ref-LERF adds 295 free-form expressions over the four LERF scenes, with code and dataset promised. Second, the method itself is a sensible stack of known ideas—per-Gaussian referring features, position-aware attention, contrastive learning—and the ablations show each component helps. I read the 29.2 mIoU vs 20.5 for GOI as real but not as evidence for the headline claim about occluded or invisible objects.\n\nWhat is actually new: the task formulation, the dataset, and the decision to compute text-Gaussian similarity in 3D before rasterization, rather than matching text to rendered 2D features like LangSplat. The paper is clearly written, the ablations are honest (including the pseudo-mask quality check against manual GT), and the limitations section admits the small scale. That is more than many new-benchmark papers do.\n\nSoft spots, in rough order of severity. First, the occlusion claim is untested. The metric is mIoU against standard 2D GT masks, which are only defined for visible parts of objects. There is no visibility stratification, no definition of GT for an entirely invisible object, and with alpha-compositing in Eq. 3 an occluder would normally suppress the response. The stress-test note is right: you cannot conclude from these numbers that the model does what the abstract says. It needs either a dedicated occlusion experiment or a qualified claim. Second, the pseudo-labels used for training have about 50% mIoU against manual GT (Table 3). That is low, and the paper does not analyze how this noise biases the trained model. It doesn't kill the result, but it limits how much you can say about true referring comprehension. Third, the baselines in Table 5 are adapted at test time and do not receive the per-scene pseudo-mask supervision that ReferSplat gets, so part of the performance gap may reflect task-specific training rather than the 3D spatial paradigm. Fourth, the two-stage refinement that improves numbers in Table 1 is left out of Table 5, which is confusing. Minor: no variance or significance tests, and four scenes are too few to draw strong generalization conclusions.\n\nWho should read this: anyone working on 3D-language grounding or open-vocabulary segmentation. The dataset is a useful first benchmark, and the evaluation gaps are instructive examples of how to (not) test an occlusion claim. The central result holds up as a first result on a new task, and the flaws are addressable rather than fatal.\n\nYes, send this to peer review. A serious referee can push for the occlusion analysis and pseudo-label noise experiments. The task and dataset deserve the field's attention even if the method numbers change.","headline":"The new R3DGS task and Ref-LERF dataset are the real contribution; the method is sound but its occlusion claim goes untested and the main comparison gives baselines no per-scene training.","tokens_in":16586,"tokens_out":4918,"would_cite":true,"duration_ms":51622,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the R3DGS task and claims that directly modeling each 3D Gaussian with language features, along with position-aware attention and contrastive learning, beats matching text to rendered 2D features for referring and…","keywords":["3D Gaussian Splatting","referring segmentation","language-guided 3D segmentation","spatial reasoning","open-vocabulary segmentation","cross-modal attention","contrastive learning","Ref-LERF dataset"],"falsifier":"Train ReferSplat on Ref-LERF using manually annotated masks instead of Grounded SAM pseudo-masks while keeping everything else fixed; if the resulting gain over render-and-match baselines shrinks or disappears, or if performance on spatial expressions like 'left of' and 'between' fails to improve, the claim that direct 3D Gaussian-language modeling provides the advantage would be falsified.","tokens_in":15499,"feed_emoji":"🎯","tokens_out":7806,"duration_ms":72977,"temperature":0.7,"pith_summary":"ReferSplat introduces a new task, Referring 3D Gaussian Splatting Segmentation (R3DGS), in which a model must produce a segmentation mask for an object described by a free-form natural-language expression in a 3D Gaussian Splatting scene. The paper argues that existing open-vocabulary 3D segmentation methods fail at this task because they match text against 2D rendered features rather than interacting with the 3D representation, which leaves them unable to reason about spatial relationships or objects hidden by occlusion. To test this, the authors construct Ref-LERF, the first R3DGS dataset, with long expressions that emphasize spatial relations, and propose ReferSplat, which gives every Gaussian a learned referring feature, refines it with position-aware cross-modal attention, and disambiguates similar expressions with contrastive learning. On Ref-LERF, ReferSplat reports 29.2 average mIoU, well above adapted baselines, and it also reaches state-of-the-art numbers on 3D open-vocabulary segmentation, supporting the paper's central claim that direct, spatially grounded 3D-language modeling is the more effective paradigm.","feed_headline":"Language-aware 3D Gaussians segment objects from descriptions","feed_subtitle":"A new dataset and method let models find objects from spatial phrases, even when they are partially hidden.","key_machinery":"The central object is the 3D Gaussian Referring Field: a learned feature vector $f^r_i$ attached to every 3D Gaussian $G_i$, so the scene itself carries a language-response field. Given a sentence, a text encoder produces word features $f^w_w$; each Gaussian scores the whole sentence by aggregating word similarities, $s_i = \\sum_w \\langle f^r_i, f^w_w \\rangle$, and these scores are rasterized with the standard splatting kernel into a 2D mask. Two mechanisms carry the performance: Position-aware Cross-Modal Interaction, which uses Gaussian center coordinates to bias cross-modal attention so spatial relations in language are grounded in 3D geometry, and Gaussian-Text Contrastive Learning, which averages the top-percentile Gaussian features as a positive embedding for the text and pulls these paired embeddings together while pushing apart other text queries. A confidence-weighted IoU selection over Grounded SAM candidates supplies the pseudo-labels, and a two-stage scheme retrains the model on its own rendered masks.","core_discovery":"The paper's central claim is that a 3D Gaussian scene can be made directly language-responsive by attaching a referring feature vector to every Gaussian, so that a natural-language expression yields a segmentation mask by rasterizing the similarity between Gaussian features and word features. This is contrasted with the prevailing render-and-match approach, where language embeddings are compared with a 2D feature map rendered from the scene. On top of this referring field, ReferSplat adds a Position-aware Cross-Modal Interaction module that injects Gaussian center coordinates into the attention between text and Gaussians, letting phrases such as 'between the pumpkin and the chair' be grounded in geometry, and a Gaussian-Text Contrastive Learning objective that treats high-similarity Gaussians as positives to pull apart semantically similar expressions that refer to different objects. The framework is trained with pseudo-masks produced by Grounded SAM and selected by a confidence-weighted IoU criterion, and a second training stage refines the masks using the model's own rendered outputs. With this design the paper reports state-of-the-art results on the new Ref-LERF benchmark (29.2 mIoU average) and on LERF-OVS and 3D-OVS open-vocabulary segmentation.","pith_inferences":["The roughly 50% mIoU of the pseudo-masks means the reported numbers may partly reflect learning a denoised version of Grounded SAM's errors; a human-mask training run on all four scenes would isolate how much of the gain is true 3D reasoning.","Selecting positive Gaussians by top-percentile similarity assumes the field is already roughly calibrated early in training; if initial scores are off-target, contrastive learning could reinforce the mistake, so a curriculum that anneals the percentile threshold is a natural extension.","Ref-LERF contains only four scenes, so the state-of-the-art claim could be scene-specific; extending the benchmark to more environments with similar language density would test whether the advantage generalizes.","The same position-aware interaction could apply to 4D Gaussian scenes, which the paper lists as future work, giving dynamic referring segmentation for robot manipulation."],"forward_implications":["On the new Ref-LERF benchmark, ReferSplat's 29.2 average mIoU exceeds the strongest adapted baseline (GOI, 20.5) and 2D-based Grounded SAM (15.8), indicating the task favors explicit 3D-language interaction.","ReferSplat also tops LERF-OVS (55.4 average mIoU) and 3D-OVS (94.1), so the same referring-field machinery transfers to open-vocabulary segmentation without being retrained for it.","Because the referring field aggregates evidence across training views, the model can produce masks for objects that are occluded or invisible in the novel view, which 2D methods cannot do.","The reported training time is 58 minutes on the ramen scene with about 3.3 MB of storage, so the approach is light enough for realistic deployment on embodied systems."],"supporting_citations":[{"why":"Provides the 3D Gaussian Splatting representation and rendering pipeline on which the referring fields are built.","marker":"(Kerbl et al., 2023)"},{"why":"Supplies the four real-world scenes whose language annotations form the Ref-LERF dataset.","marker":"(Kerr et al., 2023)"},{"why":"The main baseline and source of the LERF-OVS ground-truth masks; also supplies the default training configuration.","marker":"(Qin et al., 2024)"},{"why":"Grounded SAM generates the candidate masks that the confidence-weighted IoU selection turns into pseudo-ground-truth.","marker":"(Ren et al., 2024)"},{"why":"SPIn-NeRF's two-stage mask refinement is adopted to refine segmentation with the model's own rendered masks.","marker":"(Mirzaei et al., 2023)"},{"why":"BERT provides the text embeddings for referring expressions.","marker":"(Devlin et al., 2019)"},{"why":"Gaussian Grouping is a compared baseline for both R3DGS and open-vocabulary segmentation.","marker":"(Ye et al., 2025)"},{"why":"GOI is a compared baseline on Ref-LERF and open-vocabulary benchmarks.","marker":"(Qu et al., 2024)"},{"why":"CLIP is the vision-language feature source for several baselines and is tested as an alternative text encoder.","marker":"(Radford et al., 2021)"}],"fun_headline_variants":["Gaussians get language: segment 3D objects from text","Referring 3D segmentation: new task, dataset, method","Spatial phrases now can pick out 3D objects in Gaussians","ReferSplat: grounding words to 3D Gaussian points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training pipeline relies on pseudo-masks produced by Grounded SAM that match human-annotated masks only about half the time on the two scenes where this was measured, and the paper assumes these noisy labels are accurate enough to teach the model true referring segmentation without systematically biasing it toward simpler cases.","fun_headline_variants_meta":{"raw":{"variants":["Gaussians get language: segment 3D objects from text","Referring 3D segmentation: new task, dataset, method","Spatial phrases now can pick out 3D objects in Gaussians","ReferSplat: grounding words to 3D Gaussian points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1409,"prompt_tokens":979,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":595,"tokens_out":430,"duration_ms":4643,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:35:39.966030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ReferSplat on Ref-LERF using manually annotated masks instead of Grounded SAM pseudo-masks while keeping everything else fixed; if the resulting gain over render-and-match baselines shrinks or disappears, or if performance on spatial expressions like 'left of' and 'between' fails to improve, the claim that direct 3D Gaussian-language modeling provides the advantage would be falsified.","supporting_citations":[{"cited_title":"BERT: pre-training of deep bidirectional transformers for lan- guage understanding","cited_arxiv_id":null,"evidence_quote":"BERT provides the text embeddings for referring expressions."}],"review_version":1}