Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

ReferSplat: Referring Segmentation in 3D Gaussian Splatting

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read The paper introduces the R3DGS task and claims that directly modeling each 3D Gaussian with language features, along with position-aware attention and contrastive learning, beats matching text to rendered 2D features for referring and…

desk verdict The new R3DGS task and Ref-LERF dataset are the real contribution; the method is sound but its occlusion claim goes untested and the main comparison gives baselines no per-scene training. read the letter →

arxiv 2508.08252 v1 pith:DGHQFQJB submitted 2025-08-11 cs.CV

classification cs.CV
keywords 3DGaussianSplattingreferringsegmentationlanguage-guidedspatialreasoningopen-vocabularycross-modalattentioncontrastivelearningRef-LERFdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReferSplat introduces a new task, Referring 3D Gaussian Splatting Segmentation (R3DGS), in which a model must produce a segmentation mask for an object described by a free-form natural-language expression in a 3D Gaussian Splatting scene. The paper argues that existing open-vocabulary 3D segmentation methods fail at this task because they match text against 2D rendered features rather than interacting with the 3D representation, which leaves them unable to reason about spatial relationships or objects hidden by occlusion. To test this, the authors construct Ref-LERF, the first R3DGS dataset, with long expressions that emphasize spatial relations, and propose ReferSplat, which gives every Gaussian a learned referring feature, refines it with position-aware cross-modal attention, and disambiguates similar expressions with contrastive learning. On Ref-LERF, ReferSplat reports 29.2 average mIoU, well above adapted baselines, and it also reaches state-of-the-art numbers on 3D open-vocabulary segmentation, supporting the paper's central claim that direct, spatially grounded 3D-language modeling is the more effective paradigm.

What carries the argument

The central object is the 3D Gaussian Referring Field: a learned feature vector $f^r_i$ attached to every 3D Gaussian $G_i$, so the scene itself carries a language-response field. Given a sentence, a text encoder produces word features $f^w_w$; each Gaussian scores the whole sentence by aggregating word similarities, $s_i = \sum_w \langle f^r_i, f^w_w \rangle$, and these scores are rasterized with the standard splatting kernel into a 2D mask. Two mechanisms carry the performance: Position-aware Cross-Modal Interaction, which uses Gaussian center coordinates to bias cross-modal attention so spatial relations in language are grounded in 3D geometry, and Gaussian-Text Contrastive Learning, which averages the top-percentile Gaussian features as a positive embedding for the text and pulls these paired embeddings together while pushing apart other text queries. A confidence-weighted IoU selection over Grounded SAM candidates supplies the pseudo-labels, and a two-stage scheme retrains the model on its own rendered masks.

What would settle it

Train ReferSplat on Ref-LERF using manually annotated masks instead of Grounded SAM pseudo-masks while keeping everything else fixed; if the resulting gain over render-and-match baselines shrinks or disappears, or if performance on spatial expressions like 'left of' and 'between' fails to improve, the claim that direct 3D Gaussian-language modeling provides the advantage would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 3D Gaussian scene can be made directly language-responsive by attaching a referring feature vector to every Gaussian, so that a natural-language expression yields a segmentation mask by rasterizing the similarity between Gaussian features and word features. This is contrasted with the prevailing render-and-match approach, where language embeddings are compared with a 2D feature map rendered from the scene. On top of this referring field, ReferSplat adds a Position-aware Cross-Modal Interaction module that injects Gaussian center coordinates into the attention between text and Gaussians, letting phrases such as 'between the pumpkin and the chair' be grounded in geometry, and a Gaussian-Text Contrastive Learning objective that treats high-similarity Gaussians as positives to pull apart semantically similar expressions that refer to different objects. The framework is trained with pseudo-masks produced by Grounded SAM and selected by a confidence-weighted IoU criterion, and a second training stage refines the masks using the model's own rendered outputs. With this design the paper reports state-of-the-art results on the new Ref-LERF benchmark (29.2 mIoU average) and on LERF-OVS and 3D-OVS open-vocabulary segmentation.

Load-bearing premise

The whole training pipeline relies on pseudo-masks produced by Grounded SAM that match human-annotated masks only about half the time on the two scenes where this was measured, and the paper assumes these noisy labels are accurate enough to teach the model true referring segmentation without systematically biasing it toward simpler cases.

Editorial extensions

If this is right

  • On the new Ref-LERF benchmark, ReferSplat's 29.2 average mIoU exceeds the strongest adapted baseline (GOI, 20.5) and 2D-based Grounded SAM (15.8), indicating the task favors explicit 3D-language interaction.
  • ReferSplat also tops LERF-OVS (55.4 average mIoU) and 3D-OVS (94.1), so the same referring-field machinery transfers to open-vocabulary segmentation without being retrained for it.
  • Because the referring field aggregates evidence across training views, the model can produce masks for objects that are occluded or invisible in the novel view, which 2D methods cannot do.
  • The reported training time is 58 minutes on the ramen scene with about 3.3 MB of storage, so the approach is light enough for realistic deployment on embodied systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The roughly 50% mIoU of the pseudo-masks means the reported numbers may partly reflect learning a denoised version of Grounded SAM's errors; a human-mask training run on all four scenes would isolate how much of the gain is true 3D reasoning.
  • Selecting positive Gaussians by top-percentile similarity assumes the field is already roughly calibrated early in training; if initial scores are off-target, contrastive learning could reinforce the mistake, so a curriculum that anneals the percentile threshold is a natural extension.
  • Ref-LERF contains only four scenes, so the state-of-the-art claim could be scene-specific; extending the benchmark to more environments with similar language density would test whether the advantage generalizes.
  • The same position-aware interaction could apply to 4D Gaussian scenes, which the paper lists as future work, giving dynamic referring segmentation for robot manipulation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a new task, Referring 3D Gaussian Splatting Segmentation (R3DGS), in which a model must segment objects in a 3D Gaussian scene from natural-language expressions that often encode spatial relations and attributes. The authors construct a new dataset Ref-LERF built on the LERF scenes and propose ReferSplat, which assigns each 3D Gaussian a learnable referring feature, computes text similarity in 3D space, renders a 2D mask, and supervises it with pseudo-masks generated by Grounded SAM using a confidence-weighted IoU selection strategy. The method also includes a Position-aware Cross-Modal Interaction module and a Gaussian-Text Contrastive Learning loss, plus a two-stage self-training refinement. Experiments report state-of-the-art results on Ref-LERF (29.2 average mIoU) and on the LERF-OVS and 3D-OVS open-vocabulary benchmarks, along with a more efficient pipeline than LangSplat.

Significance. If the claims hold, the paper makes a useful contribution by introducing a new task and dataset that connect 3D Gaussian Splatting with referring expression segmentation. The proposed idea of directly modeling language-Gaussian similarity in 3D rather than matching text against rendered 2D features is reasonable and the reported numbers are strong. The authors also provide a practical pseudo-mask generation pipeline and an efficiency comparison. However, the central novelty—handling occluded or non-visible objects—is not quantitatively validated, the baseline comparisons are not controlled for the extra training supervision, and the pseudo-label quality is quite low. These issues currently prevent full confidence in the claimed state-of-the-art.

major comments (4)
  1. [Abstract, Sec. 3.2, Eq. (3), Sec. 4.1] The defining capability of the proposed task—segmenting objects that are occluded or not directly visible in a novel view—is never evaluated as a distinct claim. The evaluation protocol in Sec. 4.1 computes a single average mIoU over all test expressions with no visibility stratification, no occlusion labels, and no definition of how the ground-truth mask is obtained for an entirely invisible object. Moreover, the rasterization in Eq. (3) alpha-composites the per-Gaussian response, so an occluding object with high opacity could suppress the response of the target object. The qualitative examples in Fig. 5 do not replace quantitative evidence. I request a visibility-stratified evaluation, with explicit definitions of masks for partially and fully occluded objects, and a discussion of how the rendering operator can produce a non-empty mask for an invisible target.
  2. [Sec. 4.3-4.4, Table 5] The main comparison against 3D baselines (LangSplat, SPIn-NeRF, GS-Grouping, GOI) in Table 5 is not apples-to-apples. ReferSplat is trained per scene with pseudo-mask supervision generated by Grounded SAM, whereas the baselines are only adapted at test time. The performance gap could therefore reflect the additional task-specific training rather than the proposed 3D spatial paradigm. To support the claim that the method's design is responsible for the gains, the authors should either train the baselines under the same pseudo-mask supervision, or evaluate a version of ReferSplat that does not use task-specific training (e.g., using its pretrained RGB features with only the proposed inference procedure).
  3. [Sec. 3.3, Sec. 3.5, Table 3] The pseudo-masks used for training have only 52.9% and 49.7% mIoU against manual ground truth on ramen and kitchen scenes (Table 3). The two-stage self-training in Sec. 3.5 further uses the first-stage model's rendered masks as supervision, which can entrench errors from the noisy initial pseudo-masks. The ablation in Table 1 shows only a small gain from the second stage (35.2 to 36.9 on ramen, 24.4 to 25.2 on kitchen), but no analysis of whether the self-training improves mask quality against the manually annotated ground truth. I request an error analysis on the two scenes with manual GT, reporting how the final model's masks compare to manual GT and whether self-training reduces or propagates specific types of errors.
  4. [Table 5] The results are reported without any variance or statistical significance. With only four scenes and 59 test expressions, the reported margins (e.g., 29.2 vs. 20.5 for GOI) may be within run-to-run variation. The authors should provide standard deviations across multiple training runs or, at minimum, a per-expression breakdown, and ideally a significance test. This is particularly important because the method involves multiple random components (pseudo-mask selection, contrastive sampling, optimization).
minor comments (6)
  1. [Sec. 4.1] The dataset split is only described as 236 training and 59 test expressions for 59 objects; it is not stated whether expressions for the same object appear in both splits, which affects the difficulty and interpretation of the results.
  2. [Sec. 4.2] The hyper-parameter sentence "we set ��, � , �, and � to 16, 128, 0.3, and 0.02" is unreadable because the symbols do not render; the parameters should be defined with clear names (e.g., feature dimension D, hidden dimension, confidence threshold, loss weight). The schedule for λ is also incomplete.
  3. [Table 3] The table reports both R3DGS mIoU results and mask quality mIoU against manual GT on the same scenes; the distinction should be clarified in the caption or text so the reader does not confuse the two columns.
  4. [Sec. 4.7] The sentence "we conduct experiments comparing BERT and CLIP embeddings for language features in R3DGS in Tab. 9" contains a minor grammar issue and the table number should be referenced more gracefully.
  5. [Fig. 5] The qualitative examples would be more informative if the input text expression were displayed for each case and if the view were accompanied by a reference frame showing the target object's visibility at test time.
  6. [Sec. 6] The limitation statement acknowledges the small number of scenes; I appreciate this honesty, but it should be connected to the reported performance claims, since the four-scene benchmark limits the generality of the conclusions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported results are evaluated against independently annotated ground-truth masks and external baselines, not against the paper's own training targets.

full rationale

ReferSplat's central evaluation (Tables 5-7) is computed against ground-truth object masks and against external baselines, so the headline SOTA claim does not reduce to the paper's inputs. The pseudo-mask generation in Sec. 3.3 is a training-signal heuristic derived from Grounded SAM; it is not the evaluation target. The two-stage optimization in Sec. 3.5 supervises a second model with the first model's rendered masks, but the reported gains (Tab. 1: 36.9 vs 35.2 on ramen) are measured on the same independent GT, so this self-training loop is not used as evidence of correctness by construction. The self-citations in Related Work (He et al. 2023, 2024; He & Ding 2024) are background and are explicitly said not to be directly applicable to R3DGS; they are not load-bearing. No equation in Sec. 3 is defined in terms of the metric it is used to predict. The unsupported occlusion claim is a validity gap, not a circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a standard 3DGS rendering backbone, a set of hyperparameters tuned on the target scenes, the quality of Grounded SAM pseudo-labels, and a small dataset. The only invented internal entity is the referring feature vector, which is an ML component rather than a physical entity.

free parameters (4)
  • Referring feature dimension D = 16
    Set via ablation in Table 10; dimensions 1, 4, 16, and 32 were tested, with 16 giving the best results.
  • Positive selection count k = 128
    Used in Eq. 8 to select top-k percentile of Gaussians as positive examples for contrastive learning; fixed to 128 in Section 4.2.
  • Confidence threshold gamma = 0.3
    Threshold for Grounded SAM candidate mask confidence in pseudo-mask generation (Section 3.3).
  • Contrastive loss weight lambda = 0.02 (initial)
    Weight for contrastive loss in Eq. 10; follows a schedule during training (Section 4.2).
assumptions (4)
  • standard math 3D Gaussian Splatting alpha blending is a correct differentiable model of the scene
    Borrowed from Kerbl et al. 2023 and used throughout Section 3.1; not re-derived.
  • domain assumption Grounded SAM candidate masks contain the correct object mask with sufficiently high confidence
    The pseudo-mask generation relies on Grounded SAM outputs; Table 3 shows only about 50 percent mIoU with manual GT, making this assumption only partially valid.
  • domain assumption The Ref-LERF dataset with four scenes is representative enough to measure progress in R3DGS
    Only four real-world scenes are used, and the authors acknowledge the limitation in Section 6, noting restricted generalization.
  • ad hoc to paper Two-stage self-training with first-stage rendered masks improves, rather than entrenches, errors
    Introduced in Section 3.5 following SPIn-NeRF; only validated on ramen and kitchen scenes in the ablation.
invented entities (1)
  • Gaussian referring feature vector
    purpose: Per-Gaussian learned representation capturing language-aligned semantics for segmentation.
    It is an internal feature field trained on pseudo-labels; it has no externally falsifiable handle and is similar to features in LangSplat and Feature-3DGS.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReferSplat: Referring Segmentation in 3D Gaussian Splatting." pith.science (2026). https://pith.science/paper/DGHQFQJB

@misc{pith2026250808252,
  author       = {Pith},
  title        = {Pith review of: ReferSplat: Referring Segmentation in 3D Gaussian Splatting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DGHQFQJB}},
  note         = {Machine review of arXiv:2508.08252}
}
read the original abstract

We introduce Referring 3D Gaussian Splatting Segmentation (R3DGS), a new task that aims to segment target objects in a 3D Gaussian scene based on natural language descriptions, which often contain spatial relationships or object attributes. This task requires the model to identify newly described objects that may be occluded or not directly visible in a novel view, posing a significant challenge for 3D multi-modal understanding. Developing this capability is crucial for advancing embodied AI. To support research in this area, we construct the first R3DGS dataset, Ref-LERF. Our analysis reveals that 3D multi-modal understanding and spatial relationship modeling are key challenges for R3DGS. To address these challenges, we propose ReferSplat, a framework that explicitly models 3D Gaussian points with natural language expressions in a spatially aware paradigm. ReferSplat achieves state-of-the-art performance on both the newly proposed R3DGS task and 3D open-vocabulary segmentation benchmarks. Dataset and code are available at https://github.com/heshuting555/ReferSplat.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. E3DGS: Unified Geometric-Photometric Equivariance for 3D Gaussian Splatting via Color-as-Geometry Embedding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    3D Gaussian view-dependent colors are repacked as 3×3 matrices so geometry and color rotate together, giving exact rotation-equivariant recognition and world modeling in 3DGS.

  2. MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    MV-GEL uses a learned view selector and a fine-tuned vision-language segmentation model to localize text-described faces and edges on 3D meshes.

  3. G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Segmentation

    cs.CV 2026-01 conditional novelty 6.0 of 10

    G2P transfers opacity and scale attributes from 3D Gaussian Splatting to point clouds, improving semantic segmentation by +1.4 mIoU over PT v3 on ScanNet v2 and +2.1 on ScanNet200.

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages · cited by 3 Pith papers

  1. [3]

    Semantic anything in 3d gaussians.arXiv preprint arXiv:2401.17857,

    Hu, X., Wang, Y ., Fan, L., Fan, J., Peng, J., Lei, Z., Li, Q., and Zhang, Z. Semantic anything in 3d gaussians.arXiv preprint arXiv:2401.17857,

  2. [4]

    2d gaussian splatting for geometrically accurate radiance fields

    Huang, B., Yu, Z., Chen, A., Geiger, A., and Gao, S. 2d gaussian splatting for geometrically accurate radiance fields. InACM SIGGRAPH 2024 conference papers, pp. 1–11,

  3. [5]

    Vr-gs: A physical dynamics-aware interactive gaussian splatting system in virtual reality

    Jiang, Y ., Yu, C., Xie, T., Li, X., Feng, Y ., Wang, H., Li, M., Lau, H., Gao, F., Yang, Y ., et al. Vr-gs: A physical dynamics-aware interactive gaussian splatting system in virtual reality. InACM SIGGRAPH 2024 Conference Papers, pp. 1–1,

  4. [7]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y ., Yan, F., et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159,

  5. [2023]

    Instancegaussian: Appearance-semantic joint gaussian representation for 3d instance-level percep- tion

    Li, H., Wu, Y ., Meng, J., Gao, Q., Zhang, Z., Wang, R., and Zhang, J. Instancegaussian: Appearance-semantic joint gaussian representation for 3d instance-level percep- tion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025a. Li, Y ., Lyu, C., Di, Y ., Zhai, G., Lee, G. H., and Tombari, F. Geogaussian: Geometry-awa...

  6. [2024]

    BERT: pre-training of deep bidirectional transformers for lan- guage understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. InProceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 4171–4186,

  7. [2025]

    High- quality surface reconstruction using gaussian surfels

    Dai, P., Xu, J., Xie, W., Liu, X., Wang, H., and Xu, W. High- quality surface reconstruction using gaussian surfels. In ACM SIGGRAPH 2024 Conference Papers, pp. 1–11,

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.