{"id":"b12c19ba-f565-4b22-9165-042a038b51df","arxiv_id":"2505.24746","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new 3D language Gaussian Splatting method that clusters per-object multi-view CLIP features and reweights them to capture view-dependent semantics, improving direct 3D open-vocabulary segmentation.","lead":"This paper identifies that a 3D object can look semantically different from different viewpoints, an issue it calls view-dependent semantics, and proposes a method called LaGa that aggregates multi-view language features per 3D object. On the LERF-OVS benchmark it reports large gains (+18.7% mIoU) over prior 3D methods and uses the improved 3D understanding for direct object segmentation, editing, and retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +18.7% mIoU headline is computed against the authors' own reimplementation of OpenGaussian under a self-revised evaluation script that is neither described nor released; the comparison protocol, not the method's internal assumptions, is the most load-bearing risk.","rationale":"The reader's verdict is CONDITIONAL, and my independent reading agrees with that disposition, but for a partly different reason. The strongest claim is empirical: LaGa improves LERF-OVS mIoU by +18.7 points over prior SOTA under the same settings. That number is the paper's headline result, and it rests on a comparison against OpenGaussian‡, the authors' reimplementation with a revised evaluation script (footnote 2). Since the revision is undisclosed, the baseline number 45.3 cannot be independently checked, and the +18.7 margin is not reproducible from the paper alone. This is more load-bearing than the decomposition assumption: the decomposition is tested by epsilon and Kmax ablations and shows graceful degradation (Tables 6 and 7), whereas the evaluation protocol has no such stress test. The reader's weakest-assumption choice (3D scene decomposition) is legitimate, but it concerns generalization of the method; the evaluation concern concerns the validity of the central quantitative claim itself. I therefore recommend keeping the CONDITIONAL verdict, with the concrete check being a released, diffed evaluation script and a check of LaGa's margin under the original official script. If the revised script is shown to be unfair, the verdict should move toward REJECT or UNVERDICTED for the stated SOTA claim.","tokens_in":18906,"tokens_out":7681,"duration_ms":97707,"concrete_test":"Obtain the exact diff between the OpenGaussian official LERF-OVS evaluation script and the revised version used here, or have the authors release both scripts and the OpenGaussian‡ predictions. Then run the released LaGa model and the OpenGaussian official checkpoint through both scripts. Accept the +18.7% claim only if (a) the revised script is a demonstrable bug fix (e.g., correcting ground-truth alignment or class counting), and (b) LaGa's margin over the original OpenGaussian result under the original official script remains positive and comparable in magnitude. In addition, run an ablation that removes the 3D bilateral filter and resets the cosine and foreground thresholds to their raw-relevance equivalents; if mIoU drops by more than about 5 points, the view-aggregation contribution is not separable from post-processing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LaGa's central quantitative claim is the +18.7% mIoU improvement over the previous SOTA on LERF-OVS (Abstract, Section 6.2, Table 1). The baseline used for this figure is not the published OpenGaussian result (38.4, marked †) but the authors' reimplementation under a 'revised' evaluation script (45.3, marked ‡), which is +6.9 higher (footnote 2). The paper neither specifies what was revised nor releases the revised script. Because the same revised protocol is applied to LaGa, the headline margin could change if the revision is incorrect, inconsistent with the original benchmark, or systematically favors LaGa's binary Gaussian-mask outputs over OpenGaussian's per-Gaussian relevance outputs. Additional hand-tuned inference components (cosine similarity threshold 0.23, min-max normalization, 3D bilateral filtering, 0.6 foreground threshold, Appendix B) are not ablated, so their contribution to the reported numbers is unknown. The weakest assumption is therefore not primarily the 3D decomposition accuracy, which is partially stress-tested in Tables 6 and 7, but the validity of the evaluation protocol that produces the headline number.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the degradation of direct 3D language understanding in 3D Gaussian Splatting caused by view-dependent semantics, where an object's semantic appearance varies across viewpoints. The proposed method, LaGa, first decomposes the 3D scene into objects using contrastively trained affinity features and HDBSCAN clustering of multi-view SAM masks, then constructs per-object view-aggregated semantic representations by adaptive K-means clustering of CLIP features and weighting the resulting descriptors via a global-alignment and internal-compactness criterion. At inference, LaGa assigns Gaussians to a query by the maximum weighted descriptor relevance, producing binary Gaussian-level segmentation maps. Experiments on LERF-OVS, 3D-OVS, and ScanNet report improvements over prior 3D methods, including a headline +18.7 mIoU gain over OpenGaussian on LERF-OVS.","tokens_in":19152,"tokens_out":2947,"duration_ms":33267,"significance":"If the results hold, LaGa makes a meaningful contribution by identifying and explicitly modeling view-dependent semantics, a factor largely ignored by previous feature-lifting methods. The paper's strengths include experiments on three benchmarks, ablations of the descriptor extraction and weighting components, qualitative analyses of failure cases, and a released codebase. The view-aggregated representation with adaptive clustering and descriptor weighting is a plausible and well-motivated design. However, the headline quantitative claim currently rests on an undisclosed and unreleased revision of the OpenGaussian evaluation script, which changes the comparison baseline by +6.9 mIoU, and on several hand-tuned inference thresholds that are not ablated. These issues must be resolved before the reported margin can be considered reliable, but they are local to the evaluation protocol and do not invalidate the core methodological idea.","major_comments":[{"comment":"The claim of '+18.7% mIoU over the previous SOTA on LERF-OVS' is computed against the authors' reimplementation of OpenGaussian (45.3, marked ‡) rather than the published OpenGaussian result (38.4, marked †). Footnote 2 states that the evaluation script was revised, but the revision is not described and the revised script is not released. Because the same revised protocol is applied to LaGa, the headline margin could change if the revision is incorrect, inconsistent with the original benchmark, or systematically favors LaGa's binary Gaussian-mask outputs. Please specify the exact revisions, release the script, and report LaGa's performance against the published numbers for direct comparison.","section":"Section 6.2, Table 1, footnote 2"},{"comment":"Inference uses several hand-set components: the auxiliary cosine similarity threshold (0.23), min-max normalization, 3D bilateral filtering, and the foreground relevance threshold (0.6). None of these is ablated. If these thresholds were tuned on the evaluation sets, the reported gains could reflect threshold tuning rather than the view-aggregated semantic representation. Please ablate each component or demonstrate that the thresholds are fixed a priori and applied identically to all baselines and all scenes.","section":"Appendix B (Implementation Details), inference thresholds"}],"minor_comments":[{"comment":"The phrase '+18.7% mIoU' is ambiguous: it should be stated as '+18.7 percentage points of mIoU' or as a relative improvement, since the reported numbers are absolute mIoU values.","section":"Abstract and Section 6.2"},{"comment":"The description of how binary Gaussian segmentation maps are rendered into 2D views for evaluation would benefit from a precise statement of whether this rendering protocol matches the original LERF-OVS benchmark protocol used by the baselines, especially given the revision to OpenGaussian's evaluation script.","section":"Section 6.1"},{"comment":"This paragraph argues that SAM masks remain stable across viewpoints, but it does not provide quantitative evidence. Given that the whole pipeline depends on the decomposition grouping masks into objects correctly, a measurable analysis of multi-view mask grouping stability would strengthen the claim.","section":"Section 5.3, 'Why Is 3D Scene Decomposition Unaffected by View-Dependency?'"},{"comment":"The use of a gray background to mark superior results within each 'DW' group is difficult to discern in print; consider using bold or a footnote instead.","section":"Section 6.4, Table 4"},{"comment":"Typos and duplicated text: 'negative affect' in the introduction, 'assemantic' in Section 5.4, and a duplicated sentence in Appendix C.5 ('A more intriguing example involves transplanting a “cup” ...').","section":"Various"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth taking seriously: instead of assigning one rigid semantic vector to a 3D object, LaGa clusters multi-view CLIP features into several descriptors and weights them by directional consistency and internal compactness. That is a sensible response to the view-dependency problem they document, and the ablation study actually supports it—adaptive clustering beats fixed and average pooling, and the weighting terms both matter. The method is simple and the qualitative results look clean. I think the paper does a fair job of situating itself against OpenGaussian, LangSplat, and others, and the failure-case section is honest.\n\nThe soft spot is the comparison protocol. The +18.7% mIoU figure on LERF-OVS is computed against the authors' own reimplementation of OpenGaussian (45.3 mIoU, marked ‡), not the published number (38.4). Footnote 2 says they revised OpenGaussian's official evaluation script and gained 6.9 mIoU, but they never say what they revised, and the script is not released. That matters for two reasons: the baseline becomes non-reproducible, and the change could conceivably favor LaGa's binary mask outputs over OpenGaussian's per-Gaussian relevance outputs. The direction of the change is not obvious without seeing the script. So the exact margin is uncertain, even though the method's internal logic is sound.\n\nA second, smaller issue is that inference relies on a chain of hand-tuned thresholds and post-processing steps—0.23 cosine threshold, min-max normalization, 3D bilateral filtering, 0.6 foreground threshold. None of these are ablated, so their contribution to the reported numbers is unknown. That is fixable but should be addressed if the paper is revised.\n\nI would not let the evaluation script issue sink the paper; the method itself is new and the experiments are more extensive than most in this subfield. But if I were citing the +18.7%, I'd add a caveat. A referee should ask for the revised OpenGaussian script and a sensitivity analysis of the thresholds.\n\nFor us: this is worth discussing at reading group, and I'd cite it for the descriptor-weighting idea. I'd send it out for peer review if it were under consideration, with the evaluation script question front and center.","headline":"The view-aggregated descriptor idea is solid and the ablations are convincing, but the headline gain over OpenGaussian rests on an opaque revised evaluation script that the authors do not describe or release.","tokens_in":19740,"tokens_out":3088,"would_cite":true,"duration_ms":36191,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LaGa lifts 3D language segmentation by 18.7% mIoU","keywords":["3D Gaussian Splatting","open-vocabulary 3D segmentation","view-dependent semantics","language-grounded scene understanding","scene decomposition","view aggregation","CLIP features","SAM masks"],"falsifier":"The decisive experiment is to run LaGa with ground-truth or manually verified object groupings instead of the learned HDBSCAN decomposition on the LERF-OVS scenes. If the reported 18.7-point mIoU gain over the previous 3D state of the art persists under oracle groupings, the view-aggregated descriptors are the driver; if the gain vanishes, the improvement comes from the decomposition stage rather than from handling view-dependent semantics.","tokens_in":18663,"feed_emoji":"🧩","tokens_out":10175,"duration_ms":101462,"temperature":0.7,"pith_summary":"This paper argues that the main obstacle to direct 3D language understanding in 3D Gaussian Splatting is not how features are stored but the fact that one physical object carries different visible semantics from different viewpoints. Simply lifting per-view 2D features onto Gaussians leaves each Gaussian with only the semantics visible from the views that trained it, producing both false positives and false negatives at query time. LaGa handles this by first decomposing the 3D scene into objects, then representing each object by an adaptively chosen set of semantic descriptors obtained by clustering its multi-view features, with weights that reward descriptors aligned with the object's global semantics and internally compact. At query time the object's relevance to a text is the maximum weighted response over its descriptors, so a query that matches any reliable view retrieves the whole object. On the LERF-OVS benchmark LaGa reaches 64.0% mIoU, 18.7 points above the previous 3D state of the art and 8.8 points above the best 2D-rendering methods, and it also improves on prior methods for ScanNet point-cloud segmentation.","feed_headline":"LaGa lifts 3D language segmentation by 18.7% mIoU","feed_subtitle":"The same object looks different from different viewpoints; LaGa clusters multi-view features per object to fix it.","key_machinery":"The load-bearing object is the view-aggregated semantic representation, built per 3D object in two stages. First, a contrastively trained per-Gaussian affinity feature (32-dimensional, with masked-average-pooled mask prototypes clustered by HDBSCAN, a density-based clustering algorithm) decomposes the scene into objects and assigns every Gaussian to one object, so each object carries all its multi-view 2D semantic features. Second, adaptive K-means with silhouette-score selection partitions those features into one to twenty semantic descriptors, and each descriptor $d$ receives a weight $\\omega_d = \\langle d, \\bar{v}_{S_i}\\rangle \\cdot \\|d\\|_2$, where $\\bar{v}_{S_i}$ is the mean feature of the object: the first factor rewards directional consistency with the object's global semantics, the second rewards internal compactness of the descriptor's cluster. The object-level relevance to a text query $q$ is then $\\mathrm{REL}(G_{S_i}, q) = \\max_{d\\in D_{G_{S_i}}} \\omega_d \\cdot \\mathrm{Rel}(d,q)$, with Rel following LERF's canonical-phrase normalization. This machinery is what preserves the semantic sides of an object—for example, a book's spine and its title—instead of averaging them away or locking the object into one rigid feature.","core_discovery":"On the paper's own terms, the central discovery is that view-dependent semantics—the same 3D object producing different CLIP features from different viewpoints—is a measurable and fixable cause of degraded Gaussian-level language understanding. The paper quantifies the problem with two analyses: intra-object multi-view features are often less similar to each other than they are to features of other objects, and about half of 2D semantic features cannot fully retrieve their own 3D object. LaGa's answer is to connect views through object-level decomposition and then represent each object with a small set of view-aggregated descriptors rather than a single averaged feature: adaptive K-means selects how many descriptors an object needs, and a weighting scheme suppresses descriptors that point away from the object's dominant semantics or that are internally inconsistent. Inference takes the maximum weighted relevance over these descriptors for each object, producing direct Gaussian-level segmentation without rendering feature maps. The paper's headline evidence is a 18.7-point mIoU improvement over the previous 3D state of the art on LERF-OVS, with the largest gains on the 360-degree scenes where view-dependency is most severe.","pith_inferences":["Beyond the paper: the view-aggregation recipe is not tied to 3D-GS; it could be applied to any per-point feature field lifted from multi-view 2D encoders, including NeRF-based language fields or dynamic 4D reconstructions where visibility changes over time.","Editorial inference: the silhouette-selected cluster count is effectively the number of semantic sides an object has from the training viewpoints; geometry (surface normals or visibility maps) could predict this count more cheaply than clustering features.","Beyond the paper: the retrieval-integrity analysis could be reused as a view-dependency index for scene datasets, letting a practitioner decide in advance whether view aggregation is worth the added pipeline complexity.","Editorial inference: because LaGa inherits CLIP's bag-of-words and context-loss failures, the value of view aggregation should increase as the 2D semantic backbone improves; the method treats the backbone as an exchangeable module."],"forward_implications":["Gaussian-level open-vocabulary queries become practical for editing, deletion, and transplanting objects across scenes, since segmentation is produced directly on the 3D Gaussians rather than on rendered feature maps.","Per-object descriptor sets replace per-Gaussian high-dimensional language features, which the paper argues reduces storage and accelerates inference while sharing one semantic pool across all Gaussians of an object.","The two diagnostic analyses (intra/inter similarity overlap and the roughly 50% low-recall retrieval rate) give the field a concrete way to measure how strongly view-dependency affects any new 3D language dataset or method.","On forward-facing or nearly saturated benchmarks such as 3D-OVS, gains are small, which follows from the claim: when viewpoint diversity is low, view-dependency is not the bottleneck."],"supporting_citations":[{"why":"Provides 3D Gaussian Splatting, the explicit per-Gaussian representation and differentiable rasterizer that LaGa augments with affinity and semantic features.","marker":"Kerbl et al., 2023"},{"why":"Supplies CLIP, the vision-language model whose per-view features and text embeddings LaGa clusters and scores.","marker":"Radford et al., 2021"},{"why":"Supplies SAM, whose multi-view 2D masks LaGa groups into 3D objects.","marker":"Kirillov et al., 2023"},{"why":"Supplies the LangSplat data-preparation pipeline (SAM masks plus CLIP cropped features) that LaGa follows, and is a principal 2D-rendering baseline.","marker":"Qin et al., 2024"},{"why":"Supplies LERF, the relevance formula with canonical-phrase normalization, and the LERF-OVS benchmark used for the main comparisons.","marker":"Kerr et al., 2023"},{"why":"Supplies HDBSCAN, the clustering algorithm that groups multi-view mask prototypes into 3D objects.","marker":"McInnes et al., 2017"},{"why":"Supplies OpenGaussian, the previous direct-3D state of the art and the main 3D baseline, plus the ScanNet evaluation protocol.","marker":"Wu et al., 2024b"},{"why":"Supplies the contrastive scene-decomposition training strategy and feature norm regularization reused for Gaussian affinity features.","marker":"Cen et al., 2025a"}],"fun_headline_variants":["LaGa fixes 3D vision's viewpoint bias","View-dependent semantics solved: LaGa lifts mIoU by 18.7%","LaGa: object-level 3D language understanding","New method LaGa beats prior 3D language segmentation by 18.7%","View-dependent 3D semantics: LaGa clusters per object"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the grouping of multi-view segmentation masks produced by the Segment Anything model into 3D objects is accurate and viewpoint-invariant; if that grouping merges two different objects or splits one object, every semantic descriptor built on those groups inherits the error and the pipeline collapses.","fun_headline_variants_meta":{"raw":{"variants":["LaGa fixes 3D vision's viewpoint bias","View-dependent semantics solved: LaGa lifts mIoU by 18.7%","LaGa: object-level 3D language understanding","New method LaGa beats prior 3D language segmentation by 18.7%","View-dependent 3D semantics: LaGa clusters per object"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1448,"prompt_tokens":993,"completion_tokens":455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":609,"tokens_out":455,"duration_ms":4906,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:14:55.460324+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The decisive experiment is to run LaGa with ground-truth or manually verified object groupings instead of the learned HDBSCAN decomposition on the LERF-OVS scenes. If the reported 18.7-point mIoU gain over the previous 3D state of the art persists under oracle groupings, the view-aggregated descriptors are the driver; if the gain vanishes, the improvement comes from the decomposition stage rather than from handling view-dependent semantics.","supporting_citations":[{"cited_title":"C., Lo, W.-Y., et al","cited_arxiv_id":null,"evidence_quote":"Supplies SAM, whose multi-view 2D masks LaGa groups into 3D objects."},{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Provides 3D Gaussian Splatting, the explicit per-Gaussian representation and differentiable rasterizer that LaGa augments with affinity and semantic features."},{"cited_title":"Langsplat: 3d language gaussian splatting","cited_arxiv_id":null,"evidence_quote":"Supplies the LangSplat data-preparation pipeline (SAM masks plus CLIP cropped features) that LaGa follows, and is a principal 2D-rendering baseline."},{"cited_title":"M., Goldberg, K., Kanazawa, A., and Tancik, M","cited_arxiv_id":null,"evidence_quote":"Supplies LERF, the relevance formula with canonical-phrase normalization, and the LERF-OVS benchmark used for the main comparisons."},{"cited_title":"hdbscan: Hierarchical density based clustering","cited_arxiv_id":null,"evidence_quote":"Supplies HDBSCAN, the clustering algorithm that groups multi-view mask prototypes into 3D objects."}],"review_version":1}