Pith. sign in

REVIEW 4 major objections 4 minor 59 references

ZeroSplat claims that 3D referring segmentation can handle 0, 1, or N target objects without any per-scene training or semantic feature storage, by lifting 2D vision-language cues into the 3D Gaussian field and refining them with multi-view

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:17 UTC pith:NDOXAEPJ

load-bearing objection A genuinely new task transfer and a clean training-free pipeline, but the SOTA claims rest on author-built benchmarks and one internal contradiction that must be fixed before the headline numbers can be trusted. the 4 major comments →

arxiv 2607.18801 v1 pith:NDOXAEPJ submitted 2026-07-21 cs.CV

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Jiayu Ding , Meilu Song , Xiaoyi Zhang , Hongbo Jin , Yichen Jin , Xiangtian Si This is my paper
classification cs.CV
keywords 3D Gaussian SplattingReferring SegmentationGeneralized 0/1/N targetsVision-Language ModelMulti-view geometric constraintsTraining-freeZero-featureOpen-vocabulary segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a new task, Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS), where a language instruction may refer to zero, one, or many objects in a 3D scene. It proposes ZeroSplat, a training-free and zero-feature framework that uses a vision-language model to parse the instruction into semantic labels and 2D boxes, generates 2D masks with SAM3, and lifts them into 3D Gaussians using volume-rendering weights, then prunes and fills gaps with multi-view verification and KNN diffusion. The authors claim ZeroSplat substantially outperforms existing single-target and open-vocabulary methods on their new GR-LERF and GR-ScanNet benchmarks (e.g., 50.8 vs 31.6 mIoU on GR-LERF, 41.2 vs 24.5 on GR-ScanNet) while requiring no per-scene optimization and no extra feature storage. A sympathetic reader would see this as a shift from training-heavy, feature-storing 3D semantic models toward a decoupled, plug-and-play approach that treats the 3D Gaussians themselves as the units of understanding.

Core claim

ZeroSplat's central discovery is that robust 3D semantic understanding for referring expressions does not require altering the underlying 3D scene representation with learned semantic features. Instead, by projecting 2D vision-language model priors into the Gaussian field through geometric constraints — using each Gaussian's rendering contribution weight (transmittance times opacity) as a soft correspondence between pixels and points — a training-free pipeline can assign labels to individual Gaussians. The method also introduces a generalized task definition (0/1/N targets) and two new benchmarks, GR-LERF and GR-ScanNet, to measure pixel-level and point-level performance. The reported result

What carries the argument

The central mechanism is 'lifting with geometric constraints': 2D masks from SAM3 are back-projected into 3D by computing, for each Gaussian, the sum of its volumetric rendering weights across the pixels where it appears; a Gaussian is labeled foreground if its foreground weight exceeds its background weight. This is augmented by a cascade of multi-view filters — cross-view background cropping, box-based back-projection verification, and a KD-Tree KNN label diffusion that fills internal cavities — all operating on the original Gaussian primitives without adding any learned feature per point.

Load-bearing premise

The two new benchmarks used to demonstrate the state-of-the-art results are constructed, annotated, and evaluated by the authors themselves (with a three-person panel, but without release or reported inter-annotator agreement), and the zero-target scoring rule (IoU=1 for any correct empty mask, 0 otherwise) is a design choice that directly shapes the anti-hallucination numbers.

What would settle it

Run ZeroSplat on an independently constructed 0/1/N referring 3D segmentation benchmark where the instructions and ground-truth masks are created by annotators who are blind to the method's VLM prompts and scoring rules, then compare mean mIoU against the reported 50.8 (GR-LERF) and 41.2 (GR-ScanNet); if the margin over the best baseline shrinks to near baseline-level, the benchmark design is the load-bearing component.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, 3D referring segmentation becomes a plug-and-play operation that works directly on any existing 3DGS scene without scene-specific training, enabling real-time interactive applications.
  • The 0/1/N formulation unifies single-target, multi-target, and negative (absent-object) queries, which is closer to how users actually issue instructions and could transfer to embodied agents and editing tools.
  • Because the pipeline stores no semantic features, memory overhead is essentially zero, making deployment on resource-constrained devices feasible.
  • The explicit point-level labels produced by the method directly support object-level editing, such as translation, swapping, and appearance stylization of Gaussian primitives, as demonstrated in the appendix.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If ZeroSplat's approach generalizes, it suggests that heavy semantic feature distillation into 3D representations may be unnecessary; any improvement in the vision-language model directly translates into better 3D segmentation with no re-training, so the pipeline's ceiling is tied to VLM progress rather than to the 3D representation itself.
  • The same lifting-and-verification recipe could likely be adapted to other explicit 3D representations (point clouds, meshes, or 4D splats) by replacing the Gaussian rendering weight with the appropriate projection function, a testable extension the paper does not explore.
  • The reported state-of-the-art margins depend on author-constructed benchmarks and a binary zero-target scoring rule; an independent benchmark with fresh instructions and blind annotation might show different relative rankings, especially on anti-hallucination behavior.
  • The paper's efficiency analysis (total pipeline ~38 minutes on a scene, versus hours of training for baselines) suggests that the bottleneck for such zero-feature methods will shift to the VLM's inference cost and SAM mask quality, not to the 3D geometry stage.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS), a task requiring segmentation of an arbitrary number of targets (0, 1, or N) from a free-form text instruction, together with two new benchmarks, GR-LERF and GR-ScanNet. The proposed method, ZeroSplat, is a training-free and zero-feature framework: it uses geometry-guided keyframe selection, a two-stage VLM prompt for semantic-label extraction and 2D bounding-box localization, SAM3 masks, back-projection of 2D masks into 3D via Gaussian volume-rendering weights, cross-view background cropping and box verification, and a final KD-Tree KNN diffusion step. The main empirical claim is strong performance on GR3DGS (Table 2: 50.8 vs 31.6 mIoU on GR-LERF; 41.2 vs 24.5 on GR-ScanNet), a training-free advantage on single-target Ref-LERF (Table 3: 32.7 vs 29.2), and efficiency gains over per-scene optimization methods.

Significance. If the benchmark evidence is accepted, the paper would make a useful contribution: it formalizes a realistic 0/1/N referring-segmentation task, provides a strong training-free baseline with no auxiliary feature storage, and supports the design choices with detailed ablations (Tables 6–8) and a hyperparameter robustness study (Table 13). The method is transparent and reproducible: thresholds are specified in Section 4.2, and the exact VLM prompts are given in Supplementary H.2. However, external validity is not yet established. The GR3DGS headline margins rest entirely on two author-constructed benchmarks that are not released and that use a non-standard zero-target IoU rule, and one of the three headline claims (open-vocabulary SOTA) is directly contradicted by the paper's own Table 5. These issues are load-bearing and need to be fixed before the central claims can be accepted.

major comments (4)
  1. [§4.6, Table 5] The text states that ZeroSplat "achieves 52.4 mIoU on LERF, surpassing the previous best by 2.0 mIoU and establishing a new state of the art," but Table 5 lists 3DVLGS at 62.0 and Occam's LGS at 61.3, both well above 52.4. The claim is only true among point-based methods (where the previous best is LUDVIG at 50.4). As written, the open-vocabulary SOTA claim is unsupported and must be corrected by explicitly restricting the comparison or revising the claim.
  2. [§4.4; Supplementary A, H.1–H.2] The central GR3DGS results are evaluated exclusively on GR-LERF and GR-ScanNet, which are constructed, annotated, and scored by the authors and are not released. Supplementary A does not provide inter-annotator agreement, a full list of instructions, or details on disagreement resolution, and the zero-target scoring rule in H.1 (IoU = 1 for any correct empty mask, 0 for any non-empty prediction) is a non-standard design choice. With 25/111 zero-target instructions in GR-LERF, this rule can shift the mean by several points and systematically penalizes baselines that were not designed to abstain. Because the H.2 prompt protocol explicitly enforces a "Color + Noun" extraction that mirrors the method's own VLM prompt, there is a real risk that the benchmark is shaped to the method. The authors should release the benchmarks, report inter-annotator agreement, provide per-target-type (0/1/N) re
  3. [§4.7, Table 13] The hyperparameter sensitivity analysis is conducted on the waldo-kitchen scene, which is itself one of the four GR-LERF scenes used in Table 2. Using an evaluation scene for threshold selection or confirmation invalidates the independence of that scene's result. Moreover, the τconf row is not as flat as claimed: lowering τconf from the default 0.8 to 0.4 improves waldo-kitchen mIoU from 48.4 to 51.9, a 3.5-point gain. The sweep should be done on a held-out scene and results should also be reported separately for zero-, single-, and multi-target instructions.
  4. [§4.3, Supplementary B (Tables 10–11)] The efficiency claims are internally inconsistent. Table 10 reports 2306 s per query on the Teatime scene, with the stage "2D-to-3D Lifting & Filtering & Spatial Refinement" alone taking 1195 s. Table 11 reports Ours with Data Prep 1111 s, Lift 33 s, and a total of 1144 s, which omits the 1195 s filtering/refinement stage. Since zero-training and high efficiency are a central claimed advantage, the pipeline stage definitions must be reconciled and the runtime breakdown corrected.
minor comments (4)
  1. [Table 11] The baseline name "LUDIG" should be "LUDVIG".
  2. [Eq. (2)] The min-max normalization is undefined if σmax = σmin; a numerical guard should be specified.
  3. [§4.2 / Supplementary H.4] The ScanNet 10-class protocol in Table 4 is not discussed in the main text: Ours (49.7) is below Dr.Splat (50.8) under that protocol. The text only claims best results for 19- and 15-class protocols, which is accurate, but the discrepancy should be acknowledged for completeness.
  4. [General] The paper does not provide a code or benchmark release link. For a new task with new benchmarks, public release is essential for reproducibility and for the claims to be verifiable.

Circularity Check

0 steps flagged

No circular derivation: ZeroSplat's pipeline is a forward feature-lifting procedure with fixed thresholds, and its evaluation benchmarks/metrics are design choices rather than fitted inputs or self-citation-supported premises.

full rationale

I walked the claimed derivation chain: VLM semantic parsing (Sec. 3.2), SAM3 mask generation and 3D lifting (Sec. 3.3), cross-view verification and KNN spatial diffusion (Sec. 3.4), then evaluation on the author-constructed GR-LERF/GR-ScanNet benchmarks. None of these steps are defined in terms of the evaluation outputs. The thresholds (tau_high, tau_base, tau_conf, tau_box, tau_views, k, tau_knn, etc.) are stated constants in Sec. 4.2, not fit to GR-LERF or GR-ScanNet labels; the only sensitivity study (Table 13) is performed on the waldo-kitchen scene, which is part of the GR-LERF test set, but the paper does not claim to fit parameters from those labels, and the reported variation is small and not used to cherry-pick a best profile (e.g., tau_conf=0.4 gives 51.9 while the default 0.8 gives 48.4). This is a benchmark-integrity concern, not a fitted-input-called-prediction circularity. The zero-target IoU rule (Supp. H.1) is an author-defined metric, and the VLM prompt (Supp. H.2) tells the model to emit -1 for absent targets; this aligns the metric with the method's design, but it is an evaluation-protocol choice applied equally to all methods, not a parameter fitted to the data, and it does not make the method's outputs equal to its inputs. Self-citations (ExtrinSplat [4], etc.) appear in related-work taxonomies and are not load-bearing for the central GR3DGS claim; the novelty is explicitly contrasted with the external 3D-GRES task [37] and with external baselines. I note one non-circular internal inconsistency that affects confidence in reported numbers: Sec. 4.6 claims ZeroSplat 'surpasses the previous best by 2.0 mIoU' on LERF, but Table 5 lists Ours at 52.4 against 3DVLGS at 62.0. That contradiction is a correctness/reporting issue, not a circular-reasoning step. Overall, no equation reduces to its own input and no load-bearing argument depends solely on the authors' prior work, so the circularity score is 0.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 2 invented entities

ZeroSplat is a systems paper with no theoretical derivation. Its central claims rest on (a) the reliability of black-box VLM/SAM3 outputs, (b) the validity of self-constructed, unreleased benchmarks, and (c) roughly eight hand-set threshold groups. The 0/1/N generalization is borrowed from 3D-GRES; the new content is the specific no-feature integration and the new evaluation resources.

free parameters (8)
  • Mask confidence view-selection thresholds tau_high / tau_base / tau_safe = 0.6 / 0.3 / 0.15 (tau_safe=0 for open-vocabulary)
    Hand-set thresholds for choosing which views contribute SAM3 masks; sensitivity checked only on one evaluation scene (Table 13).
  • View quantity targets N_target / N_safe = 30 / 6
    Controls how many views are fused in Eq. 4; chosen by hand, no held-out tuning.
  • Mask fusion threshold tau_merge = 0.8
    Decides whether to merge the sub-optimal SAM3 mask in Eq. 5; hand-set.
  • Cross-view background-crop conflict ratio tau_conf = 0.8
    Prunes Gaussians whose 2D projections frequently fall on background (Eq. 9); hand-set; Table 13 shows 0.4-0.6 would score higher on the tested scene.
  • Back-projection verification thresholds tau_views / tau_box = 8 / 0.8
    Minimum observation count and out-of-bounds tolerance in Eq. 10; hand-set.
  • KNN diffusion radius k and reliability tau_knn = 40 / 0.8
    Fill internal cavities with Eq. 11; k and threshold chosen by hand.
  • Number of keyframes K = 30
    Selected greedily by curvature gain (Eq. 3); hand-set.
  • Keyframe curvature parameters s, gamma, eta, [wmin, wmax] = 0.1, 0.5, 2.5, [1, 10]
    Control sub-voxel downsampling, search radius, and saliency normalization in §3.2; hand-set.
axioms (5)
  • domain assumption The 3DGS scene geometry, camera poses, and depth maps from scene reconstruction are accurate enough for reliable back-projection.
    Used throughout §3.3-3.4; Eq. 8 depends on projected pixels and per-Gaussian contribution weights. If reconstruction is noisy, labels are wrong.
  • domain assumption Off-the-shelf VLM (Qwen3-VL-30B) and SAM3 produce correct labels, boxes, and masks for the instruction distribution.
    The entire pipeline inherits these errors; Appendix E explicitly admits failures under dense clutter and wrong label extraction.
  • domain assumption Objects are spatially contiguous in Gaussian space, so KNN diffusion (Eq. 11) heals cavities without bleeding across boundaries.
    Used in §3.4; fails for thin structures, transparent objects, and reflective surfaces as acknowledged in Appendix E.
  • domain assumption The manually annotated ground truth in GR-LERF and GR-ScanNet is correct and unbiased.
    Supplementary A describes a three-person panel but provides no release, no inter-annotator agreement, and no third-party validation; all central mIoU numbers rest on these labels.
  • standard math The volume-rendering alpha compositing model from 3DGS is standard.
    Preliminary §2; used to derive the contribution weights w(r,g_j) in Eq. 7.
invented entities (2)
  • GR3DGS task and its evaluation protocols (0/1/N target semantics, binary zero-target IoU rule) no independent evidence
    purpose: Formalize instruction ambiguity for referring segmentation in 3DGS
    New task definition with author-defined scoring (Supplementary H.1); no external benchmark or third-party check exists.
  • GR-LERF and GR-ScanNet benchmarks (111 and 83 instructions) no independent evidence
    purpose: Evaluate pixel-level and point-level generalized referring segmentation
    Created and annotated by the authors; not released, no inter-annotator agreement, so the SOTA numbers cannot be independently checked.

pith-pipeline@v1.3.0-alltime-deepseek · 18206 in / 19946 out tokens · 166965 ms · 2026-08-01T14:17:31.176678+00:00 · methodology

0 comments
read the original abstract

Recent advancements in 3D Gaussian Splatting (3DGS) have enabled language-guided scene understanding. However, existing Referring 3D Gaussian Splatting (R3DGS) methods are fundamentally restricted to single-target queries. To reflect the ambiguity of real-world instructions, we introduce the Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS) task, which requires dynamically segmenting an arbitrary number of targets (0, 1, or $N$). To facilitate comprehensive evaluation of this new task, we construct two new benchmarks: GR-LERF and GR-ScanNet. Crucially, existing R3DGS paradigms exhibit fundamental technical bottlenecks that severely limit their performance on the GR3DGS task: they lack intrinsic 3D point-level understanding by operating merely on 2D rendered pixels, and they incur prohibitive computational overhead by requiring per-scene optimization to embed heavy semantic features. To dismantle these bottlenecks, we propose ZeroSplat, a novel training-free and zero-feature framework. ZeroSplat lifts 2D Vision-Language Model (VLM) priors into 3D space through robust multi-view geometric constraints. This strategy enables intrinsic point-level understanding without incurring any additional feature storage. Extensive experiments demonstrate that ZeroSplat significantly outperforms state-of-the-art methods across generalized and single-target scenarios while maintaining exceptional efficiency. Project Page: https://inkmind-ai.github.io/ZeroSplat

Figures

Figures reproduced from arXiv: 2607.18801 by Hongbo Jin, Jiayu Ding, Meilu Song, Xiangtian Si, Xiaoyi Zhang, Yichen Jin.

Figure 1
Figure 1. Figure 1: Traditional R3DGS is limited to single-target cases(1). In contrast, GR3DGS can handle scenarios with any number of targets, including no target (2), single target, and multiple targets (3-5). names or simple noun phrases. This limitation impedes free-form language un￾derstanding, core to Embodied AI [5, 17–19, 34] and multimodal LLM agent [1, 12, 13, 39, 40], with user queries carrying fine-grained attrib… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our method. (a) Keyframes are first extracted from the input scene. A VLM then performs a two-stage parsing: the first extracts semantic labels from text via multi-view keyframe analysis, while the second localizes targets with 2D bounding boxes using these labels, referring text, and keyframes. (b) Guided by these labels, partial multi-view 2D masks are extracted across all scene views and use… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative results on object selection from the GR-LERF dataset. the original papers to ensure fair comparisons. Due to representation constraints, some pixel-only methods are not evaluated on GR-ScanNet. Results on GR-LERF. As shown in [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on object selection from the GR-ScanNet dataset. contrast, our approach accurately grounds intricate natural language descrip￾tions, isolating the exact queried instances with crisp boundaries and high se￾mantic purity [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparisons on the Ref-LERF dataset. degradation only under logical extremes, such as τsafe → 0, which disables the safety mechanism. D More Qualitative Results To further demonstrate the superiority of our approach, we provide compre￾hensive qualitative comparisons against state-of-the-art baselines on the LERF, ScanNet, and Ref-LERF datasets. Qualitative analysis on Ref-LERF [PITH_FULL_IMAGE… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparisons on the LERF dataset. accurately localizes targets and consistently produces tight boundaries. This im￾provement directly benefits from our Cross-View Background Cropping module. By enforcing spatial constraints from VLM-generated 2D bounding boxes across keyframes, we effectively prune out-of-bounds geometric artifacts and prevent background inclusion. Qualitative analysis on ScanNe… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparisons on the ScanNet dataset. contrast, our approach directly decouples and manipulates the attributes of 3D Gaussian primitives in 3D space using natural language instructions. Spatial translation and swapping. Because 3D Gaussian primitives possess explicit spatial coordinates, our method enables physical-level spatial transfor￾mations of specific instances. As illustrated in [PITH_FUL… view at source ↗
Figure 8
Figure 8. Figure 8: Spatial translation and swapping. Modifying the center coordinates of target Gaussians enables seamless translation and swapping of independent objects [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Instance-level appearance stylization. Adjusting the spherical harmonics (SH) coefficients of selected Gaussians enables high-fidelity object color editing [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative results on real-world smartphone captures. ZeroSplat robustly handles complex lighting, occlusions, and long-form descriptive instructions [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Stage 1 Prompt: Semantic label extraction. The prompt enforces spatial rea￾soning and standardizes natural language descriptions into a "Color + Noun" format [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Stage 2 Prompt: 2D geometric localization. This prompt guides the VLM to perform cross-image spatial reasoning and strictly handle the zero-target scenario by outputting -1 [PITH_FULL_IMAGE:figures/full_fig_p027_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 11 linked inside Pith

  1. [1]

    In: The 2nd Workshop on Foundation Models Meet Embodied Agents at CVPR 2026

    Chen, N., Liu, L., Li, Z., Zeng, Z., Zhu, Z., Cong, W., Hong, J., Yang, Y., Tu, Z., Wang, Y., et al.: A physics-grounded benchmark for multi-agent dynamics in world models. In: The 2nd Workshop on Foundation Models Meet Embodied Agents at CVPR 2026

    2026

  2. [2]

    In: 36th British Machine Vision Conference 2025, BMVC 2025, Sheffield, UK, November 24-27, 2025

    Cheng, J., Zaech, J.N., Gool, L.V., Paudel, D.P.: Occam’s lgs: An efficient approach for language gaussian splatting. In: 36th British Machine Vision Conference 2025, BMVC 2025, Sheffield, UK, November 24-27, 2025. BMVA (2025)

    2025

  3. [3]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Dai,A.,Chang,A.X.,Savva,M.,Halber,M.,Funkhouser,T.,Nießner,M.:Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5828–5839 (2017)

    2017

  4. [4]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ding, J., Liu, X., Pan, Z., Long, S., Li, G.: Extrinsplat: Decoupling geometry and semantics for open-vocabulary understanding in 3d gaussian splatting. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 31019–31028 (2026)

    2026

  5. [5]

    arXiv preprint arXiv:2601.05991 (2026)

    Ding, J., Tang, H., Jin, H., Gao, W., Li, G.: 3d instruction ambiguity detection. arXiv preprint arXiv:2601.05991 (2026)

    Pith/arXiv arXiv 2026

  6. [6]

    In: Proceedings of the 32nd ACM International Conference on Multimedia

    He, S., Ding, H.: Refmask3d: Language-guided transformer for 3d referring segmen- tation. In: Proceedings of the 32nd ACM International Conference on Multimedia. pp. 8316–8325 (2024)

    2024

  7. [7]

    In: International Conference on Machine Learning (ICML)

    He, S., Jie, G., Wang, C., Zhou, Y., Hu, S., Li, G., Ding, H.: ReferSplat: Referring segmentation in 3d gaussian splatting. In: International Conference on Machine Learning (ICML)

  8. [8]

    In: Proceedings of the AAAI conference on artificial intelligence

    Huang, P.H., Lee, H.H., Chen, H.T., Liu, T.L.: Text-guided graph neural networks for referring 3d instance segmentation. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35, pp. 1610–1618 (2021)

    2021

  9. [9]

    arXiv preprint arXiv:2506.22799 (2025)

    Jiang, M., Jia, S., Gu, J., Lu, X., Zhu, G., Dong, A., Zhang, L.: Votesplat: Hough voting gaussian splatting for 3d scene understanding. arXiv preprint arXiv:2506.22799 (2025)

    Pith/arXiv arXiv 2025

  10. [10]

    arXiv preprint arXiv:2505.11830 (2026)

    Jin, H., Ding, J., Xie, S., Luo, G., Li, G.: Vista: Mitigating semantic inertia in video-llms via training-free dynamic chain-of-thought routing. arXiv preprint arXiv:2505.11830 (2026)

    arXiv 2026

  11. [11]

    arXiv preprint arXiv:2605.26827 (2026)

    Jin, H., Wang, C., Tang, H., Du, Z., Jiang, X., Tian, J., Zhang, Q., Ding, J.: Contextguard: Structured self-auditing for context learning in language models. arXiv preprint arXiv:2605.26827 (2026)

    Pith/arXiv arXiv 2026

  12. [12]

    arXiv preprint arXiv:2605.25354 (2026)

    Jin, H., Zhu, M., Tian, J., Jiang, X., Du, Z., Tang, H., Xie, S., Zhang, Q., Ding, J.: Context-cot: Enhancing context learning via high-quality reasoning synthesis. arXiv preprint arXiv:2605.25354 (2026)

    Pith/arXiv arXiv 2026

  13. [13]

    arXiv preprint arXiv:2603.00977 (2026)

    Jin, H., Zhu, R., Ding, J., Zhang, W., Li, G.: Himac: Hierarchical macro-micro learning for long-horizon llm agents. arXiv preprint arXiv:2603.00977 (2026)

    Pith/arXiv arXiv 2026

  14. [14]

    splat: Directlyreferring3Dgaussiansplattingviadirectlanguageembeddingregistration

    Jun-Seong, K., Kim, G., Yu-Ji, K., Wang, Y.C.F., Choe, J., Oh, T.H.: Dr. splat: Directlyreferring3Dgaussiansplattingviadirectlanguageembeddingregistration. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 14137–14146 (2025)

    2025

  15. [15]

    arXiv preprint arXiv:2308.04079 (2023), arXiv:2308.04079

    Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D Gaussian Splatting for Real-Time Radiance Field Rendering. arXiv preprint arXiv:2308.04079 (2023), arXiv:2308.04079

    Pith/arXiv arXiv 2023

  16. [16]

    In: International Conference on Computer Vision (ICCV) (2023) 16 J

    Kerr, J., Kim, C.M., Goldberg, K., Kanazawa, A., Tancik, M.: Lerf: Language em- bedded radiance fields. In: International Conference on Computer Vision (ICCV) (2023) 16 J. Ding et al

    2023

  17. [17]

    arXiv preprint arXiv:2406.09246 (2024)

    Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024)

    Pith/arXiv arXiv 2024

  18. [18]

    arXiv preprint arXiv:2605.19390 (2026)

    Li, C., Xu, Y., Feng, J., Ding, J.: Lmm-track4d: Eliciting 4d dynamic reasoning in lmms via trajectory-grounded dialogue. arXiv preprint arXiv:2605.19390 (2026)

    Pith/arXiv arXiv 2026

  19. [19]

    arXiv preprint arXiv:2603.13615 (2026)

    Li, D., Liu, L., Liu, B., Zhou, S., Feng, J., Lu, Z., Zheng, M., You, C., Fan, Z.: Egocentric world model for photorealistic hand-object interaction synthesis. arXiv preprint arXiv:2603.13615 (2026)

    arXiv 2026

  20. [20]

    In: Proceedings of the Computer Vision and Pattern Recognition Con- ference

    Li, H., Wu, Y., Meng, J., Gao, Q., Zhang, Z., Wang, R., Zhang, J.: Instance- gaussian: Appearance-semantic joint gaussian representation for 3D instance-level perception. In: Proceedings of the Computer Vision and Pattern Recognition Con- ference. pp. 14078–14088 (2025)

    2025

  21. [21]

    arXiv preprint arXiv:2412.10231 (2024)

    Liang, S., Wang, S., Li, K., Niemeyer, M., Gasperini, S., Navab, N., Tombari, F.: Supergseg: Open-vocabulary 3d segmentation with structured super-gaussians. arXiv preprint arXiv:2412.10231 (2024)

    arXiv 2024

  22. [22]

    In: Proceedings of the IEEE international conference on computer vision

    Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., Yuille, A.: Recurrent multimodal inter- action for referring image segmentation. In: Proceedings of the IEEE international conference on computer vision. pp. 1271–1280 (2017)

    2017

  23. [23]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, L., Li, D., Liang, Y., Jiang, S., Vijay, H., Hu, H., Xu, X., Liu, Z., Shakkot- tai, S., Li, M., et al.: Egotl: Egocentric think-aloud chains for long-horizon tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2017–2027 (2026)

    2017

  24. [24]

    Advances in Neural Information Pro- cessing Systems37, 11164–11185 (2024)

    Liu, X., Xu, X., Li, J., Zhang, Q., Wang, X., Sebe, N., Ma, L.: Less: Label-efficient and single-stage referring 3d segmentation. Advances in Neural Information Pro- cessing Systems37, 11164–11185 (2024)

    2024

  25. [25]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2025)

    Marrie, J., Menegaux, R., Arbel, M., Larlus, D., Mairal, J.: Ludvig: Learning-free uplifting of 2d visual features to gaussian splatting scenes. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2025)

    2025

  26. [26]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition

    Mirzaei, A., Aumentado-Armstrong, T., Derpanis, K.G., Kelly, J., Brubaker, M.A., Gilitschenski, I., Levinshtein, A.: Spin-nerf: Multiview segmentation and percep- tual inpainting with neural radiance fields. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 20669–20679 (2023)

    2023

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Munkberg, J., Hasselgren, J., Shen, T., Gao, J., Chen, W., Evans, A., Müller, T., Fidler, S.: Extracting triangular 3d models, materials, and lighting from images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8280–8290 (2022)

    2022

  28. [28]

    arXiv preprint arXiv:2410.07577 (2024)

    Peng, Q., Planche, B., Gao, Z., Zheng, M., Choudhuri, A., Chen, T., Chen, C., Wu, Z.: 3d vision-language gaussian splatting. arXiv preprint arXiv:2410.07577 (2024)

    Pith/arXiv arXiv 2024

  29. [29]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Qin, M., Li, W., Zhou, J., Wang, H., Pfister, H.: Langsplat: 3D language gaussian splatting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20051–20060 (2024)

    2024

  30. [30]

    In: Proceedings of the 32nd ACM International Conference on Multimedia

    Qu, Y., Dai, S., Li, X., Lin, J., Cao, L., Zhang, S., Ji, R.: GOI: Find 3D gaus- sians of interest with an optimizable open-vocabulary semantic-space hyperplane. In: Proceedings of the 32nd ACM International Conference on Multimedia. ACM (2024)

    2024

  31. [31]

    arXiv preprint arXiv:2401.14159 (2024)

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., et al.: Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024)

    Pith/arXiv arXiv 2024

  32. [32]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition

    Shi, J.C., Wang, M., Duan, H.B., Guan, S.H.: Language embedded 3D gaussians for open-vocabulary scene understanding. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 5333–5343 (2024) ZeroSplat 17

    2024

  33. [33]

    arXiv preprint arXiv:2504.11893 (2025)

    Sun, W., Zhou, Y., Jiao, J., Li, Y.: Cags: Open-vocabulary 3d scene understanding with context-aware gaussian splatting. arXiv preprint arXiv:2504.11893 (2025)

    Pith/arXiv arXiv 2025

  34. [34]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Tang, H., Cao, M., Liu, R., Liang, X., Li, L., Li, G., Liang, X.: Video spatial reasoning with object-centric 3d rollout. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 9395–9403 (2026)

    2026

  35. [35]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang, Z., Lu, Y., Li, Q., Tao, X., Guo, Y., Gong, M., Liu, T.: Cris: Clip-driven referring image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11686–11695 (2022)

    2022

  36. [36]

    Advances in Neural Information Processing Systems37, 110972–110999 (2024)

    Wu, C., Ji, J., Wang, H., Ma, Y., Huang, Y., Luo, G., Fei, H., Sun, X., Ji, R., et al.: Rg-san: Rule-guided spatial awareness network for end-to-end 3d referring expression segmentation. Advances in Neural Information Processing Systems37, 110972–110999 (2024)

    2024

  37. [37]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Wu,C.,Ma,Y.,Chen,Q.,Wang,H.,Luo,G.,Ji,J.,Sun,X.:3d-stmn:Dependency- driven superpoint-text matching network for end-to-end 3d referring expression segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 5940–5948 (2024)

    2024

  38. [38]

    Advances in Neural Information Processing Systems 37, 19114–19138 (2024)

    Wu, Y., Meng, J., Li, H., Wu, C., Shi, Y., Cheng, X., Zhao, C., Feng, H., Ding, E., Wang, J., et al.: Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding. Advances in Neural Information Processing Systems 37, 19114–19138 (2024)

    2024

  39. [39]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Xiao, X., Zhang, Y., Li, X., Wang, T., Wang, X., Wei, Y., Hamm, J., Xu, M.: Visual instance-aware prompt tuning. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 2880–2889 (2025)

    2025

  40. [40]

    Transactions on Machine Learning Research (2026)

    Xiao, X., Zhang, Y., Zhao, L., Liu, Y., Liao, X., Mai, Z., Li, X., Wang, X., Xu, H., Hamm,J.,Lin,X.,Xu,M.,Wang,Q.,Wang,T.,Han,C.:Prompt-basedadaptation in large-scale vision models: A survey. Transactions on Machine Learning Research (2026)

    2026

  41. [41]

    In: European conference on computer vision

    Ye, M., Danelljan, M., Yu, F., Ke, L.: Gaussian grouping: Segment and edit any- thing in 3D scenes. In: European conference on computer vision. pp. 162–179. Springer (2024)

    2024

  42. [42]

    arXiv preprint arXiv:2503.21767 (2025)

    Yin, H., Zhan, H., Xu, Y., Yeh, R.A.: Semantic consistent language gaussian splat- ting for point-level open-vocabulary querying. arXiv preprint arXiv:2503.21767 (2025)

    arXiv 2025

  43. [43]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhou, S., Chang, H., Jiang, S., Fan, Z., Zhu, Z., Xu, D., Chari, P., You, S., Wang, Z., Kadambi, A.: Feature 3Dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21676–21685 (2024) 18 J. Ding et al. Supplementary Material A Dataset Construct...

    2024

  44. [44]

    green apple

    Evaluation and task type:Existing datasets like LERF and ScanNet focus on open-vocabulary segmentation, while referring benchmarks like Ref-LERF are limited to 2D pixel-level evaluation. In contrast, our datasets target generalized 3D referring segmentation. By evaluating directly at the 3D point or Gaussian level, they better reflect a model’s spatial an...

  45. [45]

    original expression

    Output valid JSON: {"original expression": ["category1", "category2", ...]}

  46. [46]

    You must analyze the image(s) to ensure the identified object is the one intended by the text

    SPATIAL & CONTEXTUAL TARGETING: Use spatial relationships (e.g., ’in front of’, ’next to’, ’on the top’) to locate the specific target object(s) in the image. You must analyze the image(s) to ensure the identified object is the one intended by the text

  47. [47]

    ABSTRACTION (Color + Noun): Convert the identified target object into a simplified ’Color + Noun’ phrase based on its actual appearance in the images. Examples: - Input: ’A round object with a smooth surface directly in front of the white doll’ -> If the object in the image is a blue cup, Output: [’blue cup’] - Input: ’fire alarm and smoke detector’ -> Ou...

  48. [48]

    QUANTITY: If one expression refers to multiple objects, return them all in the array

  49. [49]

    Do NOT output markdown formatting (“‘json), just the raw JSON string

  50. [50]

    Color + Noun

    COMPLETENESS: Return a mapping for EVERY expression in the input list. User Prompt: Here is the list of descriptions to convert: {input_str} Please map each description to its specific object category noun: Fig. 11:Stage 1 Prompt: Semantic label extraction. The prompt enforces spatial rea- soning and standardizes natural language descriptions into a "Colo...

  51. [51]

    VISUAL & SPATIAL CUES: Strictly rely on spatial relations, color descriptors, and object attributes. If the text describes ’A round object with a smooth surface directly in front of the white doll’, you must first identify the white doll as an anchor and then locate the specific object(s) in front of it based on spatial context

  52. [52]

    Use global spatial reasoning to determine if an object in the current frame is indeed part of the target group described by the text

    CROSS-IMAGE SPATIAL REASONING: Analyze ALL provided images to maintain consistency. Use global spatial reasoning to determine if an object in the current frame is indeed part of the target group described by the text

  53. [53]

    IMPORTANT RULES:

    ABSENCE OF TARGET: Only return -1 if the target object(s) are absolutely not present. IMPORTANT RULES:

  54. [54]

    box": [xmin, ymin, xmax, ymax],

    Coordinates must be normalized [0, 1000], order: [xmin, ymin, xmax, ymax]. Use the key "box" for the coordinates. User Prompt: Detect ALL instances of ’{target_text}’ (Categories: {short_names}) in this image. This text may refer to MULTIPLE object types. Detect ALL of them. Return format (use ONLY these exact keys, output raw JSON only): - MULTIPLE objec...

  55. [55]

    Each object MUST use key "box" (NOT bbox, NOT bbox_2d) for coordinates

  56. [56]

    Coordinates normalized [0, 1000], order: [xmin, ymin, xmax, ymax]

  57. [57]

    Use structural boundaries to distinguish objects

  58. [58]

    Return -1 ONLY if absolutely not present

  59. [59]

    12:Stage 2 Prompt: 2D geometric localization

    ’label’ field must describe each detected instance Fig. 12:Stage 2 Prompt: 2D geometric localization. This prompt guides the VLM to perform cross-image spatial reasoning and strictly handle the zero-target scenario by outputting -1