The reviewed record of science sign in
Pith

arxiv: 2607.05348 · v1 · pith:LPB7FBDY · submitted 2026-07-06 · cs.CV · cs.RO

Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-07 15:53 UTCglm-5.2pith:LPB7FBDYrecord.jsonopen to challenge →

classification cs.CV cs.RO
keywords scenesemanticunderstandingbeyondcontextualfeaturesgraphsobject
0
0 comments X

The pith

Object relationships, not just appearance, drive 3D scene understanding

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that open-vocabulary 3D semantic segmentation can be meaningfully improved by moving beyond isolated object recognition and explicitly modeling the relationships between objects in a scene. The authors propose RelGraphOV, a framework that constructs a 3D scene graph where nodes are objects and edges carry VLM-inferred relationship semantics (e.g., 'person sits on chair'), then uses a dual-stream graph attention network to propagate contextual information along those edges. The core claim is that knowing an object's spatial and functional relationships to its neighbors — such as a curtain being near a bathtub — provides a disambiguating prior that appearance alone cannot. The paper reports state-of-the-art results on ScanNetV2 (58.4 mIoU), ScanNet200 (14.5 mIoU), and zero-shot transfer to ScanNet++ (20.9 mIoU) and Replica (22.7 mIoU), with a flagship example of 'shower curtain' improving from 0.0 to 64.2 IoU when relationship context is introduced.

Core claim

The central mechanism is a pipeline that first builds a relationship-aware 3D scene graph without manual annotation: a vision-language model visually inspects pairs of nearby objects and, via chain-of-thought reasoning, either describes their relationship in subject-verb-object triplet form or prunes the edge as irrelevant. These relationship triplets are encoded as edge features. A dual-stream graph attention network then propagates features along these edges, keeping dense geometric (LSeg) features and semantic (CLIP) features in separate streams to prevent cross-contamination, with a learned gate selectively injecting semantic cues from the auxiliary CLIP stream into the main geometric-LS

What carries the argument

RelGraphOV combines: (1) automatic scene graph construction via VLM chain-of-thought relationship reasoning on visually prompted object pairs, producing SVO-triplet edge features; (2) an Adaptive Gated Dual-Stream Contextual GAT that decouples dense LSeg features (main stream) from CLIP instance features (auxiliary stream), performs edge-guided message passing independently in each stream, and fuses them via a learned gate; (3) a hierarchical contrastive loss enforcing instance-level consistency and category-level discrimination; and (4) dual-alignment losses that anchor each stream to its original feature prior while learning from VLM-generated annotations.

If this is right

  • If relationship context genuinely disambiguates categories like 'shower curtain' vs. 'curtain,' then any 3D perception system operating in structured environments (homes, offices, hospitals) could benefit from graph-based contextual refinement, particularly for rare or visually confusable object categories.
  • The modular design — where proposal generation, VLM backbone, and feature extractor are interchangeable upstream components — means improvements in any of these sub-systems should directly propagate through the relational refinement framework without architectural changes.
  • The zero-shot cross-dataset gains on ScanNet++ and Replica suggest that relationship structure generalizes across visual domains more robustly than appearance-only features, since functional and spatial relationships between objects (e.g., toilet-next-to-bathtub) are domain-invariant.
  • The dual-stream separation principle — keeping heterogeneous feature types in independent propagation paths and fusing via learned gates — could apply to other multi-modal graph learning problems where naive feature mixing causes interference.

Load-bearing premise

The entire pipeline depends on the VLM producing correct relationship descriptions for object pairs. If the VLM hallucinates relationships, returns 'none' for valid edges, or produces inconsistent descriptions, the graph topology and edge features fed into the network are corrupted. The paper does not report VLM relationship accuracy against any ground-truth scene graph, so it is unclear how robust the gains are when relationship quality degrades on novel scenes or domains.

What would settle it

Construct scenes where the VLM systematically misidentifies object relationships (e.g., cluttered scenes, unusual object configurations, poor lighting) and measure whether the segmentation gains collapse relative to the relationship-free baseline.

Figures

Figures reproduced from arXiv: 2607.05348 by Guofeng Zhang, Hujun Bao, Jiarui Hu, Tengyue Wang, Xianhao Chen, Xiyu Zhang, Yuanbo Yang, Zhaopeng Cui.

Figure 1
Figure 1. Figure 1: RelGraphOV. We propose a relationship-aware open-vocabulary 3D scene understanding framework. Left: Illustration of encoding a 3D scene into a relationship￾aware scene graph to capture contextual relationships. Right: Qualitative open￾vocabulary segmentation results produced by our method. paradigms. The first projects dense vision-language features from 2D models into 3D space [25, 33], which provides str… view at source ↗
Figure 2
Figure 2. Figure 2: Overall Pipeline of RelGraphOV. (1) Graph Construction: We build a relationship-aware 3D scene graph via multi-view VLM reasoning. (2) Annotation: A VLM chain-of-thought generates rich node semantics. (3) Adaptive Gated Dual-Stream GAT: To prevent feature interference, we decouple geometric (main) and semantic (auxiliary) feature propagation. Both undergo edge-guided message passing, while a learnable gate… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative Comparisons. Images of 3D semantic segmentation results on ScanNetV2 [7], including the example of a curtain and shower curtain as mentioned in the introduction. for two distinct reasons. Distillation-based approaches (e.g., OpenScene [33], DMA [26]) are constrained by the semantic coverage of their 2D teacher mod￾els. Conversely, methods directly using CLIP to extract instance features (e.g., … view at source ↗
read the original abstract

Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. The paper proposes RelGraphOV, a relationship-aware open-vocabulary 3D scene understanding framework. The method constructs 3D scene graphs from RGB-D sequences by using a VLM (qwen-vl-max) to infer object relationships (as SVO triplets) without manual relationship annotations, then refines node features via an Adaptive Gated Dual-Stream Contextual GAT that separates dense LSeg features (main stream) from CLIP instance features (auxiliary stream) with edge-feature-guided message passing. A hierarchical contrastive loss with dual-alignment regularization is used for training. The method is evaluated on ScanNetV2 (58.4 mIoU), ScanNet200 (14.5 mIoU), and zero-shot on ScanNet++ (20.9 mIoU) and Replica (22.7 mIoU), showing improvements over prior open-vocabulary 3D segmentation methods.

Significance. The paper addresses a genuine gap in open-vocabulary 3D scene understanding: most existing methods treat objects in isolation, and incorporating relational context via automatically constructed scene graphs is a reasonable and novel direction. The dual-stream architecture is well-motivated by the feature granularity dilemma, and the ablation studies (Tabs. 6–7) systematically isolate individual components. The cross-dataset zero-shot evaluation on ScanNet++ and Replica provides evidence of generalization beyond the training domain. The 'shower curtain' example (Tab. 2: 0.0 → 64.2 IoU) is a compelling qualitative case for the value of contextual reasoning. The overall experimental evaluation is thorough across four datasets with per-category and head/common/tail breakdowns.

major comments (3)
  1. §4.4, Table 6, 'w/o relation guidance' row: This ablation removes ALL edge features while preserving graph topology, yielding 55.5 vs. 58.4 mIoU on ScanNetV2 and 11.0 vs. 14.5 on ScanNet200. However, the paper's central novelty claim is that VLM-inferred semantic relationships (SVO triplets) provide the contextual priors. This ablation does not distinguish between (a) the VLM's specific semantic relationship content being valuable and (b) any edge features at all (e.g., spatial labels like 'near'/'adjacent', random vectors, or learned embeddings) providing a useful inductive bias for attention modulation. Since edge features enter the GAT as Key-side concatenations (§3.3, 'Edge-Feature-Guided Message Propagation'), their mere presence gives the attention mechanism additional degrees of freedom regardless of semantic content. An additional ablation replacing VLM-derived edge features with
  2. §3.1, 'VLM-based Edge Refinement and Encoding': The entire pipeline depends on the VLM producing correct relationship descriptions, but no quantitative assessment of VLM relationship quality is reported — no precision, recall, or accuracy on any ground-truth scene graph dataset. This is load-bearing because the paper claims the constructed graph provides 'reliable relational priors' (§3.1). Even a small-scale manual evaluation of the VLM's relationship predictions on a subset of ScanNetV2 scenes (e.g., 50 edges compared to human annotations) would substantially strengthen the claim. Without this, it is unclear whether the gains come from accurate relationship semantics or from the graph structure and edge-feature mechanism alone (which connects to the previous comment).
  3. Table 2, 'shower curtain' column: The flagship claim that relational context resolves the curtain/shower curtain ambiguity (0.0 → 64.2 IoU) is presented without a per-category ablation. The improvement could arise from the dual-stream CLIP features (which alone may better distinguish shower curtains via visual semantics) rather than from relation guidance specifically. A per-category ablation showing the shower curtain IoU under 'w/o relation guidance' vs. full model would directly test whether the relationship reasoning is the operative factor for this specific case.
minor comments (7)
  1. §3.1, Eq. (1): The components of S (s_comp, s_area, s_center, s_sharp) are described only briefly. A short definition or formula for each would improve reproducibility.
  2. §3.4, Eq. (6): The margin M is added only to self-pairs (via δ_ij), but the motivation for why a margin is needed specifically for self-pairs is not explained clearly. A brief justification would help.
  3. Table 1: Several entries are marked with '†' with different meanings ('reported in the original paper' vs. 'reproduced by us'). A footnote clarifying which entries are reproduced and under what protocol would improve clarity.
  4. §4.1: The keyframe scoring weights W = [w_comp, w_area, w_center, w_sharp] are listed as free parameters in the axiom ledger but their values are not specified in the main text. These should be reported or referenced to the supplementary.
  5. Figure 2: The pipeline diagram is dense and some text labels are difficult to read. Consider simplifying or enlarging key components for clarity.
  6. §3.3: The number of GAT layers L is not specified in the main text. This should be reported for reproducibility.
  7. The paper mentions 'further implementation specifics, including the exact VLM prompts, are provided in the supplementary material' (§4.1), but the supplementary was not available for review. Ensure the prompts are included in the final submission.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a thorough and constructive report. The referee correctly identifies our central novelty claims and raises three interconnected concerns about whether the gains can be attributed to VLM-inferred semantic relationship content specifically, as opposed to graph structure, edge-feature degrees of freedom, or dual-stream CLIP features. We address each comment below and commit to concrete revisions.

read point-by-point responses
  1. Referee: §4.4, Table 6, 'w/o relation guidance' ablation does not distinguish between (a) VLM semantic relationship content being valuable and (b) any edge features at all providing a useful inductive bias. An additional ablation replacing VLM-derived edge features with alternatives (spatial labels, random vectors, learned embeddings) is needed.

    Authors: The referee raises a valid and important concern. Our current 'w/o relation guidance' ablation removes all edge features entirely, which conflates the effect of having edge features with the effect of having semantically meaningful VLM-derived edge features. We agree this distinction is critical because our central novelty claim is that VLM-inferred SVO triplets provide valuable contextual priors, not merely that edge features provide additional attention modulation capacity. We will add a new ablation row in Table 6 replacing VLM-derived edge features with (i) spatial-only labels ('near'/'adjacent' encoded via CLIP text encoder) and (ii) random fixed edge vectors, while preserving the same graph topology and edge-feature-guided GAT mechanism. This will directly test whether the semantic content of VLM-inferred relationships contributes beyond the structural inductive bias of having any edge features. We expect the VLM-derived features to outperform both alternatives, particularly on categories where contextual reasoning is most beneficial (e.g., shower curtain, tail categories on ScanNet200), but we will report whatever results we obtain honestly. revision: yes

  2. Referee: §3.1, 'VLM-based Edge Refinement and Encoding': No quantitative assessment of VLM relationship quality is reported. Even a small-scale manual evaluation on a subset of ScanNetV2 scenes would substantially strengthen the claim that the constructed graph provides 'reliable relational priors.'

    Authors: We agree that this is a genuine gap in our evaluation. The claim that the constructed graph provides 'reliable relational priors' is load-bearing for our contribution, and we currently provide no quantitative evidence for it. We will conduct a small-scale human evaluation of VLM-inferred relationships on a subset of ScanNetV2 scenes. Specifically, we plan to sample approximately 100 edges across 20 scenes, have a human annotator judge whether each VLM-inferred SVO triplet is semantically correct (precision), and also estimate recall by checking whether obvious relationships are captured by the VLM. We will report precision, recall, and qualitative error patterns. This will not be a large-scale annotation effort, but it will provide the first quantitative evidence of VLM relationship quality and will allow readers to assess whether the gains plausibly come from accurate relationship semantics. We acknowledge that if the VLM relationship quality turns out to be modest, this would suggest that the graph structure and edge-feature mechanism contribute substantially on their own, which would be an honest and important finding. revision: yes

  3. Referee: Table 2, 'shower curtain' column: The 0.0 → 64.2 IoU improvement is presented without a per-category ablation. The improvement could arise from the dual-stream CLIP features rather than from relation guidance specifically. A per-category ablation showing shower curtain IoU under 'w/o relation guidance' vs. full model would directly test whether relationship reasoning is the operative factor.

    Authors: This is a fair and well-targeted criticism. The shower curtain example is our flagship qualitative case for the value of relational context, yet we have not actually isolated whether the improvement comes from relation guidance or from the dual-stream CLIP features. The referee is correct that CLIP features alone may better distinguish shower curtains via visual semantics, which would undermine the specific causal claim we make. We will add a per-category ablation for shower curtain (and ideally a few other context-dependent categories) showing IoU under: (i) w/o scene graph (baseline LSeg only), (ii) w/o relation guidance (graph + dual-stream CLIP, but no edge features), and (iii) full model. This will directly decompose the contribution of dual-stream CLIP features versus VLM relation guidance for this specific category. If the improvement is primarily from CLIP features, we will revise our claims accordingly and identify a different category where relation guidance is demonstrably the operative factor. If relation guidance does contribute specifically, the ablation will show it. Either way, the manuscript will be strengthened. revision: yes

Circularity Check

0 steps flagged

No circularity: the derivation chain is self-contained against external benchmarks

full rationale

The paper's central claim is that VLM-inferred object relationships, encoded as SVO-triplet edge features in a 3D scene graph, improve open-vocabulary 3D semantic segmentation. Walking the derivation chain: (1) Scene graph construction (Sec. 3.1) uses external pretrained models (qwen-vl-max, LSeg, CLIP) to generate edges and node features — these are not the authors' prior results. (2) The VLM annotation engine (Sec. 3.2) generates supervision targets via Describe Anything Model and Qwen-VL-Max — again external models, not self-cited prior work. (3) The Adaptive Gated Dual-Stream GAT (Sec. 3.3) is a new architectural contribution whose equations (Eqs. 3-4) define a gated fusion mechanism; no equation reduces to its input by definition. (4) The loss design (Sec. 3.4, Eqs. 5-8) defines a hierarchical contrastive objective and dual-alignment losses that are standard formulations, not tautological. (5) Evaluation uses standard external benchmarks (ScanNetV2, ScanNet200, ScanNet++, Replica) with standard metrics (mIoU, mAcc). The ablation studies (Tabs. 6-7) properly remove components to measure their contribution. The 'w/o relation guidance' ablation (55.5 vs 58.4 mIoU) removes edge features entirely rather than isolating VLM-derived semantic content from generic edge features — this is a legitimate concern about experimental design and causal attribution, but it is not circularity. The VLM annotations serve as supervision targets and the model is trained against them, but this is standard practice (using generated labels as training signal), not a self-definitional loop. No step in the derivation chain reduces to its inputs by construction, no prediction is a renamed fit, and no load-bearing argument depends on a self-citation chain. The skeptic's concern about whether gains come from VLM relationship semantics specifically versus any edge features is a correctness/attribution risk, not a circularity issue.

Axiom & Free-Parameter Ledger

8 free parameters · 4 axioms · 1 invented entities

The free parameters are numerous (at least 8) and several have unspecified values in the main text, which is a reproducibility concern. The axioms are domain assumptions about VLM capability and feature complementarity that are partially supported by ablation but not independently validated. No truly novel physical or mathematical entities are invented—the super node is a standard GNN construct.

free parameters (8)
  • λ_hie, λ_reg, λ_aux = [0.5, 1.5, 1.5]
    Loss weights for hierarchical contrastive, regularization, and auxiliary losses, set by 'preliminary hyperparameter tuning' (Sec. 4.1). No sensitivity analysis provided.
  • γ_reg, γ_aux = [0.7, 0.7]
    Dual-alignment balance factors controlling trade-off between VLM annotation alignment and original feature preservation (Sec. 4.1). No sensitivity analysis.
  • τ (similarity threshold) = Not specified
    Threshold for constructing positive mask M_pos in hierarchical contrastive loss (Sec. 3.4). Controls which instances are treated as semantically similar. Value not given in main text.
  • M (margin) = Not specified
    Margin added to self-pair similarity in Eq. 6. Affects self-consistency emphasis. Value not given.
  • η (temperature) = Not specified
    Temperature parameter in hierarchical contrastive loss (Eq. 6). Affects softmax sharpness. Value not given.
  • α (scaling factor) = Not specified
    Scaling factor for gated fusion in Eq. 4. Controls magnitude of auxiliary stream injection. Value not given.
  • W (keyframe scoring weights) = Not specified
    Weight vector [w_comp, w_area, w_center, w_sharp] for keyframe selection (Eq. 1). Described as 'customizable' but default values not provided.
  • L (number of GAT layers) = Not specified
    Number of layers in the dual-stream GAT. Affects receptive field and over-smoothing. Not stated in main text.
axioms (4)
  • domain assumption VLM (Qwen-VL-Max) can reliably infer spatial and semantic relationships between objects from 2D images with visual prompting.
    The entire scene graph construction pipeline (Sec. 3.1) depends on VLM relationship reasoning quality. No accuracy metrics are reported for VLM-generated relationships against any ground truth.
  • domain assumption Dense LSeg features and instance-level CLIP features are complementary and their combination improves open-vocabulary segmentation.
    The dual-stream architecture (Sec. 3.3) is motivated by this complementarity. The ablation (w/o dual graph fusion, 57.9 vs 58.4) provides partial support but the margin is small.
  • domain assumption Spatial proximity (3 nearest neighbors or bounding box adjacency) is a sufficient initial graph topology for relationship reasoning.
    Edge initialization (Sec. 3.1) uses this heuristic. No analysis of how the k=3 choice or adjacency criterion affects final performance.
  • domain assumption SVO triplet parsing of VLM-generated relationship descriptions produces meaningful edge features.
    Edge feature construction (Sec. 3.1) parses Subject-Verb-Object triplets and encodes them independently via CLIP text encoder. No evaluation of parsing accuracy or feature quality.
invented entities (1)
  • Super node (v_super) no independent evidence
    purpose: Virtual node connected to all real nodes for global context aggregation in the augmented graph G_aug (Sec. 3.3).
    The super node is an architectural construct. Its contribution is validated by ablation (w/o global feature: 56.0 vs 58.4 mIoU on ScanNetV2), but it is not a new physical entity—it is a standard GNN technique.

pith-pipeline@v1.1.0-glm · 20592 in / 3966 out tokens · 270375 ms · 2026-07-07T15:53:51.922853+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 58 canonical work pages · 3 internal anchors

  1. [1]

    IEEE Trans

    Campos, C., Elvira, R., Rodríguez, J.J.G., Montiel, J.M.M., Tardós, J.D.: ORB- SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM. IEEE Trans. Robotics37(6), 1874–1890 (2021) 16 Chen et al

  2. [2]

    In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pp. 9630–9640. IEEE (2021)

  3. [3]

    In: Tan, J., Toussaint, M., Darvish, K

    Chang, H., Boyalakuntla, K., Lu, S., Cai, S., Jing, E.P., Keskar, S., Geng, S., Abbas, A., Zhou, L., Bekris, K.E., Boularias, A.: Context-aware entity grounding with open-vocabulary 3d scene graphs. In: Tan, J., Toussaint, M., Darvish, K. (eds.) Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA. Proceedings of Machine Learning R...

  4. [4]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 1280–1289. IEEE (2022)

  5. [5]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024

    Cho, S., Shin, H., Hong, S., Arnab, A., Seo, P.H., Kim, S.: Cat-seg: Cost aggre- gation for open-vocabulary semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 4113–4123. IEEE (2024)

  6. [6]

    In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019

    Choy, C.B., Gwak, J., Savarese, S.: 4d spatio-temporal convnets: Minkowski con- volutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 3075–3084. Computer Vision Foundation / IEEE (2019)

  7. [7]

    In: 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017

    Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T.A., Nießner, M.: Scan- net: Richly-annotated 3d reconstructions of indoor scenes. In: 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. pp. 2432–2443. IEEE Computer Society (2017)

  8. [8]

    In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018

    Dai, A., Ritchie, D., Bokeloh, M., Reed, S., Sturm, J., Nießner, M.: Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 4578–4587. Computer Vision Founda- tion / IEEE Computer Society (2018)

  9. [9]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24,

    Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., Qi, X.: PLA: language-driven open-vocabulary 3d scene understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24,

  10. [10]

    7010–7019

    pp. 7010–7019. IEEE (2023)

  11. [11]

    In: International Conference on Learning Representa- tions (2024)

    Engelmann, F., Manhardt, F., Niemeyer, M., Tateno, K., Pollefeys, M., Tombari, F.: OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views. In: International Conference on Learning Representa- tions (2024)

  12. [12]

    In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T

    Ghiasi, G., Gu, X., Cui, Y., Lin, T.: Scaling open-vocabulary image segmentation with image-level labels. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI. Lecture Notes in Computer Science, vol. 13696, pp. 54...

  13. [13]

    In: IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024

    Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K.M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., Gan, C., de Melo, C.M., Tenenbaum, J.B.,Torralba,A.,Shkurti,F.,Paull,L.:Conceptgraphs:Open-vocabulary3dscene graphs for perception and planning. In: IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, J...

  14. [14]

    In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G

    Hu, J., Chen, X., Feng, B., Li, G., Yang, L., Bao, H., Zhang, G., Cui, Z.: CG- SLAM: efficient dense RGB-D SLAM in a consistent uncertainty-aware 3d gaussian field. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September29-October4,2024,Proceedin...

  15. [15]

    In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S

    Hu, J., Mao, M., Bao, H., Zhang, G., Cui, Z.: CP-SLAM: collaborative neural point-based SLAM system. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 20...

  16. [16]

    In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019

    Huang, J., Zhang, H., Yi, L., Funkhouser, T.A., Nießner, M., Guibas, L.J.: Tex- turenet: Consistent local parametrizations for learning from high-resolution signals on meshes. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 4440–4449. Computer Vision Foundation / IEEE (2019)

  17. [17]

    In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023

    Jain, J., Li, J., Chiu, M., Hassani, A., Orlov, N., Shi, H.: Oneformer: One trans- former to rule universal image segmentation. In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 2989–2998. IEEE (2023)

  18. [18]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024

    Jiang, L., Shi, S., Schiele, B.: Open-vocabulary 3d semantic segmentation with foundation models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 21284–21294. IEEE (2024)

  19. [19]

    In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W., Dollár, P., Girshick, R.B.: Segment anything. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. pp. 3992–4003. IEEE (2023)

  20. [20]

    In: Inter- national Conference on 3D Vision, 3DV 2024, Davos, Switzerland, March 18-21,

    Koch, S., Hermosilla, P., Vaskevicius, N., Colosi, M., Ropinski, T.: Lang3dsg: Language-based contrastive pre-training for 3d scene graph prediction. In: Inter- national Conference on 3D Vision, 3DV 2024, Davos, Switzerland, March 18-21,

  21. [21]

    1037–1047

    pp. 1037–1047. IEEE (2024)

  22. [22]

    Koch, S., Vaskevicius, N., Colosi, M., Hermosilla, P., Ropinski, T.: Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-setrelationships.In:IEEE/CVFConferenceonComputerVisionandPattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 14183–14193. IEEE (2024)

  23. [23]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025

    Koch, S., Wald, J., Colosi, M., Vaskevicius, N., Hermosilla, P., Tombari, F., Ropin- ski, T.: Relationfield: Relate anything in radiance fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 21706–21716. Computer Vision Foundation / IEEE (2025)

  24. [24]

    In: 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19,

    Lambert, J., Liu, Z., Sener, O., Hays, J., Koltun, V.: Mseg: A composite dataset for multi-domain semantic segmentation. In: 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19,

  25. [25]

    2876–2885

    pp. 2876–2885. Computer Vision Foundation / IEEE (2020)

  26. [26]

    In: IEEE/CVFConferenceonComputerVisionandPatternRecognition,CVPR2025, Nashville, TN, USA, June 11-15, 2025

    Lee, J., Park, C., Choe, J., Wang, Y.F., Kautz, J., Cho, M., Choy, C.B.: Mo- saic3d: Foundation dataset and model for open-vocabulary 3d segmentation. In: IEEE/CVFConferenceonComputerVisionandPatternRecognition,CVPR2025, Nashville, TN, USA, June 11-15, 2025. pp. 14089–14101. Computer Vision Foun- dation / IEEE (2025) 18 Chen et al

  27. [27]

    In: The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022

    Li, B., Weinberger, K.Q., Belongie, S.J., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net (2022)

  28. [28]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025

    Li, J., Saltori, C., Poiesi, F., Sebe, N.: Cross-modal and uncertainty-aware agglom- eration for open-vocabulary 3d scene understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 19390–19400. Computer Vision Foundation / IEEE (2025)

  29. [29]

    In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G

    Li, R., Zhang, Z., He, C., Ma, Z., Patel, V.M., Zhang, L.: Dense multimodal align- ment for open-vocabulary 3d scene understanding. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Pro- ceedings, Part XLIX. Lecture Note...

  30. [30]

    Describe Anything: Detailed Localized Image and Video Captioning

    Lian, L., Ding, Y., Ge, Y., Liu, S., Mao, H., Li, B., Pavone, M., Liu, M., Darrell, T., Yala,A.,Cui,Y.:Describeanything:Detailedlocalizedimageandvideocaptioning. CoRRabs/2504.16072(2025)

  31. [31]

    In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., Zhu, J., Zhang, L.: Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy...

  32. [32]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025

    Lu, J., Deng, J.: Relation3d : Enhancing relation modeling for point cloud in- stance segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 8889–8899. Computer Vision Foundation / IEEE (2025)

  33. [33]

    In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G

    Naeem, M.F., Xian, Y., Zhai, X., Hoyer, L., Gool, L.V., Tombari, F.: SILC: improv- ing vision language pretraining with self-distillation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXI. Lecture ...

  34. [34]

    In: International Conference on 3D Vi- sion, 3DV 2021, London, United Kingdom, December 1-3, 2021

    Nekrasov, A., Schult, J., Litany, O., Leibe, B., Engelmann, F.: Mix3d: Out-of- context data augmentation for 3d scenes. In: International Conference on 3D Vi- sion, 3DV 2021, London, United Kingdom, December 1-3, 2021. pp. 116–125. IEEE (2021)

  35. [35]

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual features...

  36. [36]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023

    Peng, S., Genova, K., Jiang, C.M., Tagliasacchi, A., Pollefeys, M., Funkhouser, T.A.: Openscene: 3d scene understanding with open vocabularies. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 815–824. IEEE (2023)

  37. [37]

    Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, RelGraphOV 19 P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y.,...

  38. [38]

    In: Meila, M., Zhang, T

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual E...

  39. [39]

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: Sam 2: Segment anything in images and videos (2024)

  40. [40]

    In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T

    Rozenberszki, D., Litany, O., Dai, A.: Language-grounded indoor 3d semantic seg- mentation in the wild. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII. Lecture Notes in Computer Science, vol. 13693, pp. 125...

  41. [41]

    In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020

    Schult, J., Engelmann, F., Kontogianni, T., Leibe, B.: Dualconvmesh-net: Joint geodesic and euclidean convolutions on 3d meshes. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 8609–8619. Computer Vision Foundation / IEEE (2020)

  42. [42]

    The Replica Dataset: A Digital Replica of Indoor Spaces

    Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J.J., Mur- Artal, R., Ren, C., Verma, S., Clarkson, A., Yan, M., Budge, B., Yan, Y., Pan, X., Yon, J., Zou, Y., Leon, K., Carter, N., Briales, J., Gillingham, T., Mueggler, E., Pesqueira, L., Savva, M., Batra, D., Strasdat, H.M., Nardi, R.D., Goesele, M., Lovegrove, S., Newcombe, R.:...

  43. [43]

    In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S

    Takmaz, A., Fedele, E., Sumner, R.W., Pollefeys, M., Tombari, F., Engelmann, F.: Openmask3d: Open-vocabulary 3d instance segmentation. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orle...

  44. [44]

    In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018

    Tatarchenko, M., Park, J., Koltun, V., Zhou, Q.: Tangent convolutions for dense prediction in 3d. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 3887–

  45. [45]

    Computer Vision Foundation / IEEE Computer Society (2018)

  46. [46]

    In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020

    Wald, J., Dhamo, H., Navab, N., Tombari, F.: Learning 3d semantic scene graphs from 3d indoor reconstructions. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 3960–3969. Computer Vision Foundation / IEEE (2020)

  47. [47]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) (2025)

    Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: VGGT: Visual geometry grounded transformer. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) (2025)

  48. [48]

    In: IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025

    Wang, Y., Jia, B., Zhu, Z., Huang, S.: Masked point-entity contrast for open- vocabulary 3d scene understanding. In: IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 14125–14136. Computer Vision Foundation / IEEE (2025) 20 Chen et al

  49. [49]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023

    Wang, Z., Cheng, B., Zhao, L., Xu, D., Tang, Y., Sheng, L.: VL-SAT: visual- linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud. In: IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 21560–21569. IEEE (2023)

  50. [50]

    In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Syst...

  51. [51]

    In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021

    Wu, S., Wald, J., Tateno, K., Navab, N., Tombari, F.: Scenegraphfusion: Incre- mental 3d scene graph prediction from RGB-D sequences. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 7515–7525. Computer Vision Foundation / IEEE (2021)

  52. [52]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024

    Yan,M.,Zhang, J.,Zhu, Y.,Wang,H.: Maskclustering: View consensusbased mask graph clustering for open-vocabulary 3d instance segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 28274–28284. IEEE (2024)

  53. [53]

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

    Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. CoRRabs/2310.11441 (2023)

  54. [54]

    In: IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024

    Yang, J., Ding, R., Deng, W., Wang, Z., Qi, X.: Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. In: IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 19823–19832. IEEE (2024)

  55. [55]

    In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023

    Yeshwanth, C., Liu, Y., Nießner, M., Dai, A.: Scannet++: A high-fidelity dataset of 3d indoor scenes. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. pp. 12–22. IEEE (2023)

  56. [56]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024

    Yuan, Y., Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L., Zhu, J.: Osprey: Pixel understanding with visual instruction tuning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 28202–28211. IEEE (2024)

  57. [57]

    Advances in Neural Information Processing Systems38, 15556–15580 (2026)

    Zhang, X., Bao, C., Chen, Y., Zhai, H., Dong, Y., Bao, H., Cui, Z., Zhang, G.: Atlasgs: Atlanta-world guided surface reconstruction with implicit structured gaus- sians. Advances in Neural Information Processing Systems38, 15556–15580 (2026)

  58. [58]

    In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S

    Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Wang, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,...