Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-07 15:53 UTCglm-5.2pith:LPB7FBDYrecord.jsonopen to challenge →
The pith
Object relationships, not just appearance, drive 3D scene understanding
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central mechanism is a pipeline that first builds a relationship-aware 3D scene graph without manual annotation: a vision-language model visually inspects pairs of nearby objects and, via chain-of-thought reasoning, either describes their relationship in subject-verb-object triplet form or prunes the edge as irrelevant. These relationship triplets are encoded as edge features. A dual-stream graph attention network then propagates features along these edges, keeping dense geometric (LSeg) features and semantic (CLIP) features in separate streams to prevent cross-contamination, with a learned gate selectively injecting semantic cues from the auxiliary CLIP stream into the main geometric-LS
What carries the argument
RelGraphOV combines: (1) automatic scene graph construction via VLM chain-of-thought relationship reasoning on visually prompted object pairs, producing SVO-triplet edge features; (2) an Adaptive Gated Dual-Stream Contextual GAT that decouples dense LSeg features (main stream) from CLIP instance features (auxiliary stream), performs edge-guided message passing independently in each stream, and fuses them via a learned gate; (3) a hierarchical contrastive loss enforcing instance-level consistency and category-level discrimination; and (4) dual-alignment losses that anchor each stream to its original feature prior while learning from VLM-generated annotations.
If this is right
- If relationship context genuinely disambiguates categories like 'shower curtain' vs. 'curtain,' then any 3D perception system operating in structured environments (homes, offices, hospitals) could benefit from graph-based contextual refinement, particularly for rare or visually confusable object categories.
- The modular design — where proposal generation, VLM backbone, and feature extractor are interchangeable upstream components — means improvements in any of these sub-systems should directly propagate through the relational refinement framework without architectural changes.
- The zero-shot cross-dataset gains on ScanNet++ and Replica suggest that relationship structure generalizes across visual domains more robustly than appearance-only features, since functional and spatial relationships between objects (e.g., toilet-next-to-bathtub) are domain-invariant.
- The dual-stream separation principle — keeping heterogeneous feature types in independent propagation paths and fusing via learned gates — could apply to other multi-modal graph learning problems where naive feature mixing causes interference.
Load-bearing premise
The entire pipeline depends on the VLM producing correct relationship descriptions for object pairs. If the VLM hallucinates relationships, returns 'none' for valid edges, or produces inconsistent descriptions, the graph topology and edge features fed into the network are corrupted. The paper does not report VLM relationship accuracy against any ground-truth scene graph, so it is unclear how robust the gains are when relationship quality degrades on novel scenes or domains.
What would settle it
Construct scenes where the VLM systematically misidentifies object relationships (e.g., cluttered scenes, unusual object configurations, poor lighting) and measure whether the segmentation gains collapse relative to the relationship-free baseline.
Figures
read the original abstract
Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RelGraphOV, a relationship-aware open-vocabulary 3D scene understanding framework. The method constructs 3D scene graphs from RGB-D sequences by using a VLM (qwen-vl-max) to infer object relationships (as SVO triplets) without manual relationship annotations, then refines node features via an Adaptive Gated Dual-Stream Contextual GAT that separates dense LSeg features (main stream) from CLIP instance features (auxiliary stream) with edge-feature-guided message passing. A hierarchical contrastive loss with dual-alignment regularization is used for training. The method is evaluated on ScanNetV2 (58.4 mIoU), ScanNet200 (14.5 mIoU), and zero-shot on ScanNet++ (20.9 mIoU) and Replica (22.7 mIoU), showing improvements over prior open-vocabulary 3D segmentation methods.
Significance. The paper addresses a genuine gap in open-vocabulary 3D scene understanding: most existing methods treat objects in isolation, and incorporating relational context via automatically constructed scene graphs is a reasonable and novel direction. The dual-stream architecture is well-motivated by the feature granularity dilemma, and the ablation studies (Tabs. 6–7) systematically isolate individual components. The cross-dataset zero-shot evaluation on ScanNet++ and Replica provides evidence of generalization beyond the training domain. The 'shower curtain' example (Tab. 2: 0.0 → 64.2 IoU) is a compelling qualitative case for the value of contextual reasoning. The overall experimental evaluation is thorough across four datasets with per-category and head/common/tail breakdowns.
major comments (3)
- §4.4, Table 6, 'w/o relation guidance' row: This ablation removes ALL edge features while preserving graph topology, yielding 55.5 vs. 58.4 mIoU on ScanNetV2 and 11.0 vs. 14.5 on ScanNet200. However, the paper's central novelty claim is that VLM-inferred semantic relationships (SVO triplets) provide the contextual priors. This ablation does not distinguish between (a) the VLM's specific semantic relationship content being valuable and (b) any edge features at all (e.g., spatial labels like 'near'/'adjacent', random vectors, or learned embeddings) providing a useful inductive bias for attention modulation. Since edge features enter the GAT as Key-side concatenations (§3.3, 'Edge-Feature-Guided Message Propagation'), their mere presence gives the attention mechanism additional degrees of freedom regardless of semantic content. An additional ablation replacing VLM-derived edge features with
- §3.1, 'VLM-based Edge Refinement and Encoding': The entire pipeline depends on the VLM producing correct relationship descriptions, but no quantitative assessment of VLM relationship quality is reported — no precision, recall, or accuracy on any ground-truth scene graph dataset. This is load-bearing because the paper claims the constructed graph provides 'reliable relational priors' (§3.1). Even a small-scale manual evaluation of the VLM's relationship predictions on a subset of ScanNetV2 scenes (e.g., 50 edges compared to human annotations) would substantially strengthen the claim. Without this, it is unclear whether the gains come from accurate relationship semantics or from the graph structure and edge-feature mechanism alone (which connects to the previous comment).
- Table 2, 'shower curtain' column: The flagship claim that relational context resolves the curtain/shower curtain ambiguity (0.0 → 64.2 IoU) is presented without a per-category ablation. The improvement could arise from the dual-stream CLIP features (which alone may better distinguish shower curtains via visual semantics) rather than from relation guidance specifically. A per-category ablation showing the shower curtain IoU under 'w/o relation guidance' vs. full model would directly test whether the relationship reasoning is the operative factor for this specific case.
minor comments (7)
- §3.1, Eq. (1): The components of S (s_comp, s_area, s_center, s_sharp) are described only briefly. A short definition or formula for each would improve reproducibility.
- §3.4, Eq. (6): The margin M is added only to self-pairs (via δ_ij), but the motivation for why a margin is needed specifically for self-pairs is not explained clearly. A brief justification would help.
- Table 1: Several entries are marked with '†' with different meanings ('reported in the original paper' vs. 'reproduced by us'). A footnote clarifying which entries are reproduced and under what protocol would improve clarity.
- §4.1: The keyframe scoring weights W = [w_comp, w_area, w_center, w_sharp] are listed as free parameters in the axiom ledger but their values are not specified in the main text. These should be reported or referenced to the supplementary.
- Figure 2: The pipeline diagram is dense and some text labels are difficult to read. Consider simplifying or enlarging key components for clarity.
- §3.3: The number of GAT layers L is not specified in the main text. This should be reported for reproducibility.
- The paper mentions 'further implementation specifics, including the exact VLM prompts, are provided in the supplementary material' (§4.1), but the supplementary was not available for review. Ensure the prompts are included in the final submission.
Simulated Author's Rebuttal
We thank the referee for a thorough and constructive report. The referee correctly identifies our central novelty claims and raises three interconnected concerns about whether the gains can be attributed to VLM-inferred semantic relationship content specifically, as opposed to graph structure, edge-feature degrees of freedom, or dual-stream CLIP features. We address each comment below and commit to concrete revisions.
read point-by-point responses
-
Referee: §4.4, Table 6, 'w/o relation guidance' ablation does not distinguish between (a) VLM semantic relationship content being valuable and (b) any edge features at all providing a useful inductive bias. An additional ablation replacing VLM-derived edge features with alternatives (spatial labels, random vectors, learned embeddings) is needed.
Authors: The referee raises a valid and important concern. Our current 'w/o relation guidance' ablation removes all edge features entirely, which conflates the effect of having edge features with the effect of having semantically meaningful VLM-derived edge features. We agree this distinction is critical because our central novelty claim is that VLM-inferred SVO triplets provide valuable contextual priors, not merely that edge features provide additional attention modulation capacity. We will add a new ablation row in Table 6 replacing VLM-derived edge features with (i) spatial-only labels ('near'/'adjacent' encoded via CLIP text encoder) and (ii) random fixed edge vectors, while preserving the same graph topology and edge-feature-guided GAT mechanism. This will directly test whether the semantic content of VLM-inferred relationships contributes beyond the structural inductive bias of having any edge features. We expect the VLM-derived features to outperform both alternatives, particularly on categories where contextual reasoning is most beneficial (e.g., shower curtain, tail categories on ScanNet200), but we will report whatever results we obtain honestly. revision: yes
-
Referee: §3.1, 'VLM-based Edge Refinement and Encoding': No quantitative assessment of VLM relationship quality is reported. Even a small-scale manual evaluation on a subset of ScanNetV2 scenes would substantially strengthen the claim that the constructed graph provides 'reliable relational priors.'
Authors: We agree that this is a genuine gap in our evaluation. The claim that the constructed graph provides 'reliable relational priors' is load-bearing for our contribution, and we currently provide no quantitative evidence for it. We will conduct a small-scale human evaluation of VLM-inferred relationships on a subset of ScanNetV2 scenes. Specifically, we plan to sample approximately 100 edges across 20 scenes, have a human annotator judge whether each VLM-inferred SVO triplet is semantically correct (precision), and also estimate recall by checking whether obvious relationships are captured by the VLM. We will report precision, recall, and qualitative error patterns. This will not be a large-scale annotation effort, but it will provide the first quantitative evidence of VLM relationship quality and will allow readers to assess whether the gains plausibly come from accurate relationship semantics. We acknowledge that if the VLM relationship quality turns out to be modest, this would suggest that the graph structure and edge-feature mechanism contribute substantially on their own, which would be an honest and important finding. revision: yes
-
Referee: Table 2, 'shower curtain' column: The 0.0 → 64.2 IoU improvement is presented without a per-category ablation. The improvement could arise from the dual-stream CLIP features rather than from relation guidance specifically. A per-category ablation showing shower curtain IoU under 'w/o relation guidance' vs. full model would directly test whether relationship reasoning is the operative factor.
Authors: This is a fair and well-targeted criticism. The shower curtain example is our flagship qualitative case for the value of relational context, yet we have not actually isolated whether the improvement comes from relation guidance or from the dual-stream CLIP features. The referee is correct that CLIP features alone may better distinguish shower curtains via visual semantics, which would undermine the specific causal claim we make. We will add a per-category ablation for shower curtain (and ideally a few other context-dependent categories) showing IoU under: (i) w/o scene graph (baseline LSeg only), (ii) w/o relation guidance (graph + dual-stream CLIP, but no edge features), and (iii) full model. This will directly decompose the contribution of dual-stream CLIP features versus VLM relation guidance for this specific category. If the improvement is primarily from CLIP features, we will revise our claims accordingly and identify a different category where relation guidance is demonstrably the operative factor. If relation guidance does contribute specifically, the ablation will show it. Either way, the manuscript will be strengthened. revision: yes
Circularity Check
No circularity: the derivation chain is self-contained against external benchmarks
full rationale
The paper's central claim is that VLM-inferred object relationships, encoded as SVO-triplet edge features in a 3D scene graph, improve open-vocabulary 3D semantic segmentation. Walking the derivation chain: (1) Scene graph construction (Sec. 3.1) uses external pretrained models (qwen-vl-max, LSeg, CLIP) to generate edges and node features — these are not the authors' prior results. (2) The VLM annotation engine (Sec. 3.2) generates supervision targets via Describe Anything Model and Qwen-VL-Max — again external models, not self-cited prior work. (3) The Adaptive Gated Dual-Stream GAT (Sec. 3.3) is a new architectural contribution whose equations (Eqs. 3-4) define a gated fusion mechanism; no equation reduces to its input by definition. (4) The loss design (Sec. 3.4, Eqs. 5-8) defines a hierarchical contrastive objective and dual-alignment losses that are standard formulations, not tautological. (5) Evaluation uses standard external benchmarks (ScanNetV2, ScanNet200, ScanNet++, Replica) with standard metrics (mIoU, mAcc). The ablation studies (Tabs. 6-7) properly remove components to measure their contribution. The 'w/o relation guidance' ablation (55.5 vs 58.4 mIoU) removes edge features entirely rather than isolating VLM-derived semantic content from generic edge features — this is a legitimate concern about experimental design and causal attribution, but it is not circularity. The VLM annotations serve as supervision targets and the model is trained against them, but this is standard practice (using generated labels as training signal), not a self-definitional loop. No step in the derivation chain reduces to its inputs by construction, no prediction is a renamed fit, and no load-bearing argument depends on a self-citation chain. The skeptic's concern about whether gains come from VLM relationship semantics specifically versus any edge features is a correctness/attribution risk, not a circularity issue.
Axiom & Free-Parameter Ledger
free parameters (8)
- λ_hie, λ_reg, λ_aux =
[0.5, 1.5, 1.5]
- γ_reg, γ_aux =
[0.7, 0.7]
- τ (similarity threshold) =
Not specified
- M (margin) =
Not specified
- η (temperature) =
Not specified
- α (scaling factor) =
Not specified
- W (keyframe scoring weights) =
Not specified
- L (number of GAT layers) =
Not specified
axioms (4)
- domain assumption VLM (Qwen-VL-Max) can reliably infer spatial and semantic relationships between objects from 2D images with visual prompting.
- domain assumption Dense LSeg features and instance-level CLIP features are complementary and their combination improves open-vocabulary segmentation.
- domain assumption Spatial proximity (3 nearest neighbors or bounding box adjacency) is a sufficient initial graph topology for relationship reasoning.
- domain assumption SVO triplet parsing of VLM-generated relationship descriptions produces meaningful edge features.
invented entities (1)
-
Super node (v_super)
no independent evidence
Reference graph
Works this paper leans on
-
[1]
Campos, C., Elvira, R., Rodríguez, J.J.G., Montiel, J.M.M., Tardós, J.D.: ORB- SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM. IEEE Trans. Robotics37(6), 1874–1890 (2021) 16 Chen et al
work page 2021
-
[2]
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pp. 9630–9640. IEEE (2021)
work page 2021
-
[3]
In: Tan, J., Toussaint, M., Darvish, K
Chang, H., Boyalakuntla, K., Lu, S., Cai, S., Jing, E.P., Keskar, S., Geng, S., Abbas, A., Zhou, L., Bekris, K.E., Boularias, A.: Context-aware entity grounding with open-vocabulary 3d scene graphs. In: Tan, J., Toussaint, M., Darvish, K. (eds.) Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA. Proceedings of Machine Learning R...
work page 2023
-
[4]
Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 1280–1289. IEEE (2022)
work page 2022
-
[5]
Cho, S., Shin, H., Hong, S., Arnab, A., Seo, P.H., Kim, S.: Cat-seg: Cost aggre- gation for open-vocabulary semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 4113–4123. IEEE (2024)
work page 2024
-
[6]
Choy, C.B., Gwak, J., Savarese, S.: 4d spatio-temporal convnets: Minkowski con- volutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 3075–3084. Computer Vision Foundation / IEEE (2019)
work page 2019
-
[7]
Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T.A., Nießner, M.: Scan- net: Richly-annotated 3d reconstructions of indoor scenes. In: 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. pp. 2432–2443. IEEE Computer Society (2017)
work page 2017
-
[8]
Dai, A., Ritchie, D., Bokeloh, M., Reed, S., Sturm, J., Nießner, M.: Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 4578–4587. Computer Vision Founda- tion / IEEE Computer Society (2018)
work page 2018
-
[9]
Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., Qi, X.: PLA: language-driven open-vocabulary 3d scene understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24,
work page 2023
- [10]
-
[11]
In: International Conference on Learning Representa- tions (2024)
Engelmann, F., Manhardt, F., Niemeyer, M., Tateno, K., Pollefeys, M., Tombari, F.: OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views. In: International Conference on Learning Representa- tions (2024)
work page 2024
-
[12]
In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T
Ghiasi, G., Gu, X., Cui, Y., Lin, T.: Scaling open-vocabulary image segmentation with image-level labels. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI. Lecture Notes in Computer Science, vol. 13696, pp. 54...
work page 2022
-
[13]
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K.M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., Gan, C., de Melo, C.M., Tenenbaum, J.B.,Torralba,A.,Shkurti,F.,Paull,L.:Conceptgraphs:Open-vocabulary3dscene graphs for perception and planning. In: IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, J...
work page 2024
-
[14]
In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G
Hu, J., Chen, X., Feng, B., Li, G., Yang, L., Bao, H., Zhang, G., Cui, Z.: CG- SLAM: efficient dense RGB-D SLAM in a consistent uncertainty-aware 3d gaussian field. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September29-October4,2024,Proceedin...
work page 2024
-
[15]
In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S
Hu, J., Mao, M., Bao, H., Zhang, G., Cui, Z.: CP-SLAM: collaborative neural point-based SLAM system. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 20...
work page 2023
-
[16]
Huang, J., Zhang, H., Yi, L., Funkhouser, T.A., Nießner, M., Guibas, L.J.: Tex- turenet: Consistent local parametrizations for learning from high-resolution signals on meshes. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 4440–4449. Computer Vision Foundation / IEEE (2019)
work page 2019
-
[17]
Jain, J., Li, J., Chiu, M., Hassani, A., Orlov, N., Shi, H.: Oneformer: One trans- former to rule universal image segmentation. In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 2989–2998. IEEE (2023)
work page 2023
-
[18]
Jiang, L., Shi, S., Schiele, B.: Open-vocabulary 3d semantic segmentation with foundation models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 21284–21294. IEEE (2024)
work page 2024
-
[19]
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W., Dollár, P., Girshick, R.B.: Segment anything. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. pp. 3992–4003. IEEE (2023)
work page 2023
-
[20]
In: Inter- national Conference on 3D Vision, 3DV 2024, Davos, Switzerland, March 18-21,
Koch, S., Hermosilla, P., Vaskevicius, N., Colosi, M., Ropinski, T.: Lang3dsg: Language-based contrastive pre-training for 3d scene graph prediction. In: Inter- national Conference on 3D Vision, 3DV 2024, Davos, Switzerland, March 18-21,
work page 2024
- [21]
-
[22]
Koch, S., Vaskevicius, N., Colosi, M., Hermosilla, P., Ropinski, T.: Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-setrelationships.In:IEEE/CVFConferenceonComputerVisionandPattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 14183–14193. IEEE (2024)
work page 2024
-
[23]
Koch, S., Wald, J., Colosi, M., Vaskevicius, N., Hermosilla, P., Tombari, F., Ropin- ski, T.: Relationfield: Relate anything in radiance fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 21706–21716. Computer Vision Foundation / IEEE (2025)
work page 2025
-
[24]
Lambert, J., Liu, Z., Sener, O., Hays, J., Koltun, V.: Mseg: A composite dataset for multi-domain semantic segmentation. In: 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19,
work page 2020
- [25]
-
[26]
Lee, J., Park, C., Choe, J., Wang, Y.F., Kautz, J., Cho, M., Choy, C.B.: Mo- saic3d: Foundation dataset and model for open-vocabulary 3d segmentation. In: IEEE/CVFConferenceonComputerVisionandPatternRecognition,CVPR2025, Nashville, TN, USA, June 11-15, 2025. pp. 14089–14101. Computer Vision Foun- dation / IEEE (2025) 18 Chen et al
work page 2025
-
[27]
Li, B., Weinberger, K.Q., Belongie, S.J., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net (2022)
work page 2022
-
[28]
Li, J., Saltori, C., Poiesi, F., Sebe, N.: Cross-modal and uncertainty-aware agglom- eration for open-vocabulary 3d scene understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 19390–19400. Computer Vision Foundation / IEEE (2025)
work page 2025
-
[29]
In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G
Li, R., Zhang, Z., He, C., Ma, Z., Patel, V.M., Zhang, L.: Dense multimodal align- ment for open-vocabulary 3d scene understanding. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Pro- ceedings, Part XLIX. Lecture Note...
work page 2024
-
[30]
Describe Anything: Detailed Localized Image and Video Captioning
Lian, L., Ding, Y., Ge, Y., Liu, S., Mao, H., Li, B., Pavone, M., Liu, M., Darrell, T., Yala,A.,Cui,Y.:Describeanything:Detailedlocalizedimageandvideocaptioning. CoRRabs/2504.16072(2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[31]
In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., Zhu, J., Zhang, L.: Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy...
work page 2024
-
[32]
Lu, J., Deng, J.: Relation3d : Enhancing relation modeling for point cloud in- stance segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 8889–8899. Computer Vision Foundation / IEEE (2025)
work page 2025
-
[33]
In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G
Naeem, M.F., Xian, Y., Zhai, X., Hoyer, L., Gool, L.V., Tombari, F.: SILC: improv- ing vision language pretraining with self-distillation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXI. Lecture ...
work page 2024
-
[34]
In: International Conference on 3D Vi- sion, 3DV 2021, London, United Kingdom, December 1-3, 2021
Nekrasov, A., Schult, J., Litany, O., Leibe, B., Engelmann, F.: Mix3d: Out-of- context data augmentation for 3d scenes. In: International Conference on 3D Vi- sion, 3DV 2021, London, United Kingdom, December 1-3, 2021. pp. 116–125. IEEE (2021)
work page 2021
-
[35]
Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual features...
work page 2024
-
[36]
Peng, S., Genova, K., Jiang, C.M., Tagliasacchi, A., Pollefeys, M., Funkhouser, T.A.: Openscene: 3d scene understanding with open vocabularies. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 815–824. IEEE (2023)
work page 2023
-
[37]
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, RelGraphOV 19 P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y.,...
work page 2025
-
[38]
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual E...
work page 2021
-
[39]
Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: Sam 2: Segment anything in images and videos (2024)
work page 2024
-
[40]
In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T
Rozenberszki, D., Litany, O., Dai, A.: Language-grounded indoor 3d semantic seg- mentation in the wild. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII. Lecture Notes in Computer Science, vol. 13693, pp. 125...
work page 2022
-
[41]
Schult, J., Engelmann, F., Kontogianni, T., Leibe, B.: Dualconvmesh-net: Joint geodesic and euclidean convolutions on 3d meshes. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 8609–8619. Computer Vision Foundation / IEEE (2020)
work page 2020
-
[42]
The Replica Dataset: A Digital Replica of Indoor Spaces
Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J.J., Mur- Artal, R., Ren, C., Verma, S., Clarkson, A., Yan, M., Budge, B., Yan, Y., Pan, X., Yon, J., Zou, Y., Leon, K., Carter, N., Briales, J., Gillingham, T., Mueggler, E., Pesqueira, L., Savva, M., Batra, D., Strasdat, H.M., Nardi, R.D., Goesele, M., Lovegrove, S., Newcombe, R.:...
work page internal anchor Pith review Pith/arXiv arXiv 1906
-
[43]
In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S
Takmaz, A., Fedele, E., Sumner, R.W., Pollefeys, M., Tombari, F., Engelmann, F.: Openmask3d: Open-vocabulary 3d instance segmentation. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orle...
work page 2023
-
[44]
Tatarchenko, M., Park, J., Koltun, V., Zhou, Q.: Tangent convolutions for dense prediction in 3d. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 3887–
work page 2018
-
[45]
Computer Vision Foundation / IEEE Computer Society (2018)
work page 2018
-
[46]
Wald, J., Dhamo, H., Navab, N., Tombari, F.: Learning 3d semantic scene graphs from 3d indoor reconstructions. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 3960–3969. Computer Vision Foundation / IEEE (2020)
work page 2020
-
[47]
Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: VGGT: Visual geometry grounded transformer. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) (2025)
work page 2025
-
[48]
Wang, Y., Jia, B., Zhu, Z., Huang, S.: Masked point-entity contrast for open- vocabulary 3d scene understanding. In: IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. pp. 14125–14136. Computer Vision Foundation / IEEE (2025) 20 Chen et al
work page 2025
-
[49]
Wang, Z., Cheng, B., Zhao, L., Xu, D., Tang, Y., Sheng, L.: VL-SAT: visual- linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud. In: IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 21560–21569. IEEE (2023)
work page 2023
-
[50]
In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Syst...
work page 2022
-
[51]
In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021
Wu, S., Wald, J., Tateno, K., Navab, N., Tombari, F.: Scenegraphfusion: Incre- mental 3d scene graph prediction from RGB-D sequences. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 7515–7525. Computer Vision Foundation / IEEE (2021)
work page 2021
-
[52]
Yan,M.,Zhang, J.,Zhu, Y.,Wang,H.: Maskclustering: View consensusbased mask graph clustering for open-vocabulary 3d instance segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 28274–28284. IEEE (2024)
work page 2024
-
[53]
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. CoRRabs/2310.11441 (2023)
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[54]
Yang, J., Ding, R., Deng, W., Wang, Z., Qi, X.: Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. In: IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 19823–19832. IEEE (2024)
work page 2024
-
[55]
Yeshwanth, C., Liu, Y., Nießner, M., Dai, A.: Scannet++: A high-fidelity dataset of 3d indoor scenes. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. pp. 12–22. IEEE (2023)
work page 2023
-
[56]
Yuan, Y., Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L., Zhu, J.: Osprey: Pixel understanding with visual instruction tuning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 28202–28211. IEEE (2024)
work page 2024
-
[57]
Advances in Neural Information Processing Systems38, 15556–15580 (2026)
Zhang, X., Bao, C., Chen, Y., Zhai, H., Dong, Y., Bao, H., Cui, Z., Zhang, G.: Atlasgs: Atlanta-world guided surface reconstruction with implicit structured gaus- sians. Advances in Neural Information Processing Systems38, 15556–15580 (2026)
work page 2026
-
[58]
In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S
Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Wang, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,...
work page 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.