Pith. sign in

REVIEW 4 major objections 5 minor 68 references

RelationField: Relate Anything in Radiance Fields

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RelationField embeds open-vocabulary object relationships directly inside neural radiance fields.

desk verdict The pair-of-rays relationship field is a genuinely novel idea, but the 3DSSG evaluation never controls for the teacher model's own knowledge, so the claim that relationships come 'directly from radiance fields' is not yet proven. read the letter →

arxiv 2412.13652 v2 pith:RHIMJWOY submitted 2024-12-18 cs.CV

Sebastian Koch , Johanna Wald , Mirco Colosi , Narunas Vaskevicius , Pedro Hermosilla , Federico Tombari , Timo Ropinski This is my paper · ORCID
classification cs.CV
keywords neuralradiancefieldsopen-vocabularysceneunderstandingrelationshipdistillation3Dgraphsfeatureset-of-markpromptingrelationship-guidedinstancesegmentationmultimodalLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RelationField aims to show that a neural radiance field can carry open-vocabulary knowledge about relationships between objects, not just object identities. The paper's claim is that by adding a second 'query ray' to the field's input and distilling relationship descriptions from a multimodal large language model, the field learns a dense, multi-view-consistent embedding of inter-object relations that can be queried with arbitrary text. On the 3DSSG benchmark the resulting representation sets a new state of the art for open-vocabulary 3D scene graph prediction, and it enables a new relationship-guided instance segmentation task on ScanNet++. A sympathetic reader takes away that relationship understanding can live inside the same implicit scene representation used for rendering, without depth sensors or explicit meshes.

What carries the argument

The load-bearing mechanism is a pair-of-rays relationship head: the function $g_\theta(x, d, z)$ maps a sample point, its viewing direction, and a query location to color, density, object features, and a 512-dimensional relationship feature in a language-aligned embedding space. Training uses Set-of-Mark prompting—overlaying numbered semi-transparent masks on training images—so a multimodal LLM can name object pairs and their relationships; those textual descriptions are encoded and distilled into the field with a cosine-similarity rendering loss. At query time, a textual predicate is encoded, matched against rendered relationship features, and converted to a response probability by pairwise softmax against canonical phrases such as "and", "next to", and "none". This same mechanism, combined with object-level semantic and instance heads, lets the authors extract full 3D scene graphs by clustering instance embeddings and aggregating relationship responses for instance pairs.

What would settle it

Train a controlled variant of RelationField on a fixed set of views in which the query location $z$ for each sampled ray pair is randomly permuted across object instances, holding the LLM relationship labels fixed. If 3D scene graph recall on 3DSSG stays at the same level, the relationship head is not actually using the identity of the second object and the pair-of-rays formulation is not doing the claimed work; if recall collapses, the dependency on the query object is real.

Watch

Extended reading notes

Core claim

The central discovery is that inter-object relationships can be defined as a function of two points in a radiance field: the sampled location $x$ on a ray and an additional implicit query location $z$, so the field outputs a relationship feature $r$ alongside color, density, and object features. Because the relationship feature lives in a language embedding space, any textual predicate can be matched against it by cosine similarity. The field is trained by rendering $r$ along rays and maximizing its similarity to language encodings of relationship descriptions that a multimodal LLM produces for object pairs marked in each training view. The paper argues that this distillation yields a 3D-consistent relationship representation, which is why it outperforms both 2D-only LLM inference and prior feature-field methods on scene graph prediction and relationship-guided instance segmentation.

Load-bearing premise

All relationship knowledge comes from a multimodal language model's per-view captions; if that teacher mislabels, misses, or hallucinates a relationship in the training views, the radiance field stores the error faithfully because no geometric or physical check corrects it.

Editorial extensions

If this is right

  • 3D scene graphs can be extracted from posed RGB images alone, without depth sensors, meshes, or point-cloud inputs at training time.
  • Objects that are otherwise ambiguous can be disambiguated by relationship: the same instance head plus a predicate like "standing on" separates "picture standing on the shelf" from other pictures.
  • Relationship errors from 2D view-dependent LLM inference are reduced, because volumetric rendering integrates relationship evidence across views into one consistent 3D field.
  • The approach is agnostic to the teacher model and the underlying radiance field, working with different multimodal LLMs and with both NeRF and Gaussian Splatting geometry.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the pair-of-rays input is symmetric in principle but trained with a fixed subject-object order, the authors' direction-swapping supervision trick could be made automatic by also training on swapped pairs, which would likely improve symmetric predicate recall.
  • If the teacher LLM mislabels a relationship in every view, the field will memorize that error; probing the field with deliberately contradictory predicates in a scene with known ground truth would quantify how much teacher hallucination survives distillation.
  • The same representation could support affordance and task-planning queries directly in 3D, since affordance predicates such as "turns on" and "attached to" already appear in the training prompts.
  • A natural stress test is whether the field can answer a predicate that the LLM never mentioned in any training view; if it cannot, the "open-vocabulary" claim is bounded by the teacher's coverage rather than by the embedding space.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RelationField, a NeRF extension that predicts, in addition to color, density, object-level semantic features, and instance grouping features, a relationship feature for each pair of 3D query locations. Relationship supervision is obtained by prompting a multimodal LLM (GPT-4o) with Set-of-Mark annotated training views, encoding the textual relationship descriptions with jina-embeddings-v3, and training a volumetric MLP head to reproduce those features via a cosine-similarity loss. At inference, relationship queries are answered by comparing the query text embedding to the rendered relationship features. The method is evaluated on 3D scene graph generation on the RIO10 subset of 3DSSG, where it reports state-of-the-art top-k recall, and on a newly introduced relationship-guided 3D instance segmentation benchmark on ScanNet++, with qualitative results on additional scenes. Ablations compare 2D per-view inference, different LLM teachers, and a Gaussian Splatting variant.

Significance. If the central claims are established, the paper makes a useful contribution: it shows that inter-object relationship features can be distilled into a volumetric field from posed RGB images alone, that such a field supports open-vocabulary relationship queries, and that 3D-consistent rendering reduces per-view errors relative to 2D inference. The method is clearly described, the 3D-consistency ablation in Fig. 5 is informative, and the authors' explicit statement in Sec. 5 that the relationship knowledge depends on the LLM teacher is honest. However, the quantitative evaluation does not yet disentangle the contribution of the radiance-field representation from that of the per-scene teacher, and the new benchmark is small and not released. These are substantive gaps for the paper's central and secondary claims.

major comments (4)
  1. [Sec. 4.2, Tab. 1] The main quantitative evidence for the central claim is not yet decisive because it lacks a control that isolates the radiance-field relationship representation from the per-scene GPT-4 teacher. The relationship supervision is generated per training view by GPT-4 with SoM prompting, encoded with jina-embeddings-v3, and distilled into the field (Sec. 3.2); the GPT-4 (2D+depth) baseline in Tab. 1 is not described as using the same SoM prompt, the same pair-sampling strategy, the same text encoder, or the same aggregation into the 3D point cloud. A baseline that lifts the same per-view GPT-4/SoM relationship labels to the point cloud via SAM masks and then ranks them with the same label embeddings could match or exceed RelationField's recall without any volumetric relationship field. Reporting such a matched teacher-lifting baseline is necessary to support the claim that relationships are extracted directly from the radiance field rather than inherited from the teacher; the paper's own Sec. 5 limitation statement makes this concern explicit. The concern is not tautology, since the field is queried at locations and pairs not directly supervised, but the benchmark as reported does not separate the teacher's per-view knowledge from the field's 3D-rendered knowledge.
  2. [Sec. 4.3 and supplementary Sec. F] The newly introduced relationship-guided instance segmentation benchmark is too small and too under-specified to support the quantitative claim. It contains '~30 relationship queries across 8 scenes' and the manuscript does not state that the annotations will be released or provide a download link; without the benchmark, the results in Tab. 2 cannot be reproduced or extended by other groups. With ~30 queries, the differences between LERF, OpenNeRF, LangSplat, and RelationField (IoU 0.25, 0.45, 0.49, 0.53) are not accompanied by error bars or a per-scene breakdown, so it is unclear whether any of the differences beyond LERF are statistically meaningful. For a benchmark proposed to support future research, a release plan and variance estimates are necessary.
  3. [Sec. 4.2, '3D Scene Graph Construction'] The quantitative scene-graph evaluation queries the field on the provided 3D point cloud, and the paper states the point cloud is used 'exclusively for evaluation.' This is a reasonable protocol for comparison with point-cloud-based methods, but it means the demonstrated 3D scene graph extraction is not purely from the radiance field: instance seeds and query locations come from the depth-sensor-derived geometry. The paper should either (a) describe and evaluate a fully radiance-field-based protocol in which instances and query locations are obtained by volumetric rendering (as in Fig. 3), or (b) clearly state that the quantitative graphs currently require ancillary geometry at test time. This distinction matters for the abstract's claim of extracting relationships 'directly from neural radiance fields.'
  4. [Sec. 4.2, data and metrics] The open-vocabulary claim is only measured indirectly on a closed set of 160 object classes and 27 relationship types; the paper queries these predefined labels and reports top-k recall. The qualitative results in Fig. 3 demonstrate open-vocabulary behavior, but a quantitative evaluation on held-out relationship types or an object/relationship retrieval task would directly support the 'open-vocabulary' wording in the title and abstract. In addition, the paper calls its results 'state-of-the-art in both tasks,' but the second task is newly proposed, so 'state-of-the-art' there only means 'better than the three existing feature-field baselines,' which is a weaker statement than the paper's phrasing suggests.
minor comments (5)
  1. [Abstract and Introduction] There are several language errors: 'recently even been extended' should be 'have recently been extended'; 'current method primarily' should be 'current methods primarily'; and 'multi-model LLM' should be 'multimodal LLM.'
  2. [Supplementary, Fig. 12 caption] The caption refers to 'bag-of-words behavior of CLIP [41]', but reference [41] is GPT-2; CLIP is reference [42] in the main text. Please correct the citation.
  3. [Sec. 3.1 and Sec. 3.4] Equation (2) defines the relationship field with inputs (x, d, z), but Sec. 3.4 and the scene graph construction in Sec. 4.2 appear to query the field without a direction d; please clarify whether d is used in the relationship head or whether Eq. (2) should be simplified.
  4. [Sec. 3.3, Eq. (4)] The response score depends on the choice of canonical phrases ('and', 'next to', 'none'), and the paper gives no sensitivity analysis for this choice or for the 0.5 threshold used in supplementary Sec. C. Adding such an analysis would help readers understand how much the qualitative queries depend on these free parameters.
  5. [General] The paper does not mention any plan to release code or trained models; given the complexity of the pipeline (NeRF training, SAM, GPT-4 prompting, jina encoding), a code release would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the relationship features are distilled from an external teacher and evaluated against a benchmark, with no prediction reducing to its training input by construction.

full rationale

The paper's derivation chain is a per-scene distillation: GPT-4o with SoM prompting produces textual relationships, jina-embeddings-v3 encodes them, and the radiance field is trained to reproduce those embeddings via the cosine-similarity loss in Eq. 3. Querying the field is therefore partly a retrieval of teacher knowledge, but the paper's claims require the field to generalize to object pairs and views not explicitly supervised, and it provides a 3D-consistency ablation against a 2D GPT-4 baseline. The 3DSSG evaluation uses held-out benchmark labels encoded in the same embedding space, but this is a standard open-vocabulary evaluation protocol rather than a constructional equivalence: the teacher's free-text output can disagree with the benchmark annotations, and the radiance field must render, cluster, and aggregate features over the point cloud before any label similarity is computed. No equation in the paper reduces a reported prediction to its training target, and no load-bearing claim is justified solely by a self-citation: the self-citations to SGRec3D, Lang3DSG, and Open3DSG are contextual or comparative, not foundational to RelationField's core derivation. The acknowledged limitation that the relationship knowledge depends on the multi-modal LLM prompting is a statement about supervision quality, not evidence that the method's outputs are equivalent to its inputs by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities or forces; it learns an embedding field. The main external dependencies are the teacher LLM and the text encoder, which are treated as axioms. The hand-tuned prompt template and edge threshold are free parameters affecting the results.

free parameters (3)
  • edge threshold = 0.5
    In supplementary Sec. C, a relationship response threshold of 0.5 is used to decide whether an edge is added to the scene graph. No sensitivity analysis is provided.
  • canonical phrase set = ["and", "next to", "none"]
    Equation (4) uses a hand-chosen set of canonical phrases for the softmax normalization. The choice may affect the calibrated relationship scores.
  • SoM prompt instructions
    The textual prompt in supplementary Sec. B includes many hand-tuned instructions (e.g., 'Always combine a spatial relationship with a semantic...', 'Avoid generic terms') that shape the teacher's output. These are effectively free parameters of the supervision pipeline.
assumptions (4)
  • domain assumption Multimodal LLM (GPT-4o) provides accurate relationship annotations for object pairs in training images.
    The entire relationship supervision comes from GPT-4o with SoM prompting (Sec. 3.2). The paper acknowledges in Sec. 5 that the relationship knowledge is highly dependent on the LLM prompting and output.
  • domain assumption Jina-v3 embedding space preserves semantic relationship similarity, so cosine similarity is a valid query score.
    The text encoder choice is justified with a similarity matrix in supplementary Sec. B, but the assumption that cosine similarity in this space aligns with human-judged relationship correctness is not independently validated.
  • domain assumption Volumetric rendering weights from NeRF correctly propagate relationship features from 3D to 2D.
    The relationship feature is rendered along the ray using the same weights as color and density (Sec. 3.2). This assumes that the relationship field is well-defined at each 3D point and that the rendering operator is appropriate for these learned embeddings.
  • domain assumption Instance embeddings produce clean clusters under DBSCAN, and aggregated relationship embeddings are reliable for scene graph extraction.
    Scene graph extraction (Sec. 4.2 and supp. Sec. C) first clusters instance embeddings with DBSCAN and then aggregates relationship embeddings per cluster. The quality of this clustering and aggregation is not separately evaluated, and the thresholding relies on an arbitrary 0.5 cutoff.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RelationField: Relate Anything in Radiance Fields." pith.science (2026). https://pith.science/paper/RHIMJWOY

@misc{pith2026241213652,
  author       = {Pith},
  title        = {Pith review of: RelationField: Relate Anything in Radiance Fields},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RHIMJWOY}},
  note         = {Machine review of arXiv:2412.13652}
}
read the original abstract

Neural radiance fields are an emerging 3D scene representation and recently even been extended to learn features for scene understanding by distilling open-vocabulary features from vision-language models. However, current method primarily focus on object-centric representations, supporting object segmentation or detection, while understanding semantic relationships between objects remains largely unexplored. To address this gap, we propose RelationField, the first method to extract inter-object relationships directly from neural radiance fields. RelationField represents relationships between objects as pairs of rays within a neural radiance field, effectively extending its formulation to include implicit relationship queries. To teach RelationField complex, open-vocabulary relationships, relationship knowledge is distilled from multi-modal LLMs. To evaluate RelationField, we solve open-vocabulary 3D scene graph generation tasks and relationship-guided instance segmentation, achieving state-of-the-art performance in both tasks. See the project website at https://relationfield.github.io.

Figures

Figures reproduced from arXiv: 2412.13652 by the authors.

Figure 1
Figure 1. Open-Vocabulary Relationship Understanding. We propose RelationField, the first framework to extract open-vocabulary inter-object relationships directly from neural radiance fields. RelationField can answer a wide variety of relationship queries, such as “composition”, “compare”, “spatial”, “affordance” and “support” relationships. Abstract Neural radiance fields are an emerging 3D scene rep￾resentation and recently… view at source ↗
Figure 2
Figure 2. RelationField Training. Left: RelationField learns a 3D feature field (a) that can be queried with a relationship query location (b) which changes the relationship field of the 3D volume depending on what position is selected. The relationship feature is sampled and rendered along a ray according to NeRF’s rendering weights. The language loss maximizes the cosine similarity between the extracted sparse features from… view at source ↗
Figure 3
Figure 3. Results with RelationField in 4 in-the-wild scenes. Each image shows a rendering from RelationField, along with the relationship response for each query relationship. The relevancy score describes the answer of the model to the question: What is standing on/attached to/similar to etc.? For demonstration purposes, we highlight the click as well as the outline of the clicked object, which is not needed when querying t… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: show a subset of extracted relationships with sub￾ject, predicate, and object labels, respectively, on a scene [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: 3D Consistency Ablation. Left: Extracted SoM marks per image with query. Center: Existing relationship in GPT-4 caption. Right: Relationship response from RelationField rendered into image space. While GPT-4 struggles with partially visible ob￾jects, RelationField prod…
Figure 6
Figure 6. Figure 6: Language Model Ablation. We compare GPT-4 with Llama 3.2 as the relationship extractor of RelationField for 3D scene graph prediction. we want to highlight the benefit of understanding relation￾ships from open-vocabulary textual descriptions for localiz￾ing objects of …
Figure 7
Figure 7. Figure 7: Language Encoder Ablation. We compare 5 language encoder-only model based on their separability in embedding space. For each language encoder, we plot a similarity matrix, for the pairwise cosine-similarity of 41 predicates taken from the 3DSSG dataset. embeddings gene…
Figure 8
Figure 8. Figure 8: Relationship Direction. (a) visualizes the relationship response for the question “What is standing on/lying on/similar to?”, where we localize the object in a subject-predicate-object relationship. While (b) visualizes the relationship response for the question “What …
Figure 9
Figure 9. Figure 9: Relationship Edge Construction. To extract a 3D scene graph from RelationField, we automatically query instances (a), compute the relationship response for predicates such as “lying on” (b), and aggregate the relationship response for each instance (c). We add an edge …
Figure 10
Figure 10. Figure 10: Additional Qualitative Results. We provide relationship responses for 4 additional scenes from Scannet++. The colormap visualizes the relationship response where blue is low and red is high. We visualize the relationships for the question: “What is standing on/lying o…
Figure 11
Figure 11. Figure 11: RelationField w/ NeRF or w/ Gaussian Splatting geometry. We compare the rendering speed (FPS), memory requirements and RelationField quality for the query “standing on”. mation together with the relationship information from the prompt to accurately filter predictions…
Figure 12
Figure 12. Figure 12: Releationship-guided 3D Instance Segmentation. We compare OpenNeRF with RelationField for relationship-guided 3D instance segmentation. While OpenNeRF produces many false positives because it gets confused with compositional queries arising from the bag-of-words behav…
Figure 13
Figure 13. Figure 13: Releation-guided 3D Instance Segmentation Task Overview. We visualize a few annotated segments from our labeled benchmark on Scannet++ together with annotated relationship prompts. We focus on objects which appear multiple times in the scene, but that can be uniquely …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 53 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 5, 7

    arXiv 2023

  2. [2]

    Taskography: Evaluating robot task planning over large 3d scene graphs

    Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, and Florian Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. In Proceed- ings of the 5th Conference on Robot Learning, pages 46–58. PMLR, 2022. 2

    work page 2022

  3. [3]

    Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese

    Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R. Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and cam- era. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), 2019. 2

    work page 2019

  4. [4]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the International Conference on Computer Vision (ICCV), 2021. dino. 1, 2

    work page 2021

  5. [5]

    Clip-driven open-vocabulary 3d scene graph generation via cross-modality contrastive learning

    Lianggangxu Chen, Xuejiao Wang, Jiale Lu, Shaohui Lin, Changbo Wang, and Gaoqi He. Clip-driven open-vocabulary 3d scene graph generation via cross-modality contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 27863–27873, 2024. 3

    work page 2024

  6. [6]

    Spatial- rgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language models. In NeurIPS, 2024. 3

    work page 2024

  7. [7]

    Click-gaussian: Interactive segmenta- tion to any 3d gaussians

    Seokhun Choi, Hyeonseop Song, Jaechul Kim, Taehyeong Kim, and Hoseok Do. Click-gaussian: Interactive segmenta- tion to any 3d gaussians. arXiv preprint arXiv:2407.11793,

    arXiv

  8. [8]

    4d spatio-temporal convnets: Minkowski convolutional neural networks

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075–3084,

Show all 68 references
  1. [9]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 2, 7

    2023 arXiv

  2. [10]

    Scenefun3d: Fine-grained functionality and affordance un- derstanding in 3d scenes

    Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Francis Engelmann. Scenefun3d: Fine-grained functionality and affordance un- derstanding in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (C...

    2024

  3. [11]

    Dbscan clustering algorithm based on den- sity

    Dingsheng Deng. Dbscan clustering algorithm based on den- sity. In 2020 7th International Forum on Electrical Engineer- ing and Automation (IFEEA), pages 949–953, 2020. 6

    2020

  4. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  5. [13]

    OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views

    Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, and Federico Tombari. OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views. In Interna- tional Conference on Learning Representations, 2024. 2, 3, 4, 8, 1

    2024

  6. [14]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision (ECCV), pages 540–557, 2022. 5, 1

    2022

  7. [15]

    Tenenbaum, Antonio Tor- ralba, Florian Shkurti, and Liam Paull

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Tor- ralba, Florian Shkurti, and Liam Paull. Conceptgrap...

    2024

  8. [16]

    Semantic abstraction: Open-world 3D scene understanding from 2D vision-language models

    Huy Ha and Shuran Song. Semantic abstraction: Open-world 3D scene understanding from 2D vision-language models. In Proceedings of the 2022 Conference on Robot Learning,

    2022

  9. [17]

    Nerf-rpn: A general framework for object de- tection in nerfs

    Benran Hu, Junkai Huang, Yichen Liu, Yu-Wing Tai, and Chi- Keung Tang. Nerf-rpn: A general framework for object de- tection in nerfs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23528–23538, 2023. 3

    2023

  10. [18]

    Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation

    Zhening Huang, Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby. Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation. In European Conference on Computer Vision (ECCV), 2024. 2

    2024

  11. [19]

    Hughes, Y

    N. Hughes, Y . Chang, and L. Carlone. Hydra: A real-time spatial perception system for 3D scene graph construction and optimization. In Robotics: Science and Systems (RSS) ,

  12. [20]

    Bert: Pre-training of deep bidirectional transform- ers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transform- ers for language understanding. In Proceedings of naacL- HLT, page 2. Minneapolis, Minnesota, 2019. 4, 1

    2019

  13. [21]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023. 2, 3

    2023

  14. [22]

    Lerf: Language embed- 9 ded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embed- 9 ded radiance fields. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 19729–19739, 2023. 2, 3, 4, 5, 8, 1

    2023

  15. [23]

    Garfield: Group anything with radiance fields

    Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Gold- berg, Matthew Tancik, and Angjoo Kanazawa. Garfield: Group anything with radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21530–21539, 2024. 2, 3, 4, 1

    2024

  16. [24]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Visio...

    2023

  17. [25]

    Sgrec3d: Self-supervised 3d scene graph learning via object-level scene reconstruc- tion

    Sebastian Koch, Pedro Hermosilla, Narunas Vaskevicius, Mirco Colosi, and Timo Ropinski. Sgrec3d: Self-supervised 3d scene graph learning via object-level scene reconstruc- tion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3...

  18. [26]

    Lang3dsg: Language- based contrastive pre-training for 3d scene graph prediction

    Sebastian Koch, Pedro Hermosilla, Narunas Vaskevicius, Mirco Colosi, and Timo Ropinski. Lang3dsg: Language- based contrastive pre-training for 3d scene graph prediction. In 2024 International Conference on 3D Vision (3DV), pages 1037–1047. IEEE, 2024. 3

    2024

  19. [27]

    Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships

    Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  20. [28]

    3d-relnet: Joint object and relational network for 3d prediction

    Nilesh Kulkarni, Ishan Misra, Shubham Tulsiani, and Abhi- nav Gupta. 3d-relnet: Joint object and relational network for 3d prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 2

    2019

  21. [29]

    Rethinking open-vocabulary segmentation of radiance fields in 3d space

    Hyunjee Lee, Youngsik Yun, Jeongmin Bae, Seoha Kim, and Youngjung Uh. Rethinking open-vocabulary segmentation of radiance fields in 3d space. arXiv preprint arXiv:2408.07416,

    arXiv

  22. [30]

    Roberta: A robustly optimized BERT pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:/1907.11692, 2019. 1

    1907 arXiv

  23. [31]

    Delta: Decomposed efficient long-term robot task planning using large language models

    Yuchen Liu, Luigi Palmieri, Sebastian Koch, Ilche Georgievski, and Marco Aiello. Delta: Decomposed efficient long-term robot task planning using large language models. arXiv preprint arXiv:2404.03275, 2024. 2

    2024 arXiv

  24. [32]

    3d vsg: Long-term se- mantic scene change prediction through 3d variable scene graphs

    Samuel Looper, Javier Rodriguez-Puigvert, Roland Siegwart, Cesar Cadena, and Lukas Schmid. 3d vsg: Long-term se- mantic scene change prediction through 3d variable scene graphs. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8179–8186. IEEE, 2023. 2

    2023

  25. [33]

    Visual relationship detection with language priors

    Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationship detection with language priors. In Computer Vision–ECCV 2016: 14th European Conference, Proceedings, Part I 14, pages 852–869. Springer, 2016. 7

    2016

  26. [34]

    Clio: Real-time task-driven open- set 3d scene graphs

    Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, and Luca Carlone. Clio: Real-time task-driven open- set 3d scene graphs. IEEE Robotics and Automation Letters, 9(10):8921–8928, 2024. 3

    2024

  27. [35]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. In European Conference on Computer Vision (ECCV), pages 405–421. Springer International Publishing,...

    2020

  28. [36]

    An End-to- End Transformer Model for 3D Object Detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An End-to- End Transformer Model for 3D Object Detection. In ICCV,

  29. [37]

    Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance

    Phuc Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 4...

    2024

  30. [38]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu “Max” Jiang, An- drea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 815–824, 2...

    2023

  31. [39]

    Deep hough voting for 3d object detection in point clouds

    Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEE International Conference on Com- puter Vision, 2019. 1, 2

    2019

  32. [40]

    Langsplat: 3d language gaussian splatting

    Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20051–20060,

  33. [41]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsu- pervised multitask learners. 2019. 1, 3

    2019

  34. [42]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  35. [43]

    Sayplan: Ground- ing large language models using 3d scene graphs for scalable task planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable task planning. In 7th Annual Conference on Robot Learning,

  36. [44]

    3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans

    Antoni Rosinol, Arjun Gupta, Marcus Abate, Jingnan Shi, and Luca Carlone. 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. In Robotics: Science and Systems (RSS), 2020. 2

    2020

  37. [45]

    Kimera: From slam to spatial perception with 3d 10 dynamic scene graphs

    Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupta, and Luca Carlone. Kimera: From slam to spatial perception with 3d 10 dynamic scene graphs. The International Journal of Robotics Research, 40(12-14):1510–1546, 2021. 2

    2021

  38. [46]

    Sgaligner: 3d scene alignment with scene graphs

    Sayan Deb Sarkar, Ondrej Miksik, Marc Pollefeys, Daniel Barath, and Iro Armeni. Sgaligner: 3d scene alignment with scene graphs. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 21927– 21937, 2023. 2

    2023

  39. [47]

    Mask3D: Mask Trans- former for 3D Semantic Instance Segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Trans- former for 3D Semantic Instance Segmentation. In Interna- tional Conference on Robotics and Automation (ICRA), 2023. 1, 2

    2023

  40. [48]

    Panoptic lifting for 3d scene understanding with neural fields

    Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bul `o, Nor- man M ¨uller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (C...

    2023

  41. [49]

    The replica dataset: A digital replica of indoor spaces

    Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797,

    1906 arXiv

  42. [50]

    jina-embeddings-v3: Mul- tilingual embeddings with task lora, 2023

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael G¨unther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Kouk- ounas, Nan Wang, and Han Xiao. jina-embeddings-v3: Mul- tilingual embeddings with task lora, 2023. 5, 6, 1

    2023

  43. [51]

    Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann

    Ayc ¸a Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Open- Mask3D: Open-V ocabulary 3D Instance Segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 1, 2

    2023

  44. [52]

    Nerfstudio: A mod- ular framework for neural radiance field development

    Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A mod- ular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1–...

    2023

  45. [53]

    Learning 3d semantic scene graphs from 3d indoor reconstructions

    Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3, 7

    2020

  46. [54]

    Vl-sat: Visual-linguistic semantics as- sisted training for 3d semantic scene graph prediction in point cloud

    Ziqin Wang, Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang, and Lu Sheng. Vl-sat: Visual-linguistic semantics as- sisted training for 3d semantic scene graph prediction in point cloud. arXiv preprint arXiv:2303.14408, 2023. 2

    2023 arXiv

  47. [55]

    Hierarchical Open- V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation

    Abdelrhman Werby, Chenguang Huang, Martin B¨uchner, Ab- hinav Valada, and Wolfram Burgard. Hierarchical Open- V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation. In Proceedings of Robotics: Science and Sys- tems, Delft, Netherlands, 2024. 2

    2024

  48. [56]

    Scenegraphfusion: Incremen- tal 3d scene graph prediction from rgb-d sequences

    Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremen- tal 3d scene graph prediction from rgb-d sequences. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7515–7525, 2021. 2, 3

    2021

  49. [57]

    Incremental 3d semantic scene graph prediction from rgb sequences

    Shun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari. Incremental 3d semantic scene graph prediction from rgb sequences. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 5064–5074, 2023. 3

    2023

  50. [58]

    Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection

    Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, Ruilong Li, Jialiang Wang, Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection. In Proceedings of the IEEE/CVF Interna- ...

    2023

  51. [59]

    Graph r-cnn for scene graph generation

    Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceed- ings of the European conference on computer vision (ECCV), pages 670–685, 2018. 7

    2018

  52. [61]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. 4, 1

    2023 arXiv

  53. [62]

    Scannet++: A high-fidelity dataset of 3d indoor scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 12–22, 2023. 2, 5, 8

    2023

  54. [63]

    When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022. 4

    2022 arXiv

  55. [64]

    Knowledge-inspired 3d scene graph prediction in point cloud

    Shoulong Zhang, Shuai Li, Aimin Hao, and Hong Qin. Knowledge-inspired 3d scene graph prediction in point cloud. In Advances in Neural Information Processing Systems , pages 18620–18632. Curran Associates, Inc., 2021. 3

    2021

  56. [65]

    Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and An- drew J. Davison. In-place scene labelling and understand- ing with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3 11 RelationField: Relate Any...

    2021

  57. [66]

    and SAM [24], as well as the relationship features with GPT-4 increases the training time by about 30 minutes for the first run. B. Design choices Prompting. To extract textual relationships using GPT-4 [1] or Llama [ 12], we employ a combination of visual and textual promptin...

  58. [67]

    to overlay semi-transparent masks and numeric marks. The textual prompt consists of a two-stage approach which queries the model first to extract objects by their mark-id and then to extract relationships referenced by the previ- ously extracted object-ids together with a rela...

  59. [68]

    Create a dict that maps tag id to class name

    Object Identification: Identify all objects in the image by their tag. Create a dict that maps tag id to class name

  60. [69]

    What is < > standing on/attached to/similar to etc. ?

    Affordance/Relationship Detection: For every pair of tagged ob- jects that are clearly related, describe the semantic relationships and affordances as a list of dictionaries using the format [s id: #n1, sub- ject class: x, o id: #n2, object class: y, predicates: [p1, p2, ...]]...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.