Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that feeding a large language model a learnable 3D scene graph—objects plus semantic edges to their nearest neighbors—improves accuracy on object grounding, dense captioning, and question answering over a flat list of…

desk verdict Useful engineering contribution, but the headline claim that semantic relations drive the gains is not actually isolated by the ablations. read the letter →

arxiv 2412.18450 v3 pith:H5TR6ICC submitted 2024-12-24 cs.CV

Tatiana Zemskova , Dmitry Yudin This is my paper · ORCID
classification cs.CV
keywords 3Dsceneunderstandingsemanticgraphlargelanguagemodelsvisualgroundingdensecaptioningquestionansweringlearnablerepresentationpointcloud
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

3DGraphLLM claims that a large language model can answer questions about a 3D scene more accurately when the scene is fed in as a learnable scene graph—objects plus explicit semantic relation edges between nearby objects—rather than a flat list of object embeddings. The paper builds a token sequence in which each object is represented by an identifier, its 2D and 3D features, and triplets pairing it with its two nearest neighbors through relation embeddings sourced from a pre-trained scene-graph model. Across five ScanNet benchmarks, adding these relation triplets improves object grounding, dense captioning, and question answering over the no-relations baseline, with the largest gains on the grounding tasks. The practical payoff is that an embodied agent can get these gains without a large increase in token budget: representing a 100-object scene with $k=2$ neighbors needs about 800 tokens, and inference on grounding tasks runs up to five times faster than comparable vision-language-model approaches.

What carries the argument

The key machinery is a flat, learnable token representation of a 3D scene graph. Each object becomes a small subgraph: an identifier token, a 2D object feature token from DINOv2, a 3D point-cloud feature token from Uni3D, and $k$ nearest-neighbor triplets $(F^v_i, F^e_{ij}, F^v_j)$, where $F^e_{ij}$ is the projected latent edge embedding produced by VL-SAT, a pre-trained 3D scene-graph generator. This flattens the graph into a sequence the LLM can consume, reducing a complete graph's $O(n^2)$ edges to $3nk$ triplet tokens; with $k=2$ and 100 objects the scene description is about 800 tokens. The projection layers and a low-rank fine-tuned LLM are trained to predict object identifiers, captions, or answers, so the relation edges serve as direct evidence inside the prompt rather than as a separate reasoning module.

What would settle it

A direct test would be to replace VL-SAT latent edge features with random or zero embeddings while keeping the same graph structure and training pipeline; if ScanRefer or Multi3DRefer accuracy stays at the same level, the improvement attributed to semantic relationships is actually coming from the added object-neighbor tokens or from the projection layers rather than from the relation semantics.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM can solve 3D vision-language tasks more accurately when the scene is presented as a learnable semantic scene graph rather than as a list of object embeddings. For each object, the representation includes an identifier token, a 2D DINOv2 feature token, a 3D Uni3D point-cloud token, and $k$ triplets $(F^v_i, F^e_{ij}, F^v_j)$ pairing the object with its nearest neighbors; $F^e_{ij}$ is the projected latent edge embedding from VL-SAT, taken before the relation-classification head so it can encode combinations of relations. The paper reports that this edge-aware representation outperforms the no-edges baseline on five ScanNet benchmarks, with gains of +7.5 F1@0.5 on Multi3DRefer, +6.4 Acc@0.5 on ScanRefer, and +3.9 CIDEr@0.5 on Scan2Cap, while using roughly 800 tokens to represent a 100-object scene at $k=2$. It also reports grounding accuracy on par with the strongest compared vision-language model at up to five times lower inference time.

Load-bearing premise

The approach assumes that the learned relationship features between objects, produced by a model trained on one set of indoor scans, still carry useful meaning when applied to the separate set of indoor scans used for evaluation.

Editorial extensions

If this is right

  • An embodied agent can get more accurate object grounding from an LLM by adding just a few semantic relation tokens per object, without expanding the scene description to hundreds of text tokens.
  • Semantic edges help even when instance segmentation is imperfect, so the approach remains usable when ground-truth object masks are unavailable.
  • The same graph-to-token encoding supports grounding, dense captioning, and question answering with one model, since all tasks are framed as user-assistant text generation.
  • With $k=2$ nearest neighbors, a 100-object scene uses about 800 tokens, which keeps inference fast enough for interactive use; this follows directly from the paper's token-count calculation and speed experiments.
  • Relation edges help most when the query itself is about spatial relationships; the paper's ablations show grounding and captioning gain more than question answering, because some QA questions do not need spatial context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: replace VL-SAT latent edge features with categorical relation labels or with CLIP text embeddings of relation phrases; if accuracy stays flat, the gain is coming from the extra neighbor context rather than from the learned relation geometry.
  • Because the gains persist with noisy Mask3D segmentation, the representation may tolerate imperfect perception, but the paper only evaluates indoor ScanNet-like scenes; extending the same token format to outdoor or dynamic scenes would require an edge encoder for those domains, which is an inference beyond the paper's experiments.
  • The paper's discussion of n-gram metrics suggests that some of the apparent failures in captioning and QA are actually correct answers containing spatial cues absent from the references; if that reading is right, the true advantage of semantic edges on free-form language tasks may be larger than the reported scores show.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes 3DGraphLLM, a method that converts a 3D scene into a sequence of tokens: for each object, an identifier token, a 2D DINOv2 feature, and a subgraph of k nearest neighbors encoded as triplets (object 3D feature, VL-SAT relation embedding, neighbor 3D feature), all projected into the token embedding space of a pre-trained LLM. The authors evaluate on ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D, with a two-stage training procedure using ground-truth and predicted instance segmentation. The central claim is that including semantic relationship embeddings significantly improves grounding, captioning, and question answering over a k=0 object-list baseline, and that the method achieves state-of-the-art grounding accuracy with fewer tokens and faster inference.

Significance. The method is well-motivated and the engineering is solid: it reuses strong pretrained encoders (DINOv2, Uni3D, VL-SAT), provides a compact graph tokenization (800 tokens for 100 objects at k=2), reports detailed ablations across LLMs, segmentation quality, and k values, and releases code. If the causal effect of relation embeddings were established, this would be a valuable contribution. However, the experiments as designed do not isolate the contribution of semantic relation content, so the central claim currently rests on a confounded comparison. The resource-efficiency and competitive grounding results are encouraging and should be preserved, but the causal interpretation needs additional controls.

major comments (5)
  1. [§4.2, Table 4] The comparison between 3DGraphLLM-0 and 3DGraphLLM-2 changes three ingredients at once: the token count (2n vs 2n+3nk, i.e., 200 vs 800 tokens for n=100), the presence of neighbor object 3D features Fv_j in the prompt, and the presence of semantic edge embeddings Fe_ij. Therefore the observed gains cannot be attributed specifically to semantic relationships, as the abstract claims. To isolate the effect, the authors need control conditions that keep the neighbor features and sequence length fixed while ablating the edge content, for example replacing Fe_ij with zero/random embeddings, or shuffling edge embeddings across object pairs. Such controls are missing, so the central claim is not yet supported.
  2. [§4.2, Table 8 / Appendix C] The triplet-vs-edge-only ablation is also not a control for semantic content: both conditions contain relation embeddings Fe_ij, and the triplet condition additionally includes neighbor vertex tokens Fv_j and more tokens. This experiment can at most show that neighbor vertex features help; it does not test whether the relation embeddings themselves carry useful semantics. A condition with neighbor features but without edge embeddings would be needed.
  3. [§4.2, Tables 4 and 5] All reported numbers are single point estimates. With gains as small as +0.1 EM on SQA3D (Vicuna, Table 4) and even a negative difference on ScanRefer Acc@0.5 (50.2 to 50.1), the word 'significantly' in the abstract is not justified without variance or significance tests across training seeds or validation subsets.
  4. [Supplementary Table 7 / Appendix B] The text in Appendix B states that semantic edges yield 'a consistent improvement' across all three tasks, but Table 7 shows that for OneFormer3D segmentation, 3DGraphLLM-2 decreases Scan2Cap CIDEr@0.5 (73.5 to 70.2), BLEU-4@0.5 (34.3 to 32.7), ScanQA CIDEr (87.3 to 83.3), and BLEU-4 (16.5 to 15.0) relative to 3DGraphLLM-0. This contradicts the blanket claim in the abstract that semantic relationships improve performance across the benchmarks, and it should be discussed.
  5. [§4.2 and Appendix E] The authors themselves argue that n-gram metrics are 'not adequate' for LLM-generated answers and provide a correct caption that receives CIDEr 0.0. Yet the manuscript uses CIDEr and BLEU as primary evidence for gains on Scan2Cap, ScanQA, and SQA3D. Unless a complementary evaluation (e.g., human ratings or an LLM-as-judge) is provided, the cross-task improvement claim on these benchmarks is not trustworthy.
minor comments (6)
  1. [Figure 2] The caption uses 'LoRa' where the text and standard notation write 'LoRA'; please fix the typo.
  2. [§3.2] The token count formula 2n+3nk assumes an ordered, directed triplet for each neighbor; if the graph is considered undirected, clarify why the directed count is appropriate for the prompt sequence.
  3. [§4.2 / Figure 4] The main text says 'ground-truth scene segmentation' while Figure 4's caption says 'GT instance segmentation'; unify the terminology.
  4. [§2] The related-work discussion cites Scene-LLM and MOMA but does not position 3DGraphLLM against them as other LLM-based methods that consume scene-graph-like structures; a brief comparison would help.
  5. [Table 2] The 'C' column for ScanQA is defined only in the text; add a note in the table caption indicating that C denotes CIDEr.
  6. [References] Reference [34] is a self-citation used as support for text-based graph retrieval; the relationship to the current method should be clarified explicitly in the text.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claim rests on an empirical comparison against an external baseline, not on a derivation that reduces to its own inputs. One self-citation appears but is not load-bearing.

full rationale

The paper's central claim is empirical: adding semantic-edge tokens from VL-SAT to a Chat-Scene-style object token sequence improves grounding, captioning, and QA scores on ScanNet benchmarks. There is no derivation chain in which a predicted quantity is defined in terms of the same quantity, and no fitted parameter is renamed as a prediction. Object, 2D, and edge features come from external pretrained encoders (Uni3D, DINOv2, VL-SAT), and evaluations use separate training/validation splits with standard external metrics. The only self-citation is reference [34] in the related-work discussion; it supports a peripheral statement about text-graph retrieval and is not used as the load-bearing justification for the proposed method. The skeptic's concern that the k=0 vs k=2 ablation confounds semantic relations with extra neighbor tokens and longer context is a legitimate experimental-control issue, but it is not a circular equivalence: the comparison still tests whether the added graph tokens help, even if it does not uniquely isolate the semantic content of those tokens. No enumerated circularity pattern applies, so the appropriate finding is no significant circularity, with a minor non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central contribution depends on three external feature extractors and one hand-chosen graph construction. The model itself has no invented physical entities; the identifier tokens are trainable vocabulary entries, not explanatory constructs.

free parameters (3)
  • k (number of nearest neighbors per object) = 2
    Selected via ablation (Fig. 4, Sec. 4.2). k=2 balances accuracy and inference speed; higher k improves grounding but increases tokens and memory.
  • NMS IoU threshold for neighbor deduplication = 0.99
    Chosen in Sec. 4.2 to remove duplicate object proposals in Mask3D segmentation before selecting neighbors.
  • Minimum distance between neighbor centers = 1 cm
    Chosen in Sec. 4.2 to exclude an object's own duplicated segments from its neighbor subgraph.
assumptions (3)
  • domain assumption VL-SAT edge features transfer from 3RScan to ScanNet
    Used in Sec. 3.1 to generate relation embeddings for ScanNet scenes without fine-tuning; the paper flags that scene graph methods struggle in cross-domain settings.
  • domain assumption Uni3D and DINOv2 provide object features that are useful for LLM grounding, captioning, and QA
    These pretrained encoders are fixed during training (Sec. 3.1); the method inherits their inductive biases.
  • domain assumption Nearest neighbors by 3D spatial distance are the most task-relevant relational context
    The graph construction (Sec. 3.2) chooses the k closest objects to define subgraphs; no semantic or functional neighbor selection is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding." pith.science (2026). https://pith.science/paper/H5TR6ICC

@misc{pith2026241218450,
  author       = {Pith},
  title        = {Pith review of: 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5TR6ICC}},
  note         = {Machine review of arXiv:2412.18450}
}
read the original abstract

A 3D scene graph represents a compact scene model by capturing both the objects present and the semantic relationships between them, making it a promising structure for robotic applications. To effectively interact with users, an embodied intelligent agent should be able to answer a wide range of natural language queries about the surrounding 3D environment. Large Language Models (LLMs) are beneficial solutions for user-robot interaction due to their natural language understanding and reasoning abilities. Recent methods for learning scene representations have shown that adapting these representations to the 3D world can significantly improve the quality of LLM responses. However, existing methods typically rely only on geometric information, such as object coordinates, and overlook the rich semantic relationships between objects. In this work, we propose 3DGraphLLM, a method for constructing a learnable representation of a 3D scene graph that explicitly incorporates semantic relationships. This representation is used as input to LLMs for performing 3D vision-language tasks. In our experiments on popular ScanRefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap datasets, we demonstrate that our approach outperforms baselines that do not leverage semantic relationships between objects. The code is publicly available at https://github.com/CognitiveAISystems/3DGraphLLM.

Figures

Figures reproduced from arXiv: 2412.18450 by the authors.

Figure 1
Figure 1. The proposed 3DGraphLLM approach leverages 3D semantic scene graph learnable representation supplied as input to an LLM to perform various 3D vision-language tasks. The goal of 3D dense scene captioning is to generate a textual description of a selected object in the 3D scene, including its attributes or relationships. Finally, the goal of the 3D visual question answering task is to generate text answers to various … view at source ↗
Figure 2
Figure 2. The overall architecture of our approach. We introduce trainable layers to map the extracted graph node and edge features into [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative examples of 3DGraphLLM performance on object grounding, dense captioning, and question answering tasks. We [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Dependence of inference speed and visual grounding [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Dependence of inference speed and dense scene cap [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Dependence of inference speed and question answering [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Dependence of inference speed and question answering [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Functional queries about the room and objects to the 3DGraphLLM. 3DGraphLLM is capable of answering questions about the [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Common failure cases of 3DGraphLLM related to spatial relationships. In the question answering task, 3DGraphLLM incorrectly [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Topology-aware attention over hierarchical scene graphs lets a 3D-LLM ground, caption, and answer questions across multi-room homes, with large gains on a new HM3D benchmark.

  2. Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.

  3. Open-Vocabulary Indoor Object Grounding with 3D Hierarchical Scene Graph

    cs.CV 2025-07 conditional novelty 4.0 of 10

    OVIGo-3DHSG builds a five-level scene graph (building, floor, room, location, object) and uses LLM reasoning over relevant subgraphs to ground open-vocabulary objects in multi-floor indoor scenes.

Reference graph

Works this paper leans on

70 extracted references · 51 canonical work pages · cited by 3 Pith papers

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 422–440. Springer, 2020. 5

    work page 2020

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. 6

    work page 2024

  3. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19129– 19139, 2022. 1, 2, 5, 6

    work page 2022

  4. [4]

    Context-aware entity grounding with open-vocabulary 3d scene graphs

    Haonan Chang, Kowndinya Boyalakuntla, Shiyang Lu, Si- wei Cai, Eric Jing, Shreesh Keskar, Shijie Geng, Adeeb Ab- bas, Lifeng Zhou, Kostas Bekris, et al. Context-aware entity grounding with open-vocabulary 3d scene graphs. arXiv preprint arXiv:2309.15940, 2023. 1, 2

    arXiv 2023

  5. [5]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202–221. Springer, 2020. 1, 2, 5, 6

    work page 2020

  6. [6]

    Language conditioned spatial relation reasoning for 3d object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35:20522–20535,

  7. [7]

    Ll3da: Visual interactive instruction tuning for omni-3d understand- ing, reasoning, and planning, 2023

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understand- ing, reasoning, and planning, 2023. 2, 3, 6

    work page 2023

  8. [8]

    Grounded 3d-llm with referent tokens

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370,

    arXiv

Show all 70 references
  1. [9]

    Scan2cap: Context-aware dense captioning in rgb- d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb- d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3193–3203,

  2. [10]

    Spatial- rgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language models. arXiv preprint arXiv:2406.01584, 2024. 2, 3

    2024 arXiv

  3. [11]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 4, 5

    2017

  4. [12]

    Multi-clip: Contrastive vision-language pre-training for question answering tasks in 3d scenes

    Alexandros Delitzas, Maria Parelli, Nikolas Hars, Geor- gios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Multi-clip: Contrastive vision-language pre-training for question answering tasks in 3d scenes. arXiv preprint arXiv:2306.02329, 2023. 2

    2023 arXiv

  5. [13]

    Scenegenie: Scene graph guided diffusion models for image synthesis

    Azade Farshad, Yousef Yeganeh, Yu Chi, Chengzhi Shen, B¨ojrn Ommer, and Nassir Navab. Scenegenie: Scene graph guided diffusion models for image synthesis. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 88–98, 2023. 2

    2023

  6. [14]

    Free-form description guided 3d visual graph network for object grounding in point cloud

    Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, and Aj- mal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceed- ings of the IEEE/CVF international conference on computer ...

    2021

  7. [15]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. 6

    2024 arXiv

  8. [16]

    Graphdreamer: Compositional 3d scene synthesis from scene graphs

    Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Sch¨olkopf. Graphdreamer: Compositional 3d scene synthesis from scene graphs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21295–21304, 2024. 2

    2024

  9. [17]

    Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Confere...

    2024

  10. [18]

    Zero-shot referring expression comprehension via structural similarity between images and captions

    Zeyu Han, Fangrui Zhu, Qianru Lao, and Huaizu Jiang. Zero-shot referring expression comprehension via structural similarity between images and captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14364–14374, 2024. 2

    2024

  11. [19]

    Relation-wise transformer network and reinforcement learning for visual navigation

    Yu He and Kang Zhou. Relation-wise transformer network and reinforcement learning for visual navigation. Neural Computing and Applications, pages 1–17, 2024. 1

    2024

  12. [20]

    Language-grounded dy- namic scene graphs for interactive object search with mobile manipulation

    Daniel Honerkamp, Martin B¨uchner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada. Language-grounded dy- namic scene graphs for interactive object search with mobile manipulation. IEEE Robotics and Automation Letters, 2024. 1, 2, 3

    2024

  13. [21]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NeurIPS, 2023. 3, 6

    2023

  14. [22]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023. 1, 2

    2023

  15. [23]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  16. [24]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168, 2023. 2, 3, 6

    2023 arXiv

  17. [25]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty- eighth Annual Conference on Neural Information P...

    2024

  18. [26]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baox- iong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023. 5, 6

    2023 arXiv

  19. [27]

    Multi- view transformer for 3d visual grounding

    Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi- view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524–15533, 2022. 6

    2022

  20. [28]

    Bottom up top down detection transform- ers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Kate- rina Fragkiadaki. Bottom up top down detection transform- ers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417–433. Springer, 2022. 6

    2022

  21. [29]

    Image retrieval using scene graphs

    Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668–3678, 2015. 2

    2015

  22. [30]

    Image gener- ation from scene graphs

    Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image gener- ation from scene graphs. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 1219–1228, 2018. 2

    2018

  23. [31]

    Robin3d: Improving 3d large language model via robust instruction tuning, 2025

    Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Robin3d: Improving 3d large language model via robust instruction tuning, 2025. 6

    2025

  24. [32]

    Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships

    Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  25. [33]

    Oneformer3d: One transformer for unified point cloud segmentation

    Maxim Kolodiazhnyi, Anna V orontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20943–20953, 2024. 3, 8

    2024

  26. [34]

    Beyond bare queries: Open- vocabulary object retrieval with 3d scene graph

    Sergey Linok, Tatiana Zemskova, Svetlana Ladanova, Ro- man Titkov, and Dmitry Yudin. Beyond bare queries: Open- vocabulary object retrieval with 3d scene graph. arXiv preprint arXiv:2406.07113, 2024. 2, 3

    2024 arXiv

  27. [35]

    Sqa3d: Sit- uated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Sit- uated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 5, 6

    2022 arXiv

  28. [36]

    Cross3dvg: Cross-dataset 3d visual grounding on different rgb-d scans

    Taiki Miyanishi, Daichi Azuma, Shuhei Kurita, and Motoaki Kawanabe. Cross3dvg: Cross-dataset 3d visual grounding on different rgb-d scans. In 2024 International Conference on 3D Vision (3DV), pages 717–727. IEEE, 2024. 2, 4, 5

    2024

  29. [37]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 4

    2023 arXiv

  30. [38]

    Labrad-or: lightweight memory scene graphs for accurate bimodal reasoning in dynamic operating rooms

    Ege ¨Ozsoy, Tobias Czempiel, Felix Holm, Chantal Pellegrini, and Nassir Navab. Labrad-or: lightweight memory scene graphs for accurate bimodal reasoning in dynamic operating rooms. In International Conference on Medical Image Com- puting and Computer-Assisted Intervention, pag...

    2023

  31. [39]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318,

  32. [40]

    Scene graph semantic inference for im- age and text matching

    Jiaming Pei, Kaiyang Zhong, Zhi Yu, Lukun Wang, and Ku- ruva Lakshmanna. Scene graph semantic inference for im- age and text matching. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(5):1–23,

  33. [41]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasac- chi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 815–824, 2023. 2

    2023

  34. [42]

    An approach to generate a caption for an image collection using scene graph generation

    Itthisak Phueaksri, Marc A Kastner, Yasutomo Kawanishi, Takahiro Komamizu, and Ichiro Ide. An approach to generate a caption for an image collection using scene graph generation. IEEE Access, 2023. 2

    2023

  35. [43]

    Gpt4scene: Understand 3d scenes from videos with vision-language models

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428, 2025. 6, 7

    2025 arXiv

  36. [44]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  37. [45]

    Kimera: From slam to spatial perception with 3d dynamic scene graphs

    Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupta, and Luca Carlone. Kimera: From slam to spatial perception with 3d dynamic scene graphs. The International Journal of Robotics Research, 40(12-14):1510–1546, 2021. 1

    2021

  38. [46]

    Mask3d: Mask trans- former for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask trans- former for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023. 3, 5, 6, 7, 8

    2023

  39. [47]

    Open- vocabulary object detection via scene graph discovery

    Hengcan Shi, Munawar Hayat, and Jianfei Cai. Open- vocabulary object detection via scene graph discovery. In Proceedings of the 31st ACM International Conference on Multimedia, pages 4012–4021, 2023. 2

    2023

  40. [48]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  41. [49]

    Cider: Consensus-based image description evalu- ation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalu- ation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 6

    2015

  42. [50]

    Rio: 3d object instance re-localization in changing indoor environments

    Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re-localization in changing indoor environments. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 7658–7667, 2019. 4, 5

    2019

  43. [51]

    Large language models for robotics: Opportunities, chal- lenges, and perspectives

    Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, et al. Large language models for robotics: Opportunities, chal- lenges, and perspectives. arXiv preprint arXiv:2401.04334,

    arXiv

  44. [52]

    Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud

    Ziqin Wang, Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang, and Lu Sheng. Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21560–21569,

  45. [53]

    Hierarchical open- vocabulary 3d scene graphs for language-grounded robot nav- igation

    Abdelrhman Werby, Chenguang Huang, Martin B ¨uchner, Abhinav Valada, and Wolfram Burgard. Hierarchical open- vocabulary 3d scene graphs for language-grounded robot nav- igation. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024. 1, 2, 3

    2024

  46. [54]

    3d question answering with scene graph reasoning

    Zizhao Wu, Haohan Li, Gongyi Chen, Zhou Yu, Xiaoling Gu, and Yigang Wang. 3d question answering with scene graph reasoning. In ACM Multimedia 2024, 2024. 2

    2024

  47. [55]

    Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7694–

    2024

  48. [56]

    Cross-modal rela- tionship inference for grounding referring expressions

    Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal rela- tionship inference for grounding referring expressions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4145–4154, 2019. 2

    2019

  49. [57]

    Auto-encoding scene graphs for image captioning

    Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10685–10694, 2019. 2

    2019

  50. [58]

    Visual programming for zero- shot open-vocabulary 3d visual grounding

    Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for zero- shot open-vocabulary 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20623–20633, 2024. 2, 3, 6

    2024

  51. [59]

    Com- monscenes: Generating commonsense 3d indoor scenes with scene graphs

    Guangyao Zhai, Evin Pınar ¨Ornek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Com- monscenes: Generating commonsense 3d indoor scenes with scene graphs. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  52. [60]

    Multi3drefer: Grounding text description to multiple 3d ob- jects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d ob- jects. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 15225–15236, 2023. 2, 5, 6

    2023

  53. [61]

    3dvg- transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg- transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 2928–2937, 2021. 2, 6

    2021

  54. [62]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 6

    2023

  55. [63]

    Uni3d: Exploring unified 3d representation at scale

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773,

    arXiv

  56. [64]

    Opti- mal graph transformer viterbi knowledge inference network for more successful visual navigation

    Kang Zhou, Chi Guo, Huyin Zhang, and Bohan Yang. Opti- mal graph transformer viterbi knowledge inference network for more successful visual navigation. Advanced Engineering Informatics, 55:101889, 2023. 1

    2023

  57. [65]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911– 2921, 2023. 2, 6

    2023

  58. [66]

    bath tub

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via prompt- able queries. In European Conference on Computer Vision, pages 188–206. Springer, 2025. 6 A. Ablation Stu...

    2025

  59. [67]

    It is sitting beside the tub

    It is the white toilet. It is sitting beside the tub

  60. [68]

    It has a towel on top of the lid

    It is the toilet by the curtain. It has a towel on top of the lid

  61. [69]

    There is a bathtub to the right of it and a counter to the left

    This is a white toilet. There is a bathtub to the right of it and a counter to the left. The lid is closed

  62. [70]

    It is the only toilet in the room

    The object is a toilet. It is the only toilet in the room. Figure 9. Common failure cases of 3DGraphLLM related to spatial relationships. In the question answering task, 3DGraphLLM incorrectly identifies the front/back and left/right directions relative to the observer. In the...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.