Pith. sign in

REVIEW 3 major objections 5 minor 30 references

Adding explicit spatial relations between objects in a 3D scene graph measurably improves a language model's ability to ground natural-language commands to the right object.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:29 UTC pith:D3FY2BJQ

load-bearing objection A genuinely useful, transparent empirical study on whether spatial edges in 3DSGs help LLM object grounding, but the headline result leans on a dataset where the testing procedure may inflate the benefit. the 3 major comments →

arxiv 2602.04635 v2 pith:D3FY2BJQ submitted 2026-02-04 cs.RO

Relational Scene Graphs for Object Grounding of Natural Language Commands

Julia Kuhn , Francesco Verdoja , Tsvetomila Mihaylova , Ville Kyrki This is my paper
classification cs.RO
keywords 3D scene graphsobject groundingnatural language commandsspatial relationslarge language modelsvision-language modelsopen-vocabularyrobot perception
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether a simple graph augmentation—adding labeled spatial edges between objects in a 3D scene graph—helps large language models resolve natural-language commands about objects. The authors build a pipeline that serializes scene graphs into text, then let two LLMs pick the referenced object ID. They find that adding position attributes and, more so, spatial edges consistently raises grounding accuracy across two datasets and both models, with gains up to 26 percentage points and most comparisons statistically significant. They also test whether open-vocabulary edges generated by a vision-language model from robot images outperform fixed closed-vocabulary edges; they find no significant difference, and they document generation errors that suggest reliability, not expressiveness, is the current bottleneck.

Core claim

The central claim is that explicit spatial relations in a 3D scene graph are not decorative—they carry information that LLMs use to disambiguate objects. On a large indoor-scene benchmark, grounding accuracy for GPT-4o rose from 76.97% with positions only to 84.27% with spatial edges; for GPT-5 it rose from 98.17% to 99.58%, and from a 73.0% baseline to 99.58% overall. On a robot-captured dataset with human-authored commands, both models improved substantially when edges were added. The paper further demonstrates that open-vocabulary edges generated from images by a VLM are feasible, but these generated edges sometimes describe camera-relative layouts or entities outside the graph, and on a

What carries the argument

The key object is the spatial edge added to the 3D scene graph: a directed relation between two object IDs, expressed either as a closed-vocabulary term like 'on' or as an open-vocabulary sentence. These edges are serialized into text along with object nodes and fed to an LLM that must return the target object's ID. The edges are what convert a collection of object positions into a relational description that matches how humans talk about scenes, and their presence—not just object positions—drives the accuracy gains.

Load-bearing premise

The entire comparison rests on the assumption that the spatial edges actually describe the scene the graph represents; if generated edges reference things outside the graph or depend on the camera's viewpoint, the observed effects and the open-vs-closed comparison become unreliable.

What would settle it

Run the same grounding task on a dataset where the spatial edges are randomly shuffled among object pairs while keeping the graph otherwise identical; if LLM accuracy stays at the same level as with correct edges, the edges are not carrying the claimed information.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Robots can substantially improve command understanding by storing explicit spatial relations in their scene graphs, even without retraining the language model.
  • The benefit of spatial edges grows with model capability: the stronger of the two LLMs gained more from the edges in every comparison, suggesting future models will rely even more on this kind of structured relational input.
  • Open-vocabulary edges generated from robot images are usable, but their quality must be checked for view-independence and in-graph references before deployment.
  • For large scenes, the token cost of serializing all edges is a bottleneck; approaches that select relevant subgraphs or retrieve only nearby objects will be needed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The absence of a statistically significant gap between open- and closed-vocabulary edges is likely power-limited; a larger benchmark with dozens of scenes and hundreds of generated edges could reveal a real difference once generation errors are filtered.
  • The documented failures—camera-relative descriptions and references to colors not in the graph—suggest a concrete fix: constrain VLM generation to the object set and to a fixed camera-independent reference frame, which could turn open-vocabulary edges into a strict improvement.
  • The same edge-augmentation idea could transfer to other downstream tasks such as spatial navigation instructions or manipulation target selection, where relational descriptions are equally natural.
  • Because the paper shows edges help even when they are occasionally noisy, robust relation generation may matter less than always ensuring edges are present.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether adding explicit spatial edges to 3D scene graphs helps LLMs ground natural-language referential statements to objects, and whether open-vocabulary edges generated from images by a VLM perform differently from closed-vocabulary edges. The authors build an LLM-based serialization/query pipeline, a VLM-based edge-generation pipeline, and evaluate GPT-4o and GPT-5 on VLA-3D and REACT scenes. They report accuracy gains for the graph with edges (G_{P,E}) over the position-only graph (G_P) and the node-only baseline (G), with McNemar significance in most comparisons, and conclude that open-vocabulary relations are feasible but show no significant difference from closed-vocabulary relations.

Significance. If the central claim is supported, the result is practically useful for robot scene understanding and for deciding how to enrich 3DSGs with relational edges. The paper's strengths are its use of standard off-the-shelf models, the inclusion of two datasets and two LLMs, the use of McNemar's test for paired comparisons, a human-authored command set, and a candid discussion of failure cases in VLM-generated edges. The public release of prompts is also a positive feature. However, the evidence for the main generalization is weakened by the fact that the primary evaluation uses statements generated from the very edges being tested, and the RQ2 result is based on a very small and potentially biased sample. The contribution is therefore more of a qualified empirical study than a definitive demonstration, though the approach is worth publishing after revision.

major comments (3)
  1. [V-B.1, V-C, Tables II-III] The main evidence for the central claim is partly circular. The VLA-3D referential statements are generated from the specific spatial edges present in the 3DSG, and the REACT generated statements are created with the same VLA-3D codebase. In G_{P,E}, the LLM is given those exact edges as text, so it can match the relation phrase directly; in G_P it must infer the relation from bounding boxes. The improvement may therefore reflect surface-form matching rather than improved grounding. The 119 human-authored statements in Section V-B.5 are a fairer test, but Table II reports only the 74 generated REACT statements (the accuracy values are consistent with denominators of 74), and no G_P vs G_{P,E} comparison is reported for the human-command set. For GPT-4o, the only significant edge effect is on VLA-3D; on REACT the G_P vs G_{P,E} cell is blank, i.e., not significant. Please report RQ1 on th
  2. [V-D.2, V-E] The RQ2 conclusion that open- and closed-vocabulary edges perform comparably is not well supported because many of the generated open-vocabulary edges are invalid relative to the 3DSG. The paper documents camera-relative descriptions ('left side of the table, closer to the camera'), references to objects absent from the graph ('glass wall and stairs'), and attributes not represented in the graph ('red chair'). Including such edges in the evaluation means the null result may reflect generation unreliability rather than vocabulary expressiveness. In addition, only 26 generated-command statements are tested, of which 22 are 'near' and 4 are 'on', and no McNemar statistics or power analysis are reported. The conclusion should be stated as inconclusive rather than as evidence of equivalence.
  3. [V-D.1, Table IV] The selection procedure for the RQ2 test set introduces a bias that limits external validity. Edges for which no fitting image was available were replaced with the original closed-vocabulary edge, and only the 26 statements whose edges were newly generated were tested. The image-selection criterion (highest combined pixel count of the two object masks) can also favor large objects. Because the tested set is small and skewed toward the 'near' relation, the absence of a statistically significant difference in Table IV is not informative about the general question of vocabulary choice. The paper should report edge-level statistics, exact p-values, and ideally a larger or more balanced sample.
minor comments (5)
  1. [V-D.1, notation] The notation G_{P,E} is used for both the closed- and open-vocabulary edge graphs in Experiment 2; this is confusing. Use distinct subscripts, e.g., G_{P,E}^{closed} and G_{P,E}^{open}.
  2. [Tables II-III] Table III has an empty cell for GPT-4o G vs G_P, and no p-values are reported. Please include exact McNemar p-values (or a dash with 'n.s.') for every comparison, and state the number of statements used in each test.
  3. [V-B.5] The description of the human-command collection is slightly ambiguous: 104 images, two annotations each, leading to 208 statements, of which 119 survive after ambiguity filtering. It would help to state how many objects and scenes these correspond to, and whether any scene was excluded.
  4. [V-D.2] Some example generated edges are long and would consume significant token budget; consider discussing whether edge length or graph total serialization size affects model performance, especially for larger scenes.
  5. [VI] The scalability discussion is welcome but is only qualitative. A short analysis of token counts for the graphs used would strengthen the claim that token limits are a practical barrier.

Circularity Check

0 steps flagged

No significant circularity: the grounding evaluation is empirical, uses external and human-authored data, and does not reduce to fitted inputs or self-citations.

full rationale

The paper makes no fitted-parameter prediction or first-principles derivation; it is an empirical evaluation of two off-the-shelf LLMs on a target-object grounding task. Grounding accuracy is computed against ground-truth object IDs from the external VLA-3D benchmark and from REACT scenes with independently authored human commands. The only self-citation is REACT [4] (authors include two co-authors of this paper), used as the 3DSG implementation and as one dataset source; this is a tool/dataset dependency, not a load-bearing argument, and the central RQ1 result is also supported on the external VLA-3D benchmark. The VLA-3D referential statements are generated from the dataset's spatial edges, which means the G_P,E condition has surface-form overlap with the queries; this is a potential validity limitation of that subset, not a circular derivation, and the paper also evaluates on REACT human commands that are not edge-derived. The paper explicitly discusses edge-generation failures (camera-relative 'left side' and color-referencing edges), acknowledging reliability limits; these are correctness concerns, not circularity. No step reduces a claimed result to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters or invented entities. The claims rest on the empirical validity of the serialization, the VLM edge-generation reliability, and the representativeness of the datasets.

axioms (4)
  • domain assumption LLMs can reason over text-serialized 3DSGs to identify referenced objects
    The entire grounding pipeline (Section IV-A) assumes that converting nodes and edges to text preserves enough information for GPT-4o/GPT-5 to perform the task; results are empirical evidence for this assumption, not a proof.
  • domain assumption VLM-generated open-vocabulary edges are semantically valid and view-independent
    Experiment 2 treats GPT-4o-generated edges as alternatives to closed-vocabulary edges; Section V-D documents failures (camera-relative descriptions, references to objects not in the 3DSG), so this assumption is partially violated.
  • domain assumption Accuracy on referential statements is a valid proxy for real-world object grounding
    The evaluation metric counts exact ID matches on templated or human-written statements; it does not test execution of commands or ambiguous natural conversations.
  • domain assumption The selected VLA-3D and REACT scenes are representative of typical indoor robot environments
    The VLA-3D subset consists of nine single-room scenes chosen randomly, and REACT has five scenes; generalization to larger, multi-room, cluttered environments is not established (the authors note token-limit scalability issues).

pith-pipeline@v1.3.0-alltime-deepseek · 11235 in / 10228 out tokens · 97764 ms · 2026-08-03T04:29:19.920984+00:00 · methodology

0 comments
read the original abstract

Robots are finding wider adoption in human environments, increasing the need for natural human-robot interaction. However, understanding a natural language command requires the robot to infer the intended task and how to decompose it into executable actions, and to ground those actions in the robot's knowledge of the environment, including relevant objects, agents, and locations. This challenge can be addressed by combining the capabilities of large language models (LLMs) to understand natural language with 3D scene graphs (3DSGs) for grounding inferred actions in a semantic representation of the environment. However, many 3DSGs lack explicit spatial relations between objects, even though humans often rely on these relations to describe an environment. This paper investigates whether incorporating open- or closed-vocabulary spatial relations into 3DSGs can improve the ability of LLMs to interpret natural language commands. To address this, we implement two pipelines using off-the-shelf models: an LLM-based pipeline for target object grounding from open-vocabulary language commands and a vision language model-based pipeline to add open-vocabulary spatial edges to 3DSGs from images captured while mapping. Finally, we evaluate two LLMs across 14 scenes using 905 natural language statements (786 procedurally-generated, 119 human-authored) to assess performance on the downstream task of target object grounding. Our study demonstrates that explicit spatial relations improve the ability of LLMs to ground objects, and while open-vocabulary relation generation with vision language models proves feasible from robot-captured images, our analysis did not yield evidence favoring either open- or closed-vocabulary relations.

Figures

Figures reproduced from arXiv: 2602.04635 by Francesco Verdoja, Julia Kuhn, Tsvetomila Mihaylova, Ville Kyrki.

Figure 1
Figure 1. Figure 1: Based on images captured while mapping, spatial [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the LLM-based pipeline for object [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example image used for edge generation, where the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the VLM-based pipeline for relation [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example image used for edge generation, where the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 1 linked inside Pith

  1. [1]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” inProceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, ...

    2022

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances,

    B. Ichteret al., “Do as i can, not as i say: Grounding language in robotic affordances,” inProceedings of The 6th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, Dec. 2023, pp. 287–318. [Online]. Available: https://proceedings.mlr.press/v205/ichter23a.html

    2023

  3. [3]

    Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimiza- tion,

    N. Hughes, Y . Chang, and L. Carlone, “Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimiza- tion,” inProceedings of Robotics: Science and Systems, New York City, NY , USA, Jun. 2022

    2022

  4. [4]

    REACT: Real-time Efficient Attribute Clustering and Transfer for Updatable 3D Scene Graph,

    P. Nguyen, F. Verdoja, and V . Kyrki, “REACT: Real-time Efficient Attribute Clustering and Transfer for Updatable 3D Scene Graph,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Oct. 2025

    2025

  5. [5]

    VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation,

    H. Zhang, N. Zantout, P. Kachana, Z. Wu, J. Zhang, and W. Wang, “VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation,” in2024 Robotics Science and Systems conference (RSS), Jul. 2024. [Online]. Available: https://semrob.github.io/docs/ rss semrob2024 cr paper12.pdf

    2024

  6. [6]

    Beyond bare queries: Open-vocabulary object ground- ing with 3d scene graph,

    S. Linoket al., “Beyond bare queries: Open-vocabulary object ground- ing with 3d scene graph,” in2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 13 582–13 589

    2025

  7. [7]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,

    Q. Guet al., “Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,” in2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 5021–5028

    2024

  8. [8]

    Fungraph: Functionality aware 3d scene graphs for language-prompted scene interaction,

    D. Rotondi, F. Scaparro, H. Blum, and K. O. Arras, “Fungraph: Functionality aware 3d scene graphs for language-prompted scene interaction,” 2025. [Online]. Available: https://arxiv.org/abs/2503. 07909

    2025

  9. [9]

    Gaussiangraph: 3d gaussian-based scene graph generation for open-world scene understanding,

    X. Wang, D. Yang, Y . Gao, Y . Yue, Y . Yang, and M. Fu, “Gaussiangraph: 3d gaussian-based scene graph generation for open-world scene understanding,” 2025. [Online]. Available: https: //arxiv.org/abs/2503.04034

    Pith/arXiv arXiv 2025

  10. [10]

    [Online]

    OpenAI, “Models,” 2025, accessed: 05-08-2025. [Online]. Available: https://platform.openai.com/docs/models

    2025

  11. [11]

    High resolution maps from wide an- gle sonar,

    H. Moravec and A. Elfes, “High resolution maps from wide an- gle sonar,” inProceedings. 1985 IEEE International Conference on Robotics and Automation, vol. 2, 1985, pp. 116–121

    1985

  12. [12]

    Semantic information for robot navigation: A survey,

    J. Crespo, J. C. Castillo, O. M. Mozos, and R. Barber, “Semantic information for robot navigation: A survey,”Applied Sciences, vol. 10, no. 2, 2020. [Online]. Available: https://www.mdpi.com/2076-3417/ 10/2/497

    2020

  13. [13]

    Visual relationship detection: A survey,

    J. Chenget al., “Visual relationship detection: A survey,”IEEE Transactions on Cybernetics, vol. 52, no. 8, pp. 8453–8466, 2022

    2022

  14. [14]

    Note on the sampling error of the difference between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,”Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  15. [15]

    ARKitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data,

    G. Baruchet al., “ARKitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. [Online]. Available: https://openreview.net/forum?id=tjZjv qh CE

    2021

  16. [16]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai,

    S. K. Ramakrishnanet al., “Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai,” inProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung, Eds., vol. 1, 2021. [Online]. Available: https://datasets-benchmarks-proceedings.neurips.cc/paper files/paper/ 2021/file...

    2021

  17. [17]

    Matterport3d: Learning from rgb-d data in indoor environments,

    A. Changet al., “Matterport3d: Learning from rgb-d data in indoor environments,” in2017 International Conference on 3D Vision (3DV), 2017, pp. 667–676

    2017

  18. [18]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2432–2443

    2017

  19. [19]

    Rio: 3d object instance re-localization in changing indoor environments,

    J. Wald, A. Avetisyan, N. Navab, F. Tombari, and M. Niessner, “Rio: 3d object instance re-localization in changing indoor environments,” in2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 7657–7666

    2019

  20. [20]

    [Online]

    Unity Technologies, “Unity,” 2025, accessed: 2 June 2025. [Online]. Available: https://unity.com/

    2025

  21. [21]

    VLA-3D Dataset (GitHub Repository),

    H. Zhang, N. Zantout, P. Kachana, Z. Wu, J. Zhang, and W. Wang, “VLA-3D Dataset (GitHub Repository),” 2024, accessed: 2 June

    2024

  22. [22]

    The design of stretch: A compact, lightweight mobile manipulator for indoor human environments,

    C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich, “The design of stretch: A compact, lightweight mobile manipulator for indoor human environments,” in2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 3150–3157

    2022

  23. [23]

    Microsoft coco: Common objects in context,

    T.-Y . Linet al., “Microsoft coco: Common objects in context,” in Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 740–755

    2014

  24. [24]

    Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution V olumetric Mapping and Long- term Dynamic Scene Consistency,

    L. Schmidet al., “Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution V olumetric Mapping and Long- term Dynamic Scene Consistency,” in2022 International Conference on Robotics and Automation (ICRA). IEEE, May 2022, p. 8018–8024. [Online]. Available: http://dx.doi.org/10.1109/ ICRA46639.2022.9811877

    arXiv 2022

  25. [25]

    Ultralytics yolo11,

    G. Jocher and J. Qiu, “Ultralytics yolo11,” 2024. [Online]. Available: https://github.com/ultralytics/ultralytics

    2024

  26. [26]

    Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf, “Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,” inProceedings of The 7th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, J. Tan, M. Toussaint, and K. Darvish, Eds., vol. 229. PMLR, Nov. 2023, pp. 23–72...

    2023

  27. [27]

    Retrieval-augmented generation for knowledge- intensive nlp tasks,

    P. Lewiset al., “Retrieval-augmented generation for knowledge- intensive nlp tasks,” inProceedings of the 34th International Con- ference on Neural Information Processing Systems, ser. NIPS ’20. Curran Associates Inc., 2020

    2020

  28. [28]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Weiet al., “Chain-of-thought prompting elicits reasoning in large language models,” inProceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Curran Associates Inc., 2022

    2022

  29. [29]

    Grammarly,

    Grammarly, Inc., “Grammarly,” 2025. [Online]. Available: https: //www.grammarly.com

    2025

  30. [2025]

    Available: https://github.com/HaochenZ11/VLA-3D

    [Online]. Available: https://github.com/HaochenZ11/VLA-3D