Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

ROOT: VLM based System for Indoor Scene Understanding and Beyond

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a vision-language model fine-tuned on over 610,000 indoor images can output structured scene graphs and metric distances from a single RGB image, with relation precision and recall near 90 percent.

desk verdict Useful system paper with a large new dataset, but the metric-distance claim rests on an uncalibrated monocular depth pipeline. read the letter →

arxiv 2411.15714 v1 pith:RL3YDK4V submitted 2024-11-24 cs.CV

Yonghui Wang , Shi-Yong Chen , Zhenxing Zhou , Siyi Li , Haoran Li , Wengang Zhou , Houqiang Li This is my paper · ORCID
classification cs.CV
keywords indoorsceneunderstandingvisionlanguagemodelgraphgenerationspatialreasoningdistanceestimationRGBimagesemi-automateddata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a vision-language model fine-tuned on a large synthetic indoor-scene QA dataset can read a single RGB image of a room and output a structured scene graph, a JSON tree of support, contain, hang, and attach relations rooted at the floor, wall, and ceiling, together with metric distances between objects. The authors argue that current VLMs fail at this spatial hierarchical reasoning, and that their ROOT pipeline, which uses GPT-4V for object discovery and existing foundation models for depth and masks, supplies the data needed to train SceneVLM. If correct, a generic 7B-parameter VLM could provide the spatial understanding that downstream applications such as 3D scene generation and embodied robots currently lack. The paper reports relationship precision and recall around 90 percent and 74.3 percent distance accuracy within the strict 80-120 percent error band on its 740-image test set, compared with single-digit accuracy for most general-purpose VLMs.

What carries the argument

The load-bearing object is the SceneVQA dataset and the semi-automated pipeline that builds it. GraphVQA relations are first manually annotated in JSON, then re-expressed by GPT-4 as chain-of-thought natural language before being paired with the JSON for training; DistanceVQA uses monocular depth from DepthAnything plus SAM masks and centroid distances in the unprojected 3D point cloud as distance labels. SceneVLM is a fine-tuned 7B VLM trained with a JSON-tree output format and chain-of-thought reasoning, learning to bridge directly from RGB appearance to a fixed four-relation hierarchical layout with distances given in meters.

What would settle it

Measure actual distances with a physical rangefinder for a set of indoor scenes, feed the same images to SceneVLM, and compare its meter predictions against the measured values; systematic error correlated with depth would show the model learned the labeling pipeline's bias. A second check is to run the model on images with deliberately unusual spatial layouts, such as objects hanging mid-air, and observe whether graph accuracy collapses.

Watch

Extended reading notes

Core claim

The central claim is that SceneVLM, an InternVL2 or Qwen2-VL fine-tuned on the authors' SceneVQA dataset of over 610,000 images, can generate hierarchical indoor scene graphs and estimate inter-object distances from RGB images alone, without depth sensors or explicit geometry at test time. The paper defines four spatial relationships (support, contain, hang, attach) and reports that the trained model reproduces the full JSON tree with pairwise relation precision of 91.4 percent and recall of 90.4 percent, object-wise precision of 87.7 percent with IoU around 80 percent, and node detection accuracy near 100 percent. For distances, SceneVLM achieves 74.3 percent accuracy within the strict 80-120 percent band and 97.4 percent within the looser 50-200 percent band, in both cases far ahead of general-purpose VLMs. These results are presented as evidence that VLMs can learn implicit spatial layout from RGB-only input once supplied with sufficient semi-automatically generated scene-relationship training data.

Load-bearing premise

The load-bearing premise is that distances computed by unprojecting monocular depth and taking centroid differences, together with masks and bounding boxes from the same foundation-model family, are accurate enough to serve as ground truth for both training and the 740-image test set; if that pipeline is metrically biased, the reported accuracies measure agreement with the pipeline rather than true spatial understanding.

Editorial extensions

If this is right

  • A robot or agent can go from one RGB photo to a JSON scene graph listing which objects support, contain, hang from, or attach to others, without any depth input at test time.
  • The same VLM can answer distance questions in meters, which supports embodied tasks like moving an object to a reasonable location when combined with a general-purpose VLM such as GPT-4V.
  • The retrained SceneLLM requires only an object list, not an image, so users can specify desired furniture and receive a hierarchy that drives 3D scene generation through the Holodeck pipeline.
  • The SceneVQA dataset itself, with over 610,000 images and 320,000 object categories, is a publicly released resource for future spatial-reasoning training and evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not tested in the paper is to compare SceneVLM against physically measured ground truth, for example with a laser rangefinder, to separate true metric understanding from agreement with the DepthAnything-based labeling pipeline.
  • Because the relation set is closed and anchored to the floor, wall, and ceiling, the model's success may reflect learning a constrained grammar; testing with unusual scenes such as floating or oddly stacked objects would show whether the hierarchy generalizes beyond typical room layouts.
  • The semi-automated recipe of manual JSON plus GPT-4 paraphrase into chain-of-thought before fine-tuning could transfer to other structured-output tasks, suggesting the reusable trick is the chain-of-thought-as-formatting step rather than the specific scene graph task.
  • If metric calibration is the main obstacle, combining a small amount of true-depth supervision with the large pipeline-labeled corpus could yield a model with both coverage and metric accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ROOT, a pipeline for indoor scene understanding from RGB images. The pipeline combines iterative GPT-4V-based object perception, GroundingDINO/SAM/DepthAnything for object boxes, masks and depth, and a fine-tuned 7B VLM (SceneVLM) that generates hierarchical scene graphs (support/contain/hang/attach) and metric distances between objects. To train SceneVLM the authors construct a large dataset, SceneVQA, with over 610k images from public indoor datasets, using semi-automated graph annotation and automated distance labels. Experiments on a 740-image test set report relationship precision/recall around 90% and distance accuracy of 74.32% within [80,120]% of ground truth, outperforming several open VLMs and GPT-4V. The paper also shows applications to embodied AI and Holodeck-based 3D scene generation.

Significance. If the reported results are sound, the paper demonstrates a practical route for a 7B VLM to produce structured, hierarchical scene descriptions with distance estimates, which would be valuable for embodied AI, scene synthesis, and indoor scene understanding. The authors contribute a large-scale scene graph/distance dataset, a modular pipeline, and extensive comparison with existing VLMs, and the supplementary material is unusually detailed, including prompts, algorithms, and error analyses. However, the central quantitative claim of metric distance estimation depends on an unvalidated, self-referential evaluation: distance ground truth is derived from an uncalibrated monocular depth model, and the test set is generated by the same pipeline family used for training. The significance of the distance results therefore rests on whether the model learns true metric depth or only reproduces the bias of the labeling pipeline.

major comments (4)
  1. [Section 3.2 and Supplementary D] The distance ground truth is not metric-calibrated. The paper states that DepthAnything is used to extract depth and generate a 3D point cloud, and object distances are computed as centroid distances of the corresponding point clouds. As released, DepthAnything predicts depth up to an unknown per-image scale and shift; the paper gives no calibration procedure using camera intrinsics, known object sizes, or RGB-D sensors. Consequently the 'ground truth' distances in DistanceVQA and in the 740-image test set (Supplementary C.3) are defined only up to an unknown scale factor. The [80,120]% accuracy reported in Table 3 and the absolute error analysis in Supplementary D therefore largely measure agreement with the same uncalibrated pipeline that generated the training labels, rather than true metric understanding. The authors should either provide a calibration step, compare against datasets with real metric ground truth, or clearly reframe the task as relative-depth-based distance estimation.
  2. [Supplementary C.3 and Section 3.3] The evaluation is self-referential. The test set of 740 images is drawn from the same five public datasets and labeled by the same automated/semi-automated pipeline (GPT-4V object detection, GroundingDINO boxes, SAM masks, DepthAnything distances) that produced the SceneVQA training data. The graph evaluation also uses the exact closed-vocabulary prompt and JSON format seen during training. High SceneVLM accuracy may thus reflect fitting to the label-generating process rather than generalizable scene understanding. The paper should add an independent test set with human-verified graph labels and externally validated metric distances, or at minimum report how much of the test data was manually corrected.
  3. [Table 1 and Section 4.2] The near-perfect Node Detection Accuracy (99.97 precision) is partly explained by the evaluation protocol itself: the object list is provided in the prompt, and the authors acknowledge that 'given that the list of objects is provided, generating object outputs is relatively straightforward.' This makes NDA a measure of output formatting obedience rather than object discovery. The pairwise and object-wise relation metrics inherit this advantage because the candidate object set is fixed. The comparison with baselines is still useful, but the claim in Section 4.2 that SceneVLM 'outperforms existing VLMs across all metrics' should be qualified, since the baselines may have no such list or may face a harder 'detect and relate' task.
  4. [Table 5] Several ablation conclusions are based on very small differences without error bars or repeated runs. For example, unfreezing the ViT is reported as a 1.2-point drop in Pairwise Relation Precision and a 3.5-point gain in distance [80,120], and w/o JSON is a 0.5-point drop; at this magnitude, run-to-run variance or test-set composition could change the conclusion. Reporting standard deviations across at least three seeds, or a significance test, would make the ablation claims and the 'Larger VLM' improvements (1.7-8.0 points) interpretable.
minor comments (5)
  1. [Table 5] The table header contains the typo "Unforzen ViT" and the label "Unforzen ViT" in the first row; it should read "Unfrozen ViT".
  2. [Table 1 caption] The caption writes "The best and the second results are highlighted in blod and underlined" — "blod" should be "bold".
  3. [Section 5.2] The model "SceneLLM" is introduced abruptly: the text says it is "retrained" from SceneVLM and used with Holodeck, but the paper does not describe how SceneLLM differs from SceneVLM, what data it is trained on, or why this separate model is needed. A short paragraph with training details would make the application reproducible.
  4. [Algorithm 1] The notation in Algorithm 1 is clear overall, but the variable name "unforzen" (appearing in the supplementary) should be corrected to "unfrozen", and the pseudo-code would benefit from a sentence explaining the meaning of the "max p" threshold when multiple bounding boxes are present.
  5. [Section 4.3] The paper states that the [80,120] range mirrors "the typical use of approximate descriptions in daily life", but no citation or user study is given for this claim. A brief justification or removal of the unsupported statement would be appropriate.

Circularity Check

1 steps flagged · score 6.0 of 10

Distance ground truth is generated by the same uncalibrated DepthAnything centroid pipeline used for training labels, so the reported [80,120]% distance accuracy partly measures self-consistency with the labeling pipeline rather than independent metric understanding.

  1. self definitional [Section 3.2 (Indoor Scene Parsing); Table 2; Table 3; Supplementary C.3]
    "the spatial distance between objects is determined by calculating the centroid distances of their respective point clouds. ... In this study, the test dataset consists of 740 images. ... over 20,000 instances of distances in the form [object1, object2]."

    Distance 'ground truth' is defined by the paper's own pipeline: DepthAnything produces a 3D point cloud and object distances are centroid distances of these point clouds. DepthAnything supplies affine-invariant relative depth rather than metric depth, and the paper does not describe any metric-scale calibration for the mixed public datasets. Table 2 shows the 740-image test set is a subset of the same SceneVQA dataset, so its distance labels are generated by the same DepthAnything-centroid function used to create the DistanceVQA training labels. SceneVLM is trained to output those quantities and then evaluated against those same quantities; the Table 3 [80,120]% accuracy therefore largely measures agreement with the labeling pipeline, not independently calibrated metric distance.

full rationale

The paper contains no theorem derivation and no load-bearing self-citation chain: SpatialVLM is cited only for evaluation conventions and data-filtering prompts, and GPT-4V, GroundingDINO, SAM, and DepthAnything are external tools. The scene-graph claim rests on semi-automated human annotation and is compared against strong baselines, so it has substantial independent content. The main circularity is confined to the distance-estimation claim: both training labels and test labels are defined by the same uncalibrated DepthAnything centroid-distance pipeline, so the reported metric-distance accuracy reduces, by construction, to fitting and reproducing the authors' own labeling convention. This warrants a partial-circularity score of 6 rather than a lower score; the relationship and downstream-application results keep the paper from being fully self-referential.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The system is an empirical pipeline, not a derivation. It depends on hand-chosen thresholds, monocular depth reliability, GPT-4V list quality, SAM mask quality, and the exhaustiveness of four relation types. Because the test set is built by the same pipeline, the reported numbers inherit these assumptions.

free parameters (3)
  • pm = 0.3
    GroundingDINO bounding box probability threshold selected by hand in Algorithm 1; determines which detections survive and affects the final object list.
  • pn = 0.15
    Score difference threshold for choosing between the top two bounding boxes in Algorithm 1; chosen by hand.
  • S = 1.5
    Scaling factor for cropping container regions during iterative object perception in Algorithm 1; chosen by hand.
assumptions (4)
  • domain assumption Monocular DepthAnything depth estimates yield metrically correct 3D point clouds, so centroid distances are true physical distances.
    Used to generate all DistanceVQA targets and test labels in Section 3.2. If the depth scale is wrong, reported distance accuracy is agreement with the pipeline's proxy, not reality.
  • domain assumption The four relations support, contain, hang, and attach exhaustively describe indoor object hierarchy.
    Prompts and the JSON schema restrict outputs to these relations in Section 3.3 and the supplementary prompts. Other spatial relations are ignored, so accuracy is bounded by this closed set.
  • domain assumption GPT-4V object lists and container flags are accurate and complete.
    All downstream detection, masking, graph, and distance computations depend on the initial GPT-4V object list from Section 3.1; missed objects are absent from evaluation.
  • domain assumption Human annotation of GraphVQA relationships is unbiased ground truth.
    Section 3.3 describes manual annotation and review, but reports no inter-annotator agreement or quality study.
invented entities (1)
  • SceneLLM
    purpose: Generates hierarchical room layouts for 3D scene generation in Holodeck by converting a user-specified object list into relationship constraints.
    Mentioned only in Section 5.2 as a retrained variant of SceneVLM; no architecture, training data, or quantitative evaluation is provided, so it is a placeholder concept rather than a demonstrated model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ROOT: VLM based System for Indoor Scene Understanding and Beyond." pith.science (2026). https://pith.science/paper/RL3YDK4V

@misc{pith2026241115714,
  author       = {Pith},
  title        = {Pith review of: ROOT: VLM based System for Indoor Scene Understanding and Beyond},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RL3YDK4V}},
  note         = {Machine review of arXiv:2411.15714}
}
read the original abstract

Recently, Vision Language Models (VLMs) have experienced significant advancements, yet these models still face challenges in spatial hierarchical reasoning within indoor scenes. In this study, we introduce ROOT, a VLM-based system designed to enhance the analysis of indoor scenes. Specifically, we first develop an iterative object perception algorithm using GPT-4V to detect object entities within indoor scenes. This is followed by employing vision foundation models to acquire additional meta-information about the scene, such as bounding boxes. Building on this foundational data, we propose a specialized VLM, SceneVLM, which is capable of generating spatial hierarchical scene graphs and providing distance information for objects within indoor environments. This information enhances our understanding of the spatial arrangement of indoor scenes. To train our SceneVLM, we collect over 610,000 images from various public indoor datasets and implement a scene data generation pipeline with a semi-automated technique to establish relationships and estimate distances among indoor objects. By utilizing this enriched data, we conduct various training recipes and finish SceneVLM. Our experiments demonstrate that \rootname facilitates indoor scene understanding and proves effective in diverse downstream applications, such as 3D scene generation and embodied AI. The code will be released at \url{https://github.com/harrytea/ROOT}.

Figures

Figures reproduced from arXiv: 2411.15714 by the authors.

Figure 1
Figure 1. ROOT is a system designed to interpret indoor scene images and extract various types of meta-information about the scenes. Utilizing this information, ROOT can generate hierarchical relationships and spatial distances among indoor objects. This enriched data serves to support various downstream tasks. encompasses a myriad of information, such as the entities within a room, their positions, and their interrelationshi… view at source ↗
Figure 2
Figure 2. We introduce ROOT, a system designed for understanding indoor scenes. Initially, the system utilizes an iterative object perception module based on GPT-4V to identify entities within a given image. Subsequently, the indoor scene and objects are parsed using existing vision foundation models to gather meta-information about the scene. Finally, the object information is processed by SceneVLM, resulting in a scene grap… view at source ↗
Figure 3
Figure 3. Four types of hierarchical relationships as defined. In [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: SceneVQA data generation pipeline. This diagram de [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Hierarchical scene graph visualization of our method. Each object is assigned a serial number, with the corresponding visual JSON [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: ROOT and GPT-4V integration demonstrate a promising application in indoor environments. In an office setting, this system can assist a robot in accurately identifying and manipulating objects. By leveraging ROOT’s analysis, GPT-4V identifies potential incon￾sistencies,…
Figure 7
Figure 7. Figure 7: ROOT’s newly developed SceneLLM model, integrated with Holodeck [53], effectively constructs indoor scenes from spec￾ified object lists. We show several toy examples. For example, SceneLLM can define a structured relationship such as [cabinet, support, sink], allowing …
Figure 8
Figure 8. Figure 8: The workflow for visualizing the iterative perception of objects. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The process of indoor scene parsing. • A screenshot of graphics user interface. • A piece of text. • A sketch. • A cartoon. This approach ensures that the data used in our study is highly relevant and closely aligned with the specific require￾ments of our research on i…
Figure 10
Figure 10. Figure 10: Statistical distribution of room types in our SceneVQA dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: An example of the JSON file representing hierarchical [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Analysis of distance estimation errors: both absolute [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: More results on Holodeck [53]. The integration of ROOT and Holodeck [53] enhances functionality, enabling users to specify desired objects. Consequently, this integration facilitates the automation of indoor layout and arrangement processes. 5 [PITH_FULL_IMAGE:figure…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Evidence-Driven Reasoning for Long Document Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A hierarchical multimodal RAG pipeline with GRPO-trained multi-page evidence verification and memory-guided iteration improves long-document QA accuracy by ~8% over prior open-source baselines.

  2. Generative Physical AI in Vision: A Survey

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A structured review that categorizes physics-aware generative models in vision into explicit-simulation and implicit-learning families and proposes six integration paradigms.

  3. DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A pipeline that generates editable 3D scenes from natural language by combining LLM-based layout planning, multi-timestep diffusion distillation, and staged camera sampling.

Reference graph

Works this paper leans on

88 extracted references · 62 canonical work pages · cited by 3 Pith papers

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 4

    arXiv 2023

  2. [2]

    Qwen-VL: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 ,

    arXiv

  3. [3]

    Towards in-context scene understanding

    Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelovi´c, and Olivier Henaff. Towards in-context scene understanding. In NIPS, pages 63758–63778, 2024. 1

    2024

  4. [4]

    MAPLM: A real-world large-scale vision-language benchmark for map and traffic scene under- standing

    Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M Rehg, et al. MAPLM: A real-world large-scale vision-language benchmark for map and traffic scene under- standing. In CVPR, pages 21819–21830, 2024. 1

    2024

  5. [5]

    SpatialVLM: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endow- ing vision-language models with spatial reasoning capabili- ties. In CVPR, pages 14455–14465, 2024. 2, 6, 1

    2024

  6. [6]

    Poly- Diffuse: Polygonal shape reconstruction via guided set diffu- sion models

    Jiacheng Chen, Ruizhi Deng, and Yasutaka Furukawa. Poly- Diffuse: Polygonal shape reconstruction via guided set diffu- sion models. In NIPS, pages 1863–1888, 2024. 2

    2024

  7. [7]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 1, 5, 7

    2024

  8. [8]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213– 3223, 2016. 1

    2016

Show all 88 references
  1. [9]

    InstructBLIP: towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: towards general-purpose vision-language models with instruction tuning. In NIPS, pages 49250–49267, 2023. 5, 6

    2023

  2. [10]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, pages 13142–13153, 2023. 8, 4

    2023

  3. [11]

    PLA: Language-driven open- vocabulary 3d scene understanding

    Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. PLA: Language-driven open- vocabulary 3d scene understanding. In CVPR, pages 7010– 7019, 2023. 2

    2023

  4. [12]

    Shape anchor guided holistic indoor scene understanding

    Mingyue Dong, Linxi Huan, Hanjiang Xiong, Shuhan Shen, and Xianwei Zheng. Shape anchor guided holistic indoor scene understanding. In ICCV, pages 21916–21926, 2023. 2

    2023

  5. [13]

    The Llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  6. [14]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023. 2, 4, 1

    2023 arXiv

  7. [15]

    3D-FUTURE: 3d furniture shape with texture

    Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3D-FUTURE: 3d furniture shape with texture. IJCV, 129:3313–3337, 2021. 4, 1

    2021

  8. [16]

    Dual attention network for scene segmentation

    Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, pages 3146–3154, 2019. 2

    2019

  9. [17]

    Scene-LLM: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-LLM: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. 1

    2024 arXiv

  10. [18]

    ChatGLM: A family of large language models from glm-130b to glm-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. ChatGLM: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. 5

    2024 arXiv

  11. [19]

    Semantic Abstraction: Open- world 3d scene understanding from 2d vision-language mod- els

    Huy Ha and Shuran Song. Semantic Abstraction: Open- world 3d scene understanding from 2d vision-language mod- els. arXiv preprint arXiv:2207.11514, 2022. 1

    2022 arXiv

  12. [20]

    Scene graph reasoning for visual question answering

    Marcel Hildebrandt, Hang Li, Rajat Koner, V olker Tresp, and Stephan G¨unnemann. Scene graph reasoning for visual question answering. arXiv preprint arXiv:2007.01072, 2020. 3

    2007 arXiv

  13. [21]

    Probabilistic future prediction for video scene understanding

    Anthony Hu, Fergal Cotter, Nikhil Mohan, Corina Gurau, and Alex Kendall. Probabilistic future prediction for video scene understanding. In ECCV, pages 767–785, 2020. 2

    2020

  14. [22]

    Mutual scene synthesis for mixed reality telepresence

    Mohammad Keshavarzi, Michael Zollhoefer, Allen Y Yang, Patrick Peluse, and Luisa Caldas. Mutual scene synthesis for mixed reality telepresence. arXiv preprint arXiv:2204.00161,

    arXiv

  15. [23]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, pages 4015–4026, 2023. 4, 1

    2023

  16. [24]

    Ai2-THOR: An interactive 3d environment for visual ai

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-THOR: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474,

    arXiv

  17. [25]

    TopViewRS: Vision-language models as top-view spatial reasoners

    Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, and Ivan Vuli ´c. TopViewRS: Vision-language models as top-view spatial reasoners. arXiv preprint arXiv:2406.02537, 2024. 2

    2024 arXiv

  18. [26]

    From pixels to graphs: Open-vocabulary scene graph generation with vision-language models

    Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. In CVPR, pages 28076–28086, 2024. 3

    2024

  19. [27]

    Robotic indoor scene captioning from streaming video

    Xinghang Li, Di Guo, Huaping Liu, and Fuchun Sun. Robotic indoor scene captioning from streaming video. InICRA, pages 6109–6115, 2021. 2

    2021

  20. [28]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024. 5

    2024

  21. [29]

    LLaV A-NeXT: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Improved reasoning, ocr, and world knowledge, 2024. 5

    2024

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NIPS, pages 34892–34916, 2024. 1

    2024

  23. [31]

    Grounding DINO: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding DINO: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3, 1

    2023 arXiv

  24. [32]

    GPT-4V(ision) system card

    OpenAI. GPT-4V(ision) system card. arXiv preprint arXiv:2410.21276, 2023. 2

    2023 arXiv

  25. [33]

    OpenScene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasac- chi, Marc Pollefeys, Thomas Funkhouser, et al. OpenScene: 3d scene understanding with open vocabularies. In CVPR, pages 815–824, 2023. 1, 2

    2023

  26. [34]

    Seamless scene segmentation

    Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In CVPR, pages 8277–8286, 2019. 2

    2019

  27. [35]

    Scene graph refinement network for visual question answering

    Tianwen Qian, Jingjing Chen, Shaoxiang Chen, Bo Wu, and Yu-Gang Jiang. Scene graph refinement network for visual question answering. IEEE TMM, 25:3950–3961, 2022. 3

    2022

  28. [36]

    Recognizing indoor scenes

    Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In CVPR, pages 413–420, 2009. 4, 1

    2009

  29. [37]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021. 2

    2021

  30. [38]

    Monocular depth estimation using diffusion models

    Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet. Monocular depth estimation using diffusion models. arXiv preprint arXiv:2302.14816, 2023. 2

    2023 arXiv

  31. [39]

    Structured query- based image retrieval using scene graphs

    Brigit Schroeder and Subarna Tripathi. Structured query- based image retrieval using scene graphs. In CVPR, pages 178–179, 2020. 3

    2020

  32. [40]

    Disentangling orthogonal planes for indoor panoramic room layout estimation with cross-scale distortion awareness

    Zhijie Shen, Zishuo Zheng, Chunyu Lin, Lang Nie, Kang Liao, Shuai Zheng, and Yao Zhao. Disentangling orthogonal planes for indoor panoramic room layout estimation with cross-scale distortion awareness. In CVPR, pages 17337– 17345, 2023. 2

    2023

  33. [41]

    A benchmark for the evaluation of rgb-d slam systems

    J¨urgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In ICIRS, pages 573–580, 2012. 4, 1

    2012

  34. [42]

    Distilled semantics for comprehensive scene under- standing from videos

    Fabio Tosi, Filippo Aleotti, Pierluigi Zama Ramirez, Matteo Poggi, Samuele Salti, Luigi Di Stefano, and Stefano Mat- toccia. Distilled semantics for comprehensive scene under- standing from videos. In CVPR, pages 4654–4665, 2020. 2

    2020

  35. [43]

    No more ambiguity in 360deg room layout via bi-layout estimation

    Yu-Ju Tsai, Jin-Cheng Jhang, Jingjing Zheng, Wei Wang, Albert YC Chen, Min Sun, Cheng-Hao Kuo, and Ming-Hsuan Yang. No more ambiguity in 360deg room layout via bi-layout estimation. In CVPR, pages 28056–28065, 2024. 2

    2024

  36. [44]

    LLaV A-SG: Leveraging scene graphs as visual semantic expression in vision-language models

    Jingyi Wang, Jianzhong Ju, Jian Luan, and Zhidong Deng. LLaV A-SG: Leveraging scene graphs as visual semantic expression in vision-language models. arXiv preprint arXiv:2408.16224, 2024. 3

    2024 arXiv

  37. [45]

    Is a picture worth a thousand words? delving into spatial reasoning for vision language models

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. arXiv preprint arXiv:2406.14852, 2024. 2

    2024 arXiv

  38. [46]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 5

    2024 arXiv

  39. [47]

    EmbodiedScan: A holistic multi- modal 3d perception suite towards embodied ai

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. EmbodiedScan: A holistic multi- modal 3d perception suite towards embodied ai. In CVPR, pages 19757–19767, 2024. 1

    2024

  40. [48]

    SUN Database: Exploring a large collection of scene categories

    Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Tor- ralba, and Aude Oliva. SUN Database: Exploring a large collection of scene categories. IJCV, 119:3–22, 2016. 4, 1

    2016

  41. [49]

    Unified perceptual parsing for scene understanding

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 418–434, 2018. 1

    2018

  42. [50]

    Depth Anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power of large-scale unlabeled data. In CVPR, pages 10371–10381, 2024. 2, 4, 1

    2024

  43. [51]

    Graph-structured referring expression reasoning in the wild

    Sibei Yang, Guanbin Li, and Yizhou Yu. Graph-structured referring expression reasoning in the wild. In CVPR, pages 9952–9961, 2020. 3

    2020

  44. [52]

    PHYSCENE: Physically interactable 3d scene synthesis for embodied ai

    Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. PHYSCENE: Physically interactable 3d scene synthesis for embodied ai. In CVPR, pages 16262–16272, 2024. 1

    2024

  45. [53]

    HOLODECK: Language guided generation of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Kr- ishna, Lingjie Liu, et al. HOLODECK: Language guided generation of 3d embodied ai environments. In CVPR, pages 16227–16237, 2024. 1, 8, 4, 5

    2024

  46. [54]

    Swin3D++: Effec- tive multi-source pretraining for 3d indoor scene understand- ing

    Yu-Qi Yang, Yu-Xiao Guo, and Yang Liu. Swin3D++: Effec- tive multi-source pretraining for 3d indoor scene understand- ing. arXiv preprint arXiv:2402.14215, 2024. 2

    2024 arXiv

  47. [55]

    MiniCPM-V: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. MiniCPM-V: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 5

    2024 arXiv

  48. [56]

    Human-aware object placement for visual environment reconstruction

    Hongwei Yi, Chun-Hao P Huang, Dimitrios Tzionas, Muhammed Kocabas, Mohamed Hassan, Siyu Tang, Justus 10 Thies, and Michael J Black. Human-aware object placement for visual environment reconstruction. In CVPR, pages 3959– 3970, 2022. 1

    2022

  49. [57]

    Context prior for scene segmenta- tion

    Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chun- hua Shen, and Nong Sang. Context prior for scene segmenta- tion. In CVPR, pages 12416–12425, 2020. 2

    2020

  50. [58]

    Agent3D-Zero: An agent for zero-shot 3d understanding

    Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang. Agent3D-Zero: An agent for zero-shot 3d understanding. arXiv preprint arXiv:2403.11835, 2024. 1

    2024 arXiv

  51. [59]

    DeepContext: Context-encoding neural pathways for 3d holistic scene understanding

    Yinda Zhang, Mingru Bai, Pushmeet Kohli, Shahram Izadi, and Jianxiong Xiao. DeepContext: Context-encoding neural pathways for 3d holistic scene understanding. In ICCV, pages 1192–1201, 2017. 1

    2017

  52. [60]

    LUMI- NOUS: Indoor scene generation for embodied ai challenges

    Yizhou Zhao, Kaixiang Lin, Zhiwei Jia, Qiaozi Gao, Govind Thattai, Jesse Thomason, and Gaurav S Sukhatme. LUMI- NOUS: Indoor scene generation for embodied ai challenges. arXiv preprint arXiv:2111.05527, 2021. 1

    2021 arXiv

  53. [61]

    Comprehensive image captioning via scene graph decom- position

    Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph decom- position. In ECCV, pages 211–229, 2020. 3

    2020

  54. [62]

    select prompt

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 40(6):1452–1464, 2017. 4, 1 11 ROOT: VLM based System for Indoor Scene Understanding and Beyond Supplementary Material A. Detail...

    2017

  55. [63]

    For instance, in Figure 11, the relationship [1, support, 4] is considered correct if it is correctly extracted from the JSON file

    Pairwise Relation Accuracy (PRA): PRA assesses the accuracy of the relationships between pairs of objects. For instance, in Figure 11, the relationship [1, support, 4] is considered correct if it is correctly extracted from the JSON file. There are 14 pairwise relationships in...

  56. [64]

    For example, in Figure 11, object 1 has relationships such as [[1, support, 4], [1, support, 5], [1, support, 6], [1, support, 7]]

    Object-wise Relation Accuracy (OW A):OW A evaluates the accuracy of all relationships associated with a specific object, considering its parent and child objects. For example, in Figure 11, object 1 has relationships such as [[1, support, 4], [1, support, 5], [1, support, 6], ...

  57. [65]

    For example, in Figure 11, there are four layers: the first layer includes 1: 1,2,3, the second layer contains 2: 4,5,6,7,8,9,10, and so on

    Layer-wise Accuracy (LW A):LW A measures the accu- racy of object predictions at each layer level. For example, in Figure 11, there are four layers: the first layer includes 1: 1,2,3, the second layer contains 2: 4,5,6,7,8,9,10, and so on. Accuracy is achieved when both the la...

  58. [66]

    In Figure 11, if an object, such as 1, appears in the JSON, it is considered as accurate

    Node Detection Accuracy (NDA): NDA measures the accuracy of identifying individual objects. In Figure 11, if an object, such as 1, appears in the JSON, it is considered as accurate. The figure contains a total of 17 objects. For these metrics, we utilize precision, recall, F1-...

  59. [67]

    This metric quantifies the accuracy of the positive predictions made by the model

    Precision. This metric quantifies the accuracy of the positive predictions made by the model. It is defined as the ratio of TP to the sum of TP and FP: Precision = T P T P+ F P= 3 3 + 2= 3 5 = 0.6 (1)

  60. [68]

    This metric assesses the model’s ability to identify all relevant instances

    Recall. This metric assesses the model’s ability to identify all relevant instances. It is defined as the ratio of TP to the sum of TP and FN: Recall = 3 3 + 1= 3 4 = 0.75 (2)

  61. [69]

    This metric is the harmonic mean of Precision and Recall, providing a balanced measure of both metrics

    F1 Score. This metric is the harmonic mean of Precision and Recall, providing a balanced measure of both metrics. It is computed as: F 1 = 2× Precision × Recall Precision + Recall = 2× 0.6 × 0.75 0.6 + 0.75 = 0.67 (3)

  62. [70]

    object” with a numerical suffix starting from 1. The value of each “object

    Intersection over Union (IoU). This metric evaluates the overlap between the predicted and ground truth sets. It is defined as the ratio of the area of overlap (TP) between the predicted and ground truth sets to the area of their union (TP + FP + FN): IoU = T P T P+ F P+ F N= ...

  63. [72]

    container

    For the “container” key, its value should be “True” if the object is containing or supporting other objects, and “False” otherwise

  64. [73]

    Please consider a desk and its tablecloth as one object

    The possible container that could only be a desk, shelf, bed or other similar items. Please consider a desk and its tablecloth as one object

  65. [74]

    Do not miss any suitable object

  66. [75]

    object1”: {“description

    Ensure that your output can be parsed by python’s json.loads() directly. Following is an example: {“object1”: {“description”: “trash bin with liner”, “container”: “False” }, “object2”: {“description”: “retangular dinner table with tablecloths”, “container”: “True” }, “object3”...

  67. [76]

    Each entry should uniquely describe one element without repeating values

  68. [77]

    {container}

    Only describe the objects that is on or inside the “{container}”. Please ignore other parts of the image

  69. [78]

    {container}

    Do not miss any small object that is on or inside the “{container}”

  70. [79]

    {container}

    Do not include the objects that are near, under or behind the “{container}”. If there is no suitable object, please return -1

  71. [80]

    {container}

    Do not include the “{container}” in your output

  72. [81]

    Ensure that the described objects are suitable for measuring distances between them and exclude elements like walls or floors

  73. [82]

    object1”: {“description

    Make sure that your output can be parsed by python’s json.loads() directly. Following is an example: {“object1”: {“description”: “rectangular silver tray”}, “object2”: { “description”: “bottle of wine on table”}, “object3”: {“description”: “round decorative doily”}} 6 Ojbect P...

  74. [83]

    left”, “center/middle

    You must select the most appropriate bounding box and object based on orientation words within the description, such as “left”, “center/middle” or “right”. For instance, if an image contains three side-by-side computers, and the description states “center computer”, you should...

  75. [84]

    In this situation, you still need to select the the suitable bounding box based on the relative position of these three objects

    It is possible that there are three similar objects (left, center and right respectively) in the image while only two of thems are enclosed by bounding boxes. In this situation, you still need to select the the suitable bounding box based on the relative position of these thre...

  76. [85]

    reason” and “color

    Please provide an output in JSON format with the keys “reason” and “color”. In the “reason” value, explain the rationale behind your selection, and in the “color” value, return the color of your chosen bounding box

  77. [86]

    If none of the bounding box meets the description, you should select one randomly

    If there is no orientation word, you should select the bounding box that best corresponds to the given description. If none of the bounding box meets the description, you should select one randomly

  78. [87]

    You can only select one box and the “color” value can only be one of the element from this color list: {colors}

  79. [88]

    You should select the bounding box and its corresponding color according to the description

    The order of the color list is meaningless. You should select the bounding box and its corresponding color according to the description

  80. [89]

    {description}

    Make sure that your output can be parsed by python’s json.loads() directly. Following is the provided description: “{description}” 7 An toy example of an answer from GraphVQA: CoT and JSON The art frame is hanging on the wall. The bookshelf 0, desk, and chair are supported by ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.