{"id":"87577749-368e-47ac-b82a-f26ad8bf87d5","arxiv_id":"2411.15714","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ROOT combines GPT-4V, GroundingDINO, SAM, and DepthAnything with a fine-tuned SceneVLM to produce hierarchical indoor scene graphs and object distance estimates from a single RGB image.","lead":"ROOT is a system that uses vision-language models and off-the-shelf tools to identify objects in room photos, build a hierarchy of how objects relate, and estimate distances between them. It also introduces a large training dataset and shows possible applications for robots and 3D scene generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Distance ground truth is computed from DepthAnything monocular depth with no stated metric-scale calibration; the reported [80,120]% distance accuracy may measure self-consistency with an unscaled depth model rather than true metric understanding.","rationale":"The reader's weakest_assumption correctly flags that ground truth is generated by the same pipeline family, but it does not isolate the sharpest technical defect: DepthAnything's output is affine-invariant and does not provide metric scale, so 'centroid distances' are not automatically meters. This is a concrete correctness risk in the paper's own Section 3.2, not merely a concern about benchmark independence. The proposed RGB-D test would settle whether the scale issue actually changes the reported numbers, or whether the pipeline happens to produce consistent metric-scale estimates through undocumented cues. I do not recommend switching to REJECT because the system could still be useful as a relative-layout method and the underlying VLM fine-tuning idea is plausible; the claim requiring external validation is specifically the metric distance accuracy. Therefore the reader's CONDITIONAL verdict remains appropriate, now with a sharper condition: demonstrate metric-scale validity of the distance labels on sensor depth data, or retract the absolute-meter accuracy claim. Agreement is partial because the reader's stated concern is broader and less pointed than the metric-scale defect, though directionally aligned.","tokens_in":24963,"tokens_out":3288,"duration_ms":31359,"concrete_test":"Take 100-200 RGB-D indoor images from a sensor-based benchmark not used for training (e.g., ARKitScenes or ScanNet with calibrated depth). Run the ROOT pipeline exactly as described (DepthAnything + SAM + centroid distances) to generate pipeline 'ground truth' distances, and compare them directly to the sensor depth-derived ground truth in meters. Report median absolute relative error and the fraction of labels falling within the [80,120]% band used in Table 3. If the pipeline labels deviate from sensor distances by more than a tolerated amount (e.g., median relative error > 30%) or the band accuracy is far below Table 3, then the distance claim is an artifact of uncalibrated monocular depth.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim that SceneVLM estimates metric distances is undercut by how the distance ground truth is constructed. In Section 3.2, the paper states that DepthAnything processes the RGB image to extract depth and generate a 3D point cloud, and that object distances are then taken as centroid distances of those point clouds. DepthAnything (as released and used here) outputs affine-invariant relative depth, not metric depth: its scale and shift are unknown per image. To obtain distances in meters, one must calibrate the depth scale (e.g., using known camera intrinsics, known object sizes, or RGB-D sensor depth). The paper does not describe any such calibration, nor does it report camera intrinsics for the mixed public datasets (3D-Future, TUM, SUN, MIT Indoor, Places). Consequently, the 'ground truth' distances in the DistanceVQA dataset and in the 740-image test set (Supplementary C.3) are, at best, defined only up to an unknown per-image scale factor.\n\nThis is not just a minor calibration detail: Table 3 reports 74.32% accuracy within [80,120]% of ground truth, and Section D analyzes absolute and relative errors in meters. If the labels carry an arbitrary scale, the model cannot learn true metric distances from them, and the reported numbers largely measure agreement with the same uncalibrated pipeline family used at train and test time. The relationship accuracy may have a related but milder exposure: the scene graph labels are semi-automated and internally generated, so the 90% precision/recall also faces a train/test distribution overlap issue, but the distance metric-scale problem is the sharpest single failure mode for the paper's most distinctive claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ROOT, a pipeline for indoor scene understanding from RGB images. The pipeline combines iterative GPT-4V-based object perception, GroundingDINO/SAM/DepthAnything for object boxes, masks and depth, and a fine-tuned 7B VLM (SceneVLM) that generates hierarchical scene graphs (support/contain/hang/attach) and metric distances between objects. To train SceneVLM the authors construct a large dataset, SceneVQA, with over 610k images from public indoor datasets, using semi-automated graph annotation and automated distance labels. Experiments on a 740-image test set report relationship precision/recall around 90% and distance accuracy of 74.32% within [80,120]% of ground truth, outperforming several open VLMs and GPT-4V. The paper also shows applications to embodied AI and Holodeck-based 3D scene generation.","tokens_in":25238,"tokens_out":3047,"duration_ms":30552,"significance":"If the reported results are sound, the paper demonstrates a practical route for a 7B VLM to produce structured, hierarchical scene descriptions with distance estimates, which would be valuable for embodied AI, scene synthesis, and indoor scene understanding. The authors contribute a large-scale scene graph/distance dataset, a modular pipeline, and extensive comparison with existing VLMs, and the supplementary material is unusually detailed, including prompts, algorithms, and error analyses. However, the central quantitative claim of metric distance estimation depends on an unvalidated, self-referential evaluation: distance ground truth is derived from an uncalibrated monocular depth model, and the test set is generated by the same pipeline family used for training. The significance of the distance results therefore rests on whether the model learns true metric depth or only reproduces the bias of the labeling pipeline.","major_comments":[{"comment":"The distance ground truth is not metric-calibrated. The paper states that DepthAnything is used to extract depth and generate a 3D point cloud, and object distances are computed as centroid distances of the corresponding point clouds. As released, DepthAnything predicts depth up to an unknown per-image scale and shift; the paper gives no calibration procedure using camera intrinsics, known object sizes, or RGB-D sensors. Consequently the 'ground truth' distances in DistanceVQA and in the 740-image test set (Supplementary C.3) are defined only up to an unknown scale factor. The [80,120]% accuracy reported in Table 3 and the absolute error analysis in Supplementary D therefore largely measure agreement with the same uncalibrated pipeline that generated the training labels, rather than true metric understanding. The authors should either provide a calibration step, compare against datasets with real metric ground truth, or clearly reframe the task as relative-depth-based distance estimation.","section":"Section 3.2 and Supplementary D"},{"comment":"The evaluation is self-referential. The test set of 740 images is drawn from the same five public datasets and labeled by the same automated/semi-automated pipeline (GPT-4V object detection, GroundingDINO boxes, SAM masks, DepthAnything distances) that produced the SceneVQA training data. The graph evaluation also uses the exact closed-vocabulary prompt and JSON format seen during training. High SceneVLM accuracy may thus reflect fitting to the label-generating process rather than generalizable scene understanding. The paper should add an independent test set with human-verified graph labels and externally validated metric distances, or at minimum report how much of the test data was manually corrected.","section":"Supplementary C.3 and Section 3.3"},{"comment":"The near-perfect Node Detection Accuracy (99.97 precision) is partly explained by the evaluation protocol itself: the object list is provided in the prompt, and the authors acknowledge that 'given that the list of objects is provided, generating object outputs is relatively straightforward.' This makes NDA a measure of output formatting obedience rather than object discovery. The pairwise and object-wise relation metrics inherit this advantage because the candidate object set is fixed. The comparison with baselines is still useful, but the claim in Section 4.2 that SceneVLM 'outperforms existing VLMs across all metrics' should be qualified, since the baselines may have no such list or may face a harder 'detect and relate' task.","section":"Table 1 and Section 4.2"},{"comment":"Several ablation conclusions are based on very small differences without error bars or repeated runs. For example, unfreezing the ViT is reported as a 1.2-point drop in Pairwise Relation Precision and a 3.5-point gain in distance [80,120], and w/o JSON is a 0.5-point drop; at this magnitude, run-to-run variance or test-set composition could change the conclusion. Reporting standard deviations across at least three seeds, or a significance test, would make the ablation claims and the 'Larger VLM' improvements (1.7-8.0 points) interpretable.","section":"Table 5"}],"minor_comments":[{"comment":"The table header contains the typo \"Unforzen ViT\" and the label \"Unforzen ViT\" in the first row; it should read \"Unfrozen ViT\".","section":"Table 5"},{"comment":"The caption writes \"The best and the second results are highlighted in blod and underlined\" — \"blod\" should be \"bold\".","section":"Table 1 caption"},{"comment":"The model \"SceneLLM\" is introduced abruptly: the text says it is \"retrained\" from SceneVLM and used with Holodeck, but the paper does not describe how SceneLLM differs from SceneVLM, what data it is trained on, or why this separate model is needed. A short paragraph with training details would make the application reproducible.","section":"Section 5.2"},{"comment":"The notation in Algorithm 1 is clear overall, but the variable name \"unforzen\" (appearing in the supplementary) should be corrected to \"unfrozen\", and the pseudo-code would benefit from a sentence explaining the meaning of the \"max p\" threshold when multiple bounding boxes are present.","section":"Algorithm 1"},{"comment":"The paper states that the [80,120] range mirrors \"the typical use of approximate descriptions in daily life\", but no citation or user study is given for this claim. A brief justification or removal of the unsupported statement would be appropriate.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv preprint and appears not to have been thoroughly copy-edited. The main technical concern is real: the distance task is evaluated against labels derived from uncalibrated monocular depth, and the test set shares the same generation pipeline as the training set. This is fixable within the scope of the paper by adding calibration, an independently validated test set, or by reframing the contribution as relative depth estimation. The graph evaluation, despite the human-in-the-loop annotation, would be substantially stronger with a held-out set that is manually verified. I would recommend the journal ask for these additions before publication; the system and dataset are otherwise a useful contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2411.15714. The thing to know: ROOT is a real, integrative system—GPT-4V iterative perception plus a fine-tuned 7B VLM that outputs a four-relation hierarchical scene graph with distances—and the SceneVQA dataset is large and original. The GraphVQA portion is manually annotated (9,761 images, 4-5 minutes each), which is a genuine resource. The ablation study is also decent: CoT matters, unfreezing the ViT mostly doesn't, scale helps. The Holodeck downstream demo is a nice proof of concept. I'd call this a practical subfield advance, not a breakthrough.\n\nThe soft spot is the distance metric. The paper's ground-truth distances come from DepthAnything, a monocular depth model that outputs relative depth with unknown per-image scale and shift. The authors never describe a calibration step to recover metric scale. So the '2.1m' labels in DistanceVQA—and the 740-image test set—are not actually in meters. The reported 74.3% accuracy within [80,120]% and the absolute/relative error analysis in Figure 12 largely measure consistency with the same uncalibrated pipeline used at train and test time. That's a load-bearing flaw for the paper's most distinctive claim, not a minor footnote. If DepthAnything's scale happens to be reasonably consistent across images, the model might still learn something about relative distances, but the paper neither shows that nor states it.\n\nThe relation side is healthier: GraphVQA ground truth is human-annotated, and the ~90% precision/recall is meaningful, though the test set is internally generated and the near-100% node accuracy is partly because the object list is fed to the model. Minor issues: no error bars, no direct SpatialVLM baseline in the distance table, code/data not released despite the claim, and the 'SceneLLM' in Figure 7 appears without definition in the text.\n\nOverall: the system and dataset are worth engaging with, but the distance numbers should not be quoted as evidence of metric understanding until the authors calibrate the depth scale (e.g., using known object sizes or RGB-D data) and release the test labels. I'd send this to peer review—referees can ask for exactly that. It is a legitimate paper, just oversold in one key claim.","headline":"Useful system paper with a large new dataset, but the metric-distance claim rests on an uncalibrated monocular depth pipeline.","tokens_in":25836,"tokens_out":2975,"would_cite":false,"duration_ms":26877,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a vision-language model fine-tuned on over 610,000 indoor images can output structured scene graphs and metric distances from a single RGB image, with relation precision and recall near 90 percent.","keywords":["indoor scene understanding","vision language model","scene graph generation","spatial reasoning","distance estimation","scene graph","RGB image","semi-automated data generation"],"falsifier":"Measure actual distances with a physical rangefinder for a set of indoor scenes, feed the same images to SceneVLM, and compare its meter predictions against the measured values; systematic error correlated with depth would show the model learned the labeling pipeline's bias. A second check is to run the model on images with deliberately unusual spatial layouts, such as objects hanging mid-air, and observe whether graph accuracy collapses.","tokens_in":24771,"feed_emoji":"🏠","tokens_out":5068,"duration_ms":43323,"temperature":0.7,"pith_summary":"This paper claims that a vision-language model fine-tuned on a large synthetic indoor-scene QA dataset can read a single RGB image of a room and output a structured scene graph, a JSON tree of support, contain, hang, and attach relations rooted at the floor, wall, and ceiling, together with metric distances between objects. The authors argue that current VLMs fail at this spatial hierarchical reasoning, and that their ROOT pipeline, which uses GPT-4V for object discovery and existing foundation models for depth and masks, supplies the data needed to train SceneVLM. If correct, a generic 7B-parameter VLM could provide the spatial understanding that downstream applications such as 3D scene generation and embodied robots currently lack. The paper reports relationship precision and recall around 90 percent and 74.3 percent distance accuracy within the strict 80-120 percent error band on its 740-image test set, compared with single-digit accuracy for most general-purpose VLMs.","feed_headline":"One RGB photo becomes an indoor scene graph","feed_subtitle":"Fine-tuned 7B VLM also estimates distances between objects, 74% accurate in the strict error band.","key_machinery":"The load-bearing object is the SceneVQA dataset and the semi-automated pipeline that builds it. GraphVQA relations are first manually annotated in JSON, then re-expressed by GPT-4 as chain-of-thought natural language before being paired with the JSON for training; DistanceVQA uses monocular depth from DepthAnything plus SAM masks and centroid distances in the unprojected 3D point cloud as distance labels. SceneVLM is a fine-tuned 7B VLM trained with a JSON-tree output format and chain-of-thought reasoning, learning to bridge directly from RGB appearance to a fixed four-relation hierarchical layout with distances given in meters.","core_discovery":"The central claim is that SceneVLM, an InternVL2 or Qwen2-VL fine-tuned on the authors' SceneVQA dataset of over 610,000 images, can generate hierarchical indoor scene graphs and estimate inter-object distances from RGB images alone, without depth sensors or explicit geometry at test time. The paper defines four spatial relationships (support, contain, hang, attach) and reports that the trained model reproduces the full JSON tree with pairwise relation precision of 91.4 percent and recall of 90.4 percent, object-wise precision of 87.7 percent with IoU around 80 percent, and node detection accuracy near 100 percent. For distances, SceneVLM achieves 74.3 percent accuracy within the strict 80-120 percent band and 97.4 percent within the looser 50-200 percent band, in both cases far ahead of general-purpose VLMs. These results are presented as evidence that VLMs can learn implicit spatial layout from RGB-only input once supplied with sufficient semi-automatically generated scene-relationship training data.","pith_inferences":["A natural extension not tested in the paper is to compare SceneVLM against physically measured ground truth, for example with a laser rangefinder, to separate true metric understanding from agreement with the DepthAnything-based labeling pipeline.","Because the relation set is closed and anchored to the floor, wall, and ceiling, the model's success may reflect learning a constrained grammar; testing with unusual scenes such as floating or oddly stacked objects would show whether the hierarchy generalizes beyond typical room layouts.","The semi-automated recipe of manual JSON plus GPT-4 paraphrase into chain-of-thought before fine-tuning could transfer to other structured-output tasks, suggesting the reusable trick is the chain-of-thought-as-formatting step rather than the specific scene graph task.","If metric calibration is the main obstacle, combining a small amount of true-depth supervision with the large pipeline-labeled corpus could yield a model with both coverage and metric accuracy."],"forward_implications":["A robot or agent can go from one RGB photo to a JSON scene graph listing which objects support, contain, hang from, or attach to others, without any depth input at test time.","The same VLM can answer distance questions in meters, which supports embodied tasks like moving an object to a reasonable location when combined with a general-purpose VLM such as GPT-4V.","The retrained SceneLLM requires only an object list, not an image, so users can specify desired furniture and receive a hierarchy that drives 3D scene generation through the Holodeck pipeline.","The SceneVQA dataset itself, with over 610,000 images and 320,000 object categories, is a publicly released resource for future spatial-reasoning training and evaluation."],"supporting_citations":[{"why":"Supplies the distance-evaluation protocol (50-200 percent and 80-120 percent bands) and text prompts used for data filtering.","marker":"[5]"},{"why":"Provides the object masks used in the pipeline to compute centroid distances for distance labels.","marker":"[23]"},{"why":"GroundingDINO detects objects and bounding boxes that feed the perception and parsing stages.","marker":"[31]"},{"why":"GPT-4V performs the iterative object and sub-object perception and bounding-box selection in the ROOT system.","marker":"[32]"},{"why":"DepthAnything supplies the monocular depth used to construct 3D point clouds and distance ground truth.","marker":"[50]"},{"why":"Holodeck provides the downstream 3D scene generation integration that validates SceneLLM layout output.","marker":"[53]"}],"fun_headline_variants":["Single RGB image yields indoor scene graph and distances","VLM turns one photo into a spatial scene graph","No depth sensor: VLM estimates object distances from RGB","SceneVLM builds hierarchical indoor maps from one image","From one RGB shot to a full scene graph with distances"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that distances computed by unprojecting monocular depth and taking centroid differences, together with masks and bounding boxes from the same foundation-model family, are accurate enough to serve as ground truth for both training and the 740-image test set; if that pipeline is metrically biased, the reported accuracies measure agreement with the pipeline rather than true spatial understanding.","fun_headline_variants_meta":{"raw":{"variants":["Single RGB image yields indoor scene graph and distances","VLM turns one photo into a spatial scene graph","No depth sensor: VLM estimates object distances from RGB","SceneVLM builds hierarchical indoor maps from one image","From one RGB shot to a full scene graph with distances"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3140,"prompt_tokens":985,"completion_tokens":2155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2078}},"tokens_in":601,"tokens_out":2155,"duration_ms":13267,"temperature":1.0,"reasoning_tokens":2078,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:58:59.245900+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure actual distances with a physical rangefinder for a set of indoor scenes, feed the same images to SceneVLM, and compare its meter predictions against the measured values; systematic error correlated with depth would show the model learned the labeling pipeline's bias. A second check is to run the model on images with deliberately unusual spatial layouts, such as objects hanging mid-air, and observe whether graph accuracy collapses.","supporting_citations":[{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"Provides the object masks used in the pipeline to compute centroid distances for distance labels."},{"cited_title":"Depth Anything: Unleashing the power of large-scale unlabeled data","cited_arxiv_id":null,"evidence_quote":"DepthAnything supplies the monocular depth used to construct 3D point clouds and distance ground truth."},{"cited_title":"HOLODECK: Language guided generation of 3d embodied ai environments","cited_arxiv_id":null,"evidence_quote":"Holodeck provides the downstream 3D scene generation integration that validates SceneLLM layout output."}],"review_version":1}