{"id":"8537ddb4-a08d-4ef1-a08a-d52f0975f9d7","arxiv_id":"2412.18450","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A learnable 3D scene graph representation that injects semantic relation embeddings between objects improves LLM performance on 3D grounding, captioning, and question answering.","lead":"This paper proposes 3DGraphLLM, a way to feed 3D scene graphs into a large language model by adding learned semantic relationship embeddings around each object. It reports consistent gains over a strong baseline that ignores object relationships across five 3D vision-language benchmarks, with fewer input tokens than video-based methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core ablation conflates semantic edges with added neighbor tokens and 4x context length; semantic relations are not isolated as the causal factor.","rationale":"The reader identified cross-domain transfer of VL-SAT relation features as the weakest assumption. I think the more immediate threat is attribution: the main comparison in Tab. 4 changes the token count, the neighbor-object context, and the relation embeddings simultaneously, so the observed gains cannot be unambiguously credited to semantic relationships. This is a correctness risk for the central claim, not merely a missing baseline in the related work. The permutation test above would settle both the confound and, indirectly, the transfer question: if shuffled edges match intact edges, no semantic signal is being used; if true edges beat shuffled and no-edge baselines, the cross-domain features carry usable signal. The paper has independent support — released code, two-stage training, multiple segmentation modes, and explicit discussion of n-gram metric limitations — so the appropriate outcome is still a conditional accept pending a controlled relation-content ablation, not rejection. This does not change the reader's verdict, but it sharpens the condition under which the paper should be accepted.","tokens_in":20236,"tokens_out":6351,"duration_ms":62167,"concrete_test":"Run one controlled ablation on ScanRefer/Multi3DRefer with Mask3D and the full pretraining pipeline: use the exact k=2 triplet token layout from Tab. 4, but randomly permute the relation embeddings F^e_ij across object pairs within each scene before projection, keeping the multiset of edge vectors and the token count unchanged. Retrain with the same hyperparameters (LoRA rank 16, 3 epochs, lr 5e-6). If shuffled-edge performance stays close to the intact 3DGraphLLM-2 row (e.g., 56.2/58.7 for LLAMA3), semantic edge content is not the active ingredient; if it falls toward the 3DGraphLLM-0 level (52.0/55.1), the relation features carry the claimed signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 4.2 compares 3DGraphLLM-0 (k=0, flat object list) with 3DGraphLLM-2 (k=2, per-object triplets) to conclude that semantic relationships improve grounding, captioning, and QA. The two conditions differ not only in the presence of edge embeddings F^e_ij but also in (i) the number of tokens per scene (Sec. 3.2: 2n vs 2n+3nk; for n=100, 200 vs 800), (ii) the inclusion of two neighbor objects' 3D features per object, and (iii) the overall prompt structure. Any of these could explain the gains. The supplementary subgraph ablation (Tab. 8) compares triplet vs edge-only representations but both retain relation embeddings, so it does not isolate semantic content. The paper's own conclusion in the abstract, that semantic relationships 'significantly improve' accuracy, therefore rests on an uncontrolled comparison. The reader's VL-SAT cross-domain concern is related but downstream: even if VL-SAT features transfer perfectly, the current experiment cannot show that their semantic content, rather than added context, drives the improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes 3DGraphLLM, a method that converts a 3D scene into a sequence of tokens: for each object, an identifier token, a 2D DINOv2 feature, and a subgraph of k nearest neighbors encoded as triplets (object 3D feature, VL-SAT relation embedding, neighbor 3D feature), all projected into the token embedding space of a pre-trained LLM. The authors evaluate on ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D, with a two-stage training procedure using ground-truth and predicted instance segmentation. The central claim is that including semantic relationship embeddings significantly improves grounding, captioning, and question answering over a k=0 object-list baseline, and that the method achieves state-of-the-art grounding accuracy with fewer tokens and faster inference.","tokens_in":20488,"tokens_out":6930,"duration_ms":59421,"significance":"The method is well-motivated and the engineering is solid: it reuses strong pretrained encoders (DINOv2, Uni3D, VL-SAT), provides a compact graph tokenization (800 tokens for 100 objects at k=2), reports detailed ablations across LLMs, segmentation quality, and k values, and releases code. If the causal effect of relation embeddings were established, this would be a valuable contribution. However, the experiments as designed do not isolate the contribution of semantic relation content, so the central claim currently rests on a confounded comparison. The resource-efficiency and competitive grounding results are encouraging and should be preserved, but the causal interpretation needs additional controls.","major_comments":[{"comment":"The comparison between 3DGraphLLM-0 and 3DGraphLLM-2 changes three ingredients at once: the token count (2n vs 2n+3nk, i.e., 200 vs 800 tokens for n=100), the presence of neighbor object 3D features Fv_j in the prompt, and the presence of semantic edge embeddings Fe_ij. Therefore the observed gains cannot be attributed specifically to semantic relationships, as the abstract claims. To isolate the effect, the authors need control conditions that keep the neighbor features and sequence length fixed while ablating the edge content, for example replacing Fe_ij with zero/random embeddings, or shuffling edge embeddings across object pairs. Such controls are missing, so the central claim is not yet supported.","section":"§4.2, Table 4"},{"comment":"The triplet-vs-edge-only ablation is also not a control for semantic content: both conditions contain relation embeddings Fe_ij, and the triplet condition additionally includes neighbor vertex tokens Fv_j and more tokens. This experiment can at most show that neighbor vertex features help; it does not test whether the relation embeddings themselves carry useful semantics. A condition with neighbor features but without edge embeddings would be needed.","section":"§4.2, Table 8 / Appendix C"},{"comment":"All reported numbers are single point estimates. With gains as small as +0.1 EM on SQA3D (Vicuna, Table 4) and even a negative difference on ScanRefer Acc@0.5 (50.2 to 50.1), the word 'significantly' in the abstract is not justified without variance or significance tests across training seeds or validation subsets.","section":"§4.2, Tables 4 and 5"},{"comment":"The text in Appendix B states that semantic edges yield 'a consistent improvement' across all three tasks, but Table 7 shows that for OneFormer3D segmentation, 3DGraphLLM-2 decreases Scan2Cap CIDEr@0.5 (73.5 to 70.2), BLEU-4@0.5 (34.3 to 32.7), ScanQA CIDEr (87.3 to 83.3), and BLEU-4 (16.5 to 15.0) relative to 3DGraphLLM-0. This contradicts the blanket claim in the abstract that semantic relationships improve performance across the benchmarks, and it should be discussed.","section":"Supplementary Table 7 / Appendix B"},{"comment":"The authors themselves argue that n-gram metrics are 'not adequate' for LLM-generated answers and provide a correct caption that receives CIDEr 0.0. Yet the manuscript uses CIDEr and BLEU as primary evidence for gains on Scan2Cap, ScanQA, and SQA3D. Unless a complementary evaluation (e.g., human ratings or an LLM-as-judge) is provided, the cross-task improvement claim on these benchmarks is not trustworthy.","section":"§4.2 and Appendix E"}],"minor_comments":[{"comment":"The caption uses 'LoRa' where the text and standard notation write 'LoRA'; please fix the typo.","section":"Figure 2"},{"comment":"The token count formula 2n+3nk assumes an ordered, directed triplet for each neighbor; if the graph is considered undirected, clarify why the directed count is appropriate for the prompt sequence.","section":"§3.2"},{"comment":"The main text says 'ground-truth scene segmentation' while Figure 4's caption says 'GT instance segmentation'; unify the terminology.","section":"§4.2 / Figure 4"},{"comment":"The related-work discussion cites Scene-LLM and MOMA but does not position 3DGraphLLM against them as other LLM-based methods that consume scene-graph-like structures; a brief comparison would help.","section":"§2"},{"comment":"The 'C' column for ScanQA is defined only in the text; add a note in the table caption indicating that C denotes CIDEr.","section":"Table 2"},{"comment":"Reference [34] is a self-citation used as support for text-based graph retrieval; the relationship to the current method should be clarified explicitly in the text.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is competent and the engineering is thorough, but the abstract's causal claim about semantic relationships is not supported by the current experiments because the k=0 vs k=2 comparison confounds relation content with added neighbor features and sequence length. The required control experiments are within the scope of a revision. If the authors cannot produce them, the central claim would reduce to an added-context effect, which would not support the stated contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a competent, practical paper that builds a learnable 3D scene-graph representation for LLMs and shows consistent gains over the Chat-Scene baseline across five ScanNet benchmarks. Code is public, the ablations are extensive, and the two-stage training with GT then predicted segmentation is a nice touch. Worth a serious look.\n\nThe genuinely new piece is the tokenization: each object becomes an identifier token, its 2D/3D features, and triplets (object, relation, neighbor) for k nearest neighbors, with relation embeddings taken from VL-SAT's latent edge features. That specific way of flattening a semantic graph into an LLM prompt is not in Chat-Scene or VL-SAT, so it clears novelty.\n\nWhat the paper does well: it tests across grounding, captioning, and QA; it checks GT vs Mask3D vs OneFormer3D segmentation; it evaluates different base LLMs; and it reports inference speed and token counts fairly. The +6-7 point gains on grounding under GT segmentation are not trivial.\n\nNow the soft spots. The main one is that the central causal claim is not actually isolated. The 0-edge vs 2-edge comparison in Table 4 differs in three ways at once: presence of relation embeddings, presence of the neighbor objects' 3D features, and a 4x longer context. Any of those could explain the improvement. The supplementary subgraph ablation (Table 8) compares triplets vs edge-only sequences, but both include relation embeddings, so it still doesn't separate semantic content from extra tokens. The paper's abstract says semantic relationships 'significantly improve' accuracy; that specific attribution is unsupported as the experiments stand. Adding a control with neighbor features but no relation embeddings (or vice versa) would fix this. I'd call it a moderate flaw, not fatal: the method works, but the marketing overstates what's proven.\n\nAlso, the 'state-of-the-art' claim is too strong. Looking at Table 2, GPT4Scene-HDM gets 57.0 on ScanRefer Acc@0.5; 3DGraphLLM gets 56.6. On Multi3DRefer and SQA3D it's roughly matched with Robin3D, not clearly ahead. Saying SOTA in grounding overclaims by a couple of points.\n\nMissing error bars and single-point estimates are another weakness, though common in this line of work; the paper should report variance or at least multiple seeds.\n\nBottom line: this is a solid within-subfield contribution, and the confound is addressable without redoing the whole experimental program. It deserves peer review, with a request to fix the ablation and soften the claims. I'd cite it as a baseline in future 3D VLM work. Bring it to reading group.","headline":"Useful engineering contribution, but the headline claim that semantic relations drive the gains is not actually isolated by the ablations.","tokens_in":21007,"tokens_out":2906,"would_cite":true,"duration_ms":26553,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that feeding a large language model a learnable 3D scene graph—objects plus semantic edges to their nearest neighbors—improves accuracy on object grounding, dense captioning, and question answering over a flat list of…","keywords":["3D scene understanding","semantic scene graph","large language models","3D visual grounding","3D dense captioning","3D visual question answering","learnable scene representation","point cloud"],"falsifier":"A direct test would be to replace VL-SAT latent edge features with random or zero embeddings while keeping the same graph structure and training pipeline; if ScanRefer or Multi3DRefer accuracy stays at the same level, the improvement attributed to semantic relationships is actually coming from the added object-neighbor tokens or from the projection layers rather than from the relation semantics.","tokens_in":20045,"feed_emoji":"🤖","tokens_out":7363,"duration_ms":64888,"temperature":0.7,"pith_summary":"3DGraphLLM claims that a large language model can answer questions about a 3D scene more accurately when the scene is fed in as a learnable scene graph—objects plus explicit semantic relation edges between nearby objects—rather than a flat list of object embeddings. The paper builds a token sequence in which each object is represented by an identifier, its 2D and 3D features, and triplets pairing it with its two nearest neighbors through relation embeddings sourced from a pre-trained scene-graph model. Across five ScanNet benchmarks, adding these relation triplets improves object grounding, dense captioning, and question answering over the no-relations baseline, with the largest gains on the grounding tasks. The practical payoff is that an embodied agent can get these gains without a large increase in token budget: representing a 100-object scene with $k=2$ neighbors needs about 800 tokens, and inference on grounding tasks runs up to five times faster than comparable vision-language-model approaches.","feed_headline":"Relation edges lift LLM accuracy on 3D scene tasks","feed_subtitle":"A compact learnable scene graph with two neighbors per object beats object-list baselines on five ScanNet benchmarks","key_machinery":"The key machinery is a flat, learnable token representation of a 3D scene graph. Each object becomes a small subgraph: an identifier token, a 2D object feature token from DINOv2, a 3D point-cloud feature token from Uni3D, and $k$ nearest-neighbor triplets $(F^v_i, F^e_{ij}, F^v_j)$, where $F^e_{ij}$ is the projected latent edge embedding produced by VL-SAT, a pre-trained 3D scene-graph generator. This flattens the graph into a sequence the LLM can consume, reducing a complete graph's $O(n^2)$ edges to $3nk$ triplet tokens; with $k=2$ and 100 objects the scene description is about 800 tokens. The projection layers and a low-rank fine-tuned LLM are trained to predict object identifiers, captions, or answers, so the relation edges serve as direct evidence inside the prompt rather than as a separate reasoning module.","core_discovery":"The paper's central claim is that an LLM can solve 3D vision-language tasks more accurately when the scene is presented as a learnable semantic scene graph rather than as a list of object embeddings. For each object, the representation includes an identifier token, a 2D DINOv2 feature token, a 3D Uni3D point-cloud token, and $k$ triplets $(F^v_i, F^e_{ij}, F^v_j)$ pairing the object with its nearest neighbors; $F^e_{ij}$ is the projected latent edge embedding from VL-SAT, taken before the relation-classification head so it can encode combinations of relations. The paper reports that this edge-aware representation outperforms the no-edges baseline on five ScanNet benchmarks, with gains of +7.5 F1@0.5 on Multi3DRefer, +6.4 Acc@0.5 on ScanRefer, and +3.9 CIDEr@0.5 on Scan2Cap, while using roughly 800 tokens to represent a 100-object scene at $k=2$. It also reports grounding accuracy on par with the strongest compared vision-language model at up to five times lower inference time.","pith_inferences":["A testable extension the paper does not run: replace VL-SAT latent edge features with categorical relation labels or with CLIP text embeddings of relation phrases; if accuracy stays flat, the gain is coming from the extra neighbor context rather than from the learned relation geometry.","Because the gains persist with noisy Mask3D segmentation, the representation may tolerate imperfect perception, but the paper only evaluates indoor ScanNet-like scenes; extending the same token format to outdoor or dynamic scenes would require an edge encoder for those domains, which is an inference beyond the paper's experiments.","The paper's discussion of n-gram metrics suggests that some of the apparent failures in captioning and QA are actually correct answers containing spatial cues absent from the references; if that reading is right, the true advantage of semantic edges on free-form language tasks may be larger than the reported scores show."],"forward_implications":["An embodied agent can get more accurate object grounding from an LLM by adding just a few semantic relation tokens per object, without expanding the scene description to hundreds of text tokens.","Semantic edges help even when instance segmentation is imperfect, so the approach remains usable when ground-truth object masks are unavailable.","The same graph-to-token encoding supports grounding, dense captioning, and question answering with one model, since all tasks are framed as user-assistant text generation.","With $k=2$ nearest neighbors, a 100-object scene uses about 800 tokens, which keeps inference fast enough for interactive use; this follows directly from the paper's token-count calculation and speed experiments.","Relation edges help most when the query itself is about spatial relationships; the paper's ablations show grounding and captioning gain more than question answering, because some QA questions do not need spatial context."],"supporting_citations":[{"why":"Supplies the VL-SAT latent semantic relation edge features that carry the new information added to the LLM prompt.","marker":"[52]"},{"why":"Chat-Scene is the no-relations baseline and the source of the object-identifier architecture and joint training strategy.","marker":"[25]"},{"why":"Uni3D provides the pre-trained 3D point-cloud encoder used for graph vertex features.","marker":"[63]"},{"why":"DINOv2 supplies the aggregated 2D object features that become additional tokens in each object subgraph.","marker":"[37]"},{"why":"ScanRefer provides the 3D referred-object-grounding benchmark and training data for the main grounding evaluation.","marker":"[5]"},{"why":"Multi3DRefer provides the multi-object grounding benchmark where the paper reports its largest relative gain.","marker":"[60]"},{"why":"Scan2Cap provides the dense scene captioning benchmark and metrics.","marker":"[9]"},{"why":"ScanQA provides one of the two 3D question answering benchmarks used for evaluation.","marker":"[3]"},{"why":"SQA3D provides the situated question answering benchmark and the exact-match metric.","marker":"[35]"},{"why":"Mask3D is the instance segmentation method used to build scene graphs under realistic noisy conditions.","marker":"[46]"}],"fun_headline_variants":["Semantic graph edges boost LLM accuracy on 5 ScanNet benchmarks","Edge-aware scene graph lifts LLM accuracy on 3D vision-language tasks","LLM with semantic graph edges beats object-only baselines","Compact scene graph with edges improves LLM 3D understanding","Semantic edges in scene graph lift LLM accuracy on scan tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that the learned relationship features between objects, produced by a model trained on one set of indoor scans, still carry useful meaning when applied to the separate set of indoor scans used for evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Semantic graph edges boost LLM accuracy on 5 ScanNet benchmarks","Edge-aware scene graph lifts LLM accuracy on 3D vision-language tasks","LLM with semantic graph edges beats object-only baselines","Compact scene graph with edges improves LLM 3D understanding","Semantic edges in scene graph lift LLM accuracy on scan tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3569,"prompt_tokens":1026,"completion_tokens":2543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":2452}},"tokens_in":642,"tokens_out":2543,"duration_ms":17711,"temperature":1.0,"reasoning_tokens":2452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:41:07.184749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to replace VL-SAT latent edge features with random or zero embeddings while keeping the same graph structure and training pipeline; if ScanRefer or Multi3DRefer accuracy stays at the same level, the improvement attributed to semantic relationships is actually coming from the added object-neighbor tokens or from the projection layers rather than from the relation semantics.","supporting_citations":[{"cited_title":"Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud","cited_arxiv_id":null,"evidence_quote":"Supplies the VL-SAT latent semantic relation edge features that carry the new information added to the LLM prompt."},{"cited_title":"Chat-scene: Bridging 3d scene and large language models with object identifiers","cited_arxiv_id":null,"evidence_quote":"Chat-Scene is the no-relations baseline and the source of the object-identifier architecture and joint training strategy."},{"cited_title":"Scanrefer: 3d object localization in rgb-d scans using natural language","cited_arxiv_id":null,"evidence_quote":"ScanRefer provides the 3D referred-object-grounding benchmark and training data for the main grounding evaluation."},{"cited_title":"Multi3drefer: Grounding text description to multiple 3d ob- jects","cited_arxiv_id":null,"evidence_quote":"Multi3DRefer provides the multi-object grounding benchmark where the paper reports its largest relative gain."},{"cited_title":"Scan2cap: Context-aware dense captioning in rgb- d scans","cited_arxiv_id":null,"evidence_quote":"Scan2Cap provides the dense scene captioning benchmark and metrics."},{"cited_title":"Scanqa: 3d question answering for spatial scene understanding","cited_arxiv_id":null,"evidence_quote":"ScanQA provides one of the two 3D question answering benchmarks used for evaluation."},{"cited_title":"Mask3d: Mask trans- former for 3d semantic instance segmentation","cited_arxiv_id":null,"evidence_quote":"Mask3D is the instance segmentation method used to build scene graphs under realistic noisy conditions."}],"review_version":1}