{"id":"33984096-1af8-425d-89a3-386dcd64a63f","arxiv_id":"2412.13652","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RelationField learns open-vocabulary inter-object relationship features inside a neural radiance field by distilling from a multimodal LLM, enabling relationship queries, 3D scene graph extraction, and relationship-guided segmentation.","lead":"RelationField extends neural radiance fields to represent relationships between objects as pairs of rays, distilling relationship knowledge from multimodal LLMs into a queryable 3D field. It reports state-of-the-art results on 3D scene graph prediction and introduces a relationship-guided instance segmentation benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3DSSG evaluation does not isolate the radiance-field relationship representation from the per-scene GPT-4 teacher, so the central claim that relationships are extracted directly from radiance fields lacks a decisive control.","rationale":"The reader's weakest assumption is that the relationship knowledge is entirely inherited from the multimodal LLM teacher, so teacher errors are stored faithfully. My concern is closely related but sharper: the experiments do not include a control that isolates how much of the benchmark recall is contributed by the radiance-field relationship representation as opposed to a direct lifting of the same teacher labels. This is load-bearing because the paper's strongest claim is not 'we can distill GPT-4 relationships into a NeRF' but 'we extract inter-object relationships directly from radiance fields,' and the 3DSSG numbers are the primary quantitative support for that claim. The comparison against Open3DSG and ConceptGraphs is useful but not decisive for this point, since those baselines use different supervision pipelines, and the 2D GPT-4 baseline in Tab. 1 is not an ablation of RelationField's own supervision. A teacher-lifting baseline would settle whether the 3D representation adds predictive value beyond the teacher's per-view output. Because the reader already assigned a conditional verdict with moderate confidence, my analysis does not change the verdict; it reinforces the conditionality: the central claim should be accepted only after the proposed control, or release of code/data enabling it, is provided. I credit the paper for a clear method description, qualitative demonstrations, and a Gaussian-splatting adaptation, but the missing teacher-isolation control is the main soft spot in the argument.","tokens_in":17588,"tokens_out":5197,"duration_ms":50533,"concrete_test":"Build a teacher-lifting baseline on the same 3DSSG test scenes: run the identical GPT-4o+SoM prompt and JSON parsing on the training views, assign each detected relationship to the corresponding SAM mask pair, aggregate votes across views, and evaluate with the paper's exact 3DSSG object/predicate/relationship recall protocol using the same CLIP and Jina encodings. If this baseline meets or exceeds RelationField's Tab. 1 results, the radiance-field relationship representation is not responsible for the reported performance; if it is clearly worse, the 3D-consistency argument in Sec. 4.2 is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that RelationField is the first method to extract inter-object relationships directly from neural radiance fields, with state-of-the-art 3DSSG results. But the method is a per-scene distillation: for each 3DSSG scene, GPT-4o with SoM prompting produces relationship text on training views, the text is encoded with jina-embeddings-v3, and the radiance field is optimized to reproduce those features (Sec. 3.2, Sec. 4.2). The quantitative evaluation then ranks the field's embeddings against the same closed-set 3DSSG label embeddings. The only 3D-versus-2D control in Tab. 1 is a GPT-4 baseline that the paper does not specify as using the same SoM prompt, the same pair-sampling, or the same aggregation and encoder. Consequently, the reported recall could be substantially inherited from the teacher labels themselves: a baseline that lifts the same per-view GPT-4/SoM relationships into the 3D point cloud via the SAM masks could plausibly match or exceed RelationField's numbers without any radiance-field relationship representation. Section 5 concedes that the relationship knowledge is 'highly dependent on the multi-modal LLM prompting and its output,' but no experiment measures how much of the benchmark performance is directly attributable to that teacher versus to the 3D volumetric representation. Without that control, the evidence does not yet establish that the radiance field itself contributes the relationship understanding claimed in the title and abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RelationField, a NeRF extension that predicts, in addition to color, density, object-level semantic features, and instance grouping features, a relationship feature for each pair of 3D query locations. Relationship supervision is obtained by prompting a multimodal LLM (GPT-4o) with Set-of-Mark annotated training views, encoding the textual relationship descriptions with jina-embeddings-v3, and training a volumetric MLP head to reproduce those features via a cosine-similarity loss. At inference, relationship queries are answered by comparing the query text embedding to the rendered relationship features. The method is evaluated on 3D scene graph generation on the RIO10 subset of 3DSSG, where it reports state-of-the-art top-k recall, and on a newly introduced relationship-guided 3D instance segmentation benchmark on ScanNet++, with qualitative results on additional scenes. Ablations compare 2D per-view inference, different LLM teachers, and a Gaussian Splatting variant.","tokens_in":17885,"tokens_out":8687,"duration_ms":74180,"significance":"If the central claims are established, the paper makes a useful contribution: it shows that inter-object relationship features can be distilled into a volumetric field from posed RGB images alone, that such a field supports open-vocabulary relationship queries, and that 3D-consistent rendering reduces per-view errors relative to 2D inference. The method is clearly described, the 3D-consistency ablation in Fig. 5 is informative, and the authors' explicit statement in Sec. 5 that the relationship knowledge depends on the LLM teacher is honest. However, the quantitative evaluation does not yet disentangle the contribution of the radiance-field representation from that of the per-scene teacher, and the new benchmark is small and not released. These are substantive gaps for the paper's central and secondary claims.","major_comments":[{"comment":"The main quantitative evidence for the central claim is not yet decisive because it lacks a control that isolates the radiance-field relationship representation from the per-scene GPT-4 teacher. The relationship supervision is generated per training view by GPT-4 with SoM prompting, encoded with jina-embeddings-v3, and distilled into the field (Sec. 3.2); the GPT-4 (2D+depth) baseline in Tab. 1 is not described as using the same SoM prompt, the same pair-sampling strategy, the same text encoder, or the same aggregation into the 3D point cloud. A baseline that lifts the same per-view GPT-4/SoM relationship labels to the point cloud via SAM masks and then ranks them with the same label embeddings could match or exceed RelationField's recall without any volumetric relationship field. Reporting such a matched teacher-lifting baseline is necessary to support the claim that relationships are extracted directly from the radiance field rather than inherited from the teacher; the paper's own Sec. 5 limitation statement makes this concern explicit. The concern is not tautology, since the field is queried at locations and pairs not directly supervised, but the benchmark as reported does not separate the teacher's per-view knowledge from the field's 3D-rendered knowledge.","section":"Sec. 4.2, Tab. 1"},{"comment":"The newly introduced relationship-guided instance segmentation benchmark is too small and too under-specified to support the quantitative claim. It contains '~30 relationship queries across 8 scenes' and the manuscript does not state that the annotations will be released or provide a download link; without the benchmark, the results in Tab. 2 cannot be reproduced or extended by other groups. With ~30 queries, the differences between LERF, OpenNeRF, LangSplat, and RelationField (IoU 0.25, 0.45, 0.49, 0.53) are not accompanied by error bars or a per-scene breakdown, so it is unclear whether any of the differences beyond LERF are statistically meaningful. For a benchmark proposed to support future research, a release plan and variance estimates are necessary.","section":"Sec. 4.3 and supplementary Sec. F"},{"comment":"The quantitative scene-graph evaluation queries the field on the provided 3D point cloud, and the paper states the point cloud is used 'exclusively for evaluation.' This is a reasonable protocol for comparison with point-cloud-based methods, but it means the demonstrated 3D scene graph extraction is not purely from the radiance field: instance seeds and query locations come from the depth-sensor-derived geometry. The paper should either (a) describe and evaluate a fully radiance-field-based protocol in which instances and query locations are obtained by volumetric rendering (as in Fig. 3), or (b) clearly state that the quantitative graphs currently require ancillary geometry at test time. This distinction matters for the abstract's claim of extracting relationships 'directly from neural radiance fields.'","section":"Sec. 4.2, '3D Scene Graph Construction'"},{"comment":"The open-vocabulary claim is only measured indirectly on a closed set of 160 object classes and 27 relationship types; the paper queries these predefined labels and reports top-k recall. The qualitative results in Fig. 3 demonstrate open-vocabulary behavior, but a quantitative evaluation on held-out relationship types or an object/relationship retrieval task would directly support the 'open-vocabulary' wording in the title and abstract. In addition, the paper calls its results 'state-of-the-art in both tasks,' but the second task is newly proposed, so 'state-of-the-art' there only means 'better than the three existing feature-field baselines,' which is a weaker statement than the paper's phrasing suggests.","section":"Sec. 4.2, data and metrics"}],"minor_comments":[{"comment":"There are several language errors: 'recently even been extended' should be 'have recently been extended'; 'current method primarily' should be 'current methods primarily'; and 'multi-model LLM' should be 'multimodal LLM.'","section":"Abstract and Introduction"},{"comment":"The caption refers to 'bag-of-words behavior of CLIP [41]', but reference [41] is GPT-2; CLIP is reference [42] in the main text. Please correct the citation.","section":"Supplementary, Fig. 12 caption"},{"comment":"Equation (2) defines the relationship field with inputs (x, d, z), but Sec. 3.4 and the scene graph construction in Sec. 4.2 appear to query the field without a direction d; please clarify whether d is used in the relationship head or whether Eq. (2) should be simplified.","section":"Sec. 3.1 and Sec. 3.4"},{"comment":"The response score depends on the choice of canonical phrases ('and', 'next to', 'none'), and the paper gives no sensitivity analysis for this choice or for the 0.5 threshold used in supplementary Sec. C. Adding such an analysis would help readers understand how much the qualitative queries depend on these free parameters.","section":"Sec. 3.3, Eq. (4)"},{"comment":"The paper does not mention any plan to release code or trained models; given the complexity of the pipeline (NeRF training, SAM, GPT-4 prompting, jina encoding), a code release would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the CV community, but the comparison is currently too favorable to the method. The main risk is that readers will interpret the 3DSSG numbers as evidence for the volumetric relationship representation when they may largely reflect the teacher's per-view outputs. The authors should be asked to run the matched teacher-lifting baseline and to release the new benchmark annotations before the claims can be fully evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: RelationField's pair-of-rays relationship field is a real step forward, but the headline numbers don't yet prove what the title claims. The 3DSSG evaluation has a missing control that makes the central claim—that the radiance field itself is doing the relationship understanding—under-supported.\n\nWhat's new and good: Conditioning a radiance field on a query location to output relationship features is clean and hadn't appeared in the LERF/GARField line; the \"first method\" claim looks plausible from the cited prior work. The SoM-prompted GPT-4 supervision pipeline is a sensible way to get dense relationship labels, and the ablations on language encoders and LLM choice are useful. The 3D consistency figure (Fig. 5) is compelling at show-and-tell level, and the relationship-guided segmentation task is a nice addition to the field, even if small.\n\nThe main soft spot is exactly where the stress-test points. The 2D GPT-4 baseline in Tab. 1 is not described as using SoM, the same pair-sampling, or the same aggregation, so the comparison doesn't separate the teacher's knowledge from the 3D representation's contribution. A baseline that lifts the same per-view SoM/GPT-4 relationships into the point cloud via SAM masks could plausibly match or beat RelationField's recall without any pair-of-rays field. That missing control is load-bearing for the \"directly from radiance fields\" claim. The authors are honest in Sec. 5 that the relationship knowledge depends on the LLM, but they don't measure how much.\n\nMinor issues: the new benchmark is small (~30 queries, 8 scenes) and not released; code isn't available either. The closed-set 3DSSG evaluation with provided point clouds also weakens the \"RGB-only\" claim, since evaluation requires a 3D point cloud. Those are fixable with public release and a better control.\n\nWho it's for: people working on feature fields, 3D scene graphs, and open-vocabulary scene understanding will want to read it. It deserves a serious referee—the core idea is novel and the writing is clear—but the decision should hinge on the missing control rather than a yes/no on the method. I'd send it to review with a request to add the controlled 2D-to-3D baseline and release the benchmark.","headline":"The pair-of-rays relationship field is a genuinely novel idea, but the 3DSSG evaluation never controls for the teacher model's own knowledge, so the claim that relationships come 'directly from radiance fields' is not yet proven.","tokens_in":18439,"tokens_out":3332,"would_cite":true,"duration_ms":30030,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RelationField embeds open-vocabulary object relationships directly inside neural radiance fields.","keywords":["neural radiance fields","open-vocabulary scene understanding","relationship distillation","3D scene graphs","feature fields","set-of-mark prompting","relationship-guided instance segmentation","multimodal LLM"],"falsifier":"Train a controlled variant of RelationField on a fixed set of views in which the query location $z$ for each sampled ray pair is randomly permuted across object instances, holding the LLM relationship labels fixed. If 3D scene graph recall on 3DSSG stays at the same level, the relationship head is not actually using the identity of the second object and the pair-of-rays formulation is not doing the claimed work; if recall collapses, the dependency on the query object is real.","tokens_in":17400,"feed_emoji":"🔗","tokens_out":6195,"duration_ms":54121,"temperature":0.7,"pith_summary":"RelationField aims to show that a neural radiance field can carry open-vocabulary knowledge about relationships between objects, not just object identities. The paper's claim is that by adding a second 'query ray' to the field's input and distilling relationship descriptions from a multimodal large language model, the field learns a dense, multi-view-consistent embedding of inter-object relations that can be queried with arbitrary text. On the 3DSSG benchmark the resulting representation sets a new state of the art for open-vocabulary 3D scene graph prediction, and it enables a new relationship-guided instance segmentation task on ScanNet++. A sympathetic reader takes away that relationship understanding can live inside the same implicit scene representation used for rendering, without depth sensors or explicit meshes.","feed_headline":"Radiance fields learn to relate any two objects","feed_subtitle":"New approach distills LLM relationship knowledge into 3D, enabling scene graphs and relation-guided segmentation from RGB only.","key_machinery":"The load-bearing mechanism is a pair-of-rays relationship head: the function $g_\\theta(x, d, z)$ maps a sample point, its viewing direction, and a query location to color, density, object features, and a 512-dimensional relationship feature in a language-aligned embedding space. Training uses Set-of-Mark prompting—overlaying numbered semi-transparent masks on training images—so a multimodal LLM can name object pairs and their relationships; those textual descriptions are encoded and distilled into the field with a cosine-similarity rendering loss. At query time, a textual predicate is encoded, matched against rendered relationship features, and converted to a response probability by pairwise softmax against canonical phrases such as \"and\", \"next to\", and \"none\". This same mechanism, combined with object-level semantic and instance heads, lets the authors extract full 3D scene graphs by clustering instance embeddings and aggregating relationship responses for instance pairs.","core_discovery":"The central discovery is that inter-object relationships can be defined as a function of two points in a radiance field: the sampled location $x$ on a ray and an additional implicit query location $z$, so the field outputs a relationship feature $r$ alongside color, density, and object features. Because the relationship feature lives in a language embedding space, any textual predicate can be matched against it by cosine similarity. The field is trained by rendering $r$ along rays and maximizing its similarity to language encodings of relationship descriptions that a multimodal LLM produces for object pairs marked in each training view. The paper argues that this distillation yields a 3D-consistent relationship representation, which is why it outperforms both 2D-only LLM inference and prior feature-field methods on scene graph prediction and relationship-guided instance segmentation.","pith_inferences":["Because the pair-of-rays input is symmetric in principle but trained with a fixed subject-object order, the authors' direction-swapping supervision trick could be made automatic by also training on swapped pairs, which would likely improve symmetric predicate recall.","If the teacher LLM mislabels a relationship in every view, the field will memorize that error; probing the field with deliberately contradictory predicates in a scene with known ground truth would quantify how much teacher hallucination survives distillation.","The same representation could support affordance and task-planning queries directly in 3D, since affordance predicates such as \"turns on\" and \"attached to\" already appear in the training prompts.","A natural stress test is whether the field can answer a predicate that the LLM never mentioned in any training view; if it cannot, the \"open-vocabulary\" claim is bounded by the teacher's coverage rather than by the embedding space."],"forward_implications":["3D scene graphs can be extracted from posed RGB images alone, without depth sensors, meshes, or point-cloud inputs at training time.","Objects that are otherwise ambiguous can be disambiguated by relationship: the same instance head plus a predicate like \"standing on\" separates \"picture standing on the shelf\" from other pictures.","Relationship errors from 2D view-dependent LLM inference are reduced, because volumetric rendering integrates relationship evidence across views into one consistent 3D field.","The approach is agnostic to the teacher model and the underlying radiance field, working with different multimodal LLMs and with both NeRF and Gaussian Splatting geometry."],"supporting_citations":[{"why":"Supplies the NeRF formulation of color, density, and volumetric rendering that RelationField extends with object and relationship heads.","marker":"[35]"},{"why":"Introduces the language-embedded feature field and the canonical-phrase softmax querying scheme that RelationField adapts for relationships.","marker":"[22]"},{"why":"Shows how to distill open-vocabulary CLIP features into a radiance field, the object-level precedent for the relationship head.","marker":"[13]"},{"why":"Provides the segmentation masks used for Set-of-Mark prompting and for projecting relationship features onto the image plane.","marker":"[24]"},{"why":"Defines the open-vocabulary object-embedding space that the semantic head is trained to predict.","marker":"[42]"},{"why":"Supplies the teacher that produces textual relationship descriptions between marked object pairs for distillation.","marker":"[1]"},{"why":"Is the open-vocabulary point-cloud scene graph method that RelationField compares against and exceeds on 3DSSG.","marker":"[27]"},{"why":"Provides the 3DSSG dataset and benchmark task used for the scene graph evaluation.","marker":"[53]"}],"fun_headline_variants":["RelationField lets radiance fields relate any two objects","First to extract open-vocabulary object relations from radiance fields","Radiance fields directly model inter-object relationships","LLM-distilled relationship features in neural radiance fields","Open-vocabulary 3D scene graphs from radiance fields"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All relationship knowledge comes from a multimodal language model's per-view captions; if that teacher mislabels, misses, or hallucinates a relationship in the training views, the radiance field stores the error faithfully because no geometric or physical check corrects it.","fun_headline_variants_meta":{"raw":{"variants":["RelationField lets radiance fields relate any two objects","First to extract open-vocabulary object relations from radiance fields","Radiance fields directly model inter-object relationships","LLM-distilled relationship features in neural radiance fields","Open-vocabulary 3D scene graphs from radiance fields"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000706,"raw_usage":{"total_tokens":3138,"prompt_tokens":858,"completion_tokens":2280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2199}},"tokens_in":474,"tokens_out":2280,"duration_ms":15149,"temperature":1.0,"reasoning_tokens":2199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:55:03.041097+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a controlled variant of RelationField on a fixed set of views in which the query location $z$ for each sampled ray pair is randomly permuted across object instances, holding the LLM relationship labels fixed. If 3D scene graph recall on 3DSSG stays at the same level, the relationship head is not actually using the identity of the second object and the pair-of-rays formulation is not doing the claimed work; if recall collapses, the dependency on the query object is real.","supporting_citations":[{"cited_title":"Srinivasan, Matthew Tancik, Jonathan T","cited_arxiv_id":null,"evidence_quote":"Supplies the NeRF formulation of color, density, and volumetric rendering that RelationField extends with object and relationship heads."},{"cited_title":"Lerf: Language embed- 9 ded radiance fields","cited_arxiv_id":null,"evidence_quote":"Introduces the language-embedded feature field and the canonical-phrase softmax querying scheme that RelationField adapts for relationships."},{"cited_title":"OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views","cited_arxiv_id":null,"evidence_quote":"Shows how to distill open-vocabulary CLIP features into a radiance field, the object-level precedent for the relationship head."},{"cited_title":"Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick","cited_arxiv_id":null,"evidence_quote":"Provides the segmentation masks used for Set-of-Mark prompting and for projecting relationship features onto the image plane."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Defines the open-vocabulary object-embedding space that the semantic head is trained to predict."},{"cited_title":"Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships","cited_arxiv_id":null,"evidence_quote":"Is the open-vocabulary point-cloud scene graph method that RelationField compares against and exceeds on 3DSSG."},{"cited_title":"Learning 3d semantic scene graphs from 3d indoor reconstructions","cited_arxiv_id":null,"evidence_quote":"Provides the 3DSSG dataset and benchmark task used for the scene graph evaluation."}],"review_version":1}