{"id":"e3a4adee-c52c-4eb7-86a0-5ddeba78226a","arxiv_id":"2502.05874","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MMGDreamer generates 3D indoor scenes from a mixed-modality scene graph whose nodes can be text, images, or both, and it predicts missing object relationships for more coherent layouts.","lead":"MMGDreamer is a new system that generates 3D indoor scenes from descriptions that mix text and images. It gives designers more precise control over the geometry of furniture and rooms in virtual environments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation protocol gives MMGDreamer access to ground-truth object images at test time, so the claimed geometric control may reflect copying the target geometry rather than generalization.","rationale":"I read the paper as claiming that MMGDreamer enables precise geometry control by allowing scene-graph nodes to carry images, with a dual-branch diffusion model, a visual enhancement module, and a relation predictor. For that claim to hold, the experiments must show that the image-conditioned shape branch generalizes to user-provided images and that the reported gains come from the proposed modules rather than from simply seeing the target object. The reader's weakest assumption was that CLIP features may not carry enough geometric information. I think that concern is real but secondary: the more load-bearing problem is that the evaluation uses ground-truth 3D-FUTURE renders as the image condition, so the shape branch is effectively told which exact object to produce. This is not an accusation of fraud; it is a standard evaluation confound. The paper does contain useful components: the mixed-modality graph is a reasonable extension of scene-graph-conditioned generation, the visual enhancement and relation predictor modules are clearly described, and the quantitative protocol follows prior work. However, the absence of code or data, plus the lack of an image-conditioned baseline, makes it impossible to tell whether the large margins in Tables 1 and 2 reflect the method's actual generalization or the information advantage of ground-truth images. A held-out model-ID split is a concrete, feasible check. If the method maintains high MMD/COV/1-NNA on unseen object instances, the central claim would be substantially supported; if not, the paper should be revised to claim image-conditioned shape copying within the training distribution rather than general geometry-controllable scene generation. I therefore keep the reader's conditional verdict, with the condition made explicit: a clean train/test split by object identity plus an image-conditioned baseline.","tokens_in":18867,"tokens_out":4607,"duration_ms":52350,"concrete_test":"Split SG-FRONT/3D-FUTURE by furniture model ID so that no object instance or its render appears in both training and test, then retrain MMGDreamer (I+R) and MMGDreamer (MM+R) on the training split and report MMD/COV/1-NNA on held-out model IDs. If shape quality degrades toward the text-only baselines or collapses on unseen models, the reported numbers are dominated by ground-truth image identity rather than by the mixed-modality graph. A complementary test is to compare against EchoScene with CLIP image features concatenated to its node embeddings, using the same images, to isolate the contribution of the MMG and its modules.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim of state-of-the-art geometric control rests on Tables 1 and 2, but those comparisons are confounded by the input images. In the Experimental Settings, the authors state that they 'extracted corresponding images from the 3D-FUTURE dataset based on node IDs' to construct the Full-Modality Graph. At test time, MMGDreamer (I+R) and MMGDreamer (MM+R) therefore condition the shape branch on a CLIP embedding of the very 3D-FUTURE object whose shape is used as the ground truth for MMD, COV, and 1-NNA. This gives the shape branch a near-lossless shortcut: the dataset render identifies the exact object instance, so the model can learn to map that render's CLIP embedding to the corresponding shape latent. The text-only baselines have no access to this signal, which alone can explain the large reported gains. The ablation in Table 3 does not remove the confound because every configuration uses the same ground-truth images; the visual enhancement module is only exercised on text-only nodes, but the rest of the graph still supplies ground-truth image conditions. Whether CLIP image features are semantically or geometrically informative is secondary: the current protocol guarantees that the answer is present in the input, so the experiments do not demonstrate generalizable geometry control from flexible user images.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MMGDreamer proposes a dual-branch diffusion model for 3D indoor scene generation from a Mixed-Modality Graph, where nodes can carry text, images, or both, and edges may be partially provided. The method encodes the graph with CLIP and an embedding layer, applies a visual enhancement module (VQ-VAE) to synthesize visual features for text-only nodes, and a relation predictor to infer missing edges. A triplet-GCN graph encoder with an echo mechanism conditions separate layout and shape diffusion branches. Experiments on SG-FRONT report improved FID, KID, MMD, COV, and 1-NNA over Graph-to-3D, CommonScenes, and EchoScene.","tokens_in":19133,"tokens_out":7526,"duration_ms":69740,"significance":"The idea of a mixed-modality scene graph is a useful extension of current text-only graph-based scene generation, and the dual-branch architecture is a reasonable integration of existing components. If the evaluation were fair, the method could offer a practical interface for geometry control. However, the current test protocol gives the model access to ground-truth object images, so the reported state-of-the-art numbers are not trustworthy as evidence of generalizable geometric control. The paper does not provide reproducible code or data, and the reported differences lack statistical validation.","major_comments":[{"comment":"The evaluation protocol is confounded by the test-time use of ground-truth object images. The authors state that they 'extracted corresponding images from the 3D-FUTURE dataset (Fu et al. 2021) based on node IDs to construct a Full-Modality Graph.' At test time, this means the image condition for a node is a render of the exact object instance whose shape serves as the ground truth for the object-level metrics (MMD, COV, 1-NNA) in Table 2. The text-only baselines (Graph-to-3D, CommonScenes, EchoScene) have no access to this identity-preserving signal, so the large reported gains in Tables 1 and 2 can be explained by instance copying rather than by the proposed mixed-modality graph or the visual enhancement module. The ablation in Table 3 does not resolve this issue because every configuration still uses the same ground-truth images for image-conditioned nodes. To support the central claim of state-of-the-art geometric control, the authors need to evaluate with held-out or novel user-provided images (e.g., images not corresponding to any node ID in the test graph) and compare against an image-conditioned baseline that receives the same visual input.","section":"Experimental Settings (Evaluation Dataset) and Table 2"},{"comment":"All quantitative results are reported without error bars or multiple runs. Several improvements over EchoScene are small (e.g., FID_CLIP in bedroom: 4.26 vs 3.84; KID in bedroom: 1.77 vs 1.72), and without variance estimates or a significance test it is unclear whether these differences are meaningful. The paper also does not release code or data, which makes it impossible to verify the results. The authors should report means and standard deviations over at least three random seeds and consider a statistical test for the main comparisons.","section":"Tables 1, 2, and 3"},{"comment":"The visual enhancement module is only evaluated in the full mixed-modality setting, where image-conditioned nodes still carry ground-truth image features. The paper does not isolate the module's contribution on text-only graphs, which is the scenario the module is designed for. It would be informative to report results for (i) text-only graphs with and without the visual enhancement module and (ii) image-only graphs with and without the relation predictor. Without these ablations, it is unclear whether the claimed 'enrichment of visual fidelity' actually improves geometry control beyond the direct conditioning on ground-truth images.","section":"Visual Enhancement Module and Table 3"}],"minor_comments":[{"comment":"The method name 'CommonScenes' is misspelled as 'CommoScenes' in the sentence 'EchoScene (Zhai et al. 2024b) advances the CommonScenes by incorporating...'; please fix this typo and check the supplementary for the same misspelling.","section":"Related Work"},{"comment":"The text says 'For the shape branch, as shown in Fig. 2.C.2' but in Fig. 2, C.2 is the Layout Branch and C.3 is the Shape Branch; the figure reference should be corrected.","section":"Shape and Layout Branch"},{"comment":"The metric 'mSG' (mean scene graph consistency) is used in Table 3 but is not defined in the main text; please define it in the caption or in the 'Evaluation Metrics' paragraph.","section":"Table 3 and Ablation Study"},{"comment":"The prompt in Fig. 5 states that 'only these twelve relationships are allowed' but then lists fifteen relationships; the number should be corrected to match the list.","section":"Supplementary Material (GPT-4V Prompt)"},{"comment":"The main text does not specify which CLIP model is used; the supplementary mentions ViT-B/32, but this should be stated in the main text for completeness.","section":"Implementation Details"},{"comment":"In the supplementary Table 1, under 'Relationship change' mode, MMGDreamer's 'symmetrical' score (0.53) is lower than CommonScenes (0.59), yet the text claims 'a clear superiority in the symmetrical metric compared to other methods'; this overstatement should be corrected.","section":"Ablation Study and Scene Graph Consistency"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the evaluation protocol. The authors should be asked to re-run experiments under a setting that prevents instance-identity leakage (e.g., held-out object images) or to add an image-conditioned baseline that receives the same visual input. Without such a fix, the state-of-the-art claim is not supported. This is a major revision rather than a reject because the method itself is well-motivated and the fix is within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the paper's real contribution is the Mixed-Modality Graph, which lets scene-graph nodes carry text, images, or both. That is a genuine extension of EchoScene/CommonScenes, and the visual enhancement module and relation predictor are sensible additions. But the evaluation is confounded, and the headline numbers shouldn't be trusted as they stand.\n\nWhat the paper does well: the MMG is well-specified, and the dual-branch diffusion setup is a reasonable adaptation of EchoScene. The relation-free generation experiments are a nice qualitative touch, and the writing is clear.\n\nThe soft spot is load-bearing. The authors construct test inputs by extracting images from 3D-FUTURE based on node IDs. For the object-level metrics (MMD, COV, 1-NNA), those images are of the very objects whose shapes are the ground truth. So the shape branch sees a CLIP embedding of the target object at test time, giving it a near-lossless shortcut: it can learn to map that render to the corresponding shape latent. The text-only baselines have no such signal, which alone can explain the large reported gains. The ablation doesn't remove this confound because every configuration uses the same ground-truth images. Whether CLIP features are semantically or geometrically informative is beside the point; the protocol guarantees the answer is present in the input.\n\nOther issues are minor by comparison: no error bars, no code or data release, and the \"remarkable improvement\" language outruns the evidence.\n\nWho is this for? Someone working on scene-graph-based 3D generation will want to know about the MMG representation. But the quantitative claims need rework before they can be cited. A serious referee could engage with this, and the core idea deserves a fair test, but the current paper needs major revision: image-conditioned baselines, held-out or user-provided images, and ideally code/data.\n\nI'd send it to review, with the expectation that the authors fix the protocol. I would not cite the current numbers.","headline":"The mixed-modality graph is a genuine extension of EchoScene, but the evaluation is confounded because test-time images come from the ground-truth objects, so the SOTA claims don't hold as presented.","tokens_in":19679,"tokens_out":2667,"would_cite":false,"duration_ms":25236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMGDreamer claims that a mixed-modality scene graph—where object nodes can be text, images, or both—lets a dual-branch diffusion model control object geometry in generated 3D indoor scenes, outperforming prior graph-based generators on…","keywords":["3D indoor scene generation","mixed-modality graph","scene graphs","diffusion models","geometry control","CLIP features","scene layout generation","shape generation"],"falsifier":"Generate the same scene graph with the same text node but two reference images of the same object category that differ clearly in geometry, such as an armchair and a straight-backed chair, and compare the produced shapes; if the generated shapes are statistically indistinguishable under MMD or Chamfer distance, the visual features are not carrying geometric control.","tokens_in":18670,"feed_emoji":"🛋️","tokens_out":6236,"duration_ms":54381,"temperature":0.7,"pith_summary":"This paper tries to establish that letting scene-graph nodes carry images as well as text gives a 3D indoor scene generator precise control over object geometry, something text-only graphs cannot do. It introduces a Mixed-Modality Graph in which each object node may be a text label, an image, or both, and edges can be omitted, and a dual-branch diffusion model that turns this graph into a bounding-box layout plus per-object signed-distance-field shapes. Two supporting modules carry the load: a visual enhancement module that synthesizes a visual feature for text-only nodes by quantizing their CLIP text embedding, and a relation predictor that fills in missing edges so layouts stay coherent. If the central claim holds, a user could supply a photo of a particular chair or nightstand and the model would place a faithful version of that object into a realistic room.","feed_headline":"Image nodes let 3D room generator match furniture geometry","feed_subtitle":"Scene graphs mixing text and photos drive fine-grained control of object shapes and layouts in generated 3D rooms.","key_machinery":"The central object is the Mixed-Modality Graph (MMG), a directed scene graph whose object nodes can be text-only, image-only, or both, and whose edges are optional relationship labels. The argument runs through three mechanisms: a visual enhancement module that uses a VQ-VAE-style encoder, codebook, and decoder to manufacture a visual feature from a text-only node's CLIP text embedding; a relation predictor, a GCN with MLP heads, that classifies missing edges and turns the graph into a fully connected mixed-enhanced graph; and a dual-branch latent diffusion model whose layout and shape denoisers are both conditioned on node representations produced by a triplet-GCN graph encoder with an echo mechanism. The shape branch is where geometric control is supposed to happen: its 3D-UNet denoiser receives per-node visual features, so the output signed-distance-field shape should track the geometry implied by the input image.","core_discovery":"The central claim is that a scene graph with mixed-modality nodes, combined with a dual-branch latent diffusion generator, achieves state-of-the-art geometry-controllable 3D indoor scene generation. In the paper's construction, each node carries category text, a CLIP image feature, or both, and missing relationships are either inferred by a GCN-based relation predictor or represented by zero padding; the layout branch denoises bounding-box parameters while the shape branch denoises truncated-signed-distance-field latents, with both branches conditioned on graph-encoded node representations that include the visual features. The paper reports that on the SG-FRONT dataset this design outperforms prior text-only graph-to-3D generators on scene-level realism (FID, FID-CLIP, KID), object-level shape fidelity (MMD, COV, 1-NNA), and scene-graph consistency, with the improvements attributed to the visual conditioning and to the relation predictor's role in layout coherence.","pith_inferences":["If CLIP features encode mostly semantic category rather than metric geometry, the apparent geometry control may be a category-conditioned shape prior in disguise; swapping the reference image within a category is a direct way to test this.","The relation predictor is trained on masked SG-FRONT relations, so its inferred edges are limited to the fifteen relationship types in that dataset; novel user-specified relationships outside that vocabulary would have to be mapped to synonyms or would be ignored.","The visual enhancement module is effectively a text-to-visual-feature translator; replacing it with a stronger image prior (e.g., generated reference views) would probably sharpen geometry control for text-only nodes.","Since the paper deliberately excludes texture and material, the same graph structure could later be extended to carry appearance features, giving joint geometry-and-texture control without changing the graph representation."],"forward_implications":["A user can describe some objects in a room with text and others with photos, and omit relationships between them; the model fills in the gaps rather than rejecting the input.","Giving the shape branch a node's image feature should make the generated furniture follow the reference photo's geometry, including proportions and details such as chair backrests.","Because the relation predictor supplies missing edges, the layout branch can still arrange objects coherently even when the user provides no explicit relations.","Scene-graph editing operations—changing a relation or adding a node—propagate to the generated scene, preserving spatial relations such as front/behind and left/right better than text-only baselines in the paper's measurements.","The reported object-level metrics on SG-FRONT improve for most furniture categories, which follows if the visual conditioning is genuinely shaping the generated object geometry."],"supporting_citations":[{"why":"Supplies the SG-FRONT dataset and the dual-branch scene-graph-to-3D baseline that MMGDreamer extends.","marker":"Zhai et al. 2024c"},{"why":"Supplies the echo mechanism used in the graph encoder, the strongest baseline, and the training and evaluation protocol.","marker":"Zhai et al. 2024b"},{"why":"Supplies the GCN-based graph-to-3D baseline using DeepSDF shape representation.","marker":"Dhamo et al. 2021"},{"why":"Supplies the frozen CLIP text and image encoders that produce node features and the visual features to be reconstructed.","marker":"Radford et al. 2021"},{"why":"Supplies the latent diffusion formulation and denoising objective used by both branches.","marker":"Rombach et al. 2022"},{"why":"Supplies the VQ-VAE codebook and quantization architecture used in the visual enhancement module and shape latent encoding.","marker":"Van Den Oord, Vinyals et al. 2017"},{"why":"Supplies the 3D-FUTURE furniture meshes and textures whose renderings become image nodes in the mixed-modality graphs.","marker":"Fu et al. 2021"},{"why":"Supplies the MMD, COV, and 1-NNA object-level evaluation metrics based on Chamfer distance.","marker":"Yang et al. 2019"}],"fun_headline_variants":["Mixed-modality graph gives fine-grained 3D room geometry control","Image-aware scene graphs boost 3D room geometry control","MMGDreamer: mixed-modality graph for precise 3D room generation","Text and image nodes enhance 3D scene geometry control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that CLIP image features, and the text-derived visual features the enhancement module fabricates, carry enough geometric information about an object to steer the shape denoiser; if they only capture semantic category, the paper's claimed geometry control would collapse.","fun_headline_variants_meta":{"raw":{"variants":["Mixed-modality graph gives fine-grained 3D room geometry control","Image-aware scene graphs boost 3D room geometry control","MMGDreamer: mixed-modality graph for precise 3D room generation","Text and image nodes enhance 3D scene geometry control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3151,"prompt_tokens":969,"completion_tokens":2182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2108}},"tokens_in":585,"tokens_out":2182,"duration_ms":16774,"temperature":1.0,"reasoning_tokens":2108,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:34:05.965513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate the same scene graph with the same text node but two reference images of the same object category that differ clearly in geometry, such as an armchair and a straight-backed chair, and compare the produced shapes; if the generated shapes are statistically indistinguishable under MMD or Chamfer distance, the visual features are not carrying geometric control.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GCN-based graph-to-3D baseline using DeepSDF shape representation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MMD, COV, and 1-NNA object-level evaluation metrics based on Chamfer distance."}],"review_version":1}