Pith. sign in

REVIEW 4 major objections 5 minor 79 references

FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FreeScene claims a single diffusion model, MG-DiT, can synthesize 3D indoor scenes from free-form text and images by first extracting a partial scene graph with a VLM and then denoising around that graph, outperforming prior methods…

desk verdict A solid, well-engineered combination of VLM-based graph parsing and mixed graph diffusion for 3D scenes; the results are consistently positive but the parser evaluation is too thin to fully support the free-prompt controllability claim. read the letter →

arxiv 2506.02781 v1 pith:X7VE6YWR submitted 2025-06-03 cs.CV

Tongyuan Bai , Wangyuanfan Bai , Dong Chen , Tieru Wu , Manyi Li , Rui Ma This is my paper
classification cs.CV
keywords 3Dindoorscenesynthesisgraphdiffusionmodelscontrollablegenerationvision-languagetext-to-scenegraph-to-scenelayout
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FreeScene is a framework for 3D indoor scene synthesis that claims to remove the trade-off between convenience and control: users can type a sentence, draw a sketch, show a photo, or combine them, and the system builds a room layout that follows the request. The key move is to insert an intermediate representation — a partial scene graph listing objects and their spatial relations — extracted automatically by a vision-language model, and then to generate the complete scene with a diffusion transformer that keeps that graph fixed while denoising. The paper argues this unifies tasks that previously needed separate models, including text-to-scene, graph-to-scene, rearrangement, completion, and stylization, and reports that it outperforms ATISS, DiffuScene, and InstructScene on quality and controllability metrics.

What carries the argument

The central mechanism is the partial scene graph acting as a condition that survives the entire denoising trajectory. The Graph Designer turns free-form inputs into an object list and an edge matrix of spatial relation triplets; MG-DiT then runs a mixed diffusion process in which continuous variables (size, position, orientation) are denoised with a DDPM-style Gaussian process while discrete variables (object category, quantized fVQ-VAE feature indices, relation labels) follow a D3PM-style masked categorical process, with FiLM used to pass edge information into node updates. Constrained sampling freezes whichever variables correspond to the user's condition, which is what lets one model cover many applications.

What would settle it

Run the Graph Designer on a held-out set of cluttered rooms (say, scenes with more than ten objects) and compare its object list and relation triplets against the ground-truth scene graph; the paper's controllability claim weakens if object iRecall or relation accuracy collapses as object count grows. A second check: feed identical ground-truth graphs to MG-DiT and to InstructScene's graph-to-scene model and see whether the reported iRecall advantage persists when graph-extraction noise is removed.

Watch

Extended reading notes

Core claim

The paper's central claim is that scene-graph control and free-form prompting can be combined in one diffusion model without retraining per task. Concretely, FreeScene comprises a VLM-based Graph Designer that parses user text and/or images into a partial graph (object category list, relation triplets, and a global description) using one-shot chain-of-thought prompting, and MG-DiT, a Mixed Graph Diffusion Transformer that jointly denoises continuous bounding-box attributes and discrete graph and object-feature tokens. During sampling, constrained denoising fixes the user-supplied or graph-extracted variables, so the same trained model performs text-to-scene, graph-to-scene, rearrangement, completion, stylization, and unconditioned generation. The paper claims this is the first approach to handle all these tasks with a single diffusion model, and reports lower FID, FID-CLIP, KID, and higher iRecall than the compared baselines.

Load-bearing premise

The whole pipeline relies on the vision-language model extracting a sufficiently complete and correct partial scene graph from whatever the user provides; if that extraction misses objects or gets spatial relations wrong, the generated room will not match the user's intent even if the diffusion model itself is strong.

Editorial extensions

If this is right

  • Text-to-scene generation becomes a graph-conditioned generation problem, and the paper reports that feeding the text through the Graph Designer first (Ours+GD) improves controllability metrics (iRecall) over direct text-conditioned diffusion.
  • Graph-to-scene generation can be done zero-shot by fixing the supplied graph, with no extra training.
  • A single checkpoint can perform rearrangement, completion, stylization, and unconditioned generation by fixing different subsets of variables during sampling.
  • The unified model means user interfaces can mix modalities: a photo plus a text edit (e.g., 'add a wardrobe to the right of the bed') is handled by the same pipeline.
  • The fVQ-VAE-compressed OpenCLIP features let the pipeline retrieve matching 3D furniture models from 3D-FUTURE, connecting layout generation to shape retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the VLM-based graph parser improves with stronger VLMs, FreeScene's ceiling is mostly set by the graph extraction step rather than the diffusion model, so future gains may come from better grounding rather than better layout generation.
  • The fixed category lists per room type mean the approach currently operates in closed-vocabulary territory; an extension would be to let the graph designer propose novel objects and relax the category constraint.
  • Because the diffusion model cannot precisely control exact positions and orientations (a stated limitation), the pipeline is better suited to plausible approximate layout than to strict geometric constraints; a testable extension would add interaction or affordance constraints to the diffusion loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. FreeScene proposes a two-stage framework for controllable 3D indoor scene synthesis. A VLM-based Graph Designer (GPT-4o with one-shot chain-of-thought prompting) converts free-form text and/or images (photos, sketches, top-view diagrams) into a partial scene graph consisting of object categories, spatial-relation triplets, and a textual description. This graph is then fed as a partial condition to MG-DiT, a mixed discrete/continuous diffusion transformer that jointly denoises bounding-box attributes (size, position, orientation), discrete object categories, fVQ-VAE feature indices, and edge relation categories. Constrained sampling during inference supports text-to-scene, graph-to-scene, rearrangement, completion, stylization, and unconditioned generation within a single model. The authors evaluate on 3D-FRONT bedrooms, living rooms, and dining rooms, comparing against ATISS, DiffuScene, and InstructScene across FID, FID_CLIP, KID, SCA, and iRecall, and report a supplementary user study. The main claims are state-of-the-art generation quality and controllability, with the Graph Designer providing free-form controllability.

Significance. If the reported results hold, FreeScene is a useful step toward unifying coarse and fine-grained control in 3D indoor scene synthesis: the mixed discrete/continuous diffusion formulation is a reasonable extension of MiDiffusion, and the constrained-sampling interface is elegant. The idea of using a VLM to translate free-form multimodal input into a structured graph prior is practical and could be adopted by follow-up work. The paper also provides a clear quantitative comparison against three strong baselines, and the CoT-vs-no-CoT ablation in Table 2 is informative. However, the central 'free-form controllability' claim rests on the Graph Designer, whose evaluation is thin (no sample size, no ground-truth protocol, ambiguous metrics), and the graph-to-scene SOTA claim omits several relevant graph-conditioned baselines. These gaps are load-bearing because the reported gains in Tables 3-5 and the user study all pass through either the Graph Designer or the MG-DiT graph-conditioning interface. The manuscript is technically sound in its core diffusion derivation, but the evidence currently does not fully support the advertised claims of superiority and open-vocabulary controllability.

major comments (4)
  1. [§5.1, Table 2, Supplementary Fig. 3] The Graph Designer is the linchpin of the free-form controllability claim, yet its evaluation is not sufficient to support that claim. Table 2 reports only with-CoT versus without-CoT numbers for Object iRecall and Rel Acc, but does not state the number of test inputs per modality, the ground-truth annotation protocol, the inter-annotator agreement, or the absolute accuracy relative to a human-annotated reference. The metric definition is also ambiguous: 'the ratio of correctly identified objects w.r.t. the total number of objects named' leaves unclear what denominator is used (ground-truth objects, VLM-named objects, or their union). More importantly, the prompt template in Supplementary Fig. 3 explicitly instructs the VLM to ignore objects outside a fixed per-room category list (21-24 classes), and the Limitations section admits that the VLM 'may struggle to capture all the graph priors, especially in scenes with a large number of objects.' Because MG-DiT is trained to complete a scene around the supplied partial graph, any object or relation dropped at parsing time is unrecoverable. Thus the Ours+GD improvements in Tables 3-5 and the qualitative 'free prompt' results are not yet rigorously supported. I request a proper Graph Designer evaluation: a defined test set with human-annotated ground truth, sample size per modality, precision/recall for objects and relations, and a discussion of out-of-vocabulary handling.
  2. [§5.3, Table 4] The claim of state-of-the-art graph-to-scene generation is under-supported because the only graph-conditioned baseline is InstructScene. The Related Work section discusses several graph-conditioned methods -- EchoScene, CommonScenes, SceneGraphNet, PlanIT, and Graph-to-3D -- that are directly relevant to this task, but none are included in Table 4. The sentence 'Since ATISS and DiffuScene do not involve graph priors, we only compared the graph-conditioned results of MG-DiT with InstructScene' does not justify omitting those graph-based baselines. At minimum, EchoScene, which is a recent diffusion-based scene-graph-conditioned method evaluated on 3D-FRONT, should be compared. Without these comparisons, the 'outperforming state-of-the-art' claim for graph-to-scene is not established.
  3. [§5.2-5.4, Tables 3-5] The quantitative comparisons are reported as point estimates without error bars, confidence intervals, or significance tests. Several differences are small relative to the metric scale, e.g., Table 3 living room: Ours FID 110.55 vs. InstructScene 111.52, and KID 7.95 vs. 8.65; Table 4 dining room: Ours FID 122.57 vs. InstructScene 123.82. Given that FID/KID are known to be noisy on limited test sets, these differences may not be statistically meaningful. The paper should report the number of test scenes, compute metrics over multiple seeds, and provide confidence intervals or a paired significance test (e.g., bootstrap or permutation) for the headline comparisons. This is especially important because the central narrative is 'FreeScene outperforms state-of-the-art methods,' which the current tables do not yet substantiate with statistical rigor.
  4. [§4.3, Eq. (12)] The loss in Eq. (12) appears to have an omission relative to the posterior factorization in Eq. (11). The network is defined as p_theta(b_{t-1} | b_t, z_t, y), so the noise prediction epsilon_theta should depend on the discrete noisy variables z_t as well as b_t and y. However, Eq. (12) writes epsilon_theta(sqrt(alpha_bar_t) b_0 + sqrt(1-alpha_bar_t) epsilon_b, t, y) with no z_t argument. If the loss intentionally conditions only on b_t for the continuous part, please state that explicitly; otherwise, correct Eq. (12) so that the training objective matches the architecture described in Section 4.3 and Figure 4.
minor comments (5)
  1. [Table 1] The header line 'Var . c e s[t, o]v' is garbled. Please spell out the columns explicitly, e.g., 'Variable: c, v, b=[s,t,o], e' with the corresponding fixation modes, so the reader can parse the table without inference.
  2. [§5.1, Table 2] The metric 'Object iRecall' is ambiguously defined. Clarify whether the denominator is the number of objects in the ground-truth input, the number of objects returned by the VLM, or the number of correctly matched objects divided by the union. This matters for interpreting the numbers in Table 2.
  3. [Supplementary, B.3, Table 6] The user study is reported only in the supplementary material and is not mentioned in the main text's experimental sections. Since the paper claims superiority in controllability, consider adding a summary of the user study (number of participants, number of scenes, criteria, and Top1/Top2 percentages) to the main paper, including a statistical significance test if possible.
  4. [Abstract and §1] The phrase 'the first approach capable of addressing all these tasks with a unified diffusion model' is strong. Since MiDiffusion already uses mixed diffusion for layout synthesis and InstructScene uses graph priors for several tasks, please temper the novelty claim or provide a clearer distinction of what new capability is being introduced beyond combining existing components.
  5. [Supplementary, Fig. 3] The prompt template in Supplementary Fig. 3 contains minor typographical issues (e.g., misplaced spaces in object names such as 'double _bed' and 'coffee_table'). These do not affect the method, but please clean up the template for reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: FreeScene's core derivation is self-contained; Graph Designer is an external VLM parse, MG-DiT uses standard diffusion objectives, and the self-citations are not load-bearing.

full rationale

The paper's derivation chain is: free-form inputs -> Graph Designer (GPT-4o) -> partial graph -> MG-DiT denoising (Eq. 12) -> scene layout. No equation defines the target in terms of a fitted parameter, and no predicted quantity is constructed to equal an input by definition. The fVQ-VAE codebook is borrowed from InstructScene [28] as an external pretrained component, not re-derived to match FreeScene's outputs. The constrained-sampling procedure fixes graph nodes and edges, but this is an explicit algorithmic condition rather than a hidden fit; the iRecall metric still requires the sampled continuous bounding boxes to realize the fixed relations, so the evaluation is not tautological. The only self-citation ([32], related work on language-driven scene synthesis) is a background reference and plays no role in the method, loss, or experimental comparisons. The under-specified Graph Designer evaluation in Table 2 (no sample size, no annotation protocol) is an empirical validation gap and a correctness/risk concern, not circular reasoning. Therefore, the central claims do not reduce to their inputs by construction, and the paper is self-contained against the external 3D-FRONT benchmark and standard metrics.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical system result. It depends on hand-chosen training probabilities and network hyperparameters, on the suitability of the 3D-FRONT dataset and OpenCLIP features, and on the reliability of the proprietary GPT-4o parser. No new physical or conceptual entities are introduced.

free parameters (3)
  • Training condition mixing probabilities = 0.2 text dropout, 0.2 graph preservation, 0.1 edges-only, 0.5 denoise all
    Hand-chosen in Section A.2; they control how strongly the model learns to condition on text and graph, and no sensitivity study is provided.
  • Network size = 5 blocks, 8 heads, 512 dim, dropout 0.1
    Architecture choices stated in Section A.2; results may depend on these numbers.
  • Diffusion timesteps T = unspecified
    The number of diffusion steps is never stated, but both the loss and sampling process depend on it.
assumptions (4)
  • domain assumption The 3D-FRONT dataset provides a sufficient and representative distribution of indoor scenes for training and evaluation.
    All experiments train and test on 3D-FRONT; no evidence is given that results transfer to other scene distributions.
  • domain assumption OpenCLIP features, quantized by the fVQ-VAE, capture appearance similarity for object retrieval.
    Borrowed from InstructScene [28]; the paper does not validate this choice for retrieval quality.
  • domain assumption GPT-4o with the one-shot CoT prompt reliably extracts structured graphs from arbitrary text and images.
    Central to Graph Designer; only partially tested, and the authors list VLM limitations in the Limitations section.
  • standard math Standard DDPM and D3PM formulations are valid for this mixed continuous/discrete setting.
    Used to derive the loss in Section 3 and Eq. 12; standard results from DDPM [17] and D3PM [1].

how reviews work

0 comments
Cite this review

Pith. "Pith review of FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts." pith.science (2026). https://pith.science/paper/X7VE6YWR

@misc{pith2026250602781,
  author       = {Pith},
  title        = {Pith review of: FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7VE6YWR}},
  note         = {Machine review of arXiv:2506.02781}
}
read the original abstract

Controllability plays a crucial role in the practical applications of 3D indoor scene synthesis. Existing works either allow rough language-based control, that is convenient but lacks fine-grained scene customization, or employ graph based control, which offers better controllability but demands considerable knowledge for the cumbersome graph design process. To address these challenges, we present FreeScene, a user-friendly framework that enables both convenient and effective control for indoor scene synthesis.Specifically, FreeScene supports free-form user inputs including text description and/or reference images, allowing users to express versatile design intentions. The user inputs are adequately analyzed and integrated into a graph representation by a VLM-based Graph Designer. We then propose MG-DiT, a Mixed Graph Diffusion Transformer, which performs graph-aware denoising to enhance scene generation. Our MG-DiT not only excels at preserving graph structure but also offers broad applicability to various tasks, including, but not limited to, text-to-scene, graph-to-scene, and rearrangement, all within a single model. Extensive experiments demonstrate that FreeScene provides an efficient and user-friendly solution that unifies text-based and graph based scene synthesis, outperforming state-of-the-art methods in terms of both generation quality and controllability in a range of applications.

Figures

Figures reproduced from arXiv: 2506.02781 by the authors.

Figure 1
Figure 1. We introduce FreeScene, a user-friendly controllable indoor scene synthesis framework that allows convenient and effective [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of FreeScene. Given a reference image, text, or either of them, a carefully designed agent called the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Graph Designer. The Graph Designer utilizes a one [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Mixed Graph Diffusion Transformer. MG-DiT condi [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons on text-to-scene generation. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 1
Figure 1. Figure 1: Category lists for different room types. [PITH_FULL_IMAGE:figures/full_fig_p012_1.png]
Figure 2
Figure 2. Figure 2: A comparison of the results between one-shot CoT [PITH_FULL_IMAGE:figures/full_fig_p012_2.png]
Figure 3
Figure 3. Figure 3: One-shot CoT prompt template for bedroom. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]
Figure 4
Figure 4. Figure 4: Qualitative comparisons on text-to-scene generation. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons on graph-to-scene generation. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparisons on completion. Note that, in this task, the main goal is to complete the scene by considering the given [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparisons on stylization [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 58 canonical work pages

  1. [1]

    Structured denoising dif- fusion models in discrete state-spaces.Advances in Neural Information Processing Systems, 34:17981–17993, 2021

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tar- low, and Rianne Van Den Berg. Structured denoising dif- fusion models in discrete state-spaces.Advances in Neural Information Processing Systems, 34:17981–17993, 2021. 3

    work page 2021

  2. [2]

    Demystifying mmd gans

    Mikołaj Bi ´nkowski, Danica J Sutherland, Michael Ar- bel, and Arthur Gretton. Demystifying mmd gans. arXiv:1801.01401, 2018. 6

    arXiv 2018

  3. [3]

    Interactive learning of spatial knowledge for text to 3d scene generation

    Angel Chang, Manolis Savva, and Christopher D Manning. Interactive learning of spatial knowledge for text to 3d scene generation. InProceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, pages 14– 21, 2014. 2

    work page 2014

  4. [4]

    Learning spatial knowledge for text to 3d scene generation

    Angel Chang, Manolis Savva, and Christopher D Manning. Learning spatial knowledge for text to 3d scene generation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 2028–2038, 2014

    work page 2014

  5. [5]

    Text to 3D Scene Generation with Rich Lexical Grounding

    Angel Chang, Will Monroe, Manolis Savva, Christopher Potts, and Christopher D Manning. Text to 3d scene genera- tion with rich lexical grounding.arXiv:1505.06289, 2015

    work page Pith review arXiv 2015

  6. [6]

    Sceneseer: 3d scene design with natural language.arXiv:1703.00050, 2017

    Angel X Chang, Mihail Eric, Manolis Savva, and Christo- pher D Manning. Sceneseer: 3d scene design with natural language.arXiv:1703.00050, 2017. 2

    arXiv 2017

  7. [7]

    Graph-to-3d: End-to-end generation and ma- nipulation of 3d scenes using scene graphs

    Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Fed- erico Tombari. Graph-to-3d: End-to-end generation and ma- nipulation of 3d scenes using scene graphs. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 16352–16361, 2021. 2

    work page 2021

  8. [8]

    A generalization of transformer networks to graphs.arXiv:2012.09699, 2020

    Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs.arXiv:2012.09699, 2020. 6

    arXiv 2012

Show all 79 references
  1. [9]

    Example-based synthesis of 3d object arrangements.ACM Transactions on Graphics (TOG), 31(6):1–11, 2012

    Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example-based synthesis of 3d object arrangements.ACM Transactions on Graphics (TOG), 31(6):1–11, 2012. 2

    2012

  2. [11]

    3d-front: 3d furnished rooms with layouts and semantics

    Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Bin- qiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 10933–10942,

  3. [12]

    3d-future: 3d fur- niture shape with texture.International Journal of Computer Vision, 129:3313–3337, 2021

    Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d fur- niture shape with texture.International Journal of Computer Vision, 129:3313–3337, 2021. 4, 6

    2021

  4. [13]

    Any- home: Open-vocabulary generation of structured and tex- tured 3d homes

    Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. Any- home: Open-vocabulary generation of structured and tex- tured 3d homes. InEuropean Conference on Computer Vi- sion, pages 52–70. Springer, 2025. 2

    2025

  5. [14]

    Lin Gao, Jia-Mu Sun, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Jie Yang. Scenehgn: Hierarchical graph net- works for 3d indoor scene generation with fine-grained ge- ometry.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8902–8919, 2023. 2

    2023

  6. [15]

    Generative adversarial nets.Advances in Neural Information Processing Systems, 27, 2014

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in Neural Information Processing Systems, 27, 2014. 2

    2014

  7. [16]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in Neural Information Processing Systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in Neural Information Processing Systems, 30, 2017. 6

    2017

  8. [17]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 2, 3, 5

    2020

  9. [18]

    Mixed dif- fusion for 3d indoor scene synthesis.arXiv:2405.21066,

    Siyi Hu, Diego Martin Arroyo, Stephanie Debats, Fabian Manhardt, Luca Carlone, and Federico Tombari. Mixed dif- fusion for 3d indoor scene synthesis.arXiv:2405.21066,

    arXiv

  10. [19]

    Cooperative holistic scene understanding: Unifying 3d object, layout and camera pose estimation

    Siyuan Huang, Siyuan Qi, Yinxue Xiao, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Cooperative holistic scene understanding: Unifying 3d object, layout and camera pose estimation. InNeurIPS, 2018. 2

    2018

  11. [20]

    Layoutdm: Discrete diffusion model for controllable layout generation

    Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10167–10176, 2023. 3

    2023

  12. [21]

    Hamid Izadinia, Qi Shan, and Steven M Seitz. Im2cad. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5134–5143, 2017. 2

    2017

  13. [22]

    Learning 3d scene synthesis from annotated rgb-d images

    Z Sadeghipour Kermani, Zicheng Liao, Ping Tan, and Hao Zhang. Learning 3d scene synthesis from annotated rgb-d images. InComputer Graphics Forum, pages 197–206. Wi- ley Online Library, 2016. 2

    2016

  14. [23]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv:1312.6114, 2013. 2

    2013 arXiv

  15. [24]

    The role of imagenet classes in fr’echet inception distance.arXiv:2203.06026, 2022

    Tuomas Kynk ¨a¨anniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr’echet inception distance.arXiv:2203.06026, 2022. 6

    2022 arXiv

  16. [25]

    Sketch-based search and composition of 3d models

    Jeehyung Lee and Thomas A Funkhouser. Sketch-based search and composition of 3d models. InSBIM, pages 97– 104, 2008. 2

    2008

  17. [26]

    Grains: Generative re- cursive autoencoders for indoor scenes.ACM Transactions on Graphics (TOG), 38(2):1–16, 2019

    Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. Grains: Generative re- cursive autoencoders for indoor scenes.ACM Transactions on Graphics (TOG), 38(2):1–16, 2019. 2

    2019

  18. [27]

    Interiornet: Mega- scale multi-sensor photo-realistic indoor scenes dataset

    Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, and Stefan Leutenegger. Interiornet: Mega- scale multi-sensor photo-realistic indoor scenes dataset. arXiv:1809.00716, 2018. 2

    2018 arXiv

  19. [28]

    Instructscene: Instruction- driven 3d indoor scene synthesis with semantic graph prior

    Chenguo Lin and Yadong Mu. Instructscene: Instruction- driven 3d indoor scene synthesis with semantic graph prior. arXiv:2402.04717, 2024. 2, 3, 4, 6, 8

    2024 arXiv

  20. [29]

    Graph diffusion transformers for multi-conditional molecular gen- eration.arXiv:2401.13858, 2024

    Gang Liu, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph diffusion transformers for multi-conditional molecular gen- eration.arXiv:2401.13858, 2024. 3

    2024 arXiv

  21. [30]

    Openshape: Scaling up 3d shape representation towards open-world understanding.Advances in neural information processing systems, 36, 2024

    Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xu- anlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding.Advances in neural information processing systems, 36, 2024. 4

    2024

  22. [31]

    Creating consistent scene graphs using a probabilistic grammar.ACM Transactions on Graphics (TOG), 33(6):1–12, 2014

    Tianqiang Liu, Siddhartha Chaudhuri, Vladimir G Kim, Qix- ing Huang, Niloy J Mitra, and Thomas Funkhouser. Creating consistent scene graphs using a probabilistic grammar.ACM Transactions on Graphics (TOG), 33(6):1–12, 2014. 2

    2014

  23. [32]

    Language-driven synthe- sis of 3d scenes from scene databases.ACM Transactions on Graphics (TOG), 37(6):1–16, 2018

    Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, S¨oren Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang. Language-driven synthe- sis of 3d scenes from scene databases.ACM Transactions on Graphics (TOG), 37(6):1–16, 2018. 2

    2018

  24. [33]

    Debara: Denoising-based 3d room arrangement generation.arXiv:2409.18336, 2024

    L ´eopold Maillard, Nicolas Sereyjol-Garros, Tom Durand, and Maks Ovsjanikov. Debara: Denoising-based 3d room arrangement generation.arXiv:2409.18336, 2024. 2

    2024 arXiv

  25. [34]

    Interactive furniture layout using in- terior design guidelines.ACM transactions on graphics (TOG), 30(4):1–10, 2011

    Paul Merrell, Eric Schkufza, Zeyang Li, Maneesh Agrawala, and Vladlen Koltun. Interactive furniture layout using in- terior design guidelines.ACM transactions on graphics (TOG), 30(4):1–10, 2011. 2

    2011

  26. [35]

    Total3dunderstanding: Joint lay- out, object pose and mesh reconstruction for indoor scenes from a single image

    Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang. Total3dunderstanding: Joint lay- out, object pose and mesh reconstruction for indoor scenes from a single image. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  27. [36]

    Atiss: Autoregres- sive transformers for indoor scene synthesis.Advances in Neural Information Processing Systems, 34:12013–12026,

    Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregres- sive transformers for indoor scene synthesis.Advances in Neural Information Processing Systems, 34:12013–12026,

  28. [37]

    Advances in data-driven analysis and synthesis of 3d indoor scenes.Com- put

    Akshay Gadi Patil, Supriya Gadi Patil, Manyi Li, Matthew Fisher, Manolis Savva, and Haotong Zhang. Advances in data-driven analysis and synthesis of 3d indoor scenes.Com- put. Graph. Forum, 43(1), 2024. 2

    2024

  29. [38]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  30. [39]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Du- moulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. InProceedings of the AAAI con- ference on artificial intelligence, 2018. 6

    2018

  31. [40]

    Sparse training of discrete diffusion models for graph generation

    Yiming Qin, Clement Vignac, and Pascal Frossard. Sparse training of discrete diffusion models for graph generation. arXiv:2311.02142, 2023. 3

    2023 arXiv

  32. [41]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  33. [42]

    Fast and flex- ible indoor scene synthesis via deep convolutional genera- tive models

    Daniel Ritchie, Kai Wang, and Yu-an Lin. Fast and flex- ible indoor scene synthesis via deep convolutional genera- tive models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6182– 6190, 2019. 2

    2019

  34. [43]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. InMedical image computing and computer-assisted intervention, pages 234–241, 2015. 2

    2015

  35. [44]

    Real-time automatic 3d scene generation from natural language voice and text de- scriptions

    Lee M Seversky and Lijun Yin. Real-time automatic 3d scene generation from natural language voice and text de- scriptions. InProceedings of the 14th ACM international conference on Multimedia, pages 61–64, 2006. 2

    2006

  36. [45]

    Magic canvas: interactive design of a 3-d scene prototype from freehand sketches

    HyoJong Shin and Takeo Igarashi. Magic canvas: interactive design of a 3-d scene prototype from freehand sketches. In Proceedings of graphics interface 2007, pages 63–70, 2007. 2

    2007

  37. [46]

    Semantic scene com- pletion from a single depth image

    Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Mano- lis Savva, and Thomas Funkhouser. Semantic scene com- pletion from a single depth image. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 1746–1754, 2017. 2

    2017

  38. [47]

    Generative modeling by esti- mating gradients of the data distribution.Advances in neural information processing systems, 32, 2019

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution.Advances in neural information processing systems, 32, 2019. 2, 3

    2019

  39. [48]

    Score-based generative modeling through stochastic differential equa- tions.arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions.arXiv:2011.13456, 2020. 2, 3

    2011 arXiv

  40. [49]

    Diffuscene: Denoising diffu- sion models for generative indoor scene synthesis

    Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Denoising diffu- sion models for generative indoor scene synthesis. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20507–20518, 202...

    2024

  41. [50]

    Di- gress: Discrete denoising diffusion for graph generation

    Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bo- han Wang, V olkan Cevher, and Pascal Frossard. Di- gress: Discrete denoising diffusion for graph generation. arXiv:2209.14734, 2022. 6

    2022 arXiv

  42. [51]

    Midi: Mixed graph and 3d denoising diffusion for molecule generation

    Clement Vignac, Nagham Osman, Laura Toni, and Pascal Frossard. Midi: Mixed graph and 3d denoising diffusion for molecule generation. InJoint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 560–576. Springer, 2023. 3

    2023

  43. [52]

    Deep convolutional priors for indoor scene syn- thesis.ACM Transactions on Graphics (TOG), 37(4):1–14,

    Kai Wang, Manolis Savva, Angel X Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene syn- thesis.ACM Transactions on Graphics (TOG), 37(4):1–14,

  44. [53]

    Planit: Planning and in- stantiating indoor scenes with relation graph and spatial prior networks.ACM Transactions on Graphics (TOG), 38(4):1– 15, 2019

    Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, An- gel X Chang, and Daniel Ritchie. Planit: Planning and in- stantiating indoor scenes with relation graph and spatial prior networks.ACM Transactions on Graphics (TOG), 38(4):1– 15, 2019. 2

    2019

  45. [54]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 4

    2022

  46. [55]

    Qi Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas J. Guibas. Lego- net: Learning regular rearrangements of objects in rooms. 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 19037–19047, 2023. 2, 3

    2023

  47. [56]

    Constraint- based automatic placement for scene composition

    Ken Xu, James Stewart, and Eugene Fiume. Constraint- based automatic placement for scene composition. InGraph- ics Interface, pages 25–34. Citeseer, 2002. 2

    2002

  48. [57]

    Sketch2scene: Sketch-based co-retrieval and co- placement of 3d models.ACM Transactions on Graphics (TOG), 32(4):1–15, 2013

    Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi- Min Hu. Sketch2scene: Sketch-based co-retrieval and co- placement of 3d models.ACM Transactions on Graphics (TOG), 32(4):1–15, 2013. 2

    2013

  49. [58]

    Scene synthesis via uncertainty-driven attribute syn- chronization

    Haitao Yang, Zaiwei Zhang, Siming Yan, Haibin Huang, Chongyang Ma, Yi Zheng, Chandrajit Bajaj, and Qixing Huang. Scene synthesis via uncertainty-driven attribute syn- chronization. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5630–5640, 2021. 2

    2021

  50. [59]

    Indoor scene generation from a collection of semantic- segmented depth images

    Ming-Jia Yang, Yu-Xiao Guo, Bin Zhou, and Xin Tong. Indoor scene generation from a collection of semantic- segmented depth images. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15203– 15212, 2021. 2

    2021

  51. [60]

    Physcene: Physically interactable 3d scene synthe- sis for embodied ai

    Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthe- sis for embodied ai. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 16262–16272, 2024. 2, 3

    2024

  52. [61]

    Holodeck: Language guided gen- eration of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Al- varo Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided gen- eration of 3d embodied ai environments. InProceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  53. [62]

    Make it home: automatic optimization of furniture arrangement

    Lap Fai Yu, Sai Kit Yeung, Chi Keung Tang, Demetri Terzopoulos, Tony F Chan, and Stanley J Osher. Make it home: automatic optimization of furniture arrangement. ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAPH 2011, v. 30,(4), July 2011, article no. 86, 30(4),

    2011

  54. [63]

    The clutterpalette: An interactive tool for detailing indoor scenes

    Lap-Fai Yu, Sai-Kit Yeung, and Demetri Terzopoulos. The clutterpalette: An interactive tool for detailing indoor scenes. IEEE transactions on visualization and computer graphics, 22(2):1138–1148, 2015. 2

    2015

  55. [64]

    Echoscene: Indoor scene generation via information echo over scene graph diffusion

    Guangyao Zhai, Evin Pinar ¨Ornek, Dave Zhenyu Chen, Ruo- tong Liao, Yan Di, Nassir Navab, Federico Tombari, and Benjamin Busam. Echoscene: Indoor scene generation via information echo over scene graph diffusion. InProceedings of the European Conference on Computer Vision (ECCV),

  56. [65]

    Commonscenes: Generating commonsense 3d indoor scenes with scene graphs.Advances in Neural Information Process- ing Systems, 36, 2024

    Guangyao Zhai, Evin Pınar ¨Ornek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graphs.Advances in Neural Information Process- ing Systems, 36, 2024. 2

    2024

  57. [66]

    Fast 3d indoor scene synthesis by learning spatial relation priors of objects

    Song-Hai Zhang, Shao-Kui Zhang, Wei-Yu Xie, Cheng- Yang Luo, Yong-Liang Yang, and Hongbo Fu. Fast 3d indoor scene synthesis by learning spatial relation priors of objects. IEEE Transactions on Visualization and Computer Graph- ics, 28(9):3082–3092, 2021. 2

    2021

  58. [67]

    Deep generative modeling for scene synthesis via hybrid represen- tations.ACM Transactions on Graphics (TOG), 39(2):1–21,

    Zaiwei Zhang, Zhenpei Yang, Chongyang Ma, Linjie Luo, Alexander Huth, Etienne V ouga, and Qixing Huang. Deep generative modeling for scene synthesis via hybrid represen- tations.ACM Transactions on Graphics (TOG), 39(2):1–21,

  59. [68]

    Scenegraphnet: Neural message passing for 3d indoor scene augmentation

    Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 7384–7392,

  60. [70]

    Upon receiving the triplet list output from the Graph De- signer, we construct a specialized symmetric matrix where each pair of symmetric positions stores opposite spatial re- lationships, such as ‘Left of’ and ‘Right of’. Bedroom: [0:'armchair', 1:'bookshelf', 2:'cabinet', 3...

  61. [71]

    **Viewpoint Determination**: The given image is a front-view colored image of a bedroom

  62. [72]

    **Object Identification**: - **Object 0**: Double bed, category 8 - **Object 1**: Dressing table, category 10 - **Object 2**: Dressing chair, category 9 - **Object 3**: Nightstand, category 12 - **Object 4**: Pendant lamp, category 13 - **Object 5**: Wardrobe (added per user r...

  63. [73]

    **DFS Traversal**: - Node0 (root) = Double bed (object 0)→ childlist: Dressing table (object 1), Nightstand (object 3), Wardrobe (object 5) - Node1(child of Node0) = Dressing table (object 1)→ childlist: Dressing chair (object 2) - Node2(child of Node0) = Nightstand (object 3)...

  64. [74]

    **Relationship Extraction**: - **Dressing table (object 1)** **is left of (relationship 1)** **Double bed (object 0)**→ [1, 1, 0] - **Dressing chair (object 2)** **is closely in front of (relationship 4)** **Dressing table (object 1)**→ [2, 4, 1] - **Nightstand (object 3)** **...

  65. [75]

    A wardrobe is added to the right of the bed

    **Final Output**: ''' Object List: [0:8, 1:10, 2:9, 3:12, 4:13, 5:20] Spatial Relationship Triplets: [[1, 1, 0], [2, 4, 1], [3, 6, 0], [4, 0, 3], [5, 6, 0]] Description: There is a wooden double bed with a dark dressing table on its left and a nightstand on its right. A wardro...

  66. [76]

    [[0, 6, 1], [2, 6, 0]]

    The second part is the spatial relationship triplets list e.g. [[0, 6, 1], [2, 6, 0]]. Note that for every triplet such as [0, 6, 1], each of its elements sequentially represents the object index, predicate(relationship index), and the subject index. This means the object zero...

  67. [77]

    there is a [color and texture] [obj_name] on the left of a [color and texture] [obj_name],and a [color andtexture][obj_name]ontherightofthe[colorandtexture][obj_name]

    The third part is the simple description of the room. e.g. "there is a [color and texture] [obj_name] on the left of a [color and texture] [obj_name],and a [color andtexture][obj_name]ontherightofthe[colorandtexture][obj_name] ". Youcanderiveyouranswerfollowingthesesteps. 1.As...

  68. [78]

    Identify all the objects in the image and the user text that match the specified object list, categorizing them by type. You can list every object youfind as following,notethatyouhavetotaketheobjectinusertextintoaccountas well: eg : object 0: [obj_name], catogory 12; object 1:...

  69. [79]

    Define that an object node is a main object surrounded byits several child objects,suchas a bed surrounded bynightstands and wordrobe,ora table surrounded bychairs.Youshouldstartfromthe most essentialobjectasrootnodetoid entifyits childobjects,thenrecursivelystartfromsomeofthe...

  70. [80]

    eg:object0nightstandisleftof(relationship1)theobject1[obj_name] ,triple:[0,1,1]

    Extract the relationships betweenthe root node objects and their surrounding child items, as well as the relationships between the subsequent node objects and theirownchildobjects,representingtheserelationshipsusingtheprev iouslymentionedtriples.Youcanlisteverytripletyoufindas...

  71. [2019]

    Implementation Details A.1

    2 FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts Supplementary Material A. Implementation Details A.1. Data Processing Edge Matrix Construction.In this paper, we define ten types of spatial relationships and a ‘None’ type, similar to InstructScene, i...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.