Pith. sign in

REVIEW 4 major objections 6 minor 64 references

ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ArtiScene claims that a single 2D image, drawn in isometric view by a text-to-image model, can carry enough layout and style information to assemble a full modular 3D scene without any 3D training data.

desk verdict A solid training-free 3D scene pipeline with modular assets, but the headline evaluation numbers are partly manufactured by selection and post-hoc exclusions. read the letter →

arxiv 2506.00742 v1 pith:DKLVKG4V submitted 2025-05-31 cs.CV cs.AI

Zeqi Gu , Yin Cui , Zhaoshuo Li , Fangyin Wei , Yunhao Ge , Jinwei Gu , Ming-Yu Liu , Abe Davis
show 1 more author
Yifan Ding
This is my paper
classification cs.CVcs.AI
keywords text-to-3Dscenegeneration2Dimageintermediaryisometricprojectiontraining-freepipelinelayoutestimationmonoculardepthmodular3Dscenesdiffusionmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ArtiScene claims that a skilled 3D scene can be built from free-form text without any 3D training data by first asking a text-to-image model to draw the scene in isometric view, then mining that single 2D picture for every object's shape, appearance, position, size, and rotation. The paper argues that the resulting modular scenes beat current text-to-3D scene methods, with 6–10x lower object-overlap rates than LayoutGPT, higher CLIP style scores than Holodeck, and strong user-study preferences (about 73–87% depending on the comparison). This matters because today's alternatives either learn from scarce 3D datasets or retrieve assets from fixed libraries, which limits the styles and object types they can produce. The claim is that the web-scale 2D knowledge already inside diffusion models can be repurposed as a layout engine for 3D.

What carries the argument

The load-bearing mechanism is the isometric 2D image intermediary. By requesting an isometric perspective—an orthographic view where the three coordinate axes meet at equal angles—the pipeline ensures that all three spatial dimensions are visible and that object appearance is translation-invariant, so a monocular depth model plus a modified pinhole projection (with a single hand-set scale $\alpha = 1/300$) can convert 2D bounding boxes into 3D positions and sizes. Around this proxy image, the pipeline wraps object detection (GroundedDINO), hole-filling (Pix2Gestalt), per-object 3D generation (Edify 3D), and rendered-feature pose matching (Stable Diffusion + DINO-v2) to assemble modular assets into a coherent scene.

What would settle it

Take a known 3D room, render it in isometric view, and run the same depth model on that render; compare predicted 3D box positions and sizes to ground truth. If the relative depth errors are large or the optimal depth scale $\alpha$ changes per scene, the 6–10x overlap reduction is an artifact of the specific test rooms, not a general property of the image-intermediary idea.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the spatial layout and artistic style of a 3D scene can be recovered from a single generated 2D image in isometric projection. Given a text prompt, ArtiScene generates an isometric proxy image, detects and segments each object, inpaints occlusions, generates a 3D mesh per object conditioned on the segmented image and a language description, estimates each object's 3D bounding box from monocular depth with a modified pinhole projection, and refines pose by rendered-feature matching. The same proxy image supplies floor and wall textures. The paper reports that this training-free pipeline yields scenes with object-overlap rates of 6.48% (bedrooms) and 2.19% (living rooms) versus 37.26% and 27.77% for LayoutGPT, a CLIP score of 29.45 versus Holodeck's 26.73, a 74.89% average human preference, and 95.07% in GPT-4o pairwise evaluation.

Load-bearing premise

The entire layout claim rests on the assumption that a monocular depth estimator run on a synthetic isometric picture gives positions that are accurate enough to place furniture without overlaps, using one fixed depth scale for all scenes.

Editorial extensions

If this is right

  • Because every object is generated separately and then assembled, each object can be re-textured, replaced, or re-positioned independently; the paper demonstrates this by swapping a generated car for a red Porsche and re-inserting it into the same scene.
  • The pipeline transfers to layouts not produced by text-to-image models: human-drawn or software-rendered isometric images can be fed directly, skipping the initial generation stage.
  • Since the method is training-free and component-based, replacing any stage (better depth, better 3D generation, better captioning) should improve the final scene without re-engineering the pipeline.
  • The approach can target scene types far outside indoor dataset distributions, as long as the text-to-image model can draw a plausible isometric image, because no 3D priors are learned.
  • The quantitative gains imply that overlap-free, physically plausible layouts can be obtained from 2D visual reasoning rather than from explicit spatial-relation constraints in an LLM planner.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The layout advantage hinges on monocular depth models being metric-consistent on synthetic isometric images; if that holds broadly, the same recipe could lift concept art and architectural sketches to 3D without any per-domain training.
  • The reported CLIP and GPT-4o gaps may partly reflect style preference rather than physical plausibility; a physics-based collision check (counting interpenetrating meshes) would test whether the low overlap rate is a genuine layout property or an artifact of filtered comparison categories.
  • The hand-set scale $\alpha=1/300$ and the hand-set de-occlusion margins are natural free parameters; a small study varying them across scene types would reveal whether the layout advantage is robust or tuned to the evaluated rooms.
  • An untested extension is using the same intermediary to condition lighting and camera placement as well as geometry, which the paper does not address but its modular design would support.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ArtiScene proposes a training-free text-to-3D scene generation pipeline that uses a generated 2D image as an intermediary. The pipeline first produces an isometric image from a text prompt, then detects and segments objects, generates per-object 3D assets from the segmented images with text descriptions, estimates object positions and sizes via monocular depth, and finally assembles the assets with floor/wall textures and a de-occlusion post-processing step. The authors evaluate against LayoutGPT on layout quality (OOR) and against Holodeck on end-to-end style and layout fidelity (CLIP, GPT-4o preference, user studies), reporting substantially better metrics.

Significance. The core idea of exploiting 2D image intermediaries to bypass scarce 3D training data is interesting and potentially useful for modular, editable scene generation. The paper is generally clear about the pipeline's components, and the supplementary ablations (two-pass detection, Pix2Gestalt vs. SDXL/DALL-E inpainting, image-plus-text conditioning, post-processing effect) provide useful evidence for several design choices. However, the headline quantitative claims are weakened by two load-bearing issues: an asymmetric CLIP-based input filter in the Holodeck comparison, and the heavy reliance of the layout margin on a hand-tuned post-processing step. The claims about aesthetic superiority and layout superiority are therefore not yet established at the reported magnitude. If the authors can re-run the comparisons fairly and validate the depth-based positioning, the contribution would be solid.

major comments (4)
  1. [Sec. 5.2, Table 3] The main end-to-end comparison against Holodeck is biased by an asymmetric filtering procedure. The text states: 'We use DALLE-3 to generate 3 images for each prompt, and filter out those that have CLIP score below 30, to ensure the initial intermediary guide has high quality. Our final test sets include 111 samples, and we generate Holodeck results with the same text prompt distribution.' Since the final ArtiScene scene is derived from that filtered intermediary, this selection directly inflates the CLIP score measured on the rendered scene, while Holodeck receives no equivalent filtering. The reported CLIP advantage (29.45 vs. 26.73), the GPT-4o winning rates, and the user-study preferences are all computed on these same filtered pairs, so they inherit the bias. Please re-evaluate on a common, unfiltered prompt set, or apply the same filtering to the baseline, or report results using a metric not used as a selection criterion.
  2. [Sec. 5.1, Table 6] The '6-10x lower OOR' headline claim is largely attributable to the de-occlusion post-processing rather than to the depth-derived positions. Table 6 shows that removing the post-processing raises ArtiScene's OOR from 6.48/2.19 to 27.30/18.95 for bedrooms/living rooms, reducing the margin over LayoutGPT (37.26/27.77) to roughly 1.4x. The paper should quantify the separate contributions of the 2D intermediary/depth estimation and the post-processing, and report the sensitivity of OOR to the unspecified distance margins in Sec. 3.6. Without this, the claim that image intermediaries 'indeed possess rich layout information' is not well supported by the overlap metric.
  3. [Sec. 3.2, Eq. (3)] The assertion that 'existing monocular depth models worked sufficiently well even on images with isometric perspective' is given without quantitative support, and the entire layout is scaled by a hand-set alpha = 1/300 with no sensitivity analysis. Since Eq. (1)-(3) convert 2D positions to 3D positions and sizes, and since Sec. 3.6 exists because those positions 'may contain errors', please provide a quantitative validation of the predicted 3D bounding boxes (e.g., against the image intermediary or human judgment) and show how OOR varies with alpha and with the imaginary focal length assumptions.
  4. [Supplementary Sec. 10] The supplementary material states that 'for a very small subset of examples shown in this paper, we manually excluded 3D assets from the final scene if its quality is very low, and have manually selected certain Pix2Gestalt inpainted results over the GPT suggested ones.' The paper must clarify whether these manual interventions were applied to the quantitative evaluation and user-study samples, and if so, specify how many samples were affected and how the selection was performed. Without this information, the quantitative results may not reflect a fully automated pipeline.
minor comments (6)
  1. [Abstract] The abstract reports 'averages a 74.89% winning rate in extensive user studies' but the numbers in Tables 2 and 4 do not directly yield this average; please clarify which studies are averaged and how the figure is computed.
  2. [Sec. 5.1] The user study pairs are said to be 'sampled and paired up 15 results' from an evaluation set of 42 results per room type; please describe the random sampling procedure and whether the pairs were selected before or after seeing the results.
  3. [Sec. 3.6 and Sec. 4] The 'pre-defined distance margin' and the association threshold between furniture and small objects are never given numerical values; please state them in the implementation section.
  4. [Sec. 4] The text says 'We use version 4o for ChatGPT' but the references list [30] as 'ChatGPT-4-turbo'; please align the model version and the reference.
  5. [References] References [18] and [19] are identical (both Text2Room); please remove the duplicate.
  6. [Table 1] The caption contains the typo 'Overlaping Rate'; also, the exclusion of chair-desk overlaps and object categories not in LayoutGPT should be described in the main text, not only in the caption.

Circularity Check

1 steps flagged · score 6.0 of 10

CLIP-based input filtering in Sec 5.2 selects the evaluation set on the same metric used to claim superiority, making the reported CLIP gap and downstream preferences partially forced by construction.

  1. fitted input called prediction [Section 5.2 (Scene Generation Evaluation), quantitative comparison with Holodeck]
    "We use DALLE-3 [2] to generate 3 images for each prompt, and filter out those that have CLIP score below 30, to ensure the initial intermediary guide has high quality. Our final test sets include 111 samples, and we generate Holodeck results with the same text prompt distribution."

    Table 3 then reports CLIP scores on the rendered outputs (Holodeck 26.73 vs Ours 29.45) as evidence of superiority. Because every ArtiScene output is assembled from a 2D guide that was kept only if its CLIP score already exceeded 30, while Holodeck outputs receive no such filtering, the 111-sample test set is selected for the exact property being compared. The object appearance, layout, and textures of the final scene are all extracted from that high-CLIP guide, so the rendered scene's CLIP alignment largely inherits the guide's CLIP quality. The reported 2.7-point CLIP lead, and the user-study and GPT-4o preferences computed on the same filtered pairs, are therefore partly forced by the input filter rather than by the 3D pipeline itself.

full rationale

The paper's central contribution is an empirical pipeline, not a formal derivation, and most reported results are genuine measurements. The OOR comparison is not circular because the de-occlusion post-processing is disclosed and ablated in Table 6, and the supplementary admits manual exclusions that affect only displayed examples. Self-citations (e.g. ref. [22]) are peripheral and not load-bearing. The one substantial circular element is the CLIP evaluation: the test prompts are filtered by the same CLIP metric later used for comparison, and only ArtiScene's inputs are filtered. This makes the headline CLIP superiority, and the preference scores computed on the same pairs, partially an artifact of test-set selection. The core framework still has independent content, so the analysis is not fully forced; the score reflects partial circularity in the central quantitative comparison.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The system's central claim rests on five domain assumptions about external foundation models (DALLE-3 layout fidelity, depth estimation on isometric images, Edify 3D asset quality, ChatGPT description fidelity, GroundedDINO detection recall) plus four hand-set numeric parameters. None of the assumptions are derived or benchmarked inside the paper, and the two most load-bearing ones are flagged as imperfect by the paper itself: the limitations section admits text-to-image models are weak on object quantity and location, and Sec 3.6 exists because depth-derived positions 'may contain errors'. No new theoretical entities are introduced.

free parameters (4)
  • depth scale alpha = 1/300
    Hand-set in Sec 4 ('we find alpha = 1/300 to scale the predicted depth image works for most scenes'); converts Depth-Anything-2 output to scene units in Eq. 3. No per-scene adaptation or sensitivity analysis is reported.
  • CLIP input filter threshold = 30
    Sec 5.2 keeps only DALLE-3 outputs with CLIP score above 30 before the pipeline runs; this conditions the reported output CLIP score on a selection using the same metric family and is a test-protocol parameter, not a fixed method component.
  • repetition detection threshold = 0.95 (SBERT cosine)
    Supplementary Sec 9; a hand-set cosine threshold (0.95) decides when two objects share one generated asset, with users invited to tune it (the example uses 0.89); this affects scene consistency and asset count.
  • de-occlusion distance margin = unspecified
    Sec 3.6 moves objects whose extrema lie within a 'pre-defined distance margin' of scene extrema; the margin value is never specified, so the layout post-processing contains an unquantified hand-set constant.
assumptions (5)
  • domain assumption A 2D isometric image of a scene encodes sufficient 3D layout information (relative positions, sizes, poses) to reconstruct a plausible 3D arrangement.
    Sec 3 intro and Fig. 3 argue isometric projection conveys three spatial dimensions; the paper provides no quantitative validation that DALLE-3 obeys isometric geometry for arbitrary prompts.
  • domain assumption Monocular depth models produce useful metric-relative depth on synthetic isometric images.
    Sec 3.2 states 'we found that existing monocular depth models worked sufficiently well even on images with isometric perspective'; this is an unbenchmarked empirical claim and the paper's own Sec 3.6 exists to fix its errors.
  • domain assumption DALLE-3 generates layout-plausible, style-consistent images from free-form text for the tested scene categories and styles.
    The entire pipeline depends on the intermediary image quality; Sec 6 admits the model is weak at detailed prompts about object quantity and location, bounding the assumption's validity.
  • domain assumption The proprietary single-view 3D generator (Edify 3D, ref 27) produces usable 3D assets from a segmented image plus an LLM description.
    Sec 3.3 treats the public API as a black box; its failure modes (which the supplementary partially patches via inpainting and ChatGPT selection) are outsourced to an external model.
  • domain assumption ChatGPT-4o descriptions of segmented objects are faithful, geometry-focused, and free of hallucinated or background details.
    Sec 3.3 asks the LLM to avoid viewpoint, background, and occlusion mentions, but the pipeline provides no check that descriptions match the image; wrong text conditioning would propagate into every generated asset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary." pith.science (2026). https://pith.science/paper/DKLVKG4V

@misc{pith2026250600742,
  author       = {Pith},
  title        = {Pith review of: ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DKLVKG4V}},
  note         = {Machine review of arXiv:2506.00742}
}
read the original abstract

Designing 3D scenes is traditionally a challenging task that demands both artistic expertise and proficiency with complex software. Recent advances in text-to-3D generation have greatly simplified this process by letting users create scenes based on simple text descriptions. However, as these methods generally require extra training or in-context learning, their performance is often hindered by the limited availability of high-quality 3D data. In contrast, modern text-to-image models learned from web-scale images can generate scenes with diverse, reliable spatial layouts and consistent, visually appealing styles. Our key insight is that instead of learning directly from 3D scenes, we can leverage generated 2D images as an intermediary to guide 3D synthesis. In light of this, we introduce ArtiScene, a training-free automated pipeline for scene design that integrates the flexibility of free-form text-to-image generation with the diversity and reliability of 2D intermediary layouts. First, we generate 2D images from a scene description, then extract the shape and appearance of objects to create 3D models. These models are assembled into the final scene using geometry, position, and pose information derived from the same intermediary image. Being generalizable to a wide range of scenes and styles, ArtiScene outperforms state-of-the-art benchmarks by a large margin in layout and aesthetic quality by quantitative metrics. It also averages a 74.89% winning rate in extensive user studies and 95.07% in GPT-4o evaluation. Project page: https://artiscene-cvpr.github.io/

Figures

Figures reproduced from arXiv: 2506.00742 by the authors.

Figure 1
Figure 1. We present ArtiScene, a training-free, language-driven 3D scene generation pipeline that can design diverse, aesthetic, and easily [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline Overview. Taking a text prompt as input, ArtiScene first prompts a text-to-image model for an image intermediary (yellow line). Then through object detection, inpainting, and prompting ChatGPT to describe the detected objects’ appearance and geom￾etry, we acquire a 3D model for each object (blue line). In parallel, we combine monocular depth estimation with the formerly detected 2D bounding boxes to estimat… view at source ↗
Figure 3
Figure 3. Layout from Different Views. Isometric projection conveys information about all three spatial dimensions of an object while making its appearance invariant to translation in the scene. jects are common for that type of scene. Then we prompt ChatGPT to list out objects present in that specific proxy image. The two parts complement each other to provide an abundant name list of objects that are likely to be present. A… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Qualitative Comparison w/ LayoutGPT. We visualize layout predictions as 3D bounding boxes from a side view rendering. Each furniture type is presented by a fixed color for all plots. The regions marked by red boxes contain severe overlaps, which occurs much more often …
Figure 5
Figure 5. Figure 5: Qualitative Comparison w/ Holodeck. In (a) we show more scene categories sampled from the MIT dataset, and in (b) one fixed category (bedroom) with different styles and themes that we used for our evaluation. Our method is robust across a wide range of scene categories…
Figure 7
Figure 7. Figure 7: Generation with Images Created by Human. Ar￾tiScene could be used with human-designed image input, skipping the initial text2im stage. initial text prompt input, and be directly conditioned on an isometric layout image. Moreover, it generalizes well to layout images ge…
Figure 6
Figure 6. Figure 6: Object Editing. As ArtiScene generates a scene by generating individual 3D objects and then assembling, it allows fast and simple modular editing. An important advantage of our methods is its composi￾bility and editability. As all objects are generated separately, we c…
Figure 8
Figure 8. Figure 8: Different Inpainting Methods. We found Pix2Gestalt has strong geometry awareness. This object is the television shelf in [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Pix2Gestalt and ChatGPT Selection. We use Pix2Gestalt to inpaint an incomplete audio. From seed 0 to 5, the results with seed 0 and 4 are more complete, ChatGPT success￾fully chose seed 0 as an satisfactory input for our following image conditioned 3D asset generation.…
Figure 10
Figure 10. Figure 10: Conditioning on Text and Image vs. Text Only. This object is the television in [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Top 5 Pose Candidates by FoundationPose [32]. The scores in green texts indicate the model confidence in each can￾didate pose. As we do not have ground truth camera parameters, and our 3D model is not exactly the same as the 2D counterpart in the image intermediary, e…
Figure 12
Figure 12. Figure 12: Effects of Repetition Detection. We replace the dif￾ferent chairs and armchairs in (a) with only one chair model and one armchair model, and keep the other parameters (pose, dimen￾sions, positions) the same to create (b). The changed parts are highlighted by the recta…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 40 canonical work pages

  1. [1]

    Componerf: Text-guided multi-object compositional nerf with editable 3d scene lay- out

    Haotian Bai, Yuanhuiyi Lyu, Lutao Jiang, Sijia Li, Haonan Lu, Xiaodong Lin, and Lin Wang. Componerf: Text-guided multi-object compositional nerf with editable 3d scene lay- out. arXiv preprint arXiv:2303.13843, 2023. 3

    arXiv 2023

  2. [2]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2023. 4, 5, 6, 1

    work page 2023

  3. [3]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In CVPR, pages 18392–18402, 2023. 8

    work page 2023

  4. [4]

    Comboverse: Compositional 3d as- sets creation using spatially-aware diffusion guidance

    Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Comboverse: Compositional 3d as- sets creation using spatially-aware diffusion guidance. arXiv preprint arXiv:2403.12409, 2024. 3

    arXiv 2024

  5. [5]

    Luciddreamer: Domain-free generation of 3d gaussian splatting scenes

    Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384, 2023. 2

    arXiv 2023

  6. [6]

    Set-the-scene: Global-local training for generating controllable nerf scenes

    Dana Cohen-Bar, Elad Richardson, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Set-the-scene: Global-local training for generating controllable nerf scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2920–2929, 2023. 2, 3

    work page 2023

  7. [7]

    Proc- thor: Large-scale embodied ai using procedural generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Proc- thor: Large-scale embodied ai using procedural generation. NeurIPS, 35:5982–5994, 2022. 3

    work page 2022

  8. [8]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, pages 13142– 13153, 2023

    work page 2023

Show all 64 references
  1. [9]

    Objaverse-xl: A universe of 10m+ 3d objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram V oleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. NeurIPS, 36, 2024. 3

    2024

  2. [10]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lan...

    2019

  3. [11]

    Ctrl-room: Controllable text-to-3d room meshes generation with layout constraints

    Chuan Fang, Yuan Dong, Kunming Luo, Xiaotao Hu, Rakesh Shrestha, and Ping Tan. Ctrl-room: Controllable text-to-3d room meshes generation with layout constraints. arXiv preprint arXiv:2310.03602, 2023. 2

    2023

  4. [12]

    Layoutgpt: Compositional visual plan- ning and generation with large language models

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Ar- jun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual plan- ning and generation with large language models. NeurIPS, 36, 2024. 2, 3, 5, 1

    2024

  5. [13]

    Scenescape: Text-driven consistent scene generation

    Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. NeurIPS, 36, 2024. 2

    2024

  6. [14]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. 2

    2024 arXiv

  7. [15]

    Any- home: Open-vocabulary generation of structured and tex- tured 3d homes

    Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. Any- home: Open-vocabulary generation of structured and tex- tured 3d homes. In ECCV, pages 52–70. Springer, 2025. 2, 3

    2025

  8. [16]

    Scenehgn: Hierarchical graph net- works for 3d indoor scene generation with fine-grained ge- ometry

    Lin Gao, Jia-Mu Sun, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Jie Yang. Scenehgn: Hierarchical graph net- works for 3d indoor scene generation with fine-grained ge- ometry. IEEE TPAMI, 45(7):8902–8919, 2023. 2

    2023

  9. [17]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. NeurIPS, 33:6840–6851, 2020. 2

    2020

  10. [19]

    Text2room: Extracting textured 3d meshes from 2d text-to-image models

    Lukas H ¨ollein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In ICCV, pages 7909–7920, 2023. 2

    2023

  11. [20]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023. 2

    2023 arXiv

  12. [21]

    Instructscene: Instruction- driven 3d indoor scene synthesis with semantic graph prior

    Chenguo Lin and Yadong Mu. Instructscene: Instruction- driven 3d indoor scene synthesis with semantic graph prior. arXiv preprint arXiv:2402.04717, 2024. 3

    2024 arXiv

  13. [22]

    Genusd: 3d scene generation made easy

    Tsung-Yi Lin, Chen-Hsuan Lin, Yin Cui, Yunhao Ge, Se- ungjun Nah, Arun Mallya, Zekun Hao, Yifan Ding, Hanzi Mao, Zhaoshuo Li, et al. Genusd: 3d scene generation made easy. In ACM SIGGRAPH 2024 Real-Time Live!, pages 1–

    2024

  14. [23]

    Towards language-guided in- teractive 3d generation: Llms as layout interpreter with gen- erative feedback

    Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang. Towards language-guided in- teractive 3d generation: Llms as layout interpreter with gen- erative feedback. arXiv preprint arXiv:2305.15808, 2023. 3

    2023 arXiv

  15. [24]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 5, 1

    2023 arXiv

  16. [25]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM, 65(1):99–106, 2021. 3

    2021

  17. [26]

    Housecrafter: Lifting floorplans to 3d scenes with 2d diffusion model

    Hieu T Nguyen, Yiwen Chen, Vikram V oleti, Varun Jam- pani, and Huaizu Jiang. Housecrafter: Lifting floorplans to 3d scenes with 2d diffusion model. arXiv preprint arXiv:2406.20077, 2024. 2

    2024 arXiv

  18. [27]

    Edify 3d

    NVIDIA and Shutterstock. Edify 3d. https://build. nvidia.com/shutterstock/edify-3d , 2023. Ac- cessed: 2024-11-08. 5

    2023

  19. [28]

    Sceneteller: Language-to-3d scene generation

    Bas ¸ak Melis¨Ocal, Maxim Tatarchenko, Sezer Karao˘glu, and Theo Gevers. Sceneteller: Language-to-3d scene generation. In ECCV, pages 362–378. Springer, 2024. 3

    2024

  20. [29]

    OpenAI. Chatgpt. https://openai.com/chatgpt ,

  21. [30]

    Chatgpt-4-turbo

    OpenAI. Chatgpt-4-turbo. https://openai.com/ chatgpt, 2024. Accessed: 2024-11-08. 5

    2024

  22. [31]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 5

    2023 arXiv

  23. [32]

    Found- pose: Unseen object pose estimation with foundation fea- tures

    Evin Pınar ¨Ornek, Yann Labb ´e, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. Found- pose: Unseen object pose estimation with foundation fea- tures. In ECCV, pages 163–182. Springer, 2025. 2

    2025

  24. [33]

    pix2gestalt: Amodal segmentation by synthesizing wholes

    Ege Ozguroglu, Ruoshi Liu, D ´ıdac Sur´ıs, Dian Chen, Achal Dave, Pavel Tokmakov, and Carl V ondrick. pix2gestalt: Amodal segmentation by synthesizing wholes. In CVPR, pages 3931–3940. IEEE Computer Society, 2024. 5, 1

    2024

  25. [34]

    Atiss: Autoregres- sive transformers for indoor scene synthesis

    Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregres- sive transformers for indoor scene synthesis. NeurIPS, 34: 12013–12026, 2021. 2

    2021

  26. [35]

    Compositional 3d scene generation using locally conditioned diffusion

    Ryan Po and Gordon Wetzstein. Compositional 3d scene generation using locally conditioned diffusion. In 2024 International Conference on 3D Vision (3DV), pages 651–

    2024

  27. [36]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  28. [37]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022. 3

    2022 arXiv

  29. [38]

    Recognizing indoor scenes

    Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In CVPR, pages 413–420. IEEE, 2009. 6

    2009

  30. [39]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. pages 8748–8763. PMLR, 2021. 6

    2021

  31. [40]

    Infinite photorealistic worlds using procedural generation

    Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, et al. Infinite photorealistic worlds using procedural generation. InCVPR, pages 12630– 12641, 2023. 2

    2023

  32. [41]

    In- finigen indoors: Photorealistic indoor scenes using procedu- ral generation

    Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, et al. In- finigen indoors: Photorealistic indoor scenes using procedu- ral generation. In CVPR, pages 21783–21794, 2024. 2

    2024

  33. [42]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. pages 8821–8831. Pmlr,

  34. [43]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084,

    1908 arXiv

  35. [44]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 2, 5

    2022

  36. [45]

    Controlroom3d: Room gen- eration using semantic proxy rooms

    Jonas Schult, Sam Tsai, Lukas H ¨ollein, Bichen Wu, Jialiang Wang, Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, et al. Controlroom3d: Room gen- eration using semantic proxy rooms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  37. [46]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2

    2010 arXiv

  38. [47]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017. 2

    2017

  39. [48]

    Sceneformer: Indoor scene generation with transformers

    Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 2021 International Conference on 3D Vision (3DV), pages 106–115. IEEE, 2021. 2

    2021

  40. [49]

    Luciddreaming: Controllable object-centric 3d generation

    Zhaoning Wang, Ming Li, and Chen Chen. Luciddreaming: Controllable object-centric 3d generation. arXiv preprint arXiv:2312.00588, 2023. 3

    2023 arXiv

  41. [50]

    Scene synthesis via uncertainty-driven attribute syn- chronization

    Haitao Yang, Zaiwei Zhang, Siming Yan, Haibin Huang, Chongyang Ma, Yi Zheng, Chandrajit Bajaj, and Qixing Huang. Scene synthesis via uncertainty-driven attribute syn- chronization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5630–5640, 2021. 2

    2021

  42. [51]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv:2406.09414, 2024. 5

    2024 arXiv

  43. [52]

    Scenecraft: Layout-guided 3d scene generation

    Xiuyu Yang, Yunze Man, Junkun Chen, and Yu-Xiong Wang. Scenecraft: Layout-guided 3d scene generation. NeurIPS, 37:82060–82084, 2024. 3

    2024

  44. [53]

    Llplace: The 3d in- door scene layout generation and editing via large language model

    Yixuan Yang, Junru Lu, Zixiang Zhao, Zhen Luo, James JQ Yu, Victor Sanchez, and Feng Zheng. Llplace: The 3d in- door scene layout generation and editing via large language model. arXiv preprint arXiv:2406.03866, 2024. 3

    2024 arXiv

  45. [54]

    Holodeck: Language guided gen- eration of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Kr- ishna, Lingjie Liu, et al. Holodeck: Language guided gen- eration of 3d embodied ai environments. In CVPR, pages 16227–16237, 2024. 2, 3

    2024

  46. [55]

    Wonderworld: Interactive 3d scene generation from a single image

    Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from a single image. arXiv preprint arXiv:2406.09394, 2024. 2

    2024 arXiv

  47. [56]

    Wonderjourney: Going from anywhere to everywhere

    Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, De- qing Sun, Noah Snavely, Jiajun Wu, et al. Wonderjourney: Going from anywhere to everywhere. InCVPR, pages 6658– 6667, 2024. 2

    2024

  48. [57]

    Inpaint anything: Segment anything meets image inpainting

    Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023. 5, 1

    2023 arXiv

  49. [58]

    Scenewiz3d: To- wards text-guided 3d scene composition

    Qihang Zhang, Chaoyang Wang, Aliaksandr Siarohin, Peiye Zhuang, Yinghao Xu, Ceyuan Yang, Dahua Lin, Bolei Zhou, Sergey Tulyakov, and Hsin-Ying Lee. Scenewiz3d: To- wards text-guided 3d scene composition. arXiv preprint arXiv:2312.08885, 2023. 3

    2023 arXiv

  50. [59]

    The scene language: Representing scenes with programs, words, and embeddings

    Yunzhi Zhang, Zizhang Li, Matt Zhou, Shangzhe Wu, and Jiajun Wu. The scene language: Representing scenes with programs, words, and embeddings. arXiv preprint arXiv:2410.16770, 2024. 3

    2024 arXiv

  51. [60]

    Scenex: Procedural control- lable large-scale scene generation via large-language models

    Mengqi Zhou, Jun Hou, Chuanchen Luo, Yuxi Wang, Zhaox- iang Zhang, and Junran Peng. Scenex: Procedural control- lable large-scale scene generation via large-language models. arXiv e-prints, pages arXiv–2403, 2024. 2

    2024

  52. [61]

    Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting

    Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhi- wei Lin, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv preprint arXiv:2402.07207, 2024. 3 ArtiScene: Language-Dr...

    2024 arXiv

  53. [63]

    Target Labels for Detection In Sec

    Ablation 8.1. Target Labels for Detection In Sec. 4 we mentioned we repeat the step of object detec- tion twice: the first time after detecting furniture and small objects, we inpaint away the small objects, and detect for furniture and remaining small objects again. Thus, the...

  54. [64]

    a fo�es�-t�em�d me�ti�g ro�m

    Repetition Detection (a) Generate In Separate (b) Substitute Repetitions “a fo�es�-t�em�d me�ti�g ro�m” Figure 12. Effects of Repetition Detection. We replace the dif- ferent chairs and armchairs in (a) with only one chair model and one armchair model, and keep the other param...

  55. [65]

    More Discussion on Limitation Our automated pipeline generates 3D scenes in batches, yet scene-specific manual adjustments could further improve the results. For a very small subset of examples shown in this paper, we manually excluded 3D assets from the fi- nal scene if its q...

  56. [2024]

    Accessed: 2024-11-08. 2

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.