Pith

open record

sign in

arxiv: 2607.05392 · v1 · pith:WZIWLPH6 · submitted 2026-07-06 · cs.CV

SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion

Paul Engstler , Iro Laina , Christian Rupprecht , Andrea Vedaldi This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-07 12:32 UTCglm-5.2pith:WZIWLPH6record.jsonopen to challenge →

classification cs.CV
keywords scenesdatascenesyncitycoherentconvolutionalentiregenerated
0
0 comments X

The pith

Fine-tuning 3D diffusion to run like a convolution over sliding windows

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that an object-centric 3D diffusion model (TRELLIS) can be converted into a scene-scale generator by fine-tuning it to operate convolutionally on overlapping windows, conditioned on a dimetric 2D template of the entire scene. The central mechanism is a sliding-window inference scheme: the model processes a large latent grid patch-by-patch, each patch seeing its own image crop plus surrounding 3D context, with denoising signals averaged across overlapping windows at each step. A synthetic data engine—placing random 3D objects on random terrains—provides the 320k training samples needed to teach the model this new behavior without requiring real scene-scale 3D datasets. The two-stage pipeline first generates a globally coherent 2D template via MultiDiffusion-style overlapping-window denoising in latent space, then converts that template into 3D Gaussian Splats through the convolutional 3D generator. The authors demonstrate that this approach produces scenes of arbitrary size that are more faithful to their 2D templates and more globally coherent than off-the-shelf object generators applied per-tile.

Core claim

The key finding is that fine-tuning a 3D diffusion transformer to operate as a convolutional operator—processing overlapping spatial windows with shared weights and averaged denoising—allows it to scale from single objects to arbitrarily large scenes while maintaining geometric and visual coherence. The synthetic data engine, which assembles random objects on random terrains without semantic scene structure, provides sufficient training signal for the model to learn this windowed behavior and generalize to coherent scene generation at inference time.

What carries the argument

Convolutional inference: the 3D diffusion model is applied to overlapping sub-grids of a large latent volume, with each application conditioned on a masked image crop showing the corresponding scene region plus surrounding context. Denoising predictions from overlapping windows are averaged at each diffusion step, analogous to how convolutional filters share weights across spatial positions.

If this is right

  • Scene-scale 3D generation becomes possible without curated scene-level 3D datasets, as long as a synthetic data engine can produce enough spatially diverse training samples—even random object placements suffice.
  • The convolutional fine-tuning pattern could transfer to other object-centric generative models beyond TRELLIS, converting any single-object generator into a spatially scalable one.
  • Since the 2D template and 3D generation stages are independent, artist-created templates or non-diffusion image generators could serve as inputs, making the 3D stage a general image-to-scene converter.
  • Runtime scales roughly quadratically with scene size, making very large scenes tractable on single GPUs (31 minutes for a standard scene on one H100).
  • The sliding-window averaging approach naturally suppresses artifacts from individual window predictions, similar to how MultiDiffusion smooths 2D generation boundaries.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The random-placement training data may encode spatial composition patterns (object density, scale distribution, terrain-object relationships) that implicitly teach the model plausible scene structure, even without explicit semantic layout rules.
  • If the synthetic data engine were enriched with structured scene layouts (e.g., buildings arranged along streets), the fine-tuned model might produce scenes with stronger semantic coherence without changes to the architecture.
  • The convolutional approach could be tested on other 3D representations (e.g., triplane or NeRF-based generators) to determine whether the windowed-inference pattern generalizes beyond the voxel/Gaussian Splat paradigm.
  • The dimetric-perspective requirement is an artifact of the template generation stage; alternative camera models or multi-view templates could relax this constraint if the 3D generator were fine-tuned accordingly.

Load-bearing premise

The synthetic data engine places random Objaverse objects on random terrains with no semantic scene structure, and the paper assumes this provides sufficient training signal for the model to produce coherent, structured scenes at inference time.

What would settle it

If scenes generated from complex, semantically structured prompts (e.g., 'a medieval town square with a well in the center') exhibit geometric or semantic incoherence beyond what the 2D template already encodes—duplicated structures, broken geometry, or inability to maintain object identity across window boundaries—the convolutional fine-tuning would be shown to transfer spatial locality but not scene-level compositional understanding.

Figures

Figures reproduced from arXiv: 2607.05392 by Andrea Vedaldi, Christian Rupprecht, Iro Laina, Paul Engstler.

Figure 1
Figure 1. Figure 1: Diverse 3D worlds of arbitrary size and complexity are easily created with SynCity 3000 from scratch. Our approach first constructs a visually and semantically coherent 2D template of the entire scene, and then converts it into 3D Gaussian Splats with a fine-tuned two-stage generative diffusion model. the scene. In the second stage, this template is converted into the final 3D scene. This pipeline operates… view at source ↗
Figure 2
Figure 2. Figure 2: 2D Generation Pipeline. We generate a dimetric scene template based on text prompts and optional layout constraints. After subdividing the latents into win￾dows, we apply a latent diffusion model to jointly denoise the latent canvas. Once complete, the latent canvas is then decoded into the final 2D scene template. image. The unknown parts are then completed by adding a mask in the 3D gen￾eration process. … view at source ↗
Figure 3
Figure 3. Figure 3: 3D Generation Pipeline. We use a multi-stage pipeline to synthesize full 3D scenes from 2D templates using convolutional inference. In this inference process, we divide the latent into smaller regions that we jointly denoise. Each region has a direct template correspondence. To transform a scene, we first synthesize the sparse structure—a coarse voxel grid. Then, we infer features that encode the appearanc… view at source ↗
Figure 4
Figure 4. Figure 4: Fine-Tuning Design. We utilize two objectives in our fine-tuning process. In the first (top), we provide the core and surrounding context to the model. In the second (bottom), we only provide the core, which imitates the original objective of TRELLIS [62]. In both cases, we use a masked conditioning image and apply the loss only to the core. sponding voxel grid O ∈ {0, 1} N×N×N which captures its rough sha… view at source ↗
Figure 5
Figure 5. Figure 5: Data Generation. We propose a dataset engine to generate diverse 3D scenes by placing random Objaverse-XL [14] objects on randomized terrains. Each scene is rendered in dimetric projection to match the 2D template generation [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison With SynCity [15]. We compare scenes generated by SynCity (left) and our method (right) using the same prompt for each. Overall, the worlds generated by our method are richer, more vibrant, and more coherent, both visually and semantically. They have an organic structure that lets elements flow across the whole scene. creative freedom. This becomes apparent in [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 7
Figure 7. Figure 7: Large Scene Detail Comparison. We let both our method (top) and TREL￾LIS (right) reconstruct a larger scene (3136 × 1568 pixels template, left). While the overall scene produced by TRELLIS looks decent, its resolution notably impacts its capability to reproduce details. Meanwhile, SynCity 3000 has a high adherence to the template and reproduces details more faithfully. We recommend viewing on a display and… view at source ↗
Figure 1
Figure 1. Figure 1: Example JSON file to describe a theme park scene. Color correction. Reconstructions generated by TRELLIS [62] (without any of our proposed fine-tuning) show a reduction in lightness. This can be observed in [PITH_FULL_IMAGE:figures/full_fig_p022_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A scene template generated by ChatGPT Images (left) and the resulting scene produced by our method (right). Indoor scenes. While the prior work we compare to generates open outdoor scenes, which we replicate to ensure a fair comparison, our fine-tuning setup is technically agnostic to scene design. However, we rely on FLUX to build plausible scene templates. Due to its biases, the quality and plausibility … view at source ↗
Figure 3
Figure 3. Figure 3: Indoor scenes generated by our method (left template, right reconstruction) [PITH_FULL_IMAGE:figures/full_fig_p027_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Two large scenes, shown as templates (left), as generated by our method (top) and TRELLIS (right) [PITH_FULL_IMAGE:figures/full_fig_p028_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative examples comparing TripoSG [36], Hunyuan3D-2.1 [56], TREL￾LIS [62], and SynCity 3000 [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Three scenes generated by SynCity 3000, shown from all viewpoints [PITH_FULL_IMAGE:figures/full_fig_p029_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Random selection of samples generated by the dataset engine proposed in Section 5 [PITH_FULL_IMAGE:figures/full_fig_p030_7.png] view at source ↗
read the original abstract

We present SynCity 3000, a framework for generating 3D scenes that are globally coherent while enabling fine-grained layout control. Building on the ability of current image-to-3D generators to produce complex 3D assets from a single image, we extend this capability to the scale of entire scenes by adapting the generator to be applicable as a convolutional operator. We achieve this by fine-tuning the model on scene-like data generated by a new synthetic data engine, which we propose to address the scarcity of 3D scene data for training. The convolutional generator is then applied to a dimetric image of the entire scene, generated from the user prompt, resulting in 3D scenes of arbitrary size and complexity. Across diverse prompts and layouts, SynCity 3000 produces large, coherent, and detailed scenes, addressing the shortcomings of prior approaches to 3D scene generation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 7 minor

Summary. The paper presents SynCity 3000, a two-stage framework for text-to-3D scene generation. Stage 1 generates a 2D dimetric template using a latent diffusion model with overlapping windows (MultiDiffusion-style). Stage 2 converts the template into 3D Gaussian Splats by fine-tuning TRELLIS to operate convolutionally on overlapping windows, trained on a procedurally generated synthetic dataset of random Objaverse-XL objects placed on terrains. The method is compared against SynCity, 3DTown, NuiScene, and off-the-shelf object-centric 3D generators (TRELLIS, TripoSG, Hunyuan3D-2.1) via template faithfulness metrics (Table 1), geometric reconstruction quality on synthetic scenes (Table 3), and a user study (Table 2). The central claim is that fine-tuning enables globally coherent, large-scale 3D scene generation that is more faithful to templates than off-the-shelf baselines.

Significance. The paper addresses a genuine gap: extending object-centric 3D generators to scene scale without domain-specific scene datasets. The convolutional adaptation of TRELLIS with context tokens and the dual-objective fine-tuning scheme (Sec. 3.2) is a clean, well-motivated technical contribution. The synthetic data engine (Sec. 4) is a reasonable strategy for circumventing data scarcity, and the method produces visually compelling results at scales beyond what off-the-shelf models can handle. The ablations in Table 1 are informative and isolate the contributions of fine-tuning, context, and stride. The framework is practical, with reasonable inference times reported (Supp. A).

major comments (2)
  1. §5, Table 3: The geometric quality evaluation (Chamfer Distance, F-score) uses 25 scenes generated by the same synthetic data engine used to produce the 320k training samples. The paper acknowledges this ('we can leverage our synthetic dataset engine to obtain scene-like proxies'), but this constitutes an in-distribution evaluation: the model is trained on samples from this engine and evaluated on 25 more from the same distribution. This does not test geometric quality on actual text-generated scenes. The paper should either (a) obtain an independent geometric evaluation on held-out real or synthetic-out-of-distribution scenes, or (b) explicitly frame Table 3 as a sanity check on in-distribution generalization and add a quantitative geometric evaluation on at least a subset of the 35 LLM-generated templates used in Table 1 (e.g., by manually creating rough ground-truth geometry or using多
  2. Supp. C, 'Structure duplication': The paper acknowledges that the convolutional tiling can cause duplicated or partially broken structures, but provides no quantitative measure of frequency or severity. Since the central claim includes 'globally coherent' 3D scenes, this limitation is load-bearing. A simple quantitative assessment (e.g., reporting the fraction of generated scenes exhibiting visible duplication artifacts across the 35 test templates, or a user-study rating specifically for structural coherence) would substantially strengthen the claim. Without it, the coherence claim rests on qualitative figures and a modest plausibility rating (3.57/5, N=27).
minor comments (7)
  1. Table 1: The LPIPS improvement over TRELLIS (0.3993 vs. 0.4094) is small in absolute terms. Consider reporting per-scene variance or confidence intervals to establish significance, especially given N=35 templates.
  2. Table 2: The user study (N=27) uses forced binary choices but does not report confidence intervals or significance tests. Given the modest sample size, adding intervals or a binomial test would clarify whether the preference rates are statistically meaningful.
  3. References: TripoSG appears as both [35] and [36]; Hunyuan3D-2.1 appears as both [24] and [56]; 3DTown appears as both [80] and [81]; SceneCraft appears as both [22] and [65]; DiffuScene appears as both [54] and [55]; EchoScene appears as both [71] and [72]. Please consolidate.
  4. §3.2, Eq. for ε_w: The positional encoding extension from {0,...,M-1}^3 to {-V,...,M+V-1}^3 is described, but a small figure or diagram showing how context tokens are arranged relative to the core would help readers verify the construction.
  5. Supp. A, 'Color correction': The L*a*b* color statistics transfer is applied as a post-hoc fix. It would be useful to note whether this correction was applied in the quantitative evaluations (Tables 1, 3) or only for qualitative figures, since it could affect LPIPS/SSIM.
  6. Fig. 5 (data generation): The figure caption mentions DINOv2 feature extraction but does not clarify whether visibility-aware projection was used. The supplementary (Supp. A) notes visibility is not accounted for. A brief note in the main text would help readers understand this design choice.
  7. §5, 'Experimental details': The batch size of 1 is noted, but it is unclear whether gradient accumulation was used to achieve an effective larger batch size. Please clarify.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and constructive feedback. Both major comments identify legitimate gaps in our evaluation that we will address in revision.

read point-by-point responses

  1. Referee: §5, Table 3: The geometric quality evaluation uses 25 scenes from the same synthetic data engine used for training, constituting in-distribution evaluation. The paper should either obtain independent geometric evaluation on held-out real or OOD scenes, or explicitly frame Table 3 as a sanity check and add quantitative geometric evaluation on at least a subset of the 35 LLM-generated templates.

    Authors: The referee is correct that Table 3 constitutes an in-distribution evaluation: the 25 test scenes are generated by the same procedural engine that produced the 320k training samples, albeit with different random seeds and object placements. We agree this does not test geometric quality on text-generated scenes, and we will revise the manuscript to make this limitation explicit rather than leaving it implicit. We will reframe Table 3 as an in-distribution sanity check that isolates the effect of fine-tuning on geometric reconstruction quality relative to the off-the-shelf TRELLIS baseline, controlling for scene content. We would also note that Table 1 already provides an out-of-distribution evaluation on the 35 LLM-generated templates (which are text-generated and thus out-of-distribution relative to the synthetic training data), but it measures 2D template faithfulness via perceptual metrics rather than 3D geometric quality. The fundamental challenge for option (b) is that text-generated scenes have no ground-truth geometry, so Chamfer Distance and F-score cannot be computed. However, we can and will add a supplementary evaluation on a subset of the 35 LLM-generated templates using proxy geometric metrics that do not require ground truth—specifically, we will report mesh watertightness, the fraction of disconnected components, and surface completeness (absence of holes) as measured by rendering depth maps from novel viewpoints and checking for missing geometry. This will provide at least a partial geometric quality assessment on OOD, text-generated scenes. We cannot, within the revision timeframe, obtain ground-truth geometry for real scenes at the scale our method operates, so a full OOD geometric evaluation with Chamfer Distance remains future work. revision: partial

  2. Referee: Supp. C, 'Structure duplication': The paper acknowledges that convolutional tiling can cause duplicated or partially broken structures but provides no quantitative measure. A quantitative assessment (fraction of scenes with duplication artifacts, or a user-study coherence rating) would strengthen the coherence claim.

    Authors: We agree that the coherence claim should be supported by more than qualitative figures and a modest plausibility rating. We will add a quantitative assessment of structural duplication artifacts across the 35 test templates used in Table 1. Specifically, we will have annotators label each generated scene for the presence of visible duplication artifacts (binary: present/absent) and report the fraction of affected scenes. We will also add a dedicated structural coherence rating to the user study (on the same 1–5 scale as the existing plausibility rating) to complement the binary artifact count. We expect the duplication rate to be relatively low given the ablation results in Table 1 showing that smaller strides significantly reduce duplication, but we will report whatever the data shows. This will be added to the revised manuscript and supplementary materials. revision: yes

Circularity Check

0 steps flagged

Minor in-distribution evaluation concern in Tab. 3, but central claims are independently supported by Tab. 1 and Tab. 2; no circularity in the derivation chain.

full rationale

The paper's central claim — that fine-tuning TRELLIS to operate convolutionally on overlapping windows produces more faithful 3D scene reconstructions than off-the-shelf baselines — is supported by Tab. 1 (LPIPS/SSIM/PSNR on 35 LLM-generated templates, compared against TRELLIS, TripoSG, Hunyuan3D-2.1) and Tab. 2 (user study, N=27). These evaluations use externally generated templates and independent metrics; they are not circular. The one potential concern is Tab. 3 (geometric reconstruction quality via Chamfer Distance and F-score), which uses 25 scenes from the same synthetic data engine used for fine-tuning (Sec. 5: 'we can leverage our synthetic dataset engine to obtain scene-like proxies'). This is an in-distribution evaluation: the model is trained on 320k samples from the engine and evaluated on 25 more from the same engine. However, this is a limitation of evaluation scope, not a circularity in the derivation chain. The paper is transparent about this ('as if ground truth were available'), and Tab. 3 is supplementary to the main claims, not load-bearing for them. The fine-tuning data engine (Sec. 4) generates training targets procedurally from random Objaverse-XL objects on random terrains — these are not defined in terms of the model's outputs, so there is no self-defitional circularity. No step in the method reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

8 free parameters · 4 axioms · 1 invented entities

See axiom_ledger.

free parameters (8)
  • Learning rate = 5e-6
    Stated in Sec. 5 experimental details.
  • Task probability p = 0.5
    Probability of providing context vs. no context during fine-tuning (Sec. 3.2).
  • Context size V (stage 1) = 8
    Context voxels added around core for sparse structure model (Sec. 5).
  • Context size V (stage 2) = 32
    Context voxels for structured latent model (Sec. 5).
  • Template window width = 896px
    Size of single core patch in template (Sec. 5).
  • Mask extrusion height = 60px
    Imposes maximum height of tall structures at boundaries (Sec. 5).
  • Stride during inference = 0.5 patches
    Controls overlap averaging during convolutional inference (Sec. 5).
  • Number of training samples = 320k
    Generated by the synthetic data engine (Sec. 5).
axioms (4)
  • domain assumption TRELLIS [62] produces high-quality 3D objects from images and its autoencoders are convolutional
    Sec. 3.2 relies on TRELLIS as the base model; the sparse structure VAE is stated to be already convolutional.
  • ad hoc to paper Dimetric projection provides sufficient geometric information for 3D scene reconstruction
    Sec. 3.1 and Supp. C: the method forces a dimetric perspective, which is a design choice not independently justified.
  • ad hoc to paper Random placement of Objaverse-XL objects on terrains produces scene-like data sufficient for fine-tuning
    Sec. 4: the data engine assumes random assortments teach the model convolutional scene generation, inspired by LRM-Zero [63].
  • domain assumption MultiDiffusion-style window averaging produces globally coherent latents
    Sec. 3.1: inspired by MultiDiffusion [5]; assumed to transfer from 2D image to 3D latent domain.
invented entities (1)
  • Synthetic data engine independent evidence
    purpose: Generates scene-like 3D training data by placing random Objaverse-XL objects on random terrains
    The engine produces falsifiable training data; its outputs are shown in Supp. Fig. 7 and used for quantitative evaluation (Tab. 3). The approach is independently grounded in LRM-Zero [63].

pith-pipeline@v1.1.0-glm · 22020 in / 3369 out tokens · 106883 ms · 2026-07-07T12:32:28.994665+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 78 canonical work pages · 27 internal anchors

  1. [1]

    com / alimama - creative/FLUX-Controlnet-Inpainting(2024), gitHub repository

    AlimamaCreative: Flux-controlnet-inpainting.https : / / github . com / alimama - creative/FLUX-Controlnet-Inpainting(2024), gitHub repository

    work page 2024

  2. [2]

    Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single View

    Ardelean, A., Özer, M., Egger, B.: Gen3dsr: Generalizable 3d scene reconstruction via divide and conquer from a single view. arXiv2404.03421(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Armeni, I., He, Z.Y., Gwak, J., Zamir, A.R., Fischer, M., Malik, J., Savarese, S.: 3d scene graph: A structure for unified semantics, 3d space, and camera. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5664–5673 (2019)

    work page 2019

  4. [4]

    In: British Machine Vision Con- ference (BMVC) (2022)

    Bae, G., Budvytis, I., Cipolla, R.: Irondepth: Iterative refinement of single-view depth using surface normal and its uncertainty. In: British Machine Vision Con- ference (BMVC) (2022)

    work page 2022

  5. [6]

    MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation

    Bar-Tal, O., Yariv, L., Lipman, Y., Dekel, T.: MultiDiffusion: Fusing diffusion paths for controlled image generation. arXiv.csabs/2302.08113(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [7]

    ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

    Bhat,S.F.,Birkl,R.,Wofk,D.,Wonka,P.,Müller,M.:Zoedepth:Zero-shottransfer by combining relative and metric depth. arXiv2302.12288(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [8]

    In: Proc

    Chen, D.Z., Li, H., Lee, H.Y., Tulyakov, S., Nießner, M.: Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors. In: Proc. CVPR (2024)

    work page 2024

  8. [9]

    Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model

    Chen, Y., Viégas, F., Wattenberg, M.: Beyond surface statistics: Scene represen- tations in a latent diffusion model. arXiv2306.05720(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [10]

    In: arXiv (2023)

    Chung, J., Lee, S., Nam, H., Lee, J., Lee, K.M.: LucidDreamer: Domain-free gen- eration of 3d gaussian splatting scenes. In: arXiv (2023)

    work page 2023

  10. [11]

    In: Proc

    Collins, J., Goel, S., Deng, K., Luthra, A., Xu, L., Gundogdu, E., Zhang, X., Vi- cente, T.F.Y., Dideriksen, T., Arora, H., Guillaumin, M., Malik, J.: ABO: Dataset and benchmarks for real-world 3D object understanding. In: Proc. CVPR. pp. 21126–21136 (2022)

    work page 2022

  11. [12]

    In: Proc

    Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scan- Net: Richly-annotated 3D reconstructions of indoor scenes. In: Proc. CVPR (2017)

    work page 2017

  12. [13]

    Dai, T., Wong, J., Jiang, Y., Wang, C., Gokmen, C., Zhang, R., Wu, J., Fei-Fei, L.: Automatedcreationofdigitalcousinsforrobustpolicylearning.arXiv2410.07408 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [14]

    Objaverse-XL: A Universe of 10M+ 3D Objects

    Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S.Y., VanderBilt, E., Kembhavi, A., Vondrick, C., Gkioxari, G., Ehsani, K., Schmidt, L., Farhadi, A.: Objaverse-XL: A universe of 10M+ 3D objects. CoRRabs/2307.05663(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [15]

    In: Proceedings of the International Con- ference on Computer Vision (ICCV) (2025) 16 P

    Engstler, P., Shtedritski, A., Laina, I., Rupprecht, C., Vedaldi, A.: SynCity: Training-free generation of 3D worlds. In: Proceedings of the International Con- ference on Computer Vision (ICCV) (2025) 16 P. Engstler et al

    work page 2025

  15. [16]

    In: International Conference on 3D Vision (2025)

    Engstler, P., Vedaldi, A., Laina, I., Rupprecht, C.: Invisible stitch: Generating smooth 3d scenes with depth inpainting. In: International Conference on 3D Vision (2025)

    work page 2025

  16. [17]

    Advances in Neural Information Processing Systems36, 18225–18250 (2023)

    Feng, W., Zhu, W., Fu, T.j., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems36, 18225–18250 (2023)

    work page 2023

  17. [18]

    International Journal of Computer Vision129(12), 3313–3337 (2021)

    Fu, H., Jia, R., Gao, L., Gong, M., Zhao, B., Maybank, S., Tao, D.: 3d-future: 3d furniture shape with texture. International Journal of Computer Vision129(12), 3313–3337 (2021)

    work page 2021

  18. [19]

    DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image

    Gao, D., Rozenberszki, D., Leutenegger, S., Dai, A.: DiffCAD: weakly-supervised probabilistic CAD model retrieval and alignment from an RGB image. arXiv 2311.18610(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [20]

    In: Proc

    Gümeli, C., Dai, A., Nießner, M.: ROCA: Robust CAD model retrieval and align- ment from a single image. In: Proc. CVPR (2022)

    work page 2022

  20. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Höllein, L., Cao, A., Owens, A., Johnson, J., Nießner, M.: Text2room: Extract- ing textured 3d meshes from 2d text-to-image models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7909–7920 (2023)

    work page 2023

  21. [22]

    In: Proc

    Hu, Z., Iscen, A., Jain, A., Kipf, T., Yue, Y., Ross, D.A., Schmid, C., Fathi, A.: SceneCraft: An LLM agent for synthesizing 3d scenes as Blender code. In: Proc. ICML (2024)

    work page 2024

  22. [23]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Huang, Z., Guo, Y.C., An, X., Yang, Y., Li, Y., Zou, Z.X., Liang, D., Liu, X., Cao, Y.P., Sheng, L.: Midi: Multi-instance diffusion for single image to 3D scene genera- tion. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 23646–23657 (2025)

    work page 2025

  23. [24]

    Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material

    Hunyuan3D, T., Yang, S., Yang, M., Feng, Y., Huang, X., Zhang, S., He, Z., Luo, D., Liu, H., Zhao, Y., Lin, Q., Lai, Z., Yang, X., Shi, H., Zhao, Z., Zhang, B., Yan, H., Wang, L., Liu, S., Zhang, J., Chen, M., Dong, L., Jia, Y., Cai, Y., Yu, J., Tang, Y., Guo, D., Yu, J., Zhang, H., Ye, Z., He, P., Wu, R., Wei, S., Zhang, C., Tan, Y., Sun, Y., Niu, L., Hu...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [25]

    Standard, International Commission on Illumination, Vienna, AT (1976)

    International Commission on Illumination: Colorimetry — part 4: Cie 1976 l*a*b* colour space. Standard, International Commission on Illumination, Vienna, AT (1976)

    work page 1976

  25. [26]

    In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recog- nition

    Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Re- purposing diffusion-based image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recog- nition. pp. 9492–9502 (2024)

    work page 2024

  26. [27]

    ACM Transactions on Graphics42(4) (2023)

    Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics42(4) (2023)

    work page 2023

  27. [28]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Khanna, M., Mao, Y., Jiang, H., Haresh, S., Shacklett, B., Batra, D., Clegg, A., Undersander, E., Chang, A.X., Savva, M.: Habitat synthetic scenes dataset (hssd- 200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16384–16393 (2024)

    work page 2024

  28. [29]

    In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023) SynCity 3000 17

    Kim, S.W., Brown, B., Yin, K., Kreis, K., Schwarz, K., Li, D., Rombach, R., Tor- ralba, A., Fidler, S.: Neuralfield-ldm: Scene generation with hierarchical latent dif- fusion models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023) SynCity 3000 17

    work page 2023

  29. [30]

    In: Proc

    Kuo, W., Angelova, A., Lin, T.Y., Dai, A.: Patch2CAD: Patchwise embedding learning for in-the-wild shape retrieval from a single image. In: Proc. ICCV (2021)

    work page 2021

  30. [31]

    In: Proc

    Lee, H.H., Han, Q., Chang, A.X.: NuiScene: exploring efficient generation of un- bounded outdoor scenes. In: Proc. ICCV (2025)

    work page 2025

  31. [32]

    Diffusion Probabilistic Models for Scene-Scale 3D Categorical Data

    Lee, J., Im, W., Lee, S., Yoon, S.E.: Diffusion probabilistic models for scene-scale 3d categorical data. arXiv2301.00527(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [33]

    In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition

    Lee, J., Lee, S., Jo, C., Im, W., Seon, J., Yoon, S.E.: Semcity: Semantic scene generation with triplane diffusion. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 28337–28347 (2024)

    work page 2024

  33. [34]

    SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting

    Li, W., Cai, F., Mi, Y., Yang, Z., Zuo, W., Wang, X., Fan, X.: SceneDreamer360: text-driven 3D-consistent scene generation with panoramic Gaussian splatting. arXiv2408.13711(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [36]

    TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models

    Li, Y., Zou, Z.X., Liu, Z., Wang, D., Liang, Y., Yu, Z., Liu, X., Guo, Y.C., Liang, D., Ouyang, W., et al.: Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models. arXiv2502.06608(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [37]

    In: Proc

    Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: GLIGEN: open-set grounded text-to-image generation. In: Proc. CVPR (2023)

    work page 2023

  36. [38]

    Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modeling

    Li, Z., Wang, Y., Zheng, H., Luo, Y., Wen, B.: Sparc3d: Sparse representation and construction for high-resolution 3d shapes modeling. arXiv2505.14521(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [39]

    In: International Conference on Learning Representations (ICLR) (2024)

    Lin, C., MU, Y.: InstructScene: instruction-driven 3d indoor scene synthesis with semantic graph prior. In: International Conference on Learning Representations (ICLR) (2024)

    work page 2024

  38. [40]

    InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

    Lin, C., Mu, Y.: Instructscene: Instruction-driven 3d indoor scene synthesis with semantic graph prior. arXiv2402.04717(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [41]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Meng, Q., Li, L., Nießner, M., Dai, A.: Lt3sd: Latent trees for 3d scene diffusion. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 650–660 (2025)

    work page 2025

  40. [42]

    Commu- nications of the ACM65(1), 99–106 (2021)

    Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Commu- nications of the ACM65(1), 99–106 (2021)

    work page 2021

  41. [43]

    In: Proc

    Ocal,B.M.,Tatarchenko,M.,Karaoglu,S.,Gevers,T.:SceneTeller:language-to-3D scene generation. In: Proc. ECCV (2024)

    work page 2024

  42. [44]

    DINOv2: Learning Robust Visual Features without Supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv2304.07193(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  43. [45]

    Text2Immersion: Generative Immersive Scene with 3D Gaussians

    Ouyang, H., Heal, K., Lombardi, S., Sun, T.: Text2Immersion: Generative immer- sive scene with 3D gaussians. arXiv.csabs/2312.09242(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [46]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Pan, X., Charron, N., Yang, Y., Peters, S., Whelan, T., Kong, C., Parkhi, O., Newcombe, R., Ren, Y.C.: Aria digital twin: A new benchmark dataset for ego- centric 3d machine perception. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20133–20143 (2023)

    work page 2023

  45. [47]

    Compositional 3D Scene Generation using Locally Conditioned Diffusion

    Po, R., Wetzstein, G.: Compositional 3d scene generation using locally conditioned diffusion. ArXivabs/2303.12218(2023),https://api.semanticscholar.org/ CorpusID:257663283

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  46. [48]

    Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI

    Ramakrishnan, S.K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A.X., et al.: Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for em- bodied ai. arXiv2109.08238(2021) 18 P. Engstler et al

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  47. [49]

    In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition

    Ren, X., Huang, J., Zeng, X., Museth, K., Fidler, S., Williams, F.: Xcube: Large- scale 3d generative modeling using sparse voxel hierarchies. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 4209– 4219 (2024)

    work page 2024

  48. [50]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Roberts, M., Ramapuram, J., Ranjan, A., Kumar, A., Bautista, M.A., Paczan, N., Webb, R., Susskind, J.M.: Hypersim: A photorealistic synthetic dataset for holis- tic indoor scene understanding. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10912–10922 (2021)

    work page 2021

  49. [51]

    RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

    Shriram, J., Trevithick, A., Liu, L., Ramamoorthi, R.: Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv2404.07199(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  50. [52]

    LDM3D: Latent Diffusion Model for 3D

    Stan, G.B.M., Wofk, D., Fox, S., Redden, A., Saxton, W., Yu, J., Aflalo, E., Tseng, S.Y., Nonato, F., Muller, M., et al.: Ldm3d: Latent diffusion model for 3d. arXiv 2305.10853(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  51. [53]

    LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

    Sun, F.Y., Liu, W., Gu, S., Lim, D., Bhat, G., Tombari, F., Li, M., Haber, N., Wu, J.: LayoutVLM: differentiable optimization of 3d layout via vision-language models. arXiv2412.02193(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  52. [54]

    In: Proc

    Tang, J., Nie, Y., Markhasin, L., Dai, A., Thies, J., Nießner, M.: DiffuScene: de- noising diffusion models for generative indoor scene synthesis. In: Proc. CVPR (2024)

    work page 2024

  53. [55]

    In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition

    Tang,J.,Nie,Y.,Markhasin,L.,Dai,A.,Thies,J.,Nießner,M.:Diffuscene:Denois- ing diffusion models for generative indoor scene synthesis. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 20507– 20518 (2024)

    work page 2024

  54. [56]

    Team, T.H.: Hunyuan3d 2.1: From images to high-fidelity 3d assets with production-ready pbr material (2025)

    work page 2025

  55. [57]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2024)

    Wang,G.,Wang,P.,Chen,Z.,Wang,W.,Loy,C.C.,Liu,Z.:Perf:Panoramicneural radiance field from a single panorama. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2024)

    work page 2024

  56. [58]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: VGGT: Visual geometry grounded transformer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    work page 2025

  57. [59]

    ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

    Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: ProlificDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. arXiv.csabs/2305.16213(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  58. [60]

    arXiv2307.03177(2023)

    Wu, T., Zheng, C., Cham, T.J.: Panodiffusion: 360-degree panorama outpainting via diffusion. arXiv2307.03177(2023)

    work page arXiv 2023

  59. [61]

    arXiv.cs (2024)

    Wu, Z., Li, Y., Yan, H., Shang, T., Sun, W., Wang, S., Cui, R., Liu, W., Sato, H., Li, H., Ji, P.: BlockFusion: Expandable 3D scene generation using latent tri-plane extrapolation. arXiv.cs (2024)

    work page 2024

  60. [62]

    Structured 3D Latents for Scalable and Versatile 3D Generation

    Xiang, J., Lv, Z., Xu, S., Deng, Y., Wang, R., Zhang, B., Chen, D., Tong, X., Yang, J.: Structured 3d latents for scalable and versatile 3d generation. arXiv 2412.01506(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  61. [63]

    In: Proc

    Xie, D., Bi, S., Shu, Z., Zhang, K., Xu, Z., Zhou, Y., Pirk, S., Kaufman, A., Sun, X., Tan, H.: LRM-Zero: training large reconstruction models with synthesized data. In: Proc. NeurIPS (2024)

    work page 2024

  62. [64]

    Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

    Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., Zhao, H.: Depth anything: Un- leashing the power of large-scale unlabeled data. arXiv2401.10891(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  63. [65]

    In: Advances in Neural Information Processing Systems (2024) SynCity 3000 19

    Yang, X., Man, Y., Chen, J.K., Wang, Y.X.: Scenecraft: Layout-guided 3d scene generation. In: Advances in Neural Information Processing Systems (2024) SynCity 3000 19

    work page 2024

  64. [66]

    In: Proc

    Yang, Y., Sun, F.Y., Weihs, L., VanderBilt, E., Herrasti, A., Han, W., Wu, J., Haber, N., Krishna, R., Liu, L., Callison-Burch, C., Yatskar, M., Kembhavi, A., Clark, C.: Holodeck: Language guided generation of 3d embodied AI environments. In: Proc. CVPR (2024)

    work page 2024

  65. [67]

    ACM Transactions on Graphics (TOG)44(4), 1–19 (2025)

    Yao, K., Zhang, L., Yan, X., Zeng, Y., Zhang, Q., Xu, L., Yang, W., Gu, J., Yu, J.: Cast: Component-aligned 3D scene reconstruction from an RGB image. ACM Transactions on Graphics (TOG)44(4), 1–19 (2025)

    work page 2025

  66. [68]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Yeshwanth, C., Liu, Y.C., Nießner, M., Dai, A.: Scannet++: A high-fidelity dataset of 3d indoor scenes. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12–22 (2023)

    work page 2023

  67. [69]

    WonderWorld: Interactive 3D Scene Generation from a Single Image

    Yu, H.X., Duan, H., Herrmann, C., Freeman, W.T., Wu, J.: Wonderworld: Inter- active 3d scene generation from a single image. arXiv2406.09394(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  68. [70]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yu, H.X., Duan, H., Hur, J., Sargent, K., Rubinstein, M., Freeman, W.T., Cole, F., Sun, D., Snavely, N., Wu, J., et al.: Wonderjourney: Going from anywhere to everywhere. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6658–6667 (2024)

    work page 2024

  69. [71]

    In: Proc

    Zhai, G., Örnek, E.P., Chen, D.Z., Liao, R., Di, Y., Navab, N., Tombari, F., Busam, B.: EchoScene: indoor scene generation via information echo over scene graph dif- fusion. In: Proc. ECCV (2024)

    work page 2024

  70. [72]

    In: European Conference on Computer Vision

    Zhai, G., Örnek, E.P., Chen, D.Z., Liao, R., Di, Y., Navab, N., Tombari, F., Busam, B.: Echoscene: Indoor scene generation via information echo over scene graph diffu- sion. In: European Conference on Computer Vision. pp. 167–184. Springer (2024)

    work page 2024

  71. [73]

    In: The Thirty-eighth Annual Con- ference on Neural Information Processing Systems (2024)

    Zhan, G., Zheng, C., Xie, W., Zisserman, A.: A general protocol to probe large vision models for 3d physical understanding. In: The Thirty-eighth Annual Con- ference on Neural Information Processing Systems (2024)

    work page 2024

  72. [74]

    ACM Trans

    Zhang, B., Tang, J., Nießner, M., Wonka, P.: 3dshape2vecset: A 3d shape repre- sentation for neural fields and generative diffusion models. ACM Trans. Graph. 42(4) (jul 2023).https://doi.org/10.1145/3592442,https://doi.org/10. 1145/3592442

    work page doi:10.1145/3592442 2023

  73. [75]

    Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields

    Zhang, J., Li, X., Wan, Z., Wang, C., Liao, J.: Text2NeRF: Text-driven 3D scene generation with neural radiance fields. arXiv.csabs/2305.11588(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  74. [76]

    In: Proc

    Zhang, L., Wang, Z., Zhang, Q., Qiu, Q., Pang, A., Jiang, H., Yang, W., Xu, L., Yu, J.: CLAY: A controllable large-scale generative model for creating high-quality 3D assets. In: Proc. SIGGRAPH (2024)

    work page 2024

  75. [77]

    In: Proc

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proc. CVPR (2018)

    work page 2018

  76. [79]

    Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation

    Zhao, Z., Lai, Z., Lin, Q., Zhao, Y., Liu, H., Yang, S., Feng, Y., Yang, M., Zhang, S., Yang, X., et al.: Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv2501.12202(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  77. [81]

    arXiv2505.15765(2025)

    Zheng, K., Zhang, R., Gu, J., Yang, J., Wang, X.E.: Constructing a 3d town from a single image. arXiv2505.15765(2025)

    work page arXiv 2025

  78. [82]

    GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting

    Zhou, X., Ran, X., Xiong, Y., He, J., Lin, Z., Wang, Y., Sun, D., Yang, M.H.: Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv2402.07207(2024) SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion (Supplementary Materials) Paul Engstler, Iro Laina, Christian Rupprecht, and Andrea Vedaldi Visual Geom...

    work page internal anchor Pith review Pith/arXiv arXiv 2024