Pith. sign in

REVIEW 3 major objections 6 minor 80 references

Coherent 3D Scene Diffusion From a Single RGB Image

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Framing single-image 3D scene reconstruction as conditional diffusion over all objects simultaneously, with an intra-scene attention prior and a Gaussian-based surface alignment loss, yields claimed gains of 12.04% AP3D on SUN RGB-D and…

desk verdict Genuinely new diffusion-based scene reconstruction with strong ablations; a data-leakage risk and a numeric slip keep the exact SOTA margin uncertain but the paper deserves serious review. read the letter →

arxiv 2412.10294 v1 pith:EVNU5QNZ submitted 2024-12-13 cs.CV

Manuel Dahnert , Angela Dai , Norman Müller , Matthias Nie{ss}ner This is my paper · ORCID
classification cs.CV
keywords single-image3Dscenereconstructionconditionaldiffusionpriorintra-sceneattentionsurfacealignmentlossSPAGHETTIshapecodeSUNRGB-DPix3D
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that single-view 3D scene reconstruction is better posed as a conditional diffusion problem that denoises all objects in a scene together, rather than regressing each object independently. It claims that conditioning the denoising network on every detected object at once, with attention between objects, creates a generative scene prior that captures arrangement and shape relationships, and that this prior is what fixes intersecting, floating, or incomplete reconstructions. A surface alignment loss, computed by sampling points directly from an intermediate Gaussian shape representation and comparing them with partial depth, is introduced so pose and shape can be trained jointly even where full 3D annotations are missing. If the claims hold, the same recipe should push holistic scene understanding beyond the previous regression-based baselines and transfer to other indoor datasets.

What carries the argument

The central object is the SPAGHETTI shape representation, which encodes each shape as a set of 16 oriented, anisotropic 3D Gaussians with per-Gaussian latent features; this code can be decoded into occupancy fields and meshes, and it also permits direct point sampling from the predicted Gaussians before decoding. The paper's key move is to make this representation denoiseable: during training, Gaussian parameters are noised and the network predicts the clean structure, and the same scaffold is used to form the surface alignment loss against ground-truth depth. The scene prior is carried by an intra-scene attention module that lets all objects attend to one another throughout the reverse diffusion process, so the reconstruction of one object can be informed by the positions and shapes of the others.

What would settle it

Load the paper's reimplemented SPAGHETTI prior and the original released SPAGHETTI checkpoints on identical chair and airplane inputs, then compare reconstruction Chamfer distance; if the two disagree strongly, the shape foundation of the method is not validated. A second check is to retrain the full scene model without the surface alignment loss and see whether the reported +2.11% AP3D from joint training disappears.

Watch

Extended reading notes

Core claim

The paper's central claim is that single-image 3D scene reconstruction should be treated not as per-object regression but as a conditional diffusion process over the whole scene. Starting from noise, the model denoises the 7-DoF pose and the SPAGHETTI shape code of every detected object at once, conditioned on an instance-segmented RGB image; an intra-scene attention module lets objects exchange information throughout denoising, so chairs learn to avoid intersecting tables and occluded objects get plausible completions. The paper reports that this formulation outperforms previous holistic reconstruction methods, with a 12.04% improvement in AP3D at 15% IoU on SUN RGB-D and a 13.43% increase in F-score on Pix3D, and that it generalizes to ScanNet frames without fine-tuning.

Load-bearing premise

The headline numbers rest on the paper's reimplementation of the SPAGHETTI shape prior, since the official code has no training procedure; if that reimplementation does not faithfully reproduce the original prior, the shape codes, the surface alignment loss, and the final meshes are all built on an unvalidated foundation.

Editorial extensions

If this is right

  • Replacing a one-step regression head with iterative denoising is worth an 11.08% gain in AP3D and a 0.19 improvement in Lalign in the paper's ablation.
  • Learning the joint scene prior instead of per-object marginals raises AP3D by 9.30%, showing that inter-object context is a measurable part of the result.
  • Joint training with the surface alignment loss adds 2.11% AP3D and reduces alignment error, meaning partial depth can substitute for missing full annotations.
  • Because the alignment loss samples from the Gaussian scaffold, any shape representation with the same property could be swapped in without redesigning the loss.
  • The same model can also synthesize scenes unconditionally by dropping the image condition, so the learned prior is reusable for generation, not just reconstruction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit: training with the original, unmodified SPAGHETTI code — if it were released — would isolate how much of the reported gain depends on the paper's reimplementation; until then, reproduction should treat the shape prior as a variable.
  • The surface alignment loss should transfer to video or multi-view depth observations, where partial depth from several frames could jointly supervise pose and shape without full 3D labels.
  • The dependence on instance segmentation suggests an end-to-end variant, where detection and denoising share features, might improve performance but would also couple two error sources; the paper evaluates with oracle detections, so real-world gains may be lower.
  • Extending the same joint-denoising idea to articulated or movable objects would be a natural next step, but the current static-scene assumption would need to be relaxed first.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a conditional diffusion model for joint 3D pose and shape estimation of all objects in a scene from a single RGB image. The method treats scene reconstruction as a denoising process over object poses and SPAGHETTI-style shape codes, conditions on all objects simultaneously via an intra-scene attention module, and introduces a surface alignment loss (Lalign) that enables joint training when only partial depth supervision is available. The authors report state-of-the-art results on SUN RGB-D (AP3D +12.04 percentage points over Im3D) and Pix3D (F-Score +13.43 percentage points over InstPIFu), and present ablations supporting the diffusion formulation, scene-prior modeling, and joint training.

Significance. If the results hold, the work is a meaningful advance: it demonstrates that a generative scene-level diffusion prior improves both arrangement and geometry over feed-forward baselines, and the surface alignment loss provides a practical way to train with partial annotations. The paper includes thorough per-class tables and a clear ablation study, and it benchmarks against official implementations of Total3D, Im3D, and InstPIFu. However, the central SOTA claim on Pix3D depends on a shape prior that is re-trained on ShapeNet, and the paper does not establish that Pix3D test models are excluded from that training set; this is a load-bearing unresolved risk. The numeric inconsistencies and the use of a training loss as an evaluation metric further weaken the exact margins claimed.

major comments (3)
  1. [Appendix E, Sec. 4.2] The paper does not establish that the Pix3D test 3D models are excluded from the training set of the re-implemented SPAGHETTI shape prior. Since Pix3D ground-truth meshes are drawn from ShapeNet and the prior is retrained on 'all relevant shape categories' (Appendix E), the shape decoder may have memorized test identities, which would inflate the reported F-Score and CD gains over InstPIFu. Please clarify whether any split filter was applied to the prior's training data, and if not, provide evidence that the prior does not recall test shapes (e.g., per-category F-Score with a prior trained on a ShapeNet subset disjoint from Pix3D).
  2. [Table 1 vs Table 7] The F-Score for Im3D is reported as 21.45 in Table 1 but 31.45 in Table 7, and InstPIFu's F-Score also differs (45.28 vs 45.62 in Table 7). This inconsistency changes the claimed improvement margins and must be corrected. Additionally, the abstract's '13.43% increase' and '12.04% improvement' are absolute percentage-point differences, not relative percentages; please state this unambiguously.
  3. [Eq. (11), Table 1] The surface alignment loss Lalign (Eq. 11) is used both as a training objective and as an evaluation metric in Table 1. Because the proposed method is directly trained to minimize this quantity while the baselines are not, the Lalign column is not an independent comparison. Please report an alternative geometric metric (e.g., two-sided Chamfer distance, or point-to-surface distance on a held-out depth map) or clearly contextualize that this column is a training-aligned measure.
minor comments (6)
  1. [Sec. 3.3] In the pose parameterization, the sentence 'orientation θi ∈ [−π, π) in .' is incomplete; please specify the unit (radians).
  2. [Sec. 3.2, Eq. (4)] Equation (4) appears to contain a duplicated expression for pΦ(xt−1|xt, y); please clean up the formatting.
  3. [Sec. 4.4] The text states 'resulting in a +0.2 improvement in Lalign', but Table 1 shows a difference of 0.25 relative to Im3D; please align the text and table.
  4. [General] Since the method is generative, please consider reporting standard deviations over multiple runs for the main metrics, or at least state how many seeds were used.
  5. [Sec. 4.4, Fig. 4] The generalization claim to ScanNet is supported only by qualitative results; please include quantitative pose/shape metrics on ScanNet if available.
  6. [Tables 7 and 9] Please cross-check all aggregate numbers against Table 1; in addition to the Im3D F-Score discrepancy, the InstPIFu F-Score differs between tables (45.28 vs 45.62).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: headline claims rest on external benchmarks; only a mild metric-alignment caveat on Lalign.

full rationale

The paper's central derivation chain is self-contained. The headline improvements (12.04% AP15_3D on SUN RGB-D and 13.43% F-Score on Pix3D) are measured on external benchmarks with standard metrics against official baseline implementations, and no equation in the paper defines those numbers in terms of the training objective or a self-citation. The diffusion formulation (Eqs. 2-5), pose parameterization, SPAGHETTI shape encoding, Intra-Scene Attention, and Lalign (Eq. 11) are all specified algorithmically and ablated in Table 2; none invokes a prior result by the same authors as a load-bearing premise. The self-citations present (e.g., DiffuScene [67], Scan2CAD [3], ROCA [17], panoptic reconstruction [10]) are contextual, baseline-related, or annotation sources, not evidence for the method's design choices. Appendix E discloses that the official SPAGHETTI code lacks training code and that the authors re-implemented the training procedure, which is an unvalidated fidelity assumption but not a circular step. The only mild evaluation concern is that Lalign is used both as a training loss (Eq. 8 with lambda=0.01) and as a reported metric (Sec. 4.3, Table 1), so the +0.2 Lalign gain can favor the method trained on it; this is an evaluation-alignment and fairness caveat rather than a circular derivation, and the main SOTA claim does not reduce to it. Overall, no circular step meets the quote-and-reduction threshold.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on standard DDPM theory, on a shape representation whose training code had to be reimplemented, and on depth-based alignment supervision. The free parameters are largely standard diffusion and training choices; the pose normalization constants are dataset-derived.

free parameters (6)
  • lambda (Lalign weight) = 0.01
    Weight balancing Lalign in Eq. 8, chosen by hand.
  • number of Gaussians per shape (g) = 16
    Size of the shape scaffolding; architectural choice.
  • points sampled per Gaussian in Lalign (m) = 1000
    Sampling budget for the alignment loss in Sec. 3.6.
  • pose normalization constants = d: mu=2.7 max=2.5; s: mu=3.5 max=7.0; theta: mu=0.0 max=3.14
    Dataset-specific min-max normalization for pose parameters, computed from the SUN RGB-D training set (Eqs. 12-14).
  • classifier-free guidance drop probability = 0.8
    Probability of dropping the image condition during training (Sec. 3.2).
  • diffusion noise schedule = beta1=0.0001, betaT=0.02, T=1000
    Standard linear schedule used for all diffusion processes.
assumptions (5)
  • standard math The reverse diffusion process can be learned by a neural network to recover x0 from Gaussian noise, per the DDPM formulation in Eqs. 2-5.
    Invokes the standard DDPM training objective from Ho et al. [22].
  • domain assumption The SPAGHETTI Gaussian-plus-latent representation with an occupancy decoder faithfully represents the shape distribution across the required object classes.
    The whole shape pipeline (Sec. 3.4) and the reimplementation in Appendix E rely on this representation being accurate and trainable.
  • domain assumption The 1-sided Chamfer distance against instance-segmented ground-truth depth maps provides a useful learning signal for joint pose and shape even under partial annotations.
    The surface alignment loss in Eq. 11 is the main tool for joint training on SUN RGB-D.
  • domain assumption Ground-truth 2D instance segmentation and camera parameters are used at evaluation, which isolates the 3D prediction quality from detection errors.
    Evaluation protocol in Sec. 4.3; the method's ranking is only meaningful under this oracle setting.
  • domain assumption The pose parameterization from [23] together with dataset-specific normalization is sufficient for recovering 7-DoF object poses.
    Sec. 3.3 and Appendix E define the pose representation and normalization; if the parameterization loses precision, pose results would degrade.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Coherent 3D Scene Diffusion From a Single RGB Image." pith.science (2026). https://pith.science/paper/EVNU5QNZ

@misc{pith2026241210294,
  author       = {Pith},
  title        = {Pith review of: Coherent 3D Scene Diffusion From a Single RGB Image},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVNU5QNZ}},
  note         = {Machine review of arXiv:2412.10294}
}
read the original abstract

We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. Our method utilizes an image-conditioned 3D scene diffusion model to simultaneously denoise the 3D poses and geometries of all objects within the scene. Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture the scene context and by allowing the model to learn inter-object relationships throughout the diffusion process. We further propose an efficient surface alignment loss to facilitate training even in the absence of full ground-truth annotation, which is common in publicly available datasets. This loss leverages an expressive shape representation, which enables direct point sampling from intermediate shape predictions. By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods, achieving a 12.04% improvement in AP3D on SUN RGB-D and a 13.43% increase in F-Score on Pix3D.

Figures

Figures reproduced from arXiv: 2412.10294 by the authors.

Figure 1
Figure 1. Given a single RGB image of an indoor scene, our model reconstructs the 3D scene [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Scene Prior and Surface Alignment Loss Overview. (Left) We propose a novel way to model scene priors (Sec. 3.5) by modeling the scene context and the relationships between all objects during the denoising process. (Right) For additional supervision and joint training, we use a surface alignment loss (Sec. 3.6) between a given ground truth depth map and point samples directly drawn from the intermediate shape represe… view at source ↗
Figure 3
Figure 3. Qualitative comparison of 3D scene reconstruction on SUN RGB-D [62]. While the baselines often produce noisy or incomplete shape reconstruction of intersecting or misplaced objects, our method produces plausible object arrangements as well as high-quality shape reconstructions [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Inference results on ScanNet [11]. We use our model trained on SUN RGB-D [62] and perform inference on individual frames of ScanNet without fine-tuning. We observe strong generalization capabilities with respect to different camera parameters and scene arrangements. 4.…
Figure 5
Figure 5. Figure 5: Unconditional results. Injecting ∅ as a condition to our conditional diffusion model, i.e., effectively disabling the conditioning mechanism, results in high-quality and diverse results. forms the foundation for subsequent downstream tasks like mixed reality applicatio…
Figure 6
Figure 6. Figure 6: Additional qualitative scene reconstruction results on SUN RGB [62]. Our diffusion￾based scene layout and shape prediction approach achieves accurate results even for strongly occluded objects. Object Reconstruction & Unconditional Synthesis In [PITH_FULL_IMAGE:figure…
Figure 7
Figure 7. Figure 7: Qualitative comparison of 3D pose estimation on the SUN RGB-D [62]. The input image is displayed on the left, and the predicted and ground-truth 3D arrangements are visualized as top￾down orthographic views of the scene. We observe that Total3D frequently lacks a globa…
Figure 8
Figure 8. Figure 8: Probabilistic behavior for partially occluded shapes. In the input image, the left chair is heavily occluded, which allows for multiple plausible interpretations of the non-visible part of the shape. Our diffusion-based method derives faithful modes. SUN RGB-D as the p…
Figure 9
Figure 9. Figure 9: Qualitative comparison of 3D shape reconstruction on the Pix3D [64]. While InstPIFu often produces noisy surfaces, our image-conditional 3D diffusion model synthesizes high-quality shapes that closely match the target geometries [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Shape decomposition visualization. We assign each vertex of the reconstructed mesh to the closest 3D Gaussian center and visualize the assignment with individual colors. Our scaffolding representation decomposes the shape into distinctive regions and aligns well with …
Figure 11
Figure 11. Figure 11: Comparison with retrieval baseline method ROCA [17] on frames from ScanNet [11]. While ROCA cannot always retrieve a matching mode from the shape database, such as the desk in the first row, our diffusion-based reconstruction approach reconstructs accurate shapes and …
Figure 12
Figure 12. Figure 12: Architecture Diagram of the Shape Diffusion Model. The shape diffusion model consists of 3 sub-parts: An image-conditioned diffusion model, denoising the 3D Gaussians; a 3D Gaussian-conditioned diffusion model, denoising the intrisic vectors; and an Occupancy Decoder,…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 63 canonical work pages

  1. [1]

    URL https://cocodataset.org/#detection-leaderboard

    Coco leaderboard. URL https://cocodataset.org/#detection-leaderboard

    work page

  2. [2]

    Alliegro, Y

    A. Alliegro, Y . Siddiqui, T. Tommasi, and M. Nießner. Polydiff: Generating 3d polygonal meshes with diffusion models. arXiv preprint arXiv:2312.11417, 2023

    arXiv 2023

  3. [3]

    Avetisyan, M

    A. Avetisyan, M. Dahnert, A. Dai, M. Savva, A. X. Chang, and M. Nießner. Scan2cad: Learning cad model alignment in rgb-d scans. In CVPR, 2019

    work page 2019

  4. [4]

    A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015

    arXiv 2015

  5. [5]

    Cheng, I

    B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. 2022

    2022

  6. [6]

    Cheng, H.-Y

    Y .-C. Cheng, H.-Y . Lee, S. Tulyakov, A. G. Schwing, and L.-Y . Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456–4465, 2023

    work page 2023

  7. [7]

    G. Chou, Y . Bahat, and F. Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. 2023

    work page 2023

  8. [8]

    C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Computer Vision–European Conference on Computer Vision 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 628–644. Springer, 2016

    work page 2016

Show all 80 references
  1. [9]

    T. Chu, P. Zhang, Q. Liu, and J. Wang. Buol: A bottom-up framework with occupancy-aware lifting for panoptic 3d scene reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4937–4946, 2023

    2023

  2. [10]

    Dahnert, J

    M. Dahnert, J. Hou, M. Nießner, and A. Dai. Panoptic 3d scene reconstruction from a single rgb image. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021

    2021

  3. [11]

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017

    2017

  4. [12]

    Deitke, D

    M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kemb- havi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023

    2023

  5. [13]

    Y . Du, Z. Liu, H. Basevi, A. Leonardis, B. Freeman, J. Tenenbaum, and J. Wu. Learning to exploit stability for 3d scene parsing. In Conference on Neural Information Processing Systems , 2018

    2018

  6. [14]

    H. Fan, H. Su, and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 605–613, 2017

    2017

  7. [15]

    Gkioxari, J

    G. Gkioxari, J. Malik, and J. Johnson. Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019

    2019

  8. [16]

    Groueix, M

    T. Groueix, M. Fisher, V . G. Kim, B. Russell, and M. Aubry. AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  9. [17]

    Gümeli, A

    C. Gümeli, A. Dai, and M. Nießner. Roca: Robust cad model retrieval and alignment from a single image. 2022

    2022

  10. [18]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770–778, 2016

    2016

  11. [19]

    Q. He, D. Zhou, B. Wan, and X. He. Single image 3d object estimation with primitive graph networks. In Proceedings of the 29th ACM International Conference on Multimedia , pages 2353–2361, 2021

    2021

  12. [20]

    Hertz, O

    A. Hertz, O. Perel, R. Giryes, O. Sorkine-Hornung, and D. Cohen-Or. Spaghetti: Editing implicit shapes through part aware generation. ACM Transactions on Graphics (TOG), 41(4):1–20, 2022

    2022

  13. [21]

    Ho and T

    J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 11

    2022 arXiv

  14. [22]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  15. [23]

    Huang, S

    S. Huang, S. Qi, Y . Xiao, Y . Zhu, Y . N. Wu, and S.-C. Zhu. Cooperative holistic scene understanding: Unifying 3d object, layout, and camera pose estimation. In Conference on Neural Information Processing Systems, 2018

    2018

  16. [24]

    Huang, S

    S. Huang, S. Qi, Y . Zhu, Y . Xiao, Y . Xu, and S.-C. Zhu. Holistic 3d scene parsing and reconstruction from a single rgb image. In European Conference on Computer Vision, 2018

    2018

  17. [25]

    K.-H. Hui, R. Li, J. Hu, and C.-W. Fu. Neural wavelet-domain diffusion for 3d shape generation. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022

    2022

  18. [26]

    Izadinia, Q

    H. Izadinia, Q. Shan, and S. M. Seitz. Im2cad. In CVPR, 2017

    2017

  19. [27]

    Jang and L

    W. Jang and L. Agapito. Codenerf: Disentangled neural radiance fields for object categories. InProceedings of the IEEE/CVF International Conference on Computer Vision , pages 12949–12958, 2021

    2021

  20. [28]

    Karras, M

    T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565–26577, 2022

    2022

  21. [29]

    S. W. Kim, B. Brown, K. Yin, K. Kreis, K. Schwarz, D. Li, R. Rombach, A. Torralba, and S. Fidler. Neuralfield-ldm: Scene generation with hierarchical latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  22. [30]

    J. Koo, S. Yoo, M. H. Nguyen, and M. Sung. Salad: Part-level latent diffusion for 3d shape generation and manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14441–14451, 2023

    2023

  23. [31]

    Kulkarni, I

    N. Kulkarni, I. Misra, S. Tulsiani, and A. Gupta. 3d-relnet: Joint object and relational network for 3d prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2212–2221, 2019

    2019

  24. [32]

    W. Kuo, A. Angelova, T.-y. Lin, and A. Dai. Mask2cad: 3d shape prediction by learning to segment and retrieve. In Proceedings of the European Conference on Computer Vision (European Conference on Computer Vision), 2020

    2020

  25. [33]

    W. Kuo, A. Angelova, T.-Y . Lin, and A. Dai. Patch2cad: Patchwise embedding learning for in-the-wild shape retrieval from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12589–12599, 2021

    2021

  26. [34]

    J. Lei, C. Deng, W. B. Shen, L. J. Guibas, and K. Daniilidis. Nap: Neural 3d articulated object prior. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 31878–31894. Curran Associ...

    2023

  27. [35]

    L. Li, S. Khan, and N. Barnes. Silhouette-assisted 3d object instance reconstruction from a cluttered scene. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (Proceedings of the IEEE/CVF International Conference on Computer VisionW) , pages 2080–2088, 2019...

    2019

  28. [36]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision–European Conference on Computer Vision 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proc...

    2014

  29. [37]

    H. Liu, Y . Zheng, G. Chen, S. Cui, and X. Han. Towards high-fidelity single-view holistic reconstruction of indoor scenes. In European Conference on Computer Vision, 2022

    2022

  30. [38]

    R. Liu, R. Wu, B. V . Hoorick, P. Tokmakov, S. Zakharov, and C. V ondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023

    2023

  31. [39]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  32. [40]

    W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM Trans. Gr ., 21(4):163–169, 1987. 12

    1987

  33. [41]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018

    2018

  34. [42]

    Luo and W

    S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837–2845, 2021

    2021

  35. [43]

    Mandikal, N

    P. Mandikal, N. KL, and R. Venkatesh Babu. 3d-psrnet: Part segmented 3d point cloud reconstruction from a single image. In Proceedings of the European Conference on Computer Vision (European Conference on Computer Vision) Workshops, pages 0–0, 2018

    2018

  36. [44]

    Melas-Kyriazi, C

    L. Melas-Kyriazi, C. Rupprecht, and A. Vedaldi. Pc2: Projection-conditioned point cloud diffusion for single-image 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12923–12932, 2023

    2023

  37. [45]

    Müller, Y

    N. Müller, Y . Siddiqui, L. Porzi, S. R. Bulo, P. Kontschieder, and M. Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4328–4338, 2023

    2023

  38. [46]

    C. Nash, Y . Ganin, S. A. Eslami, and P. Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference on machine learning , pages 7220–7229. PMLR, 2020

    2020

  39. [47]

    P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, 2012

    2012

  40. [48]

    Y . Nie, X. Han, S. Guo, Y . Zheng, J. Chang, and J. J. Zhang. Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In CVPR, 2020

    2020

  41. [49]

    J. Pan, X. Han, W. Chen, J. Tang, and K. Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019

    2019

  42. [50]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Information Processing Systems, 2019

    2019

  43. [51]

    Poole, A

    B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023

    2023

  44. [52]

    Popov, P

    S. Popov, P. Bauszat, and V . Ferrari. Corenet: Coherent 3d scene reconstruction from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 366–383. Springer, 2020

    2020

  45. [53]

    X. Ren, J. Huang, X. Zeng, K. Museth, S. Fidler, and F. Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  46. [54]

    L. Roberts. Machine perception of threedimensional solids. PhD thesis, Massachusetts Institute of Technology, 1963

    1963

  47. [55]

    Ronneberger, P

    O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pa...

    2015

  48. [56]

    Russakovsky, J

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015

    2015

  49. [57]

    Sella, G

    E. Sella, G. Fiebelman, P. Hedman, and H. Averbuch-Elor. V ox-e: Text-guided voxel editing of 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 430–440, 2023

    2023

  50. [58]

    J. R. Shue, E. R. Chan, R. Po, Z. Ankner, J. Wu, and G. Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20875–20886, 2023

    2023

  51. [59]

    Siddiqui, A

    Y . Siddiqui, A. Alliegro, A. Artemov, T. Tommasi, D. Sirigatti, V . Rosov, A. Dai, and M. Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2024. 13

    2024

  52. [60]

    Sohl-Dickstein, E

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256–2265. PMLR, 2015

    2015

  53. [61]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  54. [62]

    S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015

    2015

  55. [63]

    S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion from a single depth image. arXiv preprint arXiv:1611.08974, 2016

    2016 arXiv

  56. [64]

    X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B. Tenenbaum, and W. T. Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In CVPR, 2018

    2018

  57. [65]

    Szymanowicz, C

    S. Szymanowicz, C. Rupprecht, and A. Vedaldi. Viewset diffusion: (0-)image-conditioned 3d generative models from 2d data. International Conference on Computer Vision, 2023

    2023

  58. [66]

    J. Tang, X. Han, J. Pan, K. Jia, and X. Tong. A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition , pages 4541–4550, 2019

    2019

  59. [67]

    J. Tang, Y . Nie, L. Markhasin, A. Dai, J. Thies, and M. Nießner. Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303.14207 , 2023

    2023 arXiv

  60. [68]

    Tulsiani, S

    S. Tulsiani, S. Gupta, D. F. Fouhey, A. A. Efros, and J. Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 302–310, 2018

    2018

  61. [69]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    2017

  62. [70]

    N. Wang, Y . Zhang, Z. Li, Y . Fu, W. Liu, and Y .-G. Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In European Conference on Computer Vision, 2018

    2018

  63. [71]

    J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Advances in Neural Information Processing Systems, pages 82–90, 2016

    2016

  64. [72]

    H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2690–2698, 2019

    2019

  65. [73]

    A. Yu, V . Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR, 2021

    2021

  66. [74]

    X. Zeng, A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, and K. Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978, 2022

    2022 arXiv

  67. [75]

    Zhang, M

    B. Zhang, M. Nießner, and P. Wonka. 3DILG: Irregular latent grids for 3d generative modeling. In Thirty-Sixth Conference on Neural Information Processing Systems , 2022

    2022

  68. [76]

    Zhang, J

    B. Zhang, J. Tang, M. Niessner, and P. Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. arXiv preprint arXiv:2301.11445, 2023

    2023 arXiv

  69. [77]

    Zhang, Z

    C. Zhang, Z. Cui, Y . Zhang, B. Zeng, M. Pollefeys, and S. Liu. Holistic 3d scene understanding from a single image with implicit representation. In CVPR, 2021

    2021

  70. [78]

    Zhang, Z

    X. Zhang, Z. Chen, F. Wei, and Z. Tu. Uni-3d: A universal model for panoptic 3d scene reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (Proceedings of the IEEE/CVF International Conference on Computer Vision), pages 9256–9266, October 2023

    2023

  71. [79]

    Zheng, Y

    X. Zheng, Y . Liu, P. Wang, and X. Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics F orum, volume 41, pages 52–63. Wiley Online Library, 2022

    2022

  72. [80]

    dressers

    L. Zhou, Y . Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5826–5835, 2021. 14 A Appendix In the following, we show more qualitative results for scene reconstr...

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.