Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Neural LightRig claims that synthesizing nine point-light views from one object photo and regressing normals and physically based material maps from them beats single-image baselines on its synthetic benchmark.

desk verdict A solid empirical pipeline for single-image inverse rendering that generates multi-light inputs with diffusion; the SOTA claim needs cross-distribution evidence, but the core idea is real. read the letter →

arxiv 2412.09593 v1 pith:LXCA4PID submitted 2024-12-12 cs.CV

Zexin He , Tengfei Wang , Xin Huang , Xingang Pan , Ziwei Liu This is my paper · ORCID
classification cs.CV
keywords single-imageinverserenderingsurfacenormalestimationPBRmaterialmulti-lightdiffusionrelightingG-bufferpredictionphotometricstereodisambiguationsynthetictrainingdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that single-image inverse rendering can be made accurate by first synthesizing photometric-stereo-style observations. Its framework, Neural LightRig, fine-tunes a pretrained image diffusion model to turn one photo of an object into nine images lit from known directions; a second U-Net then reads these images, plus the input and the light angles, and predicts surface normals and PBR material maps. On a held-out synthetic benchmark the method reports the best numbers among the compared approaches, and the ablation shows the multi-light inputs are what cut normal error most sharply. The core claim, read sympathetically, is that generating consistent extra lighting beats capturing it, and that this largely dissolves the monocular ambiguity.

What carries the argument

The load-bearing mechanism is a generate-then-regress loop with two trained components. First, a multi-light diffusion model produces nine lighting-consistent images of the object in a single 3x3 grid, conditioned on the input photo by channel concatenation plus reference attention so the generated images keep the input's color and texture. Second, a large regression U-Net consumes the input image, the nine generated images, and sinusoidal embeddings of each light's spherical angles, and outputs an 8-channel G-buffer (normal, albedo, roughness, metallic). The chain is trained and evaluated on the paper's own rendered dataset, LightProp, which pairs 80k synthetic objects with multi-light images, ground-truth G-buffers, and randomized lighting; a set of degradations, brightness perturbations, orientation noise, and mixing of generated images is applied to bridge the render-to-diffusion domain gap.

What would settle it

Run Neural LightRig on a calibrated photometric-stereo capture of real objects: give it one image under a known flash as input, then compare predicted normals and albedo, roughness, and metallic against ground truth from the multi-light rig. If the generated nine-light images are not physically consistent with the input, the regressor's reported advantage should shrink or disappear on these real captures, since all quantitative validation in the paper is on synthetic renders and real-world results are only qualitative.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single image can be effectively expanded into a calibrated multi-light capture by a diffusion model, and that this expansion resolves much of the ambiguity that plagues single-image normal and material estimation. Concretely, Neural LightRig fine-tunes a pretrained latent diffusion model on a newly rendered dataset of 80k synthetic objects, so that given one photo it emits nine images lit by point sources at known spherical angles. Those nine images, together with their lighting angles, condition a U-Net that predicts an 8-channel G-buffer: surface normal, albedo, roughness, and metallic. Quantitative results on 1,000 held-out synthetic objects show the method outperforming prior work, for example a normal mean angular error of 6.41 degrees versus 8.03 for the closest baseline, albedo PSNR of 26.62 dB versus 23.88 dB, and relighting PSNR of 30.12 dB versus 27.98 dB. The intended upshot is that monocular inverse rendering can be recast as a photometric-stereo problem whose extra observations are synthesized rather than captured.

Load-bearing premise

The pipeline assumes that a diffusion model fine-tuned only on synthetic renders will generate physically consistent multi-light images for real in-the-wild photos, because the regressor is trained on rendered ground truth and the real-image results are shown only qualitatively.

Editorial extensions

If this is right

  • On the reported benchmark, normal estimation improves from a mean angular error of 8.03 degrees for the closest baseline to 6.41 degrees, and accuracy at the fine 3-degree threshold rises from 25.2% to 38.7%.
  • Material and relighting metrics also move in the same direction: albedo PSNR reaches 26.62 dB versus 23.88 dB, roughness 23.44 dB versus 17.25 dB, metallic 26.23 dB versus 22.00 dB, and relighting PSNR 30.12 dB versus 27.98 dB.
  • The ablation attributes the gain to the multi-light conditioning itself: moving from 0 to 9 light images lowers normal MAE from 7.56 to 4.06 degrees and raises 5-degree accuracy from 45.8% to 74.7%.
  • The reported per-frame cost is about 5 seconds, compared with 2 minutes for the optimization-based baseline, so the feed-forward pipeline is substantially faster while remaining physically grounded.
  • Because the model outputs normals and materials jointly, the relighting it produces can be re-rendered under arbitrary environment maps rather than being a purely generative image-to-image effect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is right, the same synthesize-extra-observations-then-regress recipe should transfer to other ill-posed inverse problems, such as depth from sparse views or SVBRDF from video, wherever a generative model can be made to emit consistent variations of the unknown.
  • The paper's data-mixing augmentation implies a self-improving loop: as the generator improves or is fine-tuned on real photometric-stereo captures, the downstream regressor can be retrained on the better synthetic images. The paper does not test this loop.
  • A direct testable prediction is that the nine generated lights will help most on specular and metallic objects and least on near-Lambertian ones, because multi-light cues mainly disambiguate reflectance and highlight structure; per-category breakdowns on LightProp would show this pattern.
  • The fixed nine-light layout means the method implicitly assumes one dominant point-light model per generated image; extending to continuous environment lighting would require the generator and regressor to be retrained over light-direction distributions rather than fixed angles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Neural LightRig, a two-stage framework for jointly estimating surface normals and PBR materials (albedo, roughness, metallic) from a single object image. In stage I, a Stable Diffusion-based model is fine-tuned on a new synthetic dataset (LightProp, 80k Objaverse objects rendered with Blender Cycles) to generate nine images of the same object under different point-light directions, arranged in a 3x3 grid to promote consistency. In stage II, a U-Net regression model takes the input image, the generated multi-light images, and their associated light orientations to predict an 8-channel G-buffer. The authors report state-of-the-art performance on a held-out LightProp subset for normal, material, and relighting metrics, and provide qualitative demonstrations on in-the-wild images. Ablations show that adding multi-light images and the proposed augmentation strategies improve accuracy, supporting the core mechanism.

Significance. If the claims hold, the idea of using a multi-light diffusion model to reduce the inherent ambiguity of single-image intrinsic estimation is a valuable contribution. The paper is clearly written, the ablations (Table 4) convincingly show that multi-light conditioning is the key driver of improvement, and the release of code and dataset would benefit the community. However, the quantitative state-of-the-art claim rests on an evaluation that is in-distribution for the proposed method and out-of-distribution for the baselines, and the cross-domain generalization to real images is only shown qualitatively. These issues currently limit the strength of the central claim.

major comments (3)
  1. [Sec. 4.1, Tables 1 and 2] The quantitative evaluation is performed on a held-out subset of LightProp, which is also the training distribution for both the multi-light diffusion model and the G-buffer model. The baselines (DSINE, GeoWizard, Marigold, etc.) are not trained on LightProp, so the comparison is asymmetric: the proposed method is tested in-distribution while the baselines are tested out-of-distribution. This asymmetry alone could explain part of the reported margin, and it means the claim that Neural LightRig 'surpasses existing approaches' is only established on the authors' own benchmark, not on generally harder or different distributions. To support the central claim, the authors should evaluate on a cross-dataset benchmark (e.g., a different synthetic renderer or real captured objects with ground-truth normals/materials) or, if such data are unavailable, explicitly temper the SOTA claim to the LightProp distribution and provide more substantial qualitative evidence of generalization.
  2. [Appendix A.2, Eqs. (6)-(7)] The light placement described by Eqs. (6)-(7) yields theta_0 = 0 and theta_8 = 2*pi. If theta is the polar angle relative to the overhead direction as stated in Sec. 3.3, these two angles correspond to the same physical direction (straight down the +z axis). Thus the setup provides at most 8 distinct lighting directions, not the claimed 9. This redundancy should be corrected, for example by using theta_i = i*pi/4 for i = 0..7 and a ninth distinct direction, or by clarifying the parameterization so that theta = 0 and theta = 2*pi are not coincident. The current description also affects the interpretation of the ablation in Table 4, where the 9-light configuration contains a duplicate direction.
  3. [Sec. 3.1 and Sec. 4.2] The claim that the multi-light diffusion model retains enough of Stable Diffusion's internet-scale prior to generate physically consistent lighting for in-the-wild inputs is not quantitatively validated. All quantitative results are on LightProp synthetic renders, and the only real-world evidence is qualitative (Sec. 4.2). The authors should provide a quantitative test on real images with known ground truth (e.g., light-stage captures or a small real photometric-stereo dataset) to verify that the generated multi-light images are physically consistent and that the downstream G-buffer predictions are accurate in the wild. Without this, the generalizability of the approach and the practical significance of the multi-light prior remain unsubstantiated.
minor comments (5)
  1. [Sec. 3] In the first paragraph of Sec. 3, 'the the under-constrained nature' contains a duplicated definite article.
  2. [Sec. 1] In the introduction, 'we proposed a data-driven approach' should be 'we propose' to match the present-tense style used elsewhere.
  3. [Table 2] The header 'A verage Time↓' contains an extra space and the capitalization is inconsistent with the rest of the table; it should read 'Average Time↓'.
  4. [References] Reference [1] (Stable Diffusion v2.1) has an improperly formatted URL with a line break in the middle of the link; this should be corrected to a single clickable URL.
  5. [Sec. 3.3] The statement 'the positions of light sources are fixed during the training of multi-light diffusion model g(·) and the inference of G-buffer prediction model f(·)' is repeated in a slightly different form in the appendix; the redundancy could be reduced by consolidating the description.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the pipeline is an empirical system trained and held-out evaluated on LightProp, with external baselines and no fitted parameter renamed as a prediction.

full rationale

The paper's claims are empirical rather than analytic: multi-light diffusion generates auxiliary lighting images, and a regression U-Net maps them together with the input to G-buffers. There is no equation in which a predicted quantity is defined in terms of the target, and no fitted parameter is later reported as a prediction. The central comparison is against external baselines (DSINE, GeoWizard, Marigold, StableNormal, RGB↔X, IntrinsicAnything, DiLightNet, IC-Light) on a held-out subset of LightProp, so the quantitative claim is benchmark-supported rather than reduced to a self-citation. The G-buffer model does mix diffusion-generated images into training data (Sec. 3.2, Data Mixing), and both stages use the same LightProp distribution, but this is a domain-alignment and in-distribution evaluation concern, not circular reasoning. The appendix light placement (Eqs. 6-7) indeed places light 8 at theta=2*pi, the same physical direction as light 0 at theta=0, so the 'nine-light' setup contains a redundant direction; this is a dataset-design flaw that undercuts the claimed diversity, but it does not make the derivation circular. No load-bearing self-citation or uniqueness theorem is invoked. Therefore, no specific circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the transfer of diffusion priors to synthetic multi-light generation and the representativeness of the LightProp synthetic domain; both are asserted rather than proven. The hand-chosen hyperparameters and fixed light layout are design choices that the ablations show influence results, but they are not free parameters in a derivation.

free parameters (4)
  • Number of multi-light images L = 9
    Chosen as a balance between performance and overhead (Sec 3.1). Ablation (Table 4) shows metrics improve from 0 to 3 to 6 to 9 images, so the specific choice affects the reported results.
  • Loss weighting for G-buffer training = 4:1 normal to PBR; MSE stabilization lambda_1 = 0.25
    Set by hand in Sec B.2. The balance between normal and PBR losses shapes the trained model's behavior and the reported metrics.
  • Augmentation trigger probabilities and noise scales = 0.6 for degradation/intensity/orientation; 0.3 for data mixing; N(0,0.1) for theta, N(0,0.02) for phi
    Hand-tuned in Sec B.2 to bridge the rendered-to-generated domain gap; Table 5 shows augmentation changes results substantially.
  • Fixed multi-light layout = theta_i = i*pi/4 for i=0..8; phi_i = {1,2,1,2,1,2,1,2,0}*pi/6
    A design choice in Sec A.2. The nine light directions define the conditioning signal; the G-buffer model sees this exact geometry at inference, so the layout is part of the method.
assumptions (3)
  • domain assumption Pre-trained Stable Diffusion v2-1 supplies sufficient illumination and shape priors that fine-tuning on synthetic relighting pairs transfers to in-the-wild objects.
    Invoked in Sec 3.1: 'we leverage the prior from a well-trained image diffusion model and fine-tune it for multi-light generation'. If this transfer fails, the multi-light images for real inputs are not physically consistent and the downstream G-buffer gain collapses.
  • domain assumption The LightProp synthetic distribution (Objaverse objects rendered in Blender Cycles) is representative enough of real objects for training and for benchmarking.
    Used in Sec 3.3 and A.1; all quantitative evaluation is on a held-out LightProp split. The paper provides only qualitative in-the-wild evidence for this assumption.
  • domain assumption Multiple point-light observations from a fixed set of directions reduce the ambiguity of normal and material estimation, following photometric stereo theory.
    Stated as motivation in Sec 1 and formalized by conditioning in Sec 3.2; the entire benefit of the multi-light generation rests on this established but unproved-in-paper assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion." pith.science (2026). https://pith.science/paper/LXCA4PID

@misc{pith2026241209593,
  author       = {Pith},
  title        = {Pith review of: Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LXCA4PID}},
  note         = {Machine review of arXiv:2412.09593}
}
read the original abstract

Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature. In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors. Specifically, 1) we first leverage illumination priors from large-scale diffusion models to build our multi-light diffusion model on a synthetic relighting dataset with dedicated designs. This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions. 2) By using these varied lighting images to reduce estimation uncertainty, we train a large G-buffer model with a U-Net backbone to accurately predict surface normals and materials. Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects. Code and dataset are available on our project page at https://projects.zxhezexin.com/neural-lightrig.

Figures

Figures reproduced from arXiv: 2412.09593 by the authors.

Figure 1
Figure 1. Neural LightRig takes an image as input and generates multi-light images to assist the estimation of high-quality normal and PBR materials, which can be used to render realistic relit images under various environment lighting. Abstract Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained na￾ture. In this paper, we present Neural LightRig, a novel framework … view at source ↗
Figure 2
Figure 2. Framework Overview. Multi-light diffusion generates multi-light images from an input image. These images with correspond￾ing lighting orientations are then used to predict surface normals and PBR materials with a regression U-Net. lem, as multiple combinations of materials and lighting con￾ditions could lead to the same appearance, Traditional meth￾ods attempted to employ photometric stereos [13, 51] to disambiguate… view at source ↗
Figure 3
Figure 3. Hybrid condition in multi-light diffusion. Input im [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: Qualitative comparison on surface normal estimation. Ground truth normals (G.T.) are provided for input images rendered from [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison on single-image relighting. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison on PBR material estimation. Ground truth materials (G.T.) are provided for input images rendered from [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Visualization of different conditioning strategies in [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Visualization of using different numbers of multi-light [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Failure case. φ i are perturbed with noise from N (0, 0.02) and clamped within [0, π 2 ]. The above three augmentations are triggered independently with a probability of 0.6. For Data Mixing, this augmentation is applied with a probability of 0.3. We generate multi-li…
Figure 12
Figure 12. Figure 12: More results of our method. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: More results of our method. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: More single-image relighting results of our method. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: More single-image relighting results of our method. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: More comparisons on surface normal estimation. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: More comparisons on PBR material estimation. [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: More comparisons on PBR material estimation. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: More comparisons on single-image relighting. [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Real-world Display Inverse Rendering Dataset

    cs.GR 2025-08 conditional novelty 7.0 of 10

    The paper introduces and evaluates the first real-world dataset for inverse rendering with a display-camera system, including a baseline method that recovers normals more accurately than prior methods.

  2. A Hybrid Neural-Microfacet BRDF Model for Real-Time Rendering

    cs.GR 2026-08 conditional novelty 6.0 of 10

    A hybrid BRDF model, combining a GGX analytical term with a tiny learned residual and gating network, fits measured materials more accurately than fully neural models at equal memory cost.

Reference graph

Works this paper leans on

64 extracted references · 58 canonical work pages · cited by 2 Pith papers

  1. [1]

    Stable diffusion v2.1

    Stability AI. Stable diffusion v2.1. https : / / huggingface . co / stabilityai / stable - diffusion-2-1, 2023. 4

    work page 2023

  2. [2]

    Gwangbin Bae and Andrew J. Davison. Rethinking inductive biases for surface normal estimation. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  3. [3]

    Es- timating and exploiting the aleatoric uncertainty in surface normal estimation

    Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Es- timating and exploiting the aleatoric uncertainty in surface normal estimation. In 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), page 13117–13126. IEEE,

    work page 2021

  4. [4]

    Marr revisited: 2d-3d alignment via surface normal prediction

    Aayush Bansal, Bryan Russell, and Abhinav Gupta. Marr revisited: 2d-3d alignment via surface normal prediction. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 5965–5974. IEEE, 2016. 2

    work page 2016

  5. [5]

    Freeman, and Varun Jampani

    Manel Baradad, Yuanzhen Li, Forrester Cole, Michael Ru- binstein, Antonio Torralba, William T. Freeman, and Varun Jampani. Background prompting for improved object depth,

    work page

  6. [6]

    Shape, albedo, and illumination from a single image of an unknown object

    Jonathan T Barron and Jitendra Malik. Shape, albedo, and illumination from a single image of an unknown object. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 334–341. IEEE, 2012. 2

    work page 2012

  7. [7]

    Barron, Ce Liu, and Hendrik P.A

    Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. Nerd: Neu- ral reflectance decomposition from image collections. In 2021 IEEE/CVF International Conference on Computer Vi- sion (ICCV). IEEE, 2021. 3

    work page 2021

  8. [8]

    Intrinsicanything: Learning diffusion priors for inverse rendering under unknown illumi- nation, 2024

    Xi Chen, Sida Peng, Dongchen Yang, Yuan Liu, Bowen Pan, Chengfei Lv, and Xiaowei Zhou. Intrinsicanything: Learning diffusion priors for inverse rendering under unknown illumi- nation, 2024. 3, 5

    work page 2024

Show all 64 references
  1. [9]

    Blender - a 3D modelling and rendering package

    Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 2, 12

    2018

  2. [10]

    Acquiring the reflectance field of a human face

    Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. In Proceedings of the 27th Annual Conference on Computer Graphics and In- teractive Techniques , page 145–156, USA, 2000. ACM Press/Ad...

    2000

  3. [11]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2023

  4. [12]

    Roumeliotis, and Hyun Soo Park

    Tien Do, Khiem Vuong, Stergios I. Roumeliotis, and Hyun Soo Park. Surface normal estimation of tilted images via spatial rectifier. In Proc. of the European Conference on Computer Vision, Virtual Conference, 2020. 2

    2020

  5. [13]

    Drbohlav and M

    O. Drbohlav and M. Chaniler. Can two specular pixels cal- ibrate photometric stereo? In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, pages 1850–1857 V ol. 2, 2005. 2, 3

    2005

  6. [14]

    Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans

    Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), page 10766–10776. IEEE, 2021. 2

    2021

  7. [15]

    Fouhey, Abhinav Gupta, and Martial Hebert

    David F. Fouhey, Abhinav Gupta, and Martial Hebert. Data- driven 3d primitives for single image understanding. In2013 IEEE International Conference on Computer Vision , pages 3392–3399, 2013. 2

    2013

  8. [16]

    Geowiz- ard: Unleashing the diffusion priors for 3d geometry estima- tion from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowiz- ard: Unleashing the diffusion priors for 3d geometry estima- tion from a single image. In ECCV, 2024. 2, 5

    2024

  9. [17]

    Shape, light, and material decomposition from images us- ing monte carlo rendering and denoising

    Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decomposition from images us- ing monte carlo rendering and denoising. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY , USA, 2024. Curran...

    2024

  10. [18]

    Openlrm: Open-source large reconstruction models

    Zexin He and Tengfei Wang. Openlrm: Open-source large reconstruction models. https://github.com/ 3DTopia/OpenLRM, 2023. 2

    2023

  11. [19]

    Efros, and Martial Hebert

    Derek Hoiem, Alexei A. Efros, and Martial Hebert. Auto- matic photo pop-up. ACM Trans. Graph. , 24(3):577–584,

  12. [20]

    Efros, and Martial Hebert

    Derek Hoiem, Alexei A. Efros, and Martial Hebert. Recov- ering surface layout from an image. International Journal of Computer Vision: Special Issue on Celebrating Kanade’s Vision, 75(1):151 – 172, 2007. 2

    2007

  13. [21]

    LRM: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3d. In The Twelfth International Conference on Learning Representations, 2024. 2

    2024

  14. [22]

    Ma- terial anything: Generating materials for any 3d object via diffusion

    Xin Huang, Tengfei Wang, Ziwei Liu, and Qing Wang. Ma- terial anything: Generating materials for any 3d object via diffusion. arXiv, 2024. 3

    2024

  15. [23]

    Neural gaffer: Relighting any object via diffusion

    Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. In Advances in Neural Information Processing Systems, 2024. 2, 3

    2024

  16. [24]

    Multi-view photometric stereo revis- ited

    Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Fer- rari, and Luc Van Gool. Multi-view photometric stereo revis- ited. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), page 3125–3134. IEEE, 2023. 3

    2023

  17. [25]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 5 9

    2024

  18. [26]

    Dis- criminatively trained dense surface normal estimation

    L’ubor Ladick ´y, Bernhard Zeisl, and Marc Pollefeys. Dis- criminatively trained dense surface normal estimation. In ECCV, pages 468–484. Springer International Publishing,

  19. [27]

    Modular primitives for high-performance differentiable rendering

    Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transac- tions on Graphics, 39(6), 2020. 2

    2020

  20. [28]

    The digital michelangelo project: 3d scanning of large statues

    Marc Levoy, Kari Pulli, Brian Curless, Szymon Rusinkiewicz, David Koller, Lucas Pereira, Matt Ginz- ton, Sean Anderson, James Davis, Jeremy Ginsberg, Jonathan Shade, and Duane Fulk. The digital michelangelo project: 3d scanning of large statues. In Proceedings of the 27th Annu...

    2000

  21. [29]

    Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs

    Bo Li, Chunhua Shen, Yuchao Dai, Anton van den Hen- gel, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. In 2015 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 1119– 1...

    2015

  22. [30]

    Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model

    Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In The Twelfth International Conference on Learning Represe...

    2024

  23. [31]

    Daniel Lichy, Jiaye Wu, Soumyadip Sengupta, and David W. Jacobs. Shape and material capture at home. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 6119–6129. IEEE, 2021. 3

    2021

  24. [32]

    One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion

    Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Ji- ayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In 2024 IEEE/CVF Conference on Computer Vi- sio...

    2024

  25. [33]

    Zero-1-to-3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3d object. In 2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , page 9264–9275. IEEE, 2023. 2

    2023

  26. [34]

    Unsupervised learning for intrinsic image decomposition from a single im- age

    Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. Unsupervised learning for intrinsic image decomposition from a single im- age. In CVPR, 2020. 2, 3

    2020

  27. [35]

    Syncdreamer: Gen- erating multiview-consistent images from a single-view im- age

    Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Gen- erating multiview-consistent images from a single-view im- age. In The Twelfth International Conference on Learning Representations, 2024. 2

    2024

  28. [36]

    Diffusion posterior illumination for ambiguity-aware inverse rendering

    Linjie Lyu, Ayush Tewari, Marc Habermann, Shun- suke Saito, Michael Zollh ¨ofer, Thomas Leimk ¨uehler, and Christian Theobalt. Diffusion posterior illumination for ambiguity-aware inverse rendering. ACM Transactions on Graphics, 42(6), 2023. 3

    2023

  29. [37]

    Xiaojuan Qi, Zhengzhe Liu, Renjie Liao, Philip H. S. Torr, Raquel Urtasun, and Jiaya Jia. Geonet++: Iterative geo- metric neural network with edge-aware refinement for joint depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 4...

  30. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 2

    2022

  31. [39]

    Blip-2, opt-2.7b, pre-trained only

    Salesforce. Blip-2, opt-2.7b, pre-trained only. https: //huggingface.co/Salesforce/blip2-opt-2. 7b, 2023. 5

    2023

  32. [40]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Confer- ence on Learning Representations, 2022. 4

    2022

  33. [41]

    Chandraker

    Shen Sang and M. Chandraker. Single-shot neural relighting and svbrdf estimation. In ECCV, 2020. 3

    2020

  34. [42]

    Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. Learning non-lambertian object intrinsics across shapenet categories. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5844–5853, 2017. 3

    2017

  35. [43]

    Zero123++: a single image to consistent multi-view dif- fusion base model, 2023

    Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view dif- fusion base model, 2023. 2, 3

    2023

  36. [44]

    MVDream: Multi-view diffusion for 3d gen- eration

    Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d gen- eration. In The Twelfth International Conference on Learn- ing Representations, 2024. 2

    2024

  37. [45]

    Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T

    Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. Nerv: Neural reflectance and visibility fields for relight- ing and view synthesis. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , page 74...

    2021

  38. [46]

    Lgm: Large multi-view gaussian model for high-resolution 3d content creation

    Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, P...

    2024

  39. [47]

    Matsynth: A modern pbr materials dataset

    Giuseppe Vecchio and Valentin Deschaintre. Matsynth: A modern pbr materials dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 3

    2024

  40. [48]

    Vplnet: Deep single view nor- mal estimation with vanishing points and lines

    Rui Wang, David Geraghty, Kevin Matzen, Richard Szeliski, and Jan-Michael Frahm. Vplnet: Deep single view nor- mal estimation with vanishing points and lines. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 686–695, 2020. 2

    2020

  41. [49]

    Pretraining is all you need for image-to-image translation

    Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. In arXiv, 2022. 2

    2022

  42. [50]

    Phidias: A generative 10 model for creating 3d content from text, image, and 3d con- ditions with reference-augmented diffusion

    Zhenwei Wang, Tengfei Wang, Zexin He, Gerhard Hancke, Ziwei Liu, and Rynson WH Lau. Phidias: A generative 10 model for creating 3d content from text, image, and 3d con- ditions with reference-augmented diffusion. arXiv preprint arXiv:2409.11406, 2024. 2

    2024 arXiv

  43. [51]

    Robert J. Woodham. Photometric method for determining surface orientation from multiple images , page 513–531. MIT Press, Cambridge, MA, USA, 1989. 2, 3

    1989

  44. [52]

    What matters when repurposing diffusion models for general dense perception tasks?, 2024

    Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. What matters when repurposing diffusion models for general dense perception tasks?, 2024. 2

    2024

  45. [53]

    Stablenormal: Reducing diffusion variance for stable and sharp normal

    Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 2024. 2, 5

    2024

  46. [54]

    Weakly-supervised single-view image relighting

    Renjiao Yi, Chenyang Zhu, and Kai Xu. Weakly-supervised single-view image relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8402–8411, 2023. 2, 3, 5

    2023

  47. [55]

    Ye Yu and William A. P. Smith. Inverserendernet: Learning single image inverse rendering. In 2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , page 3150–3159. IEEE, 2019. 3

    2019

  48. [56]

    Dilightnet: Fine-grained light- ing control for diffusion-based image generation

    Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained light- ing control for diffusion-based image generation. In ACM SIGGRAPH 2024 Conference Papers, 2024. 2, 3, 5

    2024

  49. [57]

    Rgb ↔x: Image decomposition and synthe- sis using material- and lighting-aware diffusion models

    Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Miloˇs Ha ˇsan. Rgb ↔x: Image decomposition and synthe- sis using material- and lighting-aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers, New York, N...

    2024

  50. [58]

    Neilf++: Inter-reflectable light fields for geometry and material es- timation

    Jingyang Zhang, Yao Yao, Shiwei Li, Jingbo Liu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. Neilf++: Inter-reflectable light fields for geometry and material es- timation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023. 3

    2023

  51. [59]

    Reference-only control

    Lyumin Zhang. Reference-only control. https : / / github.com/Mikubill/sd-webui-controlnet/ discussions/1236, 2023. 3

    2023

  52. [60]

    Ic-light github page, 2024

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Ic-light github page, 2024. 5

    2024

  53. [61]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 5

    2018

  54. [62]

    Pattern-affinitive propagation across depth, surface normal and semantic segmentation

    Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Pattern-affinitive propagation across depth, surface normal and semantic segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 4101–4110. IEEE, 2019. 2 11 Appendi...

    2019

  55. [128]

    Training on 32 NVIDIA A100 (80G) GPUs, this phase completes in approximately 20 hours

    This phase includes 1, 500 warm-up steps, a peak learning rate of 1 × 10−4, and a weight decay of 0.01, us- ing a cosine annealing schedule and the AdamW optimizer with β1 = 0.9 and β2 = 0.999. Training on 32 NVIDIA A100 (80G) GPUs, this phase completes in approximately 20 hou...

  56. [224]

    Each block contains two residual layers with Group Normalization (using 32 groups), and SiLU activation. At- tention mechanisms, implemented in a pre-norm style , are applied in all but the first down-sampling block and the last up-sampling block, using an attention head dimen...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.