Pith. sign in

REVIEW 2 major objections 3 minor 60 references

Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation

T0 review · 2 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that jointly relighting all input photos to a reference illumination with a multiview diffusion model, then fitting a radiance field with per-image normal perturbations, reconstructs shiny 3D objects from photos taken…

desk verdict A genuinely useful system paper—joint multiview relighting plus normal-warping shading embeddings—that clearly beats prior work, though the robustness mechanism is thinner than the results suggest. read the letter →

arxiv 2412.15211 v1 pith:HDVMEQK4 submitted 2024-12-19 cs.CV

Hadi Alzayer , Philipp Henzler , Jonathan T. Barron , Jia-Bin Huang , Pratul P. Srinivasan , Dor Verbin This is my paper · ORCID
classification cs.CV
keywords 3Dreconstructionmultiviewrelightingdiffusionmodelsneuralradiancefieldsspecularappearanceview-dependenteffectsimageharmonizationinverserendering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reconstructing a 3D object from photographs is hard when each photo was taken under different lighting, because the object's appearance changes from image to image. This paper argues that the right move is to remove that difficulty first: relight every input image to match one chosen reference image, using a diffusion model (a generative machine-learning model that rewrites an image under a new illumination) that processes all views together instead of one at a time, and only then reconstruct a radiance field from the harmonized images. The authors' key claim is that the residual errors left by even a good joint relighting model are mostly small warps in specular highlights, so they equip the radiance field with a per-image shading embedding that slightly re-aims the surface normals used for shading. With that mechanism, the method recovers sharp, view-dependent reflections from images captured under extreme illumination variation, where per-image appearance embeddings tend to flatten everything into diffuse color and inverse-rendering methods blur. Quantitative comparisons on synthetic and real benchmarks support the claim that this pipeline outperforms prior methods.

What carries the argument

Two components carry the argument. The first is a multiview latent diffusion model: it denoises the latent code of each view simultaneously, attending across all views with 3D self-attention and conditioning on each camera pose (encoded as raymaps) and on a binary reference mask that marks which image supplies the target illumination. This joint denoising yields relit images that share one consistent interpretation of geometry and material. The second is the shading embedding: a per-image vector that, together with the radiance field's geometry feature, is fed through a three-layer MLP to produce the surface normal used for shading that image. Because only the normal is per-image, the model can shift a specular highlight by a few degrees from one view to the next, absorbing the diffusion model's small normal-estimation errors while keeping a single albedo and geometry. The reconstruction backbone is NeRF-Casting, which models view-dependent reflections by tracing secondary reflected rays into the same radiance field.

What would settle it

Render a synthetic object under extreme lighting, run the relighting model, fit the radiance field, and inspect the residual images between the relit training views and the model's renderings: if large residuals concentrate in diffuse, shadow, or geometry-edge regions rather than near specular highlights, the normal-perturbation assumption is wrong. A direct control is to modify ground-truth relit images by intentionally warping their normals versus by additively changing diffuse albedo, and verify that the shading embedding absorbs the former but not the latter.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 3D scene can be reconstructed from inconsistently lit photographs by first jointly relighting all views with a single multiview diffusion model to match a reference illumination, then fitting a NeRF-Casting-style radiance field whose only per-image adaptation is a shading embedding that perturbs the surface normals used for reflection. The discovery reported is that joint relighting makes the images consistent enough for this to work, and that the residual inconsistencies are dominated by specular highlights being slightly tilted, an error the shading embedding can absorb without corrupting diffuse appearance. Rendered novel views retain sharp, moving reflections, and the method outperforms prior diffusion-relighting, appearance-embedding, and inverse-rendering baselines on synthetic Objaverse scenes and on real NA VI photographs.

Load-bearing premise

The load-bearing premise is that the residual errors of the joint relighting model are mostly small warps of specular highlights caused by slightly wrong surface normals, so a per-image normal perturbation is enough to make the relit images consistent, and that usable object masks and accurate camera poses are available, which are hardest to obtain for the shiny objects this method targets.

Editorial extensions

If this is right

  • Specular and mirror-like objects, which prior per-illumination-embedding and inverse-rendering methods reduce to diffuse blobs, can be reconstructed at high fidelity simply from photos taken under very different lights.
  • The method needs no known environment map and no constant-lighting input; any input image can serve as the reference, so the user chooses the illumination under which the recovered object is viewed.
  • Increasing the number of views relit jointly, from 1 to 8 to 64, steadily improves reconstruction quality, suggesting the approach benefits from larger multi-view captures rather than saturating.
  • Training the relighting model on purely mirror-like objects improves performance even for mostly diffuse objects, so high-quality synthetic shiny data is a broadly useful training signal.
  • The hard problem of jointly estimating geometry and varying illumination decomposes into relighting then reconstruction, a division that future methods can build on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The relight-then-reconstruct decomposition might extend beyond objects to whole scenes: if a multiview diffusion model can harmonize entire scenes, the same shading embedding should prevent flicker in indoor or outdoor reconstructions where lights change between captures.
  • The shading embedding effectively converts the diffusion model's geometry estimate into per-image normal corrections; a testable extension is to read out these corrections to improve the diffusion model itself, for instance by fine-tuning on the radiance field's predicted normals.
  • Because the paper flags camera pose estimation for shiny objects as the hardest unsolved input, coupling the radiance field's inferred normals with learning-based pose estimators seems a natural next step, though the paper only suggests it implicitly.
  • Real internet photo collections of the same object rarely come with masks; a practical extension would be to estimate soft masks jointly, since the mask loss used on NA VI suggests the pipeline tolerates imprecise masks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper proposes a two-stage pipeline for 3D reconstruction from a set of posed images of an object captured under differing, potentially extreme illuminations. In the first stage, a multiview latent diffusion model jointly relights all N input images to match the illumination of a selected reference image, using camera poses encoded as raymaps and a reference mask to condition the denoising process. In the second stage, the relit images are fed into a NeRF-Casting-based radiance field that incorporates a per-image "shading embedding" (Eq. 1) to perturb surface normals, with the aim of absorbing small residual inconsistencies, particularly warped specular highlights, in the relit images. The method is evaluated on synthetic Objaverse scenes and real NA VI scenes, with quantitative results reported in Tables 1 and 2 and qualitative comparison in Figures 1, 4, and 5. The paper also presents ablations on the number of jointly relit frames (Table 3), on training with mirrored materials (Table 4), and on the choice of per-image embedding (Table 5), all of which support the proposed design choices.

Significance. If the claims hold, this is a practically useful advance: it avoids the ambiguities of inverse rendering, handles highly specular objects that prior methods render as mostly diffuse, and does not require a target environment map, unlike IllumiNeRF and Neural Gaffer. The strongest evidence is the ablations: joint relighting clearly beats single-frame relighting (Table 3), mirrored-material training data helps both shiny and standard assets (Table 4), and the shading embedding outperforms both no embedding and standard appearance embeddings (Table 5). The synthetic-to-real generalization demonstrated on NA VI is encouraging. The paper's claims are empirical rather than derivational, and no formal proofs are offered; the main value is in the system and the evaluation. The work is incremental relative to recent relighting-plus-reconstruction pipelines, but the joint multiview relighting formulation and the shading-embedding idea are plausible and well-motivated contributions.

major comments (2)
  1. [Section 4 / Section 5] The synthetic evaluation may not be an independent test of the relighting model. Section 4 states that the relighting model is trained on "a dataset of ~300K high-quality objects, similar in appearance to Objaverse but with more diverse materials," while Section 5 states that the synthetic benchmark is "objects from Objaverse." The relationship between these two datasets is not specified. If any of the 20 test objects in Table 1, or close derivatives thereof, appear in the training set, then the reported PSNR/SSIM/LPIPS numbers are not evidence of generalization to unseen objects. Please state explicitly whether the training asset dataset is disjoint from the Objaverse test split, and if not, report results on a held-out set of objects that are guaranteed absent from training. This is load-bearing because Table 1 is the central quantitative claim of the paper.
  2. [Section 3.2 / Figure S2 / Table 5] The central assumption that relighting residuals are "usually due to the specular highlights being slightly warped" is supported only by the qualitative light-probe visualization in Figure S2. That evidence is specific to a spherical, perfectly mirror-like probe and does not characterize the error distribution for textured objects with diffuse albedo, cast shadows, or complex geometry. If the diffusion model also produces diffuse color shifts, shadow-boundary errors, or invented/missing reflections, the shading embedding in Eq. (1) cannot absorb them, because it only warps normals and cannot change the local diffuse color or the environment content. The 0.52 dB improvement over no embedding on shiny assets in Table 5 is modest support that this mechanism fully resolves the inconsistency problem. Please provide a quantitative analysis of relighting residuals, e.g., by decomposing per-pixel errors into diffuse-color, shadow-displacement, and specular-warp components on textured synthetic scenes, and show that the shading embedding specifically reduces the specular-warp component while not harming the others. This would substantiate the load-bearing premise of Section 3.2.
minor comments (3)
  1. [Section 5.2] In the discussion of Table 5, the sentence "using standard embeddings as in NeRF-W [32] ('Shading embeddings')" appears to contain a typo: the table row is labeled "Appearance embeddings," so the parenthetical should read "Appearance embeddings" rather than "Shading embeddings."
  2. [Section 3.1 / Figure 2] The caption of Figure 2 says "leftmost pane" but the figure layout is described as multiple panes; please clarify the intended ordering, as the relighting diffusion model is shown before the radiance field in the current caption.
  3. [Section 4] The classifier-free guidance scale is fixed at 3 and no sensitivity study is reported; since CFG is a user-set hyperparameter that can strongly affect the consistency and sharpness of diffusion outputs, please state whether the chosen value was tuned and, if so, on which validation set.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the two-stage pipeline is an empirical system evaluated on held-out views against external baselines; the shading-embedding assumption is an empirical modeling choice, not a derived prediction.

full rationale

The paper's derivation chain is not circular. The relighting model is trained on synthetic renders, and the radiance field is optimized on relit training images; novel-view results are evaluated on held-out test poses rendered under the reference illumination. The per-image shading embedding of Eq. 1 is fit to training views but is rendered at test time with the reference embedding, so it is not re-fit to the test targets. Comparisons are made against NeROIC, NeRF-Casting + AE, and IllumiNeRF on Objaverse and NA VI, so the quantitative claims are anchored to external data and baselines. The load-bearing assumption in Section 3.2 that relighting residuals are mostly warped specular highlights is motivated by the light-probe visualization in Figure S2; that evidence is qualitative and narrow, and if diffuse, shadow, or albedo errors occur the shading embedding would be insufficient. This is a correctness or robustness risk, not a circularity. The Limitations paragraph conceding the need for masks and accurate poses is similarly a scope limitation, not a circular step. Self-citations to NeRF-Casting, CAT3D, and IllumiNeRF provide architectural components and baselines; none of the central claims is justified solely by a self-citation chain or by a uniqueness theorem imported from the authors. Train/test overlap on Objaverse-like synthetic data is a possible evaluation-leakage concern, but the paper states the training assets are 'similar in appearance to Objaverse but with more diverse materials' rather than identifying them with the test assets, and no equation or quoted pipeline step makes the reported numbers equal to the training objective by construction. Under the hard rule requiring a quoted reduction, no such reduction exists here.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the empirical validity of the relighting diffusion model, the shading embedding's ability to absorb its residual errors, and transfer of a synthetic-trained model to real captures. The shading embedding is the only per-scene fitted quantity of note and is a modeling choice rather than a physical constant. No new physical entities are postulated.

free parameters (4)
  • Per-image shading embedding v_i = Learned per scene during radiance field optimization; dimension not stated in the main text
    Eq. 1: n_i(x) = normalize(MLP(f(x), v_i)). One code per training image, optimized to absorb specular-highlight warping in the relit images. This is the mechanism that makes reconstruction robust to relighting inconsistencies, and its capacity and regularization are tuned per scene.
  • Number of jointly relit frames N = 64
    Section 4 and Table 3 show performance improves monotonically with N, and 64 is the practical maximum used. This is a compute and capacity choice rather than a constant fitted to a target metric.
  • Mask loss weight lambda_mask = 0.01 for NA VI scenes
    Supplement Eq. S4. Chosen by hand to handle the imprecise camera poses and masks in NA VI, with a 7-pixel boundary erosion to avoid penalizing uncertain mask edges.
  • Classifier-free guidance scale = 3.0
    Section 4 states a CFG value of 3 for all experiments. A standard sampling hyperparameter, not fitted to the evaluation benchmarks.
assumptions (4)
  • domain assumption Residual relighting errors are predominantly specular-highlight warping, correctable by per-image normal perturbations.
    Section 3.2: 'the subtle inter-image errors made by the diffusion model are usually due to the specular highlights being slightly warped'. If errors also corrupt albedo or geometry, the shading embedding cannot fix them.
  • domain assumption The synthetic-trained relighting diffusion model generalizes to real captured photographs.
    Section 4 trains purely on synthetic data; Section 5 validates on NA VI and new captures. The real-data results support this, but the training corpus is undisclosed, so the transfer claim is only partially verifiable.
  • domain assumption Object masks and accurate camera poses are available for all inputs.
    Section 6 Limitations states these are required. They are hardest to obtain precisely for highly specular objects, which are the paper's main showcase.
  • domain assumption A Stable Diffusion 1.5-style latent backbone with 3D self-attention and raymap pose conditioning can learn to jointly relight up to 64 views consistently.
    Section 3.1 and Figure S1. Consistency is validated empirically by Table 3, but there is no theoretical guarantee that cross-view attention yields a unified material interpretation.
invented entities (1)
  • Per-image shading embedding (normal-warping code)
    purpose: Warp surface normals per training image to absorb specular-highlight inconsistencies in the relit images (Eq. 1).
    The only evidence is the paper's own ablation (Table 5) and the synthetic light-probe visualization (Figure S2). No falsifiable prediction outside the paper's benchmarks is made for this component.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation." pith.science (2026). https://pith.science/paper/HDVMEQK4

@misc{pith2026241215211,
  author       = {Pith},
  title        = {Pith review of: Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HDVMEQK4}},
  note         = {Machine review of arXiv:2412.15211}
}
read the original abstract

Reconstructing the geometry and appearance of objects from photographs taken in different environments is difficult as the illumination and therefore the object appearance vary across captured images. This is particularly challenging for more specular objects whose appearance strongly depends on the viewing direction. Some prior approaches model appearance variation across images using a per-image embedding vector, while others use physically-based rendering to recover the materials and per-image illumination. Such approaches fail at faithfully recovering view-dependent appearance given significant variation in input illumination and tend to produce mostly diffuse results. We present an approach that reconstructs objects from images taken under different illuminations by first relighting the images under a single reference illumination with a multiview relighting diffusion model and then reconstructing the object's geometry and appearance with a radiance field architecture that is robust to the small remaining inconsistencies among the relit images. We validate our proposed approach on both synthetic and real datasets and demonstrate that it greatly outperforms existing techniques at reconstructing high-fidelity appearance from images taken under extreme illumination variation. Moreover, our approach is particularly effective at recovering view-dependent "shiny" appearance which cannot be reconstructed by prior methods.

Figures

Figures reproduced from arXiv: 2412.15211 by the authors.

Figure 1
Figure 1. 3D reconstruction under extreme illumination variation. We propose a method for 3D reconstruction from a set of images captured under strongly varying illumination. Our method recovers high-fidelity appearance details including specular highlights that prior state-of-the-art approaches cannot recover (top baseline: NeRF-Casting [45] with appearance embeddings, bottom baseline: NeROIC [25]). Abstract Reconstructing t… view at source ↗
Figure 2
Figure 2. Method overview. We first apply a relighting diffusion model that converts N images I1, ..., IN with known camera poses π1, ..., πN , captured under extremely different illuminations, to a set of images with the same poses, but rendered under the illumi￾nation of the reference image I1 (highlighted in orange). We then optimize a neural radiance field to obtain a consistent 3D repre￾sentation with a novel per-image s… view at source ↗
Figure 3
Figure 3. A comparison of our multiview relighting with prior work on single-image relighting. Our method first relights a set of inconsistently-lit images (one of which is shown in (a)) to match the illumination of a selected reference image (b) in that set. Single-image relighting techniques such as IllumiNeRF [55] (c) struggle to disambiguate geometry, lighting, and materials, leading to an inaccurate relighting. In contra… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visual comparison of novel view renderings on the Objaverse dataset. (b) We show sample input images under extreme illumination variation. (c) Adding a per-image latent code to NeRF-Casting [45] (“NeRFCast + AE”) cannot accurately explain away the variations, leading t…
Figure 5
Figure 5. Figure 5: Comparison on real world photos. We use our method to reconstruct objects from in-the-wild photos taken in different environments. Our method can render novel views under the illumination conditions of any input image we select as the reference. Unlike prior work, our …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 55 canonical work pages

  1. [1]

    Learning neural light fields with ray-space embedding

    Benjamin Attal, Jia-Bin Huang, Michael Zollh ¨ofer, Johannes Kopf, and Changil Kim. Learning neural light fields with ray-space embedding. CVPR, 2022. 2

    work page 2022

  2. [2]

    Barron, Matthew O’Toole, and Pratul P

    Benjamin Attal, Dor Verbin, Ben Mildenhall, Peter Hed- man, Jonathan T. Barron, Matthew O’Toole, and Pratul P. Srinivasan. Flash Cache: Reducing Bias in Radiance Cache Based Inverse Rendering. ECCV, 2024. 3

    work page 2024

  3. [3]

    Barron, Ben Mildenhall, Dor Verbin, Pratul P

    Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR, 2022. 2

    work page 2022

  4. [4]

    Barron, Ben Mildenhall, Dor Verbin, Pratul P

    Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-NeRF: Anti-aliased grid- based neural radiance fields. ICCV, 2023. 2

    work page 2023

  5. [5]

    Tenenbaum, A

    Harry Barrow, J. Tenenbaum, A. Hanson, and E. Riseman. Recovering intrinsic scene characteristics. Computer Vision Systems, 1978. 2

    work page 1978

  6. [6]

    Eikonal fields for refractive novel-view synthesis

    Mojtaba Bemana, Karol Myszkowski, Jeppe Revall Frisvad, Hans-Peter Seidel, and Tobias Ritschel. Eikonal fields for refractive novel-view synthesis. SIGGRAPH, 2022. 2

    work page 2022

  7. [7]

    Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milo ˇs Ha ˇsan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi

    Sai Bi, Zexiang Xu, Pratul P. Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milo ˇs Ha ˇsan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv:2008.03824, 2020. 3

    arXiv 2008

  8. [8]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. CVPR, 2023. 4

    work page 2023

Show all 60 references
  1. [9]

    Optimizing the latent space of generative net- works

    Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the latent space of generative net- works. ICML, 2018. 2

    2018

  2. [10]

    Bar- ron, Ce Liu, and Hendrik P.A

    Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Bar- ron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural Re- flectance Decomposition from Image Collections. ICCV,

  3. [11]

    Barron, and Hendrik P.A

    Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P.A. Lensch. Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposi- tion. NeurIPS, 2021. 3

    2021

  4. [12]

    Scene coordinate reconstruction: Pos- ing of image collections via incremental learning of a relo- calizer

    Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cav- allari, ´Aron Monszpart, Daniyar Turmukhambetov, and Vic- tor Adrian Prisacariu. Scene coordinate reconstruction: Pos- ing of image collections via incremental learning of a relo- calizer. In ECCV, 2024. 8

    2024

  5. [13]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. CVPR, 2023. 2, 6

    2023

  6. [14]

    Bar- ron, Hendrik P.A

    Andreas Engelhardt, Amit Raj, Mark Boss, Yunzhi Zhang, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan T. Bar- ron, Hendrik P.A. Lensch, and Varun Jampani. SHINOBI: Shape and Illumination using Neural Object decomposition via BRDF optimization In-the-wild. CVPR, 2024. 3

    2024

  7. [15]

    Srinivasan, Jonathan T

    Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole. CAT3D: Create anything in 3D with multi-view diffusion models. NeurIPS, 2024. 4

    2024

  8. [16]

    Learning to predict indoor illumi- nation from a single image

    Marc-Andre Gardner, Kalyan Sunkavalli, Ersin Yumer, Xi- aohui Shen, Emiliano Gambaretto, Christian Gagne, and Jean-Francois Lalonde. Learning to predict indoor illumi- nation from a single image. SIGGRAPH Asia, 2017. 3

    2017

  9. [17]

    Fast spatially-varying in- door lighting estimation

    Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean-Francois Lalonde. Fast spatially-varying in- door lighting estimation. CVPR, 2019. 3

    2019

  10. [18]

    https://hdrmaps.com

    HDR Maps. https://hdrmaps.com

  11. [19]

    Multiview photometric stereo

    Carlos Hern ´andez, George V ogiatzis, and Roberto Cipolla. Multiview photometric stereo. IEEE TPAMI, 2008. 3

    2008

  12. [20]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022. 5

    2022 arXiv

  13. [21]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. NeurIPS, 2022. 4

    2022

  14. [22]

    NA VI: Category- agnostic image collections with high-quality 3D shape and pose annotations

    Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engel- hardt, Arjun Karpur, Karen Truong, Kyle Sargent, Stefan Popov, Andre Araujo, Ricardo Martin-Brualla, Kaushal Pa- tel, Daniel Vlasic, Vittorio Ferrari, Ameesh Makadia, Ce Liu, Yuanzhen Li, and Howard Zhou. NA VI: Category- ag...

    2023

  15. [23]

    TensoIR: Tensorial inverse rendering

    Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Song- fang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. TensoIR: Tensorial inverse rendering. CVPR, 2023. 3

    2023

  16. [24]

    Neural gaffer: Relighting any object via diffusion

    Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. NeurIPS, 2024. 2, 3, 5

    2024

  17. [25]

    NeROIC: Neural Rendering of Objects from Online Image Collections

    Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. NeROIC: Neural Rendering of Objects from Online Image Collections. SIGGRAPH, 2022. 1, 2, 3, 6, 7

    2022

  18. [26]

    Estimating the natural illumination conditions from a single outdoor image

    Jean-Franc ¸ois Lalonde, Alexei A Efros, and Srinivasa G Narasimhan. Estimating the natural illumination conditions from a single outdoor image. IJCV, 2012. 3

    2012

  19. [27]

    Deep- light: Learning illumination for unconstrained mobile mixed reality

    Chloe LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, and Paul Debevec. Deep- light: Learning illumination for unconstrained mobile mixed reality. CVPR, 2019. 3

    2019

  20. [28]

    ENVIDR: Im- plicit differentiable renderer with neural environment light- ing

    Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Sel- vakumar Panneer, and Nandita Vijaykumar. ENVIDR: Im- plicit differentiable renderer with neural environment light- ing. ICCV, 2023. 2

    2023

  21. [29]

    9 NeRO: Neural Geometry and BRDF Reconstruction of Re- flective Objects from Multiview Images

    Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. 9 NeRO: Neural Geometry and BRDF Reconstruction of Re- flective Objects from Multiview Images. SIGGRAPH, 2023. 3

    2023

  22. [30]

    SpecNeRF: Gaussian directional encoding for specular reflections

    Li Ma, Vasu Agrawal, Haithem Turki, Changil Kim, Chen Gao, Pedro Sander, Michael Zollh ¨ofer, and Christian Richardt. SpecNeRF: Gaussian directional encoding for specular reflections. CVPR, 2024. 2

    2024

  23. [31]

    Neural microfacet fields for inverse render- ing

    Alexander Mai, Dor Verbin, Falko Kuester, and Sara Fridovich-Keil. Neural microfacet fields for inverse render- ing. ICCV, 2023. 3

    2023

  24. [32]

    Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duck- worth. NeRF in the Wild: Neural Radiance Fields for Un- constrained Photo Collections. CVPR, 2021. 2, 4, 5, 8

    2021

  25. [33]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV, 2020. 2

    2020

  26. [34]

    Physi- cally based rendering: From theory to implementation

    Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physi- cally based rendering: From theory to implementation. MIT Press, 2023. 2

    2023

  27. [35]

    DiffusionLight: Light probes for free by painting a chrome ball.CVPR, 2023

    Pakkapon Phongthawee, Worameth Chinchuthakun, Non- taphat Sinsunthithet, Amit Raj, Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakorn. DiffusionLight: Light probes for free by painting a chrome ball.CVPR, 2023. 3

    2023

  28. [36]

    A Diffu- sion Approach to Radiance Field Relighting using Multi- Illumination Synthesis

    Yohan Poirier-Ginter, Alban Gauthier, Julien Philip, Jean- Franc ¸ois Lalonde, and George Drettakis. A Diffu- sion Approach to Radiance Field Relighting using Multi- Illumination Synthesis. Computer Graphics Forum , 2024. 3

    2024

  29. [37]

    https://polyhaven.com

    Poly Haven. https://polyhaven.com. 5

  30. [38]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. CVPR, 2022. 5

    2022

  31. [39]

    Scene representation transformer: Geometry-free novel view syn- thesis through set-latent scene representations

    Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani V ora, Mario Luˇci´c, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view syn- thesis through set-latent scene representations. CVPR...

    2022

  32. [40]

    Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T

    Pratul P. Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T. Barron, Richard Tucker, and Noah Snavely. Lighthouse: Predicting lighting volumes for spatially- coherent illumination. CVPR, 2020. 3

    2020

  33. [41]

    Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T

    Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. CVPR, 2021. 3

    2021

  34. [42]

    Neural-pbir reconstruction of shape, material, and illumination

    Cheng Sun, Guangyan Cai, Zhengqin Li, Kai Yan, Cheng Zhang, Carl Marshall, Jia-Bin Huang, Shuang Zhao, and Zhao Dong. Neural-pbir reconstruction of shape, material, and illumination. ICCV, 2023. 3

    2023

  35. [43]

    Srinivasan, Jonathan T

    Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Prad- han, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Bar- ron, and Henrik Kretzschmar. Block-NeRF: Scalable Large Scene Neural View Synthesis. CVPR, 2022. 2

    2022

  36. [44]

    Barron, and Pratul P

    Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022. 2

    2022

  37. [45]

    Srinivasan, Peter Hedman, Ben Milden- hall, Benjamin Attal, Richard Szeliski, and Jonathan T

    Dor Verbin, Pratul P. Srinivasan, Peter Hedman, Ben Milden- hall, Benjamin Attal, Richard Szeliski, and Jonathan T. Bar- ron. NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections. SIGGRAPH Asia, 2024. 1, 2, 3, 4, 5, 6, 7

    2024

  38. [46]

    DUSt3R: Geometric 3D vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D vision made easy. CVPR, 2024. 8

    2024

  39. [47]

    Novel view synthesis with diffusion models

    Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv:2210.04628, 2022. 4

    2022 arXiv

  40. [48]

    Neural directional encoding for efficient and accurate view- dependent appearance modeling

    Liwen Wu, Sai Bi, Zexiang Xu, Fujun Luan, Kai Zhang, Iliyan Georgiev, Kalyan Sunkavalli, and Ravi Ramamoorthi. Neural directional encoding for efficient and accurate view- dependent appearance modeling. CVPR, 2024. 2

    2024

  41. [49]

    Accidental Light Probes

    Hong-Xing Yu, Samir Agarwala, Charles Herrmann, Richard Szeliski, Noah Snavely, Jiajun Wu, and Deqing Sun. Accidental Light Probes. CVPR, 2023. 3

    2023

  42. [50]

    DiLightNet: Fine-grained lighting control for diffusion-based image generation

    Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. DiLightNet: Fine-grained lighting control for diffusion-based image generation. SIG- GRAPH, 2024. 3

    2024

  43. [51]

    Monst3r: A simple approach for estimat- ing geometry in the presence of motion

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jam- pani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming- Hsuan Yang. Monst3r: A simple approach for estimat- ing geometry in the presence of motion. arXiv preprint arXiv:2410.03825, 2024. 8

    arXiv

  44. [52]

    Cameras as rays: Pose estimation via ray diffusion

    Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. ICLR, 2024. 8

    2024

  45. [53]

    PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting

    Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. CVPR, 2021. 3

    2021

  46. [54]

    Srinivasan, Boyang Deng, Paul Debevec, William T

    Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. NeR- Factor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination. SIGGRAPH Asia, 2021. 3

    2021

  47. [55]

    Srinivasan, Dor Verbin, Keunhong Park, Ricardo Martin Brualla, and Philipp Henzler

    Xiaoming Zhao, Pratul P. Srinivasan, Dor Verbin, Keunhong Park, Ricardo Martin Brualla, and Philipp Henzler. IllumiN- eRF: 3D Relighting Without Inverse Rendering. NeurIPS,

  48. [57]

    We use the same learning rate schedule as in NeRF- Casting

    We optimize our NeRF for 25K iterations rather than 50K. We use the same learning rate schedule as in NeRF- Casting

  49. [58]

    We initialize density around exp(−1) instead of exp(2)

  50. [59]

    in [45], we set m = 16 and s = 50

    We use a faster coarse-to-fine rate: using the notation from Appendix C.1. in [45], we set m = 16 and s = 50

  51. [60]

    Finally, for scenes from NA VI [22], which have imper- cise camera poses, we found that adding a simple mask loss improved our results

    We remove the view direction as input into the color pre- diction network. Finally, for scenes from NA VI [22], which have imper- cise camera poses, we found that adding a simple mask loss improved our results. For a ray with rendering weights {wi}N i=1 we use the following lo...

  52. [2024]

    re- flection features

    2, 3, 5, 6, 7, 1 10 Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation Supplementary Material Relighting Diffusion Model raymaps reference mask conditioning input denoising vector relighting output Figure S1. Relighting model overview. F...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.