REVIEW 3 major objections 6 minor 9 cited by
DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Video diffusion models do inverse and forward rendering from one video
desk verdict A genuinely new diffusion-based renderer with strong synthetic results, but real-world claims rest on qualitative evidence and a self-labeling loop that deserves a caveat. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the video diffusion model. The forward renderer concatenates five VAE-encoded G-buffer latents (normals, depth, base color, roughness, metallic) as pixel-aligned conditioning to the denoising UNet, while an environment-map encoder converts HDR lighting into multi-resolution cross-attention features so lighting is queried at each UNet level rather than concatenated as flat pixels. The inverse renderer reuses the same architecture, concatenating the input video latent and using five optimizable domain embeddings to generate one buffer at a time. A synthetic data pipeline of 150,000 path-traced videos with ground-truth G-buffers and environment maps supplies supervision, and a LoRA is added during training on auto-labeled real data.
What would settle it
Take a set of real indoor and outdoor scenes, capture each under several known HDR lighting conditions, have the inverse renderer predict G-buffers from a single video, then run the forward renderer with the captured target light and measure PSNR, SSIM, and LPIPS against the true relit frames. If the synthetic-trained inverse estimates deviate strongly from measured normals and albedo, or the relit output is no closer to ground truth than per-frame image baselines, the central generalization claim would fail.
Extended reading notes
Core claim
On the paper's own terms, DiffusionRenderer establishes that a video diffusion model can act as a neural approximation of path-traced shading: conditioned on camera-space normals, normalized depth, base color, roughness, metallic maps, and HDR environment-map lighting, the forward model generates photorealistic images and temporally consistent videos without explicit light transport. The companion inverse model, the same video diffusion architecture with a per-attribute domain embedding, estimates those G-buffers from RGB video, and although it is trained only on synthetic data it is claimed to generalize to real-world video. That generalization is exploited: auto-labeled real videos are used as training data for the forward renderer, with a LoRA adapter absorbing the domain difference between synthetic and real data. The result is a three-step editing pipeline—estimate buffers, edit buffers or lighting, re-render—that supports relighting, material editing, and object insertion from a single video.
Load-bearing premise
The claim rests on the assumption that the inverse renderer, trained only on synthetic data, produces accurate G-buffer estimates on real-world video, because those estimates become the training labels for the forward renderer; the paper supports this assumption with qualitative examples rather than real-world ground truth.
Editorial extensions
If this is right
- If the central claim holds, relighting a real scene requires only a single input video: G-buffer estimation plus a target environment map, with no 3D reconstruction needed.
- The forward renderer's tolerance for imperfect G-buffers means editing can happen directly in buffer space, making material edits and object insertions that would break a classic PBR pipeline feasible.
- Because both models are video models, output is temporally consistent across frames, addressing a common artifact of per-frame image relighting.
- Training with auto-labeled real video expands the effective training distribution beyond synthetic data, improving real-world rendering without expensive ground-truth capture.
- A one-step deterministic variant of the inverse renderer is competitive for photometric accuracy, suggesting the approach can be made much cheaper at inference.
Reading between the lines
- Editorial: If the synthetic-trained inverse renderer generalizes in less curated settings, the same auto-labeling loop could extend to arbitrary online video, turning unlabeled footage into a scalable source of render-training data.
- Editorial: Because the forward renderer is conditioned on noisy G-buffers, a testable extension is to corrupt G-buffers at training time in a controlled way to quantify robustness and build error-aware editing tools.
- Editorial: The paper's quantitative evidence is strongest on synthetic benchmarks; a natural next experiment is measuring relighting fidelity on real scenes with a captured ground-truth target light, which would directly stress the weakest assumption.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces DiffusionRenderer, a unified framework with two video diffusion models: a neural inverse renderer that estimates per-pixel G-buffers (normals, depth, albedo, roughness, metallic) from RGB video, and a neural forward renderer that synthesizes photorealistic video from G-buffers and an HDR environment map, replacing explicit light transport. The inverse model is trained on synthetic data plus public intrinsics datasets, then applied to auto-label 150,000 real-world clips from DL3DV10k; the forward model is jointly trained on synthetic data and these pseudo-labels with a LoRA adapter for real data. Experiments report strong quantitative results on synthetic benchmarks for forward rendering, inverse rendering, and relighting, plus qualitative real-world relighting, material editing, and object insertion.
Significance. If the real-world claims hold, this is a substantial contribution: it extends image-based neural deferred shading to a video-diffusion setting, proposes a lighting conditioning method via multi-resolution environment-map features in cross-attention, and demonstrates a single framework for relighting, material editing, and object insertion from a single video. Strengths include a large curated synthetic dataset (150,000 videos with ground-truth G-buffers), systematic ablations of video vs. image models and of lighting encoders, and supplementary evaluation with ColorVideoVDP temporal consistency and a human/GPT-4V preference study. The synthetic quantitative evidence is consistent and often large-margin across all three tasks.
major comments (3)
- [4.3, 4.4, Eq. (4)] The forward renderer's real-world supervision is produced by the paper's own inverse renderer and the off-the-shelf DiffusionLight estimator, and no independent real-world ground truth is used to validate either stage. The claim in Sec. 4.3 that 'our inverse rendering model generalizes to real-world videos' is supported only by qualitative examples (Fig. 5), and the same model is then used as the labeler for the forward renderer. Because systematic errors in the pseudo-labels are inherited by the forward renderer, and the LoRA term f_{θ+Δθ} can only adapt appearance rather than correct a systematically wrong supervision signal, the real-world relighting and editing claims in Secs. 5.4 and 5.5 are not squarely supported. Please add a quantitative real-world evaluation on a dataset with ground-truth target lighting (e.g., MIT Multi-Illumination or a multi-view relighting benchmark) and/or a real-world intrinsic-image benchmark with human annotations (e.g., IIW or SAW), and report agreement between the inverse renderer's pseudo-labels and an independent estimator or a set of manual annotations.
- [Tables 1-4] No variance estimates are reported. PSNR, SSIM, and LPIPS are single numbers, and several comparisons are relatively close (e.g., Table 1 SyntheticObjects: Ours 28.3 vs. SSRT 29.4 and SplitSum 28.7; Table 2 SyntheticObjects: Ours 27.50 vs. Neural Gaffer 26.39). Without error bars, it is impossible to assess whether the claimed improvements are statistically significant. Please report mean and standard deviation over at least three runs, or bootstrap confidence intervals over test scenes, for all quantitative tables.
- [5.1, 5.2, 5.4] All quantitative evaluation for forward rendering, inverse rendering, and relighting is performed on synthetic test sets rendered by the authors' own OptiX/AgX pipeline, which may be similar in distribution to the training synthetic data. The real-world evaluation is qualitative only. The abstract and conclusion claim the method 'consistently outperforming the state-of-the-art' without scope qualification. Please either add quantitative real-world metrics or explicitly restrict the headline quantitative claims to synthetic benchmarks, with a clear statement that real-world performance is currently supported only qualitatively.
minor comments (6)
- [Abstract] The phrase 'approximates inverse and forwards rendering' contains a typo; it should be 'forward rendering'.
- [4.1] The paper defines normals in camera space but does not explain how the inverse renderer's output normal maps are transformed or kept in the same coordinate frame as the forward renderer's condition; please clarify the normal-space convention and its normalization.
- [Fig. 2] The caption references 'SSRT side view' and 'SSRT relit' but does not clearly describe what each subfigure displays; please expand the caption so the failure cases are understandable without the main text.
- [Supplement Table S2] Several GPT-4V preference scores are close to chance (e.g., 52-57% in the Objects rows); please discuss whether these differences are meaningful or present confidence intervals for the preference percentages.
- [Eq. (7)] The environment map auto-encoder objective reconstructs LDR and log-space maps but not the directional encoding; a brief sentence explaining why Edir is only an input to the direction query encoder and not a reconstruction target would improve clarity.
- [4.2] Eq. (3) uses g0 without explicitly stating that it is the VAE-encoded latent of the attribute map; please add a sentence defining the notation for g0 and z0 in both Eq. (3) and Eq. (4).
Circularity Check
No significant circularity: held-out synthetic benchmarks independently ground the central claims; the real-world auto-labeling loop is a validation gap, not a by-construction reduction.
full rationale
DiffusionRenderer's central quantitative claims are supported by held-out synthetic benchmarks with ground-truth G-buffers and path-traced references (Tables 1–4, Figs. 4 and 6), so the forward-rendering and inverse-rendering results do not reduce to the model's own outputs. The real-world pipeline does contain a self-training loop—the inverse renderer labels DL3DV10k (Sec. 4.3) and the forward renderer is trained on those labels (Sec. 4.4, Eq. 4), with real-world relighting shown only qualitatively—but this is an independent-validation gap, not a by-construction equivalence: the forward renderer is trained to reconstruct the original video from the labels and then asked to synthesize a novel lighting condition (Eq. 6), which is not one of its training targets. No fitted parameter is renamed as a prediction, no load-bearing premise is supplied solely by a self-citation, and the self-citations present (e.g., [44], [75]) are baselines or application components rather than unverified uniqueness or ansatz claims. The paper's own limitation note about auto-labeling accuracy (Sec. 6) correctly identifies the residual risk, which belongs to correctness/robustness assessment rather than circularity.
Assumptions & free parameters
free parameters (5)
- Classifier-free guidance scale =
1.2
- Denoising steps at inference =
20 for forward; 20 for inverse, with 1-step deterministic ablation
- Training iterations and batch size =
20,000 iterations, batch size 256
- Real-data LoRA =
trainable delta-theta on real data
- Condition dropout =
0.1
assumptions (5)
- domain assumption Stable Video Diffusion provides a reusable video prior that can be fine-tuned for conditional G-buffer-to-video and video-to-G-buffer tasks.
- ad hoc to paper The inverse renderer trained only on synthetic data generalizes to real-world videos.
- ad hoc to paper Auto-labeled G-buffers and DiffusionLight environment maps are accurate enough to supervise the forward renderer.
- standard math The rendering equation and Disney BRDF parameterization are an adequate model for the target imagery.
- domain assumption Diffusion denoising with EDM schedule and Stable Diffusion VAE compression preserves sufficient detail for photorealistic rendering.
Cite this review
Pith. "Pith review of DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models." pith.science (2026). https://pith.science/paper/UNW5EBMS
@misc{pith2026250118590,
author = {Pith},
title = {Pith review of: DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/UNW5EBMS}},
note = {Machine review of arXiv:2501.18590}
}
read the original abstract
Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 9 Pith papers
-
Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh
Tracking plus world-position maps in a neural G-buffer outperform depth as a geometric condition for reference-guided video diffusion rendering on a 68-clip synthetic benchmark.
-
Video Generation Models are General-Purpose Vision Learners
A video-diffusion backbone fine-tuned as a single-step multi-task perceiver matches or beats specialists on depth, normals, pose and segmentation, with high data efficiency and sim-to-real transfer.
-
LuxDiT: Lighting Estimation with Video Diffusion Transformer
A video diffusion transformer fine-tuned on synthetic and real data predicts HDR environment maps from images/videos, cutting peak light-direction error by roughly 45% on sunny outdoor scenes versus DiffusionLight.
-
InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modeling
InvRGB+L jointly estimates visible and LiDAR albedo with a physics-based specular LiDAR model and cross-modal consistency losses, improving inverse rendering and LiDAR intensity simulation for urban and indoor scenes.
-
UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
Jointly predicting albedo and relit appearance with one video-diffusion pass improves relighting fidelity and generalization over two-stage inverse-plus-forward pipelines.
-
Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models
Post-trained Cosmos world models generate controllable multi-view driving videos and LiDAR; augmenting real AV training data with these synthetic clips improves downstream perception and policy metrics, especially in ...
-
MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation
A feed-forward two-stage compositing framework that harmonizes inserted objects across views using a Hilbert-ordered Gaussian color mapping, trained and evaluated on a new 480k-scene synthetic dataset.
-
Bridging Rendering and Generative Modeling with Monte Carlo Transport Scheduling
A common variance-time SDE aligns Monte Carlo rendering noise with diffusion-model denoising, enabling low-spp render refinement and stage-ordered material control.
-
StableIntrinsic: Detail-preserving One-step Diffusion Model for Multi-view Material Estimation
StableIntrinsic estimates albedo, roughness, and metallic maps from multi-view RGB images in a single diffusion step, achieving higher PSNR and lower MSE than prior multi-step diffusion methods.
Reference graph
Works this paper leans on
-
[1]
Cosmos world foundation model platform for physical ai
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575,
-
[2]
Barron and Jitendra Malik
Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence, 37(8):1670–1687, 2014. 1, 2
2014
-
[3]
Harry Barrow and J. M. Tenenbaum. Recovering intrinsic scene characteristics from images, 1978. 1
1978
-
[4]
Harry Barrow, J. M. Tenenbaum, A. Hanson, and E. Riseman. Recovering intrinsic scene characteristics. Comput. Vis. Syst, 2:3–26, 1978. 2
1978
-
[5]
Intrinsic images in the wild
Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33(4): 159, 2014. 2, 6, 7
2014
-
[6]
Anand Bhattad, Daniel McKee, Derek Hoiem, and D. A. Forsyth. Stylegan knows normal, depth, albedo, and more,
-
[7]
Anand Bhattad, James Soole, and D.A. Forsyth. Stylitgan: Image-based relighting via latent control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3
2024
-
[8]
Stable video diffusion: Scaling latent video diffusion models to large datasets
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 3, 4, 1
arXiv 2023
Show all 91 references
-
[9]
Lensch, and Jan Kautz
Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. Two-shot spatially-varying BRDF and shape estimation. In CVPR, 2020. 2
2020
-
[10]
Bar- ron, Ce Liu, and Hendrik P.A
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Bar- ron, Ce Liu, and Hendrik P.A. Lensch. NeRD: neural re- flectance decomposition from image collections. In ICCV,
-
[11]
User- assisted intrinsic images
Adrien Bousseau, Sylvain Paris, and Frédo Durand. User- assisted intrinsic images. In ACM Transactions on Graphics (TOG), page 130. ACM, 2009. 2
2009
-
[12]
Physically-based shading at Disney
Brent Burley. Physically-based shading at Disney. In ACM SIGGRAPH, pages 1–7, 2012. 3
2012
-
[13]
DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer
Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer. In NeurIPS, 2021. 3
2021
-
[14]
Cook and Kenneth E
Robert L. Cook and Kenneth E. Torrance. A reflectance model for computer graphics. ACM Transactions on Graphics (ToG), 1(1):7–24, 1982. 3
1982
-
[15]
The triangle processor and normal vector shader: a VLSI system for high performance graphics
Michael Deering, Stephanie Winner, Bic Schediwy, Chris Duffy, and Neil Hunt. The triangle processor and normal vector shader: a VLSI system for high performance graphics. ACM Trans. on Graphics, 22(4):21–30, 1988. 2, 3
1988
-
[16]
Objaverse: A universe of annotated 3D objects
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3D objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...
2023
-
[17]
FlashTex: fast relightable mesh texturing with LightControl- Net
Kangle Deng, Timothy Omernick, Alexander Weiss, Deva Ra- manan, Jun-Yan Zhu, Tinghui Zhou, and Maneesh Agrawala. FlashTex: fast relightable mesh texturing with LightControl- Net. In European Conference on Computer Vision (ECCV),
-
[18]
Diffusion models beat GANs on image synthesis
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021. 3
2021
-
[19]
Generative models: What do they know? do they know things? let’s find out!, 2024
Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, and Anand Bhattad. Generative models: What do they know? do they know things? let’s find out!, 2024. 3
2024
-
[20]
Spotlight: Shadow-guided object relighting via diffusion, 2024
Frédéric Fortier-Chouinard, Zitian Zhang, Louis-Etienne Messier, Mathieu Garon, Anand Bhattad, and Jean-François Lalonde. Spotlight: Shadow-guided object relighting via diffusion, 2024. 2
2024
-
[21]
Ge- oWizard: unleashing the diffusion priors for 3D geometry estimation from a single image
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Ge- oWizard: unleashing the diffusion priors for 3D geometry estimation from a single image. In ECCV, 2024. 4, 5, 6
2024
-
[22]
Out- cast: Outdoor single-image relighting with cast shadows
David Griffiths, Tobias Ritschel, and Julien Philip. Out- cast: Outdoor single-image relighting with cast shadows. In Computer Graphics Forum, pages 179–193. Wiley Online Library, 2022. 3
2022
-
[23]
Johnson, Edward H
Roger Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman. Ground truth dataset and baseline evalu- ations for intrinsic image algorithms. In ICCV, pages 2335–
-
[24]
Shape, light, and material decomposition from images us- ing Monte Carlo rendering and denoising
Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decomposition from images us- ing Monte Carlo rendering and denoising. arXiv:2206.03380,
-
[25]
Deep-learning the latent space of light trans- port
Pedro Hermosilla, Sebastian Maisch, Tobias Ritschel, and Timo Ropinski. Deep-learning the latent space of light trans- port. In Computer Graphics Forum, pages 207–217. Wiley Online Library, 2019. 2
2019
-
[26]
Denoising diffu- sion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 3
2020
-
[27]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 5, 8
2022
-
[28]
Deep radiance caching: Convolutional autoencoders deeper in ray tracing
Giulio Jiang and Bernhard Kainz. Deep radiance caching: Convolutional autoencoders deeper in ray tracing. Computers & Graphics, 94:22–31, 2021. 2
2021
-
[29]
Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaox- iao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5322–5332,
-
[30]
Neural gaffer: Relighting any object via diffusion
Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. In Advances in Neural Information Processing Systems, 2024. 3, 4, 6, 7, 8, 2
2024
-
[31]
Deep scattering: Rendering at- mospheric clouds with radiance-predicting neural networks
Simon Kallweit, Thomas Müller, Brian Mcwilliams, Markus Gross, and Jan Novák. Deep scattering: Rendering at- mospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics (TOG), 36(6):1–11, 2017. 2
2017
-
[32]
Real shading in Unreal Engine 4
Brian Karis. Real shading in Unreal Engine 4. ACM SIGGRAPH Course on Physically Based Shading Theory and Practice, 4(3):1, 2013. 6
2013
-
[33]
Elucidating the design space of diffusion-based generative models
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022. 3, 5
2022
-
[34]
Repurpos- ing diffusion-based image generators for monocular depth estimation
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024...
2024
-
[35]
3D Gaussian splatting for real-time radi- ance field rendering
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radi- ance field rendering. ACM Transactions on Graphics, 42(4),
-
[36]
In- trinsic image diffusion for single-view material estimation
Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. In- trinsic image diffusion for single-view material estimation. In arxiv, 2023. 3, 6, 7
2023
-
[37]
LightIt: illumination modeling and control for diffusion models
Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, and Yannick Hold-Geoffroy. LightIt: illumination modeling and control for diffusion models. In CVPR, 2024. 3, 8
2024
-
[38]
Shading annotations in the wild
Balazs Kovacs, Sean Bell, Noah Snavely, and Kavita Bala. Shading annotations in the wild. In CVPR, pages 6998–7007,
-
[39]
Lightness and retinex theory
Edwin H Land and John J McCann. Lightness and retinex theory. Josa, 61(1):1–11, 1971. 2
1971
-
[40]
Cgintrinsics: Better intrinsic image decomposition through physically-based rendering
Zhengqi Li and Noah Snavely. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. In ECCV, pages 371–387, 2018. 2
2018
-
[41]
Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image
Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In CVPR, pages 2475–2484,
-
[42]
OpenRooms: an end-to-end open framework for photorealistic indoor scene datasets
Zhengqin Li, Ting-Wei Yu, Shen Sang, Sarah Wang, Sai Bi, Zexiang Xu, Hong-Xing Yu, Kalyan Sunkavalli, Miloš Hašan, Ravi Ramamoorthi, et al. OpenRooms: an end-to-end open framework for photorealistic indoor scene datasets. arXiv preprint arXiv:2007.12868, 2020. 2
2007 arXiv
-
[43]
Envidr: Implicit differentiable renderer with neural environment lighting
Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Sel- vakumar Panneer, and Nandita Vijaykumar. Envidr: Implicit differentiable renderer with neural environment lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 79–89, 2023. 3
2023
-
[44]
Photorealistic object insertion with diffusion-guided inverse rendering
Ruofan Liang, Zan Gojcic, Merlin Nimier-David, David Acuna, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. Photorealistic object insertion with diffusion-guided inverse rendering. In ECCV, 2024. 3, 2
2024
-
[45]
Gs-ir: 3d gaussian splatting for inverse rendering
Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. arXiv preprint arXiv:2311.16473, 2023. 3
2023 arXiv
-
[46]
UrbanIR: large-scale urban scene inverse rendering from a single video
Zhi-Hao Lin, Bohan Liu, Yi-Ting Chen, David Forsyth, Jia- Bin Huang, Anand Bhattad, and Shenlong Wang. UrbanIR: large-scale urban scene inverse rendering from a single video. arXiv preprint arXiv:2306.09349, 2023. 6, 3, 5
2023 arXiv
-
[47]
DL3DV-10K: a large-scale scene dataset for deep learning-based 3D vision
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. DL3DV-10K: a large-scale scene dataset for deep learning-based 3D vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...
2024
-
[48]
Efros, and Noah Snavely
Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros, and Noah Snavely. Learning to factorize and relight a city. In ECCV, 2020. 3
2020
-
[49]
Wonder3D: sin- gle image to 3D using cross-domain diffusion
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3D: sin- gle image to 3D using cross-domain diffusion. arXiv preprint arXiv:2310.15008, 2023. 4
-
[50]
Mantiuk, Param Hanji, Maliha Ashraf, Yuta Asano, and Alexandre Chapiro
Rafal K. Mantiuk, Param Hanji, Maliha Ashraf, Yuta Asano, and Alexandre Chapiro. Colorvideovdp: A visual difference predictor for image, video and display distortions. ACM Trans. Graph., 43(4), 2024. 3
2024
-
[51]
Fine- tuning image-conditional diffusion models is easier than you think
Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, and Bastian Leibe. Fine- tuning image-conditional diffusion models is easier than you think. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV), 2025. 8
2025
-
[52]
Srinivasan, Matthew Tancik, Jonathan T
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: representing scenes as neural radiance fields for view synthe- sis. arXiv preprint arXiv:2003.08934, 2020. 2
2003 arXiv
-
[53]
Real-time neural radiance caching for path tracing
Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. Real-time neural radiance caching for path tracing. arXiv preprint arXiv:2106.12372, 2021. 2
2021 arXiv
-
[54]
Extracting triangular 3D models, materials, and lighting from images
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fi- dler. Extracting triangular 3D models, materials, and lighting from images. arXiv:2111.12503, 2021. 3
2021 arXiv
-
[55]
A multi-illumination dataset of indoor object appear- ance
Lukas Murmann, Michael Gharbi, Miika Aittala, and Fredo Durand. A multi-illumination dataset of indoor object appear- ance. In 2019 IEEE International Conference on Computer Vision (ICCV), 2019. 3
2019
-
[56]
Seidel, and Tobias Ritschel
Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, H.-P. Seidel, and Tobias Ritschel. Deep shading: convolutional neu- ral networks for screen space shading. In Computer graphics forum, pages 65–78. Wiley Online Library, 2017. 2, 3
2017
-
[57]
Total relighting: Learning to relight portraits for background replacement
Rohit Pandey, Sergio Orts-Escolano, Chloe LeGendre, Chris- tian Haene, Sofien Bouaziz, Christoph Rhemann, Paul De- bevec, and Sean Fanello. Total relighting: Learning to relight portraits for background replacement. In ACM Transactions on Graphics (Proceedings SIGGRAPH), 2021. 3
2021
-
[58]
Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich
Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. OptiX: a general purpose ray tracing engine. ACM Trans. Graph., 29(4), 2010. 5, 1
2010
-
[59]
Physically Based Rendering - From Theory to Implementation
Matt Pharr, Wenzel Jacob, and Greg Humphreys. Physically Based Rendering - From Theory to Implementation. Morgan Kaufmann, fourth edition, 2023. 1, 3
2023
-
[60]
Multi-view relighting using a geometry- aware network
Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A Efros, and George Drettakis. Multi-view relighting using a geometry- aware network. ACM Trans. Graph., 38(4):78–1, 2019. 3
2019
-
[61]
DiffusionLight: light probes for free by painting a chrome ball
Pakkapon Phongthawee, Worameth Chinchuthakun, Non- taphat Sinsunthithet, Amit Raj, Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakorn. DiffusionLight: light probes for free by painting a chrome ball. In ArXiv,
-
[62]
A Diffusion Ap- proach to Radiance Field Relighting using Multi-Illumination Synthesis
Yohan Poirier-Ginter, Alban Gauthier, Julien Philip, Jean- François Lalonde, and George Drettakis. A Diffusion Ap- proach to Radiance Field Relighting using Multi-Illumination Synthesis. Computer Graphics Forum, 2024. 3
2024
-
[63]
Learning transferable visual models from natural language supervi- sion
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...
2021
-
[64]
Susskind
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Ku- mar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (ICCV) 2021,
2021
-
[65]
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 5
2022
-
[66]
NeRF for outdoor scene relighting
Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. NeRF for outdoor scene relighting. In ECCV, 2022. 3
2022
-
[67]
Jacobs, and Jan Kautz
Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. In ICCV, 2019. 2
2019
-
[68]
Gir: 3d gaussian inverse ren- dering for relightable scene factorization
Yahao Shi, Yanmin Wu, Chenming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang, Jian Zhang, Bin Zhou, et al. Gir: 3d gaussian inverse ren- dering for relightable scene factorization. arXiv preprint arXiv:2312.05133, 2023. 3
2023 arXiv
-
[69]
Deep unsupervised learning us- ing nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning us- ing nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015. 3
2015
-
[70]
State of the art on neural rendering
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitz- mann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin- Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. In Computer Graphics Forum, pages 701–727. Wiley Online Library, 2020. 2
2020
-
[71]
Robust Monte Carlo methods for light transport simulation
Eric Veach. Robust Monte Carlo methods for light transport simulation. Stanford University, 1998. 3
1998
-
[72]
Marschner, Hongsong Li, and Ken- neth E
Bruce Walter, Stephen R. Marschner, Hongsong Li, and Ken- neth E. Torrance. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques, page 195–206, 2007. 3
2007
-
[73]
Learn- ing indoor inverse rendering with 3D spatially-varying light- ing
Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. Learn- ing indoor inverse rendering with 3D spatially-varying light- ing. In ICCV, 2021. 2
2021
-
[74]
Neural light field estimation for street scenes with differentiable virtual object insertion
Zian Wang, Wenzheng Chen, David Acuna, Jan Kautz, and Sanja Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In ECCV, 2022. 2
2022
-
[75]
Neural fields meet explicit geometric representa- tions for inverse rendering of urban scenes
Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representa- tions for inverse rendering of urban scenes. In The IEEE Conference on Computer Vision and Pattern ...
2023
-
[76]
De-rendering 3D objects in the wild
Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. De-rendering 3D objects in the wild. In CVPR, 2022. 2
2022
-
[77]
Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation
Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation. In CVPR, 2024. 3
2024
-
[78]
Chen Xi, Peng Sida, Yang Dongchen, Liu Yuan, Pan Bowen, Lv Chengfei, and Zhou. Xiaowei. IntrinsicAnything: learning diffusion priors for inverse rendering under unknown illumi- nation. arxiv: 2404.11593, 2024. 3
2024 arXiv
-
[79]
Luminet: Latent intrinsics meets diffu- sion models for indoor scene relighting, 2024
Xiaoyan Xing, Konrad Groh, Sezer Karaoglu, Theo Gevers, and Anand Bhattad. Luminet: Latent intrinsics meets diffu- sion models for indoor scene relighting, 2024. 3
2024
-
[80]
Ye Yu and William A. P. Smith. InverseRenderNet: learning single image inverse rendering. In CVPR, 2019. 2
2019
-
[81]
Poly Haven - The Public 3D Asset Library, 2024
Greg Zaal and et al. Poly Haven - The Public 3D Asset Library, 2024. 6
2024
-
[82]
DiLightNet: fine-grained light- ing control for diffusion-based image generation
Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. DiLightNet: fine-grained light- ing control for diffusion-based image generation. In ACM SIGGRAPH 2024 Conference Papers, 2024. 3, 6, 8, 2
2024
-
[83]
RGB ↔X: image decomposition and synthesis using material-and lighting-aware diffusion models
Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Miloš Hašan. RGB ↔X: image decomposition and synthesis using material-and lighting-aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024....
2024
-
[84]
PhySG: Inverse rendering with spherical Gaus- sians for physics-based material editing and relighting
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical Gaus- sians for physics-based material editing and relighting. In CVPR, 2021. 3
2021
-
[85]
IRON: inverse rendering by optimizing neural SDFs and ma- terials from photometric images
Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. IRON: inverse rendering by optimizing neural SDFs and ma- terials from photometric images. In CVPR, 2022. 3
2022
-
[86]
Efros, Eli Shecht- man, and Oliver Wang
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 6
2018
-
[87]
Srinivasan, Boyang Deng, Paul Debevec, William T
Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. NeR- Factor: neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6):1–18, 2021. 3
2021
-
[88]
Zerocomp: Zero- shot object compositing from image intrinsics via diffusion,
Zitian Zhang, Frédéric Fortier-Chouinard, Mathieu Garon, Anand Bhattad, and Jean-François Lalonde. Zerocomp: Zero- shot object compositing from image intrinsics via diffusion,
-
[89]
A closed-form solution to retinex with nonlocal texture constraints
Qi Zhao, Ping Tan, Qiang Dai, Li Shen, Enhua Wu, and Stephen Lin. A closed-form solution to retinex with nonlocal texture constraints. TPAMI, 34(7):1437–1444, 2012. 2
2012
-
[90]
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W. Jacobs. Deep single-image portrait relighting. In ICCV, 2019. 3
2019
-
[91]
What is in this image? Describe the materials. Be concise and produce an answer with a few sentences, no more than 50 words
Jingsen Zhu, Fujun Luan, Yuchi Huo, Zihao Lin, Zhihua Zhong, Dianbing Xi, Rui Wang, Hujun Bao, Jiaxiang Zheng, and Rui Tang. Learning-based inverse rendering of complex indoor scenes with differentiable Monte Carlo raytracing. In SIGGRAPH Asia 2022 Conference Papers. ACM, 2022...
2022
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.