{"id":"8c46ba38-7b40-4706-8d24-ea1d8634a12e","arxiv_id":"2412.09593","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A diffusion model creates multi-light images from one photo, and a U-Net uses them to predict object normals and PBR materials, improving single-image inverse rendering on synthetic benchmarks.","lead":"Neural LightRig generates nine differently lit versions of a single object image with a diffusion model, then uses those images to predict the object's surface normals and material maps. The method reports large gains over prior work on synthetic tests, but real-world performance is only shown qualitatively.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on same-distribution evaluation: Tables 1-2 use held-out LightProp objects for Ours while baselines are not trained on LightProp, so cross-distribution generalization and the asserted real-world advantage are not quantitatively established.","rationale":"The reader's weakest assumption correctly identifies the real-world generalization gap: the multi-light diffusion is fine-tuned only on LightProp, and the real-world results are only qualitative. I agree this is load-bearing. My concern extends it: even the synthetic quantitative comparison is confounded because Ours is trained on LightProp while the baselines are not, so the reported superiority may reflect distribution match rather than the multi-light mechanism. This is why I say 'partial' rather than full agreement. At the same time, the paper deserves credit for a genuine ablation: Table 4 shows that adding rendered multi-light images sharply improves G-buffer prediction within the training distribution, and the augmentation study (Table 5) shows awareness of the render-to-diffusion domain gap. These are real empirical supports for the mechanism, so the paper is not without merit. The concern does not demand rejection; it demands broader evaluation: a cross-dataset synthetic benchmark and at least one real-image quantitative benchmark would settle whether the SOTA claim generalizes. The reader's conditional verdict remains appropriate, so I do not change the verdict.","tokens_in":14640,"tokens_out":8083,"duration_ms":88054,"concrete_test":"Evaluate the full pipeline on 1,000 held-out objects from an external PBR dataset (e.g., Google Scanned Objects or 3D-Front, rendered with the same Blender setup) and on a real single-image normal benchmark such as DiLiGenT. Recompute the metrics of Tables 1 and 2. If the margin over baselines shrinks substantially or real-image normal accuracy is not competitive, the SOTA claim is likely a LightProp-specific artifact rather than a generalizable multi-light prior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim 'Neural LightRig surpasses existing approaches' to hold in a meaningful sense, the method must outperform baselines outside the exact renderer and data distribution used for training. The paper does not establish this. Tables 1 and 2 measure performance on a held-out subset of LightProp, the same Blender Cycles PBR dataset used to train both the multi-light diffusion model (Sec. 3.1) and the G-buffer model (Sec. 3.2). The baselines were not trained on LightProp, so the comparison is in-distribution for Ours and out-of-distribution for baselines. This asymmetry alone can explain part of the reported margin. Real-world evidence is exclusively qualitative (Sec. 4.2), and no metric verifies that the diffusion-generated multi-light images are physically consistent with the specified light directions. If the quantitative margin is a product of LightProp-specific statistics, the central contribution reduces to 'best on the authors' own benchmark,' not a generalizable multi-light prior. As a minor but concrete sign of unreported redundancy, the appendix's light placement (Eqs. 6-7) puts light sources at theta=0 and theta=2*pi, which are the same physical direction, so the claimed '9 distinct lighting conditions' are at most 8; this suggests the multi-light setup and its evaluation contain redundancy that is not discussed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Neural LightRig, a two-stage framework for jointly estimating surface normals and PBR materials (albedo, roughness, metallic) from a single object image. In stage I, a Stable Diffusion-based model is fine-tuned on a new synthetic dataset (LightProp, 80k Objaverse objects rendered with Blender Cycles) to generate nine images of the same object under different point-light directions, arranged in a 3x3 grid to promote consistency. In stage II, a U-Net regression model takes the input image, the generated multi-light images, and their associated light orientations to predict an 8-channel G-buffer. The authors report state-of-the-art performance on a held-out LightProp subset for normal, material, and relighting metrics, and provide qualitative demonstrations on in-the-wild images. Ablations show that adding multi-light images and the proposed augmentation strategies improve accuracy, supporting the core mechanism.","tokens_in":14910,"tokens_out":4101,"duration_ms":38542,"significance":"If the claims hold, the idea of using a multi-light diffusion model to reduce the inherent ambiguity of single-image intrinsic estimation is a valuable contribution. The paper is clearly written, the ablations (Table 4) convincingly show that multi-light conditioning is the key driver of improvement, and the release of code and dataset would benefit the community. However, the quantitative state-of-the-art claim rests on an evaluation that is in-distribution for the proposed method and out-of-distribution for the baselines, and the cross-domain generalization to real images is only shown qualitatively. These issues currently limit the strength of the central claim.","major_comments":[{"comment":"The quantitative evaluation is performed on a held-out subset of LightProp, which is also the training distribution for both the multi-light diffusion model and the G-buffer model. The baselines (DSINE, GeoWizard, Marigold, etc.) are not trained on LightProp, so the comparison is asymmetric: the proposed method is tested in-distribution while the baselines are tested out-of-distribution. This asymmetry alone could explain part of the reported margin, and it means the claim that Neural LightRig 'surpasses existing approaches' is only established on the authors' own benchmark, not on generally harder or different distributions. To support the central claim, the authors should evaluate on a cross-dataset benchmark (e.g., a different synthetic renderer or real captured objects with ground-truth normals/materials) or, if such data are unavailable, explicitly temper the SOTA claim to the LightProp distribution and provide more substantial qualitative evidence of generalization.","section":"Sec. 4.1, Tables 1 and 2"},{"comment":"The light placement described by Eqs. (6)-(7) yields theta_0 = 0 and theta_8 = 2*pi. If theta is the polar angle relative to the overhead direction as stated in Sec. 3.3, these two angles correspond to the same physical direction (straight down the +z axis). Thus the setup provides at most 8 distinct lighting directions, not the claimed 9. This redundancy should be corrected, for example by using theta_i = i*pi/4 for i = 0..7 and a ninth distinct direction, or by clarifying the parameterization so that theta = 0 and theta = 2*pi are not coincident. The current description also affects the interpretation of the ablation in Table 4, where the 9-light configuration contains a duplicate direction.","section":"Appendix A.2, Eqs. (6)-(7)"},{"comment":"The claim that the multi-light diffusion model retains enough of Stable Diffusion's internet-scale prior to generate physically consistent lighting for in-the-wild inputs is not quantitatively validated. All quantitative results are on LightProp synthetic renders, and the only real-world evidence is qualitative (Sec. 4.2). The authors should provide a quantitative test on real images with known ground truth (e.g., light-stage captures or a small real photometric-stereo dataset) to verify that the generated multi-light images are physically consistent and that the downstream G-buffer predictions are accurate in the wild. Without this, the generalizability of the approach and the practical significance of the multi-light prior remain unsubstantiated.","section":"Sec. 3.1 and Sec. 4.2"}],"minor_comments":[{"comment":"In the first paragraph of Sec. 3, 'the the under-constrained nature' contains a duplicated definite article.","section":"Sec. 3"},{"comment":"In the introduction, 'we proposed a data-driven approach' should be 'we propose' to match the present-tense style used elsewhere.","section":"Sec. 1"},{"comment":"The header 'A verage Time↓' contains an extra space and the capitalization is inconsistent with the rest of the table; it should read 'Average Time↓'.","section":"Table 2"},{"comment":"Reference [1] (Stable Diffusion v2.1) has an improperly formatted URL with a line break in the middle of the link; this should be corrected to a single clickable URL.","section":"References"},{"comment":"The statement 'the positions of light sources are fixed during the training of multi-light diffusion model g(·) and the inference of G-buffer prediction model f(·)' is repeated in a slightly different form in the appendix; the redundancy could be reduced by consolidating the description.","section":"Sec. 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a promising framework and the core ablations are convincing, but the evaluation asymmetry (in-distribution for the proposed method, out-of-distribution for baselines) is a serious concern that prevents acceptance in its current form. The light-placement redundancy in the appendix is a small but concrete technical error that should be fixed. If the authors can provide cross-dataset quantitative results or appropriately qualify their claims, the paper could become a solid contribution. The scope seems appropriate for a computer vision venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a real empirical contribution, not a mirage. Fine-tuning a diffusion model to output several consistent relit images of an object, then feeding those images plus their light directions into a regression U-Net, clearly helps normal and material estimation. The ablation with 0/3/6/9 light images is the heart of the paper and it is convincing: performance jumps sharply from 0 to 3 and keeps improving. That alone justifies publishing the method.\n\nWhat is new is the particular combination and the LightProp dataset. The building blocks (diffusion-based relighting, reference attention, U-Net regression) are not new, but the multi-light consistency formulation is clean, and they release code and data, which is good for the field.\n\nNow the soft spots. The quantitative SOTA claim is not as strong as the text implies. Tables 1 and 2 are all on a held-out subset of LightProp, the same renderer and object distribution the method was trained on. The baselines were not trained on LightProp, so part of the margin may simply be an in-distribution advantage. The paper needs at least one cross-dataset quantitative evaluation (a real captured set or a different synthetic renderer) before claiming to surpass existing approaches generally. Real-world results are qualitative only, and the main tables have no error bars or significance tests, which is common in this area but still limits the strength of the claim. The stress-test note about the appendix is correct: Eqs. 6-7 place one light at theta=0 and another at theta=2*pi, which are the same physical direction, so the \"9 distinct\" lighting conditions are at most 8. That is a minor misconfiguration, not a fatal flaw, but it should be fixed.\n\nThe central idea still holds up. The paper is honest about its limitations in the appendix (highlights, resolution, object-only scope). I would send this out for review; it deserves a serious referee, and with a modest amount of additional evaluation it could become a genuinely useful reference. I would bring it to the reading group if anyone here cares about inverse rendering; otherwise it is a solid maybe. My recommendation: accept the risk of a conditional review, and push the authors for cross-dataset numbers and a corrected lighting setup before publication.","headline":"A solid empirical pipeline for single-image inverse rendering that generates multi-light inputs with diffusion; the SOTA claim needs cross-distribution evidence, but the core idea is real.","tokens_in":15453,"tokens_out":2671,"would_cite":true,"duration_ms":24881,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural LightRig claims that synthesizing nine point-light views from one object photo and regressing normals and physically based material maps from them beats single-image baselines on its synthetic benchmark.","keywords":["single-image inverse rendering","surface normal estimation","PBR material estimation","multi-light diffusion","relighting","G-buffer prediction","photometric stereo disambiguation","synthetic training dataset"],"falsifier":"Run Neural LightRig on a calibrated photometric-stereo capture of real objects: give it one image under a known flash as input, then compare predicted normals and albedo, roughness, and metallic against ground truth from the multi-light rig. If the generated nine-light images are not physically consistent with the input, the regressor's reported advantage should shrink or disappear on these real captures, since all quantitative validation in the paper is on synthetic renders and real-world results are only qualitative.","tokens_in":14434,"feed_emoji":"💡","tokens_out":11255,"duration_ms":91537,"temperature":0.7,"pith_summary":"The paper is trying to establish that single-image inverse rendering can be made accurate by first synthesizing photometric-stereo-style observations. Its framework, Neural LightRig, fine-tunes a pretrained image diffusion model to turn one photo of an object into nine images lit from known directions; a second U-Net then reads these images, plus the input and the light angles, and predicts surface normals and PBR material maps. On a held-out synthetic benchmark the method reports the best numbers among the compared approaches, and the ablation shows the multi-light inputs are what cut normal error most sharply. The core claim, read sympathetically, is that generating consistent extra lighting beats capturing it, and that this largely dissolves the monocular ambiguity.","feed_headline":"Nine synthetic lights unlock sharper normals and materials","feed_subtitle":"Nine generated lights let a regressor beat single-image normals and materials.","key_machinery":"The load-bearing mechanism is a generate-then-regress loop with two trained components. First, a multi-light diffusion model produces nine lighting-consistent images of the object in a single 3x3 grid, conditioned on the input photo by channel concatenation plus reference attention so the generated images keep the input's color and texture. Second, a large regression U-Net consumes the input image, the nine generated images, and sinusoidal embeddings of each light's spherical angles, and outputs an 8-channel G-buffer (normal, albedo, roughness, metallic). The chain is trained and evaluated on the paper's own rendered dataset, LightProp, which pairs 80k synthetic objects with multi-light images, ground-truth G-buffers, and randomized lighting; a set of degradations, brightness perturbations, orientation noise, and mixing of generated images is applied to bridge the render-to-diffusion domain gap.","core_discovery":"The paper's central claim is that a single image can be effectively expanded into a calibrated multi-light capture by a diffusion model, and that this expansion resolves much of the ambiguity that plagues single-image normal and material estimation. Concretely, Neural LightRig fine-tunes a pretrained latent diffusion model on a newly rendered dataset of 80k synthetic objects, so that given one photo it emits nine images lit by point sources at known spherical angles. Those nine images, together with their lighting angles, condition a U-Net that predicts an 8-channel G-buffer: surface normal, albedo, roughness, and metallic. Quantitative results on 1,000 held-out synthetic objects show the method outperforming prior work, for example a normal mean angular error of 6.41 degrees versus 8.03 for the closest baseline, albedo PSNR of 26.62 dB versus 23.88 dB, and relighting PSNR of 30.12 dB versus 27.98 dB. The intended upshot is that monocular inverse rendering can be recast as a photometric-stereo problem whose extra observations are synthesized rather than captured.","pith_inferences":["If the mechanism is right, the same synthesize-extra-observations-then-regress recipe should transfer to other ill-posed inverse problems, such as depth from sparse views or SVBRDF from video, wherever a generative model can be made to emit consistent variations of the unknown.","The paper's data-mixing augmentation implies a self-improving loop: as the generator improves or is fine-tuned on real photometric-stereo captures, the downstream regressor can be retrained on the better synthetic images. The paper does not test this loop.","A direct testable prediction is that the nine generated lights will help most on specular and metallic objects and least on near-Lambertian ones, because multi-light cues mainly disambiguate reflectance and highlight structure; per-category breakdowns on LightProp would show this pattern.","The fixed nine-light layout means the method implicitly assumes one dominant point-light model per generated image; extending to continuous environment lighting would require the generator and regressor to be retrained over light-direction distributions rather than fixed angles."],"forward_implications":["On the reported benchmark, normal estimation improves from a mean angular error of 8.03 degrees for the closest baseline to 6.41 degrees, and accuracy at the fine 3-degree threshold rises from 25.2% to 38.7%.","Material and relighting metrics also move in the same direction: albedo PSNR reaches 26.62 dB versus 23.88 dB, roughness 23.44 dB versus 17.25 dB, metallic 26.23 dB versus 22.00 dB, and relighting PSNR 30.12 dB versus 27.98 dB.","The ablation attributes the gain to the multi-light conditioning itself: moving from 0 to 9 light images lowers normal MAE from 7.56 to 4.06 degrees and raises 5-degree accuracy from 45.8% to 74.7%.","The reported per-frame cost is about 5 seconds, compared with 2 minutes for the optimization-based baseline, so the feed-forward pipeline is substantially faster while remaining physically grounded.","Because the model outputs normals and materials jointly, the relighting it produces can be re-rendered under arbitrary environment maps rather than being a purely generative image-to-image effect."],"supporting_citations":[{"why":"Supplies the latent diffusion architecture whose internet-scale prior is fine-tuned into the multi-light generator.","marker":"[38]"},{"why":"The specific pretrained checkpoint used as the backbone of the multi-light diffusion model.","marker":"[1]"},{"why":"Provides the reference-attention conditioning that lets generated multi-light images keep the input's color and texture.","marker":"[43]"},{"why":"Supplies the 3D object corpus from which the paper curates its 80k-object LightProp training set.","marker":"[11]"},{"why":"Supplies the photometric-stereo principle that multiple point-light views disambiguate surface orientation and reflectance.","marker":"[51]"},{"why":"Earlier demonstration that diffusion models can relight arbitrary objects, motivating lighting-conditioned generation.","marker":"[23]"},{"why":"A diffusion-based single-image decomposition method used as a baseline for normals, materials, and relighting.","marker":"[57]"},{"why":"An optimization-based inverse-rendering baseline compared for PBR quality and per-frame latency.","marker":"[8]"}],"fun_headline_variants":["Single photo becomes nine-light studio for sharper normals and materials","Diffusion generates nine lights to refine normals and materials from one image","One photo in, nine synthetic lights out: sharper normals and PBR materials","From a single image, synthetic multi-lighting yields superior normals and materials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that a diffusion model fine-tuned only on synthetic renders will generate physically consistent multi-light images for real in-the-wild photos, because the regressor is trained on rendered ground truth and the real-image results are shown only qualitatively.","fun_headline_variants_meta":{"raw":{"variants":["Single photo becomes nine-light studio for sharper normals and materials","Diffusion generates nine lights to refine normals and materials from one image","One photo in, nine synthetic lights out: sharper normals and PBR materials","From a single image, synthetic multi-lighting yields superior normals and materials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001072,"raw_usage":{"total_tokens":4490,"prompt_tokens":943,"completion_tokens":3547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3467}},"tokens_in":559,"tokens_out":3547,"duration_ms":24714,"temperature":1.0,"reasoning_tokens":3467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:53:15.978668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Neural LightRig on a calibrated photometric-stereo capture of real objects: give it one image under a known flash as input, then compare predicted normals and albedo, roughness, and metallic against ground truth from the multi-light rig. If the generated nine-light images are not physically consistent with the input, the regressor's reported advantage should shrink or disappear on these real captures, since all quantitative validation in the paper is on synthetic renders and real-world results are only qualitative.","supporting_citations":[{"cited_title":"Stable diffusion v2.1","cited_arxiv_id":null,"evidence_quote":"The specific pretrained checkpoint used as the backbone of the multi-light diffusion model."},{"cited_title":"Zero123++: a single image to consistent multi-view dif- fusion base model, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the reference-attention conditioning that lets generated multi-light images keep the input's color and texture."},{"cited_title":"Objaverse: A universe of annotated 3d objects","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D object corpus from which the paper curates its 80k-object LightProp training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the photometric-stereo principle that multiple point-light views disambiguate surface orientation and reflectance."},{"cited_title":"Neural gaffer: Relighting any object via diffusion","cited_arxiv_id":null,"evidence_quote":"Earlier demonstration that diffusion models can relight arbitrary objects, motivating lighting-conditioned generation."},{"cited_title":"Rgb ↔x: Image decomposition and synthe- sis using material- and lighting-aware diffusion models","cited_arxiv_id":null,"evidence_quote":"A diffusion-based single-image decomposition method used as a baseline for normals, materials, and relighting."},{"cited_title":"Intrinsicanything: Learning diffusion priors for inverse rendering under unknown illumi- nation, 2024","cited_arxiv_id":null,"evidence_quote":"An optimization-based inverse-rendering baseline compared for PBR quality and per-frame latency."}],"review_version":1}