{"id":"f582d914-1bf1-4951-a751-0c19ef39fad1","arxiv_id":"2411.11475","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MVLight is a light-conditioned multi-view diffusion model whose RGB, albedo, and normal outputs are distilled into a NeRF to create relightable text-to-3D assets.","lead":"This paper introduces MVLight, a multi-view diffusion model that takes an HDR lighting environment as input and generates consistent views of a 3D object together with albedo and normal maps. The authors use these outputs to drive score distillation for text-to-3D generation, aiming to produce models that can be relit under new lighting conditions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-1 SDS in Section 3.2 renders x_l as albedo times per-light ambient color, so the rendered image cannot express the directional HDR shading that MVLight is conditioned on; the optimizer can only absorb this mismatch by baking lighting into albedo, undercutting the claimed decoupling.","rationale":"The paper's central claim is that integrating the same HDR map into both SDS and PBR yields better material decomposition and relighting. For that claim, the x_l term in Eq. 2 must be a render that actually uses the conditioned HDR. The text of Section 3.2 says stage-1 x_l is albedo times ambient light with uniform intensity, i.e., it ignores the directional structure of the HDR. Because MVLight's training images contain that structure, the SDS residual cannot vanish without the albedo or geometry absorbing shading. This is not merely a missing metric; it is an internal inconsistency in the optimization objective. The reader focused on the diffusion outputs being unaligned; I see a second, arguably more basic mismatch: the rendered input to SDS is not physically lit by the HDR that conditions the diffusion model. I would keep the conditional verdict but add this specific check. If the albedo-invariance test passes, the concern is resolved; if it fails, the central relighting claim is substantially weakened. No code or data is released, so this check requires either author-provided code or a careful reimplementation based on the paper's description.","tokens_in":11564,"tokens_out":10944,"duration_ms":118200,"concrete_test":"Run the full pipeline (or a faithful reimplementation) on a fixed set of prompts twice, once with HDR set A and once with HDR set B during both stage-1 SDS and PBR fine-tuning. After training, render each model's albedo MLP under a constant neutral light and compute the mean relative absolute difference between the two albedo maps. If the albedo maps differ beyond a small tolerance (e.g., >5% mean relative error), lighting is baked into albedo and the same-HDR mechanism has not achieved light-independent material estimation. As a complementary check, relight both final models under a shared unseen HDR and compare rendered images: if they differ mainly in albedo rather than lighting, decoupling failed. Albedo should be invariant to which HDR was used during generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2, the geometry/appearance stage renders the RGB term x_l of Eq. 2 as a product of the albedo MLP with per-light 'ambient' MLPs: 'the lighting intensity is assumed to be uniform across the object's surface for each lighting environment.' However, MVLight is trained on, and conditioned to reproduce, full HDR-lit images whose directional shading (shadows, highlights) depends on the environment map. The SDS target \\hat{x}_l in Eq. 2 therefore contains high-frequency lighting that the renderer x_l is structurally unable to represent. The only free parameters that can absorb the residual x_l - \\hat{x}_l are the albedo and density fields, so the optimizer has an incentive to encode the HDR's shading into the 'albedo' and geometry. This is exactly the lighting-baking failure the paper attributes to blind PBR methods, and it means the claimed alignment of 'the same HDR map for both diffusion and PBR' is not realized in the stage-1 renderer. The final PBR stage may partially correct this, but geometry is frozen and the albedo initialization is already contaminated; the central claim of improved material decomposition therefore rests on an unverified and physically inconsistent intermediate representation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MVLight, a multi-view diffusion model conditioned on HDR environment maps that jointly generates multi-view RGB, albedo, and normal maps. This model is then used for score distillation sampling (SDS) in a two-stage text-to-3D pipeline: stage 1 optimizes NeRF geometry and appearance with an albedo MLP and per-light ambient MLPs, and stage 2 fine-tunes PBR materials (albedo, roughness, metallic) with frozen geometry, using the same HDR map for both the diffusion prior and PBR rendering. The authors claim that this light-conditioned design yields better geometric fidelity, more accurate PBR material decomposition, and improved relighting compared with DreamFusion, Fantasia3D, MVDream, and RichDreamer, supported by qualitative results, a CLIP score table, an ablation, and a user study.","tokens_in":11837,"tokens_out":6862,"duration_ms":73205,"significance":"If the central claims hold, MVLight would be a useful step toward controllable relightable text-to-3D generation: a single multi-view diffusion model explicitly conditioned on lighting, with multi-modal outputs, is a natural extension of MVDream and addresses a real limitation of blind PBR estimation in prior work. The paper is strong on implementation detail, uses a large synthesized training corpus (about 8.6M images from 90k Objaverse objects), evaluates on HDR maps unseen during training, and compares against external baselines rather than only self-comparisons. Its main weakness is that the core claims about material decomposition and relighting accuracy are supported almost entirely by qualitative figures; the only quantitative metric is CLIP score, which does not measure relighting or material correctness.","major_comments":[{"comment":"The stage-1 renderer x_l is defined as albedo multiplied by per-light ambient MLPs whose lighting intensity is uniform across the object's surface. However, the SDS target \\hat{x}_l is a full HDR-shaded image with directional shadows, highlights, and spatially varying illumination. Because x_l structurally cannot represent this directional shading, the loss in Eq. (2) contains an irreducible residual that can only be absorbed by the albedo MLP and the density field. This is exactly the lighting-baking failure the paper attributes to blind PBR baselines, and it undercuts the claim that using the same HDR map for diffusion and PBR aligns lighting conditions during stage 1. The stage-2 PBR fine-tuning cannot repair geometry that is already frozen, and it starts from an albedo initialization that may already contain baked-in lighting. Please provide a diagnostic experiment that tests whether stage-1 albedo and geometry are invariant to the HDR map used during optimization, or modify the stage-1 renderer to support directional lighting.","section":"§3.2, Eq. (2)"},{"comment":"The only quantitative comparison is the CLIP score, which measures text-image alignment and not relighting quality or material decomposition. The reported margin over MVDream is 0.44 (31.21 vs. 30.77), yet no error bars, number of views, seeds, or statistical significance test are given, so this difference may be within noise. The central claims of 'more accurate PBR material estimation' and 'superior relighting performance' are therefore not quantitatively validated. I ask for metrics that directly evaluate relighting and material separation, such as PSNR/LPIPS on held-out HDR renders, or synthetic scenes with known albedo and lighting, along with variance estimates.","section":"§4.2, Table 1"},{"comment":"The three SDS supervision terms in Eq. (2) use normal, albedo, and RGB images generated independently by the diffusion model, and the Limitation section explicitly states that 'there is no guarantee of alignment between these modalities.' If the predicted normal, albedo, and shaded color conflict, the joint SDS gradient can pull the NeRF representation in inconsistent directions, which may degrade geometry or albedo despite the qualitative ablation in Fig. 5. Since the paper's improved geometry and material decomposition claims rest on this multi-modal SDS, the authors should either provide quantitative evidence that misalignment is benign (e.g., measuring cross-modality consistency on held-out data) or modify the training to enforce alignment without sacrificing output quality.","section":"§3.2 and Limitation"},{"comment":"The user study uses only 24 participants and reports a single preference percentage (63% for MVLight) without confidence intervals or statistical testing. It is not clear how many pairwise or multi-way comparisons each participant made, how the 40 results per method were selected, or whether participants evaluated relighting performance as opposed to overall visual appeal. This evidence is suggestive but too weak to support the strong claim of superiority over all baselines; please add significance testing or at least report per-question breakdowns and participant agreement.","section":"§4.2, Fig. 4 and user study"}],"minor_comments":[{"comment":"The caption contains the typo 'MVVLight'; please correct it to 'MVLight'.","section":"§4.1, Fig. 8 caption"},{"comment":"The text says 'light-awrae PBR fine-tuning'; this should be 'light-aware'.","section":"§4.1"},{"comment":"The phrase 'the denosing U-Net' should be 'the denoising U-Net'.","section":"§2.2"},{"comment":"The order of method names in the caption ('MVDream Ours Fantasia3D DreamFusion RichDreamer') does not clearly map to the pie-chart colors and percentages; please label the chart directly or reformat the caption.","section":"Fig. 4 caption"},{"comment":"Please specify how the unbounded HDR values are scaled or clamped before VAE encoding, and clarify whether the high-frequency and low-frequency components are encoded separately and then concatenated or combined in another way.","section":"§3.1"},{"comment":"In the loss, the definition of e covers albedo, normal, and light embeddings, but for LAION text-to-image samples where ζ and L are empty it is not clear which embedding (if any) is used; please state explicitly how these samples are fed into the model.","section":"§3.1, Eq. (1)"},{"comment":"The sentence about 'random sampling from 5 unseen HDR maps' should clarify whether the same sampled HDR map is used for the diffusion conditioning and the PBR render within each SDS iteration, and whether 'unseen' means these maps were not part of the 450-map training pool.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"I do not see a circularity problem: despite the author overlap with MVDream, the paper compares against MVDream as a baseline and the light-conditioning modifications are distinct. The main concern is that the stage-1 simplified renderer may invalidate the lighting-alignment claim, and the current experimental evidence is too qualitative to establish the material-decomposition and relighting benefits. I would ask for a diagnostic experiment isolating the stage-1 lighting mismatch and for quantitative relighting metrics before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MVLight is a genuine, if incremental, advance: it takes the known recipe of conditioning a diffusion model on HDR lighting (Neural Gaffer) and combines it with multi-view generation (MVDream) to drive SDS for relightable text-to-3D. The authors also build and train on a large multi-view, multi-light dataset (~8.6M images), which is valuable in itself. The qualitative results look competitive, and the ablation separating light-aware from blind PBR training is well conceived.\n\nThe soft spots are real, and one is more serious than the paper admits. In Stage 1, the volumetric renderer produces x_l as albedo times a per-light ambient term, meaning uniform lighting across the surface. But the diffusion model is conditioned on an HDR map and predicts images with directional shading, shadows, and highlights. The SDS loss in Eq. 2 compares a renderer that cannot express those effects against a target that contains them; the only free parameters available to close that gap are albedo and density. So the optimizer has a structural incentive to bake lighting into albedo, exactly the failure the paper attributes to blind PBR methods. The paper's stated limitation is about independent normal/albedo/RGB modalities; the renderer-conditioning mismatch is separate and potentially more damaging. Stage 2 PBR fine-tuning may rescue the albedo, but the paper doesn't show that it does: geometry is frozen, and the albedo initialization is already compromised.\n\nBeyond that, the evaluation is thin. There is no quantitative metric for relighting accuracy or material decomposition. The CLIP score has no error bars, uses self-selected prompts, and is a weak proxy for the paper's actual claims. The user study is 24 participants. No code or data is released, so nobody can reproduce the dataset or the pipeline.\n\nThat said, the paper is a serious engineering effort, the central idea is novel enough, and the dataset contribution is substantial. It deserves a proper referee, but the reviewers should push on the Stage 1 inconsistency, demand quantitative relighting evaluation, and ask whether Stage 2 can actually undo the baking.","headline":"MVLight is a plausible engineering advance with a valuable dataset, but a renderer-conditioning mismatch in Stage 1 undercuts the central relighting claim.","tokens_in":12412,"tokens_out":3602,"would_cite":true,"duration_ms":35493,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MVLight feeds HDR lighting into multi-view diffusion, enabling text-to-3D assets whose albedo and relighting separate cleanly.","keywords":["relightable text-to-3d","multi-view diffusion","score distillation sampling","HDR lighting conditioning","PBR material estimation","albedo-normal decomposition","NeRF optimization","text-to-3d generation"],"falsifier":"Generate a fixed object from a fixed text prompt under two different HDR maps with MVLight, render the resulting PBR asset under a third unseen HDR map, and compare the albedo maps: if the albedo changes with the relighting environment, or if shading baked into the first render persists under the new light, the claimed decoupling is false. A second check is to compare MVLight's predicted normal map with the normal implied by its predicted shaded RGB under the known light direction; a systematic mismatch would show the independent modality outputs break SDS.","tokens_in":11309,"feed_emoji":"💡","tokens_out":5074,"duration_ms":43559,"temperature":0.7,"pith_summary":"The paper argues that existing relightable text-to-3D models bake lighting into their materials because their diffusion priors never see the lighting environment. MVLight is a multi-view diffusion model that takes an HDR environment map as an explicit condition, alongside text and camera pose, and outputs view-consistent shaded color plus normal and albedo maps under that light. These outputs are distilled into 3D optimization with Score Distillation Sampling, and the same HDR map is reused when fine-tuning PBR materials. The authors claim this alignment of lighting between generation and material estimation produces cleaner albedo and better relighting under unseen environments than blind estimation. The central thing at stake is whether lighting can be used as a conditioning signal rather than guessed after the fact.","feed_headline":"Same HDR map for diffusion and PBR gives 3D assets true relighting","feed_subtitle":"MVLight feeds the same lighting environment into generation and material estimation, so shadows stay out of albedo.","key_machinery":"The central object is the light-conditioned multi-view diffusion model MVLight. It injects an HDR environment map through a light cross-attention module, with the map decomposed into high-frequency (log-mapped detail) and low-frequency (tone-mapped color) components embedded via separate MLPs; learnable normal and albedo embeddings let the same network output those modalities. This model is then used as the score prior in a two-stage SDS pipeline: stage one optimizes a hash-grid NeRF with separate MLPs for density, albedo, and per-light ambient color, and stage two freezes geometry and fine-tunes albedo, roughness, and metallic MLPs while the diffusion model and PBR renderer share one HDR map.","core_discovery":"MVLight's central claim is that explicitly conditioning a multi-view diffusion model on an HDR lighting environment, and then using the identical HDR map during PBR material optimization, breaks the blind-estimation bottleneck in relightable text-to-3D generation. The model extends a multi-view U-Net with a light cross-attention module fed by high-frequency and low-frequency decompositions of the HDR map, and it learns to generate shaded RGB, normal maps, and albedo maps for multiple views under the specified light. Distilling these outputs through SDS jointly supervises geometry, albedo, and appearance, while the aligned lighting in the PBR fine-tuning stage prevents shadows and highlights from being baked into albedo. The authors demonstrate the claim with comparisons against prior PBR-based generators and a user study, reporting higher CLIP alignment and cleaner material decomposition.","pith_inferences":["A natural extension the author does not pursue is using the light-conditioned model for inverse rendering: given a single image, search over HDR maps and poses to recover both lighting and albedo; the trained light cross-attention would provide a prior for that search.","If the modality-alignment limitation were fixed by cross-modal attention, the same HDR-conditioned prior could supervise material estimation directly without SDS, which would be a faster and more stable optimization.","The 450 HDR maps used for training and 5 unseen maps for evaluation suggest the method's generality to arbitrary lighting is bounded by the diversity and dynamic range of those maps; testing on extreme synthetic or captured HDR probes would clarify that bound.","The claim that alignment between diffusion and PBR lighting matters could be tested directly by ablating the HDR map sharing while holding the model fixed, which the paper only does indirectly through the blind-PBR comparison."],"forward_implications":["Text-to-3D generation can be made relightable without a second text-to-image model for PBR estimation; one light-conditioned multi-view model supplies geometry, albedo, and appearance.","Sharing the HDR map between SDS and PBR fine-tuning prevents baked-in lighting, so relighting with unseen HDR environments preserves albedo.","Jointly supervising normal, albedo, and RGB during SDS yields smoother normals and more distinct albedo than supervising color alone.","Because albedo, normal, and shaded color come from one network, the approach inherits the text-to-image base's semantic flexibility for arbitrary prompts."],"supporting_citations":[{"why":"Supplies the multi-view self-attention architecture, camera pose conditioning, and the SDS noise/reconstruction formulation MVLight extends.","marker":"Shi et al., 2024"},{"why":"Introduces Score Distillation Sampling, the optimizer MVLight uses to distill the diffusion prior into 3D.","marker":"Poole et al., 2023"},{"why":"Provides the base U-Net, VAE, and text-to-image prior, and is also the PBR estimator used in prior relightable methods.","marker":"Rombach et al., 2022"},{"why":"Supplies the high/low-frequency HDR decomposition and embedding strategy used to condition on lighting.","marker":"Jin et al., 2024"},{"why":"Represents the first PBR-plus-SDS pipeline and a comparison baseline for the blind-PBR approach MVLight contrasts with.","marker":"Chen et al., 2023"},{"why":"A multi-view normal-depth diffusion baseline and a blind-PBR comparison point for relighting evaluation.","marker":"Qiu et al., 2024"},{"why":"A multi-view normal-albedo diffusion baseline, and the reference for the modality-alignment tradeoff discussed in the limitation.","marker":"Liu et al., 2023"},{"why":"Source of the roughly 90,000 objects used to synthesize the multi-view, multi-light training set.","marker":"Deitke et al., 2023"},{"why":"Source of the 450 HDR maps used for training lighting and the 5 unseen maps used for SDS and evaluation.","marker":"Poly Haven, 2024"}],"fun_headline_variants":["MVLight: shared HDR for diffusion and PBR decouples albedo","Same HDR map for diffusion and material estimation yields clean albedo","Conditions multi-view diffusion on HDR to decouple albedo from lighting","Relightable 3D from text with light-conditioned multi-view diffusion","Light-conditioned multi-view diffusion for relightable 3D assets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the diffusion model's independently sampled normal, albedo, and RGB predictions are consistent enough that a single SDS gradient can push the 3D scene in one direction; the paper's own limitation section says there is no guarantee those modalities align because they are synthesized independently.","fun_headline_variants_meta":{"raw":{"variants":["MVLight: shared HDR for diffusion and PBR decouples albedo","Same HDR map for diffusion and material estimation yields clean albedo","Conditions multi-view diffusion on HDR to decouple albedo from lighting","Relightable 3D from text with light-conditioned multi-view diffusion","Light-conditioned multi-view diffusion for relightable 3D assets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000845,"raw_usage":{"total_tokens":3638,"prompt_tokens":863,"completion_tokens":2775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":2677}},"tokens_in":479,"tokens_out":2775,"duration_ms":52481,"temperature":1.0,"reasoning_tokens":2677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:28:15.487985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a fixed object from a fixed text prompt under two different HDR maps with MVLight, render the resulting PBR asset under a third unseen HDR map, and compare the albedo maps: if the albedo changes with the relighting environment, or if shading baked into the first render persists under the new light, the claimed decoupling is false. A second check is to compare MVLight's predicted normal map with the normal implied by its predicted shaded RGB under the known light direction; a systematic mismatch would show the independent modality outputs break SDS.","supporting_citations":[],"review_version":1}