{"id":"230570ba-1c13-48f5-b651-e6335561b3ab","arxiv_id":"2507.20512","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 3D Gaussian splatting framework decomposes outdoor photos into sun, sky, indirect lighting, and reflectance, enabling editable relighting and ray-traced shadows.","lead":"This paper presents a method to relight outdoor photographs by separating each scene into sun, sky, indirect light, and surface color using 3D Gaussian splatting. It lets users edit lighting and shadows in real time from ordinary photo collections, which matters for virtual reality, cinematic effects, and driving simulation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sun-visibility masks derived from HSV brightness thresholds cannot separate albedo from illumination, so the claimed physical sun/sky/indirect decomposition is underdetermined and lacks direct quantitative validation.","rationale":"The reader's weakest assumption correctly identifies the heuristic sun and sky masks as the load-bearing support for the decomposition. My stress-test sharpens the concern: the brightness threshold in Sec. 8.2 does not just risk threshold misclassification, it is structurally unable to separate albedo from illumination, and this error propagates through Eqs. 3, 4, and 9. The decomposition is also underdetermined without independent component supervision: all training losses in Eqs. 9--11 are against the final image I, so the sun/sky/indirect split is identifiable only through the masks. No experiment in Sec. 5 measures decomposition accuracy directly; Table 1 is NVS only, and relighting/shadow results are qualitative. These issues do not invalidate the NVS contribution, which is quantitatively competitive, but they do mean the central physical-interpretability claim is conditional on additional validation. The reader's CONDITIONAL verdict is therefore appropriate, and my read does not change it.","tokens_in":12988,"tokens_out":6594,"duration_ms":83821,"concrete_test":"Render a synthetic outdoor scene with Mitsuba or Blender using known albedo R, known sun direction and binary sun visibility V, and known sky/indirect illumination, then train GaRe on views rendered under several sun positions. Compare recovered R, S_sun, S_sky, S_ind, and V against ground truth using scale-invariant MSE and mask IoU. If component errors are large or V IoU drops in scenes with strong albedo variation, the physical-interpretability claim fails. Additionally, sweep tau_V over {0.1, 0.2, 0.3, 0.4, 0.5} on a real dataset and report NVS and relighting error; if results are insensitive to the threshold, the mask concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the sun/sky/indirect decomposition of Eq. 1 is physically interpretable. This claim is underdetermined by the supervision used in Sec. 3.3. The coarse sun visibility V_c used in Eq. 3 is computed in Sec. 8.2 by thresholding brightness: after the gamma transform of Eq. 17, pixels with I'_V > 0.3 are labeled sun-visible. Brightness alone cannot separate illumination from albedo: a low-albedo sunlit surface can easily be darker than a high-albedo shadowed surface. Thus V_c can systematically mislabel sunlit and shadowed pixels, contaminating the ambient-only model trained with Eq. 3, and the residual clustering of Eq. 4 propagates that contamination into the refined visibility V used in Eq. 9. The sky mask M_sky is similarly set by a fixed disparity threshold tau_D = 0.1 on relative depth predictions, which is not scale-calibrated across images. The decomposition losses of Eqs. 9--11 compare sums of components against the same input image; no loss ever observes a single component against an independent target, so the factorization I = (V*S_sun + S_sky + S_ind)*R is identifiable only through these heuristic masks. The quantitative evaluation in Table 1 reports only novel-view synthesis; relighting and decomposition quality are shown only qualitatively. Therefore the headline physical-interpretability claim is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents GaRe, a 3D Gaussian splatting framework for relighting outdoor scenes captured from unconstrained photo collections. The method decomposes each image into reflectance and three shading components—sun, sky, and indirect—following the Lambertian model I = (V·S_sun + S_sky + S_ind)·R, where V is a sun visibility map. The framework has three main contributions: (1) residual-based sun visibility extraction via binary clustering on the difference between a globally illuminated image and an ambient-only rendered counterpart; (2) a region-based supervision framework with structural consistency loss for illumination decomposition; and (3) ray-tracing-based shadow modeling with visibility baking for real-time rendering. The authors evaluate novel-view synthesis on four outdoor datasets (Staats Theater, Brandenburg Gate, Trevi Fountain, Sacre Coeur) and provide qualitative results for decomposition, relighting, and shadow effects. The paper also includes ablations for the visibility extraction, structural consistency loss, and two-step filtering strategy.","tokens_in":13333,"tokens_out":3987,"duration_ms":31586,"significance":"If the central claim holds, GaRe is a meaningful advance over methods that encode outdoor illumination as a single latent vector: it enables per-component manipulation of sun, sky, and indirect light, and supports dynamic shadow synthesis via baked ray-traced visibility. The quantitative NVS results are competitive with state-of-the-art methods on four standard datasets, and the ablations support the individual contributions of residual-based visibility, SCL, and the filtering strategy. The paper is clearly written and the method is reproducible in structure. However, the physical interpretability claim rests on a decomposition that is supervised only through heuristic masks and never directly validated quantitatively; this is the core weakness that needs to be addressed before the central claim can be accepted.","major_comments":[{"comment":"The decomposition is fundamentally supervised by the coarse sun visibility mask V_c and the sky mask M_sky, both derived from fixed thresholds (brightness threshold tau_V=0.3 after the gamma transform in Eq. 17, and disparity threshold tau_D=0.1 in Sec. 8.1). Brightness alone cannot separate illumination from albedo: a low-albedo sunlit surface can be darker than a high-albedo shadowed surface. Similarly, relative disparity predictions from Depth Anything V2 are not scale-calibrated across images, so a fixed threshold tau_D=0.1 may misclassify sky pixels for some scenes. Since the ambient-only model is trained with Eq. 3 using V_c, the residual map in Eq. 4 inherits these errors, and the refined visibility V propagates them into Eq. 9 and the region-based losses. The manuscript provides no quantitative evidence that the masks are accurate, and the paper's own ablation in Fig. 7 shows only one example. Please provide a quantitative evaluation of the visibility and sky masks (e.g., against hand-annotated labels on a subset of images), or show that the final decomposition is insensitive to the thresholds.","section":"Sec. 8.1, Sec. 8.2, Eq. (3), Eq. (4)"},{"comment":"The transmittance recurrence in Eq. (14) is not derived and appears dimensionally suspect: the term 'n_{j-1} d_i^T' is introduced without defining the product or explaining how a normal vector and a light-direction vector enter the transmittance of a ray in a point-based traversal. If n and d are unit vectors, their dot product is a scalar between -1 and 1, which would multiply the scalar transmittance term (1-alpha_{j-1}) T_{j-1} without any physical basis in the standard 3DGS compositing equations. If n d^T is intended as an outer product, the equation would be a matrix, which contradicts the scalar nature of transmittance. Please provide the derivation of Eq. (14) and clarify the dimensions of each term, or replace it with the standard transmittance update T_j = (1 - alpha_{j-1}) T_{j-1}.","section":"Sec. 3.4, Eq. (14)"},{"comment":"The physical interpretability of the decomposition is validated only through qualitative figures; Table 1 reports only novel-view synthesis metrics. Equations (9)–(11) compare sums of components against the same input image, so no loss ever observes a single component against an independent target. To support the claim that the components are physically interpretable, please add quantitative evaluation of the decomposition itself: e.g., relighting agreement with held-out images, decomposition consistency across views in the same illumination condition, or a comparison against a baseline that removes the decomposition (e.g., using a single shading component) on the relighting tasks. At minimum, include a quantitative study of how the components change under interpolation in Fig. 5, and report standard intrinsic-image metrics if any labels are available.","section":"Sec. 3.3, Eqs. (9)–(11), Sec. 5.2, Table 1"},{"comment":"The coarse visibility mask V_c is derived from the same images used to train the ambient-only model, and the residual clustering of Eq. 4 then uses the residuals of that model to produce the refined visibility V. This is a self-referential pipeline: if V_c is incorrect, the ambient-only model learns to fit the wrong pixels, and the residuals will reflect the model's error rather than the true illumination. Please provide a quantitative analysis of the sensitivity of the final visibility to the choice of taus and V_c, including the effect on relighting quality and decomposition consistency.","section":"Sec. 3.2, Eq. (3), Sec. 8.2"}],"minor_comments":[{"comment":"The description of the sky mask threshold 'tau_D = 0.1' is unclear: it is not stated whether the condition is disparity < tau_D or disparity > tau_D, and the text says 'near-zero disparity values' but the threshold is 0.1; please clarify the assignment rule and how the threshold was chosen.","section":"Sec. 8.1"},{"comment":"The notation in Eq. (3) uses a comma inside the norm: '||(I_amb - I) · (1 - V_c)||_1' — the comma appears to be a typo and should be removed.","section":"Sec. 3.2, Eq. (3)"},{"comment":"The index convention in Eq. (14) is inconsistent with the surrounding text: T_j is defined in terms of T_{j-1}, but the text says 'After traversing, the remanent transmittance T_j corresponds to the ray-traced visibility of Gaussian G_k.' It should be clarified which index corresponds to which Gaussian along the ray.","section":"Sec. 3.4, Eq. (14)"},{"comment":"The per-Gaussian visibility prediction v_k in Eq. (15) outputs values in [-1, 1], but the ray-traced visibility v_rt is stated to be in R_+ (positive). The loss L_vis in Eq. (16) compares them directly; please clarify how the signed output is mapped to a visibility in [0,1], or why the range mismatch is acceptable.","section":"Sec. 3.4, Eq. (15)"},{"comment":"The qualitative figures would be more convincing if they included a comparison to the state-of-the-art relighting baselines (e.g., NeRF-OSR, GS-W) for the decomposition and relighting results, not just NVS.","section":"Sec. 5.2, Fig. 4–6"},{"comment":"The related-work section is thorough, but the description of 'unstructured photo collections' would benefit from a clear distinction between the fixed-illumination multi-view datasets and the unconstrained photo collections used here, especially in Table 1 where NeRF-W and 3DGS baselines are evaluated on the same data.","section":"Sec. 2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core contribution is the decomposition and its physical interpretability, but the manuscript does not currently provide quantitative evidence for this claim, and the supervision relies on heuristic thresholds that are not evaluated. The NVS results themselves are solid, and the method is well-motivated. I would consider acceptance after the authors add quantitative validation of the visibility/sky masks and the decomposition, and fix the transmittance equation issue. This is a major revision, not a rejection: the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GaRe is a genuinely useful step for outdoor relighting with 3DGS: it gets sun, sky, and indirect shading as separate, editable components and adds dynamic shadows at real-time rates, which no prior 3DGS wild-scene model does. The NVS numbers are competitive, and the ablations for residual visibility, the structural consistency loss, and the two-step filter each show a plausible gain. The three-stage training is sensible, and baking ray-traced visibility into a feature MLP is a nice practical trick.\n\nThe soft spot is the decomposition's claimed physical interpretability. The supervision (Eqs. 9-11) never sees a component against an independent target; it only fits sums and relies on masks derived from brightness and disparity thresholds. Brightness alone cannot separate albedo from illumination, so the coarse visibility mask can mislabel sunlit low-albedo and shadowed high-albedo pixels, and the residual clustering that refines it inherits that bias. Same for the fixed disparity threshold for sky. This is a real circularity, though I don't think it invalidates the NVS results -- it limits how strongly you can claim accurate physical decomposition. The paper would benefit from a sensitivity analysis of tau_V, tau_D, and the gamma parameters, and ideally a quantitative relighting evaluation (e.g., on synthetic scenes with known ground-truth shadows). As it stands, the comparison is qualitative.\n\nOne more technical concern: Eq. 14's transmittance recurrence looks dimensionally off (a scalar T times a normal vector times a direction vector?). If it's a typo, fine; otherwise the visibility query needs a derivation. It's not the paper's main load, but a referee should ask.\n\nOn balance: the engineering is credible, the novelty relative to GS-W/WildGaussians is real, and the paper is worth a serious referee. My recommendation: accept for major revision, with the decomposition claims softened and a sensitivity/quantitative relighting experiment added. I'd also push for code release; the comparisons depend on many implicit choices.","headline":"A credible engineering advance for outdoor 3DGS relighting, but the physical-decomposition claim outruns the evidence; worth revision, not rejection.","tokens_in":13858,"tokens_out":2124,"would_cite":true,"duration_ms":22645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GaRe decomposes unconstrained outdoor photo collections into reflectance plus sun, sky, and indirect shading, and recombines them with ray-traced visibility to relight scenes in real time.","keywords":["relighting","3D Gaussian splatting","intrinsic image decomposition","outdoor scenes","sun visibility","ray-traced shadows","unconstrained photo collections","real-time rendering"],"falsifier":"Render a synthetic outdoor scene with known sun position, known sky radiance, and constant albedo; run GaRe on the renderings and check whether the recovered sun visibility matches the true shadow mask and whether recovered reflectance stays constant when the sun direction is changed. Any systematic deviation between the recovered visibility and the rendered shadow mask, or reflectance drift with sun angle, would falsify the claimed decomposition.","tokens_in":12768,"feed_emoji":"☀️","tokens_out":8139,"duration_ms":79417,"temperature":0.7,"pith_summary":"This paper aims to establish that the global illumination in unconstrained outdoor photo collections—casual, pose-calibrated images taken under varying weather and times of day—can be decomposed into three physically meaningful shading layers (direct sun, sky, and indirect bounce light) and a single reflectance layer, and that these layers can be recombined to relight the scene. The reward would be practical: instead of compressing each image's lighting into one opaque appearance vector, an artist could edit sun position, sky tone, or bounce intensity separately, including casting dynamic shadows, while rendering in real time. The paper argues this is achievable by supervising the decomposition with masks derived from image residuals and ray-traced visibility, and demonstrates relit novel views and shadow changes on four landmark photo collections.","feed_headline":"Sun, sky, and indirect light separated for 3D relighting","feed_subtitle":"GaRe recomposes the three shading layers with ray-traced shadows, so outdoor photo collections become relightable in real time.","key_machinery":"The load-bearing object is the multiplicative decomposition identity I = (V·S_sun + S_sky + S_ind)·R together with the sun visibility map V. Around it, three mechanisms carry the argument: (1) residual-based visibility extraction—an ambient-only Gaussian model is trained on masked pixels, and the residual between full and ambient renderings is binary-clustered into sunlit and shadowed regions; (2) region-based supervision with a structural consistency loss—Eqs. (9)–(11) constrain each rendered shading component against ground truth in its own mask region, and the SCL term (12) propagates structure from the well-constrained reflectance to all shading components; (3) ray-traced visibility with sky and front filters, followed by baking into per-Gaussian visibility features so shadows for arbitrary light directions render in real time.","core_discovery":"GaRe models every observed image as I = (V·S_sun + S_sky + S_ind)·R, where R is Lambertian surface reflectance, S_sun, S_sky, and S_ind are shading from direct sunlight, sky radiance, and indirect light, and V is a sun visibility mask that is 1 in sunlit pixels and 0 elsewhere. The paper's central discovery is that this decomposition can be learned from unconstrained photo collections without explicit 3D modeling of shadows. A residual-based procedure first trains an ambient-only Gaussian model, computes the residual between the fully lit image and the ambient render, and binary-clusters that residual to obtain a refined sun visibility map. Then per-Gaussian features and per-image embeddings are decoded into the shading components, supervised region-wise by the visibility and sky masks and globally by a structural consistency loss. Finally, ray tracing through the Gaussians produces visibility from arbitrary sun directions, and this visibility is baked into per-Gaussian features so all shading components and shadows are computed in one rasterization pass.","pith_inferences":["If the decomposition generalizes, the per-Gaussian sun/sky/indirect features and visibility map could serve as pseudo ground truth for training single-image outdoor relighting networks, a step the paper leaves to future work.","Replacing the fixed preprocessing thresholds for sky and sunlit pixels with learned or adaptive masks could extend the method to scenes with unusual albedo or partial overcast; the paper tests only fixed thresholds.","The Lambertian assumption means specular surfaces such as water, glass, or wet roads are folded into reflectance or shading artifacts; adding a specular term to Eq. (1) is a natural testable extension.","Because visibility is baked from sampled sun directions, extremely low sun angles or a sun that moves during a sequence may require re-baking; rendering a continuous sun trajectory is an inference-time extension not demonstrated in the paper."],"forward_implications":["Users can relight an outdoor scene by editing or interpolating the sun, sky, and indirect shading embeddings independently, rather than changing a single global appearance code.","Dynamic shadows follow from choosing a new sun direction at inference time, because ray-traced visibility is baked into Gaussian features and rendered in the same pass as the shading.","Novel-view synthesis on unconstrained landmark datasets matches or exceeds existing baselines while keeping sharp light-shadow boundaries, as shown in the paper's quantitative and qualitative evaluations.","The method extends naturally to cloudy images by setting sun visibility to zero, reducing the global model to sky plus indirect shading plus reflectance.","Because all shading and shadow components are computed in a single rasterization pass, relighting runs in real time rather than requiring the expensive per-image optimization of radiance-field relighting."],"supporting_citations":[{"why":"Provides the explicit 3D Gaussian splatting representation and real-time rasterizer that all shading components are attached to.","marker":"[17]"},{"why":"Supplies the multi-view intrinsic image decomposition principle behind Eq. (1), separating reflectance from illumination.","marker":"[7]"},{"why":"Establishes rich intrinsic image decomposition of outdoor scenes that motivates splitting illumination into multiple shading components.","marker":"[22]"},{"why":"Contributes the Staats Theater and Trevi Fountain datasets and the NeRF-OSR baseline the method is compared against.","marker":"[38]"},{"why":"Contributes the Brandenburg Gate and Sacre Coeur unconstrained photo collections and the NeRF-W appearance-embedding baseline.","marker":"[30]"},{"why":"Supplies the pre-trained depth estimator whose disparity output the sky mask threshold is applied to.","marker":"[49]"},{"why":"Provides the sunny/cloudy image classification used to decide when sun visibility is estimated versus set to zero.","marker":"[51]"},{"why":"Represents the per-image latent-vector appearance model that the paper's physically interpretable decomposition is designed to surpass.","marker":"[20]"},{"why":"Prior visibility extraction by ray-surface intersection that the residual-based approach is designed to improve upon.","marker":"[23]"}],"fun_headline_variants":["GaRe untangles sun, sky, and indirect light for relighting","Ray-traced shadows in relightable 3D Gaussian splatting","One-pass decomposition of sun, sky, and indirect light","Residual visibility powers dynamic outdoor relighting","GaRe: three shading layers, one rasterization pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decomposition stands or falls on hand-set preprocessing thresholds that decide which pixels count as sky and which count as sunlit: if those thresholds mislabel shadows or sky, every shading component derived from them is wrong.","fun_headline_variants_meta":{"raw":{"variants":["GaRe untangles sun, sky, and indirect light for relighting","Ray-traced shadows in relightable 3D Gaussian splatting","One-pass decomposition of sun, sky, and indirect light","Residual visibility powers dynamic outdoor relighting","GaRe: three shading layers, one rasterization pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1279,"prompt_tokens":905,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":289}},"tokens_in":521,"tokens_out":374,"duration_ms":3539,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:42:00.537649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a synthetic outdoor scene with known sun position, known sky radiance, and constant albedo; run GaRe on the renderings and check whether the recovered sun visibility matches the true shadow mask and whether recovered reflectance stays constant when the sun direction is changed. Any systematic deviation between the recovered visibility and the rendered shadow mask, or reflectance drift with sun angle, would falsify the claimed decomposition.","supporting_citations":[{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Provides the explicit 3D Gaussian splatting representation and real-time rasterizer that all shading components are attached to."},{"cited_title":"Multi-view intrinsic images of outdoors scenes with an application to relighting","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-view intrinsic image decomposition principle behind Eq. (1), separating reflectance from illumination."},{"cited_title":"Rich intrinsic image decomposition of outdoor scenes from multiple views","cited_arxiv_id":null,"evidence_quote":"Establishes rich intrinsic image decomposition of outdoor scenes that motivates splitting illumination into multiple shading components."},{"cited_title":"Nerf for outdoor scene relighting","cited_arxiv_id":null,"evidence_quote":"Contributes the Staats Theater and Trevi Fountain datasets and the NeRF-OSR baseline the method is compared against."},{"cited_title":"Nerf in the wild: Neural radiance fields for uncon- strained photo collections","cited_arxiv_id":null,"evidence_quote":"Contributes the Brandenburg Gate and Sacre Coeur unconstrained photo collections and the NeRF-W appearance-embedding baseline."},{"cited_title":"Hierarchical disentan- gled representation learning for outdoor illumination estima- tion and editing","cited_arxiv_id":null,"evidence_quote":"Provides the sunny/cloudy image classification used to decide when sun visibility is estimated versus set to zero."},{"cited_title":"Neulighting: Neural lighting for free viewpoint outdoor scene relighting with unconstrained photo collections","cited_arxiv_id":null,"evidence_quote":"Prior visibility extraction by ray-surface intersection that the residual-based approach is designed to improve upon."}],"review_version":1}