REVIEW 5 major objections 5 minor 4 references
PractiLight: Practical Light Control Using Foundational Diffusion Models
T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Training a tiny LoRA regressor on the self-attention layers of Stable Diffusion 1.5 to predict direct-irradiance maps lets a diffusion model relight arbitrary generated images with roughly 1,000x less data than leading methods.
desk verdict Good idea and a genuinely interesting analysis of where light lives in SD, but the control-adherence claim is not supported by their own numbers—the control metric is flawed and a naive blend beats the full method on it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The direct-irradiance map is the control signal: an image of the first-bounce light arriving at each pixel, combining direct diffuse and direct specular light. The LoRA regressor is a low-rank adapter placed on the self-attention layers of Stable Diffusion 1.5 that predicts such a map from an image, and it is the mechanism that unlocks the foundation model's light knowledge. Classifier guidance uses the regressor's prediction as an energy term during denoising, steering the latent toward the user's desired direct-irradiance. Query injection from the source image's self-attention and an edge-conditioned ControlNet are supporting mechanisms that preserve identity and structure.
What would settle it
Render a diverse set of test images with known light positions, run PractiLight's regressor on them, and compare its predicted direct-irradiance map to the ground-truth render; if the prediction's light direction or shadow placement is consistently wrong outside the primitive domain, the generalization claim fails.
Extended reading notes
Core claim
The central discovery is that self-attention activations in a latent diffusion model are the right place to read and write lighting. By measuring how injecting target-light activations into a source image's denoising affects semantic similarity, the authors find self-attention layers, especially decoder ones, carry light transport better than other layers. They also find light-related attention acts early in denoising, then fades around the midpoint. Building on that, they convert the U-Net backbone into an image-to-image regressor via rank-8 LoRA on self-attention q/k/v/output projections, trained to predict clean direct-irradiance maps. Used as classifier guidance with a scheduled scale, t
Load-bearing premise
A regressor trained on only 4,000 synthetic images of a few geometric primitives lit by a single point light can estimate direct-irradiance maps accurately enough for arbitrary generated images—portraits, anime, sketches, paintings—without breaking identity.
Editorial extensions
If this is right
- Relighting can be performed with three orders of magnitude fewer parameters and data, making it feasible on a single GPU and accessible to non-specialists.
- Because the backbone is not fully fine-tuned, the model retains its generalization across image domains rather than collapsing to the training domain.
- Lighting control should be applied during early-to-mid diffusion timesteps, giving a practical scheduling rule for future light-editing methods.
- Self-attention layers, not cross-attention, are the correct interface for lighting, pointing other editing tasks toward attention-based adapters.
- Identity-preserving relighting needs both structural control (ControlNet) and query injection; each alone is insufficient.
Reading between the lines
- The per-head analysis hints at even finer control: if only some self-attention heads encode light, a per-head LoRA or direct head manipulation could relight with a fraction of the current parameters.
- The same attention-based regressor scheme should transfer to transformer-based diffusion models such as Flux, since the paper shows attention propagation in those models follows light bounces; a direct test would be training the same LoRA on a DiT backbone.
- Adding colored light, inter-reflections, or multi-bounce rendering to the synthetic training set could remove two stated limitations (no colored light, no global illumination) without changing the method's structure.
- The regressor could be reused outside relighting, for example to edit shadows or highlights locally or to serve as a lighting prior for 3D-aware generation, since it outputs a physically meaningful irradiance estimate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PractiLight, a relighting method that trains a lightweight LoRA regressor on the self-attention layers of Stable Diffusion 1.5 to predict direct-irradiance maps from images. The regressor is trained on only 4,000 synthetic images of simple geometric primitives, and is then used as a classifier-guidance energy term during sampling, together with edge-ControlNet and self-attention query injection, to relight generated images according to a user-provided lighting condition. The authors claim state-of-the-art quality and control with roughly 100x fewer parameters and 1000x less data than leading methods such as IC-Light and RGB↔x. The evidence includes quantitative metrics, user studies, ablations, and qualitative results across portraits, anime, sketches, paintings, and other domains.
Significance. If the central claims hold, PractiLight would be a practically valuable contribution: it is parameter- and data-efficient, taps into a frozen foundation model rather than fine-tuning it, and the analysis of self-attention layers as carriers of light-transport information is a useful insight for the community. The paper also ships a relatively thorough ablation study, including dataset-size effects and design choices. However, the current evaluation does not substantiate the headline claim of state-of-the-art control: the control metrics appear to measure similarity to a shading field rather than to a relit image, and a trivial blending baseline outperforms the full method on those metrics. The user study does not test whether the target lighting direction or shadow placement was achieved. These issues are load-bearing and require substantive revision.
major comments (5)
- [§5.4, Tables 1 and 2] The 'Control' metrics compare the relit image to I_c, the direct-irradiance map rendered on a depth-proxy geometry. I_c is not a picture of a relit scene; it is a shading field without albedo, texture, or full light transport. A correct relit image should not be close to I_c in pixel or deep-feature space. This is not a minor issue: Table 2 shows that the 'Blending' baseline (50/50 blend of I_s and I_c) beats the full method on all four control metrics (L2 0.057 vs 0.130; LPIPS 0.61 vs 0.74; CLIP 0.47 vs 0.56; DINO 0.59 vs 0.71) while being visually unacceptable. Moreover, in Table 1 the full method is worse than IC-Light on every control column (e.g., L2 0.13 vs 0.10; LPIPS 0.74 vs 0.63). The claim that PractiLight is 'on par' in control adherence is therefore not supported by the published numbers. Please redesign the control evaluation, for example using rendered ground-truth relit sc
- [§5.5, Table 3] The subjective evaluation asks only two questions: which method better preserves identity/style, and which produces an overall better relighting result. It never asks whether the target lighting direction, shadow placement, highlight shape, or light intensity was achieved. Since the paper's headline claim is 'quality and control,' the control half of that claim is not validated by the user study. Please add direct control questions (e.g., 'which result better matches the lighting direction shown in the condition?') or a separate perceptual control study.
- [§5.4 and Table 1] No error bars, confidence intervals, or statistical significance tests are reported. The HPSv2 differences between Ours (0.2493), IC-Light (0.2462), and RGB↔x (0.2334) may be within noise, especially with a self-constructed dataset of only 180 images. The user-study percentages in Table 3 also lack confidence intervals. Please report per-prompt variances, repeated-seed runs, or bootstrap intervals, and state whether the observed differences are significant.
- [§5.2 and §5.4] The evaluation dataset is self-constructed, and the control signals are generated using the paper's own pipeline (§4.2). Hyperparameters for all methods, including PractiLight, appear to be tuned on the same evaluation dataset: the text says CFG values and diffusion timesteps were chosen 'between dataset-provided values and manual tuning (the best of the two options)' and that hyperparameters were 'tuned and fixed ... to the best of our abilities.' This creates a risk of overfitting to the test set. Please use a separate validation split for hyperparameter selection and report results on a held-out test set, or at least report sensitivity to the chosen hyperparameters.
- [§4.1 and §5] The regressor is trained only on 4,000 synthetic images of spheres, cubes, and cuboids with a single point light source, yet it is used to guide relighting of portraits, anime, sketches, paintings, and complex scenes. The generalization of the regressor across this domain gap is a load-bearing assumption, but it is validated only qualitatively. Please provide quantitative evidence, such as regressor prediction error on held-out synthetic domains, or a correlation between regressor accuracy and relighting success on the evaluation dataset. Without this, the method's broad-domain applicability remains plausible but not established.
minor comments (5)
- [§3.1, Fig. 3] The figure uses timesteps t=900 and t=100, while later text defines t as a normalized timestep with t=0 as pure noise and t=1 as the final image. Please make the notation consistent or clarify the mapping.
- [§5.4] The text says PractiLight is 'on par in terms of control adherence,' but Table 1 numerically shows it is worse than IC-Light on all four control metrics. Please rephrase to match the reported numbers.
- [Supplemental A.6] For DiLightNet comparisons, the supplement states that 10 different random light sources were generated per prompt and 'the best looking result from each' was selected, due to a high failure rate. This selection procedure should be disclosed in the main text, as it can bias the comparison in favor of DiLightNet; conversely, it makes the comparison less standardized.
- [Throughout] There are several typos and inconsistencies: 'vasts' should be 'vast'; 'generelizability' should be 'generalizability'; 'comaprisons' should be 'comparisons'; 'DilightNet' is inconsistently capitalized; 'morphing' is misspelled in the negative prompt as 'morhping'. Please proofread.
- [§6, Limitations] The Limitations section notes that colored light is not supported, global illumination / inter-reflections are not handled, and high-frequency control is moderate. These are significant scope restrictions and should be reflected in the abstract and conclusion, where the claims are currently stated in very general terms.
Circularity Check
No significant circularity: the regressor is trained on external rendered irradiance ground truth and used as a guidance energy, not as a fitted prediction of the final relit image.
full rationale
PractiLight's derivation chain is self-contained rather than circular. The LoRA regressor is supervised on synthetic Blender-rendered direct-irradiance maps (Sec. 4.1), and at inference it is used only to define the classifier-guidance energy L = ||I_g_hat - I_c||^2 (Sec. 4.3). The final relit image I_r is produced by diffusion sampling with this energy, query injection, and ControlNet; it is not obtained by copying the regressor output or by minimizing the training loss on the test image. The guidance target I_c is a user-controlled rendered irradiance map, not the regressor's fitted training label. The control-adherence metrics in Sec. 5.4 do compare the result to I_c, which is the same kind of signal the method is guided toward and which is rendered through the same depth-estimation pipeline; this is a legitimate evaluation-validity concern (indeed, the blending baseline scores better on those metrics while being visually poor), but it is not a circular derivation step because no equation in the paper reduces the relighting prediction to the fitted regressor or to the evaluation target by construction. The only self-citation, Erel et al. 2025, appears in the supplementary DiT discussion as preliminary evidence for a possible extension and is not load-bearing for the central SD1.5-based claims. Overall, the core contribution—training a small LoRA regressor on synthetic irradiance data and using it for diffusion guidance—is an empirical pipeline with independent ground-truth supervision, not a tautology.
Assumptions & free parameters
free parameters (6)
- Guidance scale gamma_t =
2.2 for normalized t in [0.05, 0.5], 0 otherwise
- ControlNet guidance scale =
0.6
- Classifier-free guidance (CFG) scale =
7.5 (or dataset-provided values)
- LoRA rank =
8
- Query injection timesteps =
[0.0, 0.7]
- ControlNet timesteps =
[0.0, 0.6]
assumptions (4)
- domain assumption Self-attention layers in pre-trained diffusion models encode light transport more strongly than other layer types.
- domain assumption A regressor trained on 4,000 synthetic scenes of simple primitives generalizes to arbitrary image domains.
- domain assumption DDIM inversion with a fixed seed and prompt yields latents that faithfully reconstruct the original images, making the layer analysis valid.
- domain assumption The direct-irradiance map (first-bounce diffuse plus specular) is a sufficient control signal for relighting.
Cite this review
Pith. "Pith review of PractiLight: Practical Light Control Using Foundational Diffusion Models." pith.science (2026). https://pith.science/paper/OVGLQOHF
@misc{pith2026250901837,
author = {Pith},
title = {Pith review of: PractiLight: Practical Light Control Using Foundational Diffusion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/OVGLQOHF}},
note = {Machine review of arXiv:2509.01837}
}
read the original abstract
Light control in generated images is a difficult task, posing specific challenges, spanning over the entire image and frequency spectrum. Most approaches tackle this problem by training on extensive yet domain-specific datasets, limiting the inherent generalization and applicability of the foundational backbones used. Instead, PractiLight is a practical approach, effectively leveraging foundational understanding of recent generative models for the task. Our key insight is that lighting relationships in an image are similar in nature to token interaction in self-attention layers, and hence are best represented there. Based on this and other analyses regarding the importance of early diffusion iterations, PractiLight trains a lightweight LoRA regressor to produce the direct irradiance map for a given image, using a small set of training images. We then employ this regressor to incorporate the desired lighting into the generation process of another image using Classifier Guidance. This careful design generalizes well to diverse conditions and image domains. We demonstrate state-of-the-art performance in terms of quality and control with proven parameter and data efficiency compared to leading works over a wide variety of scenes types. We hope this work affirms that image lighting can feasibly be controlled by tapping into foundational knowledge, enabling practical and general relighting.
Figures
Figures from the paper (18 more)
Reference graph
Works this paper leans on
-
[2]
This follows the assumption that heads which changed more after optimization, must correlate with light. We then followed the same experiment described in Sec 4.1 in the paper, but injected features only to the top performing head (highest norm) of each self-attention layer, versus injecting features to the worst performing head per layer (Fig. 12). The r...
work page 2018
-
[2022]
High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject- driven generation. In Proce...
arXiv 2023
-
[2023]
arXiv:2302.12288 [cs.CV] https://arxiv.org/abs/2302.12288 Chris Careaga and Yağız Aksoy
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth. arXiv:2302.12288 [cs.CV] https://arxiv.org/abs/2302.12288 Chris Careaga and Yağız Aksoy. 2023. Intrinsic Image Decomposition via Ordinal Shading. ACM Trans. Graph. 43, 1, Article 12 (2023), 24 pages. Chris Careaga and Yağız Aksoy. 2024. Colorful Diffuse Intrinsic Image Decomposition in t...
arXiv 2023
-
[2024]
SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 25096–25106. Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, and Yannick Hold- Geoffroy. 2024. Lightit: Illumination modeling and control for...
work page Pith review arXiv 2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.