Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

LightLab: Controlling Light Sources in Images with Diffusion Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A diffusion model trained on on/off photo pairs can switch any visible light in a single image on, off, dimmer, or to a new color while keeping shadows and reflections plausible, with no 3D geometry recovered.

desk verdict A genuine new capability—parametric intensity/color control of visible lights in a single image—backed by solid engineering, but the quantitative evidence for the fine-grained control claim is partly self-referential and needs an independent check. read the letter →

arxiv 2505.09608 v1 pith:K3YFXBAT submitted 2025-05-14 cs.CV cs.GR

Nadav Magar , Amir Hertz , Eric Tabellion , Yael Pritch , Alex Rav-Acha , Ariel Shamir , Yedid Hoshen This is my paper · ORCID
classification cs.CVcs.GR
keywords lighteditingrelightingdiffusionmodelsparametriccontrollinearityofadditivemodeltonemappingsingle-image
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LightLab claims that fine-grained, parametric control over the light sources visible in a single photograph — switching them on or off, dimming them, recoloring them, and adjusting ambient light — can be learned by a diffusion model, without recovering 3D geometry or estimating physical light transport. The method builds its training pairs from the linearity of light: given two photographs of the same scene differing only by one light switched on, the added light is isolated as the non-negative difference between them and recombined at arbitrary intensities and colors, producing a continuous family of relit images. A small set of 600 real raw photograph pairs anchors photorealism, while large-scale physically based renders (about 600K images after augmentation) teach plausible shadows and reflections; the authors show that the mixture prevents domain drift while adding physical accuracy. The paper reports that the resulting model beats four diffusion-based baselines on the binary switch-on/off task in both reconstruction metrics and a 100-user preference study, and demonstrates sequential edits, color changes, ambient control, and consistent lighting across animation frames. If the claims hold, LightLab is the first single-image method with explicit intensity and color control over specific visible light sources.

What carries the argument

The load-bearing object is the additive light identity $i_{relit}(\alpha, \gamma, c_t) = \alpha i_{amb} + \gamma i_{change} c$, built from the clipped residual $i_{change} = \mathrm{clip}(i_{on} - i_{off}, 0)$, where $i_{on}$ and $i_{off}$ are linear-RGB captures of the same scene with a target visible light switched on and off, $c$ is the linear-RGB color-change coefficient, and $\alpha, \gamma$ are relative ambient and light intensities in $[0,1]$. This identity turns each captured or rendered pair into a parametrized sequence of training images covering the full range of intensity and color edits, which are tone-mapped either separately or with shared exposures and then fed as a condition. The conditioning scheme is the second mechanism: the target light mask scaled by $\gamma$ and colored by $c_t$ is concatenated with the input image and depth map into the latent noise, while the ambient scalar and tone-mapping choice are embedded through a Fourier-feature MLP into the cross-attention layers of a pretrained text-to-image latent diffusion model that is fine-tuned on the mixture of real and synthetic pairs.

What would settle it

Capture a scene with strong interreflections — for example a colored wall or mirror beside a lamp — at several true bulb power levels, and compare the model's output at $\gamma = 0.5$ against an actual photograph taken at half power; systematic errors in shadowed regions, color shifts where bounced light dominates, or failure to reproduce the measured intermediate exposure would show that the additive premise or the exposure calibration behind it does not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that explicit parametric relighting of visible light sources in a single image reduces to a data problem: teach a pretrained latent diffusion model the additive structure of light through paired examples. For a pair of photographs differing only by one visible light being switched on, the light's contribution is defined as the clipped residual $i_{change} = \mathrm{clip}(i_{on} - i_{off}, 0)$, the off image is treated as ambient $i_{amb}$, and every intermediate state is the linear combination $i_{relit}(\alpha, \gamma, c_t) = \alpha i_{amb} + \gamma i_{change} c$ with relative intensities $\alpha, \gamma \in [0,1]$ and target color coefficient $c$. This formula inflates each real pair by a factor of 60 and each synthetic render by 36, generating a large training set of controlled light edits; at inference the user provides a segmentation mask of the target light, a relative intensity in $[-1,1]$, a target RGB color, an ambient scalar, and a tone-mapping mode, which are fed into the diffusion model as spatial and cross-attention conditions. The paper states that no prior work offers such fine-grained control over specific visible local light sources in a real single image, and supports the claim with PSNR/SSIM results (for example 23.2 versus 15.1 PSNR against the nearest baseline) and user-study win rates of 83–89% against four diffusion-based methods.

Load-bearing premise

Everything rests on one premise: a switched-on light adds purely non-negative light to an otherwise unchanged scene, so the clipped difference between the on and off photographs isolates that light, and any blend of ambient and added light is a physically valid re-lit image.

Editorial extensions

If this is right

  • A user can edit a photograph's lighting without any 3D reconstruction: drawing a box around a lamp and sliding an intensity or color control produces a re-lit image with plausible shadows, reflections, and environmental effects.
  • Sequential edits compose: turning off daylight from a window, then an interior lamp, then switching on another fixture and recoloring it can be applied one after another while preserving geometry and material appearance.
  • Because intensity is encoded relative to the input, the same model extrapolates beyond the trained range (intensities above 1.0) and works on out-of-domain images such as cartoons, not just photorealistic indoor photos.
  • The tone-mapping condition lets the user choose whether re-lit outputs are exposed like a single well-exposed photo or like a sequence dimmed under shared exposure, separating physical light changes from display rendering.
  • Light sources with no visible geometry can be inserted into a scene, and a lamp can be moved across frames with consistent shadows and highlights, supporting stop-motion-style animation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The on/off paired-data recipe is not specific to lamps: any linear image effect that can be captured with a clean toggle — flash, projector light, screen glow, even weather-like haze — could be turned into parametric generative controls the same way, making this a general template for supervised light editing.
  • Because the paper shows synthetic-only training drifts stylistically while a small real anchor fixes the drift, a targeted fine-tune on curated synthetic pairs for under-represented fixture classes (candles, neon, screens) followed by a small real-pair correction could broaden coverage cheaply — a testable extension beyond the paper's current dataset.
  • Because the model learns light transport implicitly, probing its outputs at intermediate intensities could yield an empirical light-transport map (where shadows and reflections fall) for a single image, a cheap inverse-rendering proxy the paper does not attempt.
  • The reported metrics favor the mixed and real-only models almost equally, and the authors note that whole-image similarity scores obscure local shadow quality; shadow-region-specific error or perceptual shadow-consistency measures would likely separate the training mixtures more sharply.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. LightLab proposes a diffusion-based method for single-image relighting with explicit parametric control over visible light sources (intensity, color), ambient illumination, and tone mapping. The method fine-tunes a latent diffusion model (SDXL-like architecture) on a mixture of a small set of real raw photograph pairs and a large set of synthetically rendered indoor images. Training pairs for fractional intensity/color/ambient edits are synthesized by linear light arithmetic (Eq. 1) from disentangled ambient and change images, then tone-mapped with either joint or separate exposure strategies exposed as a condition. Experiments report PSNR/SSIM on paired real and synthetic evaluation sets, ablations over training domains, comparisons with four relighting baselines, and a 3200-answer user study; qualitative results demonstrate shadow, reflection, and sequential editing capabilities, along with some failure cases.

Significance. If the central claims hold, LightLab is a practically useful step toward fine-grained post-capture light editing from a single image, and its recipe of combining a modest number of real captures with large-scale synthetic rendering is a sensible way to keep diffusion priors grounded. The paper includes substantial empirical assets: 600 real raw pairs (inflated to ~36K images), ~600K synthetic images, a 3200-answer user study, and an interactive demo. The conditioning design is clean, and the paper is unusually honest about failure cases (Section 4.4, Section 6). However, the quantitative evidence for the headline parametric intensity/color control is partly circular, and one baseline comparison uses oracle information from the ground truth; these issues affect the strength of the central claim and need to be addressed before the paper can be accepted.

major comments (3)
  1. [Section 4.1, Table 1, Eq. (1)] The 'Intensity' and 'Color' columns of Table 1 are computed against ground-truth targets generated by the same linear light-arithmetic pipeline (Eq. 1) that creates the training pairs. The real evaluation set of 200 photo pairs is 'expanded by a factor of 60 during post-processing', so fractional-intensity and color-change ground truths are not physically captured states but linear combinations of i_amb and i_change. High PSNR/SSIM for these columns may therefore reflect the model learning the data-generation formula rather than performing physically precise light control on real scenes. The user study (Appendix D.4) only evaluates binary on/off edits, so there is currently no human or physical ground truth for the parametric claims. Please evaluate on independently captured intensity ramps (e.g., staged dimmer settings) or on synthetic held-out scenes with physically rendered ground truth, and/or add a user study for fractional intensity and color edits. At minimum, explicitly discuss this circularity as a limitation in the evaluation section.
  2. [Appendix D.3, Table 2] The RGB↔X baseline is given oracle information: the paper states that 'all conditions from the input and ground truth images' are computed, and that the irradiance condition is replaced with the ground-truth irradiance. This means RGB↔X receives information from the target relit image, which does not match the single-image setting claimed for LightLab. Although providing this information likely helps the baseline and therefore makes the reported win rate conservative, it is not a standard comparison protocol and makes the 'outperforms existing methods' claim hard to interpret. Please either run RGB↔X in its intended single-image mode (conditions estimated from the input only) or provide a clear justification for the oracle protocol and show results under both protocols.
  3. [Tables 1 and 3] The ablation tables report single PSNR/SSIM values without variance or significance tests. For example, Table 1 shows that 'Real + Synth. w/ depth' vs 'w/o depth' differ by at most 0.1 dB, and the claimed benefit of adding synthetic data is only a 2.2% averaged PSNR improvement; these differences could easily arise from run-to-run noise. Since the paper's methodological recommendation (mixing real and synthetic data) rests on these small quantitative differences, please report standard deviations over multiple training seeds or, at minimum, statistical significance tests. The qualitative evidence in Figures 7 and D.4 supports the claim, but the quantitative claim is not yet firmly established.
minor comments (6)
  1. [Section 3.2] The color-change formula c = ct ⊙ co^{-1} requires an estimate of the original light color co, but the paper does not describe how co is obtained for real captured pairs; please specify this procedure for reproducibility.
  2. [Section 4.2] The stated '2.2% averaged improvement' from adding synthetic data does not match the table entries (23.2 vs 22.9, 28.6 vs 28.3, 24.2 vs 23.75); please clarify the computation and report the relative vs absolute basis.
  3. [Section 3.3] The joint tone-mapping strategy depends on heuristically chosen deciding intensities γ_d and α_d, but the heuristic itself is not specified; please provide the exact procedure used.
  4. [Appendix B.1, Eq. (2)] Equation (2) linearly interpolates exposure, gain, and digital-gain products, but the derivation of why this interpolation is valid for the relit image is not given; a brief justification would help readers assess potential calibration errors.
  5. [Section 3.1] 600 raw pairs inflated by a factor of 60 yields exactly 36,000 images, so 'approximately 36K images' is imprecise; also clarify whether the count includes both source and target images in each pair.
  6. [References] The 'Anonymous 2024' reference (IC-Light) is listed as under review; if this work has since been published or accepted, please update the citation.

Circularity Check

1 steps flagged · score 6.0 of 10

Intensity/color control is quantitatively validated against ground truth produced by the same Eq. 1 light arithmetic that defines the conditioning and training targets.

  1. self definitional [Section 3.2 (Eq. 1), Section 3.1, Section 4.1 'Evaluation datasets']
    "For a relative ambient illumination intensity α∈[0,1], a relative target light intensity γ∈[0,1] and the target light RGB ct the relit image is computed by: irelit(α,γ,ct; iamb,ichange)=α iamb+γ ichange c (1) ... The real photograph dataset contains 200 photo pairs of different scenes and light sources, which is expanded by a factor of 60 during post-processing."

    The quantitative 'ground truth' for the Intensity and Color columns in Tables 1-3 is not a physical capture at fractional intensity or altered color; it is produced by the same post-processing pipeline (Section 3.2) used to generate all training pairs, applying Eq. 1 with the same scalar γ and RGB c that are also fed to the model as conditioning masks. A model that learns to reproduce the training distribution's light arithmetic will therefore score well on these metrics by construction, and the scores do not independently establish that fractional intensity or color edits are physically accurate on real scenes. The real evaluation pairs are 'expanded by a factor of 60 during post-processing', i.e. the continuous targets are synthetic combinations of iamb and ichange.

full rationale

LightLab is a supervised learning method, so there is no closed-form derivation chain that could be said to derive its outputs from its inputs. The training pairs are honestly and explicitly generated from real on/off photograph pairs and synthetic renders using the linearity of light (Eq. 1). The circularity is in the evaluation of the headline capability: fine-grained parametric intensity and color control. The continuous intensity/color ground truths are synthesized by the same Eq. 1 arithmetic, with the same α, γ, and c used as conditioning, so the PSNR/SSIM numbers partly measure how well the model reproduces the data-generation rule rather than physically measured light behavior. The binary on/off evaluation uses genuine physical captures and is independent, and the user study independently supports the binary editing claim, but it tests only on/off, not continuous intensity or color. No load-bearing self-citation chain or imported uniqueness theorem is present. Score 6 reflects a central quantitative claim that partially reduces by construction, without reaching the higher scores, because the core method is a trained estimator and the binary/qualitative results retain independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the additivity of light, the quality of the captured and rendered paired data, and a set of hand-chosen data-construction and tone-mapping parameters. The diffusion model itself is learned, so the scientific free parameters are mostly in the data-generation pipeline rather than in a closed-form derivation.

free parameters (4)
  • Real/synthetic mixture weighting = not reported
    Table 1 compares different weightings between domains, but the exact mixing ratio used for the final model is not given, so the data recipe is not fully specified.
  • Tone-mapping decision intensities gamma_d, alpha_d = not reported
    Section 3.3 selects these heuristically to set the fixed exposure for tone mapping image sequences; the values are not stated or ablated.
  • E_max percentile bound = top 5e-4 percentile, computed on 2000 renders
    Section 3.1 clips unbounded path-traced pixel values at this percentile to avoid noisy tone mapping; it shapes the synthetic training distribution.
  • Post-processing inflation factors = 60 for real pairs, 36 for synthetic pairs
    Section 3.1 uses these multipliers to expand the dataset; they determine the balance of intensity and color examples the model sees.
assumptions (4)
  • standard math Linearity of light: the radiance of a scene is the sum of contributions from each light source in linear RGB space.
    Used throughout Section 3.2 to construct relit images via Equation 1.
  • domain assumption The off image i_off equals ambient illumination and i_change = clip(i_on - i_off, 0) isolates the target light contribution.
    Section 3.1 defines this decomposition; it fails if the two captures differ in exposure, white balance, or ambient conditions, though calibration and clipping are used.
  • domain assumption Physically-based path-traced renders with procedural scenes are representative enough of real indoor lighting for the fine-tuned diffusion model to generalize.
    Section 3.1 and 4.2 rely on synthetic data to improve physical plausibility; the paper itself reports a domain gap when training on synthetic only.
  • ad hoc to paper Fixed-exposure tone mapping of linear image sequences can be applied jointly and exposed as a conditioning signal without losing the diffusion model's distribution.
    Section 3.3 introduces tone-mapping together and separate conditioning with heuristic exposures; Figure D.12 notes the learned control is not fully disentangled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LightLab: Controlling Light Sources in Images with Diffusion Models." pith.science (2026). https://pith.science/paper/K3YFXBAT

@misc{pith2026250509608,
  author       = {Pith},
  title        = {Pith review of: LightLab: Controlling Light Sources in Images with Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3YFXBAT}},
  note         = {Machine review of arXiv:2505.09608}
}
read the original abstract

We present a simple, yet effective diffusion-based method for fine-grained, parametric control over light sources in an image. Existing relighting methods either rely on multiple input views to perform inverse rendering at inference time, or fail to provide explicit control over light changes. Our method fine-tunes a diffusion model on a small set of real raw photograph pairs, supplemented by synthetically rendered images at scale, to elicit its photorealistic prior for relighting. We leverage the linearity of light to synthesize image pairs depicting controlled light changes of either a target light source or ambient illumination. Using this data and an appropriate fine-tuning scheme, we train a model for precise illumination changes with explicit control over light intensity and color. Lastly, we show how our method can achieve compelling light editing results, and outperforms existing methods based on user preference.

Figures

Figures reproduced from arXiv: 2505.09608 by the authors.

Figure 1
Figure 1. Light editing results of LightLab. Our method enables explicit parametric control over light sources in an image, while producing physically plausible shadows and environmental effects (top). The method can manipulate the intensity of light sources, change their color, and adjust ambient illumination (middle). LightLab can be used for intricate sequential editing of lighting in images (bottom from left to right): st… view at source ↗
Figure 2
Figure 2. Post processing pipeline. Top row. From a pair of real (raw) photograph pairs, we first isolate the target light change ichange. Bottom row. For synthetic data, we render each light component separately. After light disentanglement both domains undergo light arithmetic to create parameterized sequences of images irelit (𝛼,𝛾, ct), which are later tone mapped to SDR (either together or separately). spatial control ove… view at source ↗
Figure 3
Figure 3. Conditioning Signals. Spatial conditions (input image, target light mask and depth map) are embedded to the latent dimensions and concatenated to the input noise. Light intensity and color control are applied by scaling the intensity and color of the target light mask. Global controls (ambient light intensity and tone-mapping value) are projected to text embedding dimension and inserted through cross-attention. As w… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Tone mapping strategy. A sequence of images of increasing light intensity, tone mapped either separately or together. Top row. The images tone mapped separately, notice how the light source intensity appears constant when lit, while ambient light appears to be dimmed. …
Figure 5
Figure 5. Figure 5: Intensity control. Fine-grained control over a target light’s inten￾sity using our method. Values represent the relative intensity change with respect to the source image. image sequences, where the perceived changes in light intensity do not align with the physical ch…
Figure 7
Figure 7. Figure 7: Qualitative Comparison: The effect of adding synthetic data to the training dataset. Lime The specular reflection of the screen is simulated only in the mixed model. Red The mixed model generates a fine hard shadow for the cable which correlates with its shape and rese…
Figure 8
Figure 8. Figure 8: Qualitative Comparison. Comparison with other works on images from IIW dataset. Left. the input image where green / red contours specify which light source(s) should be turned on / off respectively. See Figures D.5, D.6 in the supplementary for more examples. succeed i…
Figure 9
Figure 9. Figure 9: Qualitative Results. Select results on publicly sourced images, demonstrating successful lighting edits (lime boxes) and failures (red boxes) of our method. See the accompanying discussion in Section 4.4. 5 APPLICATIONS We present several possible applications of our m…
Figure 10
Figure 10. Figure 10: Additional results.. Our method can control a target light intensity (third row), light color (second row) and a scene’s ambient light (fourth row) and tone-mapping effects (fifth row) across diverse scenes and image styles. The number above the results indicate the r…
Figure 11
Figure 11. Figure 11: Virtual Point Lights. By using synthetic renderings of point lights without geometry we can insert an "invisible" light source into the scene. On the left of each sequence is the input image, red "X" marks the insertion point. Notice how the localization of the light …
Figure 12
Figure 12. Figure 12: Physically Plausible Lighting. Top row. The input sequence was created by capturing photographs of a turned-off lamp rotated around the polygon dog. Bottom rows. Inference results of our method, Orange a zoom-in on the dog. Note how self occlusions on the different fa…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Semantic encoders can harm relighting, and ALI—fusing dense visual features with latent intrinsics—improves relighting on glossy and specular materials.

Reference graph

Works this paper leans on

5 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [5]

    binary". As endpoint intensities - where the light is either

    Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS (2020). Guangcong Wang, Yinuo Yang, Chen Change Loy, and Ziwei Liu. 2022. StyleLight: HDR Panorama Generation for Lighting Estimation and Editing. In European Conference on Computer Vision (ECCV) . Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset,...

    arXiv 2020

  2. [2017]

    Illuminant Spectra-based Source Separation Using Flash Photography

    Illuminant Spectra-based Source Separation Using Flash Photography. arXiv:1704.05564 [cs.CV] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017). Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. 2024. Neura...

    work page Pith review arXiv 2017

  3. [2020]

    In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Learning Physics-Guided Face Relighting Under Directional Light. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 5123–5132. https://doi.org/10.1109/cvpr42600.2020.00517 Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. GLIDE: Towards Pho- tore...

    arXiv 2022

  4. [2022]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation . Springer International Publishing, 234–241. https://doi.org/10.1007/978...

    arXiv 2022

  5. [2023]

    OutCast: Outdoor Single-image Relighting with Cast Shadows

    Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems 36 (2023), 35799–35813. David Griffiths, Tobias Ritschel, and Julien Philip. 2022. OutCast: Outdoor Single-image Relighting with Cast Shadows. arXiv:2204.09341 [cs.GR] https://arxiv.org/abs/2204. 09341 Paul Haeberli. 1992. Synthetic Lighting for Photography. htt...

    work page Pith review arXiv 2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.