REVIEW 4 major objections 5 minor 13 references
Relighting can expose what visual encoders preserve about the physical world, and this paper shows that the strongest semantic encoders often degrade, rather than improve, relighting quality—a semantic–photometric trade-off that its ALI met
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 05:38 UTC pith:H4673YH4
load-bearing objection Solid relighting pipeline with an interesting probe idea, but the headline semantic-photometric trade-off is not cleanly supported by the paper's own Table 5. the 4 major comments →
Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper reports that in a fixed latent-intrinsic diffusion relighting framework, swapping only the frozen visual encoder changes relighting quality in a way that is inverse to semantic strength: CLIP and DINOv2/DINOv3, optimized for invariance, systematically underperform MAE, trained to reconstruct pixels, and RADIO, a multi-teacher distilled encoder with pixel alignment, performs best. This is interpreted as evidence of a semantic–photometric trade-off: semantic abstraction discards dense geometry, material, and shading cues that light transfer requires. The proposed ALI method fuses RADIO features into latent intrinsics and, after a decoder-alignment and self-refinement stage, improves
What carries the argument
Latent intrinsics—lighting-invariant hierarchical scene features learned from multi-illumination image pairs without labels—form the base representation. ALI augments them by upsampling and concatenating intermediate feature maps from a frozen visual encoder into a pixel-aligned hypercolumn descriptor, then projecting that descriptor with a learnable layer and adding it to the intrinsic features. A three-stage schedule carries the argument: stage one trains the relighting encoder and projection layer under reconstruction, improved intrinsic-invariance, and hyperspherical-regularization losses; stage two freezes these and fine-tunes the diffusion decoder; stage three self-refines on pseudo-re
Load-bearing premise
The headline conclusion depends on attributing relighting differences to the encoder's training objective, but the compared encoders also differ in patch stride, resolution, and feature density, so if those confounds—not the objective—drive the gap, the semantic–photometric trade-off is overstated.
What would settle it
Take one encoder architecture and train two versions identical in patch size, input resolution, layer count, and feature dimension—one with a contrastive semantic objective, one with pixel reconstruction—then compare relighting RMSE under the same ALI pipeline; equal performance would falsify the claim that semantic objectives degrade relighting. A cheaper check is feeding DINOv3 features through the same hypercolumn alignment used for RADIO and matching feature stride before comparing.
If this is right
- If the central claim is correct, recognition scores are the wrong selection criterion for encoders used in physical image synthesis; an encoder's training objective and pixel alignment should be the primary criteria.
- Relighting can serve as a reproducible probe: ranking encoders by their performance in a fixed latent-intrinsic relighting pipeline measures how much photometric structure they preserve.
- Careful prior integration can substitute for scale and dense supervision: trained only on unlabeled real multi-illumination pairs, ALI matches or beats methods that use privileged inputs such as environment maps, G-buffers, or albedo.
- The gains concentrate on glossy, metallic, and specular materials, implying that tasks involving view-dependent reflectance are where representation choice matters most.
- The latent-intrinsic representation separates scene intrinsics from lighting codes well enough that interpolating the lighting code yields continuous, plausible lighting changes.
Where Pith is reading between the lines
- The trade-off likely extends to other inverse-graphics tasks—reflectance editing, material estimation, and view synthesis—since they share the need for pixel-aligned photometric cues; testing ALI-style fusion there would show whether the result is relighting-specific or general.
- A controlled re-test that holds architecture and feature density fixed while varying only the pre-training objective (and, separately, augmentation strength) would isolate the causal variable behind the trade-off.
- The authors' 'relightability' framing could be turned into a two-axis evaluation—recognition accuracy versus relighting RMSE/SSIM—giving model developers a cheap physical-fidelity check for any new encoder before building a generative decoder.
- Because the self-refinement stage lowers MIIW scores while improving in-the-wild generalization, a dedicated out-of-distribution relighting benchmark would make the robustness gain visible instead of letting it appear as a regression.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Augmented Latent Intrinsics (ALI), a three-stage pipeline that fuses features from a frozen pretrained visual encoder into the LumiNet latent-intrinsic relighting framework. Stage I injects a hypercolumn descriptor from the encoder into the intrinsics encoder, Stage II fine-tunes the diffusion decoder to the augmented representation, and Stage III applies self-refinement on pseudo-relit pairs. The authors report state-of-the-art MIIW results among open-sourced diffusion methods and, more centrally, claim a 'semantic–photometric trade-off': encoders trained for semantic invariance (CLIP, DINO) degrade relighting quality, while reconstruction-trained or pixel-aligned encoders (MAE, RADIO) preserve photometric fidelity. The relighting task is thus proposed as a probe of what visual representations encode about the physical world.
Significance. If the trade-off claim were cleanly established, it would be a valuable finding for representation learning in physically grounded image generation, challenging the assumption that stronger semantic encoders uniformly benefit downstream tasks. The ALI method itself appears technically sound: the multi-stage ablations and material-wise evaluations support internal consistency, and the reported relighting improvements, if reproduced, would be a useful practical contribution. The independent concurrent work RAE (Zheng et al., 2025) also supports the direction that reconstruction-based representations can outperform DINO-style features under diffusion decoding, which reduces but does not eliminate concern about self-confirmation. However, the central probe conclusion is not supported by the current evidence, because the encoder comparison in Table 5 is confounded by decoder retraining and by uncontrolled architectural differences. The paper's conceptual contribution therefore needs substantial additional analysis before the headline claim can be accepted.
major comments (4)
- [§6, Table 5] Table 5's Stage I results contradict the central narrative. At Stage I, MAE has RMSE 0.2195, which is worse than DINOv3's 0.1794 and comparable to CLIP's 0.2189. MAE overtakes the semantic encoders only after Stage II, where the LumiNet diffusion decoder is fine-tuned for each encoder with a fixed 2-epoch budget. Since Stage II retrains the decoder separately for each feature type, the final ordering can reflect feature-conditioned optimization dynamics or decoder compatibility rather than the information content of the pretrained representation. To support the claim that the training objective is decisive, the authors should measure frozen-feature relightability with a single shared decoder and controlled conditioning, or otherwise show that the Stage II gains do not depend on feature-specific decoder adaptation.
- [§5, Table 5 and §6 'Stronger Semantic Encoders' paragraph] The encoder comparison conflates multiple factors: patch stride, input resolution, feature density, layer selection, and pretraining data. The DINOv2→DINOv3 transition only partially controls for feature density, and even that changes architecture and training scale. The claim that 'the training objective is decisive' is therefore not isolated by the presented experiments. The authors should add controlled comparisons, e.g., using matched patch sizes and channel widths, or explicitly ablating feature upsampling/density while holding the pretraining objective fixed. Without such controls, the semantic–photometric trade-off framing overstates what Table 5 can establish.
- [§5, Table 3] The user-study percentages in Table 3 are uninterpretable: each row sums to greater than 1. For example, in the 'Lighting alignment' row, 0.125 + 0.415 + 0.931 = 1.471, and the other rows similarly exceed 1. If the task was forced-choice among three methods, each row should sum to 1; if the task was 'select all that apply', the protocol and aggregation should be described. Since the paper invokes this table to support 'significantly better' human judgments, the correct percentages and a clear description of the forced-choice setup are needed. This may also affect the comparison with Latent-Intrinsics on identity preservation.
- [§4, §6; baselines in Tables 1–5] The probe is built entirely inside the authors' own LumiNet framework, and the strongest baselines (Latent-Intrinsic, LumiNet) are prior work by the same group. The observed encoder ranking could be specific to this latent-intrinsic architecture. The paper cites concurrent RAE as independent support, but RAE addresses reconstruction, not relighting. To reduce this circularity concern, at least one experiment should demonstrate that the same encoder ranking holds under a different diffusion-based relighting decoder, or the paper should explicitly frame the conclusion as 'within the LumiNet/ALI framework'. As written, the abstract and discussion generalize beyond the single architecture, and this generalization is load-bearing for the probe claim.
minor comments (5)
- [Title] The arXiv metadata title ('Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics') differs from the title on the first page ('Stronger Semantic Encoders Can Harm Relighting Performance: Probing Visual Priors via Augmented Latent Intrinsics'). Please harmonize in the final version.
- [§4, Stage 1] The text states that exact layer indices and feature dimensions are provided in the supplementary material, but the submitted manuscript does not include this table. Please include the hypercolumn layer selection and dimensions, as they are needed to reproduce the Stage I fusion.
- [Table 1 abstract vs. table] The abstract states 'improving RMSE by 4.5% and SSIM by 4.9% over LumiNet'. Table 1 shows RMSE 0.294 vs 0.310 (raw) and 0.231 vs 0.240 (color-corrected); the 4.5% and 4.9% figures appear to be from the color-corrected block (3.75% RMSE and 4.9% SSIM). Please clarify which protocol the abstract refers to.
- [Table 4 and §5] The text says Stage III 'slightly reduces quantitative scores on the MIIW benchmark', but Table 4 shows mixed changes (e.g., Metallic PSNR increases from Stage I&II to All Stage, while Diffuse decreases). Specify the averaged protocol or note the exceptions to avoid confusion.
- [Appendix B.1] The material clustering relies on GPT-O3 to produce 'BRDF-style categories', but there is no validation of the clustering. Table S.1 also contains inconsistencies: Tile appears in both Glossy (27) and Uncertain/Mixed (27), and Granite/marble is placed in Uncertain. Please provide a manually verified mapping or a human-validated agreement score, as the material-wise claims depend on these categories.
Circularity Check
No circular derivation: ALI's central claims are empirical comparisons on MIIW; the trade-off is a measured correlation, not an equation that reduces to its inputs.
full rationale
The paper's derivation chain is empirical rather than definitional. ALI augments LumiNet's latent intrinsics with frozen visual features, fine-tunes the decoder, and evaluates on the MIIW test split. The 'semantic–photometric trade-off' is an observed inverse relation between ImageNet linear-probing accuracy and relighting PSNR—two externally measured quantities, not quantities defined in terms of each other. Table 5's encoder comparison is a standard representation-evaluation protocol: Stage II trains the same diffusion decoder for each feature type, so the final ranking is a measured outcome rather than a parameter fitted to the conclusion. The paper even reports Stage I numbers showing the ordering is not pre-wired (MAE trails DINOv3 at Stage I: RMSE 0.2195 vs 0.1794), which indicates a possible confound between encoder objective, spatial density, and decoder adaptation, but this is a correctness/interpretation concern, not circularity. Self-citations to Latent Intrinsics and LumiNet supply the base architecture, but the method is then validated against external baselines and the MIIW benchmark, and concurrent RAE (Zheng et al., 2025) is cited as independent support for the reconstruction-versus-semantic trend. No uniqueness theorem is imported, no ansatz is smuggled in via self-citation, and no prediction is equivalent to its fit by construction. The paper's own caveat that pixel metrics correlate weakly with perceived lighting quality further shows the authors do not treat the quantitative trade-off as a self-justifying definition.
Axiom & Free-Parameter Ledger
free parameters (4)
- Number of intermediate layers N for hypercolumn descriptor
- Hyperspherical regularization temperature λ
- Lighting zoo scenes sampled for Stage III =
1,000
- Identity relighting mixing ratio in Stage III =
10%
axioms (5)
- domain assumption Diffusion denoising objective (DDPM) is a suitable generative model for image relighting.
- domain assumption MIIW and BigTime datasets provide sufficiently diverse multi-illumination pairs to learn latent intrinsics.
- domain assumption Relighting metrics (PSNR/SSIM/RMSE) operationalize 'relightability' and physical fidelity.
- domain assumption ImageNet-1K linear probing accuracy is a valid measure of semantic strength for the compared encoders.
- ad hoc to paper GPT-O3 clustering of material labels into BRDF-style categories is physically meaningful.
invented entities (2)
-
ALI (Augmented Latent Intrinsics) representation
independent evidence
-
Lighting Zoo (pseudo-relit image pairs)
no independent evidence
read the original abstract
Image-to-image relighting requires representations that separate illumination from scene properties while preserving dense geometry, material, and photometric cues. We use this task as a probe of visual priors: unlike recognition tasks that reward invariance, relighting tests whether visual features retain the information needed for light transfer. Through a controlled generative relighting framework, we find that strong semantic encoders can degrade relighting quality, exposing a semantic--photometric trade-off between abstraction and physical fidelity. We introduce Augmented Latent Intrinsics (ALI), which balances this trade-off by fusing dense, pixel-aligned visual features into a latent-intrinsic relighting model and refining it with self-supervision on unlabeled real image pairs. ALI improves relighting quality, especially on glossy, metallic, and transparent materials, and demonstrates that generative relighting is an effective tool for quantifying what visual encoders encode about the physical world.
Figures
Reference graph
Works this paper leans on
-
[4]
Radiov2.5: Improved baselines for agglomerative vision foundation models
Heinrich, G., Ranzinger, M., Hongxu, Yin, Lu, Y ., Kautz, J., Tao, A., Catanzaro, B., and Molchanov, P. Radiov2.5: Improved baselines for agglomerative vision foundation models. InCVPR2025, 2024a. Heinrich, G., Ranzinger, M., Lu, Y ., Kautz, J., Tao, A., Catanzaro, B., Molchanov, P., et al. Radio amplified: Improved baselines for agglomerative vision foun...
-
[5]
Kocsis, P., Philip, J., Sunkavalli, K., Nießner, M., and Hold- Geoffroy, Y . Lightit: Illumination modeling and control for diffusion models.arXiv preprint arXiv:2403.10615, 2024a. Kocsis, P., Sitzmann, V ., and Nießner, M. Intrinsic image diffusion for single-view material estimation. InCVPR, 2024b. Li, J., Li, H., and Matsushita, Y . Lighting, reflectan...
-
[9]
doi: 10.1145/3450626.3459872. Phongthawee, P., Chinchuthakun, W., Sinsunthithet, N., Raj, A., Jampani, V ., Khungurn, P., and Suwajanakorn, S. Dif- fusionlight: Light probes for free by painting a chrome ball. InCVPR,
-
[12]
Latent intrinsics emerge from training to relight
Zhang, X., Gao, W., Jain, S., Maire, M., Forsyth, D., and Bhattad, A. Latent intrinsics emerge from training to relight. InNeurIPS, 2024a. Zhang, X., Jiang, R., Gao, W., Willett, R., and Maire, M. Residual connections harm generative representation learn- ing.arXiv preprint arXiv:2404.10947, 2024b. Zhang, X., Yunis, D., and Maire, M. Decipher- ing’what’an...
-
[13]
11 Stronger Semantic Encoders Can Harm Relighting Performance A
URL https://arxiv.org/abs/ 2303.07634. 11 Stronger Semantic Encoders Can Harm Relighting Performance A. Implementation details A.1. Training All experiments are conducted on a node equipped with 8 NVIDIA A6000 Ada 48GB GPUs. The model is trained at a resolution of512ˆ512with an effective batch size of 64 (including gradient accumulation). For Stage I, we ...
Pith/arXiv arXiv 2014
-
[2014]
Bharadwaj, S., Feng, H., Becherini, G., Abrevaya, V . F., and Black, M. J. Genlit: Reformulating single-image relight- ing as video generation.arXiv preprint arXiv:2412.11224,
-
[2018]
Lightlab: Controlling light sources in images with diffusion models.arXiv preprint arXiv:2505.09608,
Magar, N., Hertz, A., Tabellion, E., Pritch, Y ., Rav-Acha, A., Shamir, A., and Hoshen, Y . Lightlab: Controlling light sources in images with diffusion models.arXiv preprint arXiv:2505.09608,
-
[2020]
Masked autoencoders are scalable vision learners
9 Stronger Semantic Encoders Can Harm Relighting Performance He, K., Chen, X., Xie, S., Li, Y ., Doll ´ar, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000–16009, 2022a. He, K., Chen, X., Xie, S., Li, Y ., Doll ´ar, P., and Girshick, R. Ma...
-
[2021]
URLhttps://arxiv.org/abs/2104.09886. Li, Z. and Snavely, N. Learning intrinsic image decomposi- tion from watching the world.CVPR,
-
[2022]
Xing, X., Hu, V . T., Metzen, J. H., Groh, K., Karaoglu, S., and Gevers, T. Retinex-diffusion: On controlling illumi- nation conditions in diffusion models via retinex theory. arXiv preprint arXiv:2407.20785,
-
[2023]
URL https://arxiv.org/abs/ 2211.10206. Liang, R., Gojcic, Z., Ling, H., Munkberg, J., Hasselgren, J., Lin, Z.-H., Gao, J., Keller, A., Vijaykumar, N., Fidler, S., and Wang, Z. Diffusionrenderer: Neural inverse and forward rendering with video diffusion models. InCVPR, June
-
[2024]
Du, X., Kolkin, N., Shakhnarovich, G., and Bhattad, A. Generative models: What do they know? do they know things? let’s find out!arXiv preprint arXiv:2311.17137,
-
[2025]
Vincent, P., Larochelle, H., Bengio, Y ., and Manzagol, P.-A
URLhttps://arxiv.org/abs/2508.10104. Vincent, P., Larochelle, H., Bengio, Y ., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. InProceedings of the 25th international conference on Machine learning, pp. 1096–1103,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.