REVIEW 4 major objections 7 minor 18 references
Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models
T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Relighting a photo, with GPT-chosen colors and a pretrained relighter, can make vision-language models give wrong captions and answers while the image still looks natural.
desk verdict Plausible relighting attack on VLP models, held back by an undocumented gradient path through a diffusion model and an overstated naturalness claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pure reference lighting image $L = G(c_s, c_e, d, w)$: a synthetic light field with start color, end color, weight, and direction, generated by a lighting-image generator $G$ from parameters initially chosen by ChatGPT. The pretrained diffusion relighter IC-Light ($M^*$) imposes that light on the clean image without changing scene content, and the two-step optimization alternates between Eqs. 5 and 6: one gradient step on the lighting parameters and one SGA-style step on the reference lighting image, using multi-resolution resizing. The guiding loss is $J = \arg\max(CE(f_\phi(R), f_\varphi(T)) + CS(h_\phi(R), h_\phi(I)))$, which separates the relighted image from the text encoding while keeping it close to the original image in CLIP image-embedding space.
What would settle it
Run a human perceptual study in which raters compare LightD images against ALA and EdgeFool images at matched image-captioning degradation: if LightD is not judged more natural, the central balance claim fails. Separately, if backpropagation through IC-Light is too noisy to follow, re-running the two-step optimization with the reference-lighting-image update removed should change the reported attack scores measurably.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that semantically guided relighting is a usable and under-explored attack surface for VLP models. LightD produces $R = M^*(L, I)$ by applying a pretrained diffusion-based relighter $M^*$ to a clean image $I$ with a pure reference lighting image $L = G(c_s, c_e, d, w)$, whose colors and direction are initially recommended by a language model. The two-step collaboration optimization alternates between updating the lighting parameters and updating the reference lighting image with an SGA-style multi-resolution augmentation, using a loss that simultaneously maximises caption/VQA error and cosine similarity to the original image in a CLIP embedding space. The reported experiments claim that this combination beats prior relighting and color attacks on CLIPCap, BLIP, and BLIP2 for captioning and VQA, while producing lower NIQE values than most baselines in most settings, i.e., better measured naturalness.
Load-bearing premise
The method assumes that gradients of the attack loss can be computed through the pretrained diffusion-based IC-Light relighter, and that NIQE and CLIP cosine similarity faithfully capture whether humans would find the relighted images natural.
Editorial extensions
If this is right
- If the claim holds, captioning and VQA systems that accept arbitrary user images face a practical threat from images whose lighting alone is changed, with no noise-like perturbation for denoisers to remove.
- The general adaptation objective in Section 4.4 lets existing non-suspicious attacks from image classification be re-targeted at VLP models, and LightD demonstrates that relighting attacks can be made to work there.
- The ablation shows the two-step optimization is necessary: optimizing only lighting parameters or only the reference image degrades attack performance, and combining both gives the strongest reported drop.
- GPT-based initialization contributes visual plausibility rather than raw attack strength, since random initialization reaches comparable caption degradation on the reported MSCOCO/BLIP2 setting.
- Because the adversarial images are produced by a relighting operator rather than bounded pixel noise, standard adversarial training and input denoising defenses are unlikely to remove the perturbation.
Reading between the lines
- A testable extension is physical relighting: if the attack transfers to images printed or displayed under the adversarial illumination, the threat moves from digital editing to real-world presentation attacks.
- The naturalness claim rests on NIQE and CLIP cosine similarity, so a human-perception study could reveal whether the relighted images are really less suspicious to people than baseline attacks' images.
- The method is white-box in the sense that it needs gradients through both the victim encoder and the relighter; whether the attack transfers to black-box, API-only VLP models is an open next step.
- A defender could look for lighting signatures rather than pixel noise: training multimodal models with relighting augmentations or detecting illumination inconsistency might neutralise this class of attack.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LightD, a framework for generating natural adversarial examples for vision-language pretraining (VLP) models by relighting clean images. It uses ChatGPT to propose context-aware initial lighting parameters (start/end colors, direction), generates a reference lighting image via a lighting-image generation function G, and applies the pretrained diffusion-based IC-Light model to relight the clean image. A two-step 'lighting-based collaboration optimization' then adjusts the lighting parameters and the reference lighting image, using a loss that combines an attack term and a CLIP cosine-similarity naturalness term. The method is evaluated on image captioning (CLIPCap, BLIP, BLIP2 on MSCOCO, Flickr8K, Flickr30K) and VQA (BLIP, BLIP2 on MSCOCO, DAQUAR), and reported to outperform prior non-suspicious attacks in attack effectiveness while maintaining naturalness.
Significance. If the implementation details are resolved, LightD would be a useful contribution to adversarial robustness of VLP models, being one of the first to adapt semantic relighting attacks to VLP tasks. The broad evaluation across multiple datasets and models is a strength, as is the proposed general framework for transferring non-suspicious attacks from classification to VLP tasks. However, the paper currently omits crucial implementation details for gradient-based optimization through a diffusion model, and its naturalness claims are inconsistent with its own NIQE tables. These issues must be addressed before the claimed 'maintained visual naturalness' and 'superiority' can be accepted.
major comments (4)
- [§4.3, Eq. (5)–(6)] The optimization steps in Eqs. (5) and (6) require gradients of the loss with respect to the lighting parameters θ and the reference lighting image L, computed through the frozen diffusion-based relighting model M* in Eq. (3). The paper never explains how these gradients are obtained. Naive backpropagation through the stochastic iterative denoising chain of a diffusion model is not a standard default; it would be memory-intensive, high-variance, and is not mentioned. The only algorithmic reference is to SGA for input resizing augmentation (Appendix C), which does not address gradients through M*. Without specifying a concrete gradient approximation (e.g., one-step surrogate, score distillation, DDIM inversion, or treating M* as a black-box with finite differences), the reported improvements in Table 4 (e.g., lighting-image optimization alone reducing BLEU from 0.757 to 0.612) cannot be reproduced or causally attributed to the described optimizer.
- [§5.2, Tables 1–2] The claim that "NIQE of our method obtains the lowest values among most models on two datasets" is contradicted by the tables. On BLIP/MSCOCO (Table 1), LightD has NIQE 9.783, whereas the clean image has 5.788 and SemanticAdv has 5.672; on BLIP/Flickr30K, LightD has 9.823 versus clean 5.621 and SemanticAdv 5.542. Thus the blanket statement of maintained visual naturalness is not supported by the paper's own numbers. Moreover, NIQE is a no-reference quality index and is not a validated proxy for human-perceived naturalness; no human evaluation is reported, so the 'naturalness' contribution of the method remains only partially substantiated.
- [Eq. (7), §4.3] The loss function J is written as CE(f_phi(R), f_phi(T)) + CS(h_phi(R), h_phi(I)), where f_phi and f_phi are image and text encoders. Cross-entropy is not defined between feature vectors; it would need to be applied to the victim model's output distribution (e.g., caption token probabilities or VQA answer logits). The paper must specify the exact loss used for each task and how it is differentiated. This is central because the gradient-based optimization in Eqs. (5)–(6) is driven entirely by this loss, and the current notation is ambiguous at best.
- [§5.3, Tables 3 and 4] All reported values are single runs over 1,000 samples with no standard deviations or significance tests. Several comparisons that motivate the design choices are numerically close (e.g., Table 3: random 0.609 vs GPT 0.605 BLEU; Table 4: both steps 0.605 vs lighting-image-only 0.612). Without error bars, the claimed superiority and the benefit of the GPT initialization and the two-step optimization are not established.
minor comments (7)
- [§4.1] Typo: 'LigthD' should be 'LightD'.
- [§6] Typo: 'LighD' should be 'LightD'; also 'comtains' in §5.3 should be 'contains'.
- [§4.2, Eq. (2)] The lighting image generator G(c_s, c_e, d, w) is referenced to 'ComfyUI, 2024' but its implementation is not described; please detail how the gradient/direction and color transition are rendered.
- [§4.3] The step size α, the number of iterations, and the scheduler for the two-step optimization are not reported; these are needed for reproducibility.
- [Appendix E] Table 5 shows NIQE improving from M=1 (8.868) to M=3 (8.258) but then worsening at M=7 (8.412); the text as written ('attack performance improves as the number of resizing iterations increases') applies to the captioning metrics, but the naturalness trend is non-monotonic and should be discussed.
- [Appendix B] The GPT prompt is presented without spaces and is hard to read; please reformat.
- [Tables 1–6] Several NIQE values appear to be inconsistent with the 'clean' values across tasks (e.g., BLIP/MSCOCO clean NIQE 5.788 in both Tables 1 and 2, but LightD NIQE 9.783 in Table 1 and 5.720 in Table 2); please verify the reported values and clarify whether the adversarial images differ between tasks.
Circularity Check
No circular derivation found; LightD's attack objective and evaluation are externally benchmarked, with only minor self-citations as baselines.
full rationale
The derivation chain in Sections 4.2-4.4 is a standard white-box adversarial optimization: ChatGPT proposes initial lighting parameters (Eq. 2), IC-Light produces R = M*(G(theta), I) (Eq. 3), and Eqs. 4-7 maximize an objective J combining victim-model loss with a CLIP embedding-similarity naturalness regularizer. None of these quantities is defined in terms of the reported outcome metrics (BLEU, CIDEr, APA, NIQE). Attack performance is measured against external victim models (CLIPCap, BLIP, BLIP2) on public datasets, and naturalness is assessed with the external NIQE index rather than the CLIP similarity used in Eq. 7. The paper does not fit a parameter to a subset and then re-report that fit as a prediction. Table 3's GPT-vs-random comparison shows the GPT prior is not the source of the attack numbers. Self-citations (ALA, Jadena, face relighting) appear as related work and baselines, not as load-bearing justification, so they do not constitute circularity. The lack of detail on backpropagating through the diffusion relighter is a reproducibility and verification gap, not a circular step; no equation reduces to its own input by construction.
Assumptions & free parameters
free parameters (4)
- Number of resizing scales M in reference lighting image optimization =
5
- Initial lighting weight w =
1.0
- GPT-recommended initial lighting parameters (start color, end color, direction) =
varies per image, e.g., (darkgray, lightyellow), (olivegreen, khaki)
- Optimization step size and iteration count =
not reported
assumptions (5)
- domain assumption The pretrained IC-Light model M* relights an image using a reference lighting image while preserving semantic content (Eq. 3)
- domain assumption Gradient of the loss with respect to lighting parameters and reference lighting image can be computed through G and M* (Eqs. 5-6)
- domain assumption The victim VLP models are differentiable and accessible for white-box gradient computation
- ad hoc to paper Cosine similarity with a CLIP image encoder (h_phi) is a valid naturalness/visual similarity constraint
- ad hoc to paper NIQE score is a reliable proxy for visual naturalness
Cite this review
Pith. "Pith review of Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models." pith.science (2026). https://pith.science/paper/WDIZN3N7
@misc{pith2026250524227,
author = {Pith},
title = {Pith review of: Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/WDIZN3N7}},
note = {Machine review of arXiv:2505.24227}
}
read the original abstract
While adversarial attacks on vision-and-language pretraining (VLP) models have been explored, generating natural adversarial samples crafted through realistic and semantically meaningful perturbations remains an open challenge. Existing methods, primarily designed for classification tasks, struggle when adapted to VLP models due to their restricted optimization spaces, leading to ineffective attacks or unnatural artifacts. To address this, we propose \textbf{LightD}, a novel framework that generates natural adversarial samples for VLP models via semantically guided relighting. Specifically, LightD leverages ChatGPT to propose context-aware initial lighting parameters and integrates a pretrained relighting model (IC-light) to enable diverse lighting adjustments. LightD expands the optimization space while ensuring perturbations align with scene semantics. Additionally, gradient-based optimization is applied to the reference lighting image to further enhance attack effectiveness while maintaining visual naturalness. The effectiveness and superiority of the proposed LightD have been demonstrated across various VLP models in tasks such as image captioning and visual question answering.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Aafaq, N., Akhtar, N., Liu, W., Shah, M., and Mian, A. Con- trolled caption generation for images through adversarial attacks.arXiv preprint arXiv:2107.03050,
-
[8]
AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimization
Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Boot- strapping language-image pre-training with frozen image encoders and large language models. InICML, pp. 19730– 19742, 2023a. Li, J., Ni, M., Dong, Y ., Zhu, T., and Liu, W. AICAttack: Adversarial image captioning attack with attention-based optimization.arXiv preprint arXiv:2402.11940,
-
[12]
Tan, H. and Bansal, M. LXMERT: Learning cross-modality encoder representations from transformers.arXiv preprint arXiv:1908.07490,
arXiv 1908
-
[13]
optimization method, experimental setups, optimal parameter selection, quantitative and qualitative comparison results of image captioning tasks on the Flick8K dataset, and more visual results on both image captioning and visual question-answering (VQA) tasks. B. GPT Template We employ GPT to accommodate the initial lighting parameters (e.g., start color,...
work page 2025
-
[14]
During the optimization procedure, let Ii denote the generated adversarial image at the ith step
to deal with the optimization problem because it can enhance the diversity of adversarial examples along the optimization path by augmenting image-text pairs. During the optimization procedure, let Ii denote the generated adversarial image at the ith step. SGA conducts a data augmentation by resizingI i into multiple resolutionsM, resulting inI i ={I i1, ...
work page 2014
-
[16]
jointly and locally tunes the exposure and additive perturbations of the image according to a newly designed high-feature-level contrast-sensitive loss function. Adversarial color attacks attempt to change image color to obtain adversarial examples, including SemanticAdv (Hosseini & Poovendran, 2018), ColorFool (Shamsabadi et al., 2020b), and AdvCF (Zhao ...
work page 2018
-
[17]
is a color transformation attack that is optimized with gradient information in the parameter space of a simple color filter. Victim VLP Models.For the image caption task, we employ three typical VLP models to verify their robustness against adversarial attacks, including CLIPCap (Mokady et al., 2021), BLIP (Li et al., 2022), and BLIP2 (Li et al., 2023a) ...
work page 2021
-
[18]
APA measures the percentage of successful prediction answers among all the images
are employed to measure the model’s performance. APA measures the percentage of successful prediction answers among all the images. WUPS measures how much a predicted answer differs from the ground truth based on the difference in their semantic meaning. We employ a no-reference image quality index to assess the naturalness of the generated adversarial im...
work page 2012
Show all 18 references
-
[2005]
TASA: Deceiving question answering mod- els by twin answer sentences attack.arXiv preprint arXiv:2210.15221,
Cao, Y ., Li, D., Fang, M., Zhou, T., Gao, J., Zhan, Y ., and Tao, D. TASA: Deceiving question answering mod- els by twin answer sentences attack.arXiv preprint arXiv:2210.15221,
-
[2012]
Mokady, R., Hertz, A., and Bermano, A. H. Clip- cap: CLIP prefix for image captioning.arXiv preprint arXiv:2111.09734,
-
[2014]
In this study, we randomly choose 1,000 images from the test set of the above datasets as clean images to craft adversarial examples
datasets are employed. In this study, we randomly choose 1,000 images from the test set of the above datasets as clean images to craft adversarial examples. The detailed information is listed as follows: 12 • MSCOCO dataset can be adopted for both image captioning and VQA task...
2023
-
[2017]
Typography leads semantic diversifying: Amplify- ing adversarial transferability across multimodal large lan- guage models.arXiv preprint arXiv:2405.20090,
Cheng, H., Xiao, E., Cao, J., Yang, L., Xu, K., Gu, J., and Xu, R. Typography leads semantic diversifying: Amplify- ing adversarial transferability across multimodal large lan- guage models.arXiv preprint arXiv:2405.20090,
-
[2018]
A survey on transferabil- ity of adversarial examples across deep neural networks
Gu, J., Jia, X., de Jorge, P., Yu, W., Liu, X., Ma, A., Xun, Y ., Hu, A., Khakzar, A., Li, Z., et al. A survey on transferabil- ity of adversarial examples across deep neural networks. arXiv preprint arXiv:2310.17626,
-
[2021]
S., Oh, C., and Cavallaro, A
Shamsabadi, A. S., Oh, C., and Cavallaro, A. EdgeFool: An adversarial image enhancement filter. InICASSP, pp. 1898–1902, 2020a. Shamsabadi, A. S., Sanchez-Matilla, R., and Cavallaro, A. ColorFool: Semantic adversarial colorization. InCVPR, pp. 1151–1160, 2020b. Shamsabadi, A. ...
1902
-
[2022]
OT-Attack: Enhancing adversarial transferability of vision-language models via optimal transport optimiza- tion.arXiv preprint arXiv:2312.04403,
Han, D., Jia, X., Bai, Y ., Gu, J., Liu, Y ., and Cao, X. OT-Attack: Enhancing adversarial transferability of vision-language models via optimal transport optimiza- tion.arXiv preprint arXiv:2312.04403,
-
[2023]
SA- Attack: Improving adversarial transferability of vision- language pre-training models via self-augmentation
He, B., Jia, X., Liang, S., Lou, T., Liu, Y ., and Cao, X. SA- Attack: Improving adversarial transferability of vision- language pre-training models via self-augmentation. arXiv preprint arXiv:2312.04913,
-
[2024]
Adversarial VQA: A new benchmark for evaluating the robustness of VQA models
Li, L., Lei, J., Gan, Z., and Liu, J. Adversarial VQA: A new benchmark for evaluating the robustness of VQA models. InICCV, pp. 2042–2051, 2021b. Li, P., Zhang, Y ., Yuan, L., Zhao, J., Xu, X., and Zhang, X. Adversarial attacks on video object segmentation with hard region dis...
-
[2025]
P., Goodfellow, I., Andersen, D., and Dahl, G
Gilmer, J., Adams, R. P., Goodfellow, I., Andersen, D., and Dahl, G. E. Motivating the rules of the game for adversar- ial example research.arXiv preprint arXiv:1807.06732,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.