REVIEW 4 major objections 5 minor 40 references
A camera's color-adjustment parameters can be tuned directly by text prompts, using a pretrained vision-language model as the objective and gradient descent on a 3x3 color matrix.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 17:34 UTC pith:4WTRY5SP
load-bearing objection A simple, honest proof-of-concept for language-driven ISP color tuning; the idea is new as an application, and the paper is candid about its limits, but the quantitative evaluation is thin and partly circular. the 4 major comments →
Language-based Color ISP Tuning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that ISP parameters for color enhancement can be tuned by an objective that maximizes the cosine similarity between the CLIP embedding of the processed image and that of a text prompt. The paper implements the ISP block as a 3x3 color matrix with white-point-preserving constraints, which is differentiable, and runs gradient descent directly on the matrix entries. The result is a per-image style adjustment matching the prompt: 'warm' gives redder tones, 'cool' gives bluer tones, and cultural prompts like 'Matrix movie' produce neon greens and purples. A two-prompt variant additionally interpolates between two prompts to provide fine-grained control over the degree of styl
What carries the argument
The key machinery is a constrained, differentiable linear color transform (a 3x3 matrix whose rows sum to one) paired with a frozen vision-language model (CLIP). The objective is a cosine similarity between image and text embeddings; its gradient flows through the CLIP image encoder into the matrix parameters. A two-prompt variant replaces the single similarity with a softmax ratio between two prompt similarities, allowing interpolation. All optimization is done in the parameter space of the ISP block, so no image-processing network is trained or inferred.
Load-bearing premise
The method assumes that the vision-language model's image-text similarity is a reliable, gradient-friendly proxy for how well a human judges the style match, and that CLIP's gradients push the image in style-relevant directions.
What would settle it
A controlled human study: for a set of prompts (e.g., 'happy', 'sad', 'warm'), optimize ISP parameters on several images, then have participants rate whether each output matches the prompt's style. If CLIP similarity rises while human style-match ratings are near chance, the central proxy assumption is falsified.
If this is right
- If the method works as claimed, a user can apply a described color style directly to the ISP output with no network training and no resolution constraints or hallucination artifacts typical of neural image editors.
- The paper reports that the choice of optimizer has little effect, and that a larger vision-language model performed worse than a smaller one, suggesting gradient or representational issues with scale.
- The two-prompt objective (interpolating between, e.g., 'vibrant' and 'dull') gives continuous control over the strength of the style, which addresses the ambiguity of single-word prompts.
- Tuning is per-image: each optimization produces a matrix for that specific image, and the paper notes that global tuning across images or extensions like differentiable LUTs remain for future work.
- The method's expressiveness is bounded by the linear matrix; only global color transformations are achievable, which keeps the output natural for mild parameter clipping but limits the range of styles.
Where Pith is reading between the lines
- One natural extension is to convert a per-image tuned matrix into a reusable camera preset, so a single prompt could define a color profile applied to any image.
- The reliance on CLIP similarity could be stress-tested by adversarial examples: if optimizing for 'happy' can change semantic content (e.g., adding a smile) while keeping CLIP similarity high, the method's style-specificity would be questionable.
- Because only nine parameters are optimized, the approach could run in real time on a camera, making language-based color grading a practical consumer feature.
- The paper's observed weakness on abstract emotions points to a direct improvement: fine-tuning the vision-language model on style-labeled data could shift its representation from semantics toward aesthetics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a training-free method for tuning a color-adjustment ISP block (a 3x3 color matrix) using natural-language prompts. A differentiable linear color transform is optimized by gradient descent to maximize the CLIP cosine similarity between the processed image and a text prompt (Eq. 4). A two-prompt variant interpolates between two style descriptions (Eq. 5). Experiments on Kodak images show qualitative style changes for color, cultural, and emotional prompts, compare several CLIP models and optimizers, and report quantitative differences between 'vibrant' and 'dull' tuned images using CLIP-IQA and a colorfulness metric. The paper claims to be the first work on language-based ISP tuning.
Significance. If the central assumption holds—that CLIP gradients move in style-relevant directions rather than semantic or shortcut directions—the method provides a simple, practical way to control ISP color grading with text, with no neural-network image artifacts and no model training. The contribution is well scoped: a differentiable ISP block plus a CLIP objective is a clean and reproducible recipe. The quantitative evidence, however, is narrow and partly circular, and the paper's own limitations section concedes that the VLM was trained mostly on semantic content. The idea is promising and likely worth publishing after stronger validation of the style-matching claim.
major comments (4)
- [Sec. 2, Eq. (2)] The row-sum constraint [1,1,1]^T M = [1,1,1]^T is stated but no enforcement mechanism is described. Gradient descent with only clipping (|phi_ij| <= tau) does not preserve this linear equality. If the constraint is instead enforced by projection or by reparameterizing (e.g., optimizing only off-diagonal entries and setting each diagonal entry to 1 minus the sum of the other row entries), that must be specified. As written, the method may not conserve the white point, contradicting the claim and affecting color correctness.
- [Sec. 3.3, Table 1] The only quantitative experiment uses a single prompt pair ('vibrant' vs. 'dull'), and the primary metric, delta CLIP-IQA, is a CLIP-based quality score. Since the optimization objective (Eq. 4) is also CLIP similarity, this evaluation is at least partly circular and does not independently establish that the tuned color change matches the prompt semantics. The independent delta C metric shows a color change but not that the change corresponds to the intended style. No error bars or per-image statistics are reported, despite 24 images being available. A human study or a non-CLIP perceptual metric on several prompt pairs is needed to support the broad claim.
- [Sec. 3.1] The evidence for style matching across prompt categories is qualitative. The paper itself reports that abstract/emotional prompts 'did not work as well' and that some rows showed little stylistic change. This is in tension with the conclusion that a user can 'simply descri[b]e the desired style via a language prompt.' The qualitative figures are illustrative but not sufficient to establish the general claim. At minimum, the paper should restrict its claim to prompt types that work, or provide quantitative/behavioral evidence that the optimized color matrices are prompt-specific and consistent across images.
- [Sec. 4] The limitation statement concedes that the VLM 'was trained on images and text that largely focused on the semantic content of the image, rather than stylistic elements.' This directly affects the load-bearing assumption in Sec. 2.1 that CLIP 'correctly interpret[s] the visual style aspects' of both image and text. The paper does not test whether a high CLIP similarity (Eq. 4) corresponds to a human-perceived style match, as opposed to a low-level color or shortcut direction. A concrete diagnostic would be to compare optimized matrices for prompts with shared semantic content but opposite style (e.g., 'vibrant photo of a dog' vs. 'dull photo of a dog') and to collect human ratings.
minor comments (5)
- [Sec. 2, Eq. (5)] The softmax expression is hard to parse. Please define s_XAphi and s_XBphi explicitly and write the objective as a squared loss between the softmax probability vector and the target [alpha, 1-alpha] (or similar).
- [Sec. 2, Eq. (1)] The notation M_phi X is ambiguous for a 3xHxW tensor; clarify that the matrix multiplies the channel dimension of each pixel.
- [Sec. 3.3.A and Table 2] The caption says 'Experiment 3.2' but the section is 3.3.A; correct the cross-reference.
- [Sec. 3.3.D] The statement that tau >= 0.5 gives 'unnatural looking' images is subjective; a quantitative or perceptual evaluation would strengthen the choice of tau=0.25.
- [Sec. 2.3] The comparison with MGIE is qualitative and the MGIE result is center-cropped; state this clearly in the main text and consider a more controlled comparison at equal resolution.
Circularity Check
Evaluation partly uses CLIP-IQA, a CLIP-based metric, making one quantitative validation partially circular; the independent Hasler colorfulness metric prevents full circularity.
specific steps
-
other
[Section 2.1 Eq. (4); Section 3.3 Table 1]
"φ∗ = argmax φ S(fimage(Mφ X), ftext(t)) ... Then, we evaluated the vibrant-dull image pairs by computing the 1) difference in “Colorfulness“ CLIP-IQA score [29], and 2) the difference in colorfulness metric [40]."
The optimization objective (Eq. 4) tunes ISP parameters to maximize CLIP image-text cosine similarity. CLIP-IQA [29] is a CLIP-based metric, so the ΔCLIP-IQA column is not an independent measurement of style: it is computed from the same CLIP embedding space that the objective directly optimizes. Improving CLIP similarity for 'vibrant' vs. 'dull' is therefore expected to move CLIP-IQA colorfulness in the same direction by construction, not because human-perceived style has been validated. The paper's Hasler ΔC metric is independent and non-CLIP, which grounds the result partially, so the circularity is limited to this one evaluation column.
full rationale
The paper's core method is a direct optimization: a differentiable 3x3 ISP matrix is tuned to maximize CLIP cosine similarity between the processed image and a text prompt (Eq. 4). This is not a derivation from first principles that reduces to its inputs; it is an empirical optimization procedure. There is no load-bearing self-citation: references [27]-[29] are external CLIP/OpenCLIP works, not by the current authors, and no uniqueness theorem is imported from prior work. The only notable circularity is in the quantitative validation: one of the two metrics, ΔCLIP-IQA, is itself CLIP-based and therefore shares the model family used in the objective, making that portion of the evidence partially self-confirming. The other metric, Hasler colorfulness ΔC, is independent and non-CLIP, and the qualitative results (Figs. 5-8) provide additional, albeit subjective, grounding. Thus the central claim that CLIP-style gradients can tune ISP color is not entirely forced by construction, but the CLIP-IQA-based evaluation inflates the apparent strength of the evidence. Score 4 reflects partial circularity in evaluation while acknowledging the independent content.
Axiom & Free-Parameter Ledger
free parameters (3)
- CLIP coefficient clipping level tau =
0.25 default, varied 0.33, 0.5, 1.0
- Optimizer learning rate =
2e-3 for Adam; per-optimizer values chosen empirically
- Prompt template =
Template B: 'A {prompt} photo'
axioms (3)
- domain assumption A 3x3 color transformation matrix with rows summing to one is expressive enough to represent the language-described color styles targeted here.
- domain assumption CLIP image-text cosine similarity is a reliable, gradient-accessible proxy for perceptual style match.
- domain assumption White-point preservation is a safe constraint for style tuning.
read the original abstract
We propose a method for tuning the parameters of a color adjustment Image Signal Processor (ISP) algorithmic "block" using language prompts. This enables the user to impart a particular visual style to the ISP-processed image simply by describing it through a text prompt. To do this, we first implement the ISP block in a differentiable manner. Then, we define an objective function using an off-the-shelf, pretrained vision-language model (VLM) such that the objective is minimized when the ISP processed image is most visually similar to the input language prompt. Finally, we optimize the ISP parameters using gradient descent. Experimental results demonstrate tuning of ISP parameters with different language prompts, and compare the performance of different pretrained VLMs and optimization strategies.
Figures
Reference graph
Works this paper leans on
-
[1]
Color image processing pipeline,
R. Ramanath, W. E. Snyder, Y . Yoo, and M. S. Drew, “Color image processing pipeline,”IEEE Signal Processing Magazine, vol. 22, no. 1, pp. 34–43, 2005
2005
-
[2]
A software platform for ma- nipulating the camera imaging pipeline,
H. C. Karaimer and M. S. Brown, “A software platform for ma- nipulating the camera imaging pipeline,” inECCV Proceedings. Springer, 2016, pp. 429–444
2016
-
[3]
Replacing mobile cam- era ISP with a single deep learning model,
A. Ignatov, L. Van Gool, and R. Timofte, “Replacing mobile cam- era ISP with a single deep learning model,” inIEEE/CVF Computer Vision and Pattern Recognition Workshops, 2020
2020
-
[4]
ISP meets deep learning: A survey on deep learning methods for image signal processing,
C. F. G. d. Santoset al., “ISP meets deep learning: A survey on deep learning methods for image signal processing,”ACM Com- puting Surveys, 2025
2025
-
[5]
Nilut: Conditional neural implicit 3d lookup tables for image en- hancement,
M. V . Conde, J. Vazquez-Corral, M. S. Brown, and R. Timofte, “Nilut: Conditional neural implicit 3d lookup tables for image en- hancement,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 2, 2024, pp. 1371–1379
2024
-
[6]
PQDynamicISP: dynam- ically controlled image signal processor for any image sensors pur- suing perceptual quality,
M. Yoshimura, J. Otsuka, and T. Ohashi, “PQDynamicISP: dynam- ically controlled image signal processor for any image sensors pur- suing perceptual quality,”CoRR, 2024
2024
-
[7]
Automatic ISP image quality tuning using nonlinear optimization,
J. Nishimura, T. Gerasimow, R. Sushma, A. Sutic, C.-T. Wu, and G. Michael, “Automatic ISP image quality tuning using nonlinear optimization,” inInternational Conference on Image Processing (ICIP). IEEE, 2018, pp. 2471–2475
2018
-
[8]
Hardware-in-the-loop end-to-end optimization of cam- era image processing pipelines,
A. Mosleh, A. Sharma, E. Onzon, F. Mannan, N. Robidoux, and F. Heide, “Hardware-in-the-loop end-to-end optimization of cam- era image processing pipelines,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020
2020
-
[9]
Refactoring ISP for high-level vi- sion tasks,
Y . Shi, S. Li, X. Jia, and J. Liu, “Refactoring ISP for high-level vi- sion tasks,” inInternational Conference on Robotics and Automa- tion (ICRA). IEEE, 2022, pp. 2366–2372
2022
-
[10]
What is the cost of adding a constraint in linear least squares?
R. Kakarala and J. Wei, “What is the cost of adding a constraint in linear least squares?”arXiv preprint arXiv:2201.09935, 2022
Pith/arXiv arXiv 2022
-
[11]
Color cor- rection pipeline optimization for digital cameras,
S. Bianco, A. R. Bruna, F. Naccari, and R. Schettini, “Color cor- rection pipeline optimization for digital cameras,”Journal of Elec- tronic Imaging, vol. 22, no. 2, pp. 023 014–023 014, 2013
2013
-
[12]
DNN- based ISP parameter inference algorithm for automatic image qual- ity optimization,
Y . Kim, J. Lee, S.-S. Kim, C. Yang, T. Kim, and J. Yim, “DNN- based ISP parameter inference algorithm for automatic image qual- ity optimization,”Electronic Imaging, vol. 32, pp. 1–6, 2020
2020
-
[13]
Hyperparame- ter optimization in black-box image processing using differentiable proxies
E. Tseng, F. Yu, Y . Yang, F. Mannan, K. S. Arnaud, D. Nowrouzezahrai, J.-F. Lalonde, and F. Heide, “Hyperparame- ter optimization in black-box image processing using differentiable proxies.”ACM Transactions on Graphics, 2019
2019
-
[14]
Ef- fective ISP tuning framework based on user preference feedback,
C. Yang, J. Kim, J. Lee, Y . Kim, S.-S. Kim, T. Kim, and J. Yim, “Ef- fective ISP tuning framework based on user preference feedback,” Electronic Imaging, vol. 32, pp. 1–5, 2020
2020
-
[15]
Neural Photo-Finishing
E. Tseng, Y . Zhang, L. Jebe, X. Zhang, Z. Xia, Y . Fan, F. Heide, and J. Chen, “Neural Photo-Finishing.”ACM Transactions on Graph- ics, vol. 41, no. 6, pp. 238–1, 2022
2022
-
[16]
Learning controllable ISP for image enhancement,
H. Kim and K. M. Lee, “Learning controllable ISP for image enhancement,”IEEE Transactions on Image Processing, vol. 33, 2023
2023
-
[17]
AdaptiveISP: Learning an adaptive image signal processor for object detection,
Y . Wang, T. Xu, Z. Fan, T. Xue, and J. Gu, “AdaptiveISP: Learning an adaptive image signal processor for object detection,”Advances in Neural Information Processing Systems, vol. 37, 2024
2024
-
[18]
DynamicISP: dynamically controlled image signal processor for image recogni- tion,
M. Yoshimura, J. Otsuka, A. Irie, and T. Ohashi, “DynamicISP: dynamically controlled image signal processor for image recogni- tion,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12 866–12 876
2023
-
[19]
CCMNet: Leveraging calibrated color correction matrices for cross-camera color constancy,
D. Kim, M. Afifi, D. Kim, M. S. Brown, and S. J. Kim, “CCMNet: Leveraging calibrated color correction matrices for cross-camera color constancy,”arXiv preprint arXiv:2504.07959, 2025
arXiv 2025
-
[20]
Color correction for image-based modeling in the large,
T. Shen, J. Wang, T. Fang, S. Zhu, and L. Quan, “Color correction for image-based modeling in the large,” inACCV: Asian Confer- ence on Computer Vision. Springer, 2017, pp. 392–407
2017
-
[21]
Example-based video color grading
N. Bonneel, K. Sunkavalli, S. Paris, and H. Pfister, “Example-based video color grading.”ACM Transactions on Graphics, vol. 32, 2013
2013
-
[22]
Neural preset for color style transfer,
Z. Ke, Y . Liu, L. Zhu, N. Zhao, and R. W. Lau, “Neural preset for color style transfer,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14 173– 14 182
2023
-
[23]
Van Hurkman,Color Correction Look Book: Creative Grading Techniques for Film and Video
A. Van Hurkman,Color Correction Look Book: Creative Grading Techniques for Film and Video. Pearson Education, 2014
2014
-
[24]
Tunable color correction for noisy images,
R. Yamakabe, Y . Monno, M. Tanaka, and M. Okutomi, “Tunable color correction for noisy images,”Journal of Electronic Imaging, vol. 29, no. 3, pp. 033 012–033 012, 2020
2020
-
[25]
Learning pho- tographic global tonal adjustment with a database of input/output image pairs,
V . Bychkovsky, S. Paris, E. Chan, and F. Durand, “Learning pho- tographic global tonal adjustment with a database of input/output image pairs,” inProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. IEEE, 2011, pp. 97–104
2011
-
[26]
Color transfer with modulated flows,
M. Larchenko, A. Lobashev, D. Guskov, and V . V . Palyulin, “Color transfer with modulated flows,” inProceedings of the AAAI Confer- ence on Artificial Intelligence, vol. 39, no. 4, 2025, pp. 4464–4472
2025
-
[27]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning. PmLR, 2021
2021
-
[28]
G. Ilharco, M. Wortsman, R. Wightman, C. Gor- donet al., “OpenCLIP,” 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5143773
-
[29]
Exploring CLIP for assessing the look and feel of images,
J. Wang, K. C. Chan, and C. C. Loy, “Exploring CLIP for assessing the look and feel of images,” inAAAI, 2023
2023
-
[30]
Clip-gen: Language- free training of a text-to-image generator with clip,
Z. Wang, W. Liu, Q. He, X. Wu, and Z. Yi, “Clip-gen: Language- free training of a text-to-image generator with clip,”arXiv preprint arXiv:2203.00386, 2022
Pith/arXiv arXiv 2022
-
[31]
Guiding Instruction-based Image Editing via Multimodal Large Language Models,
T.-J. Fu, W. Hu, X. Du, W. Y . Wang, Y . Yang, and Z. Gan, “Guiding Instruction-based Image Editing via Multimodal Large Language Models,” inInt. Conference on Learning Representations, 2024
2024
-
[32]
Instruction-guided editing controls for images and multimedia: A survey in LLM era,
T. T. Nguyen, Z. Ren, T. Pham, T. T. Huynh, P. L. Nguyen, H. Yin, and Q. V . H. Nguyen, “Instruction-guided editing controls for images and multimedia: A survey in LLM era,”arXiv preprint arXiv:2411.09955, 2024
Pith/arXiv arXiv 2024
-
[33]
S. S. Chai, W. Peng, B. Hariharan, and H. Averbuch-Elor, “Not every gift comes in gold paper or with a red ribbon: Exploring color perception in text-to-image models,” 2025. [Online]. Available: https://arxiv.org/abs/2508.19791
arXiv 2025
-
[34]
Coco-lc: Colorfulness control- lable language-based colorization,
Y . Li, Y . Bai, S. Yang, and J. Liu, “Coco-lc: Colorfulness control- lable language-based colorization,” inACM MM, 2024
2024
-
[35]
A learnable color correction matrix for RAW reconstruction,
A. Liu, S. Mu, and S. Xu, “A learnable color correction matrix for RAW reconstruction,”arXiv preprint arXiv:2409.02497, 2024
Pith/arXiv arXiv 2024
-
[36]
Embedding geometries of contrastive language-image pre-training,
J. C.-C. Chou and N. Alam, “Embedding geometries of contrastive language-image pre-training,”preprint arXiv:2409.13079, 2024
Pith/arXiv arXiv 2024
-
[37]
LAION-5b: An open large-scale dataset for training next generation image-text models,
C. Schuhmann, R. Beaumont, R. Vencuet al., “LAION-5b: An open large-scale dataset for training next generation image-text models,” inConference on Neural Information Processing Systems, 2022
2022
-
[38]
PyTorch: An imperative style, high-performance deep learning library,
A. Paszke, S. Gross, F. Massaet al., “PyTorch: An imperative style, high-performance deep learning library,” CoRR, vol. abs/1912.01703, 2019. [Online]. Available: http://arxiv.org/abs/1912.01703
Pith/arXiv arXiv 1912
-
[39]
Color in visual-language models: CLIP deficiencies,
G. Arias, R. Baldrich, and M. Vanrell, “Color in visual-language models: CLIP deficiencies,”preprint arXiv:2502.04470, 2025
Pith/arXiv arXiv 2025
-
[40]
Measuring colorfulness in natural images,
D. Hasler and S. E. Suesstrunk, “Measuring colorfulness in natural images,” inHuman Vision and Electronic Imaging VIII, vol. 5007. SPIE, 2003, pp. 87–95
2003
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.