REVIEW 4 major objections 4 minor 20 references
LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read One neural codec jointly trained with a vision-language-guided diffusion restorer can compress for six restoration tasks at once, saving 22.65% BD-rate over state-of-the-art codecs.
desk verdict Novel idea, unfair baseline: the 22.65% BD-rate claim conflates codec performance with jointly fine-tuned restoration, so the headline should be read as unsubstantiated until matched controls are run. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the joint loss $\mathcal{L} = \alpha \mathcal{L}_{\text{dist}} + \beta \mathcal{L}_r + \gamma \mathcal{L}_{\text{task}}$, in which $\mathcal{L}_{\text{dist}}$ measures compressed-image fidelity, $\mathcal{L}_r$ measures bitrate, and $\mathcal{L}_{\text{task}}$ is the diffusion loss that drives the enhanced image toward the ideal image. What makes the framework general is the frozen vision-language extractor that produces feature embeddings from the compressed image; a controller then predicts the degradation type and a caption, and these guide the diffusion restorer. The codec is first trained alone and then fine-tuned jointly with the restoration network, which is the step that lets the encoder allocate bits to what restoration can fix rather than to what a human would want to see.
What would settle it
Retrain or fine-tune the baseline restoration model separately for each baseline codec, keeping the same total training effort as LL-ICM, and then recompute the BD-rate differences; if the savings shrink substantially, the reported 22.65% gain is an artifact of the unfair control.
Extended reading notes
Core claim
The central claim is that rate-perception optimization, rather than classic rate-distortion optimization, is the right objective for compressing images that will later be restored. The paper defines the distortion term as the error between the enhanced image and an ideal high-quality image, not between the compressed and original image. Under this objective the proposed LL-ICM couples the MLIC neural codec with a frozen vision-language feature extractor, a controller that predicts degradation type and caption, and an IR-SDE diffusion restorer, all trained jointly. The authors report that one trained LL-ICM model handles six low-level tasks and achieves 22.65% BD-rate reductions over the compared codecs while also improving LPIPS and Q-Align scores.
Load-bearing premise
The evaluation assumes that applying the same off-the-shelf IR-SDE to all baseline codecs is a fair control, even though LL-ICM's restoration network is fine-tuned jointly with its codec.
Editorial extensions
If this is right
- One trained LL-ICM codec can replace several task-specific codec-plus-restoration pipelines, since the same bitstream supports dehazing, deraining, raindrop removal, deshadowing, denoising, and inpainting.
- If the reported BD-rate savings hold, deploying a joint codec for a restoration-driven pipeline costs about a fifth less bitrate at equal perceptual quality compared with the best compared anchors.
- The rate-perception objective implies that the encoder can discard visually unpleasant but restoration-irrelevant content, which classic rate-distortion training would waste bits preserving.
- The vision-language conditioning makes the framework extensible: new low-level tasks can be added to the same codec as long as the diffusion restorer can use the generalized embedding.
- In some tasks such as denoising, perceptual quality can decrease as bitrate increases under ordinary codecs, a phenomenon the paper attributes to the codec preserving noise; LL-ICM still outperforms the anchors in that regime.
Reading between the lines
- The abnormal denoising curves suggest a testable extension: jointly optimized codecs should learn to allocate fewer bits to high-frequency noise, and measuring the bit allocation directly would show where the savings come from.
- The paper measures perceptual quality with LPIPS and Q-Align, but a human study on the restored outputs would test whether the BD-rate savings translate into subjectively preferred images.
- Because the vision-language embedding is frozen, the framework could in principle be applied to codecs other than MLIC, and the main empirical question is how much of the gain comes from the joint training versus the backbone codec choice.
- A practical deployment would use the no-reference metrics in the loop, since ideal high-quality references are often missing in the field; the paper's use of Q-Align and LIQE points toward this but does not close the loop with an online adaptation scheme.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LL-ICM, an image compression framework for low-level (LL) machine vision tasks. It extends a learned codec (MLIC) with a frozen CLIP feature extractor, a DA-CLIP encoder that predicts degradation type and caption, and an IR-SDE diffusion network that enhances the compressed image. In a two-stage training procedure, the codec is first trained for compression and then jointly fine-tuned with the LL restoration model under a rate-perception objective. The method is evaluated on six LL tasks (deraining, dehazing, raindrop removal, denoising, deshadowing, and inpainting) using LPIPS, Q-Align, and LIQE, with BD-rate, BD-LPIPS, and BD-QAlign metrics compared against Balle2018, Cheng2020, ECM, and MLIC. The paper claims a 22.65% BD-rate reduction over state-of-the-art codecs.
Significance. If the reported results were properly controlled, the paper would fill a genuine gap: most ICM work targets high-level tasks, while real-world images often need low-level enhancement after compression. The rate-perception formulation (Eq. 1) is a sensible extension of RDO, and the idea of using a VLM to condition one universal restorer is timely and practically appealing. The inclusion of both full-reference (LPIPS) and no-reference (Q-Align, LIQE) quality metrics is a strength, and the paper explicitly addresses the rate-perception trade-off rather than only reconstruction fidelity. However, the central quantitative claim is currently not supported because the baseline comparison confounds codec performance with downstream restorer adaptation, and the headline aggregate BD-rate number is undefined. The contribution is therefore promising but requires a re-evaluation under matched conditions before the claims can be accepted.
major comments (4)
- [Experiment (Training Setting / Testing Setting)] The comparison behind Table 2 is not matched on the downstream model. In the Training Setting, the paper states that in the second stage 'the LL task is then trained jointly with the codec and optimize it jointly,' so the IR-SDE used for LL-ICM is fine-tuned on LL-ICM's compressed images. In contrast, the Testing Setting states that for all baseline codecs 'we also use IR-SDE as the LL vision model to generate enhance outputs,' meaning the baselines are paired with an off-the-shelf IR-SDE that has not been adapted to their compression artifacts. The BD-rate, BD-LPIPS, and BD-QAlign values in Table 2 therefore compare a jointly adapted codec+restorer system against a fixed codec+generic restorer system. The headline claim in the Abstract (22.65% BD-rate reduction) and the per-task gains such as -96.07% for inpainting cannot be attributed to the codec's 'encoding ability' or to mutual adaptation per se, because the fine-tuned restorer could absorb a substantial part of the apparent gain. A matched control is required: for each baseline codec, fine-tune the same IR-SDE on its compressed images (or, alternatively, evaluate LL-ICM with the off-the-shelf IR-SDE), and report the comparison under equal restorer adaptation.
- [Abstract and Experiment (Table 2)] The 22.65% BD-rate reduction claimed in the Abstract is not defined anywhere in the experiments. Table 2 reports per-task BD-rate values with Balle2018 as the anchor, and the text also mentions '7.70%-49.23% coding gains across various LL tasks compared to state-of-the-art methods,' but no formula or table entry shows how the 22.65% aggregate is computed or which baseline is considered state-of-the-art for that number. Since this is the central quantitative claim, the paper should specify the averaging procedure (e.g., over tasks, over rate points, versus MLIC) and provide the corresponding aggregate table entry.
- [Experiment (Table 1 and Testing Setting)] The experimental setup for inpainting is underspecified. Table 1 lists 'inpainting' with sizes '29901+100' and references RePaint and CelebaHQ-256, but the text does not describe how the degraded input for inpainting is generated (e.g., mask type, mask ratio, or whether the model sees the mask). This matters because the largest claimed gain (-96.07% BD-rate in Table 2) is for inpainting; without the exact task protocol the result cannot be reproduced or interpreted. The formatting of Table 1 also makes dataset-task assignments ambiguous (e.g., the rows for denoising and inpainting are garbled).
- [Qualitative Comparison (Fig. 6)] The LPIPS values reported in Fig. 6 appear to contradict the qualitative claim. For the deraining example, MLIC (LPIPS 0.13) and ECM (LPIPS 0.09) achieve lower (better) LPIPS than LL-ICM (LPIPS 0.19), and for raindrop removal LL-ICM (LPIPS 0.30) is worse than Cheng2020 (LPIPS 0.07) and ECM (LPIPS 0.15). If these LPIPS values are computed on the enhanced images that are shown, the figure undermines the statement that 'our method preserves more intricate details and texture'; if they are computed elsewhere, the caption should say so. The paper should reconcile this inconsistency or choose qualitative examples consistent with the quantitative results.
minor comments (4)
- [Rate-perception Performance Result] The section contains a duplicated paragraph: the passage beginning 'Not all LL tasks yield consistent results' appears twice with different subfigure references (one mentions Fig. 4(g)/(h), the other Fig. 4(e)/(f)), while the Fig. 4 caption only lists subfigures (a)-(f). The text should be merged and the figure references made consistent.
- [Throughout] There are several typos and inconsistencies: 'Exsiting' should be 'Existing', 'LL-LCM' appears where 'LL-ICM' is meant, 'debluring' should be 'deblurring', and the codec name is written both as 'Balle2018' and 'Balle'. These should be corrected.
- [Experiment (Training Setting)] Hyperparameters for the second-stage joint training are not reported: the learning rate, the loss weights α, β, and γ in Eq. (5) for stage 2 are not given, and only stage-1 β values for Lr are listed. This information is necessary for reproducibility.
- [Reproducibility] No code, trained models, or evaluation scripts are released. Given the reliance on multiple pretrained components (MLIC, CLIP, DA-CLIP, IR-SDE) and the complex two-stage training, releasing the implementation would be important for reproducibility and for verifying the claimed gains.
Circularity Check
No circularity: LL-ICM's gains come from empirical evaluation; the equations are definitions and the external components are cited, not self-referential.
full rationale
The derivation chain in LL-ICM is not circular. The rate-perception objective in Eq. 1 is a problem definition, Eqs. 2-4 describe the architecture, and Eq. 5 is a weighted training loss; none of these define the reported 22.65% BD-rate reduction in terms of itself. The backbone codec MLIC, the VLM extractor, the DA-CLIP encoder, and the IR-SDE diffusion model are all cited from independent groups, and the paper does not invoke any self-derived uniqueness theorem or ansatz that smuggles the conclusion in via a citation. The central claim is a benchmark measurement, not a quantity fitted and then renamed as a prediction. One legitimate methodological concern appears in the testing setting: the authors state that 'to ensure the fairness of the comparison, we also use IR-SDE as the LL vision model to generate enhance outputs of the compressed images from these codecs,' while the training setting says the LL task is 'trained jointly with the codec and optimize it jointly.' This means the baseline restorer is off-the-shelf while LL-ICM's restorer is fine-tuned, so the comparison may be confounded. However, a fairness confound is not circularity: the reported rate savings are not algebraically forced by the loss definition, and no fitted parameter is presented as an independent prediction. Under the strict circularity criteria requiring a specific reduction via equations or self-citation, no such step is present.
Assumptions & free parameters
free parameters (5)
- Rate-distortion loss weight beta =
{0.0002, 0.0008, 0.0018, 0.0035, 0.0130, 0.0350}
- Task loss weight gamma =
1 in stage 2, 0 in stage 1
- Distortion loss weight alpha =
1
- Training patch size =
448x448
- Gaussian noise level for denoising training data =
50
assumptions (6)
- domain assumption A pre-trained CLIP model gives generalized feature embeddings that remain useful for low-level vision tasks when computed from compressed images.
- ad hoc to paper The DA-CLIP encoder can predict degradation type and caption from the compressed image, and this conditioning is sufficient to guide one diffusion network for all six tasks.
- domain assumption For every training image there exists a high-quality ground truth Xideal, and the MSE and diffusion losses against Xideal are correct training signals.
- domain assumption LPIPS and Q-Align/LIQE are valid proxies for perceptual quality of restored images, and BD-rate computed on these metrics is a fair comparison across codecs.
- domain assumption The reported training and test splits are disjoint, so test results are not leaked during joint training.
- ad hoc to paper The scalar-weighted loss L = alpha*Ldist + beta*Lr + gamma*Ltask is sufficient to reach the rate-perception optimum.
Cite this review
Pith. "Pith review of LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model." pith.science (2026). https://pith.science/paper/MWVMLGQH
@misc{pith2026241203841,
author = {Pith},
title = {Pith review of: LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/MWVMLGQH}},
note = {Machine review of arXiv:2412.03841}
}
read the original abstract
Image Compression for Machines (ICM) aims to compress images for machine vision tasks rather than human viewing. Current works predominantly concentrate on high-level tasks like object detection and semantic segmentation. However, the quality of original images is usually not guaranteed in the real world, leading to even worse perceptual quality or downstream task performance after compression. Low-level (LL) machine vision models, like image restoration models, can help improve such quality, and thereby their compression requirements should also be considered. In this paper, we propose a pioneered ICM framework for LL machine vision tasks, namely LL-ICM. By jointly optimizing compression and LL tasks, the proposed LL-ICM not only enriches its encoding ability in generalizing to versatile LL tasks but also optimizes the processing ability of down-stream LL task models, achieving mutual adaptation for image codecs and LL task models. Furthermore, we integrate large-scale vision-language models into the LL-ICM framework to generate more universal and distortion-robust feature embeddings for LL vision tasks. Therefore, one LL-ICM codec can generalize to multiple tasks. We establish a solid benchmark to evaluate LL-ICM, which includes extensive objective experiments by using both full and no-reference image quality assessments. Experimental results show that LL-ICM can achieve 22.65% BD-rate reductions over the state-of-the-art methods.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Yi-Hsin Chen, Ying-Chieh Weng, Chia-Hao Kao, Cheng Chien, Wei-Chen Chiu, and Wen- Hsiao Peng, “Transtic: Transferring transformer-based image compression from human per- ception to machine perception,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 23297–23307
work page 2023
-
[2]
Residual based hierarchical feature compression for multi-task machine vision,
Chaoran Chen, Mai Xu, Shengxi Li, Tie Liu, Minglang Qiao, and Zhuoyi Lv, “Residual based hierarchical feature compression for multi-task machine vision,” in 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023, pp. 1463–1468
work page 2023
-
[3]
Mlic: Multi-reference entropy model for learned image compression,
Wei Jiang, Jiayu Yang, Yongqi Zhai, Peirong Ning, Feng Gao, and Ronggang Wang, “Mlic: Multi-reference entropy model for learned image compression,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 7618–7627
2023
-
[4]
Learning transferable visual models from natural language supervision,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763
work page 2021
-
[5]
Controlling vision-language models for universal image restoration,
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sj ¨olund, and Thomas B Sch ¨on, “Controlling vision-language models for universal image restoration,” arXiv preprint arXiv:2310.01018, 2023
arXiv 2023
-
[6]
Image restoration with mean-reverting stochastic differential equations,
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sj ¨olund, and Thomas B Sch ¨on, “Image restoration with mean-reverting stochastic differential equations,” arXiv preprint arXiv:2301.11699, 2023
arXiv 2023
-
[7]
Deep retinex decomposition for low-light enhancement,
Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu, “Deep retinex decomposition for low-light enhancement,” arXiv preprint arXiv:1808.04560, 2018
arXiv 2018
-
[8]
Ffa-net: Feature fusion attention network for single image dehazing,
Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia, “Ffa-net: Feature fusion attention network for single image dehazing,” in Proceedings of the AAAI conference on artificial intelligence, 2020, vol. 34, pp. 11908–11915
work page 2020
Show all 20 references
-
[9]
Deep joint rain detection and removal from a single image,
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan, “Deep joint rain detection and removal from a single image,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 1357–1366
2017
-
[10]
Repaint: Inpainting using denoising diffusion probabilistic models,
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 11461–11471
2022
-
[11]
A database of human seg- mented natural images and its application to evaluating segmentation algorithms and measur- ing ecological statistics,
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik, “A database of human seg- mented natural images and its application to evaluating segmentation algorithms and measur- ing ecological statistics,” in Proceedings eighth IEEE International Conference on Computer Visi...
2001
-
[12]
De- shadownet: A multi-context embedding deep network for shadow removal,
Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson WH Lau, “De- shadownet: A multi-context embedding deep network for shadow removal,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4067–4075
2017
-
[13]
Attentive generative adversarial network for raindrop removal from a single image,
Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu, “Attentive generative adversarial network for raindrop removal from a single image,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018, pp. 2482–2491
2018
-
[14]
Video compression beyond vvc: Quantitative analysis of intra coding tools in enhanced com- pression model (ecm),
Mohsen Abdoli, Ramin G Youvalari, Karam Naser, Kevin Reuz ´e, and Fabrice Le L ´eannec, “Video compression beyond vvc: Quantitative analysis of intra coding tools in enhanced com- pression model (ecm),” arXiv preprint arXiv:2404.07872, 2024
2024 arXiv
-
[15]
Varia- tional image compression with a scale hyperprior,
Johannes Ball ´e, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston, “Varia- tional image compression with a scale hyperprior,” arXiv preprint arXiv:1802.01436, 2018
2018 arXiv
-
[16]
Learned image compres- sion with discretized gaussian mixture likelihoods and attention modules,
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto, “Learned image compres- sion with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2020, pp. 7939–7948
2020
-
[17]
The unrea- sonable effectiveness of deep features as a perceptual metric,
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang, “The unrea- sonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on Computer Vision and Oattern Recognition, 2018, pp. 586–595
2018
-
[18]
Q-align: Teaching lmms for visual scoring via discrete text-defined levels,
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al., “Q-align: Teaching lmms for visual scoring via discrete text-defined levels,” arXiv preprint arXiv:2312.17090, 2023
2023 arXiv
-
[19]
Blind image quality assessment via vision-language correspondence: A multitask learning perspective,
Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma, “Blind image quality assessment via vision-language correspondence: A multitask learning perspective,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14071–14081
2023
-
[20]
Calculation of average psnr differences between rd-curves,
Gisle Bjontegaard, “Calculation of average psnr differences between rd-curves,” ITU SG16 Doc. VCEG-M33, 2001
2001
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.