REVIEW 4 major objections 5 minor 42 references
Semantics-Guided Generative Image Compression
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A generative image codec improves quality and speed by moving segmentation to the decoder and adapting diffusion settings per image.
desk verdict A solid, incremental extension of MISC with a plausible win on compute and perceptual quality, held back mainly by thin evaluation rather than a flawed idea. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central objects are decoder-side semantic masks produced by ClipSeg, an open-vocabulary segmentation model that localizes objects from text prompts, and a content-adaptive diffusion predictor made of two small MLPs that output a diffusion step count in $[2,80]$ and a classifier-free guidance scale in $(0,10)$. The selective refinement step is $x_t = D_{1\times}(x_{t-1}; T_d[i])\cdot M[i] + x_{t-1}\cdot(1-M[i])$, which applies conditional diffusion only inside each predicted object mask $M[i]$. The MLPs are trained to predict the settings that give the best LPIPS similarity to the original, with a regularization term that penalizes large step counts.
What would settle it
Take a validation set with many small or atypical objects, compress at a range of very low bitrates below 0.06 BPP, and compare the codec's output when ClipSeg masks are computed from the compressed image versus from the original full-quality image: if the perceptual or pixel metrics converge, or the compressed-mask version drops below the MISC baseline, the load-bearing assumption fails.
Extended reading notes
Core claim
The central claim is that removing the transmitted patch-based semantic map from the MISC framework and instead running ClipSeg at the decoder on the degraded low-bitrate reconstruction, together with MLP-predicted diffusion step counts and classifier-free guidance scales, improves reconstruction quality while lowering compute. The decoder uses the segmentation masks to refine each object region selectively, so spatial conditioning costs no extra bitrate. The paper reports consistent gains over MISC across all metric groups and bitrates, with average encoding time falling from 1.19 to 0.76 seconds and decoding time from 1930.98 to 1194.00 seconds on its test setup.
Load-bearing premise
The whole scheme depends on the heavily compressed, low-resolution reconstruction still containing enough visible shape and layout that ClipSeg can find each object named in the text; if the masks are wrong, the diffusion refinement edits the wrong areas.
Editorial extensions
If this is right
- If the central claim is correct, the proposed codec delivers better perceptual similarity and quality than the MISC baseline at the same or lower bitrate, while also running over 36% faster.
- Decoder-side segmentation eliminates the bitrate formerly spent on transmitting patch-based semantic maps, shifting that cost from the encoder to the decoder without hurting reconstruction.
- Content-adaptive diffusion reduces average decoding time because most images need fewer than the fixed 40 diffusion steps that MISC uses.
- The method's per-image diffusion settings lower the chance of hallucination by avoiding unnecessary refinement steps on images that are already well reconstructed.
- The reported gains across pixel-level, perceptual-similarity, and perceptual-quality metrics suggest the two proposed components are complementary rather than redundant.
Reading between the lines
- Editorial inference: because the masks come from the compressed image, the codec's quality ceiling is set by how well the low-bitrate reconstruction preserves object outlines; a direct test would compare outputs when masks are computed from the original image instead of the compressed one.
- Editorial inference: ClipSeg is open-vocabulary, so the same decoder-side masking idea could replace transmitted segmentation maps in other semantics-driven generative codecs beyond MISC.
- Editorial inference: training the MLPs on per-image best LPIPS effectively learns a compute-quality tradeoff; retraining with a rate-aware objective could give explicit control over bitrate, runtime, and quality jointly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two modifications to the multimodal image semantic compression (MISC) framework for ultra-low-bitrate image compression. First, it replaces MISC's encoder-side patch-based semantic maps with decoder-side semantic segmentation using ClipSeg applied to the low-bitrate reconstruction. Second, it introduces content-adaptive diffusion: two MLPs predict the number of diffusion steps and the classifier-free-guidance scale from handcrafted statistical, perceptual, and semantic features of the reconstruction. The authors evaluate on the CLIC2020 professional subset (41 validation images) against seven codecs using three groups of quality metrics, and report that the proposed method improves over MISC in pixel-level and perceptual metrics while reducing encoding and decoding time by more than 36%. They also claim competitive perceptual performance against mainstream codecs at very low bitrates.
Significance. The contribution is a pragmatic, incremental improvement to an existing generative codec, with the practical appeal of simplifying the encoder (no semantic map coding) and reducing decoding cost via adaptive diffusion. The design is not circular: the MLPs are trained against an external LPIPS-based objective on a separate training set, and the evaluation is on held-out data against external baselines. The code is made available, which aids reproducibility. However, the core assumption that decoder-side segmentation on heavily compressed low-bitrate images is accurate enough for gated refinement is not quantitatively verified, and the evaluation methodology has gaps (small test set, no raw values, no bitrate-accounting details). If those issues are resolved, the work would be a solid contribution to practical generative image compression.
major comments (4)
- [Sect. 3.1, Eq. (5)] The decoder-side semantic segmentation is the central replacement for MISC's transmitted semantic maps, yet its accuracy is never quantitatively evaluated. The paper provides only one favorable visual example (Fig. 2), and the ablation in Sec. 4.3 explicitly concedes that at roughly 0.06 BPP the segmentation maps obtained from the low-bitrate reconstruction are not sufficient to improve pixel-level accuracy. Since Eq. (5) uses mask M[i] to gate which regions receive object-conditioned diffusion refinement, incorrect masks would cause diffusion to be applied to the wrong regions and could introduce hallucinated content. Please report quantitative mask accuracy (e.g., mIoU against ground-truth masks, or against the original MISC patch maps) across the operating bitrates, and analyze how mask errors impact the end-to-end reconstruction quality.
- [Sect. 4.1, Figs. 3 and 4] The evaluation is based on only 41 validation images, with per-group metric averages computed after normalizing each metric to [0,1] using IQA-PyTorch ranges. The paper does not report raw per-metric values, error bars, or statistical significance, so the claim that the proposed method 'consistently outperforms' MISC across all bitrates and metric groups is not verifiable. In addition, the bitrate accounting is unspecified: it is not stated whether the approximately 60 words of semantic text from GPT-4 Vision are included in the BPP figures, nor how the removal of MISC's semantic-map bits is reflected. Please provide per-metric tables (or BD-rate equivalents with confidence intervals), specify the exact normalization ranges, and explicitly state which components are counted in the reported BPP.
- [Sect. 4.2, Table 1] The timing comparison is based on the first 20 images of the CLIC2020 validation set on a single GPU (GTX 1060) with no repeated runs or variance reported. Since reducing complexity is one of the two main claimed contributions, this evidence is too thin. Please report mean and standard deviation over the full validation set (or a clearly justified subset) and over multiple runs, and specify exactly which operations (MLP inference, ClipSeg, DiffBIR, LMIC decode, GPT-4 Vision, etc.) are included in the measured encoding and decoding times.
- [Sect. 3.2, Eq. (3)] The training protocol for the two MLPs is under-specified. The paper does not describe how the ground-truth labels y_i (the diffusion-step count and CFG scale that achieve the best LPIPS) are obtained, including the search grid over steps in [2,80] and CFG in (0,10), the number of training samples, or the effect of the regularization coefficient λ=0.64 on the step-count distribution. Without this information, the content-adaptive diffusion component is not reproducible, and the claimed benefit of adaptivity cannot be independently assessed. Please add the full training protocol or cite a prior method that fully specifies it.
minor comments (5)
- [Abstract and Fig. 3] The abstract states that the method enhances quality 'in terms of PSNR and perceptual metrics,' but Fig. 3 shows that several learned and traditional codecs achieve higher pixel-level similarity; please qualify the PSNR claim to clarify that the improvement is relative to the MISC baseline, not to all codecs.
- [Sect. 4.1] The normalization ranges for PSNR, SSIM, MS-SSIM, and the no-reference metrics are not listed. Please specify the exact IQA-PyTorch-defined ranges used, since different normalization choices can change the shape of the averaged curves.
- [Eq. (5)] Clarify whether M[i] is the raw ClipSeg probability map or a binarized mask, and if binarized, what threshold is used; this detail affects the blending operation in Eq. (5).
- [Table 1] The caption of Table 1 does not state whether the times are per-image averages or whether they include the MLP inference, ClipSeg, and the autoencoder decode; please make the measured components explicit.
- [Sec. 4.3] The conclusion that 'our full model offers the best results across all cases' is weakened by the acknowledged tie in pixel-level metrics at about 0.06 BPP; consider adding a sentence explaining how this affects the overall claim.
Circularity Check
No significant circularity: the proposed components are trained and evaluated against external benchmarks on held-out data, and the paper's claims do not reduce to its inputs.
full rationale
The paper's central claims are supported by direct experiments on the CLIC2020 validation set against external codecs. The content-adaptive diffusion MLPs (Sec. 3.2) are trained on the training split with a ground-truth label defined as the diffusion setting that best improves LPIPS, then evaluated on held-out validation images; this is a standard supervised-learning setup and does not force the reported gains by construction. The decoder-side ClipSeg masks used in Eq. (5) are derived from the initial low-bitrate reconstruction, but the derivation does not assume the conclusion: the mask is an input feature, not the target being predicted. The only self-citation in the reference list is [34], used solely as an external evaluation metric (CLIPSim), and it is not load-bearing. The paper's own ablation (Sec. 4.3) notes that at around 0.06 BPP the segmentation maps from the low-bitrate image may be insufficient for pixel-level gains; this is an unvalidated robustness concern rather than circularity. Overall, the derivation chain is self-contained and externally benchmarked, so no circular step is exhibited.
Assumptions & free parameters
free parameters (6)
- MLP weights for step prediction =
unknown (trained)
- MLP weights for CFG scale prediction =
unknown (trained)
- Regularization coefficient lambda for step MLP =
0.64
- Output range for CFG prediction =
0 to 10
- Output range for step count =
2 to 80
- Semantic text length =
about 60 words
assumptions (5)
- domain assumption ClipSeg accurately segments objects from low-bitrate decoded images using textual descriptions.
- domain assumption Pre-trained models (GPT-4V, ClipSeg, DiffBIR, Cheng2020) work as intended in this pipeline.
- domain assumption LPIPS is a valid measure of perceptual similarity for selecting ground-truth diffusion parameters.
- domain assumption The CLIC2020 professional subset is representative of the target use case.
- domain assumption The bitrate calculation accounts for all transmitted information (text and latent).
Cite this review
Pith. "Pith review of Semantics-Guided Generative Image Compression." pith.science (2026). https://pith.science/paper/ME5VXQXQ
@misc{pith2026250524015,
author = {Pith},
title = {Pith review of: Semantics-Guided Generative Image Compression},
year = {2026},
howpublished = {\url{https://pith.science/paper/ME5VXQXQ}},
note = {Machine review of arXiv:2505.24015}
}
read the original abstract
Advancements in text-to-image generative AI with large multimodal models are spreading into the field of image compression, creating high-quality representation of images at extremely low bit rates. This work introduces novel components to the existing multimodal image semantic compression (MISC) approach, enhancing the quality of the generated images in terms of PSNR and perceptual metrics. The new components include semantic segmentation guidance for the generative decoder, as well as content-adaptive diffusion, which controls the number of diffusion steps based on image characteristics. The results show that our newly introduced methods significantly improve the baseline MISC model while also decreasing the complexity. As a result, both the encoding and decoding time are reduced by more than 36%. Moreover, the proposed compression framework outperforms mainstream codecs in terms of perceptual similarity and quality. The code and visual examples are available.
Reference graph
Works this paper leans on
-
[1]
INTRODUCTION In 2023, AI-generated images surged to over 15 billion, surpassing traditional stock photo libraries and constituting about a third of all photos ever uploaded to Instagram in just one year [1]. Models like DALL-E 2 [2], Stable Diffu- sion [3], and Midjourney enable users to create customized, high-quality visuals from simple textual prompts,...
work page 2023
-
[2]
RELA TED WORK In this section, we review related works on generative im- age codecs that can be broadly classified into Generative Adversarial Network (GAN)-based, Variational Autoencoder (V AE)-based, and diffusion-based approaches. Among these, diffusion approaches have gained significant recent interest because of their solid theoretical basis and outs...
arXiv 2025
-
[3]
METHODS End-to-end compression framework with our proposed meth- ods incorporating with MISC is depicted in Fig. 1. The Se- mantic Encoder extracts textual semantics from the input im- age, capturing a description of its most prominent objects and an overall scene description. This is accomplished by query- ing the GPT-4 Vision model [11]. These semantics...
-
[4]
EXPERIMENTS AND ANALYSIS 4.1. Settings Our method is evaluated on the professional subset of the CLIC2020 dataset [26], which includes 585 training images and 41 validation images. For consistency, we froze the parameters of the semantics encoder (GPT-4 Vision [11]), semantics segmentation model (ClipSeg [13]), and diffusion models (DiffBIR [24]) in the d...
-
[5]
Pixel-level similarity metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM) [32], and Multi-Scale SSIM (MS-SSIM) [33], which measure pixel-level fidelity relative to the origi- nal image
-
[6]
Perceptual similarity metrics: CLIPSim [34] and LPIPS [23], which measure closeness to the original image in learned feature spaces
-
[7]
Perceptual quality metrics: ClipIQA [18], MUSIQ [19], and HyperIQA [21], which are blind metrics (without reference to the original image) and simply measure the perceptual quality of the output image on its own. To summarize the performance across these groups of metrics, we normalized each metric to [0,1] using IQA- PyTorch-defined2 ranges and calculate...
work page 1930
-
[8]
CONCLUSIONS We presented a generative image codec built upon the re- cent multimodal image semantic compression (MISC) frame- work, with several important improvements. These include the replacement of coarse patch-based semantic maps with decoder-side semantic maps, as well as content-based adap- tation of diffusion parameters. Extensive experiments were...
Show all 42 references
-
[9]
AI image statistics: How much content was created by AI.,
“AI image statistics: How much content was created by AI.,”https://journal.everypixel.com/ ai-image-statistics, Aug. 2023, Accessed 28 October 2024
2023
-
[10]
Hi- erarchical text-conditional image generation with clip latents,
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hi- erarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, 2022
2022 arXiv
-
[11]
High-resolution image synthesis with latent diffusion models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Om- mer, “High-resolution image synthesis with latent diffusion models,” inCVPR, 2022, pp. 10684–10695
2022
-
[12]
Semantic based generative com- pression of images for extremely low bitrates,
T. Bordin and T. Maugey, “Semantic based generative com- pression of images for extremely low bitrates,” inProc. IEEE MMSP, 2023, pp. 1–6
2023
-
[13]
MISC: Ultra-low bitrate image se- mantic compression driven by large multimodal model,
C. Li, G. Lu, D. Feng, H. Wu, Z. Zhang, X. Liu, G. Zhai, W. Lin, and W. Zhang, “MISC: Ultra-low bitrate image se- mantic compression driven by large multimodal model,”IEEE Trans. Image Processing, vol. 34, pp. 335–349, Jan. 2025
2025
-
[14]
Denoising diffusion probabilis- tic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilis- tic models,” inNeurIPS, 2020, vol. 33, pp. 6840–6851
2020
-
[15]
Efficient diffusion models for vision: A survey,
A. Ulhaq and N. Akhtar, “Efficient diffusion models for vision: A survey,”arXiv preprint arXiv:2210.09292, 2022
2022 arXiv
-
[16]
Sample what you cant compress,
V . Birodkar, G. Barcik, J. Lyon, S. Ioffe, D. Minnen, and J. V . Dillon, “Sample what you cant compress,”arXiv preprint arXiv:2409.02529, 2024
2024
-
[17]
Lossy image compression with condi- tional diffusion models,
R. Yang and S. Mandt, “Lossy image compression with condi- tional diffusion models,” inNeurIPS, 2023, vol. 37
2023
-
[18]
Text+ sketch: Image compression at ultra low rates,
E. Lei, Y . Berkay Uslu, H. Hassani, and S. S. Bidokhti, “Text+ sketch: Image compression at ultra low rates,” inICML 2023 Workshop on Neural Compression: From Information Theory to Applications, 2023
2023
-
[19]
GPT-4 technical report,
J. Achiam et al., “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023
2023 arXiv
-
[20]
Learned image compression with discretized gaussian mixture likelihoods and attention modules,
Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” inCVPR, 2020, pp. 7939–7948
2020
-
[21]
Image segmentation using text and image prompts,
T. L ¨uddecke and A. Ecker, “Image segmentation using text and image prompts,” inCVPR, 2022, pp. 7086–7096
2022
-
[22]
Learning transferable visual models from natural language supervision,
A. Radford et al., “Learning transferable visual models from natural language supervision,” inICML, 2021
2021
-
[23]
R. C. Gonzalez and R. E. Woods,Digital Image Processing, Prentice Hall, 3rd edition, 2008
2008
-
[24]
Blind / reference- less image spatial quality evaluator,
A. Mittal, A. K. Moorthy, and A. C. Bovik, “Blind / reference- less image spatial quality evaluator,” inProc. Asilomar Conf. Signals, Systems and Computers, 2011, pp. 723–727
2011
-
[25]
Making a “completely blind
A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,”IEEE Signal Pro- cessing Letters, vol. 20, no. 3, pp. 209–212, 2012
2012
-
[26]
Exploring clip for assessing the look and feel of images,
J. Wang, K. C. K. Chan, and C. C. Loy, “Exploring clip for assessing the look and feel of images,” inProc. AAAI Conf. Artificial Intelligence, 2023, vol. 37, pp. 2555–2563
2023
-
[27]
MUSIQ: multi-scale image quality transformer,
J. Ke, Q. Wang, Y . Wang, P. Milanfar, and F. Yang, “MUSIQ: multi-scale image quality transformer,” inICCV, 2021, pp. 5148–5157
2021
-
[28]
Blind image quality assessment using a deep bilinear convolutional neural network,
W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang, “Blind image quality assessment using a deep bilinear convolutional neural network,”IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 1, pp. 36–47, 2020
2020
-
[29]
Blindly assess image quality in the wild guided by a self- adaptive hyper network,
S. Su, Q. Yan, Y . Zhu, C. Zhang, X. Ge, J. Sun, and Y . Zhang, “Blindly assess image quality in the wild guided by a self- adaptive hyper network,” inCVPR, 2020, pp. 3667–3676
2020
-
[30]
Blind im- age quality assessment via vision-language correspondence: A multitask learning perspective,
W. Zhang, G. Zhai, Y . Wei, X. Yang, and K. Ma, “Blind im- age quality assessment via vision-language correspondence: A multitask learning perspective,” inCVPR, 2023, pp. 14071– 14081
2023
-
[31]
The unreasonable effectiveness of deep features as a percep- tual metric,
R. Zhang, P. Isola, A. A Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a percep- tual metric,” inCVPR, 2018
2018
-
[32]
DiffBIR: towards blind image restoration with generative diffusion prior,
X. Lin, J. He, Z. Chen, Z. Lyu, B. Dai, F. Yu, W. Ouyang, Y . Qiao, and C. Dong, “DiffBIR: towards blind image restoration with generative diffusion prior,”arXiv preprint arXiv:2308.15070, 2023
2023 arXiv
-
[33]
Adding conditional con- trol to text-to-image diffusion models,
L. Zhang, A. Rao, and M. Agrawala, “Adding conditional con- trol to text-to-image diffusion models,” inICCV, 2023, pp. 3836–3847
2023
-
[34]
Nonlinear trans- form coding,
J. Ball ´e, P. A. Chou, D. Minnen, S. Singh, N. Johnston, E. Agustsson, S. J. Hwang, and G. Toderici, “Nonlinear trans- form coding,”IEEE J. Selected Topics in Signal Processing, vol. 15, no. 2, pp. 339–353, 2020
2020
-
[35]
The JPEG still picture compression standard,
G. K. Wallace, “The JPEG still picture compression standard,” Comm. ACM, vol. 34, no. 4, pp. 30–44, 1991
1991
-
[36]
An image format for the Web,
Google, “An image format for the Web,”https:// developers.google.com/speed/webp, 2022, Ac- cessed 28 October 2024
2022
-
[37]
BPG image format,
F. Bellard, “BPG image format,”https://bellard. org/bpg, 2018
2018
-
[38]
Overview of the versatile video coding (VVC) standard and its applications,
B. Bross, Y .-K. Wang, Y . Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the versatile video coding (VVC) standard and its applications,”IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 10, pp. 3736–3764, 2021
2021
-
[39]
High-fidelity generative image compression,
F. Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson, “High-fidelity generative image compression,” inNeurIPS, 2020, vol. 33, pp. 11913–11924
2020
-
[40]
Im- age quality assessment: from error visibility to structural simi- larity,
Z. Wang, A.C. Bovik, H. R. Sheikh, and E.P. Simoncelli, “Im- age quality assessment: from error visibility to structural simi- larity,”IEEE Trans. Image Processing, vol. 13, no. 4, pp. 600– 612, 2004
2004
-
[41]
Multiscale struc- tural similarity for image quality assessment,
Z. Wang, E.P. Simoncelli, and A.C. Bovik, “Multiscale struc- tural similarity for image quality assessment,” inProc. Asilo- mar Conf. Signals, Systems and Computers, 2003, vol. 2, pp. 1398–1402
2003
-
[42]
GODIV A: generating open-domain videos from natural descriptions,
C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan, “GODIV A: generating open-domain videos from natural descriptions,”arXiv preprint arXiv:2104.14806, 2021. 6
2021 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.