{"id":"05b8a484-0022-4eb4-904e-09d0e0fab773","arxiv_id":"2505.24015","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Decoder-side ClipSeg segmentation and content-adaptive diffusion steps improve MISC-based ultra-low-bitrate generative image compression in perceptual quality and speed.","lead":"A semantic image codec now moves object-location guidance into the decoder: it derives masks from a blurry compressed image using ClipSeg and predicts per-image diffusion effort with a small neural network. The method reports better perceptual quality than the MISC baseline and reduces encoding and decoding time by over 36%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Decoder-side segmentation quality is the load-bearing assumption: masks from ClipSeg on heavily compressed low-bitrate images are never quantitatively validated, and the paper's own ablation shows the pixel-level benefit disappears at 0.06 BPP.","rationale":"The reader's weakest_assumption pinpoints exactly the condition that must hold for the central claim: the low-bitrate reconstruction must retain enough spatial structure for ClipSeg to produce accurate object masks. This is load-bearing because the entire architectural change from MISC is the removal of coded semantic maps and their replacement with decoder-side segmentation, which then gates the selective diffusion refinement in Eq. (5). The paper provides no quantitative evidence that ClipSeg masks are accurate on compressed, downsampled inputs; the only direct evidence is a single illustrative figure, and the ablation in Sec. 4.3 explicitly notes that at 0.06 BPP the segmentation maps are not sufficient to improve pixel-level accuracy. That is an acknowledged degradation at exactly the low-bitrate regime the paper targets. A direct oracle-mask test would settle whether this degradation is severe enough to undermine the central claim. The reader's conditional verdict is appropriate: the idea is plausible and the ablation generally supports the components, but the key assumption is not yet quantitatively verified. Secondary concerns about bitrate accounting and timing subsampling are real but do not change this assessment; they are already noted in the reader's rationale. Therefore no verdict adjustment is needed beyond the existing conditional acceptance.","tokens_in":7868,"tokens_out":8915,"duration_ms":98903,"concrete_test":"On the full 41-image CLIC2020 validation set at each operating BPP, replace the decoder-side ClipSeg masks in Eq. (5) with oracle masks obtained by running ClipSeg on the original image, keeping all other components fixed. If the oracle version improves LPIPS and ClipIQA over the proposed decoder-side version by more than the reported gain of 'Ours' over 'Ours w/o ClipSeg', then mask quality from the low-bitrate reconstruction is the load-bearing bottleneck and the central claim stands or falls on it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 replaces MISC's coded 8x8 patch semantic maps with decoder-side ClipSeg masks computed from the low-bitrate reconstruction, and Eq. (5) uses these masks to gate which pixels the diffusion process refines. The central claim—better perceptual quality than MISC at similar or lower BPP—therefore depends on the initial reconstruction preserving enough structure for open-vocabulary segmentation. This is not established: ClipSeg is trained on natural images, not on heavily compressed/downsampled reconstructions, and the only support is one favorable visual example (Fig. 2). The ablation in Sec. 4.3 implicitly concedes the risk: at about 0.06 BPP, the full model is only matched by 'Ours w/o ClipSeg' on pixel-level metrics because 'the quality of semantic segmentation maps obtained from the initial low-bitrate reconstructed image is not sufficient.' If masks are wrong, Eq. (5) applies object-conditioned diffusion to incorrect regions, which can hallucinate content and hurt both pixel and perceptual fidelity; no quantitative mask-accuracy measurement is reported. This is not a circularity or internal inconsistency, but it is an unvalidated core assumption at exactly the ultra-low-bitrate regime the paper targets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two modifications to the multimodal image semantic compression (MISC) framework for ultra-low-bitrate image compression. First, it replaces MISC's encoder-side patch-based semantic maps with decoder-side semantic segmentation using ClipSeg applied to the low-bitrate reconstruction. Second, it introduces content-adaptive diffusion: two MLPs predict the number of diffusion steps and the classifier-free-guidance scale from handcrafted statistical, perceptual, and semantic features of the reconstruction. The authors evaluate on the CLIC2020 professional subset (41 validation images) against seven codecs using three groups of quality metrics, and report that the proposed method improves over MISC in pixel-level and perceptual metrics while reducing encoding and decoding time by more than 36%. They also claim competitive perceptual performance against mainstream codecs at very low bitrates.","tokens_in":8066,"tokens_out":4634,"duration_ms":46669,"significance":"The contribution is a pragmatic, incremental improvement to an existing generative codec, with the practical appeal of simplifying the encoder (no semantic map coding) and reducing decoding cost via adaptive diffusion. The design is not circular: the MLPs are trained against an external LPIPS-based objective on a separate training set, and the evaluation is on held-out data against external baselines. The code is made available, which aids reproducibility. However, the core assumption that decoder-side segmentation on heavily compressed low-bitrate images is accurate enough for gated refinement is not quantitatively verified, and the evaluation methodology has gaps (small test set, no raw values, no bitrate-accounting details). If those issues are resolved, the work would be a solid contribution to practical generative image compression.","major_comments":[{"comment":"The decoder-side semantic segmentation is the central replacement for MISC's transmitted semantic maps, yet its accuracy is never quantitatively evaluated. The paper provides only one favorable visual example (Fig. 2), and the ablation in Sec. 4.3 explicitly concedes that at roughly 0.06 BPP the segmentation maps obtained from the low-bitrate reconstruction are not sufficient to improve pixel-level accuracy. Since Eq. (5) uses mask M[i] to gate which regions receive object-conditioned diffusion refinement, incorrect masks would cause diffusion to be applied to the wrong regions and could introduce hallucinated content. Please report quantitative mask accuracy (e.g., mIoU against ground-truth masks, or against the original MISC patch maps) across the operating bitrates, and analyze how mask errors impact the end-to-end reconstruction quality.","section":"Sect. 3.1, Eq. (5)"},{"comment":"The evaluation is based on only 41 validation images, with per-group metric averages computed after normalizing each metric to [0,1] using IQA-PyTorch ranges. The paper does not report raw per-metric values, error bars, or statistical significance, so the claim that the proposed method 'consistently outperforms' MISC across all bitrates and metric groups is not verifiable. In addition, the bitrate accounting is unspecified: it is not stated whether the approximately 60 words of semantic text from GPT-4 Vision are included in the BPP figures, nor how the removal of MISC's semantic-map bits is reflected. Please provide per-metric tables (or BD-rate equivalents with confidence intervals), specify the exact normalization ranges, and explicitly state which components are counted in the reported BPP.","section":"Sect. 4.1, Figs. 3 and 4"},{"comment":"The timing comparison is based on the first 20 images of the CLIC2020 validation set on a single GPU (GTX 1060) with no repeated runs or variance reported. Since reducing complexity is one of the two main claimed contributions, this evidence is too thin. Please report mean and standard deviation over the full validation set (or a clearly justified subset) and over multiple runs, and specify exactly which operations (MLP inference, ClipSeg, DiffBIR, LMIC decode, GPT-4 Vision, etc.) are included in the measured encoding and decoding times.","section":"Sect. 4.2, Table 1"},{"comment":"The training protocol for the two MLPs is under-specified. The paper does not describe how the ground-truth labels y_i (the diffusion-step count and CFG scale that achieve the best LPIPS) are obtained, including the search grid over steps in [2,80] and CFG in (0,10), the number of training samples, or the effect of the regularization coefficient λ=0.64 on the step-count distribution. Without this information, the content-adaptive diffusion component is not reproducible, and the claimed benefit of adaptivity cannot be independently assessed. Please add the full training protocol or cite a prior method that fully specifies it.","section":"Sect. 3.2, Eq. (3)"}],"minor_comments":[{"comment":"The abstract states that the method enhances quality 'in terms of PSNR and perceptual metrics,' but Fig. 3 shows that several learned and traditional codecs achieve higher pixel-level similarity; please qualify the PSNR claim to clarify that the improvement is relative to the MISC baseline, not to all codecs.","section":"Abstract and Fig. 3"},{"comment":"The normalization ranges for PSNR, SSIM, MS-SSIM, and the no-reference metrics are not listed. Please specify the exact IQA-PyTorch-defined ranges used, since different normalization choices can change the shape of the averaged curves.","section":"Sect. 4.1"},{"comment":"Clarify whether M[i] is the raw ClipSeg probability map or a binarized mask, and if binarized, what threshold is used; this detail affects the blending operation in Eq. (5).","section":"Eq. (5)"},{"comment":"The caption of Table 1 does not state whether the times are per-image averages or whether they include the MLP inference, ClipSeg, and the autoencoder decode; please make the measured components explicit.","section":"Table 1"},{"comment":"The conclusion that 'our full model offers the best results across all cases' is weakened by the acknowledged tie in pixel-level metrics at about 0.06 BPP; consider adding a sentence explaining how this affects the overall claim.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The contribution is incremental but potentially solid if the evaluation gaps are addressed. The most important issue is the unvalidated decoder-side segmentation: since this is the main architectural change and directly gates diffusion refinement, a quantitative mask-accuracy study is necessary. I also encourage the editor to ask for raw per-metric results and bitrate accounting, as the current normalized-average curves are insufficient for a clear verdict on the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The two new pieces are decoder-side semantic segmentation with ClipSeg, replacing MISC's transmitted patch maps, and content-adaptive diffusion that predicts step count and CFG scale from the low-bitrate reconstruction. Both are known building blocks, but the combination is new and the paper shows, via ablation, that each component contributes. The method section is clear enough that a good-faith reimplementation is feasible, and the code is promised. That is real value.\n\nThe central claim — better perceptual quality than MISC at similar bitrate with over 36% less compute — is plausible and mostly supported by the figures. The compute savings are easy to believe: dropping the semantic map encoder and adapting the step count will cut time. The perceptual gains are also plausible given that decoder-side masks avoid the fixed 8x8 grid of MISC. I think the stress-test concern about ClipSeg on heavily compressed images is legitimate but not fatal. The paper itself concedes in Sec. 4.3 that at roughly 0.06 BPP the segmentation quality is insufficient to help pixel-level accuracy. That is exactly the regime the method targets. The authors do not quantify mask accuracy at all, and the one visual example (Fig. 2) is favorable. Still, the ablation shows that even when the pixel-level benefit disappears, perceptual metrics improve, so the method is not broken — it just may not always help as much as advertised.\n\nThe soft spots are in the evaluation. The validation set is 41 images with no error bars or significance tests, so the normalized average curves in Fig. 3 and Fig. 4 may hide a lot of variance. The timing comparison uses 20 images on a single GTX 1060, and, more importantly, the bitrate accounting is never specified. With a generative codec, how you count bits for text, latent, and side information changes the rate axis substantially. The authors need to state exactly what is included. These are fixable issues, not fundamental flaws.\n\nThe paper does not exhibit circularity. The MLPs are trained against LPIPS on a separate training set and evaluated on held-out images. Citation patterns look fine; MISC is the baseline and the comparison is fair.\n\nWho gets value: anyone working on semantic or generative image compression, especially at ultra-low bitrates. It is a useful engineering data point and a reasonable incremental step over MISC. The evaluation weaknesses are real but not disqualifying for a conference paper. I would send it to review and ask for error bars, a clear bitrate accounting, and at least a small quantitative check of segmentation accuracy on the low-bitrate reconstructions. With those, the 36% time claim and the perceptual quality claims would be much better supported.","headline":"A solid, incremental extension of MISC with a plausible win on compute and perceptual quality, held back mainly by thin evaluation rather than a flawed idea.","tokens_in":8654,"tokens_out":999,"would_cite":true,"duration_ms":11978,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative image codec improves quality and speed by moving segmentation to the decoder and adapting diffusion settings per image.","keywords":["generative image compression","semantic compression","diffusion models","decoder-side segmentation","content-adaptive diffusion","MISC","perceptual quality","low bitrate"],"falsifier":"Take a validation set with many small or atypical objects, compress at a range of very low bitrates below 0.06 BPP, and compare the codec's output when ClipSeg masks are computed from the compressed image versus from the original full-quality image: if the perceptual or pixel metrics converge, or the compressed-mask version drops below the MISC baseline, the load-bearing assumption fails.","tokens_in":7599,"feed_emoji":"⚡","tokens_out":5134,"duration_ms":47506,"temperature":0.7,"pith_summary":"The paper claims that a generative image codec can reconstruct images at very low bitrates more accurately and much faster than the MISC baseline by making two changes: compute object masks at the decoder from the already-decoded low-bitrate image instead of transmitting them, and let a small predictor choose how many diffusion steps and how much text guidance each image needs. The authors report that both encoding and decoding time drop by more than 36% while pixel-level and perceptual metrics improve across bitrates, and that the codec beats mainstream codecs in perceptual similarity and quality. If true, this makes semantics-guided generative compression more practical and supports a design principle where spatial side information is derived at the decoder rather than spent as bitrate.","feed_headline":"Generative codec gets sharper low-bitrate images and a 36% speedup","feed_subtitle":"By deriving masks at the decoder and adapting diffusion steps, the codec beats its baseline on quality and speed.","key_machinery":"The central objects are decoder-side semantic masks produced by ClipSeg, an open-vocabulary segmentation model that localizes objects from text prompts, and a content-adaptive diffusion predictor made of two small MLPs that output a diffusion step count in $[2,80]$ and a classifier-free guidance scale in $(0,10)$. The selective refinement step is $x_t = D_{1\\times}(x_{t-1}; T_d[i])\\cdot M[i] + x_{t-1}\\cdot(1-M[i])$, which applies conditional diffusion only inside each predicted object mask $M[i]$. The MLPs are trained to predict the settings that give the best LPIPS similarity to the original, with a regularization term that penalizes large step counts.","core_discovery":"The central claim is that removing the transmitted patch-based semantic map from the MISC framework and instead running ClipSeg at the decoder on the degraded low-bitrate reconstruction, together with MLP-predicted diffusion step counts and classifier-free guidance scales, improves reconstruction quality while lowering compute. The decoder uses the segmentation masks to refine each object region selectively, so spatial conditioning costs no extra bitrate. The paper reports consistent gains over MISC across all metric groups and bitrates, with average encoding time falling from 1.19 to 0.76 seconds and decoding time from 1930.98 to 1194.00 seconds on its test setup.","pith_inferences":["Editorial inference: because the masks come from the compressed image, the codec's quality ceiling is set by how well the low-bitrate reconstruction preserves object outlines; a direct test would compare outputs when masks are computed from the original image instead of the compressed one.","Editorial inference: ClipSeg is open-vocabulary, so the same decoder-side masking idea could replace transmitted segmentation maps in other semantics-driven generative codecs beyond MISC.","Editorial inference: training the MLPs on per-image best LPIPS effectively learns a compute-quality tradeoff; retraining with a rate-aware objective could give explicit control over bitrate, runtime, and quality jointly."],"forward_implications":["If the central claim is correct, the proposed codec delivers better perceptual similarity and quality than the MISC baseline at the same or lower bitrate, while also running over 36% faster.","Decoder-side segmentation eliminates the bitrate formerly spent on transmitting patch-based semantic maps, shifting that cost from the encoder to the decoder without hurting reconstruction.","Content-adaptive diffusion reduces average decoding time because most images need fewer than the fixed 40 diffusion steps that MISC uses.","The method's per-image diffusion settings lower the chance of hallucination by avoiding unnecessary refinement steps on images that are already well reconstructed.","The reported gains across pixel-level, perceptual-similarity, and perceptual-quality metrics suggest the two proposed components are complementary rather than redundant."],"supporting_citations":[{"why":"Supplies the baseline MISC framework, including its patch-based semantic map encoding and fixed diffusion settings that this work replaces.","marker":"[5]"},{"why":"Supplies ClipSeg, the open-vocabulary segmentation model used at the decoder to derive object masks from text and the low-bitrate image.","marker":"[13]"},{"why":"Supplies DiffBIR, the conditional diffusion decoder with ControlNet integration used for upscaling and object-wise refinement.","marker":"[24]"},{"why":"Supplies the pretrained Cheng2020 autoencoder used to produce the compact low-bitrate latent representation.","marker":"[12]"},{"why":"Supplies the latent diffusion model underlying DiffBIR and the conditional generation mechanism.","marker":"[3]"},{"why":"Supplies LPIPS, which is both the training target for the MLPs and one of the perceptual similarity metrics in the evaluation.","marker":"[23]"},{"why":"Supplies GPT-4 Vision, the semantic encoder that extracts object names, details, and the overall scene description.","marker":"[11]"}],"fun_headline_variants":["Decoder-side segmentation lifts AI compression, cuts time 36%","Semantic guidance and adaptive diffusion improve codec speed and quality","Generative codec uses on-the-fly masks to beat baseline at low bitrate","Content-aware diffusion steps slash encoding and decoding by 36%","Semantic map learned at decoder, not sent, boosts compression efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme depends on the heavily compressed, low-resolution reconstruction still containing enough visible shape and layout that ClipSeg can find each object named in the text; if the masks are wrong, the diffusion refinement edits the wrong areas.","fun_headline_variants_meta":{"raw":{"variants":["Decoder-side segmentation lifts AI compression, cuts time 36%","Semantic guidance and adaptive diffusion improve codec speed and quality","Generative codec uses on-the-fly masks to beat baseline at low bitrate","Content-aware diffusion steps slash encoding and decoding by 36%","Semantic map learned at decoder, not sent, boosts compression efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000703,"raw_usage":{"total_tokens":3103,"prompt_tokens":808,"completion_tokens":2295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":2217}},"tokens_in":424,"tokens_out":2295,"duration_ms":16998,"temperature":1.0,"reasoning_tokens":2217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:37:18.120675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a validation set with many small or atypical objects, compress at a range of very low bitrates below 0.06 BPP, and compare the codec's output when ClipSeg masks are computed from the compressed image versus from the original full-quality image: if the perceptual or pixel metrics converge, or the compressed-mask version drops below the MISC baseline, the load-bearing assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the baseline MISC framework, including its patch-based semantic map encoding and fixed diffusion settings that this work replaces."},{"cited_title":"MISC: Ultra-low bitrate image se- mantic compression driven by large multimodal model,","cited_arxiv_id":null,"evidence_quote":"Supplies ClipSeg, the open-vocabulary segmentation model used at the decoder to derive object masks from text and the low-bitrate image."},{"cited_title":"Blind / reference- less image spatial quality evaluator,","cited_arxiv_id":null,"evidence_quote":"Supplies DiffBIR, the conditional diffusion decoder with ControlNet integration used for upscaling and object-wise refinement."},{"cited_title":"Semantic based generative com- pression of images for extremely low bitrates,","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained Cheng2020 autoencoder used to produce the compact low-bitrate latent representation."},{"cited_title":"Mountain,","cited_arxiv_id":null,"evidence_quote":"Supplies the latent diffusion model underlying DiffBIR and the conditional generation mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies LPIPS, which is both the training target for the MLPs and one of the perceptual similarity metrics in the evaluation."},{"cited_title":"High-resolution image synthesis with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Supplies GPT-4 Vision, the semantic encoder that extracts object names, details, and the overall scene description."}],"review_version":1}