{"id":"3717a1a0-58e2-47ab-a1a8-436abebfd6b7","arxiv_id":"2501.04579","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"UG-ICM trains a single learned image codec with global, local, and instance-level CLIP supervision and a preference-conditioned decoder to support both human viewing and unseen machine analytics tasks from one bitstream.","lead":"This paper introduces UG-ICM, a learned image compression method that uses CLIP model supervision to preserve semantic information for both human perception and a wide range of machine analytics tasks through a single bitstream. It shows that adding multi-scale CLIP losses and a preference-conditioned decoder improves downstream task performance and perceptual quality, potentially simplifying future image coding systems for machine and human consumers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported analytics gains rest on single training runs with no error bars; the 2.8–6.0% bpp improvements may be within seed noise, so the central generalization claim is not yet established.","rationale":"The paper proposes a coherent method: CLIP-based multi-scale supervision for generalized machine analytics, a preference-conditional decoder for a unified bitstream, and a two-stage training strategy. The architecture and losses are clearly described, and the ablation studies provide internal evidence that the multi-scale loss and PCDM affect behavior. The central empirical claim, however, is that these components yield consistent analytics gains across unseen tasks. That claim currently rests on single training runs per rate point, with no error bars or significance testing. Given the modest reported margins and known sensitivity of learned compression training to seeds and hyperparameters, the gains may not be reproducible. The reader's weakest_assumption focuses on CLIP universality, which is conceptually important, but the more immediately load-bearing issue is the lack of statistical grounding for the headline numbers. I also note a secondary concern: the paper says training is 'without awareness of any specific downstream models and tasks,' yet it uses FreeSOLO, an instance segmentation model, to generate instance-level masks for the MS-CLIP loss; this is worth clarifying but does not by itself overturn the results. Overall, the verdict CONDITIONAL remains appropriate, contingent on repeated-seed validation and code release.","tokens_in":11289,"tokens_out":6292,"duration_ms":66006,"concrete_test":"Retrain the MLIC++ backbone and UG-ICM from scratch with at least 3 random seeds at the same rate point (same lambda) used in Fig. 5, and report mean±std for bpp, Top-1, mAP, mIoU, and PSNR. Run a paired bootstrap or Wilcoxon test over per-image metrics. If the 2.8–6.0% gains do not persist in direction and magnitude across seeds, the central generalization claim weakens; if they do, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the Abstract and §Experimental Results is that UG-ICM yields remarkable bpp-analytics gains (1.7–6.0%) on four unseen tasks while preserving perceptual quality. These gains are computed from one training run per rate point for each backbone and method; no standard deviation, confidence interval, or significance test is reported. Learned image compression models are known to exhibit nontrivial rate-distortion variation across random seeds and training schedules, and both anchor and proposed models are trained from scratch, making the comparison vulnerable to initialization and optimization noise. Figure 5, Figure 8, and Figure 10 all compare single curves. Because the reported margins (e.g., 2.8% classification, 3.9% detection on MLIC++) are modest relative to typical training variance, the observed improvements could be spurious. The 'unseen task' claim is therefore not yet supported with statistical confidence, independent of the conceptual validity of CLIP supervision.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UG-ICM, a learned image compression framework that combines a preference-conditional decoder with CLIP-based multi-scale semantic supervision. The encoder and entropy model produce a unified bitstream, and the decoder takes a scalar preference code (human or machine) to produce two reconstructions. The training alternates between rate-perception and rate-analytics objectives, and the CLIP losses are applied at global, random-local, and instance levels (Eqs. 4-7), with instance masks obtained from FreeSOLO. Experiments on two LIC backbones, Bmshj2018 and MLIC++, report bpp-analytics gains of 1.7-6.0% on four downstream tasks, comparable human-perceptual quality, and ablations of the PCDM and multi-scale loss.","tokens_in":11473,"tokens_out":10917,"duration_ms":106523,"significance":"The manuscript addresses a relevant and active topic, and the central idea is plausible: using a frozen multimodal model as task-agnostic supervision and conditioning the decoder on a preference code could provide a practical single-bitstream solution for both human and machine consumption. The paper provides a code link, evaluates on four downstream tasks, and uses two different backbones, which is a reasonable breadth of evidence for a conference submission. The main value is empirical; if the gains are robust, the method would be a useful contribution to the image-coding-for-machines literature. The claims of 'remarkable' and 'unseen' generalization are, however, currently stronger than the evidence supports.","major_comments":[{"comment":"The reported average bpp-analytics gains are not accompanied by any measure of variance, and the phrase 'average bpp-analytics gain' is never defined. The Implementation Details state that models are trained four times with different Lagrange parameters; that is four rate points, not four random seeds. Because learned compression training is sensitive to initialization and the claimed improvements are 1.7-6.0%, the central claim that CLIP supervision improves unseen analytics is not statistically established. Please provide multiple seeds and error bars, and specify how equal-bpp comparisons and the reported averages are computed, including the interpolation rule and bpp anchors.","section":"Experimental Results, Figs. 5 and 8"},{"comment":"The instance-level CLIP loss pairs a set of masks from the original image with a set of masks from the compressed image, but the paper does not explain how the two sets are matched. FreeSOLO can return different numbers or arrangements of masks for the original and the compressed image; if pairing is only by index, the loss may align unrelated instances. Please describe the mask extraction and matching procedure, including confidence thresholds, maximum number of masks, and whether masks are taken from the machine-preference reconstruction at every training step, or compute the loss using masks transferred from the original image only.","section":"Proposed Method, Eq. (4)"},{"comment":"The local-wise CLIP loss is underspecified: the paper does not state the crop size, the number of local crops per image, whether the same spatial coordinates are used for the original and reconstructed crops, or how crops are resized for the CLIP ViT-B/32 input. These choices directly affect the learned representation and the reported gains, so they should be documented to make the MS-CLIP loss reproducible.","section":"Proposed Method, Eq. (5) and Implementation Details"},{"comment":"The paper notes that TransTIC requires recompression into another bitstream when adapting to a new task, but it does not state whether the bpp plotted for TransTIC in Fig. 8 includes both the human-oriented and the analytics-oriented bitstreams. If only one bitstream is counted, the comparison is biased in favor of UG-ICM, whose advantage is precisely the use of a single bitstream. Please clarify the bitrate accounting for all anchor methods.","section":"Comparison to State-of-the-art ICM Methods, Fig. 8"},{"comment":"The statement that instance segmentation is an 'unseen' task is imprecise: the MS-CLIP loss uses FreeSOLO, which is itself an instance segmentation model, to generate the instance-level pairs. The compression model therefore receives instance-level segmentation information during training, even though the evaluation model, Mask R-CNN, and its annotations are not used. Please either weaken the 'unseen task' claim or explain why FreeSOLO's outputs should not be counted as task supervision.","section":"Abstract and Benchmark"}],"minor_comments":[{"comment":"Please fix typos such as 'pretraind', 'tranining', 'Multy-Scale', and 'glocal-local-instance'.","section":"Throughout"},{"comment":"The cosine term has a mismatched bracket; please correct the notation.","section":"Eq. (3)"},{"comment":"The text 'average gain of3%' and '1dB bpp-PSNR' should be formatted as '3%' and '1 dB', and the terms bpp-mAP and bpp-PSNR should be defined.","section":"Fig. 9"},{"comment":"Please report the batch size, training image resolution, number of local crops per image, and the preprocessing applied before CLIP feature extraction.","section":"Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant and active topic, and the proposed framework is plausible. My recommendation of major revision is driven by the lack of statistical validation and by several underspecified components of the MS-CLIP loss; these are addressable within the scope of the paper. I do not see a fundamental flaw in the central derivation, but the current evidence does not yet support the strength of the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the core idea is genuinely new and useful: the authors use CLIP as a task-agnostic supervision signal for learned image compression, applying it at global, local, and instance scales, and pair it with a preference-conditioned decoder so that one bitstream can be decoded for either human perception or machine analytics. That is a sensible approach to the \"unseen task\" problem in image coding for machines, and it is not something I have seen done before. Second, the evidence for the headline generalization claim is thinner than the abstract suggests. The reported gains (2.8% to 6.0% average bpp-analytics improvements on MLIC++ across four tasks) come from single training runs per rate point, with no error bars or significance tests. Learned compression training has enough seed sensitivity that those margins could shift. The stress-test note is right about that, and it is the paper's biggest soft spot.\n\nWhat the paper does well: the multi-scale CLIP supervision is a real contribution, and the offline toy experiment in Fig. 3 is a nice empirical justification for using CLIP to guide reconstruction rather than only latent constraints. The experimental design is otherwise solid — four downstream tasks, two backbones, comparisons to Chamain, TransTIC, and SA-ICM, and an ablation of the conditional decoder. The human vs. machine preference trade-off shown in Fig. 9 is a useful sanity check. The work is honest about being self-supervised and does not overclaim beyond the missing statistics.\n\nSoft spots in proportion: aside from the single-run issue, the implementation details are incomplete. I want to know how the FreeSOLO instance masks are matched between original and compressed images, what crop size is used for the local CLIP loss, and whether the equal weighting of the three CLIP losses was tuned. The code link is provided but not verified, so I cannot check reproducibility. None of these are fatal; they are standard referee asks. One thing I would not ding is the \"unseen\" label: the downstream networks are unseen, which is the right claim, and CLIP's own pretraining does not undermine that.\n\nWho this is for: anyone working on learned image compression, image coding for machines, or vision-language-guided representation learning. It is a worthwhile reading-group paper and deserves a serious peer-review round, with the statistical robustness question front and center.","headline":"A plausible and genuinely novel CLIP-based unified image coding scheme whose reported gains are real but not yet statistically pinned down; deserves a proper referee, not a desk reject.","tokens_in":12018,"tokens_out":1713,"would_cite":true,"duration_ms":18606,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a unified image codec that serves both human perception and unseen machine analytics from a single bitstream, trained without task-specific supervision.","keywords":["image compression","image coding for machine","CLIP supervision","learned image compression","conditional decoding","multi-scale semantic loss","human-machine unified coding","generalization"],"falsifier":"Pick an analytics task whose success is known to be poorly captured by CLIP-style global semantics, for instance monocular depth estimation or counting of small objects, and run the machine-preference decode of UG-ICM on it at matched bitrates. If the bpp-analytics gains over the vanilla codec disappear or reverse on that task, the claimed task-agnostic generalization is bounded by CLIP's semantic coverage; if the gains persist, the generalized claim is stronger than the paper's current evaluation shows.","tokens_in":11086,"feed_emoji":"🖼️","tokens_out":7884,"duration_ms":68511,"temperature":0.7,"pith_summary":"The paper proposes UG-ICM, an image compression scheme that trains one codec to serve two audiences at once: human viewers and machine analytics models, including models not seen during training. The key move is to use a pre-trained contrastive vision-language embedding as the semantic referee, pushing compressed images to stay close to the originals at whole-image, local-patch, and instance levels. A preference-conditioned decoder then turns one shared bitstream into either a human-pleasing reconstruction or a machine-oriented one, selected by a single tunable parameter. On the MLIC++ backbone at matched bits-per-pixel, the machine-oriented version improves classification, object detection, semantic segmentation, and instance segmentation by average bpp-analytics margins of 2.8%, 3.9%, 6.0%, and 3.2%, while the human-oriented version keeps PSNR, SSIM, and LPIPS on par with the backbone. If correct, this means task-agnostic semantic coding is achievable without task-specific labels or multiple bitstreams.","feed_headline":"CLIP-trained compression lifts unseen AI analytics on one bitstream","feed_subtitle":"Global-to-instance CLIP supervision keeps semantics intact, so one stream serves people and four vision tasks.","key_machinery":"The load-bearing machinery is the Multi-Scale CLIP loss and the Preference Conditional Decoding Module (PCDM). The MS-CLIP loss measures cosine similarity between the original and reconstructed images in the CLIP image-embedding space at three granularities: the full image (global), random local crops (local), and instance crops produced by a segmentation model (instance); summing the three pushes the codec to preserve hierarchical semantics. The PCDM is a lightweight module placed in each decoding block: a two-layer MLP turns the preference indicator beta in {beta_h, beta_m} into a bias feature that is added to the decoder's image features, steering reconstruction toward human or machine utility. A two-stage training schedule alternates human-oriented and machine-oriented objectives so the encoder produces a unified latent and the decoder learns the two branches. Together they make a single bitstream that can be decoded in two different versions without recompression.","core_discovery":"The paper claims that a learned image codec can be made simultaneously human- and machine-friendly, and generalizable to unseen analytics, by two mechanisms. First, it replaces task-specific supervision with multi-scale CLIP-based semantic supervision: a cosine-similarity loss at global, local-crop, and instance-crop levels between the original and reconstructed images, computed in the embedding space of a pre-trained CLIP image encoder. Second, a Preference Conditional Decoding Module injects a preference feature into each decoder block, so the same latent bitstream produces a human-preferred image when the preference parameter is beta_h and a machine-preferred image when it is beta_m. The whole model is trained self-supervised and alternates between rate-perception and rate-analytics objectives; at test time the downstream analytics models are unseen. On the MLIC++ backbone the machine decode gives average bpp-based gains of 2.8% for classification, 3.9% for object detection, 6.0% for semantic segmentation, and 3.2% for instance segmentation over the vanilla codec, with comparable PSNR, SSIM, and LPIPS.","pith_inferences":["Implicit extension: because CLIP embeddings are aligned with text, a language-conditioned variant could steer the machine decode toward a specified task family, such as pedestrians or road markings, without retraining the codec; the paper does not explore this.","Testable extension: the paper's ablations isolate the joint effect of the three CLIP losses but not their optimal balance; replacing the fixed equal weighting with learned or rate-dependent weights could yield larger gains.","Boundary inference: the mechanism should help any analytics task whose errors align with CLIP's notion of semantic similarity, such as common recognition tasks, but tasks like geometric measurement or fine-grained counting may see little or no benefit; the paper's task set does not cover those cases.","Consequence for codec design: if the result holds, learned codecs for machine consumption can drop task-specific heads and instead budget bits toward preserving human-interpretable semantics, making the bitstream a shared resource rather than a task-specific artifact."],"forward_implications":["A single compressed bitstream can be decoded into a human version or a machine version by switching one scalar preference, so separate human and machine streams become unnecessary.","Because training needs no downstream labels or analytics models, the same recipe can be re-run when new tasks arrive, without retraining the codec for each task.","The reported gains on unseen classification, detection, and segmentation suggest that semantic preservation in a generic embedding space transfers across task families, not just to tasks used for evaluation.","The multi-scale variant outperforms global-only CLIP supervision, indicating that small-object and local semantics are material to machine analytics and are worth explicit loss terms.","The recipe transfers across two learned codec backbones, so it acts as a training-time add-on rather than a redesign of the entropy model or architecture."],"supporting_citations":[{"why":"Supplies the pre-trained CLIP image encoder whose cosine similarity defines the multi-scale semantic supervision.","marker":"Radford et al. 2021"},{"why":"Provides the MLIC++ learned compression backbone on which the largest reported gains are measured.","marker":"Jiang and Wang 2023"},{"why":"Provides the Bmshj2018 backbone used as the second implementation target.","marker":"Ballé et al. 2018"},{"why":"Supplies the instance segmentation model whose predicted instance crops enable instance-level CLIP supervision.","marker":"Wang et al. 2022"},{"why":"Serves as the conventional supervised ICM baseline that jointly trains with a downstream analytics loss.","marker":"Chamain et al. 2021"},{"why":"Serves as the transfer-based ICM baseline that recompresses into a separate bitstream per task.","marker":"Chen et al. 2023"},{"why":"Serves as the self-supervised ICM baseline that allocates bits using segmentation information.","marker":"Shindo et al. 2024"},{"why":"Provides the YOLOv3 detector used to measure unseen object-detection performance.","marker":"Redmon and Farhadi 2018"}],"fun_headline_variants":["One bitstream, dual vision: CLIP-tuned codec adapts to any task","CLIP supervision makes one codec serve both humans and unseen AI","Self-supervised codec: single stream for human and machine eyes","Unified coding: CLIP semantics generalize to any analytics task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that closeness in the CLIP image-embedding space, at global, local, and instance scales, is a faithful proxy for what later, unseen machine analytics tasks need; if that proxy fails for a task, the training signal will not help that task.","fun_headline_variants_meta":{"raw":{"variants":["One bitstream, dual vision: CLIP-tuned codec adapts to any task","CLIP supervision makes one codec serve both humans and unseen AI","Self-supervised codec: single stream for human and machine eyes","Unified coding: CLIP semantics generalize to any analytics task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1318,"prompt_tokens":1015,"completion_tokens":303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":224}},"tokens_in":631,"tokens_out":303,"duration_ms":4030,"temperature":1.0,"reasoning_tokens":224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:28:08.426171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick an analytics task whose success is known to be poorly captured by CLIP-style global semantics, for instance monocular depth estimation or counting of small objects, and run the machine-preference decode of UG-ICM on it at matched bitrates. If the bpp-analytics gains over the vanilla codec disappear or reverse on that task, the claimed task-agnostic generalization is bounded by CLIP's semantic coverage; if the gains persist, the generalized claim is stronger than the paper's current evaluation shows.","supporting_citations":[{"cited_title":"W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained CLIP image encoder whose cosine similarity defines the multi-scale semantic supervision."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the instance segmentation model whose predicted instance crops enable instance-level CLIP supervision."},{"cited_title":"D.; Racap \\'e , F.; B \\'e gaint, J.; Pushparaja, A.; and Feltman, S","cited_arxiv_id":null,"evidence_quote":"Serves as the conventional supervised ICM baseline that jointly trains with a downstream analytics loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the transfer-based ICM baseline that recompresses into a separate bitstream per task."},{"cited_title":"Image Coding for Machines with Edge Information Learning Using Segment Anything","cited_arxiv_id":"2403.04173","evidence_quote":"Serves as the self-supervised ICM baseline that allocates bits using segmentation information."}],"review_version":1}