Pith. sign in

REVIEW 4 major objections 6 minor 85 references

UniMIC: Towards Universal Multi-modality Perceptual Image Compression

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One diffusion-based compensator, trained once on eight codecs, improves the perceptual quality of decoded images across all of them and transfers to unseen codecs.

desk verdict A solid, genuinely novel empirical contribution, but the universality claim is softer than the title suggests because of per-codec guidance tuning and a loosely used RDP framing. read the letter →

arxiv 2412.04912 v2 pith:CQX7ZNHK submitted 2024-12-06 eess.IV cs.CV

Yixin Gao , Xin Li , Xiaohan Pan , Runsen Feng , Zongyu Guo , Yiting Lu , Yulin Ren , Zhibo Chen This is my paper · ORCID
classification eess.IVcs.CV
keywords imagecompressionrate-distortion-perceptiontradeoffperceptualqualitydiffusionmodelstext-guidedrestorationcodec-agnosticenhancementmulti-modalityuniversaladapter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniMIC sets out to show that perceptual image compression does not require designing a new codec from scratch. Instead, it takes the decoded output of an existing codec—traditional or learned—and cleans it up with a generative diffusion model guided by two pieces of text: a content caption describing the image, and a compression prompt naming the codec, its quality setting, and its optimization goal. The paper argues that one trained compensator improves rate–distortion–perception performance for all eight codecs in its repository and generalizes to codecs it has never seen. If true, a single add-on module could upgrade the perceptual quality of many deployed image codecs without retraining them.

What carries the argument

The load-bearing object is the universal perception compensator: a frozen text-to-image latent diffusion model plus a trainable universal visual adapter and a trainable decoder refiner. The adapter takes the decoded image feature and injects it into the diffusion model through scale-and-shift modulation, while the compression prompt is encoded by the diffusion model's text encoder, pooled, passed through an MLP, added to the timestep embedding, and fed into every adapter block; the content prompt enters through cross-attention as in standard text-to-image diffusion. The decoder refiner connects the autoencoder encoder's features to the decoder via a skip connection so fine details from the compressed image survive the latent round trip. Multi-grained textual coding supplies the two prompts: ConP has three levels of caption length, and ComP carries the codec identity and quality syntax that lets one adapter switch between codecs.

What would settle it

Take the trained UniMIC and apply it to a held-out codec whose artifacts look structurally different from the eight training codecs—for example, a codec that outputs binary block masks or one tuned for machine-vision tasks—at several bitrates. If the compensator fails to reduce FID below the base codec on this codec while still improving the eight training codecs, the text-prompt universality claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a universal perception compensator can be trained once and then attached to any existing image codec to push decoded images toward natural-looking reconstructions. The compensator freezes a pretrained text-to-image diffusion model and learns only a small set of adapter weights; during decoding it receives the compressed image together with a content prompt (a short, moderate, or detailed caption of the original image) and a compression prompt (a text description of codec category, codec name, optimization metric, quality parameter, and prompt level). These two textual conditions steer the diffusion process so the same weights produce perceptually improved outputs whether the input came from JPEG at high compression, VVC at ultra-low bitrate, a neural codec optimized for MSE, or a neural codec optimized for MS-SSIM. On the DIV2K and CLIC2020 test sets, the paper reports consistent FID and LPIPS gains over all eight base codecs, and it shows the same trained model reduces FID when applied to WebP and to a neural codec outside the training set.

Load-bearing premise

The load-bearing premise is that naming a codec and its quality in words is enough information for one trained image-improver to know how to fix that codec's particular distortions, even for codecs it never saw during training.

Editorial extensions

If this is right

  • A single trained perceptual compensator can replace per-codec perceptual post-processing: the same weights serve JPEG, VVC, HEVC, and several learned codecs.
  • The method's gains are largest at ultra-low bitrates; the paper reports turning a VVC reconstruction into one whose FID is comparable to a dedicated state-of-the-art perceptual codec.
  • Because the compensator works on the decoder side, existing encoders and bitstreams stay untouched; only the decoding step adds text-guided refinement.
  • The codec repository can grow without retraining the compensator, provided a new codec's artifacts can be described usefully in a compression prompt.
  • Users can trade distortion against perception continuously by interpolating between the base codec output and the compensator output, with a single parameter controlling the balance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if codec-name text embeddings are the transfer mechanism, a natural stress test is to train on a subset of codecs and measure how perceptual gains change as the repository grows; the paper only reports the full eight-codec training plus two unseen codecs.
  • Editorial inference: the content prompt is transmitted losslessly, so a cheap next experiment is to generate the caption from the compressed image itself rather than the original, removing the caption's bitrate overhead entirely.
  • Editorial inference: the same mechanism—name the degradation in text, then restore with a conditional diffusion model—is likely testable on other degradations such as blur, noise, or super-resolution, but the paper only demonstrates compression artifacts.
  • Editorial inference: the strongest form of universality would require a held-out codec with qualitatively different artifacts, such as one optimized for machine-vision tasks; the two unseen codecs shown are close relatives of the training families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes UniMIC, a universal multi-modality perceptual image compression framework. The method builds a visual codec repository of eight existing codecs (JPEG, HM-intra, VTM-intra, mbt2018-mean, cheng20-anchor-mse, cheng20-anchor-msssim, ELIC, and MS-ILLM), and transmits multi-grained textual information consisting of a variable-length content prompt (generated by CogVLM) and a compression prompt (encoding codec category, codec name, optimization metric, quality indicator, and content prompt level). At the decoder, a universal perception compensator based on Stable Diffusion 2.1-base is trained once to enhance the perceptual quality of decoded images from all base codecs. The compensator uses a universal visual adapter that injects the compression prompt via scale-and-shift conditioning, and it is trained in two stages, with a decoder refiner to improve distortion. Experiments on DIV2K and CLIC report FID and LPIPS improvements over all eight training codecs and two unseen codecs (WebP and Zou2022). The paper claims unified rate-distortion-perception optimization for multiple codecs and highlights the extensibility of the codec repository.

Significance. If the central claims hold, UniMIC would offer a practical single-model post-processing enhancer that improves the perceptual quality of a wide range of existing image codecs, leveraging large pretrained generative priors and text conditioning. The breadth of codecs considered and the explicit tests on unseen codecs are valuable. The paper also provides a flexible distortion-perception trade-off via interpolation (Eq. 3) and a variable-length content prompt that lets users control the rate-perception balance. The approach reuses Stable Diffusion, and the authors report inference time and model size, which is helpful. However, the significance is tempered by two issues: the classifier-free guidance weight w is tuned per codec and bitrate at test time, and the evaluations rely on single-seed sampling without error bars. These issues need to be addressed for the universality claim to be fully convincing.

major comments (4)
  1. [Section 4.1, Test hyperparameters] The guidance weight w is manually set to 5.0 for a subset of codec/bitrate combinations and to 7.5 for all others. Since w controls the strength of the text conditioning in classifier-free guidance (Eq. 2) and directly affects the perception-distortion operating point, the reported improvements in Fig. 3 may substantially reflect per-codec and per-bitrate tuning of this sampling hyperparameter rather than the learned compression-prompt conditioning. To substantiate the universality claim, the authors should re-run inference with a fixed w (or a small set of fixed values) across all codecs and bitrates, or use an automatic rule derived from the transmitted ComP without human tuning. This experiment requires no retraining and should be added to the paper; without it, the claim that a single trained model generalizes across codecs is not fully supported.
  2. [Section 4.3, Tables 2 and 3] The ablation results are reported as single numbers without error bars, significance tests, or multiple diffusion sampling seeds. Differences such as the BD-FID values for VTM in Table 2 (-85.06 for Ours w/o ConP vs. -83.60 for Ours w/o text) are small relative to the likely stochasticity of FID estimation on 100 images and of DDIM sampling. Without repeated runs or confidence intervals, the specific contributions of the content prompt and the decoder refiner are not statistically established. The authors should provide variance estimates or multiple-seed results at least for the ablations, and ideally for the main comparisons.
  3. [Section 4.2, Fig. 3] The bitrate accounting for the textual prompts is not explicitly stated. The visual comparisons in Fig. 1 show that the UniMIC operating points include the text overhead (e.g., VTM at 0.0079 bpp vs. UniMIC at 0.0134 bpp), which suggests that Fig. 3 also plots total bitrate. The paper should explicitly state in Section 4.2 that all UniMIC points are plotted at the total bitrate including the losslessly compressed content prompt and compression prompt. If the text overhead is not included in some plots, the comparison against base codecs and HiFiC is unfair; if it is included, the exact overhead per image should be reported, including the compression prompt separately from the content prompt.
  4. [Section 4.3, Extensible codec repository and supplementary Fig. 11] The generalization to unseen codecs (WebP in Fig. 8 and Zou2022 in the supplementary) is only presented as scatter plots or qualitative statements, without numeric FID/LPIPS values, rate-distortion curves, or the operating bitrates. Since the extensibility and universality of the repository is a central claim, the paper should report concrete numbers for at least one unseen codec at representative bitrates, and these numbers should be obtained under the same fixed-guidance protocol as in the first major comment. Without quantitative evidence, the claim that text embeddings of unseen codec names transfer meaningfully is not established.
minor comments (6)
  1. [Section 4.1, Datasets] The sentence 'we set the random seed as 42 to avoid randomness' is misleading: diffusion sampling is inherently stochastic, and setting a seed only aids reproducibility. It would be more accurate to say 'to ensure reproducibility of the stochastic sampling process.'
  2. [Section 3.3, Table 1] The average byte counts for the content prompt levels are informative, but the text does not explain how the concise, moderate, and detailed levels are generated by CogVLM (e.g., whether different prompt templates are used) or how the per-image length limits are enforced. A brief description of the generation procedure would improve clarity.
  3. [Section 5, Limitation] The limitation section discusses only inference time. Other limitations, such as the dependence on the content and compression prompts for reconstruction quality, the potential mismatch of CLIP embeddings for unseen codec names, the degradation of PSNR/MS-SSIM, and the reliance on a large pretrained diffusion model, should be acknowledged.
  4. [Section 4.2, Trade-off Between Distortion and Perception] The beta interpolation is a useful practical option, but the main text only shows qualitative results in Fig. 6. The quantitative rate-distortion curves for different beta values appear only in the supplementary (Figure 10); including a small quantitative plot in the main text would make the trade-off more concrete.
  5. [References] Reference [13] is incomplete: it is listed as 'TCSVT, 2021' without author names, volume, or pages. The full citation for Bross et al. should be provided.
  6. [Abstract and Introduction] The abstract mentions code availability, but the main text does not state code availability until the abstract. It would be helpful to mention the repository link in the experimental setup or conclusion.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: UniMIC's perceptual improvements are measured against external codecs and datasets, and no claimed result reduces by construction to a fitted input or self-citation.

full rationale

UniMIC is an empirical systems paper rather than a derivation-based theory paper. The central claim is that a single trained perception compensator, conditioned on textual content prompts and compression prompts, improves FID and LPIPS across eight training codecs and generalizes to unseen codecs. This claim is evaluated on held-out datasets (DIV2K, CLIC2020) and with genuinely unseen codecs (WebP and Zou2022 in the supplementary), so the reported improvements are not defined by the model's own training targets. The compression prompt does include the codec name and quality indicator, but this is explicit conditioning rather than circularity: giving the model the codec identity as input does not by construction guarantee improved perceptual metrics, and the generalization tests to unseen codecs are real empirical checks. The classifier-free guidance weight w is set to 5.0 or 7.5 depending on codec and bitrate in Section 4.1, but this is a disclosed sampling hyperparameter, not a fitted parameter renamed as a prediction; it affects the operating point but does not make the core comparison circular. The paper cites the authors' prior work (UCIP, MoE-DiffIR) for adapter and SPADE design ideas, but those citations are architectural inspirations, not load-bearing uniqueness theorems or unverified premises; the adapters are described and ablated in this paper. No equation in the paper is equivalent to its own input by construction, and no self-citation chain forces the conclusion. The main experimental robustness concern about manual per-codec guidance-weight tuning is a correctness issue, not a circularity issue.

Assumptions & free parameters 2 free parameters · 4 assumptions · 3 invented entities

The central claim rests on standard domain assumptions about diffusion priors and CLIP text embeddings, plus one load-bearing assumption that codec names in text transfer to unseen codecs. The system introduces three new components (universal visual adapter, decoder refiner, compression prompt) that are only internally evaluated. No formal proofs or machine-checked derivations are involved.

free parameters (2)
  • Test-time guidance weight w = 5.0 for some codecs/bitrates, 7.5 for others
    The classifier-free guidance weight is set differently for different basic codecs and bitrate points (Section 4.1, 'Test hyperparameters'). It is a hand-chosen value that affects the fidelity-perception balance and is tuned to improve the reported results.
  • Stage-2 loss weights (L1, LPIPS, adversarial) = not reported
    In Section 3.5, the second-stage training loss is a combination of L1, LPIPS, and adversarial loss, similar to VQGAN. The relative weights are not specified, which means the exact objective is underspecified and any tuned values are undisclosed free parameters.
assumptions (4)
  • domain assumption Stable Diffusion provides a strong generative prior that can reconstruct natural image details from degraded inputs.
    The whole method relies on SD's ability to generate realistic textures. This is the standard assumption of the field and is used throughout Section 3.4.
  • ad hoc to paper CLIP text embeddings of codec names and quality descriptors are meaningful and transferable to unseen codecs.
    The compression prompt is embedded with a frozen CLIP text encoder. For the 'extensible' experiments with WebP and Zou2022, the codec name was not seen in training, yet the model is expected to adapt based on the text embedding. This is a core novelty assumption and is tested only through two examples.
  • domain assumption The decoded image, after base codec compression and VAE encoding, retains enough structure for the diffusion process to recover a perceptually good image.
    The universal visual adapter feeds the VAE features of the compressed image into the diffusion UNet. If the base codec erases too much structure, the diffusion model cannot restore semantics. This is a standard assumption for post-processing methods but is load-bearing here.
  • domain assumption FID and LPIPS are appropriate and sufficient measures of perceptual quality.
    The main results are reported in FID and LPIPS, and the authors claim perceptual improvement based on these. This is the accepted practice in the field, though these metrics are imperfect proxies for human perception.
invented entities (3)
  • Universal Visual Adapter
    purpose: A trainable module that injects the decoded image and the compression prompt embedding into the frozen Stable Diffusion model via scale-and-shift operations.
    This is a new architectural component proposed in the paper. Its effectiveness is only demonstrated on the paper's internal benchmarks; there is no external validation or analysis beyond the reported FID/LPIPS numbers.
  • Decoder Refiner
    purpose: A fine-tuned module added to the Stable Diffusion VAE decoder to reduce information loss and improve fidelity of the final reconstruction.
    The decoder refiner is a new module used only in this framework. Its contribution is shown in an ablation, but it is not independently validated outside the paper.
  • Compression Prompt (ComP)
    purpose: A textual conditioning signal that encodes codec category, codec name, optimization metric, quality indicator, and content prompt level, used to adapt the diffusion model to different codecs.
    ComP is a new data structure proposed by the authors. The generalization evidence is limited to a few seen and unseen codecs, and no external replication exists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniMIC: Towards Universal Multi-modality Perceptual Image Compression." pith.science (2026). https://pith.science/paper/CQX7ZNHK

@misc{pith2026241204912,
  author       = {Pith},
  title        = {Pith review of: UniMIC: Towards Universal Multi-modality Perceptual Image Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQX7ZNHK}},
  note         = {Machine review of arXiv:2412.04912}
}
read the original abstract

We present UniMIC, a universal multi-modality image compression framework, intending to unify the rate-distortion-perception (RDP) optimization for multiple image codecs simultaneously through excavating cross-modality generative priors. Unlike most existing works that need to design and optimize image codecs from scratch, our UniMIC introduces the visual codec repository, which incorporates amounts of representative image codecs and directly uses them as the basic codecs for various practical applications. Moreover, we propose multi-grained textual coding, where variable-length content prompt and compression prompt are designed and encoded to assist the perceptual reconstruction through the multi-modality conditional generation. In particular, a universal perception compensator is proposed to improve the perception quality of decoded images from all basic codecs at the decoder side by reusing text-assisted diffusion priors from stable diffusion. With the cooperation of the above three strategies, our UniMIC achieves a significant improvement of RDP optimization for different compression codecs, e.g., traditional and learnable codecs, and different compression costs, e.g., ultra-low bitrates. The code will be available in https://github.com/Amygyx/UniMIC .

Figures

Figures reproduced from arXiv: 2412.04912 by the authors.

Figure 1
Figure 1. Visual comparisons of our proposed UniMIC framework with eight representative basic codecs, including the hand-crafted [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our proposed UniMIC. The visual codec repository includes various representative basic codecs and provides visual [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overall performance comparison between our method and state-of-the-art codecs on DIV2K. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Visual comparison of the reconstructions guided by different lengths of content prompt. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Test our UniMIC on unseen bitrates in the training stage. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization results with different β values. Base Codec Ours Ours w/o ComP Ours w/o ConP Ours w/o text VTM -96.91 -94.61 -85.06 -83.60 MS-ILLM -3.71 -2.95 -1.14 -0.96 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Test our UniMIC on WebP. The significant improvement [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Rate-distortion curves with different β values. Extensible codec repository. In the paper, we validated our extensible codec repository using a traditional image codec, WebP. Here, we provide additional results using a neural codec, Zou2022 [83], which was also not pa…
Figure 9
Figure 9. Figure 9: Overall performance comparison between our method and state-of-the-art codecs on CLIC20 test split. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 11
Figure 11. Figure 11: Test our UniMIC on Zou2022 [83]. The significant improvement in FID shows our codec repository is extensible. 2 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 53 canonical work pages

  1. [1]

    python.org/3/library/zlib.html

    zlib — compression compatible with gzip.https://docs. python.org/3/library/zlib.html. 5

  2. [2]

    Multimodal biomedical ai

    Juli´an N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical ai. Nature Medicine, 28 (9):1773–1784, 2022. 3

    2022

  3. [3]

    Ntire 2017 challenge on single image super-resolution: Dataset and study

    Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceed- ings of the IEEE conference on computer vision and pattern recognition workshops, pages 126–135, 2017. 6

    2017

  4. [4]

    Generative adversarial networks for extreme learned image compression

    Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Generative adversarial networks for extreme learned image compression. In ICCV, pages 221–231, 2019. 1, 3

    2019

  5. [5]

    Spatext: Spatio-textual representation for con- trollable image generation

    Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for con- trollable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18370–18380, 2023. 5

    2023

  6. [6]

    Variational image compression with a scale hyperprior

    Johannes Ball ´e, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In ICLR, 2018. 1

    work page 2018

  7. [7]

    Compressai: a pytorch library and evaluation platform for end-to-end compression research

    Jean B´egaint, Fabien Racap´e, Simon Feltman, and Akshay Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020. 1

    arXiv 2011

  8. [8]

    Better portable graphics (bpg) image format

    Fabrice Bellard. Better portable graphics (bpg) image format. https://bellard.org/bpg/. Accessed: 2024-05-22. 1, 3

    work page 2024

Show all 85 references
  1. [9]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 1, 4

    2023

  2. [10]

    The perception-distortion tradeoff

    Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6228–6237, 2018. 3

    2018

  3. [11]

    Rethinking lossy compres- sion: The rate-distortion-perception tradeoff

    Yochai Blau and Tomer Michaeli. Rethinking lossy compres- sion: The rate-distortion-perception tradeoff. In International Conference on Machine Learning, pages 675–685. PMLR,

  4. [12]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 5

    2023

  5. [13]

    Overview of the versatile video coding (vvc) standard and its applications

    Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (vvc) standard and its applications. TCSVT, 2021. 2, 3, 5, 7, 1

    2021

  6. [14]

    Towards image compression with per- fect realism at ultra-low bitrates

    Marl`ene Careil, Matthew J Muckley, Jakob Verbeek, and St´ephane Lathuili`ere. Towards image compression with per- fect realism at ultra-low bitrates. In The Twelfth International Conference on Learning Representations, 2023. 1, 3, 5

    2023

  7. [15]

    Thousand to one: Semantic prior modeling for conceptual coding

    Jianhui Chang, Zhenghui Zhao, Lingbo Yang, Chuanmin Jia, Jian Zhang, and Siwei Ma. Thousand to one: Semantic prior modeling for conceptual coding. In 2021 IEEE international conference on multimedia and expo (ICME), pages 1–6. IEEE,

    2021

  8. [16]

    Learned image compression with discretized gaussian mixture likelihoods and attention modules

    Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In CVPR, pages 7939–7948, 2020. 1, 2, 5, 6

    2020

  9. [17]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 5

    2021

  10. [18]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim En- tezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024. 5

    2024 arXiv

  11. [19]

    Semantically structured image compression via irregu- lar group-based decoupling

    Ruoyu Feng, Yixin Gao, Xin Jin, Runsen Feng, and Zhibo Chen. Semantically structured image compression via irregu- lar group-based decoupling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17237– 17247, 2023. 1

    2023

  12. [20]

    Nvtc: Nonlinear vector transform coding

    Runsen Feng, Zongyu Guo, Weiping Li, and Zhibo Chen. Nvtc: Nonlinear vector transform coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6101–6110, 2023. 1

    2023

  13. [21]

    Rate-distortion optimization for cross modal com- pression

    Junlong Gao, Chuanmin Jia, Shanshe Wang, Siwei Ma, and Wen Gao. Rate-distortion optimization for cross modal com- pression. In 2023 Data Compression Conference (DCC) , pages 218–227. IEEE, 2023. 3

    2023

  14. [22]

    A residual diffusion model for high perceptual quality codec augmentation

    Noor Fathima Ghouse, Jens Petersen, Auke Wiggers, Tianlin Xu, and Guillaume Sautiere. A residual diffusion model for high perceptual quality codec augmentation. arXiv preprint arXiv:2301.05489, 2023. 3

    2023 arXiv

  15. [23]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27,

  16. [24]

    Theoretical foundations of transform coding

    Vivek K Goyal. Theoretical foundations of transform coding. IEEE Signal Processing Magazine, 18(5):9–21, 2001. 3

    2001

  17. [25]

    Causal contextual prediction for learned image compression

    Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Causal contextual prediction for learned image compression. TCSVT, 32(4):2329–2341, 2021. 1

    2021

  18. [26]

    Soft then hard: Rethinking the quantization in neural image compression

    Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Soft then hard: Rethinking the quantization in neural image compression. In ICML, pages 3920–3929. PMLR, 2021. 1

    2021

  19. [27]

    Minimal random code learning: Getting bits back from compressed model parameters

    Marton Havasi, Robert Peharz, and Jos´e Miguel Hern´andez- Lobato. Minimal random code learning: Getting bits back from compressed model parameters. 2019. 3 9

    2019

  20. [28]

    Elic: Efficient learned image com- pression with unevenly grouped space-channel contextual adaptive coding

    Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elic: Efficient learned image com- pression with unevenly grouped space-channel contextual adaptive coding. In CVPR, pages 5718–5727, 2022. 1, 2, 6

    2022

  21. [29]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 6

    2017

  22. [30]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 4

    2021

  23. [31]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  24. [32]

    High-fidelity im- age compression with score-based generative models

    Emiel Hoogeboom, Eirikur Agustsson, Fabian Mentzer, Luca Versari, George Toderici, and Lucas Theis. High-fidelity im- age compression with score-based generative models. arXiv preprint arXiv:2305.18231, 2023. 3

    2023 arXiv

  25. [33]

    Multi-modal sensor fusion for auto driving perception: A survey

    Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, and Yikang Li. Multi-modal sensor fusion for auto driving perception: A survey. arXiv preprint arXiv:2202.02703, 2022. 3

    2022 arXiv

  26. [34]

    Multi-modality deep network for extreme learned im- age compression

    Xuhao Jiang, Weimin Tan, Tian Tan, Bo Yan, and Liquan Shen. Multi-modality deep network for extreme learned im- age compression. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1033–1041, 2023. 3

    2023

  27. [35]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  28. [36]

    The open im- ages dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open im- ages dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  29. [37]

    Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity

    Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, and Jaeho Lee. Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity. arXiv preprint arXiv:2403.02944, 2024. 1, 3

    2024 arXiv

  30. [38]

    Text + sketch: Image compression at ultra low rates

    Eric Lei, Yigit Berkay Uslu, Hamed Hassani, and Shirin Saeedi Bidokhti. Text + sketch: Image compression at ultra low rates. In ICML 2023 Workshop Neural Compression: From Information Theory to Applications, 2023. 1, 3, 5

    2023

  31. [39]

    Misc: Ultra-low bitrate image semantic com- pression driven by large multimodal model

    Chunyi Li, Guo Lu, Donghui Feng, Haoning Wu, Zicheng Zhang, Xiaohong Liu, Guangtao Zhai, Weisi Lin, and Wen- jun Zhang. Misc: Ultra-low bitrate image semantic com- pression driven by large multimodal model. arXiv preprint arXiv:2402.16749, 2024. 3

    2024 arXiv

  32. [40]

    Cross modal compression: Towards human- comprehensible semantic compression

    Jiguo Li, Chuanmin Jia, Xinfeng Zhang, Siwei Ma, and Wen Gao. Cross modal compression: Towards human- comprehensible semantic compression. In Proceedings of the 29th ACM international conference on multimedia, pages 4230–4238, 2021. 3

    2021

  33. [41]

    Task-driven semantic coding via reinforcement learning

    Xin Li, Jun Shi, and Zhibo Chen. Task-driven semantic coding via reinforcement learning. IEEE Transactions on Image Processing, 30:6307–6320, 2021. 1

    2021

  34. [42]

    Diffusion models for image restoration and enhancement–a comprehensive sur- vey

    Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wen- jun Zeng, Xinchao Wang, and Zhibo Chen. Diffusion models for image restoration and enhancement–a comprehensive sur- vey. arXiv preprint arXiv:2308.09388, 2023. 3

    2023

  35. [43]

    Ucip: A universal framework for compressed image super-resolution using dynamic prompt

    Xin Li, Bingchen Li, Yeying Jin, Cuiling Lan, Hanxin Zhu, Yulin Ren, and Zhibo Chen. Ucip: A universal framework for compressed image super-resolution using dynamic prompt. In ECCV, 2024. 5

    2024

  36. [44]

    Learned image compression with mixed transformer-cnn architectures

    Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer-cnn architectures. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14388–14397, 2023. 1

    2023

  37. [45]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022. 8

    2022

  38. [46]

    Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022. 8

    2022 arXiv

  39. [47]

    Latent consistency models: Synthesizing high- resolution images with few-step inference

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023. 8

    2023 arXiv

  40. [48]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297–14306, 2023. 8

    2023

  41. [49]

    High-fidelity generative image compres- sion

    Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compres- sion. arXiv preprint arXiv:2006.09965, 2020. 1, 3, 6

    2006 arXiv

  42. [50]

    Channel-wise autoregres- sive entropy models for learned image compression

    David Minnen and Saurabh Singh. Channel-wise autoregres- sive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP), pages 3339–3343. IEEE, 2020. 1

    2020

  43. [51]

    Joint autoregressive and hierarchical priors for learned image com- pression

    David Minnen, Johannes Ball´e, and George Toderici. Joint autoregressive and hierarchical priors for learned image com- pression. In NeurIPS, 2018. 1, 2, 5, 6

    2018

  44. [52]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024. 5

    2024

  45. [53]

    Neuralcompres- sion

    Matthew Muckley, Jordan Juravsky, Daniel Severo, Mannat Singh, Quentin Duval, and Karen Ullrich. Neuralcompres- sion. https://github.com/facebookresearch/ NeuralCompression, 2021. 1

    2021

  46. [54]

    Improving statistical fidelity for neural image compression with implicit local likelihood models

    Matthew J Muckley, Alaaeldin El-Nouby, Karen Ullrich, Herv´e J´egou, and Jakob Verbeek. Improving statistical fidelity for neural image compression with implicit local likelihood models. In International Conference on Machine Learning, pages 25426–25443. PMLR, 2023. 1, 2, 3, 6, 7

    2023

  47. [55]

    Extreme generative image compression by learning text embedding from diffusion models

    Zhihong Pan, Xin Zhou, and Hao Tian. Extreme generative image compression by learning text embedding from diffusion models. arXiv preprint arXiv:2211.07793, 2022. 3 10

    2022 arXiv

  48. [56]

    Perceptual image compression with co- operative cross-modal side information

    Shiyu Qin, Bin Chen, Yujun Huang, Baoyi An, Tao Dai, and Shu-Tao Via. Perceptual image compression with co- operative cross-modal side information. arXiv preprint arXiv:2311.13847, 2023. 1, 3

    2023 arXiv

  49. [57]

    An overview of the jpeg 2000 still image compression standard

    Majid Rabbani and Rajan Joshi. An overview of the jpeg 2000 still image compression standard. Signal processing: Image communication, 17(1):3–48, 2002. 3

    2000

  50. [58]

    Moe-diffir: Task- customized diffusion priors for universal compressed image restoration

    Yulin Ren, Xin Li, Bingchen Li, Xingrui Wang, Mengxi Guo, Shijie Zhao, Li Zhang, and Zhibo Chen. Moe-diffir: Task- customized diffusion priors for universal compressed image restoration. In European Conference on Computer Vision , pages 116–134. Springer, 2025. 5

    2025

  51. [59]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 1, 2, 3, 4, 5

    2022

  52. [60]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 8

    2022 arXiv

  53. [61]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3

    2015

  54. [62]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 6

    2021

  55. [63]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 3

    2011 arXiv

  56. [64]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 8

    2023 arXiv

  57. [65]

    Overview of the high efficiency video coding (hevc) standard

    Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (hevc) standard. TCSVT, 22(12):1649–1668, 2012. 2, 3, 5, 1

    2012

  58. [66]

    Semantic structured image coding framework for multiple intelligent applications

    Simeng Sun, Tianyu He, and Zhibo Chen. Semantic structured image coding framework for multiple intelligent applications. TCSVT, 2020. 1

    2020

  59. [67]

    Algorithms for the communication of samples

    Lucas Theis and Noureldin Y Ahmed. Algorithms for the communication of samples. In International Conference on Machine Learning, pages 21308–21328. PMLR, 2022. 3

    2022

  60. [68]

    Lossy compression with gaussian diffusion

    Lucas Theis, Tim Salimans, Matthew D Hoffman, and Fabian Mentzer. Lossy compression with gaussian diffusion. arXiv preprint arXiv:2206.08889, 2022. 3

    2022 arXiv

  61. [69]

    Workshop and challenge on learned image compres- sion (clic2020), 2020

    George Toderici, Wenzhe Shi, Radu Timofte, Lucas Theis, Jo- hannes Balle, Eirikur Agustsson, Nick Johnston, and Fabian Mentzer. Workshop and challenge on learned image compres- sion (clic2020), 2020. 6, 1

    2020

  62. [70]

    Multimodal research in vision and language: A review of current and emerging trends

    Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmermann, and Amir Zadeh. Multimodal research in vision and language: A review of current and emerging trends. Information Fusion, 77:149–171, 2022. 3

    2022

  63. [71]

    The jpeg still picture compression stan- dard

    Gregory K Wallace. The jpeg still picture compression stan- dard. IEEE transactions on consumer electronics , 38(1): xviii–xxxiv, 1992. 1, 2, 3, 5

    1992

  64. [72]

    Chan, and Chen Change Loy

    Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin C.K. Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. International Journal of Computer Vision, 2024. 5, 1

    2024

  65. [73]

    Cogvlm: Visual expert for pretrained language models, 2023

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023. 5

    2023

  66. [74]

    Learned block-based hybrid image compression

    Yaojun Wu, Xin Li, Zhizheng Zhang, Xin Jin, and Zhibo Chen. Learned block-based hybrid image compression. IEEE Transactions on Circuits and Systems for Video Technology, 32(6):3978–3990, 2021. 1

    2021

  67. [75]

    Idempotence and perceptual image compression

    Tongda Xu, Ziran Zhu, Dailan He, Yanghao Li, Lina Guo, Yuanyuan Wang, Zhe Wang, Hongwei Qin, Yan Wang, Jingjing Liu, and Ya-Qin Zhang. Idempotence and perceptual image compression. In The Twelfth International Conference on Learning Representations, 2024. 1, 3

    2024

  68. [76]

    On perceptual lossy compression: The cost of perceptual reconstruction and an optimal training framework

    Zeyu Yan, Fei Wen, Rendong Ying, Chao Ma, and Peilin Liu. On perceptual lossy compression: The cost of perceptual reconstruction and an optimal training framework. In Interna- tional Conference on Machine Learning, pages 11682–11692. PMLR, 2021. 7

    2021

  69. [77]

    Lossy image compres- sion with conditional diffusion models

    Ruihan Yang and Stephan Mandt. Lossy image compres- sion with conditional diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 6

    2024

  70. [78]

    Improving inference for neural image compression

    Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. NeurIPS, 33:573– 584, 2020. 1

    2020

  71. [79]

    Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild

    Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiang- tao Kong, Xintao Wang, Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint arXiv:2401.13627, 2024. 1

    2024 arXiv

  72. [80]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 5

    2023

  73. [81]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595,

  74. [82]

    Transformer-based transform coding

    Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-based transform coding. In ICLR, 2022. 1

    2022

  75. [83]

    The devil is in the details: Window-based attention for image compression

    Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. The devil is in the details: Window-based attention for image compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17492–17501,

  76. [85]

    Further Experimental Details Implementation Details

    Experiments 7.1. Further Experimental Details Implementation Details. The visual codec repository composes eight representative codecs in training: 1) tra- ditional codecs: JPEG [ 71] (QF ∈ {5, 10, 15, 20}), HM- intra 18.0 [ 65] (QP ∈ {47, 42, 37}), VTM-intra 23.1 [ 13] (QP∈ {...

  77. [2022]

    This includes the further experi- mental details, additional experimental results

    1, 2 11 UniMIC: Towards Universal Multi-modality Perceptual Image Compression Supplementary Material In this document, we provide the supplementary material for the proposed universal multi-modality image compres- sion framework, UniMIC. This includes the further experi- menta...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.