Pith. sign in

REVIEW 4 major objections 5 minor 27 references

LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single large multimodal model can generate and compress image captions together, saving 65% of text bits while semantic-aware training cuts image bitrate by 41.58%.

desk verdict The LMM-driven caption compression path is a real contribution, but the headline BD-rate gain is not attributable to the proposed semantic-perceptual loss and needs a controlled re-baselining. read the letter →

arxiv 2411.13033 v1 pith:JAWA76FO submitted 2024-11-20 eess.IV cs.CV

Shimon Murai , Heming Sun , Jiro Katto This is my paper · ORCID
classification eess.IVcs.CV
keywords learnedimagecompressionlargemultimodalmodelcaptionsemantic-perceptuallosslatentdiffusionultralowbitrateCLIPLPIPS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that ultra-low-bitrate image compression can use one large multimodal model (LMM) for both caption generation and caption compression, replacing the separate text codec used by earlier text-conditioned methods. The authors also claim that fine-tuning the image codec with a weighted mixture of pixel, perceptual, and semantic losses—rather than pixel error alone—shifts the rate-distortion curve in the right direction. If true, text-conditioned generative compression becomes simpler and cheaper, and the caption model's own next-token probabilities become a usable entropy code for the caption. The headline quantitative claims are a 65.14% text bit saving from rank encoding plus gzip, and 41.58% LPIPS and 60.99% CLIP image-to-image bitrate savings against the prior method.

What carries the argument

The load-bearing mechanism is the reuse of next-token prediction: the LMM both writes the caption and acts as an entropy model for it, since its output probabilities define the rank code. The second mechanism is a semantic-perceptual loss mixture that balances pixel fidelity against perceptual and semantic objectives during fine-tuning of the image codec, with a frozen diffusion model reconstructing the final image from the compressed image plus caption.

What would settle it

Re-running the released code and weights against the prior method with the prior method's full diffusion pipeline, identical prompts, and identical rate accounting would settle the central claim: if the 41.58% LPIPS and 60.99% CLIP savings collapse to near zero, the improvement is an artifact of the configuration mismatch.

Watch

Extended reading notes

Core claim

The central discovery is that the LMM used to generate an image caption also supplies, for free, a conditional probability distribution over the next token at every step; replacing each token by its rank in that sorted distribution and gzipping the rank sequence yields a lossless text code with 65.14% fewer bits than the raw caption. On the image side, training the learned image codec with a normalized sum of MSE, LPIPS, a CLIP-based image-quality score, and a CLIP image-to-text score—instead of MSE alone—produces reconstructions that score better on LPIPS and CLIP image-to-image similarity at the same bitrate, with the largest reported gap coming when both semantic losses and text compression are active.

Load-bearing premise

The reported gains assume the prior method was evaluated with the same bitrate accounting, prompts, and diffusion settings, and that omitting its three-stage diffusion refinement—which the paper states but does not ablate—does not change comparative quality.

Editorial extensions

If this is right

  • Text-conditioned ultra-low-bitrate compression no longer needs a separate text codec; caption generation and entropy coding live in the same frozen LMM.
  • Because the LMM and diffusion model are frozen, the semantic-perceptual fine-tuning can be grafted onto any learned image codec, not just the one used here.
  • The rank-encoding scheme converts every generated token into a small integer, so text overhead stays low even when captions are long.
  • At equal bitrate, the method reports better LPIPS and CLIP image-to-image similarity than the prior text-conditioned baseline, implying better perceptual and semantic fidelity in the ultra-low-bitrate regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same rank-encoding trick should apply to any structured output an LMM emits—sketches, layouts, audio captions, or code—wherever the receiver can run the same model.
  • If semantic losses prevent overfitting to a single network-based metric, then similar loss mixtures could stabilize other perceptually trained codecs and generative compressors.
  • A natural extension is to let the decoder's diffusion model choose how much detail the caption must carry, trading caption bits against image bits at fixed total rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a two-part system for ultra-low-bitrate learned image compression: (i) an LMM-driven text compression path that entropy-codes caption tokens by ranking them with LLaVA's next-token probabilities and then gzipping the rank sequence, and (ii) a semantic-perceptual fine-tuning loss for the image coding network that combines MSE, LPIPS, CLIP-IQA, and CLIP image-to-text similarity. The reported claims are a 65.14% text bit saving, a 41.58% LPIPS BD-rate saving and a 60.99% CLIP image-to-image BD-rate saving against MISC, with code and pretrained weights released.

Significance. The central empirical claim is the 41.58%/60.99% BD-rate improvement over MISC; if this result is cleanly supported, the paper would demonstrate that a single LMM can serve as both caption generator and entropy model, and that adding semantic and perceptual losses improves ultra-low-bitrate reconstruction quality. The text-compression component is concrete, independently testable, and does not suffer from the metric-circularity issue that affects the image-loss evaluation. The public release of implementation and pretrained weights is a clear strength and should allow the community to verify the comparisons. However, the current manuscript does not isolate the source of the headline BD-rate gains, because the reported number includes the separate text-compression path and because the MISC baseline is evaluated in a modified configuration.

major comments (4)
  1. [Section IV-C and Table II] The abstract and Section IV-C attribute the 41.58% LPIPS BD-rate improvement to the proposed semantic-perceptual fine-tuning, but Table II shows that the fine-tuning component alone yields -38.07% on LPIPS without text compression, while perceptual-only loss yields -39.76%. The headline -41.58% therefore includes the separate text-compression path, and on LPIPS the semantic loss is slightly worse than the perceptual-only loss. The same issue appears on CLIP I2I, where the text-compression path contributes roughly 3.5 percentage points. Please decompose the reported gain into the contributions from (a) perceptual loss, (b) semantic loss, and (c) text compression, and rephrase the abstract, Section IV-C, and conclusion so that the 41.58% figure is not attributed to the fine-tuning method alone.
  2. [Section III-A] The MISC anchor is evaluated with its original three-stage diffusion process omitted, with the statement that it 'does not contribute to the image quality,' but no ablation or point-wise rate-distortion data is provided to justify that omission. Since every reported BD-rate saving is computed against MISC, a configuration-mismatched baseline can change the magnitude or even the sign of the gains. Please either re-run MISC with its full diffusion pipeline under identical rate accounting, prompts, and decoding settings, or provide a per-image RD comparison with and without the omitted diffusion stages for both methods.
  3. [Equation (2) and Section V] The image-coding network is trained by minimizing a loss that directly includes LPIPS, CLIP-IQA, and CLIP image-to-text similarity, and then the method is evaluated on LPIPS and CLIP I2I similarity. The reported BD-rate gains are therefore partially circular for the image-compression component, and the hand-set weights kappa_0..kappa_3 in Equation (2) are not analyzed for sensitivity. The text-compression result is not affected by this concern, but the image-coding claim should be supported by at least one metric not present in the training loss, such as FID, DISTS, or a subjective evaluation, and by a sensitivity study over the kappa values.
  4. [Table I and Section III-B] The text-compression result is reported as total bits and a compression ratio, but the manuscript does not state the number of captions evaluated, the average caption token length, the size of the token vocabulary used for ranking, whether the end-of-sequence token is included, or how the rank probabilities are converted into the gzip input. These details matter because LLMZip-style rank coding is sensitive to tokenization and to the underlying probability model; without them the 65.14% figure cannot be independently reproduced or compared with other text-compression approaches. Please report the average caption length, bit allocation between image and text streams, and the exact entropy-coding pipeline.
minor comments (5)
  1. [Section III-B] The text says the rank sequence is encoded with 'glib (adaptive Huffman coding)' but the rest of the paper and Table I refer to gzip; please unify the terminology and describe the actual implementation.
  2. [Section II-B] There is a typo: 'LIPIS-VGG' should be 'LPIPS-VGG'.
  3. [Section IV-A] The caption-generation prompt and question template are not described in the paper and are only referenced to the GitHub repository; please include the exact prompt wording used for the 50-word caption generation.
  4. [Figures 4 and 5] The RD curves are shown without point markers or error bars; providing per-image scatter or the number of test images would make the BD-rate comparison more transparent, especially because generative baselines can produce high-variance LPIPS values at low bitrates.
  5. [Abstract] The phrase 'Clip Image-to-Image Similarity' is capitalized inconsistently with 'CLIP' used elsewhere in the paper; please standardize the notation.

Circularity Check

1 steps flagged · score 5.0 of 10

LPIPS/CLIP BD-rate gains partly reduce to the training loss; the text-compression result is independent.

  1. fitted input called prediction [Section III-C (Eq. 2) and Section IV-C]
    "Our loss function combines MSE and LPIPS-VGG [11] as perceptual losses with additional CLIP-IQA [24] loss and CLIP image-to-text score [22] as semantic losses. ... For the evaluation metric, we selected LPIPS [11] and CLIP image-to-image similarity [22]."

    The reported LPIPS and CLIPI2I BD-rate gains are measured on the same metric families that appear in the training loss. Equation (2) minimizes D = κ0LMSE + κ1LLPIPS + κ2LCLIPIQA + κ3LCLIPI2T, so the fine-tuned network is explicitly optimized to reduce LPIPS and CLIP-space distances. Reporting LPIPS/CLIPI2I improvement over an MSE-trained baseline (MISC) is therefore a restatement of the training objective, not an independent prediction. The ablation confirms this: the perceptual-only variant (LPIPS loss) already achieves -39.76% LPIPS BD-rate, and adding the semantic loss actually reduces the gain to -38.07%; the remaining headline margin comes from the separate text-compression module.

full rationale

The paper has two central claims: LMM-driven text compression and semantic-perceptual fine-tuning. The text-compression claim (65.14% ratio) is a straightforward bit-count measurement on captions generated and compressed by the same LMM; it does not depend on the evaluation metrics and is not circular. The image-compression claim, however, is partially circular because the loss function (Eq. 2) includes LPIPS and CLIP-based terms, and the evaluation then uses LPIPS and CLIPI2I. The ablation in Table II shows that the perceptual-only model already attains most of the LPIPS gain, and the semantic loss slightly reduces it, so the headline 41.58% LPIPS gain is largely the expected effect of training on LPIPS plus the separate text bitrate reduction. The self-citations to Cheng [18] and the survey [9] are not load-bearing: the method is stated to be applicable to any LIC network, and the backbone choice does not drive the claimed gains. The omission of MISC's three-stage diffusion is an experimental-fairness concern, not circularity. Overall, the text-compression contribution is independent and non-circular, while the perceptual-metric headline partially reduces to the training objective.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities. The ledger shows hand-set loss weights and reliance on frozen pretrained models as the main external inputs; the text compression ratio is an independent engineering result.

free parameters (2)
  • Loss weights kappa_0..kappa_3 = (0.5, 0.2, 0.2, 0.1)
    Hand-chosen weights in Eq. (2) balancing MSE, LPIPS, CLIP-IQA, and CLIP-I2T; no sensitivity analysis or search is reported (Section IV-B).
  • Rate-distortion tradeoff lambda = {1, 2, 3, 4}
    Four target-rate values used to train separate models for the R-D curve; standard in compression but not optimized against evaluation data.
assumptions (4)
  • domain assumption LLaVA's next-token probabilities are accurate conditional distributions for entropy coding of its own captions (Section III-B).
    The entire text compression path relies on this; no calibration analysis is provided.
  • domain assumption LPIPS-VGG, CLIP-IQA, and CLIP image-to-text scores reflect perceptual and semantic quality and are suitable training targets (Sections III-C, IV-C).
    Used both in loss and evaluation, which partially aligns optimization with the benchmark.
  • domain assumption Frozen DiffBIR-v1 diffusion model restores images from decompressed latents and captions without retraining (Section III-A).
    The LIC network is the only trainable component; diffusion quality is taken as given.
  • ad hoc to paper MISC's three-stage diffusion process does not contribute to image quality (Section III-A).
    No ablation or experiment backs this; it is used to drop a baseline component.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression." pith.science (2026). https://pith.science/paper/JAWA76FO

@misc{pith2026241113033,
  author       = {Pith},
  title        = {Pith review of: LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAWA76FO}},
  note         = {Machine review of arXiv:2411.13033}
}
read the original abstract

Supported by powerful generative models, low-bitrate learned image compression (LIC) models utilizing perceptual metrics have become feasible. Some of the most advanced models achieve high compression rates and superior perceptual quality by using image captions as sub-information. This paper demonstrates that using a large multi-modal model (LMM), it is possible to generate captions and compress them within a single model. We also propose a novel semantic-perceptual-oriented fine-tuning method applicable to any LIC network, resulting in a 41.58\% improvement in LPIPS BD-rate compared to existing methods. Our implementation and pre-trained weights are available at https://github.com/tokkiwa/ImageTextCoding.

Figures

Figures reproduced from arXiv: 2411.13033 by the authors.

Figure 1
Figure 1. Compression results of Kodim15.png [8] with a zoom-in. Our model [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our network architecture. The image is compressed to image bitstream with LIC model (above path), and at the same time, transported to LMM [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The visualization of LMM text compression. [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Relationship between bpp and CLIP Similarity. [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: The visualization of kodim19.png [8] encoded with the perceptual [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 13 canonical work pages

  1. [1]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022

    2022

  2. [2]

    Generative adversarial nets,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” Advances in neural information processing systems , vol. 27, 2014

    2014

  3. [3]

    MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model,

    C. Li, G. Lu, D. Feng, H. Wu, Z. Zhang, X. Liu, G. Zhai, W. Lin, and W. Zhang, “MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model,” arXiv preprint arXiv:2402.16749, no. arXiv:2402.16749, 2024

    arXiv 2024

  4. [4]

    Text + sketch: Image compression at ultra low rates,

    E. Lei, Y . B. Uslu, H. Hassani, and S. S. Bidokhti, “Text + sketch: Image compression at ultra low rates,” ICML 2023 Workshop Neural Compression: From Information Theory to Applications , 2023

    work page 2023

  5. [5]

    Towards image compression with perfect realism at ultra-low bitrates,

    M. Careil, M. J. Muckley, J. Verbeek, and S. Lathuili`ere, “Towards image compression with perfect realism at ultra-low bitrates,” in The Twelfth International Conference on Learning Representations , 2024

    work page 2024

  6. [6]

    LLMZip: Lossless Text Compression using Large Language Models,

    C. S. K. Valmeekam, K. Narayanan, D. Kalathil, J.-F. Chamberland, and S. Shakkottai, “LLMZip: Lossless Text Compression using Large Language Models,” arXiv preprint arXiv:2306.04050 , June 2023

    arXiv 2023

  7. [7]

    Language modeling is compression,

    G. Deletang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mat- tern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness, “Language modeling is compression,” in The Twelfth International Conference on Learning Representations , 2024

    work page 2024

  8. [8]

    Kodak lossless true color image suite

    “Kodak lossless true color image suite.” https://r0k.us/graphics/kodak/

    work page

Show all 27 references
  1. [9]

    Survey on visual signal coding and processing with generative models: Technologies, standards, and optimization,

    Z. Chen, H. Sun, L. Zhang, and F. Zhang, “Survey on visual signal coding and processing with generative models: Technologies, standards, and optimization,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 14, no. 2, pp. 149–171, 2024

    2024

  2. [10]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Advances in Neural Information Processing Systems , vol. 30, Curran Associates, Inc., 2017

    2017

  3. [11]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595, 2018

    2018

  4. [12]

    High- Fidelity Generative Image Compression,

    F. Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson, “High- Fidelity Generative Image Compression,” in Advances in Neural Infor- mation Processing Systems , vol. 33, pp. 11913–11924, Curran Asso- ciates, Inc., 2020

    2020

  5. [13]

    Po-elic: Perception-oriented efficient learned image coding,

    D. He, Z. Yang, H. Yu, T. Xu, J. Luo, Y . Chen, C. Gao, X. Shi, H. Qin, and Y . Wang, “Po-elic: Perception-oriented efficient learned image coding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pp. 1764–1769, 2022

    2022

  6. [14]

    Fidelity-controllable extreme image compression with generative adversarial networks,

    S. Iwai, T. Miyazaki, Y . Sugaya, and S. Omachi, “Fidelity-controllable extreme image compression with generative adversarial networks,” in 2020 25th International Conference on Pattern Recognition (ICPR) , pp. 8235–8242, IEEE, 2021

    2020

  7. [15]

    Multi-realism image compression with a conditional generator,

    E. Agustsson, D. Minnen, G. Toderici, and F. Mentzer, “Multi-realism image compression with a conditional generator,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 22324–22333, 2023

    2023

  8. [16]

    Hoogeboom, E

    E. Hoogeboom, E. Agustsson, F. Mentzer, L. Versari, G. Toderici, and L. Theis, High-Fidelity Image Compression with Score-based Generative Models. May 2023

    2023

  9. [17]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  10. [18]

    Learned image com- pression with discretized gaussian mixture likelihoods and attention modules,

    Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image com- pression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 7939–7948, 2020

    2020

  11. [19]

    Diffbir: Towards blind image restoration with generative diffusion prior,

    X. Lin, J. He, Z. Chen, Z. Lyu, B. Dai, F. Yu, W. Ouyang, Y . Qiao, and C. Dong, “Diffbir: Towards blind image restoration with generative diffusion prior,” arXiv preprint arXiv:2308.15070 , 2024

    2024 arXiv

  12. [20]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023

    2023

  13. [21]

    LLaV A-UHD: an lmm perceiving any aspect ratio and high- resolution images,

    R. Xu, Y . Yao, Z. Guo, J. Cui, Z. Ni, C. Ge, T.-S. Chua, Z. Liu, and G. Huang, “LLaV A-UHD: an lmm perceiving any aspect ratio and high- resolution images,” arXiv preprint arXiv:2403.11703 , 2024

    2024 arXiv

  14. [22]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” arXiv preprint arXiv:2103.00020 , 2021

    2021 arXiv

  15. [23]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  16. [24]

    Exploring clip for assessing the look and feel of images,

    J. Wang, K. C. Chan, and C. C. Loy, “Exploring clip for assessing the look and feel of images,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 2555–2563, Jun. 2023

    2023

  17. [25]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Doll ´ar, “Microsoft coco: Common objects in context,” arXiv preprint arXiv:1405.0312 , 2015

    2015 arXiv

  18. [26]

    3rd workshop and challenge on learned image compression

    “3rd workshop and challenge on learned image compression.” https: //clic.compression.cc/2021/tasks/index.html

    2021

  19. [27]

    Calculation of average psnr differences between rd- curves,

    G. Bjøntegaard, “Calculation of average psnr differences between rd- curves,” 2001

    2001

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.