Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models

Deniz Gunduz; Haotian Wu; Jiangjing Hu; Ruiqi Shen; Wenjing Zhang

REVIEW 4 major objections 5 minor 21 references

Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes a reconstruction-free scheme that compresses CLIP image embeddings into about 400 bits per image, preserving zero-shot classification accuracy and using under 5% of the bitrate of learned image codecs at comparable accura

desk verdict The semantic compression idea is promising, but the reported bitrates are not reproducible as written; needs a rate-accounting revision before the headline claim is credible. read the letter →

arxiv 2509.05925 v1 pith:KYKQSMF3 submitted 2025-09-07 cs.CV cs.ITmath.IT

Ruiqi Shen , Haotian Wu , Wenjing Zhang , Jiangjing Hu , Deniz Gunduz This is my paper

classification cs.CVcs.ITmath.IT

keywords semanticcompressionCLIPproductquantizationsharedcodebookzero-shotclassificationmultimodalfoundationmodelsfeaturelearning-basedimage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that for tasks which consume image semantics rather than pixels, you do not need to reconstruct the image. It proposes compressing the feature embedding produced by CLIP—a multimodal model that maps images and text to a shared space—down to roughly 400 bits per image (0.52 bits per dimension), and claims the compressed embedding still supports zero-shot classification, captioning, and referring-object identification nearly as well as the original 32-bit-per-dimension embedding. Across benchmarks the scheme runs at 2–3×10^(-3) bits per pixel, which the paper reports as less than 5% of the bitrate needed by learned image codecs for comparable accuracy. A reader should care because this indicates a path toward sending meaning, not pixels, in rate-limited edge systems, at the cost of assuming the receiver already has the same CLIP model.

What carries the argument

The engine is PQVAE-shared, a product-quantization variational autoencoder with a single shared codebook. An encoder expands the 768-dim CLIP embedding into a 5×5×128 latent; each of the 128-dim channels is split into d subspaces; each subvector is replaced by the nearest entry of the shared codebook, producing integer indices that Huffman coding turns into a bitstream; the decoder reverses this. Training minimizes cosine distance between reconstructed and original CLIP features plus VQ codebook and commitment losses, so the discrete codebook is optimized for semantic fidelity, not pixel fidelity.

What would settle it

Take a dataset whose labels hinge on information CLIP is known to miss, such as fine-grained bird subspecies or images where two objects have identical CLIP semantics but different spatial relations, compress embeddings to 400 bits, and measure downstream accuracy versus cosine similarity. If accuracy collapses even when the reconstructed embedding's cosine similarity to the original stays above 0.99, the alignment assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that semantic content, as represented by CLIP, has an extremely low intrinsic bitrate: a 768-dimensional CLIP feature can be product-quantized and entropy-coded to about 400 bits (0.52 bpd) while retaining 81.81% zero-shot ImageNet accuracy against an 84.31% upper bound at a slightly higher rate, and roughly the same behavior across OxfordPets, Caltech101, Tiny-ImageNet, and Food101. Lower bitrates trade smoothly against accuracy, and the same compressed features feed downstream captioning and referring-object identification. The paper argues this makes semantic compression task-agnostic: unlike task-specific compressors, the same bitstream serves multiple tasks as long

Load-bearing premise

The load-bearing premise is that the geometric nearness of compressed image features to the original features, as measured by CLIP, is the right proxy for every downstream task; any information CLIP does not preserve in that geometry is simply discarded.

Editorial extensions

If this is right

Zero-shot classifiers that consume CLIP features can run on 400-bit image representations with near-original accuracy, lowering bandwidth in distributed recognition.
The same compressed bitstream is not tied to one task: classification, captioning, and referring-object identification all read recovered features, so one codec can replace multiple task-specific codecs.
Learned image codecs are off the table for semantic tasks: at equal accuracy the scheme uses about 2–3% of their bits per pixel on OxfordPets and Food101.
The rate-accuracy curve is smooth and controllable via codebook size and subspace count, letting systems pick an operating point per channel conditions.
Semantic compression doubles as a privacy mechanism: because pixels are never sent or reconstructed, visual reconstruction is prevented.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The practical bitrate comparison quietly assumes the CLIP model is already deployed at the receiver; if that model must be transmitted or licensed, the apparent 20–30x savings shrink.
The method's quality ceiling is exactly CLIP's semantics: any downstream task that depends on information CLIP's cosine geometry does not encode—fine-grained attributes, spatial layout, text inside images—will fail regardless of rate; this predicts where the approach breaks.
The same PQVAE-shared training recipe should transfer to newer multimodal embeddings, with the operating point shifting; whether 400 bits remains enough depends on how information-dense the new embedding is.
A testable extension: allocate bits adaptively per subspace by measuring each subspace's marginal contribution to downstream accuracy rather than uniform codebook size, likely improving the low-rate frontier.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is a plausible but not yet fully auditable result. The authors compress 768-dim CLIP embeddings to 0.52 bpd (400 bits/image) with only small zero-shot accuracy drops, using a product-quantization VAE with a shared codebook. The scheme is new relative to the prior work they cite. But the stress-test note is correct: every bpd number in the paper exactly matches the pre-entropy index cost defined in §3.2, and the bpp denominators are never specified. So the headline "2–3×10^-3 bpp" and "less than 5%" figure need re-checking before they can be taken at face value.

What the paper does well: it trains on ImageNet, evaluates on four datasets at multiple rate points, and compares against scalar quantization and VQ-VAE. The shared-codebook design is a practical fix for the exponential codebook growth of standard VQ-VAE, and the ablation on subspace decomposition shows a sensible tradeoff. Code and checkpoints are promised on GitHub, which is welcome.

The soft spots are real but fixable. The main one is rate accounting. §3.2 defines the cost as h×w×d×⌈log2 K⌉, then says entropy coding further compresses it. Yet every bpd value in Table 1 and Fig. 4 is an exact multiple of that pre-entropy cost, e.g., 0.78 = 600/768 and 0.52 = 400/768. Either the index distribution is uniform and Huffman is a no-op, or the reported numbers are fixed-length index costs, not actual entropy-coded lengths. The paper never states which. It needs to report real bitstream lengths or explicitly label the rate as being pre-entropy, and then do the bpp comparison with a consistent pixel count. The bpp values imply pixel counts of ~175k–232k, not 336×336=112,896, so the comparison with learned image codecs is ambiguous without specifying whether they use original image resolution or the CLIP input size.

Other minor issues: no error bars or seeds, only one learned image-codec baseline (Cheng2020-anchor), and the downstream captioning/referring tasks are qualitative. The task-agnostic claim is also a bit strong because the training loss is cosine similarity in CLIP space, so the compressor only preserves what CLIP already encodes. That is a limitation, not a fatal flaw.

Who is this for? People working on semantic communication, task-oriented compression, or edge inference with foundation models. It is a workshop-length paper, and a serious referee could sort out the rate accounting in one round. I would send it to peer review rather than desk-reject it: the idea is sound, the architecture is reasonable, and the evaluation protocol just needs to be tightened. For a reading group, it is a great case study in how subtle bitrate misreporting can be.

Recommendation: send to review, but insist on a clear, reproducible rate definition (actual entropy-coded bits or explicit fixed-length cost) and a consistent bpp denominator before acceptance.

Referee Report

4 major / 5 minor

Summary. The paper proposes PQVAE-shared, a product-quantization variational autoencoder with a shared codebook that compresses CLIP image embeddings into a small number of bits for downstream task use, without reconstructing the image. The method is trained on ImageNet with a cosine-similarity distortion plus codebook and commitment losses, and is evaluated by zero-shot classification on ImageNet, OxfordPets, Caltech101, Tiny-ImageNet, and Food101, as well as qualitatively on image captioning and referring-object identification. The headline claims are that CLIP features can be compressed to 0.52 bpd (400 bits/image) with small accuracy drop, and that the method uses only 2–3×10⁻³ bpp, less than 5% of the bitrate of a learned image codec at comparable accuracy.

Significance. If the reported rates and accuracies are correct, this is a compelling demonstration that semantic information in CLIP embeddings can be preserved at extremely low bitrates, with clear relevance to edge inference and task-oriented communication. The paper's strengths include open code and checkpoints, consistent gains over VQ-VAE and learned scalar quantization baselines across four datasets and many rate points, and a concrete comparison to Cheng2020-anchor showing a large rate advantage. The shared-codebook design is a practical solution to codebook-collapse concerns in product quantization. However, the central quantitative claims rest on the rate accounting, and the current manuscript leaves the relationship between the stated pre-entropy PQ cost and the reported bpd/bpp values ambiguous; several reported numbers are not reproducible from the text as written.

major comments (4)

[§3.2 and §4.1] The rate definition is ambiguous and the reported numbers are inconsistent with the claimed entropy coding. §3.2 states the compression cost is h×w×d×⌈log2(K)⌉ and is 'further compressed by entropy coding'; §4.1 states Huffman coding is used. However, every bpd value in Table 1 and Fig. 4 is an exact round multiple of the pre-entropy product-quantization cost: e.g., 0.78 bpd = 600 bits = 25×8×3 with d=8, K=8; 0.52 bpd = 400 bits cannot be the Huffman-coded length unless the code happens to be identity. Either the index distribution is uniform so entropy coding is a no-op (making 'further compressed' misleading), or the reported bpd values are not actual entropy-coded bitstream lengths. Please state explicitly which quantity is reported and, if entropy coding is applied, provide the actual bitstream lengths or the entropy model used. This is load-bearing because the 0.52 bpd and 400 bits/
[§4.4] The bpp denominator is never specified and the reported values are inconsistent with the CLIP input resolution. The paper reports 2.29×10⁻³ bpp for OxfordPets and 1.723×10⁻³ bpp for Food101. With a 400-bit embedding, these imply approximately 174,700 and 232,000 pixels respectively. The CLIP encoder is ViT-L/14@336px, so the preprocessed image is 336×336 = 112,896 pixels; 400 bits at that resolution would be 3.54×10⁻³ bpp, not in the claimed 2–3×10⁻³ range. If the denominator is the original image resolution, that must be stated and used consistently for a fair comparison with pixel-domain codecs. Also, the 'mainstream image compression' claim is based on a single learned codec (Cheng2020-anchor); additional baselines would be needed to support the generality of the rate comparison.
[Eq. (5), Eq. (7), §4.6] The distortion metric and training loss optimize cosine similarity in CLIP space, and the downstream tasks in §4.6 (captioning, referring identification) all operate on CLIP features. This is not circular because no test labels or downstream outputs are used to fit parameters, but it does mean the 'task-agnostic' claim is really 'agnostic to tasks whose required information is recoverable from CLIP cosine geometry.' The paper should either state this limitation explicitly or test a task that requires information CLIP is known to discard (e.g., fine-grained attribute counting, color discrimination, or text within images). Without such a test, the reader cannot judge how far the generalization claim extends.
[§4.6] The downstream-task evaluation is only qualitative. For image captioning, the paper shows example captions and says they are 'nearly identical'; for referring object identification, it reports 'robust accuracy' without giving numbers or a comparison against the uncompressed CLIP baseline. This does not support the claim that semantic integrity is preserved quantitatively. Please add standard metrics (e.g., CIDEr/BLEU for captioning, IoU or accuracy for referring identification) for both compressed and uncompressed features, ideally at several bitrates.

minor comments (5)

[§4.4 vs Table 1] For Food101, §4.4 reports 78.47% accuracy at 1.723×10⁻³ bpp, while Table 1 lists 79.39% at 0.52 bpd (400 bits). Since both should correspond to 400 bits/image, the discrepancy needs explanation or correction.
[Throughout] The bitrate comparison assumes the CLIP model is available at both encoder and decoder; the one-time cost of shipping the model is not counted. This is a standard protocol assumption in semantic communication, but it should be stated explicitly so readers do not confuse it with a property of the compression algorithm.
[§4.2] Experimental results are reported without error bars or multiple seeds. Given the small accuracy differences between some operating points (e.g., 87.30 vs 88.08 for OxfordPets), single runs make it hard to assess whether the differences are meaningful. Adding variance estimates or at least a statement that runs are deterministic would help.
[§3.2] The sentence 'This is then further compressed by entropy coding, yielding the final bit cost' is misleading given the reported bpd values; please rephrase to clarify exactly how the final bit cost is measured (see major comment 1).
[Abstract and §4.4] The abstract says 'less than 5%' of the bitrate of mainstream image compression, while §4.4 says 'less than 3%' and '2–3%'. The numbers are consistent with the reported Cheng2020-anchor values, but the percentage threshold should be aligned to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CLIP-feature compressor is trained on embeddings, evaluated with a fixed external zero-shot classifier, and no fitted parameter is relabeled as a prediction.

full rationale

The paper's derivation chain is self-contained with respect to circularity. The compressor (PQVAE-shared) is trained to reconstruct pre-trained CLIP embeddings x, using a cosine-similarity distortion D(x, x̂)=1−x̂·x/(||x̂||||x||) (Eq. 5) and a training loss containing the same term (Eq. 7). The downstream evaluations (zero-shot classification, captioning, referring-object identification) use the same pre-trained CLIP model and fixed text prompts, but no test labels, downstream outputs, or evaluation metrics are used to fit the compressor. The zero-shot classifier is a fixed external function of the reconstructed embedding, so preserving cosine similarity to the original embedding is a sufficient but not constructed condition for the reported accuracy; the actual accuracy values are empirical. The claimed task-agnosticism is explicitly scoped to tasks that CLIP embeddings support, which is an assumption, not a circular definition. The only self-citation in the introduction (ref. [6], Hu et al., same authors) is motivational ('emerging applications... prioritize semantic processing') and is not load-bearing for any technical claim; no uniqueness theorem or ansatz is imported via self-citation. The bitrate accounting concerns raised in the external critique (pre-entropy bpd values vs. entropy coding, and ambiguous bpp denominators) are quantitative consistency issues, not circularity, and do not make the derivation equivalent to its inputs. Therefore no circular step is identified and the score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the choice of CLIP as the semantic backbone and several tuned architectural parameters. The paper does not disclose exact values for all of these, so the contribution's reproducibility is limited.

free parameters (5)

codebook size K = not reported, implied by bit budget (e.g., K=8 for 3-bit subvectors)
Chosen by hand to hit target bpd values; affects rate-accuracy trade-off.
subspace dimension d_sub = varied in ablation (e.g., 8)
Number of subvectors per channel; controls codebook size and reconstruction fidelity.
latent dimensions h,w,c = 5x5x128
Intentionally increased beyond CLIP's 768-dim to facilitate codebook learning; a design choice.
loss weights alpha, beta = not reported
Balance cosine loss, codebook loss, and commitment loss; tuned but not disclosed.
bit budget / bpd operating points = 0.52, 0.78, 1, 1.04, 1.3, 1.56, 2, 2.604, 3 bpd
Points selected for evaluation; determined by codebook and subspace configuration.

assumptions (4)

domain assumption CLIP embeddings capture task-relevant semantic information
The whole method assumes CLIP's representation is sufficient for downstream tasks; if not, compression inherits the failure.
domain assumption Cosine similarity in CLIP space is a valid measure of semantic distortion
Defined in Eq. (5) and used as the training loss; no evidence that it aligns with human semantics for all tasks.
domain assumption Receiver has access to the same CLIP model and text embeddings for zero-shot classification
The bitrate excludes the cost of the shared CLIP model; the protocol assumes a shared codebook and CLIP weights.
standard math VQ-VAE training with codebook and commitment losses converges to a useful discrete codebook
Taken from reference [12]; standard in the field.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models." pith.science (2026). https://pith.science/paper/KYKQSMF3

@misc{pith2026250905925,
  author       = {Pith},
  title        = {Pith review of: Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYKQSMF3}},
  note         = {Machine review of arXiv:2509.05925}
}

read the original abstract

Recent deep learning-based methods for lossy image compression achieve competitive rate-distortion performance through extensive end-to-end training and advanced architectures. However, emerging applications increasingly prioritize semantic preservation over pixel-level reconstruction and demand robust performance across diverse data distributions and downstream tasks. These challenges call for advanced semantic compression paradigms. Motivated by the zero-shot and representational capabilities of multimodal foundation models, we propose a novel semantic compression method based on the contrastive language-image pretraining (CLIP) model. Rather than compressing images for reconstruction, we propose compressing the CLIP feature embeddings into minimal bits while preserving semantic information across different tasks. Experiments show that our method maintains semantic integrity across benchmark datasets, achieving an average bit rate of approximately 2-3* 10(-3) bits per pixel. This is less than 5% of the bitrate required by mainstream image compression approaches for comparable performance. Remarkably, even under extreme compression, the proposed approach exhibits zero-shot robustness across diverse data distributions and downstream tasks.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

[1]

INTRODUCTION Deep learning has revolutionized lossy image compression by introducing data-driven codecs that replace classical analysis and synthesis transforms with neural networks. Starting with the adoption of autoencoders and convolutional neural net- works, neural image compression has progressed through sev- eral key innovations, including the integ...

work page Pith review arXiv 2025
[2]

In this paper, we exploit the CLIP model [10], which aligns images’ visual features with corresponding textual descriptions using contrastive learning

SYSTEM MODEL Our goal is to design a robust semantic compression scheme for various downstream tasks, without requiring the recon- struction of the original image. In this paper, we exploit the CLIP model [10], which aligns images’ visual features with corresponding textual descriptions using contrastive learning. Our focus is on compressing CLIP features...

work page
[3]

Overall architecture The proposedPQVAE-sharedscheme for feature compression is illustrated in Fig

PRODUCT QUANTIZA TION-V ARIA TIONAL AUTOENCODER WITH SHARED CODEBOOK 3.1. Overall architecture The proposedPQVAE-sharedscheme for feature compression is illustrated in Fig. 2, which employs a learned product quan- tization module and a shared codebook. After obtaining the latent representationx c ∈R h×w×c as in Eq. (1), a product quantization method with ...

work page
[4]

ViT-L/14@336px

EXPERIMENTAL RESULTS 4.1. Experiment setup Our compression framework ofPQVAE-sharedis trained on ImageNet, using 800 classes for training and the other 200 classes for validation. To evaluate semantic quality of feature reconstruction, we conduct zero-shot classification, image captioning, and referring object identification at the receiver. Specifically,...

work page
[5]

Experiments demonstrate that original CLIP features can be compressed more than 30-fold while maintaining satisfactory semantic preservation

CONCLUSION We presented CLIP-driven semantic image compression, im- plemented through product quantization-variational autoen- coder with shared codebook, which is task-agnostic and fo- cuses on semantic preservation for downstream tasks rather than pixel-level reconstruction. Experiments demonstrate that original CLIP features can be compressed more than...

work page
[6]

Variational image compres- sion with a scale hyperprior,

Johannes Ball ´e, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston, “Variational image compres- sion with a scale hyperprior,” inInternational Confer- ence on Learning Representations, 2018

work page 2018
[7]

Learned image compression with discretized gaussian mixture likelihoods and attention modules,

Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 7939–7948

work page 2020
[8]

LotteryCodec: Searching the implicit representation in a random network for low-complexity image compression,

Haotian Wu, Gongpu Chen, Pier Luigi Dragotti, and Deniz Gunduz, “LotteryCodec: Searching the implicit representation in a random network for low-complexity image compression,” inF orty-second International Con- ference on Machine Learning, 2025

work page 2025

Show all 21 references

[9]

DiffCP: Ultra-low bit collaborative perception via diffusion model,

Ruiqing Mao, Haotian Wu, Yukuan Jia, Zhaojun Nan, Yuxuan Sun, Sheng Zhou, Deniz G¨und¨uz, and Zhisheng Niu, “DiffCP: Ultra-low bit collaborative perception via diffusion model,” in2025 IEEE International Confer- ence on Robotics and Automation (ICRA), 2025

2025
[10]

Edge computing with artifi- cial intelligence: A machine learning perspective,

Haochen Hua, Yutong Li, Tonghe Wang, Nanqing Dong, Wei Li, and Junwei Cao, “Edge computing with artifi- cial intelligence: A machine learning perspective,”ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023

2023
[11]

Zero- shot semantic communication with multimodal founda- tion models,

Jiangjing Hu, Haotian Wu, Wenjing Zhang, Fengyu Wang, Wenjun Xu, Hui Gao, and Deniz G¨und¨uz, “Zero- shot semantic communication with multimodal founda- tion models,”arXiv preprint arXiv:2502.18200, 2025. Fig. 7. Image captioning results. Top row: ground truth (GT); middle row...

2025 arXiv
[12]

Beyond transmitting bits: Context, semantics, and task-oriented communica- tions,

Deniz G ¨und¨uz, Zhijin Qin, Inaki Estella Aguerri, Harpreet S Dhillon, Zhaohui Yang, Aylin Yener, Kai Kit Wong, and Chan-Byoung Chae, “Beyond transmitting bits: Context, semantics, and task-oriented communica- tions,”IEEE Journal on Selected Areas in Communica- tions, vol. 41...

2022
[13]

DeepSIC: Deep seman- tic image compression,

Sihui Luo, Yezhou Yang, Yanling Yin, Chengchao Shen, Ya Zhao, and Mingli Song, “DeepSIC: Deep seman- tic image compression,” inNeural Information Process- ing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, December 13-16, 2018, Proceedings, Part I 25. Spring...

2018
[14]

Semantic-aware video compression for automotive cameras,

Yiting Wang, Pak Hung Chan, and Valentina Donzella, “Semantic-aware video compression for automotive cameras,”IEEE Transactions on Intelligent V ehicles, vol. 8, no. 6, pp. 3712–3722, 2023

2023
[15]

Learning transferable visual models from natural lan- guage supervision,

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural lan- guage supervision,” inInternational conference on ma- chine lear...

2021
[16]

Visual instruction tuning,

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, “Visual instruction tuning,”Advances in neural information processing systems, pp. 34892–34916, 2023

2023
[17]

Neural dis- crete representation learning,

Aaron Van Den Oord, Oriol Vinyals, et al., “Neural dis- crete representation learning,”Advances in neural infor- mation processing systems, vol. 30, 2017

2017
[18]

Learning product code- books using vector-quantized autoencoders for image re- trieval,

Hanwei Wu and Markus Flierl, “Learning product code- books using vector-quantized autoencoders for image re- trieval,” in2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP). IEEE, 2019

2019
[19]

Image coding for machines with edge information learning using segment anything,

Takahiro Shindo, Kein Yamada, Taiju Watanabe, and Hiroshi Watanabe, “Image coding for machines with edge information learning using segment anything,” in 2024 IEEE International Conference on Image Process- ing (ICIP). IEEE, 2024, pp. 3702–3708

2024
[20]

Clip- cap: Clip prefix for image captioning,

Ron Mokady, Amir Hertz, and Amit H Bermano, “Clip- cap: Clip prefix for image captioning,”arXiv preprint arXiv:2111.09734, 2021

2021 arXiv
[21]

Segment anything,

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al., “Segment anything,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

INTRODUCTION Deep learning has revolutionized lossy image compression by introducing data-driven codecs that replace classical analysis and synthesis transforms with neural networks. Starting with the adoption of autoencoders and convolutional neural net- works, neural image compression has progressed through sev- eral key innovations, including the integ...

work page Pith review arXiv 2025

[2] [2]

In this paper, we exploit the CLIP model [10], which aligns images’ visual features with corresponding textual descriptions using contrastive learning

SYSTEM MODEL Our goal is to design a robust semantic compression scheme for various downstream tasks, without requiring the recon- struction of the original image. In this paper, we exploit the CLIP model [10], which aligns images’ visual features with corresponding textual descriptions using contrastive learning. Our focus is on compressing CLIP features...

work page

[3] [3]

Overall architecture The proposedPQVAE-sharedscheme for feature compression is illustrated in Fig

PRODUCT QUANTIZA TION-V ARIA TIONAL AUTOENCODER WITH SHARED CODEBOOK 3.1. Overall architecture The proposedPQVAE-sharedscheme for feature compression is illustrated in Fig. 2, which employs a learned product quan- tization module and a shared codebook. After obtaining the latent representationx c ∈R h×w×c as in Eq. (1), a product quantization method with ...

work page

[4] [4]

ViT-L/14@336px

EXPERIMENTAL RESULTS 4.1. Experiment setup Our compression framework ofPQVAE-sharedis trained on ImageNet, using 800 classes for training and the other 200 classes for validation. To evaluate semantic quality of feature reconstruction, we conduct zero-shot classification, image captioning, and referring object identification at the receiver. Specifically,...

work page

[5] [5]

Experiments demonstrate that original CLIP features can be compressed more than 30-fold while maintaining satisfactory semantic preservation

CONCLUSION We presented CLIP-driven semantic image compression, im- plemented through product quantization-variational autoen- coder with shared codebook, which is task-agnostic and fo- cuses on semantic preservation for downstream tasks rather than pixel-level reconstruction. Experiments demonstrate that original CLIP features can be compressed more than...

work page

[6] [6]

Variational image compres- sion with a scale hyperprior,

Johannes Ball ´e, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston, “Variational image compres- sion with a scale hyperprior,” inInternational Confer- ence on Learning Representations, 2018

work page 2018

[7] [7]

Learned image compression with discretized gaussian mixture likelihoods and attention modules,

Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 7939–7948

work page 2020

[8] [8]

LotteryCodec: Searching the implicit representation in a random network for low-complexity image compression,

Haotian Wu, Gongpu Chen, Pier Luigi Dragotti, and Deniz Gunduz, “LotteryCodec: Searching the implicit representation in a random network for low-complexity image compression,” inF orty-second International Con- ference on Machine Learning, 2025

work page 2025