Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Pretrained vision foundation models double as low-rate image codecs without fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:32 UTC pith:3BJJFFCQ

load-bearing objection Useful exploratory study, but the headline result rests on an unvalidated bitrate estimate; worth a careful referee, not a desk reject. the 3 major comments →

arxiv 2509.05169 v1 pith:3BJJFFCQ submitted 2025-09-05 eess.IV

Exploring Autoregressive Vision Foundation Models for Image Compression

Huu-Tai Phung , Yu-Hsiang Lin , Yen-Kuan Ho , Wen-Hsiao Peng This is my paper
classification eess.IV
keywords vision foundation modelsautoregressive modelsimage compressionentropy codingperceptual qualityvector quantizationnext-token predictionlow-bitrate compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that autoregressive vision foundation models (VFMs), built for image generation, can be repurposed as perceptual image codecs by treating their next-token predictors as entropy models. The authors show that at extremely low bitrates (below 0.1 bits per pixel), several off-the-shelf VFMs match or outperform state-of-the-art learned codecs on perceptual quality metrics, with no fine-tuning. The central insight is that the same statistical model that lets a VFM generate realistic images also assigns well-calibrated probabilities to image tokens, so its self-information can drive arithmetic coding. The paper also reverses the lens, using learned codecs as generators to expose why codec-specific AR models underperform VFMs in generation.

Core claim

The paper establishes that the autoregressive transformer inside a vision foundation model can be used directly as an entropy coder for the tokens produced by its visual tokenizer, converting a generation pipeline into a lossy image codec. Concretely, the authors replace condition tokens with zeros or fixed class tokens, then compute bitrate as the summed self-information of each next-token prediction. On Kodak and CLIC2020, LlamaGen and Cosmos achieve lower LPIPS than ELIC, VTM, JPEG AI, and even perception-optimized baselines like MS-ILLM and PerCo at matching low bitrates, while keeping PSNR competitive. The study also finds that tokenizers contribute the bulk of the compression ratio (30

What carries the argument

The next-token predictor of a pre-trained AR-based VFM, used as a causal entropy model: for each token in a raster-scan (or scale-wise) order, the model outputs a probability distribution over the tokenizer's codebook, and the negative log-probability of the actual token gives its bitrate under an arithmetic coder. The paper additionally uses the VFM's own visual tokenizer (vector quantization) as the lossy front end, replacing scalar-quantized latents of learned codecs.

Load-bearing premise

The pretrained AR model's next-token probabilities, after substituting zero or fixed class tokens for conditions, form a calibrated entropy model for tokens of arbitrary test images, so that the summed self-information is a realizable bitrate under an arithmetic coder.

What would settle it

Run an actual arithmetic coder on the token sequences of the paper's test images using the VFM's reported next-token probabilities and measure the true bitstream length; if it deviates from the summed self-information by more than the standard arithmetic-coding overhead (fraction bits), the reported bitrates are not achievable. A second check: decode the generated token sequences without the condition substitutions and measure the perceptual quality drop on out-of-domain images (e.g., medical or satellite images) versus natural images.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If correct, VFM-based entropy coding offers a shortcut to extremely low-bitrate semantic compression without training a single new weight, since the AR model already encodes rich spatial and semantic context.
  • The tokenizer's codebook size becomes a first-order design knob: larger codebooks dilute the AR model's probability mass, worsening compression ratios, so future VFMs tuned for compression would use small codebooks.
  • Since the AR model is shared between generation and compression, any improvement in VFM next-token prediction directly translates to better compression, tying codec progress to foundation-model scaling.
  • The paper's generation experiments suggest that learned codecs' factorized hyperpriors and scalar quantization limit their generative abilities, implying that the two fields—compression and generation—should co-design their latent models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported bitrates likely assume ideal arithmetic coding; in practice, finite-precision arithmetic coders and the 64k codebook of Cosmos may add overhead, so real-world bitrates could be slightly higher than the self-information sums.
  • Because the conditioning substitution (zero tokens or fixed class tokens) is untested against true unconditional token distributions, the entropy model may be miscalibrated on out-of-distribution images, a gap the authors do not quantify.
  • The success at low bitrates hints that semantic fidelity—not pixel fidelity—is what perceptual metrics reward, and a VFM-based codec could be steered by adjusting the AR model's sampling temperature or top-k, a testable extension the paper does not explore.
  • If the AR model's probability estimates are well-calibrated only for images near its training distribution, then the codec will degrade gracefully for natural images but may fail on domain-shifted inputs, a boundary the paper leaves unexamined.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes using pre-trained autoregressive vision foundation models (VFMs) as image codecs without any fine-tuning. Four VFMs are adapted: VAR, LlamaGen, Cosmos, and Lumina-mGPT. The VFM tokenizer produces discrete tokens, and the pre-trained AR model is repurposed as an entropy model to estimate the bitrate of each token via self-information. For image-conditioned models the condition tokens are replaced with zeros; for class-conditioned models fixed class tokens are used. Rate-distortion comparisons on Kodak and CLIC2020 are presented against distortion-optimized codecs (VTM, JPEG AI, ELIC), perceptual codecs (MS-ILLM, PerCo), and an LMM-based codec. The paper reports extremely low bitrates (below 0.1 bpp) and claims superior perceptual quality for some VFMs. It also analyzes the generation capability of learned image codecs and connects AR modeling choices to image quality.

Significance. If the reported bitrates are realizable, the paper makes an intriguing proof-of-concept: an off-the-shelf generative VFM can act as a zero-shot perceptual compressor, and the AR next-token predictor can serve as an entropy model for the tokenizer's discrete tokens. The study's strengths are its breadth across four recent VFMs, the use of external perceptual metrics, and the complementary analysis of learned codecs as generators. However, the central empirical claim currently rests on an unvalidated rate estimate, so the significance is conditional on a successful entropy-coding demonstration or an explicit reframing as an ideal-code-length study.

major comments (3)
  1. [Section III.A, paragraph starting 'Building on these analogies'] The bitrate in Fig. 3 is computed as the sum of token self-information, not measured from a real arithmetic coder. No bitstream, entropy-coding implementation, or encode/decode verification is described. Since every R-D point, compression ratio, and the abstract's 'superior perceptual quality at extremely low bitrates' claim depend on this x-coordinate, the central result is currently unsupported. I acknowledge that a matching arithmetic coder using the same AR model could in principle realize the summed self-information within a small overhead, but the paper gives no evidence that this was done. With Cosmos's 64,000-token codebook, exact normalized full-softmax probabilities at every step and identical numerical precision at encoder and decoder are non-trivial. Please implement and report actual bitstream lengths, or clearly label the curves as ideal-code-length lower bounds and temper
  2. [Section III.A, conditioning substitution] The paper replaces image-conditioning prompt tokens with zeros (Cosmos, Lumina-mGPT) and uses fixed class tokens for VAR and LlamaGen. This is assumed to yield a valid causal coding distribution, but no validation is provided. For image-conditioned models, zero tokens are likely out-of-distribution; for class-conditioned models, a fixed class does not encode image content. Either mismatch can make the AR model's predictive probabilities miscalibrated for the actual token sequence, so the summed self-information may not be achievable. A concrete test would be to run a real entropy coder with the same condition-substitution scheme and verify (i) the bitstream length matches the self-information sum and (ii) the decoder reconstructs the exact token sequence. Without this, the R-D points for these models are not established.
  3. [Section IV.A, Fig. 3 and Fig. 5] The experimental section provides no numerical RD tables, no per-image statistics, and no error bars. Claims such as 'surpasses' and 'comparable to' are supported only by RD curves, which makes it impossible to assess the magnitude or stability of the reported differences. The ablation in Fig. 5 also lacks quantitative values for the tokenizer and AR compression ratios. Given that Kodak has only 24 images and the CLIC2020 subset is small, mean and standard deviation (or per-image scatter) should be reported to support the paper's quantitative conclusions.
minor comments (5)
  1. [Section IV.A, Table III] The table reference appears as 'Table ???III' and should be corrected to 'Table III'.
  2. [Section IV.B, Fig. 5] The typo 'accross' should be 'across'.
  3. [Section IV.A, Fig. 4 caption] The phrase 'More visualization in Github' is not a usable citation; please provide a URL or remove it.
  4. [Section IV, datasets] All images are center-cropped to 512x512, but the manuscript does not state whether the comparison baselines were run on the same crops or at their native resolutions. This should be clarified to ensure a fair comparison.
  5. [Section IV, compression ratio definition] The definition of compression ratio is introduced informally. Please define it precisely for both the tokenizer and the AR-model components, including how 'token size without entropy coding' is computed (e.g., bits = number of tokens x codebook bit depth).

Circularity Check

0 steps flagged

No circularity: the VFM codec's bitrate is computed by standard self-information summation from an off-the-shelf AR model, with no fitted parameters or self-citation carrying the claim.

full rationale

The paper repurposes pre-trained autoregressive vision foundation models as entropy coders. The rate is computed as the sum of per-token self-information under the same AR model's predictive distribution (Section III.A). This is the standard definition of ideal code length for a given probability model, not a circular reduction: the VFM is not fitted to the test images, no parameter is tuned, and the claim that certain VFMs outperform specialized codecs is an empirical comparison against external baselines on external metrics (LPIPS, CLIP-IQA, NIQE). The only self-referential element is that the rate axis uses the same model as the entropy model, which is definitional to any entropy coder, not a fitted-input-called-prediction. The cited prior work includes one authored by a co-author (ELIC [13]), but it is used only as a baseline, not as load-bearing justification. No uniqueness theorem or ansatz is imported from self-citations. Although the paper does not report an actual arithmetic-coded bitstream, that is an implementation/validity concern (miscalibration risk), not circularity: the reported bpp values are what an ideal entropy coder would achieve if the model's probabilities were exactly the coding distribution. The derivation chain is self-contained: tokenizer produces tokens, AR model supplies conditional probabilities, summation gives rate. Therefore no circular step is present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no new free parameters and no invented entities; it inherits the pretrained weights, tokenizers, and codebooks of four public VFMs. The load-bearing assumptions are that the AR distribution is a valid entropy model, that condition substitution with zeros or class tokens preserves coding validity, and that ideal entropy coding realizes the self-information sum.

axioms (4)
  • standard math The total rate of a token sequence equals the sum of per-token self-information under the AR model (ideal entropy coding).
    Invoked in Section III.A to compute bitrate; a standard Shannon coding result if the probabilities are exact and an ideal arithmetic coder is used.
  • domain assumption The pretrained AR model's next-token distribution is a calibrated entropy model for tokenizer outputs of test images.
    Central premise of Sections III.A and IV.A; if probabilities are miscalibrated, the reported bpp and compression ratios are not realizable.
  • ad hoc to paper Replacing image-conditioning prompt tokens with zeros and using fixed class tokens yields a valid causal coding distribution.
    Section III.A applies this modification for Cosmos and Lumina-mGPT; it is a paper-specific adaptation with no validation against full conditioning.
  • domain assumption The AR model's token order used for compression matches the causal order used in training.
    Section III.A and Fig. 1(d) assume the next-token probability can be accumulated in the training order; a mismatch would invalidate the entropy coding.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 6593 in / 15647 out tokens · 164577 ms · 2026-08-05T05:32:17.419836+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Exploring Autoregressive Vision Foundation Models for Image Compression." pith.science (2026). https://pith.science/paper/3BJJFFCQ

@misc{pith2026250905169,
  author       = {Pith},
  title        = {Pith review of: Exploring Autoregressive Vision Foundation Models for Image Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BJJFFCQ}},
  note         = {Machine review of arXiv:2509.05169}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This work presents the first attempt to repurpose vision foundation models (VFMs) as image codecs, aiming to explore their generation capability for low-rate image compression. VFMs are widely employed in both conditional and unconditional generation scenarios across diverse downstream tasks, e.g., physical AI applications. Many VFMs employ an encoder-decoder architecture similar to that of end-to-end learned image codecs and learn an autoregressive (AR) model to perform next-token prediction. To enable compression, we repurpose the AR model in VFM for entropy coding the next token based on previously coded tokens. This approach deviates from early semantic compression efforts that rely solely on conditional generation for reconstructing input images. Extensive experiments and analysis are conducted to compare VFM-based codec to current SOTA codecs optimized for distortion or perceptual quality. Notably, certain pre-trained, general-purpose VFMs demonstrate superior perceptual quality at extremely low bitrates compared to specialized learned image codecs. This finding paves the way for a promising research direction that leverages VFMs for low-rate, semantically rich image compression.

Figures

Figures reproduced from arXiv: 2509.05169 by Huu-Tai Phung, Wen-Hsiao Peng, Yen-Kuan Ho, Yu-Hsiang Lin.

Figure 1
Figure 1. Figure 1: A comparative overview of the operational pipelines for compression and generation tasks. This figure illustrates how [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Different types of AR models in learned image codecs. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Rate-distortion performance comparison on Kodak and CLIC2020. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visual comparison of VFMs-codecs and baselines (bpp, LPIPS). More visualization in Github . [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 14 canonical work pages

  1. [1]

    Variational image compression with a scale hyperprior,

    J. Ball ´e, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” inInternational Conference on Learning Representations, 2018

    2018

  2. [2]

    Cosmos world foundation model platform for physical ai,

    N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopad- hyay, Y . Chen, Y . Cui, Y . Dinget al., “Cosmos world foundation model platform for physical ai,”arXiv preprint arXiv:2501.03575, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Misc: Ultra-low bitrate image semantic compression driven by large multimodal model,

    C. Li, G. Lu, D. Feng, H. Wu, Z. Zhang, X. Liu, G. Zhai, W. Lin, and W. Zhang, “Misc: Ultra-low bitrate image semantic compression driven by large multimodal model,”IEEE Transactions on Image Processing, vol. 34, pp. 335–349, 2025

    work page 2025

  4. [4]

    Lmm-driven semantic image-text coding for ultra low-bitrate learned image compression,

    S. Murai, H. Sun, and J. Katto, “Lmm-driven semantic image-text coding for ultra low-bitrate learned image compression,” in2024 IEEE Inter- national Conference on Visual Communications and Image Processing (VCIP), 2024, pp. 1–5

    work page 2024

  5. [5]

    Revisiting feature prediction for learning visual representa- tions from video,

    A. Bardes, Q. Garrido, J. Ponce, M. Rabbat, Y . LeCun, M. Assran, and N. Ballas, “Revisiting feature prediction for learning visual representa- tions from video,”arXiv:2404.08471, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Gigatok: Scaling visual tokenizers to 3 billion parameters for autoregressive image generation,

    T. Xiong, J. H. Liew, Z. Huang, J. Feng, and X. Liu, “Gigatok: Scaling visual tokenizers to 3 billion parameters for autoregressive image generation,”arXiv preprint arXiv:2504.08736, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Autoregressive model beats diffusion: Llama for scalable image gener- ation,

    P. Sun, Y . Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan, “Autoregressive model beats diffusion: Llama for scalable image gener- ation,”arXiv preprint arXiv:2406.06525, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Neural discrete representation learning,

    A. van den Oord, O. Vinyals, and k. kavukcuoglu, “Neural discrete representation learning,” inAdvances in Neural Information Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017

    work page 2017

  9. [9]

    Taming transformers for high- resolution image synthesis,

    P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high- resolution image synthesis,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 12 873–12 883

    work page 2021

  10. [10]

    Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining,

    D. Liu, S. Zhao, L. Zhuo, W. Lin, Y . Xin, X. Li, Q. Qin, Y . Qiao, H. Li, and P. Gao, “Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining,”arXiv preprint arXiv:2408.02657, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction,

    K. Tian, Y . Jiang, Z. Yuan, B. Peng, and L. Wang, “Visual autoregressive modeling: Scalable image generation via next-scale prediction,” in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024, pp. 84 839–84 865

    work page 2024

  12. [12]

    Overview of the Versatile Video Coding (VVC) Standard and Its Applications,

    B. Bross, Y .-K. Wang, Y . Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the Versatile Video Coding (VVC) Standard and Its Applications,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736–3764, 2021

    work page 2021

  13. [13]

    Elic: Efficient learned image compression with unevenly grouped space- channel contextual adaptive coding,

    D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y . Wang, “Elic: Efficient learned image compression with unevenly grouped space- channel contextual adaptive coding,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5718–5727

    2022

  14. [14]

    Improving statistical fidelity for neural image compression with implicit local likelihood models,

    M. J. Muckley, A. El-Nouby, K. Ullrich, H. Jegou, and J. Verbeek, “Improving statistical fidelity for neural image compression with implicit local likelihood models,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202, 2023, pp. 25 426–25 443

    work page 2023

  15. [15]

    High-fidelity generative image compression,

    F. Mentzer, G. Toderici, M. Tschannen, and E. Agustsson, “High-fidelity generative image compression,”arXiv preprint arXiv:2006.09965, 2020

    Pith/arXiv arXiv 2006

  16. [16]

    Lossy image compression with conditional dif- fusion models,

    R. Yang and S. Mandt, “Lossy image compression with conditional dif- fusion models,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 64 971–64 995

    work page 2023

  17. [17]

    Deep hierarchical video compres- sion,

    M. Lu, Z. Duan, F. Zhu, and Z. Ma, “Deep hierarchical video compres- sion,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 8, 2024, pp. 8859–8867

    work page 2024

  18. [18]

    Kodak lossless true color image suite (photocd pcd0992),

    Eastman Kodak, “Kodak lossless true color image suite (photocd pcd0992),” http://r0k.us/graphics/kodak, 1993, accessed: 2025-07-20

    work page 1993

  19. [19]

    Workshop and challenge on learned image compression (clic),

    “Workshop and challenge on learned image compression (clic),” http: //www.compression.cc, 2020, accessed: 2025-07-20

    work page 2020

  20. [20]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018

    2018

  21. [21]

    Exploring clip for assessing the look and feel of images,

    J. Wang, K. C. Chan, and C. C. Loy, “Exploring clip for assessing the look and feel of images,” inAAAI, 2023

    work page 2023

  22. [22]

    Making a “completely blind

    A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,”IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013

    2013

  23. [23]

    VTM-17.0,

    “VTM-17.0,” https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware VTM, accessed: 2023-10-30

    work page 2023

  24. [24]

    The jpeg ai standard: Providing efficient human and machine visual data consumption,

    J. Ascenso, E. Alshina, and T. Ebrahimi, “The jpeg ai standard: Providing efficient human and machine visual data consumption,”IEEE MultiMedia, vol. 30, no. 1, pp. 100–111, 2023

    2023

  25. [25]

    Perco (SD): Open perceptual compression,

    N. K ¨orber, E. Kromer, A. Siebert, S. Hauke, D. Mueller-Gritschneder, and B. Schuller, “Perco (SD): Open perceptual compression,” inWork- shop on Machine Learning and Compression, NeurIPS, 2024

    work page 2024