REVIEW 3 major objections 4 minor 1 cited by
A training trick called knowledge distillation lets small neural networks match the compression quality of large ones while cutting compute, memory, and energy use.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Knowledge-distilled students with 64 or more channels match the rate-distortion performance of a 128-channel teacher while cutting memory by 68% and energy by 34%.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection A useful, honest empirical sweep of KD for learned compression, but the missing from-scratch baseline leaves the central causal claim untested. the 3 major comments →
Efficient Learned Image Compression Through Knowledge Distillation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the scale hyperprior compression model, knowledge distillation transfers most of a teacher's rate-distortion behavior to smaller students. A student with 64 channels reaches 34.11 dB PSNR at 0.66 bpp against the teacher's 34.53 dB at 0.67 bpp, a 1.2 percent PSNR drop with 66.6 percent fewer parameters, 71.8 percent fewer FLOPs, and 34.2 percent lower energy per frame. Students with fewer than 64 channels degrade more sharply, while 96- and 112-channel students sit nearly on the teacher's RD curve. Distilling the hyper-latent space adds only marginal gains in this setup, and using a teacher that prioritizes bit rate shifts students toward lower bit rates.
What carries the argument
The mechanism is a composite distillation loss added to the student's rate-distortion objective. The loss, L1 = λ1 MSE(ŷ_student, ŷ_teacher) + λ2 MSE(x̂_student, x̂_teacher) + λ3 RD(ŷ_student, x̂_student, x), forces the student's latent representations and reconstructed images to imitate the teacher's while still optimizing rate and distortion; a second variant L2 also distills the hyper-latent used by the entropy model. The teacher is a pre-trained 128-channel scale hyperprior model, and smaller students shrink only the channel count while keeping the latent-space size fixed.
Load-bearing premise
The paper credits the students' performance to knowledge distillation, but it never trains a same-size student from scratch with the same recipe, so it remains possible that the smaller models would reach similar quality even without a teacher.
What would settle it
Train a 64-channel scale hyperprior student from scratch for 4-5 million steps on Vimeo90K with the same rate-distortion loss; if it matches the KD student's roughly 34.1 dB PSNR at 0.66 bpp on Kodak, the attribution of the gain to distillation collapses.
If this is right
- A 64-channel KD student can replace a 5.08M-parameter teacher on memory-limited devices with roughly a one-percent PSNR drop.
- The KD student with 64 channels reduces memory by 68 percent, FLOPs by 72 percent, and per-frame energy by 34 percent, making it a direct candidate for real-time compression on edge hardware.
- Choosing a teacher trained for low bit rate shifts students toward lower bit rates, so KD gives a way to steer the compression-quality tradeoff without redesigning the architecture.
- All student models exceed 200 frames per second on the measured GPU, leaving headroom for higher resolution or lower energy operation.
- Extending KD to the hyper-latent space yields only small gains for larger students, so the main benefit comes from distilling the main latent and reconstruction.
Where Pith is reading between the lines
- The paper's causal claim would be stronger with a same-size student trained from scratch under the same recipe; without that control, part of the reported gain could be due to longer training or the choice of a 128-channel baseline.
- Energy and throughput were measured on one high-end GPU; actual performance on memory- or battery-constrained platforms is likely to differ, so the frugal-AI conclusion should be re-tested on target hardware.
- The authors note KD can be extended to transformer-based models; one could also try perceptual or GAN-based teachers to see whether students inherit better visual quality rather than just PSNR.
- Since KD on the hyper-latent gave marginal gains here, a natural next test is to distill a richer entropy model, such as an autoregressive prior, to see whether the benefit grows with the complexity of the teacher's probability model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies knowledge distillation (KD) to a learned image compression model (Ballé et al.'s scale hyperprior from compressAI) and evaluates whether smaller 'student' networks can match a larger 'teacher' model in rate–distortion performance while reducing memory, compute, and energy. Students with 16–112 channels are trained using a loss that combines latent-space, reconstruction, and rate–distortion terms (Eqs. 1–3). The central reported results are that students with 64 or more channels perform roughly on par with the 128-channel teacher on the Kodak dataset: the 64-channel student reaches 34.11 dB PSNR at 0.66 bpp versus the teacher's 34.53 dB at 0.67 bpp, while reducing parameters by about 67%, FLOPs by 72%, and per-frame energy by 34% (Tables 2–4). The paper also explores different λ tradeoffs, KD loss weights, hyper-latent distillation, and hybrid teachers, concluding that KD enables smaller models to be used on resource-constrained platforms.
Significance. If the reported results are causal and reproducible, the paper provides a practical recipe for deploying lighter learned image compression models: a 64-channel student can approximate a 128-channel teacher with only about 1% PSNR loss while saving substantial memory and compute. The measurements are grounded in external references—the teacher is a compressAI pretrained checkpoint, RD performance is evaluated on Kodak, and resource numbers are measured rather than assumed. The public code and explicit training details are strengths. The main significance is constrained by the lack of a same-size from-scratch student control, so the paper currently establishes that small architectures trained with this recipe have attractive RD/resource trade-offs, but not specifically that KD is the cause of the improvement.
major comments (3)
- [§5.1.1, Figure 2; Abstract; §5.1.2] The causal claim that KD improves student performance is not tested. The abstract states that students can achieve 'better performance than when trained independently', and §5.1.2 asserts that some students 'perform better than what they would have if trained alone', but no same-size student is ever trained without KD. All comparisons are against the 128-channel teacher or pretrained 128-channel models (Tables 2–4). A 64-channel model trained from scratch on Vimeo90K with the same 4–5M steps, batch size 16, learning-rate schedule, and the same RD loss (i.e., Eq. 3 without the KD terms) could plausibly reach close to 34.11 dB / 0.66 bpp, in which case the 'KD improves compression' claim would collapse to a claim about architecture size. This is load-bearing for the paper's central message and needs a dedicated baseline.
- [Tables 2–4 and Figures 2–12] All reported RD and resource numbers are point estimates from a single run per configuration, evaluated on 24 Kodak images, with no error bars, no multiple seeds, and no significance testing. Differences such as 34.44 vs. 34.41 dB or 0.66 vs. 0.67 bpp are used to support qualitative claims about which student size is 'best' and which KD hyperparameters matter. These differences may be within training/evaluation noise. At minimum, the paper should report variance across seeds for the main 64-channel student and the teacher, or explicitly caution that rankings by small margins are not established.
- [§5.2 and Appendix A.1] The 'resource-constrained platform' conclusion is based on measurements on an Nvidia RTX 3090, a high-end desktop GPU. Throughput and energy per frame measured there do not necessarily transfer to smartphones, IoT devices, or FPGAs, which are the platforms named in the introduction. The parameter/FLOP reductions are architecture-level and likely transfer, but the specific energy savings (e.g., 34% for the 64-channel student) and the recommendation for 'resource-constrained platforms' rest on unverified hardware transfer. Either add measurements on a representative edge device or soften the claims to be about model complexity and desktop-GPU measurements.
minor comments (4)
- [Table 5] The JPEG/WebP rows appear misaligned or duplicated (e.g., repeated WebP values and inconsistent row grouping). Please reformat so each codec/setting is a single row with matching throughput, energy, PSNR, and bpp.
- [§5.1.2, Figure 6] The names 'student_5_1', 'student_5_2', and 'student_5_3' are not introduced in the figure caption or text; clarify the mapping to hyperparameter sets to improve readability.
- [Throughout] Typographical issues: 'WepP' should be 'WebP'; §5.1.2 contains an extra space in 'weigthing'; the sentence following Figure 8 repeats 'Figure 8' twice. A light proofreading pass is needed.
- [§4.3 and Figures 19–21] When comparing with JPEG/WebP/JPEG-2000, the operating points in Table 5 are extremely different (e.g., PSNR 70 dB at 13.45 bpp for JPEG-2000). Discussing the RD and energy trade-offs would be clearer if the comparisons were at matched quality or bit rate, or if this limitation is explicitly acknowledged.
Circularity Check
No significant circularity: student RD results are measured against external Kodak/compressAI references, and the KD losses in Eqs. (1)-(3) are the method being evaluated, not a prediction derived from a fitted input.
full rationale
The paper's derivation chain is self-contained. The teacher is an external compressAI pre-trained checkpoint (Sec. 4.2), the student objectives are the stated distillation losses of Eqs. (1)-(3), and the reported outcomes are PSNR/BPP on the external Kodak set plus measured FLOPs, throughput, and energy on an RTX 3090 (Tables 2-4). None of these quantities is defined in terms of the headline result, and no equation reduces to an input by construction: the RD term in Eq. (3) is the standard rate-distortion objective, and the MSE feature/reconstruction terms in Eqs. (1)-(2) are the method being evaluated, not a prediction derived from it. The two references to prior work by co-authors ([12], [13]) are related-work mentions about FPGA deployment and are not used to justify the students' RD performance. The main weaknesses are experimental-control and selection issues rather than circularity: no same-size student trained from scratch without KD is compared (Sec. 5.1.1, Fig. 2), and the 64-channel size/KD weights were chosen after inspecting Kodak results (Sec. 5.1.2). These are important validity concerns for the causal claim that KD itself improves compression, but they do not make any stated result equivalent to its own input by definition. Therefore the appropriate circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (3)
- KD loss weights lambda1, lambda2, lambda3 (lambda4 in Eq. 2) =
(0.2, 0.2, 0.4) default; sweep (0.1, 0.1, 0.8), (0.3, 0.3, 0.4), (0.4, 0.4, 0.2)
- RD Lagrange multiplier lambda =
0.0018 to 0.18 (CompressAI quality 1-5 table)
- Number of channels N =
16, 32, 64, 96, 112
axioms (5)
- domain assumption Knowledge distillation transfers generalization: a student trained on a teacher's outputs generalizes better than one trained alone (Hinton et al. [8]).
- domain assumption The compressAI scale-hyperprior checkpoints are valid teacher models and fair baselines for the reported comparisons.
- domain assumption The entropy model's negative log-likelihood (Eq. 3) gives an accurate estimate of the true coding rate for both teacher and students.
- domain assumption The training recipe (Vimeo90K patches, 4-5M steps, batch 16, LR 1e-4 with plateau decay) reproduces CompressAI training conditions closely enough for fair comparison.
- domain assumption PSNR/BPP averaged over the 24 Kodak images from a single training run per configuration are sufficient to compare RD curves.
Cite this review
Pith. "Pith review of Efficient Learned Image Compression Through Knowledge Distillation." pith.science (2026). https://pith.science/paper/UZNRNYMT
@misc{pith2026250910366,
author = {Pith},
title = {Pith review of: Efficient Learned Image Compression Through Knowledge Distillation},
year = {2026},
howpublished = {\url{https://pith.science/paper/UZNRNYMT}},
note = {Machine review of arXiv:2509.10366}
}
read the original abstract
Learned image compression sits at the intersection of machine learning and image processing. With advances in deep learning, neural network-based compression methods have emerged. In this process, an encoder maps the image to a low-dimensional latent space, which is then quantized, entropy-coded into a binary bitstream, and transmitted to the receiver. At the receiver end, the bitstream is entropy-decoded, and a decoder reconstructs an approximation of the original image. Recent research suggests that these models consistently outperform conventional codecs. However, they require significant processing power, making them unsuitable for real-time use on resource-constrained platforms, which hinders their deployment in mainstream applications. This study aims to reduce the resource requirements of neural networks used for image compression by leveraging knowledge distillation, a training paradigm where smaller neural networks, partially trained on the outputs of larger, more complex models, can achieve better performance than when trained independently. Our work demonstrates that knowledge distillation can be effectively applied to image compression tasks: i) across various architecture sizes, ii) to achieve different image quality/bit rate tradeoffs, and iii) to save processing and energy resources. This approach introduces new settings and hyperparameters, and future research could explore the impact of different teacher models, as well as alternative loss functions. Knowledge distillation could also be extended to transformer-based models. The code is publicly available at: https://github.com/FABallemand/PRIM .
Figures
Forward citations
Cited by 1 Pith paper
-
KD-NVC: A Search-and-Distill Framework to Accelerate Neural Video Coding
KD-NVC combines acceleration-efficiency neural architecture search with energy-aware feature distillation to produce neural video codecs that reach 69 FPS 1080p decoding on RTX 5060 while matching VTM-LDB rate-distort...
Reference graph
Works this paper leans on
-
[1]
Simoncelli.End-to-end optimization of nonlinear transform codes for perceptual quality
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli.End-to-end optimization of nonlinear transform codes for perceptual quality. 2016. arXiv: 1607.05006 [cs.IT]. URL: https://arxiv.org/abs/1607.05006
Pith/arXiv arXiv 2016
-
[2]
Simoncelli.End-to-end Optimized Image Compression
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli.End-to-end Optimized Image Compression. 2017. arXiv: 1611.01704 [cs.CV]. URL: https://arxiv.org/abs/1611.01704
Pith/arXiv arXiv 2017
-
[3]
Variational image compression with a scale hyperprior
Johannes Ballé et al. Variational image compression with a scale hyperprior . 2018. arXiv: 1802 . 01436 [eess.IV]. URL: https://arxiv.org/abs/1802.01436
Pith/arXiv arXiv 2018
-
[4]
Nabajeet Barman, Maria G. Martini, and Yuriy Reznik. Bjøntegaard Delta (BD): A Tutorial Overview of the Metric, Evolution, Challenges, and Recommendations . 2024. arXiv: 2401 . 04039 [cs.MM]. URL: https : //arxiv.org/abs/2401.04039
Pith/arXiv arXiv 2024
-
[5]
Image compression
compressAI. Image compression. URL: https://interdigitalinc.github.io/CompressAI/zoo.html# training. (consulté: 30/01/2025)
2025
-
[6]
Haisheng Fu et al.Fast and High-Performance Learned Image Compression With Improved Checkerboard Context Model, Deformable Residual Module, and Knowledge Distillation. 2023. arXiv: 2309.02529 [eess.IV]. URL: https://arxiv.org/abs/2309.02529
Pith/arXiv arXiv 2023
-
[7]
Microdosing: Knowledge Distillation for GAN-based Compression
Leonhard Helminger et al. Microdosing: Knowledge Distillation for GAN-based Compression. 2022. arXiv: 2201.02624 [eess.IV]. URL: https://arxiv.org/abs/2201.02624
Pith/arXiv arXiv 2022
-
[8]
Distilling the Knowledge in a Neural Network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network . 2015. arXiv: 1503.02531 [stat.ML]. URL: https://arxiv.org/abs/1503.02531
Pith/arXiv arXiv 2015
-
[9]
FPX-NIC: An FPGA-Accelerated 4K Ultra-High-Definition Neural Video Coding System
Chuanmin Jia et al. “FPX-NIC: An FPGA-Accelerated 4K Ultra-High-Definition Neural Video Coding System”. In: IEEE Transactions on Circuits and Systems for Video Technology 32.9 (2022), pp. 6385–6399. DOI: 10. 1109/TCSVT.2022.3164059
arXiv 2022
-
[10]
Cross-Architecture Knowledge Distillation
Yufan Liu et al. Cross-Architecture Knowledge Distillation. 2022. arXiv: 2207.05273 [cs.CV]. URL: https: //arxiv.org/abs/2207.05273
Pith/arXiv arXiv 2022
-
[11]
Neural/Learned Image Compression: An Overview
loijilai. Neural/Learned Image Compression: An Overview. URL: https://medium.com/@loijilai_me/ learned-image-compression-an-overview-625f3ab709f2 . (consulté: 30/01/2025)
2025
-
[12]
Alaa Mazouz et al. “Lightweight Embedded FPGA Deployment of Learned Image Compression with Knowledge Distillation and Hybrid Quantization”. In: arXiv preprint arXiv:2503.04832 (2025)
Pith/arXiv arXiv 2025
-
[13]
Security and real-time fpga integration for learned image compression
Alaa Mazouz et al. “Security and real-time fpga integration for learned image compression”. In: arXiv preprint arXiv:2503.04867 (2025)
Pith/arXiv arXiv 2025
-
[14]
Joint Autoregressive and Hierarchical Priors for Learned Image Compression
David Minnen, Johannes Ballé, and George Toderici. Joint Autoregressive and Hierarchical Priors for Learned Image Compression. 2018. arXiv: 1809.02736 [cs.CV]. URL: https://arxiv.org/abs/1809.02736. 14
Pith/arXiv arXiv 2018
-
[15]
David Minnen and Saurabh Singh.Channel-wise Autoregressive Entropy Models for Learned Image Compression
-
[16]
Stanford EE274: Data Compression I 2023 I Lecture 16 - Learnt Image Compression
Stanford Online. Stanford EE274: Data Compression I 2023 I Lecture 16 - Learnt Image Compression. URL: https : / / www . youtube . com / watch ? v = H7dvh35xNuE & ab _ channel = StanfordOnline. (consulté: 30/01/2025)
2023
-
[17]
FPGA Codec System of Learned Image Compression With Algorithm-Architecture Co-Optimization
Heming Sun, Qingyang Yi, and Masahiro Fujita. “FPGA Codec System of Learned Image Compression With Algorithm-Architecture Co-Optimization”. In: IEEE Journal on Emerging and Selected Topics in Circuits and Systems 14.2 (2024), pp. 334–347. DOI: 10.1109/JETCAS.2024.3386328
arXiv 2024
-
[18]
FPGA Codec System of Learned Image Compression with Algorithm-Architecture Co-Optimization
Heming Sun, Qingyang Yi, and Masahiro Fujita. “FPGA Codec System of Learned Image Compression with Algorithm-Architecture Co-Optimization”. In: IEEE Journal on Emerging and Selected Topics in Circuits and Systems (2024)
2024
-
[19]
Training data-efficient image transformers & distillation through attention
Hugo Touvron et al. Training data-efficient image transformers & distillation through attention. 2021. arXiv: 2012.12877 [cs.CV]. URL: https://arxiv.org/abs/2012.12877
Pith/arXiv arXiv 2021
-
[20]
The Devil Is in the Details: Window-based Attention for Image Compression
Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. The Devil Is in the Details: Window-based Attention for Image Compression. 2022. arXiv: 2203.08450 [eess.IV]. URL: https://arxiv.org/abs/2203.08450. A Reproducibility A.1 Implementation Details All experiments are conducted using Python 3.12.7 and the version 1.2.6 of compressAI (see requirements file for ot...
Pith/arXiv arXiv 2022
-
[2020]
URL: https://arxiv.org/abs/2007.08739
arXiv: 2007.08739 [eess.IV]. URL: https://arxiv.org/abs/2007.08739
Pith/arXiv arXiv 2007
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.