Learning Optimal Linear Block Transform by Rate Distortion Minimization

Alessandro Gnutti; Chia-Hao Kao; Riccardo Leonardi; Wen-Hsiao Peng

REVIEW 3 major objections 5 minor 24 references

Learning Optimal Linear Block Transform by Rate Distortion Minimization

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a linear block transform trained end-to-end to minimize a rate-distortion loss beats the DCT by about 12% in BD-rate and the SOT by about 5% on residual blocks.

desk verdict A genuinely new learned linear transform for residual coding that reports real BD-rate gains, but the missing orthonormality check makes the headline numbers hard to fully trust. read the letter →

arxiv 2411.18494 v1 pith:IRLL7LI7 submitted 2024-11-27 eess.IV

Alessandro Gnutti , Chia-Hao Kao , Wen-Hsiao Peng , Riccardo Leonardi This is my paper · ORCID

classification eess.IV

keywords learnedlineartransformrate-distortionoptimizationresidualcodingDCTreplacementGaussianentropymodelblockimagecompressionvideo

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a single linear block transform learned by gradient descent can beat the DCT and prior data-driven transforms at coding residual blocks, without abandoning the block-transform architecture used in video and image standards. It introduces RDLT, a transform matrix updated end-to-end to minimize a rate-distortion loss, with coefficient rate estimated by a parametric Gaussian entropy model. On a mixed dataset of 8x8, 16x16, and 32x32 residual blocks, RDLT reports about 12% BD-rate saving over the DCT and about 5% over SOT. If true, the result matters because it offers a standards-compatible path to coding gains that does not require the nonlinear, decoder-heavy machinery of end-to-end learned codecs.

What carries the argument

The load-bearing object is the $n^{2}$ by $n^{2}$ transform matrix M itself, treated as a one-layer linear network whose basis functions are applied as y = xM. Training uses additive uniform noise in place of hard rounding, a Gaussian entropy model to estimate coefficient rate, and variable-rate training where a small two-layer network predicts the quantization step-size Q from $\lambda$. Reconstruction uses the transpose M^T as the inverse, a choice the paper says enforces orthonormality. The mechanism that carries the argument is the Gaussian entropy model: replacing crude proxies like the l0 norm with a differentiable estimate of true coefficient entropy is what lets gradient descent find transforms that outperform KLT and SOT.

What would settle it

Retrain RDLT under the paper's setup and compute the orthogonality error ||M M^T - I|| plus the PSNR gap between reconstructing with M^T and with the numerically inverted M; a non-negligible gap would mean part of the reported RD gain is an artifact of using a non-inverse at the decoder.

Watch

Extended reading notes

Core claim

The central claim is that the rate-distortion-optimal linear transform is learnable and measurably better than the usual choices. The paper trains the transform matrix M by back-propagating through a differentiable proxy of quantization and minimizing L = D + lambda R, where the rate term is the negative log-likelihood of the quantized coefficients under a Gaussian entropy model with learnable per-coefficient means and variances. It reports about 12% BD-rate saving over the DCT and about 5% over SOT on residual blocks of sizes 8, 16, and 32, with gains appearing at each block size. It further reports that swapping RDLT for DCT inside the multi-transform selection loop of a VVC-like codec improves rate-distortion performance. The paper's intended significance is that accurate rate estimation, not just variance-based linear optimality, is what makes a linear transform effective in practical compression.

Load-bearing premise

The load-bearing premise is that the learned matrix stays close enough to orthonormal that using its transpose as the inverse reconstruction is valid, because the training loss only nudges M in that direction and the paper reports no orthogonality measure.

Editorial extensions

If this is right

If RDLT's reported gains hold, replacing DCT with the learned matrix inside existing block codecs could yield around 12% BD-rate savings without changing the decoder's block-transform structure.
The per-block-size results suggest the gains are not an artifact of one block size; separate 8x8, 16x16, and 32x32 transforms each beat DCT.
The VVC-like MTS simulation implies RDLT can be added as one transform among several and chosen by rate-distortion optimization, rather than requiring a fixed single transform.
The comparison with SOT indicates that a differentiable Gaussian rate model outperforms an l0-based rate proxy for learning linear transforms.
Because the resulting transform is linear and intended to be invertible through orthonormality, integration into standards could be decoder-compatible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the reported gains are measured on HEVC intra residuals; extending RDLT to inter residuals or other prediction modes may require retraining, and the gain size is an open question.
Inference: because orthonormality is only encouraged by using M^T, an explicit orthogonality constraint or post-training orthogonalization could make the gains reproducible in a real codec where the inverse must be exact.
Inference: the method could be extended to a family of learned transforms, one per intra-prediction mode, along the lines of mode-dependent transforms, potentially compounding the RD gain.
Inference: since the entropy model is trained jointly, one could also tune the quantizer step-size schedule with the transform rather than treating Q as an input, but the paper does not test this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here is the short version. RDLT is worth knowing about: it is the first linear block transform trained end-to-end by minimizing a rate-distortion loss with a parametric Gaussian entropy model, and the authors report about 12% BD-rate saving over the DCT and 5% over SOT on residual blocks. Prior data-driven transforms (KLT) are optimized for linear approximation, and SOT uses an l0 rate proxy, so the RD-based training objective is genuinely new. The evaluation is also more concrete than many papers in this area: rates come from actual arithmetic coding on held-out data, and the code is public. That is real evidence and should be credited.

The load-bearing soft spot is orthonormality. Equation (3) uses M^T as the inverse, justified only if M is orthonormal. The authors say that choosing M^T 'enforces orthonormality,' but there is no orthonormality constraint, penalty, projection, or post-training measure of M M^T reported. Ordinary SGD on an unconstrained M will not keep M orthogonal. If M drifts, the decoder is a tied linear autoencoder, not an inverse transform, and the comparison against DCT/KLT/SOT—all with true orthonormal inverses—may be unfair. The reported PSNR and BD-rate could partly reflect decoder-side flexibility rather than a better transform. This is fixable: the authors should report the orthogonality error of the learned matrices and either enforce orthonormality explicitly (e.g., via a Stiefel projection or a penalty) or use M^{-1} in reconstruction for a fair comparison. If the released matrices turn out to be near-orthogonal, the central claim likely survives. As written, it is a significant gap, not a fatal one.

Minor issues: no error bars or multi-seed results; evaluation is on a held-out split of the same distribution used for training; the VVC-like integration is described briefly; and the code link lacks a commit hash. The citation pattern is normal, and the paper is honestly framed.

Who this is for: researchers in image/video compression, especially those working on transform coding or learned codecs. It is a useful new baseline. I would send it to peer review rather than desk-reject it: the idea is sound, the experiments are mostly well done, and the orthonormality concern is a specific, addressable technical point that a good referee can force the authors to clarify.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes RDLT, a linear block transform for residual blocks, trained end-to-end by minimizing a rate-distortion loss with a learned Gaussian entropy model and variable-rate training. The transform matrix M is applied as y = xM, quantization is modeled as uniform noise during training, and reconstruction uses M^T in Eq. (3). Experiments compare RDLT with DCT, KLT, and SOT on residual blocks of sizes 8, 16, and 32, using arithmetic coding on a held-out split; the authors report about 12% BD-rate saving over DCT and 5% over SOT, and also simulate integration of RDLT into a VVC-like MTS framework. The central claim is that a simple learned linear transform can substantially improve standard-compatible block transform coding.

Significance. If the reported gains hold under a standards-compliant inverse transform, this is a practically interesting result: it would show that a rudimentary linear transform trained with an RD objective and a modern entropy model can beat both the DCT and prior data-driven transforms such as KLT and SOT. The paper has genuine strengths: the rate term is evaluated with actual arithmetic coding rather than only the learned model, the code is promised to be public, and the comparison includes several relevant baselines. However, the central claim depends on the learned M being orthonormal, since Eq. (3) uses M^T as the inverse, and the currently reported experiments do not establish that property. The lack of image-disjoint evaluation and of any uncertainty quantification further weakens the strength of the reported performance margins.

major comments (3)

[Proposed method / Learning procedure, Eqs. (1)-(3)]
[Experiments / Experimental settings]
[Experiments / Rate-distortion comparison, Table 1]

minor comments (5)

[Learning procedure]
[References]
[Title/Abstract]
[Additional investigation / MTS experiment]
[Figure 5]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the performance claim rests on a held-out arithmetic-coding benchmark rather than the training objective.

full rationale

The paper's central claim—that RDLT yields around 12% BD-rate saving over DCT and 5% over SOT—is supported by a held-out evaluation in which rate is computed with arithmetic coding on quantized coefficients under the same protocol for all transforms. The Gaussian entropy model is used only as a differentiable training proxy for rate; it is not the quantity reported as the result. Training M to minimize the RD loss that also defines the evaluation metric is standard empirical optimization, not circularity, because the evaluation is performed on a held-out split and the arithmetic-coded rate is an external measurement independent of the learned entropy model's parameters. The paper's citations to prior end-to-end compression work [9,22] are external, and the authors' own GFT papers [18-20] appear only as related-work context, not as load-bearing justification for RDLT's design or performance. The orthonormality issue raised in the skeptic headline—using M^T as the inverse in Eq. (3) without an explicit orthonormality constraint—is a correctness and verification risk, not a circularity: it does not make the derivation reduce to its own inputs. No self-definitional, fitted-input-as-prediction, or self-citation-chain pattern is present, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger lists the learned parameters that the central performance claim depends on, plus the domain and modeling assumptions behind the method. No new physical entities are introduced; RDLT is a learned matrix, not a new force or particle.

free parameters (3)

Transform matrix M = learned, n^4 entries per block size (n=8,16,32)
The central learned object; trained by minimizing the RD loss on residual blocks, with no closed-form solution provided.
Gaussian entropy model parameters (means and variances) = 2n^2 per block size, learned
Used to estimate rate during training via Eq. (6); fitted jointly with the transform matrix.
Q prediction network weights = learned two-layer network
Maps lambda values to quantization step sizes during variable-rate training; trained jointly with M.

assumptions (5)

domain assumption Residual blocks from HEVC intra prediction on mixed datasets are representative of residuals in practical image/video coding.
The training set is built from ImageNet, USC-SIPI, and HEVC sequences; transfer to VVC-like residuals is assumed without retraining.
domain assumption The Gaussian entropy model without hyperprior gives a rate estimate accurate enough to find a near-optimal transform.
The rate term in Eq. (6) relies on this; the authors state that accurate rate estimation is crucial for the method.
standard math Additive uniform noise approximates hard quantization during training.
Standard practice in end-to-end learned compression, used in Eq. (5) to keep the process differentiable.
ad hoc to paper The learned transform matrix M is orthonormal, so M^T is a valid inverse.
The paper uses M^T in reconstruction Eq. (3) and claims this enforces orthonormality, but no explicit orthonormality constraint or verification is reported.
standard math Arithmetic coding achieves the entropy of the quantized coefficients.
Used to compute the actual rates in the RD evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning Optimal Linear Block Transform by Rate Distortion Minimization." pith.science (2026). https://pith.science/paper/IRLL7LI7

@misc{pith2026241118494,
  author       = {Pith},
  title        = {Pith review of: Learning Optimal Linear Block Transform by Rate Distortion Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRLL7LI7}},
  note         = {Machine review of arXiv:2411.18494}
}

read the original abstract

Linear block transform coding remains a fundamental component of image and video compression. Although the Discrete Cosine Transform (DCT) is widely employed in all current compression standards, its sub-optimality has sparked ongoing research into discovering more efficient alternative transforms even for fields where it represents a consolidated tool. In this paper, we introduce a novel linear block transform called the Rate Distortion Learned Transform (RDLT), a data-driven transform specifically designed to minimize the rate-distortion (RD) cost when approximating residual blocks. Our approach builds on the latest end-to-end learned compression frameworks, adopting back-propagation and stochastic gradient descent for optimization. However, unlike the nonlinear transforms used in variational autoencoder (VAE)-based methods, the goal is to create a simpler yet optimal linear block transform, ensuring practical integration into existing image and video compression standards. Differently from existing data-driven methods that design transforms based on sample covariance matrices, such as the Karhunen-Lo\`eve Transform (KLT), the proposed RDLT is directly optimized from an RD perspective. Experimental results show that this transform significantly outperforms the DCT or other existing data-driven transforms. Additionally, it is shown that when simulating the integration of our RDLT into a VVC-like image compression framework, the proposed transform brings substantial improvements. All the code used in our experiments has been made publicly available at [1].

Figures

Figures reproduced from arXiv: 2411.18494 by the authors.

**Figure 2.** Learning framework of the proposed RDLT. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Visualization of the basis functions of the RDLT and DCT with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: RD performance comparison [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Rate-distortion comparison for separate block sizes [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Rate-distortion comparison under multi transform selection in VVC. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages

[1]

Rate distortion learned transform,

“Rate distortion learned transform,” https://github.com/JoeK6279/Rate_ Distortion_Learned_Transform, Last accessed on 2024-10-11

work page 2024
[2]

Bhaskaran and K

V. Bhaskaran and K. Konstantinides, The H.261 Video Coding Standard, pp. 195–206, Springer US, Boston, MA, 1995

work page 1995
[3]

Intra coding of the HEVC standard,

J. Lainema, F. Bossen, W. J. Han, J. Min, and K. Ugur, “Intra coding of the HEVC standard,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1792–1801, 2012

work page 2012
[4]

Overview of the versatile video coding (VVC) standard and its applications,

B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the versatile video coding (VVC) standard and its applications,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736–3764, 2021

work page 2021
[5]

The JPEG still picture compression standard,

G. K. Wallace, “The JPEG still picture compression standard,” IEEE Trans. on Consumer Electronics, vol. 38, no. 1, pp. xviii–xxxiv, 1992

work page 1992
[6]

BPG Image format,

“BPG Image format,” https://bellard.org/bpg/, Last accessed on 2022-11-15

work page 2022
[7]

Ahmed and K

N. Ahmed and K. R. Rao, Orthogonal Transforms for Digital Signal Processing, Springer-Verlag, Berlin, 1975

work page 1975
[8]

Auto-encoding variational bayes,

Diederik P Kingma and Max Welling, “Auto-encoding variational bayes,” 2022

2022

Show all 24 references

[9]

End-to-end optimized image compression,

Johannes Ball´ e, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimized image compression,” 2017

2017
[10]

Video coding with rate-distortion op- timized transform,

Xin Zhao, Li Zhang, Siwei Ma, and Wen Gao, “Video coding with rate-distortion op- timized transform,” IEEE transactions on Circuits and Systems for Video Technology, vol. 22, no. 1, pp. 138–151, 2011

2011
[11]

Improved H.264 intra coding based on bi-directional intra prediction, directional transform, and adaptive coefficient scanning,

Yan Ye and Marta Karczewicz, “Improved H.264 intra coding based on bi-directional intra prediction, directional transform, and adaptive coefficient scanning,” in 2008 15th IEEE International Conference on Image Processing. IEEE, 2008, pp. 2116–2119

2008
[12]

On intra coding using mode dependent 2D- KLT,

Seishi Takamura and Atsushi Shimizu, “On intra coding using mode dependent 2D- KLT,” in 2013 Picture Coding Symposium (PCS). IEEE, 2013, pp. 137–140

2013
[13]

Non-separable mode dependent transforms for intra coding in HEVC,

Adri` a Arrufat, Pierrick Philippe, and Oliver D´ eforges, “Non-separable mode dependent transforms for intra coding in HEVC,” in 2014 IEEE Visual Communications and Image Processing Conference. IEEE, 2014, pp. 61–64

2014
[14]

Graph signal processing: Overview, challenges, and applications,

A. Ortega, P. Frossard, J. Kovaˇ cevi´ c, J. M. F. Moura, and P. Vandergheynst, “Graph signal processing: Overview, challenges, and applications,” Proceedings of the IEEE, vol. 106, no. 5, pp. 808–828, May 2018

2018
[15]

Graph transform optimiza- tion with application to image compression,

Giulia Fracastoro, Dorina Thanou, and Pascal Frossard, “Graph transform optimiza- tion with application to image compression,” IEEE Transactions on Image Processing, vol. 29, pp. 419–432, 2019

2019
[16]

Multiresolution graph Fourier transform for compression of piecewise smooth images,

Wei Hu, Gene Cheung, Antonio Ortega, and Oscar C. Au, “Multiresolution graph Fourier transform for compression of piecewise smooth images,” IEEE Transactions on Image Processing, vol. 24, no. 1, pp. 419–433, 2015

2015
[17]

Graph-based transforms for video coding,

Hilmi E. Egilmez, Yung-Hsuan Chao, and Antonio Ortega, “Graph-based transforms for video coding,” IEEE Transactions on Image Processing, vol. 29, 2020

2020
[18]

Symmetry-based graph Fourier transforms for image representation,

Alessandro Gnutti, Fabrizio Guerrini, Riccardo Leonardi, and Antonio Ortega, “Symmetry-based graph Fourier transforms for image representation,” in IEEE Inter- national Conference on Image Processing, 2018, pp. 2575–2579

2018
[19]

Cod- ing of image intra prediction residuals using symmetric graphs,

Alessandro Gnutti, Fabrizio Guerrini, Riccardo Leonardi, and Antonio Ortega, “Cod- ing of image intra prediction residuals using symmetric graphs,” in IEEE International Conference on Image Processing, 2019, pp. 131–135

2019
[20]

Symmetry-based graph Fourier transforms: Are they optimal for image compres- sion?,

Alessandro Gnutti, Fabrizio Guerrini, Riccardo Leonardi, and Antonio Ortega, “Symmetry-based graph Fourier transforms: Are they optimal for image compres- sion?,” in IEEE International Conference on Image Processing, 2021, pp. 1594–1598

2021
[21]

Approximation and compression with sparse orthonormal transforms,

Osman Gokhan Sezer, Onur G. Guleryuz, and Yucel Altunbasak, “Approximation and compression with sparse orthonormal transforms,” IEEE Transactions on Image Processing, vol. 24, no. 8, pp. 2328–2343, 2015

2015
[22]

Variational image compression with a scale hyperprior,

Johannes Ball´ e, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston, “Variational image compression with a scale hyperprior,” in International Conference on Learning Representations, 2018

2018
[23]

Imagenet: A large-scale hierarchical image database,

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

2009
[24]

The USC-SIPI image database,

“The USC-SIPI image database,” http://sipi.usc.edu/database/

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Rate distortion learned transform,

“Rate distortion learned transform,” https://github.com/JoeK6279/Rate_ Distortion_Learned_Transform, Last accessed on 2024-10-11

work page 2024

[2] [2]

Bhaskaran and K

V. Bhaskaran and K. Konstantinides, The H.261 Video Coding Standard, pp. 195–206, Springer US, Boston, MA, 1995

work page 1995

[3] [3]

Intra coding of the HEVC standard,

J. Lainema, F. Bossen, W. J. Han, J. Min, and K. Ugur, “Intra coding of the HEVC standard,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1792–1801, 2012

work page 2012

[4] [4]

Overview of the versatile video coding (VVC) standard and its applications,

B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the versatile video coding (VVC) standard and its applications,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736–3764, 2021

work page 2021

[5] [5]

The JPEG still picture compression standard,

G. K. Wallace, “The JPEG still picture compression standard,” IEEE Trans. on Consumer Electronics, vol. 38, no. 1, pp. xviii–xxxiv, 1992

work page 1992

[6] [6]

BPG Image format,

“BPG Image format,” https://bellard.org/bpg/, Last accessed on 2022-11-15

work page 2022

[7] [7]

Ahmed and K

N. Ahmed and K. R. Rao, Orthogonal Transforms for Digital Signal Processing, Springer-Verlag, Berlin, 1975

work page 1975

[8] [8]

Auto-encoding variational bayes,

Diederik P Kingma and Max Welling, “Auto-encoding variational bayes,” 2022

2022