Pith. sign in

REVIEW 4 minor 45 references

Three neural method families turn classical rate-distortion problems into scalable, differentiable deep-learning objectives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Neural learn-to-compute methods recast rate-distortion, information-bottleneck, and indirect RD problems as differentiable objectives via variational, mutual-information, and dual formulations.

T0 review reviewed 2026-07-12 challenge →

load-bearing objection Clean, usable survey that organizes three neural families for RD-type objectives; modest empirical side-by-side, no new theorems, but the taxonomy and caveats are solid enough to be useful.

arxiv 2607.05417 v1 pith:4HBVF7FZ submitted 2026-06-26 cs.IT math.IT

A Survey of Learn-to-Compute Paradigms for Rate-Distortion-Type Problems

Shitong Wu , Sicheng Xu , Lingyi Chen , Qiang Sun , Huihui Wu , Hao Wu , Wenyi Zhang This is my paper
classification cs.IT math.IT
keywords rate-distortion theorylearn-to-computevariational autoencodermutual information neural estimationdual formulationinformation bottleneckneural estimationhigh-dimensional information theory
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Classical rate-distortion theory tells how little information you must keep to reconstruct a signal within a chosen fidelity, but the classic algorithms that compute those limits explode in high dimensions. This survey shows that a learn-to-compute approach solves the problem by replacing probability tables with neural networks and replacing intractable mutual-information terms with differentiable surrogates. It organizes the existing work into three families—variational bounds, neural mutual-information estimators, and dual functionals—and reviews the consistency guarantees and training recipes that make each family work. Empirically the methods recover known rate-distortion and information-bottleneck curves on Gaussian and MNIST examples, while the survey itself notes remaining bias and scaling difficulties. A reader who cares about compression, representation learning, or information-theoretic analysis of deep models gains a coherent map of how to turn those classical limits into trainable losses.

Core claim

The paper establishes that variational inference, neural mutual-information estimation, and dual-form optimization together convert the classical rate-distortion Lagrangian (and its information-bottleneck and indirect-rate-distortion relatives) into sample-based, gradient-friendly objectives whose neural parameterizations are consistent under the stated theorems, thereby making high-dimensional rate-distortion computation feasible for deep learning.

What carries the argument

The Learn-to-Compute (LtC) paradigm: replace the original optimization over distributions by optimization over neural-parameterized families, and replace the original information-theoretic functional by a Monte-Carlo differentiable surrogate, so that stochastic gradient methods become applicable.

Load-bearing premise

That the modest synthetic and MNIST experiments used for comparison are representative enough to support the claimed bias sources and preferred operating regimes of the three method families.

What would settle it

Train all three families on a higher-dimensional continuous source whose true rate-distortion curve is known in closed form (or tightly bounded) and check whether the high-rate bias and complexity rankings reported in the survey’s Table I still hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. This survey organizes neural methods for computing rate-distortion (RD) and related objectives (information bottleneck, indirect RD) under a learn-to-compute (LtC) paradigm. It groups existing approaches into three families—variational reformulations (RD-VAE, VIB), neural mutual-information estimation (MINE-style min-max objectives such as RD-MINE and AIB), and dual-form optimization (NERD, MAIB, NEIRD)—and reviews their Lagrangian or dual derivations, algorithmic realizations, and consistency results drawn from the literature (Theorems 3.1, 4.2, 5.2). Section II unifies IB and iRD under generalized distortions; Sections III–V detail each family; Section VI reports illustrative experiments on a 5-D Gaussian source and MNIST and summarizes comparative trade-offs in Table I; Section VII outlines open challenges (variational gap, high-rate log-expectation bias, large-scale regimes).

Significance. The paper supplies a clear, pedagogically useful taxonomy that maps classical RD-type Lagrangians onto scalable differentiable objectives used in modern deep learning. By restating known dualities (Donsker–Varadhan, Dembo–Kontoyiannis) and consistency theorems side-by-side with concrete algorithmic templates and an honest experimental comparison, it lowers the barrier for researchers who wish to apply or extend these estimators. The explicit caveats already present in the abstract and Sections VI–VII (high-rate bias, remaining large-scale challenges) keep the claims proportionate. The work is therefore a solid contribution to the information-theory / machine-learning interface.

minor comments (4)
  1. In Section VI the batch size is written both as “12,800” and “12,80”; the latter is almost certainly a typographical error and should be corrected for consistency.
  2. Notation for the reconstruction variable Y is overloaded between classical RD and the IB setting (footnote on p. 3). A short reminder table or a consistent subscript convention would reduce occasional reader confusion.
  3. Table I lists per-iteration complexity O(BM) for dual-form methods; a brief parenthetical remark that M is typically chosen proportional to B (hence quadratic) would make the comparison fully self-contained.
  4. A few recent energy-based dual estimators (Refs. [37], [38]) are mentioned only in Remark 5.2; a one-sentence pointer in the main dual-form narrative would improve completeness without lengthening the survey.

Circularity Check

0 steps flagged

No significant circularity: survey taxonomy and cited consistency theorems stand independently of self-referential fits or definitions.

full rationale

This is a survey paper whose central claim is organizational and pedagogical: three families of neural methods (variational, MINE-style mutual-information estimation, dual-form) recast classical RD/IB/iRD Lagrangians into differentiable objectives, with consistency properties taken from the cited external theorems (Thm. 3.1 of Yang & Mandt [11], Thm. 4.2 of Belghazi et al. [13], Thm. 5.2 of Lei et al. [12]). No new quantitative prediction is derived that reduces by construction to a fitted parameter or to a self-defined quantity. Experimental curves in Sec. VI are compared against known closed-form RD expressions (Gaussian sources) or analytical lower bounds (MNIST IB/iRD), not against quantities fitted inside the paper. Self-citations to the authors’ own prior methods (MAIB [22], NEIRD [23], energy-based estimators [38]) appear only as illustrative instances inside the taxonomy; they are not invoked as uniqueness theorems that force the survey’s classification or its comparative claims in Table I. The abstract and Sec. VII already state the remaining limitations (high-rate log-expectation bias, variational gap, large-scale challenges), so the organizational claim does not rest on circular self-support. Consequently the derivation chain contains no self-definitional loop, no fitted-input-called-prediction, and no load-bearing self-citation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

As a survey the paper inherits the modeling assumptions of the methods it reviews rather than introducing new free parameters or entities. The main background axioms are standard information-theoretic definitions and the universal-approximation / empirical-process conditions used in the cited consistency theorems.

axioms (3)
  • standard math Mutual information equals KL divergence between joint and product of marginals; Donsker-Varadhan dual representation holds for the KL.
    Used throughout Sections IV and V to justify MINE-style and dual estimators.
  • domain assumption Neural networks can approximate the optimal critic or generator functions arbitrarily well under mild regularity (universal approximation + empirical-process convergence).
    Invoked for strong consistency of MINE (Thm. 4.2) and NERD (Thm. 5.2).
  • standard math The RD Lagrangian and its dual (or variational upper bounds) share the same global optimum under the stated distortion measures.
    Background for the equivalence claims that justify all three method families (Sections II-V).

reviewed 2026-07-12 · how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of Learn-to-Compute Paradigms for Rate-Distortion-Type Problems." pith.science (2026). https://pith.science/paper/4HBVF7FZ

@misc{pith2026260705417,
  author       = {Pith},
  title        = {Pith review of: A Survey of Learn-to-Compute Paradigms for Rate-Distortion-Type Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4HBVF7FZ}},
  note         = {Machine review of arXiv:2607.05417}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Rate-distortion (RD) theory and its related formulations play a central role in understanding efficient information representation, but computing these quantities remains challenging in high-dimensional settings. Classical iterative methods such as the Blahut-Arimoto algorithm become impractical in high-dimensional domains due to the curse of dimensionality and the intractability of mutual-information terms. Recent advances in neural modeling and differentiable optimization offer a promising alternative through a learn-to-compute paradigm, in which probability distributions and objective functionals are represented by flexible neural parameterizations. This survey presents an overview of neural approaches for evaluating the RD-type objectives. We present three representative families of methods: variational inference, neural mutual-information estimation, and dual-form optimization. By reviewing their theoretical principles, algorithmic techniques, and consistency properties, we elucidate how these methods collectively transform classical RD-type problems into scalable differentiable objectives suitable for deep learning, though challenges remain in large-scale applications. Together, these perspectives offer promising avenues for scaling information-theoretic computation to complex, high-dimensional machine learning systems.

Figures

Figures reproduced from arXiv: 2607.05417 by Hao Wu, Huihui Wu, Lingyi Chen, Qiang Sun, Shitong Wu, Sicheng Xu, Wenyi Zhang.

Figure 2
Figure 2. Figure 2: Comparison of neural IB estimators on the MNIST [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: NEIRD results on (left) the joint Gaussian model and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references

  1. [1]

    Coding theorems for a discrete source with a fidelity criterion,

    C. E. Shannon, “Coding theorems for a discrete source with a fidelity criterion,”IRE Nat. Conv. Rec, vol. 7, no. 142-163, p. 1, 1959

    1959

  2. [2]

    The information bottleneck method,

    N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” inProc. 37th Annual Allerton Conference on Communications, Control and Computing, 1999

    1999

  3. [3]

    Information transmission with addi- tional noise,

    R. Dobrushin and B. Tsybakov, “Information transmission with addi- tional noise,”IRE Transactions on Information Theory, vol. 8, no. 5, pp. 293–304, 1962

    1962

  4. [4]

    End-to-end optimized image compression,

    J. Ball ´e, V . Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” in5th International Conference on Learning Representa- tions, ICLR 2017, 2017

    2017

  5. [5]

    Deep learning and the information bot- tleneck principle,

    N. Tishby and N. Zaslavsky, “Deep learning and the information bot- tleneck principle,” in2015 IEEE Information Theory Workshop (ITW), 2015, pp. 1–5

    2015

  6. [6]

    Computation of channel capacity and rate-distortion func- tions,

    R. Blahut, “Computation of channel capacity and rate-distortion func- tions,”IEEE Transactions on Information Theory, vol. 18, no. 4, pp. 460–473, 1972

    1972

  7. [7]

    An algorithm for computing the capacity of arbitrary dis- crete memoryless channels,

    S. Arimoto, “An algorithm for computing the capacity of arbitrary dis- crete memoryless channels,”IEEE Transactions on Information Theory, vol. 18, no. 1, pp. 14–20, 1972

    1972

  8. [8]

    Bregman divergence based EM algorithm and its applica- tion to classical and quantum rate distortion theory,

    M. Hayashi, “Bregman divergence based EM algorithm and its applica- tion to classical and quantum rate distortion theory,”IEEE Transactions on Information Theory, vol. 69, no. 6, pp. 3460–3492, 2023

    2023

  9. [9]

    A con- strained BA algorithm for rate-distortion and distortion-rate functions,

    L. Chen, S. Wu, W. Ye, H. Wu, W. Zhang, H. Wu, and B. Bai, “A con- strained BA algorithm for rate-distortion and distortion-rate functions,” CSIAM Transactions on Applied Mathematics, vol. 6, no. 2, pp. 350–379, 2025

    2025

  10. [10]

    A communi- cation optimal transport approach to the computation of rate distortion functions,

    S. Wu, W. Ye, H. Wu, H. Wu, W. Zhang, and B. Bai, “A communi- cation optimal transport approach to the computation of rate distortion functions,” in2023 IEEE Information Theory Workshop (ITW), 2023, pp. 92–96

    2023

  11. [11]

    Towards empirical sandwich bounds on the rate-distortion function,

    Y . Yang and S. Mandt, “Towards empirical sandwich bounds on the rate-distortion function,” inInternational Conference on Learning Rep- resentations, 2022

    2022

  12. [12]

    Neural estimation of the rate- distortion function with applications to operational source coding,

    E. Lei, H. Hassani, and S. S. Bidokhti, “Neural estimation of the rate- distortion function with applications to operational source coding,”IEEE Journal on Selected Areas in Information Theory, vol. 3, no. 4, pp. 674– 686, 2022

    2022

  13. [13]

    Mutual information neural estimation,

    M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y . Bengio, A. Courville, and D. Hjelm, “Mutual information neural estimation,” inInternational Conference on Machine Learning. PMLR, 2018, pp. 531–540

    2018

  14. [14]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,”Stat, vol. 1050, p. 1, 2014

    2014

  15. [15]

    Generative adversarial nets,

    I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” Advances in Neural Information Processing Systems, vol. 27, 2014

    2014

  16. [16]

    Beyond normal: On the evaluation of mutual information estimators,

    P. Czy ˙z, F. Grabowski, J. V ogt, N. Beerenwinkel, and A. Marx, “Beyond normal: On the evaluation of mutual information estimators,”Advances in neural information processing systems, vol. 36, pp. 16 957–16 990, 2023

    2023

  17. [17]

    The information bottleneck problem and its applications in machine learning,

    Z. Goldfeld and Y . Polyanskiy, “The information bottleneck problem and its applications in machine learning,”IEEE Journal on Selected Areas in Information Theory, vol. 1, no. 1, pp. 19–38, 2020

    2020

  18. [18]

    An introduction to neural data compression,

    Y . Yang, S. Mandt, and L. Theis, “An introduction to neural data compression,”Foundations and Trends in Computer Graphics and Vision, vol. 15, no. 2, pp. 113–200, 2023

    2023

  19. [19]

    Deep variational information bottleneck,

    A. A. Alemi, I. Fischer, J. V . Dillon, and K. Murphy, “Deep variational information bottleneck,” inProc. 5th International Conference on Learn- ing Representations (ICLR), Toulon, France, Apr. 2017, pp. 1–5

    2017

  20. [20]

    Rate distortion via constrained estimated mutual information minimization,

    D. Tsur, B. Huleihel, and H. Permuter, “Rate distortion via constrained estimated mutual information minimization,” in2023 IEEE International Symposium on Information Theory (ISIT). IEEE, 2023, pp. 695–700

    2023

  21. [21]

    Adversarial information bottleneck,

    P. Zhai and S. Zhang, “Adversarial information bottleneck,”IEEE Trans- actions on Neural Networks and Learning Systems, vol. 35, no. 1, pp. 221–230, 2022

    2022

  22. [22]

    Neural estimation of the information bottleneck based on a mapping approach,

    L. Chen, S. Wu, S. Xu, W. Zhang, and H. Wu, “Neural estimation of the information bottleneck based on a mapping approach,” in2024 IEEE Information Theory Workshop (ITW), 2024, pp. 532–536

    2024

  23. [23]

    Data-driven neural estimation of indirect rate-distortion function,

    Z. Yu, Q. Sun, and W. Zhang, “Data-driven neural estimation of indirect rate-distortion function,” inICC 2025 - IEEE International Conference on Communications, 2025, pp. 2827–2832

    2025

  24. [24]

    Rate-distortion theory,

    T. Berger, “Rate-distortion theory,”Wiley Encyclopedia of Telecommuni- cations, 2003

    2003

  25. [25]

    Indirect rate distortion problems,

    H. Witsenhausen, “Indirect rate distortion problems,”IEEE Transactions on Information Theory, vol. 26, no. 5, pp. 518–521, 1980

    1980

  26. [26]

    Wasserman,All of nonparametric statistics

    L. Wasserman,All of nonparametric statistics. Springer, 2006

    2006

  27. [27]

    Stochastic backpropaga- tion and approximate inference in deep generative models,

    D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropaga- tion and approximate inference in deep generative models,” inInterna- tional Conference on Machine Learning. PMLR, 2014, pp. 1278–1286

    2014

  28. [28]

    beta-V AE: Learning basic visual concepts with a constrained variational framework

    I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner, “beta-V AE: Learning basic visual concepts with a constrained variational framework.”ICLR, vol. 3, 2017

    2017

  29. [29]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inInternational Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241

    2015

  30. [30]

    Variational inference with normalizing flows,

    D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” inInternational Conference on Machine Learning. PMLR, 2015, pp. 1530–1538

    2015

  31. [31]

    Multi-rate V AE: Train once, get the full rate-distortion curve,

    J. Bae, M. R. Zhang, M. Ruan, E. Wang, S. Hasegawa, J. Ba, and R. B. Grosse, “Multi-rate V AE: Train once, get the full rate-distortion curve,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  32. [32]

    Information dropout: Learning optimal rep- resentations through noisy computation,

    A. Achille and S. Soatto, “Information dropout: Learning optimal rep- resentations through noisy computation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 12, pp. 2897–2905, 2018

    2018

  33. [33]

    Nonlinear information bottleneck,

    A. Kolchinsky, B. D. Tracey, and D. H. Wolpert, “Nonlinear information bottleneck,”Entropy, vol. 21, no. 12, p. 1181, 2019

    2019

  34. [34]

    Asymptotic evaluation of certain Markov process expectations for large time, I,

    M. D. Donsker and S. S. Varadhan, “Asymptotic evaluation of certain Markov process expectations for large time, I,”Communications on Pure and Applied Mathematics, vol. 28, no. 1, pp. 1–47, 1975

    1975

  35. [35]

    Source coding, large deviations, and approximate pattern matching,

    A. Dembo and L. Kontoyiannis, “Source coding, large deviations, and approximate pattern matching,”IEEE Transactions on Information The- ory, vol. 48, no. 6, pp. 1590–1615, 2002

    2002

  36. [36]

    A mapping approach to rate-distortion computation and analysis,

    K. Rose, “A mapping approach to rate-distortion computation and analysis,”IEEE Transactions on Information Theory, vol. 40, no. 6, pp. 1939–1952, 1994

    1939

  37. [37]

    Rate-distortion theory by and for energy-based models,

    Q. Li and C. Guyot, “Rate-distortion theory by and for energy-based models,”IEEE Transactions on Communications, 2024

    2024

  38. [38]

    Estimating rate-distortion functions using the energy-based model,

    S. Wu, S. Xu, L. Chen, H. Wu, and W. Zhang, “Estimating rate-distortion functions using the energy-based model,” in2025 IEEE Information Theory Workshop (ITW), 2025

    2025

  39. [39]

    Correlation functions and computer simulations,

    G. Parisi, “Correlation functions and computer simulations,”Nuclear Physics B, vol. 180, no. 3, pp. 378–384, 1981

    1981

  40. [40]

    A. B. Owen,Monte Carlo Theory, Methods and Examples. Stanford, 2013

    2013

  41. [41]

    Caveats for information bottleneck in deterministic scenarios,

    A. Kolchinsky, B. D. Tracey, and S. Van Kuyk, “Caveats for information bottleneck in deterministic scenarios,” inProc. 7th International Con- ference on Learning Representations (ICLR), New Orleans, Louisiana, USA, May 2019, pp. 1–23. 12

    2019

  42. [42]

    On variational bounds of mutual information,

    B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, “On variational bounds of mutual information,” inInternational Conference on Machine Learning. PMLR, 2019, pp. 5171–5180

    2019

  43. [43]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Information Processing systems, vol. 33, pp. 6840– 6851, 2020

    2020

  44. [44]

    Score-based generative modeling through stochastic differen- tial equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differen- tial equations,” inInternational Conference on Learning Representations, 2021

    2021

  45. [45]

    Georgii,Gibbs Measures and Phase Transitions

    H.-O. Georgii,Gibbs Measures and Phase Transitions. Walter de Gruyter, 2011, vol. 9

    2011

This paper was first reviewed by grok-4.5 on July 12, 2026.