Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

Alberto Foresti; Alexander Tolmachev; Alexey Frolov; Giulio Franzese; Ivan Butakov; Pietro Michiardi

REVIEW 2 major objections 4 minor 111 references

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

T0 review · 2 major / 4 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No mutual-information estimator wins everywhere: each class beats the others on specific, designed tests.

desk verdict Solid, usable MI benchmark suite that unifies prior tests under a DPI/copula view and shows clear regime-specific winners; the comparative claim holds on the synthetic evidence alone. read the letter →

arxiv 2607.03487 v1 pith:D7IVLZIB submitted 2026-07-03 cs.LG cs.AIcs.ITmath.IT

Alberto Foresti , Ivan Butakov , Alexander Tolmachev , Giulio Franzese , Alexey Frolov , Pietro Michiardi This is my paper

classification cs.LGcs.AIcs.ITmath.IT

keywords mutualinformationestimationbenchmarkscopulasdataprocessinginequalitynon-parametricestimatorsdiscriminativegenerativediffusionmodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mutual information measures statistical dependence, but existing tests of estimators use mostly simple, low-dimensional distributions, so claims of superiority are hard to trust on realistic data. This paper unifies prior benchmarks under a copula view that separates dependency structure from marginal complexity, then builds two complementary test families: copula-first suites that systematically vary ground-truth MI, dimension and marginal difficulty (including flow-based maps), and marginals-first suites that attach controlled dependencies to real images. Across non-parametric, discriminative and generative estimators the evaluation shows no universal winner: each family systematically outperforms the others under particular setups. The authors isolate concrete barriers (sample complexity that grows with MI itself, numerical instability, high intrinsic variance of pointwise MI, and diffusion smoothing of concentrated distributions) and supply new stress tests that expose them.

What carries the argument

A unified copula-theoretic framing of all existing MI benchmarks as special cases of the data-processing inequality, which decomposes estimation difficulty into complex copulas versus complex marginals and thereby organizes the new copula-first and marginals-first test families.

What would settle it

Construct a high-MI real-image pair under the paper’s coupling whose supports overlap enough that the true mutual information is strictly less than the claimed latent value, then re-run the estimator ranking; if the same-class ranking flips, the marginals-first ground-truth claim fails.

Watch

Extended reading notes

Core claim

There is no universal winner among mutual-information estimators. For each broad class—non-parametric (k-NN), discriminative (classifier-based bounds), and generative (including diffusion)—the paper identifies concrete regimes in which that class systematically beats all competitors, overturning the informal assumption that more expensive neural methods are invariably better.

Load-bearing premise

The real-image tests assume that the latent-to-image maps keep supports disjoint (or that class labels are perfectly recoverable), so ground-truth mutual information exactly equals the controllable latent mutual information.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful takeaway is simple: there is no universal MI estimator. k-NN wins in low-d/low-MI, generative methods (MINDE-C especially) handle high MI better, and discriminative methods only look competitive after compression. The paper makes that claim with a clean taxonomy and open code rather than another ad-hoc leaderboard.

What is new is the copula-first / marginals-first split that puts existing synthetic, flow-based, and same-class tests under one DPI roof, plus a few genuinely harder distributions (rare-event channel, CNF-warped moons/circles/S-curve, continuous brightness coupling on MNIST). Appendix A derives or cites the ground-truth formulas carefully; the experiments cover continuous, discrete, flow, and image regimes with seeds and ablations on sample size, width, and k. Failures (timeouts, numerical instability of MINDE-J) are reported instead of hidden. That is real engineering work and the GitHub release makes it usable.

The soft spots are ordinary for this genre. High-dimensional k-NN runs are incomplete for cost reasons; the real-image ground truth rests on the usual disjoint-support / recoverable-class assumption. Neither sinks the headline result: the synthetic and CNF suites already reverse the rankings without that assumption. The four “fundamental barriers” (sample complexity exponential in MI, numerical instability of the Gaussian formula, high PMI variance, diffusion smoothing) are mostly known phenomena restated with clean examples; they are useful for the discussion but not revolutionary. Free parameters (MI grid, k, network width, diffusion schedule) are standard and do not look gamed.

This is for people who actually pick or design MI estimators, or who need a stress suite before claiming a new method is better. Citation pattern is appropriate; math and data look solid. I would send it to referees.

Referee Report

2 major / 4 minor

Summary. The manuscript introduces a unified copula-theoretic framework for benchmarking mutual information estimators, subsuming prior synthetic and same-class-pairing tests as special cases of the data-processing inequality. It defines two complementary families: a copula-first suite that varies ground-truth MI, dimension and marginal complexity via analytic distributions and continuous normalizing flows, and a marginals-first suite that couples real image data (MNIST, CIFAR embeddings) to controlled latent dependencies. Three estimator classes—non-parametric (KSG, WKL), discriminative/variational (MINE-DV, MINE-NWJ, InfoNCE) and generative/diffusion (MINDE-C/J)—are evaluated across these regimes. The central empirical claim is that no class is universally superior; each systematically outperforms the others under identifiable conditions. The authors further isolate fundamental barriers (exponential sample complexity in MI, numerical instability of MI-as-function-of-correlation, high PMI variance, and diffusion-time smoothing) and release open-source code.

Significance. If the reported regime-specific reversals hold, the work supplies the community with a principled, reproducible test suite that replaces the current patchwork of low-dimensional Gaussians and ad-hoc image pairings. The open-source repository, analytic ground-truth derivations (Appendix A lemmas), multi-seed averages, and ablations on sample size, network width and k-NN parameter constitute concrete engineering contributions. The identification of barriers that are intrinsic to the estimation problem rather than estimator-specific is especially useful for future method design. The paper therefore advances both evaluation practice and conceptual understanding of MI estimation difficulty.

major comments (2)

[Section 4 / Tables 2–3] Section 4 and Tables 2–3: the experimental generative class is restricted to the two MINDE variants, while the background (and Table 2) develops non-diffusion generative estimators (MIENF, VCE, DINE-Gaussian) that the authors themselves present as a distinct inductive-bias trade-off. Because the headline claim is that “each category can systematically outperform,” the generative category is under-sampled; including at least one flow-based generative baseline on the copula-first suite would make the class-level conclusion more robust.
[Section 3.2 / Tables 7–8] Section 3.2 and the constructions underlying Tables 7–8: ground-truth MI for the marginals-first family equals latent MI only under the assumption of pairwise disjoint supports (or perfect recoverability of class/brightness). For raw MNIST brightness regression this is not automatically guaranteed; a short empirical check (e.g., estimated MI between recovered and true latent, or classification accuracy of a simple decoder) would confirm that DPI equality is approximately attained and that reported errors are not inflated by an incorrect ground truth.

minor comments (4)

[Section 4] Terminology oscillates between “discriminative,” “variational” and “classifier-based” for the same family (MINE/NWJ/InfoNCE). A single consistent label in Section 4 and the figure legends would improve readability.
[Figures 3–6] Figures 3–6 plot absolute estimated MI against ground truth; adding a small inset or companion panel of absolute error (as already tabulated) would make saturation and bias patterns easier to compare across families at a glance.
[Table 5] Table 5 caption states “dX = dY = 1” but the surrounding text discusses higher-dimensional stacking; clarify whether the ablation is strictly one-dimensional or averaged over the dimensions used elsewhere.
Minor typographical inconsistencies appear (e.g., “mi” vs “MI”, “pmi” vs “PMI”, occasional missing spaces before citations). A final pass would polish presentation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ground-truth MI is constructed independently via closed-form formulas and DPI-preserving maps; estimator rankings are pure empirical comparisons against that GT.

full rationale

The paper is a benchmarking study, not a derivation of a physical or statistical law from first principles. All load-bearing ground-truth MI values are obtained either from classical closed-form expressions (multivariate Gaussian, Student-t correction, entropy of discrete uniforms, etc.) or by applying the data-processing inequality to invertible (or left-invertible) maps and Markov kernels with disjoint supports; these constructions appear in Section 3, Table 4 and Appendix A and do not involve any of the estimators under test. The experimental claim that “there is no universal winner” is simply the observed ranking of non-parametric, discriminative and generative estimators on those independently constructed suites (Figures 3–6, Tables 5–11). Self-citations (to MINDE, MIENF, prior synthetic image maps) are used only to identify the methods being re-evaluated; none of them supplies a uniqueness theorem or fitted parameter that is later re-presented as a prediction. Consequently the derivation chain contains no self-definitional step, no fitted-input-called-prediction, and no load-bearing self-citation circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The work rests on standard information-theoretic identities (DPI, MI = KL of joint vs product, copula entropy) plus the constructive assumption that the chosen Markov kernels preserve MI exactly; free parameters are ordinary estimator hyper-parameters and the controllable ground-truth MI values used to generate the test suite; the only invented objects are the concrete new test distributions and the two-family taxonomy itself.

free parameters (4)

ground-truth MI grid
Eleven linearly spaced target MI values in [0,10] used to generate every synthetic and flow-based dataset; chosen by the authors to span low-to-high regimes.
k for KSG/WKL
Nearest-neighbor order; ablated but still a free hyper-parameter of the non-parametric baselines.
network width / batch size / learning rate for neural estimators
Standard training hyper-parameters of MINE, InfoNCE and MINDE; reported but not derived from first principles.
diffusion epsilon / beta schedule for MINDE
Lower integration limit and noise schedule that effectively replace true MI by a smoothed version; chosen for numerical stability.

assumptions (3)

standard math Data Processing Inequality and invariance of MI under invertible measurable maps (Theorems 2.1–2.2)
Used throughout to guarantee that the constructed push-forwards and Markov kernels preserve the latent MI.
standard math Existence of a copula representation for absolutely continuous joint distributions (McCann, Fan & Henry)
Underpins the claim that every existing benchmark is a special case of a copula plus marginal transforms.
domain assumption Disjoint supports (or perfect class recoverability) of the image-generating kernels in the marginals-first family
Required for DPI equality so that observed MI equals latent MI; stated in Section 3.2 but not independently verified on real image manifolds.

invented entities (2)

copula-first / marginals-first benchmark taxonomy independent evidence
purpose: Organizes all prior and new tests under a single DPI view and enables systematic variation of dependence versus marginal complexity.
The two-family split and the concrete new members (rare-event channel, CNF-warped moons/circles/S-curve, brightness regression) are introduced by the paper.
Rare Event Channel distribution independent evidence
purpose: Stress-tests high intrinsic variance of pointwise MI while keeping ground-truth MI tractable.
Defined and analyzed in Section 3.1 / Appendix A; not present in prior MI benchmark suites.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation." pith.science (2026). https://pith.science/paper/D7IVLZIB

@misc{pith2026260703487,
  author       = {Pith},
  title        = {Pith review of: Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D7IVLZIB}},
  note         = {Machine review of arXiv:2607.03487}
}

read the original abstract

Mutual information (MI) estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored. We address this gap with a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective that subsumes existing benchmarks as special cases. Within this framework, we propose two complementary families of tests: a copula-first family that systematically varies ground-truth MI, dimensionality, and marginal complexity using synthetic and flow-based transformations; and a marginals-first family that couples real-world image data with controlled dependency structures, extending the classic same-class-pairing paradigm. We use this suite to extensively evaluate three classes of estimators: non-parametric, discriminative, and generative. Contrary to prevailing assumptions, our results indicate that there is no universal winner: each category can systematically outperform all other estimators under specific setups. By analyzing these cases, we identify fundamental estimation barriers and propose new tests that more effectively stress these specific limitations. We share the open source code at https://github.com/VanessB/mutinfo.

Figures

Figures reproduced from arXiv: 2607.03487 by the authors.

**Figure 2.** Examples of synthetic images from (Butakov et al., 2024b). Note that images are high-dimensional, [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** MI Estimation in Continuous Synthetic Experiment (All estimators) [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Comparison of estimator families on continuous synthetic distributions. To clearly contrast k-NN, [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: MI Estimation for discrete experiments. (a) Individual performance of all estimators. (b) Comparison of estimator families on discrete synthetic distributions. To clearly contrast k-NN, variational, and generative approaches, we aggregate the results by estimator class…

**Figure 6.** Figure 6: Normalizing Flow Experiments. (a) Individual performance of all evaluated mi estimators. (b) Aggregated performance by estimator family. To clearly contrast the approaches, each line plots the single most accurate estimate within its respective class for every given di…

**Figure 7.** Figure 7: mi vs Diffusion Time In practice, however, this intrinsic variance may manifest itself as increased bias of the estimate, as estimators are typically unable to learn the highly variable parts of the distribution. We indeed observe such behavior in our experiments with …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

111 extracted references · 10 canonical work pages

[1]

Generalized energy based models

Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0PtUPB9z6qK

2021
[2]

Arellano-Valle, Javier Contreras-Reyes, and Marc Genton

R. Arellano-Valle, Javier Contreras-Reyes, and Marc Genton. Shannon entropy and mutual information for multivariate skew-elliptical distributions. Scandinavian Journal of Statistics, 40: 0 42--62, 03 2013. doi:10.1111/j.1467-9469.2011

doi:10.1111/j.1467-9469.2011 2013
[3]

Mutual information neural estimation

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 531--540. PMLR, 07 2018....

2018
[4]

Berrett, Richard J

Thomas B. Berrett, Richard J. Samworth, and Ming Yuan. Efficient multivariate entropy estimation via k -nearest neighbour distances. Ann. Statist., 47 0 (1): 0 288--318, 02 2019. doi:10.1214/18-AOS1688. URL https://doi.org/10.1214/18-AOS1688

work page doi:10.1214/18-aos1688 2019
[5]

Mutual information estimation via normalizing flows

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov. Mutual information estimation via normalizing flows. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https://openreview.net/forum?id=JiQXsLvDls

2024
[6]

Information bottleneck analysis of deep neural networks via lossy compression

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev. Information bottleneck analysis of deep neural networks via lossy compression. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=huGECz8dPp

2024
[8]

Neural ordinary differential equations

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

2018
[9]

Infogan: Interpretable representation learning by information maximizing generative adversarial nets

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http...

2016

Show all 111 references

[10]

Neural mutual information estimation with vector copulas

Yanzhi Chen, Zijing Ou, Adrian Weller, and Michael Gutmann. Neural mutual information estimation with vector copulas. In The Thirty-ningth Annual Conference on Neural Information Processing Systems, 2025

2025
[11]

Density ratio estimation via infinitesimal classification

Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics...

2022
[12]

Cover and Joy A

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006

2006
[13]

Beyond normal: On the evaluation of mutual information estimators

Pawe Czy \.z , Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx. Beyond normal: On the evaluation of mutual information estimators. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=25vRtG56YH

2023
[14]

On the properties and estimation of pointwise mutual information profiles

Pawe Czy \.z , Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx. On the properties and estimation of pointwise mutual information profiles. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=LdflD41Gn8

2025
[15]

Flow-based variational mutual information: Fast and flexible approximations

Caleb Dahlke and Jason Pacheco. Flow-based variational mutual information: Fast and flexible approximations. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=spDUv05cEq

2025
[17]

Normalizing flows for conditional independence testing

Bao Duong and Thin Nguyen. Normalizing flows for conditional independence testing. Knowledge and Information Systems, 66, 08 2023 b . doi:10.1007/s10115-023-01964-w

2023 doi
[18]

Vector copulas, 2021

Yanqin Fan and Marc Henry. Vector copulas, 2021. URL https://arxiv.org/abs/2009.06558

2021 arXiv
[19]

MINDE : Mutual information neural diffusion estimation

Giulio Franzese, Mustapha BOUNOUA, and Pietro Michiardi. MINDE : Mutual information neural diffusion estimation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=0kWd8SJq8d

2024
[20]

Latent abstractions in generative diffusion models

Giulio Franzese, Mattia Martini, Giulio Corallo, Paolo Papotti, and Pietro Michiardi. Latent abstractions in generative diffusion models. Entropy, 27 0 (4), 2025. ISSN 1099-4300. doi:10.3390/e27040371. URL https://www.mdpi.com/1099-4300/27/4/371

2025 doi
[21]

Estimating mutual information for discrete-continuous mixtures

Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual information for discrete-continuous mixtures. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing System...

2017
[22]

Copula modeling for discrete random vectors

Gery Geenens. Copula modeling for discrete random vectors. Dependence Modeling, 8 0 (1): 0 417--440, 2020. doi:doi:10.1515/demo-2020-0022. URL https://doi.org/10.1515/demo-2020-0022

2020 doi
[23]

Goldfeld , K

Z. Goldfeld , K. Greenewald , J. Niles-Weed , and Y. Polyanskiy . Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, 66 0 (7): 0 4368--4391, 2020. doi:10.1109/TIT.2020.2975480

2020 doi
[25]

Klein, and Peng Yin

Gokul Gowri, Xiao-Kang Lun, Allon M. Klein, and Peng Yin. Approximating mutual information of high-dimensional variables using learned representations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information P...

2024
[26]

Learning deep representations by mutual information estimation and maximization

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019. URL https://ope...

2019
[27]

Time series feature selection method based on mutual information

Lin Huang, Xingqiang Zhou, Lianhui Shi, and Li Gong. Time series feature selection method based on mutual information. Applied Sciences, 14 0 (5), 2024. ISSN 2076-3417. doi:10.3390/app14051960. URL https://www.mdpi.com/2076-3417/14/5/1960

2024 doi
[28]

Infobridge: Mutual information estimation via bridge matching, 2025

Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Gushchin, and Alexander Korotin. Infobridge: Mutual information estimation via bridge matching, 2025. URL https://arxiv.org/abs/2502.01383

2025
[29]

Auto-encoding variational bayes, 2013

Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL https://arxiv.org/abs/1312.6114

2013 arXiv
[30]

L. F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Problems Inform. Transmission, 23: 0 95--101, 1987

1987
[33]

Mutual information estimation via \ f\ -divergence and data derangements

Nunzio Alexandro Letizia, Nicola Novello, and Andrea M Tonello. Mutual information estimation via \ f\ -divergence and data derangements. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=PThi9hf9UT

2024
[36]

Formal limitations on the measurement of mutual information

David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machi...

2020
[37]

Existence and uniqueness of monotone measure-preserving maps

Robert McCann. Existence and uniqueness of monotone measure-preserving maps. Duke Mathematical Journal, 80, 11 1995. doi:10.1215/S0012-7094-95-08013-2

1995 doi
[40]

Estimating divergence functionals and the likelihood ratio by convex risk minimization

XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56 0 (11): 0 5847--5861, 2010

2010
[41]

A neural difference-of-entropies estimator for mutual information, 2025

Haoran Ni and Martin Lotz. A neural difference-of-entropies estimator for mutual information, 2025. URL https://arxiv.org/abs/2502.13085

2025
[42]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in P ython. Journal of Machine Learnin...

2011
[43]

Polyanskiy and Y

Y. Polyanskiy and Y. Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2024. ISBN 9781108832908. URL https://books.google.ru/books?id=CySo0AEACAAJ

2024
[44]

On variational bounds of mutual information

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma...

2019
[45]

Benjamin Rhodes, Kai Xu, and Michael U. Gutmann. Telescoping density-ratio estimation. In The Thirty-fourth Annual Conference on Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2006.12204

2020 arXiv
[46]

U-net: Convolutional networks for biomedical image segmentation, 2015

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. URL https://arxiv.org/abs/1505.04597

2015 arXiv
[47]

Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information

Jakob Runge. Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 ...

2018
[48]

Understanding the limitations of variational mutual information estimators

Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=B1x62TNtDS

2020
[50]

Feature selection based on mutual information

Muhammad Aliyu Sulaiman and Jane Labadin. Feature selection based on mutual information. In 2015 9th International Conference on IT in Asia (CITA), pp.\ 1--6, 2015. doi:10.1109/CITA.2015.7349827

2015 doi
[52]

Rubenstein, Sylvain Gelly, and Mario Lucic

Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkxoh24FPH

2020
[53]

Representation learning with contrastive predictive coding, 2019

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. URL https://arxiv.org/abs/1807.03748

2019 arXiv
[54]

Diffusers: State-of-the-art diffusion models

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022

2022
[55]

Information theoretic text-to-image alignment

Chao Wang, Giulio Franzese, Alessandro Finamore, Massimo Gallo, and Pietro Michiardi. Information theoretic text-to-image alignment. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=Ugs2W5XFFo

2025
[56]

RFMI : Estimating mutual information on rectified flow for text-to-image alignment

Chao Wang, Giulio Franzese, Alessandro Finamore, and Pietro Michiardi. RFMI : Estimating mutual information on rectified flow for text-to-image alignment. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025 b . URL https://o...

2025
[57]

Problems Inform

Sample Estimate of the Entropy of a Random Vector , author =. Problems Inform. Transmission , pages =. 1987 , volume =

1987
[58]

Moddemeijer , keywords =

R. Moddemeijer , keywords =. On estimation of entropy and mutual information of continuous distributions , journal =. 1989 , issn =. doi:https://doi.org/10.1016/0165-1684(89)90132-1 , url =

1989 doi
[59]

Existence and uniqueness of monotone measure-preserving maps , volume =

McCann, Robert , year =. Existence and uniqueness of monotone measure-preserving maps , volume =. Duke Mathematical Journal , doi =
[60]

Estimation of mutual information using kernel density estimators , author =. Phys. Rev. E , volume =. 1995 , month =. doi:10.1103/PhysRevE.52.2318 , url =

1995 doi
[61]

Estimating mutual information , author =. Phys. Rev. E , volume =. 2004 , month =. doi:10.1103/PhysRevE.69.066138 , url =

2004 doi
[62]

and Thomas, Joy A

Cover, Thomas M. and Thomas, Joy A. , title =. 2006 , publisher =

2006
[63]

IEEE Transactions on Information Theory , volume=

Estimating divergence functionals and the likelihood ratio by convex risk minimization , author=. IEEE Transactions on Information Theory , volume=. 2010 , publisher=

2010
[64]

Mutual Information Is Copula Entropy , journal =

Jian Ma and Zengqi Sun , keywords =. Mutual Information Is Copula Entropy , journal =. 2011 , issn =. doi:https://doi.org/10.1016/S1007-0214(11)70008-6 , url =

2011 doi
[65]

and Contreras-Reyes, Javier and Genton, Marc , year =

Arellano-Valle, R. and Contreras-Reyes, Javier and Genton, Marc , year =. Shannon Entropy and Mutual Information for Multivariate Skew-Elliptical Distributions , volume =. Scandinavian Journal of Statistics , doi =
[66]

2013 , eprint=

Auto-Encoding Variational Bayes , author=. 2013 , eprint=

2013
[67]

Tabak, E. G. and Turner, Cristina V. , title =. Communications on Pure and Applied Mathematics , volume =. doi:https://doi.org/10.1002/cpa.21423 , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpa.21423 , abstract =

doi
[68]

Feature selection based on mutual information , year=

Sulaiman, Muhammad Aliyu and Labadin, Jane , booktitle=. Feature selection based on mutual information , year=
[69]

Tony Cai and Tengyuan Liang and Harrison H

T. Tony Cai and Tengyuan Liang and Harrison H. Zhou , keywords =. Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional Gaussian distributions , journal =. 2015 , issn =. doi:https://doi.org/10.1016/j.jmva.2015.0...

2015 doi
[70]

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets , url =

Chen, Xi and Duan, Yan and Houthooft, Rein and Schulman, John and Sutskever, Ilya and Abbeel, Pieter , booktitle =. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets , url =
[71]

Nonparametric independence testing via mutual information , volume =

Berrett, Thomas and Samworth, Richard , year =. Nonparametric independence testing via mutual information , volume =. Biometrika , doi =
[72]

Estimating Mutual Information for Discrete-Continuous Mixtures , url =

Gao, Weihao and Kannan, Sreeram and Oh, Sewoong and Viswanath, Pramod , booktitle =. Estimating Mutual Information for Discrete-Continuous Mixtures , url =
[73]

ITM web of conferences , volume=

Kernel density estimation and its application , author=. ITM web of conferences , volume=. 2018 , organization=

2018
[74]

Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , pages =

Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information , author =. Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , pages =. 2018 , editor =

2018
[75]

arXiv preprint arXiv:1810.05728 , year=

Estimating information flow in deep neural networks , author=. arXiv preprint arXiv:1810.05728 , year=

arXiv
[76]

International Conference on Learning Representations , year=

Learning deep representations by mutual information estimation and maximization , author=. International Conference on Learning Representations , year=
[77]

Proceedings of the 35th International Conference on Machine Learning , pages =

Mutual Information Neural Estimation , author =. Proceedings of the 35th International Conference on Machine Learning , pages =. 2018 , editor =

2018
[78]

2019 , eprint=

Representation Learning with Contrastive Predictive Coding , author=. 2019 , eprint=

2019
[79]

and Samworth, Richard J

Berrett, Thomas B. and Samworth, Richard J. and Yuan, Ming , doi =. Efficient multivariate entropy estimation via k -nearest neighbour distances , url =. Ann. Statist. , month =
[80]

Proceedings of the 36th International Conference on Machine Learning , pages =

On Variational Bounds of Mutual Information , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

2019
[81]

International Conference on Learning Representations , year=

On Mutual Information Maximization for Representation Learning , author=. International Conference on Learning Representations , year=
[82]

Dependence Modeling , doi =

Copula modeling for discrete random vectors , author =. Dependence Modeling , doi =. 2020 , lastchecked =

2020
[83]

IEEE Transactions on Information Theory , title=

Z. IEEE Transactions on Information Theory , title=. 2020 , volume=

2020
[84]

Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages =

Formal Limitations on the Measurement of Mutual Information , author =. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages =. 2020 , editor =

2020
[85]

International Conference on Learning Representations , year=

Understanding the Limitations of Variational Mutual Information Estimators , author=. International Conference on Learning Representations , year=
[86]

The Thirty-fourth Annual Conference on Neural Information Processing Systems , url=

Telescoping Density-Ratio Estimation , author=. The Thirty-fourth Annual Conference on Neural Information Processing Systems , url=. 2020 , eprint=

2020
[87]

International Conference on Learning Representations , year=

Generalized Energy Based Models , author=. International Conference on Learning Representations , year=
[88]

2021 , eprint=

Vector copulas , author=. 2021 , eprint=

2021
[89]

Proceedings of the AAAI Conference on Artificial Intelligence , author=

Entropy Estimation via Normalizing Flow , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2022 , month=. doi:10.1609/aaai.v36i9.21237 , abstractNote=

2022 doi
[90]

Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , pages =

Density Ratio Estimation via Infinitesimal Classification , author =. Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , pages =. 2022 , editor =

2022
[91]

Normalizing flows for conditional independence testing , volume =

Duong, Bao and Nguyen, Thin , year =. Normalizing flows for conditional independence testing , volume =. Knowledge and Information Systems , doi =
[92]

Thirty-seventh Conference on Neural Information Processing Systems , year=

Beyond Normal: On the Evaluation of Mutual Information Estimators , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=
[93]

Proceedings of the AAAI Conference on Artificial Intelligence , author=

Diffeomorphic Information Neural Estimation , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2023 , month=. doi:10.1609/aaai.v37i6.25908 , abstractNote=

2023 doi
[94]

2023 , eprint=

On the Effectiveness of Hybrid Mutual Information Estimation , author=. 2023 , eprint=

2023
[95]

Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Diffeomorphic information neural estimation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
[96]

arXiv preprint arXiv:2410.10924 , year=

A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets , author=. arXiv preprint arXiv:2410.10924 , year=

arXiv
[97]

2024 , url=

Giulio Franzese and Mustapha BOUNOUA and Pietro Michiardi , booktitle=. 2024 , url=

2024
[98]

The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

Mutual Information Estimation via \ f\ -Divergence and Data Derangements , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=
[99]

The Twelfth International Conference on Learning Representations , year=

Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression , author=. The Twelfth International Conference on Learning Representations , year=
[100]

The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

Mutual Information Estimation via Normalizing Flows , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=
[101]

S I: Score-based O-

Bounoua, Mustapha and Franzese, Giulio and Michiardi, Pietro , booktitle =. S I: Score-based O-. 2024 , editor =

2024
[102]

2024 , publisher=

Information Theory: From Coding to Learning , author=. 2024 , publisher=

2024
[103]

Applied Sciences , VOLUME =

Huang, Lin and Zhou, Xingqiang and Shi, Lianhui and Gong, Li , TITLE =. Applied Sciences , VOLUME =. 2024 , NUMBER =

2024
[104]

and Yin, Peng , booktitle =

Gowri, Gokul and Lun, Xiao-Kang and Klein, Allon M. and Yin, Peng , booktitle =. Approximating mutual information of high-dimensional variables using learned representations , url =
[105]

2025 , eprint=

A Neural Difference-of-Entropies Estimator for Mutual Information , author=. 2025 , eprint=

2025
[106]

Transactions on Machine Learning Research , issn=

On the Properties and Estimation of Pointwise Mutual Information Profiles , author=. Transactions on Machine Learning Research , issn=. 2025 , url=

2025
[107]

The Thirteenth International Conference on Learning Representations , year=

Information Theoretic Text-to-Image Alignment , author=. The Thirteenth International Conference on Learning Representations , year=
[108]

2025 , url=

Chao Wang and Giulio Franzese and Alessandro Finamore and Pietro Michiardi , booktitle=. 2025 , url=

2025
[109]

The Thirteenth International Conference on Learning Representations , year=

Flow-based Variational Mutual Information: Fast and Flexible Approximations , author=. The Thirteenth International Conference on Learning Representations , year=
[110]

2025 , eprint=

InfoBridge: Mutual Information estimation via Bridge Matching , author=. 2025 , eprint=

2025
[111]

Entropy , VOLUME =

Franzese, Giulio and Martini, Mattia and Corallo, Giulio and Papotti, Paolo and Michiardi, Pietro , TITLE =. Entropy , VOLUME =. 2025 , NUMBER =

2025
[112]

Scaling Learning Algorithms Towards

Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards
[113]

and Osindero, Simon and Teh, Yee Whye , journal =

Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =
[114]

2016 , publisher=

Deep learning , author=. 2016 , publisher=

2016
[115]

The Thirty-ningth Annual Conference on Neural Information Processing Systems , year=

Neural Mutual Information Estimation with Vector Copulas , author=. The Thirty-ningth Annual Conference on Neural Information Processing Systems , year=
[116]

Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , pages =

Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? , author =. Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , pages =. 2023 , editor =

2023
[117]

2015 , eprint=

U-Net: Convolutional Networks for Biomedical Image Segmentation , author=. 2015 , eprint=

2015
[118]

GitHub repository , howpublished =

Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Dhruv Nair and Sayak Paul and William Berman and Yiyi Xu and Steven Liu and Thomas Wolf , title =. GitHub repository , howpublished =. 2022 , publisher =

2022
[119]

arXiv preprint arXiv:2210.02747 , year=

Flow matching for generative modeling , author=. arXiv preprint arXiv:2210.02747 , year=

arXiv
[120]

Advances in neural information processing systems , volume=

Neural ordinary differential equations , author=. Advances in neural information processing systems , volume=
[121]

and Varoquaux, G

Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E. , journal=. S...
[122]

arXiv preprint arXiv:2011.13456 , year=

Score-based generative modeling through stochastic differential equations , author=. arXiv preprint arXiv:2011.13456 , year=

2011 arXiv

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

Generalized energy based models

Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0PtUPB9z6qK

2021

[2] [2]

Arellano-Valle, Javier Contreras-Reyes, and Marc Genton

R. Arellano-Valle, Javier Contreras-Reyes, and Marc Genton. Shannon entropy and mutual information for multivariate skew-elliptical distributions. Scandinavian Journal of Statistics, 40: 0 42--62, 03 2013. doi:10.1111/j.1467-9469.2011

doi:10.1111/j.1467-9469.2011 2013

[3] [3]

Mutual information neural estimation

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 531--540. PMLR, 07 2018....

2018

[4] [4]

Berrett, Richard J

Thomas B. Berrett, Richard J. Samworth, and Ming Yuan. Efficient multivariate entropy estimation via k -nearest neighbour distances. Ann. Statist., 47 0 (1): 0 288--318, 02 2019. doi:10.1214/18-AOS1688. URL https://doi.org/10.1214/18-AOS1688

work page doi:10.1214/18-aos1688 2019

[5] [5]

Mutual information estimation via normalizing flows

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov. Mutual information estimation via normalizing flows. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https://openreview.net/forum?id=JiQXsLvDls

2024

[6] [6]

Information bottleneck analysis of deep neural networks via lossy compression

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev. Information bottleneck analysis of deep neural networks via lossy compression. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=huGECz8dPp

2024

[7] [8]

Neural ordinary differential equations

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

2018

[8] [9]

Infogan: Interpretable representation learning by information maximizing generative adversarial nets

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http...

2016