Pith. sign in

REVIEW 3 major objections 5 minor 29 references

NMINE, a fully neural estimator combining three Donsker–Varadhan critics with entropy recovery from uniform reference distributions, estimates normalized mutual information more accurately than a KSG-based baseline on multidimensional Gauss

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:51 UTC pith:N3TDV7Y4

load-bearing objection Neural NMI via MINE+NEE is worth a look, but the entropy-recovery step is not defined for unbounded distributions, so the Gaussian benchmark may not measure NMI. the 3 major comments →

arxiv 2607.27710 v1 pith:N3TDV7Y4 submitted 2026-07-30 cs.LG

NMINE: Normalized Mutual Information Neural Estimation

Petra Eerikinharju , Marko Tuononen , Ville Hautam\"aki This is my paper
classification cs.LG MSC 62B1094A17
keywords normalized mutual informationneural estimationDonsker–Varadhan representationentropy estimationuniform reference distributionKSG estimatorcontinuous variablesasymmetric normalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Normalized mutual information is hard to estimate for continuous multidimensional variables because the raw quantity is unbounded and depends on the units of the variables. The paper proposes NMINE, a fully neural estimator that obtains the asymmetric normalized mutual information I(X;Y)/H(Y) by training three Donsker–Varadhan critics: one for the joint distribution and one for each marginal, each measuring divergence to a uniform reference distribution on the observed sample support. Entropies are recovered as log V − D, where V is the volume of the reference box, and mutual information as D_XY − D_X − D_Y. On Gaussian data from one to eight dimensions, NMINE reports lower mean absolute error than a KSG-based baseline, with relative reductions of 30.7–74.0%, and preserves the monotonic relationship between correlation and NMI even where it underestimates. A sympathetic reader would care because this is a concrete demonstration that neural estimation can replace nearest-neighbor statistics for normalized dependency measurement.

Core claim

The paper claims that a fully neural pipeline can estimate normalized mutual information for continuous variables more accurately than a nearest-neighbor-based baseline. The estimator, NMINE, uses the Donsker–Varadhan variational representation to train three separate neural critics: one for the joint distribution of (X, Y) and one for each marginal, each measuring the Kullback–Leibler divergence to a uniform reference distribution defined over the observed sample support. Differential entropy is recovered from each divergence via H = log V − D, where V is the product of the observed per-dimension ranges, and mutual information is recovered as D_XY − D_X − D_Y, following the relative-entropy

What carries the argument

The central machinery is a set of three Donsker–Varadhan variational critics, each trained to estimate the Kullback–Leibler divergence between an empirical distribution and a uniform reference distribution supported on the observed sample box. The reference density is 1/V, with V the product of the observed per-dimension ranges, which allows entropy recovery in closed form as H = log V − D. Mutual information is then obtained from the divergence decomposition I(X;Y) = D_KL(p_XY || m_X m_Y) − D_KL(p_X || m_X) − D_KL(p_Y || m_Y), and normalized mutual information as the ratio I(X;Y)/H(Y). This construction replaces nearest-neighbor statistics with neural variational estimation throughout the p

Load-bearing premise

The estimate assumes that a uniform reference distribution defined on the box spanned by the observed samples is a valid baseline for measuring entropy and divergence, even though real distributions like Gaussians have tails outside that box.

What would settle it

For a Gaussian with unbounded support, compute the true Kullback–Leibler divergence between the Gaussian and the uniform distribution on the observed range: it is infinite. A neural critic trained against box samples returns a finite number; if that finite value is an artifact of the box, then the recovered entropy and NMI should shift systematically as the sample size—and hence the observed range—changes. This can be tested directly by measuring NMINE's output across increasing sample sizes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is correct, normalized mutual information for continuous variables can be estimated with a fully neural pipeline that outperforms k-nearest-neighbor baselines in low-to-moderate dimensions.
  • The asymmetric normalization used here preserves the ranking induced by mutual information, so NMINE can be used to compare dependency strength across different variable pairs without unit-calibration artifacts.
  • The reported monotonic response to correlation, even when underestimating, suggests that NMINE can serve as a reliable dependency signal for ranking or trend detection in scientific applications.
  • The framework extends naturally to non-Gaussian settings; preliminary Student-t experiments suggest the estimator remains stable for heavy-tailed distributions, though analytical ground truth is not available there.
  • Because all components are neural and differentiable, the estimator can be embedded in gradient-based optimization pipelines, enabling end-to-end learning objectives that involve normalized mutual information.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The box-support uniform reference is a finite-sample device: for distributions with unbounded support, the true KL divergence to a compact uniform measure is infinite, so the recovered entropy and NMI should depend on sample size and extreme observations; this can be tested by sweeping sample count and observing drift.
  • The decomposition I = D_XY − D_X − D_Y means the final MI estimate is not itself a variational lower bound; its bias and variance are composition effects of three independently trained critics, so error analysis could be decomposed by critic quality.
  • The observed underestimation bias at high dimensions may be tied to the known difficulty of variational MI estimation when true MI is large relative to sample size; this suggests that variance-reduction techniques such as SMILE or critic clipping could narrow the gap.
  • Because the reference volume is computed from observed ranges, NMINE is tied to the empirical support; for heavy-tailed or outlier-prone data, robust support estimation (e.g., quantile-based ranges) may be needed to stabilize the entropy term.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes NMINE, a fully neural estimator for normalized mutual information (NMI) between continuous multidimensional variables. Mutual information is estimated via three independent Donsker–Varadhan (DV) neural critics: one for the joint distribution and one for each marginal, all relative to uniform reference distributions over the observed sample support. Marginal and joint entropies are recovered from the divergence estimates via H = log V − D, and the final NMI is computed using asymmetric normalization I(X;Y)/H(Y). The method is evaluated on multivariate Gaussian data in dimensions 1, 2, 4, and 8 against a KSG-based baseline, reporting substantially lower mean absolute error relative to analytic Gaussian NMI values.

Significance. If the method worked as claimed, it would provide a neural alternative to kNN-based NMI estimation and could be useful in applications requiring normalized dependency scores in continuous, multidimensional settings. The paper is clearly written, the experimental setup is transparent, and the decomposition I = D_XY − D_X − D_Y is algebraically correct when the reference distribution factorizes as m_XY = m_X m_Y. The Gaussian benchmark, the repeated-seed protocol, and the paired t-test are also clearly specified. However, the central methodological step—entropy recovery via a uniform reference on the observed sample support—is not a valid KL divergence for the unbounded-support Gaussian distributions used in the evaluation. This undermines the interpretation of the reported MAE results as accuracy for NMI. The core idea may be salvageable with a properly specified reference measure, but the current manuscript does not establish its central claim.

major comments (3)
  1. [Neural Entropy Estimation, Eq. (H = log V − D)] The identity H(X) = log V_X − D_X requires that the target density p_X be absolutely continuous with respect to the uniform reference m_X and that m_X have density 1/V_X on the entire support of p_X. For Gaussian data, the support is R^d. The implementation instead defines m_X as uniform over the smallest axis-aligned box containing the observed samples, so m_X has density zero outside that box. Consequently the exact D_KL(p_X || m_X) is infinite, and the DV objective with a finite-capacity critic estimates a different, function-class- and sample-dependent quantity. The same issue applies to D_Y and D_XY. Therefore bD_X, bD_Y, bD_XY, and the derived bI and bH do not correspond to the information-theoretic quantities used in the analytic Gaussian ground truth. The Discussion acknowledges this as a finite-sample approximation, but a finite-sample approximation to an infinite divergence is
  2. [Scale-Invariant Estimation vs. Proposed Method] The paper motivates the method using the relative entropy H_r(X) = −∫ p log(p/m) and claims that normalization based on H_r is scale invariant. However, the implemented estimator uses H = log V − D, which is exactly the differential entropy H_d when D is the KL divergence to a uniform density m = 1/V. Differential entropy is not scale invariant. Under x′ = a x, log V changes by d log a, while D_KL(p||m) is invariant if both p and m transform together; thus H = log V − D changes by d log a. The denominator H(Y) in the asymmetric NMI therefore changes under scaling, so the implemented NMINE is not scale invariant despite the section title. If the intended target is H_r for a fixed reference measure, the paper must show how the neural estimator targets H_r rather than H_d. As written, the invariance claim and the implemented estimator are inconsistent.
  3. [Experimental Results, Table 2] Because the reference-support mismatch makes the estimand ill-defined, the reported MAE reductions (30.7–74.0%) cannot be interpreted as evidence that NMINE more accurately estimates normalized mutual information. The finite-sample observed-range volume V is itself random and grows with sample size for unbounded supports, so the entropy recovery step introduces a sample-size-dependent bias. Lower MAE against the analytic Gaussian NMI may arise from a favorable cancellation of biases in bI and bH rather than from accurate estimation of the target. To support the central claim, the authors should either (a) use a reference distribution for which p is absolutely continuous and the reference entropy is known analytically (e.g., a fixed Gaussian reference), or (b) evaluate on bounded-support distributions where a uniform reference over the true support is legitimate. The current experiment do
minor comments (5)
  1. [Abstract] There is a duplicated phrase: 'For continuous and multidimensional variables For continuous multidimensional variables'.
  2. [Background / Neural Entropy Estimation] The method is described as 'MI-NEE-inspired', but the cited work (Chan et al. 2019) is 'Neural Entropic Estimation' (NEE), not 'MI-NEE'. Please clarify the terminology and relationship to MINE.
  3. [References] McAllester and Stratos 2018a and 2018b are the same arXiv preprint; cite only once or distinguish versions.
  4. [Figure 2 caption] The phrase 'can be estimated but is unused in asymmetric normalization' is awkward; consider rewording to explain that H(X) is estimated but not used in the final normalized ratio.
  5. [Experimental setup] The KSG baseline is described only as 'k = 5 nearest neighbors'. Please specify the entropy estimator used (e.g., Kozachenko–Leonenko) and whether the same k is used for both MI and entropy, as this affects the baseline comparison.

Circularity Check

0 steps flagged

No circularity: NMINE estimates MI and entropy from samples only; Gaussian ground truth is external and none of the fitted quantities are reused as targets.

full rationale

The derivation chain is self-contained and non-circular. The estimator's three neural critics approximate Donsker–Varadhan divergences D_XY, D_X, D_Y from samples against uniform reference boxes; the paper's formulas bI = bD_XY - bD_X - bD_Y and bH = log V - bD are algebraic identities, not constructed predictions. No parameter is fitted to the analytical Gaussian NMI values used in Table 2, and no target quantity appears as an input to the estimator. The self-citations (Tuononen and Hautamäki 2025; Tuononen, Korpi, and Hautamäki 2025) are used only to motivate numerical stability concerns and applications, not to justify the central estimation identities. The uniform-reference-support assumption for unbounded Gaussian distributions is a genuine statistical/correctness concern—the paper itself concedes it is a finite-sample approximation 'whose effect should be investigated further'—but this is a mismatch between the ideal estimand and the implemented estimator, not a circularity. The decomposition I = D(p_XY||m_X m_Y) - D(p_X||m_X) - D(p_Y||m_Y) is a mathematical identity; using it to estimate MI does not assume the result. Hence the score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No scalar constants are fitted to ground-truth NMI; the estimator uses no target-derived parameters. However, the method's output depends on three hand-chosen components: data-dependent reference volumes, critic hyperparameters, and a fixed KSG baseline configuration. The mathematical apparatus relies on the DV representation, the validity of a compact uniform reference for unbounded distributions (which fails for Gaussians), and an unproven linear combination of biased estimates. No new physical entities are introduced.

free parameters (3)
  • Reference support volumes V_X, V_Y, V_XY = data range estimates (product of observed max-min)
    Entropy estimates bH=logV-bD and therefore NMI depend on this arbitrary data-dependent support; for unbounded distributions it is a finite-sample approximation that also breaks scale invariance.
  • KSG baseline k = 5
    Headline error reductions depend on this hand-chosen neighbor count; no sensitivity analysis is reported.
  • Neural critic hyperparameters = 2x128 ReLU, LR 1e-5, batch 256, 500/100 epochs
    Set by hand without sensitivity analysis; results may depend on these choices.
axioms (4)
  • standard math Donsker-Varadhan variational representation for KL divergence
    Invoked in all three critics (DXY, DX, DY); standard theorem, but requires the reference m to be a probability measure and p to be absolutely continuous with respect to it.
  • ad hoc to paper Uniform reference measure on observed sample support is valid for entropy recovery
    Entropy recovery H=logV-D relies on m=1/V over the observed box; for Gaussian tails p is not absolutely continuous with respect to this compact-support measure, so the exact KL is infinite and the DV objective estimates a different quantity.
  • domain assumption Independent variational estimates can be linearly combined into a valid MI estimate
    I=DXY-DX-DY is exact for exact divergences, but MINE-style estimates are biased; the paper states the combination is not a variational bound and defers bias/variance analysis.
  • ad hoc to paper Asymmetric normalization I/H(Y) with differential entropy H(Y) is the intended target
    The paper motivates invariant relative-entropy normalization but implements H(Y)=log V_Y - D_Y, which is differential entropy and changes under rescaling.

pith-pipeline@v1.3.0-daily-deepseek · 9175 in / 22158 out tokens · 227041 ms · 2026-08-01T02:51:01.500603+00:00 · methodology

0 comments
read the original abstract

Mutual information is a general measure of statistical dependence that captures both linear and nonlinear relationships between random variables. For continuous and multidimensional variables For continuous multidimensional variables, mutual information must be estimated from samples. Because mutual information is unbounded, its values are not directly comparable across datasets, dimensions, or applications. Normalized mutual information addresses this limitation by converting mutual information into a normalized dependency score. Recent work has demonstrated the practical value of normalized mutual information in applications such as molecular dynamics {arXiv:2405.04980} and interpretable machine learning {arXiv:2409.16768}, but existing estimators remain sensitive to dimensionality and numerical stability {arXiv:2410.07642}. In this paper, we propose a fully neural normalized mutual information estimator for continuous variables. The proposed approach combines a MINE-based neural mutual information estimator {arXiv:1801.04062} with MI-NEE-inspired neural marginal entropy estimators {arXiv:1905.12957}. Mutual information is estimated using the Donsker--Varadhan representation, while marginal entropies are estimated by learning the divergence between each marginal distribution and a uniform reference distribution, from which entropy is recovered. The resulting estimator provides a neural alternative to k-nearest-neighbor-based normalized mutual information estimation {arXiv:2405.04980}. Experiments on Gaussian data from one to eight dimensions show that the proposed estimator improves accuracy over a KSG-based normalized mutual information baseline. These results indicate that neural estimation is a promising direction for normalized dependency measurement in continuous multidimensional settings.

Figures

Figures reproduced from arXiv: 2607.27710 by Marko Tuononen, Petra Eerikinharju, Ville Hautam\"aki.

Figure 1
Figure 1. Figure 1: Conceptual overview of the proposed NMINE [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of the proposed NMINE framework. Input samples [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Estimated normalized mutual information (NMI) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 4 linked inside Pith

  1. [1]

    2006 , publisher=

    Elements of Information Theory , author=. 2006 , publisher=

    2006

  2. [2]

    Physical Review E , volume=

    Estimating Mutual Information , author=. Physical Review E , volume=

  3. [3]

    International Conference on Machine Learning , pages=

    Mutual Information Neural Estimation , author=. International Conference on Machine Learning , pages=

  4. [4]

    Entropy , volume=

    Neural Entropic Estimation , author=. Entropy , volume=

  5. [5]

    Neural Computation , volume=

    Estimation of Entropy and Mutual Information , author=. Neural Computation , volume=

  6. [6]

    AISTATS , year=

    Efficient Estimation of Mutual Information for Strongly Dependent Variables , author=. AISTATS , year=

  7. [7]

    IEEE Transactions on Information Theory , volume=

    Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization , author=. IEEE Transactions on Information Theory , volume=

  8. [8]

    arXiv preprint arXiv:1807.03748 , year=

    Representation Learning with Contrastive Predictive Coding , author=. arXiv preprint arXiv:1807.03748 , year=

    Pith/arXiv arXiv

  9. [9]

    ICLR , year=

    Learning Deep Representations by Mutual Information Estimation and Maximization , author=. ICLR , year=

  10. [10]

    Journal of Machine Learning Research , volume=

    Information Theoretic Measures for Clusterings Comparison , author=. Journal of Machine Learning Research , volume=

  11. [11]

    IEEE Transactions on Systems Science and Cybernetics , volume=

    Prior Probabilities , author=. IEEE Transactions on Systems Science and Cybernetics , volume=

  12. [12]

    The Journal of Chemical Physics , volume =

    Nagel, Daniel and Diez, Georg and Stock, Gerhard , title =. The Journal of Chemical Physics , volume =. 2024 , doi =

    2024

  13. [13]

    Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensions , journal =

    Tuononen, Marko and Hautam. Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensions , journal =. 2025 , doi =

    2025

  14. [14]

    Jerdee, Maximilian and Kirkley, Alec and Newman, M. E. J. , title =. Nature Communications , volume =

  15. [15]

    Interpreting Deep Neural Network-Based Receiver Under Varying Signal-To-Noise Ratios , booktitle =

    Tuononen, Marko and Korpi, Dani and Hautam. Interpreting Deep Neural Network-Based Receiver Under Varying Signal-To-Noise Ratios , booktitle =. 2025 , doi =

    2025

  16. [16]

    arXiv preprint arXiv:1811.04251 , year=

    Formal Limitations on the Measurement of Mutual Information , author=. arXiv preprint arXiv:1811.04251 , year=

    Pith/arXiv arXiv

  17. [17]

    Advances in Neural Information Processing Systems , volume=

    Isolating Sources of Disentanglement in Variational Autoencoders , author=. Advances in Neural Information Processing Systems , volume=

  18. [18]

    arXiv preprint arXiv:2110.11334 , year=

    Out-of-Distribution Detection in Deep Learning: A Survey , author=. arXiv preprint arXiv:2110.11334 , year=

    Pith/arXiv arXiv

  19. [19]

    European Conference on Computer Vision Workshops , year=

    Learning Disentangled Representations via Mutual Information Estimation , author=. European Conference on Computer Vision Workshops , year=

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    International Conference on Machine Learning , pages=

    On Variational Bounds of Mutual Information , author=. International Conference on Machine Learning , pages=

  22. [22]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

  23. [23]

    Journal of Machine Learning Research , volume=

    Fast Binary Feature Selection with Conditional Mutual Information , author=. Journal of Machine Learning Research , volume=

  24. [24]

    AAAI Conference on Artificial Intelligence , pages=

    Cluster Ensembles: A Knowledge Reuse Framework for Combining Multiple Partitions , author=. AAAI Conference on Artificial Intelligence , pages=

  25. [25]

    International Conference on Learning Representations , year=

    A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , author=. International Conference on Learning Representations , year=

  26. [26]

    International Conference on Learning Representations , year=

    Adam: A Method for Stochastic Optimization , author=. International Conference on Learning Representations , year=

  27. [27]

    Communications on Pure and Applied Mathematics , volume=

    Asymptotic Evaluation of Certain Markov Process Expectations for Large Time , author=. Communications on Pure and Applied Mathematics , volume=

  28. [28]

    arXiv preprint arXiv:2104.06612 , year=

    Deep Data Density Estimation through Donsker--Varadhan Representation , author=. arXiv preprint arXiv:2104.06612 , year=

    Pith/arXiv arXiv

  29. [29]

    , title =

    Shannon, Claude E. , title =. The Bell System Technical Journal , volume =. 1948 , month =

    1948