{"id":"89fdfc94-a331-40dd-b16b-b169b1d1a8d8","arxiv_id":"2607.27710","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A fully neural estimator for normalized mutual information beats a KSG baseline on Gaussian data but fails to deliver its advertised scale-invariance property.","lead":"This paper introduces NMINE, a neural-network-based estimator for normalized mutual information, a score that measures how much one variable explains another, and tests it on synthetic Gaussian data. It reports lower estimation error than a k-nearest-neighbor baseline, but the proposed normalization is not scale-invariant as claimed, so the method does not yet deliver the advertised comparable score.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference-support mismatch makes NMINE's estimand ill-defined for Gaussian data; reported MAE may not measure accuracy for NMI.","rationale":"I read the paper in good faith. The method is clearly specified and the Gaussian experiments are reproducible in principle. The most load-bearing condition for the central claim is that the entropy-recovery identity H = log V - D produces a valid estimate of the entropy used in the analytical ground truth. That condition fails for unbounded distributions because the uniform reference is not absolutely continuous w.r.t. the Gaussian; the KL is infinite and the DV critic with finite capacity estimates a function-class-constrained surrogate. The paper's own limitation statement confirms this. I agree with the reader that this is the central soft spot. A secondary internal inconsistency is that the 'Scale-Invariant Estimation' section motivates relative-entropy normalization, but the implemented formula H = log V - D is differential entropy, so the invariance claim is not realized. I do not see a more load-bearing concern: the comparison against KSG would be meaningful if the estimand were well-defined, and the neural architecture is standard. The practical test I propose (capacity-dependence of the entropy estimate) would settle whether the Gaussian results are an artifact of finite capacity. If the test shows stability, the conditional verdict could be upgraded; if not, the central claim should be rejected. Thus I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":9540,"tokens_out":8572,"duration_ms":88873,"concrete_test":"Reproduce the paper's entropy-estimation pipeline on a univariate standard Gaussian (d=1, n=5000, 10 seeds) and compare the recovered \\hat{H}(X) to the true differential entropy H = 0.5 log(2πe). Then repeat with hidden-layer widths 128, 256, 512 (same training schedule) and with n=50000, 500000. If |\\hat{H}(X) - H| does not shrink with n or grows with width (e.g., drifts by >0.1 nat), the entropy-recovery step is capacity-dependent and has no well-defined population limit; this would directly invalidate the NMI denominator in the Gaussian experiments and the claimed accuracy gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is entropy recovery via H = log V - D (section 'Neural Entropy Estimation'). This identity is valid only if the target distribution p is absolutely continuous with respect to the uniform reference m on the observed box, and if m has density 1/V everywhere p is positive. For Gaussian data, p has unbounded support while m vanishes outside the sample box, so D_KL(p||m) = ∞. The DV objective trained with a finite-capacity critic then estimates a different, function-class- and sample-dependent quantity, not the KL divergence. Moreover, the same section computes H = log V - D, which is the differential entropy, not the relative entropy H_r defined in the 'Scale-Invariant Estimation' section; the claimed invariance therefore does not hold for the implemented estimator. Since MI is computed as D_XY - D_X - D_Y, the result is the difference of three ill-defined quantities, and the limit need not equal I(X;Y). The experimental comparison in Table 2 thus measures deviations from an analytical NMI that NMINE is not actually targeting; lower MAE may reflect a favorable finite-sample bias rather than genuine estimation of NMI. The paper's own Discussion concedes this is a finite-sample approximation whose effect should be investigated, but the concession does not repair the missing estimand.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NMINE, a fully neural estimator for normalized mutual information (NMI) between continuous multidimensional variables. Mutual information is estimated via three independent Donsker–Varadhan (DV) neural critics: one for the joint distribution and one for each marginal, all relative to uniform reference distributions over the observed sample support. Marginal and joint entropies are recovered from the divergence estimates via H = log V − D, and the final NMI is computed using asymmetric normalization I(X;Y)/H(Y). The method is evaluated on multivariate Gaussian data in dimensions 1, 2, 4, and 8 against a KSG-based baseline, reporting substantially lower mean absolute error relative to analytic Gaussian NMI values.","tokens_in":9845,"tokens_out":4739,"duration_ms":55349,"significance":"If the method worked as claimed, it would provide a neural alternative to kNN-based NMI estimation and could be useful in applications requiring normalized dependency scores in continuous, multidimensional settings. The paper is clearly written, the experimental setup is transparent, and the decomposition I = D_XY − D_X − D_Y is algebraically correct when the reference distribution factorizes as m_XY = m_X m_Y. The Gaussian benchmark, the repeated-seed protocol, and the paired t-test are also clearly specified. However, the central methodological step—entropy recovery via a uniform reference on the observed sample support—is not a valid KL divergence for the unbounded-support Gaussian distributions used in the evaluation. This undermines the interpretation of the reported MAE results as accuracy for NMI. The core idea may be salvageable with a properly specified reference measure, but the current manuscript does not establish its central claim.","major_comments":[{"comment":"The identity H(X) = log V_X − D_X requires that the target density p_X be absolutely continuous with respect to the uniform reference m_X and that m_X have density 1/V_X on the entire support of p_X. For Gaussian data, the support is R^d. The implementation instead defines m_X as uniform over the smallest axis-aligned box containing the observed samples, so m_X has density zero outside that box. Consequently the exact D_KL(p_X || m_X) is infinite, and the DV objective with a finite-capacity critic estimates a different, function-class- and sample-dependent quantity. The same issue applies to D_Y and D_XY. Therefore bD_X, bD_Y, bD_XY, and the derived bI and bH do not correspond to the information-theoretic quantities used in the analytic Gaussian ground truth. The Discussion acknowledges this as a finite-sample approximation, but a finite-sample approximation to an infinite divergence is","section":"Neural Entropy Estimation, Eq. (H = log V − D)"},{"comment":"The paper motivates the method using the relative entropy H_r(X) = −∫ p log(p/m) and claims that normalization based on H_r is scale invariant. However, the implemented estimator uses H = log V − D, which is exactly the differential entropy H_d when D is the KL divergence to a uniform density m = 1/V. Differential entropy is not scale invariant. Under x′ = a x, log V changes by d log a, while D_KL(p||m) is invariant if both p and m transform together; thus H = log V − D changes by d log a. The denominator H(Y) in the asymmetric NMI therefore changes under scaling, so the implemented NMINE is not scale invariant despite the section title. If the intended target is H_r for a fixed reference measure, the paper must show how the neural estimator targets H_r rather than H_d. As written, the invariance claim and the implemented estimator are inconsistent.","section":"Scale-Invariant Estimation vs. Proposed Method"},{"comment":"Because the reference-support mismatch makes the estimand ill-defined, the reported MAE reductions (30.7–74.0%) cannot be interpreted as evidence that NMINE more accurately estimates normalized mutual information. The finite-sample observed-range volume V is itself random and grows with sample size for unbounded supports, so the entropy recovery step introduces a sample-size-dependent bias. Lower MAE against the analytic Gaussian NMI may arise from a favorable cancellation of biases in bI and bH rather than from accurate estimation of the target. To support the central claim, the authors should either (a) use a reference distribution for which p is absolutely continuous and the reference entropy is known analytically (e.g., a fixed Gaussian reference), or (b) evaluate on bounded-support distributions where a uniform reference over the true support is legitimate. The current experiment do","section":"Experimental Results, Table 2"}],"minor_comments":[{"comment":"There is a duplicated phrase: 'For continuous and multidimensional variables For continuous multidimensional variables'.","section":"Abstract"},{"comment":"The method is described as 'MI-NEE-inspired', but the cited work (Chan et al. 2019) is 'Neural Entropic Estimation' (NEE), not 'MI-NEE'. Please clarify the terminology and relationship to MINE.","section":"Background / Neural Entropy Estimation"},{"comment":"McAllester and Stratos 2018a and 2018b are the same arXiv preprint; cite only once or distinguish versions.","section":"References"},{"comment":"The phrase 'can be estimated but is unused in asymmetric normalization' is awkward; consider rewording to explain that H(X) is estimated but not used in the final normalized ratio.","section":"Figure 2 caption"},{"comment":"The KSG baseline is described only as 'k = 5 nearest neighbors'. Please specify the entropy estimator used (e.g., Kozachenko–Leonenko) and whether the same k is used for both MI and entropy, as this affects the baseline comparison.","section":"Experimental setup"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is interesting, but the current implementation's entropy-recovery step is not valid for the Gaussian benchmark used in the evaluation. The missing estimand is a load-bearing issue, not a presentation issue. I would be willing to reconsider a revised version that replaces the observed-support uniform reference with a proper reference measure (or evaluates on bounded-support data) and re-runs the experiments accordingly. Without such a change, the reported numerical results do not support the claimed accuracy for NMI estimation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a clearly written paper that proposes the first fully neural estimator for normalized mutual information, as far as the authors cite, and it shows an empirical improvement over a KSG baseline on Gaussian data. The combination of MINE and NEE is new, the method is specified well enough to re-implement, and the experiments are honest in their simplicity. But the entropy estimation step has a load-bearing flaw: the formula H = log V - D is only valid when the data distribution is supported inside the box that defines the uniform reference. Gaussian data have unbounded support, so the true KL divergence to that uniform reference is infinite. The finite-capacity critic estimates something else, and the resulting H and MI are not the differential entropy and mutual information of the Gaussian. The paper itself concedes this in the Discussion, but the concession doesn't repair the missing estimand: the reported MAE numbers are deviations from an analytic NMI that NMINE is not actually targeting. The scale-invariance discussion also doesn't match the implementation — they define relative entropy H_r but then use the differential entropy formula.\n\nThe experiments are clean but thin: only Gaussian data with n=5000, dimensions 1-8, plus a Student-t run with no ground truth. No code, no nonlinear or real-world data. So the empirical support for a neural NMI is suggestive, not strong.\n\nMy recommendation: send it to reviewers, but with a clear expectation of major revision. The core idea is plausible and the paper is readable, but the reference-support issue needs to be fixed (e.g., by using a reference measure that dominates the target, or by explicitly redefining the estimand as NMI of truncated distributions) and the claims need to be scaled back. If a fix is possible, this could be a useful contribution to the dependency-estimation toolbox.","headline":"Neural NMI via MINE+NEE is worth a look, but the entropy-recovery step is not defined for unbounded distributions, so the Gaussian benchmark may not measure NMI.","tokens_in":10331,"tokens_out":7326,"would_cite":false,"duration_ms":73428,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62B10","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"NMINE, a fully neural estimator combining three Donsker–Varadhan critics with entropy recovery from uniform reference distributions, estimates normalized mutual information more accurately than a KSG-based baseline on multidimensional Gauss","keywords":["normalized mutual information","neural estimation","Donsker–Varadhan representation","entropy estimation","uniform reference distribution","KSG estimator","continuous variables","asymmetric normalization"],"falsifier":"For a Gaussian with unbounded support, compute the true Kullback–Leibler divergence between the Gaussian and the uniform distribution on the observed range: it is infinite. A neural critic trained against box samples returns a finite number; if that finite value is an artifact of the box, then the recovered entropy and NMI should shift systematically as the sample size—and hence the observed range—changes. This can be tested directly by measuring NMINE's output across increasing sample sizes.","tokens_in":9408,"feed_emoji":"📈","tokens_out":3610,"duration_ms":36348,"temperature":0.7,"pith_summary":"Normalized mutual information is hard to estimate for continuous multidimensional variables because the raw quantity is unbounded and depends on the units of the variables. The paper proposes NMINE, a fully neural estimator that obtains the asymmetric normalized mutual information I(X;Y)/H(Y) by training three Donsker–Varadhan critics: one for the joint distribution and one for each marginal, each measuring divergence to a uniform reference distribution on the observed sample support. Entropies are recovered as log V − D, where V is the volume of the reference box, and mutual information as D_XY − D_X − D_Y. On Gaussian data from one to eight dimensions, NMINE reports lower mean absolute error than a KSG-based baseline, with relative reductions of 30.7–74.0%, and preserves the monotonic relationship between correlation and NMI even where it underestimates. A sympathetic reader would care because this is a concrete demonstration that neural estimation can replace nearest-neighbor statistics for normalized dependency measurement.","feed_headline":"Neural NMI estimator beats KSG by up to 74% on Gaussian data","feed_subtitle":"Three Donsker–Varadhan critics and entropy recovery beat a nearest-neighbor baseline on Gaussian benchmarks.","key_machinery":"The central machinery is a set of three Donsker–Varadhan variational critics, each trained to estimate the Kullback–Leibler divergence between an empirical distribution and a uniform reference distribution supported on the observed sample box. The reference density is 1/V, with V the product of the observed per-dimension ranges, which allows entropy recovery in closed form as H = log V − D. Mutual information is then obtained from the divergence decomposition I(X;Y) = D_KL(p_XY || m_X m_Y) − D_KL(p_X || m_X) − D_KL(p_Y || m_Y), and normalized mutual information as the ratio I(X;Y)/H(Y). This construction replaces nearest-neighbor statistics with neural variational estimation throughout the p","core_discovery":"The paper claims that a fully neural pipeline can estimate normalized mutual information for continuous variables more accurately than a nearest-neighbor-based baseline. The estimator, NMINE, uses the Donsker–Varadhan variational representation to train three separate neural critics: one for the joint distribution of (X, Y) and one for each marginal, each measuring the Kullback–Leibler divergence to a uniform reference distribution defined over the observed sample support. Differential entropy is recovered from each divergence via H = log V − D, where V is the product of the observed per-dimension ranges, and mutual information is recovered as D_XY − D_X − D_Y, following the relative-entropy","pith_inferences":["The box-support uniform reference is a finite-sample device: for distributions with unbounded support, the true KL divergence to a compact uniform measure is infinite, so the recovered entropy and NMI should depend on sample size and extreme observations; this can be tested by sweeping sample count and observing drift.","The decomposition I = D_XY − D_X − D_Y means the final MI estimate is not itself a variational lower bound; its bias and variance are composition effects of three independently trained critics, so error analysis could be decomposed by critic quality.","The observed underestimation bias at high dimensions may be tied to the known difficulty of variational MI estimation when true MI is large relative to sample size; this suggests that variance-reduction techniques such as SMILE or critic clipping could narrow the gap.","Because the reference volume is computed from observed ranges, NMINE is tied to the empirical support; for heavy-tailed or outlier-prone data, robust support estimation (e.g., quantile-based ranges) may be needed to stabilize the entropy term."],"forward_implications":["If the central claim is correct, normalized mutual information for continuous variables can be estimated with a fully neural pipeline that outperforms k-nearest-neighbor baselines in low-to-moderate dimensions.","The asymmetric normalization used here preserves the ranking induced by mutual information, so NMINE can be used to compare dependency strength across different variable pairs without unit-calibration artifacts.","The reported monotonic response to correlation, even when underestimating, suggests that NMINE can serve as a reliable dependency signal for ranking or trend detection in scientific applications.","The framework extends naturally to non-Gaussian settings; preliminary Student-t experiments suggest the estimator remains stable for heavy-tailed distributions, though analytical ground truth is not available there.","Because all components are neural and differentiable, the estimator can be embedded in gradient-based optimization pipelines, enabling end-to-end learning objectives that involve normalized mutual information."],"fun_headline_variants":["Neural net for normalized mutual info outshines KSG on Gaussians","NMINE: neural normalized MI estimator trumps KSG in tests","Neural estimator for NMI beats nearest-neighbor baseline","Fully neural NMI estimation: better than KSG on Gaussians"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The estimate assumes that a uniform reference distribution defined on the box spanned by the observed samples is a valid baseline for measuring entropy and divergence, even though real distributions like Gaussians have tails outside that box.","fun_headline_variants_meta":{"raw":{"variants":["Neural net for normalized mutual info outshines KSG on Gaussians","NMINE: neural normalized MI estimator trumps KSG in tests","Neural estimator for NMI beats nearest-neighbor baseline","Fully neural NMI estimation: better than KSG on Gaussians"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000647,"raw_usage":{"total_tokens":2839,"prompt_tokens":806,"completion_tokens":2033,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1957}},"tokens_in":550,"tokens_out":2033,"duration_ms":14028,"temperature":1.0,"reasoning_tokens":1957,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:51:01.500603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a Gaussian with unbounded support, compute the true Kullback–Leibler divergence between the Gaussian and the uniform distribution on the observed range: it is infinite. A neural critic trained against box samples returns a finite number; if that finite value is an artifact of the box, then the recovered entropy and NMI should shift systematically as the sample size—and hence the observed range—changes. This can be tested directly by measuring NMINE's output across increasing sample sizes.","supporting_citations":[],"review_version":1}