Pith. sign in

REVIEW 4 major objections 5 minor 73 references

Accurate Estimation of Mutual Information in High Dimensional Data

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims neural mutual-information estimators become reliable in high-dimensional data when the dependence between the variables lives on a low-dimensional latent space, and builds a protocol with error bars and built-in failure…

desk verdict A practical protocol for neural MI estimation with error bars and honest diagnostics, but the promised CIFAR results are missing and the stopping rule needs stronger justification. read the letter →

arxiv 2506.00330 v3 pith:JJ5Z55QU submitted 2025-05-31 physics.data-an cs.ITmath.ITstat.ML

Eslam Abdelaleem , K. Michael Martini , Ilya Nemenman This is my paper
classification physics.data-ancs.ITmath.ITstat.ML MSC 62B1094A17
keywords mutualinformationestimationneuralestimatorshigh-dimensionaldatalatentdimensionalitysamplecomplexityearlystoppingheuristicconfidenceintervalsrandommatrixtheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mutual information (MI) is the standard measure of statistical dependence, but in high-dimensional data with few samples ($N \lesssim K$) all known estimators either break down or give the user no way to know they have broken down. This paper's central claim is that neural MI estimators become reliable precisely when the dependence between the two variables lives on a low-dimensional latent space: the sample size needed is set by the latent dimension $K_Z$, not the ambient dimension $K$. The authors support the claim with a random-matrix-theory argument tying the onset of a nonzero estimate to the detection threshold of a spiked covariance signal in the latent space, and they assemble a practical protocol around it—a max-test early-stopping rule, a bias-removing subsampling extrapolation, and prediction intervals. They also introduce probabilistic (VSIB) critics that stabilize estimation at high MI values where standard estimators saturate or overfit. Across synthetic $K=500$ problems, a standard 40-dataset benchmark suite, and noisy MNIST ($K=784$), the pipeline matches or beats existing methods while being the only one that reports confidence intervals and declares when it cannot be trusted.

What carries the argument

Three mechanisms carry the argument. First, the generalized critic $T(x,y) = f(g(x), h(y))$: separable embeddings force the estimator through a latent bottleneck of dimension $k_Z$, and the paper shows a critic needs $k_Z \ge K_Z$ to capture all dependence, while modestly exceeding $K_Z$ is harmless—turning the embedding dimension into a dial that reveals the structure of the data. Second, the max-test stopping rule: because MI is a nonlinear functional of the distribution, unbiased density estimates do not yield unbiased MI, and the held-out MI curve rises then collapses as the critic overfits; the paper selects the checkpoint with peak test MI and reports the corresponding training MI, arguing in Appx. A.3 that the test value is systematically biased downward while the training value at the best-generalizing checkpoint tracks the truth. Third, the random-matrix-theory detection bound: in a spiked-covariance model of the latent dependence, a signal spike separates from sampling noise only when $N$ exceeds $N^*_Z \approx 2K_Z/\theta^2$, which scales as $K_Z^2$ in the weak-signal limit—this is what makes 'sample the latent space, not the data space' quantitative. The VSIB probabilistic critics add a fourth piece: stochastic encoders with the loss $I_E(X;Z_X) + I_E(Y;Z_Y) - \beta I_D(Z_X;Z_Y)$ regularize the critic and control variance in the high-MI regime.

What would settle it

On a teacher-network generator with known latent dimension ($K_Z = 10$) embedded into $K = 500$ dimensions but with latent variables drawn from a distribution the paper did not test—e.g., heavy-tailed, discrete, or concentrated on a low-dimensional curved manifold—run the full protocol at $N = 256$ and $N = 1024$ and check whether the reported estimate plus prediction interval covers the known true MI. If the coverage fails, or if the bias changes sign when the embedding map is varied with $K_Z$ held fixed, the max-test stopping rule's core premise is falsified. A sharper test removes the paper's own escape hatch: restrict the critic family so that the optimal density-ratio critic is provably outside it, since the Appx. A.3 justification for preferring the training value over the test value requires the trained critic to be near-optimal.

Watch

Extended reading notes

Core claim

The paper establishes a regime-shift principle: the difficulty of estimating MI is governed by the dimensionality of the statistical dependence itself, not of the observed variables. For data generated by a latent model with $K_Z \le 10$ hidden variables embedded into $K = 500$ observed dimensions, trained critics recover the ground-truth MI once the number of samples satisfies $N \gg K_Z$, even when $N$ is far below $K$; the estimate only begins to form once $N$ passes a latent-space detection threshold $N^*_Z$ that scales roughly as $K_Z^2/I$, matching the spiked-covariance phase transition of random matrix theory. The paper also establishes a protocol that makes this usable: stop the critic at the epoch where held-out MI peaks and report the training value at that checkpoint; grow the critic embedding until the estimate plateaus at $k^*_Z$; subsample into $\gamma$ equal parts and extrapolate a weighted linear fit to $\gamma \to 0$; report the intercept as the MI estimate with a prediction interval, and refuse to report anything if the fit is nonlinear. A new family of probabilistic critics, VSIB, wraps InfoNCE or SMILE in stochastic encoders regularized by an information-bottleneck objective, and the paper shows this suppresses SMILE's severe overestimation at high MI. Empirically, the pipeline stays within error bars of the true MI on the benchmark suite and recovers $3.13 \pm 0.12$ bits against a true $\log_2 10 \approx 3.32$ bits on 784-dimensional noisy MNIST from 16,384 samples, without ever significantly overshooting.

Load-bearing premise

The load-bearing premise is that the training MI at the checkpoint where held-out MI peaks is a better estimate of true MI than the held-out value itself; the paper's justification of this rule indirectly assumes the trained critic is already close to the ideal critic, so distributions for which that closeness fails could make the reported values and their error bars wrong even when low-dimensional latent structure exists.

Editorial extensions

If this is right

  • Mutual information can be estimated with honest error bars from a few hundred samples in problems with ambient dimension $K \approx 500$, provided the true dependence is low-dimensional ($K_Z \approx 10$).
  • The data requirement grows roughly quadratically with the latent dimension, so the plateau in the critic-embedding curve both diagnoses the latent dimension and sets the sample budget needed for a trustworthy estimate.
  • Compressive embeddings are not an optional convenience in high dimensions; an estimator that does not project into a low-dimensional space cannot exploit the latent structure that makes estimation possible.
  • In high-MI regimes, the VSIB probabilistic critics keep estimates stable where InfoNCE saturates near $\log(\text{batch size}) \approx 7$ bits and plain SMILE overfits upward.
  • The protocol returns a falsifiable output: if the $\gamma$-extrapolation is nonlinear or the fit range shrinks below $\gamma = 5$, the pipeline refuses to report a number, giving scientists a built-in failure signal that existing neural estimators lack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the latent-dimension principle holds generally, practitioners can pre-register a data budget: estimate $K_Z$ from the critic-plateau curve and collect $N \gtrsim K_Z^2$ samples, rather than treating ambient dimension as the driver of sample complexity.
  • The random-matrix-theory connection suggests a diagnostic the paper does not itself build: comparing the observed onset of nonzero MI against the $N^*_Z$ prediction for candidate $K_Z$ values could estimate the effective latent dimension of a real dataset directly from the data.
  • The validation set is concentrated on smooth, continuous dependence structures (teacher networks, Gaussian links, image labels); an untested stress case is dependence carried by discrete or non-smooth structure, where the critic's smoothness assumptions and the linear $\gamma$-extrapolation could fail even with small $K_Z$.
  • The paper's inversion of standard practice—trust the training value at the peak-test checkpoint rather than the test value—rests on MI being a nonlinear functional; the same logic may apply to other nonlinear population functionals such as entropy or divergences, but the paper does not test that generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a practical protocol for neural mutual information (MI) estimation in high-dimensional, finite-sample settings. The protocol combines a max-test early-stopping rule, subsampling-based bias extrapolation, explicit unreliability flags, and confidence intervals, and it introduces probabilistic VSIB critics claimed to reduce bias and variance at high MI values. The central claim is that reliable MI estimation is governed by the true latent dimensionality K_Z rather than the ambient dimension K, so that MI can be estimated from hundreds to thousands of samples in problems where K is 500-784, provided the dependence is low-dimensional. The evidence includes synthetic teacher-network experiments with K=500, a subset of the Czyz et al. (2023) benchmark suite, and a noisy MNIST experiment with K=784. The abstract also claims validation on CIFAR-10/100 with a ResNet-20 backbone, but no such results appear in the body.

Significance. If the central claims hold, this would be a practically valuable contribution: it would turn neural MI estimators into more trustworthy scientific instruments by adding consistency checks, confidence intervals, and a clear statement of when estimates should not be trusted. The synthetic experiments are well designed for the latent-dimensionality hypothesis, and the MNIST result is a plausible demonstration of the proposed regime shift. The comparison to the Czyz et al. (2023) benchmarks is useful and generally favorable. However, the paper's theoretical grounding is a heuristic random-matrix calculation, and the load-bearing max-test stopping rule is justified only by a loose inequality plus empirical observation. The abstract's CIFAR-10/100 claim is unsupported in the body. These issues need to be resolved before the stronger claims of the paper can be accepted.

major comments (4)
  1. [Sec. 4.2, Appx. A.3, Eq. (33)] The max-test stopping rule is the component that converts neural training curves into point estimates with error bars, and it is used for every headline estimate in Figs. 4-7 and Tables 1-2. However, the formal justification in Appx. A.3 proves only that the expected test MI is bounded above by the expected training MI, and this only under the assumption that the trained critic is close to the globally optimal critic. That inequality does not establish that the training MI evaluated at the checkpoint selected by the argmax of a noisy test curve is close to the true MI; it is compatible with selection bias in either direction. The appendix itself labels the proof 'loose' and then falls back on empirical observation. Given the central role of this rule, the authors should either provide a sharper bound that accounts for selection over epochs or an explicit calibration/ablation study showing that the reported training value at the peak-test checkpoint is unbiased (or at least consistently closer to the truth than the test value) across a broader family of distributions.
  2. [Abstract, Sec. 4.3] The abstract states that the protocol is validated on CIFAR-10/100 with a ResNet-20 backbone, but the full text contains no CIFAR experiments, no ResNet-20 results, and no tables or figures reporting CIFAR numbers. This is a load-bearing discrepancy because the abstract's claim of reliable MI detection 'well below the ambient pixel dimension on real images' rests on that validation. The authors should either add the missing experiments or remove the CIFAR-10/100 claim from the abstract and any summary statements.
  3. [Appx. A.5, Eqs. (36)-(39)] The claim that sample complexity is 'grounded theoretically via random matrix theory' is stronger than what the appendix delivers. Equation (38) and the quadratic scaling in Eq. (39) are derived from a spiked-covariance detection threshold for linear Gaussian latent-variable models, and the derivation explicitly requires conditions ('if K_Z >> 1 and rank v << 2K_Z') that the authors state are 'neither strictly true in our model.' The extension to nonlinear teacher networks and to neural critics is asserted rather than derived, and the appendix acknowledges that the bound is optimistic because it ignores the cost of learning the nonlinear embedding. As written, this is a heuristic analogy, not a proof. The authors should either present the RMT analysis as a heuristic that motivates the empirical scaling, or prove a formal sample-complexity result for the nonlinear latent-variable setting.
  4. [Sec. 4.3, Appx. A.4, Figs. 6-7, Tables 1-2] The paper's claim to be 'the only approach to report confidence intervals and flag unreliable estimates' is not backed by any calibration study. The reported intervals in Figs. 6-7 and Tables 1-2 are prediction intervals from weighted least squares fits and subset standard deviations, but there is no experiment measuring coverage, i.e., the fraction of trials in which the reported interval contains the true MI. Similarly, the unreliability thresholds (delta > 0.1 and gamma_max <= 5) are introduced in Appx. A.4 without sensitivity analysis. Because the practical value of the protocol depends on these intervals and flags being meaningful, the authors should add a coverage analysis on synthetic data where the true MI is known, and should test how the estimates and flags vary with the choice of delta and the gamma cutoff.
minor comments (5)
  1. [Abstract and Sec. 1] There are typographical errors, e.g., 'estimators do not provide' should be 'estimators do not provide' with subject-verb agreement; a careful proofread is needed.
  2. [Fig. 5 caption] The vertical lines N*_Z and N* are defined only in Appx. A.5; the caption should give a one-sentence definition so the figure is self-contained.
  3. [Fig. 3 caption] The phrase 'we do not show the negative values' is ambiguous: it should state whether the displayed curves were clipped at zero or whether negative values simply fall outside the plotted range.
  4. [Notation throughout] The symbol k_Z is used for the critic's embedding dimension, while K_Z is the true latent dimension; the distinction is important and should be stated consistently in every figure caption where both appear.
  5. [Tables 2-3] The grey rows that mark unreliable fits may not be visually distinguishable in all rendering environments; consider adding an explicit symbol or column so the flag is readable regardless of color/greyscale reproduction.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction found; minor self-citation for VSIB origin is not load-bearing.

full rationale

The paper's central claim — that reliable neural MI estimation in high dimensions is governed by the latent dimensionality K_Z rather than the ambient dimension — is supported by controlled synthetic experiments with known ground truth and by external benchmarks (Czyz et al., noisy MNIST, CIFAR). The RMT-based sample-complexity thresholds in Appendix A.5 are derived from a stated latent Gaussian model and then compared with the empirical onset of nonzero estimates in Fig. 5; they are not fitted to the target values. The max-test stopping rule of Sec. 4.2 is a heuristic whose formal justification in Appendix A.3 explicitly assumes T*_train is close to the globally optimal critic and proves only an exchangeability inequality (Eq. 33). This is an exposed, unproven premise — a correctness risk — but not a circularity, because the reported training MI at the selected checkpoint is not defined in terms of the true MI. The VSIB probabilistic critic is imported from the authors' prior work (Abdelaleem et al., 2025), and the abstract calls it 'new', which is a minor self-citation/attribution concern; however, the empirical demonstration of reduced bias and variance is carried out in this paper against known ground truth, so the central estimation claim does not reduce to that citation. No invoked uniqueness theorem or hidden ansatz makes the output equivalent to the input by construction. Therefore no circular step is exhibited; the score reflects only the minor, non-load-bearing self-citation regarding VSIB's origin.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a series of hand-chosen hyperparameters and two heuristics: the RMT sample-complexity analogy and the max-test stopping rule. No new physical or mathematical entities are introduced. The VSIB critic is a method imported from the authors' own prior work, not a new ontology.

free parameters (7)
  • beta (VSIB weighting) = 512
    In Eq. 24, beta controls the trade-off between encoder regularization and estimated MI; the paper states beta=512 was used and that other large values yielded similar results (not shown). Chosen by hand, not fitted.
  • SMILE clipping threshold tau = 5
    Fixed clipping factor for SMILE (Sec. 4.1, Appx Eq. 8). This affects the bias-variance trade-off at high MI.
  • WLS linearity threshold delta = 0.1
    In workflow step 6 (Appx A.4), the relative quadratic coefficient delta=|a2/a1| greater than 0.1 triggers pruning of large gamma values. Threshold chosen by hand.
  • Unreliable gamma cutoff = 5
    If pruning reaches gamma <= 5, the estimator reports unreliable (Appx A.4). Chosen by hand.
  • Early-stopping patience = 50 epochs
    Appx A.6: training stops if test MI does not improve for 50 consecutive epochs. Chosen by hand.
  • Smoothing window sizes = 40 steps (Figs 1-2), 5 steps (other figures)
    Appx A.6: median filter window sizes for stopping curves. Heuristic.
  • Batch size = 128
    All estimators trained with batch size 128; InfoNCE saturates at log(batch size). This is a fixed hyperparameter affecting the regime.
assumptions (5)
  • domain assumption The data distribution admits an accurate low-dimensional latent representation with K_Z much less than K.
    Sec. 4.2 and the discussion state this is a required condition for reliable estimation. The synthetic data are constructed with known K_Z=10; for real images this is assumed but unknown.
  • standard math The DV representation and variational bounds (MINE, SMILE, InfoNCE) provide valid lower bounds on MI, and the critic family is expressive enough to approach the optimum.
    Used throughout Appx A.1.1 and Sec. 3. This is standard, but the expressivity condition is an assumption.
  • ad hoc to paper The spiked covariance detection threshold from random matrix theory predicts the sample size at which neural MI estimation becomes possible in nonlinear latent-variable models.
    Appx A.5 applies a linear Gaussian spike-detection calculation (Eqs. 36-39) to nonlinear teacher-network data. The paper itself calls the bound optimistic and says it ignores interactions, so it is a heuristic assumption rather than a proven theorem.
  • ad hoc to paper Test MI is expected to underestimate true MI, so the training MI evaluated at the checkpoint with peak test MI is the appropriate reported estimate.
    The max-test heuristic in Appx A.3 relies on this assumption. The provided proof assumes T*_train is near the globally optimal critic and that train/test sets are statistically similar; the paper acknowledges it cannot give general bounds without strong assumptions.
  • standard math Data processing inequality gives I(Z_X;Z_Y) <= I(X;Y), so estimating MI in the latent space is a valid lower bound.
    Underlies the VSIB objective in Eq. 24 and the statement that compressive embeddings are essential.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accurate Estimation of Mutual Information in High Dimensional Data." pith.science (2026). https://pith.science/paper/JJ5Z55QU

@misc{pith2026250600330,
  author       = {Pith},
  title        = {Pith review of: Accurate Estimation of Mutual Information in High Dimensional Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JJ5Z55QU}},
  note         = {Machine review of arXiv:2506.00330}
}
abstract

Mutual information (MI) quantifies statistical dependence between variables and is widely used across scientific disciplines, yet accurate estimation from finite data remains notoriously difficult. Common approaches fail in high-dimensional, undersampled regimes ($N \lesssim K$) typical of modern experiments, and no accepted tests exist to detect when neural network-based estimators fail, making them effectively unusable as scientific instruments. We show that neural MI estimators can be made reliable when the statistical dependencies admit a low-dimensional latent representation. Sample complexity is then governed by the latent dimensionality $K_Z \ll K$ rather than the ambient dimension -- a regime shift we confirm empirically and ground theoretically via random matrix theory. Building on this insight, we develop a practical protocol that provides neural estimators with explicit statistical consistency checks, bias correction, and confidence intervals. We additionally introduce a new class of probabilistic critics (the VSIB family) that substantially reduce bias and variance at higher MI values where standard estimators break down. We validate the protocol on synthetic benchmarks ($K=500$, $N$ as low as $256$), on the standard 40-dataset benchmark suite of Czyz et al. (2023), on noisy MNIST ($K=784$), and on CIFAR-10/100 ($K=3072$) with a ResNet-20 backbone. Our protocol consistently matches or exceeds existing methods while being the only approach to report confidence intervals and flag unreliable estimates, achieving reliable MI detection well below the ambient pixel dimension on real images.

Figures

Figures reproduced from arXiv: 2506.00330 by the authors.

Figure 1
Figure 1. MI estimators in the low-dimensional, infinite-data regime. Each panel plots running MI estimates over training iterations for five true MI levels (increasing every 4000 iterations). Each step introduces a fresh batch of 128 samples. We compare the CCA-based estimator (optimal for Gaussian data), InfoNCE, SMILE, and their probabilistic variants (denoted with VSIB). Faint curves show raw estimates; bold curves show s… view at source ↗
Figure 2
Figure 2. MI estimators in the high-dimensional, infinite-data regime. We extend [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The stopping heuristic. We evaluate neural MI esti￾mators for finite-data using the teacher model from [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: MI vs. sample size for low and high information. We compare InfoNCE, SMILE, and VSIB versions with the max￾test stopping for different sample sizes. Data from the frozen teacher model (10 latent, 500 data dimensions). All estimators use separable critics, kz = 32. Mean…
Figure 5
Figure 5. Figure 5: shows results for InfoNCE; not shown estimators behave similarly. When the latent space is small (KZ = 10, left), a critic with kZ ≥ KZ captures all dependencies once N ≫ KZ, and the estimate approaches the truth. With a moderate latent dimension (KZ = 100, center), se…
Figure 6
Figure 6. Figure 6: Workflow for MI estimation. We apply InfoNCE with a separable critic to data from a random-teacher model (KX = KY = 500, KZ = 10, true MI 4 bits). (left) Undersampled, N = 256. (right) Well-sampled, N = 214 = 16,384. For each panel, MI is computed on γ equal random, no…
Figure 7
Figure 7. Figure 7: MI estimation on noisy MNIST. From 16, 384 samples, our ap￾proach reliably estimates MI in this KX = KY = 784-dimensional dataset. For this, we show in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Comparison of MI estimators in the low-dimensional, infinite-data regime. We replicate the setup of [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Failure of the KSG estimator in high dimensions. Estimation of I(X;Z) with KSG (Kraskov et al., 2004) on K = 500-dimensional teacher data with latent dimension KZ = 10 and ground-truth MI of 4 bits. Left: undersampled (N = 256) regime; Right: well-sampled (N = 214). Fo…
Figure 10
Figure 10. Figure 10: Effect of continued training on MI estimation in the low-dimensional setting. Using the SMILE estimator (with a deterministic critic), we replicate the experiment from [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Sample pairs from the Noisy MNIST dataset. Each pair (X, Y ) shares the same digit class label but consists of distinct, non-overlapping digit instances. The X image is generated by applying a random rotation (uniform between 0 and π/2) and a random scaling (uniform b…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 57 canonical work pages

  1. [1]

    Michael Martini, and Ilya Nemenman

    Eslam Abdelaleem, Ahmed Roman, K. Michael Martini, and Ilya Nemenman. Simultaneous dimensionality reduction: A data efficient approach for multimodal representations learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=Ni14fXbyTV. arXiv preprint arXiv:2310.04458

    arXiv 2024

  2. [2]

    Deep variational multivariate information bottleneck-a framework for variational losses

    Eslam Abdelaleem, Ilya Nemenman, and K Michael Martini. Deep variational multivariate information bottleneck-a framework for variational losses. Journal of Machine Learning Research, 26 0 (140): 0 1--50, 2025

    work page 2025

  3. [3]

    Estimation of mutual information by the fuzzy histogram

    Maryam Amir Haeri and Mohammad Mehdi Ebadzadeh. Estimation of mutual information by the fuzzy histogram. Fuzzy Optimization and Decision Making, 13: 0 287--318, 2014

    work page 2014

  4. [4]

    Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices

    Jinho Baik, Gerard Ben Arous , and Sadrine Peche. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann.\ Probab., 33 0 (5): 0 1643--1697, 2005

    work page 2005

  5. [5]

    Information maximization in noisy channels: A variational approach

    David Barber and Felix Agakov. Information maximization in noisy channels: A variational approach. Adv.\ Neural Inf.\ Proc.\ Syst., 16, 2003

    work page 2003

  6. [6]

    The IM algorithm: A variational approach to information maximization

    David Barber and Felix Agakov. The IM algorithm: A variational approach to information maximization. Adv.\ Neural Inf.\ Proc.\ Syst., 16 0 (320): 0 201, 2004

    work page 2004

  7. [7]

    Mutual information neural estimation

    Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mutual information neural estimation . In 35th International Conference on Machine Learning, ICML 2018, volume 2, pp.\ 864--873, 2018. ISBN 9781510867963

    work page 2018

  8. [8]

    Large dimension forecasting models and random singular value spectra

    J-P Bouchaud, Laurent Laloux, M Augusta Miceli, and Marc Potters. Large dimension forecasting models and random singular value spectra. The European Physical Journal B, 55: 0 201--207, 2007

    work page 2007

Show all 73 references
  1. [9]

    Cca-zoo: A collection of regularized, deep learning based, kernel, and probabilistic cca methods in a scikit-learn style framework

    James Chapman and Hao-Ting Wang. Cca-zoo: A collection of regularized, deep learning based, kernel, and probabilistic cca methods in a scikit-learn style framework. Journal of Open Source Software, 6 0 (68): 0 3823, 2021

    2021

  2. [10]

    Regularized mutual information neural estimation

    Kwanghee Choi and Siyeong Lee. Regularized mutual information neural estimation. OpenReview, 2020

    2020

  3. [11]

    Beyond normal: On the evaluation of mutual information estimators

    Pawel Czyz, Frederic Grabowski, Julia Vogt, Niko Beerenwinkel, and Alexander Marx. Beyond normal: On the evaluation of mutual information estimators. Advances in Neural Information Processing Systems, 36: 0 16957--16990, 2023

    2023

  4. [12]

    Estimating mutual information using b-spline functions--an improved similarity measure for analysing gene expression data

    Carsten O Daub, Ralf Steuer, Joachim Selbig, and Sebastian Kloska. Estimating mutual information using b-spline functions--an improved similarity measure for analysing gene expression data. BMC bioinformatics, 5: 0 1--12, 2004

    2004

  5. [13]

    Asymptotic evaluation of certain markov process expectations for large time

    Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on pure and applied mathematics, 36 0 (2): 0 183--212, 1983

    1983

  6. [14]

    Multimodal and multiband image registration using mutual information

    Rik Fransens, Christoph Strecha, and LV Gool. Multimodal and multiband image registration using mutual information. Proc. Theory and Applications of Knowledge-Driven Image Information Mining with Focus on Earth Observation (ESA-EUSC), 2004

    2004

  7. [15]

    Multivariate information bottleneck

    Nir Friedman, Ori Mosenzon, Noam Slonim, and Naftali Tishby. Multivariate information bottleneck. arXiv preprint arXiv:1301.2270, 2013

    2013 arXiv

  8. [16]

    Efficient estimation of mutual information for strongly dependent variables

    Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Artificial intelligence and statistics, pp.\ 277--286. PMLR, 2015

    2015

  9. [17]

    Calculation of the amount of information about a random function contained in another such function

    Izrail Moiseevich Gelfand and AM Iaglom. Calculation of the amount of information about a random function contained in another such function. American Mathematical Society Providence, RI, 1959

    1959

  10. [18]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.\ 249--256. JMLR Workshop and Conference Proceedings, 2010

    2010

  11. [19]

    Sliced mutual information: A scalable measure of statistical dependence

    Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: A scalable measure of statistical dependence. Advances in Neural Information Processing Systems, 34: 0 17567--17578, 2021

    2021

  12. [20]

    Approximating mutual information of high-dimensional variables using learned representations

    Gokul Gowri, Xiaokang Lun, Allon Klein, and Peng Yin. Approximating mutual information of high-dimensional variables using learned representations. Advances in Neural Information Processing Systems, 37: 0 132843--132875, 2024

    2024

  13. [21]

    Tight mutual information estimation with contrastive fenchel-legendre optimization

    Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Jing Huang, Larry Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. Advances in Neural Information Processing Systems, 35: 0 28319--28334, 2022

    2022

  14. [22]

    Functional connectivity in resting-state fmri: is linear correlation sufficient? NeuroImage, 54 0 (3): 0 2218--2225, 2011

    Jaroslav Hlinka, Milan Palu s , Martin Vejmelka, Dante Mantini, and Maurizio Corbetta. Functional connectivity in resting-state fmri: is linear correlation sufficient? NeuroImage, 54 0 (3): 0 2218--2225, 2011

    2011

  15. [23]

    Estimation of mutual information for real-valued data with error bars and controlled bias

    Caroline M Holmes and Ilya Nemenman. Estimation of mutual information for real-valued data with error bars and controlled bias. Phys.\ Rev.\ E, 100 0 (2): 0 022404, 2019

    2019

  16. [24]

    On impossibility of learning in a reparameterization covariant way

    Timothy Holy and Ilya Nemenman. On impossibility of learning in a reparameterization covariant way. Technical report, Tech.\ Rep.\ NSF-KITP-03-123, KITP, UCSB, 2002

    2002

  17. [25]

    Relations between two sets of variates

    Harold Hotelling. Relations between two sets of variates. Biometrika, 1936. doi:10.1007/978-1-4612-4380-9_14

    1936 doi

  18. [26]

    On the distribution of the information density of gaussian random vectors: Explicit formulas and tight approximations

    Jonathan EW Huffmann and Martin Mittelbach. On the distribution of the information density of gaussian random vectors: Explicit formulas and tight approximations. Entropy, 24 0 (7): 0 924, 2022

    2022

  19. [27]

    A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula

    Robin AA Ince, Bruno L Giordano, Christoph Kayser, Guillaume A Rousselet, Joachim Gross, and Philippe G Schyns. A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula. Human brain mapping, 38 0 (3): 0 1541--1573, 2017

    2017

  20. [28]

    Nonparametric von mises estimators for entropies, divergences and mutual informations

    Kirthevasan Kandasamy, Akshay Krishnamurthy, Barnabas Poczos, Larry Wasserman, et al. Nonparametric von mises estimators for entropies, divergences and mutual informations. Advances in Neural Information Processing Systems, 28, 2015

    2015

  21. [29]

    Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data

    Shiraj Khan, Sharba Bandyopadhyay, Auroop R Ganguly, Sunil Saigal, David J Erickson III, Vladimir Protopopescu, and George Ostrouchov. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Physical Review E—Sta...

    2007

  22. [30]

    Estimating mutual information

    Alexander Kraskov, Harald St \"o gbauer, and Peter Grassberger. Estimating mutual information. Phys.\ Rev.\ E, 69 0 (6): 0 066138, 2004

    2004

  23. [31]

    Information theory and statistics

    S Kullback. Information theory and statistics. John Wiley & Sons, New York, NY, 1959

    1959

  24. [32]

    Information theory and statistics

    Solomon Kullback. Information theory and statistics. Courier Corporation, 1997

    1997

  25. [33]

    Gradient-based learning applied to document recognition

    Yann LeCun, L \'e on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc.\ IEEE, 86 0 (11): 0 2278--2324, 1998

    1998

  26. [34]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 0 (7553): 0 436--444, 2015

    2015

  27. [35]

    Functional connectivity inference from fmri data using multivariate information measures

    Qiang Li. Functional connectivity inference from fmri data using multivariate information measures. Neural Networks, 146: 0 85--97, 2022

    2022

  28. [36]

    Data-efficient mutual information neural estimator

    Xiao Lin, Indranil Sur, Samuel A Nastase, Ajay Divakaran, Uri Hasson, and Mohamed R Amer. Data-efficient mutual information neural estimator. arXiv preprint arXiv:1905.03319, 2019

    1905 arXiv

  29. [37]

    Self-supervised contrastive learning of protein representations by mutual information maximization

    Amy X Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan Moses. Self-supervised contrastive learning of protein representations by mutual information maximization. BioRxiv, pp.\ 2020--09, 2020

    2020

  30. [38]

    Aracne: an algorithm for the reconstruction of gene regulatory networks in a mammalian cellular context

    Adam A Margolin, Ilya Nemenman, Katia Basso, Chris Wiggins, Gustavo Stolovitzky, Riccardo Dalla Favera, and Andrea Califano. Aracne: an algorithm for the reconstruction of gene regulatory networks in a mammalian cellular context. BMC bioinformatics, 7: 0 S7, 2006

    2006

  31. [39]

    Protein 3d structure computed from evolutionary sequence variation

    Debora S Marks, Lucy J Colwell, Robert Sheridan, Thomas A Hopf, Andrea Pagnani, Riccardo Zecchina, and Chris Sander. Protein 3d structure computed from evolutionary sequence variation. PloS one, 6 0 (12): 0 e28766, 2011

    2011

  32. [40]

    Estimating mutual information via geodesic k-nn

    Alexander Marx and Jonas Fischer. Estimating mutual information via geodesic k-nn. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pp.\ 415--423. SIAM, 2022

    2022

  33. [41]

    Estimation of mutual information using kernel density estimators

    Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52 0 (3): 0 2318, 1995

    1995

  34. [42]

    Entropy and inference, revisited

    Ilya Nemenman, Fariel Shafee, and William Bialek. Entropy and inference, revisited. Advances in neural information processing systems, 14, 2001

    2001

  35. [43]

    Entropy and information in neural spike trains: Progress on the sampling problem

    Ilya Nemenman, William Bialek, and Rob de Ruyter van Steveninck. Entropy and information in neural spike trains: Progress on the sampling problem. Phys.\ Rev.\ E, 69 0 (5): 0 056111, 2004

    2004

  36. [44]

    Estimating divergence functionals and the likelihood ratio by convex risk minimization

    XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans.\ Information Theory, 56 0 (11): 0 5847--5861, 2010

    2010

  37. [45]

    Predictive information in a sensory population

    Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in a sensory population. Proc.\ Natl.\ Acad.\ Sci., 112 0 (22): 0 6908--6913, 2015

    2015

  38. [46]

    Estimation of entropy and mutual information

    Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 15 0 (6): 0 1191--1253, 2003

    2003

  39. [47]

    The role of spike timing in the coding of stimulus location in rat somatosensory cortex

    Stefano Panzeri, Rasmus S Petersen, Simon R Schultz, Michael Lebedev, and Mathew E Diamond. The role of spike timing in the coding of stimulus location in rat somatosensory cortex. Neuron, 29 0 (3): 0 769--777, 2001

    2001

  40. [48]

    Millisecond-scale motor coding precedes sensorimotor learning in songbirds

    Leila May M Pascual, Aanya Vusirikala, Ilya M Nemenman, Samuel J Sober, and Michael Pasek. Millisecond-scale motor coding precedes sensorimotor learning in songbirds. bioRxiv, 2024

    2024

  41. [49]

    Quantifying how much sensory information in a neural code is relevant for behavior

    Giuseppe Pica, Eugenio Piasini, Houman Safaai, Caroline Runyan, Christopher Harvey, Mathew Diamond, Christoph Kayser, Tommaso Fellin, and Stefano Panzeri. Quantifying how much sensory information in a neural code is relevant for behavior. Advances in Neural Information Process...

    2017

  42. [50]

    On variational bounds of mutual information

    Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Int.\ Conf.\ Machine Learning, pp.\ 5171--5180. PMLR, 2019

    2019

  43. [51]

    A first course in random matrix theory: for physicists, engineers and data scientists

    Marc Potters and Jean-Philippe Bouchaud. A first course in random matrix theory: for physicists, engineers and data scientists. Cambridge University Press, 2020

    2020

  44. [52]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

    2021

  45. [53]

    Mutual information between discrete and continuous data sets

    Brian C Ross. Mutual information between discrete and continuous data sets. PloS one, 9 0 (2): 0 e87357, 2014

    2014

  46. [54]

    Distinct timescales of population coding across cortex

    Caroline A Runyan, Eugenio Piasini, Stefano Panzeri, and Christopher D Harvey. Distinct timescales of population coding across cortex. Nature, 548 0 (7665): 0 92--96, 2017

    2017

  47. [55]

    Generative power of a protein language model trained on multiple sequence alignments

    Damiano Sgarbossa, Umberto Lupo, and Anne-Florence Bitbol. Generative power of a protein language model trained on multiple sequence alignments. Elife, 12: 0 e79854, 2023

    2023

  48. [56]

    A mathematical theory of communication

    Claude Elwood Shannon. A mathematical theory of communication. Bell Syst.\ Techn.\ J., 27 0 (3): 0 379--423, 1948

    1948

  49. [57]

    Understanding the limitations of variational mutual information estimators

    Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv:1910.06222, 2019

    1910 arXiv

  50. [58]

    The mutual information: detecting and evaluating dependencies between variables

    Ralf Steuer, J \"u rgen Kurths, Carsten O Daub, Janko Weise, and Joachim Selbig. The mutual information: detecting and evaluating dependencies between variables. Bioinformatics, 18 0 (suppl\_2): 0 S231--S240, 2002

    2002

  51. [59]

    Entropy and information in neural spike trains

    Steven P Strong, Roland Koberle, Rob R De Ruyter Van Steveninck, and William Bialek. Entropy and information in neural spike trains. Physical review letters, 80 0 (1): 0 197, 1998

    1998

  52. [60]

    Distribution of singular values in large sample cross-covariance matrices

    Arabind Swain, Sean Alexander Ridout, and Ilya Nemenman. Distribution of singular values in large sample cross-covariance matrices. arXiv preprint arXiv:2502.05254, 2025

    2025 arXiv

  53. [61]

    Millisecond-scale motor encoding in a cortical vocal area

    Claire Tang, Diala Chehayeb, Kyle Srivastava, Ilya Nemenman, and Samuel J Sober. Millisecond-scale motor encoding in a cortical vocal area. PLoS Biol., 12 0 (12): 0 e1002018, 2014

    2014

  54. [62]

    Input variable selection: mutual information and linear mixing measures

    Thomas Trappenberg, Jie Ouyang, and Andrew Back. Input variable selection: mutual information and linear mixing measures. IEEE Transactions on Knowledge and Data Engineering, 18 0 (1): 0 37--46, 2005

    2005

  55. [63]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  56. [64]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  57. [65]

    Alignment by maximization of mutual information

    Paul Viola and William M Wells III. Alignment by maximization of mutual information. International journal of computer vision, 24 0 (2): 0 137--154, 1997

    1997

  58. [66]

    Estimation of mutual information: A survey

    Janett Walters-Williams and Yan Li. Estimation of mutual information: A survey. In Rough Sets and Knowledge Technology: 4th International Conference, RSKT 2009, Gold Coast, Australia, July 14-16, 2009. Proceedings 4, pp.\ 389--396. Springer, 2009

    2009

  59. [67]

    On deep multi-view representation learning

    Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning, pp.\ 1083--1092. PMLR, 2015

    2015

  60. [68]

    Deep variational canonical correlation analysis

    Weiran Wang, Xinchen Yan2 Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXiv preprint arXiv:1610.03454, 2016

    2016 arXiv

  61. [69]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St \'e phane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp.\ 12310--12320. PMLR, 2021

    2021

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  63. [71]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  64. [72]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  65. [73]

    >9@:2҃Ƚ풥2^ k х @G [) GHY^Z5ln4Nx6 Ҏ @nk^Yʟg

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.