REVIEW 20 cited by
Estimating Mutual Information
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Estimating Mutual Information
read the original abstract
We present two classes of improved estimators for mutual information $M(X,Y)$, from samples of random points distributed according to some joint probability density $\mu(x,y)$. In contrast to conventional estimators based on binnings, they are based on entropy estimates from $k$-nearest neighbour distances. This means that they are data efficient (with $k=1$ we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to non-uniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of $k/N$ for $N$ points. Numerically, we find that both families become {\it exact} for independent distributions, i.e. the estimator $\hat M(X,Y)$ vanishes (up to statistical fluctuations) if $\mu(x,y) = \mu(x) \mu(y)$. This holds for all tested marginal distributions and for all dimensions of $x$ and $y$. In addition, we give estimators for redundancies between more than 2 random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.
Forward citations
Cited by 20 Pith papers
-
The Accountability Horizon: An Impossibility Theorem for Governing Human-Agent Collectives
Above a computable autonomy threshold with mixed human–AI feedback cycles, no single-locus accountability framework can jointly satisfy causal attribution, foreseeability, non-vacuity, and full allocation.
-
Same Signal, Different Story: Demystifying Receiver Effects in Wi-Fi Channel State Information
Receiver-specific CSI differences in Wi-Fi sensing stem primarily from AGC and subcarrier nonlinearities and are largely mitigated by gain alignment, restoring most cross-device model accuracy.
-
Task Relevance Is Not Local Replaceability: A Two-Axis View of Channel Information
Channel importance splits into task relevance and local replaceability; local-axis metrics predict safe removal under pruning better than target-axis metrics across multiple CNNs and datasets.
-
Mixed-Precision Information Bottlenecks for On-Device Trait-State Disentanglement in Bipolar Agitation Detection
MP-IB uses an 8x information asymmetry via FP16 trait heads and INT4 state heads to disentangle speaker identity from agitation in voice biomarkers, outperforming larger models on edge devices with low latency and sup...
-
Quantum Causal Discovery via Amplitude Estimation of Kullback-Leibler Divergence
QKLA achieves quadratic query-complexity improvement for clipped KL estimation, yielding 2.7-7.4x fewer oracle queries than classical methods when embedded in the PC causal-discovery algorithm at moderate precision.
-
The Accountability Horizon: An Impossibility Theorem for Governing Human-Agent Collectives
The Accountability Incompleteness Theorem demonstrates that human-AI collectives above the Accountability Horizon with feedback cycles cannot simultaneously meet attributability, foreseeability, non-vacuity, and compl...
-
DegenDetector: Symbolic Recovery of Parameter Degeneracies in Bayesian Posteriors
A pipeline combining mutual information screening with alternating symbolic regression recovers closed-form degeneracy equations from posterior samples.
-
A Self-Supervised Approach for Minimal-Annotation Hydroacoustic Data Exploration
Event-level MAE embeddings plus UMAP/HDBSCAN or K-Means clustering recover 15 hydroacoustic classes from multi-year Mayotte data with ~1 hour of annotation and detector-comparable F1.
-
Information-Theoretic Reliability is Robust to Analytic Choice: A 24-Specification Multiverse on Public Cognitive Test-Retest Data
NLRΔ applied to 50 primary measures from five cognitive task families on public test-retest data yields median -0.138 nats with zero cells passing the headline rule across a 24-specification multiverse.
-
Hidden Coalitions in Multi-Agent AI: A Spectral Diagnostic from Internal Representations
Spectral partitioning on pairwise mutual-information graphs from agent hidden states detects representational coalitions that behavioral measures miss in multi-agent AI.
-
Efficient characterization of general Gottesman-Kitaev-Preskill qubits
A family of positive semidefinite operators is introduced that witnesses arbitrary logical GKP qubit states and enables their efficient characterization via three quadrature measurements.
-
Efficient characterization of general Gottesman-Kitaev-Preskill qubits
A new set of positive semidefinite operators for GKP qubits gives logical infidelity via three quadrature measurements and yields finite-dimensional approximations as ground states.
-
Feature Selection via Mutual Information: New Theoretical Insights
Conditional mutual information bounds ideal prediction errors for feature subsets and supplies a stopping condition for greedy selection algorithms.
-
Investigating causality between principal components in protein dynamics
Applies causal inference to PCs from MD trajectories of two proteins to construct directed influence networks complementary to PCA and TICA.
-
Inferring stellar metallicity and elemental abundances from kinematic and spectroscopic data using machine learning -- Implications for exoplanet host stars
ML regressors trained on APOGEE DR17 red giants predict C, O, Mg, Si abundances from kinematics and [Fe/H] more accurately than [Fe/H] baseline, with external validation on HARPS FGK dwarfs and reproduction of Galacti...
-
Efficient Benchmarking Is Just Feature Selection and Multiple Regression
Kernel ridge regression combined with mRMR feature selection improves prediction of full benchmark scores from question subsets over existing efficient benchmarking techniques.
-
Data-Driven Reduction of Fault Location Errors in Onshore Wind Farm Collectors
A Gated Residual Network correction model reduces fault location error by 76% in simulated onshore wind farm collector networks compared to state-of-the-art methods.
-
Information-Theoretic Measures in AI: A Practical Decision Framework
A practical guide that organizes seven IT measures around three questions each—what it answers in AI, suitable estimators, and dangerous misuses—complete with flowchart, table, and worked examples.
-
Information-Theoretic Measures in AI: A Practical Decision Framework
The paper is a review that packages existing IT-estimator guidance into a seven-measure decision framework; a claimed validation case study is absent from the full text.
-
Information-Theoretic Measures in AI: A Practical Decision Framework
A synthesis paper offering a practical decision guide, flowchart, and table for choosing among seven established information-theoretic measures in AI and agent applications.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.