Nonnegative Decomposition of Multivariate Information
read the original abstract
Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.
This paper has not been read by Pith yet.
Forward citations
Cited by 20 Pith papers
-
Information-theoretic signatures of causality in Bayesian networks and hypergraphs
Partial information decomposition components map directly onto causal roles such as direct parents, children, and colliders in both pairwise Bayesian networks and higher-order hypergraphs.
-
Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures
The authors establish an impossibility theorem for lattice-based PID beyond three sources and introduce alternative measures of multivariate unique and synergistic information using dependency-eliminating random varia...
-
Explicit Formula for Partial Information Decomposition
Provides the first explicit formula for partial information decomposition atoms satisfying Williams and Beer's axioms via a novel do-operation.
-
Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
Introduces Synergistic Faithfulness metric based on Shapley Interaction Index to evaluate cross-modal synergy in VLM explainers, revealing over-reliance on visual salience in existing methods.
-
Closed-Form Gaussian Estimators for Multi-Source Partial Information Decomposition
Closed-form log-determinant expressions provide the first covariance-based estimators for multi-source PID quantities including redundancy, unique information, and synergy in Gaussian variables.
-
Task Relevance Is Not Local Replaceability: A Two-Axis View of Channel Information
Channel importance splits into task relevance and local replaceability; local-axis metrics predict safe removal under pruning better than target-axis metrics across multiple CNNs and datasets.
-
Quantifying Spacetime Integration across a Partition with Synergy
Synergy-based measures of spacetime integration outperform current IIT practice when tested on simple deterministic networks.
-
Quantifying Spacetime Integration across a Partition with Synergy
Synergy-based measures from partial information decomposition are found more suitable than current practice for quantifying integration in simple deterministic networks for the Information Integration Theory of Consciousness.
-
Structural Impossibility of Antichain-Lattice Partial Information Decomposition
Antichain-lattice indexing in PID is structurally insufficient to recover mutual information from information atoms for multivariate cases.
-
Emergent Coordination in Multi-Agent Language Models
Multi-agent LLM systems can be steered via prompt design from mere aggregates to higher-order collectives with identity-linked differentiation and goal-directed complementarity, as measured by partial information deco...
-
Emergence of information interference in stochastic systems with non-diagonal noise and switching environments
In stochastic systems with non-diagonal noise and switching environments, mutual information includes irreducible static and dynamic interference terms that prevent simple decomposition.
-
Partial Effective Information Decomposition for Synergistic Causality
PEID decomposes the causal effect of multiple sources on a target under maximum-entropy interventions into unique and synergistic information, enabling hyperedge causal graphs and downward causation analysis.
-
Quantifying Spacetime Integration across a Partition with Synergy
Introduces four synergy-based measures of spacetime integration from partial information decomposition and finds them more suitable than current IIT practice for simple deterministic networks.
-
Heterophily as a generative mechanism for self-organized synergistic interdependencies
Heterophily weakens pairwise couplings while inducing geometric constraints that create synergistic higher-order interdependencies in a co-evolving spin-glass model.
-
RAG-GNN: Integrating Retrieved Knowledge with Graph Neural Networks for Precision Medicine
RAG-GNN augments GNNs with retrieved literature knowledge via gated fusion to improve functional clustering of 379 proteins in cancer signaling networks, raising silhouette score by 0.093.
-
A scalable estimator of higher-order information in complex dynamical systems
Introduces M-information as a scalable measure of higher-order information integration in multivariate time series, computed via convex optimization and tested on neuronal and neuroimaging data.
-
Exo-Daisy World: Revisiting Gaia Theory through an Informational Architecture Perspective
The Exo-Daisy World model, built from stochastic differential equations, shows biosphere-environment correlations strengthening with stellar luminosity through distinct phases of information exchange quantified as rei...
-
More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystems
AI-native software ecosystems exhibit emergent behaviors best explained by complex adaptive systems theory, requiring new ecosystem-level monitoring and seven testable propositions that may extend or replace Lehman's laws.
-
ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations
ConceptTracer supplies an interactive interface and saliency/selectivity metrics to locate concept-responsive neurons in neural representations, shown on TabPFN.
-
PrismNet: Viewing Time Series Through a Multi-Modal Prism for Interpretable Power Load Forecasting
PrismNet combines text and image modalities with time series via a PID-guided contrastive learning module to boost few-shot power load forecasting accuracy and provide interpretability.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.