pith. sign in

arxiv: 2508.05530 · v2 · submitted 2025-08-07 · 💻 cs.IT · math.IT

Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures

Aobo Lyu , Andrew Clark , Netanel Raviv This is my paper

Pith reviewed 2026-05-19 00:22 UTC · model grok-4.3

classification 💻 cs.IT math.IT
keywords partial information decompositionmultivariate informationinformation atomsinconsistencyunique informationsynergistic informationauxiliary random variablesIsing model
0
0 comments X

The pith

No lattice-based partial information decomposition can stay consistent for three or more sources.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proves that any attempt to decompose mutual information into redundant, unique, and synergistic atoms via a lattice structure must fail once three or more sources are involved. It generalizes a known three-variable counterexample into a full impossibility theorem that covers every larger collection of sources. For exactly two sources it supplies closed-form expressions that meet every listed axiom. It then constructs alternative measures of unique and synergistic information that avoid lattices altogether by introducing auxiliary random variables designed to cancel higher-order dependencies.

Core claim

Lattice-based partial information decomposition cannot be made consistent for three or more sources because no assignment of the information atoms satisfies the required consistency condition on every subset simultaneously; the three-source case where the sum of atoms exceeds total mutual information is extended to this general impossibility result. For two sources, explicit closed-form formulas are given for all atoms. New measures are defined via auxiliary systems of random variables that eliminate higher-order dependencies while preserving the target atoms, and these measures obey additivity and continuity.

What carries the argument

The impossibility theorem for lattice decompositions with more than two sources, together with auxiliary random variable systems that isolate unique and synergistic information by removing higher-order dependencies.

If this is right

  • Two-source partial information decomposition now has explicit formulas that satisfy the complete set of axioms.
  • Any lattice-based assignment of atoms will produce inconsistencies for four or more sources.
  • The new measures remain additive and continuous by construction.
  • Numerical checks on the Ising model confirm that the measures capture high-order relations without lattice contradictions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The auxiliary-variable construction could be tested on other models such as neural networks or gene regulatory networks to check whether it recovers expected synergistic effects.
  • A follow-up direction is to extend the same auxiliary approach to also quantify redundant information in the multivariate case.
  • The impossibility result implies that future multivariate decomposition work should start from non-lattice structures rather than trying to repair the existing lattice.

Load-bearing premise

The auxiliary random variable systems correctly remove higher-order dependencies without introducing new artifacts or losing the original information relations.

What would settle it

A concrete three-source probability distribution on which the proposed unique or synergistic measure violates additivity or returns a negative value.

Figures

Figures reproduced from arXiv: 2508.05530 by Andrew Clark, Aobo Lyu, Netanel Raviv.

Figure 1
Figure 1. Figure 1: FIG. 1: A pictorial representation of Partial Information Decomposition ( [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FIG. 2: The structure of PID with 3 source variables. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIG. 3: Construction of ( [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: FIG. 4: Temperature dependence of (A) magnetization [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FIG. 5: Temperature dependence of PID measures. [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
read the original abstract

While mutual information effectively quantifies dependence between two variables, it does not by itself reveal the complex, fine-grained interactions among variables, i.e., how multiple sources contribute redundantly, uniquely, or synergistically to a target in multivariate settings. The Partial Information Decomposition (PID) framework was introduced to address this by decomposing the mutual information between a set of source variables and a target variable into fine-grained information atoms such as redundant, unique, and synergistic components. In this work, we review the axiomatic system and desired properties of the PID framework and make three main contributions. First, we resolve the two-source PID case by providing explicit closed-form formulas for all information atoms that satisfy the full set of axioms and desirable properties. Second, we prove that for three or more sources, PID suffers from fundamental inconsistencies: we review the known three-variable counterexample where the sum of atoms exceeds the total information, and extend it to a comprehensive impossibility theorem showing that no lattice-based decomposition can be consistent for all subsets when the number of sources exceeds three. Finally, we deviate from the PID lattice approach to avoid its inconsistencies, and present explicit measures of multivariate unique and synergistic information. Our proposed measures, which rely on new systems of random variables that eliminate higher-order dependencies, satisfy key axioms such as additivity and continuity, provide a robust theoretical explanation of high-order relations, and show strong numerical performance in comprehensive experiments on the Ising model. Our findings highlight the need for a new framework for studying multivariate information decomposition.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper reviews the axiomatic foundations of Partial Information Decomposition (PID) and makes three contributions: (1) explicit closed-form expressions for all atoms in the two-source case that satisfy the full set of axioms including non-negativity, additivity, and continuity; (2) an impossibility theorem showing that no lattice-based PID can be consistent across all subsets for three or more sources, obtained by extending the known three-variable sum-exceeding-total counterexample; and (3) alternative explicit measures of multivariate unique and synergistic information constructed via auxiliary systems of random variables that are designed to remove higher-order dependencies while preserving the target atoms. These measures are shown to obey additivity and continuity, with numerical validation on the Ising model.

Significance. If the auxiliary-variable constructions are shown to isolate atoms without residual cross terms for arbitrary alphabets, the impossibility result would be a substantial clarification of the limits of lattice-based PID, while the new measures would supply a concrete, axiom-compliant alternative that avoids the lattice inconsistencies. The two-source closed forms and the Ising experiments already provide immediate practical value.

major comments (2)
  1. [Section on proposed measures] Section on proposed measures: the claim that the auxiliary random-variable systems eliminate higher-order dependencies while preserving the information atoms lacks an explicit proof that the construction commutes with the mutual-information functional for arbitrary finite alphabets; without this, it is unclear whether non-negativity and the absence of residual cross terms hold beyond the tested regimes.
  2. [Ising experiments] Ising-model experiments: the numerical checks are confined to a narrow parameter regime (binary variables, nearest-neighbor couplings); this does not yet constitute a general verification that the auxiliary construction isolates unique and synergistic parts without artifacts for higher-order dependencies.
minor comments (2)
  1. Notation for the auxiliary variables should be introduced with an explicit diagram or table showing the original joint, the auxiliary marginals, and the induced conditional independencies.
  2. The impossibility theorem statement would benefit from a short corollary clarifying which specific axioms (e.g., monotonicity or consistency with marginals) are violated by any lattice-based attempt for n≥3.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive review. The comments correctly identify areas where additional rigor and breadth would strengthen the presentation of the auxiliary-variable constructions and their validation. We have revised the manuscript accordingly and address each point below.

read point-by-point responses

  1. Referee: Section on proposed measures: the claim that the auxiliary random-variable systems eliminate higher-order dependencies while preserving the information atoms lacks an explicit proof that the construction commutes with the mutual-information functional for arbitrary finite alphabets; without this, it is unclear whether non-negativity and the absence of residual cross terms hold beyond the tested regimes.

    Authors: We agree that an explicit general proof was not stated with sufficient clarity. The auxiliary systems are defined by adjoining independent copies chosen to match the relevant marginals while nullifying higher-order interactions; by construction this isolates the target atoms. In the revised manuscript we have added a lemma proving that the resulting mutual-information expressions commute with the original joint distribution for any finite alphabet, which directly yields non-negativity and the absence of residual cross terms. The proof relies only on the chain rule and the independence properties built into the auxiliary variables. revision: yes

  2. Referee: Ising-model experiments: the numerical checks are confined to a narrow parameter regime (binary variables, nearest-neighbor couplings); this does not yet constitute a general verification that the auxiliary construction isolates unique and synergistic parts without artifacts for higher-order dependencies.

    Authors: The original experiments were deliberately restricted to the binary nearest-neighbor Ising model to permit direct comparison with known analytic results on phase transitions. We acknowledge that this regime is narrow. In the revision we have added numerical results for ternary alphabets and for models with next-nearest-neighbor couplings, confirming that the auxiliary construction continues to isolate the intended atoms without visible artifacts. A fully exhaustive verification over all alphabets and dependency structures lies outside the scope of the present work; we have noted this limitation and the need for further study in the discussion section. revision: partial

Circularity Check

0 steps flagged

No significant circularity: explicit formulas, literature extension, and independent auxiliary constructions

full rationale

The paper resolves the two-source case with explicit closed-form formulas satisfying the full axiom set, extends a known three-variable counterexample (reviewed from prior literature) to a general impossibility theorem for n≥3 sources, and defines alternative unique/synergistic measures via explicit construction of auxiliary random variable systems that remove higher-order dependencies. None of these steps reduce by construction to fitted inputs, self-definitions, or load-bearing self-citations; the auxiliary systems are introduced as new objects and validated numerically on the Ising model. The derivation remains self-contained against external benchmarks with no reduction of predictions to their own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The work rests on standard information-theoretic axioms (non-negativity, additivity, continuity) plus the assumption that auxiliary random variable systems can isolate higher-order terms without loss of the target atoms. No free parameters are introduced; the new random variable systems are invented entities whose independent evidence is the numerical performance on the Ising model.

axioms (1)
  • standard math Standard PID axioms including non-negativity of atoms, additivity over independent sources, and continuity with respect to distributions
    Invoked throughout the review of desired properties and in the verification that new measures satisfy them
invented entities (1)
  • Auxiliary systems of random variables that eliminate higher-order dependencies independent evidence
    purpose: To define unique and synergistic information without lattice inconsistencies
    Introduced to construct the new measures; independent evidence claimed via Ising model experiments

pith-pipeline@v0.9.0 · 5805 in / 1372 out tokens · 62770 ms · 2026-05-19T00:22:44.224259+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Information-theoretic signatures of causality in Bayesian networks and hypergraphs

    cs.IT 2025-12 unverdicted novelty 8.0

    Partial information decomposition components map directly onto causal roles such as direct parents, children, and colliders in both pairwise Bayesian networks and higher-order hypergraphs.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · cited by 1 Pith paper · 2 internal anchors

  1. [1]

    Consequently, they also share the same (unconditional) joint distribution

    share the same conditional joint distribution. Consequently, they also share the same (unconditional) joint distribution. Then, we conclude that forS ′ 1 andS ′ 2 as above, we have thatI(S ′ 1;S 2) =I(S 1;S ′ 2). This, combined with Lemma 3, implies the commutativity of Red. Proof of Axioms 3 (Monotonicity) and 4 (Self-redundancy): According to Def. 3, Un...

    work page

  2. [2]

    Hence, our construction ofQvia (10) achieves the maximum entropy among allQ ′ ∈∆ P

    and (T, S2) marginals, the entropy H(S ′ 1, S2, T) is maximized whenS ′ 1 andS 2 are conditionally independent givenT: Q∗(s1, s2, t) = Q∗(s1, t)Q∗(s2, t) Q∗(t) =P(s 1 |t)P(s 2 |t)P(t). Hence, our construction ofQvia (10) achieves the maximum entropy among allQ ′ ∈∆ P . Finally, since Definition 3 defines the unique information as Un(S1 →T|S 2) =I(S ′ 1;T|...

    work page

  3. [3]

    A. Lyu, A. Clark, and N. Raviv, Explicit formula for partial information decomposition, in 2024 IEEE International Symposium on Information Theory (ISIT)(IEEE, 2024) pp. 2329– 2334. 42

    work page 2024

  4. [4]

    C. E. Shannon, A mathematical theory of communication, ACM SIGMOBILE mobile com- puting and communications review5, 3 (2001)

    work page 2001

  5. [5]

    Watanabe, Information theoretical analysis of multivariate correlation, IBM Journal of research and development4, 66 (1960)

    S. Watanabe, Information theoretical analysis of multivariate correlation, IBM Journal of research and development4, 66 (1960)

    work page 1960

  6. [6]

    P. L. Williams and R. D. Beer, Nonnegative decomposition of multivariate information, arXiv preprint arXiv:1004.2515 (2010)

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  7. [7]

    Crampton and G

    J. Crampton and G. Loizou, The completion of a poset in a lattice of antichains, International Mathematical Journal1, 223 (2001)

    work page 2001

  8. [8]

    Schneidman, W

    E. Schneidman, W. Bialek, and M. J. Berry, Synergy, redundancy, and independence in pop- ulation codes, Journal of Neuroscience23, 11539 (2003)

    work page 2003

  9. [9]

    T. F. Varley, M. Pope, M. Grazia, Joshua, and O. Sporns, Partial entropy decomposition re- veals higher-order information structures in human brain activity, Proceedings of the National Academy of Sciences120, e2300888120 (2023)

    work page 2023

  10. [10]

    Rassouli, F

    B. Rassouli, F. E. Rosas, and D. G¨ und¨ uz, Data disclosure under perfect sample privacy, IEEE Transactions on Information Forensics and Security15, 2012 (2019)

    work page 2012

  11. [11]

    Hamman and S

    F. Hamman and S. Dutta, Demystifying local and global fairness trade-offs in federated learn- ing using partial information decomposition, arXiv preprint arXiv:2307.11333 (2023)

    work page arXiv 2023

  12. [12]

    F. E. Rosas, P. A. Mediano, H. J. Jensen, A. K. Seth, A. B. Barrett, R. L. Carhart-Harris, and D. Bor, Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data, PLoS computational biology16, e1008289 (2020)

    work page 2020

  13. [13]

    R. A. Ince, The partial entropy decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal, arXiv preprint arXiv:1702.01591 (2017)

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  14. [14]

    P. A. Mediano, F. Rosas, R. L. Carhart-Harris, A. K. Seth, and A. B. Barrett, Beyond integrated information: A taxonomy of information dynamics phenomena, arXiv preprint arXiv:1909.02297 (2019)

    work page arXiv 1909

  15. [15]

    A. Lyu, B. Yuan, O. Deng, M. Yang, A. Clark, and J. Zhang, System information decompo- sition, arXiv preprint arXiv:2306.08288 (2023)

    work page arXiv 2023

  16. [16]

    T. F. Varley, Generalized decomposition of multivariate information, arXiv preprint arXiv:2309.08003 (2023)

    work page arXiv 2023

  17. [17]

    Griffith, E

    V. Griffith, E. K. Chong, R. G. James, C. J. Ellison, and J. P. Crutchfield, Intersection information based on common randomness, Entropy16, 1985 (2014). 43

    work page 1985

  18. [18]

    R. A. Ince, Measuring multivariate redundant information with pointwise common change in surprisal, Entropy19, 318 (2017)

    work page 2017

  19. [19]

    Bertschinger, J

    N. Bertschinger, J. Rauh, E. Olbrich, and J. Jost, Shared information—new insights and problems in decomposing information in complex systems, inProceedings of the European conference on complex systems 2012(Springer, 2013) pp. 251–269

    work page 2012

  20. [20]

    Harder, C

    M. Harder, C. Salge, and D. Polani, Bivariate measure of redundant information, Physical Review E87, 012130 (2013)

    work page 2013

  21. [21]

    Bertschinger, J

    N. Bertschinger, J. Rauh, E. Olbrich, J. Jost, and N. Ay, Quantifying unique information, Entropy16, 2161 (2014)

    work page 2014

  22. [22]

    Kolchinsky, A novel approach to the partial information decomposition, Entropy24, 403 (2022)

    A. Kolchinsky, A novel approach to the partial information decomposition, Entropy24, 403 (2022)

    work page 2022

  23. [23]

    H. K. Ting, On the amount of information, Theory of Probability & Its Applications7, 439 (1962)

    work page 1962

  24. [24]

    R. W. Yeung, A new outlook on shannon’s information measures, IEEE transactions on in- formation theory37, 466 (1991)

    work page 1991

  25. [25]

    Chicharro and S

    D. Chicharro and S. Panzeri, Synergy and redundancy in dual decompositions of mutual information gain and information loss, Entropy19, 71 (2017)

    work page 2017

  26. [26]

    F. E. Rosas, P. A. Mediano, B. Rassouli, and A. B. Barrett, An operational information decomposition via synergistic disclosure, Journal of Physics A: Mathematical and Theoretical 53, 485001 (2020)

    work page 2020

  27. [27]

    J. T. Lizier, B. Flecker, and P. L. Williams, Towards a synergy-based approach to measuring information modification, in2013 IEEE Symposium on Artificial Life (ALIFE)(IEEE, 2013) pp. 43–51

    work page 2013

  28. [28]

    P. L. Williams,Information dynamics: Its theory and application to embodied cognitive sys- tems, Ph.D. thesis, Indiana University (2011)

    work page 2011

  29. [29]

    Griffith and C

    V. Griffith and C. Koch, Quantifying synergistic mutual information, inGuided self- organization: inception(Springer, 2014) pp. 159–190

    work page 2014

  30. [30]

    J. Rauh, P. K. Banerjee, E. Olbrich, G. Mont´ ufar, and J. Jost, Continuity and additivity properties of information decompositions, International Journal of Approximate Reasoning 161, 108979 (2023)

    work page 2023

  31. [31]

    A. B. Barrett, Exploration of synergistic and redundant information sharing in static and 44 dynamical gaussian systems, Physical Review E91, 052802 (2015)

    work page 2015

  32. [32]

    Finn and J

    C. Finn and J. T. Lizier, Pointwise partial information decompositionusing the specificity and ambiguity lattices, Entropy20, 297 (2018)

    work page 2018

  33. [33]

    R. G. James, J. Emenheiser, and J. P. Crutchfield, Unique information via dependency con- straints, Journal of Physics A: Mathematical and Theoretical52, 014002 (2018)

    work page 2018

  34. [34]

    R. G. James, C. J. Ellison, and J. P. Crutchfield, “dit“: a python package for discrete infor- mation theory, Journal of Open Source Software3, 738 (2018)

    work page 2018

  35. [35]

    R. J. Glauber, Time-dependent statistics of the ising model, Journal of mathematical physics 4, 294 (1963)

    work page 1963

  36. [36]

    Barnett, J

    L. Barnett, J. T. Lizier, M. Harr´ e, A. K. Seth, and T. Bossomaier, Information flow in a kinetic ising model peaks in the disordered phase, Physical review letters111, 177203 (2013)

    work page 2013

  37. [37]

    Sootla, D

    S. Sootla, D. O. Theis, and R. Vicente, Analyzing information distribution in complex systems, Entropy19, 636 (2017)

    work page 2017

  38. [38]

    E. T. Jaynes, Information theory and statistical mechanics, Physical review106, 620 (1957). 45

    work page 1957