A Novel Approach to the Partial Information Decomposition

Artemy Kolchinsky

REVIEW 2 major objections 5 minor 113 references

A Novel Approach to the Partial Information Decomposition

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a single channel-optimization problem defines the unique redundancy measure for partial information decomposition, grounded in Blackwell's operational order.

desk verdict A genuinely new PID redundancy measure with a clean axiomatic story, but the Appendix D proof of the union-side Blackwell property has a chain-rule error that needs fixing before publication. read the letter →

arxiv 1908.08642 v4 pith:SYZBJ7FT submitted 2019-08-23 cs.IT math.ITq-bio.NCstat.ML

Artemy Kolchinsky This is my paper · ORCID

classification cs.ITmath.ITq-bio.NCstat.ML MSC 94A1794A1562B15

keywords partialinformationdecompositionredundancysynergyBlackwellordergarblingunionmutualaxiomatictheory

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to decompose the information a set of sources carries about a target into redundancy, synergy, unique, and union parts. Its central move is to define redundancy as the information carried by the most informative channel that is a noisy degradation (garbling) of every source channel, and union as the least informative channel that dominates every source. The author proves this redundancy measure is the unique one satisfying five natural axioms, and that it satisfies a multivariate analogue of the Blackwell property connecting information order to decision problems. The union measure is shown to equal a previously proposed quantity, so the framework also supplies a general multivariate synergy measure. The paper argues that the usual inclusion-exclusion rule of set theory should not be expected to hold between redundancy and union information.

What carries the argument

The machinery is the Blackwell order (garbling relation) $p_{B|Z} \preceq p_{C|Z}$, which holds when sampling from $p_{C|Z}$ and then applying a fixed noisy channel can reproduce $p_{B|Z}$. This order formalizes what it means for one source to be more informative than another, and its decision-theoretic content comes from Blackwell's theorem. The paper builds redundancy and union information as the information-theoretic analogues of set intersection and union: redundancy is the maximum mutual information over channels below every source, and union is the minimum over channels above every source. The feasible sets are convex polytopes, so redundancy can be computed by vertex enumeration, with Theorem 8 bounding the required cardinality of the auxiliary variable $Q$. Garbling Equality replaces the earlier Deterministic Equality axiom, and together with Existence it makes the axiom system strong enough for uniqueness.

What would settle it

Compute $I^\star_\cap$ for the classic example $Y = X_1 \,\mathrm{AND}\, X_2$ with independent uniform binary sources: the paper predicts approximately $0.311$ bits, so an independent implementation of Eq. (15) returning a different value would refute the construction. Likewise, for $Y=(X_1,X_2)$ with a full-support joint distribution, Theorem 5 predicts $I^\star_\cap = C(X_1 \wedge X_2)=0$ even when $I(X_1;X_2)>0$; any positive redundancy in that setup would falsify the identification.

Watch

Extended reading notes

Core claim

The central discovery is a channel-based definition of redundancy: $I^\star_\cap(X_1;\ldots;X_n \to Y) = \max_{s_{Q|Y}} I_s(Q;Y)$ subject to $s_{Q|Y}$ being a garbling of each source channel $p_{X_i|Y}$. This makes set-theoretic intersection operational: the intersection of information sets is the largest piece of information contained in every source, where containment is the decision-theoretic Blackwell order. Theorem 1 shows a source has no unique information exactly when its channel is a garbling of every other source. Theorem 3 shows that under Symmetry, Self-redundancy, Monotonicity, Garbling Equality, and Existence, $I^\star_\cap$ is the unique redundancy measure. The dual union measure $I^\star_\cup$ equals the previously proposed $I_p^*$ of Bertschinger et al. and Griffith and Koch, giving a multivariate synergy measure. The measures are channel-oriented, depending only on the pairwise marginals $p_{YX_i}$, so they quantify mechanistic rather than source redundancy.

Load-bearing premise

The construction rests on accepting that one source's information is contained in another's exactly when the first is a noisy degradation (a 'garbling') of the second; choose a different notion of containment, such as deterministic dependence or conditional independence, and the redundancy value and uniqueness theorem change.

Editorial extensions

If this is right

Any number of sources can be handled: the definitions and optimization problems are not restricted to the bivariate case.
A source carries no unique information exactly when its channel is a garbling of every other source, giving the Blackwell property a multivariate form.
The union measure equals the previously proposed $I_p^*$, so the resulting synergy extends those proposals to $n$ sources.
The inclusion-exclusion principle fails for these measures, and Lemma 6 shows that any measure satisfying Independent Identity must violate inclusion-exclusion for three or more sources.
When the target is the joint outcome $(X_1,X_2)$, redundancy equals Gács-Körner common information, so independent sources have zero redundancy about their joint outcome.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

As an extension, the same max/min template could be run with other 'more informative' relations: replacing Blackwell order with conditional independence recovers the $I^\mathrm{GH}_\cap$ measure, and deterministic functional containment recovers $I^\wedge_\cap$, suggesting the paper's framework is a unifying schema for existing redundancy proposals.
The convex-maximization formulation is NP-hard in general, so scaling to many high-cardinality sources will require approximation algorithms or structural assumptions; jointly Gaussian or other continuous extensions are a natural testbed where convex optimization may become tractable.
Because the measure ignores correlations among sources by construction, applying it to data with strong source dependence could undercount redundancy that arises purely from source correlations; combining it with a source-redundancy term is a possible extension the paper does not develop.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this PID paper has a genuinely new redundancy measure, a clean axiomatic characterization, and honest ties to prior union measures. The reader's report is right to accept in spirit, but it missed a real proof error in Appendix D: the second multivariate Blackwell theorem (Theorem 2) is not proven as printed. It is a local, fixable problem, not a fatal one.

What's new: I*_cap (Eq. 15) — maximum mutual information over channels sQ|Y that are garblings of every source channel — is a fresh way to define redundancy, and the Blackwell decision-theoretic reading gives it operational content. The generalized framework (Eqs. 31-32) organizes several earlier proposals as instances; the uniqueness result (Theorem 3) is a substantive contribution; and the proof that union information I*_cup equals Ip* (Theorem 9) is correct and properly credits Bertschinger et al. and Griffith-Koch. The examples, especially the COPY gate where I*_cap becomes Gács-Körner common information, are illuminating. Code is provided.

The soft spot: Appendix D. In the proof of Theorem 2, the joint sY X1Q is defined as sQ|X1 pY X1, which gives the Markov chain Y-X1-Q, not Y-Q-X1. The proof uses the wrong chain-rule expansion, so the only-if direction does not establish the condition needed; the printed argument is unsound. Fix: use the correct identity I_s(Y;Q) = I_p(Y;X1) - I_s(Y;X1|Q) (a minus, not a plus). Then the desired conclusion follows. Eq. (D2) in the Theorem 1 proof has the same sign error; there too the conclusion survives after the sign correction. The uniqueness proofs (Appendix C) and the I*_cup = Ip* equivalence (Appendix B) look fine. There's also a notational slip in Theorem 4's proof where I*_cap appears instead of I*_cup. None of this damages the main contributions, but it means the paper is not in publishable shape as-is.

A separate point: the Blackwell/garbling relation is the load-bearing modeling choice. The paper argues for it well and is explicit that it only captures mechanistic redundancy. If you think information containment should be defined more leniently or should incorporate source correlations, the measure changes. That's a scope condition, not an error.

Who it's for: anyone working on partial information decomposition or multivariate information measures. It deserves a serious referee and a request for minor-to-major revision. After the Appendix D proofs are corrected, it would be a solid contribution.

Referee Report

2 major / 5 minor

Summary. This paper proposes a channel-based partial information decomposition. The redundancy measure I*_cap (Eq. (15)) is defined as the maximum mutual information of a channel sQ|Y that is a Blackwell garbling of every source channel pXi|Y; the union information I*_cup (Eq. (19)) is defined dually. The authors prove that I*_cap satisfies a multivariate Blackwell property (Theorem 1), that I*_cup satisfies the analogous property (Theorem 2), that both measures are uniquely characterized by a set of axioms including Garbling Equality and Existence (Theorems 3 and 4), and that I*_cup agrees with the previously proposed Ip* of Bertschinger et al. and Griffith and Koch (Theorem 9). The paper also discusses the violation of the inclusion-exclusion principle, relations to Gacs-Korner common information, comparisons with prior PID measures, and generalizations of the framework.

Significance. If the results are correct, this is a significant contribution to the PID literature. The definitions are explicit, parameter-free, and have a clear decision-theoretic interpretation via Blackwell's theorem; the uniqueness theorems provide a principled way to select a multivariate redundancy measure; and the equivalence I*_cup = Ip* connects the proposal to a previously studied optimization problem. The paper also provides a practical vertex-enumeration algorithm and public code, and its treatment of the inclusion-exclusion principle is thought-provoking. The main reservations are localized proof defects in Appendix D; they do not, in my reading, invalidate the conceptual framework, but they must be repaired before the theorems can be considered proven.

major comments (2)

[Appendix D, proof of Theorem 2 (Eq. (D4))] The proof of the 'only if' direction misapplies the chain rule. The joint distribution sY X1Q is defined as sQ|X1(q|x1)pY X1(y,x1), which gives the Markov chain Y-X1-Q, so Is(Y;Q|X1)=0 and the correct identity is Is(Y;Q)=Ip(Y;X1)-Is(Y;X1|Q). The printed Eq. (D4), Is(Y;Q)=Ip(Y;X1)+Is(Y;Q|X1), follows only under the opposite Markov chain Y-Q-X1, which is not the one constructed. Consequently, from Ip(Y;X1)=I*_cup one cannot draw the printed conclusion that pX1|Y is a garbling of sQ|Y. The argument is repairable by using the correct sign and then inferring Is(Y;X1|Q)=0, but as written Theorem 2 is unproven.
[Appendix D, proof of Theorem 1 (Eq. (D2))] The same sign error appears in Eq. (D2): equating the two chain-rule expansions gives Is(Y;Q)=Ip(Y;X1)-Is(Y;X1|Q), not Is(Y;Q)=Ip(Y;X1)+Is(Y;X1|Q). With the printed plus sign, the assumption Ip(Y;X1)=I*_cap does not imply Is(Y;X1|Q)=0, so the printed 'only if' direction does not establish the multivariate Blackwell property for I*_cap. Replacing '+' with '-' fixes the immediate step; the rest of the argument then goes through.

minor comments (5)

[Section III C, Theorem 4] The theorem statement lists only Symmetry, Self-union, Garbling Equality, and Existence, but the proof in Appendix C and the axiom list in the same section include Monotonicity; the theorem statement should be corrected.
[Appendix D, proof of Theorem 2] The equation for the marginal sY Q sums over y on the right-hand side; it should sum over x1, and the notation for the marginal is inconsistent.
[Section IV A] The sentence 'Null Equality is implied by Garbling Equality (so I*_cap obeys it)' should refer to I*_cup, since Null Equality is the union-side dual of Target Equality.
[Appendix B] The constraints are written with pXi|Z instead of pXi|Y in several places; the symbol Z should be Y throughout.
[Appendix C, proof of Theorem 4] The inequality 'I*_cap(...) >= I*_cup(...)' contains a typo; the first symbol should be I*_cup.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimizations, axioms, and equivalences are independently defined and proven.

full rationale

The central redundancy measure (Eq. 15) is an explicit convex-maximization over channels that are Blackwell-garblings of every source channel, and the union measure (Eq. 19) is its explicit dual. Theorem 3's uniqueness proof does not reduce to the definition: the Existence axiom only provides a feasible witness that bounds any candidate measure above by the Eq. 15 max, while Garbling Equality, Self-redundancy, and Monotonicity bound it below; the axiom set is not a paraphrase of Eq. 15. The claimed equivalence I*_cup = Ip* (Theorem 9) is derived in Appendix B against the independently stated convex-minimization benchmark of Bertschinger et al. and Griffith and Koch, rather than being assumed or imported. Theorems 1 and 5 are proved from the definition/axioms, and the comparisons to I^BROJA, I^GH, and I^wedge are explicit external benchmarks, not renamed versions of the proposed quantity. No parameters are fitted, no prediction is statistically forced, and no load-bearing premise rests on a self-citation; prior work is cited either as standard mathematical background (Blackwell) or as alternative measures to be compared or formally equated. The suspected chain-rule/Markov-direction error in the Appendix D proof of Theorem 2 is a proof-correctness concern, not circularity, since it does not make the theorem's conclusion identical to its input. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The central results rest on standard theorems and on the paper's proposed axioms. No parameters are fitted to data, and no new ontological entities are introduced. The Existence and Garbling Equality axioms are the assumptions a critic would most plausibly attack, since they are strong and purpose-built.

assumptions (7)

standard math Blackwell-Sherman-Stein theorem: pB|Z ≼ pC|Z iff the maximum expected utility with pB is at most that with pC in every decision problem.
Used throughout to give operational meaning to the order ≼, in particular in Theorems 1, 2, and 7, and cited to [26,61].
domain assumption Finite alphabets and full-support marginals for all sources and the target.
Stated in Section II B; full support allows division by pY(y) in proofs such as Theorem 1 and Theorem 2.
domain assumption Symmetry, Self-redundancy, and Monotonicity for redundancy measures.
These are the standard Williams and Beer axioms adopted in Section III C as premises of Theorem 3; the paper does not derive them from more basic principles.
ad hoc to paper Garbling Equality for redundancy: redundancy is unchanged when a newly added source is a garbling of an existing source.
Proposed in Section III C as a Blackwell-based replacement for deterministic equality; it is essential to Theorem 3 and Theorem 1.
ad hoc to paper Existence for redundancy: for every joint distribution there is a channel sQ|Y that is less informative than each source and attains exactly the redundancy value.
Proposed in Section III C; it is the strongest of the paper's new axioms and is needed to prove uniqueness in Theorem 3.
ad hoc to paper Dual Garbling Equality and Existence axioms for union information.
Used in the proof of Theorem 4 after Section III C; they mirror the redundancy axioms and force I*_cup as the unique union measure.
standard math A convex function attains its maximum at a vertex of a compact convex polytope, and Dubins's theorem bounds the support size of extreme points.
Justifies the vertex-enumeration algorithm in Appendix A and the cardinality bound |Q| ≤ (sum_i |Xi|) - n + 1 in Theorem 8.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Novel Approach to the Partial Information Decomposition." pith.science (2026). https://pith.science/paper/SYZBJ7FT

@misc{pith2026190808642,
  author       = {Pith},
  title        = {Pith review of: A Novel Approach to the Partial Information Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SYZBJ7FT}},
  note         = {Machine review of arXiv:1908.08642}
}

read the original abstract

We consider the "partial information decomposition" (PID) problem, which aims to decompose the information that a set of source random variables provide about a target random variable into separate redundant, synergistic, union, and unique components. In the first part of this paper, we propose a general framework for constructing a multivariate PID. Our framework is defined in terms of a formal analogy with intersection and union from set theory, along with an ordering relation which specifies when one information source is more informative than another. Our definitions are algebraically and axiomatically motivated, and can be generalized to domains beyond Shannon information theory (such as algorithmic information theory and quantum information theory). In the second part of this paper, we use our general framework to define a PID in terms of the well-known Blackwell order, which has a fundamental operational interpretation. We demonstrate our approach on numerous examples and show that it overcomes many drawbacks associated with previous proposals.

Figures

Figures reproduced from arXiv: 1908.08642 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

113 extracted references · 76 canonical work pages

[1]

Thus, the Blackwell property can be formalized in terms of I∩ as I(Y ; X1) = I∩(X1; X2 /shortrightarrowY ) iﬀ pX1|Y ≼ pX2|Y

relates the unique information to the redundancy via U (X1 \ X2 /shortrightarrowY ) = I(Y ; X1) − I∩(X1; X2 /shortrightarrowY ) . Thus, the Blackwell property can be formalized in terms of I∩ as I(Y ; X1) = I∩(X1; X2 /shortrightarrowY ) iﬀ pX1|Y ≼ pX2|Y . (12) Conversely, unique information can also be related to union information via U (X1 \ X2 /shortrig...
[2]

Self-redundancy: I∩(X1 /shortrightarrowY ) = I(Y ; X1)
[3]

Xn /shortrightarrowY ) ≤ I∩(X1

Monotonicity: I∩(X1; . . . ; Xn /shortrightarrowY ) ≤ I∩(X1; . . . ; Xn−1 /shortrightarrowY )
[4]

size of the intersection

are positive. This measure can be written as I BROJA ∩ (X1; X2 /shortrightarrowY ) := Ip(Y ; X1) + Ip(Y ; X2) − Ip∗ (Y ; X1, X2), where Ip∗ (Y ; X1, X2) is deﬁned in terms of the optimization problem Ip∗ (Y ; X1, X2) = min sY X1X2 Is(Y ; X1, X2) s.t. sY X1 = pY X1 , sY X2 = pY X2 . In other words, Ip∗ (Y ; X1, X2) is the minimal mutual infor- mation that ...
[5]

more informative

Existence: For any pY X1...Xn , there exists a channel sQ|Y such that sQ|Y ≼ pXi|Y for all i ∈ [1..n] and I∩(X1; . . . ; Xn /shortrightarrowY ) = Is(Y ; Q). While the Symmetry, Self-redundancy, and Monotonicity ax- ioms are the same as in the standard PID framework (see Sec- tion II B), the other two axioms require some explanation.Gar- bling Equality is ...
[6]

Xn /shortrightarrowY ) is invariant to the per- mutation of X1,

Symmetry: I∩(X1; . . . ; Xn /shortrightarrowY ) is invariant to the per- mutation of X1, . . . , Xn
[7]

Syn- ergy, redundancy, and multivariate information measures: an experimentalist’s perspective,

N. Timme, W. Alford, B. Flecker, and J. M. Beggs, “Syn- ergy, redundancy, and multivariate information measures: an experimentalist’s perspective,” Journal of Computational Neu- roscience, vol. 36, pp. 119–140, Apr. 2014

work page 2014
[8]

Multivariate Mutual Information Inspired by Secre t- Key Agreement,

C. Chan, A. Al-Bashabsheh, J. B. Ebrahimi, T. Kaced, and T. Liu, “Multivariate Mutual Information Inspired by Secre t- Key Agreement,” Proceedings of the IEEE, vol. 103, pp. 1883– 1913, Oct. 2015

work page 1913

Show all 113 references

[9]

Xn /shortrightarrowY ) = I∩(X1

Garbling Equality : I∩(X1; . . .; Xn /shortrightarrowY ) = I∩(X1; . . . ; Xn−1 /shortrightarrowY ) if pXi|Y ≼ pXn|Y for some i < n
[10]

Nonnegative decomposition of multivariate information,

P. L. Williams and R. D. Beer, “Nonnegative decomposition of multivariate information,” arXiv preprint arXiv:1004.2515 , 2010

2010 arXiv
[11]

Xn /shortrightarrowY ) is invariant to the per- mutation of X1,

Symmetry: I∪(X1; . . . ; Xn /shortrightarrowY ) is invariant to the per- mutation of X1, . . . , Xn
[12]

( 13), which applies to the case of more than two sources

and Eq. ( 13), which applies to the case of more than two sources. Observe that the mutual information I(Y ; X1) depends only on the pairwise distribution pY X1 , and that the relation pX1|Y ≼ pX2|Y depends only on the pairwise marginal dis- tributions pY X1 and pY X2 , not th...
[13]

Self-union: I∪(X1 /shortrightarrowY ) = I(Y ; X1)
[14]

( 18), along with the decision-theoretic operationalization of the gar bling relation ≼

and Eq. ( 18), along with the decision-theoretic operationalization of the gar bling relation ≼ . Here we show that I ⋆ ∩ and I ⋆ ∪ can also be derived from an axiomatic framework. In particular, we show that I ⋆ ∩ is the unique measure that satisﬁes the following set of ﬁve axioms:
[15]

Xn /shortrightarrowY ) ≥ I∪(X1

Monotonicity: I∪(X1; . . . ; Xn /shortrightarrowY ) ≥ I∪(X1; . . . ; Xn−1 /shortrightarrowY )
[16]

Xn /shortrightarrowY ) = I∪(X1

Garbling Equality : I∪(X1; . . .; Xn /shortrightarrowY ) = I∪(X1; . . . ; Xn−1 /shortrightarrowY ) if pXn|Y ≼ pXi|Y for some i < n
[17]

Axiom of union

Existence: For any pY X1...Xn , there exists a channel sQ|Y such that pXi|Y ≼ sQ|Y for all i ∈ [1..n] and I∪(X1; . . . ; Xn /shortrightarrowY ) = I(Y ; Q). These axioms are dual to the above redundancy axioms. Com- pared to previous work (see Section II B 2), the most novel of...
[18]

local positivity

and Self-redundancy lead to the following identity: I ⋆ ∩(X1, X2 /shortrightarrowX1) = I ⋆ ∩(X2 /shortrightarrowX1) = I(X1; X2) = I ⋆ ∩(X1 /shortrightarrowX2) = I ⋆ ∩(X1, X2 /shortrightarrowX2) (24) In other words, the redundancy in sources X1 and X2 about X1 as the target, or...
[19]

Redundancy and union information should be quantiﬁed with two diﬀerent measures, I∩ and I∪. The deﬁnition of these measures should be related by a formal duality (as with I ⋆ ∩ and I ⋆ ∩), but not necessarily by a quantitative duality such as the inclusion-exclusion principle. 11
[20]

( 22), excluded information in source Xi is bounded between 0 and the conditional entropy I(Y ; X1,

and Eq. ( 22), excluded information in source Xi is bounded between 0 and the conditional entropy I(Y ; X1, . . . , Xn|Xi). Just like our redundancy measure, I ⋆ ∪ obeys a multivariate generalization of Blackwell’s property, Eq. ( 13). (See Ap- pendix D for the proof.) Theorem...
[21]

excluded information

These separate measures of redundancy and union infor- mation lead to two diﬀerent types of unique information. The ﬁrst type of unique information is deﬁned in terms of redundancy, as in Eq. ( 17), and quantiﬁes the infor- mation in a given source that is not redundant with t...
[22]

Synergy should be quantiﬁed in terms of the union in- formation measure, I∪, as in Eq. ( 21). As shown in Section III A and Section III B, all of the above quantities are guaranteed to be non-negative for our propos ed measures, I ⋆ ∩ and I ⋆ ∪. We ﬁnish by noting two things. ...
[23]

, Xn) [46]

Second, without the inclusion- exclusion principle, it becomes questionable whether it is ac- tually possible to derive a meaningful non-negative decom- position of I(Y ; X1, . . . , Xn) [46]. As an example with the bivariate case, without inclusion-exclusion it is no longe r ...
[24]

Can it be applied to more than 2 sources?
[25]

Does it obey the Monotonicity axiom?
[26]

Does it obey the inclusion-exclusion principle for the bivariate case (i.e., is Eq. (
[27]

Does it obey the Independent Identity property?
[28]

Does it obey our proposed Target Equality property (Eq. (25))?
[29]

Question marks (?) indicate aspects that we could not easily establish

Does it obey the Blackwell property (possibly in its mul- tivariate form, Theorem 1)? The results are summarized in Table I. Question marks (?) indicate aspects that we could not easily establish. In the next subsections, we perform a more detailed com- parison of our approach...
[30]

state of the environment

for this example, we ﬁnd that the optimal Q∗ is given by Q∗ = X1 OR X2, which gives the smaller redundancy value I GH ∩ (X1; X2 /shortrightarrowY ) ≈ 0.123 bits. (Note that it is incorrectly stated in [21] that in this examp le, I GH ∩ (X1; X2 /shortrightarrowY ) = 0 . In addi...
[31]

(30) 4 Assume there exists some pY X1X2Q with pQ|Y = s∗ Q|Y = pX1|Y = pX2|Y and I(Y ; Q|X1) = I(Y ; Q|X2) = 0

There is no decision problem (pY , u, A) such that for some b ∈ B , max sA′ |C E[u(A′, Y )|B = b] ≥ max sA|B E[u(A, Y )|B = b]. (30) 4 Assume there exists some pY X1X2Q with pQ|Y = s∗ Q|Y = pX1|Y = pX2|Y and I(Y ; Q|X1) = I(Y ; Q|X2) = 0 . Then, p(q|y) =∑ x1 p(q|x1)p(x1|y) = δ...
[32]

a source of information

The analogue of a set was a channel like pA|Y (this deﬁnes what is meant by “a source of information” about the target Y )
[33]

more informative

The analogue of set inclusion ⊆ was the garbling relation ≼ (this deﬁnes what is meant by the “more informative” relation)
[34]

amount of information

The analogue of set size for a given information source pA|Y was the mutual information I(A; Y ), given some ﬁxed target distribution pY ) (this deﬁnes what is meant by “amount of information”). It is possible to generalize our approach by considering oth er possible analogues...
[35]

and Eq. ( 15)). Similarly, one can deﬁne a union information in a general way as min a∈Ω φ(a) s .t. ∀i b i ⊑ a, (32) so that union information is the minimum information in any element of Ω which is more informative that each source (com- pare to Eq. (
[36]

and Eq. ( 19)). There are many options for how to actually deﬁne Ω , ⊑, and φ, though for Eq. ( 31) and Eq. ( 32) to be well-deﬁned the relation ⊑ should be a preorder. One would also commonly require φ to be monotonic with regards to ⊑, so that a ⊑ b implies φ(a) ≤ φ(b). Beyo...
[37]

more informative

One can take Ω to be the set of channels of the form pA|Y and φ to be the mutual information, but use some other relation (not the garbling relation ≼ ) as the “more informative” relation ⊑. For instance, one could say pB|Y ⊑ pA|Y if and only ifpB|Y is a garbling ofpA|Y for so...
[38]

amount of information

One can take Ω be the set of channels of the form pA|Y and keep ⊑ as the garbling relation ≼ , but change the “amount of information” function φ. For instance, one could deﬁne φ in terms “channel capac- ity”, φ(pA|Y ) := max sY I(A; Y ). This would allow one to quantify redund...
[39]

In this case, Eq

One can take Ω be the set of random variables de- ﬁned over some underlying shared probability space, take the amount of information function to be mutual information, φ(A) = I(A; Y ), and say that B ⊑ A iﬀ B is conditionally independent of Y given A. In this case, Eq. (
[40]

would become equivalent to the I GH ∩ redundancy measure discussed in the previous section (Eq. ( 29)). If instead one said that B ⊑ A iﬀ B = f (A) for some deterministic function f , one would recover the I ∧ ∩ redundancy measure discussed in the previous section (Eq. ( 28))
[41]

elephant

One can take Ω be the set of channels from source to output (such as pY |A), rather than from output to source (such as pA|Y ), and deﬁne ⊑ in terms of the input- degradation relation [54]: pY |B ⊑ pY |A iﬀ there exists some channel pA|B such that p(y|b) = ∑ a p(a|b)p(y|a). (3...
[42]

more informative

As a ﬁnal example, one could extend the framework be- yond the domain of Shannon-type information, and con- sider for instance redundancy and union information for 14 Algorithmic Information Theory [55]. Here, Ω would be the set of ﬁnite strings, the “more informative” rela- t...
[43]

and Eq. ( 32). Of course, while many generalizations are possible, the util- ity of any particular generalization will ultimately be det er- mined by factors such as: how easily can one compute the resulting redundancy and union information measures? do th e resulting measures...
[44]

The AND gate, Y = X1 AND X2, with X1 and X2 independent
[45]

The SUM gate: Y = X1 + X2, with X1 and X2 inde- pendent
[46]

In this case, I Ince ∩ (marked with ∗) gave values that increased with the amount of correlation between X1 and X2 but were typically larger than I(X1;X2)

The UNQ gate: Y = X1. In this case, I Ince ∩ (marked with ∗) gave values that increased with the amount of correlation between X1 and X2 but were typically larger than I(X1;X2)
[47]

Our redundancy mea- sure is equal to the Gács-Körner common information between X and Y , as discussed in Section IV A

The COPY gate: Y = (X1, X2). Our redundancy mea- sure is equal to the Gács-Körner common information between X and Y , as discussed in Section IV A. In this case, I Ince ∩ (marked with ∗) gave the same values as for the UNQ gate, which increase with the amount of cor- relation...
[48]

Three-way AND gate: Y = X1 AND X2 AND X3, where the sources are binary and uniformly and inde- pendently distributed
[49]

Three-way SUM gate: Y = X1 + X2 + X3, where the sources are binary and uniformly and independently distributed
[50]

channel-oriented

“Overlap” gate: we deﬁned four independent uniformly- distributed binary random variables, A, B, C, D. We then deﬁned three sources X1, X2, X3 as X1 = (A, B), X2 = ( A, C), X3 = ( A, D). The target was the joint outcome of all three sources, Y = ( X1, X2, X3) = ((A, B), (A, C)...
[51]

join” and “meet

and Eq. (32)) and the algebraic notions of “join” and “meet” on lattices. Our deﬁnitions are not based on join and meet operation, primarily because we do not require the “more informative” relation to necessarily form a lattice ( it is known, for instance, that the garbling r...
[52]

Synergy, Redun- dancy, and Independence in Population Codes,

E. Schneidman, W. Bialek, and M. J. Berry, “Synergy, Redun- dancy, and Independence in Population Codes,” The Journal of Neuroscience, vol. 23, pp. 11539–11553, Dec. 2003

2003
[53]

Quantifying collectivity,

B. C. Daniels, C. J. Ellison, D. C. Krakauer, and J. C. Flack, “Quantifying collectivity,” Current Opinion in Neurobiology , vol. 37, pp. 106–113, 2016

2016
[54]

The partial informatio n decomposition of generative neural network models,

T. Tax, P. Mediano, and M. Shanahan, “The partial informatio n decomposition of generative neural network models,” Entropy, vol. 19, no. 9, p. 474, 2017

2017
[55]

Understanding individ - ual neuron importance using information theory,

R. A. Amjad, K. Liu, and B. C. Geiger, “Understanding individ - ual neuron importance using information theory,” arXiv preprint arXiv:1804.06679, 2018

2018 arXiv
[56]

Informa tion decomposition of target eﬀects from multi-source interact ions: perspectives on previous, current and future work,

J. Lizier, N. Bertschinger, J. Jost, and M. Wibral, “Informa tion decomposition of target eﬀects from multi-source interact ions: perspectives on previous, current and future work,” 2018

2018
[57]

However, our redundancy measure is not numerically related to the union information measure, so the analysis in [57] does not apply t o our measure of redundancy

implies that — for the two sources, and a univariate Gaus - sian Y — I ⋆ ∪(X1; X2 /shortrightarrowY ) = max i I(Xi; Y ). However, our redundancy measure is not numerically related to the union information measure, so the analysis in [57] does not apply t o our measure of redun...
[58]

Partial information decomposition as a uniﬁed a p- proach to the speciﬁcation of neural goal functions,

M. Wibral, V . Priesemann, J. W. Kay, J. T. Lizier, and W. A. Phillips, “Partial information decomposition as a uniﬁed a p- proach to the speciﬁcation of neural goal functions,” Brain and Cognition, vol. 112, pp. 25–38, Mar. 2017

2017
[59]

Unique informa - tion and secret key agreement,

R. James, J. Emenheiser, and J. Crutchﬁeld, “Unique informa - tion and secret key agreement,” Entropy, vol. 21, no. 1, p. 12, 2019

2019
[60]

On the amount of information,

H. K. Ting, “On the amount of information,” Theory of Proba- bility & Its Applications , vol. 7, no. 4, pp. 439–447, 1962

1962
[61]

A new outlook on shannon’s information mea- sures,

R. W. Y eung, “A new outlook on shannon’s information mea- sures,” IEEE transactions on information theory , vol. 37, no. 3, pp. 466–474, 1991

1991
[62]

Information and its metric,

J. P. Crutchﬁeld, “Information and its metric,” in Nonlinear Structures in Physical Systems, pp. 119–130, Springer, 1990

1990
[63]

The co-information lattice,

A. J. Bell, “The co-information lattice,” in Proceedings of the Fifth International Workshop on Independent Component Anal- ysis and Blind Signal Separation: ICA , vol. 2003, 2003. 16

2003
[64]

P. L. Williams, Information dynamics: Its theory and applica- tion to embodied cognitive systems . Ph.D., Indiana University, United States – Indiana, 2011

2011
[65]

Qua n- tifying unique information,

N. Bertschinger, J. Rauh, E. Olbrich, J. Jost, and N. Ay, “Qua n- tifying unique information,” Entropy, vol. 16, no. 4, pp. 2161– 2183, 2014

2014
[66]

Quantifying syner- gistic information using intermediate stochastic variabl es,

R. Quax, O. Har-Shemesh, and P. Sloot, “Quantifying syner- gistic information using intermediate stochastic variabl es,” En- tropy, vol. 19, no. 2, p. 85, 2017

2017
[67]

Unique information via dependency constraints,

R. G. James, J. Emenheiser, and J. P. Crutchﬁeld, “Unique information via dependency constraints,” Journal of Physics A: Mathematical and Theoretical, vol. 52, no. 1, p. 014002, 2018

2018
[68]

Intersection information based on common ran - domness,

V . Griﬃth, E. K. Chong, R. G. James, C. J. Ellison, and J. P. Crutchﬁeld, “Intersection information based on common ran - domness,” Entropy, vol. 16, no. 4, pp. 1985–2000, 2014

1985
[69]

Quantifying synergistic mutual info r- mation,

V . Griﬃth and C. Koch, “Quantifying synergistic mutual info r- mation,” in Guided Self-Organization: Inception , pp. 159–190, Springer, 2014

2014
[70]

Quantifying redundant information in predicting a target random variable,

V . Griﬃth and T. Ho, “Quantifying redundant information in predicting a target random variable,” Entropy, vol. 17, no. 7, pp. 4644–4653, 2015

2015
[71]

Bivariate measure of redun- dant information,

M. Harder, C. Salge, and D. Polani, “Bivariate measure of redun- dant information,” Physical Review E, vol. 87, no. 1, p. 012130, 2013

2013
[72]

Measuring Multivariate Redundant Information wi th Pointwise Common Change in Surprisal,

R. Ince, “Measuring Multivariate Redundant Information wi th Pointwise Common Change in Surprisal,” Entropy, vol. 19, p. 318, June 2017

2017
[73]

Pointwise Partial Information Decom po- sition Using the Speciﬁcity and Ambiguity Lattices,

C. Finn and J. Lizier, “Pointwise Partial Information Decom po- sition Using the Speciﬁcity and Ambiguity Lattices,” Entropy, vol. 20, p. 297, Apr. 2018

2018
[74]

Comparison of experiments,

D. Blackwell, “Comparison of experiments,” tech. rep., How ard Washington United States, 1950

1950
[75]

Equivalent comparisons of experiments,

D. Blackwell, “Equivalent comparisons of experiments,” The annals of mathematical statistics , pp. 265–272, 1953

1953
[76]

Common information is far less than mu- tual information,

P. Gács and J. Körner, “Common information is far less than mu- tual information,” Problems of Control and Information Theory, vol. 2, no. 2, pp. 149–162, 1973

1973
[77]

Zero-error information and ap - plications in cryptography,

S. Wolf and J. Wultschleger, “Zero-error information and ap - plications in cryptography,” in Information Theory Workshop , pp. 1–6, IEEE, 2004

2004
[78]

Reconsid - ering unique information: Towards a multivariate informat ion decomposition,

J. Rauh, N. Bertschinger, E. Olbrich, and J. Jost, “Reconsid - ering unique information: Towards a multivariate informat ion decomposition,” in Information Theory (ISIT), 2014 IEEE In- ternational Symposium on, pp. 2232–2236, IEEE, 2014

2014
[79]

On extractable shared information,

J. Rauh, P. K. Banerjee, E. Olbrich, J. Jost, and N. Bertschin ger, “On extractable shared information,” Entropy, vol. 19, no. 7, p. 328, 2017

2017
[80]

Generalized measures of infor - mation transfer,

P. L. Williams and R. D. Beer, “Generalized measures of infor - mation transfer,” arXiv preprint arXiv:1102.1507, 2011

2011 arXiv
[81]

The blackwell relation deﬁnes no lattice,

N. Bertschinger and J. Rauh, “The blackwell relation deﬁnes no lattice,” in 2014 IEEE International Symposium on Information Theory, pp. 2479–2483, IEEE, 2014

2014
[82]

Coarse-Graining and the Blackwell Order,

J. Rauh, P. K. Banerjee, E. Olbrich, J. Jost, N. Bertschinger , and D. Wolpert, “Coarse-Graining and the Blackwell Order,” Entropy, vol. 19, p. 527, Oct. 2017

2017
[83]

T. M. Cover and J. A. Thomas, Elements of information theory . John Wiley & Sons, 2006

2006
[84]

Comparison of two noisy channels,

J. Korner, “Comparison of two noisy channels,” Topics in infor- mation theory, pp. 411–423, 1977

1977
[85]

Comparison of channels: criteria for domination by a symmetric channel,

A. Makur and Y . Polyanskiy, “Comparison of channels: criteria for domination by a symmetric channel,” IEEE Transactions on Information Theory, vol. 64, no. 8, pp. 5704–5725, 2018

2018
[86]

Computing the unique information,

P. K. Banerjee, J. Rauh, and G. Montúfar, “Computing the unique information,” in 2018 IEEE International Symposium on Information Theory (ISIT) , pp. 141–145, IEEE, 2018

2018
[87]

T. A. Whitelaw, Introduction to abstract algebra. Glasgow u.a: Blackie, 2. ed ed., 1988. OCLC: 17440604

1988
[88]

Concave minimization: theory, applications and algorithms,

H. P. Benson, “Concave minimization: theory, applications and algorithms,” in Handbook of global optimization , pp. 43–148, Springer, 1995

1995
[89]

A ﬁnite algorithm for concave minimization ov er a polyhedron,

H. P. Benson, “A ﬁnite algorithm for concave minimization ov er a polyhedron,” Naval Research Logistics Quarterly , vol. 32, no. 1, pp. 165–177, 1985

1985
[90]

On the global minimization of concave functions,

R. Horst, “On the global minimization of concave functions, ” Operations-Research-Spektrum, vol. 6, no. 4, pp. 195–205, 1984

1984
[91]

Concave minimization over a convex polyhedron,

H. A. Taha, “Concave minimization over a convex polyhedron,” Naval Research Logistics Quarterly, vol. 20, no. 3, pp. 533–548, 1973

1973
[92]

Concave pro- gramming for minimizing the zero-norm over polyhedral sets ,

F. Rinaldi, F. Schoen, and M. Sciandrone, “Concave pro- gramming for minimizing the zero-norm over polyhedral sets ,” Computational Optimization and Applications , vol. 46, no. 3, pp. 467–486, 2010

2010
[93]

P. R. Halmos, Naive set theory . Courier Dover Publications, 2017

2017
[94]

Shared in for- mation - new insights and problems in decomposing information in complex systems,

N. Bertschinger, J. Rauh, E. Olbrich, and J. Jost, “Shared in for- mation - new insights and problems in decomposing information in complex systems,” in Proceedings of the European Confer- ence on Complex Systems 2012 , pp. 251–269, Springer, 2013

2012
[95]

Secret Sharing and Shared Information,

J. Rauh, “Secret Sharing and Shared Information,” Entropy, vol. 19, p. 601, Nov. 2017

2017
[96]

Synergy and Redundancy in Dual Decompositions of Mutual Information Gain and Information Loss,

D. Chicharro and S. Panzeri, “Synergy and Redundancy in Dual Decompositions of Mutual Information Gain and Information Loss,” Entropy, vol. 19, p. 71, Feb. 2017

2017
[97]

Synergy, redundancy and com- mon information,

P. K. Banerjee and V . Griﬃth, “Synergy, redundancy and com- mon information,” arXiv preprint arXiv:1509.03706v1, 2015

2015 arXiv
[98]

Suﬃciency and approximate suﬃciency,

L. Le, “Suﬃciency and approximate suﬃciency,” The Annals of Mathematical Statistics, pp. 1419–1455, 1964

1964
[99]

Shannon meets blackwell and le cam: Channels, codes, and statistical experiments,

M. Raginsky, “Shannon meets blackwell and le cam: Channels, codes, and statistical experiments,” in 2011 IEEE International Symposium on Information Theory Proceedings , pp. 1220– 1224, IEEE, 2011

2011
[100]

The lattice theory of information,

C. Shannon, “The lattice theory of information,” Transactions of the IRE Professional Group on Information Theory , vol. 1, pp. 105–107, Feb. 1953

1953
[101]

A note on a partial ordering for communicatio n channels,

C. E. Shannon, “A note on a partial ordering for communicatio n channels,” Information and Control , vol. 1, pp. 390–397, Dec. 1958

1958
[102]

Cohen, J

J. Cohen, J. H. Kempermann, and G. Zbaganu, Comparisons of Stochastic Matrices with Applications in Information Th e- ory, Statistics, Economics and Population . Springer Science & Business Media, 1998

1998
[103]

On the input-degradedness and input-equivalen ce between channels,

R. Nasser, “On the input-degradedness and input-equivalen ce between channels,” in 2017 IEEE International Symposium on Information Theory (ISIT) , pp. 2453–2457, IEEE, 2017

2017
[104]

M. Li, P. Vitányi, et al. , An introduction to Kolmogorov com- plexity and its applications , vol. 3. Springer, 2008

2008
[105]

dit: a Pytho n package for discrete information theory,

R. G. James, C. J. Ellison, and J. P. Crutchﬁeld, “dit: a Pytho n package for discrete information theory,” The Journal of Open Source Software, vol. 3, no. 25, p. 738, 2018

2018
[106]

Exploration of synergistic and redundant in - formation sharing in static and dynamical Gaussian systems ,

A. B. Barrett, “Exploration of synergistic and redundant in - formation sharing in static and dynamical Gaussian systems ,” Physical Review E, vol. 91, May 2015. arXiv: 1411.2832

2015 arXiv
[107]

The parma polyhedr a library: Toward a complete set of numerical abstractions fo r the analysis and veriﬁcation of hardware and software systems,

R. Bagnara, P. M. Hill, and E. Zaﬀanella, “The parma polyhedr a library: Toward a complete set of numerical abstractions fo r the analysis and veriﬁcation of hardware and software systems, ” Science of Computer Programming , vol. 72, no. 1-2, pp. 3–21, 2008. 17

2008
[108]

On extreme points of convex sets,

L. E. Dubins, “On extreme points of convex sets,” Journal of Mathematical Analysis and Applications, vol. 5, no. 2, pp. 237– 244, 1962

1962
[109]

On lossless coding with coded side in- formation,

D. Marco and M. Eﬀros, “On lossless coding with coded side in- formation,” IEEE Transactions on Information Theory , vol. 55, no. 7, pp. 3284–3296, 2009

2009
[110]

An elementary proof of Blackwell’s theorem,

M. Leshno and Y . Spector, “An elementary proof of Blackwell’s theorem,” Mathematical Social Sciences, vol. 25, no. 1, pp. 95– 98, 1992

1992
[111]

propagate

S. Boyd and L. Vandenberghe,Convex optimization. Cambridge university press, 2004. Appendix A: Computing I ⋆ ∩ Here we consider the optimization problem that deﬁnes our proposed measure of redundancy: I ⋆ ∩(X1; . . .; Xn /shortrightarrowY ) := max sQ|Y Is(Q; Y ) s .t. ∀i sQ|Y ...

2004
[112]

for some b ∈ B . Then, max sA′ |C E[u(A′, Y )|B = b] = ∑ y,c,a p(y|b)p(c|b)s∗ A′|C(a|c)u(a, y) = ∑ y p(y|b) [∑ c p(c|b)s∗ A′|C (a|c) ] u(a, y) ≤ max sA|B E[u(A, Y )|B = b], where in the last line we’ve used that the maximum on the RHS of Eq. (
[113]

We now prove the “only if”

must be larger than the value achieved by sA|B(a|b) = ∑ c p(c|b)s∗ A′|C (a|c). We now prove the “only if”. Our approach uses a Separation Theorem, similarly to the proof of Blackwell’s theorem foun d in [61]. First, deﬁne Ω to be the set of product distributions over Y × C of ...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Thus, the Blackwell property can be formalized in terms of I∩ as I(Y ; X1) = I∩(X1; X2 /shortrightarrowY ) iﬀ pX1|Y ≼ pX2|Y

relates the unique information to the redundancy via U (X1 \ X2 /shortrightarrowY ) = I(Y ; X1) − I∩(X1; X2 /shortrightarrowY ) . Thus, the Blackwell property can be formalized in terms of I∩ as I(Y ; X1) = I∩(X1; X2 /shortrightarrowY ) iﬀ pX1|Y ≼ pX2|Y . (12) Conversely, unique information can also be related to union information via U (X1 \ X2 /shortrig...

[2] [2]

Self-redundancy: I∩(X1 /shortrightarrowY ) = I(Y ; X1)

[3] [3]

Xn /shortrightarrowY ) ≤ I∩(X1

Monotonicity: I∩(X1; . . . ; Xn /shortrightarrowY ) ≤ I∩(X1; . . . ; Xn−1 /shortrightarrowY )

[4] [4]

size of the intersection

are positive. This measure can be written as I BROJA ∩ (X1; X2 /shortrightarrowY ) := Ip(Y ; X1) + Ip(Y ; X2) − Ip∗ (Y ; X1, X2), where Ip∗ (Y ; X1, X2) is deﬁned in terms of the optimization problem Ip∗ (Y ; X1, X2) = min sY X1X2 Is(Y ; X1, X2) s.t. sY X1 = pY X1 , sY X2 = pY X2 . In other words, Ip∗ (Y ; X1, X2) is the minimal mutual infor- mation that ...

[5] [5]

more informative

Existence: For any pY X1...Xn , there exists a channel sQ|Y such that sQ|Y ≼ pXi|Y for all i ∈ [1..n] and I∩(X1; . . . ; Xn /shortrightarrowY ) = Is(Y ; Q). While the Symmetry, Self-redundancy, and Monotonicity ax- ioms are the same as in the standard PID framework (see Sec- tion II B), the other two axioms require some explanation.Gar- bling Equality is ...

[6] [6]

Xn /shortrightarrowY ) is invariant to the per- mutation of X1,

Symmetry: I∩(X1; . . . ; Xn /shortrightarrowY ) is invariant to the per- mutation of X1, . . . , Xn

[7] [7]

Syn- ergy, redundancy, and multivariate information measures: an experimentalist’s perspective,

N. Timme, W. Alford, B. Flecker, and J. M. Beggs, “Syn- ergy, redundancy, and multivariate information measures: an experimentalist’s perspective,” Journal of Computational Neu- roscience, vol. 36, pp. 119–140, Apr. 2014

work page 2014

[8] [8]

Multivariate Mutual Information Inspired by Secre t- Key Agreement,

C. Chan, A. Al-Bashabsheh, J. B. Ebrahimi, T. Kaced, and T. Liu, “Multivariate Mutual Information Inspired by Secre t- Key Agreement,” Proceedings of the IEEE, vol. 103, pp. 1883– 1913, Oct. 2015

work page 1913