Generalized Mutual Information
Pith reviewed 2026-05-24 22:39 UTC · model grok-4.3
The pith
A family of generalized mutual information measures are finite for every joint distribution on a countable alphabet.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The article proposes a family of generalized mutual information all of whose members are finitely defined for each and every distribution of two random elements on a joint countable alphabet, except the one by Shannon, and enjoy all utilities of a finite Shannon's mutual information.
What carries the argument
The family of generalized mutual information functions, each constructed to remain finite on any countable joint support while coinciding in utility with Shannon's measure when the latter is finite.
If this is right
- Information quantities can be computed directly for heavy-tailed distributions where Shannon's mutual information diverges.
- The data processing inequality and chain rule apply without additional restrictions on the alphabet or tails.
- Statistical and communication models no longer require truncation or approximation to ensure finiteness of mutual information.
- All joint distributions on countable spaces become eligible for the same information-theoretic tools.
Where Pith is reading between the lines
- Similar generalizations could be attempted for other quantities such as conditional entropy or divergence that also become infinite on countable spaces.
- Empirical estimation procedures for these new measures might be developed and compared against Shannon-based estimators on finite samples from heavy-tailed data.
- The family might admit a parameterized form that recovers Shannon's measure in a suitable limit.
Load-bearing premise
The new measures satisfy the functional properties and utilities of Shannon's mutual information whenever the latter is finite.
What would settle it
A concrete joint distribution on a countable alphabet together with a calculation showing that one of the generalized measures violates the chain rule or data processing inequality while Shannon's mutual information remains finite.
read the original abstract
Mutual information is one of the essential building blocks of information theory. Yet, it is only finitely defined for distributions with fast decaying tails on a countable joint alphabet of two random elements. The unboundedness of mutual information over the general class of all distributions on a joint alphabet prevents its potential utility to be fully realized. This is in fact a void in the foundation of information theory that needs to be filled. This article proposes a family of generalized mutual information all of whose members 1) are finitely defined for each and every distribution of two random elements on a joint countable alphabet, except the one by Shannon, and 2) enjoy all utilities of a finite Shannon's mutual information.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a family of generalized mutual information (GMI) measures defined on pairs of random elements taking values in a countable joint alphabet. The central claims are that every member of this family (except Shannon mutual information itself) is finite for every joint distribution on the alphabet, and that the family members retain the standard functional properties and utilities of finite Shannon mutual information, including non-negativity, the chain rule, and the data processing inequality.
Significance. If the proposed family is shown to be finite everywhere and to satisfy the listed functional properties, the work would address a recognized limitation of Shannon mutual information for heavy-tailed distributions on countable alphabets and could extend the applicability of mutual-information concepts to a larger class of joint distributions.
major comments (1)
- The abstract asserts that the new family members 'enjoy all utilities of a finite Shannon's mutual information,' yet the provided text supplies neither the explicit definitions of the family nor any derivation showing that the data processing inequality or chain rule continue to hold. This assumption is load-bearing for the central claim and cannot be assessed from the given material.
Simulated Author's Rebuttal
We thank the referee for their review and for acknowledging the potential significance of addressing the unboundedness of Shannon mutual information on countable alphabets. We address the single major comment below.
read point-by-point responses
-
Referee: The abstract asserts that the new family members 'enjoy all utilities of a finite Shannon's mutual information,' yet the provided text supplies neither the explicit definitions of the family nor any derivation showing that the data processing inequality or chain rule continue to hold. This assumption is load-bearing for the central claim and cannot be assessed from the given material.
Authors: We acknowledge that the version provided to the referee did not include sufficient explicit definitions of the GMI family or the derivations of the chain rule and data processing inequality. In the revised manuscript we will add these elements: the formal definition of the family will be stated in Section II, and complete proofs that the listed functional properties hold (when the GMI is finite) will be supplied in Section III, making the central claims directly verifiable. revision: yes
Circularity Check
No significant circularity
full rationale
The paper introduces a family of generalized mutual information measures via explicit new definitions on countable alphabets. The central claims (finiteness for all joint distributions except Shannon's, and preservation of non-negativity/chain rule/DPI/etc. when Shannon MI is finite) are presented as properties of the proposed definitions themselves rather than as predictions derived from fitted parameters, self-citations, or ansatzes that reduce to the inputs. No load-bearing derivation step equates an output to its own construction by the paper's equations. The work is self-contained as a definitional proposal with stated functional requirements.
Axiom & Free-Parameter Ledger
axioms (1)
- standard math Standard axioms of probability measures and information functionals on countable alphabets
invented entities (1)
-
Generalized mutual information family
no independent evidence
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.