{"id":"a8ec1148-5469-4419-a834-3c7eb36d7c79","arxiv_id":"2501.15301","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Information measures such as Wyner common information and the information bottleneck are unchanged when computed from sufficient feature representations, justifying modular estimation pipelines.","lead":"This paper proves that several information measures, including mutual information, common information, and the information bottleneck, can be computed from learned feature representations instead of raw data, as long as the features are exact sufficient statistics. It gives theoreticians and machine learning practitioners a guarantee that a modular two-step pipeline, learn features then estimate, does not lose information for these measures.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified: the central invariance theorems are internally sound; the omitted proofs are readily supplied.","rationale":"I read the paper as a set of conditional invariance theorems: if S=s(X) and T=t(Y) are exact sufficient statistics, then various information measures computed from (S,T) equal those computed from (X,Y). Under that assumption, I checked the main arguments. Proposition 1 correctly characterizes joint sufficiency via the density-ratio equality, which directly yields Corollary 1. Lemma 3, though unproved, is true by a one-line chain-rule argument, and it is used correctly in the proof of Theorem 3. The proof of Theorem 2 is the most intricate; its SVD expansion is valid for finite alphabets, the orthogonality relations justify dropping the cross terms involving residuals, and the conclusion that only singular modes with sigma_i=1 can appear in a common function follows from the nonnegative quadratic form in Eq. (25). I did not find a counterexample or a hidden assumption that would falsify the central claim. The reader's weakest assumption is the practical gap between exact sufficiency and learned features. I agree that this is a real limitation of the paper's stated motivation, but it does not undermine the conditional theorems themselves; the paper explicitly acknowledges this gap by citing prior work on learned features failing to carry the needed information. Therefore I see no reason to change the reader's conditional verdict. The missing proofs of Lemma 3 and Corollary 1 are minor exposition gaps rather than correctness risks.","tokens_in":10133,"tokens_out":24799,"duration_ms":225977,"concrete_test":"Independently re-derive the unstated Lemma 3 by expanding I(U;Y,Z|X) as I(U;Y|X)+I(U;Z|X,Y) and using the two stated Markov conditions; if the expansion does not yield I(U;Y,Z|X)=0, then Theorem 3's omitted step needs repair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. The paper's central claim is conditional on exact sufficiency X-S-T-Y, and under that condition the proofs go through. Proposition 1 supplies the density-ratio factorization needed for Corollary 1, and the proof of Theorem 3 relies on Lemma 3, which follows immediately from the chain rule: I(U;Y,Z|X)=I(U;Y|X)+I(U;Z|X,Y)=0 under the stated Markov conditions. The Gacs-Korner argument in Theorem 2 is also sound: since singular values satisfy sigma_i<=1, the quadratic form in Eq. (25) is a sum of nonnegative terms, forcing the claimed coefficient identities. The only genuine limitation is the one the paper itself flags: learned features are not guaranteed to be exactly sufficient, and no approximation bounds or finite-sample analysis is given for approximately sufficient features. This is a scope limitation for practical deployment, not an internal inconsistency in the mathematical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies when an information measure theta(X,Y) can be computed from learned feature representations s(X) and t(Y) without loss. Under the standing assumption that s(X) and t(Y) are sufficient statistics for X and Y, equivalently X-s(X)-t(Y)-Y, it proves four invariance results: mutual information and f-information satisfy I_f(X;Y)=I_f(S;T); Wyner's common information satisfies C(X,Y)=C(S,T); Gács-Körner common information is characterized by H(f_0^*(X),...,f_k^*(X)) with k the largest index such that sigma_k=1 in the modal decomposition; and the information bottleneck curve satisfies theta_{X,Y}(R)=theta_{S,T}(R) for all R>=0, with L*_IB(X,Y;beta)=L*_IB(S,T;beta) for all beta>0. The proofs use the canonical dependence kernel and its modal decomposition, together with several Markov-chain lemmas.","tokens_in":10318,"tokens_out":17663,"duration_ms":153930,"significance":"If the results hold, they give a clean conditional guarantee for modular, representation-based estimation of several information measures. The characterization of Gács-Körner common information through the singular modes with unit singular values is a nice connection. The proofs are mostly complete and self-contained, and the Markov-chain structure is made explicit. The main practical caveat, which the paper itself acknowledges, is that the guarantee is conditional on exact sufficiency; no approximation bounds or finite-sample statements are given for approximately sufficient learned features, so the advertised practical scope should be phrased carefully.","major_comments":[{"comment":"The lemma is stated with 'Proof: Omitted' and is subsequently used in the proof of Lemma 5 and in the proof of Theorem 3, so the information bottleneck invariance arguments depend on an unproved statement. The lemma is true, and a short proof can be supplied via the chain rule, but it should appear in the manuscript for the proof of Theorem 3 to be complete.","section":"Section V, Lemma 3"}],"minor_comments":[{"comment":"The denominator of the canonical dependence kernel is written as P_X(y)P_Y(y); it should be P_X(x)P_Y(y).","section":"Section II-A, Eq. (1)"},{"comment":"The proof of Corollary 1 is omitted with 'We omit the proof.' Since it follows immediately from Proposition 1, please add the two-line derivation so that the f-information invariance is fully supported.","section":"Section III-A, Corollary 1"},{"comment":"In the expansion of E[(f(X)-g(Y))^2], the term f_i^*(Y) should be f_i^*(X); the displayed formula currently misstates the argument.","section":"Section V-C, Eq. (25)"},{"comment":"The text reads 'invariance to the choices of sufficient statics'; 'statics' should be 'statistics'.","section":"Section IV"},{"comment":"The phrase 'mild assumptions' overstates the condition of exact sufficiency; consider wording that reflects that the theorems are conditional on s(X) and t(Y) being sufficient statistics, since approximate or learned features are outside the proven scope.","section":"Abstract and Introduction"}],"recommendation":"minor_revision","confidential_remarks":"For the editor: the mathematical claims appear correct, and the omitted proofs are short and readily supplied. The novelty is incremental relative to the authors' earlier modal-decomposition framework, but the unified treatment of the five information measures is a reasonable contribution. No further concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a sound, honestly scoped theory paper. The main new results are the Gács–Körner characterization via singular values of the canonical dependence kernel and the Wyner/IB invariance proofs; the mutual-information and f-information invariance is classical and the paper says so. The proofs are mostly rigorous, and the modal-decomposition machinery earns its keep.\n\nWhere it is good: Proposition 1 gives a clean equivalence among the two marginal sufficiency conditions, the joint Markov chain, and the density-ratio factorization. The proof of Theorem 2 is elegant: using σ_i ≤ 1 to force coefficient identities in the E[(f(X)-g(Y))^2] expansion is a nice trick, and the resulting characterization of CGK(X,Y) as H(f*_0(X),...,f*_k(X)) with k the largest index having σ_k=1 is genuinely new. The Wyner and IB proofs are longer but coherent, and the construction in Lemma 5 is effective.\n\nSoft spots, in proportion: the big one is the assumption. The theorems all assume S and T are exactly sufficient, i.e., X-S-T-Y. That is a strong condition for the motivating application of learned features; the paper cites [3] noting learned features can fail to carry the needed information, and then gives no approximation bounds or finite-sample statements when sufficiency holds only approximately. That limits practical applicability but does not undermine the internal math. The paper should be more explicit that the guarantees are conditional and that the design question—how to learn features that are approximately sufficient—remains open.\n\nTwo smaller issues: Lemma 3 is stated with \"Proof: Omitted\" but it is easily supplied by the chain rule under the given Markov conditions; Corollary 1's proof is also omitted but it is a direct consequence of Proposition 1. There are minor typos in Eq. (1) (the denominator should be PX(x)PY(y)) and Eq. (25) (f*_i(Y) should be f*_i(X)). None of these affect the main theorems.\n\nVerdict: this deserves a serious referee. The CGK spectral characterization in particular is a worthwhile contribution, and the paper is careful about what it claims. It needs minor revision to fill the omitted proofs and to soften the \"practical design\" language. I'd bring it to our reading group if we're doing an information-measures session, and I'd cite it if I work on feature-based estimation.","headline":"Sound and honestly scoped: exact sufficiency gives clean invariance theorems, with the Gács–Körner spectral characterization the real new money; the omitted proofs and approximation gap are fixable.","tokens_in":10788,"tokens_out":2994,"would_cite":true,"duration_ms":25204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A17","62B10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that any sufficient statistics s(X), t(Y) preserve a broad class of information measures, and it pins down Gács–Körner common information as the entropy of the singular functions with unit singular values.","keywords":["separable computation","sufficient statistics","information measures","canonical dependence kernel","modal decomposition","Wyner common information","Gács–Körner common information","information bottleneck"],"falsifier":"Take a small finite joint distribution satisfying sufficiency for a chosen $s(X)$, $t(Y)$, compute $C_{GK}$ both by brute-force enumeration of all functions $f,g$ with $P\\{f(X)=g(Y)\\}=1$ and by the formula $H(f_0^*(X),\\ldots,f_k^*(X))$ from the modal decomposition; a single distribution where they differ would refute Theorem 2. Separately, choose an intentionally non-sufficient feature map, such as one that merges two symbols with different conditional distributions of $Y$, and compute $I(X;Y)$ versus $I(s(X);Y)$; seeing the equality fail confirms the condition is doing the work.","tokens_in":9960,"feed_emoji":"📊","tokens_out":7037,"duration_ms":58752,"temperature":0.7,"pith_summary":"This paper asks when an information measure $\\theta(X,Y)$ between two variables can be computed from learned feature representations $s(X)$, $t(Y)$ instead of raw data. Its answer: if $s(X)$ and $t(Y)$ are sufficient statistics, so that $X-s(X)-t(Y)-Y$ forms a Markov chain, then $\\theta(X,Y)=\\theta(s(X),t(Y))$ for mutual information, all $f$-information measures, Wyner's common information, Gács–Körner common information, and Tishby's information bottleneck. The proofs go through the canonical dependence kernel and its singular-value (modal) decomposition, which also yields an explicit formula for Gács–Körner common information as the entropy of the singular functions whose singular value is 1. This matters because practical neural feature estimators can be interpreted as attempting to learn sufficient statistics, and the theorem supplies the condition under which such modular, feature-based estimation is exact rather than heuristic.","feed_headline":"Sufficient features preserve a whole family of information measures","feed_subtitle":"Mutual information, common information, and the bottleneck curve survive compression to sufficient features.","key_machinery":"The load-bearing object is the canonical dependence kernel (CDK), $i_{X;Y}(x,y)=\\frac{P_{X,Y}(x,y)}{P_X(x)P_Y(y)}-1$, together with its modal decomposition: a singular value decomposition $i_{X;Y}=\\sum_i \\sigma_i f_i^*(x)g_i^*(y)$, with $\\sigma_1\\ge\\sigma_2\\ge\\cdots>0$ and orthonormal singular functions. Two facts carry the argument: first, $S$ and $T$ are sufficient exactly when the density ratio of $(X,Y)$ factors through the density ratio of $(S,T)$, so the CDK is unchanged by reducing to sufficient features; second, the singular functions $f^*(X)$, $g^*(Y)$ are minimal sufficient statistics. The CDK also pinpoints the unit singular values: functions $f(X)$ and $g(Y)$ that coincide with probability 1 are exactly linear transforms of the singular functions with $\\sigma_i=1$, which yields the Gács–Körner entropy formula and the other invariance proofs.","core_discovery":"The central claim is the invariance theorem: for finite-alphabet $X,Y$, if $S=s(X)$ and $T=t(Y)$ are sufficient statistics in the sense $X-S-Y$ and $X-T-Y$ (equivalently $X-S-T-Y$), then $I(X;Y)=I(S;T)$, $I_f(X;Y)=I_f(S;T)$ for every $f$-information, $C(X,Y)=C(S,T)$ for Wyner's common information, $C_{GK}(X,Y)=C_{GK}(S,T)$ for Gács–Körner common information, $L^*_{IB}(X,Y;\\beta)=L^*_{IB}(S,T;\\beta)$ for every $\\beta>0$, and $\\vartheta_{X,Y}(R)=\\vartheta_{S,T}(R)$ for every $R\\ge 0$ for the information bottleneck. Along the way the paper characterizes Gács–Körner common information as $H(f_0^*(X),\\ldots,f_k^*(X))$, where $f_i^*$ are the left singular functions of the canonical dependence kernel and $k$ is the largest index with singular value $\\sigma_k=1$. The optimal auxiliary variables in Wyner's problem and in the bottleneck are shown to be functions of the sufficient features, which is what makes separable computation possible.","pith_inferences":["An extension the authors leave open is quantification: when features are only approximately sufficient, the equalities degrade, and the modal decomposition suggests a perturbation bound based on discarded singular modes with $\\sigma_i<1$. The paper itself stops at the exact case.","The Gács–Körner formula suggests a practical null hypothesis for learning common information: after estimating the CDK singular spectrum, the number of singular values at 1, together with $H$ of the corresponding $f_i^*$, is the quantity to test; finite-sample versions would require deciding how close to 1 counts as 1.","Because Proposition 1 phrases sufficiency purely in terms of the CDK, any information measure that is a function of the CDK may inherit the same separable-computation property; the paper demonstrates this for the listed measures but does not give a general criterion.","In continuous or weak-dependence settings, universal features have known analytic forms, so the same invariance might be derivable outside finite alphabets; the paper's proofs are restricted to finite alphabets."],"forward_implications":["Feature-based mutual information estimation is exact whenever the learned features are sufficient statistics; the estimator no longer needs access to raw $X$ and $Y$.","The optimal Wyner common-information channel $W$ can be restricted to depend on $(S,T)$, so common information can be computed from sufficient features without loss.","Gács–Körner common information is a spectral quantity: it is the entropy of the top segment of canonical features whose singular values equal 1.","For any $\\beta>0$ and any $R\\ge 0$, the information bottleneck Lagrangian optimum and the $\\vartheta(R)$ curve are invariant under sufficient-statistic reduction, so bottleneck computations can be carried out on features.","All $f$-information measures, including mutual information and divergences based on other convex functions with $f(1)=0$, are invariant under sufficient-statistic reduction."],"supporting_citations":[{"why":"Motivates the problem by showing prior learned-representation MI estimation is heuristic and can lose information, the gap this paper closes.","marker":"[3]"},{"why":"Supplies Proposition 2, that the modal-decomposition features $f^*(X)$, $g^*(Y)$ are minimal sufficient statistics, and the dependence-structure viewpoint.","marker":"[4]"},{"why":"Defines Wyner common information, the quantity whose invariance is Theorem 1.","marker":"[5]"},{"why":"Defines Gács–Körner common information, the quantity characterized in Theorem 2.","marker":"[6]"},{"why":"Defines the information bottleneck objective and its Lagrangian, whose invariance is Theorem 3.","marker":"[7]"},{"why":"Provides the universal-features framework relating information measures to $f^*(X)$, $g^*(Y)$ and to the canonical dependence kernel.","marker":"[8]"},{"why":"Introduced the modal decomposition and neural feature learning in function space used to learn $f^*$ and $g^*$.","marker":"[9]"},{"why":"Supplies the data processing inequality (Lemma 1) used in the mutual-information and bottleneck proofs.","marker":"[12]"}],"fun_headline_variants":["Sufficient statistics keep all major info measures exact","Exact info measures from sufficient feature compression","Separable info measures: one theorem for MI, common info, bottleneck","Sufficiency yields exact MI, f-info, and bottleneck values","Information measures survive compression to sufficient features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire set of equalities rests on the learned features being exactly sufficient statistics, so that $X-s(X)-t(Y)-Y$; if that Markov condition fails, the paper provides no guarantee and the equalities can break.","fun_headline_variants_meta":{"raw":{"variants":["Sufficient statistics keep all major info measures exact","Exact info measures from sufficient feature compression","Separable info measures: one theorem for MI, common info, bottleneck","Sufficiency yields exact MI, f-info, and bottleneck values","Information measures survive compression to sufficient features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1403,"prompt_tokens":902,"completion_tokens":501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":425}},"tokens_in":518,"tokens_out":501,"duration_ms":4594,"temperature":1.0,"reasoning_tokens":425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:25:37.635162+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small finite joint distribution satisfying sufficiency for a chosen $s(X)$, $t(Y)$, compute $C_{GK}$ both by brute-force enumeration of all functions $f,g$ with $P\\{f(X)=g(Y)\\}=1$ and by the formula $H(f_0^*(X),\\ldots,f_k^*(X))$ from the modal decomposition; a single distribution where they differ would refute Theorem 2. Separately, choose an intentionally non-sufficient feature map, such as one that merges two symbols with different conditional distributions of $Y$, and compute $I(X;Y)$ versus $I(s(X);Y)$; seeing the equality fail confirms the condition is doing the work.","supporting_citations":[{"cited_title":"Approximating mutual information of high-dimensional variables using le arned representations,","cited_arxiv_id":null,"evidence_quote":"Motivates the problem by showing prior learned-representation MI estimation is heuristic and can lose information, the gap this paper closes."},{"cited_title":"Dependence induced representations ,","cited_arxiv_id":null,"evidence_quote":"Supplies Proposition 2, that the modal-decomposition features $f^*(X)$, $g^*(Y)$ are minimal sufficient statistics, and the dependence-structure viewpoint."},{"cited_title":"The common information of two dependent rando m vari- ables,","cited_arxiv_id":null,"evidence_quote":"Defines Wyner common information, the quantity whose invariance is Theorem 1."},{"cited_title":"Common information is far less t han mutual information","cited_arxiv_id":null,"evidence_quote":"Defines Gács–Körner common information, the quantity characterized in Theorem 2."},{"cited_title":"Neural feature learning in function s pace,","cited_arxiv_id":null,"evidence_quote":"Introduced the modal decomposition and neural feature learning in function space used to learn $f^*$ and $g^*$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the data processing inequality (Lemma 1) used in the mutual-information and bottleneck proofs."}],"review_version":1}