Pith. sign in

REVIEW 3 major objections 4 minor 20 references

Dependence Induced Representations

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For any regular D-loss, optimal features are composed from the HGR maximal correlation functions and a loss-dependent adapter.

desk verdict Solid central theorem on dependence-induced representations, but one false claim about -Hnest and too many omitted proofs; worth refereeing with revision. read the letter →

arxiv 2411.15328 v1 pith:GPYANTEQ submitted 2024-11-22 cs.LG stat.ML

Xiangxiang Xu , Lizhong Zheng This is my paper · ORCID
classification cs.LGstat.ML MSC 62B1094A1768T07
keywords dependenceinducedrepresentationsHGRmaximalcorrelationD-lossminimalsufficientstatisticsrepresentationlearningneuralcollapsecrossentropyfeatureadapters
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a broad family of loss functions, called D-losses, all produce optimal feature representations with one shared structure: the minimizer can always be written as $(f,g)=(\varphi\circ f^*, \psi\circ g^*)$, where $f^*,g^*$ are the Hirschfeld–Gebelein–Rényi (HGR) maximal correlation functions of the pair $(X,Y)$ and $\varphi,\psi$ are loss-dependent adapters. If this is true, representations learned by cross entropy, hinge loss, their regularized variants, and variational divergence objectives are not arbitrary: they are functions of the same dependence-induced backbone and differ only through small adapters. The paper also derives necessary and sufficient conditions for a representation to be dependence induced, connects these to minimal sufficient statistics, and presents a feature-adapter design that lets hyperparameters be tuned during inference. A reader should care because the result offers a common theoretical explanation for why very different losses often learn similar features, and it gives a statistical reading of neural collapse in classifiers.

What carries the argument

The carrying object is the D-loss axiomatization together with the modal decomposition of the canonical dependence kernel. The canonical dependence kernel is $i_{X;Y}(x,y)=P_{X,Y}(x,y)/(P_X(x)P_Y(y))-1$, and its singular value decomposition in function space yields singular functions $(f_i^*, g_i^*)$ with singular values $\sigma_i$; these are the HGR maximal correlation functions. A functional $\Gamma(f,g)$ is a D-loss when it is invariant under composing features with dependence-preserving transformations and satisfies $\Gamma(f|s,g|t)\le \Gamma(f,g)$ for every Markov chain $X-s(X)-t(Y)-Y$, where $f|s$ is the conditional expectation of $f$ given $s(X)$. These two axioms force the data-processing direction that lets a minimizer be replaced by its conditional expectation onto the minimal sufficient statistics $f^*(X),g^*(Y)$ without increasing the loss, which is the step that produces the composition structure.

What would settle it

Take a small finite joint distribution, such as binary $X$ and binary $Y$, choose features $f,g$ and nontrivial coarsenings $s(X),t(Y)$, and numerically compare $\Gamma(f,g)=E_X\log E_Y\exp(f(X)^T g(Y))$ with $\Gamma(f|s,g|t)$; if the latter is larger for any choice, Fact 1 is false, and the cross-entropy and neural-collapse conclusions built on it would not follow.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is Theorem 3: for any regular D-loss $L$, the minimum value $v_k(X,Y)$ is dependence induced, and there exist dependence-induced mappings $\varphi,\psi$ such that some optimal feature pair is exactly $(\varphi\circ f^*, \psi\circ g^*)$. Consequently the map from data to the argmin of a regular D-loss is a dependence learning algorithm: the learned features depend only on the $X$-$Y$ dependence structure, not on marginal noise. The paper characterizes D-losses axiomatically by invariance to dependence-preserving transformations and a data-processing inequality, and then verifies that cross entropy (after bias calibration), hinge loss (for balanced binary labels), regularized variants, and variational forms of $\phi$-divergences belong to the family. For strictly dependent pairs, including classification data with deterministic labels, this yields the neural-collapse phenomenon $f(X)=\varphi(y(X))$ as a consequence of dependence structure rather than a special property of the softmax loss.

Load-bearing premise

The load-bearing premise is the unproved Fact 1 in Section IV-B.1: the log-sum-exp functional $E_X\log E_Y\exp(f(X)^T g(Y))$ satisfies the D-loss data-processing inequality $\Gamma(f|s,g|t)\le \Gamma(f,g)$ for every Markov chain $X-s(X)-t(Y)-Y$; if that inequality fails, cross entropy is not a D-loss and the paper's headline classifier example loses its support.

Editorial extensions

If this is right

  • For every regular D-loss, the optimal features are abstractions of the HGR maximal correlation functions, which are minimal sufficient statistics; so the learned representation discards all non-dependence information.
  • Cross entropy and hinge loss, with the stated calibrations and regularization, are included in the family; their optimal features share the same $f^*,g^*$ and differ only in $\varphi,\psi$.
  • On any training set where the label is a deterministic function of the input, every regular D-loss collapses all examples with the same label to one representation, which reproduces the NC1 component of neural collapse.
  • The composition structure permits feature adapters: train $f^*,g^*$ once, then optimize $\varphi,\psi$ for a particular loss, and even index $\varphi,\psi$ by a hyperparameter so that regularization can be tuned at inference time.
  • Convex feature constraints and weight-decay-style regularization preserve the D-loss property, so constrained and regularized variants remain within the same framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Fact 1 is verified, then a practical prediction follows: softmax classifiers trained with weight decay on noisy labels are, at their optima, dependence learning algorithms, and the calibrated bias $\beta(Y)$, not the raw bias, is the quantity that should match across runs.
  • The adapter separation suggests a testable extension: for image or text data, training only small adapters on features from a model that approximates $f^*$ should recover most of a full-network retrain for any D-loss; this could be checked on existing benchmarks.
  • The axiomatic D-loss definition is not restricted to the losses named in the paper; contrastive and adversarial objectives that satisfy the same data-processing inequality would inherit the composition theorem, though the paper does not analyze them.
  • Because all D-loss optima factor through the same minimal sufficient statistics, the theory predicts that representations learned from different D-losses are informationally equivalent after calibration; mutual information between feature pairs across losses should be equal, up to estimation error.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies representation learning from paired variables (X,Y), defining 'dependence induced' representations as those invariant to dependence preserving transformations. It proves that the HGR maximal correlation functions are dependence induced and minimal sufficient (Theorem 1, Proposition 2), characterizes dependence learning algorithms (Theorem 2), and introduces a family of 'D-losses' whose minimizers factor through the maximal correlation functions (Theorem 3). The paper then shows that cross entropy and hinge loss (with appropriate calibrations) are D-losses, interprets neural collapse as a consequence of strict dependence, and proposes a feature-adapter framework with the nested H-score for learning the maximal correlation functions.

Significance. The central result, Theorem 3, is a clean, parameter-free characterization: for every regular D-loss, the optimal feature pair is a composition of a loss-dependent map with the HGR functions. This unifies seemingly different losses and connects representation learning to minimal sufficient statistics. The derivation is self-contained for the main theorem, and the cross-entropy and hinge examples are plausible. However, the paper contains a false claim in Section V-B that the negative nested H-score is a regular D-loss; this affects the proposed method for learning f* and g* but does not undermine Theorem 3. The paper also relies on several unproved assertions (Fact 1, Property 3, Proposition 1) that should be proved or cited. The overall contribution is significant and worth publishing after revision, but the incorrect nested H-score claim must be corrected.

major comments (3)
  1. [Section V-B, Eq. (21)-(22)] The statement 'It can be verified that -H_nest(f,g) is a regular D-loss' is false. For k=1, take X,Y with P(0,0)=P(1,1)=0.4 and P(0,1)=P(1,0)=0.1, f(X)=1-2X, g(Y)=1-2Y, and constant s,t. Then f|s=0 and g|t=0, so H(f|s,g|t)=0, while H(f,g)=0.1; hence -H(f|s,g|t)=0 > -H(f,g)=-0.1, violating the D-loss inequality (9). Since this claim is used to justify learning f*,g* by maximizing the nested H-score, the section needs a corrected construction or a different justification.
  2. [Section IV-B.1, Fact 1] Fact 1, which asserts that E_X[log E_Y[exp(f(X)^T g(Y))]] is a D-loss, is stated without proof. This fact is load-bearing because it places cross entropy in the D-loss family and supports the neural collapse interpretation. The statement is true (it can be proved by two applications of conditional Jensen), but the paper should include the proof or a reference. The same applies to Property 3 in Section IV-A, which is also stated without proof and is used to regularize D-losses.
  3. [Section III-A, Proposition 1] Proposition 1, which states that the CDK of transformed variables equals the original CDK at decoded values, is used in the proof of Theorem 1 but its proof is omitted. Since Theorem 1 is foundational for the argument, this lemma should be proved in the paper or given a precise reference.
minor comments (4)
  1. [Section II-B, Eq. (3)] The denominator in the definition of the CDK is written as P_X(y)P_Y(y); it should be P_X(x)P_Y(y).
  2. [Section III-A, Definition 2] The notation xi^{-1}, eta^{-1} for the decoding maps is abusive because xi and eta are defined on X x Z and Y x W, respectively; please clarify that these are selected right inverses.
  3. [Section IV-A, Property 4] The condition that gamma is 'convex with respect to both arguments' is ambiguous; please specify whether joint convexity is required for the Jensen argument.
  4. [Section V-B] The footnote states that the weights before each H-score can be arbitrary positive numbers, but Eq. (21) uses unit weights; please reconcile the notation or clarify that this is a special case.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: Theorem 3 follows from the D-loss axioms, and the self-citations are ancillary tools; unproved Fact 1 and the Section V-B -H_nest claim are correctness risks, not circular steps.

full rationale

The central derivation is self-contained. Theorem 3 is proven directly from Definition 4: for any minimizer (fhat,ghat), with S=f*(X),T=g*(Y), the proof uses property (9) to obtain L(fhat,ghat) >= L(fhat|f*,ghat|g*) and property (8) to rewrite the latter as L(phihat,psihat;P_S,T). Equality follows from minimality; regularity then forces fhat=phihat∘f* and ghat=psihat∘g*. No parameter is fitted to data, and the D-loss axioms do not explicitly mention f*,g*, so the composition conclusion is not self-definitional. The main unproved inputs are Fact 1 (Section IV-B.1), which asserts that E_X log E_Y exp(f(X)^T g(Y)) is a D-loss, and the assertion in Section V-B that -H_nest is a regular D-loss; both are antecedent or ancillary to Theorem 3, so their unproved status is a completeness/correctness risk, not a circular step. Self-citations to [3] and [8] provide modal decomposition, H-score, nesting, and CDK facts; these are standard tools or independently derivable statements, not the target conclusion. The neural-collapse discussion is keyed to the external reference [5]. Thus the central claim has independent mathematical content, with at most minor non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central derivation is a mathematical characterization; no free parameters are fitted. The main assumptions are finite-alphabet domain, a fixed choice of HGR functions in the presence of degenerate singular values, and two unproved facts (Fact 1 and Property 3) that connect specific loss families to the D-loss definition.

assumptions (4)
  • domain assumption Discrete finite alphabets with positive probability for all symbols
    Stated in footnote 1; all results restrict to finite alphabets, so functional SVD and entropy arguments are well defined.
  • domain assumption Modal decomposition of the canonical dependence kernel into orthonormal singular modes exists and the HGR maximal correlation functions are taken as a fixed canonical choice
    Used in Eq (4); repeated singular values are not discussed, so uniqueness of f*, g* is assumed without comment.
  • ad hoc to paper Fact 1: E_X log E_Y exp(f(X)^T g(Y)) is a D-loss
    Stated in Section IV-B.1 without proof; it is the key input showing cross entropy belongs to the D-loss family.
  • ad hoc to paper Strict convexity of regularization implies regularity of D-losses (Property 3)
    Property 3 is stated without proof and underlies the claim that weight decay converts a D-loss into a regular D-loss whose argmin is a dependence learning algorithm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dependence Induced Representations." pith.science (2026). https://pith.science/paper/GPYANTEQ

@misc{pith2026241115328,
  author       = {Pith},
  title        = {Pith review of: Dependence Induced Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPYANTEQ}},
  note         = {Machine review of arXiv:2411.15328}
}
read the original abstract

We study the problem of learning feature representations from a pair of random variables, where we focus on the representations that are induced by their dependence. We provide sufficient and necessary conditions for such dependence induced representations, and illustrate their connections to Hirschfeld--Gebelein--R\'{e}nyi (HGR) maximal correlation functions and minimal sufficient statistics. We characterize a large family of loss functions that can learn dependence induced representations, including cross entropy, hinge loss, and their regularized variants. In particular, we show that the features learned from this family can be expressed as the composition of a loss-dependent function and the maximal correlation function, which reveals a key connection between representations learned from different losses. Our development also gives a statistical interpretation of the neural collapse phenomenon observed in deep classifiers. Finally, we present the learning design based on the feature separation, which allows hyperparameter tuning during inference.

Figures

Figures reproduced from arXiv: 2411.15328 by the authors.

Figure 1
Figure 1. Feature learning by training adapters φ, ψ. The feature extractors f ∗ , g ∗ can be frozen pre-trained networks or non-trainable modules. The separation enables more efficient implementation of the learning procedures. In particular, when S, T have much simpler structures compared with original X, Y , we can use lightweight neural networks as the adapters φ, ψ, and the adapter training does not require back-propagat… view at source ↗
Figure 2
Figure 2. The feature adapters φ (λ) , ψ(λ) are parameterized by the hyper￾parameter λ, which is tunable during inference. From Theorem 3, we have (f (λ) , g(λ) ) = (φ (λ) ◦f ∗ , ψ(λ)◦ g ∗ ). Note that only the adapters (φ (λ) , ψ(λ) ) depend on the λ, and can be learned by minimizing Lλ(φ (λ) , ψ(λ) ; PS,T ). Therefore, it suffices to learn feature adapters parameterized by λ. Specifically, we define the loss L(φ, ψ; PS,T ) … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 15 canonical work pages

  1. [1]

    Deep learning,

    Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015

    2015

  2. [2]

    Statistical modeling: The two cultures (wi th comments and a rejoinder by the author),

    L. Breiman, “Statistical modeling: The two cultures (wi th comments and a rejoinder by the author),” Statistical science, vol. 16, no. 3, pp. 199–231, 2001

    work page 2001

  3. [3]

    Neural feature learning in function s pace,

    X. Xu and L. Zheng, “Neural feature learning in function s pace,” Journal of Machine Learning Research , vol. 25, no. 142, pp. 1–76, 2024

    2024

  4. [4]

    Position: The Pl atonic Representation Hypothesis,

    M. Huh, B. Cheung, T. Wang, and P . Isola, “Position: The Pl atonic Representation Hypothesis,” in F orty-first International Conference on Machine Learning

    work page

  5. [5]

    Prevalence of neural collapse during the terminal phase of deep learning training,

    V . Papyan, X. Han, and D. L. Donoho, “Prevalence of neural collapse during the terminal phase of deep learning training,” Proceedings of the National Academy of Sciences , vol. 117, no. 40, pp. 24 652–24 663, 2020

    2020

  6. [6]

    The lattice theory of information,

    C. Shannon, “The lattice theory of information,” Transactions of the IRE professional Group on Information Theory , vol. 1, no. 1, pp. 105– 107, 1953

    work page 1953

  7. [7]

    On the forms of mental representation,

    H. A. Simon, “On the forms of mental representation,” in Perception and Cognition, W. Savage, Ed. University of Minnesota Press, 1978, pp. 3–18

    work page 1978

  8. [8]

    Universal features for high-dimensional learning and inf erence,

    S.-L. Huang, A. Makur, G. W. Wornell, and L. Zheng, “Universal features for high-dimensional learning and inf erence,” F oundations and Trends® in Communications and Information Theory, vol. 21, no. 1-2, pp. 1–299, 2024. [Online]. Available: http://dx.doi.org/10.1561/0100000107

    doi:10.1561/0100000107 2024

Show all 20 references
  1. [9]

    A connection between correlation and contingency,

    H. O. Hirschfeld, “A connection between correlation and contingency,” in Proceedings of the Cambridge Philosophical Society , vol. 31, no. 4, 1935, pp. 520–524

    1935

  2. [10]

    Das statistische problem der korrelatio n als variations- und eigenwertproblem und sein zusammenhang mit der ausglei ch- srechnung,

    H. Gebelein, “Das statistische problem der korrelatio n als variations- und eigenwertproblem und sein zusammenhang mit der ausglei ch- srechnung,” ZAMM-Journal of Applied Mathematics and Mechan- ics/Zeitschrift f¨ ur Angewandte Mathematik und Mechanik , vol. 21, no. 6, pp. 364...

    1941

  3. [11]

    On measures of dependence,

    A. R´ enyi, “On measures of dependence,” Acta Mathematica Academiae Scientiarum Hungarica , vol. 10, no. 3-4, pp. 441–451, 1959

    1959

  4. [12]

    Elements of information th eory (wiley series in telecommunications and signal processing),

    T. M. Cover and J. A. Thomas, “Elements of information th eory (wiley series in telecommunications and signal processing),” 200 6

  5. [13]

    Hastie, R

    T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Fried man, The elements of statistical learning: data mining, inference, and prediction. Springer, 2009, vol. 2

    2009

  6. [14]

    S. P . Boyd and L. V andenberghe, Convex optimization. Cambridge university press, 2004

    2004

  7. [15]

    Support-vector networks,

    C. Cortes and V . V apnik, “Support-vector networks,” Machine learn- ing, vol. 20, no. 3, pp. 273–297, 1995

    1995

  8. [16]

    Estimati ng diver- gence functionals and the likelihood ratio by convex risk mi nimiza- tion,

    X. Nguyen, M. J. Wainwright, and M. I. Jordan, “Estimati ng diver- gence functionals and the likelihood ratio by convex risk mi nimiza- tion,” IEEE Transactions on Information Theory , vol. 56, no. 11, pp. 5847–5861, 2010

    2010

  9. [17]

    Matryoshka representation learning,

    A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha , V . Ra- manujan, W. Howard-Snyder, K. Chen, S. Kakade, P . Jain et al. , “Matryoshka representation learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 30 233–30 249, 2022

    2022

  10. [18]

    Neural collapse wit h uncon- strained features,

    D. G. Mixon, H. Parshall, and J. Pi, “Neural collapse wit h uncon- strained features,” Sampling Theory, Signal Processing, and Data Analysis, vol. 20, no. 2, p. 11, 2022

    2022

  11. [19]

    A geometric analysis of neural collapse with unconstrained f eatures,

    Z. Zhu, T. Ding, J. Zhou, X. Li, C. Y ou, J. Sulam, and Q. Qu, “A geometric analysis of neural collapse with unconstrained f eatures,” Advances in Neural Information Processing Systems , vol. 34, pp. 29 820–29 834, 2021

    2021

  12. [20]

    Learn ing repre- sentations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learn ing repre- sentations by back-propagating errors,” nature, vol. 323, no. 6088, pp. 533–536, 1986

    1986

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.