Pith. sign in

REVIEW 4 major objections 6 minor 6 references

Information Theoretic Perspective on Representation Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that last-layer embedding reliability is governed by classical information-theoretic quantities: with a bijective input–output map, vanishing error requires embedding rate at least H(X), and no architecture can compensate f

desk verdict The paper's central capacity theorem (Theorem 8) is not proved as stated and contradicts its own noiseless result (Theorem 6) in the Y=X limit; the achievability proof solves a message-recovery problem, not the claimed source-covering problem. read the letter →

arxiv 2601.11334 v2 pith:WV27RDZP submitted 2026-01-16 cs.IT cs.LGmath.IT

Deborah Pereg , Michael Wand This is my paper · ORCID
classification cs.ITcs.LGmath.IT MSC 94A1594A1794A2968T07
keywords representationlearningembeddingrateinformationtheoryrate-distortionmutualneuralcollapseregressionShannon
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the reliability of a learned last-layer embedding—the representation a network produces just before the final output—is governed by classical information-theoretic limits, not by the architecture or training algorithm. For a bijective input–output mapping, a predictor with arbitrarily small error exists if and only if the embedding rate R = Qz/n (bits per input symbol) is at least the input entropy H(X); below that, error tends to one. In a perturbed or noisy setting, the maximum number of distinguishable signals the embedding can reliably represent is bounded by the mutual information I(X;Y), and in a compressed-output setting the minimal embedding rate is the rate-distortion function R(D). The paper also proves a source–channel separation inequality, R(D) ≤ I(X;Y) ≤ C, and argues that these limits are independent of the hypothesis class. If true, this converts 'how well can this model represent the data?' from a network property into a data property, with direct architectural implications.

What carries the argument

The central object is the embedding representation rate R = Qz/n with Qz = log2|Z|^q, the analogue of code rate in communication theory. The proofs adapt Shannon's achievability and converse arguments: the typical set and joint typicality give achievability, Fano's inequality gives the converse, and the rate-distortion covering argument gives the compressed-output bound. The embedding is treated as a codebook with one distinguishable point per resolvable input, and the input–output pair (or noisy input–clean output pair) as a source–channel pair. This machinery turns the question of how well a neural embedding can perform into a rate question set by the source entropy and mutual information.

What would settle it

On a synthetic ergodic source with known entropy H(X), train a regression model with last-layer embedding rate R just below and just above H(X) under a bijective ground truth; Theorem 6 predicts a sharp transition from test error ~1 to ~0, so any experiment showing reliable recovery for R < H(X) would refute the necessity claim.

Watch

Extended reading notes

Core claim

The central claim is that the embedding rate R = q log2|Z|/n — the log-size of the embedding space per input symbol — is the fundamental resource for last-layer representations. Theorem 6: for a bijective target, a predictor with vanishing error exists iff R ≥ H(X); conversely, R < H(X)−ε forces error to one. Theorem 8: with a noisy observation y of x, reliable representation is achievable iff R < I(X;Y), and the effective non-zero embedding support is below 2^{nI(X;Y)}. Theorem 12: for compressed output with distortion ≤ D, minimal embedding rate is R(D) = min I(X;V̂) subject to E[d]≤D. Theorem 13: R(D) ≤ I(X;V̂) ≤ I(X;Y) ≤ C. Proposition 3: class-collapsed representations are insufficient

Load-bearing premise

The achievability proofs assume that a neural network trained by the learning algorithm can realize the arbitrary typical-set code constructed in the proof; the paper does not establish that gradient-based training finds that code, and with a restricted hypothesis class (e.g., Lipschitz networks) the achievable-rate claims may fail.

Editorial extensions

If this is right

  • If the embedding rate R falls below H(X), no regression network with a bijective target can drive test error to zero, regardless of width, depth, or training method (Theorem 6).
  • In noisy settings, increasing the embedding dimension beyond the effective support set by I(X;Y) cannot add reliable discriminability; the embedding's useful capacity is capped by 2^{nI(X;Y)} (Theorem 8).
  • For compressed outputs, guaranteeing distortion ≤ D requires an embedding rate of at least R(D); any smaller rate forces the minimum achievable distortion to exceed D (Theorem 12).
  • The separation inequality R(D) ≤ I(X;Y) ≤ C implies that noise and compression combine additively: the information bottleneck is the observation channel, not the network (Theorem 13).
  • Class-collapsed representations (neural collapse) make regression impossible when the target varies within a class, because all within-class inputs map to the same embedding (Proposition 3).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit: the theorems are proved for stationary ergodic sources and bijective or surjective maps; extending the rate-distortion limit to non-ergodic or finite-sample settings (for example, natural language sequences) would require a finite-n formulation of R(D), which the paper does not give.
  • If the independence-of-hypothesis-class claim is taken at face value, it supplies a practical diagnostic: estimate H(X) or I(X;Y) from data, compute the last-layer embedding rate R, and flag models whose R is below the predicted bound—those should fail on any training objective.
  • A testable corollary of the framework is that an untrained or randomly initialized encoder should not achieve the bound, and that the onset of the transition should be visible as the embedding rate crosses the information-theoretic threshold, independent of architecture.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an information-theoretic framework for last-layer embeddings in regression tasks. It defines an embedding representation rate R = Q_z/n, a representation capacity C = max_{P_X} I(X;Y), and a representation rate-distortion function R(D). The central claims are: (i) for a bijective noise-free mapping, a predictor with vanishing error exists iff R ≥ H(X) (Theorem 6); (ii) in a noisy/degraded setting, vanishing error is achievable iff R < I(X;Y) (Theorem 8); (iii) for a compressed output, the minimal embedding rate satisfies Q_z ≥ n I(X;\hat V) (Theorem 12); and (iv) a unified source-channel separation bound R(D) ≤ I(X;Y) ≤ C (Theorem 13). The paper further claims these limits are independent of the hypothesis class (Remark 3). The proofs in Appendices A–D adapt classical source coding, channel coding, rate-distortion, and source-channel separation arguments.

Significance. If the theorems were correct, they would provide a clean information-theoretic characterization of when a last-layer embedding can reliably represent inputs. The paper has some strengths: it explicitly defines the operational quantities, attempts both achievability and converse arguments, and connects the analysis to known phenomena such as neural collapse and Lipschitz-based generalization. However, the central capacity theorem is not a theorem about the stated representation-learning problem. The achievability proof of Theorem 8 is the classical channel coding theorem for M equiprobable messages, not a statement about a fixed source distribution P_X that the predictor must cover. This is not a local gap: it contradicts the paper's own Theorem 6 in the noiseless case and undermines Theorem 13, which inherits the same message-set model. Because the main claimed contribution rests on this mismatch, the paper cannot be accepted in its present form.

major comments (4)
  1. [§5.2, Theorem 8 and Appendix B] The achievability proof proves a different operational statement. It draws M=2^{nR} messages uniformly, transmits codewords x^n(w), and decodes the message index w. But Theorem 8's predictor must have P_e→0 when y is drawn from P_Y|X(·|x) with x∼P_X. The typical support of P_X has about 2^{nH(X)} sequences, while the predictor has at most 2^{nR} output values. If R<H(X), even an arbitrary predictor can be correct on at most a fraction 2^{nR}/2^{nH(X)} of the source, so P_e→1, not 0. Taking Y=X and g=identity gives I(X;Y)=H(X), so Theorem 8 asserts P_e→0 for R<H(X), while Theorem 6's converse asserts P_e→1 for R<H(X)−ε. Thus Theorem 8 is internally inconsistent with Theorem 6. The correct statement of the proof is Shannon's channel coding theorem for M equiprobable messages, not a fixed-source representation theorem.
  2. [§5.1, Remark 3 and Appendices A–D] The claim 'there exists a trained predictor' is not derived anywhere. The achievability constructions are arbitrary assignments on typical sets; no hypothesis class, architecture, optimization procedure, or sample size is specified. This matters because Remark 3 asserts the bounds are independent of the hypothesis class. At most, the proofs show existence in the class of all functions from the typical set to the representation alphabet. Theorem 10 supplies only a Lipschitz-based generalization error bound; it does not show that a Lipschitz network can realize the code constructions. If the hypothesis class is restricted, achievability can fail even though the converses remain valid.
  3. [Appendix B, Eq (33)] The proof imposes a cost constraint Es(x,y)≤S_max and concludes C(S)=max_{P_X: Es≤S} I(X;Y). However, Theorem 8 is stated without any cost constraint. The proof never shows that the constraint is vacuous or explains how it is removed in the final statement. As written, the theorem and the proof concern different optimization problems. This is especially problematic because the capacity definition in Eq (17) is C=max_{P_X} I(X;Y), with no cost constraint.
  4. [§5.4, Theorem 13 and Eq (24)] The sandwich I(X,\hat V)<\tilde Q_z/n<I(X;Y) is not established. Equation (23) bounds R(D)≤I(X;\hat V)≤I(X;Y)≤C, but \tilde Q_z/n appears nowhere in the achievability or converse proofs of Appendix D. The theorem's condition R(D)<C is about two numerical quantities, not about the embedding's effective support. To justify Eq (24), the proof would need to show that a representation with effective support rate \tilde Q_z/n can realize, and only can realize, the source-channel code. This is exactly the point where the message-index model versus fixed-source-coverage gap reappears.
minor comments (6)
  1. [Remark 2 and Appendix A] The inequality 'd log_2 |V| ≥ 2nH(x)' should presumably be 'nH(X)' rather than '2nH(x)'.
  2. [Theorem 8 statement] The quantifier '∀y∼P_Y|X(·|x)' leaves x free; it should be read as 'for all x∼P_X and y generated according to P_Y|X(·|x)'.
  3. [Definition 14] In 'generate 2^{nR} code words x^n(w), w=1,2,...,n', the upper limit should be M=2^{nR}, not n. Also, the notation is confusing because w is used both for the message index and (elsewhere) for the source sequence.
  4. [Theorem 12] Equation (22) should specify that I(X,\hat V) is evaluated at the conditional distribution attaining the minimum in Eq (21). The same symbol R is used for the representation rate and the rate-distortion function R(D), which is confusing.
  5. [Appendix C] The event E3 is defined as 'E3 = (E3 ∪ E3)^c', which is circular and must be corrected. The definitions of E2 and E3 also appear to use complements inconsistently.
  6. [Example 1] The notation P Y n(y^n_i) is malformed, and H(E) should more precisely be H(E^n) (or the per-symbol entropy rate).

Circularity Check

4 steps flagged · score 7.0 of 10

Representation capacity and rate-distortion theorems are the classical information-theoretic results restated under new definitions; the central Theorem 8 achievability proves a different message-based statement.

  1. self definitional [Definition 9 / Eq. (17); Theorem 8; Appendix B]
    "Definition 9 (Representation Capacity) The representation capacity is C= max_{P_X(x)} \tilde Q_z = max_{P_X(x)} I(X;Y), where the maximum is taken over all possible input distributions P_X(x). ... Theorem 8: if R< I(X;Y) then there exists a trained predictor h_\Psi such that the probability of error P_e \to 0 ... Conversely, if P_e \to 0 ... then R< I(X;Y)."

    Def. 5 sets R=Q_z/n, so Q_z=nR. Definition 9 then declares the representation capacity to be max_{P_X} I(X;Y), exactly the mutual-information expression that Theorem 8 'proves' as the threshold R<I(X;Y). The theorem's achievability/converse is the standard Shannon channel-coding proof (following Kramer et al.), not a derivation from the representation-learning operational setting. The 'capacity' is therefore the defining quantity restated; no independent bound is established.

  2. renaming known result [Theorem 8 statement vs. proof in Appendix B]
    "The proof follows the outline of achievability of Shannon channel coding theorem (Kramer et al., 2008). We assume paired ground truth signals {x_i,v_i}_{i=1}^M, M=2^{nR}. Denote the original 'message' w, drawn from the index set {1,2,...,M} ... The model makes an error if \hat w is not the same as the index w that was 'transmitted'."

    The proof's formal object is an (M,n) codebook with M=2^{nR} equiprobable messages w, not the problem of vanishing error over x~P_X. The theorem's statement quantifies over y~P_{Y|X}(.|x) for source x, whose typical set has size ~2^{nH(X)}. With Y=X (noiseless), I(X;Y)=H(X); Theorem 8's condition R<H(X) would imply P_e->0, while Theorem 6's converse gives P_e->1 for R<H(X)-eps. Thus the proof establishes a different (narrower) message-recovery statement; the derived capacity is classical channel capacity imported under a mismatched rate definition.

2 more flagged steps
  1. renaming known result [Theorem 12, Eq. (21); Appendix C]
    "the rate-distortion function of a source X, and an output \hat V with distortion d(v,\hat v) R(D)= min_{P_{\hat V|X}:E[d(v,\hat v)]≤D} I(X,\hat V)=H(X)-max_{P_{\hat V|X}:E[d(v,\hat v)]≤D} H(X|\hat V) (21) is the minimum achievable rate."

    Eq. (21) defines R(D) exactly as the classical rate-distortion function (min over test channels with expected distortion ≤D of I(X;Vhat)). The theorem then states this quantity 'is the minimum achievable rate' and Appendix C follows the textbook rate-distortion achievability/converse from Kramer et al. The regression 'rate-distortion' theorem is the standard source-coding result with names changed (source X, reconstruction Vhat, embedding rate R); the only new claim Q_z≥nI is just R≥R(D).

  2. renaming known result [Theorem 13, Eq. (23); Appendix D]
    "A source with rate-distortion R(D) can be sent over a channel of capacity C and recovered with distortion D if and only if R(D)<C. ... R(D)≤I(X,\hat V)≤I(X;Y)≤C. (23) ... The proof follows similar outline to the proof of Source–Channel Separation Theorem (El Gamal and Kim, 2011)."

    Eq. (23) reproduces the source-channel separation inequality R(D)≤I(X;Vhat)≤I(X;Y)≤C, and Appendix D explicitly says it follows the Source–Channel Separation Theorem and then chains Theorems 12 and 8. Since those theorems are the classical rate-distortion and channel-coding theorems restated, the combined 'source-channel representation rate-distortion' is the classical separation theorem under new terminology.

full rationale

The central results are not derived from a new first-principles model of representation learning; they are the classical information-theoretic theorems with the vocabulary changed. Definition 9 literally equates representation capacity with max_{P_X} I(X;Y), and Theorem 8 then 'derives' R<I(X;Y) using the standard channel-coding proof. Similarly, Theorem 12 defines R(D) as the textbook rate-distortion function and Theorem 13 reproduces the source-channel separation theorem. Moreover, Theorem 8's achievability proof uses a codebook of M=2^{nR} equiprobable messages, which does not match the theorem's stated operational requirement of vanishing error over x~P_X; in the noiseless case it directly contradicts Theorem 6. This is more than a harmless resemblance: it shows the 'prediction' reduces by construction to a known result under a different problem statement. The paper does contain some independent content, e.g., Theorem 6's counting argument and Theorem 10's Lipschitz generalization bound, and there is no load-bearing self-citation chain. But because the flagship capacity/rate-distortion/separation claims are definitional renamings of Shannon theory, a circularity score of 7 is appropriate.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The paper introduces no fitted parameters and no new entities; its theorems rest on standard AEP/joint-typicality results and on domain assumptions (bijectivity, ergodicity, completeness of training data). The main conceptual entities ('representation rate', 'representation capacity') are re-labellings of classical quantities, so they carry no independent evidentiary weight.

assumptions (6)
  • standard math The AEP holds for stationary ergodic finite-alphabet sources (Theorems 1-2, after Breiman 1957/McMillan 1953).
    Used in all proofs to bound the size of the typical set A_n^epsilon(P_X).
  • domain assumption X, Y, V are stationary ergodic processes with a stationary coupling, and can be quantized to finite alphabets (Sec. 3.1).
    Underlies all entropy and mutual information definitions; excludes non-stationary or continuous unbounded data.
  • domain assumption The ground-truth mapping g: X^n → V^d is a deterministic bijection for Theorem 6 and Theorem 12, and x→v is bijective for Theorem 8.
    Required for the rate to be controlled by H(X); for many-to-one g the bound would involve H(V).
  • domain assumption Noisy inputs satisfy y = u(x) + e with y→x surjective and e independent (Sec. 5.2).
    Defines the channel P_{Y|X} for the capacity theorem.
  • domain assumption Training set is large enough (implicit in Sec. 5.3, 'we assumed that the training set is large enough').
    The coding theorems are asymptotic; finite-sample learnability is not addressed.
  • domain assumption F and G are unrestricted (arbitrary measurable functions) for achievability; Lipschitz continuity is assumed only for Theorem 10.
    This permits the standard random-coding argument but does not reflect realistic constrained hypothesis classes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Information Theoretic Perspective on Representation Learning." pith.science (2026). https://pith.science/paper/WV27RDZP

@misc{pith2026260111334,
  author       = {Pith},
  title        = {Pith review of: Information Theoretic Perspective on Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WV27RDZP}},
  note         = {Machine review of arXiv:2601.11334}
}
read the original abstract

An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting.

Figures

Figures reproduced from arXiv: 2601.11334 by the authors.

Figure 1
Figure 1. Illustration of the noise-free setting. where pixels are integers in the range [0, 255], represented in the embedding space in a vector z ∈ Z1024×1 represented in as a floating point with 32 bits, then each input pixel is represented by 128 bits in the embedding space which is a significantly larger number of bits. The Embedding Separation Distance When the model is a bijective function every input has a correspondi… view at source ↗
Figure 2
Figure 2. Illustration of a noisy environment problem setting. noise, such that generally y n = u(x n ) + e(x n ), where u(·) and e(·) are functions of x n . Indeed, most real-life practical signal-to-signal models assume an input measurement y that could be noisy and/or degraded, yet the output should correspond with the central (and/or average) clean signal, denoted here as x (e.g., EMG to speech (Hou et al., 2024), medical… view at source ↗
Figure 3
Figure 3. Illustration of a rate-distortion setting. Therefore, ∥G(F(yi)) − G(F(xi))∥ ≤ σKcKG. (19) And so the generalization error of the predictor hΨx with input y obeys, L(hΨx (y)) = E(y,v)∼PY,V ℓ [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 3 linked inside Pith

  1. [5]

    Tishby and N

    N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5,

    2015

  2. [2002]

    Luthra, T

    A. Luthra, T. Yang, and T. Galanti. Self-supervised contrastive learning is approximately supervised contrastive learning.arXiv preprint arXiv:2506.04411,

    arXiv

  3. [2015]

    Vershynin

    R. Vershynin. Memory capacity of neural networks with threshold and relu activations. arXiv preprint arXiv:2001.06938,

    arXiv 2001

  4. [2018]

    Nikolaou, T

    G. Nikolaou, T. Mencattini, D. Crisostomi, A. Santilli, Y. Panagakis, and E. Rodola. Lan- guage models are injective and hence invertible.arXiv preprint arXiv:2510.15511,

    arXiv

  5. [2023]

    URLhttps://arxiv.org/ abs/2302.08388. A. El Gamal and Y. H. Kim.Network information theory. Cambridge university press,

    arXiv

  6. [2024]

    Shwartz-Ziv and N

    R. Shwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via infor- mation.arXiv preprint arXiv:1703.00810,

    arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.