{"id":"8aeb0b86-5ed1-4b3f-bc6f-e1b61cf61b99","arxiv_id":"2602.15438","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Logit distance between two softmax models bounds their internal representational dissimilarity, while KL divergence does not, making logit-matching the right objective for preserving linear representational structure in distillation.","lead":"The paper proves a mathematical guarantee: when two neural classifiers make predictions that differ only slightly in their raw scores (logits), their internal representations are guaranteed to be nearly linearly related. This tells practitioners that if you want a small student model to truly copy a teacher's way of thinking, you should match logits, not just output probabilities.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The d_rep guarantee (Thm 3.9) relies on Assumption 3.5 and the smallest singular value σ_min of all shifted-unembedding matrices, which is likely near-zero for trained classifiers; the paper does not verify this, so the central bound may be vacuous in practice.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The weakest point is indeed Assumption 3.5. The paper's own limitation section admits this assumption is stronger than diversity, but it does not evaluate it empirically. The concrete test above would settle whether the d_rep bound has any force in the reported experiments. If σ_min is tiny, then the theoretical guarantee does not apply or is vacuous, and the paper's practical claim rests on the experiments alone. This does not falsify the theorems, but it weakens the central motivation. I therefore recommend keeping the verdict as CONDITIONAL, with a request that the authors report σ_min (and μ_m) and verify general position on their trained models.","tokens_in":42885,"tokens_out":12023,"duration_ms":118504,"concrete_test":"Run the released code on the SUB dataset (or CIFAR-100) and, for the trained teacher and L1 student checkpoints, compute the minimum singular value σ_min over all pivots and a large random sample (e.g., 10^4) of m-subsets of the shifted-unembedding matrices \\tilde{L}_J (Eq. 4). Also compute the right-hand side of Thm 3.9, C·d_logit/σ_min. If σ_min is below 1e-3 or the bound exceeds the measured d_rep by more than an order of magnitude, the theorem is vacuous in this setting; this would show that the paper's quantitative representation guarantee does not explain the empirical results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim that small logit distance implies small representational dissimilarity is split into two bounds. Theorem 3.4 (mCCA) depends only on the teacher's logit covariance μ_m>0, a comparatively mild diversity condition. However, Theorem 3.9 (d_rep) and Corollary 3.10 require Assumption 3.5: every m×m shifted-unembedding matrix \\tilde{L}_J must be invertible for every pivot \\tilde{y} and every m-subset J, and the bound scales as 1/σ_min where σ_min is the smallest singular value across all such matrices. For real classifiers, unembeddings are not random: they are learned and often occupy a lower-dimensional effective subspace or contain near-duplicate directions (e.g., CIFAR-100 with m=50 and k=100, SUB with m=10 and k=33). In that case σ_min can be arbitrarily small, making the d_rep bound vacuous; if any \\tilde{L}_J is exactly singular, d_rep is undefined and the theorem does not apply. The paper does not report σ_min for any of the trained teachers or students, so it is unknown whether the d_rep bound is operative in the experiments that support the headline claim. The experimental d_rep improvements for logit students could occur even if the theoretical bound is vacuous, leaving the central 'logit distance is the right loss' conclusion without its advertised proof support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies approximate versions of linear identifiability for softmax-type discriminative models p(y|x) ∝ exp(f(x)^T g(y)). It defines a logit distance d_logit, proves that it is a metric, shows that KL divergence both lower-bounds and (under a τ-lower-boundedness assumption) upper-bounds d_logit^2, proves an mCCA lower bound in terms of d_logit^2/(m μ_m), introduces a \"linear identifiability dissimilarity\" d_rep, and proves that d_rep is controlled by d_logit up to a 1/σ_min factor under a general-position assumption. It also gives a concept-transfer bound and reports distillation experiments on Synth, CIFAR-100, and SUB, comparing KL students with L1/L2 logit-distance students. The central claim is that small logit distance implies high linear representational similarity, and that this implication is not available with practically useful constants through KL divergence.","tokens_in":43304,"tokens_out":6506,"duration_ms":69724,"significance":"The paper addresses a real and timely question: whether the exact linear identifiability theorem for this model class is robust to distributional closeness. Theorem 3.4, if its hypotheses are met, is a nontrivial quantitative robustness statement under the comparatively mild condition μ_m > 0, and it applies to both embeddings and unembeddings. The appendix proofs are detailed and largely self-contained; I checked the algebra in the central bounds and it is correct under the stated assumptions. The paper also provides code and a clear experimental comparison. However, the headline d_rep guarantee (Theorem 3.9) is conditional on a much stronger general-position assumption and on an unverified smallest-singular-value constant. As written, the experiments do not establish that this particular theoretical bound is operative in the settings reported. The mCCA half of the paper is more robust, but the quantitative link to the experiments is also incomplete because μ_m is not reported.","major_comments":[{"comment":"The d_rep bound scales as C·d_logit/σ_min, where σ_min is the smallest singular value across every shifted-unembedding matrix ̃L_J for every pivot ̃y and every m-subset J. Assumption 3.5 requires all these matrices to be invertible. For the trained classifiers in the experiments (CIFAR-100 with m=50, k=100; SUB with m=10, k=33), unembedding vectors are learned, not random, and can be nearly collinear or confined to a lower-dimensional subspace. The paper reports d_rep values in Tables 1, 2, and 4 but never reports σ_min or checks Assumption 3.5. If any ̃L_J is singular, d_rep is undefined by Definition 3.7 and Theorem 3.9 does not apply; if σ_min is very small, the bound is vacuous. The random-subsampling procedure in §K.2 cannot certify all subsets. This is a load-bearing gap: the experimental d_rep improvements for L1/L2 students could occur even if the theoretical d_rep bound is vacuo","section":"§3.4, Assumption 3.5, Theorem 3.9"},{"comment":"Theorem 3.4 gives mCCA ≥ 1 − d_logit^2/(m μ_m), with μ_m the smallest nonzero eigenvalue of the logit covariance. The proof is correct under the full-rank assumptions in Appendix F. However, the paper never reports μ_m for any of the datasets. Without μ_m, the displayed experimental mCCA values cannot be compared with the theorem: d_logit may be large in absolute terms, and if μ_m is small the lower bound is negative. For the claim that the theory explains the observed mCCA improvements, please report μ_m (or a lower bound) together with d_logit in the extended tables, or state clearly when the bound is nontrivial.","section":"§3.2, Theorem 3.4, Appendix F.14"},{"comment":"The concept-preservation result Theorem 4.3 has a constant ∥A∥_op, where A expresses the concept weight matrix in the unembedding basis. The paper does not estimate ∥A∥_op or any related constant on SUB, so the quantitative bound is not connected to the reported Acc(C) improvements. The empirical difference between KL students and logit students on SUB is striking, but as presented it is evidence for the qualitative phenomenon rather than a validation of the specific bound. Please either estimate the relevant constants on the experimental models or explicitly delimit Theorem 4.3 as a qualitative statement.","section":"§5, Proposition 4.1, Theorem 4.3"}],"minor_comments":[{"comment":"There is an inconsistent use of 'Assumption' and 'Theorem' when referring to Assumptions 2.1, 3.2, and 3.5 (e.g., Corollary 3.6 says 'satisfy Thm. 3.5' for what is Assumption 3.5). Please standardize the cross-references.","section":"Throughout"},{"comment":"The display for d_rep^2 is malformed: the expectation and norm are missing delimiters, and the factor 1/(kJ) is ambiguous because J is used both as a label-set index and as the combinatorial count. Rename the count, e.g., N = binom(k−1,m), and add explicit parentheses.","section":"Definition 3.7"},{"comment":"The triangle-inequality proof contains typos: p(y_j|x') should be p_i(y_j|x), and some norm signs are missing in the algebra. The intended argument is clear, but the appendix should be checked carefully for these presentation errors.","section":"Appendix D.3, proof of Theorem D.8"},{"comment":"For CIFAR-100, d_rep is estimated by sampling 200 label subsets per pivot out of roughly 5×10^28 possible subsets. Since d_rep is an average over all pivots and all subsets, the estimate may have high variance, especially for the KL students with large reported standard deviations. Please report bootstrap confidence intervals or, where feasible, use a stratified sampling scheme.","section":"§K.2"},{"comment":"The statement that KL students score 'slightly above a random classifier (1/33 ≈ 0.03)' is correct (Acc(C) = 0.055), but the phrasing could be sharpened because 0.055 is also far from the teacher's 0.92. Consider reporting the chance level explicitly in the table or caption.","section":"§5.2 and Table 2"},{"comment":"The limitation paragraph for Assumption 3.5 says the assumption holds 'with high probability for random vectors in high dimension.' While true for random vectors, this does not address trained unembeddings. Please replace this argument with a concrete empirical check or a weaker alternative condition.","section":"§6, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper's algebra is largely sound and the mCCA theorem is a solid contribution. The main risk is that the d_rep half of the paper is presented as a quantitative guarantee but depends on an unverified and likely unrealistic general-position assumption. This is fixable within the manuscript's scope by reporting σ_min and μ_m, checking or relaxing Assumption 3.5, and adjusting the strength of the claims. I do not see a load-bearing error that would force rejection, but the current manuscript overstates the support for its headline conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: Theorem 3.4 is the real contribution. For the softmax family p(y|x) ∝ exp(f(x)^T g(y)), it shows small logit distance implies high mCCA between embeddings, with an explicit bound. That's new, clean, and independently meaningful. The d_rep dissimilarity and its bound (Thm 3.9) are more fragile: they require every shifted unembedding matrix to be invertible and scale with 1/σ_min. For real classifiers, σ_min is plausibly near zero, and the paper never reports it, so we can't tell whether the d_rep bound is operative in the distillation experiments. That's a gap the authors should fix, not a fatal flaw.\n\nWhat the paper does well: proofs are detailed and self-contained; I checked the algebra in the central bounds and it holds. The negative KL result from Nielsen et al. 2025 is cited properly, and this paper goes beyond by proving a positive control. The experiments are well designed: on Synth and SUB, logit-distilled students have mCCA ~0.99 vs ~0.6–0.8 for KL students, and concept recovery on SUB is dramatic. The KL-to-logit bound (Thm 3.3) is honestly acknowledged as practically vacuous; that's good scholarly behavior.\n\nSoft spots: the stress-test is right about σ_min. The paper should at least report σ_min for the trained models, or provide a relaxation of Thm 3.9 that doesn't require all subsets. Also, the 'KL is the wrong loss' slogan is stronger than the evidence: the KL baseline is not temperature-scaled, and a tempered KL (which approximates logit MSE up to scale) would be the natural control. The d_rep measure is somewhat engineered, but the mCCA bound is not circular, so I don't see a load-bearing circularity issue.\n\nBottom line: this deserves a serious referee. I'd conditionally accept with requests for (i) σ_min diagnostics, (ii) a temperature-scaled KL baseline, and (iii) a caveat about Assumption 3.5 in the abstract. For my own work, I'd cite the mCCA bound.","headline":"Small logit distance provably implies high mCCA (that's the core result); the d_rep bound needs spectral diagnostics, but the paper deserves a serious referee.","tokens_in":43818,"tokens_out":2653,"would_cite":true,"duration_ms":25074,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that small logit distance between softmax-style models forces high linear representational similarity, and that KL divergence cannot supply the same guarantee.","keywords":["logit distance","representational similarity","linear identifiability","knowledge distillation","canonical correlation analysis","softmax models","concept transfer","KL divergence"],"falsifier":"On a dataset with more labels than embedding dimension, train a teacher and a student with logit-distance distillation, then measure d_logit, mCCA, d_rep, and the smallest singular value σ_min of the teacher's shifted unembedding matrices. If the student's mCCA falls below the lower bound in Theorem 3.4, or d_rep exceeds C d_logit/σ_min, the central claim would be false; if σ_min is so small that the bound is vacuous in practice, the practical force of the claim would be unsupported.","tokens_in":42803,"feed_emoji":"📐","tokens_out":6267,"duration_ms":64275,"temperature":0.7,"pith_summary":"This paper asks whether two models that make almost the same predictions must also think almost alike. For the broad family of discriminative models where the output probability is a softmax over inner products f(x)^T g(y), exact identifiability already says equal distributions imply embeddings equal up to an invertible linear map. The paper shows this conclusion survives approximately when closeness is measured by logit differences: small average squared logit distance yields a quantitative lower bound on mean canonical correlation between representations, and an upper bound on a new dissimilarity measure that vanishes exactly on the linear equivalence class. The same guarantee cannot come from KL divergence: the paper proves KL can bound logit distance only through a factor that blows up when probabilities approach zero. That is why KL-based distillation can match a teacher's probabilities while destroying linearly encoded concepts, whereas logit-distance distillation preserves them.","feed_headline":"Logit matching provably preserves linear representations","feed_subtitle":"When two softmax models nearly agree on logits, their embeddings are close to an invertible linear map, unlike KL-based closeness.","key_machinery":"The central object is the logit distance d_logit^2 = E_x ||u(x)-u'(x)||^2, the mean squared difference in pre-softmax scores. Around it the paper builds the linear-identifiability machinery: two models (f,g) and (f',g') are considered equivalent when f = A f' and g = A^{-⊤}g', and the matrix A is recovered from shifted unembedding matrices whose columns are g(y)-g(pivot) for m labels. The smallest singular value σ_min of all such matrices converts logit closeness into representation closeness. The new dissimilarity d_rep averages the alignment error f(x) - A_J f'(x) over every pivot and every m-label subset, and it is zero exactly on the equivalence class.","core_discovery":"The paper proves two concrete inequalities. If d_logit^2 = E_x||u(x)-u'(x)||^2 is the mean squared difference in logits, then the mean canonical correlation between the two models' embedding spaces is at least 1 - d_logit^2/(m μ_m), where μ_m is the smallest nonzero eigenvalue of the logit covariance. The paper also defines a representation dissimilarity d_rep that is zero exactly when the models are related by the linear identifiability symmetry f = A f', g = A^{-⊤}g', and proves d_rep ≤ C d_logit/σ_min, where σ_min is the smallest singular value of the shifted unembedding matrices. The proof decomposes logit differences into a regression residual plus an alignment error, then uses von Neum","pith_inferences":["Editorial: the same bound suggests logit-distance matching as a generic representation-alignment objective for model stitching or cross-model transfer, not only for distillation; the paper does not discuss this use.","Editorial: Theorem 4.3 says concept degradation is controlled by the operator norm of the concept's unembedding coordinates; a testable prediction is that concepts aligned with low-variance unembedding directions will be the first to break under small d_logit.","Editorial: the general-position assumption could likely be relaxed to a lower-rank statement: when σ_min ≈ 0, only the span of the unembeddings is identifiable, and d_rep should be measured in that subspace. The authors flag the assumption but do not explore this weakening."],"forward_implications":["If two models have small d_logit and their unembedding matrices are well conditioned, their embedding spaces are nearly an invertible linear map apart, and linearly encoded concepts in one will be almost linearly recoverable in the other.","Logit-distance distillation should replace KL-based distillation whenever a teacher's linearly encodable attributes matter; the paper's experiments on synthetic and image data confirm this.","Matching KL can drive a student's representations arbitrarily far from the teacher's linear geometry while predictions look close; Theorem 3.3 shows why the KL route cannot give a practically useful bound.","For next-token predictors with very large vocabularies, the condition k > m+1 is naturally satisfied, so the guarantees extend to autoregressive language models.","Minimizing an L1 logit loss also drives d_logit down, so the framework justifies a simple, stable training objective with the same theoretical protection."],"fun_headline_variants":["Logit distance controls linear representation similarity","Logit gap bounds embedding similarity with proof","Logit closeness ensures linear representation match","Why logit matching beats KL for linear similarity","Tight logits imply linear probe recoverability"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The d_rep and concept-transfer bounds require general position—every set of m shifted unembedding vectors must be linearly independent—so if real unembeddings are nearly collinear or lie in a lower-dimensional subspace, σ_min approaches zero and those bounds become vacuous.","fun_headline_variants_meta":{"raw":{"variants":["Logit distance controls linear representation similarity","Logit gap bounds embedding similarity with proof","Logit closeness ensures linear representation match","Why logit matching beats KL for linear similarity","Tight logits imply linear probe recoverability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2140,"prompt_tokens":774,"completion_tokens":1366,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1300}},"tokens_in":518,"tokens_out":1366,"duration_ms":10137,"temperature":1.0,"reasoning_tokens":1300,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:51:54.710111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset with more labels than embedding dimension, train a teacher and a student with logit-distance distillation, then measure d_logit, mCCA, d_rep, and the smallest singular value σ_min of the teacher's shifted unembedding matrices. If the student's mCCA falls below the lower bound in Theorem 3.4, or d_rep exceeds C d_logit/σ_min, the central claim would be false; if σ_min is so small that the bound is vacuous in practice, the practical force of the claim would be unsupported.","supporting_citations":[],"review_version":1}