Pith. sign in

REVIEW 2 major objections 3 minor 86 references

In prediction-only self-distillation, an affine mix of teacher and pure-distilled student attains strictly lower ridge risk than the teacher for almost every pair of regularization levels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Mixing a deployed teacher's predictions with a pure-distilled student trained on fresh unlabeled pseudo-labels generically lowers ridge prediction risk, and the optimal mix weight is learnable from a small labeled calibration set.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A serious random-matrix theory paper proving that fresh-X prediction mixing generically beats the teacher in ridge regression; the main caveat is that the ridge results require commuting teacher and fresh covariances, which the abstract's 'general anisotropic covariance' overstates. the 2 major comments →

arxiv 2607.15450 v1 pith:Z7KKSCLY submitted 2026-07-16 math.ST cs.LGstat.MLstat.TH

Prediction-Only Distillation in Linear and Logistic Regression

Hien Dang , Pratik Patil , Alessandro Rinaldo This is my paper
classification math.ST cs.LGstat.MLstat.TH MSC 62J0762F12
keywords prediction-only distillationself-distillationfresh-X distillationridge regressiondeterministic equivalentscalibrationlogistic regressionprediction mixing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies what happens when the teacher's training data are gone and only fresh, possibly out-of-distribution unlabeled covariates can be queried. It proves that, for high-dimensional ridge regression under proportional asymptotics, the optimally mixed predictor—an affine combination of the teacher and a student trained only on the teacher's pseudo-labels—has strictly smaller prediction risk than the teacher for almost every choice of the two ridge penalties, provided a mild nondegeneracy condition holds. The improvement persists even when the fresh covariates are isotropic Gaussians, and the precise risk is captured by closed-form deterministic equivalents. The paper also shows that the optimal mixing weight is not identifiable from unlabeled data alone but can be consistently estimated from a small independent labeled calibration set, and it extends the mixing argument to logistic regression, where mixing can recover perfect classification even under label noise above 50%.

Core claim

The load-bearing identity is the oracle decomposition R*_pmsd = R − (R−C)²/D, where R and R_pd are the teacher and pure-distilled risks and C is their residual correlation; since D>0, the mixed risk beats the teacher exactly when R−C≠0. Under the assumption that the teacher covariance and the fresh-covariate covariance are simultaneously diagonalizable, the authors express R−C as a finite sum of rational terms in κs, the student fixed-point parameter. This makes the tie condition R−C=0 a polynomial equation in κs of degree at most m−1, so the set of λs for which improvement fails is finite; the exceptional λt for which improvement fails for all λs is also finite (at most 4m−1). Strict improv

What carries the argument

The machinery is the proportional-asymptotics deterministic equivalent built from two resolvents Gt=(Σt+κt I)⁻¹ and Gs=(Σs+κs I)⁻¹, with κt and κs the fixed points of the standard ridge equations. All risk functionals, residual correlations, and the optimal mixing weight are expressed as explicit traces and quadratic forms in these resolvents. The tie analysis reduces to the rational function H(κs)=Σ_j d_j/(σ̃_(j)+κs), which controls all non-improvement cases, and the oracle identity ξ*=(R−C)/D connects this function to the optimal mixing weight.

Load-bearing premise

The ridge results require the teacher's covariate covariance and the fresh covariates' covariance to be simultaneously diagonalizable (i.e., commute), and the logistic claims require unit-norm features with a constant within-class correlation and zero cross-class Gram correlation.

What would settle it

Simulate ridge distillation with two non-commuting anisotropic covariances (e.g., an AR1 teacher covariance and a fresh covariance rotated by a random orthogonal matrix) and test whether the finite-sample optimal mixed risk is still strictly below the teacher risk for all but finitely many λs; also test the isotropic prediction that improvement vanishes exactly at λt = γtσ²/r². If either fails, the strict-improvement claim is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • In the fresh-X regime, prediction mixing turns any mistuned teacher into a better predictor for almost all student penalties, including when the fresh covariates are drawn from an isotropic distribution unrelated to the teacher's data.
  • The mixing weight can be chosen in one pass from a small calibration set: evaluate the two predictors, compute three averages, plug in—no refitting, no cross-validation over ξ.
  • The optimal mixed risk is unimodal in the fresh-sample aspect ratio, so more fresh data can hurt; in the isotropic same-λ case the maximum occurs at γs = 1 + λ.
  • In the same-λ balanced regime, fresh-X mixing recovers the same-X tangent identity, tying the sign of the mixing weight to the derivative of the teacher risk curve.
  • In logistic regression under the paper's assumptions, prediction mixing can achieve perfect population accuracy even when the pseudo-label noise rate exceeds 0.5, a regime where both teacher and pure-distilled students fail.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The commuting-covariance assumption is the main technical bridge to the polynomial argument; a natural testable extension is to check whether the generic strict-improvement claim survives near-commuting or mildly anisotropic OOD designs, where the finite-tie bound may be replaced by a measure-zero tie set.
  • The calibration result appears to transfer to any fixed pair of predictors whose residuals have finite moments; as a practical recipe, one can apply the same one-shot mixing to ensemble-style predictors, not only ridge students, and select among finitely many candidates with the same calibration data.
  • The non-identifiability result suggests that any fully unlabeled tuning rule must exploit additional structure (e.g., a known covariate distribution or symmetry) to approach the oracle; otherwise a labeled calibration set is quantitatively unavoidable.
  • The classification extrapolation with ξ>1 points to a more general phenomenon: mixing beyond the [0,1] convex hull amplifies a weak corrective signal in a student—this could serve as a design principle for recovering corrupted labels in other base learners.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper studies prediction-only self-distillation: a teacher ridge predictor is fixed, its original labeled data are unavailable, and a student is trained on fresh unlabeled covariates pseudo-labeled by the teacher; the final predictor is an affine combination of teacher and student predictions. Under proportional asymptotics and Assumption A (centered sub-Gaussian-type covariates, deterministic signal, and simultaneously diagonalizable teacher/fresh covariances), the paper derives deterministic equivalents for the teacher risk, the pure-distilled student risk, the teacher-student residual correlation, and the optimally mixed PMSD risk. It then shows that the limiting optimal PMSD risk is strictly smaller than the teacher risk for almost every pair of regularization levels, with the possible exception of finitely many values, and gives sharper results for isotropic, spiked, and monotone-spectrum settings. The paper also proves that the oracle mixing weight is not identifiable from unlabeled data alone, but can be consistently estimated from a small independent labeled calibration set, and it gives analogous prediction-mixing guarantees for binary logistic regression under a block-correlation feature model. The supplement contains proofs of all main results.

Significance. If the results are correct, the paper makes a substantial contribution to the theory of self-distillation. It moves beyond the same-X setting, where geometric identities cancel, and shows that fresh-X prediction mixing can generically improve a ridge teacher, even when the fresh covariates are isotropic and hence out-of-distribution in a strong sense. The explicit deterministic equivalents, the finite-exception strict-improvement bounds, and the one-shot calibration consistency result are concrete and falsifiable. The logistic results are more speculative because of the idealized feature correlation assumed, but they are proven and the extrapolation phenomenon (ξ>1 recovering 100% accuracy) is interesting. The supplement is carefully written and supplies derivations for the central formulas rather than leaving them as black boxes. The main weakness is that the advertised scope in the abstract is broader than the assumptions actually used.

major comments (2)
  1. [§3.3, Assumption A(c), Eq. (12), Propositions 4.1–4.2; abstract] The abstract and introduction advertise deterministic equivalents and strict-improvement guarantees under 'general anisotropic covariance' and for OOD fresh covariates. The theorems, however, require Σt and Σs to be simultaneously diagonalizable (Assumption A(c)). The deterministic equivalents in Lemma C.1 use insertions that must commute with Σt, and the spectral rewrite in Eq. (12) plus the finite-tie-set polynomial argument depend on a common eigenbasis; without Assumption A(c), Propositions 4.1 and 4.2 are not established for non-commuting fresh covariances. The real-data OOD experiments use Σs = Ip, which commutes with every Σt, so they do not test the general case. Please either extend the analysis to non-commuting Σs or rewrite the abstract/contributions so the commuting-covariance restriction is explicit.
  2. [§6, Assumption B; abstract] The logistic-regression claims in the abstract are unconditional, but Theorems 6.1–6.3 are proven only under Assumption B: unit-norm features, zero cross-class Gram correlation, and constant within-class correlation c. This is a strong idealized structure, and the proofs in Section F rely on it for the exact coefficient systems (68) and (73). The results are still interesting, but the abstract should either mention the assumption or state that the gains are established under this correlated-feature model.
minor comments (3)
  1. [§7.1] The claim that the analysis extends to negative regularization and that Propositions 4.1–4.2 hold regardless of signs is given without proof. If this is intended as a result, include the verification in the supplement; otherwise label it as a conjecture.
  2. [Theorem 6.3 / Section F.4] The notation λ_n is used inconsistently: in Theorems 6.1–6.2, λ_n = 2nλ is the scaled parameter, while Theorem 6.3 appears to use λ_n for the unscaled regularization sequence, and the proof in F.4 says λ_n = 2nλ_n with λ_n = Θ(1). Please clarify the scaling in the statement.
  3. [§4.3] The non-monotonicity formal result is stated only in Section H.1 of the supplement; the main text should either state the proposition or give a precise reference when claiming 'unimodal' behavior.

Circularity Check

0 steps flagged

No significant circularity: the fresh-X PMSD derivation is self-contained; the only self-citation (same-X teacher DE) is used as an external lemma with independent content.

full rationale

The central PMSD risk identity (10) is a direct consequence of minimizing a convex quadratic (Proposition 3.1), and the strict-improvement criterion R(λt) − C(λt, λs) ≠ 0 is then evaluated using newly derived deterministic equivalents for the fresh-X pure-distilled student and the teacher–student residual correlation (Theorem 3.2, Lemmas C.2–C.3). These DEs are not fitted to data, and no parameter is both fit and then renamed as a prediction. The only imported result is the same-X teacher-risk DE cited from Dang et al. (2026) in Lemma C.1 and Theorem 3.2; that result has its own stated assumptions, does not contain the fresh-X PMSD claim, and is used as a lemma rather than as the conclusion. The finite-tie-set arguments in Propositions 4.1–4.2 are polynomial-degree and sign-change arguments on the derived DE, not circular. The calibration estimator (Theorem 5.1) is a plug-in LLN estimator for fixed predictors, not a disguised fit. Assumptions A(c) and B restrict the scope of the 'general anisotropic' and logistic claims, but under-specification of scope is a correctness/robustness concern, not circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central ridge and logistic claims rest on standard random-matrix machinery and on two clearly stated distributional assumptions: commuting covariances for ridge, and the constant-correlation feature Gram matrix for logistic. There are no ad hoc fitted constants and no newly invented entities; the PMSD predictor is a post-hoc affine combination of existing predictors.

axioms (4)
  • domain assumption Random i.i.d. design with bounded 4+μ moments and deterministic positive definite covariances (Assumption A(a),(b),(d))
    Required for the anisotropic local laws and deterministic equivalents used in Theorem 3.2; the central ridge results depend on this.
  • domain assumption Covariances Σt and Σs are simultaneously diagonalizable, i.e. they commute (Assumption A(c))
    A common eigenbasis underlies the spectral rewriting of R−C in eq (12) and the tie-set bounds in Propositions 4.1 and 4.2. Without this, the general-anisotropic result is not established.
  • domain assumption Logistic feature map has unit norm, zero cross-class Gram correlation, and constant within-class correlation c (Assumption B)
    Theorems 6.1–6.3 are proven only under this idealized structure inherited from Das and Sanghavi (2023) and Jeong and Chung (2025).
  • standard math Standard random-matrix anisotropic local laws / resolvent deterministic equivalents (Rubio and Mestre 2011; Knowles and Yin 2017; Lemma C.5)
    The proofs invoke these background results to replace sample resolvents by deterministic equivalents; they are assumed rather than reproven.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Prediction-Only Distillation in Linear and Logistic Regression." pith.science (2026). https://pith.science/paper/Z7KKSCLY

@misc{pith2026260715450,
  author       = {Pith},
  title        = {Pith review of: Prediction-Only Distillation in Linear and Logistic Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7KKSCLY}},
  note         = {Machine review of arXiv:2607.15450}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.

Figures

Figures reproduced from arXiv: 2607.15450 by Alessandro Rinaldo, Hien Dang, Pratik Patil.

Figure 1
Figure 1. Figure 1: Schematic of the prediction-only setting and fresh- [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Strict improvement of the PMSD student. Test squared prediction risks of the ridge teacher (R, in blue), the fresh-X pure-distilled student (Rpd, in light blue), and the optimally mixed PMSD student (Rpmsd, in olive), plotted as functions of the teacher’s ridge penalty λt . The oracle optimal mixing weight ξ ⋆ is shown in red. Setting: UCI Blog Feedback data with nt = 2619, ns = 5240, and p = 280. We use t… view at source ↗
Figure 3
Figure 3. Figure 3: Setting: Σt = Σs and λt = λs = λ. Both panels use an AR1 covariance design, and the true signal β aligns with the top eigenvectors of Σt . We set p = 200, nt = ns = 400, and σ 2 = 1 in both panels; the right panel has r 2 = 1. Similar figures with Σt ̸= Σs and λt ̸= λs appear in Section I.3. Ze ∈ R ns×p has i.i.d. entries with mean 0, variance 1, and uniformly bounded (4 + µe)-th moment for some µ >e 0. (c… view at source ↗
Figure 4
Figure 4. Figure 4: PMSD using fresh covariates from an isotropic Gaussian distribution on real￾world regression tasks. At each value of λt , the student penalty λs is tuned separately over a grid for the PD and PMSD students. particular, R(λ) − C(λ, λ) = − κ 2b 2 t R′ (λ). (15) Hence, whenever R′ (λ) ̸= 0, R⋆ pmsd(λ, λ) < R(λ) and sign ξ ⋆ (λ, λ)  = − sign R′ (λ)  . The deterministic limiting risks and the sign rule for th… view at source ↗
Figure 5
Figure 5. Figure 5: Non-monotonicity of the PMSD risk with the amount of fresh unlabeled data in the [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: PMSD risk estimated using a small calibration set of size [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Fresh-X PMSD versus same-X loss-mixed SD. Setting: p = 400, nt = 200, ns = 4000, and Σt = Σs. The ns fresh unlabeled samples are used only by the fresh-X PD and PMSD students. For each value of |λt | on the x-axis, we evaluate both signs of the teacher regularization, tune λs over a grid on [−100, 100] in each case, and report the smaller risk. The signal is β = au1 + √ r 2 − a 2v, where a = 1.7, r = 2, u1… view at source ↗
Figure 8
Figure 8. Figure 8: Communities and Crime with nt = 400, ns = 800, and p = 99. The student regularization is fixed at λs = 0.1 for all values of λt . 10 2 10 1 10 0 10 1 10 2 Teacher ridge penalty t 0.4 0.5 0.6 0.7 0.8 Squared prediction risk Communities and Crime R Rpd Rpmsd 100 80 60 40 20 0 Optimal mixing parameter (a) Fresh unlabeled covariates are drawn from the same distribution as the teacher’s training covariates. 10 … view at source ↗
Figure 9
Figure 9. Figure 9: Communities and Crime with nt = 400, ns = 800, and p = 99. The student regularization is fixed at λs = 1 for all values of λt . 60 [PITH_FULL_IMAGE:figures/full_fig_p060_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Communities and Crime with nt = 400, ns = 800, and p = 99. At each value of λt , the student regularization λs is tuned separately over a grid for the PD and PMSD students. I.1.2 Blog Feedback 10 2 10 1 10 0 10 1 10 2 Teacher ridge penalty t 0.8 0.9 1 1.1 Squared prediction risk Blog Feedback R Rpd Rpmsd 600 500 400 300 200 100 0 Optimal mixing parameter (a) Fresh unlabeled covariates are drawn from the s… view at source ↗
Figure 11
Figure 11. Figure 11: Blog Feedback with nt = 2619, ns = 5240, and p = 280. The student regularization is fixed at λs = 0.1 for all values of λt . 61 [PITH_FULL_IMAGE:figures/full_fig_p061_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Blog Feedback with nt = 2619, ns = 5240, and p = 280. The student regularization is fixed at λs = 1 for all values of λt . 10 2 10 1 10 0 10 1 10 2 Teacher ridge penalty t 0.8 0.9 1 1.1 Squared prediction risk Blog Feedback R( t) Rpd( s ) Rpmsd ( t, s ) ( t, s ) 50 40 30 20 10 0 Optimal mixing parameter (a) Fresh unlabeled covariates are drawn from the same distribution as the teacher’s training covariate… view at source ↗
Figure 13
Figure 13. Figure 13: Blog Feedback with nt = 2619, ns = 5240, and p = 280. At each value of λt , the student regularization λs is tuned separately over a grid for the PD and PMSD students. 62 [PITH_FULL_IMAGE:figures/full_fig_p062_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Empirical and theoretical results when Σ [PITH_FULL_IMAGE:figures/full_fig_p067_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Empirical and theoretical results when Σ [PITH_FULL_IMAGE:figures/full_fig_p067_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Empirical and theoretical results when Σ [PITH_FULL_IMAGE:figures/full_fig_p068_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Comparison with the same-X SD student when p = 400, nt = 200, ns = 4000, and Σt = Σs follow an AR1 covariance model. At each value of |λt | on the x-axis, we evaluate the teacher regularizations λt and −λt , tune λs over a grid on [−100, 100] in each case, and report the lower risks. The signal is β = au1 + √ r 2 − a 2v, where a = 1.7, r = 2, u1 is the leading eigenvector of Σt , and v is orthogonal to u1… view at source ↗
Figure 18
Figure 18. Figure 18: Synthetic logistic regression with constant feature correlation [PITH_FULL_IMAGE:figures/full_fig_p069_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Synthetic logistic regression with constant feature correlation [PITH_FULL_IMAGE:figures/full_fig_p069_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Synthetic logistic regression with uniform feature correlation (defined in Section [PITH_FULL_IMAGE:figures/full_fig_p070_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Synthetic logistic regression with uniform feature correlation (defined in Section [PITH_FULL_IMAGE:figures/full_fig_p070_21.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references

  1. [1]

    International Conference on Machine Learning , year =

    Rudrajit Das and Sujay Sanghavi , title =. International Conference on Machine Learning , year =

  2. [2]

    Du and Sewoong Oh , title =

    Divyansh Pareek and Simon S. Du and Sewoong Oh , title =. Advances in Neural Information Processing Systems , year =

  3. [3]

    Emrullah Ildiz and Halil Alperen Gozeten and Ege Onur Taga and Marco Mondelli and Samet Oymak , title =

    M. Emrullah Ildiz and Halil Alperen Gozeten and Ege Onur Taga and Marco Mondelli and Samet Oymak , title =. International Conference on Learning Representations , year =

  4. [4]

    Advances in Neural Information Processing Systems , year =

    Behrad Moniri and Hamed Hassani , title =. Advances in Neural Information Processing Systems , year =

  5. [5]

    Advances in Neural Information Processing Systems , year =

    Adel Javanmard and Rudrajit Das and Alessandro Epasto and Vahab Mirrokni , title =. Advances in Neural Information Processing Systems , year =

  6. [6]

    Advances in Neural Information Processing Systems , year =

    Elvis Dohmatob and Yunzhen Feng and Julia Kempe , title =. Advances in Neural Information Processing Systems , year =

  7. [7]

    Bartlett , title =

    Hossein Mobahi and Mehrdad Farajtabar and Peter L. Bartlett , title =. Advances in Neural Information Processing Systems , year =

  8. [8]

    Tibshirani , title =

    Trevor Hastie and Andrea Montanari and Saharon Rosset and Ryan J. Tibshirani , title =. Annals of Statistics , number =. 2022 , url =

    2022

  9. [9]

    International Conference on Machine Learning , year =

    Tommaso Furlanello and Zachary Lipton and Michael Tschannen and Laurent Itti and Anima Anandkumar , title =. International Conference on Machine Learning , year =

  10. [10]

    2015 , url =

    Geoffrey Hinton and Oriol Vinyals and Jeff Dean , title =. 2015 , url =

    2015

  11. [11]

    Unifying distillation and privileged information , note =

    David Lopez-Paz and L. Unifying distillation and privileged information , note =. 2015 , url =

    2015

  12. [12]

    International Conference on Machine Learning , year =

    Mary Phuong and Christoph Lampert , title =. International Conference on Machine Learning , year =

  13. [13]

    International Conference on Learning Representations , year =

    Anvit Garg and Sohom Bhattacharya and Pragya Sur , title =. International Conference on Learning Representations , year =

  14. [14]

    I. S. Gradshteyn and I. M. Ryzhik , title =

  15. [15]

    Bartlett and Andrea Montanari and Alexander Rakhlin , title =

    Peter L. Bartlett and Andrea Montanari and Alexander Rakhlin , title =. Acta Numerica , pages =. 2021 , url =

    2021

  16. [16]

    Silverstein , title =

    Zhidong Bai and Jack W. Silverstein , title =

  17. [17]

    2022 , url =

    Linfeng Zhang and Chenglong Bao and Kaisheng Ma , title =. 2022 , url =

    2022

  18. [18]

    Maybank and Dacheng Tao , title =

    Jianping Gou and Baosheng Yu and Stephen J. Maybank and Dacheng Tao , title =. International Journal of Computer Vision , number =. 2021 , url =

    2021

  19. [19]

    Conference on Computer Vision and Pattern Recognition , year =

    Defang Chen and Jian-Ping Mei and Hailin Zhang and Can Wang and Yan Feng and Chun Chen , title =. Conference on Computer Vision and Pattern Recognition , year =

  20. [20]

    Advances in Neural Information Processing Systems , year =

    Jimmy Ba and Rich Caruana , title =. Advances in Neural Information Processing Systems , year =

  21. [21]

    Advances in Neural Information Processing Systems , year =

    Guangda Ji and Zhanxing Zhu , title =. Advances in Neural Information Processing Systems , year =

  22. [22]

    2023 , url =

    Collin Burns and Pavel Izmailov and Jan Hendrik Kirchner and Bowen Baker and Leo Gao and Leopold Aschenbrenner and Yining Chen and Adrien Ecoffet and Manas Joglekar and Jan Leike and Ilya Sutskever and Jeff Wu , title =. 2023 , url =

    2023

  23. [23]

    Advances in Neural Information Processing Systems , year =

    Guobin Chen and Wongun Choi and Xiang Yu and Tony Han and Manmohan Chandraker , title =. Advances in Neural Information Processing Systems , year =

  24. [24]

    International Conference on Computer Vision , year =

    Yuncheng Li and Jianchao Yang and Yale Song and Liangliang Cao and Jiebo Luo and Li-Jia Li , title =. International Conference on Computer Vision , year =

  25. [25]

    Lawrence and Zhenwen Dai , title =

    Sungsoo Ahn and Shell Xu Hu and Andreas Damianou and Neil D. Lawrence and Zhenwen Dai , title =. Conference on Computer Vision and Pattern Recognition , year =

  26. [26]

    Advances in Neural Information Processing Systems , year =

    Junnan Li and Ramprasaath Selvaraju and Akhilesh Gotmare and Shafiq Joty and Caiming Xiong and Steven Chu Hong Hoi , title =. Advances in Neural Information Processing Systems , year =

  27. [27]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , pages =

    Hengzhi He and Shirong Xu and Guang Cheng , title =. Journal of the Royal Statistical Society Series B: Statistical Methodology , pages =. 2026 , url =

    2026

  28. [28]

    Nature , number =

    Ilia Shumailov and Zakhar Shumaylov and Yiren Zhao and Nicolas Papernot and Ross Anderson and Yarin Gal , title =. Nature , number =. 2024 , url =

    2024

  29. [29]

    International Conference on Learning Representations , year =

    Elvis Dohmatob and Yunzhen Feng and Arjun Subramonian and Julia Kempe , title =. International Conference on Learning Representations , year =

  30. [30]

    International Conference on Learning Representations , year =

    Sina Alemohammad and Josue Casco-Rodriguez and Lorenzo Luzi and Ahmed Imtiaz Humayun and Hossein Babaei and Daniel LeJeune and Ali Siahkoohi and Richard Baraniuk , title =. International Conference on Learning Representations , year =

  31. [31]

    Model compression , booktitle =

    Cristian Bucilu. Model compression , booktitle =. 2006 , url =

    2006

  32. [32]

    Roberts and Diyi Yang and David L

    Matthias Gerstgrasser and Rylan Schaeffer and Apratim Dey and Rafael Rafailov and Tomasz Korbak and Henry Sleight and Rajashree Agrawal and John Hughes and Dhruv Bhandarkar Pai and Andrey Gromov and Daniel A. Roberts and Diyi Yang and David L. Donoho and Sanmi Koyejo , title =. Conference on Language Modeling , year =

  33. [33]

    2008 , url =

    Saverio De Vito , title =. 2008 , url =

    2008

  34. [34]

    2014 , url =

    Krisztian Buza , title =. 2014 , url =

    2014

  35. [35]

    2002 , url =

    Michael Redmond , title =. 2002 , url =

    2002

  36. [36]

    Conference on Computer Vision and Pattern Recognition , year =

    Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun , title =. Conference on Computer Vision and Pattern Recognition , year =

  37. [37]

    Annals of Statistics , number =

    Edgar Dobriban and Stefan Wager , title =. Annals of Statistics , number =. 2018 , url =

    2018

  38. [38]

    Proceedings of the National Academy of Sciences , number =

    Mikhail Belkin and Daniel Hsu and Siyuan Ma and Soumik Mandal , title =. Proceedings of the National Academy of Sciences , number =. 2019 , url =

    2019

  39. [39]

    Bartlett and Philip M

    Peter L. Bartlett and Philip M. Long and G. Benign overfitting in linear regression , journal =. 2020 , url =

    2020

  40. [40]

    Tibshirani , title =

    Pratik Patil and Alessandro Rinaldo and Ryan J. Tibshirani , title =. International Conference on Artificial Intelligence and Statistics , year =

  41. [41]

    Tibshirani , title =

    Pratik Patil and Yuting Wei and Alessandro Rinaldo and Ryan J. Tibshirani , title =. International Conference on Artificial Intelligence and Statistics , year =

  42. [42]

    Advances in Neural Information Processing Systems , year =

    Pratik Patil and Jin-Hong Du , title =. Advances in Neural Information Processing Systems , year =

  43. [43]

    Trevor Hastie and Robert Tibshirani , title =

  44. [44]

    Stein , title =

    Charles M. Stein , title =. Annals of Statistics , number =. 1981 , url =

    1981

  45. [45]

    Journal of the American Statistical Association , number =

    Bradley Efron , title =. Journal of the American Statistical Association , number =. 1983 , url =

    1983

  46. [46]

    Journal of the American Statistical Association , number =

    Bradley Efron , title =. Journal of the American Statistical Association , number =. 1986 , url =

    1986

  47. [47]

    Probability Theory and Related Fields , number =

    Antti Knowles and Jun Yin , title =. Probability Theory and Related Fields , number =. 2017 , url =

    2017

  48. [48]

    Journal of Machine Learning Research , number =

    Edgar Dobriban and Yue Sheng , title =. Journal of Machine Learning Research , number =. 2020 , url =

    2020

  49. [49]

    Journal of Machine Learning Research , number =

    Pratik Patil and Jin-Hong Du and Arun Kumar Kuchibhotla , title =. Journal of Machine Learning Research , number =. 2023 , url =

    2023

  50. [50]

    2022 , url =

    Pratik Patil and Arun Kumar Kuchibhotla and Yuting Wei and Alessandro Rinaldo , title =. 2022 , url =

    2022

  51. [51]

    Statistics and Probability Letters , number =

    Francisco Rubio and Xavier Mestre , title =. Statistics and Probability Letters , number =. 2011 , url =

    2011

  52. [52]

    Tibshirani , title =

    Pratik Patil and Jin-Hong Du and Ryan J. Tibshirani , title =. International Conference on Machine Learning , year =

  53. [53]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , number =

    Kamiar Rahnama Rad and Arian Maleki , title =. Journal of the Royal Statistical Society Series B: Statistical Methodology , number =. 2020 , url =

    2020

  54. [54]

    International Conference on Machine Learning , year =

    Alexander Wei and Wei Hu and Jacob Steinhardt , title =. International Conference on Machine Learning , year =

  55. [55]

    Journal of Machine Learning Research , number =

    Qiyang Han and Xiaocong Xu , title =. Journal of Machine Learning Research , number =. 2026 , url =

    2026

  56. [56]

    Bellec and Jin-Hong Du and Takuya Koriyama and Pratik Patil and Kai Tan , title =

    Pierre C. Bellec and Jin-Hong Du and Takuya Koriyama and Pratik Patil and Kai Tan , title =. Journal of the Royal Statistical Society Series B: Statistical Methodology , number =. 2025 , url =

    2025

  57. [57]

    Bellec , title =

    Takuya Koriyama and Pratik Patil and Jin-Hong Du and Kai Tan and Pierre C. Bellec , title =. Annals of Statistics , number =. 2026 , url =

    2026

  58. [58]

    Journal of Statistical Mechanics: Theory and Experiment , number =

    Theodor Misiakiewicz and Andrea Montanari , title =. Journal of Statistical Mechanics: Theory and Experiment , number =. 2024 , url =

    2024

  59. [59]

    Advances in Neural Information Processing Systems , year =

    Nilesh Tripuraneni and Ben Adlam and Jeffrey Pennington , title =. Advances in Neural Information Processing Systems , year =

  60. [60]

    Advances in Neural Information Processing Systems , year =

    Will Stephenson and Zachary Frangella and Madeleine Udell and Tamara Broderick , title =. Advances in Neural Information Processing Systems , year =

  61. [61]

    International Conference on Machine Learning , year =

    Hien Dang and Pratik Patil and Alessandro Rinaldo , title =. International Conference on Machine Learning , year =

  62. [62]

    Le , title =

    Qizhe Xie and Minh-Thang Luong and Eduard Hovy and Quoc V. Le , title =. Conference on Computer Vision and Pattern Recognition , year =

  63. [63]

    International Conference on Learning Representations , year =

    Hyeonsu Jeong and Hye Won Chung , title =. International Conference on Learning Representations , year =

  64. [64]

    Advances in Neural Information Processing Systems , year =

    Kaito Takanami and Takashi Takahashi and Ayaka Sakata , title =. Advances in Neural Information Processing Systems , year =

  65. [65]

    Kakade and Karthik Sridharan and Ambuj Tewari , title =

    Sham M. Kakade and Karthik Sridharan and Ambuj Tewari , title =. Advances in Neural Information Processing Systems , year =

  66. [66]

    Annals of Statistics , number =

    Vladimir Koltchinskii and Dmitry Panchenko , title =. Annals of Statistics , number =. 2002 , url =

    2002

  67. [67]

    2024 , url =

    Yanke Song and Sohom Bhattacharya and Pragya Sur , title =. 2024 , url =

    2024

  68. [68]

    International Conference on Machine Learning , year =

    Neil Rohit Mallinar and Austin Zane and Spencer Frei and Bin Yu , title =. International Conference on Machine Learning , year =

  69. [69]

    Duchi and Percy S

    Yair Carmon and Aditi Raghunathan and Ludwig Schmidt and John C. Duchi and Percy S. Liang , title =. Advances in Neural Information Processing Systems , year =

  70. [70]

    Dhillon and Alessandro Epasto and Adel Javanmard and Jieming Mao and Vahab Mirrokni and Sujay Sanghavi and Peilin Zhong , title =

    Rudrajit Das and Inderjit S. Dhillon and Alessandro Epasto and Adel Javanmard and Jieming Mao and Vahab Mirrokni and Sujay Sanghavi and Peilin Zhong , title =. International Conference on Machine Learning , year =

  71. [71]

    Le , title =

    Hieu Pham and Zihang Dai and Qizhe Xie and Quoc V. Le , title =. Conference on Computer Vision and Pattern Recognition , year =

  72. [72]

    Solvable model for inheriting the regularization through knowledge distillation , booktitle =

    Luca Saglietti and Lenka Zdeborov. Solvable model for inheriting the regularization through knowledge distillation , booktitle =. 2022 , url =

    2022

  73. [73]

    International Conference on Machine Learning , year =

    Diyuan Wu and Lehan Chen and Theodor Misiakiewicz and Marco Mondelli , title =. International Conference on Machine Learning , year =

  74. [74]

    Brooks and D

    Thomas F. Brooks and D. Stuart Pope and Michael A. Marcolini , title =. 1989 , url =

    1989

  75. [75]

    2009 , url =

    Alex Krizhevsky , title =. 2009 , url =

    2009

  76. [76]

    2007 , url =

    Gregory Griffin and Alex Holub and Pietro Perona , title =. 2007 , url =

    2007

  77. [77]

    Byrd and Peihuang Lu and Jorge Nocedal and Ciyou Zhu , title =

    Richard H. Byrd and Peihuang Lu and Jorge Nocedal and Ciyou Zhu , title =. 1995 , url =

    1995

  78. [78]

    2025 , url =

    Rylan Schaeffer and Joshua Kazdan and Alvan Caleb Arulandu and Sanmi Koyejo , title =. 2025 , url =

    2025

  79. [79]

    2026 , url =

    Radu Lecoiu and Debarghya Mukherjee and Pragya Sur , title =. 2026 , url =

    2026

  80. [80]

    Advances in Neural Information Processing Systems , year =

    Denny Wu and Ji Xu , title =. Advances in Neural Information Processing Systems , year =

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.