REVIEW 2 major objections 5 minor 1 cited by
Analysis of Overparameterization in Continual Learning under a Linear Model
T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Overparameterization alone can curb catastrophic forgetting
desk verdict Useful single-task bound; the forgetting-ratio theorem needs an extra regime condition before it is true. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a latent-space linear regression model in which observed features are noisy $p$-dimensional projections of $d$-dimensional latent features, with the idealized assumption $W^T W = p\gamma I_d$ making the feature columns pairwise orthogonal and equal in length. Training is characterized in closed form: from an initialization $\beta_0$, gradient descent on square loss converges to the minimum-norm interpolating solution $\beta_0 + A^T(AA^T)^{-1}(y - A\beta_0)$, so each estimator is an orthogonal projection of its initialization onto the affine solution set. Risk decomposes through the projection $P_{A^T}$ onto the row space of the data and the projection $P_W$ onto the column space of $W$; the proof bounds the relevant projection norms using non-asymptotic random matrix estimates (in particular singular value concentration bounds) and then combines a single-task risk bound with a terminal-forgetting bound from the authors' earlier work.
What would settle it
Compute the denominator $1 - 72\sqrt{d/n} - 18n/p$ for parameters allowed by the theorem, such as $d = n/2$, $p = 20n$, $\gamma = 1/\sqrt{nd}$; it is negative, so inequality (20) would assert that a nonnegative forgetting ratio is at most a negative number. Running the paper's two-task linear model at those parameters and measuring the forgetting ratio would settle whether the theorem's conclusion holds as stated.
Extended reading notes
Core claim
The paper's main theorem (Theorem 2) states that if $W^T W = p\gamma I_d$, $n \ge d$, $p \ge 20n$, and $\gamma \ge 1/\sqrt{nd}$, then with probability at least $1 - 20e^{-cd}$, the minimum-norm estimator for task A trained from zero has risk $R(\hat{\beta}_A) \le (72\sqrt{d/n} + 18n/p)\|\theta\|^2$, the estimator obtained by then training on task B from $\hat{\beta}_A$ has risk $R(\hat{\beta}_{BA}) \le (72\sqrt{d/n} + 96\sqrt{n/p})\|\theta\|^2$, and the forgetting ratio $(R(\hat{\beta}_{BA}) - R(\hat{\beta}_A))/(R(0) - R(\hat{\beta}_A))$ is at most $78\sqrt{n/p}/(1 - 72\sqrt{d/n} - 18n/p)$. The message is that if $d \ll n \ll p$, the first task is well-learned initially, remains well-learned after the second task, and the forgetting due to the second task is negligible compared with the learning gained from the first. The theorem formalizes the geometric picture that in the overparameterized regime the constant-risk ellipses for the two tasks become nearly orthogonal and highly elongated, so moving along task B's solution set barely changes risk on task A.
Load-bearing premise
The forgetting-ratio bound is proved by dividing by $1 - 72\sqrt{d/n} - 18n/p$, and the theorem's stated conditions ($n \ge d$, $p \ge 20n$, $\gamma \ge 1/\sqrt{nd}$) do not by themselves ensure this denominator is positive; if it is not positive, the bound as stated is not meaningful.
Editorial extensions
If this is right
- When $d/n$ and $n/p$ are small, the bounds show $R(\hat{\beta}_A)$ and $R(\hat{\beta}_{BA})$ are both small multiples of the null risk $\|\theta\|^2$, so task A is well-learned before and after task B.
- The forgetting ratio tends to $0$ as $p/n \to \infty$ with $d/n$ fixed, meaning overparameterization alone fully mitigates forgetting in this linear model.
- Because the single-task risk bound decreases monotonically in $p$, it provides a finite-sample demonstration of the double-descent phenomenon.
- The bounds hold even at the low signal-to-noise level $\gamma = 1/\sqrt{nd}$, so the qualitative conclusion does not require strong signal.
Reading between the lines
- A natural extension is whether the same qualitative picture holds when $W$ has independent Gaussian entries rather than the idealized orthogonal equal-length columns; if the projection arguments survive approximate orthogonality, similar risk bounds should hold with high probability.
- The geometric mechanism suggests a concrete neural-network analogue: increasing layer width should align the constant-risk surfaces of successive tasks and reduce forgetting even without explicit continual-learning algorithms, which could be checked by varying width in a permuted-label linear or two-layer network.
- The proof's reliance on closed-form minimum-norm interpolating solutions means the result likely extends to other interpolating algorithms, such as ridgeless regression from a generic initialization, but not directly to early-stopped or regularized training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies two-task continual learning in a linear regression model with latent-space data. Task A has Gaussian latent features mapped through a fixed matrix W (with W^T W = pγ I_d), and task B is obtained by applying a random orthogonal transformation to the observations while sharing the same labels. Training is by gradient descent with square loss from initialization 0 for task A and from the task-A solution for task B. The paper claims three high-probability non-asymptotic bounds: the single-task risk R(β_hat_A), the risk on task A after sequential training R(β_hat_BA), and the forgetting ratio (R(β_hat_BA)-R(β_hat_A))/(R(0)-R(β_hat_A)). The first two are proved in Theorems 8 and 10, and the third in Theorem 11. The advertised conclusion is that, when the latent dimension is small and the model is sufficiently overparameterized, task A is well-learned initially, remains well-learned after training on task B, and the forgetting is small relative to the initial learning.
Significance. If the main theorem is correct, the paper gives a clean formal demonstration of a phenomenon that has been observed empirically: overparameterization alone can mitigate catastrophic forgetting, at least in a simplified linear model with random orthogonal task transformations. The non-asymptotic single-task risk bound in Theorem 8 is also a useful contribution to the double-descent literature, since it extends the asymptotic analysis of Hastie et al. to finite-size problems. The proof strategy is transparent and uses standard random-matrix concentration tools. However, the central forgetting-ratio claim has a load-bearing correctness gap that must be resolved before the advertised conclusion can be accepted.
major comments (2)
- [Theorem 11, Eqs. (82)-(85)] The proof of Theorem 11 divides by 1 - 72√(d/n) - 18n/p, but the hypotheses n ≥ d, p ≥ 20n, γ ≥ 1/√(nd) do not ensure that this quantity is positive. For example, n = d and p = 20n give 72√(d/n) + 18n/p = 72.9, so the displayed denominator is -71.9. In Eq. (83), the denominator R(f0)-R(fβ_A) is bounded below by (1 - 72√(d/n) - 18n/p)||θ||²; if this lower bound is non-positive or negative, the division step is invalid and the inequality direction is not controlled. The theorem as stated is therefore not established. The fix is to add an explicit positivity condition, such as 72√(d/n) + 18n/p < 1, and to verify that the actual denominator is positive on that event. The same issue appears in the informal Eq. (3) and in Theorem 2, Eq. (20).
- [Appendix A, Theorem 9] Theorem 9, which bounds the terminal forgetting R(f_β_BA) - R(f_β_A), is stated without proof and attributed to Theorem F.11 of the authors' prior AISTATS paper. Since Theorem 10 and Theorem 11 both rely directly on this bound, the current manuscript is not self-contained at a load-bearing point. Please either reproduce the proof of Theorem 9 or restate it as an imported result with a complete statement of its hypotheses and a precise reference. This is not a circularity, but it is an omitted proof that the referee cannot verify from the submitted text alone.
minor comments (5)
- [General] The assumption numbering is inconsistent: it is Assumption 1 in Section 3 and Assumption 3 in Appendix A. Please renumber.
- [Theorem 8, Eq. (73)] The inequality (a+b)² ≤ 2a² + 2b² is used, but it is attributed to Jensen's inequality for the quadratic function; the attribution is incorrect and should be replaced by the elementary convexity bound or a direct expansion.
- [References] The bibliography entry for Goldfarb and Hand (2023) has the corrupted title "m." and should give the full title of the AISTATS paper.
- [Figure 1] The caption of Figure 1 contains garbled tokens ("/uni0422" and "uni0422") that should be repaired before publication.
- [Notation] The notation is not perfectly uniform: the abstract and Section 2 use R(β), while Appendix A uses R(f_β); likewise R(0) and R(f0) are used for the same null risk. Please standardize.
Circularity Check
No circularity: the derivation is a genuine proof from stated model assumptions; the main step depends on independent prior-work lemmas, and the known denominator issue in Eq. (20) is a correctness gap, not circular reasoning.
full rationale
The paper's central claims are derived, not fitted or assumed. Theorem 8 proves a new non-asymptotic single-task risk bound directly from the linear model, the closed-form least-squares solutions, and random-matrix estimates from Vershynin (2010); no parameter is calibrated to the risk quantity being bounded. The two-task bounds in Theorems 10 and 11 combine this new bound with Theorem 9, which is quoted from the authors' prior AISTATS paper (Goldfarb and Hand 2023). That self-citation is load-bearing but not circular: Theorem F.11 of the prior work is an independent published result with stated assumptions that do not include the target risk bounds, and it is not a restatement of the present paper's conclusions. Likewise, Lemmas F.2, F.3, and F.9 are cited supporting identities from the same prior paper, not assumptions that smuggle in the result. The notable weakness in the paper is a correctness gap in Theorem 11: the proof divides by 1 - 72*sqrt(d/n) - 18*n/p in Eqs. (83)-(85) without showing this denominator is positive under the hypotheses n >= d, p >= 20n, gamma >= 1/sqrt(nd). In some allowed regimes, e.g. n = d and p = 20n, the denominator is negative, so the inequality reversal is invalid. This is a proof error, not circularity: the derivation does not assume its own conclusion, and the advertised result could fail as stated for those parameters. Since no prediction reduces by construction to its input and no unverified self-citation is used as the sole justification of the central claim, the circularity score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption W^T W = pγ I_d, i.e. columns of W are pairwise orthogonal and of equal length.
- domain assumption The risk formula R(β) = σ² + (β - β*)^T Σ (β - β*) and its equivalence to the noisy-feature model.
- domain assumption Lemma F.9 of Goldfarb and Hand (2023): ||A^T(AA^T)^{-1}ε||² ≤ n||θ||² / (pγ(√(p-d) - 2√n)²).
- domain assumption Theorem F.11 of Goldfarb and Hand (2023): R(β_BA) - R(β_A) ≤ (66√(n/p) + 12/(pγ))||θ||².
- standard math Standard random matrix concentration results, e.g. Vershynin Theorem 5.39, for singular values of Gaussian matrices.
Cite this review
Pith. "Pith review of Analysis of Overparameterization in Continual Learning under a Linear Model." pith.science (2026). https://pith.science/paper/NLKNQOLJ
@misc{pith2026250210442,
author = {Pith},
title = {Pith review of: Analysis of Overparameterization in Continual Learning under a Linear Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/NLKNQOLJ}},
note = {Machine review of arXiv:2502.10442}
}
read the original abstract
Autonomous machine learning systems that learn many tasks in sequence are prone to the catastrophic forgetting problem. Mathematical theory is needed in order to understand the extent of forgetting during continual learning. As a foundational step towards this goal, we study continual learning and catastrophic forgetting from a theoretical perspective in the simple setting of gradient descent with no explicit algorithmic mechanism to prevent forgetting. In this setting, we analytically demonstrate that overparameterization alone can mitigate forgetting in the context of a linear regression model. We consider a two-task setting motivated by permutation tasks, and show that as the overparameterization ratio becomes sufficiently high, a model trained on both tasks in sequence results in a low-risk estimator for the first task. As part of this work, we establish a non-asymptotic bound of the risk of a single linear regression task, which may be of independent interest to the field of double descent theory.
Figures
Forward citations
Cited by 1 Pith paper
-
High-dimensional Asymptotics of Generalization Performance in Continual Ridge Regression
Exact asymptotic prediction risk and transfer metrics for continual ridge regression in proportional-dimension linear models, with risk curves under three covariance dynamics.
Reference graph
Works this paper leans on
-
[1]
Understanding double descent requires a fine-grained bias-variance decomposition
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained bias-variance decomposition. Advances in neural information processing systems, 33: 0 11022--11032, 2020
work page 2020
-
[2]
Benign overfitting in linear regression
Peter L Bartlett, Philip M Long, G \'a bor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117 0 (48): 0 30063--30070, 2020
2020
-
[3]
Two models of double descent for weak features
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2 0 (4): 0 1167--1180, 2020
work page 2020
-
[4]
Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory, pages 483--513. PMLR, 2020
work page 2020
-
[5]
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Springer Nature, 2022
work page 2022
-
[6]
Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355, 2021
arXiv 2021
-
[7]
A continual learning survey: Defying forgetting in classification tasks
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44 0 (7): 0 3366--3385, 2021
work page 2021
-
[8]
Continual learning in linear classification on separable data
Itay Evron, Edward Moroshko, Gon Buzaglo, Maroun Khriesh, Badea Marjieh, Nathan Srebro, and Daniel Soudry. Continual learning in linear classification on separable data. In International Conference on Machine Learning, pages 9440--9484. PMLR, 2023
work page 2023
Show all 22 references
-
[9]
Orthogonal gradient descent for continual learning
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762--3773. PMLR, 2020
2020
-
[10]
Catastrophic forgetting in connectionist networks
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 0 (4): 0 128--135, 1999
1999
-
[11]
Daniel Goldfarb and Paul Hand. m. In International Conference on Artificial Intelligence and Statistics, pages 2975--2993. PMLR, 2023
2023
-
[12]
Surprises in high-dimensional ridgeless least squares interpolation
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50 0 (2): 0 949, 2022
2022
-
[13]
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...
2017
-
[14]
Learning without forgetting
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40 0 (12): 0 2935--2947, 2017
2017
-
[15]
Theory on forgetting and generalization of continual learning
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pages 21078--21100. PMLR, 2023
2023
-
[16]
Wide neural networks forget less catastrophically
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. In International Conference on Machine Learning, pages 15699--15717. PMLR, 2022
2022
-
[17]
Harmless interpolation of noisy data in regression
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1 0 (1): 0 67--83, 2020
2020
-
[18]
Deep double descent: Where bigger models and more data hurt
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 0 (12): 0 124003, 2021
2021
-
[19]
Continual learning with deep generative replay
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017
2017
-
[20]
Introduction to the non-asymptotic analysis of random matrices
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010
2010 arXiv
-
[21]
Continual learning through synaptic intelligence
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987--3995. PMLR, 2017
2017
-
[22]
Understanding deep learning (still) requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64 0 (3): 0 107--115, 2021
2021
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.