Pith. sign in

REVIEW 1 major objections 5 minor 107 references

Optimal Training-Time Scaling in Gradual Adaptation

T0 review · 1 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A gradual adaptation path divided into N tasks is best traversed by training each task for a time proportional to 1/N, because progress vanishes if the total budget is too small or too large.

desk verdict A sound and clean theory result for whole-path common-solution progress, but the abstract's unqualified Θ(N^{-1}) claim is misleading because the same model makes infinite per-task training optimal for final-task loss. read the letter →

arxiv 2608.04927 v2 pith:DWHJBIUS submitted 2026-08-05 cs.LG

Zonghuan Xu , Krishna Harish This is my paper · ORCID
classification cs.LG
keywords gradualdomainadaptationcontinuallearningtraining-timescalingoverparameterizedlinearregressiongradientflowwhole-pathprogresseffectivetrainingtime1/N
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how the per-task training time should change when the same gradual adaptation path is sliced into more intermediate tasks. It analyzes overparameterized linear regression where tasks change smoothly along a path and all share a zero-loss solution, and it treats the time spent on each task as a tunable design variable. The central result is that learning progress depends on the total effective training time $\tau = N s_N$: as $N$ grows with $N s_N \to \tau$, the discrete sequence converges to a limiting continuum curve, and the limiting progress is $\Theta(\tau)$ for small $\tau$ and $\Theta(\tau^{-1})$ for large $\tau$. Only a constant-order total budget $N s_N = \Theta(1)$ preserves positive progress, so the optimal per-task time scales as $s_N^* = \Theta(N^{-1})$. This gives a concrete rule for scheduling updates in gradual adaptation, and experiments on rotated MNIST and temporal Yearbook data are consistent with less per-task training as the path is divided more finely.

What carries the argument

The central object is the effective training time $\tau = N s_N$ together with the continuum evolution $\dot{x}_\tau = -\tau H(t) x_\tau$; its progress identity $P_\tau = 2\tau \int x_\tau^\top H x_\tau\,dt$ turns the whole-path progress into an accumulated cost that can be compared across all discretizations. The argument combines exact exponential updates $v_j = e^{-s_N H(t_j)}v_{j-1}$, a Taylor expansion of these updates against the continuum ODE with uniform $O(N^{-1})$ approximation, and Kato transport $U(t)$, defined by $\dot U = [\dot P, P]U$, which rotates the moving zero-loss spaces into a fixed frame so the large-$N s_N$ regime can be expanded into the operators $G_s$ and $G_H$. Two tail expansions, $P_\tau = 2\tau\langle v_0, A_H v_0\rangle + O(\tau^2)$ and $P_\tau = 2\tau^{-1}\langle v_0, G_H v_0\rangle + o(\tau^{-1})$, deliver the $\Theta(\tau)$ and $\Theta(\tau^{-1})$ bounds.

What would settle it

On a smooth rank-one rotating regression path with known matrices, compute the exact progress $P_{N,s}(v_0)$ on a fine grid of $s$ for $N=16,32,\ldots,256$ and locate the maximizer $N s_N^*$; Theorem 2 predicts it stays in a fixed bounded interval with all accumulation points maximizing the limiting curve, so observing $N s_N^* \to 0$ or $N s_N^* \to \infty$ would refute the scaling law.

Watch

Extended reading notes

Core claim

Under a $C^3$, constant-rank path of Hessians $H(t)$ with a common zero-loss parameter $w^*$, the learning progress $P_{N,s_N}(v_0) = \|v_0\|^2 - \|v_N\|^2$ (distance squared to the common solution set, after removing the component invisible to all tasks) has a sharp continuum limit. If $N s_N \to \tau \in [0,\infty)$, the endpoint converges to the solution of $\dot{x}_\tau(t) = -\tau H(t) x_\tau(t)$, and the progress converges to $P_\tau(v_0)$, which satisfies the exact identity $P_\tau(v_0) = 2\tau \int_0^1 x_\tau(t)^\top H(t) x_\tau(t)\,dt$. The limiting curve rises linearly for small $\tau$ and decays as $\tau^{-1}$ for large $\tau$, so its maximum lies at a finite positive $\tau$. The paper therefore proves that for every sufficiently large $N$, all optimizers of per-task training time are finite and satisfy $s_N^*(v_0)=\Theta(N^{-1})$, equivalently $N s_N^* = \Theta(1)$, meaning the total effective training budget stays constant as the path is refined. An explicit corollary is that training every task to convergence belongs to the vanishing-progress regime.

Load-bearing premise

The $N^{-1}$ law depends on measuring progress as whole-path squared-distance reduction to a common zero-loss solution; the paper's own figures show it need not transfer to target-endpoint loss, because final-task loss can keep improving after whole-path progress starts to decline, and the proof also assumes this shared solution and a smooth constant-rank Hessian path.

Editorial extensions

If this is right

  • When a fixed task path is divided into more steps, each task should be trained proportionally less; the optimal total budget $N s_N^*$ stays at the same finite scale rather than growing with $N$.
  • Training each intermediate task to convergence is not harmless: it is the overtraining regime and its progress falls like $N^{-1}$, tending to zero as the discretization is refined.
  • Task resolution and per-task optimization time cannot be chosen independently: keeping $s_N$ fixed while increasing $N$ sends the total effective time to infinity and moves the system to the wrong regime.
  • At any fixed nonzero per-task time, finite fitting gives a strictly larger first-order progress coefficient than exact fitting, and this ordering survives any strictly increasing transformation of squared distance.
  • Rotated MNIST and temporal Yearbook experiments show fitted exponents whose confidence intervals contain the predicted inverse scaling, supporting the qualitative rule of less per-task training under finer discretization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: in practical test-time or continual adaptation, the update budget should be set by the rate of distribution change rather than by a fixed number of steps per task, since the total effective training time along the path is what determines progress.
  • A testable extension is to measure the same scaling under an endpoint-weighted objective; the paper's own right-hand panels in Figures 3 and 4 show final-task loss still improving after whole-path mean loss rises, so a different optimum should appear.
  • A neighbouring problem it suggests: for a path with non-uniform task spacing, the analogous law would likely become a local update density proportional to the local rate of change of the Hessian, an extension the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper studies how per-task training time should scale when a fixed gradual-adaptation path is sampled into N tasks. It considers a smooth path of overparameterized linear regression tasks t∈[0,1] that share a zero-loss parameter w*, with Hessians H(t) of constant rank, and trains each of the N tasks for time s under gradient flow, so each update is the matrix exponential e^{-sH(t_j)}. Defining learning progress P_{N,s}(v0) as the reduction in squared distance to the common solution set K=∩_t ker H(t) (Eq. (6)), the paper proves (Theorem 1) that when Ns_N→τ the discrete updates converge uniformly to the continuum ODE ẋ=-τH(t)x, with O(N^{-1}) rates on compact τ ranges. Theorem 2 characterizes the limiting progress as Θ(τ) for τ→0 and Θ(τ^{-1}) for τ→∞, and concludes that positive limiting progress occurs exactly when Ns_N=Θ(1), giving s_N^*(v0)=Θ(N^{-1}). Proposition 3 compares fixed finite per-task training with exact fitting at first order, and the paper reports experiments on rotating rank-one regression, rotated MNIST, and Yearbook as qualitative consistency checks.

Significance. The mathematical core is nontrivial and, for the measure it analyzes, appears sound and self-contained: the paper gives an exact telescoping decomposition (Proposition 2), a dense-task limit with explicit uniform rates (Lemma 1), Kato-transport expansions for the large-τ tail (Lemma 2), finite-time first-order expansions with a strict coefficient comparison (Lemma 3), and an explicit anisotropic counterexample showing that the finite-vs-exact comparison can reverse under a different distance (Section A.4). The experiments are honestly labeled as qualitative checks, with fixed seeds, bootstrap intervals, and explicit caveats about departures from the theorem's assumptions. The main qualification is scope: the optimal N^{-1} scaling law concerns the whole-path common-kernel distance measure, not final-task loss, and the abstract and introduction currently state the result without this qualification.

major comments (1)
  1. [Section 2.2 (Eq. (6)), Section 3.2 (Theorem 2), Abstract] The scaling law s_N^* = Θ(N^{-1}) is established only for the progress measure P_{N,s}(v0) = ||v0||^2 - ||Φ_{N,s}v0||^2, i.e., the reduction of squared distance to the common solution set K = ∩_t V(t). This is not the usual gradual-adaptation objective of good final-task performance. Under the same assumptions, the final-task population loss L_1(w_N) = ½ v_N^T H(1) v_N from Eq. (1) is exactly zero at s = ∞, because Φ_{N,∞}v0 = P(t_N)⋯P(t_1)v0 lies in V(1) = ker H(1). Hence for target-endpoint loss, s = ∞ is an optimizer for every N, not Θ(N^{-1}); the N^{-1} rule is contradicted by the same model when the objective is final-task loss rather than whole-path common-kernel distance. The paper's own Figure 3 (right) and Figure 4 (right) demonstrate the operative distinction: final-angle and target-year losses keep improving after whole-path mean loss has turned upward. The abstract's unqualified statement that optimal per-task training times scale as s_N^* = Θ(N^{-1}) is therefore misleading. The claims must either be explicitly restricted to the whole-path common-kernel distance objective, or the target-endpoint objective must be analyzed separately.
minor comments (5)
  1. [Section 1] In the sentence 'coupling its resolution N to the local training times N', the second 'N' should be 's_N'; as printed, the sentence is self-referential and confusing.
  2. [Table 2] The Yearbook interpolated optima are non-monotone between N=16 (4.23) and N=32 (5.01). The text says the optimum 'falls overall' but should explicitly acknowledge this non-monotonicity, since it weakens the visual support for a monotone N^{-1} trend.
  3. [Section B.3 and B.4] The experiments use mean cross-entropy over the path as the empirical objective, whereas Proposition 1 relates the theoretical progress measure to the worst-case path loss sup_t L_t(w), not the average. Since the experiments are qualitative consistency checks this is acceptable, but the mismatch should be stated where the proxy is introduced.
  4. [Proposition 1, Eq. (7)] The upper bound uses L from Eq. (3) but L is not redefined in Proposition 1; readers must infer it from the spectral bound. A one-line reminder would improve readability.
  5. [Contributions, Section 1] In the second bullet, 'prove its Θ(τ) growth' should read 'prove its Θ(τ) growth' or 'prove the Θ(τ) growth'; the sentence as printed has an awkward missing subject.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the N^{-1} law is derived from the ODE and Kato-transport analysis; the sole self-citation is not load-bearing.

full rationale

The paper's central derivation is self-contained. Theorem 1's dense-task limit is proved by Taylor expansion of the matrix exponentials and contraction estimates, giving the uniform O_T(N^{-1}) rates in Eqs. (13)-(14) and the identity in Eq. (15). Theorem 2's small- and large-budget tails are proved from the limiting ODE and the Kato-transport block decomposition in Lemmas 2-4, with the large-τ decay following from the spectral-gap estimate and the explicit operator G_H in Eq. (32). The optimal scaling s_N^* = Θ(N^{-1}) then follows from the two tail bounds and compactness of the limiting maximizer set, not from any fitted constant or from the definition of P_N,s. The progress measure P_N,s(v0)=||v0||^2-||Φ_{N,s}v0||^2 is explicitly introduced in Eq. (6), and the optimal time in Eq. (9) is a maximizer of that named objective; choosing a different objective such as final-task loss is a scope limitation that the paper itself acknowledges ('continued training can improve the target task while worsening full-path performance'), but this is objective-dependence, not circularity. The only overlapping-author citation is Xu and Ma 2026, which appears in a related-work sentence ('while recent work studies continual models under SGD and task distributions (Evron et al., 2026; Xu & Ma, 2026)') and is not used in any proof, definition, or assumption. The experiments fit empirical exponents and report confidence intervals containing 1, while the text explicitly calls these qualitative consistency checks rather than exact verifications, so the empirical claims are not fitted parameters renamed as predictions. No equation in the paper reduces by construction to its own input, and no load-bearing result is imported solely from the authors' prior work.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central scaling law is derived without fitted constants. The main assumptions are a shared zero-loss solution, a smooth constant-rank Hessian path, and a path-averaged squared-distance objective. Standard tools are Kato's spectral transport and contraction estimates. No new entities are postulated.

assumptions (6)
  • domain assumption All tasks share a zero-loss parameter w* with y = x^T w* almost surely.
    Defines the common solution set K and the progress measure; Section 2.1, Eq (1).
  • domain assumption The Hessian path H(t) is C^3 with constant rank and nonzero eigenvalues in [mu, L].
    Needed for Kato transport, contraction bounds, and O(N^-1) rates; Section 2.1, Eq (3).
  • domain assumption The initial error lies in V(0) intersect K-perp after fitting the first task.
    Reduction in Section 2.2; progress depends only on this component.
  • standard math Kato's adiabatic transport theorem and the generator identities for continuously varying projections.
    Used in Section A.3, Eqs (30) and (31), to diagonalize the moving projector P(t).
  • domain assumption Learning progress is squared-distance reduction to the common solution set, equivalent to whole-path mean loss.
    Objective choice in Eq (6); if replaced by target-endpoint loss, the scaling need not hold, as shown in Figures 3 and 4.
  • domain assumption Gradient-flow time approximates finite-step SGD in the experiments.
    Used to map theory to discrete update counts; Section 2.2 and Sections B.3-B.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimal Training-Time Scaling in Gradual Adaptation." pith.science (2026). https://pith.science/paper/DWHJBIUS

@misc{pith2026260804927,
  author       = {Pith},
  title        = {Pith review of: Optimal Training-Time Scaling in Gradual Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DWHJBIUS}},
  note         = {Machine review of arXiv:2608.04927}
}
abstract

In gradual adaptation, how should the training time on each task change as the number of intermediate tasks increases? We study this question for overparameterized linear regression tasks that change smoothly and share a zero-loss solution. With $N$ tasks and training time $s_N$ on each, the final learning progress converges to a continuum curve when $Ns_N\to\tau$. The limiting progress is $\Theta(\tau)$ for small $\tau$ and $\Theta(\tau^{-1})$ for large $\tau$, so both very short and very long training produce little progress. It follows that optimal per-task training times scale as $s_N^\star=\Theta(N^{-1})$, equivalently $Ns_N^\star=\Theta(1)$. Experiments on gradually rotated MNIST and a natural Yearbook time shift are consistent with less per-task training as the path is divided more finely.

Figures

Figures reproduced from arXiv: 2608.04927 by the authors.

Figure 1
Figure 1. Limiting trajectories for the rotating-task example in [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Training time has an intermediate optimum on a rotating task path. Left: Finite-task progress curves approach the same rise-and-decay limit. Right: The best total training time Ns⋆ N approaches τ ⋆ ≈ 2.20, so the best per-task time is approximately 2.20/N. Rotated MNIST. We next train an MLP on upright MNIST and update it along a fixed rotation path from 0 ◦ to 60◦ . Every task count is evaluated on the same 13 angl… view at source ↗
Figure 3
Figure 3. Finer rotation steps favor fewer updates per task. Left: Mean loss across the rotation path against total updates N k. Middle: Estimated optimal updates per task and the N −1 reference. Right: At N = 32, final-angle loss keeps decreasing after the mean loss across all angles turns upward. Error bars and bands are 95% intervals over ten seeds. Yearbook over time. Finally, we use Yearbook photographs spanning 1930–201… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Finer temporal steps favor less training per task overall. Left: Mean loss across years against total updates N k. Middle: Estimated optimal updates per task and the N −1 reference. Right: At N = 32, target-year loss keeps decreasing while the mean loss across years ev…
Figure 5
Figure 5. Figure 5: High-dimensional stress test over ten random smooth Hessian paths. Left: median learning [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

107 extracted references · 68 canonical work pages

  1. [1]

    Neural Networks , volume=

    Continual Lifelong Learning with Neural Networks: A Review , author=. Neural Networks , volume=. 2019 , doi=

    2019

  2. [2]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    A Continual Learning Survey: Defying Forgetting in Classification Tasks , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 2022 , doi=

    2022

  3. [3]

    Proceedings of the 37th International Conference on Machine Learning , series=

    Understanding Self-Training for Gradual Domain Adaptation , author=. Proceedings of the 37th International Conference on Machine Learning , series=. 2020 , publisher=

    2020

  4. [4]

    Advances in Neural Information Processing Systems , volume=

    Gradual Domain Adaptation without Indexed Intermediate Domains , author=. Advances in Neural Information Processing Systems , volume=. 2021 , url=

    2021

  5. [5]

    Proceedings of the 39th International Conference on Machine Learning , series=

    Understanding Gradual Domain Adaptation: Improved Analysis, Optimal Path and Beyond , author=. Proceedings of the 39th International Conference on Machine Learning , series=. 2022 , publisher=

    2022

  6. [6]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Continual Test-Time Domain Adaptation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=. 2022 , url=

    2022

  7. [7]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Robust Mean Teacher for Continual and Gradual Test-Time Adaptation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=. 2023 , url=

    2023

  8. [8]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Robust Test-Time Adaptation in Dynamic Scenarios , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=. 2023 , url=

    2023

Show all 107 references
  1. [9]

    IEEE Robotics and Automation Letters , volume=

    Test-Time Adaptation in the Dynamic World with Compound Domain Knowledge Management , author=. IEEE Robotics and Automation Letters , volume=. 2023 , url=

    2023

  2. [10]

    2025 , publisher=

    Zhang, Yunbei and Mehra, Akshay and Niu, Shuaicheng and Hamm, Jihun , booktitle=. 2025 , publisher=

    2025

  3. [11]

    The Eleventh International Conference on Learning Representations , year=

    Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting , author=. The Eleventh International Conference on Learning Representations , year=

  4. [12]

    Proceedings of the 34th International Conference on Machine Learning , series=

    Automated Curriculum Learning for Neural Networks , author=. Proceedings of the 34th International Conference on Machine Learning , series=. 2017 , publisher=

    2017

  5. [13]

    Proceedings of the 39th International Conference on Machine Learning , series=

    Curriculum Reinforcement Learning via Constrained Optimal Transport , author=. Proceedings of the 39th International Conference on Machine Learning , series=. 2022 , publisher=

    2022

  6. [14]

    Advances in Neural Information Processing Systems , volume=

    Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation , author=. Advances in Neural Information Processing Systems , volume=. 2022 , url=

    2022

  7. [15]

    The Thirteenth International Conference on Learning Representations , year=

    Optimal Protocols for Continual Learning via Statistical Physics and Control Theory , author=. The Thirteenth International Conference on Learning Representations , year=

  8. [16]

    Constructive Approximation , volume=

    On Early Stopping in Gradient Descent Learning , author=. Constructive Approximation , volume=. 2007 , doi=

    2007

  9. [17]

    Journal of Machine Learning Research , volume=

    Early Stopping and Non-parametric Regression: An Optimal Data-dependent Stopping Rule , author=. Journal of Machine Learning Research , volume=. 2014 , url=

    2014

  10. [18]

    Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , series=

    A Continuous-Time View of Early Stopping for Least Squares Regression , author=. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , series=. 2019 , publisher=

    2019

  11. [19]

    Proceedings of the 33rd International Conference on Machine Learning , series=

    Train Faster, Generalize Better: Stability of Stochastic Gradient Descent , author=. Proceedings of the 33rd International Conference on Machine Learning , series=. 2016 , publisher=

    2016

  12. [20]

    Advances in Neural Information Processing Systems , volume=

    Understanding the Role of Training Regimes in Continual Learning , author=. Advances in Neural Information Processing Systems , volume=

  13. [21]

    Proceedings of the 41st International Conference on Machine Learning , series=

    A Statistical Theory of Regularization-Based Continual Learning , author=. Proceedings of the 41st International Conference on Machine Learning , series=. 2024 , publisher=

    2024

  14. [22]

    Advances in Neural Information Processing Systems , volume=

    Optimal Rates in Continual Linear Regression via Increasing Regularization , author=. Advances in Neural Information Processing Systems , volume=. 2025 , url=

    2025

  15. [23]

    Proceedings of the 2nd Conference on Lifelong Learning Agents , series=

    Challenging Common Assumptions about Catastrophic Forgetting and Knowledge Accumulation , author=. Proceedings of the 2nd Conference on Lifelong Learning Agents , series=. 2023 , publisher=

    2023

  16. [24]

    Proceedings of the 42nd International Conference on Machine Learning , series=

    The Importance of Being Lazy: Scaling Limits of Continual Learning , author=. Proceedings of the 42nd International Conference on Machine Learning , series=. 2025 , publisher=

    2025

  17. [25]

    Proceedings of the 35th Conference on Learning Theory , series=

    How Catastrophic Can Catastrophic Forgetting Be in Linear Regression? , author=. Proceedings of the 35th Conference on Learning Theory , series=. 2022 , publisher=

    2022

  18. [26]

    Advances in Neural Information Processing Systems , volume=

    Nearly Optimal Bounds for Cyclic Forgetting , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    2023

  19. [27]

    From Continual Learning to

    Evron, Itay and Levinstein, Ran and Schliserman, Matan and Sherman, Uri and Koren, Tomer and Soudry, Daniel and Srebro, Nathan , booktitle=. From Continual Learning to. 2026 , publisher=

    2026

  20. [28]

    Journal of the Physical Society of Japan , volume=

    On the Adiabatic Theorem of Quantum Mechanics , author=. Journal of the Physical Society of Japan , volume=. 1950 , doi=

    1950

  21. [29]

    Facchi, Paolo and Pascazio, Saverio , journal=. Quantum. 2008 , doi=

    2008

  22. [30]

    Communications in Mathematical Physics , volume=

    Adiabatic Theorems for Generators of Contracting Evolutions , author=. Communications in Mathematical Physics , volume=. 2012 , doi=

    2012

  23. [32]

    Journal of Dynamics and Games , volume=

    Asymptotic Behavior of Compositions of Under-Relaxed Nonexpansive Operators , author=. Journal of Dynamics and Games , volume=. 2014 , doi=

    2014

  24. [33]

    Proceedings of the 26th International Conference on Machine Learning , pages=

    Curriculum Learning , author=. Proceedings of the 26th International Conference on Machine Learning , pages=. 2009 , doi=

    2009

  25. [34]

    Advances in Neural Information Processing Systems , volume=

    Self-Paced Learning for Latent Variable Models , author=. Advances in Neural Information Processing Systems , volume=

  26. [35]

    Proceedings of the 35th International Conference on Machine Learning , series=

    Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks , author=. Proceedings of the 35th International Conference on Machine Learning , series=. 2018 , publisher=

    2018

  27. [36]

    Proceedings of the 36th International Conference on Machine Learning , series=

    On the Power of Curriculum Learning in Training Deep Networks , author=. Proceedings of the 36th International Conference on Machine Learning , series=. 2019 , publisher=

    2019

  28. [37]

    Proceedings of the 37th International Conference on Machine Learning , series=

    Test-Time Training with Self-Supervision for Generalization under Distribution Shifts , author=. Proceedings of the 37th International Conference on Machine Learning , series=. 2020 , publisher=

    2020

  29. [38]

    2021 , url=

    Wang, Dequan and Shelhamer, Evan and Liu, Shaoteng and Olshausen, Bruno and Darrell, Trevor , booktitle=. 2021 , url=

    2021

  30. [39]

    Proceedings of the 39th International Conference on Machine Learning , series=

    Efficient Test-Time Model Adaptation without Forgetting , author=. Proceedings of the 39th International Conference on Machine Learning , series=. 2022 , publisher=

    2022

  31. [40]

    Zhang, Marvin and Levine, Sergey and Finn, Chelsea , booktitle=

  32. [41]

    International Conference on Learning Representations , year=

    Towards Stable Test-Time Adaptation in Dynamic Wild World , author=. International Conference on Learning Representations , year=

  33. [42]

    Machine Learning , volume=

    A Theory of Learning from Different Domains , author=. Machine Learning , volume=. 2010 , doi=

    2010

  34. [43]

    Journal of Machine Learning Research , volume=

    Domain-Adversarial Training of Neural Networks , author=. Journal of Machine Learning Research , volume=

  35. [44]

    and Darrell, Trevor , booktitle=

    Hoffman, Judy and Tzeng, Eric and Park, Taesung and Zhu, Jun-Yan and Isola, Phillip and Saenko, Kate and Efros, Alexei A. and Darrell, Trevor , booktitle=. 2018 , publisher=

    2018

  36. [45]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Maximum Classifier Discrepancy for Unsupervised Domain Adaptation , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  37. [46]

    Proceedings of the National Academy of Sciences , volume=

    Overcoming Catastrophic Forgetting in Neural Networks , author=. Proceedings of the National Academy of Sciences , volume=. 2017 , doi=

    2017

  38. [47]

    Proceedings of the 34th International Conference on Machine Learning , series=

    Continual Learning Through Synaptic Intelligence , author=. Proceedings of the 34th International Conference on Machine Learning , series=. 2017 , publisher=

    2017

  39. [48]

    Advances in Neural Information Processing Systems , volume=

    Gradient Episodic Memory for Continual Learning , author=. Advances in Neural Information Processing Systems , volume=

  40. [49]

    Efficient Lifelong Learning with

    Chaudhry, Arslan and Ranzato, Marc'Aurelio and Rohrbach, Marcus and Elhoseiny, Mohamed , booktitle=. Efficient Lifelong Learning with. 2019 , url=

    2019

  41. [50]

    Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , series=

    Orthogonal Gradient Descent for Continual Learning , author=. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , series=. 2020 , publisher=

    2020

  42. [51]

    Advances in Neural Information Processing Systems , volume=

    Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting , author=. Advances in Neural Information Processing Systems , volume=

  43. [52]

    Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , series=

    Unifying Importance Based Regularisation Methods for Continual Learning , author=. Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , series=. 2022 , publisher=

    2022

  44. [53]

    Nature Machine Intelligence , volume=

    Three Types of Incremental Learning , author=. Nature Machine Intelligence , volume=. 2022 , doi=

    2022

  45. [54]

    IEEE Transactions on Computational Imaging , volume=

    A Century of Portraits: A Visual Historical Record of American High School Yearbooks , author=. IEEE Transactions on Computational Imaging , volume=. 2017 , doi=

    2017

  46. [55]

    Advances in Neural Information Processing Systems Datasets and Benchmarks Track , year=

    Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time , author=. Advances in Neural Information Processing Systems Datasets and Benchmarks Track , year=

  47. [56]

    Zico Kolter, and Ryan J

    Alnur Ali, J. Zico Kolter, and Ryan J. Tibshirani. A continuous-time view of early stopping for least squares regression. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp.\ ...

    2019

  48. [57]

    J. E. Avron, M. Fraas, G. M. Graf, and P. Grech. Adiabatic theorems for generators of contracting evolutions. Communications in Mathematical Physics, 314: 0 163--191, 2012. doi:10.1007/s00220-012-1504-1

    2012 doi

  49. [58]

    A theory of learning from different domains

    Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79: 0 151--175, 2010. doi:10.1007/s10994-009-5152-4

    2010 doi

  50. [59]

    Curriculum learning

    Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, pp.\ 41--48, 2009. doi:10.1145/1553374.1553380

    2009

  51. [60]

    Unifying importance based regularisation methods for continual learning

    Frederik Benzing. Unifying importance based regularisation methods for continual learning. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp.\ 2372--2396. PMLR, 2022

    2022

  52. [61]

    Efficient lifelong learning with A-GEM

    Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM . In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Hkf2_sC5FX

    2019

  53. [62]

    Gradual domain adaptation without indexed intermediate domains

    Hong-You Chen and Wei-Lun Chao. Gradual domain adaptation without indexed intermediate domains. In Advances in Neural Information Processing Systems, volume 34, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/45017f6511f91be700fda3d118034994-Abstract.html

    2021

  54. [63]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 0 (7): 0 3...

    2022

  55. [64]

    Marsden, and Bin Yang

    Mario D \"o bler, Robert A. Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7704--7714, 2023. URL https://openaccess.thecvf.com/content/CVPR202...

    2023

  56. [65]

    Dynamic update-to-data ratio: Minimizing world model overfitting

    Nicolai Dorka, Tim Welschehold, and Wolfram Burgard. Dynamic update-to-data ratio: Minimizing world model overfitting. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=BJv0jo4hqEZ

    2023

  57. [66]

    Ward, Nathan Srebro, and Daniel Soudry

    Itay Evron, Edward Moroshko, Rachel A. Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Proceedings of the 35th Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pp.\ 4028--407...

    2022

  58. [67]

    From continual learning to SGD and back: Better rates for continual linear models

    Itay Evron, Ran Levinstein, Matan Schliserman, Uri Sherman, Tomer Koren, Daniel Soudry, and Nathan Srebro. From continual learning to SGD and back: Better rates for continual linear models. In Proceedings of the 37th International Conference on Algorithmic Learning Theory, vol...

    2026

  59. [68]

    Orthogonal gradient descent for continual learning

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp.\ 3762--3773. ...

    2020

  60. [69]

    Domain-adversarial training of neural networks

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Fran c ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17 0 (59): 0 1--35, 2016

    2016

  61. [70]

    a henb \

    Shiry Ginosar, Kate Rakelly, Sarah M. Sachs, Brian Yin, Crystal Lee, Philipp Kr \"a henb \"u hl, and Alexei A. Efros. A century of portraits: A visual historical record of american high school yearbooks. IEEE Transactions on Computational Imaging, 3 0 (3): 0 421--431, 2017. do...

    2017

  62. [71]

    The importance of being lazy: Scaling limits of continual learning

    Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta, Thomas Hofmann, and Lorenzo Noci. The importance of being lazy: Scaling limits of continual learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning R...

    2025

  63. [72]

    Bellemare, Jacob Menick, R \'e mi Munos, and Koray Kavukcuoglu

    Alex Graves, Marc G. Bellemare, Jacob Menick, R \'e mi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1311--13...

    2017

  64. [73]

    On the power of curriculum learning in training deep networks

    Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.\ 2535--2544. PMLR, 2019

    2019

  65. [74]

    Train faster, generalize better: Stability of stochastic gradient descent

    Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp.\ 1225--1234. PMLR, 2016. UR...

    2016

  66. [75]

    Efros, and Trevor Darrell

    Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. CyCADA : Cycle-consistent adversarial domain adaptation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of M...

    1989

  67. [76]

    Curriculum reinforcement learning using optimal transport via gradual domain adaptation

    Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips.cc/paper_fi...

    2022

  68. [77]

    On the adiabatic theorem of quantum mechanics

    Tosio Kato. On the adiabatic theorem of quantum mechanics. Journal of the Physical Society of Japan, 5 0 (6): 0 435--439, 1950. doi:10.1143/JPSJ.5.435

    1950 doi

  69. [78]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting i...

    2017 doi

  70. [79]

    Curriculum reinforcement learning via constrained optimal transport

    Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 1...

    2022

  71. [80]

    Understanding self-training for gradual domain adaptation

    Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 5468--5479. PMLR, 2020. URL https://procee...

    2020

  72. [81]

    Pawan Kumar, Benjamin Packer, and Daphne Koller

    M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, volume 23, 2010

    2010

  73. [82]

    Challenging common assumptions about catastrophic forgetting and knowledge accumulation

    Timoth \'e e Lesort, Oleksiy Ostapenko, Pau Rodr \'i guez, Diganta Misra, Md Rifat Arefin, Laurent Charlin, and Irina Rish. Challenging common assumptions about catastrophic forgetting and knowledge accumulation. In Proceedings of the 2nd Conference on Lifelong Learning Agents...

    2023

  74. [83]

    Optimal rates in continual linear regression via increasing regularization

    Ran Levinstein, Amit Attia, Matan Schliserman, Uri Sherman, Daniel Soudry, Tomer Koren, and Itay Evron. Optimal rates in continual linear regression via increasing regularization. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.ne...

    2025

  75. [84]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  76. [85]

    Understanding the role of training regimes in continual learning

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. In Advances in Neural Information Processing Systems, volume 33, pp.\ 7308--7320, 2020

    2020

  77. [86]

    Optimal protocols for continual learning via statistical physics and control theory

    Francesco Mori, Stefano Sarao Mannelli, and Francesca Mignacco. Optimal protocols for continual learning via statistical physics and control theory. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  78. [87]

    Efficient test-time model adaptation without forgetting

    Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Re...

    2022

  79. [88]

    Towards stable test-time adaptation in dynamic wild world

    Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=g2YraF75Tj

    2023

  80. [89]

    Parisi, Ronald Kemker, Jose L

    German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113: 0 54--71, 2019. doi:10.1016/j.neunet.2019.01.012

    2019 doi

  81. [90]

    Wainwright, and Bin Yu

    Garvesh Raskutti, Martin J. Wainwright, and Bin Yu. Early stopping and non-parametric regression: An optimal data-dependent stopping rule. Journal of Machine Learning Research, 15 0 (11): 0 335--366, 2014. URL https://jmlr.org/papers/v15/raskutti14a.html

    2014

  82. [91]

    Online structured laplace approximations for overcoming catastrophic forgetting

    Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, volume 31, 2018

    2018

  83. [92]

    Maximum classifier discrepancy for unsupervised domain adaptation

    Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 3723--3732, 2018

    2018

  84. [93]

    Test-time adaptation in the dynamic world with compound domain knowledge management

    Junha Song, Kwanyong Park, InKyu Shin, Sanghyun Woo, Chaoning Zhang, and In So Kweon. Test-time adaptation in the dynamic world with compound domain knowledge management. IEEE Robotics and Automation Letters, 8 0 (11): 0 7583--7590, 2023. URL https://arxiv.org/abs/2212.08356

    2023 arXiv

  85. [94]

    Efros, and Moritz Hardt

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machi...

    2020

  86. [95]

    Ward, Mark Kong, and Halyun Jeong

    William Swartworth, Deanna Needell, Rachel A. Ward, Mark Kong, and Halyun Jeong. Nearly optimal bounds for cyclic forgetting. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/d72ae75abaa70a3b1...

    2023

  87. [96]

    van de Ven, Tinne Tuytelaars, and Andreas S

    Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types of incremental learning. Nature Machine Intelligence, 4: 0 1185--1197, 2022. doi:10.1038/s42256-022-00568-3

    2022 doi

  88. [97]

    Tent : Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent : Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=uXl3bZLkr3c

    2021

  89. [98]

    Understanding gradual domain adaptation: Improved analysis, optimal path and beyond

    Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 22784--22801. PMLR, 20...

    2022

  90. [99]

    Continual test-time domain adaptation

    Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7201--7211, 2022 b . URL https://openaccess.thecvf.com/content/CVPR2022/html/Wang_Continual_Tes...

    2022

  91. [100]

    Curriculum learning by transfer learning: Theory and experiments with deep networks

    Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 5238--5246. PMLR, 2018

    2018

  92. [101]

    From order to distribution: A spectral characterization of forgetting in continual learning

    Zonghuan Xu and Xingjun Ma. From order to distribution: A spectral characterization of forgetting in continual learning. arXiv preprint arXiv:2604.13460, 2026

    2026 arXiv

  93. [102]

    Wild-time: A benchmark of in-the-wild distribution shift over time

    Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2022

    2022

  94. [103]

    On early stopping in gradient descent learning

    Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26 0 (2): 0 289--315, 2007. doi:10.1007/s00365-006-0663-2

    2007 doi

  95. [104]

    Robust test-time adaptation in dynamic scenarios

    Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15922--15932, 2023. URL https://openaccess.thecvf.com/content/CVPR2023/html/Yuan_Robust_Test-Ti...

    2023

  96. [105]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 3987--3995. PMLR, 2017

    2017

  97. [106]

    MEMO : Test time robustness via adaptation and augmentation

    Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO : Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems, volume 35, 2022

    2022

  98. [107]

    DPC ore: Dynamic prompt coreset for continual test-time adaptation

    Yunbei Zhang, Akshay Mehra, Shuaicheng Niu, and Jihun Hamm. DPC ore: Dynamic prompt coreset for continual test-time adaptation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp.\ 75757--75778. P...

    2025

  99. [108]

    A statistical theory of regularization-based continual learning

    Xuyang Zhao, Huiyuan Wang, Weiran Huang, and Wei Lin. A statistical theory of regularization-based continual learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.\ 61021--61039. PMLR, 2024...

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.