Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Optimal Task Order for Continual Learning of Multiple Tasks

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Train on the least typical tasks first, and keep similar tasks apart.

desk verdict New analytical result for task-order dependence in linear continual learning, two useful but heuristic ordering rules, and a real-data validation that is suggestive rather than conclusive. read the letter →

arxiv 2502.03350 v2 pith:BFJNGJ4K submitted 2025-02-05 stat.ML cs.LG

Ziyan Li , Naoki Hiratani This is my paper · ORCID
classification stat.MLcs.LG MSC 68T0562H20
keywords continuallearningtaskorderoptimizationsimilarityteacher-studentmodelcatastrophicforgettingcurriculumHamiltonianpathzero-shottransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that in continual learning, where a network is trained on one task after another, the order in which tasks appear is a controllable factor that can be optimized, not a nuisance variable, and it derives exactly how order enters the final error. Working in a linear teacher-student model in which each task's inputs and outputs are generated from a small set of shared latent factors, the authors prove that the final average error is determined by the upper-triangular half of the task similarity matrix, so permuting tasks changes performance in a way that can be computed in closed form. From a perturbation analysis of that formula they extract two ordering principles that hold across a wide parameter range: learn the least typical, most peripheral tasks first and the most typical task last, and make adjacent tasks as dissimilar as possible. They validate both rules on Fashion-MNIST, CIFAR-10, and CIFAR-100 using fully connected and convolutional networks, where task similarities are estimated from zero-shot transfer performance and the recommended orders outperform random, min-path, and core-to-periphery orderings. They further show that estimating similarity from only about one percent of the data preserves most of the benefit.

What carries the argument

The load-bearing object is a linear teacher-student model with latent factors: task $\mu$ generates inputs $x = A_\mu s$ and targets $y^* = B_\mu s$ from shared Gaussian latent $s$, with mixing matrices $A_\mu, B_\mu$ correlated across tasks through matrices $C^{\mathrm{in}}, C^{\mathrm{out}}$. In the $N_s/N_x \to 0$ limit the gradient-descent update on each task reduces to a product of projection operators, $W_\mu \approx \gamma\sum_{\nu=1}^\mu (B_\nu A_\nu^T)\prod_{\rho=\nu+1}^\mu (I-\gamma A_\rho A_\rho^T)$, and the expectation over the random matrices contracts, via Isserlis' theorem, to products of task correlations $\prod C^{\mathrm{in}}_{\rho\sigma}$. The identity that carries the argument is the geometric series $\sum_{k\ge 0}(-1)^k (C^{\mathrm{in},U})^k = (I + C^{\mathrm{in},U})^{-1}$, which turns the nested sums over task indices into a single matrix inverse and makes the upper-triangular dependence explicit. The second workhorse is the linear perturbation $G_{\mu\nu} = G^+_{\mu\nu} + G^-_{\mu\nu}$ around a homogeneous task similarity $m$: $G^+$ depends on position sums through the quantities $(1-m)^\mu$ and yields the periphery-to-core rule by a rearrangement inequality, while $G^-$ depends on position differences $(1-m)^{\nu-\mu}$ and connects the error to the length of the Hamiltonian path on the task dissimilarity graph.

What would settle it

A direct calculation would settle the linear-model claim: construct two input correlation matrices $C^{\mathrm{in}}$ that agree on every upper-triangular entry but differ on lower-triangular entries, simulate gradient descent to convergence for both, and compare the final average error; Theorem 3.1 predicts identical errors in the $N_s/N_x \to 0$ limit, and any systematic difference would refute the closed form. On real networks, the counterpart is to find task sets where the transfer-estimated ordering violates both rules (for instance, most typical task first followed by a min-path traversal) yet still beats the recommended ordering across many random sets, which would show the rules do not transfer beyond the linear model.

Watch

Extended reading notes

Core claim

The paper's central claim is that task-order dependence in continual learning is not an accident of nonlinear dynamics but a structural feature that already appears in the simplest tractable setting: a linear teacher-student model with latent factors, in the limit where the input dimension far exceeds the latent dimension. In that setting the final error averaged over random task generations is exactly $\bar{\epsilon}_f = \|(C^{\mathrm{out}})^{1/2}(I-(I+C^{\mathrm{in},U})^{-1}C^{\mathrm{in}})\|_F^2$ (Theorem 3.1), where $C^{\mathrm{in},U}$ is the strictly upper-triangular part of the input correlation matrix between tasks; because only the upper-triangular part appears, the error cannot be permutation-invariant. Expanding the formula around a uniform similarity baseline, the order dependence splits into two interpretable components (Theorem 4.1): a typicality term, minimized by ordering tasks from least to most representative (periphery-to-core rule), and a relative-distance term, minimized by maximizing the length of the Hamiltonian path through the task-dissimilarity graph so that adjacent tasks are dissimilar (max-path rule). The paper then argues these rules transfer to real networks, demonstrating improved average accuracy on image classification benchmarks when task similarity is estimated from zero-shot transfer errors.

Load-bearing premise

The load-bearing premise is that task similarity in real, nonlinear networks is faithfully captured by the zero-shot transfer estimator of Eq. 16, whose square-root link between transfer error and correlation is derived from the linear model; if that mapping is inaccurate, the estimated orderings are wrong and the real-data experiments do not actually test the theory.

Editorial extensions

If this is right

  • Task order can be chosen before training begins: with a pilot estimate of pairwise task similarity, the recommended order is computable in $O(P)$ time instead of the $O(P!)$ cost of searching all orders.
  • Order matters quantitatively: in the linear model with five chain-structured tasks, the best order beats the worst by roughly a factor of seven in final error, so ordering is a first-order effect rather than a tuning detail.
  • The max-path rule explains the earlier empirical finding that longest Hamiltonian paths outperform shortest ones in image classification, and it gives the theoretical condition, a $G^-$ term with negative coefficient, under which that finding should hold.
  • The periphery-to-core advantage grows with the number of tasks, as demonstrated on CIFAR-100 with $P=3,5,7$, and both rules remain beneficial when similarity is estimated from about one percent of the training data.
  • The two rules can be applied without access to model internals, since the similarity estimator uses only the inputs and outputs of trained networks, which the paper notes makes the method usable even for closed systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the similarity estimator uses only the inputs and outputs of a trained network, the ordering rules could in principle be applied where internal representations are inaccessible, for example in designing practice schedules for human learners or training protocols in animal experiments; the paper raises these settings as motivation but does not test them.
  • The theory treats each task as learned once to convergence; an untested extension is an online scheduler that reorders only the remaining task queue from a running dissimilarity estimate, which the linear model's closed form suggests would capture most of the benefit at much lower cost.
  • The upper-triangular structure implies a sharper diagnostic: the gain of any ordering should be predictable from where the large pairwise similarities sit along the sequence, so a new benchmark could test the mechanism, rather than the estimator, by checking whether that placement predicts the accuracy gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies how the order of tasks affects the final error in task-incremental continual learning. In a linear teacher-student model with a low-dimensional latent factor, the authors derive a closed-form expression for the average final error (Theorem 3.1) showing that the error depends on the task ordering through the strictly upper-triangular part of the input correlation matrix. A perturbation expansion around a constant task-similarity matrix then decomposes the order dependence into two terms, leading to two ordering principles: the periphery-to-core rule (least typical tasks should be learned first) and the max-path rule (adjacent tasks should be as dissimilar as possible). The principles are illustrated on synthetic tasks with chain, ring, and tree similarity structures and are claimed to be validated on Fashion-MNIST, CIFAR-10, and CIFAR-100 using an estimator of task similarity based on zero-shot transfer loss (Eq. 16). The paper also reports that estimating the order from about 1% of the data still yields a significant advantage over random ordering.

Significance. If the claims are established, this paper would provide a rare analytical handle on task-order dependence in continual learning, explaining the previously empirical max-path finding of Bell & Lawrence (2022) and offering a practical, low-cost way to choose task order. The derivation of Eq. (8) is clean and the perturbation decomposition in Theorem 4.1 is a useful technical contribution; the numerical simulations in the linear model match the theory, and the source code is publicly available. However, the theoretical support for the two ordering rules is only heuristic in parts, and the real-data validation depends on an untested mapping between cross-entropy transfer losses and the linear-model similarity measure. These gaps do not invalidate the core derivation but do mean that the paper's strongest claims are not yet fully supported.

major comments (3)
  1. [Section 5.1, Eq. (16), Appendix C.4] The estimator in Eq. (16) is the sole bridge between the linear theory and the real-data validation. The square-root transform is justified in Appendix C.4 by Eq. (83), an identity for MSE loss in a linear teacher-student model with output correlation rho_o = 1. The image experiments, however, use cross-entropy loss on ReLU networks with a shared output head; no derivation or empirical calibration establishes that the same functional relation holds for cross-entropy transfer losses, or that the label-shuffling normalization makes the expression monotone in the true input similarity. The robustness checks in Figs. 7c,d, 9, and 10 vary the data fraction and the train/test split used for evaluation, but they never vary the functional form of the estimator itself. The observed advantages of the periphery-to-core and max-path orderings could therefore, in principle, be artifacts of the estimator rather than evidence for the theoretical similarity structure. Please add a calibration experiment that compares Eq. (16) with an independent measure of task similarity (for example, a representation-similarity probe or a controlled linear-model analogue with cross-entropy loss), or otherwise establish the monotonicity of the transfer-loss-to-similarity mapping for the architectures and losses used in Section 5.
  2. [Section 4.2, Eq. (82)] The periphery-to-core rule is not proven. The rearrangement inequality is applied only to the first term of the G+ contribution in Eq. (82), and the paper itself states that the second term 'may not be minimized' under the same ordering. Because the sign and magnitude of the second term are not controlled, the abstract's claim that the two principles 'hold under a wide parameter range' exceeds what the analysis establishes. The numerical evidence in Fig. 3e also shows that the periphery-to-core rule does not outperform the alternatives for small positive average correlation. Either supply a bound on the neglected term or a sufficient condition for its non-interference, or restate the periphery-to-core rule as a heuristic with an explicitly qualified validity regime.
  3. [Section 4.3, Eq. (15)] The max-path rule is also derived heuristically. From Eq. (14), the text observes that the coefficient of D_H in Eq. (15) is the smallest among the distance sums and concludes that choosing the largest adjacent dissimilarity is beneficial. This is plausible but does not prove that, among all Hamiltonian paths, the max-path assignment minimizes the full weighted sum of pairwise dissimilarities; the relationship between D_H and the remaining distance sums is not analyzed. Since the same rule was already an empirical finding of Bell & Lawrence (2022), the paper would be strengthened by either stating a precise sense in which max-path is near-optimal (for example, a bound on the suboptimality gap) or clearly labeling the rule as a heuristic inferred from coefficient sizes.
minor comments (4)
  1. [Theorem 3.1 and Appendix A.4] The theorem statement says the result holds at N_s/N_x -> 0, but the proof of the required Wick-contraction dominance in Appendix A.4 relies on P << N_x/N_s. This condition is mentioned later in the appendix but is omitted from the theorem statement; it should be included or noted there.
  2. [Section 5.1, Eq. (16)] The label-shuffled error is described as the 'chance-level error.' For cross-entropy with a shared output head, the chance-level loss after training on task A is not necessarily the same as the loss on a randomly labeled version of task B, because the output-layer bias and the learned features affect both. The text should clarify what exactly is being normalized and why the shuffled loss is the appropriate baseline.
  3. [Section 5.2] In the sentence 'the max-path rule outperformed both the min-path rule and the random ordering (Fig. 5b,d,e; see also Bell & Lawrence (2022))', the reference should be to Fig. 5b,d,f, since panels e and f cover the CIFAR-100 results and panel f is the max-path comparison for that dataset.
  4. [Throughout] The paper uses 'task-incremental continual learning' and 'task-incremental learning' interchangeably; the terminology should be made consistent, especially because the shared-head binary and multi-class setups are not the standard 'task-incremental' scenario with separate output heads.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ordering rules are derived from an independent linear-model perturbation analysis and validated without fitting to the target metric.

full rationale

The paper's central derivation is self-contained. Theorem 3.1 (Eq. 8) is obtained from explicit gradient-descent dynamics in the linear teacher-student model (Eqs. 5-7) and is checked against numerical gradient descent over randomly sampled task-correlation matrices (Fig. 2b), with no parameter fitted to the final error. Theorem 4.1 provides analytical perturbation coefficients G_mu_nu as explicit functions of m, rho_o, and P (Eqs. 12, 57); the periphery-to-core and max-path rules are presented as heuristics derived from the monotonicity/sign structure of G^+ and alpha^- (Eqs. 13-15), and then tested on independently sampled C_in matrices (Figs. 3e,g, 4). These are genuine predictions, not restatements of the inputs. The real-data validation uses the zero-shot-transfer similarity estimator of Eq. 16, whose square-root transform is justified in Appendix C.4 by a linear-model identity (Eq. 83) cited to prior work by one of the authors (Hiratani). This is a self-citation, but it is a parameter-free analytical identity with stated assumptions (linear model, output similarity one), and it does not contain the ordering-rule result. Whether the same square-root relationship holds for cross-entropy losses on ReLU networks is an external-validity assumption, and the paper explicitly describes Eq. 16 as 'Motivated by' the linear relationship rather than as a proven consequence for nonlinear networks. That concern is a correctness/robustness risk, not circularity: the empirical ordering rules are not extracted from the final test accuracies used to evaluate them, and no fitted parameter is renamed as a prediction. No uniqueness theorem is imported from the authors' prior work, and no known empirical result is presented as a new derivation. The only mildly notable item is the self-citation supporting the similarity estimator, but because the estimator is not fitted to the target outcome and the theory's core results do not depend on it, it does not constitute a circular step under the stated criteria.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The theory contributes a closed-form error expression that depends on no fitted constants, but the ordering rules rest on the linear-model generative assumptions, the asymptotic wide-input limit, and a heuristic perturbation argument. The real-data tests additionally assume the zero-shot transfer similarity estimate is valid outside the linear model.

assumptions (5)
  • domain assumption Task inputs are generated by a shared low-dimensional latent variable s with task-specific mixing matrices A_mu, B_mu, sampled from correlated Gaussians with element-wise correlation matrices C_in, C_out.
    Defines the teacher-student model in Section 3.1; all subsequent theory builds on this generative process.
  • domain assumption Asymptotic regime Ns/Nx -> 0 with P << Nx/Ns, where the pseudo-inverse is approximated by a scaled transpose and cross-task traces concentrate.
    Used in Appendix A.3 and A.4 to derive Theorem 3.1; limits direct applicability to wide networks with few latent factors.
  • ad hoc to paper In the perturbation analysis, all non-diagonal entries of C_in are m+deltaM_mu_nu with small deltaM and C_out is constant rho_o across tasks.
    Assumed in Theorem 4.1 and the derivation of G_mu_nu in Appendix B to make the matrix inversion analytically tractable.
  • domain assumption For real image tasks, task similarity is faithfully estimated by zero-shot transfer performance via Eq. 16, including the square-root mapping.
    Connects the linear-model notion of similarity to nonlinear networks; if the mapping is wrong, the ordering rules are misapplied.
  • ad hoc to paper The second term in the G+ decomposition (Eq. 82) is assumed not to overturn the rearrangement inequality; the paper notes it may not be minimized.
    Used to justify the periphery-to-core rule; the paper itself flags this is a heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimal Task Order for Continual Learning of Multiple Tasks." pith.science (2026). https://pith.science/paper/BFJNGJ4K

@misc{pith2026250203350,
  author       = {Pith},
  title        = {Pith review of: Optimal Task Order for Continual Learning of Multiple Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BFJNGJ4K}},
  note         = {Machine review of arXiv:2502.03350}
}
read the original abstract

Continual learning of multiple tasks remains a major challenge for neural networks. Here, we investigate how task order influences continual learning and propose a strategy for optimizing it. Leveraging a linear teacher-student model with latent factors, we derive an analytical expression relating task similarity and ordering to learning performance. Our analysis reveals two principles that hold under a wide parameter range: (1) tasks should be arranged from the least representative to the most typical, and (2) adjacent tasks should be dissimilar. We validate these rules on both synthetic data and real-world image classification datasets (Fashion-MNIST, CIFAR-10, CIFAR-100), demonstrating consistent performance improvements in both multilayer perceptrons and convolutional neural networks. Our work thus presents a generalizable framework for task-order optimization in task-incremental continual learning.

Figures

Figures reproduced from arXiv: 2502.03350 by the authors.

Figure 1
Figure 1. Schematic figure of the task-order dependence. a, b) Continual learning of binary classification with two different task orders. c) Average test accuracy on the three classification tasks at the end of learning under task orders depicted in panels a and b. Error bars represent the standard error of mean over 10 random seeds. Similar constraints arise in designing of machine-learning￾based teaching curricula for scho… view at source ↗
Figure 2
Figure 2. a) Schematic of the teacher-student model. b) Comparison between the analytical and numerical evaluations of the error ϵf under various number of tasks. Each point represents the errors under a randomly sampled task similarity matrices (C in, Cout) (see Appendix C for implementation details). c) Optimal task order for three task learning. In the white regions, C in is not a positive-definite matrix, hence the tasks … view at source ↗
Figure 3
Figure 3. Linear perturbation analysis of task order dependence. a) Decomposition of the contribution of task similarity to the final error under ρo = 1, m = 0.3. b) Schematic representations of periphery-to-core ordering and core-to-periphery ordering. Circles A, B, ..., I represent tasks and their spatial positions represent similarity between tasks. Here, tasks A-C are central whereas tasks D-I are periphery. c) Schematic … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Optimal task orders for tasks with simple graph-like similarity structures. a,b) Total error ϵf under all task orders when the similarity structure of five tasks follows chain (a) and ring (b) structures. Each gray line represents one of 120 (=5!) task order, while red…
Figure 5
Figure 5. Figure 5: Task order preference in continuous image classification tasks. a–f) Continual learning performance, defined as the average test accuracy across all the tasks after learning, under various task orders. Panels (a, c, e) compare the periphery-to-core rule against the cor…
Figure 6
Figure 6. Figure 6: a, b) The same as Fig. 3e and g, but without normalization. The average correlation dominates the error ϵf , but the order dependence is observed robustly. c) The ratio of networks where the periphery-to-core order outperforms core-to-periphery order. As before, we gen…
Figure 7
Figure 7. Figure 7: Task order preference in continuous image classification tasks. a,b) Average classification performance after continual learning of CIFAR-100 where each task consists of 5 label classifications. Here, we randomly picked 25 labels from CIFAR-100 dataset and generated 5 …
Figure 8
Figure 8. Figure 8: Schematic of the task similarity estimation. From the error in zero-shot transfer ϵµ[Wν] (left), we estimated task similarity ρµν (right) [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Task order preference estimated from various fraction of training data under CIFAR-10 task. For instance, in 0.1% data results depicted in panels a and b, we first used 0.1% of training data (∼ 10 images) for training the network with one task and used 0.1% of training…
Figure 10
Figure 10. Figure 10: Task order preference in continuous image classification tasks where the task similarity was estimated from 1% of training data, as opposed to [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    TeLAPA preserves behaviorally diverse policy neighborhoods in a shared latent space, improving MiniGrid continual RL transfer, revisit recovery, and retention over single-model preservation.

Reference graph

Works this paper leans on

60 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S., Saxe, A

    Advani, M. S., Saxe, A. M., and Sompolinsky, H. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132: 0 428--446, 2020

    2020

  3. [3]

    Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks

    Asanuma, H., Takagi, S., Nagano, Y., Yoshida, Y., Igarashi, Y., and Okada, M. Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks. Journal of the Physical Society of Japan, 90 0 (10): 0 104001, 2021

    2021

  4. [4]

    S., Sohl-Dickstein, J., and Ganguli, S

    Bahri, Y., Kadmon, J., Pennington, J., Schoenholz, S. S., Sohl-Dickstein, J., and Ganguli, S. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11: 0 501--528, 2020

    work page 2020

  5. [5]

    Bell, S. J. and Lawrence, N. D. The effect of task ordering in continual learning. arXiv preprint arXiv:2205.13323, 2022

    arXiv 2022

  6. [6]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  7. [7]

    D., and Sompolinsky, H

    Cohen, U., Chung, S., Lee, D. D., and Sompolinsky, H. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11 0 (1): 0 746, 2020

    work page 2020

  8. [8]

    Elman, J. L. Learning and development in neural networks: The importance of starting small. Cognition, 48 0 (1): 0 71--99, 1993

    work page 1993

Show all 60 references
  1. [9]

    How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp.\ 4028--4079

    Evron, I., Moroshko, E., Ward, R., Srebro, N., and Soudry, D. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp.\ 4028--4079. PMLR, 2022

    2022

  2. [10]

    Continual learning in linear classification on separable data

    Evron, I., Moroshko, E., Buzaglo, G., Khriesh, M., Marjieh, B., Srebro, N., and Soudry, D. Continual learning in linear classification on separable data. In International Conference on Machine Learning, pp.\ 9440--9484. PMLR, 2023

    2023

  3. [11]

    French, R. M. Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks. In Proceedings of the 13th annual cognitive science society conference, volume 1, pp.\ 173--178, 1991

    1991

  4. [12]

    French, R. M. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 0 (4): 0 128--135, 1999

    1999

  5. [13]

    and Derrida, B

    Gardner, E. and Derrida, B. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General, 22 0 (12): 0 1983, 1989

    1983

  6. [14]

    and Hand, P

    Goldfarb, D. and Hand, P. Analysis of catastrophic forgetting for random orthogonal transformation tasks in the overparameterized regime. In International Conference on Artificial Intelligence and Statistics, pp.\ 2975--2993. PMLR, 2023

    2023

  7. [15]

    The joint effect of task similarity and overparameterization on catastrophic forgetting—an analytical model

    Goldfarb, D., Evron, I., Weinberger, N., Soudry, D., and HAnd, P. The joint effect of task similarity and overparameterization on catastrophic forgetting—an analytical model. In The Twelfth International Conference on Learning Representations

  8. [16]

    Dynamic task prioritization for multitask learning

    Guo, M., Haque, A., Huang, D.-A., Yeung, S., and Fei-Fei, L. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pp.\ 270--287, 2018

    2018

  9. [17]

    A., and Pascanu, R

    Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24 0 (12): 0 1028--1040, 2020

    2020

  10. [18]

    F lax: A neural network library and ecosystem for JAX , 2024

    Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Z ee, M. F lax: A neural network library and ecosystem for JAX , 2024. URL http://github.com/google/flax

    2024

  11. [19]

    and Dahmen, D

    Helias, M. and Dahmen, D. Statistical field theory for neural networks, volume 970. Springer, 2020

    2020

  12. [20]

    Disentangling and mitigating the impact of task similarity for continual learning

    Hiratani, N. Disentangling and mitigating the impact of task similarity for continual learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

  13. [21]

    A., Welbl, J., Clark, A., et al

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processi...

    2022

  14. [22]

    How to train your robot with deep reinforcement learning: lessons we have learned

    Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., and Levine, S. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40 0 (4-5): 0 698--721, 2021

    2021

  15. [23]

    Sub-network discovery and soft-masking for continual learning of mixed tasks

    Ke, Z., Liu, B., Xiong, W., Celikyilmaz, A., and Li, H. Sub-network discovery and soft-masking for continual learning of mixed tasks. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  16. [24]

    Continual learning of a mixed sequence of similar and dissimilar tasks

    Ke, Z., Liu, B., and Huang, X. Continual learning of a mixed sequence of similar and dissimilar tasks. Advances in neural information processing systems, 33: 0 18493--18504, 2020

    2020

  17. [25]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 0 (13): 0 3521--3526, 2017

    2017

  18. [26]

    Is continual learning ready for real-world challenges? CoRR, 2024

    Kontogianni, T., Yue, Y., Tang, S., and Schindler, K. Is continual learning ready for real-world challenges? CoRR, 2024

    2024

  19. [27]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  20. [28]

    Krueger, K. A. and Dayan, P. Flexible shaping: How learning in small steps helps. Cognition, 110 0 (3): 0 380--394, 2009

    2009

  21. [29]

    Continual learning in medical imaging analysis: A comprehensive review of recent advancements and future prospects

    Kumari, P., Chauhan, J., Bozorgpour, A., Azad, R., and Merhof, D. Continual learning in medical imaging analysis: A comprehensive review of recent advancements and future prospects. CoRR, 2023

    2023

  22. [30]

    Toward optimal ordering of prediction tasks

    Lad, A., Ghani, R., Yang, Y., and Kisiel, B. Toward optimal ordering of prediction tasks. In Proceedings of the 2009 SIAM International Conference on Data Mining, pp.\ 884--893. SIAM, 2009

    2009

  23. [31]

    Continual learning in the teacher-student setup: Impact of task similarity

    Lee, S., Goldt, S., and Saxe, A. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pp.\ 6109--6119. PMLR, 2021

    2021

  24. [32]

    Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges

    Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., and D \' az-Rodr \' guez, N. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information fusion, 58: 0 52--68, 2020

    2020

  25. [33]

    Statistical mechanics of continual learning: Variational principle and mean-field potential

    Li, C., Huang, Z., Zou, W., and Huang, H. Statistical mechanics of continual learning: Variational principle and mean-field potential. Physical Review E, 108 0 (1): 0 014309, 2023

    2023

  26. [34]

    Beyond not-forgetting: Continual learning with backward knowledge transfer

    Lin, S., Yang, L., Fan, D., and Zhang, J. Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 35: 0 16165--16177, 2022

    2022

  27. [35]

    Theory on forgetting and generalization of continual learning

    Lin, S., Ju, P., Liang, Y., and Shroff, N. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pp.\ 21078--21100. PMLR, 2023

    2023

  28. [36]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning

    Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023

    2023 arXiv

  29. [37]

    and Cohen, N

    McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp.\ 109--165. Elsevier, 1989

    1989

  30. [38]

    S., and Mignacco, F

    Mori, F., Mannelli, S. S., and Mignacco, F. Optimal protocols for continual learning via statistical physics and control theory. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning

    2024

  31. [39]

    E., and Stone, P

    Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21 0 (181): 0 1--50, 2020

    2020

  32. [40]

    The ideal continual learner: An agent that never forgets

    Peng, L., Giampouras, P., and Vidal, R. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, pp.\ 27585--27610. PMLR, 2023

    2023

  33. [41]

    Pentina, A., Sharmanska, V., and Lampert, C. H. Curriculum learning of multiple tasks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 5492--5500, 2015

    2015

  34. [42]

    N., Brunskill, E., Griffiths, T

    Rafferty, A. N., Brunskill, E., Griffiths, T. L., and Shafto, P. Faster teaching via pomdp planning. Cognitive science, 40 0 (6): 0 1290--1332, 2016

    2016

  35. [43]

    V., Dyer, E., and Raghu, M

    Ramasesh, V. V., Dyer, E., and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations

  36. [44]

    Catastrophic forgetting, rehearsal and pseudorehearsal

    Robins, A. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7 0 (2): 0 123--146, 1995

    1995

  37. [45]

    Experience replay for continual learning

    Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019

    2019

  38. [46]

    and Solla, S

    Saad, D. and Solla, S. A. On-line learning in soft committee machines. Physical Review E, 52 0 (4): 0 4225, 1995

    1995

  39. [47]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Saxe, A., McClelland, J., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the International Conference on Learning Represenatations, 2014

    2014

  40. [48]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 0 (23): 0 11537--11546, 2019

    2019

  41. [49]

    Overcoming catastrophic forgetting with hard attention to the task

    Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp.\ 4548--4557. PMLR, 2018

    2018

  42. [50]

    S., Sompolinsky, H., and Tishby, N

    Seung, H. S., Sompolinsky, H., and Tishby, N. Statistical mechanics of learning from examples. Physical review A, 45 0 (8): 0 6056, 1992

    1992

  43. [51]

    K., Kim, J., and Kim, J

    Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017

    2017

  44. [52]

    W., Gao, D., Talbot, M

    Singh, P., Li, Y., Sikarwar, A., Lei, S. W., Gao, D., Talbot, M. B., Sun, Y., Shou, M. Z., Kreiman, G., and Zhang, M. Learning to learn: How to continuously teach humans and machines. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 11708--11719, 2023

    2023

  45. [53]

    Understanding self-supervised learning dynamics without contrastive pairs

    Tian, Y., Chen, X., and Ganguli, S. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp.\ 10268--10278. PMLR, 2021

    2021

  46. [54]

    Van de Ven, G. M. and Tolias, A. S. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019

    1904 arXiv

  47. [55]

    Clad: A realistic continual learning benchmark for autonomous driving

    Verwimp, E., Yang, K., Parisot, S., Hong, L., McDonagh, S., P \'e rez-Pellitero, E., De Lange, M., and Tuytelaars, T. Clad: A realistic continual learning benchmark for autonomous driving. Neural Networks, 161: 0 659--669, 2023

    2023

  48. [56]

    Learning curves for stochastic gradient descent in linear feedforward networks

    Werfel, J., Xie, X., and Seung, H. Learning curves for stochastic gradient descent in linear feedforward networks. Advances in neural information processing systems, 16, 2003

    2003

  49. [57]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  50. [58]

    On compressing deep models by low rank and sparse decomposition

    Yu, X., Liu, T., Wang, X., and Tao, D. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 7370--7379, 2017

    2017

  51. [59]

    and Krzakala, F

    Zdeborov \'a , L. and Krzakala, F. Statistical physics of inference: Thresholds and algorithms. Advances in Physics, 65 0 (5): 0 453--552, 2016

    2016

  52. [60]

    Zhu, X., Singla, A., Zilles, S., and Rafferty, A. N. An overview of machine teaching. arXiv preprint arXiv:1801.05927, 2018

    2018 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.