Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Achieving Deep Continual Learning via Evolution

T0 review · 5 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that evolving a fresh neural architecture for each task and archiving it as a dedicated expert solves continual learning's stability-plasticity dilemma without replay.

desk verdict ECL is a clean per-task evolutionary NAS + expert-archiving recipe that deserves serious review, but the headline SOTA claims currently rest on un-matched baselines and no variance estimates. read the letter →

arxiv 2502.06210 v2 pith:RZKHOAEL submitted 2025-02-10 cs.LG

Aojun Lu , Junchao Ke , Chunhui Ding , Jiahao Fan , Jiancheng Lv , Yanan Sun This is my paper · ORCID
classification cs.LG
keywords continuallearningevolutionaryalgorithmsneuralarchitecturesearchcatastrophicforgettingtask-incrementalclass-incrementalexpertmodelsreplay-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that continual learning does not have to be squeezed into one model: instead, a population of candidate architectures can evolve a specialized network for each incoming task, and the finished models are archived as experts. It argues that this dissolves catastrophic forgetting by construction, because old experts are never retrained, and boosts plasticity, because each task gets an architecture tuned to its own data distribution. The payoff, if true, is a replay-free continual learner that beats single-model methods on standard benchmarks while keeping total parameters low.

What carries the argument

The engine is an evolutionary architecture-search loop. A population of cell-based networks, initialized randomly for the first task and inherited thereafter, is evolved for a fixed number of generations per task: parents are selected by fitness, offspring are produced by crossover and mutation, and the next population is chosen from the combined pool. Fitness is multi-objective, balancing validation error on the current task against parameter count, so the search favors both accuracy and cheap models. The top architecture is then trained for 200 epochs and archived, while the rest of the population passes to the next task, which is how earlier architectural discoveries propagate forward. The key enabling assumption is that a 10-epoch fitness estimate predicts final converged performance, which is what makes the search affordable.

What would settle it

A matched-budget head-to-head would settle it: give the strongest single-model baseline the same total compute and parameter budget ECL uses, including the search epochs, train it on the same 10-task sequence, and compare Last Accuracy. If the baseline matches or exceeds ECL under equal budget, the claimed advantage is an artifact of evaluation budget.

Watch

Extended reading notes

Core claim

ECL claims that running an evolutionary architecture search per task, then training the winning architecture to convergence and storing it as a dedicated expert, yields higher Last Accuracy than existing single-model continual learning methods in both Task-IL and Class-IL settings. On CIFAR-100 the reported gain over the next-best method is +15.9% in Task-IL and +0.4% in Class-IL; on Tiny-ImageNet it is +27.8% and +6.6%. The method is rehearsal-free, and the ten CIFAR-100 experts together contain only 4.04M parameters, fewer than a standard ResNet-18 backbone. In the authors' own comparison, the improvement comes from the evolved architectures rather than merely from using multiple models: a multi-ResNet-32 baseline with one fixed architecture per task scores lower per-task accuracy and confuses task identities more often.

Load-bearing premise

The load-bearing premise is that the published baseline numbers are a fair yardstick: ECL invests heavily in architecture search and trains one full model per task, so if the comparison does not control for compute and capacity, the accuracy gaps could come from budget rather than from evolution.

Editorial extensions

If this is right

  • If the reported gains hold under fair comparison, replay-free continual learning can be achieved by per-task architectural specialization rather than by protecting a shared model's weights.
  • ECL's parameter efficiency suggests a growing library of experts can stay competitive with a single large backbone, so the approach is not automatically ruled out by memory constraints.
  • Better task discrimination in Class-IL follows from the architecture search, which would make task-ID inference easier for any downstream selection mechanism.
  • Population inheritance provides a concrete mechanism for forward transfer across tasks: solutions that were good for earlier tasks seed the search for later ones.
  • The method turns continual learning into a per-task architecture search problem, connecting CL to AutoML and making future progress in NAS directly applicable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit is that ECL's compute profile is dominated by search (20 candidates, up to 10 generations per task, 10-epoch fitness runs), so its practical advantage depends on whether that compute is affordable relative to the task sequence.
  • If ECL is right, hybrid designs become natural: use evolution to spawn new experts only when a shared model's plasticity fails, rather than for every task, which would reduce total search cost.
  • The Class-IL confidence-based expert selection could be stress-tested as the number of tasks grows; the paper's task-confusion matrices cover 10 tasks, and scaling to dozens of tasks is an untested extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes Evolving Continual Learning (ECL), in which a population of cell-based architectures is evolved with a multi-objective evolutionary algorithm for each new task; the best architecture is then trained and archived as a task-specific expert, and the population is inherited by the next task. At inference, Task-IL uses the provided task identity, while Class-IL selects the expert with the highest prediction confidence. The authors report Last Accuracy gains over several single-model continual learning baselines on CIFAR-100 and Tiny-ImageNet, and they claim a total of only 4.04M parameters for the ten CIFAR-100 experts.

Significance. If the empirical claims were established, ECL would be a meaningful contribution to continual learning: it is replay-free, decouples stability from plasticity by construction, and brings a neural-architecture-search mechanism into the continual learning setting in a way that is not present in most single-model baselines. The comparison against a multi-ResNet-32 baseline is also a useful attempt to isolate the contribution of architecture search. However, the central evidence is currently not sufficient: the baseline comparisons are not compute-matched, no variability information is reported, and the architecture-search component is not cleanly separated from capacity and training-budget effects. The conceptual idea is worth taking seriously, but the manuscript needs substantial additional experiments and reporting before the claimed state-of-the-art results can be accepted.

major comments (5)
  1. [§4.4, §2.1] The claimed superiority over baselines is not established because ECL and the baselines are not compared under matched conditions. Section 4.4 says the Task-IL and Class-IL baseline results are "sourced from prior studies" and use AlexNet or ResNet-18, whereas ECL spends an additional compute budget of population size 20, up to 10 generations for task 1 and 5 for later tasks, 10-epoch fitness training per candidate, and 200-epoch final training for each per-task expert. No FLOPs, wall-clock time, or search cost is reported. The reported margins (+15.9%, +27.8% Task-IL; +0.4%, +6.6% Class-IL) could therefore be explained by evaluation budget and per-task isolation rather than by the evolutionary search mechanism. Please rerun the baselines under matched compute/backbone conditions, or explicitly bound the budget difference and show that the conclusions are insensitive to it.
  2. [§2.1] No measure of variability is reported anywhere in Section 2. The claim that ECL "consistently and significantly outperforms" all competing single-model methods is unsupported without error bars, confidence intervals, or the number of seeds over the Last Accuracy metric. The 0.4% Class-IL advantage on CIFAR-100 is within the typical run-to-run variation of such benchmarks, so the word "significantly" should be removed or justified with a proper statistical comparison.
  3. [§2.2, §4.4] The control against a multi-ResNet-32 baseline does not isolate the contribution of architecture search. The final ECL experts are trained for 200 epochs with channel width 64 and N=3, while the ResNet-32 baseline's training protocol, capacity, and number of epochs are not matched; if the ResNet-32 baseline is trained for fewer epochs or has a different capacity, the per-task accuracy gap in Figure 3(a) may reflect training budget or model size rather than the evolved architecture. A stronger control would use a fixed per-task architecture trained with the same epochs, batch size, and parameter budget, or an ablation in which ECL's search is replaced by random architecture selection while keeping everything else constant.
  4. [§4.3, §4.4] The construction of the validation split used for architecture selection is unspecified. Section 4.3 says each candidate is evaluated on the validation set of the current task, but Section 4.4 does not state how that validation set is carved out of the task's training data, whether it is disjoint from the test set, or whether it contains the same class distribution as the training portion. Without this information, the architecture-selection step may overfit the test distribution, and the reported gains may not transfer to a correctly held-out evaluation.
  5. [§4.4] The search and final-training configurations differ substantially: during search, models use channel width 16 and N=1, while final experts are trained with channel width 64 and N=3. Because the fitness that drives architecture selection is measured on the smaller configuration, the architecture chosen as "best" is not the architecture whose accuracy is reported after final training. The paper should either justify this transfer or search at the final scale, and if the transfer is retained, it should report both the early-search and final-convergence performance for the selected architectures.
minor comments (4)
  1. [§1] There is a grammatical error in Section 1: "the prevailing CL paradigm remains focuses on" should read "remains focused on".
  2. [§2.1] The paper writes "CIF AR-100" with an unwanted space in several places; please standardize to "CIFAR-100".
  3. [§2.2] The sentence "...where task identity is not explicitly provided.." ends with two periods; please correct the punctuation.
  4. [Figure 2(b), §4.4] The paper reports only bar charts and no numeric table of Last Accuracy values with standard deviations; including such a table would greatly improve reproducibility and allow readers to assess the size of the reported margins.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ECL's reported benchmark gains are empirical comparisons to external results, not consequences of its own definitions or fits.

full rationale

The paper's central claim (Section 2.1) is that ECL outperforms single-model CL methods on Last Accuracy. That claim is supported by test-set measurements against baselines that, as Section 4.4 states, 'are sourced from prior studies [25, 40]'; nothing in the method's definitions makes those numbers equal to ECL's fitness values. The fitness proxy from [33] (10-epoch training during search) selects an architecture, but the reported accuracy comes from a separate 200-epoch training and test evaluation, so the outcome is not fixed by construction. The only self-citation is [32], cited alongside external reference [31] to justify variable cell depth in the search space (Section 4.2); it is a design motivation and does not carry the performance claim. The multi-model control in Section 2.2 is a direct experiment rather than a definitional identity. The comparability limitation that baselines were not rerun under ECL's compute budget is an evidence-quality concern, not a circularity, and does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

This is an empirical machine learning paper with no invented physical or mathematical entities. The free parameters are hyperparameters and search-budget choices that are plausible but not systematically ablated. The main load-bearing assumptions are the early-training fitness proxy, validation-set independence, confidence-based expert selection, and comparability of externally sourced baselines.

free parameters (5)
  • Search population size and generations = population 20; 10 generations for first task, 5 for later tasks
    Chosen without sensitivity analysis in the paper; the search budget directly affects both performance and fairness of the comparison.
  • Fitness evaluation training epochs = 10 epochs
    Used as a cheap proxy for final accuracy; this is a core design choice and is not validated for the CL benchmarks.
  • Final training epochs = 200 epochs
    Applied to the selected expert; the full training budget is part of why ECL may not be compute-comparable to baselines.
  • Final architecture capacity = initial width 16, final width 64, N=1 during search, N=3 for final
    These capacity settings determine per-task accuracy and the reported memory claim of 4.04M parameters.
  • Cell node range = 5 to 12 nodes per cell
    The chosen search-space size controls depth; no ablation shows how sensitive results are to this range.
assumptions (4)
  • domain assumption Early 10-epoch validation error is correlated with final 200-epoch converged accuracy.
    Invoked in Section 4.3 via reference [33]; if false, the search may select suboptimal architectures even when the final training is strong.
  • domain assumption The validation set used during architecture search is independent of the data used for final training.
    Section 4.3 does not describe a held-out split; standard NAS assumes this, and violating it would optimistically bias the selected architecture's accuracy.
  • domain assumption Confidence-based expert selection identifies the correct task in Class-IL inference.
    Section 2.1 and Figure 2(a) describe selecting the highest-confidence expert; the paper evaluates confusion rates but provides no guarantee or bound.
  • domain assumption Baseline accuracy numbers from prior studies [25,40] are directly comparable without a matched compute budget.
    Section 4.4 sources baselines from prior papers, which is necessary for the headline Last Accuracy comparison; unequal budgets would invalidate the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Achieving Deep Continual Learning via Evolution." pith.science (2026). https://pith.science/paper/RZKHOAEL

@misc{pith2026250206210,
  author       = {Pith},
  title        = {Pith review of: Achieving Deep Continual Learning via Evolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RZKHOAEL}},
  note         = {Machine review of arXiv:2502.06210}
}
read the original abstract

Deep neural networks, despite their remarkable success, remain fundamentally limited in their ability to perform Continual Learning (CL). While most current methods aim to enhance the capabilities of a single model, Inspired by the collective learning mechanisms of human populations, we introduce Evolving Continual Learning (ECL), a framework that maintains and evolves a diverse population of neural network models. ECL continually searches for an optimal architecture for each introduced incremental task. This tailored model is trained on the corresponding task and archived as a specialized expert, contributing to a growing collection of skills. This approach inherently resolves the core CL challenges: stability is achieved through the isolation of expert models, while plasticity is greatly enhanced by evolving unique, task-specific architectures. Experimental results demonstrate that ECL significantly outperforms state-of-the-art individual-level CL methods. By shifting the focus from individual adaptation to collective evolution, ECL presents a novel path toward AI systems capable of CL.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SEAL: Searching Expandable Architectures for Incremental Learning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SEAL jointly searches a neural network architecture and an expansion policy, expanding the network only when a capacity threshold is exceeded, and reports competitive accuracy with lower average forgetting on CIFAR-10...

Reference graph

Works this paper leans on

42 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    M., Tuytelaars, T

    Van de Ven, G. M., Tuytelaars, T. & Tolias, A. S. Three types of incremental learning. Nature Machine Intelligence 4, 1185–1197 (2022)

    work page 2022

  2. [2]

    & Zhu, J

    Wang, L., Zhang, X., Su, H. & Zhu, J. A comprehensive survey of continual learn- ing: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    work page 2024

  3. [3]

    Silver, D. L. Machine lifelong learning: Challenges and benefits for artificial gen- eral intelligence. In Artificial General Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3-6, 2011. Proceedings 4 , 370–375 (Springer, 2011)

    work page 2011

  4. [4]

    & Grefenstette, E

    Jiang, M., Rockt¨ aschel, T. & Grefenstette, E. General intelligence requires rethinking exploration. Royal Society Open Science 10, 230539 (2023)

    work page 2023

  5. [5]

    & Cohen, N

    McCloskey, M. & Cohen, N. J. in Catastrophic interference in connectionist net- works: The sequential learning problem In Psychology of learning and motivation, Vol. 24 109–165 (Elsevier, 1989)

    work page 1989

  6. [6]

    J., Mirza, M., Xiao, D., Courville, A

    Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A. & Bengio, Y. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013). 10

    arXiv 2013

  7. [7]

    Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize a changing world

    Grossberg, S. Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize a changing world. Neural networks 37, 1–47 (2013)

    work page 2013

  8. [8]

    Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (2017)

    work page 2017

Show all 42 references
  1. [9]

    K., Ajanthan, T

    Chaudhry, A., Dokania, P. K., Ajanthan, T. & Torr, P. H. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV) , 532–547 (2018)

    2018

  2. [10]

    & Lampert, C

    Rebuffi, S.-A., Kolesnikov, A., Sperl, G. & Lampert, C. H. icarl: Incremental classifier and representation learning . In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , 2001–2010 (2017)

    2017

  3. [11]

    Wang, L. et al. Memory replay with data compression for continual learning . In International Conference on Learning Representations (2021)

    2021

  4. [12]

    & Wayne, G

    Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. & Wayne, G. Experience replay for continual learning. Advances in neural information processing systems 32 (2019)

    2019

  5. [13]

    & Karatzoglou, A

    Serra, J., Suris, D., Miron, M. & Karatzoglou, A. Overcoming catastrophic for- getting with hard attention to the task . In International conference on machine learning, 4548–4557 (PMLR, 2018)

    2018

  6. [14]

    Yan, S., Xie, J. & He, X. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3014–3023 (2021)

    2021

  7. [15]

    M., Siegelmann, H

    Van de Ven, G. M., Siegelmann, H. T. & Tolias, A. S. Brain-inspired replay for continual learning with artificial neural networks. Nature communications 11, 4069 (2020)

    2020

  8. [16]

    Gurbuz, M. B. & Dovrolis, C. Nispa: Neuro-inspired stability-plasticity adapta- tion for continual learning in sparse networks . In International Conference on Machine Learning, 8157–8174 (PMLR, 2022)

    2022

  9. [17]

    Kudithipudi, D. et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence 4, 196–210 (2022)

    2022

  10. [18]

    Hadsell, R., Rao, D., Rusu, A. A. & Pascanu, R. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences 24, 1028–1040 (2020)

    2020

  11. [19]

    & Ganguli, S

    Zenke, F., Poole, B. & Ganguli, S. Continual learning through synaptic intel- ligence. In International conference on machine learning , 3987–3995 (PMLR, 2017). 11

    2017

  12. [20]

    Wang, L. et al. Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence 5, 1356–1368 (2023)

    2023

  13. [21]

    Dohare, S. et al. Loss of plasticity in deep continual learning. Nature 632, 768–774 (2024)

    2024

  14. [22]

    & Jozefiak, A

    Farias, V. & Jozefiak, A. D. Self-normalized resets for plasticity in continual learn- ing. In The Thirteenth International Conference on Learning Representations (2025)

    2025

  15. [23]

    Zoph, B. & Le, Q. Neural architecture search with reinforcement learning . In International Conference on Learning Representations (2017)

    2017

  16. [24]

    Ren, P. et al. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) 54, 1–34 (2021)

    2021

  17. [25]

    Konishi, T. et al. Parameter-level soft-masking for continual learning . In International Conference on Machine Learning , 17492–17505 (PMLR, 2023)

    2023

  18. [26]

    & Meyarivan, T

    Deb, K., Pratap, A., Agarwal, S. & Meyarivan, T. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation 6, 182–197 (2002)

    2002

  19. [27]

    P., Girshick, R., He, K

    Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K. & Doll´ ar, P. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10428–10436 (2020)

    2020

  20. [28]

    Wan, X., Ru, B., Esperan¸ ca, P. M. & Li, Z. On redundancy and diversity in cell-based neural architecture search . In International Conference on Learning Representations (2022)

    2022

  21. [29]

    Zoph, B., Vasudevan, V., Shlens, J. & Le, Q. V.Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 8697–8710 (2018)

    2018

  22. [30]

    & Yang, Y

    Liu, H., Simonyan, K. & Yang, Y. Darts: Differentiable architecture search . In International Conference on Learning Representations (2019)

    2019

  23. [31]

    Mirzadeh, S. I. et al. Wide neural networks forget less catastrophically . In International Conference on Machine Learning , 15699–15717 (PMLR, 2022)

    2022

  24. [32]

    & Sun, Y

    Lu, A., Feng, T., Yuan, H., Song, X. & Sun, Y. Revisiting neural networks for continual learning: an architectural perspective. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , 4651–4659 (2024)

    2024

  25. [33]

    Xue, Y. et al. Neural architecture search with progressive evaluation and sub-population preservation. IEEE Transactions on Evolutionary Computation (2024). 12

    2024

  26. [34]

    Wortsman, M. et al. Supermasks in superposition. Advances in Neural Information Processing Systems 33, 15173–15184 (2020)

    2020

  27. [35]

    Kang, H. et al. Forget-free continual learning with winning subnetworks . In International Conference on Machine Learning , 10734–10750 (PMLR, 2022)

    2022

  28. [36]

    & Liu, C.-L

    Zhu, F., Zhang, X.-Y., Wang, C., Yin, F. & Liu, C.-L. Prototype augmentation and self-supervision for incremental learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    2021

  29. [37]

    & Delezoide, B

    Petit, G., Popescu, A., Schindler, H., Picard, D. & Delezoide, B. Fetril: Fea- ture translation for exemplar-free class-incremental learning . In Proceedings of the IEEE/CVF winter conference on applications of computer vision , 3911–3920 (2023)

    2023

  30. [38]

    & van de Weijer, J

    Goswami, D., Liu, Y., Twardowski, B. & van de Weijer, J. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  31. [39]

    & Bagdanov, A

    Magistri, S., Trinci, T., Soutif, A., van de Weijer, J. & Bagdanov, A. D. Elastic feature consolidation for cold start exemplar-free incremental learning . In The Twelfth International Conference on Learning Representations (2024)

    2024

  32. [40]

    Gomez-Villa, A. et al. Exemplar-free continual representation learning via learn- able drift compensation . In European Conference on Computer Vision , 473–490 (Springer, 2025)

    2025

  33. [41]

    Learning multiple layers of features from tiny images (2009)

    Krizhevsky, A. Learning multiple layers of features from tiny images (2009). URL https://api.semanticscholar.org/CorpusID:18268744

    2009

  34. [42]

    & Yang, X

    Le, Y. & Yang, X. Tiny imagenet visual recognition challenge. CS 231N 7, 3 (2015). 13

    2015

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.