{"id":"c57ba428-c882-485c-935b-482fcf2055b8","arxiv_id":"2502.06210","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ECL searches for a separate evolved neural network architecture for each task, archives it as an expert, and reports replay-free continual learning gains over single-model baselines on CIFAR-100 and Tiny-ImageNet.","lead":"This paper proposes ECL, a continual learning method that evolves a population of neural network architectures and archives one specialized model per incoming task. It reports large accuracy gains over single-model continual learning baselines on CIFAR-100 and Tiny-ImageNet, but the comparison does not control compute or report error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline results are taken from prior studies without matching compute or search budget, so the reported SOTA gains may reflect evaluation budget rather than the evolutionary mechanism.","rationale":"The reader's weakest assumption — that baseline numbers from prior studies are not directly comparable because ECL uses far more compute and one model per task — is exactly the most load-bearing concern I find. The paper's own Sec. 4.4 says baseline results 'are sourced from prior studies' and do not appear to be rerun under matched conditions. The multi-model control in Sec. 2.2 is a useful step, but it is only one fixed architecture (ResNet-32) and is not a strong per-task baseline, so it does not resolve the budget confound. I found no internal logical inconsistency in the method; the risk is experimental fairness and reproducibility. The 0.4% Class-IL CIFAR-100 gap could easily be noise, and no code, data, or error bars are provided. Since the reader already issued a CONDITIONAL verdict and my concern lands on the same point, the appropriate action is to keep the verdict unchanged pending a compute-matched rerun and release of artifacts. If such a rerun shows the gap disappears, the verdict should move to REJECT. The concrete test I propose directly checks whether the comparison survives a matched budget.","tokens_in":8173,"tokens_out":4329,"duration_ms":42469,"concrete_test":"Rerun the leading Class-IL baseline (e.g., LDC) on CIFAR-100 under ECL's exact data order, optimizer, and compute envelope (same total training epochs/FLOPs), or equivalently run ECL with search disabled (population 1, no generations) while keeping the same 200-epoch final training and per-task inference. Compare Last Accuracy over at least 3 seeds. If the ECL margin over LDC shrinks to within seed noise or reverses, the headline comparison is not settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ECL 'consistently and significantly outperforms' single-model CL methods rests on comparisons to numbers 'sourced from prior studies' (Sec. 4.4). Those baselines use AlexNet (Task-IL) or ResNet-18 (Class-IL) and are trained once. ECL, by contrast, spends a large additional compute budget: a population of 20, up to 10 generations of search for the first task and 5 for later tasks, 10-epoch fitness training per candidate, plus 200-epoch final training for each per-task expert. No FLOPs, runtime, or seed variance is reported. If the comparison is not compute-matched, the reported gains (Task-IL: +15.9%, +27.8%; Class-IL: +0.4%, +6.6%) could stem from the extra search budget and from per-task isolation rather than from the evolutionary search itself. The 0.4% Class-IL CIFAR-100 margin is especially fragile without error bars. The control in Sec. 2.2 compares only against a fixed multi-ResNet-32, not a strong per-task fixed-architecture baseline, so the specific contribution of architecture search is not cleanly isolated. This is an evidence/correctness concern, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Evolving Continual Learning (ECL), in which a population of cell-based architectures is evolved with a multi-objective evolutionary algorithm for each new task; the best architecture is then trained and archived as a task-specific expert, and the population is inherited by the next task. At inference, Task-IL uses the provided task identity, while Class-IL selects the expert with the highest prediction confidence. The authors report Last Accuracy gains over several single-model continual learning baselines on CIFAR-100 and Tiny-ImageNet, and they claim a total of only 4.04M parameters for the ten CIFAR-100 experts.","tokens_in":8395,"tokens_out":5445,"duration_ms":50608,"significance":"If the empirical claims were established, ECL would be a meaningful contribution to continual learning: it is replay-free, decouples stability from plasticity by construction, and brings a neural-architecture-search mechanism into the continual learning setting in a way that is not present in most single-model baselines. The comparison against a multi-ResNet-32 baseline is also a useful attempt to isolate the contribution of architecture search. However, the central evidence is currently not sufficient: the baseline comparisons are not compute-matched, no variability information is reported, and the architecture-search component is not cleanly separated from capacity and training-budget effects. The conceptual idea is worth taking seriously, but the manuscript needs substantial additional experiments and reporting before the claimed state-of-the-art results can be accepted.","major_comments":[{"comment":"The claimed superiority over baselines is not established because ECL and the baselines are not compared under matched conditions. Section 4.4 says the Task-IL and Class-IL baseline results are \"sourced from prior studies\" and use AlexNet or ResNet-18, whereas ECL spends an additional compute budget of population size 20, up to 10 generations for task 1 and 5 for later tasks, 10-epoch fitness training per candidate, and 200-epoch final training for each per-task expert. No FLOPs, wall-clock time, or search cost is reported. The reported margins (+15.9%, +27.8% Task-IL; +0.4%, +6.6% Class-IL) could therefore be explained by evaluation budget and per-task isolation rather than by the evolutionary search mechanism. Please rerun the baselines under matched compute/backbone conditions, or explicitly bound the budget difference and show that the conclusions are insensitive to it.","section":"§4.4, §2.1"},{"comment":"No measure of variability is reported anywhere in Section 2. The claim that ECL \"consistently and significantly outperforms\" all competing single-model methods is unsupported without error bars, confidence intervals, or the number of seeds over the Last Accuracy metric. The 0.4% Class-IL advantage on CIFAR-100 is within the typical run-to-run variation of such benchmarks, so the word \"significantly\" should be removed or justified with a proper statistical comparison.","section":"§2.1"},{"comment":"The control against a multi-ResNet-32 baseline does not isolate the contribution of architecture search. The final ECL experts are trained for 200 epochs with channel width 64 and N=3, while the ResNet-32 baseline's training protocol, capacity, and number of epochs are not matched; if the ResNet-32 baseline is trained for fewer epochs or has a different capacity, the per-task accuracy gap in Figure 3(a) may reflect training budget or model size rather than the evolved architecture. A stronger control would use a fixed per-task architecture trained with the same epochs, batch size, and parameter budget, or an ablation in which ECL's search is replaced by random architecture selection while keeping everything else constant.","section":"§2.2, §4.4"},{"comment":"The construction of the validation split used for architecture selection is unspecified. Section 4.3 says each candidate is evaluated on the validation set of the current task, but Section 4.4 does not state how that validation set is carved out of the task's training data, whether it is disjoint from the test set, or whether it contains the same class distribution as the training portion. Without this information, the architecture-selection step may overfit the test distribution, and the reported gains may not transfer to a correctly held-out evaluation.","section":"§4.3, §4.4"},{"comment":"The search and final-training configurations differ substantially: during search, models use channel width 16 and N=1, while final experts are trained with channel width 64 and N=3. Because the fitness that drives architecture selection is measured on the smaller configuration, the architecture chosen as \"best\" is not the architecture whose accuracy is reported after final training. The paper should either justify this transfer or search at the final scale, and if the transfer is retained, it should report both the early-search and final-convergence performance for the selected architectures.","section":"§4.4"}],"minor_comments":[{"comment":"There is a grammatical error in Section 1: \"the prevailing CL paradigm remains focuses on\" should read \"remains focused on\".","section":"§1"},{"comment":"The paper writes \"CIF AR-100\" with an unwanted space in several places; please standardize to \"CIFAR-100\".","section":"§2.1"},{"comment":"The sentence \"...where task identity is not explicitly provided..\" ends with two periods; please correct the punctuation.","section":"§2.2"},{"comment":"The paper reports only bar charts and no numeric table of Last Accuracy values with standard deviations; including such a table would greatly improve reproducibility and allow readers to assess the size of the reported margins.","section":"Figure 2(b), §4.4"}],"recommendation":"major_revision","confidential_remarks":"The main weakness is evidentiary rather than conceptual. I see no circularity in the core argument: the paper's contribution is empirical and its numbers are compared against external benchmarks. The framework is novel enough to warrant a major revision rather than rejection, but the authors must provide matched-condition baselines, statistical error bars, and a precise description of the validation split before the central claim can be trusted. If the authors cannot rerun the baselines due to compute constraints, they should at least weaken the 'consistently and significantly outperforms' language and make the compute asymmetry explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ECL is a clean and useful recipe—per-task evolutionary NAS plus expert archiving—but the headline numbers currently rest on un-matched baselines and no variance estimates, so treat the SOTA claims as unproven until the authors rerun the comparison under equal budget and report seeds.\n\nWhat's actually new: the specific integration of an NSGA-II-style population search, variable-depth cells, and per-task expert archival for replay-free CL. The ingredients are known—parameter isolation, NAS, early-stage fitness proxies—but the combination is not in the cited literature, and the multi-objective pressure on parameter count is a nice touch. The framework is straightforward to explain, the algorithm is concrete, and the parameter efficiency claim (4.04M total for ten CIFAR-100 experts) is genuinely interesting.\n\nWhere it gets soft: the central comparison to 'state-of-the-art single-model methods' reuses baseline numbers from [25,40] with no FLOPs, runtime, or seed variance. That is the load-bearing wall of the paper. ECL spends a population of 20, 10+5 generations, 10-epoch fitness training, then 200-epoch final training; the baselines are trained once. Without a compute-matched rerun, the Task-IL margins (+15.9%, +27.8%) and even the Class-IL +6.6% are hard to attribute to the evolutionary mechanism. The 0.4% Class-IL CIFAR-100 edge is within noise for all we know—no error bars anywhere. The control against multi-ResNet-32 is a step in the right direction, but ResNet-32 is a weak per-task baseline; a stronger fixed per-task architecture would isolate the value of architecture search more convincingly. No code or data is released either.\n\nNone of this is an internal contradiction; the method is coherent and the paper is honest about sourcing baselines. It just needs empirical rigor. For a reader in CL or NAS, this is a worthwhile paper to know. It deserves peer review and likely conditional acceptance with matched-budget comparisons, error bars, and released artifacts. I'd bring it to reading group once the authors supply those.","headline":"ECL is a clean per-task evolutionary NAS + expert-archiving recipe that deserves serious review, but the headline SOTA claims currently rest on un-matched baselines and no variance estimates.","tokens_in":8944,"tokens_out":1849,"would_cite":false,"duration_ms":16551,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that evolving a fresh neural architecture for each task and archiving it as a dedicated expert solves continual learning's stability-plasticity dilemma without replay.","keywords":["continual learning","evolutionary algorithms","neural architecture search","catastrophic forgetting","task-incremental learning","class-incremental learning","expert models","replay-free learning"],"falsifier":"A matched-budget head-to-head would settle it: give the strongest single-model baseline the same total compute and parameter budget ECL uses, including the search epochs, train it on the same 10-task sequence, and compare Last Accuracy. If the baseline matches or exceeds ECL under equal budget, the claimed advantage is an artifact of evaluation budget.","tokens_in":7958,"feed_emoji":"🧬","tokens_out":5376,"duration_ms":46084,"temperature":0.7,"pith_summary":"This paper tries to show that continual learning does not have to be squeezed into one model: instead, a population of candidate architectures can evolve a specialized network for each incoming task, and the finished models are archived as experts. It argues that this dissolves catastrophic forgetting by construction, because old experts are never retrained, and boosts plasticity, because each task gets an architecture tuned to its own data distribution. The payoff, if true, is a replay-free continual learner that beats single-model methods on standard benchmarks while keeping total parameters low.","feed_headline":"Evolving a new network per task beats continual learning baselines","feed_subtitle":"Per-task evolved experts gain up to 27.8% in accuracy on Tiny-ImageNet, with only 4.04M parameters on CIFAR-100.","key_machinery":"The engine is an evolutionary architecture-search loop. A population of cell-based networks, initialized randomly for the first task and inherited thereafter, is evolved for a fixed number of generations per task: parents are selected by fitness, offspring are produced by crossover and mutation, and the next population is chosen from the combined pool. Fitness is multi-objective, balancing validation error on the current task against parameter count, so the search favors both accuracy and cheap models. The top architecture is then trained for 200 epochs and archived, while the rest of the population passes to the next task, which is how earlier architectural discoveries propagate forward. The key enabling assumption is that a 10-epoch fitness estimate predicts final converged performance, which is what makes the search affordable.","core_discovery":"ECL claims that running an evolutionary architecture search per task, then training the winning architecture to convergence and storing it as a dedicated expert, yields higher Last Accuracy than existing single-model continual learning methods in both Task-IL and Class-IL settings. On CIFAR-100 the reported gain over the next-best method is +15.9% in Task-IL and +0.4% in Class-IL; on Tiny-ImageNet it is +27.8% and +6.6%. The method is rehearsal-free, and the ten CIFAR-100 experts together contain only 4.04M parameters, fewer than a standard ResNet-18 backbone. In the authors' own comparison, the improvement comes from the evolved architectures rather than merely from using multiple models: a multi-ResNet-32 baseline with one fixed architecture per task scores lower per-task accuracy and confuses task identities more often.","pith_inferences":["A consequence the paper leaves implicit is that ECL's compute profile is dominated by search (20 candidates, up to 10 generations per task, 10-epoch fitness runs), so its practical advantage depends on whether that compute is affordable relative to the task sequence.","If ECL is right, hybrid designs become natural: use evolution to spawn new experts only when a shared model's plasticity fails, rather than for every task, which would reduce total search cost.","The Class-IL confidence-based expert selection could be stress-tested as the number of tasks grows; the paper's task-confusion matrices cover 10 tasks, and scaling to dozens of tasks is an untested extension."],"forward_implications":["If the reported gains hold under fair comparison, replay-free continual learning can be achieved by per-task architectural specialization rather than by protecting a shared model's weights.","ECL's parameter efficiency suggests a growing library of experts can stay competitive with a single large backbone, so the approach is not automatically ruled out by memory constraints.","Better task discrimination in Class-IL follows from the architecture search, which would make task-ID inference easier for any downstream selection mechanism.","Population inheritance provides a concrete mechanism for forward transfer across tasks: solutions that were good for earlier tasks seed the search for later ones.","The method turns continual learning into a per-task architecture search problem, connecting CL to AutoML and making future progress in NAS directly applicable."],"supporting_citations":[{"why":"Supplies the Task-IL baseline results and the experimental protocol whose reported numbers ECL is compared against.","marker":"[25]"},{"why":"Supplies the Class-IL baseline results and the experimental protocol whose reported numbers ECL is compared against.","marker":"[40]"},{"why":"Supplies the multi-objective evolutionary algorithm used to rank candidate architectures by error rate and parameter count.","marker":"[26]"},{"why":"Defines the cell-based search-space design that ECL adapts, including normal and reduction cells and operation choices.","marker":"[30]"},{"why":"Provides the observation that early-stage training performance predicts final performance, justifying the cheap 10-epoch fitness evaluation.","marker":"[33]"},{"why":"Provides the CIFAR-100 benchmark dataset used for the main continual learning comparisons.","marker":"[41]"},{"why":"Provides the Tiny-ImageNet benchmark dataset used for the main continual learning comparisons.","marker":"[42]"}],"fun_headline_variants":["Evolve a net per task: Continual learning wins big","Per-task evolution gives AI a continual learning edge","Collective evolution: New path to continual learning","Evolving experts beat continual learning by 27.8%","Forget single models: Evolve a network for each task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the published baseline numbers are a fair yardstick: ECL invests heavily in architecture search and trains one full model per task, so if the comparison does not control for compute and capacity, the accuracy gaps could come from budget rather than from evolution.","fun_headline_variants_meta":{"raw":{"variants":["Evolve a net per task: Continual learning wins big","Per-task evolution gives AI a continual learning edge","Collective evolution: New path to continual learning","Evolving experts beat continual learning by 27.8%","Forget single models: Evolve a network for each task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1184,"prompt_tokens":888,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":215}},"tokens_in":504,"tokens_out":296,"duration_ms":3225,"temperature":1.0,"reasoning_tokens":215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:23:01.376489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A matched-budget head-to-head would settle it: give the strongest single-model baseline the same total compute and parameter budget ECL uses, including the search epochs, train it on the same 10-task sequence, and compare Last Accuracy. If the baseline matches or exceeds ECL under equal budget, the claimed advantage is an artifact of evaluation budget.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Task-IL baseline results and the experimental protocol whose reported numbers ECL is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Class-IL baseline results and the experimental protocol whose reported numbers ECL is compared against."},{"cited_title":"& Meyarivan, T","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-objective evolutionary algorithm used to rank candidate architectures by error rate and parameter count."},{"cited_title":"& Yang, Y","cited_arxiv_id":null,"evidence_quote":"Defines the cell-based search-space design that ECL adapts, including normal and reduction cells and operation choices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the observation that early-stage training performance predicts final performance, justifying the cheap 10-epoch fitness evaluation."},{"cited_title":"Learning multiple layers of features from tiny images (2009)","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR-100 benchmark dataset used for the main continual learning comparisons."},{"cited_title":"& Yang, X","cited_arxiv_id":null,"evidence_quote":"Provides the Tiny-ImageNet benchmark dataset used for the main continual learning comparisons."}],"review_version":1}