{"id":"e6fa6330-7f94-4254-b418-1009a43e39bd","arxiv_id":"2505.10457","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SEAL jointly searches a neural network architecture and an expansion policy, expanding the network only when a capacity threshold is exceeded, and reports competitive accuracy with lower average forgetting on CIFAR-10/100 and ImageNet16-120.","lead":"A new method lets an image classifier automatically decide when to grow larger as new data arrives, instead of growing at every step. Tests on three standard image benchmarks show that it can match or beat several existing continual-learning methods while using smaller models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'expand only when necessary' claim rests on a hand-tuned trigger (Eq. 10) that is not part of the NAS search and is never validated by expansion counts or sensitivity analysis.","rationale":"The reader's weakest_assumption identifies Eq. 10's expansion trigger as relying on a relative loss ratio whose reliability is not established, with dataset-specific thresholds and no sensitivity analysis. My stress-test converges on the same point and sharpens it: tau is not merely an unanalyzed hyperparameter, it is the component of the expansion policy that decides when to expand, and it is excluded from the NAS optimization that the paper claims jointly searches architecture and policy. The paper's own conclusion acknowledges that the trigger could be made more adaptive, which supports the concern. I checked for other possible weaknesses: the comparison omits NAS-based incremental learning baselines, and Table 2 does not report baseline parameter counts despite the 'fewer parameters' claim; these are real limitations but secondary to the expansion-trigger issue. There is no fundamental mathematical error, no circular derivation, and the released code makes the proposed test feasible. Since the reader already marked the paper CONDITIONAL and this concern is the same one, I recommend no change to the verdict.","tokens_in":11807,"tokens_out":5448,"duration_ms":55788,"concrete_test":"Instrument the released SEAL code to log, for each of the K=5 tasks on CIFAR-10, CIFAR-100, and ImageNet16-120, whether Eq. 10 triggers an expansion, using the reported tau values and 5 seeds. Then run two additional experiments on the same searched architectures: (1) grid-search tau in {0.05, 0.1, 0.13, 0.2, 0.3, 0.5} and report expansion count, average accuracy, forgetting, and FWT; and (2) compare against always-expand-every-task and never-expand control policies. If the chosen tau does not produce a distinct expansion profile (for example, it expands at every task), or if accuracy and forgetting are insensitive to tau, then the 'expand only when necessary' claim and the 'jointly searched expansion policy' claim are not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SEAL's central claim is that it jointly searches an architecture and an expansion policy that expands the model only when capacity is insufficient. The 'when' is decided by Eq. 10: expand if 1 - L(W_{t-1}, D_t)/L_{t-1} < tau. This trigger is not a searched or learned component: the NAS objectives in Eqs. 8-14 optimize the architecture x and the 3-bit expansion vector d, while tau is a user-set hyperparameter (0.2 for CIFAR-10, 0.13 for the other datasets). Thus the claimed single-phase joint search of the expansion policy is only partial; the most consequential part of the policy, the expansion decision, is exogenous and dataset-specific. Moreover, Eq. 10 is not a clean capacity estimator. The ratio compares the loss on new data under the previous weights to the previous task's training loss, which depends on optimization state, learning rate, the fixed 3-epoch budget, and task similarity, not just on capacity saturation. If L_{t-1} is very small (overfitted), the ratio can be large and trigger expansion at every task; if L_{t-1} is large, the trigger may never fire. The paper reports no expansion counts, no tau sensitivity analysis, and no ablation against always-expand or never-expand policies, so the empirical gains in Tables 1-2 cannot be attributed to selective expansion rather than to NAS architecture selection or to expanding at every task. The conclusion even notes that incorporating a capacity-aware control signal into the NAS process would enable 'more adaptive expansion decisions', an in-paper admission that the current trigger is not adaptive. This is an empirical-validation gap in the central claim, not a mathematical contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SEAL proposes a NAS framework for data-incremental learning. It uses an OFA supernet to define a constrained search space, and applies NSGA-II with two surrogate objectives to jointly select a base architecture and a 3-bit expansion vector. During incremental deployment, expansion is triggered by a capacity criterion (Eq. 10), and stability is maintained by cross-distillation training after each expansion. The method is evaluated on CIFAR-10, CIFAR-100, and ImageNet16-120 with five splits, comparing accuracy, forgetting, and forward transfer against EWC, SI, LwF, naive, and joint-training baselines.","tokens_in":12148,"tokens_out":6400,"duration_ms":58072,"significance":"If the central claims held, SEAL would be a useful contribution: it combines NAS with selective network expansion for incremental learning, introduces a flatness-aware objective, and the authors release their code, which aids reproducibility. The paper is clearly written and the objectives in Eqs. (12)–(14) are straightforward. However, the main advertised contribution—jointly searching an architecture and an expansion policy—is only partially realized, because the expansion trigger is a hand-tuned formula rather than a searched component. The empirical support is also mixed: SEAL performs well on CIFAR-100 and ImageNet16-120, but on CIFAR-10 its forgetting and forward transfer are worse than several baselines. These are fixable issues, so I view the paper as a candidate for major revision rather than rejection.","major_comments":[{"comment":"The expansion trigger in Eq. (10) is not learned or searched: τ is a hand-tuned, dataset-specific hyperparameter (τ=0.2 for CIFAR-10, τ=0.13 for CIFAR-100 and ImageNet16-120 in §5.2), while the NAS optimizes only the 3-bit expansion vector d (§4.4). The paper nevertheless claims in the abstract and §1 that the NAS 'jointly searches for both the architecture and the optimal expansion policy.' This is a partial claim: the most consequential decision—when to expand—is exogenous. The trigger is also not a clean capacity estimator: 1−L(W_{t−1},D_t)/L_{t−1} depends on the scale of L_{t−1}, the optimizer state, the fixed 3-epoch budget, and task similarity, so it can fire spuriously or remain silent regardless of true capacity saturation. No expansion counts, no τ-sensitivity analysis, and no ablation against always-expand or never-expand policies are provided, so the 'expand only when necessary' claim and the memory advantages in Tables 1–2 cannot be attributed to selective expansion. The Conclusion itself lists a 'capacity-aware control signal (e.g., a threshold-bit)' as future work, which corroborates this gap.","section":"§4.3, Eq. (10); §5.2"},{"comment":"The cross-dataset claim that SEAL 'effectively reduces forgetting' is contradicted by the CIFAR-10 row: SEAL's Forgetting is 1.76, worse than Naive (1.41), EWC (1.13), and SI (0.71), and its Forward Transfer is 0.18, far worse than Naive (0.68), EWC (1.33), SI (1.44), and LwF (1.22). Only on CIFAR-100 and ImageNet16-120 does SEAL achieve the best forgetting. The text acknowledges this in part, but the abstract and Section 5.2 still make unqualified statements about reducing forgetting. Please either restrict the claim to the datasets where it holds or provide a principled aggregation (e.g., average ranks with significance tests) that supports a cross-dataset conclusion.","section":"Table 2, CIFAR-10@5"},{"comment":"The flatness objective H requires two unspecified hyperparameters, the noise intensity σ and the number of perturbations N, and neither is reported in Section 5 nor subject to sensitivity analysis. Table 1 does not support the claim of 'consistent improvements' from flatness: relative to SingleObj, MultiObj-Flat has lower ACC on CIFAR-10 (94.53 vs 94.82) and on ImageNet-16 (50.40 vs 52.31), and on CIFAR-100 the best ACC is MultiObj-Acc, not MultiObj-Flat. Please specify σ and N, add a sensitivity study, and soften the claim to what the data show.","section":"§4.6, Eqs. (13)–(14); Table 1"},{"comment":"The accuracy objective uses a fixed size exponent w=0.07, but no justification or ablation is given for this value. Since the paper's memory-efficiency claim depends on the trade-off between accuracy and parameter count encoded by this exponent, its influence on the Pareto front and on the final selected models should be assessed.","section":"§4.6, Eq. (12)"}],"minor_comments":[{"comment":"There is a typo: '0.13 fo the other datasets' should be '0.13 for the other datasets.'","section":"§5.2"},{"comment":"The notation ACC_{DK}(W_K) is used without defining the subscript D_K; please clarify the indexing over datasets and tasks.","section":"Eq. (13)"},{"comment":"The baseline name is spelled both LwF and LWF; please use one consistent spelling throughout.","section":"§2 and Table 2"},{"comment":"The discussion of expanding early versus late layers would benefit from quantitative support, such as the exact forward-transfer values and a significance test, since the text calls the early-layer result 'intriguing' but provides no numbers.","section":"Fig. 5 and §5.2"},{"comment":"No pairwise significance tests or confidence intervals are reported, so several apparent differences may be within noise; adding such tests would strengthen the comparison.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope and the core NAS/IL combination is reasonable. I recommend major revision rather than rejection: the framework is sound in outline, but the headline 'jointly search expansion policy' needs to be reconciled with the hand-tuned trigger, and the CIFAR-10 numbers need to be addressed honestly. The authors are in a good position to add the missing sensitivity and ablation experiments; the released code should make these feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on SEAL (arXiv:2505.10457). The genuinely new piece is the constrained expansion operator: by restricting expansions to valid OFA subnetworks, the expanded models inherit pretrained weights and the search stays cheap. That is a real engineering contribution, and the single-phase search of architecture plus expansion vector is a fair framing. The code is up, and the evaluation against EWC, SI, LwF, naive, and joint training is reasonable. I'd rather see the NAS-based IL baselines (ENAS-S, CLEAS, Learn-to-Grow, ArchCraft) included, since those are the direct competitors they survey.\n\nThe soft spot, and it's a significant one, is the expansion trigger. Eq. 10 does not participate in the search. The threshold tau is set by hand, 0.2 for CIFAR-10 and 0.13 for the others. The paper reports no sensitivity analysis, no expansion counts, and no ablation against always-expand or never-expand. So the headline claim that SEAL expands 'only when necessary' is not actually supported by the experiments, and the sentence in the conclusion about adding a capacity-aware threshold-bit is essentially an admission that the current trigger is not learned or adaptive. The trigger is also a noisy proxy: the loss ratio depends on optimization state and task similarity, not just capacity saturation.\n\nAlso, the flatness metric's sigma and N are never specified, and the CIFAR-10 numbers show SEAL with higher forgetting and much lower forward transfer than EWC or SI. That doesn't torpedo the paper—the average across datasets favors SEAL—but it does temper the general claim.\n\nThe math is straightforward and there's no circularity in the derivation. The paper is honest about limitations. It's a solid, citable engineering contribution with a validation gap on the central mechanism. I'd send it to peer review and ask the authors to ablate the expansion policy, run a tau sweep, and report expansion counts. Without that, the 'only when necessary' claim remains a hand-tuned hyperparameter in disguise.\n\nI wouldn't cite it in my own work right now, but I'd bring it to a reading group focused on AutoML or continual learning.","headline":"A neat constrained-OFA expansion idea undermined by an exogenous, hand-tuned expansion trigger that the paper never validates.","tokens_in":12676,"tokens_out":2252,"would_cite":false,"duration_ms":19821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEAL searches for a starting network and a growth policy in a single NAS pass, expanding the model only when a capacity estimate says it must, and reports competitive accuracy with lower forgetting than EWC, SI, and LwF on three benchmarks.","keywords":["incremental learning","neural architecture search","dynamic network expansion","data-incremental learning","catastrophic forgetting","cross-distillation","once-for-all supernet","multi-objective optimization"],"falsifier":"Run SEAL on a data-incremental stream where a new split is nearly identical to the previous one but the learning rate is lowered so the loss drops slowly; if the trigger fires an expansion despite ample capacity, the capacity estimator is confounded by optimization dynamics. Conversely, a sweep of $\\tau$ on a single dataset would reveal whether the threshold, rather than the architecture search, is responsible for the reported accuracy and forgetting values.","tokens_in":11544,"feed_emoji":"🧠","tokens_out":3800,"duration_ms":33390,"temperature":0.7,"pith_summary":"SEAL claims that an incremental learner can be built by searching, in a single neural architecture search phase, both the initial DNN architecture and a policy for expanding it, so the model grows only when a capacity estimator says it cannot learn the new data. This matters because prior NAS-based continual learning either re-runs search at every task or expands unconditionally, which is wasteful in resource-constrained settings. SEAL targets data-incremental learning, where disjoint data samples arrive sequentially and are not stored for future access, and reports accuracy competitive with or better than EWC, SI, and LwF while lowering average forgetting and using fewer parameters. The central claim is that one search phase suffices to produce an architecture and expansion policy that generalize across future data splits.","feed_headline":"SEAL grows a model only when capacity runs out","feed_subtitle":"Single-phase architecture search beats EWC, SI, and LwF with fewer parameters on incremental benchmarks.","key_machinery":"The central object is the pair consisting of a candidate architecture $\\bar{x}$ and a 3-bit expansion vector $\\vec{d}$, together with the synergy between an OFA-constrained search space and the relative capacity trigger of Eq. 10. The OFA supernet guarantees that every expansion is a valid subnetwork, so candidate evaluation becomes cheap fine-tuning from pretrained weights; the trigger, $1 - L(W_{t-1}, D_t)/L_{t-1} < \\tau$, decides when the model grows; cross-distillation loss of Eq. 11 stabilizes the model after expansion; and the flatness metric of Eq. 14 guides the multi-objective NSGA-II search. This machinery carries the claim that the model expands only when necessary, in a single search phase.","core_discovery":"On its own terms, SEAL is the first NAS framework to jointly optimize the starting architecture and the expansion policy, casting the problem as a bi-objective optimization of incremental accuracy (traded against parameter count) and a flatness-based robustness metric. The model expands only when a relative loss comparison indicates capacity saturation, the expansion operator is constrained to produce valid OFA subnetworks so newly added components initialize from pretrained weights, and cross-distillation training follows each expansion. Across CIFAR-10, CIFAR-100, and ImageNet16-120 with five data splits, SEAL reports accuracies of 95.35, 83.94, and 52.45 with average forgetting of 1.76, 4.81, and 3.23, outperforming the EWC, SI, and LwF baselines on most of these metrics while using smaller models.","pith_inferences":["The relative loss trigger may conflate capacity saturation with optimization plateaus; a control experiment that varies learning rate or task similarity could separate these causes and reveal when the expansion policy is confounded.","The threshold $\\tau$ is hand-tuned per dataset and no sensitivity analysis is reported, so learning $\\tau$ inside the NAS objective, as the paper itself hints with a threshold-bit, is a natural testable extension.","The observed correlation between flatter minima and smaller models suggests that flatness could serve as a proxy for efficient capacity allocation in continual learning beyond this specific benchmark suite.","Because expansions are constrained to MobileNetV3-style OFA blocks, the 'single search phase' benefit may transfer to other weight-sharing supernets if the same joint architecture-and-policy search is applied."],"forward_implications":["SEAL eliminates the cost of re-running NAS at every task, because one search phase yields both the architecture and a fixed expansion policy.","In data-incremental settings the model grows only when the capacity trigger fires, bounding the final parameter count relative to approaches that expand at every task.","Multi-objective search that includes a flatness objective yields smaller models and lower forgetting than single-objective accuracy-only search.","Cross-distillation after expansion reduces forgetting and improves knowledge transfer compared to training expanded models without distillation.","Expanding the last layers rather than early layers better preserves fine-grained features and gives higher accuracy in this data-incremental setting."],"supporting_citations":[{"why":"Supplies the OFA supernet whose pretrained weights and constrained subnetworks make fast fine-tuning and valid expansions possible.","marker":"(Cai et al. 2020)"},{"why":"EWC is a regularization baseline SEAL compares against for accuracy, forgetting, and forward transfer.","marker":"(Kirkpatrick et al. 2016)"},{"why":"LwF is both a distillation baseline and the basis for the cross-distillation loss used after each expansion.","marker":"(Li and Hoiem 2017)"},{"why":"SI is the second regularization baseline used in the comparison tables.","marker":"(Zenke, Poole, and Ganguli 2017)"},{"why":"Defines the data-incremental learning scenario and motivates the choice of baselines and metrics.","marker":"(van de Ven, Tuytelaars, and Tolias 2022)"},{"why":"Provides the average accuracy and average forgetting metrics that structure all reported results.","marker":"(Chaudhry et al. 2018)"},{"why":"Contributes the rescaling-invariant flatness metric reused as the second NAS objective.","marker":"(Gambella, Pittorino, and Roveri 2024)"},{"why":"Supplies the NSGA-II evolutionary search and adaptive-switching surrogate models used to solve the bi-objective problem.","marker":"(Lu et al. 2020)"},{"why":"DEN is the prior expandable-network method whose loss-threshold expansion idea SEAL refines into a relative capacity trigger.","marker":"(Yoone et al. 2017)"}],"fun_headline_variants":["SEAL expands models only when capacity runs out","NAS framework grows networks only when needed for incremental learning","SEAL: jointly optimize architecture and expansion policy for incremental tasks","Selective expansion beats EWC, SI, LwF with fewer parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole 'expand only when necessary' behavior rests on the claim that the ratio $1 - L(W_{t-1}, D_t)/L_{t-1}$ falling below a threshold $\\tau$ really means the model's capacity is exhausted; if the loss stagnates because of optimization noise, learning-rate effects, or task similarity instead, SEAL expands too early or too late, and the threshold must be hand-tuned per dataset (0.2 for CIFAR-10, 0.13 for the others).","fun_headline_variants_meta":{"raw":{"variants":["SEAL expands models only when capacity runs out","NAS framework grows networks only when needed for incremental learning","SEAL: jointly optimize architecture and expansion policy for incremental tasks","Selective expansion beats EWC, SI, LwF with fewer parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1514,"prompt_tokens":920,"completion_tokens":594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":536,"tokens_out":594,"duration_ms":5948,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:08:36.412070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SEAL on a data-incremental stream where a new split is nearly identical to the previous one but the learning rate is lowered so the loss drops slowly; if the trigger fires an expansion despite ample capacity, the capacity estimator is confounded by optimization dynamics. Conversely, a sweep of $\\tau$ on a single dataset would reveal whether the threshold, rather than the architecture search, is responsible for the reported accuracy and forgetting values.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EWC is a regularization baseline SEAL compares against for accuracy, forgetting, and forward transfer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the data-incremental learning scenario and motivates the choice of baselines and metrics."},{"cited_title":"K.; Ajanthan, T.; and Torr, P","cited_arxiv_id":null,"evidence_quote":"Provides the average accuracy and average forgetting metrics that structure all reported results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the rescaling-invariant flatness metric reused as the second NAS objective."}],"review_version":1}