{"id":"486c17c0-7abb-408e-959e-a2b6b4ab308b","arxiv_id":"2504.12503","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On nine new engineering regression continual-learning benchmarks, Experience Replay consistently beats EWC and GEM, approaching joint-retraining accuracy at roughly half the compute in several cases.","lead":"This paper benchmarks three continual learning methods (Experience Replay, EWC, and GEM) on nine new engineering regression benchmarks built from five 3D design datasets. It finds that replay-based learning comes closest to full retraining accuracy while cutting training time roughly in half for several tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never describes how test sets were split relative to experience bins; if the replay buffer overlaps the test set, Replay's near-Joint accuracy could be an evaluation artifact.","rationale":"The reader's verdict is already CONDITIONAL, and this stress-test supports that conditionality without escalating it. The most load-bearing unresolved issue is not the representativeness of the proposed regression CL scenarios, which the authors explicitly acknowledge in Section 7, but the undisclosed test-set construction: the central Replay claim is an empirical comparison, and its validity requires that evaluation samples not be contaminated by the replay buffer or training data. The reader did flag this as a secondary unstated premise, so there is partial agreement. The GEM runtime contradiction in Table 13 is a separate internal inconsistency that reinforces the need for the promised code and datasets before the numerical conclusions can be trusted. However, none of this proves the Replay result is wrong; it only means the paper is not yet verifiable as written. Hence the appropriate verdict remains CONDITIONAL, requiring release of code, datasets, and explicit split/replay-construction documentation, rather than ACCEPT or REJECT.","tokens_in":24497,"tokens_out":5623,"duration_ms":63214,"concrete_test":"Request the exact train/test split construction and rerun SplitDRIVAERNET-PC with a strict disjoint partition: (1) split each target bin into 80/20 train/test before assigning experiences, (2) build the replay buffer only from training samples, (3) verify no test sample appears in any training set or replay buffer, then recompute Replay MPE, MAE, and forgetting ratio versus the Joint baseline. If Replay's performance shifts materially or its rank against Joint changes, the headline claim is not robust to test leakage; if the numbers are unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Replay ≈ Joint at roughly half the training time) depends on the evaluation protocol, yet the manuscript nowhere specifies how test sets were constructed. Section 5 says models were tuned on 'a separate validation set outside the continual learning setting,' but it does not state whether each experience's test samples are disjoint from the training and replay data, how the global train/test partition was made, or whether duplicate or near-duplicate geometries across experiences were removed. In rehearsal-based CL, this matters directly: if the replay buffer is sampled from the same pool that is later scored, ER can trivially retain test-like samples, and the reported MAE and forgetting advantages over EWC and GEM become inflated. A second reliability signal is that Section 5 asserts all methods have shorter total runtime than Joint, but Table 13 lists GEM at 8168.27 s on SplitDRIVAERNET++-PC (Bin) versus Joint at 7662.63 s, so the stated evaluation constraint is not consistently met. Because the code and datasets are only promised, not released, these evaluation details cannot currently be checked by readers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces continual learning (CL) to 3D engineering regression problems by proposing three regression-specific CL scenarios (bin-incremental, input-incremental, and multi-target incremental) and constructing nine benchmarks from five engineering datasets (SHIPD, DrivAerNet, DrivAerNet++, ShapeNet Car, RAADL). It benchmarks three standard CL strategies from the Avalanche library—Experience Replay (ER), Elastic Weight Consolidation (EWC), and Gradient Episodic Memory (GEM)—against naive incremental training and a Joint retraining baseline, reporting mean percent error, mean absolute error, forgetting ratios, and total runtime. The central empirical claim is that Replay achieves performance comparable to Joint retraining on several benchmarks while reducing training time by roughly half, whereas EWC and GEM are less effective.","tokens_in":24659,"tokens_out":4112,"duration_ms":42176,"significance":"If the central claim holds, the paper provides practically valuable evidence that a simple replay buffer of 20% of past data can nearly match joint retraining for evolving engineering surrogate models at substantially reduced compute. The proposed benchmarks and scenarios fill a clear gap, since most CL research targets classification rather than regression, and engineering domains have been largely overlooked. The use of standard Avalanche implementations and an independent Joint baseline are strengths that increase the credibility of the comparison. However, the evaluation protocol—especially the construction of test sets relative to experience bins—is not described, and several hyperparameters are unreported. These gaps currently prevent the reader from verifying whether the headline result is an artifact of leakage or unfair configuration, so the contribution is not yet fully substantiated.","major_comments":[{"comment":"The manuscript never specifies how the test sets were constructed relative to the experience bins. It is not stated whether each experience's test samples are disjoint from the training and replay data, how the global train/test partition was made, or whether duplicate or near-duplicate geometries across experiences were removed. This is load-bearing for the central claim: if the replay buffer is sampled from the same pool that is later scored, Replay's reported MAE and forgetting advantages over EWC and GEM could be inflated. Please describe the exact train/test split procedure, confirm that replay and test sets do not overlap, and provide the number of samples in each split.","section":"Section 5 (Implementation and Evaluation Metrics), Tables 3–12"},{"comment":"Section 5 states that all methods were constrained to have a shorter total runtime than the Joint retraining baseline, but Table 13 lists GEM at 8168.27 s on SplitDRIVAERNET++-PC (Bin) versus Joint at 7662.63 s, violating the constraint. Additionally, Section 7 concedes that GEM's patterns-per-experience had to be reduced to meet this cap, which 'likely contributed to GEM's diminished overall performance.' Thus the runtime constraint is not consistently enforced, and the comparison of GEM against Replay and Joint is confounded by an unequal compute budget. Please either re-run GEM without the artificial pattern reduction, or explicitly relax the constraint and discuss the implications for the ranking.","section":"Table 13 and Section 5 (Implementation and Evaluation Metrics)"},{"comment":"Hyperparameters for the continual learning strategies are not reported: the EWC regularization strength lambda, GEM's patterns per experience (PPE), the replay buffer size, learning rate, number of epochs, and batch size are all absent. Section 5 states that the surrogate models were tuned 'using a hit-and-trial approach,' but no values or search ranges are given. Since the paper's headline conclusion is a ranking of strategies, the validity of that ranking depends on all methods being reasonably configured; unreported or arbitrarily chosen hyperparameters can change the ordering (e.g., a poorly chosen EWC lambda could explain EWC's near-Naive performance). Please report the exact hyperparameter values, the tuning procedure, and ideally perform a sensitivity analysis for the key parameters.","section":"Section 5 (Surrogate Models; Continual Learning Strategies)"},{"comment":"The Mann-Whitney U test analysis and critical-difference plots are described only briefly. It is unclear what the unit of analysis is (nine benchmarks, or multiple seeds per benchmark), how the metrics were normalized before ranking, and whether any multiple-comparison correction was applied. The claim that Replay and GEM are 'statistically indistinguishable' on forgetting, while Replay is significantly better on final MAE, needs these details to be assessable. Please specify the exact statistical procedure, including the number of independent measurements per strategy, the normalization formula, and the significance level used.","section":"Section 7 (Statistical Trends Across Benchmarks)"}],"minor_comments":[{"comment":"The definition of 'best forgetting ratio' as 'the lowest FRk achieved over all trials' is unclear; please specify the number of trials/seeds and justify why the best rather than the mean (or median) is reported. Reporting the mean with variance would be more standard.","section":"Section 5 (Implementation and Evaluation Metrics)"},{"comment":"There is a typo: 'illustrted' should be 'illustrated' in the sentence describing the benchmarks in Figure 4.","section":"Section 4 (Benchmarks)"},{"comment":"Some references are incomplete: [16] and [17] (Rolnick et al.; Lopez-Paz and Ranzato) lack year and venue information, and [42] is listed as 'n.d.' with only a URL. Please provide full bibliographic details for all entries.","section":"References"},{"comment":"The subplots in Figure 6 are small and the axis labels appear difficult to read at publication size. Consider enlarging the figure or splitting it into separate panels.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The test-set construction issue is the highest-risk point for the paper's central claim. If the test split is not disjoint from the replay buffer, the main result could be an artifact. I would recommend the editor prioritize verification of this point during revision. The authors should also be asked to release the code and data at least as a supplementary artifact for review, since the manuscript promises but does not provide them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful benchmarking study. It applies established CL methods to engineering regression surrogates, proposes three regression-specific scenarios, and builds nine new benchmarks from five real engineering datasets. The central finding—a simple 20% replay buffer nearly matches joint retraining at roughly half the compute—is plausible and consistent with the broader CL literature. The paper is not a methodological breakthrough, but it fills a real gap. It deserves peer review, but needs a reporting pass before the numbers are taken at face value.\n\nWhat's actually new: the bin/input/multi-target scenario definitions are a sensible adaptation of classification CL to continuous targets, and the benchmark suite is the first systematic comparison of ER/EWC/GEM on these engineering datasets. Replay's consistency across point-cloud and parametric representations, and its runtime advantage, are practically interesting for surrogate-model workflows. The Mann-Whitney/CD-plot analysis is a reasonable way to rank strategies across heterogeneous scales, and the paper's treatment of its own limitations is honest.\n\nSoft spots, in order:\n\n1. Evaluation protocol under-specified. The paper never says how each experience's test set was constructed relative to the training and replay pools. In rehearsal-based CL this matters: if replay samples overlap the scored test set, Replay's near-Joint results could be inflated. I see no direct evidence of leakage—Avalanche's standard split would keep them disjoint—but the manuscript doesn't state it, and with code and data only promised, readers can't check. This is the most important fix.\n\n2. GEM was deliberately compute-constrained. Section 7 admits that reducing patterns per experience to stay under the Joint runtime likely hurt GEM. That's honest, but it means the 'Replay beats GEM' conclusion is not a comparison of the methods as normally configured.\n\n3. Hyperparameters are largely unreported: EWC lambda, GEM PPE, learning rates, epochs, batch sizes, and number of bins are all missing; tuning is described only as hit-and-trial. For a benchmarking paper, that's a serious reproducibility gap.\n\n4. Table 13 contradicts the stated constraint: GEM on SplitDRIVAERNET++-PC (Bin) takes 8168s vs Joint 7662s. Minor by itself, but it undermines the 'all methods beat Joint runtime' claim and suggests runtime accounting needs checking.\n\nCitation pattern looks fine: proper credit to the CL classification literature and the engineering surrogate datasets. No red flags.\n\nBottom line: useful empirical contribution with a plausible central claim, but conditional on clarifying the evaluation protocol, reporting hyperparameters, and re-running GEM fairly. I'd send it to review, and ask for the code/data release and a clear description of the train/test/replay split before accepting the headline result.","headline":"Solid first benchmark of continual learning on engineering regression surrogates; Replay's near-Joint accuracy is plausible and useful, but missing evaluation details and a handicapped GEM make the headline result conditional.","tokens_in":25267,"tokens_out":2930,"would_cite":true,"duration_ms":29435,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmarking study claims that a simple replay buffer lets engineering surrogate models learn from new data nearly as well as retraining from scratch, at about half the compute.","keywords":["continual learning","catastrophic forgetting","engineering regression","surrogate modeling","experience replay","benchmarking","point clouds","drag coefficient prediction"],"falsifier":"Run Experience Replay and EWC on a new engineering regression stream where target bins overlap and tasks are defined by operating conditions rather than value ranges, evaluating on a held-out test set excluded from every replay buffer; if Replay's final error does not come close to Joint retraining, or if EWC matches Replay, the paper's central advantage fails.","tokens_in":24214,"feed_emoji":"⚙️","tokens_out":6686,"duration_ms":66403,"temperature":0.7,"pith_summary":"Engineers often train surrogate models—neural networks that predict continuous quantities such as drag coefficients from 3D geometry—on static datasets, but real design pipelines add new data over time, making retraining from scratch expensive. This paper transfers continual learning, the study of training models on sequential data without catastrophic forgetting, to engineering regression by constructing nine benchmarks from five engineering datasets. It proposes three regression-specific task sequences: bin-incremental (target values partitioned into ranges), input-incremental (input categories or operating conditions), and multi-target-incremental (new output quantities). Benchmarking Experience Replay, Elastic Weight Consolidation, and Gradient Episodic Memory, the paper claims that Replay, using only 20% of past data as a memory buffer, reaches performance comparable to fully retraining on several benchmarks while reducing training time by nearly half. A sympathetic reader would take the central message to be that simple rehearsal, not parameter regularization, is the practical route to continual surrogate models.","feed_headline":"Replay buffer nearly matches full retraining on engineering models","feed_subtitle":"Benchmarking nine evolving-design tasks, rehearsal beats regularization and cuts training time by nearly half.","key_machinery":"The central device is the pairing of regression-specific task sequences with a rehearsal buffer. In the bin-incremental scenario the continuous target is cut into ordered value ranges and each range becomes one experience; in the input-incremental scenario experiences are defined by input categories such as car type; a third multi-target scenario, noted but not benchmarked, would add new output quantities. Experience Replay carries the argument by storing a random 20% of past samples in a memory buffer and mixing them into training on each new experience, which is what lets the model retain old mappings. EWC's Fisher-information penalty and GEM's gradient-projection constraint serve as the comparators that Replay beats. The forgetting ratio, a normalized version of the maximum regression-error increase on earlier experiences, is the metric that makes the cross-benchmark ranking possible.","core_discovery":"On its nine benchmarks, the paper claims a clear ranking: Experience Replay—randomly replaying 20% of prior samples—is the best continual learning strategy, with final mean percent error close to the Joint retraining baseline (for instance 2.07% vs. 1.57% on DrivAerNet point clouds and 3.64% vs. 3.36% on DrivAerNet++ point clouds). GEM, which constrains gradients using stored past patterns, ranks between Replay and the regularization method EWC, and on forgetting Replay and GEM are statistically indistinguishable and sit just below Joint. EWC performs no better than naive sequential training on most benchmarks. The input-incremental scenario (car category as task) produces less forgetting overall than the bin-incremental scenario (drag-value range as task), and on the small glider dataset all methods show negative forgetting, meaning learning later experiences improved predictions on earlier ones.","pith_inferences":["The paper's compute comparison counts total runtime including evaluation; the memory cost of a 20% replay buffer is not tabulated. If buffer memory is counted, replay's advantage over memory-free EWC could narrow in settings where storage is the bottleneck.","A testable extension follows from the input-incremental results: because the second car category caused an error spike followed by recovery, replay methods may act as a cheap concept-drift response whenever shifting operating conditions periodically revisit previously seen regimes—something the paper does not claim.","If replay at 20% buffer generalizes beyond these nine benchmarks, engineering MLOps pipelines could replace full retraining with rolling buffers, freeing compute for simulation; that practical workflow implication is implicit, not proven.","The proposed multi-target-incremental scenario (adding new output quantities over time) is not benchmarked, so the paper's ranking of strategies does not extend to that regime; testing it is a direct next step."],"forward_implications":["Engineers can update surrogate models on streaming design data with a 20% replay buffer and expect final prediction error close to what full retraining would give, at roughly half the training time.","Rehearsal-based strategies (Replay and GEM) are statistically indistinguishable on forgetting and rank just below the ideal joint-retraining baseline, so the choice between them can be made on compute grounds.","Elastic Weight Consolidation does not beat naive sequential training on most of these regression benchmarks, suggesting regularization penalties are not enough when targets are continuous.","In the input-incremental car-category scenario, forgetting is less severe than in the bin-incremental scenario, and models recover after the second category, implying shared geometry between tasks eases retention.","On small datasets with narrow target ranges, learning later experiences can improve predictions on earlier ones (negative forgetting), so catastrophic forgetting is not automatic in engineering regression."],"supporting_citations":[{"why":"Defines Experience Replay, the random past-sample buffer strategy that the paper finds best.","marker":"[16]"},{"why":"Defines Elastic Weight Consolidation and the Fisher-information penalty used as the regularization baseline.","marker":"[15]"},{"why":"Defines Gradient Episodic Memory, the gradient-constraint rehearsal baseline.","marker":"[17]"},{"why":"Supplies the three incremental-learning scenario types that the paper adapts to regression.","marker":"[14]"},{"why":"Provides the continual-learning survey and scenario categorization underlying the benchmark design.","marker":"[19]"},{"why":"Notes the lack of prior work on catastrophic forgetting in regression, motivating the gap the paper fills.","marker":"[13]"},{"why":"Supplies the DrivAerNet car drag dataset used for the bin-incremental benchmark.","marker":"[2]"},{"why":"Supplies the DrivAerNet++ car dataset whose car categories define the input-incremental benchmarks.","marker":"[39]"},{"why":"Supplies the SHIPD ship hull dataset with both parametric and point-cloud representations.","marker":"[37]"},{"why":"Supplies the small RAADL glider dataset where backward transfer is observed and marks the small-data regime.","marker":"[40]"}],"fun_headline_variants":["Replay nearly matches joint training on engineering benchmarks","Replay cuts training time nearly in half while matching retraining","Simple replay beats sophisticated continual learning on engineering","Replay is best continual learning strategy for engineering regression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions assume the benchmark scenarios—splitting data into disjoint target-value bins or by car category—mirror how engineering data actually arrives over time; if real streams have overlapping target ranges or different task boundaries, the measured ranking of strategies may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Replay nearly matches joint training on engineering benchmarks","Replay cuts training time nearly in half while matching retraining","Simple replay beats sophisticated continual learning on engineering","Replay is best continual learning strategy for engineering regression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001475,"raw_usage":{"total_tokens":5912,"prompt_tokens":911,"completion_tokens":5001,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":4939}},"tokens_in":527,"tokens_out":5001,"duration_ms":32008,"temperature":1.0,"reasoning_tokens":4939,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:30:04.354768+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Experience Replay and EWC on a new engineering regression stream where target bins overlap and tasks are defined by operating conditions rather than value ranges, evaluating on a held-out test set excluded from every replay buffer; if Replay's final error does not come close to Joint retraining, or if EWC matches Replay, the paper's central advantage fails.","supporting_citations":[{"cited_title":"P., and Wayne, G., 2019, Experience replay for continual learning","cited_arxiv_id":null,"evidence_quote":"Defines Experience Replay, the random past-sample buffer strategy that the paper finds best."},{"cited_title":"Overcoming catastrophic forgetting in neural net- works,","cited_arxiv_id":null,"evidence_quote":"Defines Elastic Weight Consolidation and the Fisher-information penalty used as the regularization baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Gradient Episodic Memory, the gradient-constraint rehearsal baseline."},{"cited_title":"Three types of incremental learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the three incremental-learning scenario types that the paper adapts to regression."},{"cited_title":"A continual learning survey: Defying forgetting in classification tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the continual-learning survey and scenario categorization underlying the benchmark design."},{"cited_title":"Clear: An adaptive contin- ual learning framework for regression tasks,","cited_arxiv_id":null,"evidence_quote":"Notes the lack of prior work on catastrophic forgetting in regression, motivating the gap the paper fills."},{"cited_title":"Drivaer- net: A parametric car dataset for data-driven aerody- namic design and prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the DrivAerNet car drag dataset used for the bin-incremental benchmark."},{"cited_title":"Drivaernet++: A large-scale multimodal car dataset with computational fluid dynamics simulations and deep learning benchmarks,","cited_arxiv_id":null,"evidence_quote":"Supplies the DrivAerNet++ car dataset whose car categories define the input-incremental benchmarks."},{"cited_title":"Ship-d: Ship hull dataset for design optimization using machine learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the SHIPD ship hull dataset with both parametric and point-cloud representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the small RAADL glider dataset where backward transfer is observed and marks the small-data regime."}],"review_version":1}