{"id":"2e6f7952-233f-4613-9978-5d73d6c83007","arxiv_id":"2506.15223","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid replay-plus-EWC fine-tuning strategy lets a universal machine-learning interatomic potential learn a new solid electrolyte without forgetting its pretrained knowledge, and the paper demonstrates this on several Li-ion conductors.","lead":"The paper introduces reEWC, a fine-tuning recipe for pretrained machine-learning interatomic potentials that combines experience replay with elastic weight consolidation, and tests it on a lithium solid-state electrolyte target. It reports that reEWC prevents catastrophic forgetting, preserves the base model's general accuracy, and transfers well to other solid electrolytes, which matters for high-throughput materials discovery.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Replay set and the sMPtrj forgetting test set may overlap; if they do, reEWC's headline forgetting-prevention and synergy results are partly memorization, not general memory retention.","rationale":"The paper's central empirical claim is that reEWC balances plasticity and stability, preserves generalizability, and gives synergistic benefits over Replay and EWC alone. The main evidence for stability/generalizability on the original pretraining domain is the sMPtrj loss in Fig. 2a and Table 1. Both sMPtrj and the Replay set are defined as random 10% samples of MPtrj with no explicit statement that they are disjoint, so the forgetting metric is vulnerable to training/test overlap for the very methods that train on the Replay set. This is not a stylistic issue: it changes the interpretation of Fig. 2b, where reEWC's lower sMPtrj loss despite larger parameter shifts is used as evidence that reEWC's parameter updates are more effective than EWC's. If a non-trivial fraction of sMPtrj was seen during replay, that comparison is confounded and the 'synergistic benefit' claim is weakened. The other benchmarks (argyrodite and non-argyrodite AIMD sets, diffusivity, quasi-melting) are independent of MPtrj and do support knowledge transfer, so the paper would not be rejected outright; but the forgetting-prevention and 'preserves generalizability' headline needs a clean held-out MPtrj split. The reader's weakest assumption concerns FIM calibration on a hand-filtered subset; that is a real secondary issue, but the potential Replay/sMPtrj overlap more directly undercuts the main forgetting metric, which is why I disagree with the reader's identification of the single most load-bearing assumption.","tokens_in":22400,"tokens_out":6213,"duration_ms":65066,"concrete_test":"Using the released Zenodo data and GitHub code, list configuration identifiers (or hashes of structures with their energy/force/stress labels) for the sMPtrj test set and for the Replay set, and compute their intersection. If the intersection is non-empty, re-evaluate the already-trained Replay and reEWC models on the sMPtrj subset that is disjoint from the Replay set, and recompute the Fig. 2a and Table 1 forgetting losses on that disjoint subset. If reEWC's loss is still close to SevenNet-0 and below EWC on the disjoint subset, the concern is resolved; if the advantage shrinks or vanishes, the forgetting-prevention and synergy claims need revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that reEWC preserves generalizability on the pretraining domain rests heavily on the sMPtrj loss in Fig. 2a and Table 1. In Sec. 2.2, the Replay set is described as a random 10% sample of the MPtrj dataset, and in Sec. 2.3, the sMPtrj test set is also described as a random 10% sample of the MPtrj dataset. No disjointness is stated. If the two samples are independent, about 1% of MPtrj, i.e. roughly 10% of sMPtrj, would appear in the Replay training set. Replay and reEWC train directly on those configurations, so their lower sMPtrj loss, including reEWC's lower loss than EWC despite larger parameter shifts in Fig. 2b, could reflect memorization rather than genuine forgetting prevention. That would inflate the 'preserves generalizability' claim and the claimed synergistic advantage over EWC. The FIM subset in Methods 4.2 may also overlap sMPtrj, though EWC does not train on it and this is less directly biasing. The paper must either document a disjoint split or re-evaluate forgetting on a clean held-out MPtrj subset.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes reEWC, a hybrid continual-learning fine-tuning strategy that combines Experience Replay and Elastic Weight Consolidation (EWC) for pretrained universal machine-learning interatomic potentials (MLIPs). Using SevenNet-0, pretrained on the MPtrj dataset, as the base model and Li6PS5Cl (LPSC) as the fine-tuning target, the authors compare Vanilla fine-tuning, EWC, Replay, and reEWC. They report that reEWC simultaneously achieves low target-domain losses, reduced forgetting on the pretraining domain (as measured by loss on a random 10% MPtrj subset they call sMPtrj), improved energy/force accuracy and softening scales on 126 argyrodite and 9 non-argyrodite solid-electrolyte test sets, Li diffusivities close to DFT references, and quasi-melting ratios consistent with AIMD. The central claim is that reEWC provides a synergistic balance of stability and plasticity, preserving generalizability while learning the target system, at training cost comparable to Replay and with greater robustness to the composition of the Replay set.","tokens_in":22633,"tokens_out":3676,"duration_ms":40263,"significance":"If the central claim holds, the paper offers a practical, computationally efficient recipe for fine-tuning universal MLIPs without catastrophic forgetting, which is directly relevant to materials discovery workflows. The study is unusually thorough in its validation: energy/force MAEs, softening scales, diffusivities, and quasi-melting ratios are benchmarked against independent DFT/AIMD references that were not used in training, and the source data and code are publicly released. The separation of learning, forgetting, and generalizability metrics, the explicit examination of parameter shifts (MAD), the FIM-aware noise-perturbation experiment, and the MD-stability case studies (Li3N, high-entropy argyrodites) are notable strengths. However, the central forgetting metric, sMPtrj, may be contaminated by overlap with the Replay training set, and the FIM-based regularizer relies on hand-tuned-though-robust hyperparameters and on a hand-filtered Fisher subset; these issues must be resolved before the synergistic-advantage claim can be fully credited.","major_comments":[{"comment":"The FIM is computed on a hand-filtered subset of the pretrained dataset (1,154,095 configurations with Huber loss below 0.000726), and the regularization strengths are set to different values for EWC (λ=10^6) and reEWC (λ=10^5) after manual exploration. While the paper reports in Fig. S9 that the conclusions are insensitive to reasonable variations in the λ values, no equivalent sensitivity analysis is provided for the FIM-subset loss threshold. Because the EWC and reEWC regularizers rely entirely on the FIM to identify parameters that must stay fixed, an ill-calibrated subset could cause either under-regularization (forgetting) or over-regularization (limited target learning) in a way that is not captured by the current experiments. Please add an ablation over the threshold (or at least report the fraction of parameters that change materially as the threshold is varied) to substantiate that the filter is not load-bearing for the reported comparisons.","section":"§4.2, Fig. S9"}],"minor_comments":[{"comment":"Table 1 reports training cost as ×1 for Vanilla and EWC and ×2 for Replay and reEWC, but the main text (§3) only states that reEWC cost is 'comparable' to Replay; clarifying whether the ×2 factor refers to wall-clock time on the same GPU and whether it includes the FIM computation would improve the efficiency claim.","section":"§2.2, §4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-executed and the extensive external DFT validation is a real asset. The primary blocker is the potential Replay/sMPtrj overlap, which directly affects the headline forgetting-prevention and synergy claims; this is fixable by documenting a disjoint split or adding a clean held-out evaluation. The FIM-subset sensitivity is a secondary but important robustness check. If these are addressed, the paper would likely be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, useful empirical paper on fine-tuning pretrained universal MLIPs, but the central forgetting-prevention claim has a load-bearing hole that needs to be fixed before I would trust the synergy headline.\n\nThe method itself is not conceptually new—replay plus EWC is a standard continual-learning combination—but the paper's contribution is real: the first systematic demonstration on a pretrained universal potential, with MLIP-specific failure-mode analysis. The spurious Li–Li bonds under Replay, the atomic-energy shifts under EWC, and the quasi-melting checks are genuinely informative. The benchmark suite is extensive: energy and force MAEs, softening scales, Li diffusivities, and structural stability, all against external DFT references that were not used in training. They also shipped data and code. That is reproducible evidence and deserves credit.\n\nThe main problem is the sMPtrj forgetting test. In Section 2.2 the Replay set is a random 10% sample of MPtrj; in Section 2.3 the test set is also a random 10% sample of MPtrj. No disjointness is stated. If they are independent draws, roughly 10% of sMPtrj appears in the Replay training set, and Replay/reEWC train directly on those configurations. That means the lower sMPtrj losses—including reEWC beating EWC despite larger parameter shifts—could be partly memorization rather than genuine retention. This does not refute the whole paper: the transfer benchmarks on argyrodites and non-argyrodites use fresh DFT data and are independent of the Replay set. But the specific claim that reEWC 'preserves generalizability' on the pretraining domain, and the claimed synergistic advantage over EWC, rest heavily on this contaminated metric. The authors must either document a disjoint split or re-evaluate forgetting on a clean held-out MPtrj subset. This is a serious flaw, not a cosmetic one.\n\nOther soft spots are more minor. The lambda values differ between EWC and reEWC (1e6 vs 1e5), and while the authors say results are insensitive, a systematic sweep on both methods with matched lambdas would be more convincing. The FIM is computed on a hand-filtered high-accuracy subset; the noise-perturbation experiment supports the FIM's usefulness, but the subset selection itself is not justified beyond practicality. The abstract's 'knowledge transfer to chemically distinct systems' is stronger than the nine-material non-argyrodite evidence supports: only LGPS and LPS show clear gains, and the other seven roughly hold even in the best case. The atomic-energy analysis in Fig. 7 uses an MLIP reference rather than DFT; the authors acknowledge this, so it is a caveat rather than a hidden flaw. The GitHub link lacks a pinned commit, which is a minor reproducibility gap.\n\nWho is this for? Anyone fine-tuning universal MLIPs for solid-state electrolyte screening or similar target-specific specialization. It is worth a serious referee and a reading-group discussion, but the revision must fix the split or redo the forgetting evaluation, and the knowledge-transfer language should be dialed back. Send it out, conditionally.","headline":"A genuinely useful empirical study of replay-plus-EWC fine-tuning for universal MLIPs, but its headline forgetting metric may be partly memorization because the Replay set and the sMPtrj test set are both random 10% samples of MPtrj with no disjointness stated.","tokens_in":23214,"tokens_out":1985,"would_cite":true,"duration_ms":23065,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining Experience Replay with Elastic Weight Consolidation lets a pretrained universal machine-learning interatomic potential be fine-tuned to a new material while preserving the general knowledge it was pretrained on.","keywords":["machine-learning interatomic potentials","catastrophic forgetting","fine-tuning","elastic weight consolidation","experience replay","solid-state electrolytes","potential energy surface softening","continual learning"],"falsifier":"Recompute the Fisher matrix on the full MPtrj dataset instead of the 1,154,095 high-accuracy subset, retrain reEWC with that matrix, and compare held-out pretraining (sMPtrj) loss and LPSC accuracy; if the results are nearly identical, the subset choice is not load-bearing, and if they differ materially, the core mechanism depends on that filtering.","tokens_in":22178,"feed_emoji":"🔋","tokens_out":6508,"duration_ms":61591,"temperature":0.7,"pith_summary":"This paper tries to show that a universal machine-learning interatomic potential (an MLIP, a neural-network model that predicts atomic energies and forces) can be fine-tuned to a new material without losing the broad chemical knowledge it was pretrained on. The proposed method, reEWC, combines two established forgetting-prevention techniques: Experience Replay, which re-trains on a sample of the original data, and Elastic Weight Consolidation (EWC), which penalizes changes to parameters the pretrained model depends on. The authors test it by fine-tuning the pretrained SevenNet-0 model on the solid electrolyte Li$_6$PS$_5$Cl, and report that reEWC corrects the known 'softening' of the potential-energy surface, gives Li diffusivities close to DFT reference values, and improves accuracy on other sulfide, oxide, nitride, and halide electrolytes. The deeper claim is that Replay and EWC are not redundant: each covers a weakness of the other, and their combination is more robust to replay-set composition than Replay alone. A reader should care because a practical anti-forgetting fine-tuning recipe would let one universal model be continually adapted to new materials while remaining trustworthy for high-throughput screening.","feed_headline":"Replay plus weight consolidation stops MLIP fine-tuning forgetting","feed_subtitle":"Fine-tuning one solid electrolyte boosts accuracy on many and keeps the pretrained model's general knowledge intact.","key_machinery":"The load-bearing object is the diagonal Fisher information matrix, computed on a filtered subset of 1,154,095 MPtrj configurations with low Huber loss. It supplies per-parameter importance scores $F_i$; EWC then adds the penalty $\\frac{\\lambda}{2}\\sum_i F_i(\\theta_i-\\theta_{i,\\mathrm{pre}})^2$ to the fine-tuning loss, pinning down parameters whose movement would destroy pretrained knowledge. Replay contributes a second ingredient: in each epoch, after a mini-batch from the LPSC fine-tuning set, a same-size mini-batch drawn from a 10\\% random sample of MPtrj is used for an extra parameter update. reEWC is simply the two mechanisms run together, with $\\lambda=10^5$ (compared with $10^6$ for EWC alone), and the claim is that the replay signal makes the Fisher penalty more effective while the penalty prevents the large, unconstrained parameter shifts that make Replay-only models unstable on out-of-domain materials. The noise-perturbation experiment (accuracy survives larger perturbations when noise is scaled by the inverse Fisher matrix) is the direct evidence that these importance scores are meaningful.","core_discovery":"The central claim is that reEWC—running the Replay mini-batch schedule while adding the EWC Fisher-information penalty—achieves a favorable stability–plasticity balance that neither method reaches alone. The paper reports that on a held-out sample of the pretraining set (sMPtrj), reEWC keeps loss close to the pretrained model's own loss, better than EWC even though its parameters move more; on the target LPSC dataset it reduces energy and force errors and removes the systematic softening that makes the pretrained model overestimate Li diffusivity. It also reports that reEWC transfers improved accuracy across chemically distinct solid electrolytes, reproduces DFT quasi-melting behavior, and avoids the unphysical short Li–Li bonds that arise in MD with a Vanilla- or Replay-only fine-tuned model. The conclusion the authors draw is that reEWC should be a default fine-tuning strategy for continual learning in pretrained MLIPs.","pith_inferences":["If this recipe is as robust as reported, the same replay-plus-Fisher combination should apply to other pretrained universal interatomic potentials and other target materials; the paper demonstrates it on one model and one target, so that extension is unverified.","The filtered-Fisher trick also suggests a cheap pre-deployment check: compute where the pretrained model is confident, and use those parameters as anchors before adapting it to a niche.","The Li$_3$N finding implies that loss-based forgetting metrics can miss catastrophic dynamical failure; reEWC's advantage may be as much about bounding parameter shifts as about protecting loss values. This is an editorial reading, not a claim the paper makes explicitly."],"forward_implications":["Fine-tuned reEWC models keep near-pretrained accuracy on the original pretraining domain (sMPtrj loss comparable to SevenNet-0), so they remain safe for general materials screening.","The known PES softening and overestimated Li diffusivity of the pretrained model are corrected on LPSC, making simulated ion transport realistic.","Knowledge gained from one fine-tuning target transfers to chemically related systems, including PS$_4$-containing sulfides and also oxides, nitrides, and halides.","reEWC remains stable when the replay set is changed, e.g., restricted to Li-containing compounds, whereas Replay alone forgets under the same change.","The added cost is modest: reEWC takes about twice the vanilla fine-tuning time, which the authors consider acceptable for lab-scale GPU resources."],"supporting_citations":[{"why":"Introduces Elastic Weight Consolidation and the Fisher-information penalty that reEWC inherits.","marker":"[33]"},{"why":"Supplies the Experience Replay strategy whose mini-batch schedule reEWC adopts.","marker":"[35]"},{"why":"The pretrained SevenNet-0 model that all fine-tuning starts from.","marker":"[17]"},{"why":"Source of the MPtrj pretraining dataset and of the sMPtrj test set used to measure forgetting.","marker":"[18]"},{"why":"Documents systematic PES softening in universal MLIPs, the target problem fine-tuning must cure.","marker":"[27]"},{"why":"Identifies severe PES softening in LPSC, the chosen fine-tuning target.","marker":"[39]"}],"fun_headline_variants":["Replay plus EWC fine-tuning prevents MLIP catastrophic forgetting","Hybrid replay-EWC keeps pretrained interatomic potentials general","reEWC: forget-free fine-tuning for universal machine-learning interatomic potentials","Combining replay and weight consolidation stops MLIP fine-tuning memory loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that Fisher-importance scores computed from a filtered subset of the pretraining data correctly identify which parameters must stay fixed, even though fine-tuning follows structured loss gradients rather than random noise.","fun_headline_variants_meta":{"raw":{"variants":["Replay plus EWC fine-tuning prevents MLIP catastrophic forgetting","Hybrid replay-EWC keeps pretrained interatomic potentials general","reEWC: forget-free fine-tuning for universal machine-learning interatomic potentials","Combining replay and weight consolidation stops MLIP fine-tuning memory loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1572,"prompt_tokens":992,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":504}},"tokens_in":608,"tokens_out":580,"duration_ms":6547,"temperature":1.0,"reasoning_tokens":504,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:40:33.313279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Fisher matrix on the full MPtrj dataset instead of the 1,154,095 high-accuracy subset, retrain reEWC with that matrix, and compare held-out pretraining (sMPtrj) loss and LPSC accuracy; if the results are nearly identical, the subset choice is not load-bearing, and if they differ materially, the core mechanism depends on that filtering.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Elastic Weight Consolidation and the Fisher-information penalty that reEWC inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Experience Replay strategy whose mini-batch schedule reEWC adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The pretrained SevenNet-0 model that all fine-tuning starts from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the MPtrj pretraining dataset and of the sMPtrj test set used to measure forgetting."},{"cited_title":"npj Comput","cited_arxiv_id":null,"evidence_quote":"Documents systematic PES softening in universal MLIPs, the target problem fine-tuning must cure."}],"review_version":2}