{"id":"363b29ef-c075-4ce6-98ec-b1babddc45b1","arxiv_id":"2505.05946","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"EWC during Lithuanian continual pretraining of Gemma2 reduces catastrophic forgetting of English and improves Lithuanian fluency and accuracy on 5 of 7 knowledge benchmarks.","lead":"This paper tests whether a standard continual-learning trick, Elastic Weight Consolidation, can teach a 2-billion-parameter Gemma2 model Lithuanian without destroying its English knowledge. The authors report that EWC preserves English fluency and accuracy while improving Lithuanian in most benchmarks, offering a practical path for adapting LLMs to under-represented languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MMLU is both Fisher source and evaluation benchmark; no held-out check shows EWC protects general knowledge beyond MMLU-like skills.","rationale":"The paper is a straightforward empirical study: it asks whether EWC with a Fisher estimated on MMLU can add Lithuanian fluency to Gemma2-2B while preserving English fluency and domain knowledge. The empirical setup is transparent, hyperparameters are reported, and code is released. I read the central claim as the reader does: EWC, as implemented with MMLU-derived Fisher information, preserves English and improves Lithuanian competence across most benchmarks. The weakest link is not the training procedure itself but the evidential value of the benchmark results for the claimed generality. Because MMLU is used both to define parameter importance and as one of the evaluation benchmarks, the 7/7 and 5/7 counts partially measure the model's ability to stay close to MMLU-competent parameters rather than to preserve general domain knowledge. The other benchmarks may correlate with MMLU competence, and the Lithuanian translations may inherit English-centric reasoning, so the empirical result does not yet distinguish between 'broad knowledge protection' and 'protection of skills aligned with the Fisher source.' The authors acknowledge the missing theoretical justification, but the absence of a held-out Fisher source leaves the central generalization undemonstrated. The lack of error bars and multiple seeds compounds this: without variance estimates, the claimed improvements could be within noise. These concerns do not invalidate the paper, but they justify treating the conclusions as provisional and requiring the proposed de-circularization check before the general claim is accepted. Since the reader already recommended conditional acceptance, my assessment does not change the verdict.","tokens_in":7670,"tokens_out":6903,"duration_ms":75791,"concrete_test":"Hold MMLU out entirely: estimate the Fisher diagonal on the remaining six English benchmarks (ARC-Easy, Belebele, GSM8K, HellaSwag, TruthfulQA, Winogrande) as a combined source, then run the same Lithuanian continual pretraining with that Fisher and evaluate English and Lithuanian performance on those same six benchmarks, excluding MMLU. If the preservation/gain pattern survives this disjoint Fisher source, the MMLU circularity is not the driver; if the pattern weakens or reverses, the reported generalization is an artifact of using MMLU both as the importance source and as a benchmark. Report per-benchmark accuracies and, ideally, 3-seed means and standard deviations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claims—English preservation in 7/7 benchmarks and Lithuanian gains in 5/7—rest on EWC with Fisher information estimated from MMLU (Eq. 4, Sec. 3). The central premise is that this Fisher diagonal marks parameters important for domain knowledge in general, so the quadratic penalty protects broad competence. That premise is not actually tested. MMLU is included in the evaluation suite, so one of the seven benchmarks is favored by construction; the remaining six could correlate with MMLU-specific skills without establishing general-domain protection. The Lithuanian benchmarks [21,22] are translations of English tasks, so preserving the English semantic subspace may inflate Lithuanian accuracy even if no Lithuanian-specific domain knowledge is acquired. The authors' Limitations section acknowledges that 'Our approach to evaluating Fisher information via MMLU also asks for a theoretical justification,' but no empirical de-circularization is provided. Additionally, all counts are read from single-run curves without error bars or seeds, making 7/7 and 5/7 fragile comparisons. The concern is load-bearing because the practical conclusion—that this simple EWC recipe is a general mechanism for adding languages—requires that the Fisher source not be a proxy for the evaluation suite.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This preprint reports a continual-pretraining experiment in which the authors take Gemma2-2B-it, autoregressively train on 10% of the Lithuanian portion of CulturaX, and use Elastic Weight Consolidation with Fisher information computed on MMLU to preserve prior knowledge. The authors evaluate English and Lithuanian linguistic fluency via perplexity and domain knowledge via seven standard benchmarks, and report that EWC preserves English fluency and accuracy on all seven benchmarks while improving Lithuanian performance on five of seven. The paper is a technical report with hyperparameters and open-sourced code.","tokens_in":7900,"tokens_out":4682,"duration_ms":46623,"significance":"If the results hold, the paper makes a useful practical contribution: it suggests that a simple regularizer can add a low-resource language to an LLM without original training data and with modest degradation of general competence. The paper is also honest about limitations, including the lack of theoretical justification for the Fisher source. However, the strongest evidence is weakened by using MMLU as both the Fisher source and an evaluation benchmark, and by the absence of uncertainty quantification, so the practical significance hinges on further validation.","major_comments":[{"comment":"The claim of '7/7 cases' in the abstract and Section 1 is inflated by circularity: Eq. (4) in Section 3 estimates Fisher information on MMLU, and MMLU is one of the seven benchmarks in Table 1. The English preservation result should be recomputed excluding MMLU (6/6) and, ideally, with the Fisher computed on a different benchmark to demonstrate that the penalty protects general domain knowledge rather than MMLU-specific skills. As written, the paper's own Limitations section acknowledges the need for a theoretical justification, but an empirical de-circularization is absent.","section":"Section 3, Eq. (4) and Table 1"},{"comment":"All headline counts in Section 4 and Figure 3 come from single-run evaluations without error bars, confidence intervals, or multiple seeds. The differences between λ values that determine '7/7' and '5/7' may be within run-to-run noise, especially on benchmarks like TruthfulQA where the gaps appear small. The paper should report variance estimates at least for the selected λ values, or explicitly frame the results as observations from a single run rather than statistically supported claims.","section":"Section 4, Figures 1-3"},{"comment":"The contributions do not specify which value of λ produces the reported improvements. Figure 3 shows that the optimal λ varies by benchmark (e.g., GSM8K appears preserved at high λ while other benchmarks may require lower λ). The authors should define a prespecified selection rule (for example, the largest λ that preserves English fluency within a tolerance) and then report the Lithuanian benchmark counts for that λ, rather than inferring 5/7 from visual inspection of the full sweep.","section":"Section 4 and Conclusions"}],"minor_comments":[{"comment":"The perplexity computation on TruthfulQA is underspecified; please state whether the perplexity is computed over the answer tokens conditioned on the question, and how the Lithuanian translation is obtained.","section":"Section 3, Linguistic fluency"},{"comment":"Figure captions refer to 'Fine-tuned' as 'autoregressive pretraining with EWC regularisation,' but the λ=0 curve is also fine-tuned without EWC; the label should be clarified (e.g., 'continually pretrained' versus 'initial model').","section":"Figure captions"},{"comment":"The Lithuanian benchmarks from references [21,22] are used for evaluation, but their construction, number of instances, and translation methodology are not described; this is important for assessing the validity of the Lithuanian domain-knowledge results.","section":"Section 3, Domain knowledge"},{"comment":"Figure 2 caption says 'LT-Llama-13B' while the text uses 'LT-Llama2-13B'; please make the notation consistent.","section":"Figure 2 and Section 3"},{"comment":"The Limitations section should explicitly mention the lack of multi-seed runs and the MMLU circularity, since both are central to the strength of the evidence.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report with useful empirical results, but the evidence as presented is not yet at the standard of a journal publication. The authors are also the creators of the Lithuanian evaluation benchmarks [21,22]; this is not per se a problem, but the paper should state this connection and describe the benchmark construction. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: if you work on continual pretraining for under-resourced languages, this is a useful data point. The authors take Gemma2-2B, continue pretraining on 10% of Lithuanian CulturaX with EWC, and report that a simple regularizer keeps English fluency and domain knowledge intact while improving Lithuanian on most benchmarks. That is a concrete new measurement, not a new method.\n\nThe paper does several things well. The setup is clean and reproducible: they report hyperparameters, a sweep over lambda, and a public codebase. They evaluate both perplexity and accuracy on seven benchmarks in both languages, which is more thorough than most CL reports. The finding that EWC sometimes improves Lithuanian accuracy is interesting and not obvious.\n\nThe soft spots are real but not fatal. The main one is that MMLU is used both to compute the Fisher information and as one of the evaluation benchmarks. That makes the 7/7 claim for English preservation look better than it should, because one of the seven is favored by construction. The authors acknowledge this and include MMLU anyway, which is honest but does not fix the circularity. The six other benchmarks are independent, so the general claim is not purely circular, but the lack of error bars or multiple seeds means the 7/7 and 5/7 counts are fragile. Single-run curves can shift.\n\nA second concern: the Lithuanian benchmarks are translations of English tasks. If EWC protects the English semantic subspace, some of the Lithuanian gains could be cross-lingual transfer rather than genuinely new Lithuanian competence. The authors don't discuss this, and it would be easy to test by comparing against a model trained without EWC on an untranslated Lithuanian benchmark.\n\nThe Limitations section is candid: they admit the MMLU-Fisher link needs theoretical justification and that comparisons with other CL methods are missing. That helps trust but doesn't resolve the issues.\n\nBottom line: this deserves a serious referee. It would be a solid workshop paper or technical report. For a full conference, I'd want error bars, an MMLU-free Fisher variant, and ideally a second CL baseline. I'd accept it into review and ask for those.","headline":"A clean, reproducible empirical data point: EWC lets you add Lithuanian to Gemma2-2B without wrecking English, but the MMLU circularity and single runs make the headline counts softer than they look.","tokens_in":8405,"tokens_out":2316,"would_cite":false,"duration_ms":21993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using Elastic Weight Consolidation, this paper continually pretrains all of Gemma2's parameters on Lithuanian text and reports that English fluency and domain knowledge are preserved on every benchmark tested, while Lithuanian improves on…","keywords":["continual learning","elastic weight consolidation","catastrophic forgetting","large language models","Lithuanian language","perplexity","domain knowledge","full-parameter pretraining"],"falsifier":"Run the identical Lithuanian continual-pretraining protocol but estimate Fisher information on a single held-out benchmark (for example ARC-Easy) instead of MMLU; if the other six English benchmarks still stay flat the choice of MMLU is incidental, but if they drop, the paper's claim that MMLU-based Fisher protects domain knowledge fails for this setup.","tokens_in":7515,"feed_emoji":"🇱🇹","tokens_out":7684,"duration_ms":66635,"temperature":0.7,"pith_summary":"This technical report asks whether a large language model can learn a new language fluently without discarding what it already knows, and answers with a qualified yes: autoregressively pretraining all parameters of Gemma2 (2B version) on 10% of the Lithuanian part of the CulturaX dataset, with an Elastic Weight Consolidation (EWC) penalty, preserved English linguistic fluency and English domain knowledge on all seven benchmarks tested. The same regularizer also improved Lithuanian fluency and raised Lithuanian accuracy on five of the seven benchmarks. The authors interpret this as evidence that a simple regularizer, informed only by Fisher information computed on the MMLU benchmark, can add a low-resource language to a general-purpose model without access to the original training data.","feed_headline":"EWC lets Gemma2 learn Lithuanian without forgetting English","feed_subtitle":"All 7 English benchmarks hold; Lithuanian accuracy improves on 5 of 7 after adding 10% of CulturaX.","key_machinery":"The load-bearing object is the EWC regularizer, $L_B(\\theta)+\\frac{\\lambda}{2}\\sum_i F_i(\\theta_i-\\theta_{A,i})^2$, where $F_i$ is the empirical Fisher information of the model on MMLU data before continual pretraining. This penalty slows the movement of parameters that MMLU gradients mark as carrying domain knowledge, while leaving the rest of the model free to adapt to Lithuanian. The authors compute $F_i$ from gradients of the log-likelihood on MMLU question-answer pairs, so the identity of the protected knowledge is set entirely by what MMLU represents.","core_discovery":"The central claim is that EWC regularisation, with Fisher information estimated on MMLU, turns what would otherwise be a catastrophic forgetting event into a net gain. Without EWC ($\\lambda = 0$), English benchmark accuracy drops substantially after Lithuanian pretraining; with an intermediate regularisation strength (roughly $\\lambda \\in [10^2, 10^{11}]$), English accuracy is preserved on all seven benchmarks, and Lithuanian accuracy is higher than the untrained model on five of them. The paper states the result plainly: EWC mitigated catastrophic forgetting in the English component in linguistic fluency and domain knowledge in 7 of 7 cases, and improved the Lithuanian component in fluency and domain knowledge on 5 of 7 language-understanding benchmarks.","pith_inferences":["Nothing in the results ties the protected knowledge to a specific language, so a natural extension is to compute Fisher information on a mix of MMLU and Lithuanian text and check whether preservation survives on a held-out domain that MMLU underrepresents, such as code or medicine.","If the MMLU-derived Fisher mask generalises across domains, the same mask could be reused when adding other languages or modalities, turning EWC into a one-time audit of which parameters carry a pretrained model's competence.","Because the report covers one random seed and one data slice (10%), the 5/7 Lithuanian improvements could partly reflect dataset ordering or benchmark proximity to MMLU; repeating the run on a different slice of the Lithuanian corpus would test the stability of that result.","The observed co-occurrence of lower perplexity and higher benchmark accuracy, if causal, would make fluency training a proxy objective for domain knowledge in low-resource languages, a claim that is testable on other language pairs."],"forward_implications":["Adding Lithuanian under EWC preserves English fluency (perplexity) and English accuracy on all seven tested benchmarks, whereas the same pretraining without EWC degrades them.","The same run raises Lithuanian accuracy above the untrained baseline on ARC-Easy, GSM8K, HellaSwag, MMLU, and WinoGrande, while keeping Belebele and TruthfulQA stable.","Very strong regularisation ($\\lambda>10^{11}$) freezes the model so that it neither forgets English nor learns Lithuanian, bracketing an operational range for $\\lambda$ between roughly $10^2$ and $10^{11}$."],"supporting_citations":[{"why":"Supplies the Elastic Weight Consolidation regulariser that the paper adds on top of the language-model objective.","marker":"[5]"},{"why":"Justifies the empirical Fisher estimator used to compute parameter relevance from MMLU gradients.","marker":"[17]"},{"why":"Provides the conceptual template of continually pretraining a foundation model for a new language that the experiment follows.","marker":"[18]"},{"why":"Defines the Gemma2 2B model that is continually pretrained in all experiments.","marker":"[19]"},{"why":"Source of the Lithuanian web-crawl corpus whose first 10% forms the new-language training task.","marker":"[20]"},{"why":"Supplies the Lithuanian translations of the evaluation benchmarks and the Lithuanian Q/A data used for perplexity checks.","marker":"[21, 22]"},{"why":"The LT-Llama2-13B model used as a reference for measuring grammatical fluency of generated Lithuanian answers.","marker":"[23]"},{"why":"Defines the MMLU benchmark that provides both the Fisher-information data and one of the seven evaluation sets.","marker":"[24, 25]"}],"fun_headline_variants":["EWC lets Gemma2 add Lithuanian, keep English","Catastrophic forgetting? EWC says no for Gemma2","Gemma2 learns Lithuanian, retains all 7 English benchmarks","EWC turns catastrophic forgetting into net gain for Gemma2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the assumption that Fisher information computed on MMLU identifies the parameters underlying general domain knowledge, so that freezing them protects competence beyond MMLU; the paper itself notes that this step lacks a theoretical justification.","fun_headline_variants_meta":{"raw":{"variants":["EWC lets Gemma2 add Lithuanian, keep English","Catastrophic forgetting? EWC says no for Gemma2","Gemma2 learns Lithuanian, retains all 7 English benchmarks","EWC turns catastrophic forgetting into net gain for Gemma2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1890,"prompt_tokens":947,"completion_tokens":943,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":872}},"tokens_in":563,"tokens_out":943,"duration_ms":7188,"temperature":1.0,"reasoning_tokens":872,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:51:32.211443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical Lithuanian continual-pretraining protocol but estimate Fisher information on a single held-out benchmark (for example ARC-Easy) instead of MMLU; if the other six English benchmarks still stay flat the choice of MMLU is incidental, but if they drop, the paper's claim that MMLU-based Fisher protects domain knowledge fails for this setup.","supporting_citations":[{"cited_title":"Overcoming Catastrophic Forgetting in Neural Networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the Elastic Weight Consolidation regulariser that the paper adds on top of the language-model objective."},{"cited_title":"van de Ven","cited_arxiv_id":null,"evidence_quote":"Justifies the empirical Fisher estimator used to compute parameter relevance from MMLU gradients."},{"cited_title":"Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States","cited_arxiv_id":"2501.03952","evidence_quote":"The LT-Llama2-13B model used as a reference for measuring grammatical fluency of generated Lithuanian answers."}],"review_version":1}