Pith. sign in

REVIEW 4 major objections 6 minor 23 references

What Causes Knowledge Loss in Multilingual Language Models?

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Non-Latin scripts lose more knowledge in multilingual models

desk verdict A useful 52-language empirical map of donor/receiver languages, but the script-causality headline and the partial-sharing conclusion outrun the experiments. read the letter →

arxiv 2504.20356 v1 pith:FOFXCBMM submitted 2025-04-29 cs.CL

Maria Khelli , Samuel Cahyawijaya , Ayu Purwarianti , Genta Indra Winata This is my paper · ORCID
classification cs.CL
keywords catastrophicforgettingcross-lingualtransfermultilingualcontinuallearningLoRAadaptersscriptsimilarityMASSIVEdatasetmulti-hopmetricsslotfilling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to explain why a multilingual language model forgets some languages but not others when languages are added one by one. Training XLM-R sequentially on 52 languages with LoRA adapters, the authors claim that the writing system is the deciding factor: languages written in non-Latin scripts—Chinese, Japanese, Thai, Hindi—are the most likely to suffer catastrophic forgetting and the least able to pass knowledge on to later languages, while Latin-script languages transfer and retain better. The paper also proposes that adapter-based parameter sharing determines how much forgetting occurs: separate per-language adapters hold performance near joint-training levels, whereas a single shared adapter loses 15–30 F1 points. The abstract and conclusion mention partial parameter sharing as a mitigation, but no partially shared adapter condition actually appears in the experiments. The paper contributes a donor–receiver view of cross-lingual transfer with new multi-hop forward and backward transfer metrics to separate a language's effect on later languages from its vulnerability to earlier ones.

What carries the argument

The central machinery is a frozen base model (XLM-R) with LoRA adapters trained sequentially over the 52 MASSIVE languages, measured through split transfer metrics. A LoRA adapter is a low-rank update $\Delta W = BA$ to the frozen weights, so a SHARED LoRA condition updates one adapter across all languages while a NON-SHARED LoRA condition gives each language its own adapter. The paper adds Multi-Hop Forward Transfer and Multi-Hop Backward Transfer, which measure how training on language $i$ affects a language $h$ steps later or earlier in the sequence; these turn each language into a quantified donor or receiver and expose asymmetries that single-step performance shifts miss.

What would settle it

Train the same sequential loop with one language in two scripts: Hindi transliterated into Latin and in Devanagari, or Japanese romanized and in kana/kanji, keeping training order and adapter rank identical. If the Latin version forgets markedly less, the script/token-overlap story is causal; if both forget equally, the effect belongs to language family, morphology, or resource differences rather than script.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that knowledge loss in sequential multilingual training is language-dependent and predictable from script. Non-Latin-script languages act as both weak donors and vulnerable receivers: when zh-CN, ja-JP, or th-TH are trained late in the sequence, prior language performance drops sharply, and these languages themselves are easily overwritten. Latin-script languages such as Spanish, French, and German are the opposite—they donate knowledge to later tasks and remain stable after later training. The proposed mechanism is shared subword tokenization: Latin-script languages overlap heavily in subword tokens and vocabulary, so shared representations protect them, while non-Latin languages have token distributions that the tokenizer, optimized for Latin alphabets, fragments and that later training overwrites. The paper frames this as representational dominance of Latin-script languages in the shared model.

Load-bearing premise

The claim that script causes the forgetting differences rests on observational comparisons in which non-Latin script travels together with tokenizer fragmentation, language family, resource level, and training order, so the paper's own data cannot tell script apart from those confounds.

Editorial extensions

If this is right

  • If script is the cause, multilingual continual learning should avoid training non-Latin-script languages late in a shared-adapter sequence, or should isolate them in their own adapters.
  • A single shared LoRA adapter is a poor default for incremental language additions: in the paper's runs it drops 15–30 F1 points and shows the worst backward transfer, while non-shared adapters keep performance within about 2 points of joint training.
  • Because increasing LoRA rank from 32 to 256 barely changes shared-adapter forward transfer (CFT stays 0.51–0.53), capacity alone does not undo forgetting.
  • The same script/forgetting pattern appears with a second base model, so the effect is not tied to one architecture.
  • A language can look like a good donor simply because it appears before destructive languages in the training order, so position effects must be corrected by sequence-agnostic metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A directly testable extension the paper does not run: transliterate Hindi, Thai, or Japanese into Latin script and retrain the same sequence. If the script claim is causal, forgetting should shrink; if it does not, tokenizer fragmentation or language family is the real driver.
  • The paper's 'partial parameter sharing' conclusion is a statement of intent, not a measured result: the experiments include only fully shared and fully separate adapters, so any hybrid benefit remains an inference, not a finding.
  • The donor–receiver asymmetry suggests an allocation rule the authors do not state: give high-donor languages larger adapters and vulnerable receiver languages isolated or earlier slots; the multi-hop metrics could directly evaluate such schedules.
  • The metrics themselves could be applied to other continual-learning remedies—replay, regularization, adapter fusion—to see whether script asymmetries persist when forgetting is otherwise controlled.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies catastrophic forgetting in multilingual slot filling, comparing models trained jointly (MULTI), per language (MONO), sequentially (VANILLA), and with shared or non-shared LoRA adapters across 52 languages from MASSIVE, using XLM-R and E5 as base models. It introduces Multi-Hop Forward Transfer (MFT) and Multi-Hop Backward Transfer (MBT) metrics to analyze cross-lingual transfer over multiple training steps, and qualitatively relates the observed forgetting patterns to writing script (Latin vs. non-Latin), language family, and language vitality. The abstract states that non-Latin-script languages are more susceptible to catastrophic forgetting while Latin-script languages enable more effective cross-lingual transfer, and the conclusion claims that partial parameter sharing can mitigate forgetting while maintaining performance.

Significance. If the causal claim about script were substantiated, this would be a useful finding for multilingual continual learning, pointing to tokenization and script-aware scheduling as design levers. The paper has notable strengths: a 52-language setup, multiple random training orders, two base models, and a LoRA rank sweep provide a solid descriptive basis. However, the central causal claim is not supported by the experimental design, and one of the paper's own stated conclusions (partial parameter sharing) corresponds to no experiment actually run. The MFT/MBT definitions are incomplete, and the vitality analysis is circular. With targeted reanalysis and revised claims, the descriptive contribution could become a solid empirical study, but the current manuscript overstates its findings.

major comments (4)
  1. [Abstract and §5.2] The headline claim that non-Latin scripts cause greater catastrophic forgetting is not established by the presented experiments. Script is confounded with tokenizer fragmentation, language family, and training order; in §5.2 the authors themselves say the pattern 'likely reflects greater subword token overlap and lexical similarity,' which is a vocabulary-based mechanism, not script per se. A controlled comparison (e.g., matching languages on subword overlap or conditioning on language family) is needed to separate these explanations, or the claim should be weakened to a correlational observation.
  2. [§7 Conclusion and §2.2] The conclusion that 'partial parameter sharing can effectively mitigate forgetting while maintaining performance' is unsupported because no partial-sharing condition was run. The experimental methods include MULTI (fully shared), SHARED LoRA (one shared adapter), NON-SHARED LoRA (per-language adapters), VANILLA, and MONO. There is no condition in which some parameters are shared and others are not. The claim must either be removed or supported by new experiments.
  3. [§3.4, Eqs. (13) and (14)] The definitions of MFT and MBT are incomplete and ambiguous. In Eq. (13), P_i is stated to be the average performance on tasks seen up to step i, but the formula uses P_{i+h} - P_{i-1} with an unspecified set L and no explicit relation to the score matrix R in Eq. (10). Similarly, Eq. (14) uses P_i - P_{i-h-1}, but the indexing relative to the training trajectory is not defined. As written, these equations are not computable from the described data, which undermines the reproducibility of the central analysis in §5.
  4. [§5.4 and Table 1] The language vitality categories 'low,' 'mid,' and 'high' are used to interpret performance differences, but no definition or external source for these categories is provided, and the categorization appears to be derived post hoc from the same data it is used to explain. This is circular. The paper should specify the assignment criteria (e.g., speaker population, digital presence) and provide per-language labels, ideally from an independent resource.
minor comments (6)
  1. [Abstract and §1] The abstract and introduction state that the experiments cover 'non-shared, partially shared, and fully shared parameters,' but the methodology only describes shared and non-shared LoRA; the word 'partially' should be removed or a partial-sharing condition should be added.
  2. [§3.3.1, Eq. (9)] The performance shift metric is defined with N as the number of task sequences, but it is unclear how the average over five orders is integrated with the formula; please clarify the notation.
  3. [Figure 3 caption] The caption says 'on x-axis' but the figure appears to plot performance change for each language as categorical labels; please clarify the axes.
  4. [Table 1] The column 'Params (M)' mixes trainable and total parameter counts (e.g., MULTI, VANILLA, and NON-SHARED LoRA r=32 all report 278.04M); please specify whether these are trainable or total parameters and be consistent for MONO.
  5. [§2.3] There is a typo: 'It freezes the pre-trained weights' should be lowercase 'it'.
  6. [§4, Figure 4] The comparison of XLM-R and E5 is mentioned but the E5 training setup (rank, learning rate, epochs) is not specified; please provide these details for reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity; findings are observational, with interpretive/causal overclaims and a post-hoc training-order construction as the main caveats.

full rationale

The reported analyses are empirical and not definitionally circular. F1, CBT/CFT, MFT, and MBT are computed from held-out MASSIVE test scores under the training orders in Table 3; no parameter is fitted to the script or vitality findings, so those findings are not equivalent to their inputs by construction. The self-citation to Winata et al. (2023a) for CFT/CBT is not load-bearing: Section 3.4 redefines the metrics, and they are standard descriptive comparisons. The central caveats are interpretation and support gaps, not equation-level circularity. First, Section 5.2 attributes the Latin-script pattern to 'greater subword token overlap and lexical similarity,' which undercuts the abstract's causal script claim; script is confounded with tokenization and language family, making the headline overstatement a correctness risk rather than a circular derivation. Second, the Conclusion's 'partial parameter sharing can effectively mitigate forgetting' is not derived, because no partially shared adapter condition is implemented; only fully shared and non-shared LoRA conditions appear in Section 2.2. Third, Appendix A.1 constructs the fifth order by placing languages 'found to be particularly destructive' at the end and then includes that order in the five-order averages used to report which languages cause forgetting, a post-hoc selection loop. None of these issues reduces a result to a fitted parameter, a self-citation chain, or an equation identity, so the circularity score remains low.

Assumptions & free parameters 1 free parameters · 4 assumptions · 2 invented entities

The central claim rests on ordinary supervised training assumptions and on the interpretive assumption that script rather than tokenizer or family drives the observed differences. The vitality categorization is an authored construct with no external grounding, and the LoRA ranks are chosen hyperparameters.

free parameters (1)
  • LoRA rank r = 32, 64, 256
    LoRA rank is a chosen hyperparameter grid, not fitted to the target result. It directly controls trainable parameter counts in Table 1, but it is not a fitted constant.
assumptions (4)
  • domain assumption MASSIVE slot-filling labels are correct and uniform across all 52 languages with the stated train/validation/test splits.
    All performance measurements depend on the MASSIVE dataset's gold annotations and the per-language sample counts stated in Section 3.1.
  • domain assumption Average F1 over five language orders is a sufficient statistic for forgetting behavior.
    Section 3.3.1 averages performance shifts over five orderings (N=5) but never reports variance or significance, so conclusions assume these five orders are representative.
  • domain assumption XLM-RBASE and E5 instruct are representative multilingual encoders for the conclusions.
    Only two models are tested, and the authors acknowledge this in the Limitations section, so the generality of the findings rests on this assumption.
  • domain assumption Subword tokenizer segmentation is the principal mechanism underlying the observed script differences.
    Section 5.2 hypothesizes that the tokenizer is optimized for Latin scripts and causes representational dominance, but the paper provides no token-overlap statistics or controlled test of this mechanism.
invented entities (2)
  • Multi-hop transfer metrics MFT and MBT
    purpose: Measure how a language donates or receives knowledge across multiple training steps in the sequential learning setting.
    These metrics are introduced in Section 3.4, but their definitions leave indices unspecified and they are not validated against any external benchmark or correlated with any independent measure.
  • Language vitality categories (low, mid, high)
    purpose: Group languages in Table 1 and Section 5.4 to analyze how speaker population and digital presence affect continual learning outcomes.
    No data source, threshold, or reproducible definition is given for the low/mid/high labels; the categorization appears authored ad hoc and is then used to interpret the same data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Causes Knowledge Loss in Multilingual Language Models?." pith.science (2026). https://pith.science/paper/FOFXCBMM

@misc{pith2026250420356,
  author       = {Pith},
  title        = {Pith review of: What Causes Knowledge Loss in Multilingual Language Models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FOFXCBMM}},
  note         = {Machine review of arXiv:2504.20356}
}
read the original abstract

Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that process all data simultaneously often fail to mimic real-world scenarios, leading to challenges like catastrophic forgetting, where fine-tuning on new tasks degrades performance on previously learned ones. Our study explores this issue in multilingual contexts, focusing on linguistic differences affecting representational learning rather than just model parameters. We experiment with 52 languages using LoRA adapters of varying ranks to evaluate non-shared, partially shared, and fully shared parameters. Our aim is to see if parameter sharing through adapters can mitigate forgetting while preserving prior knowledge. We find that languages using non-Latin scripts are more susceptible to catastrophic forgetting, whereas those written in Latin script facilitate more effective cross-lingual transfer.

Figures

Figures reproduced from arXiv: 2504.20356 by the authors.

Figure 1
Figure 1. Pipeline for various approaches in lifelong [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance results after training each lan [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance change after training a certain language on x-axis in sequential training ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison results between XLM-R and E5 models. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of Multi-hop Backward Transfer (MBT), illustrates how training on later languages affects [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Heatmap of Multi-hop Forward Transfer (MFT), represents each language’s ability to donate knowledge [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Heatmap on VANILLA method for first language order. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

  1. [1]

    Kartikeya Badola, Shachi Dave, and Partha Talukdar. 2023. Parameter-efficient finetuning for robust continual multilingual learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9763--9780

    work page 2023

  2. [2]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, \'E douard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440--8451

    2020

  3. [3]

    Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, et al. 2023. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Li...

    work page 2023

  4. [4]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, et al. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of the 36th International Conference on Machine Learning (ICML)

    work page 2019

  5. [5]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . arXiv preprint arXiv:2106.09685

    arXiv 2021

  6. [6]

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. https://proceedings.mlr.press/v119/hu20b.html XTREME : A massively multilingual multi-task benchmark for evaluating cross-lingual generalization . In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine L...

    work page 2020

  7. [7]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences

    work page 2017

  8. [8]

    Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2021. Preserving cross-linguality of pre-trained models via continual learning. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 64--71, Bangkok, Thailand (Online). Association for Computational Linguistics

    work page 2021

Show all 23 references
  1. [9]

    Michael McCloskey and Neal J. Cohen. 1989. https://doi.org/10.1016/S0079-7421(08)60536-8 Catastrophic interference in connectionist networks: The sequential learning problem . In Psychology of Learning and Motivation, volume 24, pages 109--165. Elsevier

    1989 doi

  2. [10]

    Meryem M’hamdi, Xiang Ren, and Jonathan May. 2023. Cross-lingual continual learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3908--3943, July 9-14, 2023. Association for Computational Linguistics

    2023

  3. [11]

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020 a . https://arxiv.org/abs/2005.00247 Adapterfusion: Non-destructive task composition for transfer learning . arXiv preprint arXiv:2005.00247

    2020 arXiv

  4. [12]

    Jonas Pfeiffer, Ivan Vuli \'c , Iryna Gurevych, and Sebastian Ruder. 2020 b . MAD-X : An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654--7673. Associat...

    2020

  5. [13]

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32

    2019

  6. [14]

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, et al. 2016. Progressive neural networks. In arXiv preprint arXiv:1606.04671

    2016 arXiv

  7. [15]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672

    2024 arXiv

  8. [16]

    Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preo t iuc-Pietro. 2023 a . Overcoming catastrophic forgetting in massively multilingual continual learning. In Findings of the Association for Computational Li...

    2023

  9. [17]

    Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, et al. 2023 b . Nusax: Multilingual parallel sentiment dataset for 10 indonesian local languages. In Proceed...

    2023

  10. [18]

    Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, et al. 2024. Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual ...

    2024 arXiv

  11. [19]

    Genta Indra Winata, Guangsen Wang, Caiming Xiong, and Steven Hoi. 2021. Adapt-and-adjust: Overcoming the long-tail problem of multilingual speech recognition. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, page 361

    2021

  12. [20]

    Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. 2024. Towards lifelong learning of large language models: A survey. arXiv preprint arXiv:2406.06391

    2024 arXiv

  13. [21]

    Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. 2025. Lifelong learning of large language model based agents: A roadmap. arXiv preprint arXiv:2501.07278

    2025

  14. [22]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  15. [23]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.