{"id":"80b6020c-1b9f-4993-8608-df9dbf10963a","arxiv_id":"2504.20356","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In sequential multilingual fine-tuning, non-Latin-script languages are forgotten more severely than Latin-script ones, and separating LoRA adapters per language reduces forgetting.","lead":"When a multilingual model is fine-tuned one language at a time, languages written in non-Latin scripts lose more accuracy than Latin-script languages, and separate adapters for each language retain knowledge better than one shared adapter. This matters for deploying multilingual assistants that must learn new languages without forgetting old ones.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Script effect is confounded with subword tokenization and language family; the paper's own §5.2 attribution to token overlap makes the causal script claim unsupported.","rationale":"The reader's weakest assumption identifies exactly this confound. I agree. The paper contains useful empirical material: 52-language MASSIVE experiments, five training orders, XLM-R/E5 consistency, and LoRA adapter comparisons. Those support the existence of language-dependent forgetting and the general inefficiency of shared adapters. They do not support the script-causal headline because script, tokenizer fragmentation, and family are entangled in the design. The proposed matched-family/token-fertility test would move the paper from descriptive correlation toward a defensible mechanistic claim. Since the reader already assigned CONDITIONAL on this basis, no verdict change is needed.","tokens_in":10691,"tokens_out":4605,"duration_ms":51769,"concrete_test":"Run a matched tokenizer-fertility analysis: for each of the 52 languages, compute average subword tokens per utterance on the MASSIVE train set and the overlap between the language's 20k most frequent tokens and the union of all other languages' top-20k sets. Then, within language-family blocks that contain both Latin and non-Latin script languages (e.g., Indo-Aryan hi-IN vs ur-PK; Slavic ru-RU vs pl-PL), compare mean MBT/CBT. If the non-Latin penalty persists within family after matching on token fertility, script survives; if it reverses or disappears, the headline must be reframed as a tokenizer/vocabulary-overlap effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that non-Latin script causes greater catastrophic forgetting is supported only by an observational contrast. In §5.2 the authors themselves write that the pattern 'likely reflects greater subword token overlap and lexical similarity,' and in §5.3 that family benefits 'may arise from shared scripts and vocabulary rather than deep structural similarity,' yet all analyses aggregate languages by script without controlling for tokenizer fragmentation, shared vocabulary, language family, or training-order resource. In XLM-R's shared SentencePiece vocabulary, non-Latin languages are systematically split into more, less-shared subwords, so the script dummy is a proxy for tokenization difficulty and family distance. The reported MBT/MFT differences therefore do not establish that writing system per se drives forgetting; a tokenizer or vocabulary-overlap explanation is equally or more consistent with the data and is in fact the paper's own preferred mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies catastrophic forgetting in multilingual slot filling, comparing models trained jointly (MULTI), per language (MONO), sequentially (VANILLA), and with shared or non-shared LoRA adapters across 52 languages from MASSIVE, using XLM-R and E5 as base models. It introduces Multi-Hop Forward Transfer (MFT) and Multi-Hop Backward Transfer (MBT) metrics to analyze cross-lingual transfer over multiple training steps, and qualitatively relates the observed forgetting patterns to writing script (Latin vs. non-Latin), language family, and language vitality. The abstract states that non-Latin-script languages are more susceptible to catastrophic forgetting while Latin-script languages enable more effective cross-lingual transfer, and the conclusion claims that partial parameter sharing can mitigate forgetting while maintaining performance.","tokens_in":10860,"tokens_out":2663,"duration_ms":28554,"significance":"If the causal claim about script were substantiated, this would be a useful finding for multilingual continual learning, pointing to tokenization and script-aware scheduling as design levers. The paper has notable strengths: a 52-language setup, multiple random training orders, two base models, and a LoRA rank sweep provide a solid descriptive basis. However, the central causal claim is not supported by the experimental design, and one of the paper's own stated conclusions (partial parameter sharing) corresponds to no experiment actually run. The MFT/MBT definitions are incomplete, and the vitality analysis is circular. With targeted reanalysis and revised claims, the descriptive contribution could become a solid empirical study, but the current manuscript overstates its findings.","major_comments":[{"comment":"The headline claim that non-Latin scripts cause greater catastrophic forgetting is not established by the presented experiments. Script is confounded with tokenizer fragmentation, language family, and training order; in §5.2 the authors themselves say the pattern 'likely reflects greater subword token overlap and lexical similarity,' which is a vocabulary-based mechanism, not script per se. A controlled comparison (e.g., matching languages on subword overlap or conditioning on language family) is needed to separate these explanations, or the claim should be weakened to a correlational observation.","section":"Abstract and §5.2"},{"comment":"The conclusion that 'partial parameter sharing can effectively mitigate forgetting while maintaining performance' is unsupported because no partial-sharing condition was run. The experimental methods include MULTI (fully shared), SHARED LoRA (one shared adapter), NON-SHARED LoRA (per-language adapters), VANILLA, and MONO. There is no condition in which some parameters are shared and others are not. The claim must either be removed or supported by new experiments.","section":"§7 Conclusion and §2.2"},{"comment":"The definitions of MFT and MBT are incomplete and ambiguous. In Eq. (13), P_i is stated to be the average performance on tasks seen up to step i, but the formula uses P_{i+h} - P_{i-1} with an unspecified set L and no explicit relation to the score matrix R in Eq. (10). Similarly, Eq. (14) uses P_i - P_{i-h-1}, but the indexing relative to the training trajectory is not defined. As written, these equations are not computable from the described data, which undermines the reproducibility of the central analysis in §5.","section":"§3.4, Eqs. (13) and (14)"},{"comment":"The language vitality categories 'low,' 'mid,' and 'high' are used to interpret performance differences, but no definition or external source for these categories is provided, and the categorization appears to be derived post hoc from the same data it is used to explain. This is circular. The paper should specify the assignment criteria (e.g., speaker population, digital presence) and provide per-language labels, ideally from an independent resource.","section":"§5.4 and Table 1"}],"minor_comments":[{"comment":"The abstract and introduction state that the experiments cover 'non-shared, partially shared, and fully shared parameters,' but the methodology only describes shared and non-shared LoRA; the word 'partially' should be removed or a partial-sharing condition should be added.","section":"Abstract and §1"},{"comment":"The performance shift metric is defined with N as the number of task sequences, but it is unclear how the average over five orders is integrated with the formula; please clarify the notation.","section":"§3.3.1, Eq. (9)"},{"comment":"The caption says 'on x-axis' but the figure appears to plot performance change for each language as categorical labels; please clarify the axes.","section":"Figure 3 caption"},{"comment":"The column 'Params (M)' mixes trainable and total parameter counts (e.g., MULTI, VANILLA, and NON-SHARED LoRA r=32 all report 278.04M); please specify whether these are trainable or total parameters and be consistent for MONO.","section":"Table 1"},{"comment":"There is a typo: 'It freezes the pre-trained weights' should be lowercase 'it'.","section":"§2.3"},{"comment":"The comparison of XLM-R and E5 is mentioned but the E5 training setup (rank, learning rate, epochs) is not specified; please provide these details for reproducibility.","section":"§4, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's descriptive results are potentially useful, but the manuscript as submitted overstates the causal role of script and claims a mitigation result (partial sharing) that was not tested. The missing partial-sharing condition is a particularly serious mismatch with the paper's own stated contributions. I would recommend requiring either new experiments or a substantial revision of the claims, and a careful reanalysis that controls for tokenizer overlap and language family before the paper is considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is the empirical setup: 52 languages, MASSIVE slot filling, XLM-R and E5, LoRA adapters, five training orders, and heatmaps that show which languages hurt or help when learned sequentially. The multi-hop transfer metrics (MFT/MBT) are a reasonable idea—measuring transfer at a distance rather than only next-task—and they do surface patterns that single-step CFT/CBT would miss. Credit is due for trying to separate donor and receiver roles.\n\nThe soft spots are real. The central abstract claim—that non-Latin script causes susceptibility to forgetting—is not supported by the experiments. Script is confounded with tokenizer fragmentation, language family, and training order. The authors themselves say the Latin-script advantage “likely reflects greater subword token overlap and lexical similarity” and that family benefits “may arise from shared scripts and vocabulary.” That is an alternative mechanism, not confirmation of script as the cause. There are no error bars or significance tests; the script comparison is descriptive heatmaps only.\n\nMore damaging: the conclusion says partial parameter sharing mitigates forgetting while maintaining performance, but no partial-sharing condition was run. The methods are shared LoRA and non-shared LoRA only. That sentence in the conclusion is not about any experiment in the paper.\n\nThe MFT/MBT definitions are also incomplete. Equations 13–14 use Pi without defining what set L is, what h ranges over, or how the average is indexed across the 52-language sequence. A reader cannot reproduce the heatmaps from the text.\n\nThe vitality labels (low/mid/high) are ad hoc—no formal definition is given—and they are used to interpret the same data they were derived from, which is mildly circular.\n\nWhat holds up: the shared vs non-shared LoRA comparison is clear and useful. NON-SHARED LoRA is far more stable and nearly matches MULTI, while SHARED LoRA degrades heavily. That is a solid, reproducible observation even if not surprising. The XLM-R vs E5 comparison adds robustness. The related work is appropriate; they build on Winata et al. 2023 and Badola et al. 2023 without inflating novelty.\n\nWho should read this: people working on multilingual continual learning who want a broad empirical map of donor and receiver languages. But they should treat the script finding as a hypothesis, not a result.\n\nRecommendation: send to peer review, but ask for major revision. The authors need to either control for tokenizer overlap and language family, or downgrade the script claim to an observational pattern; define MFT/MBT precisely; and remove or explicitly retract the partial-sharing conclusion, or actually run a partial-sharing condition.","headline":"A useful 52-language empirical map of donor/receiver languages, but the script-causality headline and the partial-sharing conclusion outrun the experiments.","tokens_in":11375,"tokens_out":2774,"would_cite":false,"duration_ms":29195,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Non-Latin scripts lose more knowledge in multilingual models","keywords":["catastrophic forgetting","cross-lingual transfer","multilingual continual learning","LoRA adapters","script similarity","MASSIVE dataset","multi-hop transfer metrics","slot filling"],"falsifier":"Train the same sequential loop with one language in two scripts: Hindi transliterated into Latin and in Devanagari, or Japanese romanized and in kana/kanji, keeping training order and adapter rank identical. If the Latin version forgets markedly less, the script/token-overlap story is causal; if both forget equally, the effect belongs to language family, morphology, or resource differences rather than script.","tokens_in":10514,"feed_emoji":"🧠","tokens_out":6056,"duration_ms":59927,"temperature":0.7,"pith_summary":"This paper sets out to explain why a multilingual language model forgets some languages but not others when languages are added one by one. Training XLM-R sequentially on 52 languages with LoRA adapters, the authors claim that the writing system is the deciding factor: languages written in non-Latin scripts—Chinese, Japanese, Thai, Hindi—are the most likely to suffer catastrophic forgetting and the least able to pass knowledge on to later languages, while Latin-script languages transfer and retain better. The paper also proposes that adapter-based parameter sharing determines how much forgetting occurs: separate per-language adapters hold performance near joint-training levels, whereas a single shared adapter loses 15–30 F1 points. The abstract and conclusion mention partial parameter sharing as a mitigation, but no partially shared adapter condition actually appears in the experiments. The paper contributes a donor–receiver view of cross-lingual transfer with new multi-hop forward and backward transfer metrics to separate a language's effect on later languages from its vulnerability to earlier ones.","feed_headline":"Non-Latin scripts lose more knowledge in multilingual models","feed_subtitle":"A 52-language study shows Latin-script languages transfer and retain better; separate adapters resist forgetting.","key_machinery":"The central machinery is a frozen base model (XLM-R) with LoRA adapters trained sequentially over the 52 MASSIVE languages, measured through split transfer metrics. A LoRA adapter is a low-rank update $\\Delta W = BA$ to the frozen weights, so a SHARED LoRA condition updates one adapter across all languages while a NON-SHARED LoRA condition gives each language its own adapter. The paper adds Multi-Hop Forward Transfer and Multi-Hop Backward Transfer, which measure how training on language $i$ affects a language $h$ steps later or earlier in the sequence; these turn each language into a quantified donor or receiver and expose asymmetries that single-step performance shifts miss.","core_discovery":"On the paper's own terms, the central discovery is that knowledge loss in sequential multilingual training is language-dependent and predictable from script. Non-Latin-script languages act as both weak donors and vulnerable receivers: when zh-CN, ja-JP, or th-TH are trained late in the sequence, prior language performance drops sharply, and these languages themselves are easily overwritten. Latin-script languages such as Spanish, French, and German are the opposite—they donate knowledge to later tasks and remain stable after later training. The proposed mechanism is shared subword tokenization: Latin-script languages overlap heavily in subword tokens and vocabulary, so shared representations protect them, while non-Latin languages have token distributions that the tokenizer, optimized for Latin alphabets, fragments and that later training overwrites. The paper frames this as representational dominance of Latin-script languages in the shared model.","pith_inferences":["A directly testable extension the paper does not run: transliterate Hindi, Thai, or Japanese into Latin script and retrain the same sequence. If the script claim is causal, forgetting should shrink; if it does not, tokenizer fragmentation or language family is the real driver.","The paper's 'partial parameter sharing' conclusion is a statement of intent, not a measured result: the experiments include only fully shared and fully separate adapters, so any hybrid benefit remains an inference, not a finding.","The donor–receiver asymmetry suggests an allocation rule the authors do not state: give high-donor languages larger adapters and vulnerable receiver languages isolated or earlier slots; the multi-hop metrics could directly evaluate such schedules.","The metrics themselves could be applied to other continual-learning remedies—replay, regularization, adapter fusion—to see whether script asymmetries persist when forgetting is otherwise controlled."],"forward_implications":["If script is the cause, multilingual continual learning should avoid training non-Latin-script languages late in a shared-adapter sequence, or should isolate them in their own adapters.","A single shared LoRA adapter is a poor default for incremental language additions: in the paper's runs it drops 15–30 F1 points and shows the worst backward transfer, while non-shared adapters keep performance within about 2 points of joint training.","Because increasing LoRA rank from 32 to 256 barely changes shared-adapter forward transfer (CFT stays 0.51–0.53), capacity alone does not undo forgetting.","The same script/forgetting pattern appears with a second base model, so the effect is not tied to one architecture.","A language can look like a good donor simply because it appears before destructive languages in the training order, so position effects must be corrected by sequence-agnostic metrics."],"supporting_citations":[{"why":"Supplies XLM-R, the frozen base model whose tokenizer and representations produce the script-related forgetting patterns.","marker":"(Conneau et al., 2020)"},{"why":"Supplies MASSIVE, the 52-language slot-filling dataset used for every training and evaluation run.","marker":"(FitzGerald et al., 2023)"},{"why":"Defines LoRA, the low-rank adapter mechanism whose shared and non-shared variants are the experimental comparison.","marker":"(Hu et al., 2021)"},{"why":"Provides the continual-learning setting and the CBT/CFT metrics that the paper extends with multi-hop transfer.","marker":"(Winata et al., 2023a)"},{"why":"Prior work on parameter-efficient adapters for continual multilingual learning that motivates the adapter-based design.","marker":"(Badola et al., 2023)"}],"fun_headline_variants":["Script predicts which languages forget in multilingual training","Latin scripts shield language models from forgetting","Why non-Latin languages get forgotten in multilingual AI","Multilingual models: Latin script languages retain better","Non-Latin languages more prone to catastrophic forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that script causes the forgetting differences rests on observational comparisons in which non-Latin script travels together with tokenizer fragmentation, language family, resource level, and training order, so the paper's own data cannot tell script apart from those confounds.","fun_headline_variants_meta":{"raw":{"variants":["Script predicts which languages forget in multilingual training","Latin scripts shield language models from forgetting","Why non-Latin languages get forgotten in multilingual AI","Multilingual models: Latin script languages retain better","Non-Latin languages more prone to catastrophic forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1251,"prompt_tokens":845,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":461,"tokens_out":406,"duration_ms":4192,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:30:34.669125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same sequential loop with one language in two scripts: Hindi transliterated into Latin and in Devanagari, or Japanese romanized and in kana/kanji, keeping training order and adapter rank identical. If the Latin version forgets markedly less, the script/token-overlap story is causal; if both forget equally, the effect belongs to language family, morphology, or resource differences rather than script.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MASSIVE, the 52-language slot-filling dataset used for every training and evaluation run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on parameter-efficient adapters for continual multilingual learning that motivates the adapter-based design."}],"review_version":1}