A Study of Crosslinguistic Influence in Language Models

Abderrahmane Issam; Gerasimos Spanakis; Jan Scholtes; Yusuf Can Semerci

REVIEW 5 major objections 6 minor 5 references

A Study of Crosslinguistic Influence in Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In bilingual language models, a dominant first language systematically warps the second language's grammar in proportion to syntactic distance.

desk verdict Behavioral age-of-exposure effect is plausible, but the abstract promises a mechanistic attention ablation that the paper never runs—fix that mismatch before treating the causal claims seriously. read the letter →

arxiv 2601.21587 v2 pith:FJADFFDN submitted 2026-01-29 cs.CL

Abderrahmane Issam , Yusuf Can Semerci , Jan Scholtes , Gerasimos Spanakis This is my paper · ORCID

classification cs.CL MSC 68T50

keywords crosslinguisticinfluencelanguagemodelsbilingualtrainingstructuralprimingsyntacticdistancedominancesecondacquisitionneuronoverlap

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that crosslinguistic influence (CLI) in language models is not a haphazard side effect of limited capacity, but a structured phenomenon driven by two factors: how dominant the first language is and how proficient the second language is. Using bilingual models trained on one language first and a second language later, the authors show that the more entrenched the L1, the stronger the correlation between structural interference and typological distance between L1 and L2. They also show that a late-introduced L2 has less capacity for positive transfer, making it vulnerable to persistent interference from distant L1s. The paper further provides mechanistic evidence: L1 is co-activated during L2 processing, and syntactically similar L1s share more L2 neurons. If correct, this means training order and language similarity jointly determine how much a model's second language is shaped by its first, giving researchers a controlled way to study bilingualism.

What carries the argument

The central object is the Step of Exposure (SoE)—the training step at which the L2 is introduced—which jointly manipulates L1 dominance and L2 proficiency. The argument runs on several interlocking tools: syntactic distance measured as the number of non-shared WALS features between L1 and English; crosslinguistic structural priming using NLLB translations of BLiMP sentences as primes; LogitLens to decode L1 token presence during L2 processing; and Parallel Language-specific Neuron Detection (PLND) to identify L2 neurons and measure their overlap across L1 models. These tools connect behavioral accuracy differences (CLI scores) to internal architectural signatures (L1 co-activation, neuron ov

What would settle it

Train the same bilingual framework on a substantially larger set of L1s (e.g., 20 languages with densely sampled syntactic distances) and check whether the CLI-distance correlation and the neuron-overlap gradient survive; if a language with few non-shared WALS features behaves like a distant language, or if the correlation vanishes under an alternative distance measure (e.g., human-rated typological similarity), the paper's central claim would be refuted. A second direct falsifier: replace NLLB primes with translations that are known to preserve vs. distort the target English structure; if dis

Watch

Extended reading notes

Core claim

The central claim is that CLI in language models follows a predictable gradient: as the L1 becomes more dominant (operationalized as later introduction of the L2 during training), the correlation between structural transfer and syntactic distance strengthens dramatically, while reduced L2 proficiency suppresses the model's ability to benefit from positive transfer. Crosslinguistic structural priming—prepending an L1 translation of a grammatical sentence—amplifies these effects, facilitating processing for similar languages and degrading it for distant ones. Mechanistically, the paper finds that the L1 is physically co-activated in hidden states during L2 processing (measured via LogitLens) a

Load-bearing premise

The entire distance-gradient analysis rests on assuming that the number of non-shared WALS features between L1 and L2 is a faithful measure of the syntactic distance that a language model's learning is actually sensitive to; if that feature-based distance does not capture what the model's representations encode, the central correlations and the neuron-overlap interpretation lose their meaning.

Editorial extensions

If this is right

If L1 entrenchment amplifies distance-correlated interference, then the order and timing of languages in multilingual pretraining is not neutral: a curriculum that entrenches one language first will systematically shape downstream performance in the second.
Crosslinguistic priming effects that scale with syntactic distance give a behavioral test for whether a model has internalized shared syntactic representations, making priming a usable diagnostic for transfer in multilingual models.
The neuron-overlap result implies that typological similarity physically determines how much neural circuitry is shared between languages, which could inform architectural choices for low-resource language transfer.
The romanization finding indicates that orthographic overlap facilitates CLI, suggesting that surface script similarity is a controllable factor in cross-lingual transfer, not just deep syntax.
The asymmetry result—shared structures prime bidirectionally but ungrammatical structures only from dominant L1 to L2—suggests that the shared-versus-connected debate about bilingual syntax can be resolved by considering structural overlap rather than a single universal account.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An extension the paper leaves implicit: if CLI scales with syntactic distance and dominance, then multilingual models trained with a dominant pivot language (e.g., English-first) may systematically exhibit predictable negative transfer for structurally distant languages—a testable prediction for large-scale multilingual pretraining.
One could test the causal claim more directly by independently manipulating L2 proficiency (e.g., by varying L2 training volume while holding SoE constant) to see whether the distance gradient persists when dominance is not confounded with proficiency.
The WALS-feature distance measure is a coarse proxy; a stronger test would use a behavioral or human-judgment-based syntactic similarity measure, or a larger set of L1s, to check whether the correlation between distance and interference is robust across alternative distance metrics.
If the mechanistic account is right, then targeted interventions—such as pruning or dampening L1-specific neurons during L2 inference—should predictably modulate CLI, offering a direct causal test of the neuron-overlap claim.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: the paper's advertised contribution is two results at different strengths. The behavioral one—later L2 introduction strengthens the correlation between WALS-based syntactic distance and L2 interference on BLiMP—looks plausible and is worth testing further. The mechanistic one, as advertised in the abstract, is not in the paper. There is no attention ablation anywhere in Section 5 or the appendices. The abstract says 'targeted causal ablations ... confirm that deep-layer attention mechanisms exclusively drive this crosslinguistic transfer.' No such experiment exists. That's a load-bearing gap, not a cosmetic one.

What's actually new: a clean-ish setup for varying age of exposure across five typologically distinct L1s, with a monolingual baseline that shares the bilingual tokenizer, and a metric for CLI that separates positive and negative transfer. The FCE non-native preference analysis, the LogitLens co-activation measure, and the neuron-overlap correlation are all reasonable exploratory instruments. The paper also engages the SLA-LM literature properly; the related-work discussion positions itself against Oba, Yadavalli, Constantinescu, and Arnett without hand-waving.

Soft spots, in order of severity. First, the abstract overclaims on both the number of languages (15 vs. 5) and the attention ablation. That's a manuscript integrity issue that an editor should catch before review. Second, the mechanistic language—'physically dictates,' 'exclusively drive'—goes beyond correlation. The neuron-overlap correlation is interesting, but it doesn't rule out MLP or residual contributions or tokenizer artifacts. Third, the main Figure 2 has no error bars or confidence intervals; the permutation tests are referenced and Figure 12 shows something, but the main text doesn't describe the test procedure or how many permutations. That's fixable. Fourth, the WALS feature count as syntactic distance is a crude but defensible proxy; with only five languages, the correlation is fragile. The translations from NLLB are a genuine confound, though Figure 11 gives some reassurance.

Bottom line: the core behavioral finding could survive a serious referee, but the current manuscript doesn't earn the mechanistic headline. Worth engaging, but with the abstract heavily reframed and the missing analysis either supplied or retracted. I'd send it to review, but I'd expect major revision. For your own work, I wouldn't cite the causality claims, but the distance-by-soe interaction is potentially useful once it's backed by numbers.

Referee Report

5 major / 6 minor

Summary. The paper trains bilingual GPT-2 models on English and one of five L1s, varying the step at which L2 is introduced (SoE), and studies crosslinguistic influence (CLI) via BLiMP accuracy, FCE non-native preference, crosslinguistic priming, LogitLens, and L2 neuron overlap. The central claim is that later L2 introduction (higher L1 dominance, lower L2 proficiency) produces a syntactic-distance-dependent interference gradient, and that this gradient is mechanistically rooted in L1 co-activation and overlapping L2 neurons. The submitted body, however, does not contain the causal attention ablation advertised in the arXiv abstract, and the statistical support for the core correlation is incomplete.

Significance. If the behavioral findings hold, the paper would provide a controlled computational demonstration that age of exposure and typological distance jointly shape CLI, with potential implications for bilingualism theory and multilingual LM training. The use of external measures (BLiMP, FCE, WALS-based distances) and the attempt to connect behavior to internal representations are strengths. However, the advertised mechanistic exclusivity claim is unsupported, and the five-language, single-seed evidence base makes the central correlation fragile. The paper's contribution is therefore currently more modest than its abstract claims.

major comments (5)

[Abstract vs. §3.2, §5] The abstract states that the study covers '15 typologically diverse L1s' and that 'targeted causal ablations ... confirm that deep-layer attention mechanisms exclusively drive' CLI. §3.2 trains only five L1s, and §5 reports no attention manipulation. The exclusivity claim is load-bearing and is not substantiated anywhere in the body or appendices. Either provide the attention ablation with appropriate controls, or remove the claim and reconcile the abstract with the actual scope.
[§4.1, Table 1, Appendix B.3] The central distance-gradient result rests on five languages and is plotted without error bars or confidence intervals. The text invokes permutation tests ('Appendix B.3') to support the result, but that appendix contains only Figure 12 and no test statistic, p-value, or permutation distribution. With five points, a single outlier can drive the correlation. Report the permutation results, add confidence intervals, and ideally increase the number of L1s or use multiple seeds.
[§3.3.3, Appendix B.2] The priming paradigm depends on NLLB translations preserving the target L1 structure, and the translation-quality ablation is presented only as two figures with no quantitative summary or description of the comparison. The text in §4.1 states that the ablation 'confirm[s]' results are not explained by lexical overlap or translation quality, but the submitted material does not support that assertion. Provide the actual comparison statistics and a narrative description of what was contrasted.
[§4.3.2, Figure 6] The neuron-overlap result is reported as a 'consistent negative correlation' without a correlation coefficient, confidence interval, or sample size. The top-25% activation threshold and 'consistently ranking' criterion are not operationally defined. Since this is the main mechanistic evidence for the claim that 'L1 typological proximity physically dictates' L2 circuitry, it needs statistical rigor and a clearer description of the neuron identification procedure.
[§3.3.2, Eq. (3)] The FCE preference measure normalizes by the model's own surprisal differences summed over all learner groups, so each model's baseline is confounded with its own L1. A control — such as a monolingual English model or a model without L1-specific training — is needed to demonstrate that the ΔS pattern reflects L1-specific preference rather than overall surprisal differences. Single-seed results (Appendix A, seed 123) also provide no uncertainty estimate.

minor comments (6)

[Table 1] Rename the 'Latin' column to 'Latin script' to avoid confusion with the Latin language family; the current columns 'Indo-European' and 'Latin' overlap in meaning.
[Eq. (2)] The notation Acc_{L1,L2} is ambiguous; clarify that it is the bilingual model's accuracy and explicitly state why a positive value is labeled 'interference' rather than 'negative transfer'.
[§2, References] There is a typo 'the the shared-syntax hypothesis', and the reference 'NICOLADIS (2006)' should be normalized to title case.
[Appendices B.1–B.3] The appendix headings are followed by figures but almost no prose. Add at least a paragraph per subsection describing the method, the metric, and the conclusion.
[Figure 6] Add error bars or scatter distributions; the claimed 'wider confidence interval' for German is not visible from the current figure.
[Title] The full-text title differs from the arXiv title; align them in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's core correlations and mechanistic analyses use independent external measures and training-schedule manipulations, with no fitted parameter renamed as a prediction or load-bearing self-citation chain.

full rationale

The paper's central empirical chain is non-circular. The CLI metric in Eq. 2 is defined as the relative BLiMP accuracy of a bilingual model against a monolingual baseline trained with the same bilingual tokenizer; the regressor (WALS-based syntactic distance) is an external feature count and the target (BLiMP accuracy) is an external benchmark. The dominance/proficiency manipulation (age of exposure / SoE) is a training-schedule variable, not a parameter fitted to the outcomes. The FCE measure in Eq. 3 is self-referential by design—each model is tested on its own L1 learner errors—but that is the experimental setup, not a derivation: the normalized surprisal gap is measured, not optimized to produce the claimed preference ordering, and the preference claim is then compared across models rather than derived from the metric's definition. The mechanistic analyses are also independent of the behavioral results: LogitLens L1-token ratios are decoded from hidden states during L2 processing, and L2 neuron overlap is correlated with an externally defined WALS distance; no parameter is fitted to the reported correlations and then presented as a prediction. The ablations (early imbalanced training, L1/L2 reversal, model scale, translation-model sensitivity, and permutation controls) are control comparisons rather than fitted predictions. All load-bearing method citations—Papadimitriou & Jurafsky for syntactic distance, Warstadt et al. for BLiMP, Yannakoudakis et al. for FCE, Zhao et al. for PLND, and NLLB for translations—are external prior work by other authors, so there is no self-citation chain carrying the argument. Concerns about the unsupported abstract claim that 'deep-layer attention mechanisms exclusively drive' CLI, and about the validity of WALS feature counts as the operative syntactic distance, are evidentiary or correctness risks, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. The central claims rest on domain assumptions about the validity of BLiMP, WALS distance, NLLB translations, and interpretability thresholds. The age-of-exposure steps and neuron/LogitLens thresholds are hand-chosen experimental parameters, not fitted constants.

free parameters (4)

Age of exposure steps (SoE) = 0, 16K, 32K, 48K
Hand-chosen experimental conditions; not fitted to data, but they define the dominance/proficiency gradient on which the central claim rests.
Top-25% activation threshold for language neurons = 25%
Threshold chosen following PLND (Zhao et al. 2024); directly determines the neuron-overlap numbers in §4.3.2.
Token classification frequency ratio = 2x
A token is classified L1- or L2-specific if its relative frequency in one corpus is at least twice that in the other; affects LogitLens language-presence measurements.
WALS feature set (75 features) = 75 reported features
Syntactic distance is the number of non-shared features after filtering to features reported for all languages; the choice of features determines the language ordering used in every correlation.

assumptions (5)

domain assumption BLiMP accuracy is a valid proxy for L2 grammatical proficiency.
Used throughout §4.1 to define CLI and to measure the effect of L1 dominance.
domain assumption WALS feature dissimilarity is a valid measure of syntactic distance for CLI.
Set in §3.2/Table 1 and used as the x-axis for all distance correlations.
domain assumption NLLB translations preserve the target grammatical structure of the English acceptable sentence.
Primes in §3.3.3 are L1 translations of the acceptable BLiMP sentence; the translation-quality ablation in Appendix B.2 is referenced but not described.
domain assumption GPT-2 trained on 1B tokens per language from OSCAR is a plausible artificial learner of human grammars.
The paper's framing as a model of human CLI depends on this assumption, which is argued but not independently established.
domain assumption PLND language neurons correspond to syntactic/structural representations.
The neuron-overlap analysis in §4.3.2 interprets shared L2 neurons as 'syntactic neurons' despite PLND identifying language-specific neurons, not syntax-specific ones.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Study of Crosslinguistic Influence in Language Models." pith.science (2026). https://pith.science/paper/FJADFFDN

@misc{pith2026260121587,
  author       = {Pith},
  title        = {Pith review of: A Study of Crosslinguistic Influence in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FJADFFDN}},
  note         = {Machine review of arXiv:2601.21587}
}

read the original abstract

The sequential acquisition of languages inevitably leads to Crosslinguistic Influence (CLI), where the syntactic properties of a first language (L1) impact the processing of a second language (L2). While modern language models exhibit robust cross-lingual transfer, the exact mechanisms governing how language dominance, relative proficiency, and typological distance dictate structural interference warrant deeper investigation. In this work, we systematically investigate CLI in artificial learners by training simultaneous and sequential bilingual models across 15 typologically diverse L1s and varying the Step of Exposure (SoE), defined as the specific training step at which the L2 is introduced. Utilizing crosslinguistic structural priming, we decouple latent CLI into distinct positive and negative transfer rates. Our evaluations reveal a critical computational tradeoff: while increased L1 dominance (higher SoE) strongly amplifies the correlation between structural transfer and syntactic distance, diminished L2 proficiency bottlenecks the model's capacity for positive transfer, leaving it highly vulnerable to persistent negative interference from distant L1s. Mechanistically, we demonstrate that L1 typological proximity physically dictates the cross-lingual overlap of L2 syntactic neurons. Furthermore, we uncover that explicit priming induces a dynamic layer-wise migration of grammatical resolution to the terminal layers of the network. Through targeted causal ablations, we confirm that deep-layer attention mechanisms exclusively drive this crosslinguistic transfer by routing the L1 structural prior into the final prediction. Ultimately, our findings demonstrate that CLI in language models is not an arbitrary artifact of capacity constraints, but a structured phenomenon fundamentally governed by the interplay of language dominance and proficiency.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references

[2011]

Aafke Hulk and Natascha Müller

Dual language exposure and early bilin- gual development*.Journal of Child Lan- guage, 39:1 – 27. Aafke Hulk and Natascha Müller. 2000. Bilin- gual first language acquisition at the interface between syntax and pragmatics.Bilingualism: Language and Cognition, 3(3):227–244. Catharine Keatley, John Spinks, and Beatrice Gelder. 1994. Asymmetrical cross-langu...

2000
[2014]

Taku Kudo and John Richardson

Shared abstract representation of linguis- tic structure in bilingual sentence comprehen- sion.Psychonomic bulletin & review, 22. Taku Kudo and John Richardson. 2018. Senten- cePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. InProceedings of the 2018 Conference on Empirical Methods in Natural Language...

2018
[2016]

Robert J

Cross-linguistic structural priming in multilinguals: Further evidence for shared syn- tax.Journal of Memory and Language, 90:14– 30. Robert J. Hartsuiker and Sarah Bertnolet. 2017. The development of shared syntax in second language learning.Bilingualism: Language and Cognition, 20(2):219–234. Robert J. Hartsuiker, Martin J. Pickering, and Eline Veltkamp...

2017
[2018]

InProceedings of ACL 2018, Sys- tem Demonstrations, pages 13–18, Melbourne, Australia

Out-of-the-box universal Romanization tool uroman. InProceedings of ACL 2018, Sys- tem Demonstrations, pages 13–18, Melbourne, Australia. Association for Computational Lin- guistics. Erika Hoff, Cynthia Core, Silvia Place, Rosario Rumiche, Melissa Señor, and Marisol Parra

2018
[2021]

Danbi Ahn and Victor S Ferreira

Limerick, 12 July 2021 (Online-Event), pages 1 – 9. Danbi Ahn and Victor S Ferreira. 2024. Shared vs separate structural representations: Evi- dence from cumulative cross-language struc- tural priming.Quarterly Journal of Exper- imental Psychology, 77(1):174–190. PMID: 36960936. Belen Alastruey, João Maria Janeiro, Alexandre Allauzen, Maha Elbayad, Loïc B...

2021

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2011]

Aafke Hulk and Natascha Müller

Dual language exposure and early bilin- gual development*.Journal of Child Lan- guage, 39:1 – 27. Aafke Hulk and Natascha Müller. 2000. Bilin- gual first language acquisition at the interface between syntax and pragmatics.Bilingualism: Language and Cognition, 3(3):227–244. Catharine Keatley, John Spinks, and Beatrice Gelder. 1994. Asymmetrical cross-langu...

2000

[2] [2014]

Taku Kudo and John Richardson

Shared abstract representation of linguis- tic structure in bilingual sentence comprehen- sion.Psychonomic bulletin & review, 22. Taku Kudo and John Richardson. 2018. Senten- cePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. InProceedings of the 2018 Conference on Empirical Methods in Natural Language...

2018

[3] [2016]

Robert J

Cross-linguistic structural priming in multilinguals: Further evidence for shared syn- tax.Journal of Memory and Language, 90:14– 30. Robert J. Hartsuiker and Sarah Bertnolet. 2017. The development of shared syntax in second language learning.Bilingualism: Language and Cognition, 20(2):219–234. Robert J. Hartsuiker, Martin J. Pickering, and Eline Veltkamp...

2017

[4] [2018]

InProceedings of ACL 2018, Sys- tem Demonstrations, pages 13–18, Melbourne, Australia

Out-of-the-box universal Romanization tool uroman. InProceedings of ACL 2018, Sys- tem Demonstrations, pages 13–18, Melbourne, Australia. Association for Computational Lin- guistics. Erika Hoff, Cynthia Core, Silvia Place, Rosario Rumiche, Melissa Señor, and Marisol Parra

2018

[5] [2021]

Danbi Ahn and Victor S Ferreira

Limerick, 12 July 2021 (Online-Event), pages 1 – 9. Danbi Ahn and Victor S Ferreira. 2024. Shared vs separate structural representations: Evi- dence from cumulative cross-language struc- tural priming.Quarterly Journal of Exper- imental Psychology, 77(1):174–190. PMID: 36960936. Belen Alastruey, João Maria Janeiro, Alexandre Allauzen, Maha Elbayad, Loïc B...

2021