Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Probing shows MLLMs build language-free concept neurons

desk verdict Solid evidence for growing cross-lingual neuron alignment during pre-training; the steering experiment oversells the causal claim, but the correlational core holds up. read the letter →

arxiv 2506.01629 v1 pith:5IIKSOTG submitted 2025-06-02 cs.CL

Frederick Riemenschneider , Anette Frank This is my paper
classification cs.CL
keywords cross-lingualgeneralizationcompressionhypothesismultilinguallanguagemodelsmechanisticinterpretabilityconceptneuronspre-trainingdynamicsalignmentneuronsteering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that multilingual language models develop a language-independent semantic layer during pre-training, driven by compression pressure that forces the model to reuse the same neurons for the same concepts across languages. The authors track individual concept expert neurons over training checkpoints in BLOOM models and a custom model, showing that neurons initially tied to one language gradually align across languages. The decisive behavioral evidence is that activating concept neurons identified from Spanish text makes the model generate coherent English text about that concept, showing the neurons encode the concept, not the language. If right, this reframes cross-lingual transfer as a byproduct of compression rather than of shared subwords or linguistic similarity.

What carries the argument

The central object is the concept expert neuron: an MLP neuron whose activation pattern, scored by average precision against positive and negative sentence sets for a WordNet sense, reliably predicts the presence of a concept. The paper defines per-language expert score vectors over all neurons, then measures cross-lingual alignment via Pearson correlation (averaged with Fisher's Z transform), mutual information (via k-nearest-neighbor entropy estimation), and direct overlap of top-500 neuron sets. The layer-wise version of the overlap measure separates lexical (early-layer) from semantic (middle-layer) sharing, and the generation steering protocol, setting top expert neurons to their median concept activation and sampling with no prompt tokens, is the causal test that carries the argument.

What would settle it

Take two sets of human-written, non-parallel sentences about the same 200 WordNet concepts in, say, Spanish and English, measure top-500 concept-neuron overlap in a late BLOOM checkpoint, and compare it to the overlap measured on the NLLB-translated parallel corpus used in the paper; if the overlap drops to chance or near-zero on the independent texts, the compression-to-shared-neurons conclusion would be an artifact of translated input. A second check: rerun the steering experiment with the top 500 expert neurons replaced by 500 random neurons at the same layers, and if random-neuron activation also shifts generated language toward English, the effect is not concept-specific.

Watch

Extended reading notes

Core claim

The central claim is that cross-lingual generalization in decoder-only multilingual language models is realized by concrete shared neurons: individual MLP neurons that respond to the same semantic concept across languages, emerging and strengthening during pre-training. The authors show that by late training, roughly one sixth of the top 500 concept-selective neurons are shared between any pair of languages, that this overlap peaks in middle layers (10-17) where language-identification ability is weakest, and that the trend replicates from a 257M-parameter model to BLOOM-7B1. The most direct evidence is causal: setting the top 500 concept expert neurons (identified from Spanish or Simplified Chinese text) to their concept-median activations, with no language cue given, biases the model to generate semantically coherent text, and in later checkpoints that text is overwhelmingly English. The authors conclude that aligned concept neurons encode generalized semantics independent of the surface language, and that their emergence is consistent with the information bottleneck and compression hypothesis.

Load-bearing premise

The entire alignment measurement rests on parallel data: the English concept sentences are machine-translated into every target language with NLLB, so the growing top-neuron overlap across languages could in principle reflect translation artifacts like shared subwords, cognates, or translationese rather than a genuinely shared semantic code.

Editorial extensions

If this is right

  • If shared concept neurons are a general feature of multilingual models, zero-shot cross-lingual transfer should become predictable from neuron overlap measured before fine-tuning, rather than from downstream task scores.
  • The layer-resolved overlap curve suggests that interventions targeting middle layers should be the route for language-agnostic concept control in generation.
  • The same measurement pipeline can be applied to other model families with released checkpoints to test whether the compression-to-sharing trajectory is universal or architecture-dependent.
  • Concept steering becomes a practical technique: a concept learned from one language can be used to elicit that concept from the model while it generates in another language.
  • The observed English and Chinese dominance in steering outputs implies that shared representations are not neutral; the compressed layer may be anchored to the most frequent pre-training languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the compression account is causal rather than correlational, deliberately increasing model capacity should delay or reduce cross-lingual neuron sharing, a testable prediction the paper does not run.
  • The reliance on NLLB-translated parallel data leaves open that translationese, calques, or cognate retention inflate the measured overlap; an orthogonal check would measure overlap on independently written same-concept sentences in each language.
  • The English-output steering result may partly reflect the model's output prior favoring high-resource languages, since Swahili-derived neurons never yield Swahili text; a cleaner demonstration would control for output-language priors.
  • Extending the concept inventory from concrete nouns to verbs, adjectives, or relational concepts could reveal whether shared encoding is a general property of lexical semantics or specific to the sampled word senses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates how multilingual language models (BLOOM-560M, BLOOM-7B1, and a custom XGLM-based model) evolve from language-specific to cross-lingual representations during pre-training. Using linear probing, the authors show that language identity becomes less decodable in middle layers over training. They then compute concept-specific 'expert neuron' scores (average precision) for 200 WordNet concepts across languages, using NLLB-translated parallel sentences, and measure cross-lingual alignment via correlation, mutual information, and top-500 neuron overlap. All three measures increase during training, and the effect is replicated across model scales. A layer-wise analysis localizes the strongest cross-lingual overlap in middle layers. Finally, a neuron-steering experiment manipulates concept neurons derived from Spanish/Chinese data and shows a shift from source-language to English generation at later checkpoints, which the authors interpret as evidence that aligned neurons encode language-independent semantic content.

Significance. If the central claim holds, this would be a valuable mechanistic account of cross-lingual generalization: shared concept neurons forming a language-independent semantic layer during pre-training, with practical implications for controllable generation and for understanding high-resource language bias. The paper's strengths are its longitudinal design, the use of three independent alignment measures, replication across BLOOM-560M, BLOOM-7B1 and a custom model, and the public release of checkpoints and code. The layer-wise analysis (Section 7) is a thoughtful attempt to separate lexical overlap from semantic sharing. However, the headline claim rests on a generation experiment that currently lacks the control conditions needed to attribute the observed language shift to the manipulated concept neurons, and the alignment measures are computed on machine-translated parallel data, leaving a translation-induced artifact as a plausible alternative explanation. These issues are addressable, but they are load-bearing for the strongest interpretation.

major comments (3)
  1. [Section 8 / Appendix E] The steering experiment described in Section 8 and Appendix E has no no-manipulation baseline and no random-neuron control. The protocol sets the top-500 concept neurons to concept-specific median values, prompts only with a BOS token, and measures the language of generated text. The shift from Spanish at step 10k to English at step 400k could simply reflect the model's default output distribution from an unconstrained prompt, especially given English's dominance in BLOOM's training corpus; the paper itself invokes corpus prominence to explain Chinese generations but does not apply the same reasoning to English. Without a baseline (e.g., generation from BOS with no neuron manipulation) and a control with the same number of random neurons set to their medians, the language shift cannot be attributed to the semantic content of the selected neurons. Additionally, 'concept-relevant' is supported only by qualitative examples in Table 1; a quantitative measure (classifier, human rating, or concept-specific metric) is needed to show that the manipulated neurons specifically evoke the target concept rather than generic fluent text. This is load-bearing for the abstract's claim that aligned neurons 'encode generalized semantic representations, independent of the original language encoding.'
  2. [Section 6, Data] The expert neuron scores are computed on the same English sentences machine-translated by NLLB into each target language, creating parallel corpora with identical semantic content. Cross-lingual alignment of AP vectors could therefore be inflated by translationese, shared subwords, cognates, or the trivial preservation of content across languages, rather than by the model's formation of language-independent abstractions. The layer analysis in Section 7 partially separates subword overlap from semantic overlap by showing that alignment is strongest in middle layers, but it does not control for the fact that the underlying text is semantically identical by construction. To support the compression conclusion, the authors should add a control: e.g., compute alignment on non-parallel monolingual corpora for the same concepts, randomly permute concept labels within one language before computing cross-lingual correlation, or compare the observed overlap against a shuffled-neuron null distribution. Without such a control, the increasing alignment is consistent with the model simply reflecting the parallel data's shared content.
  3. [Section 6, Neuron Overlap (k-sensitivity)] The top-k overlap measure is reported only for k=500, and the conclusion that 'approx. 1/6 of the top 500 concept-selective neurons are shared' is given without sensitivity analysis. Since the degree of overlap trivially decreases with k, and the choice of k is a free parameter, reporting at least one other k (e.g., k=100 and k=1000) would establish that the temporal trend is not an artifact of the specific threshold.
minor comments (5)
  1. [Section 6, Figure 4] The three panels of Figure 4 share the x-axis 'Step' but the y-axis labels and units are not visible in the text; please label them explicitly (e.g., Pearson r after Fisher Z, overlap proportion, MI in nats).
  2. [Section 6, Neuron Overlap] The phrase 'approx. 1/6 of the top 500 concept-selective neurons are shared between any pair of languages' is ambiguous; it should state that the mean pairwise overlap proportion is approximately 1/6, i.e., about 83 of 500 neurons on average.
  3. [Limitations] The Limitations section does not mention the absence of control conditions in the generation experiment or the reliance on translated parallel data; these are the two main threats to the paper's central claim and should be acknowledged.
  4. [Section 2, Related Work] The claim of being 'the first to analyze the development of cross-lingual semantic generalization during pre-training' (contribution ii) should be tempered in light of Blevins et al. (2022), who analyze cross-lingual pretraining dynamics even if with different tasks; suggest rephrasing to 'the first to our knowledge at the level of individual concept neurons in decoder-only models.'
  5. [Section 6, Expert score definition] The notation 'zc,l m = {zc,l m,i}N i=1' is hard to parse; define zc,l m,i as the activation of neuron m on the i-th sentence of concept c in language l.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical measures are direct activation statistics and the compression framing is an external hypothesis, not a fitted consequence.

full rationale

The paper does not fit any parameter to the phenomena it claims to predict, and no equation reduces to its inputs by construction. The cross-lingual alignment evidence (Section 6) is computed directly from neuron activation statistics (AP vectors, Pearson correlations, mutual information, top-k overlap) on parallel translated concept datasets; these quantities are not defined in terms of the paper's conclusions. The layer analysis in Section 7 explicitly controls for subword overlap, and the generation experiment in Section 8 is a behavioral probe that is not statistically forced by the Section 6 scores. The compression hypothesis is imported from external prior work (Tishby et al.; Shwartz-Ziv and Tishby) as a framing hypothesis, not derived from the paper's own assumptions, and no load-bearing self-citation or uniqueness theorem is invoked. The main weaknesses—parallel-data dependence of the alignment measures and the lack of no-manipulation/random-neuron baselines in the steering experiment—are threats to the soundness or interpretation of the evidence, not circularity, since the observed outputs are not equivalent to the inputs by construction. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The analysis depends on interpretability assumptions (probing and neuron-as-expert) and on the compression hypothesis borrowed from information bottleneck literature. No physical constants or fitted parameters are used; the only hand-chosen quantity affecting reported numbers is the top-k neuron count. The parallel-translation data construction and the language-prior interpretation are load-bearing domain assumptions.

free parameters (1)
  • top-k neuron count k = 500
    The overlap proportion O_{l1,l2}=|S_{c,l1} ∩ S_{c,l2}|/k and the generation manipulation use k=500 expert neurons; the reported ~1/6 overlap figure depends on this choice, and no sensitivity analysis across k is provided.
assumptions (5)
  • domain assumption Linear probe accuracy reflects how much language-specific information a layer encodes.
    Section 5 interprets logistic-regression accuracy on sampled hidden states as evidence about language information content; linear probes measure accessible, linearly separable information, not the full information content.
  • domain assumption AP-based expert neuron scores on parallel translated sentences isolate language-independent concept encoding.
    Section 6 uses ONESEC English sentences translated by NLLB into all languages; if translation artifacts or cognates drive the scores, alignment measures would not reflect abstraction. The layer analysis addresses subword overlap only indirectly.
  • domain assumption The shift to English output in the steering experiment is caused by shared semantic neurons, not by the model's language prior.
    Section 8 interprets late-checkpoint English generation as evidence of language-independent concept representations; the authors record a strong English/Chinese generation bias, so this causal reading is not established without control manipulations.
  • domain assumption Individual MLP neurons can serve as concept experts as defined by Suau et al.
    Section 6 adopts the neuron-as-expert methodology; neurons are likely polysemantic, and max-pooled activations may reflect distributed codes, so the top-k set is an approximation.
  • domain assumption The small custom model trained on uniformly sampled MC4 is a faithful proxy for large-scale multilingual pre-training.
    Appendix A: BLOOM has only sparse checkpoints, so the authors train a 257M XGLM variant on 16 languages to obtain dense training trajectories; this setup differs in scale and data balance from BLOOM.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons." pith.science (2026). https://pith.science/paper/5IIKSOTG

@misc{pith2026250601629,
  author       = {Pith},
  title        = {Pith review of: Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5IIKSOTG}},
  note         = {Machine review of arXiv:2506.01629}
}
read the original abstract

Multilingual language models (MLLMs) have demonstrated remarkable abilities to transfer knowledge across languages, despite being trained without explicit cross-lingual supervision. We analyze the parameter spaces of three MLLMs to study how their representations evolve during pre-training, observing patterns consistent with compression: models initially form language-specific representations, which gradually converge into cross-lingual abstractions as training progresses. Through probing experiments, we observe a clear transition from uniform language identification capabilities across layers to more specialized layer functions. For deeper analysis, we focus on neurons that encode distinct semantic concepts. By tracing their development during pre-training, we show how they gradually align across languages. Notably, we identify specific neurons that emerge as increasingly reliable predictors for the same concepts across languages.

Figures

Figures reproduced from arXiv: 2506.01629 by the authors.

Figure 1
Figure 1. Language identity probing classification accuracy across layers of the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Language identification probing accuracy [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Expert neuron alignment across languages in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: (1) Correlation Coefficient, (2) Neuron Over [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Layer-wise distribution of BLOOM-560M’s top 500 expert neurons, averaged across languages and concepts. for different layers. For each layer ℓ, we com￾pute the average pairwise overlap between top k expert neurons by comparing the sets S c,l ℓ across languages, measuri…
Figure 6
Figure 6. Figure 6: Cross-lingual overlap of BLOOM-560M’s top 500 expert neurons per layer, showing the averaged proportion of shared neurons between language pairs. concept (e.g., earthquake), we identify its top 500 expert neurons using data from one language (e.g., Spanish) and manipul…
Figure 7
Figure 7. Figure 7: Relative frequency distribution of the top 10 detected languages when manipulating neurons derived from [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Language identification probing accuracy [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Language identification probing accuracy [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Language identity probing classification accuracy across layers of the [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Layer-wise language identity probing on XLM-R BASE. Higher accuracy indicates that language￾specific information is more easily extractable from the hidden states at that layer. Error bars show standard deviation across three random seeds [PITH_FULL_IMAGE:figures/ful…
Figure 12
Figure 12. Figure 12: Layer-wise language identity probing on MBERT BASE CASED. Higher accuracy indicates that language-specific information is more easily extractable from the hidden states at that layer. Error bars show standard deviation across three random seeds [PITH_FULL_IMAGE:figur…
Figure 13
Figure 13. Figure 13: Expert neuron alignment of BLOOM-560M at different training stages [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Expert neuron alignment of our toy model at different training stages. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Correlation Coefficient and Neuron Overlap [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Layer-wise distribution of BLOOM-7B1’s top 500 expert neurons, averaged across languages and concepts. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Layer 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Expert Overlap Proportion Step 0 Step 1000 …
Figure 17
Figure 17. Figure 17: Cross-lingual overlap of BLOOM-7B1’s top 500 expert neurons per layer, showing the averaged proportion of shared neurons between language pairs [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Text generation experiments: Relative frequency distribution of the top 10 detected languages when [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Text generation experiments: Relative frequency distribution of the top 10 detected languages when [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Text generation experiments: Relative frequency distribution of the top 10 detected languages when [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages

  1. [1]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. https://doi.org/10.18653/v1/2020.acl-main.421 On the cross-lingual transferability of monolingual representations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.421 2020

  2. [2]

    BigScience Workshop . 2022. https://doi.org/10.57967/hf/0003 BLOOM (revision 4ab0472)

    doi:10.57967/hf/0003 2022

  3. [3]

    Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.234 Analyzing the mono- and cross-lingual pretraining dynamics of multilingual language models . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3575--3590, Abu Dhabi, United Arab Emirates. Association for Com...

    doi:10.18653/v1/2022.emnlp-main.234 2022

  4. [4]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  5. [5]

    Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. https://doi.org/10.18653/v1/2021.naacl-main.280 I nfo XLM : An information-theoretic framework for cross-lingual language model pre-training . In Proceedings of the 2021 Conference of the North American Chapter of the Associatio...

    doi:10.18653/v1/2021.naacl-main.280 2021

  6. [6]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2020.acl-main.747 2020

  7. [7]

    Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672

    arXiv 2022

  8. [8]

    Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2022. https://doi.org/10.18653/v1/2022.naacl-main.264 When is BERT multilingual? isolating crucial ingredients for cross-lingual transfer . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3610--...

    doi:10.18653/v1/2022.naacl-main.264 2022

Show all 40 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  2. [10]

    Philipp Dufter and Hinrich Sch \"u tze. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.358 Identifying elements essential for BERT ' s multilinguality . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423--4437, Onli...

    2020 doi

  3. [11]

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. https://openreview.net/forum?id=JYs1R9IMJr Finding neurons in a haystack: Case studies with sparse probing . Transactions on Machine Learning Research

    2023

  4. [12]

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In Proceedings of the 37th International Conference on Machine Learning, ICML'2...

    2020

  5. [13]

    Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. https://openreview.net/forum?id=HJeT3yrtDr Cross-lingual ability of multilingual bert: An empirical study . In International Conference on Learning Representations

    2020

  6. [14]

    Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. https://doi.org/10.18653/v1/2024.naacl-long.384 On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons . In Proceedings ...

    2024 doi

  7. [15]

    Alexander Kraskov, Harald St\"ogbauer, and Peter Grassberger. 2004. https://doi.org/10.1103/PhysRevE.69.066138 Estimating mutual information . Phys. Rev. E, 69:066138

    2004 doi

  8. [16]

    Anne Lauscher, Vinit Ravishankar, Ivan Vuli \'c , and Goran Glava s . 2020. https://doi.org/10.18653/v1/2020.emnlp-main.363 From zero to hero: O n the limitations of zero-shot language transfer with multilingual T ransformers . In Proceedings of the 2020 Conference on Empirica...

    2020 doi

  9. [17]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022 doi

  10. [18]

    Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. https://doi.org/10.18653/v1/P19-1301 Choosing transfer languages for cross-lin...

    2019 doi

  11. [19]

    George A. Miller. 1994. https://aclanthology.org/H94-1111 W ord N et: A lexical database for E nglish . In H uman L anguage T echnology: Proceedings of a Workshop held at P lainsboro, N ew J ersey, M arch 8-11, 1994

    1994

  12. [20]

    Pedro Javier Ortiz Suárez , Benoit Sagot, and Laurent Romary. 2019. https://doi.org/10.14618/ids-pub-9021 Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures . Proceedings of the Workshop on Challenges in the Management of Large Corpora...

    2019 doi

  13. [21]

    Isabel Papadimitriou, Kezia Lopez, and Dan Jurafsky. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.89 Multilingual BERT has an accent: Evaluating E nglish influences on fluency in multilingual models . In Findings of the Association for Computational Linguistics: EACL 2...

    2023 doi

  14. [22]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...

    2011

  15. [23]

    Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. https://doi.org/10.18653/v1/2023.acl-long.323 Towards a common understanding of contributing factors for cross-lingual transfer in multilingual language models: A review . In Proceedings of the 61st Annual Meeting of the As...

    2023 doi

  16. [24]

    Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. https://doi.org/10.18653/v1/P19-1493 How multilingual is multilingual BERT ? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996--5001, Florence, Italy. Association for Compu...

    2019 doi

  17. [25]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  18. [26]

    Sara Rajaee and Christof Monz. 2024. https://aclanthology.org/2024.eacl-long.177 Analyzing the evaluation of cross-lingual knowledge transfer in multilingual language models . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Li...

    2024

  19. [27]

    Brian C. Ross. 2014. https://doi.org/10.1371/journal.pone.0087357 Mutual information between discrete and continuous data sets . PLOS ONE, 9(2):1--5

    2014 doi

  20. [28]

    Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2019. https://doi.org/10.18653/v1/P19-1069 Just `` O ne S e C '' for producing multilingual sense-annotated data . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 699--709, ...

    2019 doi

  21. [29]

    Nakatani Shuyo. 2010. http://code.google.com/p/language-detection/ Language detection library for java

    2010

  22. [30]

    Ravid Shwartz-Ziv and Naftali Tishby. 2017. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810

    2017 arXiv

  23. [31]

    Xavier Suau, Luca Zappella, and Nicholas Apostoloff. 2022. Self-conditioning pre-trained language models. International Conference on Machine Learning

    2022

  24. [32]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. https://doi.org/10.18653/v1/2024.acl-long.309 Language-specific neurons: The key to multilingual capabilities in large language models . In Proceedings of the 62nd...

    2024 doi

  25. [33]

    Pereira, and William Bialek

    Naftali Tishby, Fernando C. Pereira, and William Bialek. 1999. https://arxiv.org/abs/physics/0004057 The information bottleneck method . In Proceedings of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368--377

    1999 arXiv

  26. [34]

    Naftali Tishby and Noga Zaslavsky. 2015. https://doi.org/10.1109/ITW.2015.7133169 Deep learning and the information bottleneck principle . In 2015 IEEE Information Theory Workshop (ITW), pages 1--5

    2015

  27. [35]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    O'Reilly Media, Inc

    Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. 2022. Natural language processing with transformers. "O'Reilly Media, Inc."

    2022

  29. [37]

    Elena Voita, Rico Sennrich, and Ivan Titov. 2019. https://doi.org/10.18653/v1/D19-1448 The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives . In Proceedings of the 2019 Conference on Empirical Methods ...

    2019 doi

  30. [38]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. https://doi.org/10.18653/v1/2024.acl-long.820 Do llamas work in E nglish? on the latent language of multilingual transformers . In Proceedings of the 62nd Annual Meeting of the Association for Computati...

    2024 doi

  31. [39]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  32. [40]

    Shijie Wu and Mark Dredze. 2019. https://doi.org/10.18653/v1/D19-1077 Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o...

    2019 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.