Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Investigating Multilingual NMT Representations at Scale

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Using SVCCA on a 103-language translation model, this paper finds that encoder representations cluster by linguistic family, shift with target language, and predict fine-tuning distortion.

desk verdict The paper's three headline claims all rest on an unvalidated mean-pooling heuristic, and the appendix shows the key divergence disappears under an alternative pooling—so treat the conclusions as promising but not established. read the letter →

arxiv 1909.02197 v2 pith:WATTINVZ submitted 2019-09-05 cs.CL cs.LG

Sneha Reddy Kudugunta , Ankur Bapna , Isaac Caswell , Naveen Arivazhagan , Orhan Firat This is my paper · ORCID
classification cs.CLcs.LG
keywords multilingualneuralmachinetranslationSVCCArepresentationsimilaritycross-lingualtransferlinguisticfine-tuningrobustnesszero-shotlanguageclustering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make the internal geometry of a massively multilingual translation model legible. It argues that a 103-language encoder organizes languages by linguistic family and branch, that a language's representation is shaped by what it is being translated into, and that these two facts predict how much a language's representation shifts when the model is fine-tuned on an arbitrary other language pair. If the claims hold, they give practitioners a principled way to choose which language pairs to fine-tune on for zero- or few-shot transfer, and explain why linguistically distant languages transfer poorly.

What carries the argument

SVCCA (Singular Value Canonical Correlation Analysis) is the measuring instrument. It reduces each layer's activation matrix to the subspace retaining 99% of variance via SVD, then applies CCA to find linear transformations that maximize correlations between two such subspaces; the mean canonical correlation $\bar{\rho}$ is the reported similarity score. Because the compared sentences are unaligned across languages, the paper mean-pools each sentence's token activations over time steps before computing SVCCA, assuming that every token is equally likely to align with every token of its equivalent sentence. This pooled SVCCA score is what produces the language clusters, the layer-wise similarity curves, and the fine-tuning distortion measurements.

What would settle it

Re-run the same SVCCA comparisons on the same 103-language model using token-level alignment derived from the multi-way parallel evaluation set, for example by using word alignments or an alignment-agnostic CCA objective. If the top-encoder family clusters and the high-resource versus low-resource fine-tuning ordering do not reappear, the mean-pooling assumption, not the model, produced the reported structure.

Watch

Extended reading notes

Core claim

The central claim is that a massively multilingual NMT encoder organizes languages by linguistic similarity, and that this organization predicts fine-tuning behavior. Concretely, the paper reports three findings. First, languages from the same family, branch, and even dialect group cluster in SVCCA similarity space, with clusters sharpening from embeddings to the top encoder layer; Serbian and Croatian, written in different scripts, become each other's nearest neighbors, showing the clustering is distributional rather than purely lexical. Second, the representation of a source language depends on the target language: in X-to-English, source-language representations grow more alike up the encoder, while English decoder representations diverge by source language, and the reverse happens in English-to-X, so the source-target boundary across encoder and decoder is gradual. Third, after fine-tuning on one language pair, high-resource languages and languages related to the fine-tuning pair keep their representations, low-resource unrelated languages distort most, and the amount of representation shift tracks the amount of BLEU degradation.

Load-bearing premise

The analysis assumes that mean-pooling a sentence's activations over its tokens gives a fair way to compare unaligned sentences across languages; if that pooling discards the structural information that actually drives representational similarity, the reported clusters and fine-tuning ordering could be artifacts of the averaging heuristic.

Editorial extensions

If this is right

  • Fine-tuning on a high-resource language like Spanish or Russian distorts few other representations, so those language pairs can serve as relatively safe anchors for adapting a multilingual model without destroying existing languages.
  • For a low-resource language, fine-tuning on a linguistically near neighbor distorts the target less than fine-tuning on a distant language, providing a measurable criterion for choosing adaptation languages.
  • Because representation shift correlates with BLEU degradation, SVCCA similarity computed before and after fine-tuning could serve as a cheap proxy for translation-quality loss, even for language pairs without test sets.
  • Input embeddings overlap less than top-encoder representations, so methods that rely on shared multilingual embeddings alone are likely weaker for cross-lingual transfer than methods that use or align encoder outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the top-encoder SVCCA nearest-neighbor list could serve directly as a training-language selector for few-shot adaptation: pick the fine-tuning language whose representation is already closest to the target language. The paper demonstrates that such neighbor lists are linguistically coherent but never runs this decision rule itself.
  • The Serbian-Croatian result suggests that when the training objective rewards semantic equivalence, script difference is learnable noise. A testable extension would transliterate all languages into a single script before training and check whether the family clusters compress further or whether the script signal still visible in the embeddings disappears.
  • Sinhala's shift from the Indo-Aryan cluster in the embeddings to the Dravidian cluster at the encoder top hints that areal contact or topic overlap can override genealogical family in deeper layers; a direct test would compare Sinhala's neighbors under a topic-controlled evaluation set versus the current multi-way set.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript studies a 103-language multilingual NMT model (Transformer-Big, 375M parameters) using SVCCA to compare layer representations across languages, using a multi-way parallel evaluation set. It proposes mean-pooling over sequence time-steps to handle unaligned sentences, and draws three conclusions: (i) encoder representations cluster by linguistic similarity, (ii) encoder source representations depend on the target language and decoder target representations on the source, and (iii) high-resource and linguistically similar languages are more robust to fine-tuning on arbitrary language pairs. The paper supports these with spectral-embedding visualizations, pairwise SVCCA distributions across layers, nearest-neighbor lists, and fine-tuning experiments with BLEU comparisons. Supplemental material includes token-level CCA comparisons, additional family-specific clusterings, and per-layer fine-tuning sensitivity.

Significance. If the three claims survive scrutiny, this would be an important step toward understanding why multilingual NMT transfers, with practical implications for language selection in zero/few-shot adaptation and for architecture choices such as consistency losses. The study's scale—103 languages, a shared 64k SentencePiece vocabulary, and a multi-way aligned 3k-sentence evaluation set—is unusual and valuable, and the attempt to connect representational change to BLEU changes is commendable. However, the analysis is observational and depends on a similarity metric whose central assumption is explicitly acknowledged as a heuristic and is shown in the paper itself to alter a headline result under an alternative aggregation. The lack of quantitative validation and statistical tests prevents the conclusions from being accepted as established.

major comments (4)
  1. [§2.2, Appendix A.3] The mean-pooling heuristic in §2.2 is load-bearing but unvalidated. The text acknowledges that it assumes a given token in language A is equally likely to be aligned to each token in the equivalent language B sentence, but no evidence is offered that this preserves the representational information relevant to cross-language similarity. More seriously, Appendix A.3 shows that a token-level strategy reproduces the En-X divergence results only up to the top encoder layer; at the top layer the divergence disappears and is dismissed as 'possibly an artifact.' Since the same pooled SVCCA scores underlie the clustering analysis of §3 and the fine-tuning analysis of §4, all three headline conclusions are conditional on this pooling choice. The authors should either validate the pooling against a ground-truth alignment or a semantic equivalence criterion, or demonstrate robustness to the aggregation method and justify why the token-level top-layer result is the artifact rather than the mean-pooled result.
  2. [§3, Figures 2–4] The cluster claims are based on visual inspection of spectral embedding plots. Statements such as 'we can observe some overlapping clusters' are subjective; no cluster validity indices, adjusted Rand index against language family labels, or statistical significance tests are reported. The paper acknowledges confounding factors including script, family, and web-crawl domain, but the strength of the claims (that representations 'cluster based on linguistic similarity') needs quantitative support. I recommend adding quantitative cluster metrics and controlling for resource size and script when evaluating the contribution of linguistic family.
  3. [§4, Figures 6a and 6b] The fine-tuning analysis claims that variations in SVCCA scores 'correspond very well' with changes in BLEU, but no correlation coefficient or confidence interval is given. The observations about high-resource robustness and family-specific robustness rest on visual patterns in heatmaps over 12 fine-tuning languages, and no seed or initialization variability is reported. Because these observations directly support claim (iii) and the zero/few-shot implications in §5, the paper should include quantitative correlations, uncertainty estimates, and a sensitivity analysis for the resource-size threshold in Table 2 (the gap between 10^7 and 10^8 sentences has no empirical support).
  4. [§3.2, Appendix A.5] The sentence that Serbian and Croatian 'cluster purely based on distributional similarity' overstates what can be concluded. The absence of subword overlap controls for lexical/script overlap, but other confounding factors (corpus domain, topic distribution, data size, target-side text effects) are not controlled; the manuscript itself later notes that further analysis is required for Sinhala in A.5. The claim should be softened or supported with controlled experiments that isolate distributional similarity from other correlates.
minor comments (5)
  1. [§5] There is duplicated wording: 'it might not be might not be effective' should be corrected.
  2. [Appendix A.1] The phrase 'we followed a learning rate of a learning rate of 3.0' contains a duplication and should be reworded.
  3. [§4] 'Ukranian' should be 'Ukrainian'.
  4. [Appendix A.3] The sentence 'Our data is unaligned for compared other components of our experiment, so we do not discuss those results' is grammatically incomplete and obscures why the token-level clustering is not compared more fully with the mean-pooling clustering.
  5. [Footnote 1] The paper says tools for online visualization and representation similarity will be open-sourced, but no repository or URL is provided; adding a link would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are observational measurements from a trained model, not quantities derived from their own inputs.

full rationale

The paper's three headline conclusions are empirical observations about representations of a trained multilingual NMT model, not quantities fitted from the quantities they are claimed to explain. In Section 2.2, SVCCA similarity is defined on mean-pooled activations; Sections 3 and 4 then measure pairwise SVCCA scores across languages and across fine-tuning runs, and compare those measurements to external linguistic-family labels and corpus resource sizes. The language-similarity clusters are not used to define the SVCCA similarity measure, and the resource-size/robustness relationship is reported as an observed trend, not as a predicted value obtained by fitting a parameter to the data. The self-citations (Arivazhagan et al., 2019b; Chen et al., 2018) supply the model architecture, corpus, and training recipe; they do not supply the representational-similarity conclusions, and no uniqueness theorem or ansatz is imported from those works to force the paper's choices. Appendix A.3 honestly reports that an alternate token-level CCA changes the top-layer divergence result, which is a validity concern about the pooling heuristic, not a circularity: both pooling choices are inputs to the analysis, and the conclusions do not reduce to those choices by construction. The paper also grounds its model quality against bilingual BLEU baselines in Appendix A.2, so the studied model is independently characterized. No fitted parameter is renamed as a prediction, and no central claim is definitionally equivalent to its inputs. Accordingly, the circularity burden is not met; this is a normal non-finding.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on three assumptions: the validity of mean-pooling for SVCCA comparison of sentences, the semantic equivalence of the multi-way parallel evaluation set, and the use of linguistic family and branch membership as ground truth. No novel entities are introduced. Methodological cutoffs (SVCCA variance retention 99%, fine-tuning for 40k steps, and the set of 12 fine-tuning languages) are user choices that shape the robustness results.

free parameters (3)
  • SVCCA variance retention = 0.99
    Fraction of variance retained when computing SVD subspaces in SVCCA; chosen following Raghu et al. (2017) and used for all comparisons.
  • Fine-tuning steps = 40k
    All 12 fine-tuning runs use 40k steps; this duration affects how much representation change and BLEU change are observed.
  • Fine-tuning language set = 12 language pairs
    Hand-picked mix of high and low resource languages from 6 subfamilies; the robustness conclusions depend on this selection.
assumptions (3)
  • ad hoc to paper Mean-pooling over sequence time-steps is a valid alignment for SVCCA across languages.
    Introduced in Section 2.2 (SVCCA for Sequences). The authors note token-level alignment is infeasible; Appendix A.3 shows token-level SVCCA gives possibly artifactual results at the top layer, so the pooling choice is load-bearing.
  • domain assumption The multi-way parallel evaluation set contains semantically equivalent sentences across all 103 languages.
    Section 2.1 states the evaluation set is used to compare representations while controlling for semantics; any semantic drift would bias SVCCA similarities.
  • domain assumption Linguistic similarity is captured by language family and branch membership.
    Section 3.1 defines similarity via families and branches; the paper acknowledges confounding with script and data size that it does not fully disentangle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Multilingual NMT Representations at Scale." pith.science (2026). https://pith.science/paper/WATTINVZ

@misc{pith2026190902197,
  author       = {Pith},
  title        = {Pith review of: Investigating Multilingual NMT Representations at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WATTINVZ}},
  note         = {Machine review of arXiv:1909.02197}
}
read the original abstract

Multilingual Neural Machine Translation (NMT) models have yielded large empirical success in transfer learning settings. However, these black-box representations are poorly understood, and their mode of transfer remains elusive. In this work, we attempt to understand massively multilingual NMT representations (with 103 languages) using Singular Value Canonical Correlation Analysis (SVCCA), a representation similarity framework that allows us to compare representations across different languages, layers and models. Our analysis validates several empirical results and long-standing intuitions, and unveils new observations regarding how representations evolve in a multilingual translation model. We draw three major conclusions from our analysis, with implications on cross-lingual transfer learning: (i) Encoder representations of different languages cluster based on linguistic similarity, (ii) Representations of a source language learned by the encoder are dependent on the target language, and vice-versa, and (iii) Representations of high resource and/or linguistically similar languages are more robust when fine-tuning on an arbitrary language pair, which is critical to determining how much cross-lingual transfer can be expected in a zero or few-shot setting. We further connect our findings with existing empirical observations in multilingual NMT and transfer learning.

Figures

Figures reproduced from arXiv: 1909.02197 by the authors.

Figure 1
Figure 1. Per language pair data distribution of the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualizing clustering of the encoder representations of all languages, based on their SVCCA similarity. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparing clusterings in the subword embeddings of the Slavic languages in our dataset with their [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Representations of the Turkic and Slavic lan [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The change in distribution of pairwise SVCCA scores between language pairs across layers of a multi [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Visualization depicting the (a) change in representations (using SVCCA) and (b) relative change in per [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Trendlines depicting translation performance [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Top layer of the encoder for En-X language pairs using token level SVCCA as a similarity measure. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The change in distribution of pairwise SVCCA scores using our pooling strategy and a naive token-wise strategy between English-to-Any language pairs across the encoder layers of a multilingual NMT model. We see that while the encoder representations diverge in both cas…
Figure 10
Figure 10. Figure 10: Visualization of the top layer of the encoder and decoder. Both the encoder and decoder show clustering [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: All-to-All (X-X) clustering of the encoder and decoder representations of all languages, based on their [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Visualization of the Indo-Aryan languages, the Iranian languages, and the Dravidian languages, for [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Visualization of the embedding layer for three branches of the Indo-European language family, coloring [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Visualization of the Slavic and Turkic languages, for the embeddings (left column) and the top layer of [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Comparing average change in representation space over finetuning steps across layers for various [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 25 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. arXiv preprint arXiv:1903.00089

    arXiv 2019

  4. [4]

    Maruan Al - Shedivat and Ankur P. Parikh. 2019. http://arxiv.org/abs/1904.02338 Consistency by agreement in zero-shot neural machine translation . CoRR, abs/1904.02338

    arXiv 2019

  5. [5]

    Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019 a . The missing ingredient in zero-shot neural machine translation. arXiv preprint arXiv:1903.07091

    arXiv 2019

  6. [6]

    Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019 b . http://arxiv.org/abs/1907.05019 Massively multilingual neural machine translation in the wild: Findings and challenges

    arXiv 2019

  7. [7]

    Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464

    arXiv 2018

  8. [8]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. http://arxiv.org/abs/1409.0473 Neural machine translation by jointly learning to align and translate . In International Conference on Learning Representations

    arXiv 2015

Show all 52 references
  1. [9]

    Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Identifying and controlling important neurons in neural machine translation. arXiv preprint arXiv:1811.01157

    2018 arXiv

  2. [10]

    Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? arXiv preprint arXiv:1704.03471

    2017 arXiv

  3. [11]

    Yonatan Belinkov, Llu \' s M \`a rquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. arXiv preprint arXiv:1801.07772

    2018 arXiv

  4. [12]

    Mikhail Belkin and Partha Niyogi. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373--1396

    2003

  5. [13]

    https://www.britannica.com/topic/South-Slavic-languages/media/556411/61368 Slavic languages' family tree

    Wayles Browne and Vyacheslav Vsevolodovich Ivanov. https://www.britannica.com/topic/South-Slavic-languages/media/556411/61368 Slavic languages' family tree . Encyclopædia Britannica

  6. [14]

    Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. http://www.aclweb.org/anthology/P...

    2018

  7. [15]

    Kyunghyun Cho, Bart Van Merri \"e nboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

    2014 arXiv

  8. [16]

    Sandagomi Coperahawa. 2007. Language Contact and Linguistic Area: the Sinhala - Tamil Contact Situation . Journal of the Royal Asiatic Society of Sri Lanka, 53:133 -- 152

    2007

  9. [17]

    Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 7

    2019

  10. [18]

    Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingual neural machine translation. arXiv preprint arXiv:1809.04686

    2018 arXiv

  11. [19]

    Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. arXiv preprint arXiv:1601.01073

    2016 arXiv

  12. [20]

    Harald Hammarstr \" o m, Robert Forkel, and Martin Haspelmath. 2017. Hindustani

    2017

  13. [21]

    David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):2639--2664

    2004

  14. [22]

    https://www.britannica.com/topic/West-Germanic-languages/German#ref603830 West Germanic Languages

    Marvin Irving Herzog. https://www.britannica.com/topic/West-Germanic-languages/German#ref603830 West Germanic Languages

  15. [23]

    Pavle Ivi \' c . 2011. https://www.britannica.com/science/linguistics/Language-classification Encyclopaedia Britannica

    2011

  16. [24]

    Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Asso...

    2017

  17. [25]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of...

    2017

  18. [26]

    Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1318--1326. Association for Computational Linguistics

    2011

  19. [27]

    Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226

    2018 arXiv

  20. [28]

    Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291

    2019 arXiv

  21. [29]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier

    1989

  22. [30]

    Etukoori Balaraama Moorti. 2011. https://lists.hcs.harvard.edu/mailman/listinfo/proto-dravidian Proto Dravidian . Study of Dravidian Linguistics and Civilization

    2011

  23. [31]

    Morcos, Maithra Raghu, and Samy Bengio

    Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. http://dl.acm.org/citation.cfm?id=3327345.3327475 Insights on representational similarity in neural networks with canonical correlation . In Proceedings of the 32Nd International Conference on Neural Information Processing S...

    2018

  24. [32]

    Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. arXiv preprint arXiv:1808.04189

    2018 arXiv

  25. [33]

    Nguyen and David Chiang

    Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proc. IJCNLP, volume 2, pages 296--301

    2017

  26. [34]

    Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill, 3(3):e10

    2018

  27. [35]

    O stling and J \

    Robert \"O stling and J \"o rg Tiedemann. 2016. Continuous multilinguality with language vectors. arXiv preprint arXiv:1612.07486

    2016 arXiv

  28. [36]

    Phillips and M Davis

    A. Phillips and M Davis. 2009. https://tools.ietf.org/html/bcp47 Tags for Identifying Languages . RFC 5646, RFC Editor

    2009

  29. [37]

    Alessandro Raganato and J \"o rg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287--297

    2018

  30. [38]

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pages 6076--6085

    2017

  31. [39]

    Hichem Sahbi. 2018. Learning cca representations for misaligned data. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0--0

    2018

  32. [40]

    Naomi Saphra and Adam Lopez. 2018. Understanding learning dynamics of language models with svcca. arXiv preprint arXiv:1811.00225

    2018 arXiv

  33. [41]

    Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235

    2018 arXiv

  34. [42]

    Mohammad Tahsin Siddiqi. 1994. Hindustani-English code-mixing in modern literary texts . University of Wisconsin

    1994

  35. [43]

    http://assets.cambridge.org/97805212/94485/excerpt/9780521294485_excerpt.pdf The Slavic Languages

    Roland Sussex and Paul Cubberley. http://assets.cambridge.org/97805212/94485/excerpt/9780521294485_excerpt.pdf The Slavic Languages

  36. [44]

    Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104--3112

    2014

  37. [45]

    Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with language clustering. arXiv preprint arXiv:1908.09324

    2019 arXiv

  38. [46]

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXi...

    2019 arXiv

  39. [47]

    J \"o rg Tiedemann. 2018. Emerging language spaces learned from massively multilingual corpora. arXiv preprint arXiv:1802.00273

    2018 arXiv

  40. [48]

    Jakob Uszkoreit, Jay M Ponte, Ashok C Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1101--1109. Association for Computational Linguistics

    2010

  41. [49]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008

    2017

  42. [50]

    Shijie Wu and Mark Dredze. 2019. http://arxiv.org/abs/1904.09077 Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT . CoRR, abs/1904.09077

    2019 arXiv

  43. [51]

    Kelly W Zhang and Samuel R Bowman. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv preprint arXiv:1809.10040

    2018 arXiv

  44. [52]

    Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568--1575

    2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.