Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Extending a multilingual model to low-resource languages works better when new vocabulary embeddings start from translation-related source subwords rather than from random vectors.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:47 UTC pith:5UCF4LJ3

load-bearing objection A sensible initialization trick for expanding mPLM vocabularies to low-resource languages, but the headline gains over random initialization are not yet statistically supported. the 3 major comments →

arxiv 2602.09388 v2 pith:5UCF4LJ3 submitted 2026-02-10 cs.CL

Effective vocabulary expansion of multilingual language models for extremely low-resource languages

Jianyu Zheng This is my paper
classification cs.CL
keywords vocabulary expansionmultilingual language modelslow-resource languagescontinued pre-trainingbilingual dictionarysubword embeddingsfastTextembedding initialization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper addresses how to extend a multilingual pre-trained model such as mBERT to extremely low-resource languages it was not trained on. The proposed method expands the model's vocabulary with target-language subwords, then initializes the new embeddings as a temperature-smoothed weighted average of source-language (English) subword embeddings, where the weights come from cosine similarity in a fastText embedding space aligned with the target via bilingual dictionaries. Across seven low-resource languages, this initialization outperforms random initialization by 0.54 F1 on POS tagging and 2.60 F1 on NER after continued pre-training, while preserving English performance. The gains are modest but consistent for NER and appear robust to the choice of training corpus.

Core claim

The central claim is that a similarity-based initialization, instead of random initialization, gives the expanded vocabulary a better starting point for continued pre-training, yielding measurable gains on downstream tasks. Specifically, the paper reports that after continued pre-training, models initialized with the proposed method achieve average macro-F1 improvements of 0.54% on POS tagging and 2.60% on NER compared to random initialization across seven low-resource languages (Amharic, Maltese, Manx, North Sami, Scottish Gaelic, Uyghur, Sinhala). It also claims the method is robust to the choice of target-language corpus and does not degrade the model's ability to process English.

What carries the argument

The key object is the cross-lingual (sub)word similarity matrix S computed from fastText-style subword embeddings (the sum of n-gram word embeddings) for source and target, aligned through an orthogonal mapping learned from a bilingual dictionary. For each new target subword, the method selects its k most similar source subwords (k=10) and initializes its embedding as a weighted average of the mBERT embeddings of those source subwords, with weights given by exp(S/τ) and temperature τ=0.1. This construction is what transfers information from the source language's well-supported subwords to the target's new subwords.

Load-bearing premise

The method assumes that cosine similarity between fastText-derived subword embeddings (source vs target) points to genuinely related subwords whose mBERT embeddings can be averaged; the paper does not verify that the fastText space and mBERT's embedding space are comparable.

What would settle it

Take the same screening step, the same target vocabulary, and the same weighted-average formula, but replace the similarity scores S with random values (or shuffle the assignment of source neighbors to target subwords). If downstream F1 on NER does not fall back to the level of random initialization, the reported gains are not caused by the semantic alignment. Alternatively, compute for a sample of target subwords whether their top-k source neighbors in the fastText space actually correspond to correct translations in the bilingual dictionary; a low hit rate (e.g., below 10%) would indicate th

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Expanding the vocabulary and continued pre-training alone produce large gains over zero-shot mBERT (average +32.54 F1 on POS, +18.45 on NER), so vocabulary expansion is the primary driver of improvement.
  • The similarity-based initialization adds a smaller but consistent gain on NER (+2.60 average, up to +6.68), suggesting the initialization matters more for tasks with fewer label classes.
  • English capability is essentially preserved after continued pre-training, meaning the method does not trade away source-language performance.
  • The method's benefit is larger for languages linguistically distant from English (e.g., Maltese, Uyghur) and smaller for close ones (Scottish Gaelic), consistent with the intuition that distant languages need more help from the source.
  • The method remains effective when the training corpus is changed from MADLAD-400 to Wikipedia, though the best-performing corpus differs by task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The small POS gain (0.54%) and the failure on Amharic and Sinhala—languages with fewer than 1,000 bilingual word pairs—suggest that the quality of the bilingual dictionary, not the initialization scheme itself, is the real bottleneck; a more reliable unsupervised alignment might yield bigger gains.
  • The similarity matrix is computed in the fastText space while the averaged embeddings come from mBERT; this cross-space leap is unvalidated, and the improvement might partly come from the temperature-smoothed averaging having a regularizing effect regardless of which neighbors are picked. A quick ablation (shuffling the neighbor assignments) would separate those effects.
  • The method could be extended to initialize embeddings for multiple low-resource languages at once if a shared multilingual dictionary or unsupervised alignment is used, potentially amortizing the cost of continued pre-training.
  • The released models (the author says they will be released) would let the community test whether the initialization advantage compounds after further task-specific fine-tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes a vocabulary-expansion method for extending mBERT to seven very low-resource languages (Amharic, Maltese, Manx, North Sami, Scottish Gaelic, Uyghur, Sinhala). It screens a subset of mBERT's vocabulary believed to encode English, builds aligned fastText word embeddings from bilingual dictionaries, computes subword embeddings with a fastText-style n-gram sum, and initializes new target-language subwords as a similarity-weighted average of k nearest source subword embeddings. The model is then continued pre-training with MLM on target corpora. Evaluated on UD POS tagging and wikiann NER against mBERT and a random-initialization baseline, it reports average gains of 0.54 and 2.60 points respectively, plus robustness checks on corpus source, source vocabulary size, and English retention.

Significance. The problem is timely and the experimental design has good elements: the random-initialization control is the right baseline, the evaluation uses standard benchmarks, and the robustness analyses cover corpus source, source vocabulary size, and English retention. If the reported gains are statistically reliable, the method is a simple, practical contribution to extending mPLMs to extremely low-resource languages. However, the current evidence for the headline claim is incomplete: no repeated runs or uncertainty estimates are reported on very small evaluation sets, and the described initialization mixes embedding spaces without an explicit alignment. The contribution is therefore conditional on additional validation and clarification.

major comments (3)
  1. [Table 5, §5.1, Table 3] The headline claim (average gains of 0.54 POS and 2.60 NER over random initialization) is not supported with any measure of uncertainty. No seed information, standard deviation, confidence interval, or significance test is reported. The NER test sets are 100 sentences for all five languages and the POS test set for Sinhala is 99 sentences; the NER splits also appear to be a subsample of wikiann, but the sampling procedure is not described. Under these conditions, per-language differences such as Amharic POS (37.62 vs 40.26) and Sinhala POS (28.79 vs 29.63) are as likely to reflect noise as the method's effect. Section 5.2.4 acknowledges scarce evaluation data but does not quantify uncertainty. Please add multiple seeds, variance, and paired significance tests, or clearly state that only one run was performed and temper the claims accordingly.
  2. [§3.4, Eqs. (3)-(4)] There is a mismatch between the space in which similarities are computed and the space from which averaged embeddings are taken. In §3.3, Us and Ut are derived from aligned fastText static word embeddings. Eq. (4) then averages 'e_s_x', defined as the source embedding for subword x in the mPLM's vocabulary, i.e., an mBERT embedding. The paper gives no mapping between the fastText space and mBERT's embedding space. Without such a mapping, the k nearest neighbors found by Eq. (3) are not guaranteed to correspond to the mBERT embeddings used in Eq. (4). Please clarify the intended relation or add an explicit alignment/projection step and ablate it.
  3. [§4.4 and Table 3] The POS evaluation protocol is underspecified for Amharic and Sinhala. Their POS rows have no train or validation splits, yet Table 5 reports scores for these languages. It is therefore unclear how supervised fine-tuning was performed (e.g., zero-shot from English, or training on some other split), and the reader cannot tell whether all languages use the same protocol. Please state the exact training and evaluation procedure for each language.
minor comments (4)
  1. [§3.4, Eq. (3)] The denominator is written as ||Us|| ||Us||; it should presumably be ||Us|| ||Ut||. Please correct the typo in this central equation.
  2. [§3.1] The 'monolingual PLM' used for screening the source vocabulary is never identified. Which model and checkpoint were used for the intersection with mBERT's vocabulary? Without this, the screening step cannot be reproduced.
  3. [§5.2.3, Figure 5] The English-retention analysis reports only an arithmetic average over models trained for all target languages, with no per-language or variance information. Please report the individual values or provide a table.
  4. [Data Availability] The data availability statement says materials are available from the author upon request. For reproducibility, consider releasing code, configuration files, and random seeds, especially because the empirical claims currently rest on small performance differences.

Circularity Check

0 steps flagged

No significant circularity: the reported gains are empirical outcomes, not forced by the method's definitions or by self-citation.

full rationale

The paper's derivation chain is empirical and externally benchmarked. The method constructs a target-language vocabulary from MADLAD-400, screens a source-oriented subset of mBERT's vocabulary by intersection with an English fastText vocabulary, aligns separately trained fastText word embeddings with a bilingual dictionary, derives subword embeddings via fastText n-gram summation, and initializes the expanded embeddings as a similarity-weighted average of mBERT source embeddings. The claimed improvements of 0.54% (POS) and 2.60% (NER) over the random-initialization baseline are measured after continued pre-training and fine-tuning on UD and wikiann, not algebraically implied by Eqs. (2)-(4). Random initialization is the control under the same continued-pretraining setup, so the comparison isolates the initialization scheme. Hyperparameters k=10 and tau=0.1 are hand-chosen rather than fitted to the target test sets. Citations to prior work such as WECHSEL, FOCUS, and Wang et al. are external prior work, not self-citations, and they supply components or inspiration rather than a uniqueness argument that forces the paper's conclusions. The acknowledged limitations—small bilingual dictionaries and scarce evaluation data—are robustness and generalization concerns, not evidence of circularity. The denominator typo in Eq. (3) and the unstated cross-space comparability assumption are correctness risks but do not make any result definitionally identical to its inputs. The paper is self-contained against external benchmarks, so the appropriate finding is no circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No fundamentally new entities are introduced. The method depends on several unverified domain assumptions about embedding-space compatibility and on hand-chosen hyperparameters (k, tau, vocabulary size, training steps), none of which are fitted to the evaluation results.

free parameters (5)
  • k (number of nearest neighbors) = 10
    Number of source subwords used in the weighted average (Eq. 4); chosen by hand without sensitivity analysis.
  • tau (temperature) = 0.1
    Softmax temperature in Eq. (4); chosen by hand.
  • fastText embedding dimension = 300
    Dimension of source/target static word embeddings (Section 4.5).
  • target vocabulary size = 30k
    WordPiece vocabulary size for target languages (Table 4).
  • continued pre-training steps = ~500k
    Number of MLM training steps (Table 4).
axioms (4)
  • domain assumption mPLM vocabulary can be approximated in a shared cross-lingual embedding space, and the English-biased subset can be identified by intersection with a monolingual English model's vocabulary.
    Section 3.1; the screening step presupposes this without verification.
  • domain assumption FastText n-gram subword embeddings computed from word vectors are comparable to mBERT subword embeddings.
    Section 3.3, Eq. (2); the paper does not validate that the two embedding spaces are compatible.
  • domain assumption Orthogonal Procrustes alignment with small bilingual dictionaries (0.2k-8.2k pairs) yields reliable cross-lingual word embeddings.
    Section 3.2; the authors themselves note failures for Amharic and Sinhala due to small dictionaries, so the assumption only partially holds.
  • domain assumption Continued pre-training with MLM only is sufficient for the expanded model to learn the target language.
    Section 4.5; training uses MLM without additional objectives or adapters.

pith-pipeline@v1.3.0-alltime-deepseek · 14284 in / 10498 out tokens · 94273 ms · 2026-08-03T02:47:53.624763+00:00 · methodology

0 comments
read the original abstract

Multilingual pre-trained language models(mPLMs) offer significant benefits for many low-resource languages. To further expand the range of languages these models can support, many works focus on continued pre-training of these models. However, few works address how to extend mPLMs to low-resource languages that were previously unsupported. To tackle this issue, we expand the model's vocabulary using a target language corpus. We then screen out a subset from the model's original vocabulary, which is biased towards representing the source language(e.g. English), and utilize bilingual dictionaries to initialize the representations of the expanded vocabulary. Subsequently, we continue to pre-train the mPLMs using the target language corpus, based on the representations of these expanded vocabulary. Experimental results show that our proposed method outperforms the baseline, which uses randomly initialized expanded vocabulary for continued pre-training, in POS tagging and NER tasks, achieving improvements by 0.54% and 2.60%, respectively. Furthermore, our method demonstrates high robustness in selecting the training corpora, and the models' performance on the source language does not degrade after continued pre-training.

Figures

Figures reproduced from arXiv: 2602.09388 by Jianyu Zheng.

Figure 1
Figure 1. Figure 1: The number of languages supported by commonly used mPLMs. model [18; 23]. To address this issue, some previous works initialize the representation of target language vocabulary by aligning source and target word embeddings[24] and calculating cross-lingual word similarities[18; 25]. However, these works typically focus on low-resource languages already supported by mPLMs[18], or on extending monolingual mo… view at source ↗
Figure 2
Figure 2. Figure 2: The steps for initializing the representations of the expanded vocabulary for target languages. 3.1. Screening source language vocabulary In mPLMs, most (sub)words in the vocabulary are shared and encode information across multiple languages. As a result, aligning the entire vocabulary of the model with the expanded vocabulary for the target language would incur substantial computational costs. To mitigate… view at source ↗
Figure 3
Figure 3. Figure 3: The impact of source language vocabulary size on the POS tagging task. 5.2.2. The impact of training corpus We also investigate whether the size of the target language corpus affects the models’ performance. Combing the results in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The impact of source language vocabulary size on the NER task. Moreover, we also observe that the source of the training corpus does impact on the models’ performance on specific downstream tasks. The models trained with MADLAD data consistently outperform those trained with Wikipedia data on the POS tagging task, while the opposite performance is observed for the NER task. This indicates that the choice o… view at source ↗
Figure 5
Figure 5. Figure 5: The performance of each method for English on the evaluation tasks. 6. Conclusion In this work, to enable existing mPLMs to better process low-resource languages, we first expand the models’ vocabulary to include these languages. Then, using the source language’s word representations and word alignment provided by bilingual dictionaries, we better initialize the representation of the expanded vocabulary fo… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 2 linked inside Pith

  1. [1]

    A primer on pretrained multilingual language models,

    S. Doddapaneni, G. Ramesh, M. Khapra, A. Kunchukuttan, and P. Kumar, “A primer on pretrained multilingual language models,”ACM Computing Surveys, vol. 57, no. 9, pp. 1–39, 2025

    2025

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton, L. K. Toutanovaet al., “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1, no. 2. Minneapolis, Minnesota, 2019

    2019

  3. [3]

    “cloze procedure

    W. L. Taylor, ““cloze procedure”: A new tool for measuring readability,”Journalism quarterly, vol. 30, no. 4, pp. 415–433, 1953

    1953

  4. [4]

    Cross-lingual language model pretraining,

    A. Conneau and G. Lample, “Cross-lingual language model pretraining,”Advances in neural information processing systems, vol. 32, 2019

    2019

  5. [5]

    Unsupervisedcross-lingualrepresentationlearningatscale,

    A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervisedcross-lingualrepresentationlearningatscale,”inProceedingsofthe58thannualmeetingoftheassociationforcomputational linguistics, 2020, pp. 8440–8451

    2020

  6. [6]

    mt5: a massively multilingual pre-trained text-to-text transformer,

    M. Kale, L. Xue, N. Constant, A. Roberts, R. Al-Rfou, A. Siddhant, and A. Barua, “mt5: a massively multilingual pre-trained text-to-text transformer,”arXiv preprint arXiv:2010.11934, 2020

    Pith/arXiv arXiv 2010

  7. [7]

    Mass: Masked sequence to sequence pre-training for language generation

    T. Qin, “Mass: Masked sequence to sequence pre-training for language generation.”

  8. [8]

    Multifit: Efficient multi-lingual language model fine-tuning,

    J. Eisenschlos, S. Ruder, P. Czapla, M. Kadras, S. Gugger, and J. Howard, “Multifit: Efficient multi-lingual language model fine-tuning,” in Proceedingsofthe2019conferenceonempiricalmethodsinnaturallanguageprocessingandthe9thinternationaljointconferenceonnatural language processing (EMNLP-IJCNLP), 2019, pp. 5702–5707

    2019

  9. [9]

    Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks

    H. Huang, Y. Liang, N. Duan, M. Gong, L. Shou, D. Jiang, and M. Zhou, “Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks.”

  10. [10]

    Multilingual denoising pre-training for neural machine translation,

    Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,”Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, 2020. Jianyu Zheng:Preprint submitted to ElsevierPage 11 of 12 Vocabulary expanding of multilingual language models

    2020

  11. [11]

    mgpt: Few-shot learners go multilingual,

    O. Shliazhko, A. Fenogenova, M. Tikhonova, A. Kozlova, V. Mikhailov, and T. Shavrina, “mgpt: Few-shot learners go multilingual,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 58–79, 2024

    2024

  12. [12]

    75languages,1model:Parsinguniversaldependenciesuniversally,

    D.KondratyukandM.Straka,“75languages,1model:Parsinguniversaldependenciesuniversally,”inProceedingsofthe2019Conferenceon Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019, pp. 2779–2795

    2019

  13. [13]

    Ernie-m:Enhancedmultilingualrepresentationbyaligningcross-lingual semanticswithmonolingualcorpora,

    X.Ouyang,S.Wang,C.Pang,Y.Sun,H.Tian,H.Wu,andH.Wang,“Ernie-m:Enhancedmultilingualrepresentationbyaligningcross-lingual semanticswithmonolingualcorpora,”inProceedingsofthe2021conferenceonempiricalmethodsinnaturallanguageprocessing,2021,pp. 27–38

    2021

  14. [14]

    Beyondenglish-centric multilingual machine translation,

    A.Fan,S.Bhosale,H.Schwenk,Z.Ma,A.El-Kishky,S.Goyal,M.Baines,O.Celebi,G.Wenzek,V.Chaudharyetal.,“Beyondenglish-centric multilingual machine translation,”Journal of Machine Learning Research, vol. 22, no. 107, pp. 1–48, 2021

    2021

  15. [15]

    Ethnologue: Languages of the world,

    L. Campbell and V. Grondona, “Ethnologue: Languages of the world,”Language, vol. 84, no. 3, pp. 636–641, 2008

    2008

  16. [16]

    How to adapt your pretrained multilingual model to 1600 languages,

    A. Ebrahimi and K. von der Wense, “How to adapt your pretrained multilingual model to 1600 languages,” inProceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers), 2021, pp. 4555–4567

    2021

  17. [17]

    Expandingpretrainedmodelstothousandsmorelanguagesvialexicon-basedadaptation,

    X.Wang,S.Ruder,andG.Neubig,“Expandingpretrainedmodelstothousandsmorelanguagesvialexicon-basedadaptation,”inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022

    2022

  18. [18]

    Focus:Effectiveembeddinginitializationformonolingualspecializationofmultilingualmodels,

    K.DoblerandG.DeMelo,“Focus:Effectiveembeddinginitializationformonolingualspecializationofmultilingualmodels,”arXivpreprint arXiv:2305.14481, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Extending multilingual bert to low-resource languages,

    Z. Wang, K. Karthikeyan, S. Mayhew, and D. Roth, “Extending multilingual bert to low-resource languages,” inFindings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 2649–2656

    2020

  20. [20]

    The johns hopkins university bible corpus: 1600+ tongues for typological exploration,

    A. D. McCarthy, R. Wicks, D. Lewis, A. Mueller, W. Wu, O. Adams, G. Nicolai, M. Post, and D. Yarowsky, “The johns hopkins university bible corpus: 1600+ tongues for typological exploration,” inProceedings of the Twelfth Language Resources and Evaluation Conference, 2020, pp. 2884–2892

    2020

  21. [21]

    Learningmultiplevisualdomainswithresidualadapters,

    S.-A.Rebuffi,H.Bilen,andA.Vedaldi,“Learningmultiplevisualdomainswithresidualadapters,”Advancesinneuralinformationprocessing systems, vol. 30, 2017

    2017

  22. [22]

    Load what you need: Smaller versions of mutililingual bert,

    A. Abdaoui, C. Pradel, and G. Sigel, “Load what you need: Smaller versions of mutililingual bert,” inProceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, 2020, pp. 119–123

    2020

  23. [23]

    Wechsel: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models,

    B. Minixhofer, F. Paischer, and N. Rekabsaz, “Wechsel: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 3992–4006

    2022

  24. [24]

    A survey of cross-lingual word embedding models,

    S. Ruder, I. Vulić, and A. Søgaard, “A survey of cross-lingual word embedding models,”Journal of Artificial Intelligence Research, vol. 65, pp. 569–631, 2019

    2019

  25. [25]

    From english to foreign languages: transferring pre-trained language models,

    K. Tran, “From english to foreign languages: transferring pre-trained language models,” 2019

    2019

  26. [26]

    Subword mapping and anchoring across languages,

    G. Vernikos and A. Popescu-Belis, “Subword mapping and anchoring across languages,” inFindings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 2633–2647

    2021

  27. [27]

    As good as new. how to successfully recycle english gpt-2 to make models for other languages,

    W. de Vries and M. Nissim, “As good as new. how to successfully recycle english gpt-2 to make models for other languages,” inFindings of the association for computational linguistics: ACL-IJCNLP 2021, 2021, pp. 836–846

    2021

  28. [28]

    Arealllanguagescreatedequalinmultilingualbert?

    S.WuandM.Dredze,“Arealllanguagescreatedequalinmultilingualbert?”in5thWorkshoponRepresentationLearningforNLP,RepL4NLP 2020 at the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020. Association for Computational Linguistics (ACL), 2020, pp. 120–130

    2020

  29. [29]

    Improving pre-trained multilingual model with vocabulary expansion,

    H. Wang, D. Yu, K. Sun, J. Chen, and D. Yu, “Improving pre-trained multilingual model with vocabulary expansion,” inProceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), 2019, pp. 316–327

    2019

  30. [30]

    Lorelei language packs: Data, tools, and resources for technology development in low resource languages,

    S. Strassel and J. Tracey, “Lorelei language packs: Data, tools, and resources for technology development in low resource languages,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), 2016, pp. 3273–3280

    2016

  31. [31]

    Massively multilingual word embeddings

    W. Ammar, G. Mulcaire, Y. Tsvetkov, G. Lample, C. Dyer, and N. A. Smith, “Massively multilingual word embeddings.”

  32. [32]

    Learning multilingual word embeddings in latent metric space: a geometric approach,

    P. Jawanpuria, A. Balgovind, A. Kunchukuttan, and B. Mishra, “Learning multilingual word embeddings in latent metric space: a geometric approach,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 107–120, 2019

    2019

  33. [33]

    Generalizing procrustes analysis for better bilingual dictionary induction,

    Y. KementchedjhievaO, S. Ruder, R. Cotterell, and A. SøgaardO, “Generalizing procrustes analysis for better bilingual dictionary induction,” CoNLL 2018, p. 211, 2018

    2018

  34. [34]

    Enriching word vectors with subword information,

    P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,”Transactions of the association for computational linguistics, vol. 5, pp. 135–146, 2017

    2017

  35. [35]

    A generalized solution of the orthogonal procrustes problem,

    P. H. Schönemann, “A generalized solution of the orthogonal procrustes problem,”Psychometrika, vol. 31, no. 1, pp. 1–10, 1966

    1966

  36. [36]

    Word translation without parallel data,

    G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou, “Word translation without parallel data,” inInternational conference on learning representations, 2018

    2018

  37. [37]

    Are all good word vector spaces isomorphic?

    I. Vulić, S. Ruder, and A. Søgaard, “Are all good word vector spaces isomorphic?” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 3178–3192

    2020

  38. [38]

    Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization,

    A. Siddhant, J. Hu, M. Johnson, O. Firat, and S. Ruder, “Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization,” inProceedings of the International Conference on Machine Learning, vol. 2020, 2020, pp. 4411–4421. Jianyu Zheng:Preprint submitted to ElsevierPage 12 of 12

    2020