REVIEW 4 major objections 4 minor 1 cited by
On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper argues that supervised cross-lingual word embedding learning is more robust than unsupervised learning once training corpora are noisy or the language pair is distant.
desk verdict A useful broad evaluation that tempers the unsupervised-CLWE claim, but the social-media evidence is dimensionality-confounded and an appendix inconsistency needs fixing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the alignment of two independently trained monolingual embedding spaces by a linear transformation learned through an iterative bootstrap of a bilingual dictionary. VecMap uses an orthogonal transformation with a self-learning refinement; MUSE uses a similar orthogonal Procrustes solution with an adversarial or dictionary-based initialization. The paper also evaluates Meemi, a postprocessing step that refines aligned vectors with an unconstrained linear map toward the midpoint of each translation pair. These methods are tested under varying corpus types, supervision amounts, and language pairs, making the alignment procedure itself the object under stress.
What would settle it
Train the social-media monolingual embeddings at 300 dimensions (and perhaps Wikipedia embeddings at 100 dimensions) and repeat the dictionary induction and similarity evaluations; if unsupervised P@1 on noisy text recovers substantially, the corpus-noise explanation loses its main quantitative support. A second check is whether the near-zero Finnish and Farsi results persist when those languages are trained on larger, cleaner corpora of the same type.
Extended reading notes
Core claim
The central claim is that state-of-the-art unsupervised cross-lingual embedding aligners are only competitive under idealized conditions. On Wikipedia data with closely related languages, dictionary-free VecMap and MUSE match supervised performance, but on social-media corpora their dictionary induction precision drops to near zero for Finnish and Farsi, and on web corpora they also fall well below supervised variants. MUSE is additionally unstable with small seed dictionaries, sometimes doing worse with a 100-pair dictionary than with none, while VecMap remains consistent across supervision levels. Postprocessing with a large clean bilingual dictionary helps, especially on non-comparable corpora, but hurts when the dictionary is small or noisy. From these results the paper concludes that the claim that high-quality cross-lingual embeddings can always be learned without much supervision is not supported.
Load-bearing premise
The comparison across corpus types assumes that the social-media performance drop is caused by noise and non-comparability, but social-media embeddings are also trained at a lower dimensionality (100 versus 300), so part of the drop could be an artifact of representational capacity.
Editorial extensions
If this is right
- Unsupervised cross-lingual alignment should be treated as reliable only for clean comparable corpora and closely related languages.
- Noisy user-generated text breaks dictionary-free alignment, and even supervised models degrade sharply, so cross-lingual social-media tasks need dedicated training signals or postprocessing.
- Distant language pairs such as English-Finnish, English-Farsi, and English-Russian remain major challenges regardless of method; supervision matters most there.
- The Meemi postprocessing step improves alignments on non-comparable corpora when paired with a large clean dictionary, but is harmful with small or noisy dictionaries.
- MUSE's sensitivity to seed-dictionary size means that unstable convergence, not just average accuracy, should be part of method comparisons.
Reading between the lines
- Editorial inference: the paper's flagship corpus comparison is partly confounded, because the social-media embeddings use 100 dimensions while Wikipedia and web embeddings use 300; a retest at matched dimensionality is needed to separate corpus noise from representational capacity.
- Editorial inference: the identical-words dictionary behaves as cheap distant supervision and helps most on code-switching social-media text, suggesting that automatically harvested cognates or shared tokens may substitute for seed dictionaries on noisy data.
- Editorial inference: the reported correlations among tasks (dictionary induction correlates with NLI at 0.73, similarity with NLI at 0.28) suggest that lexicon induction may be a better proxy for downstream transfer than word similarity, although the paper presents these figures as indicative only.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an extensive empirical evaluation of alignment-based cross-lingual word embedding methods, comparing VecMap and MUSE in unsupervised, identical-word-dictionary, and 100/1K/8K supervised settings, with and without the Meemi post-processing step. The evaluation spans Wikipedia, web, and Twitter corpora; five or six target languages; and three tasks: bilingual dictionary induction, cross-lingual word similarity (SemEval-17), and zero-shot XNLI. The authors conclude that supervised cross-lingual alignment is more robust than purely unsupervised alignment, particularly on noisy, non-comparable corpora and for distant language pairs, and that Meemi can improve results when a sufficiently large clean dictionary is available.
Significance. The question addressed is timely and important: recent unsupervised cross-lingual embedding methods have mostly been validated on comparable, clean Wikipedia data, and the paper broadens the evidence to noisy and non-comparable settings that are closer to practical use. The breadth of the study—three corpus types, multiple language families and scripts, two base models, a post-processing method, and three evaluation tasks—is a genuine contribution. The authors are also candid about failure cases, including MUSE instability and near-zero results on Finnish and Farsi social media. If the robustness conclusion survives the experimental concerns below, the paper would usefully qualify the claims of Conneau et al. (2018a) and Artetxe et al. (2018b). However, the decisive evidence currently rests on a confounded corpus-type comparison and on differences that are not statistically quantified, so the significance is conditional on the issues being resolved.
major comments (4)
- [Section 4.4 (footnote 5), Section 6, Section 7] The central corpus-type comparison is confounded by embedding dimensionality: social media embeddings are trained at 100 dimensions while Wikipedia and web embeddings use 300 dimensions. The social media condition is where the supervised-versus-unsupervised gap is largest (e.g., Table 3 average Spearman 46.9 vs 19.6 for MUSE 8K vs unsupervised; Table 2 dictionary induction P@5 9.3 vs 2.9), and Section 6 attributes the Twitter collapse "largely to the noise and informal nature of the conversation topics." Because lower dimensionality degrades nearest-neighbour structure, which the unsupervised bootstrap mechanisms of both VecMap and MUSE directly depend on, the collapse could be a technical artifact of dimension rather than a property of noisy user-generated text. The authors should either train and align 300-dimensional social media embeddings or run a controlled dimensionality sweep on the same corpus (e.g., 100-d vs 300-d Wikipedia) to rule out this alternative explanation. This is load-bearing for the paper's main robustness claim.
- [Appendix Table 6 vs Table 2 (Wikipedia, Meemi rows)] There is a clear mismatch between the main text and appendix for the Meemi conditions on English-Farsi Wikipedia with the 8K dictionary. Table 2 reports MeemiVM P@1/P@5/P@10 as 23.4/41.7/47.7 and MeemiMS as 20.7/38.6/45.1, while Appendix Table 6 reports 0.0/0.0/0.0 for both methods under the same conditions. This is not a cosmetic inconsistency: the appendix values would overturn the claim in Section 6 that Meemi helps on distant languages, and the main-table values support it. The discrepancy must be resolved, and because one such mismatch was found, the full set of tables should be re-audited for transcription errors.
- [Tables 2-4 and Section 6] No error bars, confidence intervals, or significance tests are reported, yet several conclusions rest on small or unstable differences. For example, in Table 2 Wikipedia VecMap P@1 is 39.6 for both unsupervised and 8K supervision, and in Table 4 XNLI VecMap is 43.3 unsupervised versus 43.1 with 8K supervision. The claim that supervision is "a key component" in non-ideal settings is not statistically quantified, especially given that the authors themselves note MUSE convergence instability in Section 6. Repeated runs with variance reporting, or at minimum a statement of how many runs were used and whether the reported numbers are representative, are needed before broad robustness conclusions can be accepted.
- [Section 7 (Conclusions)] The concluding statement that "supervised cross-lingual word embedding learning is more robust than purely unsupervised cross-lingual learning" is stronger than the evidence in the non-confounded conditions. On Wikipedia and web corpora at 300 dimensions, VecMap's unsupervised mode is often within one point of its supervised mode (Table 2), and on XNLI the supervision gap is negligible or reversed (Table 4, Wikipedia VecMap 43.3 unsupervised vs 43.1 with 8K). The conclusion should be scoped to the specific noisy-corpus conditions that actually show large gaps, after the dimensionality confound is addressed, rather than stated as a general property of supervision.
minor comments (4)
- [Section 4.4] The use of 100 dimensions for social media and 300 dimensions for Wikipedia/web is disclosed only in a footnote; given its importance for interpreting Tables 2 and 3, it should be stated in Section 4.1 alongside the corpus statistics and discussed explicitly in Section 6.
- [Section 6] The error analysis for Twitter ("discover" to "descubr", "timeline" to "instas") is illustrative but anecdotal; quantifying error types (e.g., orthographic noise versus topic shift) would make the causal claim about noise more convincing.
- [Table 2 caption] The social media block has no Russian column, consistent with the note in Section 4.1, but this should be stated in the table caption for clarity.
- [Section 5.1] The three seed dictionaries are described as splits of the Conneau et al. (2018a) training dictionaries; it would be helpful to state whether the 8K/1K/100 splits are nested and to confirm explicitly that the test dictionaries are disjoint from all training splits.
Circularity Check
No significant circularity: the paper is an empirical evaluation against external benchmarks, and its central claim is supported by measured performance differences rather than by construction or self-citation.
full rationale
This paper is an empirical evaluation study, not a derivation: it compares VecMap and MUSE under different corpora, supervision levels, and language pairs, and evaluates them on the MUSE test dictionaries, SemEval-2017 similarity data, and XNLI. The central claim that supervised cross-lingual learning is more robust than unsupervised learning is supported by external benchmark numbers in Tables 2-4, so it is not equivalent to any fitted input by construction. The only self-cited system, Meemi (Doval et al., 2018), is one of several evaluated methods and is used as a postprocessing baseline; improvements are not assumed but measured, and the paper explicitly documents cases where Meemi fails (e.g., with small dictionaries and for Farsi in Tables 6-7). No equation reduces the conclusion to a parameter definition, and no load-bearing argument relies on a self-citation chain. The dimensionality confound between social media (100-d) and other corpora (300-d), noted in footnote 5 of Section 4.4, is a genuine threat to the corpus-type interpretation, but it is a correctness/validity risk rather than a circularity: the comparison does not define its conclusion into existence, and the paper does not rename a fitted quantity as a prediction. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Embedding dimensionality =
100 for social media; 300 for Wikipedia and web corpora
assumptions (3)
- domain assumption Monolingual embedding spaces of different languages are approximately isomorphic.
- domain assumption Words that appear in both monolingual corpora tend to have the same meaning.
- domain assumption The MUSE test dictionaries and SemEval-17 similarity judgments are valid gold resources for Finnish, Farsi, and Russian.
Cite this review
Pith. "Pith review of On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning." pith.science (2026). https://pith.science/paper/35NVLWF6
@misc{pith2026190807742,
author = {Pith},
title = {Pith review of: On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/35NVLWF6}},
note = {Machine review of arXiv:1908.07742}
}
read the original abstract
Cross-lingual word embeddings are vector representations of words in different languages where words with similar meaning are represented by similar vectors, regardless of the language. Recent developments which construct these embeddings by aligning monolingual spaces have shown that accurate alignments can be obtained with little or no supervision. However, the focus has been on a particular controlled scenario for evaluation, and there is no strong evidence on how current state-of-the-art systems would fare with noisy text or for language pairs with major linguistic differences. In this paper we present an extensive evaluation over multiple cross-lingual embedding models, analyzing their strengths and limitations with respect to different variables such as target language, training corpora and amount of supervision. Our conclusions put in doubt the view that high-quality cross-lingual embeddings can always be learned without much supervision.
Figures
Forward citations
Cited by 1 Pith paper
-
Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?
In a 210-pair bilingual lexicon induction evaluation, unsupervised cross-lingual embeddings underperform weakly supervised methods on every pair, including 87 pairs where they collapse to near-zero accuracy.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
U., Zhang, Z., Ma, X., Hovy, E., Chang, K.-W., and Peng, N
Ahmad, W. U., Zhang, Z., Ma, X., Hovy, E., Chang, K.-W., and Peng, N. (2019). On difficulties of cross-lingual transfer with order differences: A case study on dependency parsing. In Proceedings of NAACL
work page 2019
-
[3]
Al-Rfou, R., Perozzi, B., and Skiena, S. (2013). Polyglot: Distributed word representations for multilingual nlp. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning , pages 183--192, Sofia, Bulgaria
work page 2013
-
[4]
AleAhmad, A., Amiri, H., Darrudi, E., Rahgozar, M., and Oroumchian, F. (2009). Hamshahri: A standard persian text collection. Knowledge-Based Systems , 22(5):382--387
work page 2009
-
[5]
Artetxe, M., Labaka, G., and Agirre, E. (2017). Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 451--462, Vancouver, Canada, July. Association for Computational Linguistics
work page 2017
-
[6]
Artetxe, M., Labaka, G., and Agirre, E. (2018a). Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)
work page 2018
-
[7]
Artetxe, M., Labaka, G., and Agirre, E. (2018b). A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of ACL , pages 789--798
work page 2018
-
[8]
Bakarov, A., Suvorov, R., and Sochenkov, I. (2018). The limitations of cross-language word embeddings evaluation. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics , pages 94--100
work page 2018
Show all 43 references
-
[9]
Barone, A. V. M. (2016). Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders. In Proceedings of the 1st Workshop on Representation Learning for NLP , pages 121--126
2016
-
[10]
Baroni, M., Bernardini, S., Ferraresi, A., and Zanchetta, E. (2009). The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language resources and evaluation , 43(3):209--226
2009
-
[11]
Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association of Computational Linguistics , 5(1):135--146
2017
-
[12]
T., Collier, N., and Navigli, R
Camacho-Collados, J., Pilehvar, M. T., Collier, N., and Navigli, R. (2017). Semeval-2017 task 2: Multilingual and cross-lingual semantic word similarity. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , pages 15--26
2017
-
[13]
Camacho-Collados, J., Doval, Y., Mart \' nez-C \'a mara, E., Espinosa-Anke, L., Barbieri, F., and Schockaert, S. (2020). Learning cross-lingual embeddings from T witter via distant supervision. In Proceedings of ICWSM
2020
-
[14]
Cardellino, C. (2016). Spanish B illion W ords C orpus and E mbeddings. http://crscardellino.me/SBWCE/ , March
2016
-
[15]
C., and Saha, A
Chandar, S., Lauly, S., Larochelle, H., Khapra, M., Ravindran, B., Raykar, V. C., and Saha, A. (2014). An autoencoder approach to learning bilingual word representations. In Advances in Neural Information Processing Systems , pages 1853--1861
2014
-
[16]
and Cardie, C
Chen, X. and Cardie, C. (2018). Unsupervised multilingual word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 261--270
2018
-
[17]
and Kiela, D
Conneau, A. and Kiela, D. (2018). SentEval: An Evaluation Toolkit for Universal Sentence Representations . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , Miyazaki, Japan. European Language Resources Association (ELRA)
2018
-
[18]
Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and J \'e gou, H. (2018a). Word translation without parallel data. In Proceedings of ICLR
2018
-
[19]
R., Schwenk, H., and Stoyanov, V
Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., and Stoyanov, V. (2018b). Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computa...
2018
-
[20]
Doval, Y., Camacho-Collados, J., Espinosa-Anke, L., and Schockaert, S. (2018). Improving cross-lingual word embeddings by meeting in the middle. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 294--304, Brussels, Belgium. Assoc...
2018
-
[21]
Doval, Y., Camacho-Collados, J., Espinosa-Anke, L., and Schockaert, S. (2019). Meemi: A simple method for post-processing and integrating cross-lingual word embeddings. arXiv preprint arXiv:1910.07221
2019 arXiv
-
[22]
Glavas, G., Litschko, R., Ruder, S., and Vulic, I. (2019). How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions . CoRR , abs/1902.00508
2019 arXiv
-
[23]
Godin, F., Vandersmissen, B., De Neve, W., and Van de Walle, R. (2015). Named Entity Recognition for Twitter Microposts using Distributed Word Representations . In Proceedings of ACL Workshop on Noisy User-generated Text , pages 146--153
2015
-
[24]
Han, L., Kashyap, A., Finin, T., Mayfield, J., and Weese, J. (2013). UMBC EBIQUITY-CORE: Semantic textual similarity systems . In Proceedings of the Second Joint Conference on Lexical and Computational Semantics , pages 44--52
2013
-
[25]
Hartmann, M., Kementchedjhieva, Y., and S gaard, A. (2018). Why is unsupervised alignment of english embeddings from different algorithms so hard? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 582--586
2018
-
[26]
Heyman, G., Verreet, B., Vuli \'c , I., and Moens, M.-F. (2019). Learning unsupervised multilingual word embeddings with incremental multilingual hubs. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human ...
2019
-
[27]
and Wolf, L
Hoshen, Y. and Wolf, L. (2018). Non-adversarial unsupervised word translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 469--478, Brussels, Belgium, October-November. Association for Computational Linguistics
2018
-
[28]
Kementchedjhieva, Y., Ruder, S., Cotterell, R., and S gaard, A. (2018). Generalizing procrustes analysis for better bilingual dictionary induction. In Proceedings of the Conference on Computational Natural Language Learning
2018
-
[29]
Klementiev, A., Titov, I., and Bhattarai, B. (2012). Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012 , pages 1459--1474
2012
-
[30]
Luong, T., Pham, H., and Manning, C. D. (2015). Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing , pages 151--159
2015
-
[31]
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. CoRR , abs/1301.3781
2013 arXiv
-
[32]
Pennington, J., Socher, R., and Manning, C. D. (2014). Glo V e: Global vectors for word representation. In Proceedings of EMNLP , pages 1532--1543
2014
-
[33]
Ruder, S., Vuli \'c , I., and S gaard, A. (2018). A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research
2018
-
[34]
L., Turban, D
Smith, S. L., Turban, D. H., Hamblin, S., and Hammerla, N. Y. (2017). Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of ICLR
2017
-
[35]
S gaard, A., Ruder, S., and Vuli \' c , I. (2018). On the limitations of unsupervised bilingual dictionary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 778--788. Association for Computatio...
2018
-
[36]
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., and Qin, B. (2014). Learning sentiment-specific word embedding for twitter sentiment classification. In Proceedings of ACL , pages 1555--1565
2014
-
[37]
Upadhyay, S., Faruqui, M., Dyer, C., and Roth, D. (2016). Cross-lingual models of word embeddings: An empirical comparison. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pages 1661--1670
2016
-
[38]
and Korhonen, A
Vuli \'c , I. and Korhonen, A. (2016). On the role of seed lexicons in learning bilingual word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pages 247--257
2016
-
[39]
and Moens, M.-F
Vuli \'c , I. and Moens, M.-F. (2015). Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conferenc...
2015
-
[40]
Vuli \'c , I., Glava s , G., Reichart, R., and Korhonen, A. (2019). Do we really need fully unsupervised cross-lingual embeddings? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La...
2019
-
[41]
Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V...
2018
-
[42]
Xu, R., Yang, Y., Otani, N., and Wu, Y. (2018). Unsupervised cross-lingual transfer of word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2465--2474. Association for Computational Linguistics
2018
-
[43]
Yang, X., Macdonald, C., and Ounis, I. (2018). Using word embeddings in twitter election classification. Information Retrieval Journal , 21(2):183--207, Jun
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.