{"id":"b0b7dd43-7057-40f2-a3ba-d232aa69811d","arxiv_id":"1908.07742","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Unsupervised cross-lingual embeddings are much less reliable on noisy social media text and on distant language pairs, so supervision remains important for robust alignment.","lead":"This paper stress-tests popular cross-lingual word embedding methods on clean encyclopedic text, web text, and noisy social media, across six target languages. It finds that unsupervised alignment is brittle on distant languages and noisy corpora, so supervised bilingual dictionaries still matter.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that noisy user-generated text defeats unsupervised alignment is confounded with embedding dimension (Section 4.4 footnote): social media embeddings are 100-d, Wikipedia/web are 300-d.","rationale":"Good faith reading: the paper is an evaluation study whose value is the breadth of corpora, languages, supervision levels and tasks, and it uses standard tools and published test dictionaries. Its independent support includes a genuinely uncontrolled corpus-quality comparison within the 300-d regime (Wikipedia vs web corpora), qualitative error analysis specific to Twitter content, and consistent trends across three tasks. The central claim requires that (a) unsupervised methods fail in non-ideal settings and (b) supervision recovers the loss because the difficulty is linguistic or corpus-related rather than technical. The least secure condition is (b)-as-explained: the flagship evidence for the supervision gap lives in the social media rows of Tables 2 and 3, and those rows are the only ones with 100-d embeddings. I weighed three alternatives. First, missing error bars and single seeds: genuine, and the paper itself flags MUSE's convergence instability (Section 6, footnote 12), but the corpus effect is consistent across two models and across tasks, and VecMap also collapses, so seed noise alone would not erase the pattern. Second, the appendix inconsistency in Table 6 (MeemiVM/MeemiMS English-Farsi Wikipedia/web at 8K/1K shown as 0.0, contradicting positive values in Table 2): real and should be corrected, but it affects the Meemi postprocessing discussion rather than the supervised-vs-unsupervised comparison itself. Third, the observation that supervision gains are concentrated in the social media condition while 300-d conditions show little or no gain (XNLI Table 4) — this is not a separate objection; it is the same fact that makes the dimensionality confound load-bearing, because it identifies where the evidence for the headline claim actually sits. The dimensionality check is the cleanest discriminator: it is explicitly admitted in the text, and its outcome changes what the paper can claim about noisy text. I therefore keep the reader's CONDITIONAL verdict; the claim is plausible and partially supported, but not definitive until the controlled rerun is done.","tokens_in":20597,"tokens_out":14780,"duration_ms":137312,"concrete_test":"Retrain the five Twitter monolingual embeddings at 300 dimensions (same FastText settings, tokenization and preprocessing as the Wikipedia/web runs) and rerun the full battery: VecMap and MUSE under unsupervised, identical-dictionary and 8K-dictionary supervision; report bilingual dictionary induction P@1/P@5/P@10 and SemEval-17 Spearman. If unsupervised P@1 on the 300-d Twitter embeddings stays near the current values (roughly 0-9) while supervised results stay near 4-10, the noise-based explanation survives. If unsupervised P@1 rises substantially (e.g., toward the 20-30 range typical of 300-d web corpora for the same language pairs), the central explanation is false or incomplete and the claim must be re-scoped to low-dimensional embeddings rather than noisy text.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 7) is that supervised cross-lingual alignment is more robust than unsupervised learning, with the decisive evidence coming from the social media condition, where supervised-to-unsupervised gaps are largest (e.g., MUSE word similarity jumps from 19.6 to 46.9 average Spearman; dictionary induction P@5 from 2.9 to 9.3, Table 2). Section 6 ('Impact of corpora') attributes the Twitter collapse 'largely to the noise and the informal nature of the conversation topics,' supported by qualitative errors such as 'discover' -> 'descubr' and 'timeline' -> 'instas'. This explanation is not established because corpus type is perfectly confounded with embedding dimension: footnote 5 of Section 4.4 states that Wikipedia and web embeddings use 300 dimensions while social media embeddings use 100 dimensions. Lower dimensionality degrades nearest-neighbour structure, which is precisely what the unsupervised bootstrap mechanisms (VecMap's similarity-based seed and MUSE's adversarial initialization) depend on, so the collapse could be a technical artifact rather than a property of noisy user-generated text. The concern is load-bearing because the paper's own non-confounded conditions show much weaker supervision effects: XNLI (Table 4, both corpora at 300-d) shows no supervised-vs-unsupervised gap (43.3 vs 43.1 VecMap Wikipedia), and Wikipedia dictionary induction gains are small for VecMap (30.2 to 31.2 P@1, Table 2). The web-vs-Wikipedia contrast (both 300-d) does support a comparability effect, but the specific conclusion that noisy text defeats unsupervised alignment rests on the one condition that is confounded, and Section 4.4 explicitly lists this dimensionality change as an 'external factor' rather than controlling it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an extensive empirical evaluation of alignment-based cross-lingual word embedding methods, comparing VecMap and MUSE in unsupervised, identical-word-dictionary, and 100/1K/8K supervised settings, with and without the Meemi post-processing step. The evaluation spans Wikipedia, web, and Twitter corpora; five or six target languages; and three tasks: bilingual dictionary induction, cross-lingual word similarity (SemEval-17), and zero-shot XNLI. The authors conclude that supervised cross-lingual alignment is more robust than purely unsupervised alignment, particularly on noisy, non-comparable corpora and for distant language pairs, and that Meemi can improve results when a sufficiently large clean dictionary is available.","tokens_in":20928,"tokens_out":6647,"duration_ms":58454,"significance":"The question addressed is timely and important: recent unsupervised cross-lingual embedding methods have mostly been validated on comparable, clean Wikipedia data, and the paper broadens the evidence to noisy and non-comparable settings that are closer to practical use. The breadth of the study—three corpus types, multiple language families and scripts, two base models, a post-processing method, and three evaluation tasks—is a genuine contribution. The authors are also candid about failure cases, including MUSE instability and near-zero results on Finnish and Farsi social media. If the robustness conclusion survives the experimental concerns below, the paper would usefully qualify the claims of Conneau et al. (2018a) and Artetxe et al. (2018b). However, the decisive evidence currently rests on a confounded corpus-type comparison and on differences that are not statistically quantified, so the significance is conditional on the issues being resolved.","major_comments":[{"comment":"The central corpus-type comparison is confounded by embedding dimensionality: social media embeddings are trained at 100 dimensions while Wikipedia and web embeddings use 300 dimensions. The social media condition is where the supervised-versus-unsupervised gap is largest (e.g., Table 3 average Spearman 46.9 vs 19.6 for MUSE 8K vs unsupervised; Table 2 dictionary induction P@5 9.3 vs 2.9), and Section 6 attributes the Twitter collapse \"largely to the noise and informal nature of the conversation topics.\" Because lower dimensionality degrades nearest-neighbour structure, which the unsupervised bootstrap mechanisms of both VecMap and MUSE directly depend on, the collapse could be a technical artifact of dimension rather than a property of noisy user-generated text. The authors should either train and align 300-dimensional social media embeddings or run a controlled dimensionality sweep on the same corpus (e.g., 100-d vs 300-d Wikipedia) to rule out this alternative explanation. This is load-bearing for the paper's main robustness claim.","section":"Section 4.4 (footnote 5), Section 6, Section 7"},{"comment":"There is a clear mismatch between the main text and appendix for the Meemi conditions on English-Farsi Wikipedia with the 8K dictionary. Table 2 reports MeemiVM P@1/P@5/P@10 as 23.4/41.7/47.7 and MeemiMS as 20.7/38.6/45.1, while Appendix Table 6 reports 0.0/0.0/0.0 for both methods under the same conditions. This is not a cosmetic inconsistency: the appendix values would overturn the claim in Section 6 that Meemi helps on distant languages, and the main-table values support it. The discrepancy must be resolved, and because one such mismatch was found, the full set of tables should be re-audited for transcription errors.","section":"Appendix Table 6 vs Table 2 (Wikipedia, Meemi rows)"},{"comment":"No error bars, confidence intervals, or significance tests are reported, yet several conclusions rest on small or unstable differences. For example, in Table 2 Wikipedia VecMap P@1 is 39.6 for both unsupervised and 8K supervision, and in Table 4 XNLI VecMap is 43.3 unsupervised versus 43.1 with 8K supervision. The claim that supervision is \"a key component\" in non-ideal settings is not statistically quantified, especially given that the authors themselves note MUSE convergence instability in Section 6. Repeated runs with variance reporting, or at minimum a statement of how many runs were used and whether the reported numbers are representative, are needed before broad robustness conclusions can be accepted.","section":"Tables 2-4 and Section 6"},{"comment":"The concluding statement that \"supervised cross-lingual word embedding learning is more robust than purely unsupervised cross-lingual learning\" is stronger than the evidence in the non-confounded conditions. On Wikipedia and web corpora at 300 dimensions, VecMap's unsupervised mode is often within one point of its supervised mode (Table 2), and on XNLI the supervision gap is negligible or reversed (Table 4, Wikipedia VecMap 43.3 unsupervised vs 43.1 with 8K). The conclusion should be scoped to the specific noisy-corpus conditions that actually show large gaps, after the dimensionality confound is addressed, rather than stated as a general property of supervision.","section":"Section 7 (Conclusions)"}],"minor_comments":[{"comment":"The use of 100 dimensions for social media and 300 dimensions for Wikipedia/web is disclosed only in a footnote; given its importance for interpreting Tables 2 and 3, it should be stated in Section 4.1 alongside the corpus statistics and discussed explicitly in Section 6.","section":"Section 4.4"},{"comment":"The error analysis for Twitter (\"discover\" to \"descubr\", \"timeline\" to \"instas\") is illustrative but anecdotal; quantifying error types (e.g., orthographic noise versus topic shift) would make the causal claim about noise more convincing.","section":"Section 6"},{"comment":"The social media block has no Russian column, consistent with the note in Section 4.1, but this should be stated in the table caption for clarity.","section":"Table 2 caption"},{"comment":"The three seed dictionaries are described as splits of the Conneau et al. (2018a) training dictionaries; it would be helpful to state whether the 8K/1K/100 splits are nested and to confirm explicitly that the test dictionaries are disjoint from all training splits.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"This is an honest and useful empirical study, and the central question is well motivated. The headline claim, however, currently rests on a dimension-confounded comparison and on an appendix/main-text inconsistency that directly affects one of the stated findings. Both issues are fixable within the scope of the manuscript, and I would not reject. If the authors provide dimension-matched social media results, significance or variance estimates, and a corrected, re-audited set of tables, the paper could be acceptable for publication. I saw no circularity concern: Meemi is one of the evaluated systems, and the main claim does not depend on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, honest evaluation that should temper the 'unsupervised alignment just works' claim, but the most dramatic evidence—the social-media collapse—is confounded, and one appendix table doesn't agree with the main table. The stress-test note lands.\n\nThe genuinely new part is the grid itself: three corpus types, six target languages, VecMap/MUSE with and without supervision plus Meemi postprocessing, and three tasks including XNLI. That broadens Søgaard et al. and Vulić/Korhonen, and the Meemi dictionary-size finding is a real addition—Meemi needs a big clean dictionary and can hurt otherwise. The cross-task correlations are a nice practical check. The paper also documents its own failure cases and is not circular: Meemi is self-cited but it is an evaluated system, not a premise.\n\nThe central empirical pattern is visible in the non-confounded conditions too. On web corpora at 300 dimensions, unsupervised VecMap collapses on Finnish (0.0 P@1) and unsupervised MUSE collapses on Finnish and Russian, with 8K dictionaries recovering them. XNLI shows almost no supervision gap, so the claim should be 'supervised helps for word-level tasks and distant/non-comparable corpora,' not a universal law. But the specifically loud result—that noisy social-media text destroys unsupervised alignment—is not established, because the social-media embeddings are 100-d while Wikipedia and web are 300-d. Dimensionality changes nearest-neighbor structure and hits the bootstrap mechanisms these methods rely on. The footnote lists this as an external factor rather than controlling it. That is a real flaw in a load-bearing spot of the narrative, though not in the entire paper.\n\nTwo smaller issues: no error bars or significance tests, so some gaps are just noise; and Table 6 in the appendix shows MeemiVM/MeemiMS at 8K scoring 0.0 on English-Farsi Wikipedia while Table 2 reports around 20/40. That cannot both be right, and it touches exactly the claim that Meemi helps distant languages. Also, all source languages are English, and corpus sizes vary (Finnish/Farsi Wikipedia are small), so the language-family story is partly corpus-size confounded.\n\nWho should read it: anyone working on cross-lingual embeddings or relying on unsupervised alignment for low-resource transfer. It deserves a serious referee. I would accept for review and ask for dimensionality-controlled social-media embeddings, corrected tables, and at least a few seeds before publication.","headline":"A useful broad evaluation that tempers the unsupervised-CLWE claim, but the social-media evidence is dimensionality-confounded and an appendix inconsistency needs fixing.","tokens_in":21468,"tokens_out":4682,"would_cite":true,"duration_ms":134667,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that supervised cross-lingual word embedding learning is more robust than unsupervised learning once training corpora are noisy or the language pair is distant.","keywords":["cross-lingual word embeddings","unsupervised alignment","bilingual dictionary induction","robustness evaluation","noisy user-generated text","distant languages","VecMap","MUSE"],"falsifier":"Train the social-media monolingual embeddings at 300 dimensions (and perhaps Wikipedia embeddings at 100 dimensions) and repeat the dictionary induction and similarity evaluations; if unsupervised P@1 on noisy text recovers substantially, the corpus-noise explanation loses its main quantitative support. A second check is whether the near-zero Finnish and Farsi results persist when those languages are trained on larger, cleaner corpora of the same type.","tokens_in":20439,"feed_emoji":"🌐","tokens_out":7182,"duration_ms":278180,"temperature":0.7,"pith_summary":"This paper tries to establish that the recent success of unsupervised cross-lingual word embeddings is tied to a favorable setting: clean, comparable corpora such as Wikipedia and closely related languages. It reports an extensive comparison of two alignment methods, VecMap and MUSE, with and without bilingual seed dictionaries, across Wikipedia, web, and social-media corpora and six target languages. The results show that on noisy user-generated text and for languages with different alphabets or morphology, unsupervised alignment collapses while supervision becomes decisive. The authors conclude that supervised cross-lingual learning is more robust than purely unsupervised learning, and that bilingual supervision signals are a key component in non-ideal settings.","feed_headline":"Dictionary-free word alignment fails on noisy, distant languages","feed_subtitle":"A six-language, three-corpus study finds unsupervised alignment only works on clean, comparable data.","key_machinery":"The load-bearing machinery is the alignment of two independently trained monolingual embedding spaces by a linear transformation learned through an iterative bootstrap of a bilingual dictionary. VecMap uses an orthogonal transformation with a self-learning refinement; MUSE uses a similar orthogonal Procrustes solution with an adversarial or dictionary-based initialization. The paper also evaluates Meemi, a postprocessing step that refines aligned vectors with an unconstrained linear map toward the midpoint of each translation pair. These methods are tested under varying corpus types, supervision amounts, and language pairs, making the alignment procedure itself the object under stress.","core_discovery":"The central claim is that state-of-the-art unsupervised cross-lingual embedding aligners are only competitive under idealized conditions. On Wikipedia data with closely related languages, dictionary-free VecMap and MUSE match supervised performance, but on social-media corpora their dictionary induction precision drops to near zero for Finnish and Farsi, and on web corpora they also fall well below supervised variants. MUSE is additionally unstable with small seed dictionaries, sometimes doing worse with a 100-pair dictionary than with none, while VecMap remains consistent across supervision levels. Postprocessing with a large clean bilingual dictionary helps, especially on non-comparable corpora, but hurts when the dictionary is small or noisy. From these results the paper concludes that the claim that high-quality cross-lingual embeddings can always be learned without much supervision is not supported.","pith_inferences":["Editorial inference: the paper's flagship corpus comparison is partly confounded, because the social-media embeddings use 100 dimensions while Wikipedia and web embeddings use 300; a retest at matched dimensionality is needed to separate corpus noise from representational capacity.","Editorial inference: the identical-words dictionary behaves as cheap distant supervision and helps most on code-switching social-media text, suggesting that automatically harvested cognates or shared tokens may substitute for seed dictionaries on noisy data.","Editorial inference: the reported correlations among tasks (dictionary induction correlates with NLI at 0.73, similarity with NLI at 0.28) suggest that lexicon induction may be a better proxy for downstream transfer than word similarity, although the paper presents these figures as indicative only."],"forward_implications":["Unsupervised cross-lingual alignment should be treated as reliable only for clean comparable corpora and closely related languages.","Noisy user-generated text breaks dictionary-free alignment, and even supervised models degrade sharply, so cross-lingual social-media tasks need dedicated training signals or postprocessing.","Distant language pairs such as English-Finnish, English-Farsi, and English-Russian remain major challenges regardless of method; supervision matters most there.","The Meemi postprocessing step improves alignments on non-comparable corpora when paired with a large clean dictionary, but is harmful with small or noisy dictionaries.","MUSE's sensitivity to seed-dictionary size means that unstable convergence, not just average accuracy, should be part of method comparisons."],"supporting_citations":[{"why":"Provides the MUSE method and the test dictionaries used for bilingual dictionary induction, and is the main target of the paper's challenge to unsupervised alignment.","marker":"Conneau et al. (2018a)"},{"why":"Provides the fully unsupervised VecMap variant whose claimed robustness the paper evaluates and questions.","marker":"Artetxe et al. (2018b)"},{"why":"Supplies the seeded orthogonal Procrustes alignment procedure that both VecMap and MUSE build on.","marker":"Artetxe et al. (2017)"},{"why":"Earlier analysis of limitations of unsupervised bilingual dictionary induction that this paper extends to more systems, tasks, and supervision types.","marker":"Søgaard et al. (2018)"},{"why":"Introduces the Meemi postprocessing technique whose dictionary-size sensitivity and benefit on non-comparable corpora are analyzed.","marker":"Doval et al. (2018)"},{"why":"Concurrent analysis cited as reaching a similar conclusion that fully unsupervised cross-lingual embeddings are not generally needed or reliable.","marker":"Vuli´c et al. (2019)"},{"why":"Comparative evaluation framework for cross-lingual embeddings that the paper contrasts with by going beyond Wikipedia-only training.","marker":"Glavas et al. (2019)"}],"fun_headline_variants":["Unsupervised word alignment crumbles on noisy, distant languages","Dictionary-free cross-lingual embeddings not robust to real-world data","When unsupervised alignment fails: noisy text and linguistic distance","Cross-lingual embeddings need supervision on messier data","Unsupervised alignment only works on clean, close languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison across corpus types assumes that the social-media performance drop is caused by noise and non-comparability, but social-media embeddings are also trained at a lower dimensionality (100 versus 300), so part of the drop could be an artifact of representational capacity.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised word alignment crumbles on noisy, distant languages","Dictionary-free cross-lingual embeddings not robust to real-world data","When unsupervised alignment fails: noisy text and linguistic distance","Cross-lingual embeddings need supervision on messier data","Unsupervised alignment only works on clean, close languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1227,"prompt_tokens":840,"completion_tokens":387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":456,"tokens_out":387,"duration_ms":4332,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:58:12.867128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the social-media monolingual embeddings at 300 dimensions (and perhaps Wikipedia embeddings at 100 dimensions) and repeat the dictionary induction and similarity evaluations; if unsupervised P@1 on noisy text recovers substantially, the corpus-noise explanation loses its main quantitative support. A second check is whether the near-zero Finnish and Farsi results persist when those languages are trained on larger, cleaner corpora of the same type.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the seeded orthogonal Procrustes alignment procedure that both VecMap and MUSE build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Meemi postprocessing technique whose dictionary-size sensitivity and benefit on non-comparable corpora are analyzed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Concurrent analysis cited as reaching a similar conclusion that fully unsupervised cross-lingual embeddings are not generally needed or reliable."},{"cited_title":"How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions","cited_arxiv_id":"1902.00508","evidence_quote":"Comparative evaluation framework for cross-lingual embeddings that the paper contrasts with by going beyond Wikipedia-only training."}],"review_version":1}