{"id":"95b18412-ba3e-4ce3-893f-1a0e492132b1","arxiv_id":"2506.13020","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For English-Yoruba word translation, normalization greatly improves precision for curated embeddings but only slightly for Wikipedia embeddings.","lead":"This paper applies a standard supervised embedding alignment method to translate words from English to Yoruba, comparing two Yoruba embedding sources with and without normalization. It reports that both embedding quality and normalization affect translation precision, with an interaction between the two.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training and evaluation dictionaries may overlap; without an explicit disjointness guarantee, reported precisions could reflect memorization rather than transfer.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Section 2.1 derives both the training and evaluation dictionaries from Alabi et al. (2020) without stating that overlapping English-Yoruba pairs were removed. This is the most direct threat to the paper's central claim, because the reported precision values are the only quantitative evidence for the interaction between embedding quality and normalization. If evaluation pairs appear in the training dictionary, the Procrustes alignment has effectively seen the answers, and the 19.38% top-1 precision for curated-normalized embeddings could be inflated by memorization rather than measuring genuine cross-lingual transfer. This concern is concrete and testable: an explicit overlap check and a clean re-run on the disjoint subset would settle it. A secondary concern is the absence of error bars or repeated trials for the four conditions, since each condition is a single deterministic run, but that issue is less fundamental than potential train/eval leakage. If the overlap check finds no intersection, the paper's main claims survive this particular objection; if overlap is found, the reported numbers need substantial revision. Because the reader already issued a conditional verdict and identified this same weakness, the stress-test does not change the recommended verdict. The appropriate response is to require the overlap check and, if needed, a reanalysis on the non-overlapping evaluation pairs before the empirical claims are accepted.","tokens_in":4577,"tokens_out":3383,"duration_ms":37125,"concrete_test":"Apply the same deduplication and diacritic normalization described in Section 2.1 to both dictionaries and compute the set intersection of (source English, target Yoruba) pairs between the 3,693 training pairs and the 161 evaluation pairs. If the intersection is nonempty, rerun the four alignment conditions with all intersecting pairs, or at minimum all English source words appearing in the evaluation set, removed from the training dictionary, and report precisions on the remaining evaluation pairs. If the intersection is empty, report that explicit disjointness check in the paper. If curated-normalized top-1 precision drops materially below 19.38% after removal, then overlap, rather than embedding quality and normalization, was driving the headline result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on the 161-pair evaluation dictionary being disjoint from the 3,693-pair supervised training dictionary. Section 2.1 describes both as derived from Alabi et al. (2020): the training set is scraped from their online English-Yoruba dictionary, and the evaluation set comes from their Yoruba translation of WordSim353. The paper nowhere states that overlapping pairs, or even overlapping English source words, were removed. Since common words such as 'sea' are likely to appear in both resources, the curated-normalized condition's top-1 precision of 19.38% may be inflated because the Procrustes mapping already saw those test pairs during training. The claimed interaction effect, in which normalization helps the curated embeddings much more than the wiki embeddings, is only credible if it persists after removing all train/eval overlap. If no overlap exists, the concern evaporates; if overlap exists, all four conditions' precision values, and especially the memorization-prone curated conditions, must be recomputed on the disjoint evaluation subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a supervised embedding-alignment study for English-to-Yoruba word translation. The authors align English fastText embeddings with two Yoruba embedding sources (a Wikipedia fastText embedding and a larger curated embedding from Alabi et al. 2020) under two preprocessing conditions (with or without centering and normalization), yielding four conditions. They evaluate top-1/5/10 translation precision on a 161-pair English-Yoruba evaluation set derived from WordSim353 and report that normalization improves precision, with a larger gain for the curated embeddings than for the Wikipedia embeddings; they interpret this as an interaction effect between embedding quality and normalization.","tokens_in":4741,"tokens_out":4756,"duration_ms":52448,"significance":"If the reported results were statistically robust and free of data leakage, the paper would provide a useful, if preliminary, demonstration that for low-resource languages the quality of monolingual embeddings and the preprocessing of embedding spaces matter more than the alignment algorithm itself, challenging a direct transfer of high-resource results. The study is fully empirical, uses publicly available data, and reports concrete numbers that could be reproduced. However, the central claims are currently undermined by the likely overlap between the training and evaluation dictionaries and by the absence of any uncertainty quantification or significance testing; these issues must be resolved before the interaction effect can be taken as established.","major_comments":[{"comment":"The training dictionary (3,693 pairs scraped from Alabi et al.'s online dictionary) and the evaluation dictionary (161 pairs from Alabi et al.'s Yoruba translation of WordSim353) are both derived from Alabi et al. (2020), and the paper does not state that the two sets are disjoint. If common English-Yoruba pairs, or even English source words, appear in both resources, the reported top-k precision values, especially the curated-normalized 19.38% at k=1, could reflect memorization during supervised Procrustes alignment rather than genuine transfer. The authors should either verify and report that no overlapping pairs (or overlapping English source words) exist between the two resources, or remove all overlaps from the evaluation set and recompute all four conditions on the disjoint subset; without such a check the main quantitative claim is not trustworthy.","section":"Section 2.1"},{"comment":"Each condition is evaluated on only 161 word pairs with a single run, and no error bars, confidence intervals, or significance tests are reported. The claimed interaction effect rests entirely on point estimates; for example, the key contrast between curated-normalized (19.38%, roughly 31/161) and wiki-normalized (12.42%, roughly 20/161) is a difference of about 11 correct translations, and a two-proportion z-test would not reject equality at the 5% level. The authors should provide at least bootstrap confidence intervals or a permutation test for the precision differences, and ideally multiple training/evaluation splits, to establish that the interaction is not an artifact of small-sample variation.","section":"Section 3, Table 1"},{"comment":"The paper describes the method as following Conneau et al. (2017) but does not state whether the evaluation used the CSLS (cross-domain similarity local scaling) retrieval metric that the MUSE system employs; it appears to use plain nearest-neighbor retrieval. Since Section 4.1 directly compares the obtained absolute precision values to MUSE's published results for Spanish, German, and Chinese (77.4, 68.4, and 40.6), the difference in retrieval metric could account for part of the gap. The authors should specify the retrieval procedure precisely and, if CSLS was not used, avoid direct numerical comparison with MUSE results without noting this protocol difference.","section":"Section 2.2 and Section 4.1"}],"minor_comments":[{"comment":"The Procrustes problem is written as an arg max over W of ||WX - Y||_F, but the goal is to minimize that Frobenius norm; the displayed optimization should be arg min, and the stated SVD solution is for the minimizing orthogonal matrix.","section":"Section 2.2, Eq. (1)"},{"comment":"The sentence \"Overall, the wiki-unnormalized condition consistently yields the lowest precision, whereas the curated-normalized condition the highest\" is missing a verb; it should read \"and the curated-normalized condition yields the highest precision.\"","section":"Section 3"},{"comment":"\"The top retrievals of the curated-unnormalized model, however, bares no visible semblance to the meaning of 'sea'\" contains subject-verb agreement and word-choice errors; it should be \"bear no visible resemblance.\"","section":"Section 3"},{"comment":"The authors write that they \"scrape the online English-Yoruba dictionary Alabi et al. (2020)\"; Alabi et al. (2020) is a research paper, not the dictionary itself, so please identify the actual web resource being scraped and provide its URL or a data availability statement.","section":"Section 2.1"},{"comment":"The normalization step is described only as \"center and normalize the embeddings using the tensor operations from the PyTorch package\"; please specify whether \"normalize\" means L2 unit-length normalization and whether centering is mean subtraction across the vocabulary for each dimension, as these choices affect reproducibility.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a short empirical report, possibly from a course project, and its novelty is limited relative to a full research paper. The methodological gaps identified in the major comments — data overlap and lack of statistical support — are fixable within the manuscript's scope, but until they are addressed the central claims are not supported. If the authors can show disjointness and provide uncertainty estimates, a revised version could be acceptable as a workshop-style empirical contribution. The direct comparison to MUSE numbers should also be toned down."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a small, honest experiment applying supervised Procrustes alignment to English–Yoruba, comparing two embedding sources and normalization. The value is in the concrete low-resource setting and the clear reporting of weak precision numbers. The soft spot is the one that matters: the training and evaluation dictionaries are both derived from Alabi et al. (2020), and the paper never states that overlapping pairs were removed. That makes the reported interaction effect—normalization helping the curated embedding much more than the wiki embedding—hard to credit. ‘Sea’ is a perfectly common word; if any of the 161 eval pairs appear in the 3,693 training pairs, those results are inflated by memorization, and the curated conditions are precisely the ones that would benefit most.\n\nWhat the paper does well: it takes a genuinely low-resource language pair, uses two publicly available embeddings, and honestly reports that even the best condition gets only 19.4% top-1 precision. The qualitative example with ‘sea’ is helpful, though not systematic. The discussion of why normalization might matter more for the curated embedding is plausible.\n\nThe flaws are in proportion. There are no error bars, each condition ran once, and no significance test. That alone would make the interaction claim premature. But the bigger problem is the dictionary overlap. If the authors can show the evaluation set is disjoint from training, or recompute on the disjoint subset, then the paper becomes a useful descriptive study. As it stands, the numbers should not be taken at face value. There is also no code or data, which makes the single-run result impossible to check.\n\nThe paper is worth a serious referee because the topic matters and the experiment is easy to fix. My recommendation: send it to review, but with a clear request to address the overlap, run multiple seeds, and share code/data. If the authors cannot rule out overlap, they should drop the interaction claim and present this as a case study of low-resource alignment challenges.\n\nFor a reading group, it might be useful as a cautionary example of how easy it is to accidentally leak test data into training in low-resource settings.","headline":"An honest small-scale study whose headline interaction effect is not credible until train/eval overlap is ruled out.","tokens_in":5240,"tokens_out":2623,"would_cite":false,"duration_ms":27773,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Normalizing embeddings nearly triples top-1 English-Yoruba word translation precision.","keywords":["embedding alignment","Procrustes alignment","word translation precision","Yoruba","low-resource language","fastText embeddings","embedding normalization","English-Yoruba translation"],"falsifier":"Re-run the four alignments after deleting from the 161-pair evaluation set every word pair that also appears in the 3,693-pair training set; if curated-normalized top-1 precision drops well below 19.38% while the unnormalized conditions do not, the claimed interaction effect is an artifact of dictionary overlap.","tokens_in":4384,"feed_emoji":"🌍","tokens_out":6894,"duration_ms":62852,"temperature":0.7,"pith_summary":"The paper asks whether supervised embedding alignment, a method that maps two monolingual word-vector spaces onto each other without parallel corpora, can work for a low-resource language pair. It tests English-to-Yoruba translation with four alignments that vary the Yoruba embedding source (noisy Wikipedia versus a curated, diacritic-corrected corpus) and whether embeddings are normalized before Procrustes alignment. The result is that both factors matter, and they interact: normalization lifts curated precision from 6.88% to 19.38% at k=1, while its effect on the Wikipedia embedding is far smaller. The paper concludes that the state-of-the-art method transfers to Yoruba only when the monolingual embeddings are high quality and put on the same scale, and that low-resource language work must therefore attend to embedding curation, not just the alignment algorithm.","feed_headline":"Normalizing embeddings nearly triples top-1 English-Yoruba precision","feed_subtitle":"Curated Yoruba embeddings jump from 6.88% to 19.38% top-1 precision after normalization, while Wikipedia embeddings gain less.","key_machinery":"The load-bearing object is the orthogonal mapping matrix $W$ found by Procrustes alignment: $W^* = \\arg\\max_{W\\in O_d(\\mathbb{R})} \\|WX-Y\\|_F = UV^T$, where $U\\Sigma V^T = \\mathrm{SVD}(YX^T)$. Given a set of anchor word pairs, the method rotates the source embedding space $X$ onto the target space $Y$ so that nearest-neighbor retrieval in the target space acts as translation. The second piece of machinery is the pre-alignment centering and normalization applied to half the conditions, which puts the two vector distributions on a common scale and unlocks the curated embedding's advantage.","core_discovery":"On the paper's own terms, the discovery is that supervised Procrustes alignment produces low but usable word translation precision for English-Yoruba only under the right conditions. Normalizing the embedding vectors before alignment increases precision for both embedding sources, and the gain is much larger for the curated Yoruba embedding (from 6.88% to 19.38% top-1) than for the Wikipedia embedding (from 10.56% to 12.42%). The authors interpret this interaction as reflecting the initial similarity of the two spaces: the Wikipedia embeddings are both trained with the same widely used embedding method and thus closer in shape, while the curated Yoruba embedding is trained on a distinct, higher-quality corpus, so it needs normalization to map onto English. Across all conditions, the top-1 precision stays below the published figures for high-resource language pairs, which the authors attribute to embedding size, data quality, and morphological distance.","pith_inferences":["The reported precision likely depends on the separation between the training and evaluation dictionaries; if overlapping English-Yoruba pairs were not removed, a strict holdout replication could find lower, though still positive, precision.","A direct extension would test whether subword-aware or contextualized Yoruba embeddings trained on the same curated corpus beat word-level vectors, and whether normalization still matters for them.","The factorial design (embedding source × normalization) could be rerun on other low-resource pairs to see whether the interaction generalizes or is an artifact of the particular Wikipedia-versus-curated contrast studied here.","If the interaction is general, then the practical bottleneck for low-resource embedding alignment is monolingual data curation, not the alignment algorithm itself."],"forward_implications":["For low-resource pairs, supplying larger and diacritically correct monolingual embeddings raises nearest-neighbor translation precision without any parallel training data.","Normalizing before alignment is not optional; it yields a large gain for embeddings trained on different corpora and a smaller but positive gain for same-method embeddings.","The interaction between embedding source and normalization means that these two factors cannot be tuned independently when building a low-resource translation system.","The best English-Yoruba condition still translates only about a fifth of source words correctly at k=1, so at current resource levels this method is a baseline, not a deployment-ready translator.","Supervised embedding alignment with a human-created dictionary is a workable starting point for English-Yoruba, with top-1 precision of 19.38% in the curated-normalized condition."],"supporting_citations":[{"why":"Supplies the supervised Procrustes alignment method and the high-resource language precision baselines the paper compares against.","marker":"Conneau et al. (2017)"},{"why":"Provides the curated Yoruba embedding trained on diacritic-corrected news and web corpora and the ground-truth and evaluation dictionaries used for training and testing.","marker":"Alabi et al. (2020)"},{"why":"Describes the embedding model that produced the Wikipedia Yoruba vectors, one of the two embedding sources tested.","marker":"Bojanowski et al. (2017)"},{"why":"Supplies the pretrained Wikipedia embeddings and the token-count figures that quantify the size gap between Yoruba and English.","marker":"Mikolov et al. (2018)"},{"why":"Provides the pretrained English word vectors used as the source embedding in all alignments.","marker":"Mikolov et al. (2017)"}],"fun_headline_variants":["Normalizing embeddings nearly triples curated Yoruba precision","Curated Yoruba embeddings get biggest lift from normalization","Normalization boosts English-Yoruba translation most for curated data","Low-resource word translation improves with embedding normalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported precision figures assume the 161-pair evaluation dictionary and the 3,693-pair training dictionary are effectively disjoint; if the same English-Yoruba pairs appear in both, the numbers are inflated by memorization of anchor points instead of genuine transfer.","fun_headline_variants_meta":{"raw":{"variants":["Normalizing embeddings nearly triples curated Yoruba precision","Curated Yoruba embeddings get biggest lift from normalization","Normalization boosts English-Yoruba translation most for curated data","Low-resource word translation improves with embedding normalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":3133,"prompt_tokens":917,"completion_tokens":2216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2152}},"tokens_in":533,"tokens_out":2216,"duration_ms":16988,"temperature":1.0,"reasoning_tokens":2152,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:35:58.081131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four alignments after deleting from the 161-pair evaluation set every word pair that also appears in the 3,693-pair training set; if curated-normalized top-1 precision drops well below 19.38% while the unnormalized conditions do not, the claimed interaction effect is an artifact of dictionary overlap.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained Wikipedia embeddings and the token-count figures that quantify the size gap between Yoruba and English."}],"review_version":1}