{"id":"1b9b96ce-8533-460f-aa46-bd423a98599a","arxiv_id":"2412.16894","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Combining CSCBLI, linear transformation, and the iterative VecMap framework yields the top lexicon-induction accuracy on English with Sinhala, Tamil, and Punjabi, but the gains are small and the reported scores come from tuning on the evaluation sets.","lead":"This paper tests whether stacking several known improvements to unsupervised bilingual lexicon induction works on three low-resource language pairs. It reports small accuracy gains for some combinations and releases two new evaluation dictionaries, for English-Sinhala and English-Punjabi.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-recipe claim rests on tuning and scoring with the same evaluation dictionaries; without a held-out split or significance test, M17's small margins over simpler baselines are not established.","rationale":"The central claim is that combining CSCBLI, Linear Transformation, and UVecMap (M17) is the best recipe. What must be true for that claim is that M17's advantage is real and generalizes beyond the particular evaluation dictionaries used. The weakest spot is exactly that the final Table 5 scores are computed on the same dictionaries used to select alpha values and frequency thresholds, with a single run per configuration and no significance testing; small advantages such as +0.11 on EnTa W2V are within plausible noise. I agree with the reader's weakest assumption. The released EnSi and EnPa dictionaries are a useful independent contribution even if the ranking fails to generalize, but the EnSi dictionary is described as created by machine translation with back-translation filtering, so calling it human-curated overstates its quality ceiling and makes tuning on it doubly risky. The proposed held-out split followed by re-tuning and bootstrap confidence intervals would settle whether M17 remains best; because the reader already conditions acceptance on such validation, the verdict stays CONDITIONAL.","tokens_in":16421,"tokens_out":3500,"duration_ms":31883,"concrete_test":"Before running any further configuration, split each evaluation dictionary once into a tuning split (80%) and a held-out split (20%), stratified by source-word frequency bucket. Re-run the entire selection pipeline in §4.4 (alpha grids in §4.4.1 and frequency-threshold ablation) using only the tuning split, then compute pr@1 for M1, M3, M17, M19, and M20 on the held-out split, with bootstrap 95% confidence intervals over dictionary items. If M17 is not strictly best in at least five of the six language/embedding columns, or if its lead over the next-best configuration is smaller than the bootstrap interval, the paper's central recipe claim does not survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The ranking in Table 5 is produced under selection on the evaluation set. The alpha grids in §4.4.1 (Tables 6–11) and the frequency thresholds in §4.4 (Tables 12–14) were chosen by maximizing pr@1 on the same Section 4.2 dictionaries that are later used as the final test in Table 5. Thus every non-baseline configuration is effectively tuned to those dictionaries; the reported margins are in-sample, not out-of-sample. The margins are also small or tied: M17 beats M1 by only +0.11 on EnTa W2V and +0.45 on EnPa W2V, ties M19/M20 on EnPa W2V and EnPa FastText, and is worse than M3 on EnSi W2V (32.84 vs 33.18). No variance estimate or significance test is reported, so a 0.11-point lead is not distinguishable from dictionary noise. The EnSi evaluation dictionary is machine-translated with back-translation filtering, not human-curated as the contributions state; errors there would bias both tuning and final scores. The conclusion already concedes manual hyper-parameter setting and memory-constrained CSCBLI, but the unaddressed issue is that the final comparison lacks any held-out component.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies unsupervised bilingual lexicon induction (UBLI) for three low-resource language pairs (English-Sinhala, English-Tamil, English-Punjabi) by extending the unsupervised VecMap (UVecMap) framework with techniques from prior work: linear transformation, embedding fusion, iterative and effective dimensionality reduction, and CSCBLI (combining static and contextual embeddings). It reports an extensive grid of 20 method combinations and evaluates them with precision@1 on newly created evaluation dictionaries, concluding that CSCBLI with linear transformation over UVecMap (M17) generally performs best. The paper also releases bilingual dictionaries for English-Sinhala and English-Punjabi.","tokens_in":16843,"tokens_out":4380,"duration_ms":39113,"significance":"If the evaluation were sound, the paper would make a useful practical contribution: it systematically compares combinations of previously independently tested UBLI extensions on genuinely low-resource language pairs, and it releases new evaluation dictionaries for English-Sinhala and English-Punjabi, which are valuable resources. The detailed hyperparameter tables are also a useful record. However, the central claim about the best combination is currently not established because the evaluation protocol selects hyperparameters on the same dictionaries used for final scoring, reports no uncertainty estimates, and contains internal inconsistencies in the baseline numbers. The contribution of the dictionaries is real, but the paper's headline conclusion needs a stronger evaluation design.","major_comments":[{"comment":"The final ranking in Table 5 is in-sample, not an independent test. The alpha values in Table 3 were selected by running UVecMap/CSCBLI on the evaluation dictionaries built in §4.2 (stated in §4.4.1: 'These embeddings were evaluated against the evaluation dictionaries we created'), and the Min_Freq thresholds in §4.4 were likewise chosen through an ablation study whose accuracy columns (Tables 12–14) report pr@1 on those same dictionaries. Consequently, every non-baseline configuration in Table 5 has been tuned to the test set, and the margins over the baseline are not out-of-sample evidence. The problem is compounded by the absence of significance tests or error bars: for example, M17 exceeds M1 by only +0.11 on EnTa Word2Vec and +0.45 on EnPa Word2Vec, while M17 ties M19/M20 on EnPa Word2Vec and EnPa FastText. A held-out split or nested validation, together with variance estimates, is needed before the 'best combination' claim can be supported.","section":"§4.4, §4.4.1, §5, Table 5"},{"comment":"The claim in §5 that 'CSCBLI with Linear Transformation and UVecMap (M17) yielded the best results for both Word2Vec and Fasttext' is contradicted by the paper's own Table 5. For EnSi Word2Vec, M3 (Linear Transformation + UVecMap) achieves 33.18, which is higher than M17's 32.84; the same table also shows M17 tied with M19 on EnPa Word2Vec and with M20 on EnPa FastText. The sentence immediately following acknowledges the EnSi exception, but the generalized conclusion 'best results for both Word2Vec and Fasttext' is therefore not supported. The recommendation should be rephrased to identify a family of competitive configurations rather than a single winner.","section":"§5, Table 5"},{"comment":"The paper states as a contribution that it releases 'human-curated bilingual lexicons' for English-Sinhala and English-Punjabi, but §4.2.1 describes the EnSi dictionary as generated by machine translation (Google Translate) followed by back-translation filtering and automatic consistency checks, with no human verification step. Since this dictionary is used both for hyperparameter selection and for the final accuracy numbers, the contradiction matters: machine-translated evaluation data can carry systematic errors that bias the ranking. The authors should either add genuine human curation and document it, or explicitly describe the EnSi dictionary as machine-translated with automatic filtering and discuss the quality ceiling this imposes.","section":"§4.2.1 and Contributions (Abstract, §1)"},{"comment":"The frequency-threshold ablation tables are inconsistent with the baseline results in Table 5. For EnSi, Table 14 with Min_Freq=8 reports Word2Vec=27.48 and FastText=31.49, while Table 5 M1 reports Word2Vec=31.49 and FastText=27.48 — the values are swapped. For EnTa, Table 12 with Min_Freq=6 reports Word2Vec=15.39 and FastText=12.81, while Table 5 M1 reports 16.74 and 11.69. These discrepancies make it impossible to verify which threshold configuration produced the reported baselines, and they undermine the claim that the thresholds were 'carefully selected' as described. Please align the appendix with the final experimental configuration and state explicitly which threshold was used for each language pair and embedding type.","section":"Appendix B, Tables 12–14; Table 5"},{"comment":"Several combinations collapse to near-zero accuracy on EnTa FastText (e.g., M2=0.11, M4=0, M7=14.16, M10=0, M18=0), and the paper offers only a qualitative explanation that 'the mapping relies heavily on a few key embeddings.' This instability is itself a threat to the reliability of the ranking: small changes in preprocessing can produce total mapping failure, so differences of a few points between surviving configurations may reflect initialization or numerical sensitivity rather than genuine method quality. A robustness analysis (e.g., multiple random seeds or multiple training runs) is needed to establish that the reported ordering is stable.","section":"§5, Table 5"}],"minor_comments":[{"comment":"The text says dimensionality reduction 'halving the original dimensions' but gives no ablation or justification for choosing 150 (Word2Vec/FastText) and 512 (XLM-R) rather than other target dimensions; a brief explanation or reference would be helpful.","section":"§4.4.2"},{"comment":"The line 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' appears to be a leftover template artifact and should be removed.","section":"Table 9"},{"comment":"The notation for the spring network weights γ1 and γ2 is introduced as vectors initialized to zero, but the subsequent equations use scalar-looking multiplication; clarifying whether the multiplication is elementwise would improve readability.","section":"§3.3"},{"comment":"The spelling 'Fasttext' and 'FastText' is used inconsistently; please standardize.","section":"Throughout"},{"comment":"The phrase 'best result is in boldface, second best is in italics' is not consistently applied in Table 5: for EnSi Word2Vec the best is M3 (33.18), and M17 (32.84) is the second best, so it should be italicized; please check all columns.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper's resource contribution is real, but the evaluation design has a load-bearing circularity: hyperparameters are tuned on the same dictionaries that are later used as the test set, and no uncertainty estimates are provided. This is fixable with a held-out split, significance testing or repeated runs, and clarification of the machine-translated EnSi dictionary. The internal inconsistencies between Appendix B and Table 5 need to be resolved before the paper can be accepted. I would support major revision rather than rejection because the underlying comparison framework and the released dictionaries are valuable, and the methodological gaps are addressable within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the new Sinhala-English and Punjabi-English evaluation dictionaries are the real contribution. The paper's claimed best recipe (M17) is not established because the authors tuned alpha values and frequency thresholds on the very dictionaries they then use for final scoring. That's a genuine circularity, and the stress-test note is right to flag it.\n\nWhat's new: a systematic scan of 20 combinations of published extensions on UVecMap for three low-resource pairs, plus two new evaluation sets. Sinhala and Punjabi are underserved, and a public eval set for either is useful to the community. The paper is also transparent about manual hyper-parameter setting and GPU memory limits, which is more than many papers do.\n\nSoft spots, in proportion: the selection-on-test problem is load-bearing. Section 4.4.1 says alpha grids were evaluated against the evaluation dictionaries; Section 4.4 says Min_Freq thresholds came from an ablation study on the same eval sets. So Table 5 rankings are in-sample, not out-of-sample. The margins are small: M17 beats M1 by 0.11 on EnTa W2V and 0.45 on EnPa W2V, ties M19/M20 on EnPa W2V and FastText, and loses to M3 on EnSi W2V. No significance tests or error bars, so those gaps are noise-level. The EnTa FastText collapses to zero for several configs adds further instability. And the EnSi dictionary is called human-curated in the contributions but the methodology describes Google Translate with back-translation filtering. That mismatch should be fixed.\n\nNone of this invalidates the resource value. The evaluation dictionaries are useful even if the ranking doesn't generalize. But the best-recipe claim needs held-out tuning and a significance check before I'd trust it.\n\nRecommendation: send to peer review. A serious referee can insist on a held-out split and cleaned-up claims. The dictionaries alone justify the referee time.\n\nBest","headline":"The combination ranking is undercut by tuning on the test dictionaries, but the new En-Si and En-Pa evaluation sets are worth having.","tokens_in":17226,"tokens_out":2667,"would_cite":true,"duration_ms":23951,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that stacking CSCBLI on a linear-transformed UVecMap yields the best unsupervised bilingual lexicon induction for three low-resource language pairs.","keywords":["bilingual lexicon induction","unsupervised BLI","low-resource languages","VecMap","word embeddings","contextual embeddings","Sinhala","Tamil"],"falsifier":"Build a held-out, human-verified test set for English–Sinhala, English–Tamil, and English–Punjabi that is never touched during alpha or frequency tuning, then re-run M17 and the M1 baseline. If M17's advantage over M1 disappears or reverses on the held-out set, or if a different combination wins there, the paper's central claim that M17 is the best recipe would be falsified.","tokens_in":1793,"feed_emoji":"🔤","tokens_out":2548,"duration_ms":59332,"temperature":0.7,"pith_summary":"The paper asks whether improvements to structure-based unsupervised bilingual lexicon induction (UBLI), developed and tested separately, still help when stacked onto the same framework. On three low-resource pairs – English–Sinhala, English–Tamil, and English–Punjabi – the authors extend the unsupervised VecMap framework (UVecMap) with embedding-creation, pre-processing, and initialization upgrades, including their own PCA-based dimensionality reduction. They report that the best overall accuracy comes from CSCBLI, a method that adds XLM-R contextual representations on top of static embeddings, combined with a linear transformation and UVecMap (combination M17), for both Word2Vec and FastText embeddings. The paper also releases human-curated evaluation dictionaries for English–Sinhala and English–Punjabi, and it treats the question of whether simultaneous use produces equal gains as an empirical one answered by their experiments.","feed_headline":"Stacked extensions beat solo recipes in unsupervised word translation","feed_subtitle":"Combining static-plus-contextual embeddings with linear-transformed VecMap wins on Sinhala, Tamil, Punjabi; datasets released.","key_machinery":"The central object is the UVecMap framework, an unsupervised self-learning method that iterates two steps: an optimal orthogonal mapping between the source and target embedding spaces, obtained via SVD of $X^T D Z$, and a new dictionary induced over the mapped similarities using CSLS retrieval. Onto this base, the paper stacks three families of extensions: a linear transformation of the input monolingual spaces, based on $n$-th order similarity matrices $M_n(X) = (XX^T)^n$ parameterized by $\\alpha$; CSCBLI, which builds a unified word representation by adding contextual offsets from XLM-R through a spring network and then interpolates similarity with a weight $\\lambda$; and PCA-based dimensionality reduction applied either as pre-processing or iteratively during initialization. The best recipe, M17, is the specific pipeline where CSCBLI runs on top of a linear-transformed UVecMap alignment.","core_discovery":"The paper's central claim is that combining multiple independently proposed extensions inside one structure-based UBLI pipeline gives a recipe that outperforms the baseline UVecMap and most individual extensions. In their experiments, the best configurations were M17 (CSCBLI + Linear Transformation + UVecMap) for both Word2Vec and FastText, with M19 and M20 close behind; the one exception was English–Sinhala with Word2Vec, where M3 (linear transformation alone) won. The gains over the baseline are positive but modest, ranging roughly from 0.1 to 3.6 absolute points of precision@1. The paper also reports that embedding fusion consistently hurt performance, and that several combinations on English–Tamil FastText collapsed to near-zero accuracy, showing that stacked extensions can destabilize alignment rather than improve it.","pith_inferences":["The gains of M17 over the baseline are small enough that tuning on the same test dictionaries may be inflating the recipe's apparent advantage; a held-out human-verified test set could reveal whether the stacked pipeline truly generalizes.","Since the English–Sinhala evaluation dictionary was created by machine translation with back-translation filtering, its quality ceiling is uncertain, and part of the reported accuracy on that pair may reflect quirks of the translation tool rather than real lexical correspondence.","The method's dependence on manually tuned alpha values per language pair suggests a natural extension: learn alpha from intrinsic monolingual statistics, such as the spectral properties of the embedding spaces, to make the recipe applicable to new low-resource languages without additional tuning effort.","The three tested pairs all involve English and South Asian languages; testing M17 on typologically more distant or script-divergent pairs would clarify whether the stacked combination is broadly useful or pair-specific."],"forward_implications":["Practitioners working on low-resource language pairs can adopt M17 as a default unsupervised BLI recipe when static Word2Vec or FastText embeddings and XLM-R contextual embeddings are available.","The released human-curated dictionaries for English–Sinhala and English–Punjabi give the community evaluation sets for two low-resource languages that previously relied on automatically built lexicons.","The frequent collapse to zero accuracy on English–Tamil FastText shows that stacking extensions is not universally safe, so each combination should be validated before use.","Because the margins over the baseline are small and the hyperparameters were selected on the same evaluation dictionaries, the reported ranking should be re-checked against a genuinely held-out human dictionary."],"supporting_citations":[{"why":"Supplies the UVecMap baseline, the unsupervised self-learning framework that all combinations build on.","marker":"[3]"},{"why":"Contributes the CSCBLI method of combining static and contextual embeddings, which is the central extension in the best-performing M17 recipe.","marker":"[50]"},{"why":"Introduces the linear transformation post-processing of monolingual spaces that M17 stacks with CSCBLI.","marker":"[45]"},{"why":"Provides the mathematical underpinning, the $n$-th order similarity transformation $M_n(X) = M_1(X R^{(n-1)/2})$, used by the linear transformation extension.","marker":"[5]"},{"why":"Provides the CSLS retrieval method that UVecMap uses when inducing the dictionary from aligned embeddings.","marker":"[27]"},{"why":"Supplies XLM-R, the contextual embedding model that CSCBLI uses alongside static Word2Vec or FastText embeddings.","marker":"[14]"},{"why":"Proposes dimensionality reduction during initialization, which the paper adapts and also tests as a separate preprocessing step.","marker":"[28]"},{"why":"Introduces the original VecMap supervised framework that the unsupervised version extends.","marker":"[2]"}],"fun_headline_variants":["Stacking extensions improves low-resource word translation","Combining methods wins for Sinhala, Tamil, Punjabi lexicons","Best recipe found for unsupervised bilingual lexicons","Extensions stack up for low-resource lexicon induction","Modest gains from stacked BLI extensions on three LRLs"],"cache_read_input_tokens":19328,"weakest_assumption_plain":"The entire ranking of combinations is computed on evaluation dictionaries that were also used to pick frequency thresholds and alpha values, and for English–Sinhala that dictionary was produced by machine translation with back-translation filtering; if those lists are biased or noisy, the reported best combination may not be the best on genuinely held-out data.","fun_headline_variants_meta":{"raw":{"variants":["Stacking extensions improves low-resource word translation","Combining methods wins for Sinhala, Tamil, Punjabi lexicons","Best recipe found for unsupervised bilingual lexicons","Extensions stack up for low-resource lexicon induction","Modest gains from stacked BLI extensions on three LRLs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1555,"prompt_tokens":910,"completion_tokens":645,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":565}},"tokens_in":526,"tokens_out":645,"duration_ms":5904,"temperature":1.0,"reasoning_tokens":565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T06:00:04.464080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a held-out, human-verified test set for English–Sinhala, English–Tamil, and English–Punjabi that is never touched during alpha or frequency tuning, then re-run M17 and the M1 baseline. If M17's advantage over M1 disappears or reverses on the held-out set, or if a different combination wins there, the paper's central claim that M17 is the best recipe would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the UVecMap baseline, the unsupervised self-learning framework that all combinations build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the CSCBLI method of combining static and contextual embeddings, which is the central extension in the best-performing M17 recipe."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the linear transformation post-processing of monolingual spaces that M17 stacks with CSCBLI."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the mathematical underpinning, the $n$-th order similarity transformation $M_n(X) = M_1(X R^{(n-1)/2})$, used by the linear transformation extension."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CSLS retrieval method that UVecMap uses when inducing the dictionary from aligned embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies XLM-R, the contextual embedding model that CSCBLI uses alongside static Word2Vec or FastText embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes dimensionality reduction during initialization, which the paper adapts and also tests as a separate preprocessing step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the original VecMap supervised framework that the unsupervised version extends."}],"review_version":1}