{"id":"3cf4570e-0e3a-4739-95a0-871a2860147e","arxiv_id":"2411.15768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A frequency-aware extension of orthogonal Procrustes alignment finds old and new Turkish synonym pairs better than a linear transformation baseline on a 221-pair test set.","lead":"This paper tests two ways to find Turkish words that replaced older Arabic or Persian words after Turkey's 20th century language reform. It aligns word embeddings from different decades and adds a frequency-correlation re-ranking step, reporting gains over a baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set construction selects on the frequency signal that OP+SC exploits, so the reported advantage over LT may largely reflect evaluation bias rather than generalizable detection ability.","rationale":"The reader's weakest_assumption identifies the same issue: the test set is constructed from JSD-ranked words, which is the same frequency signal OP+SC uses for re-ranking. I agree this is the most load-bearing concern because it attacks the external validity of the main comparison, not just a single number. The impossible MRR values are a separate alarm that the reported metrics are unreliable. However, the methods themselves are clearly described and the code is shared, and the limitation section acknowledges corpus-specific noise. A conditional verdict is appropriate: the authors should re-evaluate on a frequency-neutral test set and correct the metric computations before the claimed superiority can be accepted. Since the reader already reached this conclusion, no verdict change is needed.","tokens_in":8228,"tokens_out":6709,"duration_ms":55945,"concrete_test":"Build a frequency-neutral test set: e.g., take the TDK dictionary's historical equivalents (eski/yeni sözcük pairs) or have annotators match modern words to 1930s words from a random sample of vocabulary, without access to frequency divergence statistics. Run the released OP, OP+SC, and LT pipeline on this set and compare Recall@10 and MRR. If OP+SC's margin over LT shrinks to near zero (or reverses), the reported gains are an artifact of the original test set's selection on frequency divergence. Additionally, recompute MRR from the per-query ranks in the released code to confirm the inconsistency between MRR and Recall@1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation set in Section V-A is built by ranking the top 5,000 words by Jensen-Shannon divergence between the 1930s and 1980s unigram distributions, and then manually pairing words whose frequencies changed sharply. The OP+SC method (Section IV-B) re-ranks candidate neighbors by Spearman's rank correlation of frequency time series. Thus the test pairs are selected precisely for the property OP+SC is designed to exploit: a strong divergence in relative frequencies. The LT baseline (Section V-A) uses only a linear transformation and does not receive frequency information, so it is systematically handicapped on this test set. If the test set had been assembled from an external lexicon (e.g., TDK's lists of replaced words) or by annotators who did not consult frequency divergence, the margin in Table I (Recall@10 0.74 vs. 0.34; MRR 0.81 vs. 0.17) could shrink materially. Compounding this, the reported MRR values (0.81 for CBOW, 0.85 for SVD) are mathematically impossible given the same table's Recall@1 values (0.35 and 0.33): with one correct answer per query, MRR is bounded above by (1+Recall@1)/2, which yields maxima of 0.675 and 0.665. This internal inconsistency signals that the evaluation pipeline or its reporting contains a bug, further reducing confidence in the quantitative comparison. The central claim therefore rests on a potentially circular evaluation and at least one erroneous metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the task of detecting synonyms used in different time periods for Turkish, a language that underwent rapid lexical change during the 20th-century language reform. The authors propose two methods: Orthogonal Procrustes alignment (OP), which aligns embedding spaces from two time periods via an orthogonal transformation, and OP+SC, which extends OP by re-ranking the nearest neighbors according to Spearman rank correlation of the words' frequency time series. They evaluate on the Turkronicles diachronic corpus using CBOW and SVD embeddings, comparing against a linear transformation (LT) baseline from Zhang et al. (2015). The reported results (Tables I and II) claim that OP+SC outperforms LT and OP on Recall@1, Recall@10, and MRR, and that the methods are more robust than LT when the target period shifts from the 1960s to the 1980s.","tokens_in":8516,"tokens_out":3196,"duration_ms":28526,"significance":"If the reported results are trustworthy, this would be a useful contribution to diachronic NLP for Turkish, offering an automatic way to pair obsolete words with their modern replacements (e.g., vesika–belge). The work is the first computational study of Turkish neologism of this type, and the authors share their code for reproducibility. The methodological idea of combining orthogonal alignment with frequency-based re-ranking is natural and clearly presented. However, the quantitative evaluation contains at least one internal inconsistency and a potential circularity, so the central claim depends on fixes that the authors should be able to make within the scope of a revision.","major_comments":[{"comment":"The reported MRR values are mathematically inconsistent with the reported Recall@1 values. With a single correct answer per query, MRR for a set of queries is bounded above by (1 + Recall@1) / 2, since the reciprocal rank is 1 for queries answered at rank 1 and at most 1/2 for all others. For CBOW (Table I), Recall@1=0.35 gives a maximum possible MRR of 0.675, but the table reports MRR=0.81. For SVD (Table II), Recall@1=0.33 gives a maximum of 0.665, but MRR=0.85 is reported. This indicates a bug in the evaluation pipeline or in the reporting of the metrics. Because the central claim that OP+SC outperforms the baseline rests on these numbers, the authors must re-run the evaluation and ensure the metrics are computed consistently with the single-ground-truth ranking setup described in Section V-A.","section":"Section V-B, Tables I and II"},{"comment":"The test set construction and the signal exploited by OP+SC overlap in a way that inflates the reported advantage. The test set is built by ranking the top 5,000 words by Jensen–Shannon divergence between the base (1930s) and target (1980s) unigram distributions and manually pairing words whose frequencies changed sharply. OP+SC then re-ranks candidate neighbors by Spearman's rank correlation of the frequency time series (Section IV-B). This means that the test pairs are selected precisely for the property that OP+SC is designed to exploit, while the LT baseline (Section V-A) receives no frequency information and is thus systematically handicapped on this test set. To support the claim of generalizable detection ability, the evaluation should be repeated on a test set constructed without reference to frequency divergence, for example by using official TDK lists of replaced words or by having annotators create pairs while blind to frequency statistics. If that is not feasible, the authors should at least report results on a subset of pairs that are not selected by frequency divergence and discuss the potential bias.","section":"Section V-A and Section IV-B"},{"comment":"No error bars, confidence intervals, or significance tests are provided for any of the metrics in Tables I–II or Figures 2–3. With only 221 test pairs (220 in the temporal-distance experiment), the differences between methods—for instance Recall@1 = 0.33 vs. 0.35 between OP and OP+SC in Table I—may be within the noise of the evaluation. The authors should include bootstrap confidence intervals or a paired significance test (e.g., Wilcoxon signed-rank on the ranks) to establish that the observed improvements are not due to chance. This is particularly important given the small test set and the high variance typical of embedding-based retrieval.","section":"Section V-B"}],"minor_comments":[{"comment":"Equation (1) uses the notation E_Tb and E_Tt for the intersection embeddings, but the preceding paragraph uses W_Tb and W_Tt; please harmonize the notation, and clarify that K = W_Tb W_Tt^T is the cross-covariance matrix whose SVD yields U and V, with the optimal orthogonal matrix Q = U V^T.","section":"Section IV-A"},{"comment":"In the paragraph discussing temporal-distance results, the text reads 'the performance of LS tends to decrease'; this appears to be a typo for 'LT'.","section":"Section V-B"},{"comment":"Reference [13] and reference [15] are the same paper (Hamilton et al., 'Diachronic word embeddings reveal statistical laws of semantic change'); the duplicate should be removed. Also, reference [20] contains a placeholder '[ ?]' that should be completed, and several reference entries are missing volume/page information.","section":"References"},{"comment":"The description of the test set construction states that the base period is 1930 and the target period is 1980, but the experiments actually use the decade spans 1930–1939 and 1980–1989; this should be stated precisely at first mention to avoid ambiguity.","section":"Section V-A"}],"recommendation":"major_revision","confidential_remarks":"The MRR inconsistency is a serious red flag that the authors must address before the paper can be considered publishable. The circularity concern raised by the test-set construction is also substantial and, if left unaddressed, would weaken the paper's central claim. I recommend major revision: the technical core is plausible and the problem is timely, but the evaluation needs to be reworked and re-reported. The authors should also be urged to release the test set and a clear evaluation script so that the metrics can be independently verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a modest but legitimate diachronic NLP paper for Turkish. The genuinely new pieces are the first computational test set for Turkish neologisms (221 manually curated pairs like belge/vesika) and the OP+SC extension that re-ranks Procrustes neighbors by Spearman correlation of frequency trajectories. The code is shared. That is a real, if small, contribution.\n\nWhat holds up: the OP method is a clean application of Hamilton et al.'s alignment, and it beats the linear transformation baseline even without the Spearman re-ranking. So the central claim about Procrustes alignment is probably robust. The paper is clearly written and the setup is transparent enough to identify the problems.\n\nThe first problem: the MRR values in Tables I and II are mathematically impossible. With one correct answer per query, MRR cannot exceed (1+Recall@1)/2. For CBOW, Recall@1 is 0.35, so MRR is capped at 0.675, yet they report 0.81. For SVD, the cap is 0.665 and they report 0.85. This is not a rounding artifact; something in the metric computation or the evaluation pipeline is broken. The recall numbers might still be usable, but the MRR column needs to be recomputed or removed.\n\nThe second problem is test-set circularity. The 221 pairs were selected by taking the 5,000 words with the highest Jensen-Shannon divergence between the 1930s and 1980s unigram distributions and then manually pairing words whose frequencies changed sharply. OP+SC then re-ranks candidates by Spearman correlation of frequency time series. The test set is literally constructed to contain the signal the method exploits. The margin over LT could shrink if the test set were built independently, e.g., from TDK's replacement lists or without consulting frequency divergence. I would want to see that robustness check before trusting the quantitative comparison.\n\nMinor but worth saying: no error bars, no significance tests, and the test set is not released. For a 221-pair manually built set, that is a legitimate request, not a nitpick.\n\nWhom is this for? Researchers working on diachronic NLP, especially for morphologically rich or reform-affected languages. It is not a theoretical breakthrough; it is an applied paper with a useful dataset and a simple method. The direction is sensible and the issues are fixable.\n\nRecommendation: send it to peer review. A serious referee should ask for a corrected MRR computation, error bars, and ideally a re-evaluation on a test set built without frequency divergence. But the core idea deserves referee time.","headline":"Small but genuine step for Turkish diachronic NLP; the OP method is fine, but the reported MRR values are arithmetically impossible and the test set is biased toward the frequency signal OP+SC exploits.","tokens_in":9068,"tokens_out":2179,"would_cite":false,"duration_ms":20948,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligned embedding spaces, re-ranked by frequency correlation, identify the exact Turkish words that replaced older Arabic and Persian terms during the 20th-century language reform.","keywords":["semantic change","natural language processing","Turkish language reform","diachronic word embeddings","orthogonal Procrustes alignment","neologism detection","Spearman rank correlation"],"falsifier":"Build a matched test set of 221 old/new Turkish word pairs chosen by a linguist without access to the Jensen-Shannon divergence rankings used here, and measure OP+SC's Recall@10 on the same CBOW embeddings. If the score falls far below 0.74, the evaluation procedure is what carries the result, not the alignment.","tokens_in":1581,"feed_emoji":"🔄","tokens_out":6931,"duration_ms":104798,"temperature":0.7,"pith_summary":"The paper tries to establish that a purely computational pipeline can pair a word introduced by the Turkish language reform with the older Arabic- or Persian-origin word it replaced, without looking up a dictionary. The pipeline aligns the word-embedding spaces of two time periods with an orthogonal rotation, then re-ranks the nearest neighbours by the Spearman correlation of the two words' frequency histories across decades. On a manually built set of 221 pairs spanning the 1930s and 1980s, the method with re-ranking retrieves the correct counterpart in the top 10 for 74% of queries and reaches a mean reciprocal rank of 0.81 with CBOW embeddings, above the linear-transformation baseline. The authors argue this matters because historical Turkish documents are hard for modern NLP tools, and automatically connecting old and new vocabulary could make those documents readable and processable. They also claim the method stays stable when the target decade moves from the 1960s to the 1980s, with only gradual decline for later periods.","feed_headline":"Find old Turkish words via frequency re-ranking","feed_subtitle":"Orthogonal rotation plus frequency re-ranking retrieves the right old-to-new synonym pair in 74% of top-10 tries.","key_machinery":"The central object is the orthogonal Procrustes transformation matrix $Q$ that maps the base-period embedding space onto the target-period space by minimizing $\\|E_{T_b}Q - E_{T_t}\\|_F^2$ over the intersection vocabulary, with the optimal $Q$ obtained from the SVD of $K = W_{T_b} W_{T_t}^T$. A second mechanism is Spearman's rank correlation computed on the relative-frequency time series of the query word and each candidate word, used to re-rank the $k$-nearest neighbours returned by OP. The intuition is that a replacement word should appear in the same contexts as the word it displaced, so after alignment the old word's vector should fall near the new word's vector.","core_discovery":"The central claim is that aligning diachronic embedding spaces with an orthogonal Procrustes transformation, rather than a general linear transformation, is the right mechanism for finding exact synonym replacements across time, and that a frequency-based re-ranking step makes it better. On the 1930s-to-1980s test set, OP+SC reaches Recall@10 of 0.74 and MRR of 0.81 with CBOW embeddings, compared with 0.34 and 0.17 for the linear-transformation baseline (Table I); with SVD embeddings the same pattern holds (0.70 Recall@10 and 0.85 MRR, Table II). The orthogonal map preserves vector geometry while rotating the base space onto the target space, so a query word's transformed vector lands near the vector of its replacement. The Spearman re-ranking then exploits the frequency signature of replacement: candidates whose decade-by-decade frequency curves are most consistent with the query's curve are promoted in the nearest-neighbour list.","pith_inferences":["A decisive test the paper does not run: build the evaluation set from synonym pairs selected without looking at frequency divergence between the two decades. If OP+SC's advantage over the linear baseline shrinks, part of the reported gain is built into the test set rather than in the alignment.","The method's reliance on decade-level frequency time series means it would likely struggle with short-lived or low-frequency neologisms; a natural extension is to compute correlations over smaller time windows or from contextual co-occurrence counts.","Because the authors note Turkronicles contains noise from text extraction and covers only formal state documents, an open question is whether the same accuracy holds on newspapers or fiction, where register and vocabulary differ."],"forward_implications":["Old-to-new word pairs produced by the method could be used to build or extend diachronic dictionaries for Turkish, covering words that current dictionaries omit.","The same pipeline transfers to other languages that underwent rapid lexical replacement, since it uses only embeddings and frequency counts, not Turkish-specific resources.","Historical-document NLP systems could preprocess texts by replacing era-specific words with their modern counterparts, reducing the temporal misalignment that degrades model performance.","Because performance is stable across moderate time gaps such as the 1960s to the 1980s, the method can be applied without retraining for every decade pair."],"supporting_citations":[{"why":"Supplies Turkronicles, the diachronic Turkish corpus of parliament records and official gazette texts from 1920 to 2020 used for all embeddings and frequency time series.","marker":"[14]"},{"why":"Provides the generalized orthogonal Procrustes solution used to compute the alignment matrix Q.","marker":"[22]"},{"why":"Establishes the orthogonal Procrustes alignment approach for diachronic word embeddings that the paper adapts to synonym detection.","marker":"[13]"},{"why":"Provides the linear transformation baseline and the seed-pair strategy for constructing the transformation matrix M.","marker":"[25]"},{"why":"Motivates the need for alignment by showing that word-embedding spaces from different periods are not directly comparable due to arbitrary axes.","marker":"[9]"},{"why":"Guides the embedding hyperparameters, such as the smoothed unigram distribution and negative sampling, used in training the CBOW and SVD vectors.","marker":"[11]"}],"fun_headline_variants":["Orthogonal alignment cracks Turkish synonym shifts","Turkish word evolution decoded via frequency rank","Re-ranking boosts old-to-new Turkish synonym search","Aligning embeddings finds Turkish synonyms across time","Old Turkish synonyms via orthogonal Procrustes"],"cache_read_input_tokens":11136,"weakest_assumption_plain":"The method assumes that a replaced word and its replacement have opposite frequency trajectories across decades, because both the test set and the Spearman re-ranking are built from that frequency-divergence signal.","fun_headline_variants_meta":{"raw":{"variants":["Orthogonal alignment cracks Turkish synonym shifts","Turkish word evolution decoded via frequency rank","Re-ranking boosts old-to-new Turkish synonym search","Aligning embeddings finds Turkish synonyms across time","Old Turkish synonyms via orthogonal Procrustes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1285,"prompt_tokens":890,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":329}},"tokens_in":506,"tokens_out":395,"duration_ms":4094,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:54:52.969234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a matched test set of 221 old/new Turkish word pairs chosen by a linguist without access to the Jensen-Shannon divergence rankings used here, and measure OP+SC's Recall@10 on the same CBOW embeddings. If the score falls far below 0.74, the evaluation procedure is what carries the result, not the alignment.","supporting_citations":[{"cited_title":"Turkronicles: Diachronic Resources for the Fast Evolving Turkish Language","cited_arxiv_id":"2405.10133","evidence_quote":"Supplies Turkronicles, the diachronic Turkish corpus of parliament records and official gazette texts from 1920 to 2020 used for all embeddings and frequency time series."},{"cited_title":"A generalized solution of the orthogonal pro- crustes problem","cited_arxiv_id":null,"evidence_quote":"Provides the generalized orthogonal Procrustes solution used to compute the alignment matrix Q."},{"cited_title":"Omnia mutantur, nihil interit: Connecting past with present by finding corresponding terms across time","cited_arxiv_id":null,"evidence_quote":"Provides the linear transformation baseline and the seed-pair strategy for constructing the transformation matrix M."},{"cited_title":"Statistically significant detection of linguistic change","cited_arxiv_id":null,"evidence_quote":"Motivates the need for alignment by showing that word-embedding spaces from different periods are not directly comparable due to arbitrary axes."},{"cited_title":"Improving distributional similarity with lessons learned from word embeddings","cited_arxiv_id":null,"evidence_quote":"Guides the embedding hyperparameters, such as the smoothed unigram distribution and negative sampling, used in training the CBOW and SVD vectors."}],"review_version":1}