Pith. sign in

REVIEW 3 major objections 4 minor 29 references

Detecting Turkish Synonyms Used in Different Time Periods

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Aligned embedding spaces, re-ranked by frequency correlation, identify the exact Turkish words that replaced older Arabic and Persian terms during the 20th-century language reform.

desk verdict Small but genuine step for Turkish diachronic NLP; the OP method is fine, but the reported MRR values are arithmetically impossible and the test set is biased toward the frequency signal OP+SC exploits. read the letter →

arxiv 2411.15768 v1 pith:K5TKMZJM submitted 2024-11-24 cs.CL

Umur Togay Yazar , Mucahid Kutlu This is my paper · ORCID
classification cs.CL
keywords semanticchangenaturallanguageprocessingTurkishreformdiachronicwordembeddingsorthogonalProcrustesalignmentneologismdetectionSpearmanrankcorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a purely computational pipeline can pair a word introduced by the Turkish language reform with the older Arabic- or Persian-origin word it replaced, without looking up a dictionary. The pipeline aligns the word-embedding spaces of two time periods with an orthogonal rotation, then re-ranks the nearest neighbours by the Spearman correlation of the two words' frequency histories across decades. On a manually built set of 221 pairs spanning the 1930s and 1980s, the method with re-ranking retrieves the correct counterpart in the top 10 for 74% of queries and reaches a mean reciprocal rank of 0.81 with CBOW embeddings, above the linear-transformation baseline. The authors argue this matters because historical Turkish documents are hard for modern NLP tools, and automatically connecting old and new vocabulary could make those documents readable and processable. They also claim the method stays stable when the target decade moves from the 1960s to the 1980s, with only gradual decline for later periods.

What carries the argument

The central object is the orthogonal Procrustes transformation matrix $Q$ that maps the base-period embedding space onto the target-period space by minimizing $\|E_{T_b}Q - E_{T_t}\|_F^2$ over the intersection vocabulary, with the optimal $Q$ obtained from the SVD of $K = W_{T_b} W_{T_t}^T$. A second mechanism is Spearman's rank correlation computed on the relative-frequency time series of the query word and each candidate word, used to re-rank the $k$-nearest neighbours returned by OP. The intuition is that a replacement word should appear in the same contexts as the word it displaced, so after alignment the old word's vector should fall near the new word's vector.

What would settle it

Build a matched test set of 221 old/new Turkish word pairs chosen by a linguist without access to the Jensen-Shannon divergence rankings used here, and measure OP+SC's Recall@10 on the same CBOW embeddings. If the score falls far below 0.74, the evaluation procedure is what carries the result, not the alignment.

Watch

Extended reading notes

Core claim

The central claim is that aligning diachronic embedding spaces with an orthogonal Procrustes transformation, rather than a general linear transformation, is the right mechanism for finding exact synonym replacements across time, and that a frequency-based re-ranking step makes it better. On the 1930s-to-1980s test set, OP+SC reaches Recall@10 of 0.74 and MRR of 0.81 with CBOW embeddings, compared with 0.34 and 0.17 for the linear-transformation baseline (Table I); with SVD embeddings the same pattern holds (0.70 Recall@10 and 0.85 MRR, Table II). The orthogonal map preserves vector geometry while rotating the base space onto the target space, so a query word's transformed vector lands near the vector of its replacement. The Spearman re-ranking then exploits the frequency signature of replacement: candidates whose decade-by-decade frequency curves are most consistent with the query's curve are promoted in the nearest-neighbour list.

Load-bearing premise

The method assumes that a replaced word and its replacement have opposite frequency trajectories across decades, because both the test set and the Spearman re-ranking are built from that frequency-divergence signal.

Editorial extensions

If this is right

  • Old-to-new word pairs produced by the method could be used to build or extend diachronic dictionaries for Turkish, covering words that current dictionaries omit.
  • The same pipeline transfers to other languages that underwent rapid lexical replacement, since it uses only embeddings and frequency counts, not Turkish-specific resources.
  • Historical-document NLP systems could preprocess texts by replacing era-specific words with their modern counterparts, reducing the temporal misalignment that degrades model performance.
  • Because performance is stable across moderate time gaps such as the 1960s to the 1980s, the method can be applied without retraining for every decade pair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A decisive test the paper does not run: build the evaluation set from synonym pairs selected without looking at frequency divergence between the two decades. If OP+SC's advantage over the linear baseline shrinks, part of the reported gain is built into the test set rather than in the alignment.
  • The method's reliance on decade-level frequency time series means it would likely struggle with short-lived or low-frequency neologisms; a natural extension is to compute correlations over smaller time windows or from contextual co-occurrence counts.
  • Because the authors note Turkronicles contains noise from text extraction and covers only formal state documents, an open question is whether the same accuracy holds on newspapers or fiction, where register and vocabulary differ.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses the task of detecting synonyms used in different time periods for Turkish, a language that underwent rapid lexical change during the 20th-century language reform. The authors propose two methods: Orthogonal Procrustes alignment (OP), which aligns embedding spaces from two time periods via an orthogonal transformation, and OP+SC, which extends OP by re-ranking the nearest neighbors according to Spearman rank correlation of the words' frequency time series. They evaluate on the Turkronicles diachronic corpus using CBOW and SVD embeddings, comparing against a linear transformation (LT) baseline from Zhang et al. (2015). The reported results (Tables I and II) claim that OP+SC outperforms LT and OP on Recall@1, Recall@10, and MRR, and that the methods are more robust than LT when the target period shifts from the 1960s to the 1980s.

Significance. If the reported results are trustworthy, this would be a useful contribution to diachronic NLP for Turkish, offering an automatic way to pair obsolete words with their modern replacements (e.g., vesika–belge). The work is the first computational study of Turkish neologism of this type, and the authors share their code for reproducibility. The methodological idea of combining orthogonal alignment with frequency-based re-ranking is natural and clearly presented. However, the quantitative evaluation contains at least one internal inconsistency and a potential circularity, so the central claim depends on fixes that the authors should be able to make within the scope of a revision.

major comments (3)
  1. [Section V-B, Tables I and II] The reported MRR values are mathematically inconsistent with the reported Recall@1 values. With a single correct answer per query, MRR for a set of queries is bounded above by (1 + Recall@1) / 2, since the reciprocal rank is 1 for queries answered at rank 1 and at most 1/2 for all others. For CBOW (Table I), Recall@1=0.35 gives a maximum possible MRR of 0.675, but the table reports MRR=0.81. For SVD (Table II), Recall@1=0.33 gives a maximum of 0.665, but MRR=0.85 is reported. This indicates a bug in the evaluation pipeline or in the reporting of the metrics. Because the central claim that OP+SC outperforms the baseline rests on these numbers, the authors must re-run the evaluation and ensure the metrics are computed consistently with the single-ground-truth ranking setup described in Section V-A.
  2. [Section V-A and Section IV-B] The test set construction and the signal exploited by OP+SC overlap in a way that inflates the reported advantage. The test set is built by ranking the top 5,000 words by Jensen–Shannon divergence between the base (1930s) and target (1980s) unigram distributions and manually pairing words whose frequencies changed sharply. OP+SC then re-ranks candidate neighbors by Spearman's rank correlation of the frequency time series (Section IV-B). This means that the test pairs are selected precisely for the property that OP+SC is designed to exploit, while the LT baseline (Section V-A) receives no frequency information and is thus systematically handicapped on this test set. To support the claim of generalizable detection ability, the evaluation should be repeated on a test set constructed without reference to frequency divergence, for example by using official TDK lists of replaced words or by having annotators create pairs while blind to frequency statistics. If that is not feasible, the authors should at least report results on a subset of pairs that are not selected by frequency divergence and discuss the potential bias.
  3. [Section V-B] No error bars, confidence intervals, or significance tests are provided for any of the metrics in Tables I–II or Figures 2–3. With only 221 test pairs (220 in the temporal-distance experiment), the differences between methods—for instance Recall@1 = 0.33 vs. 0.35 between OP and OP+SC in Table I—may be within the noise of the evaluation. The authors should include bootstrap confidence intervals or a paired significance test (e.g., Wilcoxon signed-rank on the ranks) to establish that the observed improvements are not due to chance. This is particularly important given the small test set and the high variance typical of embedding-based retrieval.
minor comments (4)
  1. [Section IV-A] Equation (1) uses the notation E_Tb and E_Tt for the intersection embeddings, but the preceding paragraph uses W_Tb and W_Tt; please harmonize the notation, and clarify that K = W_Tb W_Tt^T is the cross-covariance matrix whose SVD yields U and V, with the optimal orthogonal matrix Q = U V^T.
  2. [Section V-B] In the paragraph discussing temporal-distance results, the text reads 'the performance of LS tends to decrease'; this appears to be a typo for 'LT'.
  3. [References] Reference [13] and reference [15] are the same paper (Hamilton et al., 'Diachronic word embeddings reveal statistical laws of semantic change'); the duplicate should be removed. Also, reference [20] contains a placeholder '[ ?]' that should be completed, and several reference entries are missing volume/page information.
  4. [Section V-A] The description of the test set construction states that the base period is 1930 and the target period is 1980, but the experiments actually use the decade spans 1930–1939 and 1980–1989; this should be stated precisely at first mention to avoid ambiguity.

Circularity Check

1 steps flagged · score 5.0 of 10

Evaluation set is built from the same frequency-divergence signal that OP+SC re-ranks by, so the reported advantage over the baseline is partly by construction.

  1. other [Section IV-B (OP+SC) and Section V-A (Dataset)]
    "Next, a frequency time series is constructed for each candidate pair (w, c), where c ∈ K. Subsequently, we re-rank the pairs according to Spearman's correlation coefficient in decreasing order. ... In this calculation, we selected the top 5,000 words that contributed the most to the divergence score. ... This process enables us to identify words whose frequency changed dramatically between these periods, indicating potentially replaced or newly added words."

    The test set is constructed by taking the 5,000 words with the largest Jensen-Shannon divergence between the 1930s and 1980s unigram distributions, i.e., words whose frequencies changed dramatically. OP+SC's only novel component is re-ranking OP's nearest neighbors by Spearman's rank correlation of the same kind of frequency time series, while the LT baseline receives no frequency signal. Thus the evaluation pairs are enriched for exactly the frequency-divergence property OP+SC uses to re-rank, so the reported OP+SC gain over LT is partly a consequence of test-set construction rather than of generalizable semantic alignment. This is not a fitted-parameter loop, but it is a circular evaluation of the frequency-based component.

full rationale

The OP alignment itself is a standard external technique and is not derived from the test set, so the core alignment claim has independent content. The main circularity is in the evaluation design: the 221 test pairs were deliberately selected from the 5,000 words with the largest frequency divergence between the two periods, which is precisely the signal OP+SC uses for its Spearman re-ranking. Because LT does not use frequency information, the large reported margin (Recall@10 0.74 vs. 0.34) partially reflects this selection rather than general detection ability. OP alone also outperforms LT, so the paper is not wholly circular, but the OP+SC-specific improvement is suspect. Separately, the reported MRR values (0.81 for CBOW and 0.85 for SVD) are mathematically impossible given the same tables' Recall@1 values (0.35 and 0.33) under the paper's own single-answer ranking setup, indicating an evaluation or reporting bug; this is a correctness concern rather than a circularity concern, but it further weakens the quantitative comparison. The Turkronicles self-citation is a dataset release, not an unverified load-bearing theorem, so it does not independently raise the circularity score.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on the standard Procrustes/SVD math, on the assumption that a single orthogonal rotation aligns decade-specific embedding spaces, and on the frequency-divergence-based construction of the manual test set. The most consequential ledger item is the test set selection rule, which is entangled with the frequency-correlation signal used by OP+SC.

free parameters (8)
  • Embedding dimension d = 300
    Chosen for both SVD and CBOW; not fitted to the test set, but affects alignment quality.
  • Context window size = 2
    Symmetric kernel used for both embeddings; standard configuration, not justified by experiments.
  • Negative samples (CBOW) = 5
    CBOW training hyperparameter.
  • Downsampling rate = 1e-5
    CBOW training hyperparameter.
  • Unigram smoothing alpha = 0.75
    From Levy et al.; affects embeddings.
  • Ridge regression alpha for baseline = 0.2
    Baseline LT hyperparameter.
  • Neighborhood sizes for OP+SC = 5, 15, 150 for Recall@1, @10, @100
    Chosen so that the re-ranked candidate set matches the evaluation cutoff; not tuned on a held-out set, but directly tied to metrics.
  • Top divergent words for test set construction = 5,000
    Chosen by the authors; directly shapes the test set and thus all reported results.
assumptions (5)
  • standard math Orthogonal Procrustes optimal solution via SVD
    Section IV.A, Eq. 1; standard result from Schönemann (1966).
  • domain assumption A single orthogonal transformation aligns two embedding spaces
    Section IV.A; assumes semantic geometry of the base period maps linearly into the target period; Hamilton et al. make a similar assumption.
  • domain assumption Frequency trajectories of replaced and replacing words move in opposite directions
    Section IV.B and Figure 1; not true for all replacement pairs, e.g., stable or co-existing words.
  • domain assumption Turkronicles corpus represents Turkish language change
    Section V.A; corpus is formal parliamentary and gazette records, not colloquial Turkish.
  • domain assumption Manual test pairs are correct
    Section V.A; annotated by the authors using the TDK dictionary; no inter-annotator agreement reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Turkish Synonyms Used in Different Time Periods." pith.science (2026). https://pith.science/paper/K5TKMZJM

@misc{pith2026241115768,
  author       = {Pith},
  title        = {Pith review of: Detecting Turkish Synonyms Used in Different Time Periods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5TKMZJM}},
  note         = {Machine review of arXiv:2411.15768}
}
read the original abstract

Dynamic structure of languages poses significant challenges in applying natural language processing models on historical texts, causing decreased performance in various downstream tasks. Turkish is a prominent example of rapid linguistic transformation due to the language reform in the 20th century. In this paper, we propose two methods for detecting synonyms used in different time periods, focusing on Turkish. In our first method, we use Orthogonal Procrustes method to align the embedding spaces created using documents written in the corresponding time periods. In our second method, we extend the first one by incorporating Spearman's correlation between frequencies of words throughout the years. In our experiments, we show that our proposed methods outperform the baseline method. Furthermore, we observe that the efficacy of our methods remains consistent when the target time period shifts from the 1960s to the 1980s. However, their performance slightly decreases for subsequent time periods.

Figures

Figures reproduced from arXiv: 2411.15768 by the authors.

Figure 1
Figure 1. Relative frequency of words belge and vesika between 1920 and 2022 in ten-year periods in Turkronicles dataset. c ∈ K. Subsequently, we re-rank the pairs according to Spearman’s correlation coefficient in decreasing order. This method increases the rank of the true counterpart of w in the k-nearest neighbor set. V. EXPERIMENTS A. Experimental Setup Dataset. In our experiments, we utilized the Turkronicles dataset [1… view at source ↗
Figure 2
Figure 2. Performance of the methods across time periods in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance of the methods across time periods in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 23 canonical work pages

  1. [1]

    The Turkish language reform: A catastrophic success: A catastrophic success

    Lewis, Geoffrey. The Turkish language reform: A catastrophic success: A catastrophic success. OUP Oxford, 1999

    work page 1999

  2. [2]

    Lexical change in present-day English: A corpus- based study of the motivation, institutionalization, and productivity of creative neologisms

    Fischer. Roswitha, "Lexical change in present-day English: A corpus- based study of the motivation, institutionalization, and productivity of creative neologisms". V ol. 17. Gunter Narr Verlag, 1998

    work page 1998

  3. [3]

    Dictionaries of Neologisms: a Review and Proposals for its Improvement

    Rodríguez Guerra, Alexandre. "Dictionaries of Neologisms: a Review and Proposals for its Improvement" Open Linguistics, vol. 2, no. 1, 2016. https://doi.org/10.1515/opli-2016-0028

    work page doi:10.1515/opli-2016-0028 2016

  4. [4]

    John R Taylor and Anthony P. Grant. 2014. Lexical Borrowing. Oxford University Press, Oxford

    work page 2014

  5. [5]

    Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution

    Pechenick, Eitan Adam, Christopher M. Danforth, and Peter Sheridan Dodds. "Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution." PloS one 10.10 (2015): e0137041

    work page 2015

  6. [6]

    Quantitative analysis of culture using millions of digitized books

    Michel, Jean-Baptiste, et al. "Quantitative analysis of culture using millions of digitized books." science 331.6014 (2011): 176-182

    work page 2011

  7. [7]

    The Road to Success: Assessing the Fate of Linguistic Innovations in Online Communities

    Del Tredici, Marco, and Raquel Fernández. "The road to success: Assessing the fate of linguistic innovations in online communities." arXiv preprint arXiv:1806.05838 (2018)

    work page Pith review arXiv 2018

  8. [8]

    The reason why

    Aitchison. J, "The reason why", Language change: Progress or decay? , ISBN: 9781107023628, Cambridge University Press, 2012

    work page 2012

Show all 29 references
  1. [9]

    Statistically significant detection of linguistic change

    Kulkarni, Vivek, et al. "Statistically significant detection of linguistic change." Proceedings of the 24th international conference on world wide web. 2015

    2015

  2. [10]

    Temporal analog retrieval using transformation over dual hierarchical structures

    Zhang, Yating, Adam Jatowt, and Katsumi Tanaka. "Temporal analog retrieval using transformation over dual hierarchical structures." Proceed- ings of the 2017 ACM on Conference on Information and Knowledge Management. 2017

    2017

  3. [11]

    Improving distributional similarity with lessons learned from word embeddings

    Levy, Omer, Yoav Goldberg, and Ido Dagan. "Improving distributional similarity with lessons learned from word embeddings." Transactions of the association for computational linguistics 3 (2015): 211-225

    2015

  4. [12]

    Diachronic word embeddings and semantic shifts: a survey

    Kutuzov, Andrey, et al. "Diachronic word embeddings and semantic shifts: a survey." arXiv preprint arXiv:1806.03537 (2018)

    2018 arXiv

  5. [14]

    Turkronicles: Diachronic Resources for the Fast Evolving Turkish Language

    Yazar, Togay, Mucahid Kutlu, and ˙Isa Kerem Bayırlı. "Turkronicles: Diachronic Resources for the Fast Evolving Turkish Language." arXiv preprint arXiv:2405.10133 (2024)

    2024 arXiv

  6. [15]

    Diachronic word embeddings reveal statistical laws of semantic change

    Hamilton, William L., Jure Leskovec, and Dan Jurafsky. "Diachronic word embeddings reveal statistical laws of semantic change." arXiv preprint arXiv:1605.09096 (2016)

    2016 arXiv

  7. [16]

    Measuring historical word sense variation

    Bamman, David, and Gregory Crane. "Measuring historical word sense variation." Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries. 2011

    2011

  8. [17]

    Jean Aitchison, Language change: progress or de- cay?(Fontana Linguistics.) London: Fontana, 1981. Pp. 266

    Denison, David. "Jean Aitchison, Language change: progress or de- cay?(Fontana Linguistics.) London: Fontana, 1981. Pp. 266." Journal of Linguistics 19.2 (1983): 503-504

    1983

  9. [18]

    Detecting Lexical Se- mantic Change across Corpora with Smooth Manifolds (Student Ab- stract)

    Goel, Anmol, and Ponnurangam Kumaraguru. "Detecting Lexical Se- mantic Change across Corpora with Smooth Manifolds (Student Ab- stract)." Proceedings of the AAAI Conference on Artificial Intelligence. V ol. 35. No. 18., 2021

    2021

  10. [19]

    Language change

    Bybee, Joan. Language change. Cambridge University Press, 2015

    2015

  11. [20]

    Efficient estimation of word representations in vector space

    Mikolov, Tomas, et al. "Efficient estimation of word representations in vector space." arXiv preprint arXiv:1301.3781 (2013)

    2013 arXiv

  12. [21]

    A Computational Evaluation of Two Laws of Semantic Change

    Xu, Yang, and Charles Kemp. "A Computational Evaluation of Two Laws of Semantic Change." CogSci. 2015

    2015

  13. [22]

    A generalized solution of the orthogonal pro- crustes problem

    Schönemann, Peter H. "A generalized solution of the orthogonal pro- crustes problem." Psychometrika 31.1 (1966): 1-10

    1966

  14. [23]

    Dynamic embeddings for language evolution

    Rudolph, Maja, and David Blei. "Dynamic embeddings for language evolution." Proceedings of the 2018 world wide web conference. 2018

    2018

  15. [24]

    Where new words are born: Distributional semantic analysis of neologisms and their semantic neighborhoods

    Ryskina, Maria, et al. "Where new words are born: Distributional semantic analysis of neologisms and their semantic neighborhoods." arXiv preprint arXiv:2001.07740 (2020)

    2020 arXiv

  16. [25]

    Omnia mutantur, nihil interit: Connecting past with present by finding corresponding terms across time

    Zhang, Yating, et al. "Omnia mutantur, nihil interit: Connecting past with present by finding corresponding terms across time." Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Natural Langu...

    2015

  17. [26]

    Entity cloze by date: What LMs know about unseen entities

    Onoe, Yasumasa, et al. "Entity cloze by date: What LMs know about unseen entities." arXiv preprint arXiv:2205.02832 (2022)

    2022 arXiv

  18. [27]

    Dynamic word embeddings for evolving semantic discovery

    Yao, Zijun, et al. "Dynamic word embeddings for evolving semantic discovery." Proceedings of the eleventh acm international conference on web search and data mining. 2018

    2018

  19. [28]

    Training temporal word embeddings with a compass

    Di Carlo, Valerio, Federico Bianchi, and Matteo Palmonari. "Training temporal word embeddings with a compass." Proceedings of the AAAI conference on artificial intelligence. V ol. 33. No. 01. 2019

    2019

  20. [29]

    Time waits for no one! analysis and challenges of temporal misalignment

    Luu, Kelvin, et al. "Time waits for no one! analysis and challenges of temporal misalignment." arXiv preprint arXiv:2111.07408 (2021)

    2021 arXiv

  21. [30]

    NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms

    Zheng, Jonathan, Alan Ritter, and Wei Xu. "NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms." arXiv preprint arXiv:2402.12261 (2024)

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.