Pith. sign in

REVIEW 3 major objections 6 minor 4 references

Feature-Refined Unsupervised Model for Loanword Detection

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A purely language-internal, unsupervised feature-refinement model can detect loanwords in monolingual wordlists and outperforms earlier unsupervised baselines.

desk verdict Genuinely new unsupervised architecture for loanword detection, but the headline F1 looks partly fitted to the labeled set; needs held-out validation before the central claim is fully supported. read the letter →

arxiv 2508.17923 v1 pith:BJ76QB3K submitted 2025-08-25 cs.CL

Promise Dodzi Kpoglu This is my paper · ORCID
classification cs.CL
keywords loanworddetectionhistoricallinguisticsunsupervisedlearninglinguisticfeaturespatternrecognitionphonotacticsiterativerefinementlanguagecontact
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to prove that loanwords can be detected without any information from outside the language being examined. Its model takes a single monolingual wordlist, scores every word on a set of internally derived linguistic cues, and then iteratively refines the scoring until the predicted set of borrowings stabilises. The reported result is that this approach finds substantially more loanwords than the earlier unsupervised baseline it is compared against, reaching 0.71 recall and 0.63 precision in the basic version and 0.91 recall in a cross-linguistically scaled version. The point of the exercise is practical: if the method works, historical linguists can generate candidate borrowings from a wordlist alone, without first committing to a language family tree or to similarity comparisons with other languages, both of which the paper argues can introduce circularity.

What carries the argument

The load-bearing mechanism is an iterative refinement loop built around a composite probability score. Each word is reduced to a feature vector: how rare its sub-word segments are, how skewed their distribution is, how uncommon the transitions between neighbouring sounds are, how predictable those transitions are on average, and how far the word's length deviates from the vocabulary average. These features are weighted, adjusted for word length and part-of-speech, and mapped through a smooth curve (a sigmoid) to a borrowing probability. The loop then re-estimates the native-word statistics from the words it currently believes to be native, adjusts scores using pattern databases of frequent prefixes, suffixes, and three-character segments, and averages probabilities over iterations until the predicted loanword set stops changing. In the scaled variant, each word's phonological distance from its concept-matched counterparts in other languages is computed by pairwise alignment with context-sensitive match probabilities and articulatory feature distances, then combined with the base probability in a weighted composite score.

What would settle it

Run the published code on a newly transcribed wordlist from a language outside the six studied, using the paper's exact threshold and constants with no re-tuning, and compare against a fresh gold standard; if F1 falls to the baseline level, the gains come from test-set tuning rather than the method itself.

Watch

Extended reading notes

Core claim

The paper's central claim is that language-internal statistical and structural cues, modelled in an iterative refinement loop, are sufficient to detect a large share of loanwords in monolingual wordlists. The model's design reflects the linguistic assumption that borrowed words stand out on several independent axes at once: rare sub-word sequences, improbable phoneme transitions, unusual word length, and the tendency of borrowings to be nouns rather than function words. These signals are combined into a weighted, sigmoid-mapped probability, and the model then re-estimates the native-vs-loan statistics from the words it currently believes to be native, converging after a few iterations. The discovery, as the paper frames it, is that this hybrid feature-plus-statistics architecture with iterative self-refinement outperforms the prior unsupervised baseline on six languages, and that adding cross-linguistic comparability scores raises recall considerably.

Load-bearing premise

The reported scores are only trustworthy if the hand-annotated gold standard is accurate and if the decision threshold, feature weights, and scaling constants were not tuned against the same 5,092 entries on which precision and recall are computed.

Editorial extensions

If this is right

  • Historical linguists can obtain loanword candidates from a single language's wordlist, avoiding the circularity the paper identifies in phylogenetic and cross-linguistic methods.
  • The model's unsupervised, self-training design is extensible to low-resource languages where no labelled loanword data exist.
  • The scaled model's high recall (0.91 aggregated) makes it usable as a first-pass filtering tool whose output a linguist checks manually.
  • The error analysis indicates the model implicitly learns additional cues such as consonant clusters and vowel quality, suggesting that even unencoded phonological regularities contribute to the results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: a decisive test would be to run the published code, with the paper's threshold and constants fixed, on a wordlist from a language outside the six studied and compare against a fresh gold standard; if the gains vanish, the reported scores reflect tuning to the evaluation set.
  • Extension: if the language-internal approach generalises, it could enable loanword studies in language isolates and poorly documented contact zones, where the phylogenetic information that older methods need is unavailable.
  • Extension: the model's dependence on IPA transcriptions suggests a natural port to tone languages by adding tonal features to the feature vector, which the current design does not exploit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes AUT_BOR, an unsupervised model for loanword detection that uses only language-internal features: n-gram rarity, transition probabilities, length z-scores, and part-of-speech weights are combined into a sigmoid-mapped composite score, then refined via iterative self-training until convergence. A scaled variant adds cross-linguistic alignment scores. The method is evaluated on a self-curated dataset of 5,092 IPA-transcribed entries from six Indo-European languages. The reported results (Table 2) show AUT_BOR achieving F1=0.67 versus F1=0.38 for a reimplemented Prakhya & P (2020) baseline, and the scaled model reaching F1=0.73 and recall=0.91 (Table 4). The central claim is that a purely language-internal, unsupervised approach can outperform earlier unsupervised methods and is extensible to multilingual data.

Significance. If the reported numbers represent genuine out-of-sample predictions, the paper would be a useful contribution to computational historical linguistics: it demonstrates that language-internal cues can detect loanwords without relying on phylogenetic or cross-family information, thereby avoiding the circularity the authors rightly identify in external-information methods. The architecture is novel and clearly described, the code is promised for release, and the error analysis (Section 7.1) is thoughtful and diagnostic. However, the evaluation protocol contains load-bearing flaws—threshold and hyperparameter tuning on the same data used for evaluation, a modified baseline whose fairness is not established, and no variance or out-of-sample checks—so the headline performance figures are not currently credible as evidence for the paper's central claim.

major comments (3)
  1. [Section 4 and Appendix A2] The decision threshold is tuned on the evaluation data itself. Section 4 states that the model outputs labels through 'a tuned decision threshold that balances precision and recall,' and balancing precision and recall requires access to the gold labels. Section 3 similarly says threshold τ is 'to be determined empirically,' and Appendix A2 lists numerous hyperparameters (n-gram rarity thresholds ε1=0.005, ε2=0.02; transition thresholds 0.01 and 0.05; feature weights; POS weights; sigmoid parameters γ and θ; iteration limit 7; convergence criterion 1%; anomaly boosts) that were 'established through preliminary, iterative experimentation.' No held-out split, cross-validation, or nested validation is reported anywhere. Consequently, the scores in Table 2 and Table 4 may be partly fitted values rather than predictions, and the claimed advantage over the UNS baseline may reflect a tuning advantage. The authors must re-run the evaluation with a fixed a-priori threshold or a separate validation set, or at minimum report the tuning procedure and results on a truly held-out portion of the data.
  2. [Section 6.1] The baseline reimplementation is not demonstrated to be fair. The paper introduces 'a key modification' to Prakhya & P (2020) by defining stems as the first two syllables, asserting this 'ensures that our baseline remains faithful to the original model's spirit.' However, no evidence is given that the original stem definition is inapplicable to the six target languages, nor that the modified stem definition does not systematically handicap the baseline. Since the paper's main evidence is the large gap between AUT_BOR (F1 0.67) and UNS (F1 0.38) in Table 2, the comparator must be validated. Please report results with the original stem definition as well, or otherwise demonstrate that the modification is neutral or favorable to the baseline, and consider tuning the baseline's own parameters under the same protocol used for AUT_BOR.
  3. [Section 5] The scaled model uses additional empirically fitted constants. Section 5 defines a dynamic threshold θ(x) = α + β((1-C(x)) - B(x)) with 'α, β ∈ R are empirically determined constants,' and also introduces hyperparameters w1, w2, and λ. As with the basic model, no indication is given that these were set on a separate validation set, and no sensitivity analysis is reported. The claim in Section 7 that cross-linguistic scaling yields a '6% performance increase' over the basic model is therefore also at risk of being an artifact of fitting to the same 5,092 labeled entries. Please provide a validation-based selection procedure for these constants or report performance over a grid with error bars.
minor comments (6)
  1. [Section 6.2] The notation ℳ−𝒫3 is confusing: the text says this model 'introduces new features,' which is an augmentation, not an ablation. Rename it (e.g., ℳ+𝒫3 or AUT_BOR-AUG) for consistency with Table 2.
  2. [Section 4.1.1] The rare_ngram_score formula is typeset confusingly: the indicator notation 𝟙 is mixed with the subscripts and the roles of ε1, ε2, c1, c2 are not clearly separated. Please rewrite the equation with standard indicator functions and define each parameter explicitly.
  3. [Section 4.2] The length-modifier formula λlen(w) appears to contain a stray '0.5' and unbalanced parentheses: λlen(w) = 1 + (0.5 ⋅ (1/(1+e^{-3(|z_len(w)|-1.5)}) 0.5)). This should be corrected to a clear mathematical expression.
  4. [Figure 3] The caption reads 'F1 score as a function of training data proportion,' but the model is unsupervised and Section 7's description suggests varying the proportion of the input dataset, not a training split. Clarify what is being varied and how the curve in Figure 3 was obtained.
  5. [Section 3.1] The dataset is self-curated and is not made available for reproduction. Please release the gold-standard wordlists and annotations (or provide a link), and report inter-annotator agreement or a second annotation pass given the stated adjudication decisions.
  6. [Appendix A1] The table numbering in the appendix conflicts with the main text: 'Table 1' appears both in Section 3.1 and in Appendix A1. Renumber the appendix tables.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported F1 is partly fitted: the decision threshold is tuned to balance precision and recall on the same 5,092 labeled entries, and Appendix A2 describes iterative hyperparameter experimentation with no reported held-out split.

  1. fitted input called prediction [Section 4 (Basic model), final paragraph; also Section 3 definition of f(w) with threshold τ.]
    "Following this, t he model outputs both probabilities and binary labels per word, with a tuned decision threshold that balances precision and recall."

    The threshold τ is the decision rule that turns scores into the TP/FP/TN/FN counts reported in Table 2. Balancing precision and recall is an operation that requires the gold-standard loanword labels from Section 3.1, and the paper describes no held-out split, validation set, or nested tuning procedure. Section 3 only says τ is 'determined empirically'. Therefore the reported precision 0.63, recall 0.71, and F1 0.67 are not out-of-sample predictions; they are partly constructed by choosing τ to optimize the very metrics then reported. The UNS baseline receives no analogous label-guided threshold tuning in the description, so the comparison also mixes model quality with a tuning advantage.

  2. fitted input called prediction [Section 5.1 (Scaled model), dynamic threshold definition; Appendix A2 (Hyperparameters).]
    "𝜃(𝑥) = 𝛼 + 𝛽 ((1 − 𝐶(𝑥)) − 𝐵(𝑥)) where 𝛼, 𝛽 ∈ 𝑅 are empirically determined constants controlling the sensitivity to discrepancies between cross-linguistic evidence and the basic model’s prediction."

    The dynamic threshold θ(x) determines the scaled model's final binary labels y-hat(x), and α and β are 'empirically determined constants'. Appendix A2 states that the model's components are 'governed by a set of hyperparameters established through preliminary, iterative experimentation', including n-gram penalty breakpoints, part-of-speech weights, sigmoid boost thresholds, initial threshold 0.3, and iteration limits. Because no development/test partition is reported anywhere, the experiments that set these constants appear to be the same 5,092-entry evaluation used for Tables 2–4. The scaled model's high recall (0.91) and F1 (0.73) are therefore partially fitted descriptions of the evaluation set rather than demonstrated predictions on unseen words or languages.

full rationale

The derivation is not circular by definition: loanword status is not defined in terms of any feature used by the model, and the core scoring features (rare n-grams, transition probabilities, length z-scores, pattern databases) are computed from the input vocabulary rather than from the gold labels. The iterative refinement loop uses the model's own predicted sets, not the annotated loanword labels, so that part of the architecture is self-contained. There is also no load-bearing self-citation chain: citations to List, Prakhya & P, and others are contextual or baseline-related, not invoked to justify the model's predictions. The significant circularity is in the evaluation protocol. Section 4 explicitly says the decision threshold is 'tuned' to balance precision and recall, which requires access to the gold labels used to compute those metrics. Section 5.1 introduces empirically determined constants α and β for the scaled model's decision rule, and Appendix A2 discloses that hyperparameters were set through iterative experimentation. Since the paper never reports a train/dev/test split, cross-validation, or any separation between parameter selection and evaluation, the headline numbers in Tables 2 and 4 cannot be read as clean out-of-sample predictions. Some of the reported advantage over the UNS baseline may reflect label-informed threshold and hyperparameter selection rather than the model's unsupervised inference ability. This warrants a 6 rather than a higher score because the central feature-based derivation has independent content and is not equivalent to its inputs by construction; the circularity is confined to the fitted decision boundaries and tuning choices that produce the reported performance figures.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The model depends on many hand-set coefficients and on assumptions about the validity of the manual gold standard. In the absence of released data or a held-out split, the reported gains are best read as evidence of an architecture that can work, not a validated off-the-shelf detector.

free parameters (9)
  • N-gram rarity thresholds epsilon1, epsilon2 and penalties c1, c2 = epsilon1=0.005, epsilon2=0.02; c1,c2 not numerically specified
    Controls rare_ngram_score in Section 4.1.1; chosen by iterative experimentation in Appendix A2.
  • Transition rarity thresholds and penalty coefficients = p<0.01 gets coefficient 100; 0.01<=p<0.05 gets coefficient 20
    Appears in rare_transition_score in Section 4.1.2; thresholds fixed by hand in Appendix A2.
  • Feature weights alpha_i = Order claimed: rare n-gram > rare transition > transition entropy > n-gram entropy > avg transition prob > length…
    Weighted sum in Section 4.2; relative importance assigned by the authors, not learned.
  • Part-of-speech weights beta_POS = noun 1.0, adjective 0.5, verb 0.3, adverb 0.2, function word 0.05
    POS adjustment lambda_pos in Section 4.2; values from Appendix A2.
  • Sigmoid steepness gamma and center theta = theta=0.5; gamma unspecified
    Probability mapping in Section 4.2; gamma controls how sharply scores become probabilities.
  • Decision threshold tau = 0.3 for provisional classification
    Section 4.3 uses tau; Appendix A2 says words with probability at least 0.3 are provisionally borrowed; Section 4 also refers to a tuned threshold balancing precision and recall.
  • Iteration limits T_max and convergence threshold = T_max=7; early stop if set change <1%
    Convergence control in Section 4.3, values in Appendix A2.
  • Anomaly boost coefficients eta_f and anomaly thresholds = not specified
    Boosts in P'(w)=min(P(w)*prod(1+eta_f delta_f(w)),1) in Section 4.2; values absent.
  • Scaled model hyperparameters lambda, w1, w2, alpha, beta = not specified
    Weights and dynamic threshold in Section 5; described as empirically determined but not reported.
assumptions (6)
  • domain assumption Native and loanwords have statistically distinguishable feature distributions within a single wordlist.
    Stated as the guiding hypothesis in Section 3; if false the entire scoring function has no discriminative signal.
  • domain assumption The manually annotated gold standard labels are accurate and consistent.
    Section 3.1 describes curation with etymological evidence and judgment for ambiguous cases; no second annotation or external benchmark is provided.
  • domain assumption Normalized IPA transcriptions preserve loanword-diagnostic information after removing length and stress marks.
    Section 3.1 preprocesses IPA by deleting length and stress; the model then operates only on these strings.
  • ad hoc to paper Iterative self-training converges to a better classification than the initial scoring pass.
    Section 4.3 assumes provisional native sets become more reliable and that pattern databases constructed from them improve decisions; no proof or convergence diagnostics are shown.
  • domain assumption Fixed borrowability weights for POS categories transfer across all six languages.
    The POS adjustment in Section 4.2 uses one set of beta weights, taken from typological tendencies, not estimated per language.
  • ad hoc to paper The reimplemented Prakhya and P baseline is a fair comparator.
    Section 6.1 changes the stem definition to the first two syllables; if this weakens the baseline, the reported outperformance is inflated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Feature-Refined Unsupervised Model for Loanword Detection." pith.science (2026). https://pith.science/paper/BJ76QB3K

@misc{pith2026250817923,
  author       = {Pith},
  title        = {Pith review of: Feature-Refined Unsupervised Model for Loanword Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BJ76QB3K}},
  note         = {Machine review of arXiv:2508.17923}
}
read the original abstract

We propose an unsupervised method for detecting loanwords i.e., words borrowed from one language into another. While prior work has primarily relied on language-external information to identify loanwords, such approaches can introduce circularity and constraints into the historical linguistics workflow. In contrast, our model relies solely on language-internal information to process both native and borrowed words in monolingual and multilingual wordlists. By extracting pertinent linguistic features, scoring them, and mapping them probabilistically, we iteratively refine initial results by identifying and generalizing from emerging patterns until convergence. This hybrid approach leverages both linguistic and statistical cues to guide the discovery process. We evaluate our method on the task of isolating loanwords in datasets from six standard Indo-European languages: English, German, French, Italian, Spanish, and Portuguese. Experimental results demonstrate that our model outperforms baseline methods, with strong performance gains observed when scaling to cross-linguistic data.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages

  1. [1]

    Alvarez-Mellado, E., Porta -Zamorano, J., Lignos, C., & Gonzalo, J. (2025). Overview of ADoBo at IberLEF 2025 : Automatic Detection of Anglicisms in Spanish (arXiv:2507.21813). Crawford, C. J. (2009). Adaptation and transmission in Japanese loanword phonology. Cornell University. Grant, A. P. (2000). Fabric, pattern, shift and diffusion: What change in Or...

    work page Pith review arXiv 2025

  2. [79]

    Mennecier, P., Nerbonne, J., Heyer, E., & Manni, F. (2016). A Central Asian language survey: Collecting data, measuring relatedness and detecting loans. Language Dynamics and Change, 6(1), 57‑98. Miller, J. E., Tresoldi, T., Zariquiey, R., Beltran Castanon, C. A., Morozova, N., & List, J.-M. (2020). Using lexical language models to detect borrowings in mo...

    work page Pith review arXiv 2016

  3. [167]

    Rama, T., & List, J. -M. (2019). An automated framework for fast cognate detection and Bayesian phylogenetic inference in computational historical linguistics . Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 6225–6235. Swadesh, M. (1955). Towards Greater Accuracy in Lexicostatistic Dating. International Journal of...

    work page 2019

  4. [201]

    -M., & Forkel, R

    List, J. -M., & Forkel, R. (2022). Automated identification of borrowings in multilingual wordlists. Open Research Europe, 1,

    work page 2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.