{"id":"a705dc04-cd49-42d6-9e93-9a57a2530034","arxiv_id":"2508.17923","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"An unsupervised language-internal feature-scoring model detects loanwords in six Indo-European languages with F1 0.67, rising to 0.73 when cross-linguistic wordlists are added.","lead":"This paper presents a computer model that tries to spot borrowed words in a language by looking only at patterns inside that language's own word list, such as unusual letter combinations and word length. On six European languages it finds more loanwords than an older method, and adding word lists from other languages raises recall to 0.91.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported F1 may be partly fitted: Section 4 says the decision threshold is tuned to balance precision and recall, and Appendix A2 describes iterative experimentation, with no held-out split, cross-validation, or error bars.","rationale":"The reader's weakest_assumption already identifies the same core risk: constants and thresholds may have been chosen against the labeled evaluation set rather than inferred purely from language-internal structure. My read of the full text strengthens that concern with explicit passages: Section 4's 'tuned decision threshold that balances precision and recall' can only be implemented with access to gold labels, and Appendix A2 confirms that the hyperparameters emerged from iterative experimentation without any described validation split. This is not an external-consensus objection; it is an evaluation-protocol objection. If the tuned constants were selected on the test entries, the reported confusion matrices in Tables 2 and 4 are partly fitted values, and the claimed superiority over UNS could shrink or vanish under an honest split. That said, the paper does have genuine assets: the architecture is clearly described, the model is unsupervised in its inference loop, the feature combination is linguistically motivated, and code is promised at a public repository. The concern is therefore conditional rather than fatal, and the reader's CONDITIONAL verdict remains appropriate; it should be accepted only with a clean evaluation protocol. I also note the baseline modification in Section 6.1 is a secondary concern: redefining the stem as the first two syllables is a plausible adaptation, but without a sensitivity analysis it is unclear whether the large UNS gap is partly an artifact of that particular choice. The primary concrete test — a held-out tuning/evaluation split applied to both the proposed model and the baseline — would settle whether the central claim transfers beyond the current, potentially leaked evaluation.","tokens_in":16101,"tokens_out":3007,"duration_ms":34684,"concrete_test":"Using the released code and dataset, perform a strict split evaluation: hold out 20% of entries per language as a test set; tune tau and all Appendix A2 constants, including POS weights, n-gram and transition penalty breakpoints, alpha, beta, w1, and w2, only on the remaining 80% development data; freeze the selected constants and report precision, recall, and F1 on the held-out test folds, averaging over five random splits. Apply the identical tuning protocol to the UNS reimplementation. If the frozen model's pooled F1 falls below roughly 0.55 or no longer clearly separates from UNS under the same protocol, the headline outperformance claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that an unsupervised, language-internal model detects loanwords and outperforms UNS. That claim holds only if the Table 2 and Table 4 numbers are predictions under a fixed decision rule, not the result of fitting constants to the same 5,092 labeled entries. The text contains direct evidence of label-informed selection. Section 4 describes 'a tuned decision threshold that balances precision and recall' — balancing precision and recall requires the gold labels. Section 5 uses 'empirically determined constants' alpha and beta for the scaled model. Appendix A2 says hyperparameters were 'established through preliminary, iterative experimentation', including n-gram penalty breakpoints, POS weights, initial threshold 0.3, iteration limits, and feature weights. No held-out split, nested validation, or error bars are reported. Since the scoring function is a weighted combination with at least nine tunable constants plus tau, alpha, beta, w1, and w2, many configurations can be fit to the labeled set, so the reported F1=0.67 versus UNS's 0.38 may partly reflect a tuning advantage rather than model quality. The comparison is also asymmetric if the UNS reimplementation was not given the same tuning budget. The load-bearing weakness is therefore evaluation leakage: the headline numbers are not demonstrated to be genuine out-of-sample predictions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AUT_BOR, an unsupervised model for loanword detection that uses only language-internal features: n-gram rarity, transition probabilities, length z-scores, and part-of-speech weights are combined into a sigmoid-mapped composite score, then refined via iterative self-training until convergence. A scaled variant adds cross-linguistic alignment scores. The method is evaluated on a self-curated dataset of 5,092 IPA-transcribed entries from six Indo-European languages. The reported results (Table 2) show AUT_BOR achieving F1=0.67 versus F1=0.38 for a reimplemented Prakhya & P (2020) baseline, and the scaled model reaching F1=0.73 and recall=0.91 (Table 4). The central claim is that a purely language-internal, unsupervised approach can outperform earlier unsupervised methods and is extensible to multilingual data.","tokens_in":16430,"tokens_out":5113,"duration_ms":51277,"significance":"If the reported numbers represent genuine out-of-sample predictions, the paper would be a useful contribution to computational historical linguistics: it demonstrates that language-internal cues can detect loanwords without relying on phylogenetic or cross-family information, thereby avoiding the circularity the authors rightly identify in external-information methods. The architecture is novel and clearly described, the code is promised for release, and the error analysis (Section 7.1) is thoughtful and diagnostic. However, the evaluation protocol contains load-bearing flaws—threshold and hyperparameter tuning on the same data used for evaluation, a modified baseline whose fairness is not established, and no variance or out-of-sample checks—so the headline performance figures are not currently credible as evidence for the paper's central claim.","major_comments":[{"comment":"The decision threshold is tuned on the evaluation data itself. Section 4 states that the model outputs labels through 'a tuned decision threshold that balances precision and recall,' and balancing precision and recall requires access to the gold labels. Section 3 similarly says threshold τ is 'to be determined empirically,' and Appendix A2 lists numerous hyperparameters (n-gram rarity thresholds ε1=0.005, ε2=0.02; transition thresholds 0.01 and 0.05; feature weights; POS weights; sigmoid parameters γ and θ; iteration limit 7; convergence criterion 1%; anomaly boosts) that were 'established through preliminary, iterative experimentation.' No held-out split, cross-validation, or nested validation is reported anywhere. Consequently, the scores in Table 2 and Table 4 may be partly fitted values rather than predictions, and the claimed advantage over the UNS baseline may reflect a tuning advantage. The authors must re-run the evaluation with a fixed a-priori threshold or a separate validation set, or at minimum report the tuning procedure and results on a truly held-out portion of the data.","section":"Section 4 and Appendix A2"},{"comment":"The baseline reimplementation is not demonstrated to be fair. The paper introduces 'a key modification' to Prakhya & P (2020) by defining stems as the first two syllables, asserting this 'ensures that our baseline remains faithful to the original model's spirit.' However, no evidence is given that the original stem definition is inapplicable to the six target languages, nor that the modified stem definition does not systematically handicap the baseline. Since the paper's main evidence is the large gap between AUT_BOR (F1 0.67) and UNS (F1 0.38) in Table 2, the comparator must be validated. Please report results with the original stem definition as well, or otherwise demonstrate that the modification is neutral or favorable to the baseline, and consider tuning the baseline's own parameters under the same protocol used for AUT_BOR.","section":"Section 6.1"},{"comment":"The scaled model uses additional empirically fitted constants. Section 5 defines a dynamic threshold θ(x) = α + β((1-C(x)) - B(x)) with 'α, β ∈ R are empirically determined constants,' and also introduces hyperparameters w1, w2, and λ. As with the basic model, no indication is given that these were set on a separate validation set, and no sensitivity analysis is reported. The claim in Section 7 that cross-linguistic scaling yields a '6% performance increase' over the basic model is therefore also at risk of being an artifact of fitting to the same 5,092 labeled entries. Please provide a validation-based selection procedure for these constants or report performance over a grid with error bars.","section":"Section 5"}],"minor_comments":[{"comment":"The notation ℳ−𝒫3 is confusing: the text says this model 'introduces new features,' which is an augmentation, not an ablation. Rename it (e.g., ℳ+𝒫3 or AUT_BOR-AUG) for consistency with Table 2.","section":"Section 6.2"},{"comment":"The rare_ngram_score formula is typeset confusingly: the indicator notation 𝟙 is mixed with the subscripts and the roles of ε1, ε2, c1, c2 are not clearly separated. Please rewrite the equation with standard indicator functions and define each parameter explicitly.","section":"Section 4.1.1"},{"comment":"The length-modifier formula λlen(w) appears to contain a stray '0.5' and unbalanced parentheses: λlen(w) = 1 + (0.5 ⋅ (1/(1+e^{-3(|z_len(w)|-1.5)}) 0.5)). This should be corrected to a clear mathematical expression.","section":"Section 4.2"},{"comment":"The caption reads 'F1 score as a function of training data proportion,' but the model is unsupervised and Section 7's description suggests varying the proportion of the input dataset, not a training split. Clarify what is being varied and how the curve in Figure 3 was obtained.","section":"Figure 3"},{"comment":"The dataset is self-curated and is not made available for reproduction. Please release the gold-standard wordlists and annotations (or provide a link), and report inter-annotator agreement or a second annotation pass given the stated adjudication decisions.","section":"Section 3.1"},{"comment":"The table numbering in the appendix conflicts with the main text: 'Table 1' appears both in Section 3.1 and in Appendix A1. Renumber the appendix tables.","section":"Appendix A1"}],"recommendation":"major_revision","confidential_remarks":"The evaluation-leakage problem is serious and load-bearing, but it is fixable: the authors can re-run the evaluation with a proper held-out split for threshold and hyperparameter selection, report error bars or cross-validation, and address the baseline fairness concern. The paper is within scope for the journal and the architectural contribution is interesting, so I do not recommend rejection at this stage; however, the current version does not support the headline claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this carefully and I mostly line up with the stress-test note. The genuinely new thing here is the architecture: instead of relying on cross-linguistic similarity or phylogeny, it combines rare n-gram statistics, transition probabilities, length normalization, POS weighting, and an iterative self-training loop that refines patterns until convergence. That is a real step beyond the syllabic features in Prakhya & P, and the scaled variant that adds cross-linguistic comparability is a sensible extension. The paper also does honest work in places: per-language results, an ablation that shows n-grams matter, a thoughtful error analysis, and a public code release. The high recall for the scaled model (0.91 aggregate) is interesting if true.\n\nThe soft spot is the one the stress test flags: the numbers look partly fitted. Section 4 explicitly says the decision threshold is \"tuned\" to balance precision and recall, which requires the gold labels. Section 5 uses \"empirically determined constants\" alpha and beta. Appendix A2 says hyperparameters were \"established through preliminary, iterative experimentation,\" including n-gram penalties, POS weights, threshold 0.3, iteration limits, and feature weights. No held-out split, no cross-validation, no error bars. That means the F1=0.67 vs. UNS 0.38 comparison may be a tuning advantage rather than a model advantage. I also worry the baseline reimplementation changed the stem definition to \"first two syllables,\" which could weaken the comparator. The dataset is self-curated, not shared, so I cannot check the gold standard.\n\nThere are minor issues too: some equations are mangled in the PDF (the length modifier formula especially), and the paper casually says Figure 3 shows something while the text around it is thin. These are fixable.\n\nThe central idea is sound, and the paper is honest enough about its own design choices that the evaluation leakage is likely a reporting and protocol problem rather than deliberate deception. But the claim of outperforming the baseline is not established by the current evidence. I would send this to a serious referee, with the explicit request that the author provide a held-out evaluation protocol, error bars, and shared data/code. As submitted, I would cite it cautiously and not rely on the numbers.\n\nFor you: this is worth a look if you care about computational historical linguistics or low-resource loanword detection, but treat Table 2 and Table 4 as promising proofs-of-concept rather than validated results.","headline":"Genuinely new unsupervised architecture for loanword detection, but the headline F1 looks partly fitted to the labeled set; needs held-out validation before the central claim is fully supported.","tokens_in":16928,"tokens_out":1264,"would_cite":true,"duration_ms":15396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A purely language-internal, unsupervised feature-refinement model can detect loanwords in monolingual wordlists and outperforms earlier unsupervised baselines.","keywords":["loanword detection","historical linguistics","unsupervised learning","linguistic features","pattern recognition","phonotactics","iterative refinement","language contact"],"falsifier":"Run the published code on a newly transcribed wordlist from a language outside the six studied, using the paper's exact threshold and constants with no re-tuning, and compare against a fresh gold standard; if F1 falls to the baseline level, the gains come from test-set tuning rather than the method itself.","tokens_in":15868,"feed_emoji":"🔤","tokens_out":11120,"duration_ms":94915,"temperature":0.7,"pith_summary":"This paper tries to prove that loanwords can be detected without any information from outside the language being examined. Its model takes a single monolingual wordlist, scores every word on a set of internally derived linguistic cues, and then iteratively refines the scoring until the predicted set of borrowings stabilises. The reported result is that this approach finds substantially more loanwords than the earlier unsupervised baseline it is compared against, reaching 0.71 recall and 0.63 precision in the basic version and 0.91 recall in a cross-linguistically scaled version. The point of the exercise is practical: if the method works, historical linguists can generate candidate borrowings from a wordlist alone, without first committing to a language family tree or to similarity comparisons with other languages, both of which the paper argues can introduce circularity.","feed_headline":"Unsupervised model finds 71% of loanwords, beats baseline","feed_subtitle":"A wordlist-only method outdoes earlier unsupervised detection across six European languages.","key_machinery":"The load-bearing mechanism is an iterative refinement loop built around a composite probability score. Each word is reduced to a feature vector: how rare its sub-word segments are, how skewed their distribution is, how uncommon the transitions between neighbouring sounds are, how predictable those transitions are on average, and how far the word's length deviates from the vocabulary average. These features are weighted, adjusted for word length and part-of-speech, and mapped through a smooth curve (a sigmoid) to a borrowing probability. The loop then re-estimates the native-word statistics from the words it currently believes to be native, adjusts scores using pattern databases of frequent prefixes, suffixes, and three-character segments, and averages probabilities over iterations until the predicted loanword set stops changing. In the scaled variant, each word's phonological distance from its concept-matched counterparts in other languages is computed by pairwise alignment with context-sensitive match probabilities and articulatory feature distances, then combined with the base probability in a weighted composite score.","core_discovery":"The paper's central claim is that language-internal statistical and structural cues, modelled in an iterative refinement loop, are sufficient to detect a large share of loanwords in monolingual wordlists. The model's design reflects the linguistic assumption that borrowed words stand out on several independent axes at once: rare sub-word sequences, improbable phoneme transitions, unusual word length, and the tendency of borrowings to be nouns rather than function words. These signals are combined into a weighted, sigmoid-mapped probability, and the model then re-estimates the native-vs-loan statistics from the words it currently believes to be native, converging after a few iterations. The discovery, as the paper frames it, is that this hybrid feature-plus-statistics architecture with iterative self-refinement outperforms the prior unsupervised baseline on six languages, and that adding cross-linguistic comparability scores raises recall considerably.","pith_inferences":["Extension: a decisive test would be to run the published code, with the paper's threshold and constants fixed, on a wordlist from a language outside the six studied and compare against a fresh gold standard; if the gains vanish, the reported scores reflect tuning to the evaluation set.","Extension: if the language-internal approach generalises, it could enable loanword studies in language isolates and poorly documented contact zones, where the phylogenetic information that older methods need is unavailable.","Extension: the model's dependence on IPA transcriptions suggests a natural port to tone languages by adding tonal features to the feature vector, which the current design does not exploit."],"forward_implications":["Historical linguists can obtain loanword candidates from a single language's wordlist, avoiding the circularity the paper identifies in phylogenetic and cross-linguistic methods.","The model's unsupervised, self-training design is extensible to low-resource languages where no labelled loanword data exist.","The scaled model's high recall (0.91 aggregated) makes it usable as a first-pass filtering tool whose output a linguist checks manually.","The error analysis indicates the model implicitly learns additional cues such as consonant clusters and vowel quality, suggesting that even unencoded phonological regularities contribute to the results."],"supporting_citations":[{"why":"Supplies the unsupervised baseline the basic model must outperform, using stem-diversity scores that the reimplementation adapts.","marker":"Prakhya & P (2020)"},{"why":"Sets the benchmark precision and recall for non-autonomous loanword detection that motivates the paper's language-internal alternative.","marker":"List & Forkel (2022)"},{"why":"Provides the fabric/pattern transfer distinction that grounds the choice of phonological and phonotactic features as diagnostics.","marker":"Grant (2000)"},{"why":"Gives the typological borrowability statistics (nouns outnumber verbs among borrowings) that justify the part-of-speech weights.","marker":"Tadmor et al., (2010)"},{"why":"Defines loanword as direct lexical transfer, the precise object the model is built to identify.","marker":"Haspelmath (2009)"},{"why":"Argues historical linguistics tasks require models that go beyond supervised inference, motivating the whole unsupervised design.","marker":"List (2024)"}],"fun_headline_variants":["Language-internal cues alone reveal loanwords in six tongues","Iterative self-refinement finds loanwords unsupervised","No external gold: loanword detection goes monolingual","Borrowed-word radar: features plus stats beats baseline","Six-language loanword spotter improves with cross-lingual data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported scores are only trustworthy if the hand-annotated gold standard is accurate and if the decision threshold, feature weights, and scaling constants were not tuned against the same 5,092 entries on which precision and recall are computed.","fun_headline_variants_meta":{"raw":{"variants":["Language-internal cues alone reveal loanwords in six tongues","Iterative self-refinement finds loanwords unsupervised","No external gold: loanword detection goes monolingual","Borrowed-word radar: features plus stats beats baseline","Six-language loanword spotter improves with cross-lingual data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1526,"prompt_tokens":867,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":578}},"tokens_in":483,"tokens_out":659,"duration_ms":6541,"temperature":1.0,"reasoning_tokens":578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:59:27.770562+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published code on a newly transcribed wordlist from a language outside the six studied, using the paper's exact threshold and constants with no re-tuning, and compare against a fresh gold standard; if F1 falls to the baseline level, the gains come from test-set tuning rather than the method itself.","supporting_citations":[{"cited_title":"-M., & Forkel, R","cited_arxiv_id":null,"evidence_quote":"Sets the benchmark precision and recall for non-autonomous loanword detection that motivates the paper's language-internal alternative."}],"review_version":2}