{"id":"3a5c3c2e-8e99-45a4-8e0e-b167ffa339c0","arxiv_id":"2506.10614","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A hybrid parsimony, rule-guided transformation, and evolutionary search pipeline improves unsupervised Latin protoform reconstruction from Romance cognates over a probabilistic baseline.","lead":"This paper presents an unsupervised computer pipeline that reconstructs ancestral Latin word forms from modern Romance language words, using parsimony rules plus an evolutionary search. The hybrid approach outperforms a probabilistic baseline on character accuracy and error rates, and is intended for linguists studying language change.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline is a modified reimplementation with a simplified phoneme model and no original published comparison; the claimed margin over established baselines is not yet established.","rationale":"The reader's weakest assumption concerned the undisclosed and test-tuned rule inventory. I agree that is a real problem, but the most load-bearing issue for the specific claim 'outperforms established baselines' is the baseline itself. The paper's only comparison is a self-modified reimplementation; no numbers from the original Bouchard-Côté et al. (2007) paper or from He et al. (2022) are given. Because the modifications are one-sided (they simplify the baseline while the proposed model gets expert rules and a test-set-tuned rule count), the effect size is not interpretable. This is fixable: a faithful baseline comparison would either confirm or refute the headline. I therefore keep the verdict CONDITIONAL, unchanged relative to the reader: the method is promising and the code is released, but the empirical claim currently exceeds the evidence.","tokens_in":15604,"tokens_out":8527,"duration_ms":89995,"concrete_test":"Re-implement the original Bouchard-Côté et al. (2007) model with its natural phoneme classes and original preprocessing (or obtain the authors' code), run it on the same 5,419 cognate sets with the same data split, and compare C_ACC, CER, VER, EDIT_DIST, and FEAT_DIST against Table 2. If Ranked Prob-Evo's C_ACC advantage over this faithful baseline is less than 5 points, or if the advantage reverses on CER or FEAT_DIST, the central empirical claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in Table 2 is that Ranked Prob-Evo beats the Base model by 15.8 points in character accuracy (50.64 vs 34.85). But Section 8.2 says this Base model is a reimplementation of Bouchard-Côté et al. (2007) 'as implemented by He et al. (2022)', modified by (i) removing the natural phoneme classes and using a simpler vowel/consonant split, (ii) retaining Romanian, and (iii) omitting the 'complex preprocessing pipelines' of the original studies. Each modification is apt to lower baseline accuracy: natural phoneme classes are central to Bouchard-Côté's context-dependent sound-change model, Romanian adds a divergent reflex that complicates inference, and preprocessing (e.g., removing length marks) was part of the original method. The original Bouchard-Côté results are not reported, so the 15.8-point gap could reflect a handicapped baseline rather than the hybrid architecture. This is compounded by Section 9 and Figure 3, where the 14-rule configuration is selected by inspecting test-set performance; the baseline receives no equivalent tuning. Thus the headline 'substantial improvements over established baselines' is not yet supported by a like-for-like comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised three-phase system for reconstructing protoforms from aligned cognate sets: column-based parsimony with probabilistic ranking (Phase I), reverse application of language-specific weighted phonological rules (Phase II), and an evolutionary search that combines both seed sets (Phase III). The method is evaluated on 5,419 Latin/Romance cognate sets from Ciobanu & Dinu (2014), using character accuracy, consonant/vowel error rates, edit distance, and feature distance. The central claim is that the full model (Ranked Prob-Evo) substantially outperforms a reimplemented Bouchard-Côté-style baseline (C_ACC 50.64 vs 34.85; CER 22.31 vs 34.24) and all ablation variants, while also recovering phonemes absent from the attested reflexes.","tokens_in":15809,"tokens_out":5493,"duration_ms":61518,"significance":"If the reported gains are robust, the paper makes a useful contribution: it combines parsimony heuristics, rule-based transformations, and evolutionary search in an unsupervised setting, and it provides a concrete demonstration that injecting linguistically motivated inverse rules can expand a reconstruction system's phonemic inventory beyond what is present in reflexes. The ablation design (ℳranked, ℳunranked, Ranked Path-Prob, Ranked Prob-Evo-Ext) is informative, the evaluation uses linguistically grounded metrics rather than raw accuracy only, and the paper states that code is publicly released. The central quantitative claim, however, currently rests on a modified baseline and on rule-set/hyperparameter selection performed on the same data used for the reported metrics; until those two issues are addressed, the magnitude of the improvement over prior unsupervised methods is uncertain.","major_comments":[{"comment":"The headline 15.8-point C_ACC gap over the base model is not a like-for-like comparison. Section 8.2 states that the baseline is a reimplementation of Bouchard-Côté et al. (2007), as implemented by He et al. (2022), modified by removing natural phoneme classes (using a simple vowel/consonant split), retaining Romanian, and omitting the original preprocessing pipelines. Each of these changes can be expected to lower baseline accuracy, and no result from the original system or from He et al. (2022) is reported on the same data. Please report the unmodified Bouchard-Côté/He baseline under matched preprocessing, or justify quantitatively that the modifications do not systematically disadvantage the baseline.","section":"§8.2, Table 2"},{"comment":"The final model configuration is selected on the same data used for the reported metrics. Figure 3 is used to justify adopting the 14-rule configuration, and Appendix A.2 says hyperparameters \"were established through preliminary iterative assessment\"; no train/development/test split or nested validation is described. As a result, the Table 2 numbers, including the comparison with the baseline, are in-sample estimates and the baseline receives no equivalent tuning. Please provide a split-based or nested evaluation protocol, and report model behavior for all rule-count configurations on held-out data.","section":"§9, Figure 3, Appendix A.2"},{"comment":"The method assumes a complete, a priori set of language-specific transformation rules, but the final inventory is not disclosed. Figure 1 shows only a few sample rules, and Section 9 says the final model uses a manually assembled 14-rule configuration; without the full rule set and weights, the experiments cannot be reproduced and the direction of bias from incomplete or incorrect rules cannot be assessed. Please publish the complete rule inventory with weights and add a sensitivity analysis (e.g., leave-one-rule-out or rule-count subsampling) to show that the main result is not an artifact of particular hand-chosen rules.","section":"§3, §6, Figure 1"},{"comment":"The abstract and conclusions claim \"substantial improvements over established baselines,\" but the experiments compare only with the reimplemented Bouchard-Côté-style model and the paper's own ablations. Since He et al. (2022) is described as having a publicly available implementation and is the most direct unsupervised predecessor, the paper should include that comparison (or a matched reproduction) to support the claim; otherwise the \"established baselines\" language should be scaled back.","section":"§1, §8"}],"minor_comments":[{"comment":"The quantity P(D|S) is called a \"log-scaled score\" but written like a probability; rename it (e.g., S(D|S)) to avoid implying a normalized likelihood.","section":"§5.2"},{"comment":"The likelihood equation contains a typo: \"∑ 1 ∈ ℒψ1\" should be Σ_{l∈ℒ} ψ_1, and the superscript on w_c^l is mismatched; correct the notation.","section":"§7"},{"comment":"\"Portugeuse\" should be \"Portuguese.\"","section":"Figure 1"},{"comment":"The table column order (French, Spanish, Italian, Romanian, Portuguese, Latin, Reconstruction) differs from the prose ordering in the caption and from the earlier description of the dataset; align them for readability.","section":"Appendix Table 1"},{"comment":"The penalty δ treats a gap in a reflex as a mismatch regardless of the candidate character; clarify whether the candidate character can itself be a gap and how column-wise gaps are handled during alignment.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript's contribution is plausible but the evaluation is not yet convincing. I would not reject outright because the issues are fixable, but I would require (i) a matched or original baseline comparison, (ii) a real split or nested validation for rule-count and hyperparameter selection, and (iii) the complete rule inventory. The paper also cites He et al. (2022) but does not compare against their public implementation, which is the most direct unsupervised competitor; this should be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first unsupervised protoform-reconstruction pipeline I know that combines parsimony beam search, manually assembled sound-change rules, and evolutionary search. That combination is new relative to Bouchard-Côté, He et al., and the supervised sequence models, and the paper describes each phase clearly enough to reimplement. The central idea—that reverse-applying expert sound laws can introduce phonemes absent from modern reflexes, which then get selected by the evolutionary search—is linguistically sensible and worth taking seriously. The ablations are a real plus; they let you see that the rule-based transformation phase is doing most of the work.\n\nThat said, the headline number (C_ACC 50.64 vs 34.85) overstates what is currently supported. The baseline is not Bouchard-Côté 2007; it's a reimplementation with natural phoneme classes removed, Romanian retained, and preprocessing omitted. Each of those changes is apt to reduce baseline accuracy, and the original baseline numbers are not reported. So the 15.8-point gap could be partly a handicapped baseline, not the hybrid architecture. The rule-count threshold (14) is also chosen after inspecting test-set performance (Section 9, Figure 3, A.2.2), and the hyperparameters are described as 'established through preliminary iterative assessment' with no held-out validation and no error bars. That is an in-sample selection problem, not a fatal one, but it means the margin is optimistic.\n\nMinor: the assumption of given language-specific rules {r_l} is acknowledged in Section 3, but it's a strong one and the full 14-rule inventory is not published. For a paper selling rule-guided methods, that's a disclosure gap. Also the text in Section 9 has a small contradiction about EDIT_DIST/FEAT_DIST; not important.\n\nOverall: the architecture is a genuine contribution to a small subfield, and the problems are fixable. I'd send it to review, but I'd insist the authors either report the original Bouchard-Côté numbers or run a properly matched baseline, add a validation protocol, and release the exact rule set and code commit. If the gap survives that, it's a solid method paper. If not, it's a useful negative result about how much the baseline matters.","headline":"Genuinely novel hybrid pipeline for protoform reconstruction, but the large accuracy margin is against a modified baseline and a test-set-tuned rule count, so the empirical claim needs a fairer comparison before it convinces.","tokens_in":16424,"tokens_out":2527,"would_cite":false,"duration_ms":30019,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a three-phase hybrid of parsimony-based heuristics, weighted phonological rules, and evolutionary search reconstructs Latin protoforms with higher accuracy and phonological coherence than a purely statistical…","keywords":["protoform reconstruction","historical linguistics","evolutionary algorithms","phonological modeling","rule-based inference","unsupervised learning","Romance languages"],"falsifier":"Run the full Ranked Prob-Evo pipeline on the same 5,419 cognate sets with the rule inventory fixed by cross-validation on a training subset of cognate sets rather than by inspecting test performance; if character accuracy then falls to the baseline level or the held-out feature distance no longer improves, the central claim of rule-guided gain is falsified. A second direct check is to port the claimed 14-rule inventory to another Romance subset or another family and measure whether the accuracy gap persists.","tokens_in":15287,"feed_emoji":"🧬","tokens_out":6837,"duration_ms":75443,"temperature":0.7,"pith_summary":"This paper claims that unsupervised protoform reconstruction is improved by mixing parsimony-based heuristics with manually encoded phonological rules inside an evolutionary search. The central proposal is a three-phase pipeline: generate top-ranked protoform candidates by column-wise parsimony and probabilistic phonological scoring, reverse-apply weighted language-specific sound-change rules to those candidates to produce alternative diachronic pathways, then run an evolutionary algorithm seeded with both sets. On 5,419 Romance cognate sets with Latin gold standards, the full model reaches 50.64% character accuracy, 22.31% consonant error rate, and 26.95% vowel error rate, against 34.85%, 34.24%, and 30.80% for the reimplemented probabilistic baseline. A sympathetic reader would take the contribution to be evidence that data-driven inference alone underuses the phonological inventory, and that linguistically motivated rules can expand the set of phonemes a reconstruction can propose.","feed_headline":"Rule-guided evolution lifts protoform accuracy to 50.6%","feed_subtitle":"Injecting linguist-style sound-change rules into an evolutionary search beats a purely data-driven model.","key_machinery":"The load-bearing mechanism is the hybrid seed population for the evolutionary loop. Phase I builds protoform candidates column by column over pre-aligned reflexes with beam search under a parsimony cost in which a gap and a substitution each cost 1, then ranks the top candidates with a log-scaled phonological plausibility score that rewards context-sensitive changes, brevity, and low Levenshtein distance. Phase II takes the top-ranked candidates and recursively applies inverse language-specific rules $\\mathcal{R}_\\ell^{-1}$, each weighted by a naturalness factor $\\omega(s_i,t_i)=1/(1+\\alpha|\\mathrm{len}(s_i)-\\mathrm{len}(t_i)|\\varphi(s_i,t_i))$ that favors same-class, length-preserving changes, with morphological and phonotactic well-formedness scores filtering the resulting pathways. Phase III runs an evolutionary algorithm seeded with the union of the parsimony candidates and the rule-transformed pathways, eliminating the bottom fifth of candidates each round, mutating vowels, suffixes, and consonant clusters when diversity falls below a threshold, and stopping at one survivor, maximum generations, or fitness convergence. The key point is that the rule-transformed seeds are what let the search escape the phonemic inventory of the reflexes.","core_discovery":"On the paper's own terms, the discovery is that injecting rule-guided transformations into the candidate population materially improves reconstruction quality and phonological coherence. The highest-ranked parsimony reconstruction tends to be phonemically conservative, for example '-anõ' for the Latin '-anum', because it only recombines segments that appear in the reflexes. By reverse-applying weighted sound-change rules to the top parsimony candidates, the model produces seeds like '-anum' that contain phonemes such as /ʊ/ and /m/ absent from all observed reflexes, and the evolutionary search then selects among these seeds under a fitness function combining reflexive likelihood, edit similarity, phonetic-class similarity, and morphological and phonotactic priors. Compared with a reimplementation of the probabilistic expectation-maximization baseline, the resulting Ranked Prob-Evo model improves character accuracy from 34.85% to 50.64%, reduces consonant error rate from 34.24% to 22.31% and vowel error rate from 30.80% to 26.95%, while mean edit distance and feature distance also improve (4.63 versus 5.87, and 2.68 versus 3.36). The paper further reports that the rule-based transformation phase is the single largest source of gain, and that the 14-rule configuration marks the point where surface edit efficiency and phonological feature fidelity balance best.","pith_inferences":["Because the final 14-rule inventory is not published and its size was chosen by inspecting test-set curves, the cleanest test of the paper's central claim is an independent selection of rules on a training split followed by evaluation on a held-out split; that experiment is not reported here.","If the hybrid recipe generalizes, the comparative method itself becomes the inductive bias: any language family with attested sound laws could get a similar parsimony-plus-inverse-rules seed set, so the approach is naturally extensible beyond Romance despite the paper's single-family evaluation.","The model's probabilistic ranking favors phonotactically well-formed forms, which can inflate feature distance on some ablations; a feature-aware ranking term or a feature-weighted edit cost might recover the edit-distance gains without sacrificing phonological fidelity.","The reported non-monotonic effect of rule count suggests an upper bound on useful rule-set size per family, so a practical deployment would need a principled rule-selection criterion rather than a fixed manual configuration."],"forward_implications":["Rule-transformed seeds let the model propose protoform phonemes and phonotactic patterns that never appear in the observed reflexes, directly addressing the inventory conservatism of purely data-driven reconstruction.","The full model outperforms the reimplemented probabilistic baseline on character accuracy, consonant and vowel error rates, edit distance, and feature distance, so the hybrid seed strategy transfers to a standard unsupervised benchmark.","Ablation results indicate that the rule-based transformation phase contributes the largest performance gain, while the evolutionary phase synthesizes the structural variation from parsimony and rule-based candidates.","Rule-set size is a tradeoff knob: growing the rule inventory from 1 to 16 rules gives modest, non-monotonic gains, and the 14-rule configuration is where edit-distance and feature-distance trends balance best.","The same architecture is positioned as a usable baseline for low-resource language contexts where a linguist can supply regular sound correspondences but little data."],"supporting_citations":[{"why":"Defines the probabilistic generative formulation of diachronic phonology that the paper's baseline reimplements and against which the hybrid model is compared.","marker":"Bouchard et al., 2007"},{"why":"Provides the public baseline reimplementation and the initial probability and smoothing settings used for the base model in this paper's experiments.","marker":"He et al., 2022"},{"why":"Supplies the 5,419 IPA-transcribed Romance cognate sets with Latin gold protoforms that the experiments are run on.","marker":"Ciobanu & Dinu, 2014"},{"why":"Curates the Romance benchmark dataset mentioned as evaluation context and provides the supervised sequence-to-sequence baseline discussed in related work.","marker":"Meloni et al., 2021"},{"why":"Defines the edit distance used in the evaluation metrics for reconstruction quality.","marker":"Levenshtein, 1966"}],"fun_headline_variants":["Evolutionary search plus linguist rules revives dead words","Unsupervised protoform reconstruction hits 50.6% accuracy","Hybrid heuristics outlearn data-only models for lost language","Rule-guided evolution improves ancient word reconstruction by 15%","Turning sound-change rules into seeds boosts protoform accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on a handwritten, language-specific set of phonological rules supplied in advance, and the final 14-rule configuration is neither fully published nor selected on a separate validation set; if those rules are wrong, incomplete, or tuned to the test data, the reported advantage over the data-driven baseline would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Evolutionary search plus linguist rules revives dead words","Unsupervised protoform reconstruction hits 50.6% accuracy","Hybrid heuristics outlearn data-only models for lost language","Rule-guided evolution improves ancient word reconstruction by 15%","Turning sound-change rules into seeds boosts protoform accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000138,"raw_usage":{"total_tokens":1148,"prompt_tokens":934,"completion_tokens":214,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":131}},"tokens_in":550,"tokens_out":214,"duration_ms":3597,"temperature":1.0,"reasoning_tokens":131,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:21:39.463145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full Ranked Prob-Evo pipeline on the same 5,419 cognate sets with the rule inventory fixed by cross-validation on a training subset of cognate sets rather than by inspecting test performance; if character accuracy then falls to the baseline level or the held-out feature distance no longer improves, the central claim of rule-guided gain is falsified. A second direct check is to port the claimed 14-rule inventory to another Romance subset or another family and measure whether the accuracy gap persists.","supporting_citations":[],"review_version":1}