REVIEW 3 major objections 5 minor 33 references
The Cognate Data Bottleneck in Language Phylogenetics
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read No feasible automatic route to larger cognate datasets exists, the paper argues, and BabelNet extraction fails the phylogenetic test.
desk verdict Solid BabelNet-specific negative result, but the abstract's universal 'no feasible approach' claim overreaches. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the BabelNet synset: a group of senses from different languages that encode the same concept, which provides the parallelism needed to build columns of a character matrix. Around this, the pipeline selects languages and synsets, obtains IPA transcriptions either directly from BabelNet or automatically via epitran, tokenizes them with ipatok, clusters cognates with LexStat, and encodes presence and absence as binary columns. The argument's check on this machinery is the generalized quartet (GQ) distance between the inferred maximum-likelihood tree and the Glottolog gold standard, together with Pythia difficulty scores; the GQ distance staying above 0.4 is the main evidence that the automatic pipeline destroys phylogenetic signal.
What would settle it
Run the same extraction and inference protocol on a different multilingual resource with broader phonetic coverage, such as Wiktionary, and compare the resulting maximum-likelihood tree to the Glottolog gold standard; a GQ distance substantially below 0.4 would show that the bottleneck is not universal.
Extended reading notes
Core claim
The central claim is that automatically generated cognate character matrices from BabelNet are not suitable for phylogenetic inference, and that no feasible automatic route to larger cognate datasets currently exists. The authors construct binary presence-absence matrices from BabelNet synsets—sets of senses from different languages that express the same concept—by automatically transcribing words with epitran, tokenizing with ipatok, and clustering cognates with LexStat. Maximum likelihood trees inferred from these matrices with RAxML-NG under a BIN+G model all have GQ distances above 0.4 to the Glottolog gold standard, and Pythia difficulty scores between 0.620 and 0.926, indicating weak phylogenetic signal. Manual data, by contrast, reaches a GQ distance of 0.024 to the same gold standard. The paper therefore concludes that more complex evolutionary models and machine-learning-based phylogenetics cannot currently be applied to cognate data, and that constructing larger datasets requires fundamentally new ways of acquiring language data, such as machine learning on sound recordings.
Load-bearing premise
The conclusion that no automatic method can produce larger cognate datasets rests on assuming that BabelNet's failure is representative of every other multilingual resource, even though only BabelNet was actually tested.
Editorial extensions
If this is right
- More complex evolutionary models for cognate data cannot be parameterized reliably until much larger datasets exist, so their use in historical linguistics is blocked.
- Machine-learning-based phylogenetic inference methods cannot currently be applied to cognate data with available resources.
- Manually assembled cognate datasets, such as the iecor database, remain the practical standard for phylogenetic inference in historical linguistics.
- Efforts to enlarge cognate datasets must shift from harvesting written multilingual resources to fundamentally different data acquisition, for instance machine learning on sound recordings.
- The evaluation protocol combining GQ distance with Pythia difficulty scores offers a reusable test for any future claim of an automatically generated cognate dataset.
Reading between the lines
- The universal negative 'no feasible approach' is extrapolated from BabelNet alone; a phonetically richer resource such as Wiktionary, which the paper mentions but does not test, could in principle support high-quality extraction and refute the universal claim.
- If the bottleneck holds, progress in computational historical linguistics may come from bypassing written text entirely, for example by training models directly on audio corpora to infer language relationships.
- The observed failure profile suggests a quantitative target for future extraction methods: improving grapheme-to-phoneme conversion, joint transcription and tokenization, and concept-list curation would need to move GQ distance from roughly 0.4 down toward 0.1 to be useful.
- Because the authors deliberately kept only one 'main sense' per synset and excluded synonyms, the extracted matrices contain no polymorphism; testing matrices that retain synonyms is a natural next step to see whether some of the lost signal is recoverable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that computational phylogenetics for cognate data faces a data bottleneck: sophisticated models and machine-learning-based methods require substantially larger cognate datasets than the manually curated ones currently available. The authors propose fully automatic extraction as a solution and implement a pipeline based on BabelNet: they select synsets, obtain main senses, transcribe orthography to IPA with epitran, tokenize with ipatok, and cluster cognates with LexStat. They evaluate the resulting binary character matrices by density/AMC, by Maximum Likelihood tree inference compared to Glottolog gold-standard trees via the GQ distance, and by Pythia difficulty scores. Their empirical finding is that all twelve BabelNet-derived matrices are sparse (AMC at most 0.49) and that inferred trees have GQ distances of 0.394–0.647, versus 0.024 for the manually assembled iecor dataset, with Pythia scores of 0.620–0.926. From this they conclude that automatically extracted BabelNet matrices are not suitable for phylogenetic inference and, more strongly, that no feasible approach to automatically generate larger cognate datasets currently exists.
Significance. If confined to the BabelNet extraction pipeline, the paper is a useful negative empirical result: it demonstrates with external gold standards (Glottolog trees, NorthEuraLex, lexibank) that a specific automatic pipeline yields character matrices with weak phylogenetic signal. The manuscript ships reproducible code, reports error rates for epitran and ipatok, and is not circular because evaluations are anchored to independently constructed references. However, the significance of the broader claim is limited. The universal negative in the abstract and Section 4, namely that no feasible automatic approach exists for larger cognate datasets, is not supported by the experiments, which test a single resource and a single extraction design. The BabelNet-specific evidence is solid, but the title and abstract make a much stronger statement that goes beyond the data.
major comments (3)
- [Abstract / Section 4] The central claim 'there exists no feasible approach to automatically generate larger cognate datasets' is a universal negative, but the evidence consists of one extraction pipeline applied to one resource. Section 4 itself hedges: other resources 'might encounter analogous, potentially more pronounced, challenges' and 'we therefore expect' poorer quality. This is extrapolation, not demonstration. The paper itself notes in Section 2.2 that Wiktionary provides IPA transcriptions in 4,400 languages, and a Wiktionary/DBnary-based extraction with native IPA would bypass the epitran step identified as a major error source in Section 3.2; this is a concrete untested alternative that the universal claim must address. Please either restrict the conclusion to BabelNet-based extraction or provide evidence from additional resources.
- [Section 3.2] The error-rate audit for epitran and ipatok is performed only on NorthEuraLex, and the authors explicitly state that 'the quality of the corresponding transcriptions is not examined here' for the many BabelNet dense languages not covered by that resource. This does not invalidate the BabelNet-specific negative conclusion, since the GQ and Pythia results in Tables 5 and 6 are computed on the actual extracted matrices. However, Section 4 uses these measured error rates as a reason to expect analogous or worse problems in other resources; that transfer is unsupported and should be explicitly labeled as an assumption rather than a result.
- [Section 3.1.2 / Tables 5-6] The extraction design excludes all non-main senses, thereby removing synonyms and polymorphism by construction. The paper does not test whether this filtering itself removes phylogenetic signal. Since the conclusion that the BabelNet data are unsuitable is based on matrices produced with this restriction, the negative result is about one particular extraction design, not about BabelNet as a resource in general. A sensitivity analysis that retains additional senses, or at least an explicit discussion of this design choice as a potential cause of signal loss, is needed before generalizing to 'no feasible approach.'
minor comments (5)
- [Abstract] Typo: 'models an machine learning-based techniques' should read 'models and machine learning-based techniques.'
- [Section 2.2] 'Wikitionary' should be 'Wiktionary.'
- [Section 3.1.2 / Section 3.3.1] Cross-references to 'section 3.1.2 illustrates' and 'section 3.3.1 provides sparsity plots' should refer to Figure 1 and Figures 2/3, respectively; Section 3.2 refers to 'section 3.2' for the e1/e2 values, which are presented in Table 1.
- [Section 3.1] The resource version is described as 'BabelNet v 5.0' at the start of Section 3 and as 'version 5.3' in Section 3.1; please make the version identifiers consistent.
- [References] Several citations appear as 'Häuser et al.' with no year or full reference; please complete these entries.
Circularity Check
No circular derivation: the negative result is anchored to external Glottolog gold-standard trees and NorthEuraLex/lexibank error audits, with only a peripheral self-citation in the Pythia difficulty scores.
full rationale
The paper's load-bearing evidence is external. Section 3.3.2 compares maximum-likelihood trees inferred from BabelNet character matrices against Glottolog gold-standard trees using the GQ distance, reporting distances of 0.394-0.647 for the BabelNet matrices and 0.024 for manually curated iecor. This comparison does not reduce to any fitted parameter or to the paper's own construction. Section 3.2 similarly measures epitran and ipatok error rates against externally supplied NorthEuraLex transcriptions and lexibank tokenizations, so those quality audits are also externally grounded. The Pythia difficulty scores cited from Haag et al. and Haag and Stamatakis are a self-citation by one of the present authors, and the phrase 'Pythia ground truth difficulty scores' is somewhat overstated, but these scores are only corroborative: the 'not suitable for phylogenetic inference' conclusion is already carried by the GQ distances, and the Pythia scores are not defined in terms of the paper's target claim about cognate data acquisition. The abstract's universal statement that no feasible automatic approach exists is an extrapolation from the BabelNet experiment, and it is explicitly hedged in Section 4 as an expectation about other resources, but extrapolation is not circularity. No equation is defined in terms of the result it is supposed to support, and no fitted input is renamed as a prediction. The only mild self-referential element is the Pythia tool, and it is not load-bearing; therefore the circularity score is minimal.
Assumptions & free parameters
free parameters (3)
- top-k synset threshold =
5000
- number of ML tree searches =
20 per matrix
- main-sense exclusion rule =
1 sense per language per synset
assumptions (5)
- domain assumption Glottolog trees are a reliable gold standard for assessing inferred language trees (GQ distance benchmark).
- domain assumption Epitran and ipatok error rates measured on NorthEuraLex apply to BabelNet languages.
- ad hoc to paper The BabelNet main-sense restriction faithfully represents the most frequent sense, and excluding other senses does not remove informative variation.
- domain assumption The current manually collected cognate datasets are too small for complex models and ML phylogenetics.
- domain assumption LexStat with default settings produces suitable cognate clusters for automatically transcribed data.
Cite this review
Pith. "Pith review of The Cognate Data Bottleneck in Language Phylogenetics." pith.science (2026). https://pith.science/paper/4KSPQLLU
@misc{pith2026250700911,
author = {Pith},
title = {Pith review of: The Cognate Data Bottleneck in Language Phylogenetics},
year = {2026},
howpublished = {\url{https://pith.science/paper/4KSPQLLU}},
note = {Machine review of arXiv:2507.00911}
}
read the original abstract
To fully exploit the potential of computational phylogenetic methods for cognate data one needs to leverage specific (complex) models an machine learning-based techniques. However, both approaches require datasets that are substantially larger than the manually collected cognate data currently available. To the best of our knowledge, there exists no feasible approach to automatically generate larger cognate datasets. We substantiate this claim by automatically extracting datasets from BabelNet, a large multilingual encyclopedic dictionary. We demonstrate that phylogenetic inferences on the respective character matrices yield trees that are largely inconsistent with the established gold standard ground truth trees. We also discuss why we consider it as being unlikely to be able to extract more suitable character matrices from other multilingual resources. Phylogenetic data analysis approaches that require larger datasets can therefore not be applied to cognate data. Thus, it remains an open question how, and if these computational approaches can be applied in historical linguistics.
Figures
Reference graph
Works this paper leans on
-
[4]
URL https://doi.org/10.1007/978-3-319-53817-4_4 . Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. Xlm-v: Overcoming the vocabulary bottleneck in multi- lingual masked language models,
-
[7]
doi: 10.18653/v1/2023.acl-long.235
Associa- tion for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.235. URL https: //aclanthology.org/2023.acl-long.235. Ramanathan Guha, Dan Brickley, and Steve MacBeth. Schema.org: Evolution of structured data on the web. Queue, 13:10–37, 11
-
[10]
URL https://aclanthology.org/2024.naacl-long.141
18653/v1/2024.naacl-long.141. URL https://aclanthology.org/2024.naacl-long.141. Felipe Almeida and Geraldo Xex´ eo. Word embeddings: A survey,
work page 2024
-
[11]
Benjamin Heinzerling and Michael Strube
doi: 10.1126/science.abg0818. Benjamin Heinzerling and Michael Strube. BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Mae- gaard, Joseph Mariani, H´ el` ene Mazo, Asuncion Moreno, Jan O...
-
[14]
URL http://dx.doi.org/10.18653/v1/2023.acl-long.61
doi: 10.18653/ v1/2023.acl-long.61. URL http://dx.doi.org/10.18653/v1/2023.acl-long.61. Gerhard J¨ ager, Johann-Mattis List, and Pavel Sofroniev. Using support vector machines and state-of-the-art algorithms for phonetic alignment to identify cognates in multi-lingual wordlists. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, Proceedings o...
-
[15]
doi: 10.1038/sdata.2018.189. Gerhard J¨ ager. Phylogenetic linguistic inference from acoustic speech data: Ideas for a novel research paradigm,
-
[18]
doi: 10.1007/978-3-319-53817-4
ISBN 978-3-319-53817-4. doi: 10.1007/978-3-319-53817-4
-
[20]
URL https://arxiv.org/abs/2301.10472. Johann-Mattis List. Lexstat: automatic detection of cognates in multilingual wordlists. pages 117–125, 04
Show all 33 references
-
[21]
URL https: //doi.org/10.31826/jlr-2014-110111
doi: doi:10.31826/jlr-2014-110111. URL https: //doi.org/10.31826/jlr-2014-110111. Johann-Mattis List, Simon J. Greenhill, Cormac Anderson, Thomas Mayer, Tiago Tresoldi, and Robert Forkel. Clics2: An improved database of cross-linguistic colexifications assem- bling lexical dat...
2014 doi
-
[22]
URL https://doi.org/10.1515/ lingty-2018-0010
doi: doi:10.1515/lingty-2018-0010. URL https://doi.org/10.1515/ lingty-2018-0010. Johann-Mattis List, Robert Forkel, Simon Greenhill, Christoph Rzymski, Johannes Englisch, and Russell Gray. Lexibank, a public repository of standardized wordlists with com- puted phonological an...
2018 doi
-
[23]
Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Cecconi
doi: 10.1016/j.artint.2012.07.001. Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Cecconi. Ten years of babelnet: A survey. pages 4559–4567, 08
2012 doi
-
[24]
Luca Nesterenko, Luc Blassel, Philippe Veber, Bastien Boussau, and Laurent Jacob
doi: 10.24963/ijcai.2021/620. Luca Nesterenko, Luc Blassel, Philippe Veber, Bastien Boussau, and Laurent Jacob. Phylo- former: Fast, accurate and versatile phylogenetic reconstruction with deep neural networks. working paper or preprint, November
2021 doi
-
[25]
Cross-lingual name tagging and linking for 282 languages
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross-lingual name tagging and linking for 282 languages. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vo...
1946
-
[28]
doi: 10.18653/v1/P19-1627
Association for Computational Linguistics. doi: 10.18653/v1/P19-1627. URL https://aclanthology.org/P19-1627. Laurent Sagart, Guillaume Jacques, Yunfan Lai, Robin Ryder, Valentin Thouzeau, Simon J. Greenhill, and Johann-Mattis List. Dated language phylogenies shed light on the ...
-
[29]
URL https://www
doi: https://doi.org/10.1073/pnas.1817972116. URL https://www. pnas.org/content/early/2019/04/30/1817972116. Rahul Sharma, Kunal Dhawan, and Balakrishna Pailla. Phonetic word embeddings,
2019 doi
-
[30]
Robyn Speer, Joshua Chin, and Catherine Havasi
URL https://arxiv.org/abs/2109.14796. Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge,
-
[31]
doi: 10.1093/molbev/msad277
ISSN 1537-1719. doi: 10.1093/molbev/msad277. URL https://doi.org/10.1093/molbev/msad277. Piek Vossen, editor. EuroWordNet: a multilingual database with lexical semantic networks for European Languages. Kluwer,
-
[33]
Vil´ em Zouhar, Kalvin Chang, Chenxuan Cui, Nathaniel Carlson, Nathaniel Robinson, Mrin- maya Sachan, and David Mortensen
doi: 10.23919/ChiCC.2017.8028251. Vil´ em Zouhar, Kalvin Chang, Chenxuan Cui, Nathaniel Carlson, Nathaniel Robinson, Mrin- maya Sachan, and David Mortensen. Pwesuite: Phonetic word embeddings and tasks they facilitate,
2017
-
[1998]
Automatic synonym extraction using word2vec and spectral clustering
Li Zhang, Jun Li, and Chao Wang. Automatic synonym extraction using word2vec and spectral clustering. In 2017 36th Chinese Control Conference (CCC) , pages 5629–5632,
2017
-
[2011]
Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli
doi: 10.1371/journal.pone.0020109. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. Word sense disambigua- tion: A unified evaluation framework and empirical comparison. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, Proceedings of the 15th Con...
-
[2012]
20 doi: https://doi.org/10.1016/j.websem.2012.01.001
ISSN 1570-8268. 20 doi: https://doi.org/10.1016/j.websem.2012.01.001. URL https://www.sciencedirect. com/science/article/pii/S1570826812000030. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick...
2012 doi
-
[2014]
Yang Li and Tao Yang
doi: 10.3233/SW-140134. Yang Li and Tao Yang. Word Embedding for Understanding Natural Language: A Survey , pages 83–104. Springer International Publishing, Cham,
-
[2015]
Julia Haag and Alexandros Stamatakis
doi: 10.1145/2857274.2857276. Julia Haag and Alexandros Stamatakis. Pythia 2.0: New data, new prediction model, new features. bioRxiv,
-
[2016]
URL https: //aclanthology.org/2016.gwc-1.9
Global Wordnet Association. URL https: //aclanthology.org/2016.gwc-1.9. Jordan Boyd-Graber, Christiane Fellbaum, Daniel Osherson, and Robert Schapire. Adding dense, weighted connections to WordNet. GWC 2006: 3rd International Global WordNet Conference, Proceedings, 01
2016
-
[2017]
doi: 10.18653/v1/P17-1178
Association for Computational Linguistics. doi: 10.18653/v1/P17-1178. URL https:// aclanthology.org/P17-1178. Simone Pompei, Vittorio Loreto, and Francesca Tria. On the accuracy of language trees. PloS one, 6:e20109, 06
-
[2018]
, Miyazaki, Japan, May 7-12, 2018
2018
-
[2019]
org/abs/1810.04805
URL https://arxiv. org/abs/1810.04805. Aron B. Dolgopolsky. Gipoteza drevnejˇ sego rodstva jazykovych semej severnoj evrazii s vero- jatnostej toˇ cky zrenija,
-
[2021]
URL https://ideas.repec.org/a/ nat/natcom/v12y2021i1d10.1038_s41467-021-22073-8.html
doi: 10.1038/s41467-021-22073-. URL https://ideas.repec.org/a/ nat/natcom/v12y2021i1d10.1038_s41467-021-22073-8.html . Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madia...
-
[2022]
20.496790
doi: 10.1101/2022.06. 20.496790. Harald Hammarstr¨ om, Robert Forkel, Martin Haspelmath, and Sebastian Bank. Glottolog 4.7,
2022 doi
-
[2023]
org/abs/1901.09069
URLhttps://arxiv. org/abs/1901.09069. Dana Azouri, Shiran Abadi, Yishay Mansour, Itay Mayrose, and Tal Pupko. Harnessing ma- chine learning to guide phylogenetic-tree search algorithms. Nature Communications, 12(1): 1–9, December
1901 arXiv
-
[2024]
URL https: //aclanthology.org/2024.eacl-long.14
Association for Computational Linguistics. URL https: //aclanthology.org/2024.eacl-long.14. V. S. D. S. Mahesh Akavarapu and Arnab Bhattacharya. Automated cognate detection as a supervised link prediction task with cognate transformer, 2024a. V.S.D.S.Mahesh Akavarapu and Arnab...
2024
-
[2025]
URL https://www.biorxiv.org/ content/early/2025/03/28/2025.03.25.645182
doi: 10.1101/2025.03.25.645182. URL https://www.biorxiv.org/ content/early/2025/03/28/2025.03.25.645182. Julia Haag, Dimitri H¨ ohler, Ben Bettisworth, and Alexandros Stamatakis. From easy to hope- less - predicting the difficulty of phylogenetic analyses. bioRxiv,
2025 doi
-
[8227]
A systematic exploration of current limitations of cognate-based phylogenetic inference
Luise H¨ auser, Gerhard J¨ ager, and Alexandros Stamatakis. A systematic exploration of current limitations of cognate-based phylogenetic inference. Luise H¨ auser, Gerhard J¨ ager, Taraka Rama, Johann-Mattis List, and Alexandros Stamatakis. Are sounds sound for phylogenetic r...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.