REVIEW 3 major objections 7 minor 43 references
Mining for Species, Locations, Habitats, and Ecosystems from Scientific Papers in Invasion Biology: A Large-Scale Exploratory Study with Large Language Models
T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A general-purpose LLM, without any fine-tuning, can extract species, locations, habitats, ecosystems, and their relationships from invasion biology abstracts at scale.
desk verdict A transparent exploratory corpus paper that ships a reusable workflow and dataset, but the lack of any quantitative accuracy check is a real limit that keeps the headline result at 'potential' rather than 'validated'. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the specialize-generalize-extract schema workflow. The LLM first acts as a research assistant in invasion biology and proposes a semantic model for each paper, then reviews those paper-specific models and proposes a standardized JSON schema for the whole corpus, and finally uses that schema as the extraction target for every abstract. The schema itself is the load-bearing output format: it defines species by name, role, and taxonomy level; locations by name, category, and geopolitical detail; ecosystems by type and scope; habitats by type and ecosystem link; and relationships by name, type, directionality, and context. This two-stage schema discovery lets an open-ended extraction task operate without a pre-existing labeled dataset or ontology.
What would settle it
Select 100 abstracts at random from the released corpus, have ecologists annotate the four entity types and relations, and measure agreement with the LLM output; if precision or recall falls below a level ecologists would accept for systematic review, the central claim that the dataset supports ecological research is not established.
Extended reading notes
Core claim
The central claim is that a single general-purpose LLM, prompted with a two-stage schema-discovery workflow, can structure ecological knowledge from scientific abstracts in invasion biology. In the specialization stage the model proposes per-paper schemas for the four target entities; in the generalization stage it merges those into one standardized JSON schema; in the extraction stage it populates that schema across the whole corpus. The paper reports that of 12,636 abstracts, the model classified 1,740 as outside invasion biology and returned structured extractions for the remaining 10,896. The extracted data include species names and roles, geopolitical locations, habitat and ecosystem types, and typed relationships, and the paper presents qualitative summaries of the most frequent species, locations, ecosystems, and relation categories.
Load-bearing premise
The load-bearing premise is that the LLM's extractions are accurate enough to be useful, since the paper reports no quantitative evaluation and explicitly says the results are explored qualitatively.
Editorial extensions
If this is right
- The released corpus gives invasion biology researchers a structured index of more than 10,000 papers, searchable by species, location, habitat, ecosystem, and relation type.
- The extracted species roles, such as native, introduced, invasive, biological control agent, and ecosystem engineer, support large-scale comparisons of how different kinds of organisms spread.
- The relation categories, dominated by invasion, competition, impact, and predation, offer a first-pass network of ecological interactions derived from abstracts alone.
- The specialize-generalize workflow can be reapplied to other scientific domains that lack annotated information-extraction datasets.
- The habitat-ecosystem pairings highlight where invasions occur, from freshwater systems and Mediterranean Sea ecosystems to urban and agricultural landscapes.
Reading between the lines
- A natural next step is to human-annotate a sample of the 10,896 outputs and measure precision and recall; that single step would convert this proof of concept into a quantitative resource.
- Because most papers contributed only abstracts, running the same schema on the roughly 2,800 available full texts could reveal finer habitat-ecosystem relations and rarer species mentions that abstracts omit.
- If extraction quality is adequate, the workflow could be run periodically over newly published abstracts to track emerging invaders and shifting ranges in near-real time.
- The paper's own examples of noise, such as generic phrases like 'native species' recorded as species names, suggest that a lightweight post-filtering step would improve the dataset's usability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports an exploratory study in which GPT-4o, with no domain-specific fine-tuning, extracts four entity types—species, location, habitat, and ecosystem—together with relationships from the titles and abstracts (and some full texts) of invasion-biology papers. The authors assemble a corpus of 12,636 papers with abstracts from the Invasion Biology Corpus via the ORKG ASK service, develop a standardized JSON schema through a two-stage, LLM-driven process (per-paper schemas followed by a merge of three model-generated variants), and apply the resulting schema to the full corpus; the model labels 1,740 papers as out of scope, leaving 10,896 papers with structured extractions. The results are presented qualitatively: species roles and names, frequently mentioned locations, ecosystems and habitats, and relation types such as invasion (814 mentions), competition (429), and impact (349). The released Zenodo corpus (doi:10.5281/zenodo.13956882) and GitHub code are the paper's concrete deliverables, and the authors state plainly in Section 1 that they cannot evaluate the results quantitatively.
Significance. If the extractions are faithful to the source texts, this is a useful large-scale resource for invasion biology: structured extractions for 10,896 papers would support systematic reviews, hypothesis generation, and downstream curation, and the two-stage schema-discovery workflow is a reusable methodological contribution. The paper's strengths should be credited: the corpus and code are publicly released with a DOI; the prompt-based pipeline is described in sufficient detail to reproduce; an ecologist was consulted to fix the entity definitions in Table 1; and the authors are unusually candid, explicitly stating in Section 1 that quantitative evaluation was not performed and acknowledging in Section 3.4 that generic terms such as 'native species' were extracted as species names. This candor is also the paper's central vulnerability: the load-bearing assumption of acceptable extraction accuracy is stated as an assumption, not measured, so the significance is conditional on a validation the manuscript does not provide.
major comments (3)
- [Section 1; Sections 3.4-5] The paper states in Section 1 that 'we cannot evaluate the results quantitatively,' and no precision, recall, F1, manual sample verification, or inter-annotator agreement appears anywhere in the paper; Section 3.4 offers only descriptive statistics and selected examples. This is load-bearing because the conclusion in Section 5 claims that the study 'demonstrates the potential of LLMs for advancing IE in invasion biology' and Section 3.4 recommends the output for tasks such as systematic or scoping reviews, recommendations that presuppose that the extracted JSON reflects the biology in the abstracts. The paper itself documents noise in Section 3.4 ('the extraction also included generic terms (e.g., 'native species' and 'native plants') as species names') but never quantifies how much of the 10,896-paper output is affected, and examples such as 'ballast water associated with the marine ecosystem' in the habitat analysis suggest that vector-habitat conflation is present. A human audit of a random sample, say 200 papers with per-entity precision and recall computed against the abstract text, would directly test the central claim and is clearly feasible; without such an error profile, the qualitative insights cannot be distinguished from model artifacts.
- [Sections 3.1.1, 3.1.2, 3.2.1] The schema-discovery pipeline is circular in a way that affects the validity of the results: GPT-4o proposes the per-paper schemas in the specialize stage (Section 3.1.1), merges three of its own outputs into the final schema in the generalize stage (Section 3.1.2), and then populates that schema in the extract stage (Section 3.2.1). The internal coherence of the resulting JSON is therefore partly an artifact of a single model family being consistent with itself, not evidence that the schema captures the ecology described in the papers. The ecologist consultation behind Table 1 mitigates the entity definitions, but the relationship schema and the container structure were model-generated, and the generalize stage merged schemas derived from only nine papers; no check indicates that the final schema covers the phenomena present in the full 12,636-paper corpus. An external check, for example having a domain expert independently evaluate the merged schema against a held-out set of abstracts, should be added before the schema is presented as the 'most flexible' representation.
- [Section 3.4] The model classified 1,740 of the 12,636 papers as outside the scope of invasion biology and responded with 'N/A,' yet the accuracy of this binary relevance filter is never assessed. The specialize stage (Section 3.1.1) encountered one out-of-scope paper in a sample of ten, showing that such false positives exist in the source corpus, but nothing is reported about the precision of the 1,740 exclusions or about false negatives, that is, in-scope papers the model wrongly discarded. Because the 10,896-paper set is the denominator for every count in Section 3.4 and the basis of the released dataset, an unmeasured filter-error rate propagates into all of the paper's descriptive findings. A manual audit of a random sample of the excluded papers would quantify this and should be reported alongside the extraction evaluation.
minor comments (7)
- [Section 3.4] Several 'available here' placeholders (for the roles list, the species list, the location data, and the habitat-ecosystem pairs) are not resolved to URLs in the manuscript; each should point to a concrete file or query in the repository.
- [Section 3.4] It is not defined whether counts such as 'Australia (406)' and 'invasion (814)' refer to papers, mentions, or unique extractions; the counting unit should be stated once and used consistently.
- [Tables 1 and 2] Both Table 1's Habitat example and Table 2's 'specifics' property write 'litoral,' which should be 'littoral.'
- [Section 2.3 / Figure 1] The text states that full-text availability begins in 1990, while Figure 1's horizontal axis starts in 2004; the caption says the figure covers the past 20 years, but the axis range should be stated explicitly to avoid an apparent contradiction.
- [Footnote 1, Section 3.1.1] Footnote 1 redefines in-context learning as 'only providing a detailed task description without examples,' which conflicts with the standard few-shot meaning of the term in the cited Brown et al. (2020); replacing 'in-context learning' with 'zero-shot instruction following' would avoid terminological confusion.
- [Section 3.4] The habitat-ecosystem examples include 'ballast water associated with the marine ecosystem,' but ballast water is a transport vector in invasion biology rather than a habitat; if this pairing is representative, the extraction is conflating vectors with habitats, and the discussion should address such systematic errors explicitly.
- [Figure 2] The label 'Regional Euro-Asian Biological...' is truncated in Figure 2; the full publisher name should be given.
Circularity Check
No significant circularity: the study is an exploratory, unvalidated extraction pipeline, not a derivation whose outputs are forced by its inputs.
full rationale
The paper makes no quantitative accuracy claim and explicitly disclaims one: 'Since we cannot evaluate the results quantitatively, we explore them qualitatively' (Section 1). The extraction targets (species, location, habitat, ecosystem) are defined by the authors in Table 1, and the final extraction schema was human-finalized after comparing three LLM-generated variants (Section 3.1.2), so the schema is not purely the model's own output. The same LLM both proposes and populates the schema, which is a genuine validation weakness, but it is not an input-output equivalence by construction: the model could return wrong species or locations, and the paper does not present a fitted parameter renamed as a prediction. The only self-citation of note (Brinner et al. 2022, co-authored by Zarrieß) is used to claim scarcity of invasion-biology resources and is not load-bearing for the central claim. Lack of a gold-standard benchmark is a correctness risk, not circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The four entity types (species, location, habitat, ecosystem), with the definitions given in Table 1, are the correct and sufficient extraction targets for invasion biology information extraction.
- domain assumption The subset of the Invasion Biology Corpus available through ASK (12,636 of 49,438 DOIs) is representative of the full literature on invasion biology.
- ad hoc to paper Zero-shot prompting without few-shot examples is sufficient for the LLM to follow the extraction schema accurately.
- ad hoc to paper The final schema merged from only three LLM-generated variants is an adequate and unbiased representation of the domain.
Cite this review
Pith. "Pith review of Mining for Species, Locations, Habitats, and Ecosystems from Scientific Papers in Invasion Biology: A Large-Scale Exploratory Study with Large Language Models." pith.science (2026). https://pith.science/paper/2ZT2JGUM
@misc{pith2026250118287,
author = {Pith},
title = {Pith review of: Mining for Species, Locations, Habitats, and Ecosystems from Scientific Papers in Invasion Biology: A Large-Scale Exploratory Study with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/2ZT2JGUM}},
note = {Machine review of arXiv:2501.18287}
}
read the original abstract
This paper presents an exploratory study that harnesses the capabilities of large language models (LLMs) to mine key ecological entities from invasion biology literature. Specifically, we focus on extracting species names, their locations, associated habitats, and ecosystems, information that is critical for understanding species spread, predicting future invasions, and informing conservation efforts. Traditional text mining approaches often struggle with the complexity of ecological terminology and the subtle linguistic patterns found in these texts. By applying general-purpose LLMs without domain-specific fine-tuning, we uncover both the promise and limitations of using these models for ecological entity extraction. In doing so, this study lays the groundwork for more advanced, automated knowledge extraction tools that can aid researchers and practitioners in understanding and managing biological invasions.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Stein Aerts, Diether Lambrechts, Sunit Maity, Peter Van Loo, Bert Coessens, Frederik De Smet, Leon-Charles Tranchevent, Bart De Moor, Peter Marynen, Bassem Hassan, et al. 2006. Gene prioritization through genomic data fusion. Nature biotechnology, 24(5):537--544
work page 2006
-
[4]
Dean Allemang and Juan Sequeda. 2024. Increasing the llm accuracy for question answering: Ontologies to the rescue! arXiv preprint arXiv:2405.11706
arXiv 2024
-
[5]
Xavier Amatriain, Ananth Sankar, Jie Bing, Praveen Kumar Bodigutla, Timothy J Hazen, and Michaeel Kazi. 2023. Transformer models: an introduction and catalog. arXiv preprint arXiv:2302.07730
arXiv 2023
-
[6]
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew Mccallum. 2017. Semeval 2017 task 10: Scienceie-extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 546--555
work page 2017
-
[7]
Mohamed-Amine Baazizi, Cl \'e ment Berti, Dario Colazzo, Giorgio Ghelli, and Carlo Sartiani. 2020. Human-in-the-loop schema inference for massive json datasets. In EDBT 2020-23nd International Conference on Extending Database Technology, pages 635--638. OpenProceedings. org
work page 2020
-
[8]
Mohamed-Amine Baazizi, Houssem Ben Lahmar, Dario Colazzo, Giorgio Ghelli, and Carlo Sartiani. 2017. Schema inference for massive json datasets. In Extending Database Technology (EDBT)
work page 2017
Show all 43 references
-
[9]
Hamed Babaei Giglou, Jennifer D’Souza, and S \"o ren Auer. 2023. Llms4ol: Large language models for ontology learning. In International Semantic Web Conference, pages 408--427. Springer
2023
-
[10]
Hamed Babaei Giglou, Jennifer D’Souza, and Sören Auer. 2024. https://doi.org/10.52825/ocp.v4i.2472 Preface for llms4ol 2024: The 1st large language models for ontology learning challenge at the 23rd iswc . Open Conference Proceedings, 4:1–2
2024 doi
-
[11]
Arthur Brack, Jennifer D’Souza, Anett Hoppe, S \"o ren Auer, and Ralph Ewerth. 2020. Domain-independent extraction of scientific concepts from research articles. In European Conference on Information Retrieval, pages 251--266. Springer
2020
-
[12]
Marc Brinner, Tina Heger, and Sina Zarriess. 2022. https://doi.org/10.18653/v1/2022.wiesp-1.5 Linking a hypothesis network from the domain of invasion biology to a corpus of scientific abstracts: The INAS dataset . In Proceedings of the first Workshop on Information Extraction...
2022 doi
-
[13]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...
2020
-
[14]
Phillip Cassey, Pablo Garc \' a-D \' az, Julie L Lockwood, and Tim M Blackburn. 2018. Invasion biology: searching for predictions and prevention, and avoiding lost causes. In Invasion biology: hypotheses and evidence, pages 3--13. CAB International Wallingford UK
2018
-
[15]
J Harry Caufield, Harshad Hegde, Vincent Emonet, Nomi L Harris, Marcin P Joachimiak, Nicolas Matentzoglu, HyeongSik Kim, Sierra Moxon, Justin T Reese, Melissa A Haendel, et al. 2024. Structured prompt interrogation and recursive extraction of semantics (spires): A method for p...
2024
-
[16]
Danilo Dess \` , Francesco Osborne, Diego Reforgiato Recupero, Davide Buscaldi, Enrico Motta, and Harald Sack. 2020. Ai-kg: an automatically generated knowledge graph of artificial intelligence. In The Semantic Web--ISWC 2020: 19th International Semantic Web Conference, Athens...
2020
-
[17]
Jennifer D ' Souza, S \"o ren Auer, and Ted Pedersen. 2021. https://doi.org/10.18653/v1/2021.semeval-1.44 S em E val-2021 task 11: NLPC ontribution G raph - structuring scholarly NLP contributions for a research knowledge graph . In Proceedings of the 15th International Worksh...
2021 doi
-
[18]
Jennifer D’Souza. 2023. https://doi.org/10.48366/R609337 A catalog of transformer models
2023 doi
-
[19]
Jennifer D’Souza. 2024. Agriculture named entity recognition—towards fair, reusable scholarly contributions in agriculture. Knowledge, 4(1):1--26
2024
-
[20]
Jennifer D’Souza and S \"o ren Auer. 2022. Computer science named entity recognition in the open research knowledge graph. In International Conference on Asian Digital Libraries, pages 35--45. Springer
2022
-
[21]
Jennifer D’Souza, Salomon Kabongo, Hamed Babaei Giglou, and Yue Zhang. 2024. Overview of the clef 2024 simpletext task 4: Sota? tracking the state-of-the-art in scholarly publications. In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), pages 3163--3173
2024
-
[22]
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extraction from the web. Communications of the ACM, 51(12):68--74
2008
-
[23]
Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, et al. 2011. Open information extraction: The second generation. In Twenty-Second International Joint Conference on Artificial Intelligence. Citeseer
2011
-
[24]
Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 1535--1545
2011
-
[25]
Kata G \'a bor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang QasemiZadeh, Ha \" fa Zargayouna, and Thierry Charnois. 2018. https://doi.org/10.18653/v1/S18-1111 S em E val-2018 task 7: Semantic relation extraction and classification in scientific papers . In Proceedings of t...
2018 doi
-
[26]
John Giorgi, Xindi Wang, Nicola Sahar, Won Young Shin, Gary D Bader, and Bo Wang. 2019. End-to-end named entity recognition and relation extraction using pre-trained language models. arXiv preprint arXiv:1912.13415
2019 arXiv
-
[27]
Jonathan M Jeschke and Tina Heger. 2018. Invasion biology: hypotheses and evidence. CAB International
2018
-
[28]
Salomon Kabongo, Jennifer D’Souza, and S \"o ren Auer. 2021. Automated mining of leaderboards for empirical ai research. In International Conference on Asian Digital Libraries, pages 453--470
2021
-
[29]
Petr Knoth, Drahomira Herrmannova, Matteo Cancellieri, Lucas Anastasiou, Nancy Pontika, Samuel Pearce, Bikash Gyawali, and David Pride. 2023. Core: a global aggregation service for open access papers. Scientific Data, 10(1):366
2023
-
[30]
Vamsi Krishna Kommineni, Birgitta K \"o nig-Ries, and Sheeba Samuel. 2024. From human experts to machines: An llm supported approach to ontology and knowledge graph construction. arXiv preprint arXiv:2403.08345
2024 arXiv
-
[31]
Gang Li, Karen E Ross, Cecilia N Arighi, Yifan Peng, Cathy H Wu, and K Vijay-Shanker. 2015. mirtex: a text mining system for mirna-gene relation extraction. PLoS computational biology, 11(9):e1004391
2015
-
[32]
Haoyang Liu, M Janina Sarol, and Halil Kilicoglu. 2021. Uiuc\_bionlp at semeval-2021 task 11: A cascade of neural models for structuring scholarly nlp contributions. arXiv preprint arXiv:2105.05435
2021 arXiv
-
[33]
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219--3232
2018
-
[34]
Jeschke, Maud Bernard-Verdier, Tina Heger, Camille Musseau, and Steph Tyszka
Daniel Mietchen, Jonathan M. Jeschke, Maud Bernard-Verdier, Tina Heger, Camille Musseau, and Steph Tyszka. 2024. https://doi.org/10.5281/zenodo.12518037 Invasion biology corpus 2024-07
2024 doi
-
[35]
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. 2018. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 19(6):1236--1246
2018
-
[36]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9
2019
-
[37]
Francesco Ronzano and Jay Nanavati. 2024. Towards ontology-enhanced representation learning for large language models. arXiv preprint arXiv:2405.20527
2024 arXiv
-
[38]
Helen E Roy, An \' bal Pauchard, Peter Stoett, Tanara Renard Truong, Sven Bacher, Bella S Galil, Philip E Hulme, Tohru Ikeda, Kavileveettil Sankaran, Melodie A McGeoch, et al. 2023. Ipbes invasive alien species assessment: summary for policymakers. IPBES
2023
-
[39]
Mahsa Shamsabadi, Jennifer D ' Souza, and S \"o ren Auer. 2024. https://aclanthology.org/2024.findings-eacl.26 Large language models for scientific information extraction: An empirical study for virology . In Findings of the Association for Computational Linguistics: EACL 2024...
2024
-
[40]
Karthik Soman, Peter W Rose, John H Morris, Rabia E Akbas, Brett Smith, Braian Peetoom, Catalina Villouta-Reyes, Gabriel Cerono, Yongmei Shi, Angela Rizk-Jackson, et al. 2024. Biomedical knowledge graph-optimized prompt generation for large language models. Bioinformatics, 40(...
2024
-
[41]
Rudi Studer, V Richard Benjamins, and Dieter Fensel. 1998. Knowledge engineering: Principles and methods. Data & knowledge engineering, 25(1-2):161--197
1998
-
[42]
Zhong-Yi Wang and Hong-Yu Zhang. 2013. Rational drug repositioning by medical genetics. Nature biotechnology, 31(12):1080--1082
2013
-
[43]
XueZhong Zhou, J \"o rg Menche, Albert-L \'a szl \'o Barab \'a si, and Amitabh Sharma. 2014. Human symptoms--disease network. Nature communications, 5(1):4212
2014
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.