Pith. sign in

REVIEW 3 major objections 4 minor 78 references

The paper claims that an ensemble of three open-source LLMs, decided by majority vote, extracts disease, country, date, case, and death fields from WHO outbreak bulletins at least as accurately as commercial models, making the released eKG

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The authors release a structured dataset and knowledge graph extracted from WHO outbreak reports by an LLM ensemble, with benchmark F1 scores between 0.66 and 0.96 for four of five extracted fields.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful public resource with a real validation gap: the numeric fields—case counts and deaths—are never actually checked against gold-standard numbers. the 3 major comments →

arxiv 2509.02258 v1 pith:LVBC766G submitted 2025-09-02 cs.AI

An Epidemiological Knowledge Graph extracted from the World Health Organization's Disease Outbreak News

Sergio Consoli , Pietro Coletti , Peter V. Markov , Lia Orfei , Indaco Biazzo , Lea Schuh , Nicolas Stefanovitch , Lorenzo Bertolini
show 2 more authors
Mario Ceresa Nikolaos I. Stilianakis
This is my paper
classification cs.AI
keywords epidemiological surveillancelarge language modelsknowledge graphinformation extractiondisease outbreak newsensemble learninglinked open dataevent-based surveillance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that an ensemble of three open-source large language models, combined by majority voting, can turn the WHO's prose Disease Outbreak News bulletins into a structured, queryable record of outbreaks with accuracy at least matching proprietary models. To establish this, the authors benchmark disease-name, country, date, and case-count extraction against expert-annotated reports and report that the ensemble reaches the top F1 score on all four tasks. On that basis, they release eKG, a daily-updated knowledge graph of roughly 2,900 outbreak events and about 26,000 triples, mapped to standard biomedical and geographic ontologies and accessible through a public SPARQL endpoint. A reader should care because the resource offers timely epidemic signals without the cost, rate limits, or confidentiality constraints of closed commercial APIs, and because the full pipeline is open-source and reusable.

Core claim

The core discovery is that committee agreement among three open-source LLMs—Mistral-7B-OpenOrca, Meta-Llama-3-70B-Instruct, and Zephyr-7B-Beta—beats every individual model tested, including large proprietary ones, on this extraction task. On the 171-report benchmark, the ensemble scores F1 values of 0.851 for disease names, 0.962 for countries, 0.658 for confirmed-case counts, and 0.869 for event dates, the highest in all four comparisons. The authors take this as evidence that the fully automatic eKG pipeline is technically sound: reconstructed time series for MERS-CoV in Saudi Arabia correlate significantly with WHO-confirmed yearly counts, with the fit improving after excluding the 2013 o

What carries the argument

The load-bearing mechanism is the Ensemble: three open-source LLMs are each prompted with a fixed extraction template, and their JSON outputs are reconciled by majority voting. For textual fields, votes are grouped using synonym dictionaries built from syntactic matching, WordNet synonymy, and semantic similarity via Sentence-BERT for country names and BioBERT for disease names, with a cosine threshold of 0.8. The majority result is then packaged as RDF triples in a knowledge graph that reuses the Infectious Disease Ontology and GeoNames classes, with an imputed date field drawn from the report title when no event date is found. The ensemble is what carries the accuracy claim: it outperforms

Load-bearing premise

The dataset's headline death counts are never checked against expert annotations; the paper assumes deaths are extracted as reliably as case counts, because deaths were not annotated in the benchmark.

What would settle it

Take a random sample of DON reports that report deaths, run the ensemble to extract the deaths field, and compare each value against the figure stated in the report and, wherever possible, against WHO's own outbreak updates; a material drop in deaths-F1 relative to cases-F1 would refute the comparability assumption and would call for a mortality-specific prompt or validator.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • eKG can be queried with SPARQL, letting researchers test hypotheses about outbreak timing, geography, and case loads without reading thousands of prose reports.
  • The pipeline runs daily and is built only from open-source models, so it can be redeployed on the full archive of over 3,000 DON reports without API quotas or per-token costs.
  • A qualitative check reproduces known epidemic curves—MERS-CoV in Saudi Arabia, Ebola in the Democratic Republic of the Congo and Guinea, and SARS in China—supporting the dataset's use for retrospective epidemiological analysis.
  • The imputed date field, taken from the report title, is flagged as potentially later than the true event date, meaning time-series analyses should prefer the extracted date when present.
  • Because the knowledge graph maps to standard ontologies, eKG can be linked to other linked-open-data resources to combine outbreak data with geographic, demographic, or clinical datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The unvalidated deaths field is worth an explicit audit before eKG feeds any mortality-focused surveillance: the paper's similarity argument is plausible, but death counts are often reported as cumulative totals or with caveats, so majority voting over raw numbers could silently under- or over-count.
  • A stratified evaluation by disease-report frequency would test where the ensemble degrades: the 171-report benchmark may not reflect the long tail of rare pathogens that makes up much of the full DONs archive.
  • If extraction errors are largely idiosyncratic to each model rather than shared, adding a fourth diverse open-source model might further raise F1; the same architecture directly supports that experiment.
  • The synonym-clustering step for disease names could be turned into a benchmark of its own: comparing the ensemble's labels against ICD-10 codes would measure whether the ontology mapping layer adds value beyond string matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript describes the construction and release of eKG, a knowledge graph and daily-updated dataset built by automatically extracting epidemiological fields (disease name, country, event date, case count, death count) from WHO Disease Outbreak News reports. Extraction is performed by an ensemble of three open-source LLMs (Mistral-7B-OpenOrca, Meta-Llama-3-70B-Instruct, Zephyr-7B-Beta) with majority voting and synonym/semantic-similarity normalization. The paper reports benchmarking against 171 expert-annotated samples from the Incident Database, compares the ensemble with commercial and open models, and presents a yearly MERS-CoV regression as a quantitative check. The paper also documents the FAIR publication of the data, SPARQL endpoint, and several visualization services.

Significance. The resource is potentially valuable: it converts a large, underutilized narrative corpus into structured, queryable data, is publicly released under CC BY 4.0, and is accompanied by code and interactive endpoints. The open-source LLM ensemble approach is also practically relevant because it avoids commercial API constraints. However, the validation in its current form does not substantiate the central reliability claim. The reported F1 scores measure only the presence/absence of a field, not whether the extracted numeric values are correct; case counts are therefore not truly validated, and death counts are not validated at all. If the authors can provide per-record numeric validation for case and death counts, the contribution would be substantially stronger. As it stands, the significance is contingent on additional validation.

major comments (3)
  1. [Technical Validation, Tables 2-4] The evaluation is explicitly a binary presence/absence test: 'Each IE task was assessed as a binary classification problem.' This means a true positive is recorded whenever the model outputs any value for a report whose IDB record contains that field, not when the extracted number matches the gold standard. For cases, Table 4 reports ensemble F1=0.658 and recall=0.591, so even at the presence level the pipeline misses roughly 41% of reports containing a case count. The F1 scores therefore say nothing about whether the numeric contents of cases_extracted are accurate. The later MERS-CoV comparison in Figure 5 aggregates yearly totals for one disease/country pair and does not test per-record numeric correctness. The claim that the ensemble results 'support the technical quality of the extracted eKG dataset' is not supported for the numeric fields.
  2. [Technical Validation and Methods (prompt)] The paper states that death extraction could not be evaluated because the IDB does not annotate deaths, and then assumes that its performance is 'likely to be comparable' to case extraction. This is an unsupported premise, and it is load-bearing because deaths_extracted is a headline field in eKG. The assumption is also fragile: mortality in DON reports is often reported as ranges, cumulative totals, or confirmed vs. suspected counts, so it is not methodologically similar to extracting a single case count. In addition, the extraction prompt shown in the Methods section asks for the number of deaths but specifies JSON keys only for 'disease name, country, date, cases'; death appears only in the illustrative output. This under-specification compounds the lack of validation. The manuscript should either annotate and evaluate death extraction or explicitly mark deaths_extracted as unvalidate
  3. [Technical Validation, Figure 5] The only quantitative validation of numeric values is the yearly regression of reconstructed MERS-CoV cases in Saudi Arabia against WHO totals. This is a single disease/country pair, and the comparison excludes 2013 because the authors suspect the WHO DONs included potential cases in that year. The exclusion is post hoc and is not described as a pre-specified criterion. Moreover, yearly aggregation cannot detect errors in individual extracted counts, such as systematic over- or under-reporting that cancels out at the annual level. This evidence is too narrow to establish the general numeric reliability of cases_extracted across the 2384-event dataset, and it provides no direct evidence for deaths_extracted.
minor comments (4)
  1. [Methods, KGs Services & Interfaces] The text reports '2.3 generalized axioms'; this likely should be '2.3k' or '2,300'. Please correct the typo and verify the number.
  2. [Methods, LLMs for Epidemic IE] The ensemble description says the models share an 8K context length, while individual model descriptions state a 4,096-token context. Clarify which context length is actually used.
  3. [Methods, prompt] The JSON keys listed in the prompt omit 'deaths' even though the prompt asks for it. If the final pipeline includes deaths_extracted, the prompt should reflect that; otherwise the provenance of the field is unclear.
  4. [Technical Validation, Tables 2-5] No confidence intervals or significance tests are reported for the F1 differences, despite the small 171-sample benchmark. Many of the differences among top models are small (e.g., disease F1 0.851 vs. 0.842), so a statement about statistical significance would be useful.

Circularity Check

0 steps flagged

No significant circularity; independent validation against external IDB benchmark; minor non-load-bearing self-citation.

full rationale

The paper does not exhibit a circular derivation. The central pipeline (ensemble of Mistral-7B-OpenOrca, Meta-Llama-3-70B-Instruct, Zephyr-7B-Beta with majority voting) is motivated by the authors' prior conference paper (ref 11), which is a self-citation; however, the current manuscript independently benchmarks this ensemble against the IDB gold standard (171 expert-annotated WHO DON-derived reports) in Tables 2-5 and against external WHO yearly case counts for MERS-CoV in Figure 5. The F1 scores are computed as binary presence/absence against these external annotations; no extracted value is obtained from a fitted parameter or from the evaluation metric itself. The death-extraction field is explicitly not evaluated ('It was not possible to evaluate the task of extraction of the number of deaths'), and the 'likely to be comparable' statement is an untested extrapolation, not a circular step. The regression line in Figure 5 is a post hoc comparison, not an input to extraction. The only mild issue is reliance on the authors' prior result as motivation, but because the validation is reproduced here on external data, that self-citation is not load-bearing. Hence a score of 2 at most.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claim rests on the quality of the LLM extractions, benchmarked on a small expert-annotated set, and on the unvalidated assumption that deaths are extracted as reliably as cases.

free parameters (2)
  • Semantic similarity threshold = 0.8
    Cosine similarity threshold used to group disease/country synonyms in the majority voting step; chosen experimentally with SBERT/BioBERT.
  • 2013 exclusion in MERS comparison = 2013 removed
    The regression in Figure 5b drops 2013, the epidemic onset year, to improve the fit; this is a post hoc data selection.
axioms (4)
  • domain assumption IDB annotations are an accurate gold standard
    The 171-sample subset of the Incident Database is treated as ground truth for all four validated fields; errors there propagate into F1 scores.
  • domain assumption WHO DON reports are reliable narratives of outbreaks
    The pipeline trusts the content of DONs as the source of disease, case, and death counts.
  • ad hoc to paper Deaths extraction resembles case extraction
    The paper asserts 'given the similarity of this task to the extraction of the number of cases, its performance is likely to be comparable' without any evaluation.
  • domain assumption LLMs do not invent information
    The prompt instructs models to use 'None' rather than invent, but no manual audit of a sample is reported.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of An Epidemiological Knowledge Graph extracted from the World Health Organization's Disease Outbreak News." pith.science (2026). https://pith.science/paper/LVBC766G

@misc{pith2026250902258,
  author       = {Pith},
  title        = {Pith review of: An Epidemiological Knowledge Graph extracted from the World Health Organization's Disease Outbreak News},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LVBC766G}},
  note         = {Machine review of arXiv:2509.02258}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rapid evolution of artificial intelligence (AI), together with the increased availability of social media and news for epidemiological surveillance, are marking a pivotal moment in epidemiology and public health research. Leveraging the power of generative AI, we use an ensemble approach which incorporates multiple Large Language Models (LLMs) to extract valuable actionable epidemiological information from the World Health Organization (WHO) Disease Outbreak News (DONs). DONs is a collection of regular reports on global outbreaks curated by the WHO and the adopted decision-making processes to respond to them. The extracted information is made available in a daily-updated dataset and a knowledge graph, referred to as eKG, derived to provide a nuanced representation of the public health domain knowledge. We provide an overview of this new dataset and describe the structure of eKG, along with the services and tools used to access and utilize the data that we are building on top. These innovative data resources open altogether new opportunities for epidemiological research, and the analysis and surveillance of disease outbreaks.

Figures

Figures reproduced from arXiv: 2509.02258 by Indaco Biazzo, Lea Schuh, Lia Orfei, Lorenzo Bertolini, Mario Ceresa, Nicolas Stefanovitch, Nikolaos I. Stilianakis, Peter V. Markov, Pietro Coletti, Sergio Consoli.

Figure 1
Figure 1. Figure 1: provides a schematic overview of the pipeline, which will be described in more detail in the “Methods” section. A daily process is triggered to extract, transform and load (ETL) new reports from WHO DONs, which are then processed by the ensemble of LLMs for the epidemiological information extraction (IE) task. The extracted information are then published using FAIR principles along with the derived eKG. Th… view at source ↗
Figure 2
Figure 2. Figure 2: Snippet of eKG exploration services and interfaces. 6/22 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Main classes of the epidemiological knowledge graph. In our data engineering work to model the knowledge graph of epidemiological information, we adhered to standard style guidelines for identifying and describing linked data resources62. Specifically, within eKG, data names were expressed in lowercase and any potential space characters replaced with dashes, as per the established norms for naming and labe… view at source ↗
Figure 4
Figure 4. Figure 4: Number of extracted cases vs. time for 4 case studies: MERS-Cov in Saudi Arabia (Panel a), Ebola cases in the Democratic Republic of Congo (Panel b), SARS cases in China (Panel c), and Ebola cases in Guinea (Panel d). These four case studies correspond to the Disease Outbreak Names - Country pairs reported in bold in [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: WHO yearly number of cases vs the number of cases from the reconstructed events, together with regression line: including year 2013 (Panel a) and without year 2013 (Panel b). Since 2013 was the epidemic’s onset, the inclusion of potential and confirmed cases by WHO DONS could lead to a lower correlation coefficient due to overestimated reconstructed cases. Correlation and p-values are reported in the top r… view at source ↗
Figure 6
Figure 6. Figure 6: Faceted Browser showing the extracted information for Nipah virus - India. 15/22 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Content negotiation of the Nipah virus - India report by means of LodView [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Graph visualization of Nipah virus - India in LodLive. 16/22 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: An extract from the eKG representation in LODE. 17/22 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: A snapshot of the WebVOWL visualization of eKG. FILTER (?label = “Nipah Virus”)} The same but with a more flexible regular expression in the SPARQL query: SELECT ?event FROM <eKG> WHERE {?event eKG:virus_extracted ?label . FILTER regex(str(?label), “nipah”, “i”)} The following SPARQL query finds all the disease outbreaks that occurred in “Italy”: SELECT ?event FROM <eKG> WHERE {?event eKG:country_extracte… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 65 canonical work pages

  1. [1]

    Salathé, M. et al. Digital epidemiology. PLoS Comput. Biol. 8, e1002616 (2012)

    work page 2012

  2. [2]

    S., Rader, B., Astley, C

    Brownstein, J. S., Rader, B., Astley, C. M. & Tian, H. Advances in Artificial Intelligence for Infectious-Disease Surveillance. New Engl. J. Medicine 388, 1597 – 1607, 10.1056/NEJMra2119215 (2023)

    work page doi:10.1056/nejmra2119215 2023

  3. [3]

    M., Chen, Y ., Pennock, D

    Polgreen, P. M., Chen, Y ., Pennock, D. M., Nelson, F. D. & Weinstein, R. A. Using internet searches for influenza surveillance. Clin. infectious diseases 47, 1443–1448 (2008)

    work page 2008

  4. [4]

    & Checchi, F

    Warsame, A., Murray, J., Gimma, A. & Checchi, F. The practice of evaluating epidemic response in humanitarian and low-income settings: a systematic review. BMC Medicine 18 (2020)

    work page 2020

  5. [5]

    Disease Outbreak News (2024)

    World Health Organization. Disease Outbreak News (2024). Last accessed: 2024-10-01, https://www.who.int/emergencies/ disease-outbreak-news/

    work page 2024

  6. [6]

    Oppenheim, B. et al. Assessing global preparedness for the next pandemic: Development and application of an Epidemic Preparedness Index. BMJ Glob. Heal. 4 (2019)

    work page 2019

  7. [7]

    Mondor, L. et al. Timeliness of nongovernmental versus governmental global outbreak communications. Emerg. Infect. Dis. 18, 1184 – 1187 (2012)

    work page 2012

  8. [8]

    & Brett-Major, D

    Lugo-Robles, R., Garges, E., Olsen, C. & Brett-Major, D. Identifying nontraditional epidemic disease risk factors associated with major health events from world health organization and world bank open data. Am. J. Trop. Medicine Hyg. 105, 896–902 (2021)

    work page 2021

  9. [9]

    Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems, 5999–6009 (2017)

    work page 2017

  10. [10]

    Brown, T. B. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, vol. 2020-December (2020)

    work page 2020

  11. [11]

    Consoli, S. et al. Epidemic Information Extraction for Event-Based Surveillance Using Large Language Models. In Proceedings of Ninth International Congress on Information and Communication Technology (ICICT 2024), vol. 1011 LNNS, 241 – 252 (Lecture Notes in Networks and Systems, 2024)

    work page 2024

  12. [12]

    & Bizer, C

    Heath, T. & Bizer, C. Linked Data: Evolving the Web into a Global Data Space. Synth. Lect. on Semantic Web: Theory Technol. 1, 1–121 (2011)

    work page 2011

  13. [13]

    Auer, S. et al. Towards a Knowledge Graph for Science. In Proceedings of the 8th International Conference on Web Intelligence, Mining and Semantics, WIMS ’18 (Association for Computing Machinery, New York, NY , USA, 2018). 19/22

    work page 2018

  14. [14]

    & Osborne, F

    Peng, C., Xia, F., Naseriparsa, M. & Osborne, F. Knowledge Graphs: Opportunities and Challenges. Artif. Intell. Rev. 56, 10.1007/s10462-023-10465-9 (2023)

    doi:10.1007/s10462-023-10465-9 2023

  15. [15]

    Hogan, A. et al. Knowledge graphs. ACM Comput. Surv. 54, 10.1145/3447772 (2021)

    doi:10.1145/3447772 2021

  16. [16]

    Resource Description Framework (RDF) (2024)

    World Wide Web Consortium. Resource Description Framework (RDF) (2024). Accessed: 2024-10-01, https://www.w3. org/RDF/

    work page 2024

  17. [17]

    Ji, S., Pan, S., Cambria, E., Marttinen, P. & Yu, P. S. A Survey on Knowledge Graphs: Representation, Acquisition, and Applications. IEEE Transactions on Neural Networks Learn. Syst. 33, 10.1109/TNNLS.2021.3070843 (2022)

    arXiv 2021

  18. [18]

    B., Usip, P

    Tiwari, S., Ortíz-Rodriguez, F., Abbés, S. B., Usip, P. U. & Hantach, R. Semantic AI in Knowledge Graphs (Taylor & Francis, Boca Raton, US, 2023)

    work page 2023

  19. [19]

    McDonald, D. J. et al. Can auxiliary indicators improve COVID-19 forecasting and hotspot prediction? Proc. Natl. Acad. Sci. United States Am. 118, 10.1073/pnas.2111453118 (2021)

    work page doi:10.1073/pnas.2111453118 2021

  20. [20]

    Pepe, E. et al. COVID-19 outbreak response, a dataset to assess mobility changes in Italy following national lockdown. Sci. data 7, 1–7 (2020)

    work page 2020

  21. [21]

    M., Tedijanto, C., Goldstein, E., Grad, Y

    Kissler, S. M., Tedijanto, C., Goldstein, E., Grad, Y . H. & Lipsitch, M. Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period. Science (2020)

    work page 2020

  22. [22]

    Sutskever, I., Vinyals, O. & Le, Q. V . Sequence to sequence learning with neural networks. In Advances in neural information processing systems, 3104–3112 (2014)

    work page 2014

  23. [23]

    MistralOrca: Mistral-7B Model Instruct-tuned on Filtered OpenOrcaV1 GPT-4 Dataset (2023)

    Lian, W.et al. MistralOrca: Mistral-7B Model Instruct-tuned on Filtered OpenOrcaV1 GPT-4 Dataset (2023). HuggingFace repository, https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca

    work page 2023

  24. [24]

    Lian, W. et al. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces (2023). HuggingFace repository, https://https://huggingface.co/Open-Orca/OpenOrca

    work page 2023

  25. [25]

    Mukherjee, S. et al. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (2023). arXiv:2306.02707

    Pith/arXiv arXiv 2023

  26. [26]

    Beltagy, I., Peters, M. E. & Cohan, A. Longformer: The Long-document Transformer (2020). arXiv:2004.05150

    Pith/arXiv arXiv 2020

  27. [27]

    Jiang, A. Q. et al. Mistral 7B (2023). arXiv:2310.06825

    Pith/arXiv arXiv 2023

  28. [28]

    Tunstall, L. et al. Zephyr: Direct Distillation of LM Alignment (2023). arXiv:2310.16944

    Pith/arXiv arXiv 2023

  29. [29]

    Dubey, A. et al. The Llama 3 Herd of Models (2024). arXiv:2407.21783

    Pith/arXiv arXiv 2024

  30. [30]

    Touvron, H. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023). arXiv:2307.09288

    Pith/arXiv arXiv 2023

  31. [31]

    Singh, A. et al. Llama 3 Model Card (2024). GitHub repository, https://github.com/meta-llama/llama3/blob/main/MODEL_ CARD.md

    work page 2024

  32. [32]

    & Rokach, L

    Sagi, O. & Rokach, L. Ensemble learning: A survey. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 8, 10.1002/widm. 1249 (2018)

    doi:10.1002/widm 2018

  33. [33]

    Ensemble methods: Foundations and algorithms (Chapman & Hall/CRC, 2012)

    Zhou, Z.-H. Ensemble methods: Foundations and algorithms (Chapman & Hall/CRC, 2012)

    work page 2012

  34. [34]

    Miller, G. A. WordNet: A lexical database for English. Commun. ACM 38, 39–41 (1995)

    work page 1995

  35. [35]

    & Gurevych, I

    Reimers, N. & Gurevych, I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, 2019). https://arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  36. [36]

    & Toutanova, K

    Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In ACL-HLT 2019 Conference Proceedings, vol. 1, 4171–4186 (2019)

    work page 2019

  37. [37]

    & Consoli, S

    Korst, J., Pronk, V ., Barbieri, M. & Consoli, S. Introduction to classification algorithms and their performance analysis using medical examples. In Consoli, S., Recupero, D. R. & Petkovic, M. (eds.)Data Science for Healthcare: Methodologies and Applications, 39–73 (Springer Nature, 2019)

    work page 2019

  38. [38]

    Sentence Transformers all-mpnet-base-v2 model (2024)

    Hugging Face. Sentence Transformers all-mpnet-base-v2 model (2024). Last Accessed: 2024-10-01, https://huggingface. co/sentence-transformers/all-mpnet-base-v2

    work page 2024

  39. [39]

    Lee, J. et al. Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234 – 1240 (2020)

    work page 2020

  40. [40]

    Deka, P., Jurek-Loughrey, A. & P, D. Evidence extraction to validate medical claims in fake news detection. In Traina, A. et al. (eds.) Health Information Science, 3–15 (Springer Nature Switzerland, Cham, 2022). 20/22

    work page 2022

  41. [41]

    & van Harmelen, F

    Antoniou, G. & van Harmelen, F. Web Ontology Language: OWL, 67–92 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2004)

    work page 2004

  42. [42]

    Ontology design patterns for semantic web content

    Gangemi, A. Ontology design patterns for semantic web content. Lect. Notes Comput. Sci. 3729 LNCS, 262–276 (2005)

    work page 2005

  43. [43]

    & Hitzler, P

    Shimizu, C., Hirt, Q. & Hitzler, P. A protégé plug-in for annotating OWL ontologies with OPLa. Lect. Notes Comput. Sci. 11155 LNCS, 23–27 (2018)

    work page 2018

  44. [44]

    & Presutti, V

    Asprino, L., Carriero, V ., Colonna, C. & Presutti, V . OPLaX: Annotating ontology design patterns at conceptual and instance level. In CEUR Workshop Proceedings, vol. 3011 (2021)

    work page 2021

  45. [45]

    Cowell, L. G. & Smith, B. Infectious Disease Ontology (Springer Berlin Heidelberg, 2010)

    work page 2010

  46. [46]

    GeoNames geographical database (2024)

    GeoNames. GeoNames geographical database (2024). Last Accessed: 2024-10-01, https://www.geonames.org/

    work page 2024

  47. [47]

    & Ruttenberg, A

    Neil Otte, J., Beverley, J. & Ruttenberg, A. BFO: Basic Formal Ontology.Appl. Ontology 17, 17 – 43, 10.3233/AO-220262 (2022)

    work page doi:10.3233/ao-220262 2022

  48. [48]

    Jackson, R. et al. OBO Foundry in 2021: operationalizing open data principles to evaluate ontologies. Database 2021, 10.1093/database/baab069 (2021)

    work page doi:10.1093/database/baab069 2021

  49. [49]

    International Classification of Diseases, Version 10 (ICD10) (2024)

    World Health Organization. International Classification of Diseases, Version 10 (ICD10) (2024). Last Accessed: 2024-10- 01, https://icd.who.int/browse10/2019/en

    work page 2024

  50. [50]

    & Schmid, B

    Krauer, F. & Schmid, B. V . Mapping the plague through natural language processing.Epidemics 41 (2022)

    work page 2022

  51. [51]

    Soille, P. et al. A versatile data-intensive computing platform for information retrieval from big geospatial data. Futur. Gener. Comput. Syst. 81, 30 – 40, 10.1016/j.future.2017.11.007 (2018)

    doi:10.1016/j.future.2017.11.007 2017

  52. [52]

    [dataset] European Commission, Joint Research Centre (JRC). Epidemic Information Extraction from WHO Disease Outbreak News, European Data Portal, Joint Research Centre Data Catalogue, https: //doi.org/10.2905/89056048-7f5d-4d7c-96ad-f99d1c0f6601 (2024). PID: http://data.jrc.ec.europa.eu/dataset/ 89056048-7f5d-4d7c-96ad-f99d1c0f6601

    work page doi:10.2905/89056048-7f5d-4d7c-96ad-f99d1c0f6601 2024

  53. [53]

    RDF Turtle: Terse RDF Triple Language (2024)

    World Wide Web Consortium. RDF Turtle: Terse RDF Triple Language (2024). Accessed: 2024-10-01, https://www.w3. org/TR/turtle/

    work page 2024

  54. [54]

    Joint Research Centre Data Catalogue (2024)

    European Commission, Joint Research Centre (JRC). Joint Research Centre Data Catalogue (2024). Last Accessed: 2024-10-01, https://data.jrc.ec.europa.eu/

    work page 2024

  55. [55]

    Kirstein, F. et al. Linked Data in the European Data Portal: A Comprehensive Platform for Applying DCAT-AP. Lect. Notes Comput. Sci. (including subseries Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) 11685 LNCS, 192 – 204, 10.1007/978-3-030-27325-5_15 (2019)

    work page doi:10.1007/978-3-030-27325-5_15 2019

  56. [56]

    & Gutierrez, C

    Pérez, J., Arenas, M. & Gutierrez, C. Semantics and complexity of SPARQL. ACM Transactions on Database Syst. 34 (2009)

    work page 2009

  57. [57]

    Consoli, S. et al. eKG Virtuoso SPARQL endpoint (2024). Password protected access, with credentials available upon request to authors. For the purpose of the review process, please use the following provisional login details: Username: user, Password: user2024, https://api-vast.jrc.service.ec.europa.eu/sparql/

    work page 2024

  58. [58]

    & Nikolaev, K

    Kirillovich, A. & Nikolaev, K. Adapting the LodView RDF Browser for Navigation over the Multilingual Linguistic Linked Open Data Cloud. In 2022 IEEE 9th International Conference on Sciences of Electronics, Technologies of Information and Telecommunications, SETIT 2022, 143 – 149, 10.1109/SETIT54465.2022.9875628 (2022)

    work page arXiv 2022

  59. [59]

    & Schafer, V

    Griset, P. & Schafer, V . Hosting the world wide web consortium for Europe: From CERN to INRIA.Hist. Technol. 27, 353 – 370, 10.1080/07341512.2011.604177 (2011)

    work page arXiv 2011

  60. [60]

    V ., Mazzini, S

    Camarda, D. V ., Mazzini, S. & Antonuccio, A. LodLive, exploring the Web of Data. InACM International Conference Proceeding Series, 197 – 200, 10.1145/2362499.2362532 (2012)

    work page arXiv 2012

  61. [61]

    Last accessed: 2024-10-01, https://creativecommons.org/licenses/ by/4.0/

    Creative Commons Attribution 4.0 International license. Last accessed: 2024-10-01, https://creativecommons.org/licenses/ by/4.0/

    work page 2024

  62. [62]

    & Presutti, V

    Gangemi, A. & Presutti, V . Ontology design patterns. InHandbook on Ontologies, 221–243 (International Handbooks on Information Systems, 2009)

    work page 2009

  63. [63]

    Weibel, S. L. & Koch, T. The Dublin core metadata initiative: Mission, current activities, and future directions. D-Lib Mag. 6, 10.1045/december2000-weibel (2000). 21/22

    work page doi:10.1045/december2000-weibel 2000

  64. [64]

    Middle East respiratory syndrome coronavirus-Kingdom of Saudi Arabia (2024)

    World Health Organization Disease Outbreak News. Middle East respiratory syndrome coronavirus-Kingdom of Saudi Arabia (2024). 2024-05-08, https://www.who.int/emergencies/disease-outbreak-news/item/2024-DON516

    work page 2024

  65. [65]

    Ebola outbreak 2018-2020- North Kivu-Ituri (2020)

    World Health Organization Disease Outbreak News. Ebola outbreak 2018-2020- North Kivu-Ituri (2020). 2020-07-03, https://www.who.int/emergencies/situations/Ebola-2019-drc-

    work page 2018

  66. [66]

    Ebola outbreak 2014-2016 - West Africa (2016)

    World Health Organization Disease Outbreak News. Ebola outbreak 2014-2016 - West Africa (2016). https://www.who. int/emergencies/situations/ebola-outbreak-2014-2016-West-Africa

    work page 2014

  67. [67]

    Wikipedia, Page Version ID: 1247640845, https://en.wikipedia.org/w/index.php?title= 2002%E2%80%932004_SARS_outbreak

    2002–2004 SARS outbreak (2024). Wikipedia, Page Version ID: 1247640845, https://en.wikipedia.org/w/index.php?title= 2002%E2%80%932004_SARS_outbreak

    work page 2002

  68. [68]

    & Ghozzi, S

    Abbood, A., Ullrich, A., Busche, R. & Ghozzi, S. EventEpi-A natural language processing framework for event-based surveillance. PLoS Comput. Biol. 16, 10.1371/journal.pcbi.1008277 (2020)

    work page doi:10.1371/journal.pcbi.1008277 2020

  69. [69]

    & Petkovic, M

    Consoli, S., Reforgiato Recupero, D. & Petkovic, M. (eds.) Data Science for Healthcare - Methodologies and Applications (Springer Nature, 2019)

    work page 2019

  70. [70]

    Biderman, S. et al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In Proceedings of Machine Learning Research, vol. 202, 2397 – 2430 (2023)

    work page 2023

  71. [71]

    Introducing MPT-30B: Raising the bar for open-source foundation models (2023)

    MosaicML NLP Team. Introducing MPT-30B: Raising the bar for open-source foundation models (2023). Last Accessed: 2024-10-01, www.mosaicml.com/blog/mpt-30b

    work page 2023

  72. [72]

    GPT Models Family (2024)

    OpenAI. GPT Models Family (2024). Last Accessed: 2024-10-01, https://platform.openai.com/docs/models

    work page 2024

  73. [73]

    Dong, Q. et al. A survey on in-context learning (2023). arXiv:2301.00234

    Pith/arXiv arXiv 2023

  74. [74]

    Paolotti, D. et al. Web-based participatory surveillance of infectious diseases: the Influenzanet participatory surveillance experience. Clin. Microbiol. Infect. 20, 17–21 (2014)

    work page 2014

  75. [75]

    Lewnard, J. A. & Reingold, A. L. Emerging challenges and opportunities in infectious disease epidemiology. Am. J. Epidemiol. 188, 873 – 882, 10.1093/aje/kwy264 (2019)

    work page doi:10.1093/aje/kwy264 2019

  76. [76]

    The Joint Research Centre (JRC) of the European Commission (2024)

    European Commission. The Joint Research Centre (JRC) of the European Commission (2024). Last Accessed: 2024-10-01, https://ec.europa.eu/info/departments/joint-research-centre_en

    work page 2024

  77. [77]

    EIOS: Epidemic Intelligence from Open Sources

    World Health Organization. EIOS: Epidemic Intelligence from Open Sources. https://www.who.int/initiatives/eios (2024). Last Accessed: 2024-10-01

    work page 2024

  78. [78]

    & Lausen, G

    Schmidt, M., Meier, M. & Lausen, G. Foundations of SPARQL query optimization. In ACM International Conference Proceeding Series, 4 – 33 (2010). Acknowledgements We would like to thank the colleagues of the Digital Health Unit (JRC.F7) at the Joint Research Centre of the European Commission for helpful guidance and support. The views expressed are purely t...

    work page 2010

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.