{"id":"02984def-573c-47af-bfbb-26582dbe76f1","arxiv_id":"2509.02258","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors release a structured dataset and knowledge graph extracted from WHO outbreak reports by an LLM ensemble, with benchmark F1 scores between 0.66 and 0.96 for four of five extracted fields.","lead":"This paper describes eKG, a daily-updated knowledge graph built from WHO Disease Outbreak News reports using an ensemble of three open-source LLMs that extract disease, country, date, cases, and deaths. It is a practical resource for tracking outbreaks, though validation covers only four of the five extracted fields.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation measures only presence/absence of cases; exact case counts and all death counts are untested, so the reliability claim for eKG's numeric fields is unsupported.","rationale":"The reader's weakest_assumption focuses on the unvalidated death field and the unsupported similarity assumption to cases. My read agrees that deaths are a critical gap, but the deeper issue is broader: the validation protocol does not actually test the numeric correctness of either cases or deaths. The binary-presence framing means the F1 scores validate only whether a value was extracted, not what value was extracted. This is a load-bearing flaw because eKG's headline use case—surveillance and epidemiological modeling—requires accurate counts. The paper does provide some independent support: the data and code are publicly released, and the MERS-Cov regression gives a limited quantitative check, though it covers only one disease/country and excludes 2013 post hoc. These do not compensate for the absence of exact-value evaluation. The concern is not internal inconsistency or fraud; it is an unmet validation requirement. Since the reader already assigned CONDITIONAL, this concern does not change the verdict, but it sharpens the condition: the authors should provide exact-match evaluation for case counts and at least a sampled gold-standard evaluation for deaths before the reliability claim can be accepted.","tokens_in":19893,"tokens_out":4357,"duration_ms":56166,"concrete_test":"Using the same 171-report IDB subset, recompute precision/recall/F1 for the cases field with exact integer matching (after normalizing strings, e.g., '1,500' vs '1500') instead of presence/absence. For deaths, manually annotate 50–100 randomly sampled DON reports from the IDB subset (or use an independent gold standard such as WHO situation reports) and compute exact-match F1 for deaths. If cases exact-match F1 falls materially below the reported 0.658 binary-presence F1, or death F1 is not comparable to case F1, the paper's 'likely comparable' assumption and the reliability claim for eKG's numeric fields fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that eKG is a reliable surveillance resource depends on the extracted numeric fields—case counts and deaths—being accurate. However, the Technical Validation section evaluates each extraction task as a binary classification: a true positive is recorded when the model outputs any value for a report whose IDB record contains that field, not when the extracted number matches the gold-standard number. The reported F1 scores in Tables 2–5 therefore say nothing about whether the actual numeric contents of cases_extracted and deaths_extracted are correct. In fact, Table 4 shows the ensemble's binary-presence F1 for cases is only 0.658, with recall 0.591, meaning it misses case counts in about 41% of reports even at the presence/absence level. Deaths are never evaluated at all; the paper's stated assumption that death extraction is 'likely to be comparable' to case extraction is unsupported, especially because DON mortality is frequently reported as ranges, cumulative totals, confirmed vs. suspected counts, or with different denominators. The only numeric validation is the MERS-Cov Saudi Arabia yearly comparison, which covers one disease/country pair and relies on post hoc exclusion of 2013. Since a wrong count in a surveillance dataset is potentially more harmful than a missing count, the unvalidated numeric accuracy of the eKG fields is the most load-bearing weakness in the paper's argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes the construction and release of eKG, a knowledge graph and daily-updated dataset built by automatically extracting epidemiological fields (disease name, country, event date, case count, death count) from WHO Disease Outbreak News reports. Extraction is performed by an ensemble of three open-source LLMs (Mistral-7B-OpenOrca, Meta-Llama-3-70B-Instruct, Zephyr-7B-Beta) with majority voting and synonym/semantic-similarity normalization. The paper reports benchmarking against 171 expert-annotated samples from the Incident Database, compares the ensemble with commercial and open models, and presents a yearly MERS-CoV regression as a quantitative check. The paper also documents the FAIR publication of the data, SPARQL endpoint, and several visualization services.","tokens_in":20240,"tokens_out":5060,"duration_ms":64888,"significance":"The resource is potentially valuable: it converts a large, underutilized narrative corpus into structured, queryable data, is publicly released under CC BY 4.0, and is accompanied by code and interactive endpoints. The open-source LLM ensemble approach is also practically relevant because it avoids commercial API constraints. However, the validation in its current form does not substantiate the central reliability claim. The reported F1 scores measure only the presence/absence of a field, not whether the extracted numeric values are correct; case counts are therefore not truly validated, and death counts are not validated at all. If the authors can provide per-record numeric validation for case and death counts, the contribution would be substantially stronger. As it stands, the significance is contingent on additional validation.","major_comments":[{"comment":"The evaluation is explicitly a binary presence/absence test: 'Each IE task was assessed as a binary classification problem.' This means a true positive is recorded whenever the model outputs any value for a report whose IDB record contains that field, not when the extracted number matches the gold standard. For cases, Table 4 reports ensemble F1=0.658 and recall=0.591, so even at the presence level the pipeline misses roughly 41% of reports containing a case count. The F1 scores therefore say nothing about whether the numeric contents of cases_extracted are accurate. The later MERS-CoV comparison in Figure 5 aggregates yearly totals for one disease/country pair and does not test per-record numeric correctness. The claim that the ensemble results 'support the technical quality of the extracted eKG dataset' is not supported for the numeric fields.","section":"Technical Validation, Tables 2-4"},{"comment":"The paper states that death extraction could not be evaluated because the IDB does not annotate deaths, and then assumes that its performance is 'likely to be comparable' to case extraction. This is an unsupported premise, and it is load-bearing because deaths_extracted is a headline field in eKG. The assumption is also fragile: mortality in DON reports is often reported as ranges, cumulative totals, or confirmed vs. suspected counts, so it is not methodologically similar to extracting a single case count. In addition, the extraction prompt shown in the Methods section asks for the number of deaths but specifies JSON keys only for 'disease name, country, date, cases'; death appears only in the illustrative output. This under-specification compounds the lack of validation. The manuscript should either annotate and evaluate death extraction or explicitly mark deaths_extracted as unvalidate","section":"Technical Validation and Methods (prompt)"},{"comment":"The only quantitative validation of numeric values is the yearly regression of reconstructed MERS-CoV cases in Saudi Arabia against WHO totals. This is a single disease/country pair, and the comparison excludes 2013 because the authors suspect the WHO DONs included potential cases in that year. The exclusion is post hoc and is not described as a pre-specified criterion. Moreover, yearly aggregation cannot detect errors in individual extracted counts, such as systematic over- or under-reporting that cancels out at the annual level. This evidence is too narrow to establish the general numeric reliability of cases_extracted across the 2384-event dataset, and it provides no direct evidence for deaths_extracted.","section":"Technical Validation, Figure 5"}],"minor_comments":[{"comment":"The text reports '2.3 generalized axioms'; this likely should be '2.3k' or '2,300'. Please correct the typo and verify the number.","section":"Methods, KGs Services & Interfaces"},{"comment":"The ensemble description says the models share an 8K context length, while individual model descriptions state a 4,096-token context. Clarify which context length is actually used.","section":"Methods, LLMs for Epidemic IE"},{"comment":"The JSON keys listed in the prompt omit 'deaths' even though the prompt asks for it. If the final pipeline includes deaths_extracted, the prompt should reflect that; otherwise the provenance of the field is unclear.","section":"Methods, prompt"},{"comment":"No confidence intervals or significance tests are reported for the F1 differences, despite the small 171-sample benchmark. Many of the differences among top models are small (e.g., disease F1 0.851 vs. 0.842), so a statement about statistical significance would be useful.","section":"Technical Validation, Tables 2-5"}],"recommendation":"major_revision","confidential_remarks":"The main gap is between the paper's reliability claims and the evidence: numeric case counts and all death counts are unvalidated, and the prompt specification for deaths is internally inconsistent. This is close to a rejection in a data-descriptor venue because the primary deliverable is a dataset whose headline numeric fields are not verified. I nevertheless recommend major revision rather than rejection because the resource is genuinely useful, the code and data are publicly available, and targeted per-record validation of numeric fields appears feasible within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Genuinely new here is the resource, not the evaluation. eKG gives a daily-updated, structured extraction of WHO Disease Outbreak News, published as CSV and RDF, with ontology alignments, SPARQL access, and code on GitHub. That is useful and worth having. The pipeline is practical: three open LLMs, majority voting with synonym dictionaries built from SBERT and BioBERT, and a sensible summarization step for long reports. Credit is due for shipping data and code.\n\nThe soft spot is validation, and it is load-bearing. The F1 scores in Tables 2-5 measure each extraction task as binary presence/absence: did the model output any value for a report whose IDB record contains that field? They do not measure whether the extracted number matches the gold-standard number. So the case-count F1 of 0.658 with recall 0.591 means the ensemble misses case counts in about 41% of reports even at presence/absence level, and we know nothing about whether the numbers it outputs are correct. Deaths are never evaluated at all. The paper says death extraction is likely to be comparable because the tasks are similar. That is unsupported, and mortality reporting in DONs is messy: ranges, cumulative totals, confirmed versus suspected counts, different denominators. For a surveillance resource, a wrong count is more harmful than a missing one.\n\nThe MERS-CoV comparison is the only numeric validation, and it covers one disease-country pair. Excluding 2013 post hoc to improve the regression has a plausible explanation, but the fragility is visible in the difference between the two panels of Figure 5. That does not sink the resource, but it cannot carry a general reliability claim.\n\nMinor: SPARQL endpoint is password protected despite text suggesting otherwise, '2.3 generalized axioms' looks like a typo, and dataset size should be stated more clearly (2.9K events before cleaning, 2,384 after).\n\nThis paper should go to peer review, not desk reject. A serious referee should ask for exact-match evaluation of cases and deaths on an annotated sample, or at minimum for explicit caveats about numeric accuracy. The resource is worth publishing; the current validation does not support the strong reliability framing.","headline":"Useful public resource with a real validation gap: the numeric fields—case counts and deaths—are never actually checked against gold-standard numbers.","tokens_in":739,"tokens_out":1804,"would_cite":false,"duration_ms":42449,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an ensemble of three open-source LLMs, decided by majority vote, extracts disease, country, date, case, and death fields from WHO outbreak bulletins at least as accurately as commercial models, making the released eKG","keywords":["epidemiological surveillance","large language models","knowledge graph","information extraction","disease outbreak news","ensemble learning","linked open data","event-based surveillance"],"falsifier":"Take a random sample of DON reports that report deaths, run the ensemble to extract the deaths field, and compare each value against the figure stated in the report and, wherever possible, against WHO's own outbreak updates; a material drop in deaths-F1 relative to cases-F1 would refute the comparability assumption and would call for a mortality-specific prompt or validator.","tokens_in":19830,"feed_emoji":"🦠","tokens_out":6927,"duration_ms":66843,"temperature":0.7,"pith_summary":"The paper's central claim is that an ensemble of three open-source large language models, combined by majority voting, can turn the WHO's prose Disease Outbreak News bulletins into a structured, queryable record of outbreaks with accuracy at least matching proprietary models. To establish this, the authors benchmark disease-name, country, date, and case-count extraction against expert-annotated reports and report that the ensemble reaches the top F1 score on all four tasks. On that basis, they release eKG, a daily-updated knowledge graph of roughly 2,900 outbreak events and about 26,000 triples, mapped to standard biomedical and geographic ontologies and accessible through a public SPARQL endpoint. A reader should care because the resource offers timely epidemic signals without the cost, rate limits, or confidentiality constraints of closed commercial APIs, and because the full pipeline is open-source and reusable.","feed_headline":"Ensemble of open-source LLMs tops outbreak-info extraction scores","feed_subtitle":"Majority vote across three models beats every single model tested on disease, country, date, and case-count tasks.","key_machinery":"The load-bearing mechanism is the Ensemble: three open-source LLMs are each prompted with a fixed extraction template, and their JSON outputs are reconciled by majority voting. For textual fields, votes are grouped using synonym dictionaries built from syntactic matching, WordNet synonymy, and semantic similarity via Sentence-BERT for country names and BioBERT for disease names, with a cosine threshold of 0.8. The majority result is then packaged as RDF triples in a knowledge graph that reuses the Infectious Disease Ontology and GeoNames classes, with an imputed date field drawn from the report title when no event date is found. The ensemble is what carries the accuracy claim: it outperforms","core_discovery":"The core discovery is that committee agreement among three open-source LLMs—Mistral-7B-OpenOrca, Meta-Llama-3-70B-Instruct, and Zephyr-7B-Beta—beats every individual model tested, including large proprietary ones, on this extraction task. On the 171-report benchmark, the ensemble scores F1 values of 0.851 for disease names, 0.962 for countries, 0.658 for confirmed-case counts, and 0.869 for event dates, the highest in all four comparisons. The authors take this as evidence that the fully automatic eKG pipeline is technically sound: reconstructed time series for MERS-CoV in Saudi Arabia correlate significantly with WHO-confirmed yearly counts, with the fit improving after excluding the 2013 o","pith_inferences":["The unvalidated deaths field is worth an explicit audit before eKG feeds any mortality-focused surveillance: the paper's similarity argument is plausible, but death counts are often reported as cumulative totals or with caveats, so majority voting over raw numbers could silently under- or over-count.","A stratified evaluation by disease-report frequency would test where the ensemble degrades: the 171-report benchmark may not reflect the long tail of rare pathogens that makes up much of the full DONs archive.","If extraction errors are largely idiosyncratic to each model rather than shared, adding a fourth diverse open-source model might further raise F1; the same architecture directly supports that experiment.","The synonym-clustering step for disease names could be turned into a benchmark of its own: comparing the ensemble's labels against ICD-10 codes would measure whether the ontology mapping layer adds value beyond string matching."],"forward_implications":["eKG can be queried with SPARQL, letting researchers test hypotheses about outbreak timing, geography, and case loads without reading thousands of prose reports.","The pipeline runs daily and is built only from open-source models, so it can be redeployed on the full archive of over 3,000 DON reports without API quotas or per-token costs.","A qualitative check reproduces known epidemic curves—MERS-CoV in Saudi Arabia, Ebola in the Democratic Republic of the Congo and Guinea, and SARS in China—supporting the dataset's use for retrospective epidemiological analysis.","The imputed date field, taken from the report title, is flagged as potentially later than the true event date, meaning time-series analyses should prefer the extracted date when present.","Because the knowledge graph maps to standard ontologies, eKG can be linked to other linked-open-data resources to combine outbreak data with geographic, demographic, or clinical datasets."],"supporting_citations":[{"why":"Defines the source corpus: WHO Disease Outbreak News, the set of narrative outbreak reports the pipeline processes.","marker":"[5]"},{"why":"Supplies the Transformer architecture underlying all three LLMs used in the ensemble.","marker":"[9]"},{"why":"Previous benchmark of LLMs for epidemic information extraction that the ensemble builds on and is compared against.","marker":"[11]"},{"why":"One of the three ensemble models (Mistral-7B-OpenOrca); the paper evaluates its individual extraction performance.","marker":"[23]"},{"why":"One of the three ensemble models (Zephyr-7B-Beta); the paper evaluates its individual extraction performance.","marker":"[28]"},{"why":"The third ensemble model (Meta-Llama-3-70B-Instruct); the paper evaluates its individual extraction performance.","marker":"[29]"},{"why":"Infectious Disease Ontology, whose classes are reused for disease-outbreak events in eKG, linking to ICD-10.","marker":"[45]"},{"why":"GeoNames, reused for geographic entities in eKG (countries and regions).","marker":"[46]"},{"why":"WHO-confirmed MERS-CoV yearly counts used for qualitative regression validation of the reconstructed time series.","marker":"[64]"},{"why":"Incident Database, the expert-annotated gold standard that all extraction models are scored against.","marker":"[68]"}],"fun_headline_variants":["Three open-source LLMs beat all rivals on outbreak data extraction","Ensemble of three LLMs outperforms all in outbreak extraction","Majority vote of 3 open-source LLMs beats all on outbreak data","Committee voting among three LLMs wins outbreak data extraction","Ensemble beats every single LLM on WHO outbreak info extraction"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The dataset's headline death counts are never checked against expert annotations; the paper assumes deaths are extracted as reliably as case counts, because deaths were not annotated in the benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Three open-source LLMs beat all rivals on outbreak data extraction","Ensemble of three LLMs outperforms all in outbreak extraction","Majority vote of 3 open-source LLMs beats all on outbreak data","Committee voting among three LLMs wins outbreak data extraction","Ensemble beats every single LLM on WHO outbreak info extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2792,"prompt_tokens":727,"completion_tokens":2065,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1992}},"tokens_in":471,"tokens_out":2065,"duration_ms":16751,"temperature":1.0,"reasoning_tokens":1992,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:41:47.705239+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of DON reports that report deaths, run the ensemble to extract the deaths field, and compare each value against the figure stated in the report and, wherever possible, against WHO's own outbreak updates; a material drop in deaths-F1 relative to cases-F1 would refute the comparability assumption and would call for a mortality-specific prompt or validator.","supporting_citations":[{"cited_title":"Disease Outbreak News (2024)","cited_arxiv_id":null,"evidence_quote":"Defines the source corpus: WHO Disease Outbreak News, the set of narrative outbreak reports the pipeline processes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer architecture underlying all three LLMs used in the ensemble."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Previous benchmark of LLMs for epidemic information extraction that the ensemble builds on and is compared against."},{"cited_title":"MistralOrca: Mistral-7B Model Instruct-tuned on Filtered OpenOrcaV1 GPT-4 Dataset (2023)","cited_arxiv_id":null,"evidence_quote":"One of the three ensemble models (Mistral-7B-OpenOrca); the paper evaluates its individual extraction performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Infectious Disease Ontology, whose classes are reused for disease-outbreak events in eKG, linking to ICD-10."},{"cited_title":"GeoNames geographical database (2024)","cited_arxiv_id":null,"evidence_quote":"GeoNames, reused for geographic entities in eKG (countries and regions)."},{"cited_title":"Middle East respiratory syndrome coronavirus-Kingdom of Saudi Arabia (2024)","cited_arxiv_id":null,"evidence_quote":"WHO-confirmed MERS-CoV yearly counts used for qualitative regression validation of the reconstructed time series."},{"cited_title":"& Ghozzi, S","cited_arxiv_id":null,"evidence_quote":"Incident Database, the expert-annotated gold standard that all extraction models are scored against."}],"review_version":1}