{"id":"6022d37c-6a91-4ec8-81b3-dbb423c8da89","arxiv_id":"2412.15256","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM plus knowledge graph pipeline extracts HPO phenotypes from EHR notes and identifies 12 suspected undiagnosed BPAN patients among 33.6 million, with benchmark validation on three public datasets.","lead":"Large language models extract symptom terms from electronic health records and map them to standard medical ontologies to build patient knowledge graphs. The authors apply this pipeline to 33.6 million patients and report 12 suspected undiagnosed cases of the rare disease BPAN.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BPAN discovery claim rests on an undisclosed scoring method calibrated on n=2 confirmed cases; with no external validation or molecular follow-up, '12 very high probability cases' is unsupported.","rationale":"I read the paper as a credible applied ML report: the benchmark experiments are independent, reproducible in principle, and the HPO extraction results broadly align with prior published work. The conditional verdict is appropriate because the paper's distinctive contribution is the real-world BPAN discovery, and that discovery rests on a step that is neither disclosed nor validated. The public benchmarks cover entity extraction and phenotype classification, but they do not cover the proprietary 0-9 likelihood rating that converts extracted features into the final 12-patient shortlist. With only two confirmed BPAN patients available, any scoring rule can be overfit to idiosyncratic properties of those two records, and no statistic is reported to rule this out. The 12 candidates are described as 'very high probability' without clinical or molecular confirmation, so the claim should be read as hypothesis generation rather than validated discovery. This is exactly the weakest assumption identified by the reader, and the proposed tests would settle it: leave-one-out ranking of the two known positives, false-positive rate estimation on code-matched controls, and WDR45 or expert clinical review of the final candidates. Since the reader already recommends a conditional verdict rather than acceptance, my stress-test does not move the verdict; it reinforces the need for disclosure and validation before the discovery claim is accepted.","tokens_in":13936,"tokens_out":5872,"duration_ms":58051,"concrete_test":"Disclose the proprietary scoring method and perform a leave-one-out validation of the ranking step: fit or freeze the rule using one of the two confirmed BPAN patients, then check whether the other confirmed BPAN patient ranks in the top 12 of the ~24k candidate pool, and measure the false-positive rate at the 7-9 threshold on a random sample of patients sharing the same six generic ICD-10 codes but lacking any BPAN mention. If the held-out patient is not ranked in the top 12, or if controls are frequently scored 7 or higher, the final 12-candidate list is not evidence of BPAN discovery. Supplement this with WDR45 sequencing or blinded expert chart review of the 12 candidates if available.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in the paper is Section V's proprietary 0-9 BPAN likelihood rating applied to the roughly 24k patients matching one of six generic ICD-10 codes. The only positives available to design or calibrate that rating are two patients found by a keyword search for 'BPAN'. With n=2, the rating cannot be tested for overfitting, and the final 12 'very high probability' candidates may simply resemble the two index patients in note-writing style, demographic mix, or code co-occurrence rather than in true BPAN phenotype. No precision, recall, AUC, or calibration statistic is reported for the real-world ranking, and the 12 candidates have no clinical confirmation or WDR45 variant data. The public benchmark results (e.g., HPO extraction F1 up to 0.72 on BiolarkGSC+) validate entity extraction, not the proprietary scoring that actually produces the discovery. Given a target of 12 positives in a pool of about 24k, the required specificity exceeds 99.95%, a level that cannot be established from two positive examples. The claim 'very high probability cases of BPAN' therefore outruns the evidence and should be treated as an unverified computational hypothesis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for constructing patient-specific knowledge graphs from unstructured EHR notes using LLM-based extraction, with extracted entities mapped to standard ontologies (MeSH, SNOMED-CT, RxNORM, HPO). The method is benchmarked on three public tasks: BC5CDR chemical/disease NER (Table I), MIMIC-III multi-label phenotype classification (Table II), and BiolarkGSC+ HPO extraction and mapping (Table III). The authors then apply the approach to two rare diseases in a 33.6M-patient ambulatory EHR dataset. For Dravet syndrome, they use 38 patients with ICD-10 codes as ground truth, extract 46 HPO phenotypes per patient, and compare observed phenotype frequencies with HPO frequency categories. For BPAN, where no ICD-10 code exists, they identify two patients via a keyword search, use a proprietary 0-9 likelihood rating over roughly 24k patients matching six generic ICD-10 codes, and report having narrowed the list to 12 'very high probability cases of BPAN.' The central contribution is the claim that LLM-based HPO extraction plus a proprietary scoring method can discover undiagnosed rare-disease patients in real-world EHR data.","tokens_in":14153,"tokens_out":4601,"duration_ms":46307,"significance":"If the BPAN discovery claim were properly supported, this would be a meaningful demonstration of using LLM-based phenotype extraction for rare-disease cohort discovery in a very large EHR database. The public benchmark work is a genuine strength: it covers three standard datasets, multiple model families, static and dynamic few-shot prompting, LoRA fine-tuning, and gleaning, with clear tables that allow comparison with prior systems. The authors also honestly report negative or mixed lessons, including their inability to reproduce PhenoGPT's full improvement and their skepticism about narrow fine-tuning generalization. However, the significance of the paper's headline result is currently conditional: the BPAN discovery rests on an undisclosed proprietary scoring method, calibrated against only two keyword-identified patients, with no reported precision, recall, calibration, or clinical validation. The Dravet frequency analysis also lacks manual validation of the LLM extractions. These issues must be resolved before the paper's real-world discovery claims can be accepted.","major_comments":[{"comment":"The BPAN discovery claim ('We were able to finally narrow this patient list down to 12 very high probability cases of BPAN') is the paper's headline result, but the evidence presented in Section V does not support it. The proprietary 0-9 rating method is not described, and the only apparent positives available for designing or calibrating it are two patients found by a keyword search for 'BPAN' in clinical notes. No precision, recall, AUC, or calibration statistic is reported for the ranking, and the final 12 candidates have no clinical confirmation, molecular (WDR45) follow-up, or manual chart review. With roughly 24k patients in the pool, identifying 12 true positives requires extremely high specificity, which cannot be established from two positive examples. Please disclose the scoring method, validate it on labeled positives and negatives, report calibration at the chosen threshold, and either provide independent confirmation of the 12 candidates or explicitly re-label them as unverified computational candidates rather than 'very high probability cases.'","section":"V"},{"comment":"The Dravet HPO extraction results are presented as if they are measurements, but no manual validation or precision/recall analysis is reported for the LLM extractions on the 38 real-world patients. Table IV lists patient counts per HPO term, and Figure 3 compares those counts against HPO frequency categories, yet there is no evidence that the extracted terms match the patients' actual records. Without a manual chart-review sample or inter-annotator agreement, the observed frequency discrepancies may reflect extraction errors rather than true phenotype presentation. Please report a validation set of the extracted HPO terms against clinician annotation or at least a random sample of patient charts, and discuss how the prompt's inclusion of the Dravet syndrome description may bias extraction.","section":"IV-C and Table IV"},{"comment":"The comparison between HPO frequencies and observed frequencies in the 38 Dravet patients is not statistically grounded. HPO frequency categories are coarse ranges (e.g., Frequent means 30-79%), while Table IV reports raw counts in a 38-patient cohort, and rows marked '-' are ambiguous (zero, not extracted, or not applicable). Figure 3 appears to be a heat map, but the text does not define its axes, color scale, or the denominator used for observed frequencies. This makes it impossible to assess whether the claimed differences between HPO annotations and real-world data are meaningful. Please specify how unmentioned phenotypes were handled, provide confidence intervals or exact tests for the observed proportions, and clarify the figure's construction.","section":"IV-D and Figure 3"},{"comment":"The two 'confirmed' BPAN patients are identified only by a keyword search for the string 'BPAN' in unstructured notes. The manuscript does not state whether these patients have genetic confirmation, a clinical diagnosis documented by a specialist, or merely a note mentioning BPAN in a differential or rule-out context. Because these two patients are the sole anchor points for the proprietary scoring method, the meaning of 'confirmed' is load-bearing. Please describe how confirmation was established and how the two index patients were separated from other notes that might mention BPAN without having the disease.","section":"V"}],"minor_comments":[{"comment":"The phrase 'RH's test dataset' is confusing because the 33.6M-patient EHR resource is not a machine-learning test set; please call it a de-identified research dataset or real-world database.","section":"II-D"},{"comment":"The column labeled 'Frequency' actually contains patient counts; please rename it 'Patient count' or 'Patients with HPO term (n=38).' Also, replace the '-' entries with 0 or 'not observed' so the table is unambiguous.","section":"Table IV"},{"comment":"The dataset name appears as both 'BiolarkGSC+' and 'Bio-Lark Gold Standard Corpus'; please standardize the spelling to 'Bio-Lark GSC+' and cite the original dataset description.","section":"III-C"},{"comment":"For the LoRA fine-tuning, only rank (64) and epochs (2) are reported; please add learning rate, LoRA alpha/dropout, quantization details, and the validation procedure so the experiment is reproducible.","section":"III-C"},{"comment":"The heat map is described only in a caption and lacks visible axis labels, a color scale, and a description of how missing HPO terms were treated; without these, the figure is not interpretable.","section":"Figure 3"},{"comment":"The caption says the knowledge graph consists of 'structured elements from the EHR, as well as a dearth of unstructured information in the medical notes'; this appears to be a typo, since the paper elsewhere emphasizes extensive unstructured notes. Please replace 'dearth' with 'body' or 'wealth.'","section":"IV-B, Figure 2 caption"},{"comment":"The manuscript says 'We then utilize a proprietary method' without defining the 0-9 score's anchors, inputs, or threshold; at minimum, please state what a score of 7-9 means and how the threshold was chosen.","section":"V"},{"comment":"The prompt instructs the model to output reasoning for each HPO match, but the response format also says 'NO additional fields beyond the specified format'; please reconcile this minor inconsistency.","section":"Appendix, Listing 1"}],"recommendation":"major_revision","confidential_remarks":"The public benchmark experiments appear sound and are presented with useful comparisons, and the authors deserve credit for reporting negative lessons and for making the Dravet extraction prompt available. The main barrier to acceptance is the proprietary BPAN scoring method: because it is undisclosed and validated against only two keyword-identified patients, the paper's most significant claim cannot currently be verified. I would urge the editor to require either full disclosure of the scoring method or independent validation (e.g., clinician chart review or genetic follow-up of the 12 candidates) before publication. The current framing of the 12 candidates as 'very high probability cases' is disproportionate to the evidence. This is a fixable issue if the authors add validation or soften the claim, so I am not recommending rejection on this basis alone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The part worth reading is the benchmark work and the Dravet case study. They run standard datasets (BC5CDR, MIMIC-III, BiolarkGSC+), report clear tables, and the lessons are sensible: fine-tuned encoders still win binary NER, dynamic few-shot helps LLMs, gleaning improves recall, and narrow LoRA fine-tuning can look good on a benchmark while generalizing poorly. They are also honest about not replicating PhenoGPT's boost. That is real applied ML reporting and I trust those numbers more than most arXiv papers of this type.\n\nThe genuinely new piece is applying HPO-grounded LLM extraction to a 33.6M-patient ambulatory EHR and trying to find undiagnosed BPAN patients. The Dravet experiment gives the method a plausibility check, and the BPAN search is a real-world application with no ground truth. That is a worthwhile question.\n\nThe soft spot is exactly where the stress-test note lands. The BPAN discovery rests on a proprietary 0–9 scoring method applied to ~24k patients. The only positive examples are two patients found by keyword search. No precision, recall, AUC, or calibration is reported for the ranking, and the 12 “very high probability” candidates have no clinical confirmation or WDR45 variant data. With 12 positives out of 24k, the required specificity is north of 99.95%, and you cannot establish that from two index patients. The candidates could easily resemble the two known patients in note style or code co-occurrence rather than true BPAN phenotype. The phrase “very high probability cases” outruns the evidence. Also, Table IV reports extraction counts without precision/recall or error bars, and the Dravet cohort is only 38 patients. These are proportional concerns: the benchmark sections stand on their own, but the discovery claim does not.\n\nThe citation pattern looks fine. The methods are grounded in public ontologies and benchmarks, and the paper acknowledges limitations openly. There is no code or data shipped, and the load-bearing scoring method is undisclosed, so the central result is not reproducible as written.\n\nWho is this for? Clinical NLP researchers and anyone building LLM-based phenotype extraction pipelines for EHR cohort discovery. It deserves a serious referee, but the referee should demand either disclosure of the scoring method and a validation set, or reframing the 12 BPAN candidates as an unverified computational hypothesis. As written, the paper is a conditional reject; the benchmark half is publishable and the BPAN half needs much more evidence.","headline":"Solid benchmark survey and an interesting rare-disease search use case, but the headline BPAN discovery is an unvalidated hypothesis built on an undisclosed scoring method with n=2 positives.","tokens_in":14687,"tokens_out":1666,"would_cite":false,"duration_ms":17441,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A patient knowledge graph built by LLM extraction can flag undiagnosed rare disease patients, as demonstrated by searching 33.6 million EHR records for BPAN.","keywords":["large language models","knowledge graphs","ontology mapping","structured extraction","Dravet syndrome","BPAN","Human Phenotype Ontology","electronic health records"],"falsifier":"Enroll the 12 candidate patients and a matched random sample of the remaining 24,000 generic-code patients in a blinded neurological evaluation or WDR45 gene test; if the 12 do not show a markedly higher BPAN rate than the 24,000-patient pool, the ranking carries no discriminative information. A simpler check is to re-score the patients after removing all information from the two known BPAN cases from the ranking procedure; if the top 12 shift completely, the method was overfit.","tokens_in":13737,"feed_emoji":"🧬","tokens_out":8326,"duration_ms":68306,"temperature":0.7,"pith_summary":"This paper argues that large language models can replace rigid medical coding as the front end of patient search: instead of matching patients to exact ICD-10 or SNOMED codes, an LLM reads unstructured clinical notes and extracts symptom entities, which are then grounded in standard ontologies and assembled into a per-patient knowledge graph. The authors validate the extraction pipeline on public benchmark corpora for named-entity recognition, multi-label phenotype classification, and HPO term mapping, and then apply it to two rare diseases in a 33.6-million-patient ambulatory EHR database. Using 38 ICD-10-confirmed Dravet syndrome patients as ground truth, they show that their extracted HPO phenotype frequencies broadly track the frequencies the HPO assigns to the disease. They then run the same method on BPAN, a rare disease with no ICD-10 code, and report narrowing a search of over 24,000 candidates with generic epilepsy- and development-related codes to 12 patients they describe as very high probability undiagnosed BPAN cases. If the result holds, the approach would give researchers a scalable way to find and study rare disease patients years before coding systems catch up.","feed_headline":"12 rare-disease patients surfaced from 33.6M EHR records","feed_subtitle":"Expert-written notes become HPO phenotypes that rank candidates without an ICD-10 code.","key_machinery":"The load-bearing object is the patient-specific knowledge graph: a directed labeled graph in which clinical entities (patients, symptoms, diagnoses, treatments) are nodes and their relationships are edges, with the unstructured text of medical notes attached as node attributes. The extraction step is a single LLM prompt that takes a patient's history and a list of candidate HPO terms, and returns a JSON structure per patient listing each matched HPO identifier with a confidence score and a reasoning sentence. HPO (Human Phenotype Ontology) supplies the standardized vocabulary of over 13,000 phenotype terms that anchors the extraction and makes different patients comparable. For the BPAN ranking, a proprietary method (not described) converts each candidate's knowledge graph into a 0-9 likelihood score, and the top-scoring patients are re-extracted for phenotypic review. The machinery's role is to convert free-text evidence into a searchable, ontology-anchored representation that can be filtered by symptom rather than by code.","core_discovery":"On the paper's own terms, the central claim is that a patient knowledge graph grounded in HPO, MeSH, SNOMED-CT, and RxNORM, with nodes extracted by LLM prompting from unstructured notes, is a workable search index for rare disease discovery. The evidence has two parts. In the Dravet syndrome case, the method extracts 48 HPO phenotypes from the notes of 38 ICD-10-coded patients and compares the observed frequencies with HPO's standard frequencies, finding broad agreement: the most frequent real-world phenotypes (complex febrile seizure, febrile seizure, cognitive impairment) match HPO's expectations, while head, neck, and limb presentations are largely absent from the notes. In the BPAN case, there is no ground truth: after a keyword search finds only two confirmed patients, the authors broaden to six generic ICD-10 codes covering 24,000 patients, score each with a proprietary 0-9 likelihood method, and report that a knowledge-graph review of the 268 patients scoring 7 to 9 reduces the list to 12 very high probability undiagnosed BPAN cases. The paper therefore claims that phenotype-based search via LLM extraction can identify rare disease patients whose condition has no codified diagnosis.","pith_inferences":["The 12 BPAN candidates have no clinical confirmation, so the paper's headline number is an unvalidated prediction; a chart review or WDR45 genetic test would be the decisive next experiment.","Because the 0-9 scoring method is proprietary and undisclosed, the result cannot be independently reproduced; replacing it with a published algorithm would turn the pipeline into a testable baseline.","The same approach could be applied to a disease with delayed code adoption, using notes written before the code existed, to test retrospectively how many 'undiagnosed' patients the method recovers once the code appears.","The absence of head, neck, and limb phenotypes in the Dravet notes may reflect documentation bias in ambulatory care rather than true phenotype rarity; linking extraction to structured exam fields would separate those possibilities."],"forward_implications":["Rare diseases without their own ICD-10 code become searchable by phenotype, so patient recruitment for trials and natural-history studies can start years earlier.","The observed mismatch between HPO's reference frequencies and the frequencies in real-world notes suggests ontology entries should be updated with data-driven evidence, not only expert curation.","The same extraction pipeline can be reused across diseases by swapping the candidate HPO list, so the marginal cost of adding a new disease search is low.","Documentation artifacts are visible in the frequency comparison, which means the method could also serve as a quality signal for EHR note completeness."],"supporting_citations":[{"why":"Supplies the Human Phenotype Ontology vocabulary and frequency annotations that the extraction maps to and compares against.","marker":"[29]"},{"why":"Supplies the benchmark corpus for chemical and disease entity extraction in Experiment 1.","marker":"[35]"},{"why":"Supplies the benchmark notes and labels for multi-label phenotype classification in Experiment 2.","marker":"[33]"},{"why":"Provides the HPO extraction benchmark validation set and the PhenoReRank comparison in Experiment 3.","marker":"[48]"},{"why":"Supplies the method and LoRA fine-tuning approach the authors re-implement as an HPO extraction baseline.","marker":"[20]"},{"why":"Supplies the fine-tuned BERT baseline that outperforms LLMs on binary chemical and disease NER in Experiment 1.","marker":"[15]"},{"why":"Supplies the 33.6-million-patient real-world EHR dataset used for the Dravet and BPAN case studies.","marker":"[34]"},{"why":"Supplies the G40.83 codes used as ground truth for confirmed Dravet syndrome patients.","marker":"[27]"}],"fun_headline_variants":["LLM graph finds 12 undiagnosed BPAN cases in 33M records","No code? LLM phenotype search still nets 12 rare patients","HPO from notes: 12 BPAN leads without a ground truth code","Phenotype extraction from EHR notes uncovers 12 BPAN candidates","From 33.6M records, LLM-built graph isolates 12 likely BPAN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the undisclosed proprietary 0-9 BPAN scoring is calibrated and not overfit to the two known BPAN patients, and that the LLM-extracted HPO terms are accurate enough to support the final ranking, with no measured precision for the real-world extraction and no clinical confirmation of the 12 finalists.","fun_headline_variants_meta":{"raw":{"variants":["LLM graph finds 12 undiagnosed BPAN cases in 33M records","No code? LLM phenotype search still nets 12 rare patients","HPO from notes: 12 BPAN leads without a ground truth code","Phenotype extraction from EHR notes uncovers 12 BPAN candidates","From 33.6M records, LLM-built graph isolates 12 likely BPAN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000375,"raw_usage":{"total_tokens":2027,"prompt_tokens":1000,"completion_tokens":1027,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":924}},"tokens_in":616,"tokens_out":1027,"duration_ms":8209,"temperature":1.0,"reasoning_tokens":924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:58:00.344906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Enroll the 12 candidate patients and a matched random sample of the remaining 24,000 generic-code patients in a blinded neurological evaluation or WDR45 gene test; if the 12 do not show a markedly higher BPAN rate than the 24,000-patient pool, the ranking carries no discriminative information. A simpler check is to re-score the patients after removing all information from the two known BPAN cases from the ranking procedure; if the top 12 shift completely, the method was overfit.","supporting_citations":[{"cited_title":"Biocreative v cdr task corpus: a resource for chemical disease relation extraction","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark corpus for chemical and disease entity extraction in Experiment 1."},{"cited_title":"Phenotype annotations for patient notes in the mimic-iii database","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark notes and labels for multi-label phenotype classification in Experiment 2."},{"cited_title":"Phenorerank: A re-ranking model for phenotypic concept recognition pre-trained on human phenotype ontology","cited_arxiv_id":null,"evidence_quote":"Provides the HPO extraction benchmark validation set and the PhenoReRank comparison in Experiment 3."},{"cited_title":"Enhancing phenotype recognition in clinical notes using large language models: Phenobcbert and phenogpt","cited_arxiv_id":null,"evidence_quote":"Supplies the method and LoRA fine-tuning approach the authors re-implement as an HPO extraction baseline."},{"cited_title":"Biobert: a pre-trained biomedical language representation model for biomedical text mining","cited_arxiv_id":null,"evidence_quote":"Supplies the fine-tuned BERT baseline that outperforms LLMs on binary chemical and disease NER in Experiment 1."},{"cited_title":"Respond.health – clinical discovery that responds to the real world","cited_arxiv_id":null,"evidence_quote":"Supplies the 33.6-million-patient real-world EHR dataset used for the Dravet and BPAN case studies."},{"cited_title":"International Statistical Classification of Diseases and related health problems: Alphabetical index , volume 3","cited_arxiv_id":null,"evidence_quote":"Supplies the G40.83 codes used as ground truth for confirmed Dravet syndrome patients."}],"review_version":1}