{"id":"4c715c8d-f7a8-4674-bf30-be4a52a091d5","arxiv_id":"2501.00031","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Small BERT models distilled from LLM and ontology labels match the teachers on medication and disease extraction at a fraction of cost, but trail on symptoms.","lead":"The paper trains small BERT models on labels extracted by large language models and medical ontologies to identify medications, diseases, and symptoms in clinical notes. The distilled models run far faster and cheaper than the LLM teachers, with matching F1 scores for medications and diseases, though not for symptoms.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cost comparison uses inconsistent GPU pricing; the claimed 101x savings drops to roughly 17x under the paper's stated $28/hr rate, and the Gemini Flash comparison inverts.","rationale":"The paper's core methodology—distilling from LLM and ontology teachers into BERT for three NER tasks, with a thorough teacher-combination search and external validation—is sound and potentially useful. The strongest claim, however, is the economic one: distilled BERT is 'up to 101x cheaper' while maintaining similar performance. The internal inconsistency between the $28/hour rate stated in Methods and the $4.74/hour rate used in Table 4 is a concrete, load-bearing flaw because the 101x figure is computed from the lower rate. Recomputing with the paper's stated rate reduces the maximum savings to about 17x and makes the Gemini Flash comparison unfavorable to BERT. This is not a disagreement with an external consensus; it is an internal inconsistency that any careful reader can verify. The rest of the paper's claims (performance, speed, generalizability) are less affected: the performance numbers are credible, and the error analysis even suggests F1 scores may be lower bounds. The external validation sample is small, but that is a limitation, not a fatal flaw. The reader's conditional verdict is appropriate; our concern reinforces it rather than changing it. We recommend keeping CONDITIONAL, with the required revision being a consistent, clearly stated hardware-cost assumption and a recomputed cost-benefit table.","tokens_in":16534,"tokens_out":4787,"duration_ms":42814,"concrete_test":"Recompute the cost per note and cost-ratio column of Table 4 using the paper's stated $28/hour rate for 4xH100 (or, alternatively, confirm from the released code which GPU and hourly rate were actually used for inference). If the corrected ratios fall to approximately 15-17x and the Gemini Flash comparison becomes negative, the abstract's 'up to 101x cheaper' claim must be revised or accompanied by an explicit statement that the lower $4.74/hour rate for a single A100 is the deployment assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—'up to 101x cheaper'—rests on an internally inconsistent cost model. Methods ('Inference Time and Cost Analysis') states: 'We estimate cost per note for BERT models by multiplying inference time by $28/hour, which is the average cost of a virtual machine with 4xH100.' Yet Table 4 and Table 6 use $4.74/hour for a 1xA100 80GB, and the Results text attributes this to a '4xH100 virtual machine' (a hardware mismatch). Using the paper's stated $28/hour rate, the distilled BioBERT cost per note becomes 0.14 s / 3600 s * $28 = $0.00109, not $0.000187. The cost ratios then become: GPT-4o $0.0159 / $0.00109 ≈ 14.6x, o1-mini $0.0189 / $0.00109 ≈ 17.3x, and Gemini Flash $0.00046 / $0.00109 ≈ 0.42x, meaning the '2x cheaper' claim for Gemini Flash reverses to BERT being about 2.4x more expensive. Thus the headline 'up to 101x cheaper' is not robust to the paper's own stated hardware cost. The $4.74 rate may be defensible for inference on a single A100, but the manuscript never reconciles this with the Methods text, and the abstract/conclusion do not hedge on this dependency. This directly undermines the economic motivation for distillation, which is a primary contribution of the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a knowledge-distillation pipeline for clinical named entity recognition (NER) in which large language models (GPT-4o, GPT-4o-mini, o1-mini, Gemini 1.5 Flash) and medical ontologies (RxNorm, SNOMED) serve as teacher labelers for medication, disease, and symptom extraction. The authors evaluate all 31 teacher combinations on development sets, select the best per task, fine-tune BERT variants on the resulting hard labels, and compare performance against the teachers and against BERT fine-tuned on human labels across five datasets, with external validation on 10 notes from MedAlign. The headline claims are that distilled BioBERT achieves F1 comparable to its teachers (0.84 vs 0.82 for disease; 0.87 vs 0.84 for medication; 0.68 vs 0.73 for symptoms) and is up to 101x cheaper and 12x faster in inference.","tokens_in":16874,"tokens_out":5270,"duration_ms":45534,"significance":"If the efficiency claims hold, distillation from LLM teacher labelers is a practically useful strategy for clinical NER, and the paper's direct comparison of teacher-only versus distilled-BERT performance, combined with error analysis across multiple datasets, is a valuable contribution. The authors also release code and report inter-annotator agreement, which supports reproducibility. However, the central cost and speed claims rest on an internally inconsistent computational-cost model, and the external validation is based on a very small sample, so the strength of the practical-deployment conclusion is currently not commensurate with the evidence presented.","major_comments":[{"comment":"The cost model is internally inconsistent. Methods states that BERT inference cost is estimated at $28/hour for a 4xH100 virtual machine, while Table 4 and the Results text use $4.74/hour for a 1xA100 80GB, and Table 6 lists only 1xA100 vendor rates. Under the paper's stated $28/hour rate, the distilled BioBERT cost per note becomes about $0.00109 rather than $0.000187, which changes the headline ratios from 85x/101x/2x to roughly 14.6x/17.3x/0.42x for GPT-4o/o1-mini/Gemini Flash. The Gemini Flash comparison would reverse, with BERT being more expensive. The abstract and conclusions do not hedge on this dependency. The authors must reconcile the stated hardware cost with the numbers used in the cost tables and re-evaluate the economic claims accordingly.","section":"Methods (Inference Time and Cost Analysis) and Results (Inference Time and Cost), Table 4, Table 6"},{"comment":"The external validation is based on only 10 manually annotated notes from MedAlign, with just 2 notes doubly annotated and a reported Cohen's kappa of 0.61. A kappa of 0.61 indicates moderate agreement and casts doubt on the reliability of the gold standard for this small sample. The paper nonetheless claims 'strong performance' on the external dataset. Confidence intervals or other uncertainty estimates are not provided, and the Discussion does not list this as a limitation. The generalizability claim is therefore not well supported by the presented evidence.","section":"Methods (External Validation), Table 11, Table 13"},{"comment":"The Discussion states that 'distilled BERT models outperformed teacher labelers,' but this is contradicted by the symptom extraction results in Table 2, where the teacher-only F1 is 0.73 and the distilled BioBERT F1 is 0.68. The blanket claim is therefore not accurate as written, and the symptom result is a negative result relative to the teacher. The Discussion should be revised to acknowledge that distillation underperformed the teacher for symptoms and to discuss possible reasons.","section":"Discussion, paragraph 1; Table 2"},{"comment":"The reported speed-up factors are inconsistent: the Abstract says distilled BERT was '12x, 4x, 8x faster' than GPT-4o, o1-mini, and Gemini Flash, respectively, while the Discussion says '2x, 4x, 8x'. Table 4's timing data (1.66 s vs 0.14 s) support 12x, but the Discussion's 2x appears to be an error. This inconsistency in a headline quantitative claim must be corrected.","section":"Abstract and Discussion, speed multipliers"},{"comment":"The inference-cost comparison omits the one-time cost of generating teacher labels via LLM API calls and the cost of fine-tuning the BERT models. For a fair 'cheaper' claim to be sustained in a deployment scenario, these fixed costs should be reported and amortized over a plausible volume of notes, or the authors should explicitly state that the comparison is inference-only and justify that the fixed costs are negligible for the intended use case.","section":"Results (Inference Time and Cost), Table 4"},{"comment":"The cost calculation for the LLM teachers is not fully transparent. The paper says tiktoken was used to compute token counts, but it does not report the number of input and output tokens per note, the pricing per million tokens used, or whether the long system prompts (which are included in the Supplementary) were counted. Providing these details is necessary for reproducibility and for assessing whether the reported per-note costs are accurate.","section":"Results (Inference Time and Cost), Table 4"}],"minor_comments":[{"comment":"The Results text says the cost estimate is for 'a 4xH100 virtual machine,' but Table 6 reports '1xA100 80GB' vendor prices. Unify the hardware description and the cost source.","section":"Results (Inference Time and Cost), Table 6 and accompanying text"},{"comment":"The learning rate is written as '2x10e-5'; this should be '2e-5' or '2 × 10⁻⁵' to avoid ambiguity.","section":"Methods (Model Distillation Implementation)"},{"comment":"The table caption states development splits were used for 'prompt engineering,' but the Methods describe using them for selecting the optimal teacher labeler combination. Align the terminology.","section":"Table 12"},{"comment":"The abstract says 'over 3,300 clinical notes,' but the teacher-labeling set is described as 2,096 documents; clarify whether the 3,300 figure includes test and development notes from all datasets, and make the arithmetic explicit.","section":"Abstract"},{"comment":"The output format instructions say 'Output a string delimited by //', but the example shows a JSON object with 'entities' and 'rationale' keys. This mismatch could confuse readers trying to replicate the teacher-labeling pipeline.","section":"Supplementary Material, Medication Prompt"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the efficiency claim (cost and speed) alongside comparable F1. The internal inconsistency in the cost model is not a mere presentation issue; it changes the direction of the Gemini Flash comparison and weakens the headline numbers. The external validation is also too small to support the generalizability claim. I would recommend major revision with the hope that the authors can fix the cost model, correct the speed multiplier inconsistency, soften the over-generalizing discussion, and clarify the limitations of the 10-note external validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [name],\n\nQuick take on arXiv:2501.00031: the paper shows you can distill labels from a mix of LLMs and ontologies into BERT-sized models for medication and disease NER and get F1 close to human-supervised BERT. The exhaustive 31-subset teacher search is the real contribution, and the error analysis is refreshingly honest. But the headline economy claim—up to 101x cheaper—doesn't survive contact with the paper's own cost assumptions. The Methods say $28/hour for a 4xH100 VM; Tables 4 and 6 use $4.74/hour for a 1xA100. At $28/hour, the savings vs GPT-4o and o1-mini drop to roughly 15–17x, and the Gemini Flash comparison flips: BERT becomes about 2.4x more expensive per note. That's a real problem because cost is one of the two headline claims.\n\nWhat's genuinely new: prior work distilled from a single LLM teacher; this paper systematically tests all 31 combinations of four LLMs plus an ontology, picks per-task winners on a dev set, and validates externally on MedAlign (though only 10 annotated notes, with Cohen's kappa 0.61). For medications and diseases, the distilled BioBERT (F1 0.87/0.84) beats the teacher labelers (0.84/0.82) and approaches human-supervised BERT (0.91/0.89). That's a useful result for anyone wanting HIPAA-friendly local NER. Symptoms are weaker: distilled BERT (0.68) is worse than the teacher (0.73), and the paper acknowledges the teacher was inconsistent there.\n\nOther caveats: the external validation is tiny, and the error analysis shows 63–82% of false positives were actually correct in the gold standard, so the reported F1s are lower bounds. That's a strength in transparency, but it means the point estimates are noisier than they look.\n\nThe cost inconsistency is fixable—acknowledge the $4.74 rate is for a single A100 and re-run the headline numbers—but as written the abstract overstates the economy claim. I'd still send this to a serious reviewer: the empirical design is thorough, the code is public, and the core distillation result for meds and diseases is solid. The authors just need to reconcile the hardware pricing and temper the 'up to 101x' language. Useful for the clinical NLP crowd, less so for anyone outside that niche.","headline":"Solid empirical distillation study for clinical NER, but the headline cost savings rest on an inconsistent GPU rate and the Gemini Flash comparison inverts under the paper's own stated price.","tokens_in":17391,"tokens_out":2551,"would_cite":true,"duration_ms":21573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distilled BERT models rival LLM teachers on clinical NER at 101x lower cost.","keywords":["knowledge distillation","clinical NER","BERT","large language models","named entity recognition","electronic health records","BioBERT","medication extraction"],"falsifier":"Recompute the cost per note using the $28/hour for 4xH100 stated in the Methods; if the cost advantage over o1-mini falls from 101x to roughly 15x, the headline economy claim as stated does not hold.","tokens_in":16351,"feed_emoji":"🩺","tokens_out":8380,"duration_ms":62748,"temperature":0.7,"pith_summary":"This paper asks whether the accuracy of large language models on clinical named-entity recognition can be transferred to a model roughly 1,000 times smaller, making the task cheap enough for routine use. The authors generate training labels for medications, diseases, and symptoms from API-based LLMs (GPT-4o, o1-mini, Gemini Flash) and medical ontologies (RxNorm, SNOMED), then fine-tune BERT-family models on those labels. On three public clinical datasets, the distilled BioBERT model reaches F1 of 0.84 for diseases and 0.87 for medications, close to the 0.89 and 0.91 of BERT trained on human labels, and above the LLM teachers' own direct F1 of 0.82 and 0.84. The distilled model runs up to 12 times faster and, at the paper's chosen pricing, up to 101 times cheaper than the teacher LLMs. The paper argues that distillation is a practical route to scalable clinical information extraction without large expert-annotated corpora.","feed_headline":"Distilled BERT rivals LLM teachers on clinical NER at 101x lower cost","feed_subtitle":"A model 1,000x smaller nearly matches LLM teacher labels on diseases and drugs, running in 0.14 seconds per note.","key_machinery":"The load-bearing mechanism is teacher labeler ensembling followed by supervised fine-tuning. Five teachers are used: four LLMs (GPT-4o, GPT-4o-mini, o1-mini, Gemini 1.5 Flash) and one ontology matcher (RxNorm for medications, SNOMED CT for diseases and symptoms). All 31 non-empty subsets of these teachers are evaluated on a development set, and the union of entities from the best-scoring combination is converted into Inside-Outside token labels for fine-tuning a BERT variant (BERT-base, BioBERT, or BioClinBERT). This union-of-entities operation is what transfers the teachers' knowledge into the small model, and the per-task selection of the best teacher subset is what makes the transfer work across different entity types.","core_discovery":"On its own terms, the paper's central claim is that knowledge distillation from multiple teacher labelers—commercial LLMs plus ontology matchers—produces small BERT models that perform essentially as well as those same LLMs do directly, and nearly as well as BERT fine-tuned on human annotations. For diseases, o1-mini alone was the best teacher (direct F1 0.787), yet the distilled BioBERT scored 0.84; for medications, Gemini Flash plus GPT-4o was the best teacher (0.881) and distilled BioBERT scored 0.87; for symptoms, the best teacher combination scored 0.801 and the distilled model 0.68. In external validation on notes from a different health system, the distilled model's F1 was 0.883 for medications, 0.726 for diseases, and 0.699 for symptoms. An error analysis found that 63–82% of the models' false positives were entities correctly identified but missing from the human test labels, so the paper argues the reported precision and F1 are lower bounds. The paper concludes that distillation offers an efficient and scalable alternative to LLM API calls for clinical NER.","pith_inferences":["Recomputing the cost comparison with the paper's stated $28/hour 4xH100 rate instead of the $4.74/hour 1xA100 rate used in Table 4 drops the headline 101x advantage over o1-mini to roughly 15x; the qualitative claim of large savings holds, but the exact multiplier depends on hardware pricing.","The union-of-entities rule for combining teachers may need per-task tuning for other entity types, since the paper finds that adding ontology labels sometimes hurts F1 by increasing false positives and that no single teacher wins across all tasks.","If test-set labels are as noisy as the error analysis suggests, published benchmark scores across the field may systematically understate real-world performance; cleaning and re-annotating the n2c2, NCBI, and CORAL test sets could change the ranking of many clinical NER models.","A self-improving loop is possible: as LLM teachers improve, regenerating teacher labels and re-fine-tuning BERT would improve the distilled models without any new human annotation, provided the teacher combination is re-selected on a stable development set."],"forward_implications":["A healthcare system can run clinical NER on local hardware with a roughly 110-million-parameter BERT model, avoiding per-query API costs and the need to send protected health information to external endpoints.","The distilled model's F1 on medications and diseases is within 0.02–0.05 of BERT trained on fully human-labeled data, so the practical trade-off for most downstream uses such as cohort selection and phenotyping is small.","Because teacher labels come from APIs and ontologies rather than manual annotation, the pipeline can be re-run at scale on new note types or new entity types at a fraction of the cost of building new gold-standard corpora.","External validation on MedAlign notes suggests the distilled model generalizes across health systems for medication and disease extraction, although symptom extraction remains below 0.70 F1."],"supporting_citations":[{"why":"Supplies BioBERT, the biomedical pretrained model that is distilled in the experiments.","marker":"[13]"},{"why":"Defines the knowledge-distillation framework that the paper applies to transfer LLM and ontology labels to BERT.","marker":"[18]"},{"why":"Provides the ontology-driven weak supervision baseline that motivates replacing human labels with teacher labels.","marker":"[12]"},{"why":"Prior work distilling LLMs for biomedical extraction, which this paper extends with multiple teachers and external validation.","marker":"[21]"},{"why":"Supplies the n2c2 2018 medication extraction dataset and its human annotations used for training and testing.","marker":"[23]"},{"why":"Supplies the NCBI Disease Corpus and its human annotations used for disease extraction.","marker":"[25]"},{"why":"Supplies the CORAL symptom extraction dataset used for teacher labeling and testing.","marker":"[26]"},{"why":"Supplies the MedAlign EHR notes used for external validation of the distilled models.","marker":"[27]"}],"fun_headline_variants":["Distilled BERT matches LLM teachers on clinical NER at 101x lower cost","1000x smaller BERT achieves similar clinical NER accuracy to LLMs","Clinical NER: tiny BERT 12x faster, 101x cheaper, and nearly as accurate","Distillation makes BERT rival LLMs for clinical entity extraction","Small BERT models match LLM teacher labels on clinical NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cost and latency comparison assumes that running BERT on a rented GPU at a single hourly rate is directly comparable to calling commercial LLM APIs at list token prices, so the claimed savings multiplier depends on which hardware price and which API pricing are current.","fun_headline_variants_meta":{"raw":{"variants":["Distilled BERT matches LLM teachers on clinical NER at 101x lower cost","1000x smaller BERT achieves similar clinical NER accuracy to LLMs","Clinical NER: tiny BERT 12x faster, 101x cheaper, and nearly as accurate","Distillation makes BERT rival LLMs for clinical entity extraction","Small BERT models match LLM teacher labels on clinical NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000518,"raw_usage":{"total_tokens":2635,"prompt_tokens":1195,"completion_tokens":1440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":811,"completion_tokens_details":{"reasoning_tokens":1335}},"tokens_in":811,"tokens_out":1440,"duration_ms":11274,"temperature":1.0,"reasoning_tokens":1335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:34:17.866041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the cost per note using the $28/hour for 4xH100 stated in the Methods; if the cost advantage over o1-mini falls from 101x to roughly 15x, the headline economy claim as stated does not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ontology-driven weak supervision baseline that motivates replacing human labels with teacher labels."},{"cited_title":"& Uzuner, O","cited_arxiv_id":null,"evidence_quote":"Supplies the n2c2 2018 medication extraction dataset and its human annotations used for training and testing."},{"cited_title":"I., Leaman, R","cited_arxiv_id":null,"evidence_quote":"Supplies the NCBI Disease Corpus and its human annotations used for disease extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CORAL symptom extraction dataset used for teacher labeling and testing."}],"review_version":1}