{"id":"54244e2e-46b3-40a3-9b4c-3386d4cffd4f","arxiv_id":"2505.08704","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"An ensemble of prompted GPT-4o outputs scores F1=0.95 on medical entity classification in EHR text, but only after excluding the 63 percent of gold entities that the extraction step missed.","lead":"The authors compared GPT-4o and DeepSeek-R1 for extracting medical terms from electronic health records using different prompt styles, and found that an ensemble of prompts gave the highest classification F1 of 0.95 on a single test document. The number is calculated only on entities that were first matched, because the raw entity extraction accuracy for the ensemble was 37 percent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1=0.95 is computed only on 70 matched gold entities (37% recall); end-to-end ensemble F1 is at most ~0.53, so the abstract claim is unsupported.","rationale":"The reader's weakest_assumption identifies the same load-bearing problem: Table V metrics are conditional on the matched subset. Reading the full text confirms this is the decisive issue. The paper's stated contribution is 'reliable medical entity recognition from EHRs', and the abstract compares against the i2b2/VA system's F1 of 0.924. That comparison is invalid if the 0.95 F1 covers only the 37% of gold entities that survived the cosine-similarity filter. The paper does disclose Table IV, so this is not a hidden error, but the text fails to connect the two tables and instead presents the conditional F1 as the headline result. The single-document test set (190 entities) is a secondary problem: even a correct end-to-end number would have no confidence interval and no evidence of generalizability. I do not see a reason to move away from the reader's REJECT verdict. The authors' fix would be to report standard end-to-end precision, recall, and F1 over all gold entities, using the raw model outputs, and to evaluate on a larger sample. The recommended verdict is therefore unchanged.","tokens_in":8225,"tokens_out":6518,"duration_ms":61113,"concrete_test":"Recompute end-to-end NER metrics for the GPT-4o ensemble run from the raw prediction logs: count a true positive only when a predicted entity matches a gold entity (cosine similarity >= 0.92) and receives the correct label; treat all unmatched gold entities as false negatives and all unmatched predicted entities as false positives. Report precision, recall, and F1 over the full 190-entity gold set. If those numbers are approximately 0.93/0.37/0.53 instead of 0.92/0.98/0.95, the abstract's central claim is falsified. Apply the same recomputation to every row of Table V.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that GPT-4o with the prompt ensemble achieves F1=0.95 and recall=0.98 for medical entity recognition is not supported by its own tables. Table IV shows the ensemble predicted 75 entities, matched 70 of 190 gold entities, and achieved extraction accuracy 0.37. Table V reports precision=0.92, recall=0.98, F1=0.95 for the same run. These numbers can only be reconciled if the classification metrics are computed over the 70 matched entities after cosine-similarity filtering at threshold 0.92. Under standard NER evaluation, the 120 unmatched gold mentions are false negatives and unmatched predictions are false positives. Even granting perfect classification of all 70 matched entities, precision is at most 70/75 = 0.933, recall is 70/190 = 0.368, and F1 is about 0.528, not 0.95. The paper even concedes this tension: 'Although entity extraction performance is low, the classification F1-score surpasses the 0.924 achieved by the i2b2/VA concept extraction system' — a conditional classification number is being compared with an end-to-end extraction benchmark. The abstract's headline claim is therefore an artifact of the matching threshold, not a measure of the full recognition pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a prompt-ensemble method for medical named entity recognition (NER) from EHRs, combining zero-shot, few-shot (document, sentence, and entity levels), and a majority-voting ensemble over four prompting configurations. The authors evaluate GPT-4o and DeepSeek-R1 on a single test document from the 2010 i2b2/VA dataset, reporting entity extraction accuracy (Table IV) and classification precision/recall/F1 (Table V). The abstract claims that GPT-4o with the prompt ensemble achieves an F1-score of 0.95 and recall of 0.98, outperforming DeepSeek-R1 and surpassing the i2b2/VA system's F1 of 0.924.","tokens_in":8485,"tokens_out":5040,"duration_ms":47210,"significance":"If the reported F1 of 0.95 represented end-to-end NER performance, the paper would demonstrate a practical and flexible LLM-based alternative to specialized clinical NER systems, with the prompt ensemble providing a novel aggregation mechanism. The use of a public benchmark (i2b2/VA) is a strength, as is the explicit pseudocode for the ensemble and the attempt to mitigate hallucination via an 'unknown' label. However, the central evaluation is fundamentally flawed: the classification metrics are computed only on entities that survive a data-dependent cosine-similarity threshold, so the headline numbers do not measure full extraction and classification. The paper's own acknowledgment that 'entity extraction performance is low' (Section IV) is in direct tension with the abstract's unqualified claim, and the comparison with the i2b2/VA end-to-end baseline is not valid. As a result, the claimed significance is not supported by the evidence in the manuscript.","major_comments":[{"comment":"The abstract's claim of F1=0.95 and recall=0.98 is unsupported by the paper's own tables. Table IV reports that the GPT-4o ensemble predicted 75 entities, matched only 70 of 190 gold entities, and achieved an extraction accuracy of 0.37. Table V reports precision=0.92, recall=0.98, and F1=0.95 for the same configuration. These numbers can only be reconciled if precision, recall, and F1 are computed exclusively on the 70 matched entities. Under standard NER evaluation, the 120 unmatched gold mentions are false negatives, giving recall at most 70/190≈0.37 and F1 at most approximately 0.53 (assuming 75 predictions and 70 true positives). The headline F1 is therefore conditional on the matching filter and does not measure the complete extraction-plus-classification pipeline, which is the task claimed in the abstract and introduction.","section":"Section IV, Tables IV and V"},{"comment":"The cosine similarity threshold τ=0.92 is described as 'empirically chosen' in Section IV. This threshold directly determines which predicted entities are included in the classification evaluation, because entities whose similarity to a gold mention falls below τ are not scored. The reported F1 is thus not a model performance measure independent of the evaluation pipeline; it is partly an artifact of a fitted filter. The authors provide no sensitivity analysis (e.g., τ=0.90 or 0.95), no held-out validation for the threshold, and no justification for choosing 0.92 beyond post hoc matching behavior. This circularity undermines the central claim that the ensemble 'achieved' an F1 of 0.95.","section":"Section IV, first paragraph; Algorithm 1"},{"comment":"The comparison of the ensemble F1=0.95 with the i2b2/VA system's F1=0.924 is invalid. The i2b2/VA F1 is an end-to-end concept extraction and classification measure computed over all gold entities in the benchmark, whereas the reported 0.95 is computed only on the matched subset of 70 entities. These are different quantities, and the paper even concedes this tension in the sentence 'Although entity extraction performance is low, the classification F1-score surpasses the 0.924 achieved by the i2b2/VA concept extraction system.' A conditional classification score on a small filtered subset cannot be compared with an end-to-end benchmark score, and the abstract's 'surpassing' claim is therefore not supported.","section":"Section IV, discussion of Table V"},{"comment":"The entire evaluation is performed on a single test document containing 190 gold entities. This sample is far too small to support the strong comparative claims made in the paper, such as GPT-4o consistently outperforming DeepSeek-R1 and the rank ordering of prompting strategies. No confidence intervals, significance tests, or error bars are provided, and no analysis of variance across documents is offered. The single-document design compounds the metric-definition problem and makes the headline numbers unreliable even as descriptive statistics.","section":"Section III-A, Table II"}],"minor_comments":[{"comment":"The handling of predictions labeled 'unknown' (those clusters with fewer than two votes) is not specified in the precision/recall computation. The paper should state clearly whether 'unknown' counts as an incorrect label, is excluded from the denominator, or is treated as a separate category, because this materially affects the reported metrics in Table V.","section":"Algorithm 1 and Table V"},{"comment":"The prompt template is described at a high level but the exact prompts used for each few-shot configuration and for the ensemble are not included. Providing the full prompts (or an appendix) would be necessary for reproducibility, especially since the paper positions prompt design as a core contribution.","section":"Section III-B, Figure 2"},{"comment":"The sentence 'Shen et al. evaluated ChatGPT...' cites reference [13], but reference [13] is the Huang et al. paper. Please verify the intended citation for the Shen et al. work.","section":"Section II-C, paragraph 2"},{"comment":"The corpus name is typeset as 'i2b2/V A' with a stray space in multiple places (e.g., Section II-A, Section III-A, Table IV context); it should be 'i2b2/VA'.","section":"Throughout"},{"comment":"The phrase 'top p value of 1' is ambiguous; specify whether this refers to nucleus sampling with p=1.0 or to another parameter, and clarify how this interacts with temperature=0.2 in the API configuration.","section":"Section III-B, paragraph 1"}],"recommendation":"reject","confidential_remarks":"This manuscript has been accepted at IEEE IRI, but the evaluation methodology contains a load-bearing error: the headline F1 is computed on a filtered subset of matched entities, not on the full test set, and the threshold is empirically chosen in a way that is circular with the metric. A corrected re-evaluation using standard end-to-end NER metrics would likely change the conclusions qualitatively (estimated F1 around 0.53 instead of 0.95). The single-document test set is also far too small for the comparative claims. These issues are not minor presentation fixes; they require a redesign of the evaluation, so rejection seems appropriate for the current form. If the authors are willing to substantially rework the evaluation and temper the claims, a new submission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about arXiv:2505.08704 is that the headline F1 of 0.95 is not an end-to-end entity recognition score. It is classification accuracy computed only on the 70 entities that survived a cosine-similarity filter at threshold 0.92. Table IV shows the ensemble matched only 70 of 190 gold entities (37% extraction recall). If you compute standard NER metrics over the full gold set, the ensemble's F1 is about 0.53, not 0.95.\n\nWhat is genuinely new here is a concrete test of a prompt ensemble—three few-shot variants whose outputs are aligned via ClinicalBERT embeddings and resolved by majority voting—on two current LLMs, GPT-4o and DeepSeek-R1, on the 2010 i2b2/VA challenge. I don't know of another paper that runs exactly this combination on these models. The writing is clear, the authors separate extraction from classification, and they report timing and token-limit problems, which is useful practical detail.\n\nThe soft spot is the evaluation. The 0.95 F1 and 0.98 recall are conditional on the matched subset, and the paper itself concedes 'entity extraction performance is low.' The comparison with the i2b2/VA system's 0.924 F1 is invalid because that is an end-to-end extraction score, not a classification-only score on matched entities. On top of that, the similarity threshold is empirically chosen on the same test document, so the headline metric is partly fitted. The test set is a single document with 190 entities, so there is no confidence interval and no evidence of generality. The 'reliability' claim is also hard to swallow: the ensemble has the lowest extraction accuracy of any GPT-4o prompt (0.37, tied with zero-shot), meaning it reduces coverage while improving consistency on a small selected subset.\n\nWho gets value from this? Someone working on prompt-based clinical NER might read it as a reminder of the difference between extraction and classification, and as a worked example of embedding-based alignment. But the abstract oversells the result, and the evaluation needs rework. I would not cite it in its current form.\n\nI'd send it to a serious referee rather than desk reject, because the experiment is checkable and the flaws are fixable. But the referee should be told that the F1 metric needs to be recomputed end-to-end, the dataset split expanded, and the claims scaled back. As it stands, the central claim is unsupported.","headline":"The F1=0.95 headline is conditional on a matched subset covering only 37% of gold entities; end-to-end ensemble F1 is about 0.53.","tokens_in":9048,"tokens_out":3982,"would_cite":false,"duration_ms":37831,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a prompt ensemble—four prompts aggregated by embedding similarity and majority voting—lets GPT-4o classify medical entities from EHRs with F1 0.95 and recall 0.98, exceeding both DeepSeek-R1 and the 2010 challenge…","keywords":["medical entity recognition","electronic health records","prompt engineering","prompt ensemble","GPT-4o","DeepSeek-R1","ClinicalBERT embeddings","clinical natural language processing"],"falsifier":"Recompute precision, recall, and F1 over all 190 gold entities in the test document, treating each unmatched gold mention as a false negative and each ensemble output that matches no gold mention as a false positive, then compare with each single-prompt result and with 0.924. If the full-extraction F1 drops below the best single prompt or below the benchmark, the claim that the prompt ensemble delivers reliable medical entity recognition would be refuted.","tokens_in":8034,"feed_emoji":"🩺","tokens_out":7677,"duration_ms":66685,"temperature":0.7,"pith_summary":"The paper tries to show that prompt engineering can make general-purpose large language models reliable enough for medical entity recognition in electronic health records without any fine-tuning. Its central result is that GPT-4o, when asked to extract entities through four different prompts and then combining those outputs by embedding similarity and majority voting, classifies medical entities with F1 0.95 and recall 0.98, above DeepSeek-R1's best F1 of 0.94 and above the 0.924 scored by the 2010 concept-extraction challenge system. The ensemble's purpose is reliability: requiring agreement across prompt formats filters hallucinations and label noise, and an 'unknown' label lets the model abstain. The paper reports the same ensemble extracts only 70 of the test document's 190 gold entities, so the headline F1 describes classification on the matched subset.","feed_headline":"GPT-4o prompt ensemble hits 0.95 F1 on medical entities","feed_subtitle":"Four prompt formats, merged by embedding similarity and majority voting, outscore DeepSeek-R1 and the 2010 clinical benchmark.","key_machinery":"The central mechanism is the prompt ensemble: four prompts (zero-shot, one-document few-shot, 100-sentence few-shot, and 5,355-entity few-shot) each return entity–label pairs; the pairs are embedded with ClinicalBERT, greedily clustered when cosine similarity is at least 0.92, and each cluster is given the majority label if two or more prompts agree, otherwise 'unknown'. This turns several individually noisy prompt outputs into a single consensus output with an explicit abstention option, and it is the mechanism the paper credits for reducing hallucination and label noise.","core_discovery":"On the standard 2010 clinical concept extraction benchmark, the authors claim that GPT-4o with a prompt ensemble outperforms all other tested configurations: an F1 of 0.95 and recall of 0.98 for classifying entities as Problem, Test, or Treatment, surpassing DeepSeek-R1's best F1 of 0.94 and exceeding the challenge baseline's 0.924. The ensemble uses three few-shot prompts—one annotated document, 100 annotated sentences, and 5,355 annotated entity examples—plus a zero-shot prompt, embeds each extracted entity with ClinicalBERT, clusters mentions with cosine similarity at least 0.92, and assigns the majority label when at least two prompts agree, otherwise 'unknown'. The paper also reports that entity extraction coverage is low: GPT-4o's ensemble matched only 70 of 190 gold entities in the single test document, giving extraction accuracy 0.37, and the classification metrics are conditional on that matched set. It interprets the ensemble as a reliability mechanism rather than a coverage booster, with the 'unknown' abstention designed to reduce hallucination in clinical use.","pith_inferences":["My reading: the purported superiority over the 2010 challenge's 0.924 F1 would only hold as a like-for-like comparison if the baseline were also scored on matched entities; with the 120 missed gold mentions counted as false negatives, GPT-4o's effective F1 would be far below 0.95.","A direct test of the ensemble's value is to run the same four prompts on several held-out documents and report recall over all gold mentions, with and without the 0.92 matching gate; if full recall does not beat single few-shot prompts, the ensemble should be described as a classification filter rather than a recognizer.","Because the threshold 0.92 is tuned on one test document, a useful extension would sweep the threshold and report precision/recall curves, which would show whether the headline numbers are stable or an artifact of one cutoff.","The abstention label shifts errors from the model to human reviewers; deployment would need to weigh how many 'unknown' items a clinical team can manually adjudicate."],"forward_implications":["GPT-4o's ensemble F1 of 0.95 is the highest among all prompt strategies tested, supporting the paper's claim that consensus across prompt formats is more reliable than any single prompt format.","DeepSeek-R1 performs nearly as well in zero-shot (F1 0.91) and with one document (0.94), but its ensemble score drops to 0.89, so the benefit of prompt ensembling is model-specific.","GPT-4o runs faster than DeepSeek-R1 in every configuration, with the largest gap in zero-shot (8.88 seconds versus 34.88 seconds), suggesting that prompt-based clinical NER can be practical in time-sensitive settings.","Entity extraction coverage did not improve with ensembling: GPT-4o's ensemble matched 70 of 190 gold entities, the same extraction accuracy as zero-shot (0.37), meaning the ensemble's gain is in classification reliability, not recall of mentions."],"supporting_citations":[{"why":"Provides the benchmark dataset and the 0.924 F1 baseline the paper claims to surpass.","marker":"[14]"},{"why":"Supplies the ClinicalBERT embeddings used to align entity mentions across prompt outputs and to apply the cosine-similarity threshold.","marker":"[21]"},{"why":"Identifies GPT-4o, the model whose prompt ensemble produces the headline results.","marker":"[15]"},{"why":"Identifies DeepSeek-R1, the comparison model evaluated under the same prompt strategies.","marker":"[16]"},{"why":"Shows that structured prompts let GPT-4 approach specialized clinical models, motivating the prompt-engineering approach.","marker":"[5]"},{"why":"Documents LLM hallucination in clinical note extraction, supporting the design of the 'unknown' abstention label.","marker":"[13]"}],"fun_headline_variants":["GPT-4o ensemble hits 0.95 F1 but on only 70 of 190 entities","0.95 F1 on matched entities: GPT-4o prompt ensemble abstains on the rest","Prompt ensemble gives GPT-4o high precision but low coverage on clinical NER","GPT-4o ensemble: 0.95 F1 yet matches only 37% of gold entities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that it is fair to grade classification only on the 70 gold entities whose extracted mentions cleared the 0.92 similarity cutoff, ignoring the other 120 gold entities the ensemble missed; if those misses were counted as errors, the reported F1 and recall would drop.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o ensemble hits 0.95 F1 but on only 70 of 190 entities","0.95 F1 on matched entities: GPT-4o prompt ensemble abstains on the rest","Prompt ensemble gives GPT-4o high precision but low coverage on clinical NER","GPT-4o ensemble: 0.95 F1 yet matches only 37% of gold entities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001235,"raw_usage":{"total_tokens":5055,"prompt_tokens":912,"completion_tokens":4143,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":4042}},"tokens_in":528,"tokens_out":4143,"duration_ms":26116,"temperature":1.0,"reasoning_tokens":4042,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:47:20.539226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute precision, recall, and F1 over all 190 gold entities in the test document, treating each unmatched gold mention as a false negative and each ensemble output that matches no gold mention as a false positive, then compare with each single-prompt result and with 0.924. If the full-extraction F1 drops below the best single prompt or below the benchmark, the claim that the prompt ensemble delivers reliable medical entity recognition would be refuted.","supporting_citations":[{"cited_title":"2010 i2b2/va challenge on concepts, assertions, and relations in clinical text,","cited_arxiv_id":null,"evidence_quote":"Provides the benchmark dataset and the 0.924 F1 baseline the paper claims to surpass."},{"cited_title":"Improving large language models for clinical named entity recognition via prompt engineering,","cited_arxiv_id":null,"evidence_quote":"Shows that structured prompts let GPT-4 approach specialized clinical models, motivating the prompt-engineering approach."},{"cited_title":"A critical assessment of using chatgpt for extracting structured data from clinical notes,","cited_arxiv_id":null,"evidence_quote":"Documents LLM hallucination in clinical note extraction, supporting the design of the 'unknown' abstention label."}],"review_version":1}