{"id":"ba2f9a71-481f-44a8-9b46-f768c776cc65","arxiv_id":"2505.17265","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CaseReportBench tests LLMs on dense information extraction from 138 rare-disease case reports and reports that Qwen2.5-7B outperforms GPT-4o under string-based metrics.","lead":"This paper introduces CaseReportBench, a dataset of 138 expert-annotated clinical case reports for testing how well LLMs extract structured medical information. It evaluates five LLMs and finds that a 7-billion-parameter open-source model extracts information more faithfully than GPT-4o under the paper's metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Qwen-vs-GPT ranking rests on token-overlap metrics that explicitly penalize paraphrase; without a semantic or clinician-grounded comparison of those two models, the headline claim may be a metric artifact.","rationale":"The reader's weakest assumption correctly identifies the load-bearing risk: string-based token similarity may not capture clinically meaningful extraction quality, and the paper's own discussion concedes that GPT-4o is penalized for synonyms and rewording. The clinician evaluation in Section 5.1 is too narrow to rescue the ranking because it evaluates only Llama3-8B and uses the same clinicians who constructed the gold standard. The dataset, annotation effort, and open-source pipeline are genuine contributions, and the paper is internally consistent; the problem is external validity of the metric used for the central comparative claim. A semantic or blinded clinician evaluation comparing Qwen2.5-7B and GPT-4o directly would settle whether the ranking reflects extraction fidelity or merely format/paraphrase adherence. Until then, the claim should remain conditional, as the reader concluded.","tokens_in":15504,"tokens_out":3353,"duration_ms":42641,"concrete_test":"Run a blinded clinician study: for 30 randomly selected CaseReportBench cases, present the FS-FCSP outputs of Qwen2.5-7B and GPT-4o to two clinicians (not the gold annotators) and score completeness, accuracy, and actionability with the Section 5.1 rubric. Also report BERTScore or MedSTS per category. If GPT-4o equals or exceeds Qwen on clinical relevance or semantic similarity, the headline ranking is a metric artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline finding—Qwen2.5-7B outperforms GPT-4o for dense clinical extraction—is established exclusively through string-overlap metrics (TSR, Levenshtein, EM; Tables 2–3). The authors themselves state that GPT-4o's lower scores are 'likely due to the stringent evaluation metrics used, which penalize minor variations (e.g., synonyms, rewording)' (Section 5). The clinician evaluation that could adjudicate clinical equivalence was run only on Llama3-8B outputs (Section 5.1), not on the Qwen2.5-7B vs. GPT-4o comparison. Thus the ranking conflates instruction-following and verbatim copying with extraction quality. This is not a minor caveat: Table 5 shows two expert annotators agree at only 74.65 average TSR, so the gold standard is itself one of several acceptable extractions; a model that picks different but clinically equivalent wording is heavily discounted. Additional extractions by GPT-4o are counted as 'hallucination' if absent from the gold set, even though the annotation reconciliation (Appendix B/C) shows experts disagreed on what belongs in categories. The 'open-source beats GPT-4o' claim therefore needs a metric or clinician check that does not penalize semantic equivalence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CaseReportBench, an expert-annotated benchmark for dense information extraction from 138 clinical case reports focused on inborn errors of metabolism, using 14 structured clinical categories. The authors evaluate five LLMs (Qwen2-7B, Qwen2.5-7B, Qwen2.5-32B, Llama3-8B, and GPT-4o) under three data integration methods (FCSP, UCP, UGP) and several prompting strategies, and report that Qwen2.5-7B outperforms GPT-4o on string-based metrics. They also present a clinician evaluation of Llama3-8B outputs. The dataset and code are publicly released.","tokens_in":15924,"tokens_out":2845,"duration_ms":22029,"significance":"If the central claim is upheld, the benchmark would be a useful resource for evaluating clinical dense extraction, and the finding that a compact open-weight model outperforms GPT-4o would be notable for practical medical NLP. The paper has concrete strengths: a publicly released, expert-annotated dataset with a documented reconciliation pipeline; systematic comparisons of prompting and data integration strategies; and a clinician assessment that goes beyond automatic metrics. The main weakness is that the headline model ranking is established exclusively through token-overlap metrics, which the authors themselves acknowledge penalize clinically valid paraphrases, and the clinician evaluation was not applied to the models being compared in that ranking.","major_comments":[{"comment":"The headline claim that Qwen2.5-7B outperforms GPT-4o is supported only by TSR, Levenshtein, exact match, BLEU, and ROUGE. These metrics penalize synonyms and rewording, as the paper concedes in Section 5: GPT-4o's lower performance is 'likely due to the stringent evaluation metrics used, which penalize minor variations (e.g., synonyms, rewording).' The clinician evaluation in Section 5.1 was run only on Llama3-8B outputs, not on Qwen2.5-7B versus GPT-4o, so there is no human or semantic check on the comparison that drives the abstract's main claim. Without a metric that credits semantically equivalent extractions, or a clinician adjudication of the two models' outputs, the ranking may reflect formatting fidelity rather than clinical extraction quality. The authors should either add such an evaluation or substantially soften the comparative claim.","section":"Section 5, Tables 2 and 3"},{"comment":"Inter-annotator agreement is low in key categories: pairwise F1 is 0.18 for ENDO, 0.33 for GU, 0.33 for Vitals Hema, and 0.36 for MSK, and the average TSR between the two specialists is only 74.65, with History at 42.50. This means the gold standard is one of several acceptable extractions, and a model that chooses a different but clinically equivalent wording is heavily discounted. The evaluation should report model performance against each annotator's original annotations as a robustness check, and the discussion should explicitly address how the reconciled gold standard interacts with the string-based metrics.","section":"Appendix C, Table 7 and Appendix B, Table 5"},{"comment":"All results appear to come from a single run per configuration, with no error bars, confidence intervals, or significance tests. Differences among the top configurations (e.g., qwen2.5:7b FS FCSP TSR 56.377 versus qwen2.5:32b FS FCSP 51.467 versus llama3:8b FS FCSP 50.351) may be within run-to-run noise, especially for greedy or low-temperature decoding with quantized local models. At minimum, the authors should report multiple runs or bootstrap resampling to establish that the ordering is stable.","section":"Section 5, Tables 2 and 3"},{"comment":"Hallucination is defined as the percentage of LLM-extracted information not present in the benchmark set. Given the low inter-annotator agreement documented in Appendix C, many non-gold extractions may be clinically valid details that the annotators simply did not include or assigned to a different category. The hallucination metric therefore conflates clinically false content with format deviation and missing from gold. The paper should distinguish these cases, for example by having clinicians review a sample of non-matching extractions for the models being compared, rather than labeling all non-gold content as hallucination.","section":"Section 5, hallucination definition"}],"minor_comments":[{"comment":"The figure caption says 'a single combined prompt for all 13 categories,' while the text describes 14 clinical categories; please make the count consistent.","section":"Figure 3C"},{"comment":"The model identifier 'qwen2.532b' appears without a colon in Table 3, unlike the other Qwen entries; please standardize the notation.","section":"Section 5 and Table 3"},{"comment":"The text contains the typo 'textbffactual precision' and should read 'textual precision' or similar.","section":"Section 5, 'Open-Source Models and GPT-4o' paragraph"},{"comment":"The abbreviation is introduced as FCSP, but the text later uses 'FSCSP' in the clinician evaluation section and in the conclusions; please use one consistent abbreviation.","section":"Section 5 and throughout"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a useful contribution and the paper is within scope for CHIL. The main issue is that the central comparative claim requires a human or semantic validation step that is currently missing. I would not reject the paper, but the revision should either add that validation or reframe the claim as a token-level alignment result rather than a general claim about clinical extraction quality."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this one if you work on clinical NLP benchmarks. It ships a real artifact: CaseReportBench, 138 rare-disease case reports from PMC, expert-annotated across 14 clinical categories, with code and data on GitHub/HF. That alone is worth your time. The paper also compares five LLMs under three prompting strategies, and the authors are upfront that string-based metrics like TSR and EM penalize synonyms and rewording. The trouble is their central claim — Qwen2.5-7B outperforms GPT-4o — is built entirely on those metrics. GPT-4o paraphrases; TSR counts that as error. The paper calls GPT-4o's extractions 'hallucinations' when they are simply absent from the gold set, even though the appendix shows expert annotators themselves disagreed on category boundaries (pairwise F1 of 0.18 for ENDO, 0.33 for GU). The clinician evaluation that could adjudicate clinical equivalence was run only on Llama3-8B outputs, not on Qwen vs GPT-4o. So the ranking is a metric artifact risk, and the authors' own discussion concedes as much. That said, the benchmark construction is careful: IEMbase-based selection, reconciliation of low-TSR disagreements, public annotation guidelines, and reproducible code. The efficiency finding — FCSP cuts compute 6.2% with comparable scores — is plausible and useful. CoT adding little is a reasonable negative result. But the paper needs a semantic similarity or clinician-grounded comparison, at minimum on the Qwen vs GPT-4o outputs, plus error bars or significance testing across runs (only single runs reported). Minor: LYMPH has only 1.45% coverage, so claims there are uninformative. Overall: worth serious peer review, not desk reject, but the headline claim should be softened or re-evaluated before publication.","headline":"Valuable new benchmark, but the headline Qwen-beats-GPT-4o claim rests on token-overlap metrics the paper itself admits penalize paraphrase, so it needs a semantic or clinician-grounded comparison before being believed.","tokens_in":16281,"tokens_out":2647,"would_cite":true,"duration_ms":18733,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CaseReportBench shows that a compact open-weight model, Qwen2.5-7B, extracts structured clinical case-report information more faithfully than GPT-4o.","keywords":["clinical dense information extraction","case reports","inborn errors of metabolism","large language models","benchmark dataset","prompting strategies","token set ratio","rare disease diagnosis"],"falsifier":"Run the same extraction task with a clinician-graded semantic metric, or have clinicians rate whether GPT-4o's paraphrased outputs contain the same clinical facts as the gold annotations; if GPT-4o's facts are judged clinically equivalent, the token-similarity ranking collapses.","tokens_in":15359,"feed_emoji":"🩺","tokens_out":4562,"duration_ms":40358,"temperature":0.7,"pith_summary":"This paper introduces CaseReportBench, an expert-annotated benchmark of 138 clinical case reports on inborn errors of metabolism, organized into 14 clinical categories, and uses it to evaluate how well large language models turn case-report narratives into structured medical information. The central claim is that a compact open-weight model, Qwen2.5-7B, extracts this information more faithfully than GPT-4o when scored by token-level similarity, and that category-specific prompting improves alignment with expert annotations. If true, this matters because case reports are a largely untapped resource for rare-disease diagnosis, and a small, controllable model that follows instructions could make large-scale structured extraction practical without relying on proprietary APIs. The paper also reports that zero-shot chain-of-thought adds little, that unified global prompting fails to follow the required output schema, and that clinician review finds the extracted outputs usable but incomplete on negative findings.","feed_headline":"A 7B open-source model beats GPT-4o on clinical case-report extraction","feed_subtitle":"On a new expert-annotated benchmark for rare-disease case reports, a small model wins on token-level fidelity.","key_machinery":"The load-bearing machinery is CaseReportBench itself: 138 case reports curated from open-access PubMed Central, annotated by two rare-disease specialists into 14 categories such as History, Lab Image, Neurology, and Pregnancy, with disagreements reconciled by a third reviewer. The primary evaluation metric is Token Set Ratio (TSR), a fuzzy string-similarity score over token sets, alongside Levenshtein similarity and exact match. The prompting strategies compared are category-specific prompting (each category gets a tailored prompt) versus unified global prompting, combined with few-shot examples and with subheading-filtered text integration (FCSP) versus unfiltered full-text application (UCP). The paper's key methodological claim is that category-specific, few-shot prompting with filtered subheadings gives the best alignment with expert annotations, while a single unified prompt collapses.","core_discovery":"The paper's core claim is that for dense information extraction from clinical case reports, the open-source Qwen2.5-7B-Instruct model outperforms all tested alternatives, including GPT-4o and the larger Qwen2.5-32B, under string-based evaluation metrics. With few-shot category-specific prompting and filtered subheading data integration, Qwen2.5-7B reaches a Token Set Ratio of 56.4%, Exact Match of 38.3%, and the lowest hallucination rate among the five models, while GPT-4o scores lowest on these metrics. The authors attribute GPT-4o's weak numerical performance partly to evaluation metrics that penalize synonym use and rephrasing, and they argue that instruction fidelity and output-format compliance matter more than model size for structured clinical extraction. Clinician evaluations of Llama3-8B outputs rate readability and trustworthiness highly but note lower actionability and omissions of negative findings, suggesting that token-level accuracy does not automatically equal clinical utility.","pith_inferences":["If a clinician-grounded or semantic similarity metric replaced TSR, the model ranking might shift, since the paper notes GPT-4o's paraphrases are penalized; a clinician study with meaning-based scoring could test this.","The benchmark's category imbalance (e.g., Lymphatic System appears in only 2 of 138 reports) means per-category conclusions about rare categories are weak; rebalancing or pooling across more reports could sharpen comparisons.","The same category-specific prompting recipe may transfer to other rare-disease domains beyond inborn errors of metabolism, since the 14 categories are drawn from a standard inpatient work-up form rather than an IEM-specific ontology.","Future work could use CaseReportBench to test whether semantic evaluators, such as an LLM judge with clinical grounding, change the ranking, or whether explicitly extracting negative findings improves differential-diagnosis support."],"forward_implications":["A 7-billion-parameter open-weight model can serve as a practical extractor for structured clinical information from case reports, reducing dependence on proprietary APIs.","Category-specific prompting with few-shot examples is a transferable recipe for dense extraction tasks that require many output fields.","Unified global prompting is poorly suited to this task because models fail to follow the full output schema, suggesting that multi-category extraction should be decomposed.","Token-level similarity metrics and hallucination rate can distinguish instruction-following ability among LLMs, even when traditional fluency metrics do not.","LLM-assisted extraction can reduce manual annotation time by roughly 24 hours across 138 cases while still needing expert oversight, especially to catch negative findings."],"supporting_citations":[{"why":"Supplies the Qwen2.5 model family that includes the top-performing Qwen2.5-7B-Instruct.","marker":"(Yang et al., 2024)"},{"why":"Supplies Llama3-8B-Instruct, one of the five evaluated models and the one used in the clinician case study.","marker":"(Dubey et al., 2024)"},{"why":"Supplies GPT-4o, the proprietary model that the paper's central comparison targets.","marker":"(Islam and Moushi, 2024)"},{"why":"Supplies Qwen2-7B-Instruct, an earlier-generation model used as a baseline in the evaluation.","marker":"(Hui et al., 2024)"},{"why":"Provides the IEMbase knowledge base and ICIMD classification used to identify case reports for the benchmark.","marker":"(Ferreira et al., 2021)"},{"why":"Supplies the Prodigy annotation tool used by the two specialists to create the gold-standard annotations.","marker":"(Montani and Honnibal, 2018)"}],"fun_headline_variants":["Small 7B model beats GPT-4o on clinical extraction benchmark","Open-source 7B wins over GPT-4o in case-report data extraction","Qwen2.5-7B outperforms GPT-4o on dense clinical extraction","Benchmark shows 7B model tops GPT-4o for case-report info mining","Rare disease case reports: 7B model beats GPT-4o in extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of models rests on the assumption that string-level token similarity (TSR) captures what matters in clinical information extraction; if synonym use and rephrasing are clinically acceptable, GPT-4o's lower scores may overstate its deficiency.","fun_headline_variants_meta":{"raw":{"variants":["Small 7B model beats GPT-4o on clinical extraction benchmark","Open-source 7B wins over GPT-4o in case-report data extraction","Qwen2.5-7B outperforms GPT-4o on dense clinical extraction","Benchmark shows 7B model tops GPT-4o for case-report info mining","Rare disease case reports: 7B model beats GPT-4o in extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1542,"prompt_tokens":969,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":465}},"tokens_in":585,"tokens_out":573,"duration_ms":3962,"temperature":1.0,"reasoning_tokens":465,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:48:31.395349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same extraction task with a clinician-graded semantic metric, or have clinicians rate whether GPT-4o's paraphrased outputs contain the same clinical facts as the gold annotations; if GPT-4o's facts are judged clinically equivalent, the token-similarity ranking collapses.","supporting_citations":[],"review_version":1}