{"id":"ec835f22-352f-416c-8bb3-a9da7572fee0","arxiv_id":"2505.10717","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 3.8B clinical SLM built with pre-instruction tuning, model merging, and 2.5M synthetic instructions beats GPT-4-0125 on ICD-10 coding and approaches Llama3-8B on the extended CLUE+ benchmark.","lead":"A team from Microsoft and Essen University Hospital built small clinical language models by combining pre-instruction tuning, model merging, and synthetic instruction data, then measured them on an expanded medical benchmark. They report large gains over the Phi-3.5 base model and near-parity with an 8B model, but the evaluation sets were partly rewritten with another LLM, so the exact numbers need scrutiny.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark-label rewrites by Llama-3.3-70B are the load-bearing risk; headline gains on RRS QA and the average CLUE+ improvement depend on these modified labels, not the original test data.","rationale":"The reader's weakest assumption identifies the same concern I would raise: the CLUE+ evaluation is not a pristine, externally fixed benchmark because two of its constituent datasets were rewritten by an LLM. I examined the methodology sections and the appendix and found the engineering pipeline coherent and the ablations informative, so I do not object to the directional claim that pre-instruction tuning plus merging plus alignment helps a 3.8B model on clinical NLP. However, the paper's strongest quantitative claims are benchmark numbers, and the benchmark labels are partly synthetic. The exact-string check in A.8.2 is necessary but not sufficient to establish label fidelity, and the MedicationQA filtering in A.8.1 changes the test set size and content. The proposed original-label rerun would settle whether the rewritten labels are driving the gains. Until that check is done, the headline magnitudes should be treated as conditional on the CLUE+ v2 labels, which is exactly the reader's verdict. I therefore recommend no change to the conditional verdict.","tokens_in":24679,"tokens_out":6081,"duration_ms":65651,"concrete_test":"Re-run the two affected evaluations on the original, unmodified test sets with the same models and greedy decoding: (1) MEDIQA-RRS as originally released, findings-to-impression summarization scored with ROUGE-1 F1 under the CLUE-style summarization prompt; (2) MedicationQA with all 674 original questions and the original source-excerpt answers, scored with the same Rouge-1 F1 protocol. Compare Phi-3.5-mini, MediPhi-Instruct, and GPT-4-0125. If the RRS and MedicationQA gains over the base model, and the GPT-4 comparison, largely persist, the label-rewrite concern is settled; if the gains shrink or reverse, the headline numbers must be restated as specific to the rewritten CLUE+ labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is measured entirely on CLUE+, but CLUE+ is not the original benchmark for two of its datasets. Appendix A.8.1 rewrites MedicationQA answers with Llama-3.3-70B and drops 189 of the original 674 questions, leaving 485 reformulated pairs. Appendix A.8.2 converts MEDIQA-RRS impressions from summarization targets into LLM-generated question-answer pairs, with exact-string matching used only to confirm that the answer substrings are verbatim from the impression. That check does not ensure that the generated questions faithfully cover the original impression, that no content was dropped, or that the reformulation is unbiased across models. Because MediPhi is aligned on GPT-4o-generated synthetic instructions in the same direct-answer style, its large RRS QA gain (49.5% relative) and part of the 18.9% average alignment gain may reflect better mimicry of the rewrite style rather than better clinical capability. This is a correctness risk, not an internal inconsistency, but it is load-bearing because the headline comparisons—including the comparison to GPT-4—are numbers on this rewritten benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a modular framework for adapting a 3.8B-parameter SLM (Phi-3.5 mini) to clinical NLP tasks without using real clinical records for training. The pipeline has three stages: (1) pre-instruction tuning (PIT) of five domain experts on PubMed, clinical notes, medical coding pages, guidelines, and MedWiki; (2) model merging, first SLERP-merging each expert with the base model and then combining all experts into a unified model (MediPhi) via BreadCrumbs merging selected by an evolutionary search; and (3) alignment on MediFlow, a synthetic instruction dataset of 2.5M examples, using SFT and DPO to obtain MediPhi-Instruct. The authors introduce CLUE+, a 12-dataset benchmark extending CLUE, and report that the experts improve over the base model on several tasks (64.3% relative on SDoH, 49.5% on RRS QA, 44% on ICD-10 coding, with the latter surpassing GPT-4-0125 by 14%), and that alignment adds 18.9% average relative gain. The paper also releases the models, the MediFlow corpus, and synthetic validation sets.","tokens_in":24942,"tokens_out":4884,"duration_ms":48286,"significance":"If the results hold, this is a strong practical contribution: a 3.8B model that approaches or exceeds 8B and much larger models on a broad set of clinical tasks, built without real clinical records and with a modular, reusable pipeline. The paper's strengths include careful ablations (DAPT vs. PIT vs. Explainer, SLERP vs. no merging), transparent flagging of the fine-tuned Med42 comparison, and the release of models, the 2.5M-instruction MediFlow corpus, and synthetic validation sets that enable reproducibility of the merge selection. The CLUE+ extension itself is a useful benchmarking resource. The quantitative claims are credible relative to the base model on the datasets that were not modified, but the benchmark-label rewrites described in Appendix A.8 create a correctness risk for part of the headline results; the lack of uncertainty estimates is an additional concern.","major_comments":[{"comment":"The CLUE+ evaluation modifies two test sets with Llama-3.3-70B: MedicationQA is reduced from 674 to 485 pairs with rewritten answers, and MEDIQA-RRS impressions are converted from summarization targets into generated QA pairs. The reported headline gains on RRS QA (49.5% relative) and on MedicationQA (11.2 to 19.3), as well as part of the 18.9% average alignment gain, are computed on these reformulated labels. The verification for RRS QA only uses exact-string matching on answer substrings, which does not establish that the generated questions faithfully cover the impression or that the reformulation is unbiased across models. Because MediPhi-Instruct is aligned on GPT-4o-generated QA-style synthetic instructions, the measured gains may partly reflect better mimicry of the rewrite style rather than improved clinical capability. The paper should report results on the original task formats—e.g., MEDIQA-RRS summarization with the original impressions scored by ROUGE, and MedicationQA on all 674 original pairs with a reference-based metric—or provide human validation of the reformulations and show that the rankings are stable.","section":"Appendix A.8.1 and A.8.2"},{"comment":"All reported accuracies are single-run point estimates with no error bars, confidence intervals, or significance tests. Several decisive-looking differences are small in absolute terms (e.g., MediPhi-SFT 43.0 vs. MediPhi-Instruct 43.4; Task-Arithmetic 39.4 vs. Ties 39.3; MediPhi 39.3 vs. MedWiki 39.7), and without variance estimates it is not possible to judge whether the reported relative improvements (64.3%, 49.5%, 44%, 18.9%) are stable. I ask the authors to report multiple seeds (at least 3) for the main model comparisons, or at minimum bootstrap confidence intervals over the benchmark datasets, and to indicate which differences are statistically reliable.","section":"Tables 2, 3, 4, and 5"},{"comment":"The selection of the merge operator, the evolutionary search budget, and the SLERP proportions is performed on GPT-4o-generated synthetic validation sets that are explicitly designed to be consistent with the CLUE+ tasks, and the MediFlow alignment data covers the same task types and output formats as CLUE+. While this is not test-set fitting, it means the reported CLUE+ numbers are not a fully independent held-out evaluation: the pipeline is tuned on benchmark-aligned synthetic data. The claim of 'no task-specific fine-tuning' should be qualified to state that merge weights, operator choice, and data filters were selected using synthetic sets aligned with the evaluation distribution. An evaluation on an external clinical benchmark would substantially strengthen the generality claim.","section":"Section 3.1.4 and Appendix A.3"}],"minor_comments":[{"comment":"The contribution bullet claims the models and datasets are released under a 'commercially permissive license,' but the footnote for this claim reads 'TBD.' This is internally inconsistent and should be resolved before publication.","section":"Introduction, footnote 2"},{"comment":"The rows for 'Phi3.5 mini +SFT 800K' and '+DPO' both report 42.2. If the DPO row means SFT followed by DPO, the notation is unclear; please label the training stages explicitly and verify that the duplicated value is not a typo.","section":"Table 4"},{"comment":"The Med42 comparison is fine-tuned on the ACI-Bench training set, which the authors mark with an asterisk. This is good transparency, but the main text states that 'MediPhi surpasses Med42 in relative percentages on four tasks' and 'outperforms LLaMA3 on four key datasets' without repeating the caveat; the limitation of the Med42 comparison should be stated in the results discussion itself, not only in the table footnote.","section":"Table 5 and surrounding text"},{"comment":"The legend labels 'GPT-4o-0806' and 'GPT-4-0125' are inconsistent with the abstract and Section 4.2, which mention GPT-4-0125; please verify the model versions and use one consistent naming convention.","section":"Figures 4 and 5"},{"comment":"When stating that easy and medium difficulty ICD10CM questions were excluded, the paper should report the number of remaining test examples and confirm that the same subset is used for the GPT-4 comparison.","section":"Section 4.2"},{"comment":"In Table 1, the PubMed group lists both PMC full articles (3.8M docs) and PMC abstracts (36M docs). Clarify whether the full articles include the abstracts, to avoid apparent double counting of token totals.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a potentially valuable contribution, but the central quantitative claims are partly anchored to LLM-rewritten test labels. The fix is straightforward—report original-format results or human validation—and the rest of the paper is solid enough to be salvageable. I would also flag that the 'commercially permissive license' claim currently contradicts the 'TBD' footnote, which matters for the release promise. The benchmark-aligned synthetic validation is a methodological concern worth stating explicitly in the paper, but I do not see it as a fatal circularity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. My take: it's a competent engineering integration with genuinely useful new artifacts—MediFlow (2.5M synthetic clinical instructions), CLUE+ (12 datasets), and the extension of pre-instruction tuning beyond QA to NER, relation extraction, and summarization. The ablation comparing DAPT, textbook-style data, and PIT on ICD10CM is clear and informative, and the fact that a 3.8B model beats GPT-4-0125 on ICD10CM coding is a real result worth paying attention to. The modular pipeline (PIT experts, SLERP merge with base, multi-expert merge, then SFT+DPO) is described in enough detail that someone could reproduce it, and the authors are upfront about compute and limitations.\n\nThe soft spots are real but localized. The biggest is the modification of two test sets. Appendix A.8.1 rewrites MedicationQA answers with Llama-3.3-70B and drops 189 questions; A.8.2 converts MEDIQA-RRS impressions into LLM-generated QA pairs, with only an exact-string check that answers are verbatim substrings. That check does not validate the questions. Since MediPhi is trained on GPT-4o-generated direct-answer instructions, the large relative gains on RRS QA (49.5%) and part of the 18.9% average alignment gain may reflect style alignment rather than clinical capability. This doesn't sink the paper—the ICD10CM result and the SDoH gain are on cleaner benchmarks—but it means the headline numbers on the rewritten datasets should be presented as a separate, provisional category, and ideally re-run on the original labels or at least with a human-validated subset.\n\nAlso minor: no error bars or significance tests, and the Med42 comparison is contaminated because Med42 was fine-tuned on ACI-Bench, which is in CLUE+; the asterisk is there, but the number still appears in the main comparison. The merge selection uses synthetic validation sets designed to mimic the benchmark tasks—disclosed, and not test-set fitting, but it's a form of indirect optimization worth acknowledging more explicitly.\n\nNet: I'd send it to peer review. The resources are valuable, the recipe is plausible, and the core result on ICD10CM stands. The revision should make the modified benchmark results clearly separable and add robustness checks on unmodified data.","headline":"Solid modular recipe for clinical SLMs with useful new resources, but headline gains on two rewritten benchmark datasets need to be re-examined before the magnitudes are taken at face value.","tokens_in":25503,"tokens_out":3040,"would_cite":true,"duration_ms":30331,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3.8B clinical language model built from merged experts and synthetic alignment matches much larger medical models and beats GPT-4-0125 on ICD-10 coding.","keywords":["clinical NLP","small language models","pre-instruction tuning","model merging","synthetic instruction data","direct preference optimization","ICD-10 coding","CLUE+ benchmark"],"falsifier":"Re-run the CLUE+ evaluation on the original, unmodified MedicationQA answers and MEDIQA-RRS impressions; if the 18.9% average gain or the GPT-4 comparison shrinks materially, the reported improvements depend on the reformulation step rather than on the model itself.","tokens_in":24512,"feed_emoji":"🩺","tokens_out":7418,"duration_ms":60770,"temperature":0.7,"pith_summary":"The paper claims that a 3.8B-parameter model, MediPhi-Instruct, can reach near-parity with 8B models and beat GPT-4-0125 on ICD-10 coding while using no real clinical records for training. It builds this from three steps: pre-instruction tuning of five experts on permissively licensed medical corpora, merging those experts back into one model, and aligning the merged model on MediFlow, a synthetic dataset of 2.5 million instructions. The authors report relative gains over the base model of 64.3% on medical entities, 49.5% on radiology reports, and 44% on ICD-10 coding, with a further 18.9% average gain from alignment. The paper also contributes CLUE+, a doubled clinical benchmark, and the MediFlow dataset. If the results hold, small models become a practical route to deployable clinical NLP without touching sensitive patient data.","feed_headline":"Merged 3.8B clinical AI beats GPT-4 on ICD-10 coding","feed_subtitle":"Pre-instruction tuning, expert merging, and synthetic instruction data lift a small model 18.9% on clinical tasks.","key_machinery":"The central machinery is pre-instruction tuning (PIT) — a two-phase continual pre-training that first fine-tunes on GPT-4o-generated task outputs for each document (summaries, QA, entities, relations) and then on the concatenation of those task data with the original corpus — followed by model merging and synthetic-data alignment. Each of five experts is trained on one corpus group (PubMed, Clinical, MedCode, Guidelines, MedWiki), then merged back with the base model via SLERP to restore general skills lost to catastrophic forgetting, and finally all experts are unified with BreadCrumbs merging guided by an evolutionary search over synthetic validation sets. The aligned model is produced by SFT on the 2.5M-instruction MediFlow set (quality-filtered to 800K) and then DPO on 130K triplets with GPT-4o-generated marginally wrong rejected responses. This stack converts domain corpora into a single small model without using real clinical records.","core_discovery":"The paper claims to construct a 3.8B clinical language model, MediPhi-Instruct, whose modular pipeline — pre-instruction tuning of five domain experts, model merging, and SFT plus DPO alignment on synthetic instructions — yields relative gains of 18.9% over its base model on CLUE+, a new 12-dataset clinical benchmark it introduces. The merged experts already improve over the base without task-specific fine-tuning by 64.3% on medical entities, 49.5% on radiology reports, and 44% on ICD-10 coding, with the coding expert surpassing GPT-4-0125 by 14% relative. After alignment, the model clips 3.8B parameters within 1% of Llama-3-8B on CLUE+ while beating it on four datasets, and exceeds Llama3-Med42-8B on four clinical tasks. The authors treat these results as evidence that domain adaptation for small models can be achieved modularly, with permissively licensed data and synthetic tasks replacing sensitive clinical records.","pith_inferences":["Re-running CLUE+ on the original, untouched MedicationQA answers and MEDIQA-RRS impressions would test whether the 18.9% gain is a property of the model or of the answer reformulation step.","The modular recipe — pre-instruction tuning on permissively licensed corpora, merging, and DPO on synthetic tasks — looks transplantable to other data-scarce, license-constrained domains such as legal document analysis, where an instruction-following teacher can synthesize tasks from public corpora.","Because MediFlow spans 98 document types but only 14 task types, its unused formats such as nursing notes, laboratory reports, and consent forms are natural next evaluation targets the paper has not measured."],"forward_implications":["A single 3.8B model, MediPhi-Instruct, approaches the CLUE+ average of an 8B Llama-3 model while using roughly half the parameters.","The pipeline needs no real clinical records: permissively licensed corpora plus a synthetic instruction dataset suffice for the reported gains.","Merging experts back with the base model after pre-instruction tuning recovers general abilities that continual pre-training degrades.","Specialized expert models, such as the ICD-10 coding expert, already surpass GPT-4-0125 without any task-specific fine-tuning.","Alignment on a quality-filtered subset of synthetic instructions (800K of 2.5M) beats using the full set, supporting data quality over quantity."],"supporting_citations":[{"why":"Provides pre-instruction tuning, the continual pre-training method the paper extends beyond QA to summarization, NER, and relation extraction.","marker":"(Jiang et al., 2024)"},{"why":"Its SLERP-based merging of a medical model with the base establishes the expert-base merge recipe used here.","marker":"(Labrak et al., 2024)"},{"why":"The CLUE benchmark and evaluation configuration that CLUE+ extends to 12 datasets.","marker":"(Dada et al., 2024)"},{"why":"MergeKit supplies the evolutionary merge optimization used to unify the five experts.","marker":"(Goddard et al., 2024)"},{"why":"BreadCrumbs is the multi-expert merging technique selected to form MediPhi.","marker":"(Davari and Belilovsky, 2024)"},{"why":"Source of MedicationQA, one of the two datasets whose answers were reformulated for CLUE+.","marker":"(Ben Abacha et al., 2019)"},{"why":"Source of MEDIQA-RRS radiology reports, reformulated into QA pairs for CLUE+.","marker":"(Ben Abacha et al., 2021)"},{"why":"MedConceptsQA ICD10CM, the benchmark where the coding expert surpasses GPT-4-0125.","marker":"(Shoham and Rappoport, 2024)"},{"why":"Phi-3.5 mini, the 3.8B base model on which all experts and MediPhi are built.","marker":"(Abdin et al., 2024a)"}],"fun_headline_variants":["Modular SLM merges experts to beat GPT-4 on ICD-10 coding","Clinical SLM beats GPT-4 on ICD-10 using synthetic data and merging","Small model tops GPT-4 on ICD-10 via synthetic tuning and merging","3.8B clinical model merges experts to surpass GPT-4 on ICD-10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on a benchmark whose answers for MedicationQA and MEDIQA-RRS were rewritten by Llama-3.3-70B, with content preservation confirmed only by exact string matching for the radiology QA reformulation.","fun_headline_variants_meta":{"raw":{"variants":["Modular SLM merges experts to beat GPT-4 on ICD-10 coding","Clinical SLM beats GPT-4 on ICD-10 using synthetic data and merging","Small model tops GPT-4 on ICD-10 via synthetic tuning and merging","3.8B clinical model merges experts to surpass GPT-4 on ICD-10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001694,"raw_usage":{"total_tokens":6764,"prompt_tokens":1055,"completion_tokens":5709,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":5620}},"tokens_in":671,"tokens_out":5709,"duration_ms":35688,"temperature":1.0,"reasoning_tokens":5620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:04:50.200630+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CLUE+ evaluation on the original, unmodified MedicationQA answers and MEDIQA-RRS impressions; if the 18.9% average gain or the GPT-4 comparison shrinks materially, the reported improvements depend on the reformulation step rather than on the model itself.","supporting_citations":[],"review_version":1}