Pith. sign in

REVIEW 4 major objections 5 minor 31 references

ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning smaller models beats larger ones at clinical data extraction.

desk verdict Useful dataset and a plausible result, but the GPT-4-generated gold labels make the headline comparison unreliable without a human evaluation of outputs. read the letter →

arxiv 2502.05638 v1 pith:6YQ5JEEO submitted 2025-02-08 cs.CL cs.AI

Aynur Guluzade , Naguib Heiba , Zeyd Boukhers , Florim Hamiti , Jahid Hasan Polash , Yehya Mohamad , Carlos A Velasco This is my paper · ORCID
classification cs.CLcs.AI
keywords clinicalinformationextractionlargelanguagemodelsfine-tuningLoRAstructureddatareportspatientsummariesdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports that fine-tuning a relatively small language model on clinical reports can produce better structured information extraction than much larger models used with prompting alone. In the reported experiments, a LoRA-fine-tuned Llama 3.1 8B Instruct model outperforms Llama 3.1 405B Instruct, even when the 405B model is given detailed category definitions and in-context examples; the 8B model leads on ROUGE, BERTScore, and entity-level F1. The authors built and released a 60,000-report English dataset (plus 24,000 German translations) with annotations across 15 clinical categories, generated with GPT-4 and checked by automated and manual validation. The result matters because it suggests that accurate, structured extraction from legacy clinical text can be achieved with models small enough to run on modest hardware.

What carries the argument

The central object is the learned mapping $f_\theta$ from a clinical report $R$ to a structured representation $S$ over 15 predefined categories, trained with Low-Rank Adaptation (LoRA). LoRA freezes the original weights and adds a low-rank update, which is what makes fine-tuning feasible on a single GPU and lets a compact model absorb the task without losing general language ability. Around this mapping, the dataset-generation workflow is the second load-bearing piece: categories were defined with input from previous work and physicians, a small set was manually annotated, then GPT-4 with in-context learning produced annotations for 60,000 English reports, with manual-validation error rates recorded per category. The argument is that once the category structure is internalised by fine-tuning, the model no longer needs detailed prompts to know what belongs in each field.

What would settle it

Have independent clinicians re-annotate a random sample of the English reports and compare their labels with the released GPT-4-derived gold labels and with the fine-tuned model's output; if agreement with the gold labels is much lower than the reported manual-validation rates, or if the model's score against clinician labels does not reproduce the ranking over Llama 405B, the paper's performance comparison would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that parameter-efficient fine-tuning closes the gap between small and large language models for structured clinical information extraction. On the ELMTEX dataset, fine-tuning Llama 3.1 8B Instruct with Low-Rank Adaptation (LoRA) yields the best scores on all three evaluation families: ROUGE-1 0.7771, ROUGE-2 0.6841, ROUGE-L 0.7626, BERTScore F1 0.8253, and entity-level F1 0.7869, beating Llama 3.1 405B Instruct with advanced prompting and in-context learning (0.6969, 0.5716, 0.6714, 0.7287, 0.7359). Even the fine-tuned Llama 3.2 1B and 3B models outperform all non-fine-tuned models, despite the 1B model being prone to hallucination when only prompted. The authors interpret this as evidence that fine-tuning lets a model internalize category definitions, output formatting, and task scope, whereas even very large models must infer these from the prompt alone.

Load-bearing premise

The gold-standard labels are treated as accurate: they were produced by GPT-4 with in-context learning and manual validation, and the manual-validation error rates in Section 3.2 are up to 11% for comorbidities and 8% for diagnosis; if those labels are systematically biased, both fine-tuning and evaluation inherit the bias.

Editorial extensions

If this is right

  • Fine-tuned Llama 3.1 8B achieves the best overall performance across all metrics, surpassing all non-fine-tuned models including Llama 3.1 405B with advanced prompting.
  • Fine-tuned 1B and 3B models also beat all prompting-only approaches, suggesting that a small model on a single GPU or edge device can handle structured clinical extraction.
  • Advanced prompting with retrieved in-context examples substantially improves medium and large models over naive prompting, but none of those setups catches the fine-tuned 8B model.
  • The released ELMTEX dataset provides a 60,000-instance English benchmark (plus 24,000 German translations) for training and evaluating future clinical information extraction systems.
  • Fine-tuning smaller models reduces the hardware and cost barrier for clinical deployment, which is directly relevant to resource-limited healthcare settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the gold labels are GPT-4-derived, the fine-tuned model may be matching GPT-4's annotation style rather than an independent clinical gold standard; a clinician-annotated test subset would separate those explanations.
  • Editorial inference: the same fine-tuning advantage is likely to transfer to other structured-extraction tasks with well-defined category schemas (e.g., radiology or pathology reports), but the ELMTEX category definitions are their own schema, so domain transfer should be tested.
  • Editorial inference: a direct test of the German subset would show whether cross-lingual transfer or per-language fine-tuning is needed; the paper releases the translations but does not report German performance in the main text.
  • Editorial inference: if the pattern generalizes, procurement decisions for clinical IE could shift from renting access to very large proprietary models toward owning small fine-tuned models, with implications for data privacy and the EU AI Act the paper mentions as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ELMTEX, a dataset of 60,000 English clinical report summaries annotated with structured information over 15 categories, plus 24,000 German translations, and evaluates open-source LLMs for structured clinical information extraction. The authors compare naive prompting, advanced prompting with in-context learning, and LoRA fine-tuning on Llama 3.1 8B/70B/405B and Llama 3.2 1B/3B, using ROUGE, BERTScore, and entity-level metrics. The main claim is that the fine-tuned Llama-3.1-8B model achieves the best performance across all metrics, outperforming all non-fine-tuned models including Llama-405B, and that fine-tuning enables small models to compete with much larger ones. The dataset and code are released publicly.

Significance. The practical significance is clear: if the result holds, resource-limited clinical settings could deploy small fine-tuned models instead of massive proprietary or GPU-hungry models. The released dataset is a sizeable resource for clinical IE research, and the multi-metric evaluation covers surface, semantic, and entity-level agreement. The manual validation effort (Table 1) is a positive step, as is the public code release. However, the significance is conditional on the evaluation reference: the ground truth is GPT-4-generated, and the paper does not report an independent human evaluation or correction of validation errors. The reported comparison is therefore best interpreted as fidelity to the annotation protocol rather than a fully independent measure of clinical extraction correctness.

major comments (4)
  1. [Section 3.2 (Dataset Generation Workflow) and Table 2] The central comparison in Table 2 is not an independent measure of clinical extraction quality. Section 3.2 states that 'We then used the GPT-4 model with advanced prompting and in-context learning to generate the initial annotations,' and Table 1 reports manual-validation error rates up to 11% for comorbidities and 8% for diagnosis without stating whether these errors were corrected in the released dataset. Because the fine-tuned models are trained on these GPT-4 labels and all metrics (ROUGE, BERTScore, entity-level) compare model outputs to the same labels, the observed advantage of the fine-tuned Llama-3.1-8B over the non-fine-tuned 405B partly reflects the fine-tuned model's success at reproducing the GPT-4 annotation convention. To substantiate the claim that fine-tuned small models 'match or surpass larger counterparts' for clinical IE, the authors should add either a human evaluation of a random sample of outputs from the top-performing models or an evaluation against an independent gold standard (e.g., manually corrected annotations), and report how the validation errors in Table 1 were handled.
  2. [Section 3.1 (Evaluation) and Section 4.1 (Experimental Setup)] The test set is not described. The fine-tuning paragraph states that the training set comprises 90% of D, but the paper does not report the size of the test set, the splitting method, or whether the same test set is used in the prompting and fine-tuning evaluations. With only single point estimates in Table 2 and no variance or significance testing, the reader cannot assess whether the differences between the fine-tuned 8B and the 405B advanced-prompting condition are stable or within noise. Please report the test-set size, split protocol (including whether in-context retrieval examples are disjoint from the test set, as Eq. (2) claims), and confidence intervals or repeated-run variability for the main comparisons.
  3. [Section 4.1 (Experimental Setup)] Details needed for reproducibility are missing. The LoRA hyperparameters (rank, alpha, learning rate, number of epochs, batch size), the number of in-context examples m in Eq. (2), the identity of the retrieval model g_phi, and the quantization/decoding settings for Llama-405B are not reported. These choices can materially affect the magnitude of the fine-tuning gains that drive the central claim, so they should be specified in the main text or a supplementary table.
  4. [Section 3.2 (Entity-level evaluation) and Table 2] The entity-level evaluation relies on an external SciSpacy NER model, but the paper does not explain how the entities extracted by SciSpacy map onto the 15 custom categories, nor how precision/recall are aggregated across categories. Since SciSpacy is trained on biomedical text and does not necessarily recognize social history or lifestyle concepts in the same way as the annotation protocol, the entity-level column may reflect NER errors as much as model performance. At minimum, the aggregation procedure and the per-category entity-level results should be reported; if this is not feasible, the entity-level column should be interpreted with appropriate caveats.
minor comments (5)
  1. [Throughout] The model name is spelled inconsistently as 'LLama' in Section 4.2 and the abstract; use 'Llama' for consistency with the Llama 3 series citation.
  2. [Section 3.2] The number of manually validated samples per category is not reported; only error-rate percentages are given in Table 1. Please state the sample size for the manual validation.
  3. [Section 3.2] The German dataset is mentioned in the abstract and Section 3.2 as part of the contribution, but no German results appear in the main text; either include them or clarify that the evaluation is English-only.
  4. [Table 2] The table reports ROUGE scores without stating whether they are F1-scores or recall; please specify the ROUGE variant used.
  5. [Title] The title contains an extra space in 'Fine-T uning'; please correct.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the headline comparison is a held-out empirical evaluation, though the GPT-4-derived reference labels are a validity caveat.

full rationale

The paper's main claim—that a fine-tuned Llama-3.1-8B outperforms larger prompted models on the ELMTEX benchmark—rests on a supervised train/test comparison, not on a circular derivation. Fine-tuning minimizes cross-entropy against the reference S* (Eq. 3) on a 90% training split, while the evaluation metrics in Section 3.1 compare all systems to S* on the held-out test portion; Section 3.1 explicitly states that retrieval-based in-context examples use a training set disjoint from the test set. The reference S* was initially generated by GPT-4 (Section 3.2) and then manually validated, with Table 1 reporting per-category error rates up to 11%. This is a genuine gold-standard quality limitation: the ranking may partly reflect fidelity to GPT-4's annotation style rather than independent clinical truth, and no independent human evaluation of model outputs is reported. However, this is a benchmark-construction and validity caveat, not circularity: all systems are scored against the same fixed reference, the evaluated reports are not the ones on which the fine-tuned model was trained, and the fine-tuned model must generalize to unseen reports. The only self-citations are to the released dataset [12] and the appendix [11] containing prompts and German-language results; these are not load-bearing for the English headline result, and no uniqueness theorem or prior-work ansatz is invoked to force the conclusion. Accordingly, no step reduces to its own input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result depends on treating GPT-4 outputs as gold, on unstated modeling choices such as LoRA hyperparameters and retrieval settings, and on the corpus being representative of clinical practice. No new physical or conceptual entities are introduced.

free parameters (3)
  • LoRA hyperparameters (rank, alpha, learning rate, epochs, batch size) = not reported
    Section 4.1 only says LoRA fine-tuning was used; no values are given, so the reported model gains cannot be reconstructed or checked.
  • Number of in-context examples m = not reported
    Equation (2) defines retrieval of m examples, but m is never specified and the retrieval encoder g_phi is unnamed.
  • GPT-4 annotation generation settings = not reported
    The initial annotations were produced by GPT-4 with advanced prompting; sampling temperature, prompt details, and post-processing are not specified.
assumptions (4)
  • domain assumption The 15 categories with semicolon-delimited concepts form a valid and complete target schema for structured clinical extraction.
    Category definitions came from related work and physician consultations, but no ontology mapping or inter-annotator agreement is reported; ambiguity between categories is acknowledged in the error analysis.
  • ad hoc to paper GPT-4-generated JSON labels are an acceptable stand-in for clinical ground truth.
    These labels are used as both training targets and evaluation references; manual validation covers only a sample, and several categories retain error rates of 7-11%.
  • domain assumption The PMC-Patients subset is representative of real clinical reports.
    The corpus is derived from published PubMed Central case reports rather than electronic health records, so distribution shift to clinical practice is unmeasured.
  • domain assumption SciSpacy medical NER provides sufficiently accurate entity extraction for evaluation.
    Entity-level metrics rely on this external NER model; its detection errors are shared across systems but may systematically affect specific categories.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports." pith.science (2026). https://pith.science/paper/6YQ5JEEO

@misc{pith2026250205638,
  author       = {Pith},
  title        = {Pith review of: ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YQ5JEEO}},
  note         = {Machine review of arXiv:2502.05638}
}
read the original abstract

Europe's healthcare systems require enhanced interoperability and digitalization, driving a demand for innovative solutions to process legacy clinical data. This paper presents the results of our project, which aims to leverage Large Language Models (LLMs) to extract structured information from unstructured clinical reports, focusing on patient history, diagnoses, treatments, and other predefined categories. We developed a workflow with a user interface and evaluated LLMs of varying sizes through prompting strategies and fine-tuning. Our results show that fine-tuned smaller models match or surpass larger counterparts in performance, offering efficiency for resource-limited settings. A new dataset of 60,000 annotated English clinical summaries and 24,000 German translations was validated with automated and manual checks. The evaluations used ROUGE, BERTScore, and entity-level metrics. The work highlights the approach's viability and outlines future improvements.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages

  1. [1]

    Frontiers i n Pharmacology (2023)

    Adamson, B., et al.: Approach to machine learning for extr action of real-world data variables from electronic health records. Frontiers i n Pharmacology (2023)

    work page 2023

  2. [2]

    Internati onal Journal of Engineer- ing Business Management (2019)

    Adnan, K., Akbar, R.: Limitations of information extract ion methods and tech- niques for heterogeneous unstructured big data. Internati onal Journal of Engineer- ing Business Management (2019)

    work page 2019

  3. [3]

    Scientific data (2018)

    Caufield, J.H., et al.: A reference set of curated biomedic al data and metadata from clinical case reports. Scientific data (2018)

    work page 2018

  4. [4]

    In: BIOCOMPUTING 2021: Proceedings of the Pacific Symposium (2020)

    Chen, I.Y., Alsentzer, E., Park, H., Thomas, R., Gosangi, B., Gujrathi, R., Khu- rana, B.: Intimate partner violence and injury prediction f rom radiology reports. In: BIOCOMPUTING 2021: Proceedings of the Pacific Symposium (2020)

    work page 2020

  5. [5]

    https://doi.org/10.48550/arXiv.1810.04805

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre -training of Deep Bidirectional Transformers for Language Understan ding. https://doi.org/10.48550/arXiv.1810.04805

    arXiv doi:10.48550/arxiv.1810.04805

  6. [6]

    European Commission: Commission Recommendation (EU) 20 19/243 on a European Electronic Health Record exchange format (201 9), https://eur-lex.europa.eu/eli/reco/2019/243

    work page 2019

  7. [7]

    European Parliament: Artificial Intelligence Act: AI-Ac t (13 June 2024), http://data.europa.eu/eli/reg/2024/1689/oj

    work page 2024

  8. [8]

    European Parliament: Proposal for a REGULATION OF THE EP A ND OF THE COUNCIL on the European Health Data Space: COM/2022/197 fina l (352022), https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:52022PC0197

    work page 2022

Show all 31 references
  1. [9]

    https://doi.org/10.48550/arXiv.2307.03042

    Gema, A.P., Minervini, P., Daines, L., Hope, T., Alex, B.: Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Do main. https://doi.org/10.48550/arXiv.2307.03042

    doi arXiv

  2. [10]

    https://doi.org/10.48550/arXiv.2407.21783

    Grattafiori, A., et al.: The Llama 3 Herd of Models. https://doi.org/10.48550/arXiv.2407.21783

    doi arXiv

  3. [11]

    https://doi.org/10.5281/zenodo.14795642

    Guluzade, A., et al.: Appendix - ELMTEX. https://doi.org/10.5281/zenodo.14795642

    doi

  4. [12]

    https://doi.org/10.5281/zenodo.14793810

    Guluzade, A., et al.: ELMTEX Dataset. https://doi.org/10.5281/zenodo.14793810

    doi

  5. [13]

    Yearbook of medical informatics (2020)

    Hahn, U., Oleynik, M.: Medical information extraction i n the age of deep learning. Yearbook of medical informatics (2020)

    2020

  6. [14]

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wan g, S., Wang, L., Chen, W.: LoRA: Low-Rank Adaptation of Large Language Mo dels, http://arxiv.org/pdf/2106.09685v2

    arXiv

  7. [15]

    Journal of the American Medi cal Informatics Asso- ciation (2024)

    Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V.K., Zuo, X., Z hou, Y., Li, Z., Jiang, X., Lu, Z., et al.: Improving large language models for clini cal named entity recog- nition via prompt engineering. Journal of the American Medi cal Informatics Asso- ciation (2024)

    2024

  8. [16]

    Scott MARCUS, Bertin MARTENS, Christophe CARUGATI, A .B., and Ilsa GODLOVITCH: The European Health Data Space

    J. Scott MARCUS, Bertin MARTENS, Christophe CARUGATI, A .B., and Ilsa GODLOVITCH: The European Health Data Space

  9. [17]

    ACM Com- puting Surveys 55(12), 1–38 (2023)

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of Hallucination in Natural Language Ge neration. ACM Com- puting Surveys 55(12), 1–38 (2023)

    2023

  10. [18]

    Knowl- edge and Information Systems (2023)

    Landolsi, M.Y., Hlaoua, L., Ben Romdhane, L.: Informati on extraction from elec- tronic medical documents: state of the art and future resear ch directions. Knowl- edge and Information Systems (2023)

    2023

  11. [19]

    https://doi.org/10.48550/arXiv.2109.02555 Fine-Tuning LLMs for Structured Clinical Information Extr action 11

    Moradi, M., Blagec, K., Haberl, F., Samwald, M.: GPT-3 Mo d- els are Poor Few-Shot Learners in the Biomedical Domain. https://doi.org/10.48550/arXiv.2109.02555 Fine-Tuning LLMs for Structured Clinical Information Extr action 11

    doi arXiv

  12. [20]

    Journal of biomedical informatic s (2009)

    Mykowiecka, A., Marciniak, M., Kupść, A.: Rule-based in formation extraction from patients’ clinical data. Journal of biomedical informatic s (2009)

    2009

  13. [21]

    : Natural language pro- cessing of MIMIC-III clinical notes for identifying diagno sis and procedures with neural networks

    Nuthakki, S., Neela, S., Gichoya, J.W., Purkayastha, S. : Natural language pro- cessing of MIMIC-III clinical notes for identifying diagno sis and procedures with neural networks. https://doi.org/10.48550/arXiv.1912.12397

    1912 doi arXiv

  14. [22]

    In: Jiang, J., Reit ter, D., Deng, S

    Pal, A., Umapathi, L.K., Sankarasubbu, M.: Med-HALT: Me dical domain hallu- cination test for large language models. In: Jiang, J., Reit ter, D., Deng, S. (eds.) Proceedings of the 27th Conference on Computational Natura l Language Learning (CoNLL). pp. 314–334 (2023). https:/...

    2023 doi

  15. [23]

    , Wu, Y.: Model tuning or prompt tuning? a study of large language models for clinic al concept and relation extraction

    Peng, C., Yang, X., Smith, K.E., Yu, Z., Chen, A., Bian, J. , Wu, Y.: Model tuning or prompt tuning? a study of large language models for clinic al concept and relation extraction. Journal of biomedical informatics (2024)

    2024

  16. [24]

    OpenAI blog (2 019)

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Suts kever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog (2 019)

  17. [25]

    https://doi.org/10.48550/arXiv.2207.03885

    Roller, R., et al.: A Medical Information Extraction Wor kbench to Process German Clinical Text. https://doi.org/10.48550/arXiv.2207.03885

    doi arXiv

  18. [26]

    Advances in ne ural information pro- cessing systems (2017)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jon es, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in ne ural information pro- cessing systems (2017)

    2017

  19. [27]

    https://doi.org/10.48550/arXiv.2306.09968

    Wang, G., Yang, G., Du, Z., Fan, L., Li, X.: ClinicalGPT: L arge Language Models Finetuned with Diverse Medical Data and Comprehensi ve Evaluation. https://doi.org/10.48550/arXiv.2306.09968

    doi arXiv

  20. [28]

    Journal of the American Medical Informatics Associ ation (2020)

    Wu, S., Roberts, K., Datta, S., Du, J., Ji, Z., Si, Y., Soni , S., Wang, Q., Wei, Q., Xiang, Y., et al.: Deep learning in clinical natural languag e processing: a methodical review. Journal of the American Medical Informatics Associ ation (2020)

    2020

  21. [29]

    In: Proceedings of BioNLP 15 (2015)

    Wu, Y., Xu, J., Zhang, Y., Xu, H.: Clinical abbreviation d isambiguation using neural word embeddings. In: Proceedings of BioNLP 15 (2015)

    2015

  22. [30]

    In: Procs

    Zając, H.D., et al.: Ground truth or dare: Factors affecti ng the creation of medical datasets for training ai. In: Procs. of the 2023 AAAI/ACM AIE S Conference. p. 351–362 (2023). https://doi.org/10.1145/3600211.3604766

    2023

  23. [31]

    Scientific data 10(1), 909 (2023)

    Zhao, Z., Jin, Q., Chen, F., Peng, T., Yu, S.: A large-scal e dataset of patient sum- maries for retrieval-based clinical decision support syst ems. Scientific data 10(1), 909 (2023). https://doi.org/10.1038/s41597-023-02814-8

    2023 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.