Pith. sign in

REVIEW 5 major objections 6 minor 15 references

Decoder-style generative LLMs with LoRA and multi-task instruction tuning match full fine-tuning using only 20% of labeled clinical data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:57 UTC pith:VNUJW2CU

load-bearing objection Useful benchmark with plausible directions, but the headline data-efficiency claim is under-specified and the UF Health overlap is unaddressed. the 5 major comments →

arxiv 2509.04753 v1 pith:VNUJW2CU submitted 2025-09-05 cs.CL cs.AI

A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning

Cheng Peng , Xinyu Dong , Mengxian Lyu , Daniel Paredes , Yaoyun Zhang , Yonghui Wu This is my paper
classification cs.CL cs.AI
keywords clinical concept extractionclinical relation extractionlarge language modelsparameter-efficient fine-tuningLoRAmulti-task instruction tuningfew-shot learningpatient information extraction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks how best to adapt large language models to extract patient information from clinical notes—finding disease mentions, treatments, medications, and the relations among them. It claims that decoder-style generative models tuned with a parameter-efficient method (LoRA adapters, which train under 1% of weights) match or beat the best encoder-style models on both tasks at a fraction of the training cost. Its larger claim is about generalization: when three generative models are first instruction-tuned on four clinical datasets together, then tested on a held-out dataset with 5, 10, 20, or 50 examples, their zero-shot and few-shot F1 scores jump substantially, and with only 20% of a dataset they match models trained on the full data. If true, clinical NLP teams could spend far less on annotation and compute, reusing a multi-task instruction-tuning pool to adapt to new datasets cheaply.

Core claim

The paper's central claim is that the best recipe for clinical concept and relation extraction is a decoder-only generative LLM adapted with prompt-based LoRA parameter-efficient fine-tuning, followed by multi-task instruction tuning. On five benchmark datasets, the clinical continued-pretrained Llama variant reached average F1 of 0.8981 for concept extraction and 0.8978 for relation extraction, slightly ahead of the largest encoder-only models. Prompt-based PEFT improved relation extraction F1 by up to 15.9% over traditional full fine-tuning. In a leave-one-dataset-out evaluation, multi-task instruction tuning lifted zero-shot F1 from near zero to as high as 0.3596 for concept extraction an

What carries the argument

The argument runs on two mechanisms. First, prompt-based PEFT: instead of adding dataset-specific classification heads, the model is given natural-language instructions and generates concept spans or relation labels as text; LoRA injects small trainable low-rank matrices into attention layers, so fewer than 1% of parameters are updated. Second, multi-task instruction tuning: the same decoder model is fine-tuned jointly on four datasets spanning two tasks with multiple prompt templates, then tested on a held-out dataset. The leave-one-dataset-out protocol is what lets the paper attribute gains in zero-shot and few-shot performance to the instruction-tuning stage.

Load-bearing premise

The headline few-shot gains assume the single-dataset baselines were trained under identical conditions (same adapter settings, prompts, optimizer, and decoding) as the multi-task models, and that the clinical continued-pretraining corpus does not include the evaluation notes; if either assumption fails, the measured boosts are not solely due to multi-task instruction tuning.

What would settle it

Run the leave-one-dataset-out experiment with a single base model, holding LoRA rank, prompt templates, optimizer, learning rate, and decoding constant, and compare single-task fine-tuning against multi-task instruction tuning; if the 1.1–37.8% F1 boosts shrink to noise, the multitask claim is refuted. Separately, evaluate the clinically continued-pretrained model on notes from an external institution to test whether pretraining/evaluation overlap explains the high scores.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A single generative model can be prompted for both concept extraction and relation extraction, replacing task-specific classification heads with one unified text-to-text interface.
  • LoRA makes adapting an 8-billion-parameter clinical model practical: about 8 GPU hours versus 48 for full fine-tuning of a 9-billion-parameter model, with inference latency nearly unchanged because adapters merge into the base weights.
  • Multi-task instruction tuning raises zero-shot F1 from near zero to usable levels (about 0.27–0.40 across models), making annotation-free extraction more realistic for new clinical domains.
  • For a new dataset, labeling roughly 20% of the available examples after instruction tuning can reach the F1 of a fully fine-tuned model, cutting annotation cost by about 80%.
  • For small encoder models with abundant in-domain data, full fine-tuning can still beat prompt-based PEFT, so the best architecture depends on model scale and data availability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct ablation that holds the base model, LoRA rank, prompt templates, optimizer, and decoding identical between single-task fine-tuning and multi-task instruction tuning would cleanly isolate the instruction-tuning effect the paper attributes to the multi-task stage.
  • The 20%-data result suggests a practical deployment recipe: maintain a shared multi-task instruction-tuning pool, then add a small number of target-domain examples for each new annotation schema; whether the pool should include non-clinical datasets is testable but not explored here.
  • Because one of the best-performing models was continued-pretrained on the same institution's notes used for evaluation, a cross-institution replication on notes from different health systems would show how much of the gain reflects broad method versus domain familiarity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper reports an empirical benchmark of encoder-only (BERT, GatorTron) and decoder-only (GatorTronGPT, Llama 3.1, GatorTronLlama) LLMs for clinical concept extraction (CE) and relation extraction (RE) on five datasets. It compares traditional full-size fine-tuning with prompt-based PEFT/LoRA, and evaluates multi-task instruction tuning with a leave-one-dataset-out protocol for zero- and few-shot generalization. The headline claims are that decoder-based generative LLMs with PEFT match or exceed the best encoder-only models, that prompt-based PEFT improves RE by up to 15.9% over full fine-tuning, and that multi-task instruction tuning using only 20% of the full dataset achieves F1 within 0.005 of full-size fine-tuning.

Significance. If the claims hold, the paper would provide practical guidance for building clinical IE systems with substantially lower annotation and compute costs: decoder LLMs with LoRA and multi-task instruction tuning could replace expensive full fine-tuning of encoder models. The study covers diverse, widely used datasets and reports an efficiency comparison (Table 4), which is useful for practitioners. The strengths are the breadth of the benchmark, the inclusion of several model families, and the explicit efficiency measurements. However, several load-bearing comparisons are not adequately controlled, and no variance information is reported, so the quantitative conclusions are not yet established at the level claimed.

major comments (5)
  1. The abstract's central data-efficiency claim—'using only 20% of the full dataset achieved similar performance comparable to the full-size fine-tuning, with a very small gap less than 0.005 in F1 scores'—is not supported by the reported methodology. The full-data comparator is never defined: it is not stated whether the baseline is the same decoder model with LoRA, the same prompt templates, the same output parsing, or the same training budget. Figure 3 is only described qualitatively; no axis labels, error bars, or per-fold numbers are given. Because all results appear to be single runs with no seeds or confidence intervals, a gap of 0.005 is indistinguishable from run-to-run variance. The authors should specify the comparator precisely and report multiple seeds or confidence intervals for this key claim.
  2. The claim of an 'F1 improvement up to 15.9% over traditional fine-tuning' for prompt-based PEFT is misleading. In Table 3, the 15.9% gap is between GatorTron-base with full fine-tuning on RadGraph (0.6925) and GatorTronLlama with prompt-based PEFT on RadGraph (0.8514). This comparison varies both the architecture (encoder vs. decoder) and the fine-tuning strategy simultaneously. Within the encoder-only family, the largest improvement from PEFT over full fine-tuning is 2.84 percentage points (GatorTron-large-MRC 0.8661 vs. GatorTron-large 0.8375). The abstract should either report the within-architecture comparison or explicitly state that the 15.9% figure includes architecture differences.
  3. The few-shot and zero-shot baselines are underspecified. The text states that multi-task models are compared with 'single-dataset fine-tuned models', but it does not state whether those baselines use the same LoRA configuration (rank 256, dropout 0.2), the same prompt templates, the same base model, the same optimizer, or the same decoding/parsing of generated text. Without this information, the reported multi-task improvements (e.g., 0.8–9.7% for CE, 1.8–6.0% for RE, and the zero-shot gains) conflate multi-task instruction tuning with differences in the adaptation protocol. The authors should provide the full baseline protocol or run baselines that differ only in the multi-task training data.
  4. There is a potential pretraining/evaluation overlap that is not addressed. GatorTronLlama is described as having been continue-pretrained on over 100 billion words of clinical text collected from UF Health, and the UF Health evaluation dataset is also drawn from UF Health IDR. Similarly, GatorTron was pretrained on de-identified UF Health notes. The paper does not report any overlap analysis (e.g., n-gram overlap between pretraining corpora and the UF Health test set). Because the UF Health dataset is not public, readers cannot assess this independently. Please report overlap statistics or otherwise justify that the UF Health test set was not seen during pretraining.
  5. No measure of variability is reported anywhere. All F1 scores in Tables 2 and 3 and Figures 2 and 3 appear to be from single runs. Given that several headline conclusions involve small differences (e.g., GatorTronLlama 0.8981 vs. Llama 3.1 0.8964 for CE; the <0.005 gap in Figure 3), the absence of seeds, confidence intervals, or significance tests is a major limitation. The authors should report at least three seeds for the key comparisons, or a bootstrap confidence interval, to establish that the reported rankings and gaps are stable.
minor comments (6)
  1. 'similar performance comparable to' is redundant; consider 'performance comparable to'.
  2. The definition says decoder-based LLMs are 'trained using the encoder component of the transformer architecture'; this appears to be a typo for 'decoder component'.
  3. Typo: 'GatoTronGPT-base' should be 'GatorTronGPT-base'.
  4. Typo in row label: 'GaotTronLlma' should be 'GatorTronLlama'.
  5. The reported improvements are given in percentage points but phrased as percentages (e.g., 'by 34.41%'); please clarify whether these are relative or absolute differences.
  6. The figure captions describe content, but the figures themselves are not included in the manuscript text. Ensure the final version includes axis labels, legends, and ideally error bars for the few-shot trajectories.

Circularity Check

0 steps flagged

No circular derivation; central claims are empirical benchmark observations, with minor non-load-bearing self-citations.

full rationale

This paper is an empirical benchmark study: every headline result (CE/RE F1 scores, PEFT vs full fine-tuning, zero-/few-shot multitask gains, the 20%-data/0.005-gap claim) is a measured outcome on held-out test sets, not a quantity derived from a fitted parameter or from a definition. I specifically examined the load-bearing 20%/0.005 claim (Abstract, Figure 3, Discussion): it is presented as an observed learning-curve trajectory, not as a prediction forced by construction; there is no equation in which the 20% result is defined in terms of the full-data comparator. The self-cited artifacts—GatorTron/GatorTronGPT/GatorTronLlama, the prompt-based MRC framework (ref. 25), and the internal UF Health dataset (ref. 39)—are inputs to the benchmark, not premises that entail the conclusions. The methods state 'We adopted a prompt-based Machine Reading Comprehension (MRC) framework developed in our previous work to fine-tune encoder-based LLMs'; this is an implementation choice, and the conclusion that decoder-based models match or exceed encoder-based models is not logically forced by that citation. The paper's limitations section acknowledges prompt-quality dependence, which is a validity caveat, not circularity. The main methodological weaknesses—the under-specified few-shot baseline comparator and the possible pretraining/evaluation overlap for UF Health—are reproducibility and data-leakage concerns, not circularity, and I do not count them toward the circularity score. The self-citations are numerous but none is load-bearing to the central measured claims, so the score is 2 rather than 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

This is an empirical benchmark, so the 'free parameters' are experimental hyperparameters and hand-tuning choices rather than fitted constants in a model. The claims rest mostly on standard ML assumptions; the two nonstandard load-bearing assumptions are the undisclosed UF Health pretraining/evaluation overlap and the comparability of the multi-task and single-task few-shot baselines. GatorTronLlama is a new model artifact (continued-pretrained Llama 3.1) whose only evidence is this paper's evaluation, but it is an empirical artifact, not a theoretical postulate.

free parameters (4)
  • LoRA rank (adapter_dim) = 256
    Global hyperparameter for all decoder PEFT runs; controls adaptation capacity; no ablation reported.
  • LoRA dropout = 0.2
    Applied to adapter layers in all decoder runs; no ablation reported.
  • Learning rates = 2e-5 (encoders), 1e-4 (decoders)
    Chosen per model family; no tuning grid reported.
  • Hand-selected prompt templates = Per-task templates listed in supplementary Table S1
    The paper states 'We identified the best prompts for each task based on prompt engineering'; the selection protocol is not described, and prompt choice can inflate task scores.
axioms (4)
  • domain assumption Micro-averaged F1 with strict span/type match is an adequate measure of clinical IE quality.
    Standard metric for i2b2/n2c2 style evaluation; adopted without discussion of its limitations for overlapping or discontinuous entities noted in the error analysis.
  • ad hoc to paper The pretraining corpora of GatorTron, GatorTronGPT, and GatorTronLlama are effectively disjoint from the UF Health evaluation dataset.
    GatorTronLlama was continued-pretrained on over 100 billion words of UF Health clinical text and the UF Health test set is drawn from UF Health IDR; the paper never addresses possible overlap. Load-bearing for GatorTronLlama's top average F1.
  • domain assumption Leave-one-dataset-out multi-task tuning is a faithful test of generalizability to unseen annotation schemas.
    The held-out datasets share the clinical IE task family, so zero-shot transfer here may overstate transfer to genuinely novel schema designs.
  • ad hoc to paper Generated text from decoder LLMs was parsed into concept and relation spans without systematic error that favors particular models.
    The parsing and post-processing procedure for text-to-text outputs is not described anywhere in the methods or results.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 11553 in / 25882 out tokens · 221637 ms · 2026-08-05T05:57:26.249479+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning." pith.science (2026). https://pith.science/paper/VNUJW2CU

@misc{pith2026250904753,
  author       = {Pith},
  title        = {Pith review of: A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNUJW2CU}},
  note         = {Machine review of arXiv:2509.04753}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Natural language processing (NLP) is a key technology to extract important patient information from clinical narratives to support healthcare applications. The rapid development of large language models (LLMs) has revolutionized many NLP tasks in the clinical domain, yet their optimal use in patient information extraction tasks requires further exploration. This study examines LLMs' effectiveness in patient information extraction, focusing on LLM architectures, fine-tuning strategies, and multi-task instruction tuning techniques for developing robust and generalizable patient information extraction systems. This study aims to explore key concepts of using LLMs for clinical concept and relation extraction tasks, including: (1) encoder-only or decoder-only LLMs, (2) prompt-based parameter-efficient fine-tuning (PEFT) algorithms, and (3) multi-task instruction tuning on few-shot learning performance. We benchmarked a suite of LLMs, including encoder-based LLMs (BERT, GatorTron) and decoder-based LLMs (GatorTronGPT, Llama 3.1, GatorTronLlama), across five datasets. We compared traditional full-size fine-tuning and prompt-based PEFT. We explored a multi-task instruction tuning framework that combines both tasks across four datasets to evaluate the zero-shot and few-shot learning performance using the leave-one-dataset-out strategy.

Figures

Figures reproduced from arXiv: 2509.04753 by Cheng Peng, Daniel Paredes, Mengxian Lyu, Xinyu Dong, Yaoyun Zhang, Yonghui Wu.

Figure 2
Figure 2. Figure 2: Zero-shot and few-shot performance between multi-task instruction tuned LLMs and baseline models without instruction tuning across CCE and CRE tasks [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

  1. [1]

    Wells, Jr

    A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning Authors: Cheng Peng, PhD1, * Xinyu Dong, PhD3, * Mengxian Lyu, MS1 Daniel Paredes, MS1 Yaoyun Zhang, PhD3, † Yonghui Wu, PhD1,2, † Affiliation of the authors: 1Department of Health Outcomes and Biomedical Informa...

    work page 2006

  2. [2]

    Performance comparison of LLMs with different architectures and fine-tuning strategies for clinical concept extraction. Model Architecture Params Fine-tuning strategy Dataset (micro average F1 score) 2010 i2b2 2018 n2c2 2022 n2c2 RadGraph UF Health Average BERT-large Encoder-only 340M 0.8694 0.8807 0.8318 0.8852 0.9048 0.8744 GatorTron-base 345M Full fine...

    work page 2010

  3. [3]

    Performance comparison of LLMs with different architectures and fine-tuning strategies for clinical relation extraction Model Architecture Params Fine-tuning strategy Dataset (micro average F1 score) 2018 n2c2 2022 n2c2 RadGraph UFHealth Average BERT-large Encoder-only 340M Full fine-tuning 0.9561 0.7807 0.6985 0.8952 0.8326 GatorTron-base 345M 0.9575 0.7...

    work page 2018

  4. [4]

    severe headache

    Comparison of computational efficiency for different LLMs. Model Params Fine-Tuning Strategy Trainable Params Average Training Time (GPU Hours) Average Inference Time (ms/note) GatorTron-base 345 million Full fine-tuning 100% ~8 ~15 GatorTron-large 9 billion ~48 ~22 GatorTron-base-MRC 345 million Prompt-based LoRA ~0.5% ~2 ~16 GatorTron-large-MRC 9 billio...

    work page 2022

  5. [8]

    All prompt-based PEFT was conducted within NeMo

    Experiments for all decoder-only generative models (GatorTronGPT, Llama 3.1, GatorTronLlama) were performed using the NVIDIA NeMo framework. All prompt-based PEFT was conducted within NeMo. For these models, a learning rate of 1e-4 was used with the AdamW optimizer. The specific LoRA configuration included a rank (adapter_dim) of 256 and a dropout of 0.2 ...

    work page 2022

  6. [9]

    Testing and evaluation of health care applications of large language models: A systematic review

    doi: 10.1093/jamia/ocad012 18 Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of large language models: A systematic review. JAMA. 2025;333:319–28. 19 Yao L, Mao C, Luo Y. Clinical text classification with rule-based features and knowledge-guided convolutional neural networks. BMC Med Inform Decis Mak. 2019;19:71. 20 ...

    doi:10.1093/jamia/ocad012 2025

  7. [12]

    Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review

    doi: 10.1093/jamia/ocad107 26 Fraile Navarro D, Ijaz K, Rezazadegan D, et al. Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review. Int J Med Inform. 2023;177:105122. 27 Touvron H, Lavril T, Izacard G, et al. LLaMA: Open and efficient foundation language models. arXiv [cs.CL]

    work page doi:10.1093/jamia/ocad107 2023

  8. [15]

    Identifying social determinants of health from clinical narratives: A study of performance, documentation ratio, and potential bias

    39 Yu Z, Peng C, Yang X, et al. Identifying social determinants of health from clinical narratives: A study of performance, documentation ratio, and potential bias. J Biomed Inform. 2024;153:104642

    work page 2024

  9. [20]

    have limitations in generalizability, recent advancements in LLMs provide promising solutions. The original breakthrough of LLMs came from the encoder-based transformer models, such as BERT[21], which were later customized for biomedical and clinical applications through pre-training using biomedical and clinical corpora, contributing to domain-specific t...

    work page 2020

  10. [33]

    find the drug events including names, brand names and collective names

    and the trained classification layer only can be used for one task-specific type of IE. As LLMs typically have massive amounts of parameters typically over billions, it is very expensive to update all parameters during fine-tuning. Parameter-efficient fine-tuning (PEFT), such as P-tuning[34,35] and Low-Rank Adaptation (LoRA)[36], were proposed to reduce t...

    work page 2010

  11. [2018]

    BioBERT: a pre-trained biomedical language representation model for biomedical text mining

    22 Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36:1234–40. 23 Huang K, Altosaar J, Ranganath R. ClinicalBERT: Modeling clinical notes and predicting hospital readmission. arXiv [cs.CL]

    work page 2020

  12. [2019]

    A large language model for electronic health records

    24 Yang X, Chen A, PourNejatian N, et al. A large language model for electronic health records. NPJ Digit Med. 2022;5:194. 25 Peng C, Yang X, Yu Z, et al. Clinical concept and relation extraction using prompt-based machine reading comprehension. J Am Med Inform Assoc. Published Online First: 14 June

    work page 2022

  13. [2021]

    Contextualized medication information extraction using Transformer-based deep learning architectures

    33 Chen A, Yu Z, Yang X, et al. Contextualized medication information extraction using Transformer-based deep learning architectures. J Biomed Inform. 2023;142:104370. 34 Liu X, Zheng Y, Du Z, et al. GPT Understands, Too. arXiv [cs.CL]

    work page 2023

  14. [2023]

    2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text

    9 Uzuner Ö, South BR, Shen S, et al. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc. 2011;18:552–6. 10 Uzuner O, Bodnari A, Shen S, et al. Evaluating the state of the art in coreference resolution for electronic medical records. J Am Med Inform Assoc. 2012;19:786–91. 11 Uzuner O, Solti I, Cadag E. Ext...

    work page 2010

  15. [2024]

    A study of generative large language model for medical research and healthcare

    30 Peng C, Yang X, Chen A, et al. A study of generative large language model for medical research and healthcare. NPJ Digit Med. 2023;6:210. 31 Brown TB, Mann B, Ryder N, et al. Language Models are Few-Shot Learners. arXiv [cs.CL]

    work page 2023