Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Prompting Large Language Models for Clinical Temporal Relation Extraction

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hard-prompt fine-tuning of a 345M-parameter clinical encoder sets a new state of the art for clinical temporal relation extraction, reaching 89.54% micro F1 against the prior 85.70%.

desk verdict A plausible new SOTA on the i2b2 CTRE benchmark, but the 3.74% gain over prior work is unverified until the evaluation protocol is shown to match. read the letter →

arxiv 2412.04512 v1 pith:L6MVUO7V submitted 2024-12-04 cs.CL cs.LG

Jianping He , Laila Rasmy , Haifang Li , Jianfu Li , Zenan Sun , Evan Yu , Degui Zhi , Cui Tao This is my paper · ORCID
classification cs.CLcs.LG
keywords clinicaltemporalrelationextractionlargelanguagemodelshardpromptingprompttuningparameter-efficientfine-tuningGatorTronfew-shotlearning2012i2b2task
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that prompt-based fine-tuning of clinical large language models can push clinical temporal relation extraction past the prior state of the art on the standard 2012 i2b2 temporal-relations benchmark. Its main finding is that GatorTron-Base, a 345M-parameter clinical encoder, fine-tuned with a hard-prompt template that reformulates the task as mask filling, reaches 89.54% micro F1, beating the previous best of 85.70% by 3.74 points. The paper also reports that quantized LoRA variants of an 8.9B GatorTron model exceed the prior best, and that in few-shot settings frozen encoder models outperform frozen decoder models. If the comparison holds under identical preprocessing and evaluation, the result offers a practical recipe for improving timeline reconstruction from electronic health records without requiring the largest models.

What carries the argument

The load-bearing object is the hard-prompt reformulation of clinical temporal relation extraction as masked language modeling. Input text is appended with the template "Subject [MASK] [MASK] Object," and each temporal relation is mapped to label words, with BEFORE mapped to "happened before," OVERLAP to "happened overlap," and AFTER to "happened after." The model predicts the masked positions, and classification is the dot product between the contextual vectors of the masked tokens and the embeddings of the label words, with the highest-scoring label word mapped back to the relation. This lets an encoder LLM perform relation classification through its pretraining objective rather than through a separate classification head. Soft prompting, LoRA, and QLoRA serve as comparison strategies, but the hard-prompt masked-language-modeling step is what carries the reported gain over standard fine-tuning.

What would settle it

Run the same hard-prompting recipe on the 2012 i2b2 clinical temporal relation extraction data using the exact preprocessing of the prior baseline and the same official evaluation script; if the micro F1 drops to or below 85.70%, the central claim collapses. A simpler audit would confirm whether the reported 89.54% is reproducible from the released code and whether the test-set relation distribution matches the standard 27,736 relations split.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that hard-prompting with an unfrozen GatorTron-Base sets a new state of the art for clinical temporal relation extraction: micro F1 of 89.54% on the 2012 i2b2 task, 3.74% higher than the previous SOTA of 85.70%. The claim is grounded in a systematic comparison of six fine-tuning strategies across encoder and decoder large language models. In the fully supervised setting, the top four variants are all GatorTron-based, including standard fine-tuning (85.95%) and two QLoRA configurations on GatorTron-Large (88.06% and 87.58%), while decoder models top out below the SOTA. In few-shot settings, the paper reports that performance rises from 1-shot to 4-shot and then plateaus, and that frozen encoder models beat frozen decoder models, reversing the fully supervised trend. The paper attributes the encoder advantage to GatorTron's pretraining on a large de-identified clinical corpus and to the suitability of encoder architectures for classification.

Load-bearing premise

The claimed lead over the previous state of the art depends on the new runs sharing the same test split, preprocessing, and scoring as the 85.70% baseline, but the paper does not specify that preprocessing in the main text or provide an evaluation script; it also does not describe a held-out validation set, so the winning variant may have been selected on the test set.

Editorial extensions

If this is right

  • A 345M-parameter clinical encoder with hard prompting outperforms the prior best model on the 2012 i2b2 clinical temporal relation extraction benchmark by 3.74 micro-F1 points.
  • Parameter-efficient QLoRA on the 8.9B GatorTron also exceeds the prior SOTA, so strong CTRE can be achieved while keeping most weights frozen.
  • In few-shot regimes, frozen encoder models are the better choice, and adding examples beyond 4 shots yields no significant further gain.
  • The choice of fine-tuning strategy matters more than raw model size for this classification task: the best model is the smallest encoder, not the largest decoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hard-prompt result replicates, the same "verbalize the label, fill the mask" trick is a plausible drop-in for other clinical relation extraction tasks whose relation labels can be turned into short phrases; the authors themselves note that prompts must be redesigned per task.
  • The reported few-shot plateau at 4 examples is consistent with prompt-based methods needing only a tiny anchor set; a testable extension would be to check whether the same plateau shape appears on other relation extraction benchmarks.
  • Because the prior SOTA was set on the same benchmark with a different model family, the paper's protocol relies on cross-paper comparability; an independent reproduction with a shared evaluation harness would settle whether the 3.74-point gap is real or partly an artifact of scoring differences.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This manuscript evaluates full and parameter-efficient fine-tuning strategies for four large language models on the 2012 i2b2 clinical temporal relation extraction (CTRE) task. The authors compare 17 model/strategy variants under fully supervised and few-shot settings and report that hard-prompting with unfrozen GatorTron-Base achieves a micro-F1 of 89.54%, surpassing the previous SOTA of 85.70% by 3.74 percentage points. They also report that several QLoRA variants of GatorTron-Large and standard fine-tuning of GatorTron-Base exceed SOTA, while decoder-based models behave differently in few-shot regimes.

Significance. If the reported numbers are reproducible, the paper provides a useful practical comparison of fine-tuning strategies for clinical temporal relation extraction and demonstrates a substantial gain from a simple hard-prompting formulation on a clinical encoder model. The comparison of 17 variants across two data regimes and four LLM families is a valuable empirical resource, and the authors are explicit that the study is limited to a single dataset. However, the headline claim currently rests on an unverified protocol match with prior work and on single-run evaluations, so the significance is conditional on the availability of the missing evaluation details and code. The paper does not ship code, predictions, or an evaluation script, and no validation set or repeated-seed variance is reported.

major comments (5)
  1. [Materials and Methods, Dataset; Results, Table 4] The claimed 3.74% improvement over SOTA is only meaningful if the comparison is on the identical test split, preprocessing, relation-label set, and micro-F1 scoring. The main text says only that the authors 'adopted the same preprocessing methodology as outlined in the previous work by Chen et al. [35]' and does not describe that methodology; the SOTA papers [10,11] are not shown to have used that pipeline, and Background cites an 'F-measure' of 85.7 while Table 4 reports 'Micro F1'. Please provide the complete preprocessing and evaluation pipeline, or release the test predictions and scoring code, so that the 85.70% baseline can be reproduced under identical conditions.
  2. [Results, Table 4] All results appear to come from a single run per variant; no error bars, confidence intervals, or significance tests are reported. The comparison that is closest, GatorTron-Base at 85.95% versus SOTA at 85.70%, is a 0.25-point gap that is well within the typical run-to-run variance of fine-tuned transformers, so the claim that this variant 'exceeds' SOTA is not supported. Please report mean and standard deviation over at least 3-5 seeds and a significance test appropriate to document-level evaluation.
  3. [Materials and Methods; Results, Table 4] The paper reports 17 model/strategy variants on the same test set and selects the best (hard-prompting) without describing any held-out validation set or model-selection procedure. This raises the risk that the top result was chosen by peeking at the test set. Please state explicitly how the hyperparameters and the best variant were selected (e.g., validation set, cross-validation) and, if the test set was used for selection, describe the multiple-testing correction or qualify the claim accordingly.
  4. [Few-Shot Scenarios, Figure 6; Supplementary Table 5] The few-shot experimental protocol is not described in the main text: the paper does not say how the 1-, 2-, 4-, 8-, and 16-shot training sets are constructed (sampling procedure, seed, relation-type coverage), nor whether all models see identical examples, and the actual scores appear only in a supplementary table. Because the few-shot comparison is one of the two central contributions, please move the protocol description into the main text and include the per-seed or bootstrapped variance in the results.
  5. [Materials and Methods, Hard-Prompting with Unfrozen LLMs] The label words for the three relation classes are multi-token ('happened before', 'happened overlap', 'happened after'), but the scoring procedure is not fully specified: the text mentions a dot-product between 'the contextual vectors of the masked tokens' and 'the embeddings of the predefined label words' without stating how multiple label-token embeddings are combined or normalized. Please clarify the exact scoring and whether the same two [MASK] positions are used for all classes.
minor comments (6)
  1. [Table 2] The word 'Decocer' appears in both architecture rows and should be 'Decoder'.
  2. [Abstract; Table 4] The abstract refers to 'two variants of QLoRA adapted to GatorTron-Large', but Table 4 labels these rows as 'GatorTron-Large* + LoRA+' and 'GatorTron-Large* + LoRA++' without the QLoRA designation; please align the terminology.
  3. [Employed Large Language Models (LLMs)] The name 'Gatortron-Base' appears with a lowercase 't' in one place; use 'GatorTron' consistently throughout.
  4. [Parameter-Efficient Fine-Tuning Strategies for Transformer Decoder-based Models] The sentence beginning 'The second PEFT strategy, QLoRA with frozen LLMs utilizing standard fine-tuning, Instead of generating a textual label...' is grammatically incomplete; 'Instead' should begin a new sentence.
  5. [Background; Results] The 85.70% SOTA value is cited to both [10,11], but the Related Works section reports [10] as achieving 82.04%; please attribute the 85.70% figure to [11] and reconcile the citation.
  6. [Discussions and Conclusions] The claim that the i2b2 dataset was not used in GatorTron pretraining is stated without supporting evidence or a data-contamination analysis; please add a citation or an explicit leakage check.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the SOTA gain is an empirical benchmark comparison, and the only self-citations are motivational design choices, not load-bearing reductions.

full rationale

The paper's central claim is an empirical measurement on the 2012 i2b2 CTRE benchmark: 'GatorTron-Base + hard-prompting achieved the highest score of 89.54%, surpassing the SOTA model's score of 85.70% by 3.74%.' There is no derivation chain in which an equation or fitted parameter is renamed as a prediction; hard-prompting is described by an architectural mechanism (dot-product between mask-token contextual vectors and label embeddings), not by an identity that forces the reported Micro F1. The only self-citations, [42,43], support the design decision that 'reformatting downstream tasks to align with a model's pretraining format significantly enhances performance,' which is a motivational prior result, not the target result, and does not by itself imply the 89.54% score. The SOTA comparison is against external prior work [10,11], and the preprocessing-follows-Chen-et-al. statement is a reproducibility concern, not a circularity concern. Unverifiability of the exact protocol is a correctness/risk issue, not evidence that the result reduces to its inputs. Accordingly, the appropriate finding is no significant circularity, with a score of 1 reflecting the presence of minor non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper relies on standard pretrained models and benchmark data. The main 'free' design choices are the prompt templates, label words, and PEFT hyperparameters, which are not specified in the main text.

free parameters (3)
  • Hard-prompt label words = 'happened before', 'happened overlap', 'happened after'
    The verbalizers used for the masked language modeling prediction are a design choice; different label words could affect the F1 score.
  • Soft prompt length = 32 and 64
    Two fixed lengths were tested for soft-prompting; these are hyperparameters chosen by the authors and affect the soft-prompting results, though not the central best result.
  • LoRA configuration = query-value layers (+) vs all linear layers (++); rank not specified in main text
    The LoRA variants differ in which layers are adapted, and the rank/alpha values are deferred to the supplementary file. These choices affect the LoRA results.
assumptions (3)
  • domain assumption The i2b2 evaluation protocol used by the authors matches that of the SOTA works [10,11].
    The claimed SOTA improvement depends on this; the paper does not provide the evaluation script or a detailed description of the preprocessing.
  • domain assumption The i2b2 test set was not used to select among the many model variants.
    The paper does not describe a validation set, so it is unclear whether the reported best result is a genuine hold-out evaluation or a selection on test data.
  • domain assumption GatorTron's pretraining corpus excludes i2b2 notes.
    The authors assert this to rule out test contamination, but provide no evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompting Large Language Models for Clinical Temporal Relation Extraction." pith.science (2026). https://pith.science/paper/L6MVUO7V

@misc{pith2026241204512,
  author       = {Pith},
  title        = {Pith review of: Prompting Large Language Models for Clinical Temporal Relation Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L6MVUO7V}},
  note         = {Machine review of arXiv:2412.04512}
}
read the original abstract

Objective: This paper aims to prompt large language models (LLMs) for clinical temporal relation extraction (CTRE) in both few-shot and fully supervised settings. Materials and Methods: This study utilizes four LLMs: Encoder-based GatorTron-Base (345M)/Large (8.9B); Decoder-based LLaMA3-8B/MeLLaMA-13B. We developed full (FFT) and parameter-efficient (PEFT) fine-tuning strategies and evaluated these strategies on the 2012 i2b2 CTRE task. We explored four fine-tuning strategies for GatorTron-Base: (1) Standard Fine-Tuning, (2) Hard-Prompting with Unfrozen LLMs, (3) Soft-Prompting with Frozen LLMs, and (4) Low-Rank Adaptation (LoRA) with Frozen LLMs. For GatorTron-Large, we assessed two PEFT strategies-Soft-Prompting and LoRA with Frozen LLMs-leveraging Quantization techniques. Additionally, LLaMA3-8B and MeLLaMA-13B employed two PEFT strategies: LoRA strategy with Quantization (QLoRA) applied to Frozen LLMs using instruction tuning and standard fine-tuning. Results: Under fully supervised settings, Hard-Prompting with Unfrozen GatorTron-Base achieved the highest F1 score (89.54%), surpassing the SOTA model (85.70%) by 3.74%. Additionally, two variants of QLoRA adapted to GatorTron-Large and Standard Fine-Tuning of GatorTron-Base exceeded the SOTA model by 2.36%, 1.88%, and 0.25%, respectively. Decoder-based models with frozen parameters outperformed their Encoder-based counterparts in this setting; however, the trend reversed in few-shot scenarios. Discussions and Conclusions: This study presented new methods that significantly improved CTRE performance, benefiting downstream tasks reliant on CTRE systems. The findings underscore the importance of selecting appropriate models and fine-tuning strategies based on task requirements and data availability. Future work will explore larger models and broader CTRE applications.

Figures

Figures reproduced from arXiv: 2412.04512 by the authors.

Figure 1
Figure 1. The architectures of LLMs [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. The principle of primary fine-tuning strategies for LLMs. Fine-Tuning Strategies for GatorTron-Base We developed and evaluated four fine-tuning strategies for GatorTron-Base: (1) Standard Fine-Tuning without the Use of Prompts, (2) Hard-Prompting with Unfrozen LLMs, (3) Soft￾Prompting with Frozen LLMs, and (4) LoRA with Frozen LLMs [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The principle of four primary fine-tuning strategies for GatorTron-Base [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The principle of two primary PEFT strategies for the Transformer Decoder [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The performance (micro F1) of various models under the fully supervised setting. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: The performance (micro F1) of various models across different few [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 1
Figure 1. Figure 1: The architectures of LLMs [PITH_FULL_IMAGE:figures/full_fig_p019_1.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives

    cs.CL 2026-08 conditional novelty 6.0 of 10

    An LLM generator-verifier refinement loop improves temporal ordering of symptoms in single-report vaccine narratives, evaluated on a new 5,347-report benchmark.

Reference graph

Works this paper leans on

3 extracted references · cited by 1 Pith paper

  1. [7]

    Review of Temporal Reasoning in the Clinical Domain for Timeline Extraction: Where we are and where we need to be

    doi: 10.1016/j.jbi.2013.11.001 4 Olex AL, McInnes BT. Review of Temporal Reasoning in the Clinical Domain for Timeline Extraction: Where we are and where we need to be. J Biomed Inform. 2021;118:103784. doi: 10.1016/j.jbi.2021.103784 5 Leaman R, Khare R, Lu Z. Challenges in clinical natural language processing for automated disorder normalization. J Biome...

    arXiv 2013

  2. [43]

    Multimodal learning for temporal relation extraction in clinical texts

    doi: 10.1145/182.358434 10 Knez T, Žitnik S. Multimodal learning for temporal relation extraction in clinical texts. J Am Med Inform Assoc. 2024;31:1380–7. doi: 10.1093/jamia/ocae059 11 Uma K, Francis S, Moens M-F. Masking language model mechanism with event-driven knowledge graphs for temporal relations extraction from clinical narratives. Complex Networ...

    arXiv 2024

  3. [75]

    LIMSI-COT at SemEval-2016 Task 12: Temporal relation identification using a pipeline of classifiers

    doi: 10.1136/amiajnl-2013-001619 29 Tourille J, Ferret O, Névéol A, et al. LIMSI-COT at SemEval-2016 Task 12: Temporal relation identification using a pipeline of classifiers. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Stroudsburg, PA, USA: Association for Computational Linguistics 2016. 30 Cohan A, Meurer K, Goh...

    arXiv 2013

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.