{"id":"79eda85b-2852-452b-b3d0-73b9b85c7584","arxiv_id":"2412.04512","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Hard-prompting fine-tuning of GatorTron-Base achieves 89.54% micro F1 on the 2012 i2b2 clinical temporal relation extraction task, surpassing the previous best score of 85.70%.","lead":"This study fine-tunes four large language models, including GatorTron, LLaMA3, and MeLLaMA, to extract temporal relations such as 'before' and 'after' from clinical notes. The best configuration, hard-prompting with an unfrozen GatorTron-Base encoder, reaches 89.54% micro F1 on the 2012 i2b2 benchmark, beating the previous state of the art by 3.74 percentage points.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3.74% SOTA gain is only meaningful under identical preprocessing and evaluation; the paper says it follows Chen et al. [35] without describing that pipeline and gives no prediction files or official scorer, so the headline comparison is unverifiable.","rationale":"The reader's weakest_assumption matches the point I consider most load-bearing. The paper repeatedly frames the contribution as 'surpassing SOTA by 3.74%', but the entire foundation is an uncontrolled comparison between a newly implemented pipeline and a published number. The discrepancy could easily arise from sentence splitting, truncated 512-token context, entity masking, dropped relation types, or the use of micro vs macro F1. These are mundane pipeline choices, not accusations of misconduct. The method is plausible and GatorTron-Base + hard-prompting may well generalize, but the preprint as written does not give enough to verify the central empirical claim. The lack of a validation set and error bars compounds the problem, but the protocol mismatch is the primary attack. I would not reject the work; I would keep the reader's CONDITIONAL verdict: the claim should not be treated as established until predictions and evaluation code are available and the comparison is run under identical conditions.","tokens_in":12727,"tokens_out":8610,"duration_ms":84421,"concrete_test":"Run an independent reproduction using only the paper's described hard-prompt template ('Subject [MASK] [MASK] Object' with label words 'happened before/overlap/after') and the official 2012 i2b2 test split, scoring with the official i2b2 evaluation script rather than any private implementation. Then apply the same preprocessing and scorer to the prior SOTA system (or have the authors share its predictions) and compare micro-F1. If the reproduced hard-prompting score does not exceed the reproduced SOTA under identical conditions, the headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a single number: GatorTron-Base + hard-prompting reaches 89.54% vs. the 85.70% SOTA. That comparison can only hold if both systems are measured on the same 120 test notes, with the same relation label set, the same sentence/context preprocessing, and the same micro-F1 scoring. The paper's Methods only say 'We adopted the same preprocessing methodology as outlined in the previous work by Chen et al. [35]' and relegate details to a supplementary file; it does not show that the SOTA papers [10,11] used that pipeline or that their 85.70 was micro-F1 (Background says 'F-measure', Table 4 says 'Micro F1'). No evaluation script, predictions, or trained model are released, and no held-out validation set or repeated-seed variance is reported, so the reader cannot rule out protocol mismatch or test-set selection among the 19 variants in Table 4. If the preprocessing or scoring differs, the claimed 3.74% improvement is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript evaluates full and parameter-efficient fine-tuning strategies for four large language models on the 2012 i2b2 clinical temporal relation extraction (CTRE) task. The authors compare 17 model/strategy variants under fully supervised and few-shot settings and report that hard-prompting with unfrozen GatorTron-Base achieves a micro-F1 of 89.54%, surpassing the previous SOTA of 85.70% by 3.74 percentage points. They also report that several QLoRA variants of GatorTron-Large and standard fine-tuning of GatorTron-Base exceed SOTA, while decoder-based models behave differently in few-shot regimes.","tokens_in":12855,"tokens_out":7328,"duration_ms":62331,"significance":"If the reported numbers are reproducible, the paper provides a useful practical comparison of fine-tuning strategies for clinical temporal relation extraction and demonstrates a substantial gain from a simple hard-prompting formulation on a clinical encoder model. The comparison of 17 variants across two data regimes and four LLM families is a valuable empirical resource, and the authors are explicit that the study is limited to a single dataset. However, the headline claim currently rests on an unverified protocol match with prior work and on single-run evaluations, so the significance is conditional on the availability of the missing evaluation details and code. The paper does not ship code, predictions, or an evaluation script, and no validation set or repeated-seed variance is reported.","major_comments":[{"comment":"The claimed 3.74% improvement over SOTA is only meaningful if the comparison is on the identical test split, preprocessing, relation-label set, and micro-F1 scoring. The main text says only that the authors 'adopted the same preprocessing methodology as outlined in the previous work by Chen et al. [35]' and does not describe that methodology; the SOTA papers [10,11] are not shown to have used that pipeline, and Background cites an 'F-measure' of 85.7 while Table 4 reports 'Micro F1'. Please provide the complete preprocessing and evaluation pipeline, or release the test predictions and scoring code, so that the 85.70% baseline can be reproduced under identical conditions.","section":"Materials and Methods, Dataset; Results, Table 4"},{"comment":"All results appear to come from a single run per variant; no error bars, confidence intervals, or significance tests are reported. The comparison that is closest, GatorTron-Base at 85.95% versus SOTA at 85.70%, is a 0.25-point gap that is well within the typical run-to-run variance of fine-tuned transformers, so the claim that this variant 'exceeds' SOTA is not supported. Please report mean and standard deviation over at least 3-5 seeds and a significance test appropriate to document-level evaluation.","section":"Results, Table 4"},{"comment":"The paper reports 17 model/strategy variants on the same test set and selects the best (hard-prompting) without describing any held-out validation set or model-selection procedure. This raises the risk that the top result was chosen by peeking at the test set. Please state explicitly how the hyperparameters and the best variant were selected (e.g., validation set, cross-validation) and, if the test set was used for selection, describe the multiple-testing correction or qualify the claim accordingly.","section":"Materials and Methods; Results, Table 4"},{"comment":"The few-shot experimental protocol is not described in the main text: the paper does not say how the 1-, 2-, 4-, 8-, and 16-shot training sets are constructed (sampling procedure, seed, relation-type coverage), nor whether all models see identical examples, and the actual scores appear only in a supplementary table. Because the few-shot comparison is one of the two central contributions, please move the protocol description into the main text and include the per-seed or bootstrapped variance in the results.","section":"Few-Shot Scenarios, Figure 6; Supplementary Table 5"},{"comment":"The label words for the three relation classes are multi-token ('happened before', 'happened overlap', 'happened after'), but the scoring procedure is not fully specified: the text mentions a dot-product between 'the contextual vectors of the masked tokens' and 'the embeddings of the predefined label words' without stating how multiple label-token embeddings are combined or normalized. Please clarify the exact scoring and whether the same two [MASK] positions are used for all classes.","section":"Materials and Methods, Hard-Prompting with Unfrozen LLMs"}],"minor_comments":[{"comment":"The word 'Decocer' appears in both architecture rows and should be 'Decoder'.","section":"Table 2"},{"comment":"The abstract refers to 'two variants of QLoRA adapted to GatorTron-Large', but Table 4 labels these rows as 'GatorTron-Large* + LoRA+' and 'GatorTron-Large* + LoRA++' without the QLoRA designation; please align the terminology.","section":"Abstract; Table 4"},{"comment":"The name 'Gatortron-Base' appears with a lowercase 't' in one place; use 'GatorTron' consistently throughout.","section":"Employed Large Language Models (LLMs)"},{"comment":"The sentence beginning 'The second PEFT strategy, QLoRA with frozen LLMs utilizing standard fine-tuning, Instead of generating a textual label...' is grammatically incomplete; 'Instead' should begin a new sentence.","section":"Parameter-Efficient Fine-Tuning Strategies for Transformer Decoder-based Models"},{"comment":"The 85.70% SOTA value is cited to both [10,11], but the Related Works section reports [10] as achieving 82.04%; please attribute the 85.70% figure to [11] and reconcile the citation.","section":"Background; Results"},{"comment":"The claim that the i2b2 dataset was not used in GatorTron pretraining is stated without supporting evidence or a data-contamination analysis; please add a citation or an explicit leakage check.","section":"Discussions and Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the empirical matrix is potentially useful, but the headline result depends on an evaluation protocol that is not described and cannot be audited from the manuscript alone. I recommend that the editor require the authors to provide the evaluation script and trained-model predictions during revision, and to report multi-seed variance for the claimed SOTA improvement. The citation inconsistency around [10,11] and the missing validation-set description should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: if the 89.54% F1 holds up under identical evaluation, this is a legitimate new state of the art on a standard clinical NLP benchmark. But the paper doesn't yet provide enough to verify that. The methods are standard prompt tuning and PEFT; the new part is the empirical result and the breadth of the comparison.\n\nWhat's genuinely useful: the paper runs a structured comparison across four models (GatorTron-Base/Large, LLaMA3-8B, MeLLaMA-13B) and eight fine-tuning variants under both fully supervised and few-shot settings. That is a useful map for practitioners. The finding that hard-prompting on the small GatorTron-Base beats large frozen models, and that few-shot trends invert relative to fully supervised, is clearly described and interesting. The authors are honest that only the i2b2 dataset was used.\n\nThe soft spots are about verification. The headline gain over the 85.70% SOTA depends on the assumption that the comparison used the same test notes, preprocessing, and micro-F1 scoring. The paper says it followed Chen et al.'s preprocessing but does not describe it or show that the cited SOTA papers used the same pipeline. No evaluation script, prediction files, or trained models are released. No held-out validation set or repeated runs are reported, and Table 4 lists 19 model variants, so the best variant may have been selected on test performance. These are real concerns, but they are fixable; they weaken the strength of the claim, not the plausibility of the result. A single run of a 345M model will vary by maybe a point; the 3.74% gap is larger than typical noise, which helps, but the protocol-matching issue is not resolved by that.\n\nThe paper is a solid empirical study in its subfield, not a methodological breakthrough. The citation pattern is fine; self-citations are to the authors' prior prompt-tuning work and do not create circularity.\n\nRecommendation: send to peer review. A serious referee should ask for code/data release, a validation-set protocol, repeated runs, and a direct comparison against the prior SOTA's own preprocessing and scorer before accepting 89.54% as the new SOTA.","headline":"A plausible new SOTA on the i2b2 CTRE benchmark, but the 3.74% gain over prior work is unverified until the evaluation protocol is shown to match.","tokens_in":13477,"tokens_out":2073,"would_cite":true,"duration_ms":19813,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hard-prompt fine-tuning of a 345M-parameter clinical encoder sets a new state of the art for clinical temporal relation extraction, reaching 89.54% micro F1 against the prior 85.70%.","keywords":["clinical temporal relation extraction","large language models","hard prompting","prompt tuning","parameter-efficient fine-tuning","GatorTron","few-shot learning","2012 i2b2 temporal relation task"],"falsifier":"Run the same hard-prompting recipe on the 2012 i2b2 clinical temporal relation extraction data using the exact preprocessing of the prior baseline and the same official evaluation script; if the micro F1 drops to or below 85.70%, the central claim collapses. A simpler audit would confirm whether the reported 89.54% is reproducible from the released code and whether the test-set relation distribution matches the standard 27,736 relations split.","tokens_in":12469,"feed_emoji":"🩺","tokens_out":5255,"duration_ms":47812,"temperature":0.7,"pith_summary":"The paper tries to show that prompt-based fine-tuning of clinical large language models can push clinical temporal relation extraction past the prior state of the art on the standard 2012 i2b2 temporal-relations benchmark. Its main finding is that GatorTron-Base, a 345M-parameter clinical encoder, fine-tuned with a hard-prompt template that reformulates the task as mask filling, reaches 89.54% micro F1, beating the previous best of 85.70% by 3.74 points. The paper also reports that quantized LoRA variants of an 8.9B GatorTron model exceed the prior best, and that in few-shot settings frozen encoder models outperform frozen decoder models. If the comparison holds under identical preprocessing and evaluation, the result offers a practical recipe for improving timeline reconstruction from electronic health records without requiring the largest models.","feed_headline":"Hard-prompting lifts clinical temporal-relation score by 3.7 points","feed_subtitle":"A 345M-parameter clinical encoder with mask-filling prompts hits 89.54% micro F1 on the 2012 i2b2 temporal-relation task.","key_machinery":"The load-bearing object is the hard-prompt reformulation of clinical temporal relation extraction as masked language modeling. Input text is appended with the template \"Subject [MASK] [MASK] Object,\" and each temporal relation is mapped to label words, with BEFORE mapped to \"happened before,\" OVERLAP to \"happened overlap,\" and AFTER to \"happened after.\" The model predicts the masked positions, and classification is the dot product between the contextual vectors of the masked tokens and the embeddings of the label words, with the highest-scoring label word mapped back to the relation. This lets an encoder LLM perform relation classification through its pretraining objective rather than through a separate classification head. Soft prompting, LoRA, and QLoRA serve as comparison strategies, but the hard-prompt masked-language-modeling step is what carries the reported gain over standard fine-tuning.","core_discovery":"On its own terms, the central claim is that hard-prompting with an unfrozen GatorTron-Base sets a new state of the art for clinical temporal relation extraction: micro F1 of 89.54% on the 2012 i2b2 task, 3.74% higher than the previous SOTA of 85.70%. The claim is grounded in a systematic comparison of six fine-tuning strategies across encoder and decoder large language models. In the fully supervised setting, the top four variants are all GatorTron-based, including standard fine-tuning (85.95%) and two QLoRA configurations on GatorTron-Large (88.06% and 87.58%), while decoder models top out below the SOTA. In few-shot settings, the paper reports that performance rises from 1-shot to 4-shot and then plateaus, and that frozen encoder models beat frozen decoder models, reversing the fully supervised trend. The paper attributes the encoder advantage to GatorTron's pretraining on a large de-identified clinical corpus and to the suitability of encoder architectures for classification.","pith_inferences":["If the hard-prompt result replicates, the same \"verbalize the label, fill the mask\" trick is a plausible drop-in for other clinical relation extraction tasks whose relation labels can be turned into short phrases; the authors themselves note that prompts must be redesigned per task.","The reported few-shot plateau at 4 examples is consistent with prompt-based methods needing only a tiny anchor set; a testable extension would be to check whether the same plateau shape appears on other relation extraction benchmarks.","Because the prior SOTA was set on the same benchmark with a different model family, the paper's protocol relies on cross-paper comparability; an independent reproduction with a shared evaluation harness would settle whether the 3.74-point gap is real or partly an artifact of scoring differences."],"forward_implications":["A 345M-parameter clinical encoder with hard prompting outperforms the prior best model on the 2012 i2b2 clinical temporal relation extraction benchmark by 3.74 micro-F1 points.","Parameter-efficient QLoRA on the 8.9B GatorTron also exceeds the prior SOTA, so strong CTRE can be achieved while keeping most weights frozen.","In few-shot regimes, frozen encoder models are the better choice, and adding examples beyond 4 shots yields no significant further gain.","The choice of fine-tuning strategy matters more than raw model size for this classification task: the best model is the smallest encoder, not the largest decoder."],"supporting_citations":[{"why":"The previous state-of-the-art result of 85.70% that the paper's best model must beat.","marker":"[11]"},{"why":"The GatorTron encoder LLMs that carry the top results, pretrained on a large clinical corpus.","marker":"[21]"},{"why":"The preprocessing methodology the paper adopts, so the comparison with prior work depends on it.","marker":"[35]"},{"why":"The 2012 i2b2 temporal relations corpus used for training and test.","marker":"[38]"},{"why":"LoRA, the low-rank adaptation strategy used for parameter-efficient baselines.","marker":"[39]"},{"why":"QLoRA, the quantized LoRA strategy used for the larger frozen models.","marker":"[41]"},{"why":"The earlier prompt-tuning architecture for GatorTron on other clinical extraction tasks that this study extends to CTRE.","marker":"[17]"}],"fun_headline_variants":["Hard-prompting clinical LLMs sets new SOTA for temporal relations","GatorTron with hard prompts beats SOTA by 3.74% F1","Unfrozen hard prompts lift clinical temporal extraction to 89.54%","Encoder LLM with hard prompts wins clinical temporal task","Hard-prompting beats fine-tuning for clinical temporal relations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed lead over the previous state of the art depends on the new runs sharing the same test split, preprocessing, and scoring as the 85.70% baseline, but the paper does not specify that preprocessing in the main text or provide an evaluation script; it also does not describe a held-out validation set, so the winning variant may have been selected on the test set.","fun_headline_variants_meta":{"raw":{"variants":["Hard-prompting clinical LLMs sets new SOTA for temporal relations","GatorTron with hard prompts beats SOTA by 3.74% F1","Unfrozen hard prompts lift clinical temporal extraction to 89.54%","Encoder LLM with hard prompts wins clinical temporal task","Hard-prompting beats fine-tuning for clinical temporal relations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2302,"prompt_tokens":1201,"completion_tokens":1101,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":817,"completion_tokens_details":{"reasoning_tokens":1007}},"tokens_in":817,"tokens_out":1101,"duration_ms":8095,"temperature":1.0,"reasoning_tokens":1007,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:16:50.901954+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same hard-prompting recipe on the 2012 i2b2 clinical temporal relation extraction data using the exact preprocessing of the prior baseline and the same official evaluation script; if the micro F1 drops to or below 85.70%, the central claim collapses. A simpler audit would confirm whether the reported 89.54% is reproducible from the released code and whether the test-set relation distribution matches the standard 27,736 relations split.","supporting_citations":[],"review_version":1}