REVIEW 1 major objections 7 minor 22 references
Converting Annotated Clinical Cases into Structured Case Report Forms
T0 review · 1 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims a general semi-automatic methodology that converts annotated clinical case corpora into gold-standard-filled Case Report Forms, with zero-shot GPT-4o reaching 59.7% micro F1 in Italian and 67.3% in English on the…
desk verdict Useful new CRF resource from E3C, but test item construction leaks test annotations into the benchmark and the reported F1 needs a fix before it is a clean result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-step conversion pipeline. First, clinical cases are clustered using a graph whose edge weight is $s=3d+e$ (for E3C, $s=3d+\frac{1}{2}(e+b)$), where $d$ is diagnosis similarity computed from UMLS-augmented embeddings and $e,b$ are ratios of shared clinical entities and body parts; Louvain community detection over weakly connected components forms the groups. Second, each group receives a CRF generated by collecting annotation-derived items per section, normalizing them via UMLS and manual review, and filling each case from its annotations. The E3C adaptation creates diagnosis, history, and exam items from the corpus's entity, event, relation, and attribute annotations.
What would settle it
Regenerate the E3C CRF item sets using only training-split annotations, rerun the same zero-shot GPT-4o prompts on the test cases, and compare micro F1 to the reported 59.7% (Italian) and 67.3% (English); a substantial drop would show the current scores are inflated by item-set leakage.
Extended reading notes
Core claim
The central claim is that a clinically meaningful CRF benchmark can be produced from an existing annotated corpus without designing forms from scratch. The pipeline clusters clinical notes by diagnosis-weighted semantic similarity, creates one group-specific CRF item set per cluster, and fills each case's CRF from its annotations to produce gold labels. Applied to E3C, this yields three sub-tasks on which zero-shot GPT-4o is the strongest tested model, reaching 59.7% micro F1 (Italian) and 67.3% (English); exam-result generation is the hardest sub-task, and models score about 7.5 micro-F1 points higher on English than on Italian on average.
Load-bearing premise
The load-bearing premise is that generating the group-specific CRF item sets from the whole corpus, including test cases, does not bias the evaluation; if that assumption fails, the reported F1 scores overstate how well the models can fill CRFs for unseen notes.
Editorial extensions
If this is right
- Other annotated clinical corpora can be converted into CRF benchmarks with the same pipeline, lowering the cost of obtaining task-specific forms without manual design.
- The released dataset gives a multilingual public testbed for CRF slot filling, allowing future models to be compared on identical items and gold fillings.
- The strongest models still lag on exam-result extraction, so the benchmark isolates a generative extraction skill rather than only classification.
- The simplified three-value history schema keeps the dataset usable across languages, while the underlying E3C annotations still allow the full twelve-value schema to be restored.
Reading between the lines
- Because group-specific CRFs are built from all cases including test cases, the test items are partly determined by test annotations; a train-only CRF version would likely yield lower scores and would test item generalization more strictly.
- With about 90% of CRF items left unfilled, micro F1 is dominated by correct 'not available' answers, so the task-level precision and recall the paper reports are the more informative comparison.
- A natural next experiment is to fine-tune an open-weight model on the training split and measure the gap to zero-shot GPT-4o, since all reported numbers are from zero-shot prompting.
- The methodology's portability to other datasets depends on matching the available annotation types to CRF sections; the paper itself notes that treatment items were excluded because E3C lacks treatment annotations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a semi-automatic methodology for converting information-extraction-annotated clinical case corpora into structured, filled Case Report Forms (CRFs). The method clusters clinical notes by semantic similarity (with diagnosis as the primary dimension), generates a group-specific CRF item set per cluster, and populates it from the original annotations. The methodology is applied to the E3C corpus in English and Italian, and the resulting dataset is released publicly. The authors evaluate several LLMs in a zero-shot setting, reporting that GPT-4o achieves 59.7% micro F1 on Italian and 67.3% on English, with open-weight models performing worse, and conclude that CRF slot filling remains challenging.
Significance. If the evaluation is valid, the released dataset constitutes a new public benchmark for CRF filling, a task for which annotated datasets are scarce. The methodology itself is a useful contribution and is described with enough detail to be adapted to other IE corpora. The paper is transparent about many construction choices and releases the data on Hugging Face, which supports reproducibility. However, the headline F1 scores rest on an evaluation protocol whose validity is questionable (see major comments), so the benchmark's value as a reliable measurement tool is not yet established.
major comments (1)
- [§3.1 and §5.1 (diagnosis extraction, item normalization)] GPT-4o is used both during dataset construction (for diagnosis selection in §3.1 and for the 'close source Large Language Model' that proposes item mappings during data revision in §3.2) and as the best-performing evaluation model. This creates a potential circularity: the gold answers and item phrasing are shaped by the same model family that is later evaluated against them, which could inflate GPT-4o's scores relative to open-weight models that had no role in dataset construction. The manual review of 10 examples is a positive step but does not rule out systematic favoritism. The authors should discuss this risk and, if feasible, report an additional evaluation using a model not involved in dataset construction, or at least quantify the overlap between GPT-4o's construction outputs and its test predictions.
minor comments (7)
- [Abstract] The text reads 'We release the datest at ...' — 'datest' should be 'dataset'.
- [§3.1 vs. §5.1] The similarity formula is given as s = 3d + e in §3.1 (Eq. 1) but as s = 3d + 1/2(e + b) in §5.1. The relationship between the general formulation and the E3C-specific adaptation should be clarified, including the role of the body-part ratio b.
- [Table 3] The 'Filled' column percentages are not clearly tied to the item counts; adding a note on how 'filled' is defined (e.g., value different from 'not available') would improve readability.
- [Table 8] Some entries have inconsistent spacing (e.g., '13/ 9.7' and '11 / 9.7'), and the table would benefit from a footnote explaining the units for mean and standard deviation.
- [§4 and §6] The phrase 'closed Large Language Models (zero-shot)' is awkward; 'closed' likely means proprietary, and the sentence could be rephrased for clarity.
- [Appendix A.2 Table 6] The CRF value for 'Finite' permanence is listed as 'Certainly not chronic', which is confusing because a finite condition is not necessarily 'certainly not chronic'; consider rewording or adding an example.
- [§7 (Results)] No confidence intervals or significance tests are reported for the differences between models. Given the small test sizes, a statement about variability would help readers assess the robustness of the ranking.
Circularity Check
Test CRF items are generated from all cases (including test cases), so exam and diagnosis questions are defined from the same annotations that provide the gold answers.
-
self definitional
[Section 5.1, 'Train-test split', 'Exams', and 'Diagnosis']
"group-specific CRFs are generated on all the cases in the corpus. ... constructing comprehensive item sets from the full dataset is necessary and does not introduce bias beyond the task’s inherent structure. ... creating clinical history item sets ... excluding from the final set the ones filled only for test cases after data revision. ... A CRF exam item was created for each textual span with a corresponding RML, representing its filling value. ... For each diagnosis, an item was created and populated with 'yes' if it applied to the clinical case and 'not available' otherwise."
The test CRF item sets are built from annotations of all cases, including test cases. Exam items are one-to-one with textual spans carrying an RML (the value the model must predict), and diagnosis items are one-to-one with diagnoses selected from the case annotations. An annotation appearing only in a test case therefore generates a test item whose gold value is exactly that annotation. The authors exclude test-only items for clinical history but do not describe the same exclusion for exams or diagnosis; the history filter shows they regard test-only items as a contamination source.
full rationale
The claimed headline results (GPT-4o micro-F1 59.7 on Italian, 67.3 on English) are computed on test items whose sets are generated from the full corpus, including the test cases themselves. For exams, every item is created from a textual span linked to an RML, i.e., from the annotation that also supplies the gold value; for diagnosis, every item comes from a diagnosis selected from the case annotations. The authors explicitly filter out test-only items for clinical history but not for exams or diagnosis, so the test questions are partly selected from the gold answers. This is a genuine reduction by construction, but it does not make the model outputs tautological: the model still has to read the clinical note and decide on a value, and the paper's claim that CRF filling is challenging retains independent content. The train-test split comes from Ghosh et al. (2025), whose author list overlaps with this paper, but using an existing split is ordinary practice and is not load-bearing evidence. GPT-4o was also used in diagnosis selection and item mapping, but labels were manually reviewed, and the model is evaluated zero-shot, so that overlap is not itself a fitted-input circularity. Overall score 5: one central benchmark-construction step is definitionally coupled to the gold labels, but the quantitative conclusion is not forced.
Assumptions & free parameters
free parameters (3)
- Similarity weight for diagnosis (d) =
3
- Similarity weight for entities and body parts (e and b) =
0.5
- Maximum UMLS related terms per mention =
5
assumptions (4)
- domain assumption E3C annotations are accurate and sufficient to define gold-standard CRF values.
- domain assumption The seven CRF areas observed in 200 hospital CRFs generalize to the E3C clinical cases.
- domain assumption Clustering based on diagnosis similarity produces clinically coherent CRF item sets.
- domain assumption GPT-4o diagnosis extraction is accurate enough for group construction.
Cite this review
Pith. "Pith review of Converting Annotated Clinical Cases into Structured Case Report Forms." pith.science (2026). https://pith.science/paper/Q62HNRZV
@misc{pith2026250611666,
author = {Pith},
title = {Pith review of: Converting Annotated Clinical Cases into Structured Case Report Forms},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q62HNRZV}},
note = {Machine review of arXiv:2506.11666}
}
read the original abstract
Case Report Forms (CRFs) are largely used in medical research as they ensure accuracy, reliability, and validity of results in clinical studies. However, publicly available, wellannotated CRF datasets are scarce, limiting the development of CRF slot filling systems able to fill in a CRF from clinical notes. To mitigate the scarcity of CRF datasets, we propose to take advantage of available datasets annotated for information extraction tasks and to convert them into structured CRFs. We present a semi-automatic conversion methodology, which has been applied to the E3C dataset in two languages (English and Italian), resulting in a new, high-quality dataset for CRF slot filling. Through several experiments on the created dataset, we report that slot filling achieves 59.7% for Italian and 67.3% for English on a closed Large Language Models (zero-shot) and worse performances on three families of open-source models, showing that filling CRFs is challenging even for recent state-of-the-art LLMs. We release the datest at https://huggingface.co/collections/NLP-FBK/e3c-to-crf-67b9844065460cbe42f80166
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
- [3]
-
[4]
Emma Chiaramello, Francesco Pinciroli, Alberico Bonalumi, Angelo Caroli, and Gabriella Tognola. 2016. https://doi.org/10.1016/j.jbi.2016.07.017 Use of “off-the-shelf” information extraction algorithms in clinical informatics: A feasibility study of MetaMap annotation of Italian medical notes . Journal of Biomedical Informatics, 63:22--32. @inproceedings g...
-
[5]
Aaron Grattafiori et al. 2024. https://arxiv.org/abs/2407.21783 The Llama 3 herd of models . Preprint, arXiv:2407.21783
arXiv 2024
-
[6]
Robert Fleischmann, Anne-Marie Decker, Antje Kraft, Knut Mai, and Sein Schmidt. 2017. https://doi.org/10.1186/s12874-017-0429-y Mobile electronic versus paper case report forms in clinical trials: a randomized controlled trial . BMC Medical Research Methodology, 17(1):153
-
[7]
Soumitra Ghosh, Begona Altuna, Saeed Farzi, Pietro Ferrazzi, Alberto Lavelli, Giulia Mezzanotte, Manuela Speranza, and Bernardo Magnini. 2025. Low-resource information extraction with the European Clinical Case Corpus . ArXiv preprint
work page 2025
-
[8]
Alba Gutiérrez-Sacristán, Simran Makwana, Audrey Dionne, Simran Mahanta, Karla J. Dyer, Faridis Serrano, Carmen Watrin, Pierre Pages, Sajad Mousavi, Anil Degala, Jessica Lyons, Danielle Pillion, Joany M. Zachariasse, Lara S. Shekerdemian, Dongngan T. Truong, Jane W. Newburger, and Paul Avillach. 2024. https://doi.org/10.1016/j.ebiom.2024.105337 Developmen...
Show all 22 references
-
[9]
Gonzalez, Hao Zhang, and Ion Stoica
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the ACM SIGOPS 29th Symposium on Operating ...
2023
-
[10]
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. https://arxiv.org/abs/2405.17428 NV-Embed : Improved techniques for training LLMs as generalist embedding models . Preprint, arXiv:2405.17428
2024 arXiv
-
[11]
Ching-Heng Lin, Nai-Yuan Wu, and Der-Ming Liou. 2015. https://doi.org/10.1016/j.jbi.2014.08.013 A multi-technique approach to bridge electronic case report form design and data standard adoption . Journal of Biomedical Informatics, 53:49--57
2015 doi
-
[12]
Hao Lu, Mahantesh Halappanavar, and Ananth Kalyanaraman. 2015. https://doi.org/10.1016/j.parco.2015.03.003 Parallel heuristics for scalable community detection . Parallel Computing, 47:19--37
2015 doi
-
[13]
W. R. Mac Kenzie, A. J. Davidson, A. Wiesenthal, J. P. Engel, K. Turner, L. Conn, S. J. Becker, S. Moffatt, S. L. Groseclose, J. Jellison, J. Stinn, N. Y. Garrett, L. Helmus, B. Harmon, C. L. Richards, J. R. Lumpkin, and M. F. Iademarco. 2016. https://doi.org/10.1177/003335491...
2016 doi
-
[14]
Bernardo Magnini, Bego \ n a Altuna, Alberto Lavelli, Anne-Lyse Minard, Manuela Speranza, and Roberto Zanoli. 2023. https://doi.org/10.1007/978-3-031-17258-8_17 European Clinical Case Corpus , pages 283--288. Springer International Publishing, Cham
2023 doi
-
[15]
National Library of Medicine (US) . 2024. http://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html UMLS knowledge sources [dataset on the internet]. release 2024aa . Cited 2025 March
2024
-
[16]
OpenAI and Josh Achiam et al. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774
2024 arXiv
-
[17]
Frank Pétavy et al. 2019. https://www.appliedclinicaltrialsonline.com/view/global-standardization-clinical-research-data Global standardization of clinical research data . Applied Clinical Trials, 28(4):20--23
2019
-
[18]
Qwen and An Yang et al. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115
2025 arXiv
-
[19]
Richesson and Prakash Nadkarni
Rachel L. Richesson and Prakash Nadkarni. 2011. https://doi.org/10.1136/amiajnl-2011-000107 Data standards for clinical research data collection forms: current status and challenges . Journal of the American Medical Informatics Association, 18(3):341--346
2011 doi
-
[20]
Eugenia Rinaldi, Caroline Stellmach, and Sylvia Thun. 2025. https://doi.org/10.2196/51598 How to design electronic case report form (ecrf) questions to maximize semantic interoperability in clinical research . Interactive Journal of Medical Research, 14:e51598
2025 doi
-
[21]
Dongkuan Xu and Yingjie Tian. 2015. https://doi.org/10.1007/s40745-015-0040-1 A comprehensive survey of clustering algorithms . Annals of Data Science, 2(2):165--193
2015 doi
-
[22]
Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. https://arxiv.org/abs/2412.19048 Jasper and Stella : distillation of SOTA embedding models . Preprint, arXiv:2412.19048
2025 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.