REVIEW 3 major objections 4 minor
Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper establishes that MedRoBERTa.nl, a Dutch transformer pre-trained on electronic health records, detects adverse drug events in Dutch clinical free text better than a Bi-LSTM baseline and three other transformer models, reaching a…
desk verdict A careful, honest benchmark for Dutch ADE detection; the MedRoBERTa.nl ranking looks credible, but the external recall numbers rest on a 4-sentence window assumption that isn't verified on the external corpora. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-stage pipeline whose second stage never trains its own transformer. A fine-tuned encoder (BERTje, RobBERT, MedRoBERTa.nl, or NuNER-multilingual, each with a conditional random field layer) predicts drug and disorder spans; the same encoder's contextual embeddings for the [CLS] token, for the candidate drug, and for the candidate disorder, plus the NER label probabilities, are concatenated into a 3,088-dimensional feature vector and fed to a four-layer multi-layer perceptron that classifies the pair as ADE or not. Candidate pairs are limited to drug-disorder mentions at most four sentences apart, a window chosen because it covers 99.42 percent of gold ADE relations in the primary corpus, and class imbalance is handled by SMOTE oversampling plus random undersampling. Thresholds are chosen from precision-recall curves on the validation set under F1 and F2 objectives, and ADE groups let the authors evaluate both 'easy' success, where any mention of a unique clinical ADE is found, and 'hard' success, where every mention of the same clinical ADE is found.
What would settle it
Annotate the sentence distance between drug and disorder for every gold ADE relation in the WINGS discharge letters and the ICU AKI notes. If a substantial share of relations span more than four sentences, the reported 0.67 to 0.74 external recalls are pessimistic ceilings imposed by the candidate window rather than true end-to-end ability, and the benchmark would need re-running with a longer window or with document-level aggregation.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that ADE detection in Dutch clinical free text is best treated as two linked subtasks, recognizing drug and disorder mentions and then classifying drug-disorder pairs as ADE, prescribing indication, or neither, and that both can be served by one domain-pretrained encoder. MedRoBERTa.nl achieves the highest macro-averaged F1 on the ADE relation task (0.6301 with gold entities and 0.6222 end-to-end), and the same model detects the presence of an ADE in 67 to 74 percent of external discharge letters, outperforming the next best model by a clear margin (0.681 versus 0.622 F1 under an F1-optimised threshold). The paper further claims that evaluation choices hide this result: micro-averaged F1 puts all models near 0.98 because the majority 'no relation' class dominates, and it recommends macro-averaged F1, precision-recall curves, and F2-based threshold selection as the appropriate measures when missing an ADE is costlier than a false alarm. The authors also acknowledge that part of MedRoBERTa.nl's internal advantage may come from pre-training on notes that overlap the training corpus, and they lean on the external validation to argue the model generalises rather than overfits.
Load-bearing premise
The candidate-pair generator links a drug and a disorder only when they lie within four sentences of each other, a window chosen on the training corpus because it covers 99.42 percent of its gold ADE relations; the same fixed window is applied to the external discharge letters and ICU notes without checking whether their ADE relations also fall inside it, so if external relations span longer distances every model's document-level recall is capped below its true potential.
Editorial extensions
If this is right
- Macro-averaged F1, not the near-perfect micro-averaged F1, is the number that describes real ADE detection performance; on that measure all models land between 0.58 and 0.63, so the task remains hard.
- Choosing thresholds with the F2 score roughly raises MedRoBERTa.nl's retrieval of complete ADE groups (37 to 60 percent of groups detected in the easy setting), at the price of more false positives.
- The winning model transfers across document type, specialty, and hospital: discharge letters from two non-academic internal medicine wards are detected at 67 to 74 percent recall, so the approach is applicable to hospital-wide medication safety monitoring and not just the training ICU.
- One fine-tuned encoder can serve both named entity recognition and relation classification because the relation head reuses its embeddings, avoiding the cost of training a second transformer.
- Document-flagging for manual causality assessment, the workflow matched by the 'easy' ADE-group evaluation, appears deployable now: MedRoBERTa.nl retrieves 60 percent of ADE groups at the F2 threshold, a large gain over the estimated 6 to 33 percent of ADEs currently captured in structured form.
Reading between the lines
- The four-sentence pairing window is carried unchanged from the training corpus to the external corpora; measuring the actual sentence distances of gold ADE relations in the WINGS and ICU AKI notes would show whether the reported recalls are true end-to-end performance or a ceiling imposed by the candidate generator.
- Because the relation head is a small multi-layer perceptron over reused embeddings, adding another relation type or another rare clinical event (falls, pressure ulcers, delirium) would cost little; the same pipeline pattern could be lifted to other rare-event monitoring tasks in clinical text.
- The paper's deliberate exclusion of decoder-only large language models is a testable bet: a reader could run the identical two-stage evaluation with a generative model and check whether the encoder-only design, not just the domain pre-training, is what carries the advantage in low-resource Dutch clinical settings.
- The easy-versus-hard ADE-group evaluation mirrors two real workflows, triage and comprehensive registration; future work could report cost-weighted operating points, such as the number of notes needing pharmacist review per confirmed ADE, to turn precision-recall curves into staffing numbers.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks five models (Bi-LSTM, BERTje, RobBERT, MedRoBERTa.nl, NuNER-multilingual) for adverse drug event (ADE) detection in Dutch clinical free text. Using 102 richly annotated ICU progress notes as the primary corpus, the authors train and evaluate named entity recognition (NER) and relation classification (RC) models in both a two-step (gold entities) and an end-to-end (predicted entities) setting, reporting micro- and macro-averaged F1, precision-recall curves, and F1/F2 threshold selection. They further evaluate document-level ADE detection on two additional corpora: an ICU AKI corpus and the WINGS corpus of 100 discharge letters from non-academic hospitals, the latter serving as external validation. The central claim is that MedRoBERTa.nl performs best overall, with macro-averaged F1 of 0.63 (gold) and 0.62 (predicted) on the primary corpus, and external document-level recall of 0.67-0.74 on WINGS discharge letters.
Significance. If the results hold, this is a useful benchmark: it is, to the authors' knowledge, the first comparison of transformer models for ADE detection in Dutch clinical free text, and one of the few ADE NLP studies with external validation on a different document type and hospital setting. The evaluation design is careful in several respects: it reports macro-averaged F1 alongside micro-averaged F1 in a highly imbalanced setting, uses precision-recall curves and F2-based threshold selection, evaluates at both relation and ADE-group levels, and explicitly discloses the potential pre-training overlap between MedRoBERTa.nl and the primary corpus. The external WINGS validation, where MedRoBERTa.nl wins by a larger margin, is an important piece of evidence that the internal result is not solely due to data leakage. The main limitation is that the external document-level recall figures depend on a candidate-pair window whose adequacy is only demonstrated on the primary corpus, and on a data-split description that is inconsistent between the main text and the supplement.
major comments (3)
- [Supplement 1 (Relation Classification models; ICU AKI and WINGS dataset experiments)] The 4-sentence window used to generate candidate drug-disorder pairs is justified only on the primary corpus: the supplement states 'We chose 4 sentences as it covered 99.42% of all ADE relations in the gold-standard ADE corpus.' The same window is then applied unchanged to the ICU AKI and WINGS corpora ('the same way as we did for other experiments'). The paper does not report the distribution of relation distances in these external corpora. If ADE relations in discharge letters or AKI notes span more than four sentences, the document-level recall reported in Table 4 is capped below its true value for every model, and the external validation becomes a test of the window assumption rather than of the models. This directly affects the abstract's headline claim that 67-74% of discharge letters with ADEs were detected. Please measure and report the distance coverage of the 4-sentence window in both external corpora, or use a window/aggregation strategy that does not cap recall, and re-report the affected results if needed.
- [Methods (Experimental set up) vs. Supplement 1 (Experimental Design)] The data split proportions are described inconsistently. The main text says 'we used 5-fold cross validation with 60/20/20 train/validation/test percentual splits,' while Supplement 1 says 'We used 5-fold cross validation with 80/10/10 train/validation/test percentual splits.' These are incompatible, and neither is the usual description of a 5-fold cross-validation split (which would typically use 80% of the data for training and 20% for testing, with a further split for validation). Please clarify the exact split, ensure the main text and supplement agree, and describe how the five folds relate to the train/validation/test partitions.
- [Methods (Datasets)] The ICU AKI corpus is used for a full set of document-level experiments in Table 3, but its size is never reported: the dataset description does not state the number of notes or the number of ADE-positive documents. Without this information, the reader cannot assess the reliability of the precision, recall, and specificity estimates in Table 3, which are averaged over the five model folds. Please add the corpus size, the number of ADE-positive notes, and the annotation statistics for this corpus.
minor comments (4)
- [Methods (Part 1)] The text contains a literal placeholder '[ADD CITATION]' after 'BIO format' in the description of the experimental setup; this should be replaced with the actual citation.
- [Table 2] The model name is typoed as 'B-iLSTM' in the table header; it should be 'Bi-LSTM'.
- [References] Reference [23] ('Transformer-Based Models for Named Entity Recognition: A Comparative Study') lacks author names and publication details; please complete the citation.
- [General] The term 'end-to-end task' is used both for relation classification with predicted entities on the primary corpus and for document-level ADE detection on the external corpora; consider distinguishing these, e.g., 'end-to-end RC' versus 'document-level detection,' to avoid ambiguity.
Circularity Check
No significant circularity: the benchmark is an empirical comparison, and the central claim is independently supported by external validation on the WINGS corpus.
full rationale
This paper is an empirical benchmark study rather than a derivation chain, so the main circularity patterns do not apply. The strongest internal result (MedRoBERTa.nl macro-averaged F1 of 0.63 with gold entities and 0.62 with predicted entities) is accompanied by the authors' own disclosure that roughly half of the Dutch ADE corpus notes may have been included in MedRoBERTa.nl pretraining data. That disclosure is a data-leakage caveat, not a circular argument, and the paper provides independent external support: on the WINGS discharge-letter corpus, MedRoBERTa.nl outperforms the other models by a larger margin (F1 0.681 vs. 0.622 with the F1 threshold, and 0.736 vs. 0.680 with the F2 threshold). Self-citations to the Dutch ADE corpus creation [12], the scoping review [6], and prior AKI work [5] are sources of data and background; they are not load-bearing arguments that force the conclusions. The four-sentence candidate-pair window is a preprocessing choice justified on the primary corpus ('We chose 4 sentences as it covered 99.42% of all ADE relations in the gold-standard ADE corpus') and then applied unchanged to the ICU AKI and WINGS experiments. If external ADE relations span longer distances, the reported external document-level recall would be capped for all models; that is a generalizability and validity limitation, not circularity, because the reported recall is an empirical measurement and the paper does not derive an external relation-distance distribution from the primary corpus. The paper's limitations section notes the small training corpus and the sparse annotations in external sets, but it does not explicitly flag the unverified distance assumption; under the review rule, this missing support is weighed here as a correctness risk rather than as evidence of circularity. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is repackaged under new coordinates. The central quantitative claim therefore does not reduce to its own inputs by construction.
Assumptions & free parameters
free parameters (4)
- F1-optimized and F2-optimized classification thresholds =
chosen per fold on validation PR curves
- SMOTE/undersampling positive-to-negative ratio =
0.4
- Maximum sentence distance for candidate drug-disorder pairs =
4 sentences
- NER and RC hyperparameters =
e.g., lr 3e-5, batch 8, epochs 10, warmup 0.2, patience 3 (NER); lr 1e-6, batch 128, dropout 0.5, patience 20 (RC)
assumptions (4)
- domain assumption The Dutch ADE corpus and WINGS corpus annotations are correct gold standards for drugs, disorders, ADE relations, and ADE groups.
- domain assumption A binary document-level label is a sufficient proxy for ADE presence in external validation.
- ad hoc to paper Synthetic oversampling (SMOTE) produces training pairs that do not distort the relation classification task.
- ad hoc to paper The 4-sentence window for candidate pairs is sufficient for external documents.
Cite this review
Pith. "Pith review of Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study." pith.science (2026). https://pith.science/paper/PSL763JZ
@misc{pith2026250719396,
author = {Pith},
title = {Pith review of: Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study},
year = {2026},
howpublished = {\url{https://pith.science/paper/PSL763JZ}},
note = {Machine review of arXiv:2507.19396}
}
read the original abstract
In this study, we establish a benchmark for adverse drug event (ADE) detection in Dutch clinical free-text documents using several transformer models, clinical scenarios, and fit-for-purpose performance measures. We trained a Bidirectional Long Short-Term Memory (Bi-LSTM) model and four transformer-based Dutch and/or multilingual encoder models (BERTje, RobBERT, MedRoBERTa(.)nl, and NuNER) for the tasks of named entity recognition (NER) and relation classification (RC) using 102 richly annotated Dutch ICU clinical progress notes. Anonymized free-text clinical progress notes of patients admitted to the intensive care unit (ICU) of one academic hospital and discharge letters of patients admitted to Internal Medicine wards of two non-academic hospitals were reused. We evaluated our ADE RC models internally using the gold standard (two-step task) and predicted entities (end-to-end task). In addition, all models were externally validated for detecting ADEs at the document level. We report both micro- and macro-averaged F1 scores, given the dataset imbalance in ADEs. Although differences for the ADE RC task between the models were small, MedRoBERTa(.)nl was the best performing model with a macro-averaged F1 score of 0.63 using the gold standard and 0.62 using predicted entities. The MedRoBERTa(.)nl models also performed the best in our external validation and achieved a recall of between 0.67 to 0.74 using predicted entities, meaning between 67 to 74% of discharge letters with ADEs were detected. Our benchmark study presents a robust and clinically meaningful approach for evaluating language models for ADE detection in clinical free-text documents. Our study highlights the need to use appropriate performance measures fit for the task of ADE detection in clinical free-text documents and envisioned future clinical use.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.