{"id":"4cff2669-bf51-4b1e-bf27-a4d7b59ef5ed","arxiv_id":"2507.19396","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A benchmark of five models for detecting adverse drug events in Dutch clinical free text finds MedRoBERTa.nl best, with macro-F1 around 0.63 and external recall of 0.67 to 0.74.","lead":"This paper benchmarks five NLP models, from a Bi-LSTM to Dutch and multilingual transformers, for detecting adverse drug events in Dutch hospital notes and discharge letters. MedRoBERTa.nl performed best, finding between 67 and 74 percent of discharge letters that mention an adverse drug event.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 4-sentence candidate-pair window, tuned to the primary corpus, is applied unchanged to WINGS and ICU AKI; if external ADE relations span longer distances, the reported external recall is an artifact of that window.","rationale":"The reader's weakest assumption is the same one I consider most load-bearing: the 4-sentence candidate window is transferred from the Dutch ADE corpus to the external WINGS and ICU-AKI corpora without validating that it covers their ADE relations. The paper's central quantitative claims include external document-level recall of 0.67 to 0.74, and that number is conditional on the window. If external ADE relations are longer-range, the reported recall is a lower bound imposed by the pipeline, not a measure of the models' actual ability, and the external validation is less informative than presented. This concern is concrete, testable, and does not require access to model weights. I do not see a more serious threat to the central claim. The disclosed MedRoBERTa.nl pretraining overlap is a real limitation for the internal comparison, but the WINGS external validation is from different hospitals and document types, and the paper acknowledges the issue. The lack of released code and data hinders replication but is not a scientific flaw in the argument. Differences between models are sometimes small and no significance tests are reported, but the pattern is consistent across internal and external metrics, so I would not elevate that into the primary concern. If the proposed distance check shows no long-distance external relations, the concern is resolved and the external recall figures stand as reported. The reader's conditional verdict already captures this uncertainty, so I recommend no change to the verdict.","tokens_in":26276,"tokens_out":12684,"duration_ms":145812,"concrete_test":"Using the gold ADE annotations in the WINGS and ICU-AKI corpora, compute the sentence-distance distribution between each gold drug-disorder ADE relation, and report the fraction exceeding 4 sentences. Then re-run the end-to-end document-level detection pipeline with the 4-sentence cap removed (or with caps of 8, 16, and full note) and compare document-level recall and F1. If recall rises materially or the model ranking changes, the reported 0.67-0.74 external recall is a window artifact and should be re-reported; if recall is unchanged, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The external document-level results in Table 4 and the abstract's '67 to 74% of discharge letters with ADEs were detected' depend on a candidate-generation step whose only justification is measured on the primary corpus. Supplement 1 (Relation Classification models) states: 'We chose 4 sentences as it covered 99.42% of all ADE relations in the gold-standard ADE corpus.' The ICU AKI and WINGS experiments then create drug-disorder pairs 'the same way as we did for other experiments.' No distance distribution is reported for external ADE relations. Discharge letters from internal-medicine wards differ in document type and annotation density from ICU progress notes; if their drug and disorder mentions are more than 4 sentences apart, those ADE relations are never presented to the RC model, so no model can detect them. Document-level recall is then capped below its true value for every model, and the external validation becomes a test of the window assumption rather than of the models. This does not necessarily change the ranking, but it directly affects the central quantitative claim about external recall and the paper's contribution of externally validating ADE detection. The concern is testable and would be settled by measuring relation distances in the external corpora.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks five models (Bi-LSTM, BERTje, RobBERT, MedRoBERTa.nl, NuNER-multilingual) for adverse drug event (ADE) detection in Dutch clinical free text. Using 102 richly annotated ICU progress notes as the primary corpus, the authors train and evaluate named entity recognition (NER) and relation classification (RC) models in both a two-step (gold entities) and an end-to-end (predicted entities) setting, reporting micro- and macro-averaged F1, precision-recall curves, and F1/F2 threshold selection. They further evaluate document-level ADE detection on two additional corpora: an ICU AKI corpus and the WINGS corpus of 100 discharge letters from non-academic hospitals, the latter serving as external validation. The central claim is that MedRoBERTa.nl performs best overall, with macro-averaged F1 of 0.63 (gold) and 0.62 (predicted) on the primary corpus, and external document-level recall of 0.67-0.74 on WINGS discharge letters.","tokens_in":26540,"tokens_out":6156,"duration_ms":71142,"significance":"If the results hold, this is a useful benchmark: it is, to the authors' knowledge, the first comparison of transformer models for ADE detection in Dutch clinical free text, and one of the few ADE NLP studies with external validation on a different document type and hospital setting. The evaluation design is careful in several respects: it reports macro-averaged F1 alongside micro-averaged F1 in a highly imbalanced setting, uses precision-recall curves and F2-based threshold selection, evaluates at both relation and ADE-group levels, and explicitly discloses the potential pre-training overlap between MedRoBERTa.nl and the primary corpus. The external WINGS validation, where MedRoBERTa.nl wins by a larger margin, is an important piece of evidence that the internal result is not solely due to data leakage. The main limitation is that the external document-level recall figures depend on a candidate-pair window whose adequacy is only demonstrated on the primary corpus, and on a data-split description that is inconsistent between the main text and the supplement.","major_comments":[{"comment":"The 4-sentence window used to generate candidate drug-disorder pairs is justified only on the primary corpus: the supplement states 'We chose 4 sentences as it covered 99.42% of all ADE relations in the gold-standard ADE corpus.' The same window is then applied unchanged to the ICU AKI and WINGS corpora ('the same way as we did for other experiments'). The paper does not report the distribution of relation distances in these external corpora. If ADE relations in discharge letters or AKI notes span more than four sentences, the document-level recall reported in Table 4 is capped below its true value for every model, and the external validation becomes a test of the window assumption rather than of the models. This directly affects the abstract's headline claim that 67-74% of discharge letters with ADEs were detected. Please measure and report the distance coverage of the 4-sentence window in both external corpora, or use a window/aggregation strategy that does not cap recall, and re-report the affected results if needed.","section":"Supplement 1 (Relation Classification models; ICU AKI and WINGS dataset experiments)"},{"comment":"The data split proportions are described inconsistently. The main text says 'we used 5-fold cross validation with 60/20/20 train/validation/test percentual splits,' while Supplement 1 says 'We used 5-fold cross validation with 80/10/10 train/validation/test percentual splits.' These are incompatible, and neither is the usual description of a 5-fold cross-validation split (which would typically use 80% of the data for training and 20% for testing, with a further split for validation). Please clarify the exact split, ensure the main text and supplement agree, and describe how the five folds relate to the train/validation/test partitions.","section":"Methods (Experimental set up) vs. Supplement 1 (Experimental Design)"},{"comment":"The ICU AKI corpus is used for a full set of document-level experiments in Table 3, but its size is never reported: the dataset description does not state the number of notes or the number of ADE-positive documents. Without this information, the reader cannot assess the reliability of the precision, recall, and specificity estimates in Table 3, which are averaged over the five model folds. Please add the corpus size, the number of ADE-positive notes, and the annotation statistics for this corpus.","section":"Methods (Datasets)"}],"minor_comments":[{"comment":"The text contains a literal placeholder '[ADD CITATION]' after 'BIO format' in the description of the experimental setup; this should be replaced with the actual citation.","section":"Methods (Part 1)"},{"comment":"The model name is typoed as 'B-iLSTM' in the table header; it should be 'Bi-LSTM'.","section":"Table 2"},{"comment":"Reference [23] ('Transformer-Based Models for Named Entity Recognition: A Comparative Study') lacks author names and publication details; please complete the citation.","section":"References"},{"comment":"The term 'end-to-end task' is used both for relation classification with predicted entities on the primary corpus and for document-level ADE detection on the external corpora; consider distinguishing these, e.g., 'end-to-end RC' versus 'document-level detection,' to avoid ambiguity.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a clinical NLP journal and the experimental design is generally thoughtful. The main concerns are fixable: the 4-sentence window needs external validation, and the data-split inconsistency must be resolved. I would encourage the editor to ask the authors to provide the distance distribution for the external corpora, since this is the most direct way to settle whether the external recall figures are trustworthy. The missing ICU AKI corpus size is a smaller but still important reporting gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper does something genuinely useful: it gives the first benchmark of transformer models for ADE detection in Dutch clinical free text, and the first external validation of such models on Dutch discharge letters. The ADE group-level scoring and the use of macro-F1 and F2 thresholds over micro-F1 are sensible methodological additions. The experimental design is careful — 5-fold CV, two-step vs end-to-end, PR curves, threshold selection — and the authors disclose the likely MedRoBERTa.nl pre-training overlap with the primary corpus. Given that MedRoBERTa.nl wins by a larger margin on the external WINGS corpus, the central ranking holds up.\n\nThe soft spots are real but manageable. The biggest is the 4-sentence candidate-pair window. It is justified only on the primary corpus (covers 99.42% of ADE relations there) and then applied unchanged to the external corpora. If external ADE relations span longer distances, document-level recall is capped for every model, and the external validation becomes a test of the window rather than of the models. That directly affects the headline '67–74% recall' claim. It's testable — report relation-distance distributions in WINGS and ICU AKI and re-run with a larger window. I'd call this a moderate concern, not a fatal one; the ranking would likely survive.\n\nTwo lesser issues: the primary corpus is only 102 notes, so internal comparisons carry wide standard deviations (reported honestly), and no code or data is released. The privacy constraints on the clinical corpora are understandable, but the Dutch ADE corpus is at least available on request, and code would help.\n\nNet: this is a benchmark paper that deserves referee time. A solid referee would ask for the distance analysis and possibly a larger window sensitivity check. If that comes back clean, the paper is a useful baseline for Dutch clinical NLP and for ADE detection in non-English EHR text. My recommendation: send it to peer review.","headline":"A careful, honest benchmark for Dutch ADE detection; the MedRoBERTa.nl ranking looks credible, but the external recall numbers rest on a 4-sentence window assumption that isn't verified on the external corpora.","tokens_in":27077,"tokens_out":3036,"would_cite":true,"duration_ms":32149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that MedRoBERTa.nl, a Dutch transformer pre-trained on electronic health records, detects adverse drug events in Dutch clinical free text better than a Bi-LSTM baseline and three other transformer models, reaching a…","keywords":["adverse drug events","Dutch clinical free text","transformer models","named entity recognition","relation classification","external validation","class imbalance metrics","MedRoBERTa.nl"],"falsifier":"Annotate the sentence distance between drug and disorder for every gold ADE relation in the WINGS discharge letters and the ICU AKI notes. If a substantial share of relations span more than four sentences, the reported 0.67 to 0.74 external recalls are pessimistic ceilings imposed by the candidate window rather than true end-to-end ability, and the benchmark would need re-running with a longer window or with document-level aggregation.","tokens_in":26072,"feed_emoji":"💊","tokens_out":8887,"duration_ms":86931,"temperature":0.7,"pith_summary":"This paper establishes the first benchmark of transformer-based language models for adverse drug event (ADE) detection in Dutch clinical free text, using richly annotated intensive care unit progress notes for training and discharge letters from two other hospitals for external validation. The central finding is that MedRoBERTa.nl, a model pre-trained on Dutch electronic health records, performs best in every realistic setting: a macro-averaged F1 of 0.63 with gold entities, 0.62 end-to-end with predicted entities, and document-level recall of 0.67 to 0.74 on external discharge letters. The authors argue that the micro-averaged F1 scores near 0.98 that look excellent are an artifact of class imbalance, because adverse drug events make up only 0.64 percent of candidate drug-disorder pairs, and that macro-averaged F1 and precision-recall curves are the honest measures for this task. If the benchmark holds, a single fine-tuned Dutch clinical model could flag between two-thirds and three-quarters of discharge letters mentioning an ADE for pharmacist review, a large gain over the estimated 6 to 33 percent of ADEs currently captured in structured form.","feed_headline":"Catches 74% of drug-harm discharge letters with a Dutch clinical model","feed_subtitle":"First transformer benchmark for adverse drug events in Dutch clinical text, validated on real discharge letters.","key_machinery":"The load-bearing mechanism is a two-stage pipeline whose second stage never trains its own transformer. A fine-tuned encoder (BERTje, RobBERT, MedRoBERTa.nl, or NuNER-multilingual, each with a conditional random field layer) predicts drug and disorder spans; the same encoder's contextual embeddings for the [CLS] token, for the candidate drug, and for the candidate disorder, plus the NER label probabilities, are concatenated into a 3,088-dimensional feature vector and fed to a four-layer multi-layer perceptron that classifies the pair as ADE or not. Candidate pairs are limited to drug-disorder mentions at most four sentences apart, a window chosen because it covers 99.42 percent of gold ADE relations in the primary corpus, and class imbalance is handled by SMOTE oversampling plus random undersampling. Thresholds are chosen from precision-recall curves on the validation set under F1 and F2 objectives, and ADE groups let the authors evaluate both 'easy' success, where any mention of a unique clinical ADE is found, and 'hard' success, where every mention of the same clinical ADE is found.","core_discovery":"On the paper's own terms, the discovery is that ADE detection in Dutch clinical free text is best treated as two linked subtasks, recognizing drug and disorder mentions and then classifying drug-disorder pairs as ADE, prescribing indication, or neither, and that both can be served by one domain-pretrained encoder. MedRoBERTa.nl achieves the highest macro-averaged F1 on the ADE relation task (0.6301 with gold entities and 0.6222 end-to-end), and the same model detects the presence of an ADE in 67 to 74 percent of external discharge letters, outperforming the next best model by a clear margin (0.681 versus 0.622 F1 under an F1-optimised threshold). The paper further claims that evaluation choices hide this result: micro-averaged F1 puts all models near 0.98 because the majority 'no relation' class dominates, and it recommends macro-averaged F1, precision-recall curves, and F2-based threshold selection as the appropriate measures when missing an ADE is costlier than a false alarm. The authors also acknowledge that part of MedRoBERTa.nl's internal advantage may come from pre-training on notes that overlap the training corpus, and they lean on the external validation to argue the model generalises rather than overfits.","pith_inferences":["The four-sentence pairing window is carried unchanged from the training corpus to the external corpora; measuring the actual sentence distances of gold ADE relations in the WINGS and ICU AKI notes would show whether the reported recalls are true end-to-end performance or a ceiling imposed by the candidate generator.","Because the relation head is a small multi-layer perceptron over reused embeddings, adding another relation type or another rare clinical event (falls, pressure ulcers, delirium) would cost little; the same pipeline pattern could be lifted to other rare-event monitoring tasks in clinical text.","The paper's deliberate exclusion of decoder-only large language models is a testable bet: a reader could run the identical two-stage evaluation with a generative model and check whether the encoder-only design, not just the domain pre-training, is what carries the advantage in low-resource Dutch clinical settings.","The easy-versus-hard ADE-group evaluation mirrors two real workflows, triage and comprehensive registration; future work could report cost-weighted operating points, such as the number of notes needing pharmacist review per confirmed ADE, to turn precision-recall curves into staffing numbers."],"forward_implications":["Macro-averaged F1, not the near-perfect micro-averaged F1, is the number that describes real ADE detection performance; on that measure all models land between 0.58 and 0.63, so the task remains hard.","Choosing thresholds with the F2 score roughly raises MedRoBERTa.nl's retrieval of complete ADE groups (37 to 60 percent of groups detected in the easy setting), at the price of more false positives.","The winning model transfers across document type, specialty, and hospital: discharge letters from two non-academic internal medicine wards are detected at 67 to 74 percent recall, so the approach is applicable to hospital-wide medication safety monitoring and not just the training ICU.","One fine-tuned encoder can serve both named entity recognition and relation classification because the relation head reuses its embeddings, avoiding the cost of training a second transformer.","Document-flagging for manual causality assessment, the workflow matched by the 'easy' ADE-group evaluation, appears deployable now: MedRoBERTa.nl retrieves 60 percent of ADE groups at the F2 threshold, a large gain over the estimated 6 to 33 percent of ADEs currently captured in structured form."],"supporting_citations":[{"why":"Supplies the primary Dutch ADE corpus of 102 richly annotated ICU progress notes with ADE-group mappings that train and internally test all models.","marker":"[12]"},{"why":"Provides the WINGS corpus of 100 discharge letters, 36 with expert-confirmed ADEs, used for external document-level validation.","marker":"[21]"},{"why":"Introduces MedRoBERTa.nl, the Dutch EHR-pretrained model that finishes best on the internal and external ADE tasks.","marker":"[27]"},{"why":"Defines the n2c2 2018 shared-task evaluation of ADE and medication extraction whose relation-level metrics the paper's tasks extend.","marker":"[15]"},{"why":"The MADE 1.0 challenge, the other standard ADE extraction benchmark whose entity-relation design supports the paper's task framing.","marker":"[16]"},{"why":"The authors' scoping review documenting why Bi-LSTM is the standard baseline and why prior evaluations misreport performance on imbalanced ADE data.","marker":"[6]"},{"why":"Prior transformer results on English ADE relation classification that frame why the paper reports macro-F1 alongside micro-F1.","marker":"[8]"},{"why":"The best prior Dutch ADE detection by rule-based text mining (F1 0.71), the practical baseline the benchmark seeks to improve upon.","marker":"[14]"}],"fun_headline_variants":["Dutch clinical model finds ADEs in 67–74% of discharge letters","MedRoBERTa tops Dutch ADE detection with 74% recall on letters","Benchmark: Transformer model spots ADEs in Dutch clinical text","Dutch ICU notes: transformer model catches 74% of drug harm cases","Best Dutch ADE detector: MedRoBERTa with 0.74 recall on letters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The candidate-pair generator links a drug and a disorder only when they lie within four sentences of each other, a window chosen on the training corpus because it covers 99.42 percent of its gold ADE relations; the same fixed window is applied to the external discharge letters and ICU notes without checking whether their ADE relations also fall inside it, so if external relations span longer distances every model's document-level recall is capped below its true potential.","fun_headline_variants_meta":{"raw":{"variants":["Dutch clinical model finds ADEs in 67–74% of discharge letters","MedRoBERTa tops Dutch ADE detection with 74% recall on letters","Benchmark: Transformer model spots ADEs in Dutch clinical text","Dutch ICU notes: transformer model catches 74% of drug harm cases","Best Dutch ADE detector: MedRoBERTa with 0.74 recall on letters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000552,"raw_usage":{"total_tokens":2742,"prompt_tokens":1163,"completion_tokens":1579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":779,"completion_tokens_details":{"reasoning_tokens":1478}},"tokens_in":779,"tokens_out":1579,"duration_ms":13268,"temperature":1.0,"reasoning_tokens":1478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:19:53.454631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate the sentence distance between drug and disorder for every gold ADE relation in the WINGS discharge letters and the ICU AKI notes. If a substantial share of relations span more than four sentences, the reported 0.67 to 0.74 external recalls are pessimistic ceilings imposed by the candidate window rather than true end-to-end ability, and the benchmark would need re-running with a longer window or with document-level aggregation.","supporting_citations":[],"review_version":1}