REVIEW 4 major objections 6 minor 22 references
Ensemble BERT for Medication Event Classification on Electronic Health Records (EHRs)
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Majority voting over fine-tuned BERT models lifts medication-event F1 by 4–6 points on the CMED benchmark.
desk verdict Plausible but under-supported empirical claim: majority voting over fine-tuned BERT models gains 4–6 F1 points on CMED, but single-run results and post-hoc selection of the voting rule leave the real gain unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ensemble: eleven publicly available pretrained BERT-family checkpoints (BERT base, BioBERT variants, ClinicalBERT, DischargeBERT, BioClinicalBERT, BioDischargeBERT, BioRedditBERT, RoBERTa base, RoBERTa large), each fine-tuned on CMED training data and applied to the test notes to produce token-level BIO (begin/inside/outside) predictions. Majority soft voting sums predicted class probabilities across models and selects the class with the largest total; the paper also tests weighted voting with weights inversely related to expected calibration error, which underperforms. Strict versus lenient offset matching and micro/macro averaging define the scored outcomes.
What would settle it
Run the same fine-tuning and majority-voting protocol on the CMED training set with, say, five to ten random seeds, and compare the mean strict Micro-F and Macro-F against Robertalarge on the official test split: if the 4–6 point gap falls within the seed-to-seed spread, the claimed gain is not a stable effect.
Extended reading notes
Core claim
On the CMED test split, the paper's majority-voting ensemble of fine-tuned BERT variants reaches strict Micro-F 0.8401 and strict Macro-F 0.7744, compared with Robertalarge's 0.8003 and 0.7176; lenient scores improve similarly. The comparison holds across strict and lenient matching and across micro- and macro-averaged precision, recall, and F1. The same voting ensemble, after a simple chunk-combining post-process, also improves medication identification, reaching strict F1 0.9092 versus 0.8844 for the best single model. A weighted variant that assigns weights from expected calibration error does not improve on the single models, so the claim is specifically about majority voting.
Load-bearing premise
The claim rests on a single evaluation run on the official CMED test split: no repeated runs or variance are reported, and the paper does not fully specify the training/validation split, hyperparameters, and seeds, so the 4–6 point gap could in principle be run-to-run noise.
Editorial extensions
If this is right
- If the claim holds, teams working on this medication-event task can gain 4–6 F1 points by majority-voting existing pretrained models, without changing the underlying model architecture.
- The majority-voting result implies that different pretraining corpora (general, biomedical, clinical, Reddit) supply complementary signal even after fine-tuning on the same small clinical training set.
- The failure of ECE-weighted voting suggests that calibration-based weights are not a safe way to combine these models; simple equal-vote aggregation is more reliable here.
- Post-processing the ensemble's event predictions into medication spans also improves medication identification, so the ensemble benefit carries to the downstream extraction task.
Reading between the lines
- The paper reports a single pass over the official test split; a natural extension would be repeated fine-tuning with different seeds to attach error bars to the 4–6 point gain, since seed variance alone could cover the gap.
- The gain presumably depends on diversity among the ensemble members; a testable extension is to measure pairwise prediction disagreement on a validation set and ask whether a smaller, maximally disagreeing subset of three or four models retains most of the gain at lower inference cost.
- If the gain comes from error decorrelation rather than from any specific checkpoint, the same majority-voting recipe may transfer to other clinical token-classification tasks such as medication-related relation extraction, though this is not tested in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an ensemble of 11 fine-tuned BERT-style models for medication event classification (subtask 2, Track 1) on the n2c2 2022 CMED dataset. It evaluates individual models and two voting strategies (majority voting and ECE-weighted voting) under strict and lenient micro/macro metrics, and also reports medication identification F1. The main claim is that majority voting improves strict MicroF from 0.8003 (Robertalarge) to 0.8401 and strict MacroF from 0.7176 to 0.7744.
Significance. If the reported improvements are stable, the paper demonstrates a simple, practical ensemble gain over strong single models on an external benchmark, which would be of interest to clinical NLP practitioners. The strength is the breadth of pretrained models compared and the use of an established shared-task benchmark. However, the current evidence is a single run with no error bars, no hyperparameter or split details, and no code; moreover, the conclusion contains a statement that could imply fine-tuning on the test data. These issues must be resolved before the central claim can be accepted. The claim itself is not circular; it is an empirical measurement, but its validity depends on experimental controls that are currently not documented.
major comments (4)
- [Section V] Section V states that 'these pretrained BERT models were fine-tuned on CMED testing datasets.' This directly contradicts the methodology in Sections II and III-A, where fine-tuning is on the CMED training data. If this statement is literal, the models have been exposed to the test labels and all reported results are invalid; if it is a typo, it must be corrected, and the abstract's 'fine-tuned on CMED training data' suggests the intended meaning. Please clarify this in the revision.
- [Section III-D, Tables III and IV] The central improvement claim (about 5% relative gain in strict MicroF and 6% in strict MacroF) rests on a single pass over the official test split. No error bars, repeated runs, or seeds are reported. Given the small training set (7,230 mentions) and the 7% minority class, fine-tuned transformer results are known to be seed-sensitive, and the reported gap could be within run-to-run noise. Provide multiple random seeds or bootstrap confidence intervals, and report the variance across runs for the best single model and the majority-voting ensemble.
- [Section III-D, Tables III and IV] The majority-voting rule appears to have been selected after observing its superior test performance, while the ECE-weighted variant underperforms the best single model (Robertalarge) on strict MacroF and on lenient MicroF and MacroF. This is test-set-based model selection. Use a held-out validation split to choose the voting rule, or explicitly describe the selection procedure; otherwise, the claimed improvement may be an optimistic artifact of selecting the best-scoring variant on the test set.
- [Sections II-B and III-C] Reproducibility details are missing: fine-tuning hyperparameters (learning rate, epochs, batch size, sequence length, optimizer), the training/validation split, the number of runs, and the exact computation of ECE weights are not reported. Without these, the experiment cannot be repeated or independently verified. Please include a detailed experimental setup or a pointer to code that supplies these details.
minor comments (6)
- [Section II-B] The text says 'majority soft voting' but the preceding description explains hard voting (summing predictions for each label and choosing the most frequent class). Clarify which mechanism is actually used and how ties are broken.
- [Abstract and Introduction] There are typographical issues: 'text-ming tasks' should be 'text-mining tasks', and 'N2C2 2022' vs 'n2c2 2022' is inconsistent throughout the paper.
- [Section III-D.2] The phrase 'In addition, In addition,' appears duplicated and should be reduced to one instance.
- [References] Reference [17] (Macosko et al., droplet-based single-cell RNA sequencing) appears unrelated to the topic of clinical NLP or ensemble learning; please verify that this citation is used appropriately or remove it.
- [Table II] The model name 'Robertabase' is non-standard; specify the exact checkpoint (e.g., RoBERTa-base) and indicate whether the same tokenizer and sequence length settings are used across all models.
- [Section III-B] In the metric equations, C is described as 'the total number of medication events,' but macro-averaging requires C to be the number of classes. Clarify the notation.
Circularity Check
No significant circularity: the reported gains are empirical measurements on an external benchmark, not derivations that reduce to the method's own inputs.
full rationale
The paper's central claim is an empirical performance comparison on the external n2c2 CMED benchmark. The ensemble is formed by fine-tuning externally released pretrained BERT models on CMED training data and combining their predictions by voting; the improvement numbers in Tables III and IV are obtained by scoring the resulting predictions against gold annotations. Nothing in the method defines the reported Micro-F or Macro-F values in terms of themselves: the majority-voting rule is a fixed aggregation procedure, not a parameter fitted to the test labels. The ECE-weighted variant is computed from training-data calibration and actually performs worse than Robertalarge, which further indicates the voting scheme was not forced by construction. No load-bearing self-citation appears; the cited prior works are external. The main validity concerns are not circularity: the majority-voting variant appears to have been highlighted after inspecting test results, no repeated runs or variance estimates are reported, the training/validation split and hyperparameters are not described, and the Conclusion contains an internal inconsistency stating that the pretrained BERT models were fine-tuned on CMED testing datasets (Section V). These affect robustness and reproducibility, but they do not reduce the claimed result to the paper's own inputs. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (1)
- ECE-based ensemble weights =
not reported
assumptions (5)
- domain assumption The CMED gold annotations and the official strict/lenient matching definitions are correct and representative.
- domain assumption Fine-tuning the named pretrained BERT checkpoints on the CMED training data produces valid medication-event classifiers.
- domain assumption The single reported evaluation on the CMED test split is stable enough to support a 4-6 point improvement claim.
- domain assumption The named models such as BioBERT, ClinicalBERT, and BioClinicalBERT correspond to the publicly released checkpoints with the stated pretraining corpora.
- standard math The standard definitions of precision, recall, and F-score in Equations 1-6 apply to the task.
Cite this review
Pith. "Pith review of Ensemble BERT for Medication Event Classification on Electronic Health Records (EHRs)." pith.science (2026). https://pith.science/paper/WW4LZT5N
@misc{pith2026250623315,
author = {Pith},
title = {Pith review of: Ensemble BERT for Medication Event Classification on Electronic Health Records (EHRs)},
year = {2026},
howpublished = {\url{https://pith.science/paper/WW4LZT5N}},
note = {Machine review of arXiv:2506.23315}
}
read the original abstract
Identification of key variables such as medications, diseases, relations from health records and clinical notes has a wide range of applications in the clinical domain. n2c2 2022 provided shared tasks on challenges in natural language processing for clinical data analytics on electronic health records (EHR), where it built a comprehensive annotated clinical data Contextualized Medication Event Dataset (CMED). This study focuses on subtask 2 in Track 1 of this challenge that is to detect and classify medication events from clinical notes through building a novel BERT-based ensemble model. It started with pretraining BERT models on different types of big data such as Wikipedia and MIMIC. Afterwards, these pretrained BERT models were fine-tuned on CMED training data. These fine-tuned BERT models were employed to accomplish medication event classification on CMED testing data with multiple predictions. These multiple predictions generated by these fine-tuned BERT models were integrated to build final prediction with voting strategies. Experimental results demonstrated that BERT-based ensemble models can effectively improve strict Micro-F score by about 5% and strict Macro-F score by about 6%, respectively.
Figures
Reference graph
Works this paper leans on
-
[1]
I. J. Goodfellow, Y . Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016. urlhttp://www.deeplearningbook.org
work page 2016
-
[2]
Toward understanding clinical context of medication change events in clinical narratives,
D. Mahajan, J. J. Liang, and C.-H. Tsou, “Toward understanding clinical context of medication change events in clinical narratives,” AMIA ... Annual Symposium proceedings. AMIA Symposium , vol. 2021, pp. 833– 842, 2021
work page 2021
-
[3]
BERT: Pre- training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), (Minneapolis, Minnesota), pp. 4171–4186...
work page 2019
-
[4]
BioBERT: a pre-trained biomedical language representation model for biomedical text mining,
J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “BioBERT: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, sep 2019
work page 2019
-
[5]
An overview of named entity recognition,
P. Sun, X. Yang, X. Zhao, and Z. Wang, “An overview of named entity recognition,” in 2018 International Conference on Asian Language Processing (IALP), pp. 273–278, 2018
work page 2018
-
[6]
A survey on deep learning for named entity recognition,
J. Li, A. Sun, J. Han, and C. Li, “A survey on deep learning for named entity recognition,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 1, pp. 50–70, 2022
work page 2022
-
[7]
Z. Yang, C. Zhu, V . Sachidananda, and E. F. Darve, “Out-of-vocabulary embedding imputation with grounded language information by graph convolutional networks.,” arXiv: Computation and Language , 2020
work page 2020
-
[8]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Curran Associates, Inc., 2017
2017
Show all 22 references
-
[9]
Efficient estimation of word representations in vector space,
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” in 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings , 2013
2013
-
[10]
Fast wordpiece tokenization,
X. Song, A. Salcianu, Y . Song, D. Dopson, and D. Zhou, “Fast wordpiece tokenization,” arXiv, 2020
2020
-
[11]
An ensemble of pre- trained transformer models for imbalanced multiclass malware classifi- cation,
F. Demirkıran, A. C ¸ ayır, U. ¨Unal, and H. Da ˘g, “An ensemble of pre- trained transformer models for imbalanced multiclass malware classifi- cation,” Computers & Security , vol. 121, p. 102846, 2022
2022
-
[12]
Emotion detection from tweets using a bert and svm ensemble model,
I. Albu and S. Sp ˆınu, “Emotion detection from tweets using a bert and svm ensemble model,” ArXiv, vol. abs/2208.04547, 2022
2022 arXiv
-
[13]
Estimating expected calibration errors,
N. Posocco and A. Bonnefoy, “Estimating expected calibration errors,” in ICANN, 2021
2021
-
[14]
Text chunking using transformation-based learning,
L. Ramshaw and M. Marcus, “Text chunking using transformation-based learning,” in Third Workshop on Very Large Corpora , 1995
1995
-
[15]
Macro-and micro-averaged evaluation measures [[basic draft]],
V . Van Asch, “Macro-and micro-averaged evaluation measures [[basic draft]],” Belgium: CLiPS, vol. 49, 2013
2013
-
[16]
A study of thresholding strategies for text categorization,
Y . Yang, “A study of thresholding strategies for text categorization,” in Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval , pp. 137–145, 2001
2001
-
[17]
Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets,
E. Z. Macosko, A. Basu, R. Satija, J. Nemesh, K. Shekhar, M. Goldman, I. Tirosh, A. R. Bialas, N. Kamitaki, E. M. Martersteck, et al., “Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets,” Cell, vol. 161, no. 5, pp. 1202–1214, 2015
2015
-
[18]
Data mining for imbalanced datasets: An overview,
N. V . Chawla, “Data mining for imbalanced datasets: An overview,” in Data mining and knowledge discovery handbook, pp. 875–886, Springer, 2009
2009
-
[19]
Metrics for multi-class classifi- cation: an overview,
M. Grandini, E. Bagli, and G. Visani, “Metrics for multi-class classifi- cation: an overview,” ArXiv, vol. abs/2008.05756, 2020
2008 arXiv
-
[20]
Bidirectional lstm-crf models for sequence tagging,
Z. Huang, W. Xu, and K. Yu, “Bidirectional lstm-crf models for sequence tagging,” ArXiv, vol. abs/1508.01991, 2015
2015 arXiv
-
[21]
Deep contextualized word representations,
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies...
2018
-
[22]
Publicly available clinical BERT embeddings,
E. Alsentzer, J. Murphy, W. Boag, W.-H. Weng, D. Jindi, T. Naumann, and M. McDermott, “Publicly available clinical BERT embeddings,” in Proceedings of the 2nd Clinical Natural Language Processing Workshop, (Minneapolis, Minnesota, USA), pp. 72–78, Association for Computa- tion...
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.