{"id":"e2d3c8e8-15fa-44da-b17e-fe6c34f1f784","arxiv_id":"2506.23315","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Majority voting over multiple pretrained BERT models improves medication event classification on the n2c2 CMED dataset, but the paper lacks error bars and code.","lead":"This paper combines several pretrained BERT models with majority voting to classify medication change events in clinical notes from the n2c2 2022 shared task. The best ensemble improves strict F-scores by about 4 to 6 percentage points over single models, but only for one voting scheme.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 4–6 point gain is credible only if it survives seed variance and is not an artifact of choosing the majority-voting rule after seeing test results; the paper supplies neither repeated runs nor a validation split.","rationale":"This is an empirical paper, so the central claim depends on the evaluation being a stable, unbiased measurement. The strongest support for the claim is the majority-voting row in Table III (strict Micro-F 0.8401 vs. Robertalarge 0.8003; strict Macro-F 0.7744 vs. 0.7176), but the paper provides no repeated runs, no error bars, no validation split, no hyperparameters, and no code. The fact that the weighted ECE ensemble underperforms Robertalarge while the majority ensemble is highlighted suggests that the voting rule may have been selected after seeing test results. This is exactly the condition under which a 4–6 point gap could be inflated. The conclusion's 'fine-tuned on CMED testing datasets' sentence, while probably a typo, should be corrected because a literal reading would imply test leakage. The reader's conditional verdict is appropriate: the result is plausible and cheap to test, but the manuscript needs experimental rigor and a precise statement of which ensemble variant is claimed before the result can be accepted.","tokens_in":8968,"tokens_out":5784,"duration_ms":65311,"concrete_test":"Rerun the full pipeline with 5 random seeds, choosing the voting rule (majority vs weighted) on a held-out 20% validation split rather than on the test set, and report mean and 95% bootstrap confidence intervals for strict/lenient Micro-F and Macro-F on the official test set; if the 95% CI for the improvement over Robertalarge includes zero, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that majority voting over fine-tuned BERT models raises strict Micro-F by about 4–5 points and strict Macro-F by about 6 points above Robertalarge (Tables III–IV). This empirical comparison rests on a single pass over the official CMED test set. No training/validation split, hyperparameters, seeds, error bars, or code are reported, and the conclusion even contains an internal inconsistency, saying the pretrained BERT models were fine-tuned on CMED testing datasets (Section V). More importantly, the paper evaluates two voting variants; the weighted ECE variant is worse than Robertalarge, while the majority variant that supports the abstract appears to have been selected after inspecting test results. Because fine-tuning 11 transformer models on 7,230 mentions with a 7% minority class is seed-sensitive, the reported gap could be within run-to-run noise or an optimistic selection artifact. Without independent scoring using the official n2c2 scorer and without variance estimates, the claimed improvement is not established as a stable ensemble effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ensemble of 11 fine-tuned BERT-style models for medication event classification (subtask 2, Track 1) on the n2c2 2022 CMED dataset. It evaluates individual models and two voting strategies (majority voting and ECE-weighted voting) under strict and lenient micro/macro metrics, and also reports medication identification F1. The main claim is that majority voting improves strict MicroF from 0.8003 (Robertalarge) to 0.8401 and strict MacroF from 0.7176 to 0.7744.","tokens_in":9100,"tokens_out":3839,"duration_ms":40067,"significance":"If the reported improvements are stable, the paper demonstrates a simple, practical ensemble gain over strong single models on an external benchmark, which would be of interest to clinical NLP practitioners. The strength is the breadth of pretrained models compared and the use of an established shared-task benchmark. However, the current evidence is a single run with no error bars, no hyperparameter or split details, and no code; moreover, the conclusion contains a statement that could imply fine-tuning on the test data. These issues must be resolved before the central claim can be accepted. The claim itself is not circular; it is an empirical measurement, but its validity depends on experimental controls that are currently not documented.","major_comments":[{"comment":"Section V states that 'these pretrained BERT models were fine-tuned on CMED testing datasets.' This directly contradicts the methodology in Sections II and III-A, where fine-tuning is on the CMED training data. If this statement is literal, the models have been exposed to the test labels and all reported results are invalid; if it is a typo, it must be corrected, and the abstract's 'fine-tuned on CMED training data' suggests the intended meaning. Please clarify this in the revision.","section":"Section V"},{"comment":"The central improvement claim (about 5% relative gain in strict MicroF and 6% in strict MacroF) rests on a single pass over the official test split. No error bars, repeated runs, or seeds are reported. Given the small training set (7,230 mentions) and the 7% minority class, fine-tuned transformer results are known to be seed-sensitive, and the reported gap could be within run-to-run noise. Provide multiple random seeds or bootstrap confidence intervals, and report the variance across runs for the best single model and the majority-voting ensemble.","section":"Section III-D, Tables III and IV"},{"comment":"The majority-voting rule appears to have been selected after observing its superior test performance, while the ECE-weighted variant underperforms the best single model (Robertalarge) on strict MacroF and on lenient MicroF and MacroF. This is test-set-based model selection. Use a held-out validation split to choose the voting rule, or explicitly describe the selection procedure; otherwise, the claimed improvement may be an optimistic artifact of selecting the best-scoring variant on the test set.","section":"Section III-D, Tables III and IV"},{"comment":"Reproducibility details are missing: fine-tuning hyperparameters (learning rate, epochs, batch size, sequence length, optimizer), the training/validation split, the number of runs, and the exact computation of ECE weights are not reported. Without these, the experiment cannot be repeated or independently verified. Please include a detailed experimental setup or a pointer to code that supplies these details.","section":"Sections II-B and III-C"}],"minor_comments":[{"comment":"The text says 'majority soft voting' but the preceding description explains hard voting (summing predictions for each label and choosing the most frequent class). Clarify which mechanism is actually used and how ties are broken.","section":"Section II-B"},{"comment":"There are typographical issues: 'text-ming tasks' should be 'text-mining tasks', and 'N2C2 2022' vs 'n2c2 2022' is inconsistent throughout the paper.","section":"Abstract and Introduction"},{"comment":"The phrase 'In addition, In addition,' appears duplicated and should be reduced to one instance.","section":"Section III-D.2"},{"comment":"Reference [17] (Macosko et al., droplet-based single-cell RNA sequencing) appears unrelated to the topic of clinical NLP or ensemble learning; please verify that this citation is used appropriately or remove it.","section":"References"},{"comment":"The model name 'Robertabase' is non-standard; specify the exact checkpoint (e.g., RoBERTa-base) and indicate whether the same tokenizer and sequence length settings are used across all models.","section":"Table II"},{"comment":"In the metric equations, C is described as 'the total number of medication events,' but macro-averaging requires C to be the number of classes. Clarify the notation.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The sentence in the conclusion about fine-tuning on CMED testing datasets is a serious red flag that needs an explicit editorial correction or clarification. If it is a typo, the paper still lacks the experimental detail needed to support the 4-6 point improvement claim. The manuscript's current form is not suitable for publication until the variance analysis, model-selection procedure, and reproducibility details are addressed. The topic is within scope for a clinical NLP venue, but the 'novel ensemble' contribution is modest since majority voting is a standard technique; the empirical comparison is the main value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper reports a concrete, cheap-to-test result — majority voting over 11 fine-tuned BERT variants beats the best single model (Robertalarge) by about 4–5 points strict MicroF and 6 points strict MacroF on the n2c2 CMED medication event task. The tables are clear, the baseline coverage is decent, and they honestly report that their weighted ECE ensemble underperforms the best single model. That honesty is a point in their favor; they don't hide the negative result.\n\nWhat's actually new is the specific CMED numbers; the method itself is standard hard voting over fine-tuned pretrained transformers. Calling it a \"novel ensemble model\" is a stretch, but the empirical outcome could still be useful to clinical NLP teams.\n\nThe soft spots are real, though not fatal on their own. Most importantly, there are no error bars, no repeated runs, no seeds, and no training/validation split details. With 7,230 mentions and a 7% minority class, fine-tuning is genuinely seed-sensitive, so a 4–6 point gap could shrink or vanish. Second, they evaluated two voting schemes and report the one that worked on the test set; there's no validation-based selection of the voting rule, which risks optimistic selection. Third, no code or data release, which makes independent verification harder even though the dataset is public. Fourth, the abstract's phrase \"ensemble models can effectively improve\" is too broad when only majority voting helps; the weighted variant doesn't. Finally, the conclusion contains an internal inconsistency — it says the models were fine-tuned on CMED testing datasets — which suggests a careless final proofread.\n\nNone of this makes the central claim impossible; it just means the paper hasn't yet established that the improvement is stable. The fix is straightforward: release code, run each model and the ensemble over multiple seeds, report mean and variance, and show that majority voting was chosen on a validation split, not the test set. A comparison to the official n2c2 shared task results would also help calibrate where this stands.\n\nFor a serious venue, I'd send it to review with a request for major revisions along those lines. It's a credible empirical data point, but it needs rigor before the numbers can be trusted. As is, it reads like a workshop paper with a useful observation and an overconfident abstract.","headline":"Plausible but under-supported empirical claim: majority voting over fine-tuned BERT models gains 4–6 F1 points on CMED, but single-run results and post-hoc selection of the voting rule leave the real gain unproven.","tokens_in":9699,"tokens_out":2235,"would_cite":false,"duration_ms":26276,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Majority voting over fine-tuned BERT models lifts medication-event F1 by 4–6 points on the CMED benchmark.","keywords":["medication event classification","electronic health records","BERT ensemble","majority voting","CMED dataset","clinical natural language processing","BioBERT","token classification"],"falsifier":"Run the same fine-tuning and majority-voting protocol on the CMED training set with, say, five to ten random seeds, and compare the mean strict Micro-F and Macro-F against Robertalarge on the official test split: if the 4–6 point gap falls within the seed-to-seed spread, the claimed gain is not a stable effect.","tokens_in":8697,"feed_emoji":"💊","tokens_out":7130,"duration_ms":61775,"temperature":0.7,"pith_summary":"Medication event classification in electronic health records asks a model to tag each medication mention as a change (Disposition), no change (NoDisposition), or undetermined. This paper argues that no single pretrained transformer is enough: combining eleven BERT-family models, each fine-tuned on the CMED training set, through majority soft voting improves strict Micro-F by about 5% and strict Macro-F by about 6% over the best single model. The practical payoff is that the gain comes from model integration and post-processing alone, not from a new architecture or task-specific annotation. If the result holds, cheap ensemble voting is a ready upgrade for medication-event extraction pipelines built on existing clinical language models.","feed_headline":"Majority-voting BERTs beat single models on medication event F1","feed_subtitle":"Eleven fine-tuned BERT variants vote to gain 4–6 F1 points on the CMED medication-event benchmark.","key_machinery":"The central object is the ensemble: eleven publicly available pretrained BERT-family checkpoints (BERT base, BioBERT variants, ClinicalBERT, DischargeBERT, BioClinicalBERT, BioDischargeBERT, BioRedditBERT, RoBERTa base, RoBERTa large), each fine-tuned on CMED training data and applied to the test notes to produce token-level BIO (begin/inside/outside) predictions. Majority soft voting sums predicted class probabilities across models and selects the class with the largest total; the paper also tests weighted voting with weights inversely related to expected calibration error, which underperforms. Strict versus lenient offset matching and micro/macro averaging define the scored outcomes.","core_discovery":"On the CMED test split, the paper's majority-voting ensemble of fine-tuned BERT variants reaches strict Micro-F 0.8401 and strict Macro-F 0.7744, compared with Robertalarge's 0.8003 and 0.7176; lenient scores improve similarly. The comparison holds across strict and lenient matching and across micro- and macro-averaged precision, recall, and F1. The same voting ensemble, after a simple chunk-combining post-process, also improves medication identification, reaching strict F1 0.9092 versus 0.8844 for the best single model. A weighted variant that assigns weights from expected calibration error does not improve on the single models, so the claim is specifically about majority voting.","pith_inferences":["The paper reports a single pass over the official test split; a natural extension would be repeated fine-tuning with different seeds to attach error bars to the 4–6 point gain, since seed variance alone could cover the gap.","The gain presumably depends on diversity among the ensemble members; a testable extension is to measure pairwise prediction disagreement on a validation set and ask whether a smaller, maximally disagreeing subset of three or four models retains most of the gain at lower inference cost.","If the gain comes from error decorrelation rather than from any specific checkpoint, the same majority-voting recipe may transfer to other clinical token-classification tasks such as medication-related relation extraction, though this is not tested in the paper."],"forward_implications":["If the claim holds, teams working on this medication-event task can gain 4–6 F1 points by majority-voting existing pretrained models, without changing the underlying model architecture.","The majority-voting result implies that different pretraining corpora (general, biomedical, clinical, Reddit) supply complementary signal even after fine-tuning on the same small clinical training set.","The failure of ECE-weighted voting suggests that calibration-based weights are not a safe way to combine these models; simple equal-vote aggregation is more reliable here.","Post-processing the ensemble's event predictions into medication spans also improves medication identification, so the ensemble benefit carries to the downstream extraction task."],"supporting_citations":[{"why":"Supplies the CMED dataset, its three-class medication-event annotation scheme, and the training/testing split used in all experiments.","marker":"[2]"},{"why":"Defines BERT and its pretraining objectives, the base model that every ensemble member starts from.","marker":"[3]"},{"why":"Introduces BioBERT and establishes that biomedical pretraining improves biomedical text mining, motivating the clinical and biomedical checkpoints ensembled here.","marker":"[4]"},{"why":"Provides the voting-ensemble rationale and hard/soft voting definitions used to combine the fine-tuned models.","marker":"[11]"},{"why":"Supplies the expected calibration error estimation used to build the weighted ensemble variant.","marker":"[13]"},{"why":"Defines macro- and micro-averaged precision, recall, and F1, the metric family all reported scores use.","marker":"[15]"},{"why":"Defines strict versus lenient matching for offset pairs, the two evaluation modes reported in the tables.","marker":"[19]"},{"why":"Introduces publicly available clinical BERT embeddings, the source of the ClinicalBERT and related clinical checkpoints in the ensemble.","marker":"[22]"}],"fun_headline_variants":["Voting BERT ensemble lifts medication event F1 by 5 points","Majority-vote BERTs beat single models in EHR medication tagging","Ensemble of BERTs improves CMED medication event scores","BERT voting ensemble gains macro-F1 on clinical notes","Fine-tuned BERTs vote to boost medication event detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on a single evaluation run on the official CMED test split: no repeated runs or variance are reported, and the paper does not fully specify the training/validation split, hyperparameters, and seeds, so the 4–6 point gap could in principle be run-to-run noise.","fun_headline_variants_meta":{"raw":{"variants":["Voting BERT ensemble lifts medication event F1 by 5 points","Majority-vote BERTs beat single models in EHR medication tagging","Ensemble of BERTs improves CMED medication event scores","BERT voting ensemble gains macro-F1 on clinical notes","Fine-tuned BERTs vote to boost medication event detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1250,"prompt_tokens":907,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":255}},"tokens_in":523,"tokens_out":343,"duration_ms":3807,"temperature":1.0,"reasoning_tokens":255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:46:40.301079+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning and majority-voting protocol on the CMED training set with, say, five to ten random seeds, and compare the mean strict Micro-F and Macro-F against Robertalarge on the official test split: if the 4–6 point gap falls within the seed-to-seed spread, the claimed gain is not a stable effect.","supporting_citations":[{"cited_title":"Toward understanding clinical context of medication change events in clinical narratives,","cited_arxiv_id":null,"evidence_quote":"Supplies the CMED dataset, its three-class medication-event annotation scheme, and the training/testing split used in all experiments."},{"cited_title":"BERT: Pre- training of deep bidirectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"Defines BERT and its pretraining objectives, the base model that every ensemble member starts from."},{"cited_title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining,","cited_arxiv_id":null,"evidence_quote":"Introduces BioBERT and establishes that biomedical pretraining improves biomedical text mining, motivating the clinical and biomedical checkpoints ensembled here."},{"cited_title":"An ensemble of pre- trained transformer models for imbalanced multiclass malware classifi- cation,","cited_arxiv_id":null,"evidence_quote":"Provides the voting-ensemble rationale and hard/soft voting definitions used to combine the fine-tuned models."},{"cited_title":"Estimating expected calibration errors,","cited_arxiv_id":null,"evidence_quote":"Supplies the expected calibration error estimation used to build the weighted ensemble variant."},{"cited_title":"Macro-and micro-averaged evaluation measures [[basic draft]],","cited_arxiv_id":null,"evidence_quote":"Defines macro- and micro-averaged precision, recall, and F1, the metric family all reported scores use."},{"cited_title":"Publicly available clinical BERT embeddings,","cited_arxiv_id":null,"evidence_quote":"Introduces publicly available clinical BERT embeddings, the source of the ClinicalBERT and related clinical checkpoints in the ensemble."}],"review_version":1}