{"id":"303e5cfe-50e2-4007-ba3c-6dfba1c19d84","arxiv_id":"2412.05583","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A Bi-LSTM and a 1D-CNN trained on public ECG datasets classify arrhythmias with reported accuracies up to 99%, but the evaluation likely suffers from data leakage.","lead":"This paper applies standard machine learning models to public ECG datasets, reporting 92% accuracy in detecting atrial fibrillation and 98-99% accuracy in classifying five heartbeat types. The authors also surveyed medical professionals and built a local web demo, but the evaluation protocol has methodological flaws that likely inflate the reported accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section III resamples the full MIT-BIH beat set before the stratified split and never groups beats by patient, so the same patient's (and possibly duplicated) beats can appear in both training and test folds.","rationale":"The reader's weakest assumption correctly identified the most load-bearing threat: resampling before cross-validation and beat-level (rather than patient-level) splitting create train/test information overlap. The paper's own statements in Section III describe exactly this pipeline, and no held-out patient evaluation is reported. This is not a disagreement with an external consensus; it is an internal validity problem: under the stated protocol, the evaluation statistic cannot support the generalization claim made in Section IV-B. I also note the F1 arithmetic inconsistency in Table I and the absence of code or error bars, which reinforce caution, but the leakage issue is the decisive problem. The limitations section acknowledges demographic transfer and computational constraints, showing the authors were willing to state limitations, yet it omits the most consequential one. If a patient/record-level split still produced roughly 99% accuracy, the claim would be credible; the concrete test is designed to allow that outcome. Given the present protocol, the reader's REJECT verdict stands, and my pass does not move it.","tokens_in":8621,"tokens_out":7519,"duration_ms":69761,"concrete_test":"Re-run the CNN with patient/record-wise cross-validation: group all beats by MIT-BIH record (and merge records belonging to the same patient where known), train on 36 records and test on 12 records (or a 5-fold record-level split), and perform all resampling only inside the training folds. Compare the held-out-record accuracy with the paper's 98.66-99.24% range. In parallel, instrument the original pipeline to count how many test beats have an exact or near-duplicate counterpart in the training folds from the resampling step. If held-out-record accuracy drops materially below the reported range, or if any duplicate overlap exists, the near-perfect accuracy claim is invalidated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a 1D-CNN classifies five MIT-BIH beat types at 98.66-99.24% accuracy across stratified 5-fold CV (Section IV-B). The load-bearing condition is that those CV accuracies estimate performance on new patients. That condition fails because of the order of operations in Section III: the full 100,012-beat dataset is normalized and resampled to 8,000 beats per class before any split, and the split is stratified at the beat level, not grouped by the 47 patients or the originating MIT-BIH record. Resampling before splitting means minority-class beats are duplicated or synthesized before the train/test boundary is drawn, so copies of the same original beat can fall on both sides of the boundary. Even setting duplication aside, beat-level splitting puts many beats from the same patient into both training and test folds; the CNN (conv blocks 7/32, 5/64, 3/128 with 50 epochs) can memorize patient-specific QRS morphology and achieve high test accuracy without learning a general arrhythmia rule. The paper's limitation section (IV-C) discusses demographic transfer and computational cost but never acknowledges this more basic leakage risk. A supporting red flag is Table I: for class A, precision 0.9867 and recall 0.9800 imply F1 approximately 0.9833, not the reported 0.9933, so the metric table needs independent verification. The leakage path alone, however, is sufficient to undermine the headline accuracy claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops and evaluates machine learning models for ECG-based arrhythmia detection and classification. It first presents a binary Bi-LSTM classifier distinguishing normal rhythm from atrial fibrillation using the PhysioNet Challenge 2017 dataset, reporting 92.44% test accuracy. It then describes a 1D-CNN for five-class classification (normal, LBBB, RBBB, PAC, PVC) using the MIT-BIH arrhythmia dataset, with stratified 5-fold cross-validation and reported accuracy between 98.66% and 99.24%. The paper also includes a survey of medical professionals on the utility of such systems and a locally deployed web portal for real-time demonstration. The central claim is that the 1D-CNN achieves near-perfect multiclass accuracy, indicating robust generalization.","tokens_in":8950,"tokens_out":3065,"duration_ms":29933,"significance":"If the reported accuracies were valid, the work would provide a useful confirmation that deep learning can classify common arrhythmia types from ECG beats, and the clinician survey adds a user-oriented perspective. The paper uses standard public datasets, gives a clear architecture description, and reports per-class precision, recall, and F1. However, the evaluation protocol has a fundamental leakage problem that invalidates the headline accuracy as an estimate of performance on new patients. The survey and web portal are minor contributions, and the multiclass result is not supported by the experiments as designed.","major_comments":[{"comment":"The evaluation protocol is invalid for estimating generalization. The MIT-BIH dataset is resampled to 8000 beats per class before the stratified 5-fold split, and the split is performed at the beat level without grouping by patient. This causes two leakage paths: (i) resampling before the split can place duplicated or synthesized copies of the same minority-class beat in both training and test folds, and (ii) beats from the same patient appear in both training and test sets, allowing the CNN to memorize patient-specific QRS morphology. Consequently, the 98.66-99.24% accuracy reported in Section IV-B does not reflect performance on new patients, and the paper's central claim is unsupported.","section":"Section III (Training and Testing Dataset)"},{"comment":"The reported F1-score for class A is internally inconsistent: with precision 0.9867 and recall 0.9800, the F1-score is approximately 0.9833, not the reported 0.9933. This arithmetic inconsistency suggests that the metric table has not been carefully verified, further undermining confidence in the numerical claims.","section":"Table I (Class A row)"},{"comment":"The limitations discussion addresses demographic transfer and computational cost but omits the train/test leakage caused by pre-split resampling and beat-level splitting. This omission is significant because the leakage is a more basic threat to validity than the issues discussed; the authors should have acknowledged and addressed it before claiming robust cross-validated performance.","section":"Section IV-C (Scope and Limitations)"},{"comment":"The same methodological issue affects the binary Bi-LSTM experiment: the PhysioNet 2017 dataset is resampled before the 9:1 train/test split, and the split is not patient-wise. The reported 92.44% test accuracy is therefore also likely optimistic, although this is a secondary claim relative to the multiclass result.","section":"Section II and III (Binary classification)"}],"minor_comments":[{"comment":"The affiliation contains the typo 'Electtionics' instead of 'Electronics'.","section":"Affiliations"},{"comment":"The word 'survelogical' appears to be a typo; likely 'survey' or 'survey-based' was intended.","section":"Section I"},{"comment":"The reference title contains the typo 'Identication' instead of 'Identification'.","section":"Reference [4]"},{"comment":"The resampling technique is not specified beyond 'resampling techniques'; the authors should state whether random oversampling, SMOTE, or another method was used, as this affects the leakage assessment.","section":"Section III (Resampling)"},{"comment":"The cross-validation results are reported only as a range for accuracy; the paper does not report the standard deviation or per-fold class-wise metrics, making it difficult to assess stability.","section":"Section IV-B and Table I"},{"comment":"The survey section does not report the number of respondents, their specialty distribution, or the exact questions, which limits the reproducibility and interpretability of the 'medical professional opinion' results.","section":"Section II (Survey)"}],"recommendation":"reject","confidential_remarks":"This manuscript appears to be a short conference paper with limited novelty beyond the combination of known methods. The evaluation flaw (resampling before cross-validation and beat-level splitting) is fundamental and invalidates the central accuracy claim. The metric inconsistency in Table I adds further doubt. The paper would require a complete re-analysis with patient-wise splitting and resampling performed inside each fold, which is beyond a routine revision. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: don't trust the headline accuracy numbers. The stress-test note is right: Section III resamples the full MIT-BIH set to 8000 beats per class before the 5-fold split and never groups beats by patient. That puts duplicated and same-patient beats on both sides of the train/test boundary, so the 98.66–99.24% range is not an honest estimate of performance on new patients. Table I also has a concrete arithmetic slip: for class A, precision 0.9867 and recall 0.9800 give F1 ≈ 0.9833, not the reported 0.9933. Those two issues are enough to undermine the central claim.\n\nWhat the paper does well: it is clearly written, the pipeline is reproducible in principle (public datasets, standard architecture), and the authors are honest in Section IV-C about demographic transfer and computational limits. The survey of medical professionals is a nice user-needs exercise—not a scientific result, but it did shape the multiclass design. The web demo is a reasonable engineering deliverable.\n\nSoft spots beyond the leakage: no code or data released; no confidence intervals or per-patient metrics; no baseline comparison against existing ECG classifiers; and the binary part (Bi-LSTM on PhysioNet 2017) is a routine application with 92% accuracy, presented without context against challenge results. Novelty is low: standard Bi-LSTM and 1D-CNN on classic datasets.\n\nMy take: this is an honest student project, but the evaluation protocol makes the headline claim unsupported. The leakage path is real, not a nitpick. If the authors redo the CV with patient-wise grouping, avoid resampling before splitting, and verify the metrics table, the multiclass result might be worth a short engineering report. As is, it doesn't deserve a full referee cycle. I'd desk reject with guidance—or send back for major revision if the venue is lenient. I wouldn't cite it.","headline":"The reported 98-99% accuracy is likely inflated by resampling-before-split and beat-level cross-validation; otherwise a standard, clearly written ECG classification study.","tokens_in":9444,"tokens_out":1902,"would_cite":false,"duration_ms":17601,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that a 1D convolutional neural network trained on individual ECG heartbeats classifies five arrhythmia types with 98.66–99.24% accuracy under stratified 5-fold cross-validation.","keywords":["Electrocardiogram (ECG)","Cardiac arrhythmia","Machine learning","Bidirectional LSTM","Convolutional Neural Network","Stratified cross-validation","Classification accuracy","Atrial fibrillation detection"],"falsifier":"Run the identical 1D-CNN pipeline with a patient-disjoint 5-fold split—ensuring no beats from the same recording appear in both training and test folds—and compare accuracy; if the 98.66–99.24% range collapses, the original cross-patient generalization claim is refuted. Additionally, check for duplicated beats introduced by the resampling step: any duplicate crossing a fold boundary would inflate the metric.","tokens_in":8444,"feed_emoji":"🫀","tokens_out":6851,"duration_ms":58502,"temperature":0.7,"pith_summary":"The paper tries to establish that a comparatively simple 1D convolutional neural network, trained on individual ECG heartbeats, can tell apart normal beats and four common arrhythmia types with near-perfect accuracy. It reports 98.66% to 99.24% accuracy across the five folds of a stratified cross-validation, with per-class precision, recall, and F1 scores all near or above 0.98. The motivation is practical: reliable automatic arrhythmia detection could support remote monitoring, assistive diagnosis, and early warning before serious cardiac events. The paper also argues, from a survey of medical professionals, that such tools are welcome in clinical practice if false negatives are minimized.","feed_headline":"One CNN sorts five heartbeat types at up to 99% accuracy","feed_subtitle":"Stratified five-fold testing keeps every fold above 98.6 percent, pointing toward real-time ECG screening tools","key_machinery":"The mechanism that carries the argument is the 1D-CNN beat classifier plus the validation protocol around it. Heartbeats are extracted from the raw ECG using sym4 wavelet detection, z-score normalized, and fed as one-dimensional signals through three convolutional blocks (kernel sizes 7, 5, 3; filters 32, 64, 128) with batch normalization and pooling, then dropout-regularized fully connected layers and a softmax output. Stratified 5-fold cross-validation ensures every beat is used once for testing, and per-class resampling to 8000 beats makes the training set balanced. For the binary branch, a Bi-LSTM with 100 hidden units over extracted morphological features (heart rate, R-R intervals, spectral entropy, instantaneous frequency) is the mechanism.","core_discovery":"The central discovery, stated as the authors would state it, is that a relatively shallow 1D-CNN can classify five ECG beat types—normal, left and right bundle branch block beats, premature ventricular contraction, and premature atrial contraction—with a maximum fold accuracy of 99.24% and a minimum of 98.66% after each class is resampled to 8000 beats. Combined fold confusion matrices show class-wise precision between roughly 0.977 and 0.996 and specificity above 0.994. In the earlier binary stage, a bidirectional LSTM distinguished normal from atrial-fibrillation signals with 92.44% testing accuracy and outperformed decision tree, naïve Bayes, and plain neural network classifiers, which the paper takes as evidence that sequence models fit ECG time series well.","pith_inferences":["Because the dataset is resampled to equal class sizes before the stratified split, the reported 98–99% accuracy is a class-balanced number, not a prevalence-weighted one; a deployment tool would need recalibration to real-world arrhythmia rates.","The per-beat split treats consecutive beats from one patient as independent, so the figures likely overstate cross-patient generalization; a per-patient split would likely be lower.","A natural stress test is to run the same architecture on noise-contaminated leads or on recordings from different ECG hardware; the sym4 wavelet preprocessing may not transfer cleanly to wearable-grade signals."],"forward_implications":["A heartbeat-level 1D-CNN can serve as the core of a five-class arrhythmia screener with per-class F1 scores near 0.98, because the model itself does the feature extraction from raw beat morphology.","The stable folds imply that the reported accuracy is not an artifact of one lucky split; every fold stays above 98.6% after class balancing.","The binary Bi-LSTM result means one model can reliably separate normal rhythm from atrial fibrillation before escalation to multi-class analysis.","The clinician survey and web portal suggest the model family is ready to be embedded in a real-time or near-real-time assistive tool, not just an offline benchmark."],"supporting_citations":[{"why":"Supplies the single-lead ECG recordings used for the binary normal-versus-atrial-fibrillation classifier.","marker":"[16]"},{"why":"Supplies the annotated multi-class heartbeat data on which the five-class CNN is trained and tested.","marker":"[18]"},{"why":"Provides the stratified cross-validation scheme that produces the reported fold-stable accuracy.","marker":"[20]"},{"why":"Provides the wavelet-based QRS detection and beat extraction that prepare the CNN input.","marker":"[22]"},{"why":"Provides the resampling technique used to balance the class distribution before cross-validation.","marker":"[23]"},{"why":"Defines the precision, recall, F1, and specificity metrics used for per-class evaluation.","marker":"[21]"},{"why":"Supplies the review of deep learning ECG classifiers that motivates the Bi-LSTM and CNN architecture choices.","marker":"[11]"}],"fun_headline_variants":["CNN hits 99.2% on five-type ECG arrhythmia detection","Five heartbeat types, one CNN, 99% accuracy","ECG AI classifies five arrhythmias with 99% precision","Bi-LSTM+CNN: arrhythmia screening reaches 99% accuracy","Multiclass ECG beat classifier hits 99% across five types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on treating balanced, individual heartbeats as independent samples; if the same patient's beats appear in both training and testing folds, the 98–99% accuracies are not a measure of how the model would perform on a new patient.","fun_headline_variants_meta":{"raw":{"variants":["CNN hits 99.2% on five-type ECG arrhythmia detection","Five heartbeat types, one CNN, 99% accuracy","ECG AI classifies five arrhythmias with 99% precision","Bi-LSTM+CNN: arrhythmia screening reaches 99% accuracy","Multiclass ECG beat classifier hits 99% across five types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3589,"prompt_tokens":958,"completion_tokens":2631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2540}},"tokens_in":574,"tokens_out":2631,"duration_ms":16098,"temperature":1.0,"reasoning_tokens":2540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:33:21.465874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical 1D-CNN pipeline with a patient-disjoint 5-fold split—ensuring no beats from the same recording appear in both training and test folds—and compare accuracy; if the 98.66–99.24% range collapses, the original cross-patient generalization claim is refuted. Additionally, check for duplicated beats introduced by the resampling step: any duplicate crossing a fold boundary would inflate the metric.","supporting_citations":[{"cited_title":"A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross- Validation in Imbalanced Learning,","cited_arxiv_id":null,"evidence_quote":"Provides the stratified cross-validation scheme that produces the reported fold-stable accuracy."},{"cited_title":"QRS Wave Detection In Matlab Using Wavelet Transform,","cited_arxiv_id":null,"evidence_quote":"Provides the wavelet-based QRS detection and beat extraction that prepare the CNN input."},{"cited_title":"A Resampling Method for Imbalanced Datasets Considering Noise and Overlap,","cited_arxiv_id":null,"evidence_quote":"Provides the resampling technique used to balance the class distribution before cross-validation."}],"review_version":1}