{"id":"098cb601-6c95-42a4-8e6b-3d34dd597746","arxiv_id":"2505.09812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Oversampling and SMOTE improve accuracy of stroke prediction models on the fedesoriano Kaggle dataset, with Random Forest and SVM exceeding 99% test accuracy under oversampling, though clinical sensitivity remains limited.","lead":"This paper compares five machine learning models for predicting stroke risk using a public dataset of 5,110 patient records. It finds that random forests, SVM, and XGBoost reach high accuracy after balancing the data, but sensitivity remains a limiting factor for clinical use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported >99% accuracy likely stems from evaluating on a test set that was balanced before splitting, so synthetic or duplicated minority instances inflate the metrics; the manuscript never states that balancing was confined to training.","rationale":"After reading the full text and the reader's verdict, I find the single most load-bearing concern is not merely the absence of external validation but a concrete and very likely leakage mechanism in the evaluation protocol. Section II-E describes balancing the dataset; Section V-A then reports 'test accuracy' on the balanced data. The natural reading is that resampling happened before the split, and the manuscript gives no indication that the test fold was excluded from resampling. Under that protocol, oversampling duplicates real minority patients across the split, so the same patient appears in both training and test; SMOTE synthesizes minority instances that are not independent of training. Both mechanisms can inflate accuracy and recall to the ~99% level reported. The paper's own abstract concedes sensitivity is limiting, which is hard to reconcile with recall 0.997 unless the test set was balanced. This is more than a generalization caveat; it calls into question the quantitative central claim. The reader's weakest_assumption already gestured at 'held-out split of the already-balanced data,' so I agree with the reader. The requested condition—clarify whether metrics were computed on synthetic or real instances—is exactly the right fix. I therefore recommend no change to the CONDITIONAL verdict, but I would elevate the condition from a request to a hard gate: the authors must either demonstrate split-before-balancing or rerun with that protocol before the 99% accuracy claims can be cited.","tokens_in":4675,"tokens_out":6287,"duration_ms":66067,"concrete_test":"Re-run the pipeline with a strict split-before-balancing protocol: split the original 5,110 records into train/test (or use k-fold) before any oversampling or SMOTE; apply balancing only to the training folds; evaluate on the untouched original test records; report accuracy, precision, recall, and F1 separately for real minority and majority instances. If RF and SVM no longer reach ~99% accuracy, or if recall on real stroke cases drops sharply, the reported headline numbers are artifacts of evaluating on synthetic or duplicated test instances. A secondary check is to inspect any released code for the order of resampling and splitting, and to count how many test-set rows are exact duplicates of training rows or SMOTE-generated; if nonzero, leakage is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II-E says oversampling and SMOTE were applied to balance the dataset, and Section V reports test accuracy on the resulting data, but the manuscript never states that balancing was applied only to the training fold. If the whole dataset was balanced before the split, the test set contains randomly duplicated real patients (oversampling) and synthetic patients (SMOTE). Duplicates of training rows can be memorized, and synthetic test rows are generated by the same procedure as training rows, so the reported figures do not measure generalization to new patients. The internal inconsistency supports this reading: the abstract says 'sensitivity remains a limiting factor for real-world clinical applications,' yet Section V-C reports Random Forest recall = 0.997 and accuracy 99.02% under oversampling; on the original 4.87%-positive distribution such numbers are implausible, while on a balanced test set they are trivial. No code, data split description, or confidence intervals are provided, so the ambiguity cannot be resolved from the text. The feature importance and SHAP analyses in Section VI inherit the same risk if they were computed on balanced data. This concern is load-bearing because the paper's headline claim of >99% accuracy is exactly what would be invalidated by leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a comparative evaluation of five machine learning classifiers (Logistic Regression, Random Forest, Decision Tree, SVM, and XGBoost) on the publicly available Stroke Prediction Dataset (5,110 records, 4.87% stroke cases). The authors describe a preprocessing pipeline that imputes missing BMI values with an iterative random forest imputer, encodes categorical variables, and applies oversampling, undersampling, and SMOTE to address class imbalance. They report that SVM and Random Forest achieve over 99% test accuracy under oversampling, with Random Forest reaching precision, recall, and F1 of 0.997. The paper also presents feature importance analyses, including a SHAP-based comparison for the full dataset and for elderly patients aged 65–80, and concludes that age, average glucose level, and BMI are the most influential predictors while sensitivity remains a limiting factor for clinical use.","tokens_in":4829,"tokens_out":2845,"duration_ms":30958,"significance":"If the reported accuracy figures were valid, the paper would provide a useful benchmark for stroke prediction on a widely used public dataset and a reasonable illustration of imbalance-handling techniques. The SHAP analysis for the elderly subgroup is a constructive direction. However, the central empirical claim is undermined by an unresolved methodological ambiguity: the manuscript does not state whether class balancing was applied before or after the train/test split. Because the reported metrics are the paper's main contribution, the current version cannot be considered reliable as a scientific record. The paper also lacks confidence intervals, a detailed data-split description, or code, which limits reproducibility. These issues must be resolved before the findings can be assessed.","major_comments":[{"comment":"The manuscript never states whether balancing was applied to the full dataset before splitting or only to the training partition. If oversampling and SMOTE were applied before the split, then the test set contains duplicated real patients and synthetic patients generated from training data, so the reported test accuracy, precision, and recall do not measure generalization to new patients. This concern is load-bearing because the headline result of 99.28% accuracy for SVM under oversampling depends entirely on this choice. The authors must specify the exact pipeline order and, ideally, re-run the evaluation with balancing performed inside each training fold only.","section":"Section II-E and Section V-A"},{"comment":"There is an internal inconsistency between the abstract's statement that 'sensitivity remains a limiting factor for real-world clinical applications' and the reported Random Forest recall of 0.997 under oversampling. On the original imbalanced distribution (4.87% positives), such a recall is implausible, whereas on a balanced test set it is trivially achievable. This tension supports the interpretation that the test set was balanced before splitting, and it must be resolved by reporting performance on an untouched, original-distribution test set.","section":"Section V-C and Abstract"},{"comment":"Hyperparameter tuning with RandomizedSearchCV appears to have been performed on the balanced dataset, since the tuning stage is described after the balancing techniques are introduced in Section II-E. If cross-validation during tuning also includes synthetic or duplicated minority instances, the selected hyperparameters may be optimized for artifacts of the balancing procedure rather than for genuine signal. The authors should clarify the timing of balancing relative to hyperparameter tuning and ensure that no information from synthetic or duplicated instances leaks into model selection.","section":"Section IV-B"},{"comment":"The paper reports accuracy differences as small as 0.26% (e.g., SVM 99.28% versus Random Forest 99.02% under oversampling) without confidence intervals, statistical significance tests, or repeated-seed variation. Given that the original dataset contains only 249 positive cases, such differences are likely within sampling noise. The authors should report confidence intervals or use repeated stratified cross-validation with variance estimates to support claims of relative model superiority.","section":"Section V-A"}],"minor_comments":[{"comment":"The 'previous studies' cited in the research gap are Kaggle forum posts and dataset discussion pages, not peer-reviewed comparative machine learning studies; the authors should cite actual published work on stroke prediction with class imbalance and feature importance.","section":"Section I-C"},{"comment":"The phrase 'By combining these techniques' is misleading because the experimental results in Section V treat oversampling, undersampling, and SMOTE as separate alternatives, not as a combination; the text should be reworded to reflect the actual procedure.","section":"Section II-E"},{"comment":"Hyperparameter names are inconsistently formatted (e.g., 'n estimators' should be 'n_estimators', 'max depth' should be 'max_depth') and the table would benefit from a consistent code-style notation.","section":"Table I"},{"comment":"The reported test accuracies are not accompanied by the specific train/test split ratio, random seed, or number of repeated runs, so the reader cannot assess the stability of the numbers.","section":"Section V-B"},{"comment":"The text refers to 'Feature 7', 'Feature 8', and 'Feature 9' without providing a feature-to-index mapping anywhere in the paper; please include a table or list that defines these indices.","section":"Section VI"},{"comment":"References [7] and [8] cite blog posts rather than the original methodological sources; for SMOTE, the authors should cite Chawla et al. (2002), and for class imbalance handling, a more standard textbook or survey reference would be appropriate.","section":"References"},{"comment":"The notation 'Precision, Recall, and F1-score all at 0.997' would be clearer if each metric were reported with a model-specific label and an explicit number of significant figures, since the classification report likely gives more decimal places.","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an initial student project report. The central methodological ambiguity around data leakage is serious, but it is potentially fixable by re-running the evaluation with balancing confined to training folds and reporting performance on an original-distribution test set. I would not recommend acceptance in the current form. The novelty is limited relative to the existing machine learning benchmark literature on this dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a routine benchmark on the public Kaggle stroke dataset, and the headline numbers are probably not real. The manuscript never states that class balancing was applied only to the training fold; it says the dataset was balanced. If the split happened after balancing, the test set contains duplicates or synthetic positives, and the 99% accuracy / 0.997 recall become trivial rather than evidence of generalization. The abstract itself concedes sensitivity is limited, which contradicts the reported recall unless the test set is balanced.\n\nWhat the paper does well: it is clearly organized, covers five models and three balancing techniques, and the age-restricted analysis (65–80) is a small but genuine extension, with SHAP plots adding some interpretability. The authors also deserve credit for acknowledging that sensitivity is a problem in the abstract, even if the results section doesn't match that caution.\n\nSoft spots: the evaluation leak is load-bearing. There are no confidence intervals, no repeated cross-validation, no code, and no external validation. The reference list is thin, with an unusual number of blog posts and one questionable journal citation, but that's minor relative to the methodological issue. The feature importance and SHAP analyses inherit the same risk if computed on balanced data.\n\nBottom line: this is a paper for readers who want a quick comparison of standard ML models on a well-known dataset, but the reported results are not reliable as they stand. It could be salvageable with a corrected evaluation (split before balancing, SMOTE inside CV, CIs, code release). If the authors can do that, it's a modest but legitimate benchmark contribution; if not, there's not much here.\n\nRecommendation: I'd send it to peer review, but with a clear request for major revision focused on the evaluation. A serious referee would catch the leak immediately, and the paper would be a useful cautionary example either way.","headline":"The paper's 99% accuracy is likely a balancing-before-split artifact; the age-specific feature analysis is the only fresh part.","tokens_in":5449,"tokens_out":3051,"would_cite":false,"duration_ms":31006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Random Forest and SVM reach 99% stroke-prediction accuracy after resampling","keywords":["stroke prediction","machine learning","class imbalance","SMOTE","oversampling","Random Forest","feature importance","SHAP"],"falsifier":"Split the original 5,110 records into train and test before applying any balancing, then train with oversampling or SMOTE only on the training side and evaluate on the untouched test fold that keeps the natural 4.87% stroke rate; if Random Forest and SVM accuracy falls well below 99% and sensitivity is low, the reported metrics depend on evaluating already-balanced data.","tokens_in":4399,"feed_emoji":"🧠","tokens_out":5974,"duration_ms":56404,"temperature":0.7,"pith_summary":"The paper compares five machine learning models for stroke prediction on a public 5,110-patient dataset and argues that the main obstacle is class imbalance, not model choice. After correcting the imbalance with oversampling or SMOTE, the paper reports that Random Forest and Support Vector Machines exceed 99% test accuracy, with Random Forest reaching precision, recall, and F1-scores of 0.997 under oversampling. It also claims that age, average glucose level, and BMI are the most influential predictors, and that these rankings shift for patients aged 65-80, where work type, glucose, hypertension, and heart disease gain weight. If the claims hold, the practical message for stroke-risk screening is that tuned ensemble models plus a balancing step can give very high discrimination on benchmark data, but the authors note sensitivity must improve before clinical deployment.","feed_headline":"Random Forest and SVM hit 99% stroke accuracy after resampling","feed_subtitle":"In 5,110 records, age, glucose, and BMI drive risk; sensitivity still limits real-world clinical use.","key_machinery":"The load-bearing mechanism is the combination of resampling and hyperparameter-tuned classifiers. Oversampling duplicates minority-class stroke records; SMOTE (creating synthetic minority examples by interpolating between neighboring samples) generates new ones instead, and randomized hyperparameter search tunes each model before evaluation. This balancing step is what allows Random Forest and SVM to reach the reported 99% accuracy, while tree-based feature importance and SHAP (Shapley-value-based feature attribution) plots carry the interpretability claim about age, glucose, BMI, and the elderly-specific shift in feature rankings.","core_discovery":"The central claim, stated on the paper's own terms, is that Random Forest and SVM benefit dramatically from resampling: with oversampling they reach 99.02% and 99.28% test accuracy respectively, and Random Forest achieves precision, recall, and F1 all at 0.997. SMOTE yields slightly lower but still strong results (Random Forest 93.98%, SVM 93.52%), while undersampling degrades accuracy to 74-83%, leading the authors to conclude oversampling and SMOTE are the appropriate fixes for this severely imbalanced dataset. On feature importance, age is consistently the top predictor, followed by average glucose level and BMI; restricting the cohort to patients aged 65-80 makes work type and glucose level more influential and raises the prominence of hypertension and heart disease. The authors accordingly claim that ensemble models handle imbalance better than linear models, and that age-specific predictive models would increase clinical usefulness, while acknowledging that sensitivity remains the limiting factor for real-world stroke screening.","pith_inferences":["The 99% figure is best read as an upper bound: because the test split comes from the same dataset after synthetic or duplicated samples were added, it cannot be read as expected performance on a fresh hospital population with the natural 4.87% stroke rate.","A stronger validation design would split before balancing, train on balanced data, and test on an untouched, naturally imbalanced cohort; I would expect reported sensitivity to drop under that design.","The age-stratified SHAP result suggests a testable extension: build separate risk scores for the 65-80 band and compare net reclassification against a single all-ages model on external data.","Because SMOTE interpolates between existing minority points, it can hide poor generalization when the minority class is small; a direct check is to see how performance changes as the number of synthetic examples is reduced."],"forward_implications":["On this dataset, class imbalance rather than model choice appears to be the binding constraint: once balanced, even a kernel SVM reaches 99% test accuracy.","Because sensitivity remains the reported weak spot, clinical use would require threshold tuning or cost-sensitive learning, not simply maximizing accuracy.","XGBoost offers the most balanced precision-recall profile under SMOTE (F1 0.926), making it the candidate to prefer when missing a stroke is costly.","Screening instruments for patients aged 65-80 should weight glucose, work type, hypertension, and heart disease more heavily than general-population rankings suggest.","Age, average glucose level, and BMI are sufficient as the primary screening variables in a general adult population."],"supporting_citations":[{"why":"Supplies the 5,110-record Stroke Prediction Dataset, including the 4.87% stroke class imbalance the whole comparison is built on.","marker":"[5]"},{"why":"Provides the SMOTE algorithm used to generate synthetic minority-class samples.","marker":"[7]"},{"why":"Supplies the oversampling and undersampling techniques compared against SMOTE.","marker":"[8]"},{"why":"Gives the XGBoost model and feature-importance approach used in the comparison and SHAP analysis.","marker":"[9]"},{"why":"Supports the iterative imputation of missing BMI values and the multivariate feature analysis.","marker":"[6]"}],"fun_headline_variants":["Oversampling pushes stroke models to 99% accuracy","Age, glucose, BMI lead stroke risk in ML models","Resampling key to high stroke prediction accuracy","99% stroke accuracy, but sensitivity limits clinical use","Ensemble models beat linear ones on stroke data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that test accuracy on a held-out slice of the same dataset, measured after synthetic or duplicated stroke cases were added, reflects how well the model would predict strokes for patients it has never seen.","fun_headline_variants_meta":{"raw":{"variants":["Oversampling pushes stroke models to 99% accuracy","Age, glucose, BMI lead stroke risk in ML models","Resampling key to high stroke prediction accuracy","99% stroke accuracy, but sensitivity limits clinical use","Ensemble models beat linear ones on stroke data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3024,"prompt_tokens":867,"completion_tokens":2157,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":2082}},"tokens_in":483,"tokens_out":2157,"duration_ms":15677,"temperature":1.0,"reasoning_tokens":2082,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:22:19.585243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split the original 5,110 records into train and test before applying any balancing, then train with oversampling or SMOTE only on the training side and evaluate on the untouched test fold that keeps the natural 4.87% stroke rate; if Random Forest and SVM accuracy falls well below 99% and sensitivity is low, the reported metrics depend on evaluating already-balanced data.","supporting_citations":[{"cited_title":"Soriano, Stroke Prediction Dataset, Kaggle, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the 5,110-record Stroke Prediction Dataset, including the 4.87% stroke class imbalance the whole comparison is built on."},{"cited_title":"Chawla et al., SMOTE for Classification, Domino AI Blog, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the SMOTE algorithm used to generate synthetic minority-class samples."},{"cited_title":"https://blog.roboflow.com/handling-unbalanced-classes/","cited_arxiv_id":null,"evidence_quote":"Supplies the oversampling and undersampling techniques compared against SMOTE."},{"cited_title":"http://ijlbpr.com/uploadfiles/45vol13issue7pp249-253.pdf","cited_arxiv_id":null,"evidence_quote":"Gives the XGBoost model and feature-importance approach used in the comparison and SHAP analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the iterative imputation of missing BMI values and the multivariate feature analysis."}],"review_version":1}