{"id":"ce84a69f-a217-4a8b-99f9-f7a050bbbc8f","arxiv_id":"2505.09969","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A standard comparison of Logistic Regression, KNN, and Random Forest on the Cleveland heart disease dataset gives conflicting accuracy numbers and no code.","lead":"This paper compares three machine learning models for predicting heart disease from a standard 303-patient dataset and reports that Random Forest is the most accurate. The study is a routine benchmark with no new method, and the paper's own numbers are internally inconsistent.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Random Forest metrics are internally inconsistent across abstract, discussion, conclusion, and §3.5; no single evaluation run supports the headline 91% accuracy and F1 0.89.","rationale":"The reader's REJECT verdict is well supported. I looked for ways to rescue the central claim — for instance, that the abstract, discussion, and conclusion might be reporting different tuned models or different random splits — but the manuscript itself forecloses this: it describes a single 80/20 split, three models, and one 'best' model, and it states incompatible accuracy/F1 values for that model without any indication that they come from different runs or different hyperparameter settings. The confusion-matrix paragraph in §3.5 is arithmetically inconsistent with both the 91% accuracy claim and the adjacent classification report. Because no code, random seed, or hyperparameter values are provided, the only way to decide which number is real is to rerun the experiment, and that rerun cannot be performed from the manuscript as written. This is a soundness failure of the central quantitative claim, not merely a disagreement with the broader literature. A revised manuscript that reports one consistent set of metrics, specifies the exact split and tuning procedure, and provides code or a precise reproducibility statement could change this assessment, but the current version does not support its headline result.","tokens_in":8011,"tokens_out":3895,"duration_ms":37271,"concrete_test":"Reproduce the §3 pipeline on the public Cleveland heart-disease.csv (303 instances, 14 features). For a fixed grid of random seeds and GridSearchCV/RandomizedSearchCV parameter ranges consistent with the text, record Random Forest accuracy and F1 on the held-out 20%. Then check whether any single run simultaneously yields (a) 91% accuracy and F1 0.89; (b) a held-out set of size 61 or 64; and (c) a confusion matrix with 25 true negatives and 29 true positives out of 61 instances. Also check whether the displayed classification report with support 31/33 and accuracy 0.91 is producible by any model in the pipeline. If no run reproduces the abstract metrics and the §3.5 report, the headline claim is unsupported. Independently, recompute accuracy from the confusion-matrix text to confirm 54/61 = 88.5%, not 91%.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that Random Forest achieves 91% accuracy and F1-score 0.89 (abstract), cannot be assigned to any well-defined evaluation run. The manuscript gives at least three incompatible versions: abstract says 91% accuracy / 0.89 F1; discussion says 91% accuracy / 0.91 F1; conclusion says 89% accuracy / 0.91 F1. In §3.5, the confusion-matrix text states 25/29 class-0 correct and 29/32 class-1 correct, which is 54/61 = 88.5% accuracy, not 91%. The classification report printed immediately after shows class supports 31 and 33, whose sum is 64, while the accuracy line reports support 61 and macro/weighted rows report support 63; these numbers cannot come from one test set. Figure 4's caption says Logistic Regression achieved the highest accuracy, directly contradicting the text's claim that Random Forest outperformed the others. No code, random seed, or hyperparameter values are provided, so there is no basis for selecting among these incompatible numbers or for verifying that any of them came from a leakage-free single 80/20 split. The absence of a reproducible protocol is not cosmetic: the only quantitative evidence for the paper's main conclusion is a number the manuscript itself refutes multiple times.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a machine-learning pipeline for binary heart-disease prediction on the Cleveland heart-disease dataset (303 instances, 14 attributes). Three classifiers are compared: Logistic Regression, K-Nearest Neighbors, and Random Forest, with hyperparameter tuning via GridSearchCV and RandomizedSearchCV and an 80/20 train-test split. The abstract claims Random Forest achieved the best performance with 91% accuracy and an F1-score of 0.89; the discussion restates 91% accuracy but an F1-score of 0.91; the conclusion states 89% accuracy and an F1-score of 0.91. Section 3.5 includes a confusion matrix, a classification report, and a feature-importance figure, and the paper concludes that Random Forest is the most suitable model, while acknowledging the small dataset size as a limitation and calling for external validation.","tokens_in":8409,"tokens_out":3531,"duration_ms":33179,"significance":"If the reported results were reliable, this would be a modest benchmark comparison on a well-known public dataset, providing limited new methodological insight. The paper's explicit recognition of the small dataset and its claim to have avoided data leakage by fitting scalers on the training set only are positive features. However, the significance is heavily undercut by internal contradictions in the headline metrics, an internally inconsistent classification report, and the absence of code, random seeds, or hyperparameter values. The central claim—that Random Forest achieves roughly 91% accuracy—is not supported by any single, well-defined evaluation run described in the manuscript, and the paper therefore does not currently provide a dependable empirical contribution.","major_comments":[{"comment":"The headline Random Forest results are inconsistent across the manuscript. The abstract reports accuracy 91% and F1-score 0.89; §4 reports accuracy 91% and F1-score 0.91; §5 reports accuracy 89% and F1-score 0.91. These numbers cannot all describe the same model evaluated on the same test split, yet no explanation or reconciliation is offered. Since the central conclusion rests on these metrics, the paper does not currently state its main result in a coherent way.","section":"Abstract; §4 Discussion; §5 Conclusion"},{"comment":"The classification report and the confusion-matrix description are internally inconsistent. The report lists class supports 31 and 33, which sum to 64 test instances, while the accuracy line reports support 61 and the macro/weighted rows report support 63. For an 80/20 split of 303 samples, the test set has either 60 or 61 instances, so supports of 63 and 64 are impossible from a single split. The confusion-matrix text says 25 of 29 class-0 instances and 29 of 32 class-1 instances were correct, which corresponds to 54/61 = 88.5% accuracy, not the stated 91%. These contradictions prevent the reader from assigning any reported accuracy or F1-score to a concrete evaluation run.","section":"§3.5 Evaluation and Results"},{"comment":"Figure 4's caption states that Logistic Regression achieved the highest accuracy and that Random Forest was second, directly contradicting the text in §3.5 and §4, which claim Random Forest outperformed the other models. This is not a minor wording issue: it reverses the paper's central comparative conclusion. As printed, the manuscript does not establish which model produced the headline results.","section":"Figure 4 caption; §3.5 text"},{"comment":"The evaluation protocol is not reproducible from the information provided. The manuscript does not report the random seed used in train_test_split, the best hyperparameter values found by GridSearchCV or RandomizedSearchCV, or any code. Section 3.2 asserts that scaling was fitted on the training set only to prevent leakage, but nothing in the manuscript allows verification of this claim. Given the metric inconsistencies described above, the accuracy and F1 values cannot be traced to a well-defined procedure.","section":"§3.2 Data Preprocessing; §3.4 Model Training and Evaluation"},{"comment":"All performance claims are based on a single 80/20 split after hyperparameter tuning, with no repeated random splits, nested cross-validation, or variance estimates reported. On a dataset of 303 instances, selecting the best model after tuning on the same split risks overfitting the test set, and the reported single-point accuracy is not sufficient to support the general claim of clinical utility or model superiority.","section":"§3.4 and §3.5"}],"minor_comments":[{"comment":"The table title contains a typo: \"Tabel 1\" should be \"Table 1.\"","section":"§3.1"},{"comment":"Figure 1 and Figure 2 appear to present identical scatter plots with identical captions, and the caption text under Figure 2 repeats the Figure 1 caption verbatim. The authors should either remove the duplicate or provide distinct figures and captions.","section":"Figures 1 and 2"},{"comment":"The description of the 'thal' feature lists \"1: Normal, 3: Normal,\" which appears to be a typo; the standard encoding has distinct values for normal, fixed defect, and reversible defect.","section":"Table 1"},{"comment":"Figure 6's caption identifies the confusion matrix as being for the K-Nearest Neighbors classifier, but the surrounding paragraph and classification report discuss the best-performing model (stated to be Random Forest). The figure and report should be clearly labeled with the corresponding model.","section":"Figure 6 and surrounding text"},{"comment":"The classification report is presented as plain text without a model name or table caption; it should be formatted as a proper table and explicitly attributed to the model whose results are being reported.","section":"§3.5"},{"comment":"The reference list contains several entries that are not cited in the text, and the dataset itself (the UCI Heart Disease dataset) is not formally cited. Please add the appropriate dataset citation and remove or integrate uncited references.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's reported experimental results are internally contradictory (abstract, discussion, and conclusion give different accuracies/F1-scores; the classification report has impossible support counts; Figure 4's caption contradicts the text's main claim), and the evaluation protocol is not reproducible. These are load-bearing problems that cannot be resolved by local corrections within the scope of a revision; the experimental basis for the central claim is absent. The novelty of the methodological contribution is also very limited relative to the existing literature on this well-studied benchmark dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a textbook three-classifier comparison on the UCI Cleveland heart-disease dataset, a benchmark that already exists in the literature (including Ref 10, which does random forest on the same data with 10-fold CV). The paper does not claim a new method or new features. What it does well: the preprocessing section explicitly says scaling and transformations were fitted on the training set only, which is the right way to avoid leakage. The related work is on-topic and the writing is clear about the dataset's small size and limited diversity as limitations.\n\nThe soft spots are not minor. The central claim—Random Forest at 91% accuracy and 0.89 F1—does not survive contact with the rest of the manuscript. The discussion says 91% accuracy and 0.91 F1; the conclusion says 89% accuracy and 0.91 F1; the classification report in §3.5 shows a confusion matrix with supports 29 and 32 (total 61) but the printed report has supports 31 and 33 (total 64), and the macro/weighted rows say support 63. The accuracy line says 0.91 but the confusion matrix as described gives 54/61 = 88.5%. Figure 4's caption says Logistic Regression achieved the highest accuracy, directly contradicting the text that says Random Forest outperformed the others. In short, there is no single evaluation run that supports the headline number.\n\nEqually important, no code, no random seed, no hyperparameter values, and no data split details are given. The claim that hyperparameter tuning via GridSearchCV/RandomizedSearchCV was used cannot be checked, and model selection on a 303-sample dataset with a single 80/20 split is itself prone to overfitting the test set. These are not cosmetic concerns; the paper's only quantitative evidence is a number the manuscript itself refutes in several places.\n\nThe result, if true, would still be marginal. Random forest on Cleveland is not new, and 91% accuracy is in line with many prior reports. There is no external validation, no clinical endpoint, and no analysis of error cases.\n\nBottom line: this paper is for a reader who wants a brief survey of related work on this dataset plus a cautionary example of how reporting inconsistencies can undermine an otherwise simple benchmark. I would not bring it to reading group, and I would not cite it. It does not deserve a serious referee until the authors provide a single consistent evaluation protocol and the code or seed needed to reproduce it. Desk reject is the right call.","headline":"Routine Cleveland heart-disease benchmark whose headline accuracy is contradicted by the paper's own tables and captions; no code or seeds, so nothing is reproducible.","tokens_in":8807,"tokens_out":1847,"would_cite":false,"duration_ms":16317,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tuned Random Forest classifier is claimed to reach 91% accuracy and an F1-score of 0.89 for heart disease prediction on a 303-sample benchmark, beating Logistic Regression and KNN.","keywords":["heart disease prediction","Random Forest","Logistic Regression","K-Nearest Neighbors","hyperparameter tuning","Cleveland heart disease dataset","classification metrics","clinical decision support"],"falsifier":"Re-running the described pipeline on the public 303-sample heart-disease.csv with a fixed random seed and an 80/20 split would settle it: if the held-out test accuracy is below 90% or the F1-score is not around 0.89–0.91, the central claim collapses. Concretely, a test set of 61 samples with 54 correct predictions yields 88.5%, not 91%.","tokens_in":7811,"feed_emoji":"❤️","tokens_out":5757,"duration_ms":51214,"temperature":0.7,"pith_summary":"The paper aims to show that a standard machine-learning pipeline—data cleaning, feature scaling, training three classifiers, and hyperparameter tuning—can predict the presence of heart disease from 14 clinical measurements. The headline claim is that a tuned Random Forest reaches 91% accuracy and an F1-score of 0.89, beating Logistic Regression and K-Nearest Neighbors. If that holds, it gives clinicians a cheap, interpretable screening tool that runs on routine hospital data. The paper also argues that Random Forest's ensemble structure is what lets it capture nonlinear feature interactions that linear and distance-based models miss, while cautioning that the 303-sample dataset limits generalizability.","feed_headline":"Study: Random Forest hits 91% accuracy on heart disease data","feed_subtitle":"Tuned ensemble beats logistic regression and KNN on a 303-patient benchmark, though the paper's own numbers vary.","key_machinery":"The load-bearing mechanism is the Random Forest ensemble: a collection of decision trees whose individual predictions are averaged, which the paper uses to reduce variance and capture nonlinear feature interactions. Hyperparameter tuning via GridSearchCV and RandomizedSearchCV selects settings such as tree count and neighbor count, and the 80/20 train/test split with scaling fit only on the training set is meant to prevent leakage. The evaluation machinery—precision, recall, F1-score, and confusion matrix—is what supports the claim that performance is balanced across disease-positive and disease-negative patients.","core_discovery":"On the paper's own terms, the central discovery is that a Random Forest classifier, tuned with GridSearchCV and RandomizedSearchCV and evaluated on a single 80/20 split of the 303-sample heart-disease data, outperforms both Logistic Regression and KNN. The authors report an accuracy of 91% and an F1-score of 0.89 for Random Forest, with balanced precision and recall across the two classes. They attribute the result to the ensemble's ability to combine many decision trees, reduce overfitting, and handle complex feature relationships better than a linear model or a distance-based method. The same text later gives 89% accuracy and an F1-score of 0.91 in the conclusion, so the exact numbers are not fixed; the stable claim is that Random Forest gives the best overall balance of precision and recall among the three models tested.","pith_inferences":["Because the paper does not report a random seed, code, or exact hyperparameter values, the headline accuracy should be read as approximate until the protocol is reproduced; the manuscript's own conflicting numbers (91%, 89%, and a confusion matrix implying about 88.5%) reinforce this.","A straightforward extension would replace the single 80/20 split with repeated stratified k-fold cross-validation, which would give confidence intervals and almost certainly lower the point estimate.","The same pipeline applied to other public heart-disease datasets, such as the Hungarian or Statlog variants, would test whether Random Forest's advantage is dataset-specific or general."],"forward_implications":["If the 91% figure is reproducible, a tuned Random Forest can serve as a usable first-pass screening tool for heart disease from 14 routine clinical measurements.","Logistic Regression and KNN are presented as simpler alternatives that trade accuracy for interpretability or computational efficiency.","The paper's own limitation section implies the 303-sample result is a benchmark rather than a generalizable clinical model; larger and more diverse datasets are needed before deployment.","Hyperparameter tuning is treated as a necessary step, since untuned KNN and Random Forest do not reach the reported performance."],"supporting_citations":[{"why":"Prior random-forest model on the same Cleveland heart-disease dataset that this study directly extends.","marker":"[10]"},{"why":"Multi-classifier comparison on heart-failure data that provides the benchmark set this paper positions Random Forest against.","marker":"[12]"},{"why":"Ensemble stacking method reaching 96% accuracy on heart sounds, used to argue ensemble approaches are competitive.","marker":"[13]"},{"why":"Decision-tree risk model on Cleveland data with 86.53% accuracy, an earlier same-data baseline.","marker":"[14]"}],"fun_headline_variants":["Random Forest tops heart disease prediction at 91% accuracy","Ensemble model beats logistic regression and KNN in heart disease study","Tuned Random Forest predicts heart disease with 91% accuracy","Random Forest wins heart disease benchmark, but dataset is tiny","Ensemble outperforms linear and lazy learners in heart disease prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result assumes that all reported metrics come from one clean, consistent evaluation split, but the paper itself gives different accuracy figures: 91% in the abstract, 89% in the conclusion, and a confusion matrix that implies roughly 88.5%.","fun_headline_variants_meta":{"raw":{"variants":["Random Forest tops heart disease prediction at 91% accuracy","Ensemble model beats logistic regression and KNN in heart disease study","Tuned Random Forest predicts heart disease with 91% accuracy","Random Forest wins heart disease benchmark, but dataset is tiny","Ensemble outperforms linear and lazy learners in heart disease prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2649,"prompt_tokens":873,"completion_tokens":1776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1691}},"tokens_in":489,"tokens_out":1776,"duration_ms":12716,"temperature":1.0,"reasoning_tokens":1691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:18:47.530181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the described pipeline on the public 303-sample heart-disease.csv with a fixed random seed and an 80/20 split would settle it: if the held-out test accuracy is below 90% or the F1-score is not around 0.89–0.91, the central claim collapses. Concretely, a test set of 61 samples with 54 correct predictions yields 88.5%, not 91%.","supporting_citations":[{"cited_title":"Heart disease prediction system using random forest","cited_arxiv_id":null,"evidence_quote":"Prior random-forest model on the same Cleveland heart-disease dataset that this study directly extends."},{"cited_title":"A machine learning approach for chronic heart failure diagnosis","cited_arxiv_id":null,"evidence_quote":"Multi-classifier comparison on heart-failure data that provides the benchmark set this paper positions Random Forest against."},{"cited_title":"Chronic heart failure detection from heart sounds using a stack of machine -learning classifiers","cited_arxiv_id":null,"evidence_quote":"Ensemble stacking method reaching 96% accuracy on heart sounds, used to argue ensemble approaches are competitive."},{"cited_title":"Predicting the likelihood of heart failure with a multi level risk assessment using decision tree","cited_arxiv_id":null,"evidence_quote":"Decision-tree risk model on Cleveland data with 86.53% accuracy, an earlier same-data baseline."}],"review_version":1}