{"id":"4b58044f-0236-41d8-a6a3-b8589bbe62ed","arxiv_id":"2412.05666","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An ensemble of two low-parameter CNNs reaches 99.92% accuracy on a Kaggle Alzheimer's MRI dataset, but SMOTE is applied before splitting the data, which can leak synthetic samples into the test set.","lead":"This paper proposes two small convolutional networks and an ensemble that averages their predictions to classify Alzheimer's disease stages from MRI images on a Kaggle dataset, reporting up to 99.92% accuracy. The practical appeal is low computational cost, but the reported results are undermined by a data-splitting flaw: SMOTE is applied before the train/test split.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SMOTE is applied before the train/test split (Algorithm 1 lines 4-5), so synthetic test images can be generated from training data; the 99.92% SMOTE accuracy is not an independent estimate and the headline result is unsupported.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing flaw: SMOTE is applied before the train/validation/test split, so the test set is contaminated by synthetic instances derived from training data. This is not a matter of disagreement with consensus or an internal inconsistency; it is a direct violation of the independence required for an unbiased performance estimate. The paper's strongest claim, the 99.92% SMOTE-scenario ensemble accuracy, is the headline result and the basis for the claimed superiority over prior work. Because the leakage undermines that number, the central claim is unsupported as stated, and the rejection verdict stands. I see no need to change the reader's verdict; the concern is decisive and concrete, and a simple split-before-SMOTE rerun would settle it.","tokens_in":20432,"tokens_out":1944,"duration_ms":20105,"concrete_test":"Modify Algorithm 1 to split the original dataset into train/validation/test first, then apply SMOTE only to the training partition (e.g., after line 5, run SMOTE on XTrain/YTrain only). Retrain both CNNs and the ensemble with the same hyperparameters on the untouched test set. If the SMOTE-scenario ensemble accuracy falls materially below the reported 99.92%, the reported result is an artifact of pre-split SMOTE leakage. As a secondary check, verify whether any test image in the original pipeline is a synthetic SMOTE interpolation by checking whether its nearest neighbor in the training set is an original image used to generate it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 applies SMOTE to the entire image set before splitting: line 4 computes XSMOTE, YSMOTE from all Images and Labels, and line 5 then partitions X, Y into training, validation, and test sets. SMOTE creates each synthetic instance as x_new = x_i + rand[0,1] * (x_ij - x_i), i.e., a convex combination of a minority sample and one of its k-nearest neighbors. When the split is performed after SMOTE, test-set instances can be synthetic images that are interpolations of training-set images, or original images that were used as neighbors to generate synthetic training instances. Either way, the test set is not independent of the training set, and the reported SMOTE-scenario metrics (ensemble accuracy 99.92%, precision/recall/F1 99.92%, Table 2; per-class results in Table 3; comparisons in Table 5) are optimistically biased and cannot be interpreted as generalization performance on unseen MRI scans. The NO-SMOTE results are also vulnerable to known duplicate images in this Kaggle dataset, but the SMOTE leakage alone is sufficient to invalidate the central claim of state-of-the-art four-class AD staging.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two low-parameter convolutional neural networks, IR-BRAINNET and Modified-DEMNET, and an ensemble that averages their softmax outputs for four-class Alzheimer's disease staging on the Kaggle Alzheimer's MRI dataset. The authors report 98.28% accuracy without SMOTE and 99.92% accuracy with SMOTE for the ensemble, claiming improvements over both individual CNNs and prior work, and they report FLOPs, memory use, and training times to support practical deployment. The central claim is that averaging two compact CNNs yields state-of-the-art accuracy on this dataset in both balanced and imbalanced settings.","tokens_in":20663,"tokens_out":4801,"duration_ms":46450,"significance":"If the evaluation were valid, the contribution would be moderately significant for resource-constrained clinical imaging: the models are compact, the ensemble is simple, and the paper includes useful efficiency estimates. The manuscript also has transparency strengths, including explicit pseudocode, hyperparameters, per-class tables, learning curves, and confusion matrices. However, the main empirical claim is undermined by a methodological flaw in the evaluation protocol: synthetic SMOTE instances are generated before the train/test split, so the reported SMOTE-scenario accuracies do not measure generalization to unseen MRI scans. The NO-SMOTE results are additionally vulnerable to known duplicate-image issues in this dataset. Because the headline results cannot be interpreted as valid estimates, the significance of the contribution as submitted is low.","major_comments":[{"comment":"SMOTE is applied before the train/validation/test split: Algorithm 1 line 4 computes XSMOTE, YSMOTE from all Images and Labels, and line 5 then splits X and Y. Equation (2) defines each synthetic instance as a convex combination of an existing minority sample and one of its nearest neighbors, so the test partition can contain synthetic images generated from training images, and original test images can have served as neighbors for synthetic training instances. The test set is therefore not independent of the training set. All SMOTE-scenario metrics in Table 2, Table 3, and Table 5, including the headline 99.92% accuracy, are optimistically biased and cannot be interpreted as performance on unseen MRI scans. To support the claims, the authors would need to fit SMOTE on the training partition only and keep the test set entirely original.","section":"Section 3.3, Algorithm 1, Eqs. (1)-(2)"},{"comment":"SMOTE is applied to image data as flat feature vectors, and Eqs. (1)-(2) generate new instances by pixel-wise linear interpolation between MRI scans. Such interpolations are not anatomically valid MRI images. As implemented, the test set contains these synthetic images, so the reported accuracy reflects the model's ability to classify interpolated pixel vectors at least as much as real clinical scans. Even if the split were corrected, the use of SMOTE as a data-generation method for MRI should be justified with evidence that the synthetic images are plausible, rather than assumed from accuracy improvements on a contaminated test set.","section":"Section 3.3, Table 1"},{"comment":"The Kaggle Alzheimer's dataset is known to contain duplicate and near-duplicate images, and the paper does not report any deduplication or subject-level partitioning. The NO-SMOTE results, including the ensemble's 98.28% accuracy in Table 2 and the per-class results in Table 3, are therefore also at risk of inflation from training/test overlap of identical or near-identical images. The authors should demonstrate that no patient or image-level overlap exists between the training and test partitions before the NO-SMOTE generalization claim can be accepted.","section":"Section 3.2, Section 4.4, Table 2"}],"minor_comments":[{"comment":"The text says \"Fig. 8 illustrates the ROC curves,\" but the figure is numbered Fig. 6; the cross-reference should be corrected.","section":"Section 4.4, Fig. 6 reference"},{"comment":"Reported values are inconsistent across the abstract, text, and tables: the abstract and text give e.g. 99.80%, 99.72%, and 99.92%, while Table 2 shows rounded values of 0.99, and Table 3's values are rounded to two decimals. Exact metrics should be reported consistently.","section":"Table 2, Table 3, and abstract"},{"comment":"The text states a MOD recall of 92.30% in the NO-SMOTE scenario, but Table 3 shows 0.92; the text also cites a precision of 98.84% for MildDemented while Table 3 shows 0.98. The text should be aligned with the table.","section":"Section 4.4, Table 3 discussion"},{"comment":"The training-set sizes stated in the text (4,608 images for NO-SMOTE and 9,216 for SMOTE) are inconsistent with the stated 70/10/20 split of 6,400 and 12,800 total images, which would yield 4,480 and 8,960 training images. This discrepancy should be explained or corrected.","section":"Table 4"},{"comment":"There are several typos: \"Preprcoess\" in Algorithm 1 line 5, \"Psuedo-Algorithm\" in the algorithm title, \"DEMENT\" in Table 5, and \"Modified-DEMENT\" in the Fig. 3 discussion. These should be corrected to \"DEMNET.\"","section":"Algorithm 1, Table 5, and Fig. 3 caption"}],"recommendation":"reject","confidential_remarks":"The SMOTE-before-split protocol is a load-bearing flaw: it invalidates the principal quantitative claim, not merely a secondary analysis. Fixing it requires rerunning all experiments with a presplit SMOTE procedure and a subject-disjoint, deduplicated test split. That is beyond a normal minor or major revision of the submitted results, so I recommend rejection. If the authors redo the evaluation and the revised numbers remain competitive, a fresh submission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one-paragraph version: this paper proposes two low-parameter CNNs—IR-BRAINNET and Modified-DEMNET—and an ensemble that averages their softmax outputs for four-class staging of the Kaggle Alzheimer's MRI dataset. The architecture details are concrete and the authors are careful about reporting FLOPs, memory, training times, and even Wilcoxon tests showing the ensemble outperforms the individual models. That part is honest, workmanlike engineering on a well-worn benchmark.\n\nThe problem is that the SMOTE scenario, which produces the headline 99.92% accuracy, is invalid. Algorithm 1 applies SMOTE to the entire image set before the split (lines 4–5). SMOTE creates synthetic instances by interpolating between a minority sample and one of its neighbors. When you split after SMOTE, test-set instances can be convex combinations of training images, or original images that served as neighbors to synthetic training images. Either way the test set is not independent. The 99.92% number is an in-sample estimate, not a measure of generalization. The paper even shows the pseudocode, so this isn't hidden; it's just a serious methodological error.\n\nThe NO-SMOTE results are less contaminated but still not solid: the authors use a single fixed split, report no error bars, don't address the known duplicate images in this Kaggle dataset, and don't release code. So the NO-SMOTE 98.28% accuracy is plausible but not independently verifiable. The comparison to prior work (DEMNET, ADD-Net, etc.) is also apples-to-oranges because the field consistently uses different splits and sometimes different preprocessing; the table in Section 5 doesn't control for that.\n\nWhat's genuinely new here is limited: IR-BRAINNET and Modified-DEMNET are specific architecture variants with low parameter counts, and the ensemble averaging is straightforward. If the evaluation were clean, this would be a modest incremental contribution to resource-constrained AD classification. As it stands, the central empirical claim is unsupported.\n\nWho should read this? Someone mining the literature for architecture ideas might skim the model descriptions, and anyone teaching evaluation pitfalls could use the Algorithm 1 leak as a textbook example of train/test contamination. But as a research result, I wouldn't cite it or send it to a serious referee in its current form. If the authors rerun with SMOTE applied only inside the training folds, tackle the duplicates, and provide a reproducible split, the comparison could be worth another look at a workshop level.","headline":"The 99.92% SMOTE accuracy is a training-set leak, not a real result; the paper's architecture work is competent but the evaluation invalidates the main claim.","tokens_in":21176,"tokens_out":3398,"would_cite":false,"duration_ms":29385,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Averaging two compact CNNs classifies four stages of Alzheimer's from MRI at 98.28% accuracy, rising to 99.92% with oversampling.","keywords":["Alzheimer's disease","dementia","MRI images","convolutional neural network","ensemble learning","SMOTE","transfer learning","deep learning"],"falsifier":"Re-run the experiment with SMOTE applied only inside the training split — for example, generating synthetic instances within each fold of stratified cross-validation — and compare the ensemble's test accuracy with the reported 99.92%; a material drop would show the published SMOTE number was inflated by leakage. A complementary check is to record, for every SMOTE-scenario test image, the distance to its nearest training image: if many test samples sit at near-zero distance from training samples, they are essentially memorized rather than classified.","tokens_in":20215,"feed_emoji":"🧠","tokens_out":11553,"duration_ms":89007,"temperature":0.7,"pith_summary":"The paper sets out to show that accurate four-class staging of Alzheimer's disease from brain MRI does not require large models: two deliberately compact convolutional networks, combined by averaging their probability outputs, reach 98.28% test accuracy on an imbalanced public dataset, and 99.92% after rebalancing the minority classes with the Synthetic Minority Over-sampling Technique (SMOTE), which creates new images by interpolating between existing ones. The authors argue this matters because many hospitals lack the GPU resources that heavy CNN architectures demand, so small models that still catch early stages of dementia — especially the tiny Moderate Demented class — could make automated screening practical. The central claim is that the average of two diverse low-parameter networks is more accurate than either network alone (Wilcoxon p-values of 2.9e-3 and 5.2e-6), and that the same ensemble beats earlier models on this dataset in both balanced and imbalanced settings.","feed_headline":"Averaged CNNs hit 99.92% on Alzheimer's MRI staging","feed_subtitle":"Two low-parameter networks, combined by averaging, beat heavier rivals on four-class dementia classification.","key_machinery":"The load-bearing mechanism is the average operator $\\frac{1}{n}\\sum_{i=1}^{n}\\mathrm{Output}_{CNN_i}$ applied to the two networks' Softmax vectors; averaging, the paper argues, reduces prediction variance across architecturally distinct CNNs and costs only linear time in the number of test images. The two members are IR-BRAINNET, a six-convolutional-layer network whose second layer is initialized with the weights of VGG-19's third convolutional layer, and Modified-DEMNET, which keeps DEMNET's convolutional blocks but replaces two max-pooling layers with average pooling and compresses the fully connected head to a single 100-neuron dense layer. SMOTE supplies the synthetic diversity: each new image is a convex combination of a minority-class sample and one of its chosen nearest neighbours, $x_{new}=x_i+\\mathrm{rand}[0,1]\\cdot(\\hat{x}_{ij}-x_i)$, and the training pipeline then fine-tunes both networks with the Adam optimizer for 50 epochs.","core_discovery":"On its own terms, the paper's finding is empirical: averaging the final Softmax probability vectors of IR-BRAINNET (about 1.8 million parameters, with one convolutional layer initialized from pre-trained VGG-19 weights) and Modified-DEMNET (a slimmed-down DEMNET with mixed max and average pooling) yields a four-class dementia classifier that outperforms both constituent networks. In the imbalanced no-SMOTE scenario the ensemble scores 98.28% accuracy, 98.80% precision and 96.28% recall; after SMOTE balancing it scores 99.92% on all four metrics, including perfect precision and recall for the Moderate Demented minority class. The paper reads this as evidence that simple output averaging is an effective variance-reduction strategy for low-resource AD diagnosis, and that the synthetic diversity introduced by SMOTE is what carries performance above 99.9%.","pith_inferences":["Because the paper applies SMOTE before the 70/10/20 split, some test images are synthetic points interpolated between training samples; if SMOTE is confined to the training fold, the 99.92% figure would likely fall, making the no-SMOTE 98.28% the more honest estimate of real-world accuracy.","The ensemble's gain presumably comes from the two networks making different errors; counting the test images on which IR-BRAINNET and Modified-DEMNET disagree, and verifying that the average resolves most of them, would test that mechanism directly.","The comparison against earlier models is made under different preprocessing, splitting and oversampling protocols, so a shared-protocol benchmark with identical folds and identical SMOTE handling could settle whether the compact ensemble genuinely beats heavier architectures.","The authors' stated plan to train the ensemble as a single network, or to replace averaging with fuzzy integrals, offers a direct way to test whether linear averaging is already close to the ceiling for these two backbones."],"forward_implications":["A four-class AD staging system can run in about 14 MB of memory for the two networks combined, putting it within reach of hospital workstations and edge devices that cannot host heavy CNN backbones.","Simple averaging of Softmax outputs buys a statistically significant accuracy gain over both constituent networks, so ensembling low-parameter models is a cheap upgrade path for medical-image classifiers.","The Moderate Demented minority class, the one clinicians most need not to miss, is detected at 100% precision in both scenarios and at 92-100% recall, suggesting the pipeline is usable for early-intervention screening.","Rebalancing the public dataset with SMOTE adds roughly 1.6 percentage points of accuracy on top of the ensemble, indicating that class imbalance, not model capacity, is the dominant remaining source of error."],"supporting_citations":[{"why":"Supplies the four-class Alzheimer's MRI dataset on which both CNNs are trained and all accuracy figures are computed.","marker":"[27]"},{"why":"DEMNET, the prior model whose architecture Modified-DEMNET adapts and the main accuracy baseline to beat.","marker":"[7]"},{"why":"Defines the SMOTE oversampling technique used to generate synthetic minority-class images.","marker":"[29]"},{"why":"The ensemble-learning review that grounds the claim that averaging predictions reduces variance and improves generalization.","marker":"[24]"},{"why":"ImageNet, the source of the pre-trained weights transferred into IR-BRAINNET's second convolutional layer.","marker":"[26]"},{"why":"ADD-Net, the closest prior four-class competitor on the same dataset, outperformed in the comparison table.","marker":"[31]"},{"why":"A prior binary CNN on the same dataset whose 99.98% accuracy serves as a comparison point in Table 5.","marker":"[15]"},{"why":"TLBS-optimized deep CNN on the Kaggle dataset, another four-class comparison point in Table 5.","marker":"[37]"}],"fun_headline_variants":["Ensemble of lean CNNs hits 99.92% on Alzheimer's MRI","Averaging two CNNs yields 99.92% accuracy for dementia","SMOTE plus averaging: MRI dementia detection reaches 99.92%","Low-parameter CNN ensemble achieves 99.92% Alzheimer's accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The SMOTE-scenario result rests on the assumption that the test set contains genuinely unseen data, but the paper's pipeline applies SMOTE to the whole dataset before the 70/10/20 split, so a synthetic test image can be a point interpolated between two training images; if that contamination is real, the reported 99.92% accuracy would not survive an honest train/test separation.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble of lean CNNs hits 99.92% on Alzheimer's MRI","Averaging two CNNs yields 99.92% accuracy for dementia","SMOTE plus averaging: MRI dementia detection reaches 99.92%","Low-parameter CNN ensemble achieves 99.92% Alzheimer's accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000516,"raw_usage":{"total_tokens":2572,"prompt_tokens":1082,"completion_tokens":1490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":1408}},"tokens_in":698,"tokens_out":1490,"duration_ms":11197,"temperature":1.0,"reasoning_tokens":1408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:28:19.444908+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment with SMOTE applied only inside the training split — for example, generating synthetic instances within each fold of stratified cross-validation — and compare the ensemble's test accuracy with the reported 99.92%; a material drop would show the published SMOTE number was inflated by leakage. A complementary check is to record, for every SMOTE-scenario test image, the distance to its nearest training image: if many test samples sit at near-zero distance from training samples, they are essentially memorized rather than classified.","supporting_citations":[{"cited_title":"In: www.kaggle.com","cited_arxiv_id":null,"evidence_quote":"Supplies the four-class Alzheimer's MRI dataset on which both CNNs are trained and all accuracy figures are computed."},{"cited_title":"In: 2009 IEEE Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"ImageNet, the source of the pre-trained weights transferred into IR-BRAINNET's second convolutional layer."},{"cited_title":"Zikria, Ahmed G, et al (2022) ADDNet: An Effective Deep Learning Model for Early Detection of Alzheimer Disease in MRI Scans","cited_arxiv_id":null,"evidence_quote":"ADD-Net, the closest prior four-class competitor on the same dataset, outperformed in the comparison table."},{"cited_title":"Multimedia Tools and Applications 83:3767–3799","cited_arxiv_id":null,"evidence_quote":"A prior binary CNN on the same dataset whose 99.98% accuracy serves as a comparison point in Table 5."},{"cited_title":"Mohana Roopa, B","cited_arxiv_id":null,"evidence_quote":"TLBS-optimized deep CNN on the Kaggle dataset, another four-class comparison point in Table 5."}],"review_version":1}