{"id":"4b279387-894f-416c-9b25-5ac35e9bb05a","arxiv_id":"2412.20235","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Combining SMOTE with a voting classifier on transfer-learned features improved diabetic retinopathy recall by 5.4 points, while brain tumor accuracy stayed near 99.85%.","lead":"Five pretrained image models were tested on brain tumor MRI and diabetic retinopathy photos, then paired with SMOTE resampling and classical machine learning classifiers. On the imbalanced diabetic retinopathy data, the hybrid pipeline lifted sensitivity by about 5.4 percentage points over the best single transfer-learning model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SMOTE-driven gains are confounded by a simultaneous switch to an ensemble and balanced class weights; a 2x2 ablation is required before attributing the recall improvement to SMOTE.","rationale":"The reader's weakest assumption identifies exactly the same confound: SMOTE is never tested alone, while the classifier and class weights change simultaneously. I agree with that assessment. The tables are internally consistent (90.17 + 1.97 = 92.14; 80.06 + 5.44 = 85.50; specificity 97.46 + 0.72 = 98.18), and the direction of the result is plausible, but the abstract's language is causal ('we integrate SMOTE ... which improves'), and the experimental design cannot rule out that the Voting Classifier's ensemble or balanced class weights produce the recall gain. The BT dataset result, where SMOTE did not enhance performance, further indicates that SMOTE may be inert in this pipeline. Secondary concerns such as in-sample CNN probabilities, lack of error bars, and possible test-set model selection from the 'top 5' reinforce the conditionality rather than overturn the paper's contribution. Since the reader already returned CONDITIONAL, no verdict adjustment is needed.","tokens_in":8378,"tokens_out":4213,"duration_ms":44232,"concrete_test":"Run a 2x2 ablation on the APTOS 2019 test split using the authors' Xception probabilities: (i) Voting Classifier with default settings, no SMOTE; (ii) same with class_weight='balanced' on classifiers that support it; (iii) SMOTE-resampled probabilities without balanced weights; (iv) SMOTE plus balanced weights. If condition (ii) attains recall within sampling noise of 85.50%, or if condition (iii) does not, then SMOTE is not the causal driver of the reported gain. Report bootstrap 95% confidence intervals for each condition to assess whether the 5.44-point gap is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests entirely on comparing Xception transfer learning alone (Table V) with a Voting Classifier trained on SMOTE-resampled Xception probabilities (Table VI). The pipeline changes three variables at once: the classifier becomes a five-model ensemble, the ML stage applies class_weight='balanced' where available (Section III.F), and SMOTE is applied (Section III.E). No ablation isolates SMOTE. Therefore the recall gain from 80.06% to 85.50% could equally be caused by ensembling or class reweighting rather than by SMOTE. This confound is load-bearing because the abstract states that the SMOTE integration improves accuracy and recall. A second, compounding issue is that ML features are in-sample CNN probabilities (Sections III.D-E), so synthetic SMOTE points occupy a probability-space geometry that may not match test-time predictions; no out-of-fold scheme is used. The paper's own observation that SMOTE gave no benefit on the balanced BT dataset is consistent with SMOTE being an inert component in this pipeline, reinforcing the need to test it separately. The finding is plausible but not yet causally attributed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates five ImageNet-pretrained CNN models (MobileNet, Xception, InceptionV3, ResNet50, DenseNet201) on two medical imaging datasets: a brain tumor MRI dataset and the APTOS 2019 diabetic retinopathy dataset. For the brain tumor dataset, transfer learning alone achieves near-perfect results. For the imbalanced diabetic retinopathy dataset, the authors propose a hybrid pipeline in which CNN class probabilities are used as features, SMOTE is applied to balance the training set, and traditional machine learning classifiers are trained on the resampled features. They report that a Voting Classifier trained on SMOTE-resampled Xception probabilities improves accuracy from 90.17% to 92.14%, recall from 80.06% to 85.50%, and specificity from 97.46% to 98.18%, compared with Xception transfer learning alone. The paper concludes that combining transfer learning with SMOTE and traditional classifiers is a computationally light way to mitigate class imbalance in medical image classification.","tokens_in":8570,"tokens_out":2897,"duration_ms":32228,"significance":"If the attribution to SMOTE were established, the paper would offer a practically useful and computationally inexpensive post-processing method for improving sensitivity on imbalanced medical imaging datasets. The study has clear strengths: it uses two public datasets, compares several CNN architectures, and reports a consistent set of classification metrics. The internal consistency of Tables III–VI is good, and the observation that SMOTE provides no benefit on the already balanced brain tumor dataset is a useful negative control. However, the central claim is not yet causally supported because the improvement is measured between a single CNN baseline and a pipeline that changes multiple components at once, with no ablation, no repeated runs, and no statistical uncertainty assessment. The core result is plausible but currently under-identified.","major_comments":[{"comment":"The reported 1.97% accuracy gain and 5.44% recall gain are attributed to SMOTE, but the comparison changes three variables simultaneously: the single Xception classifier is replaced by a five-model Voting Classifier, class weights are set to 'balanced' where available (Section III.F), and SMOTE is applied (Section III.E). A 2x2 ablation is needed—SMOTE on/off crossed with class weighting on/off, and ideally with a single logistic regression as well as the ensemble—to determine whether the recall improvement comes from SMOTE, from the balanced class weights, or from ensemble averaging. Without this ablation, the abstract's causal statement that 'we integrate SMOTE... which improves accuracy by 1.97%' is not justified by the experimental design.","section":"IV.B, Tables V and VI; Abstract"},{"comment":"The machine-learning stage is trained on in-sample CNN probabilities: Section III.D states that 'predicted class probabilities for training and test datasets were retained,' and Section III.E says these training probabilities form the new dataset. No out-of-fold or cross-validated probability generation is described. Because SMOTE interpolates between training points in probability space, synthetic samples may lie in regions that do not correspond to test-time CNN outputs, especially given that the same training set was used to fit the CNN. The authors should generate the feature set using out-of-fold predictions (or at least evaluate with an inner cross-validation loop) to avoid optimistic bias in the SMOTE + ML stage.","section":"III.D–III.E"},{"comment":"All reported numbers come from a single training run, with no error bars, confidence intervals, or significance tests. The improvement from 90.17% to 92.14% may be within stochastic variation, particularly because training uses stochastic augmentation and early stopping. In addition, Section IV says 'We'll only use the top 5 of the final results to evaluate,' which implies that the best of many ML models was selected on the test set; this selection effect is not accounted for in the reported gains. The authors should report repeated runs or use a nested validation scheme so that the headline improvement is not an artifact of model selection or random seed.","section":"IV (overall evaluation protocol)"}],"minor_comments":[{"comment":"The paper uses 'multilabel image classification' but the datasets are multiclass, single-label problems; this terminology should be corrected.","section":"Abstract and throughout"},{"comment":"The text says the ImageDataGenerator class comes from scikit-learn, but ImageDataGenerator is provided by Keras/TensorFlow, not scikit-learn.","section":"III.B"},{"comment":"The narrative switches to first person ('I further divided the remaining 20%') in a multi-author paper; this should be 'we' throughout.","section":"III.A"},{"comment":"The sentence 'the moderate class was predominantly misclassified, particularly as Proliferate and Severe classes' appears to contradict Figure 5, which likely shows misclassifications into adjacent stages. Please verify the confusion matrix interpretation and clarify that misclassification into more severe stages is not necessarily clinically 'less critical.'","section":"V, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical comparison, and the reported effect is plausible, but the central attribution to SMOTE is not isolated. The requested ablation and out-of-fold features are within the scope of the current experimental setup and should be doable. The manuscript would also benefit from clearer reporting of the number of runs and the model-selection protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a straightforward empirical comparison: five ImageNet CNNs fine-tuned on two public medical datasets, then class probabilities fed to classical ML classifiers with SMOTE. The one concrete new number is on APTOS: the Voting Classifier on SMOTE-resampled Xception probabilities gets 92.14% accuracy and 85.50% recall versus 90.17% and 80.06% for Xception fine-tuning alone. That is a plausible and potentially useful gain for screening, and it costs little compute.\n\nCredit where due: the experiments are reproducible in shape (public datasets, standard models), the metric tables are internally consistent, and the authors explicitly note that SMOTE did nothing on the balanced brain-tumor dataset, which is the kind of negative result that lends the work some credibility.\n\nHowever, the central attribution is not isolated. The hybrid pipeline changes three things at once: the classifier becomes a five-model voting ensemble, 'balanced' class weights are applied where available, and SMOTE is applied. The abstract says SMOTE improves recall by 5.43%, but the comparison in Tables V and VI is Xception-only versus Voting+SMOTE. An ablation (Xception probabilities + Voting without SMOTE, Xception probabilities + SMOTE + single ML, etc.) is needed before crediting SMOTE. Given that SMOTE gave nothing on the BT dataset, it may well be the ensemble or class weighting doing the work.\n\nThere is also a leakage-adjacent design issue: the ML stage trains on in-sample CNN probabilities with no out-of-fold scheme. SMOTE interpolates in a probability simplex geometry that may not reflect test-time predictions. No error bars or significance tests either, so the 5-point recall gain could be within run-to-run noise.\n\nAll that said, this is a working-paper-level contribution that a serious referee could improve with a 2x2 ablation and a nested cross-validation. It is not a conceptual breakthrough, but it is an honest measurement worth reporting. I would send it to peer review expecting revision. I would not cite it for the SMOTE claim until the ablation is done.","headline":"A useful but confounded empirical result: the reported SMOTE gains on APTOS need an ablation before causal credit.","tokens_in":9148,"tokens_out":1690,"would_cite":false,"duration_ms":15879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SMOTE-plus-transfer-learning lifts diabetic retinopathy recall by 5.4 points.","keywords":["transfer learning","SMOTE","diabetic retinopathy","brain tumor MRI","class imbalance","ensemble learning","medical image classification","Xception"],"falsifier":"Holding Xception probabilities and the Voting Classifier fixed, retrain once with SMOTE and once without it, keeping balanced class weights in both runs; if recall stays near 85% without SMOTE, the reported gain is not attributable to SMOTE.","tokens_in":8148,"feed_emoji":"🩺","tokens_out":4656,"duration_ms":43531,"temperature":0.7,"pith_summary":"The paper tests whether transfer-learned CNN features, rebalanced with SMOTE and fed to classical machine-learning classifiers, can fix the sensitivity shortfall of transfer learning alone on imbalanced medical images. On the APTOS 2019 diabetic retinopathy dataset it reports that the best hybrid, a Voting Classifier trained on SMOTE-resampled Xception probabilities, reaches 92.14% accuracy and 85.50% recall versus 90.17% and 80.06% for Xception alone. On the balanced brain tumor MRI dataset, by contrast, transfer learning is already near-perfect and SMOTE adds nothing. The practical stake is a computationally light way to improve detection of underrepresented disease classes without retraining large networks.","feed_headline":"SMOTE boosts DR recall by 5.4 points over transfer learning","feed_subtitle":"Voting classifier on resampled Xception probabilities lifts sensitivity to 85.5% on APTOS 2019.","key_machinery":"The carried mechanism is a three-stage pipeline: a frozen ImageNet pre-trained CNN (Xception for the diabetic retinopathy set) outputs class probabilities for each training image; SMOTE synthesizes new probability-vector samples for minority classes by interpolating between existing ones; then classical classifiers, configured with balanced class weights, are trained on the resampled probability vectors. This converts an imbalanced image-classification problem into a balanced tabular classification problem, avoiding the cost of oversampling raw images.","core_discovery":"On imbalanced medical image data, replacing the transfer-learned CNN's final classifier with a classical machine-learning pipeline trained on SMOTE-resampled CNN probabilities improves sensitivity more than accuracy. The paper's central result is on the five-class APTOS 2019 diabetic retinopathy set: Xception transfer learning alone gives 90.17% accuracy and 80.06% recall, while the Voting Classifier (Logistic Regression, Linear SVM, SGD, Extra Trees, Random Forest) on SMOTE-balanced Xception probabilities gives 92.14% accuracy and 85.50% recall, with specificity rising from 97.46% to 98.18%. The same pipeline leaves the balanced brain tumor benchmark essentially unchanged, which the paper takes as evidence that resampling matters only when class imbalance is the bottleneck.","pith_inferences":["The paper's attribution of the gain to SMOTE is underdetermined: the final comparison changes three variables at once (the classifier type, the balanced class-weight setting, and SMOTE), so an ablation holding the classifier and class weights fixed would be needed to isolate SMOTE's true contribution.","SMOTE here is applied to CNN probability vectors, not to raw images; a testable extension is whether oversampling raw image embeddings rather than probabilities would yield larger gains or better calibration.","The claim that no test image was falsely labeled 'No DR' when it actually had disease is class-specific; a per-class confusion matrix would show whether the sensitivity gain is concentrated in particular severity stages.","The same probability-plus-SMOTE recipe could be tried on other imbalanced medical datasets (e.g., rare cancers or lesion types) where a pre-trained CNN is already available but its raw sensitivity is too low for screening use."],"forward_implications":["If this holds, transfer learning alone is sufficient on near-balanced medical datasets, and SMOTE-based rebalancing is only needed when class imbalance dominates.","The reported 5.44-point recall gain on the APTOS dataset means more true cases of diabetic retinopathy would be caught than with the transfer-learning baseline, at minimal added compute.","The hybrid approach can be applied to any pre-trained CNN by saving its training-set probabilities, so it does not require re-training the deep network.","The small accuracy gain (1.97 points) alongside the larger recall gain suggests the method trades little overall correctness for much better sensitivity.","Because the BT results show no degradation after SMOTE, the method appears safe on balanced data as well, though it offers no gain there."],"supporting_citations":[{"why":"Supplies the APTOS 2019 dataset and the ensemble-learning baseline (91% accuracy) that the paper's hybrid pipeline is designed to beat.","marker":"[2]"},{"why":"Provides the prior SVM-plus-MobileNetV2 result on a balanced two-class DR task that motivates the imbalanced multi-class comparison.","marker":"[15]"},{"why":"Offers a CNN-DELM result on the MESSIDOR DR dataset that establishes the state of the art the paper contrasts with.","marker":"[13]"},{"why":"Demonstrates VGGNet transfer learning for DR on EYEPACS, supporting the transfer-learning premise.","marker":"[14]"},{"why":"Uses pre-trained CNN features with machine-learning classifiers for brain tumor MRI, a template for the probability-as-feature pipeline.","marker":"[11]"},{"why":"Shows feature fusion and ensemble learning in a CNN for mammography, supporting the ensemble component.","marker":"[5]"}],"fun_headline_variants":["SMOTE lifts diabetic retinopathy recall by 5.4 points","Resample CNN probabilities to boost diabetic retinopathy recall by 5.4%","Voting classifier on SMOTE-resampled features boosts DR sensitivity","SMOTE + voting classifier bests transfer learning recall by 5.4 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument that SMOTE produces the improvement assumes nothing else in the final pipeline changed, but the comparison swaps the classifier for an ensemble and turns on balanced class weights at the same time.","fun_headline_variants_meta":{"raw":{"variants":["SMOTE lifts diabetic retinopathy recall by 5.4 points","Resample CNN probabilities to boost diabetic retinopathy recall by 5.4%","Voting classifier on SMOTE-resampled features boosts DR sensitivity","SMOTE + voting classifier bests transfer learning recall by 5.4 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2955,"prompt_tokens":917,"completion_tokens":2038,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1957}},"tokens_in":533,"tokens_out":2038,"duration_ms":15602,"temperature":1.0,"reasoning_tokens":1957,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:24:42.516002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Holding Xception probabilities and the Voting Classifier fixed, retrain once with SMOTE and once without it, keeping balanced class weights in both runs; if recall stays near 85% without SMOTE, the reported gain is not attributable to SMOTE.","supporting_citations":[{"cited_title":"Sikder, M","cited_arxiv_id":null,"evidence_quote":"Supplies the APTOS 2019 dataset and the ensemble-learning baseline (91% accuracy) that the paper's hybrid pipeline is designed to beat."},{"cited_title":"Classifying Diabetic Retinopathy using CNN and Machine Learning,","cited_arxiv_id":null,"evidence_quote":"Provides the prior SVM-plus-MobileNetV2 result on a balanced two-class DR task that motivates the imbalanced multi-class comparison."},{"cited_title":"Image Fundus Classification System for Diabetic Retinopathy Stage Detection Using Hybrid CNN-DELM,","cited_arxiv_id":null,"evidence_quote":"Offers a CNN-DELM result on the MESSIDOR DR dataset that establishes the state of the art the paper contrasts with."},{"cited_title":"Trans- fer Learning-Based Model for Diabetic Retinopathy Diagnosis Using Retinal Images,","cited_arxiv_id":null,"evidence_quote":"Demonstrates VGGNet transfer learning for DR on EYEPACS, supporting the transfer-learning premise."},{"cited_title":"MRI-Based Brain Tumor Classification Using Ensemble of Deep Features and Machine Learning Classifiers,","cited_arxiv_id":null,"evidence_quote":"Uses pre-trained CNN features with machine-learning classifiers for brain tumor MRI, a template for the probability-as-feature pipeline."},{"cited_title":"Feature fusion and Ensemble learning-based CNN model for mammographic image classification,","cited_arxiv_id":null,"evidence_quote":"Shows feature fusion and ensemble learning in a CNN for mammography, supporting the ensemble component."}],"review_version":1}