{"id":"55577e3a-a4f9-4df1-845c-56fb80afac80","arxiv_id":"2504.17540","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An Xception-plus-NGBoost pipeline tuned by African Vultures Optimization reaches 97.53% accuracy and 97.72% F1 on the Monkeypox Skin Lesion Dataset, but the evaluation uses augmented images and may leak patient data across folds.","lead":"This paper tests a machine-learning setup that sorts skin-lesion photos into monkeypox or not, using a pretrained image network plus a tuned boosting classifier. The authors report 97.5% accuracy and argue the tool could help diagnose monkeypox where lab tests are scarce.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"5-fold CV is run on augmented images without patient-level grouping, so augmented copies of the same original lesion likely appear in both train and test folds; this inflates the headline 97.53% accuracy and the SOTA claim is unsupported until a grouped-CV check is done.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I find. The paper's strongest claim is the state-of-the-art accuracy, and that number is produced by 5-fold CV over augmented images without patient-level grouping. Because augmentation multiplies each original image fourteen-fold, random image-level folds inevitably leak augmented copies of the same lesion into both training and test sets. This is a known and severe evaluation flaw for medical image classifiers, and the paper provides no evidence that the splits were grouped by original image or patient. The internal numbers in Table 11 are consistent with random image-level splitting: 429 test samples per fold equals 20% of the 2,142 augmented Train images. No external validation, no nested model selection, and no code are provided, so there is no independent check on the inflated estimate. The concern is not merely that the method might be weaker; it is that the reported 97.53% accuracy does not estimate the quantity the paper claims, namely generalization to new patients. I agree with the reader's REJECT verdict, and a grouped-CV rerun is the single test that would settle the issue.","tokens_in":26354,"tokens_out":3347,"duration_ms":35669,"concrete_test":"Reproduce the proposed pipeline (Xception frozen features, PCA to 530 components, NGBoost with SVR base learner, learning rate 0.1092, n_estimators 5) on the MSLD augmented Train folder, but replace the random 5-fold split with grouped 5-fold CV in which all augmented images sharing the same original image are assigned to the same fold, and ideally group by patient ID using the 162-patient metadata. Report mean and standard deviation of accuracy, F1, and AUC. Additionally, train and test on the original 228 un-augmented images with patient-level grouped CV as a lower-bound sanity check. If grouped CV accuracy falls materially below 97.53%, for example below 92% or overlapping the 90% range reported by prior work, the state-of-the-art claim is unsupported.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—AVOA-NGBoost achieves state-of-the-art 97.53% accuracy, 97.72% F1, and 97.47% AUC—rests entirely on the evaluation in Section 6. Section 3.2 states that fourteen-fold augmentation was applied to the MSLD, yielding 1,428 Monkeypox and 1,764 Others images from only 228 originals (162 unique patients), and that the augmented Train folder containing 2,142 images was used. Section 6 then says 5-fold cross-validation randomly divided the dataset into folds. There is no statement that augmented images derived from the same original image or same patient are kept together. With image-level random splitting, a typical original image has augmented siblings scattered across training and test folds; the test fold then contains near-duplicates of training images under rotation, shear, hue, noise, and other transformations. Since the Xception features are frozen ImageNet features and PCA/NGBoost operate on those features, the classifier can exploit idiosyncratic pixel-level similarities shared by augmented siblings rather than learning to recognize new patients' lesions. Table 11 confirms this protocol: each fold has 429 test samples, exactly 20% of the 2,142 augmented Train images. The conclusion in Section 7 that the model provides a highly precise diagnostic tool for new patients is therefore not supported. The missing patient-level grouping is the load-bearing weakness: if it holds, the headline numbers are inflated and the SOTA claim collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an explainable framework for monkeypox diagnosis from skin lesion images, combining Xception deep features, PCA dimensionality reduction, and an NGBoost classifier whose hyperparameters are tuned with the African Vultures Optimization Algorithm (AVOA). The authors report state-of-the-art performance on the Monkeypox Skin Lesion Dataset (MSLD), with 97.53% accuracy, 97.72% F1-score, and 97.47% AUC under 5-fold cross-validation, and they provide Grad-CAM and LIME visualizations for interpretability. The paper includes extensive comparisons across thirteen feature extractors and four classifiers, as well as comparisons of AVOA with other metaheuristic optimizers.","tokens_in":26679,"tokens_out":5293,"duration_ms":48750,"significance":"If the reported results were valid, the paper would make a useful practical contribution: it offers a complete, coherent pipeline for automated triage of monkeypox versus other rash illnesses, uses a publicly available dataset, and incorporates probabilistic classification and explainability, which are desirable for clinical decision support. The experimental breadth is substantial, with 52 feature-extractor/classifier combinations and multiple optimization baselines. However, the central claim of state-of-the-art performance is not supported by the experiments as reported, because the evaluation protocol is compromised by likely patient leakage and by model selection on the same data used for final metrics. These are not cosmetic issues; they directly affect the headline accuracy, F1-score, and AUC, and they undermine the generalization claims in the abstract and conclusion.","major_comments":[{"comment":"The 5-fold cross-validation in Section 6 randomly splits the augmented Train folder into image-level folds, but Section 3.2 and Table 1 show that the MSLD consists of only 162 unique patients and was augmented fourteen-fold, so each original lesion has many augmented siblings. With random image-level splits, augmented copies of the same original image can appear in both training and test folds. Because the Xception features are frozen ImageNet features and PCA/NGBoost operate directly on those features, the classifier can exploit pixel-level near-duplicates shared by augmented siblings rather than learning to recognize new patients' lesions. As a result, the mean accuracy of 97.53% and F1-score of 97.72% in Table 11, and the corresponding state-of-the-art claim in the abstract, are inflated and do not measure generalization to new patients. The authors need to repeat the evaluation with patient-level (or at least original-image-level) grouped cross-validation and, ideally, an external validation set.","section":"Section 3.2 and Section 6"},{"comment":"The PCA variance ratio is selected as the value yielding the highest classification accuracy on the augmented dataset (Figure 6 and Table 3), and AVOA is used on the same data to select NGBoost hyperparameters (Table 5). The final metrics in Table 10 are then reported from 5-fold cross-validation on that same dataset. This is model selection on the evaluation data: the reported accuracy, F1-score, and AUC are optimistically biased because the same data have been used both to choose the configuration and to estimate performance. A nested cross-validation, or a held-out test set that is completely untouched during PCA variance selection and hyperparameter optimization, is required to obtain unbiased performance estimates.","section":"Section 6 (Figure 6, Tables 5 and 10)"},{"comment":"The conclusion claims that the low standard deviation across folds in Table 11 demonstrates generalizability and supports the model as a 'highly precise diagnostic tool' for new patients. This claim is not supported because the cross-validation folds are drawn from the same patient population and contain overlapping augmented images, as described in the first major comment. The stability of results across folds within this leakage-prone setup says nothing about performance on new patients, who would present lesions not derived from the same original images. The conclusion overreaches the evidence.","section":"Section 7 and Table 11"}],"minor_comments":[{"comment":"The F1-score formula is misprinted: it is written as TP / (TP + 0.5(FP + FN)), which is not the standard F1-score; the correct form is 2TP / (2TP + FP + FN). The reported numbers appear to have been computed correctly, but the formula should be fixed.","section":"Section 5, Eq. (27)"},{"comment":"The split sizes are inconsistent: the text says the dataset was split in a 70:10:20 proportion, and Table 1 lists 3,192 augmented images, yet the augmented Train folder is stated to contain 2,142 images, which is 67.1% of 3,192 rather than 70%. The relationship among the original images, the 3,192 augmented images, and the 2,142-image Train folder should be clarified.","section":"Section 3.2 and Table 1"},{"comment":"The sentence 'The core concept of SVM is to identify an optimal decision boundary...' is duplicated verbatim in the SVM description.","section":"Section 4.2.2.2"},{"comment":"In the summary of Saha et al., the architecture name is misspelled as 'Densnet'; it should be 'DenseNet'.","section":"Section 2"},{"comment":"The notation 'Γ(1 + β2)' in the Lévy flight expression is ambiguous; it should be written as Γ((1 + β)/2), which is the standard form.","section":"Section 4.2.3.1, Eq. (22)"},{"comment":"The caption refers to the class 'Normal', but the dataset labels are 'Monkeypox' and 'Others'; this inconsistency should be corrected.","section":"Figure 11 caption"},{"comment":"The t-test applied to t-SNE coordinates is statistically questionable because t-SNE embeddings are stochastic and the coordinates are not independent samples; the interpretation of these t-values should be justified or removed.","section":"Section 6, Figure 7"}],"recommendation":"reject","confidential_remarks":"The paper is a preprint submitted to Elsevier and is not accompanied by a code-availability statement, which limits reproducibility. The leakage issue is likely to be fatal: even if the authors re-run the experiments with patient-grouped cross-validation, the reported headline numbers will almost certainly drop, and the claimed state-of-the-art status would need to be reassessed. In my view, the current evaluation does not support the paper's central claim, and a revision would require substantially new experiments rather than local fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper is a standard transfer-learning plus boosting pipeline applied to monkeypox images, and it has one genuinely useful piece: a broad comparison of 13 feature extractors and four classifiers, with careful reporting of metrics and some interpretability analysis. Second, the headline claim of 97.53% accuracy is not supported by the experiments as described, because the 5-fold CV is run on the augmented image set without patient-level grouping. Augmented siblings of the same original lesion almost certainly land in both train and test folds, so the test fold is full of near-duplicates of training images. That inflates the accuracy and makes the SOTA claim fragile.\n\nGive credit where due. The authors do a systematic comparison of 52 combinations (13 CNN feature extractors x four classifiers), which is more thorough than many papers in this space. They also report per-fold results, kappa, AUC, training times, and t-SNE plots, and they make a reasonable effort to show why PCA helps. The AVOA vs. other optimizers comparison is a fair empirical test, not circular. No invented entities, no self-citation dependence.\n\nThe soft spots are real and load-bearing. Section 3.2 describes fourteen-fold augmentation producing 2,142 augmented training images from 228 originals (162 patients). Section 6 says 5-fold CV randomly divided the dataset, with 429 test samples per fold, exactly 20% of 2,142. There is no grouping by original image or patient. With rotation, shear, hue, and noise augmentations, siblings are near-duplicates; Xception features from frozen ImageNet weights will make the classifier exploit those idiosyncrasies. That alone is enough to explain the gap between baseline NGBoost on PCA features (95.57%) and the AVOA-tuned version (97.53%).\n\nSecond, the model selection is not nested. They use the same 5-fold CV to choose the feature extractor, the PCA variance ratio, and the AVOA hyperparameters, then report the final metrics from those same folds. That is selection on the test data, and it further inflates the numbers. Third, there is no external validation and no code or data shipped, so the claimed superiority over prior work cannot be checked.\n\nNone of this makes the authors sloppy in an obvious way; the pipeline is coherent and the writing is clear. It just means the central empirical claim does not hold as reported. A serious referee should ask for patient-level grouped CV and a nested model-selection protocol. If the accuracy survives that, the paper is a useful incremental addition. If not, at least the comparison table is worth having.\n\nMy recommendation: send it to peer review, not desk reject, because the paper is substantial and the flaw is fixable with a re-analysis. The version you'd publish would likely be a more modest one, but the comparison data is usable. I would not cite the headline numbers.","headline":"Competent incremental pipeline undone by likely augmentation leakage in the cross-validation, so the SOTA claim doesn't hold as reported.","tokens_in":27192,"tokens_out":2265,"would_cite":false,"duration_ms":21770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T10","92C55"],"pacs":[],"model":"deepseek-v4-flash","headline":"The monkeypox model hits 97.5% accuracy with optimized deep features.","keywords":["monkeypox diagnosis","skin lesion classification","transfer learning","Xception","NGBoost","African Vultures Optimization Algorithm","PCA dimensionality reduction","explainable artificial intelligence"],"falsifier":"Re-run the AVOA-NGBoost pipeline with 5-fold cross-validation stratified by the 162 unique patients so all augmented copies of one patient stay in the same fold; if mean accuracy drops materially below 97.53%, the claim that the model generalizes to new patients is not supported.","tokens_in":26173,"feed_emoji":"🦠","tokens_out":6228,"duration_ms":57172,"temperature":0.7,"pith_summary":"The paper argues that a low-cost image-classification pipeline can diagnose monkeypox from skin-lesion photos accurately enough to serve as a triage tool. The proposed framework takes frozen Xception deep features, compresses them with PCA, and classifies them with NGBoost whose hyperparameters are tuned by the African Vultures Optimization Algorithm. On the Monkeypox Skin Lesion Dataset, the paper reports 97.53% accuracy, 97.72% F1-score, and 97.47% AUC, beating the thirteen CNN feature extractors and four classifiers it compares against. If those numbers hold for new patients, the framework would give a fast, explainable alternative or complement to PCR testing in resource-constrained settings.","feed_headline":"Monkeypox model hits 97.5% accuracy with optimized deep features","feed_subtitle":"Frozen Xception features, PCA, and vulture-tuned NGBoost top earlier classifiers on lesion photos.","key_machinery":"The load-bearing mechanism is a four-stage pipeline: Xception, a deep CNN built on depthwise separable convolutions, is frozen and used to extract 2048 features per image; PCA cuts those features to 530; NGBoost, a boosting algorithm that fits a probabilistic output by natural gradients, classifies the reduced features; and AVOA, a swarm metaheuristic modeled on vulture foraging, searches the learning-rate and estimator-count space. What carries the argument is the coupling: PCA removes redundancy so the tuned NGBoost sees a low-dimensional, well-separated representation, and the optimizer finds hyperparameters that the paper shows beat other metaheuristics on the same folds.","core_discovery":"On the paper's own terms, the discovery is that a transfer-learned Xception network, reduced features, and a probabilistically trained NGBoost classifier with AVOA-tuned hyperparameters outperform both the plain deep feature extractors and several other boosting and SVM classifiers on the Monkeypox Skin Lesion Dataset. The authors report that the proposed AVOA-NGBoost reaches 97.53% accuracy, 97.72% F1, and 97.47% AUC in 5-fold cross-validation, with per-class accuracy of 98.10% for monkeypox and 96.84% for non-monkeypox. They also show that PCA cuts training time by roughly four-fifths and that AVOA beats five other metaheuristics in convergence and final fitness on the same folds.","pith_inferences":["A fair test of the headline number requires patient-level cross-validation; the paper does not report this, so I would not treat 97.53% as a generalization estimate for new patients.","The paper compares AVOA only with other metaheuristics. Comparing it with cheap baselines at equal budget, such as random search or coarse grid search, would isolate how much the optimizer itself contributes.","Because NGBoost outputs a probabilistic prediction, the framework could support an uncertainty-aware referral rule that flags low-confidence cases for PCR or clinician review; the paper does not evaluate calibration, so this is a natural next step rather than a demonstrated property.","The frozen Xception features are trained on ImageNet; fine-tuning the backbone or using a medical-imaging pretrained backbone is a testable variant that could change the accuracy-versus-speed tradeoff."],"forward_implications":["If the reported figures hold, a frozen pretrained CNN plus PCA plus an optimized gradient-boosting classifier is enough to separate monkeypox from chickenpox and measles at clinical-grade accuracy on this dataset.","PCA removed roughly three-quarters of the features and cut training time from 85.97 seconds to 22.58 seconds for NGBoost, supporting use in settings where computational resources are limited.","AVOA tuning improves all four classifiers, and the largest gain appears for NGBoost, so hyperparameter search is a necessary part of the claimed result rather than a cosmetic addition.","Grad-CAM and LIME highlight lesion regions in the explanations, giving clinicians a visual reason to trust or question individual predictions."],"supporting_citations":[{"why":"Supplies the Monkeypox Skin Lesion Dataset on which all experiments and reported metrics are based.","marker":"Ali et al., 2022"},{"why":"Defines the Xception architecture used as the frozen deep feature extractor.","marker":"Chollet, 2016"},{"why":"Introduces NGBoost, the probabilistic gradient-boosting classifier whose hyperparameters are tuned.","marker":"Duan et al., 2020"},{"why":"Introduces the African Vultures Optimization Algorithm used for hyperparameter search.","marker":"Abdollahzadeh et al., 2021"},{"why":"Provides the prior ensemble benchmark that the paper compares against as the strongest existing result on the dataset.","marker":"Pramanik et al., 2023"},{"why":"Provides additional MSLD baselines with InceptionV3 and MobileNet that the proposed method is compared with.","marker":"Almufareh et al., 2023"},{"why":"Reports comparative CNN results across monkeypox datasets, used as context for the accuracy levels claimed.","marker":"Saha et al., 2023"},{"why":"Offers an earlier deep CNN monkeypox classifier and is cited in the data preprocessing discussion.","marker":"Bala et al., 2023"}],"fun_headline_variants":["Vulture-optimized AI detects monkeypox with 97.5% accuracy","Monkeypox AI hits 97.5% with Xception features and vulture tuning","Explainable monkeypox model reaches 97.5% accuracy via NGBoost","Vulture-tuned NGBoost spots monkeypox at 97.5% precision","Deep features + AVOA optimization yield 97.5% monkeypox detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that random 5-fold splitting of the fourteen-fold augmented image set estimates accuracy on new patients; because augmented versions of the same original lesion can appear in both training and test folds, the folds may leak patient identity and inflate the reported figures.","fun_headline_variants_meta":{"raw":{"variants":["Vulture-optimized AI detects monkeypox with 97.5% accuracy","Monkeypox AI hits 97.5% with Xception features and vulture tuning","Explainable monkeypox model reaches 97.5% accuracy via NGBoost","Vulture-tuned NGBoost spots monkeypox at 97.5% precision","Deep features + AVOA optimization yield 97.5% monkeypox detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2393,"prompt_tokens":988,"completion_tokens":1405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1295}},"tokens_in":604,"tokens_out":1405,"duration_ms":12112,"temperature":1.0,"reasoning_tokens":1295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:37:29.981971+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the AVOA-NGBoost pipeline with 5-fold cross-validation stratified by the 162 unique patients so all augmented copies of one patient stay in the same fold; if mean accuracy drops materially below 97.53%, the claim that the model generalizes to new patients is not supported.","supporting_citations":[{"cited_title":"Ngboost: natural gradient boosting for probabilistic prediction","cited_arxiv_id":null,"evidence_quote":"Introduces NGBoost, the probabilistic gradient-boosting classifier whose hyperparameters are tuned."},{"cited_title":"Monkeypox detection from skin lesion images using an amalgamation of cnn models aided with beta function-based normalization scheme","cited_arxiv_id":null,"evidence_quote":"Provides the prior ensemble benchmark that the paper compares against as the strongest existing result on the dataset."},{"cited_title":"A transfer learning approach for clinical detection support of monkeypox skin lesions","cited_arxiv_id":null,"evidence_quote":"Provides additional MSLD baselines with InceptionV3 and MobileNet that the proposed method is compared with."},{"cited_title":"A Comparative Analysis of CNN-Based Pretrained Models for the Detection and Prediction of Monkeypox","cited_arxiv_id":"2302.10277","evidence_quote":"Reports comparative CNN results across monkeypox datasets, used as context for the accuracy levels claimed."}],"review_version":1}