{"id":"1c95ea96-9d2f-454e-9f34-b18fc9ff2268","arxiv_id":"2507.15915","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuned InceptionV3 reaches 95% accuracy on a binary monkeypox skin image dataset and MobileNetV2 reaches 93% on a six-class dataset, but no code or error bars are provided and the reported micro-average AUC values contradict the accuracies.","lead":"This paper fine-tunes four pretrained CNN models to classify monkeypox from skin lesion images and uses Grad-CAM to show which image regions drive predictions. It reports 95% accuracy for InceptionV3 on a binary dataset and 93% for MobileNetV2 on a six-class dataset, but the results lack error bars, code, and contain metric inconsistencies.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Augmentation-before-split data leakage is the load-bearing flaw: the reported 95%/93% test accuracies likely reflect memorization of augmented variants of the same source images rather than generalization to unseen patients.","rationale":"The reader identified the augmentation-before-split leakage as the weakest assumption, and the manuscript text supports that concern directly: Figure 3 puts augmentation before the train/validation/test split, Section 3.2 states that only the already-augmented Kaggle versions were used, and no code or duplicate-removal step is reported. This is exactly the condition that must hold for the headline accuracies to measure generalization. The concern is load-bearing because the paper's main contribution is empirical accuracy on these two datasets; if the test partitions contain augmented variants of training images, the 95% and 93% numbers do not support the claimed clinical applicability. I also note the internally inconsistent micro-average AUC values as a secondary indicator that the evaluation pipeline needs correction, but the leakage issue alone is sufficient to reject the paper in its current form. The appropriate action is to retain the reader's REJECT verdict and require a corrected, leakage-free evaluation as a condition for resubmission.","tokens_in":21400,"tokens_out":4638,"duration_ms":53502,"concrete_test":"Download the same Kaggle MSLD and MSLD v2.0 augmented datasets, compute perceptual hashes (e.g., pHash) of all images, and check whether any of the 320 binary test images or 754 multi-class test images is a near-duplicate of a training image (same source original). Then retrain InceptionV3 on the binary dataset and MobileNetV2 on the multi-class dataset using a split performed on original-image identity before augmentation, with the same hyperparameters; if the test accuracy drops materially below 0.95 and 0.93, the reported figures are inflated by leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central accuracy claims stand only if the test sets contain genuinely unseen images. The methodology described in Figure 3 states that after preprocessing, 'data augmentation techniques were applied to increase the number of images and enhance diversity. The datasets were then split into training, validation, and test sets.' Section 3.2 confirms that only the pre-augmented Kaggle versions are used, and for MSLD v2.0 the augmented images are already partitioned into folds. Splitting after augmentation means augmented copies of the same original lesion image can appear in both the training and test partitions, so the classifier can exploit near-duplicate pixel patterns rather than learning to recognize new patients. The paper never establishes that the split was performed on original-image identity before augmentation, and no code is provided to rule out leakage. If the reported 0.95 and 0.93 test accuracies are inflated by such leakage, the central claim that fine-tuned pretrained CNNs generalize to unseen skin-lesion images is unsupported. A corroborating sign that the metrics are unreliable is the reported micro-average AUC of 0.33-0.39 for binary models whose per-class AUC is 0.95-0.99, which is mathematically inconsistent with a working binary classifier and suggests the evaluation pipeline has additional errors. The load-bearing issue, however, is the uncontrolled overlap between training and test data created by augmenting before splitting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical benchmark of four pretrained CNN architectures (VGG16, VGG19, InceptionV3, MobileNetV2) fine-tuned with a small custom head for early detection of mpox from skin lesion images. Two public Kaggle datasets are used: a two-class MSLD dataset and a six-class MSLD v2.0 dataset. The authors report that InceptionV3 achieves 95% accuracy on the binary dataset and MobileNetV2 achieves 93% accuracy on the multi-class dataset, and they use Grad-CAM to visualize which image regions drive predictions. The paper also reports precision, recall, F1-score, loss, wall time, and CPU time, and compares the results with selected prior work.","tokens_in":21676,"tokens_out":8548,"duration_ms":89780,"significance":"If the reported numbers were reliable, this would be a useful but incremental empirical confirmation that standard transfer learning with frozen pretrained backbones can classify skin-lesion images on these specific public datasets. The comparison across four architectures, the reporting of wall/CPU time, and the use of two datasets are positive features, as is the candid discussion of some limitations in Sections 6 and 7. However, the contribution is primarily an evaluation rather than a methodological advance, and the paper provides no code or evaluation scripts to support reproducibility. The significance is therefore conditional on the trustworthiness of the reported metrics, and the concerns below put that trustworthiness in question.","major_comments":[{"comment":"The reported test accuracies are intended to measure generalization to unseen images, but the pipeline described in Figure 3 and Section 3.3.2 applies data augmentation before the train/validation/test split, and Section 3.2 states that only the pre-augmented Kaggle versions are used. Consequently, augmented copies of the same original lesion image can appear in both the training and test partitions, and the manuscript does not state that the split was performed on original-image identity (nor is code provided to rule this out). This directly threatens the central claims of 0.95 accuracy for InceptionV3 in Table 8 and 0.93 accuracy for MobileNetV2 in Table 9. The authors must re-run the experiments with augmentation contained within the training split, or provide evidence that no source-image overlap exists across splits.","section":"§3.1 (Figure 3), §3.2, §3.3.2"},{"comment":"The micro-average AUC values are mathematically inconsistent with the other reported metrics. In a binary problem the micro-average ROC is identical to the overall ROC, so values of 0.33–0.39 for models with per-class AUCs of 0.95–0.99 cannot arise from a correct ROC computation. In the multi-class setting, micro-average AUCs of 0.37–0.61 alongside accuracies of 0.85–0.93 and per-class AUCs of 0.93–1.00 are likewise implausible. This indicates a systematic error in the ROC/AUC computation and raises doubt about all metrics produced by the same evaluation pipeline.","section":"§4.1, §4.2 (ROC analysis)"},{"comment":"The paper's metric definitions are incorrect: Eq. (1) defines accuracy as (TP+FP)/Total instead of (TP+TN)/Total, and the surrounding text defines FP as \"Actual Positive but predicted negative\" and FN as \"Actual negative but predicted positive\", interchanging the two. Because accuracy, precision, recall, and F1-score are the load-bearing outcomes of the study, the authors need to correct these definitions and confirm that the reported numbers in Tables 8 and 9 were computed with a standard implementation.","section":"§3.3.7 (Eq. 1)"},{"comment":"The authors themselves state in Section 7 that \"the relationship between Grad-CAM outputs and the CNN model's performance was not explicitly established, leaving the relevance of CNN predictions and Grad-CAM visualizations unclear.\" This admission undercuts the interpretability component of the contribution: Section 5 describes the Grad-CAM heatmaps as validating \"the clinical relevance of predictions,\" which is stronger than the evidence supports. The paper should either provide a quantitative or structured evaluation linking Grad-CAM localizations to model decisions, or substantially weaken the claim. If this is not done, the XAI claim should be treated as illustrative rather than validated.","section":"§7 (Future work) and §5 (Discussion)"},{"comment":"All results are reported from a single 75/15/10 split, with no repeated runs, cross-validation, confidence intervals, or significance tests. Given the small number of original images (roughly 400 and 1000 before augmentation, as noted in Section 3.2 and Table 3), the 0.93–0.95 accuracies could vary substantially across splits. The authors should report variability, for example through repeated stratified splits or k-fold cross-validation, before claiming that one architecture demonstrated the best performance.","section":"§3.3.2, Tables 4–5"}],"minor_comments":[{"comment":"The learning rate is stated as 0.001 in the text of Section 3.3.6 but as 0.0001 in Table 6, and the Conclusion repeats 0.0001; these values should be reconciled.","section":"§3.3.6, Table 6, §8"},{"comment":"Table 3 reports roughly 1,200 augmented images for MSLD and more than 3,000 for MSLD v2.0, but Tables 1 and 2 and the text in Section 3.2 use 3,192 and 7,532 images for the augmented versions; the counts should be made consistent.","section":"Table 3 vs Tables 1–2"},{"comment":"In Section 4.1, the CPU time for InceptionV3 is reported as \"3m 60s\" in Table 8, while Table 11 lists 3m 16s for the same model; the entry should be corrected, and \"3m 60s\" should be written as 4m.","section":"Tables 8 and 11"},{"comment":"Several cross-references are inaccurate: Section 5 refers to confusion matrices in \"figure 7 and figure 10,\" and the research-question roadmap in Section 3.1.2 points to sections that do not match the actual numbering; these need to be updated.","section":"§3.1.2, §5"},{"comment":"The manuscript contains many typographical and grammatical errors (for example, \"this models were basically trained,\" \"videnced,\" \"Orthodox virus genus\" for Orthopoxvirus, and \"Figure 5 and 6 displays\") that should be corrected by a thorough language edit.","section":"Throughout"},{"comment":"No code or training/evaluation script is provided, and the only availability statement is a Kaggle link; providing the split and evaluation code would materially improve reproducibility and would help resolve the leakage concern raised in the first major comment.","section":"§9"}],"recommendation":"reject","confidential_remarks":"This is an empirical benchmark with no methodological novelty, so its value depends entirely on the reliability of the reported numbers. The augmentation-before-split protocol is a serious validity threat to the central accuracy claims, and the micro-average AUC values and incorrect metric definitions indicate that the evaluation pipeline contains errors. I would be willing to reconsider a substantially revised version that re-runs the experiments with a contamination-free split, corrects the ROC/AUC computation, and provides code, but as submitted the empirical basis of the paper is not dependable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a routine transfer-learning benchmark with headline numbers I don't trust. The paper's own Figure 3 shows augmentation before splitting, while Section 3.3 describes splitting before augmentation. The binary dataset is the augmented Kaggle version (3,192 images from ~400 originals) and the paper never states that the split was done on original-image identity. If augmentation precedes the split, near-duplicate copies of the same lesion image end up in both training and test, which would inflate the 95% and 93% accuracies. That is the load-bearing issue. The reported micro-average AUCs (0.33–0.61) alongside per-class AUCs (0.93–1.00) are mathematically inconsistent for a binary classifier, so the evaluation pipeline also has an independent error. No code or error bars are provided; a single 75/15/10 split means the point estimates have no variance.\n\nWhat the paper does well: it is clearly written, compares four standard pretrained CNNs on two public datasets, reports wall/CPU time, and honestly discusses overfitting and threats to validity. That is useful as a survey of what these models do on these Kaggle sets, but the actual contribution is thin. Similar benchmarks with Grad-CAM appear in the cited literature (PoxNet22, attention-MobileNetV2, MonkeyNet) with comparable or better accuracy, so the novelty is marginal. The 'lightweight' claim is also shaky: MobileNetV2 has 16M trainable parameters here.\n\nI would not spend referee time on this in its current form. The central accuracy claims rest on a split that is at best ambiguous and at worst leaky, and the AUC numbers show the metrics were computed incorrectly. The right move is to ask the authors to fix the experimental protocol (explicit split-before-augmentation on original images, correct ROC computation, code or detailed logs, confidence intervals) and resubmit. If they can show the numbers hold after a clean split, the benchmark could be a useful data point for the community.","headline":"Routine transfer-learning benchmark whose headline accuracies are unreliable due to ambiguous augmentation/split order and internally inconsistent AUC metrics.","tokens_in":22281,"tokens_out":3881,"would_cite":false,"duration_ms":40560,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InceptionV3 hits 95% on Mpox images; MobileNetV2 93% on six classes","keywords":["Mpox detection","transfer learning","pretrained CNN","InceptionV3","MobileNetV2","Grad-CAM","explainable AI","skin lesion classification"],"falsifier":"Rerun the same four-model pipeline on the two Kaggle datasets with augmentation applied strictly after the train/validation/test split, and report accuracy on the untouched original test images; if the figures fall materially below 95% on binary and 93% on six-class data, the reported numbers were inflated by leakage of augmented copies into training.","tokens_in":21177,"feed_emoji":"🦠","tokens_out":6972,"duration_ms":61031,"temperature":0.7,"pith_summary":"This paper claims that fine-tuning four ImageNet-pretrained convolutional networks—VGG16, VGG19, InceptionV3, and MobileNetV2—produces accurate classifiers for Mpox from skin-lesion photos, with InceptionV3 reaching 95% accuracy on a two-class Kaggle dataset and MobileNetV2 reaching 93% on a six-class dataset. The authors argue this is evidence that lightweight transfer learning can support early detection of a disease that is easy to confuse with chickenpox, measles, and other rashes. They also claim Grad-CAM heatmaps highlight the lesion regions behind each prediction, giving clinicians a visual reason to trust or question the output. Because the study validates on both binary and multi-class data, the authors present the results as a step toward deployable, interpretable screening tools.","feed_headline":"InceptionV3 hits 95% on Mpox images; MobileNetV2 93% on six classes","feed_subtitle":"Fine-tuned pretrained CNNs plus Grad-CAM heatmaps could give doctors a transparent early-warning screen for Mpox.","key_machinery":"The mechanism is transfer learning from ImageNet: every pretrained convolutional base is frozen, and only a custom classification head is trained. The head consists of a flatten layer, a dense layer with 256 ReLU units, a dropout layer at rate 0.5, and a softmax output; early stopping and model checkpointing cap training at 30 epochs. Grad-CAM then computes gradient-weighted activations from the final convolutional layer to produce a heatmap of the image regions that most influenced the prediction. This combination is what the paper claims makes the models lightweight, fast, and interpretable.","core_discovery":"On the paper's own terms, the core discovery is that a frozen pretrained feature extractor plus a small custom head—a flatten layer, a dense layer of 256 ReLU units, a dropout layer at 0.5, and a softmax output—is enough to reach near-clinical accuracy on both datasets. InceptionV3 achieves 0.95 accuracy, precision, recall, and F1 on the binary Monkeypox Skin Lesion Dataset, and MobileNetV2 achieves 0.93 accuracy and precision with 0.90 recall on the six-class MSLD v2.0. The paper further reports that freezing the ImageNet base shrinks trainable parameters substantially, with VGG16 dropping from about 21 million total parameters to about 6.4 million trainable, and that Grad-CAM highlights regions corresponding to lesion areas. The stated conclusion is that these lightweight, interpretable models generalize across two independent datasets and are competitive with prior state-of-the-art results.","pith_inferences":["The paper's own future-work statement admits the link between Grad-CAM heatmaps and model performance was not explicitly established, so the heatmaps should be treated as illustrative rather than validated clinical evidence.","If the augmentation-before-split pipeline inflated the reported accuracies, the true ranking of the four models on unseen patient images is unknown and could differ from InceptionV3/MobileNetV2 leading.","A direct deployment test would classify only the original, non-augmented images, one per patient, to measure the accuracy a clinician would actually experience.","The method's applicability to other rash-like diseases could be tested by fine-tuning the same frozen bases on an independent dermatology dataset with a held-out patient split."],"forward_implications":["If correct, InceptionV3 provides a binary Mpox-versus-others screen at 95% accuracy and MobileNetV2 a six-class differential at 93% accuracy on these Kaggle datasets.","Per-class AUC values up to 0.99 on binary and 1.00 on some multi-class categories indicate these models separate Mpox from common look-alikes on the images tested.","Grad-CAM heatmaps give a clinician a visual region to check, which is the core of the paper's interpretability claim.","Wall times of about 16 minutes for InceptionV3 on binary and 57 minutes for MobileNetV2 on multi-class suggest the fine-tuned heads can be retrained on moderate hardware.","The reduced trainable parameter counts, such as VGG16 shrinking from about 21M to 6.4M parameters, support deployment on resource-limited devices."],"supporting_citations":[{"why":"Supplies the VGG16 and VGG19 base models that are fine-tuned for both tasks.","marker":"[37]"},{"why":"Supplies the MobileNetV2 base model, the best performer on the multi-class dataset.","marker":"[38]"},{"why":"Supplies the InceptionV3 base model, the best performer on the binary dataset.","marker":"[39]"},{"why":"Supplies the Grad-CAM technique used to highlight image regions that drive predictions, supporting the interpretability claim.","marker":"[43]"},{"why":"Provides the six-class Kaggle dataset (MSLD v2.0) used for the multi-class experiments.","marker":"[49]"},{"why":"Provides the binary Kaggle dataset (Monkeypox Skin Lesion Dataset) used for the two-class experiments.","marker":"[50]"},{"why":"Justifies the augmentation-only training approach by showing augmented models resist input distortions, supporting the data expansion step.","marker":"[35]"}],"fun_headline_variants":["Mpox detection: InceptionV3 95%, MobileNetV2 93% with Grad-CAM heatmaps","Frozen CNNs detect Mpox: 95% and 93% with Grad-CAM localization","Pretrained CNNs for Mpox: 95% accuracy, XAI heatmaps pinpoint lesions","Lightweight CNNs hit 95% Mpox accuracy; Grad-CAM explains it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies are only meaningful as generalization estimates if augmented copies of the same original photograph never appear in both the training and test sets, and the paper does not show that the data split was performed before augmentation.","fun_headline_variants_meta":{"raw":{"variants":["Mpox detection: InceptionV3 95%, MobileNetV2 93% with Grad-CAM heatmaps","Frozen CNNs detect Mpox: 95% and 93% with Grad-CAM localization","Pretrained CNNs for Mpox: 95% accuracy, XAI heatmaps pinpoint lesions","Lightweight CNNs hit 95% Mpox accuracy; Grad-CAM explains it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001164,"raw_usage":{"total_tokens":4887,"prompt_tokens":1084,"completion_tokens":3803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":3700}},"tokens_in":700,"tokens_out":3803,"duration_ms":26089,"temperature":1.0,"reasoning_tokens":3700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:23:04.948819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same four-model pipeline on the two Kaggle datasets with augmentation applied strictly after the train/validation/test split, and report accuracy on the untouched original test images; if the figures fall materially below 95% on binary and 93% on six-class data, the reported numbers were inflated by leakage of augmented copies into training.","supporting_citations":[{"cited_title":"R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Ba- tra, D","cited_arxiv_id":null,"evidence_quote":"Supplies the Grad-CAM technique used to highlight image regions that drive predictions, supporting the interpretability claim."},{"cited_title":"A Web-based Mpox Skin Lesion Detection System Using State-of-the-art Deep Learning Models Considering Racial Diversity","cited_arxiv_id":"2306.14169","evidence_quote":"Provides the six-class Kaggle dataset (MSLD v2.0) used for the multi-class experiments."},{"cited_title":"Monkeypox Skin Lesion Detection Using Deep Learning Models: A Feasibility Study","cited_arxiv_id":"2207.03342","evidence_quote":"Provides the binary Kaggle dataset (Monkeypox Skin Lesion Dataset) used for the two-class experiments."},{"cited_title":"Improving deep learning with generic data aug- mentation , in: 2018 IEEE sympo- sium series on computational intelligence (SSCI), IEEE","cited_arxiv_id":null,"evidence_quote":"Justifies the augmentation-only training approach by showing augmented models resist input distortions, supporting the data expansion step."}],"review_version":1}