{"id":"b0418c33-84fd-434b-b44a-801b8c057c85","arxiv_id":"2506.06007","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Pretrained ResNet-50 features plus simple classifiers reach about 90 to 92 percent accuracy on Monkeypox Skin Images without augmentation, while exposing data leakage in prior augmented test-set studies.","lead":"This paper tests a hybrid pipeline that uses a pretrained neural network to extract features from skin images and then trains simple classifiers, achieving about 90 percent accuracy on an orthopox dataset without any data augmentation. It also argues that several earlier high-accuracy results were inflated because those studies augmented data before splitting train and test sets, leaking test information.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline '91.49%' is a cross-validation mean (Table 7), not the fixed 77-image hold-out accuracy that Section 3.5 says underpins unbiased evaluation, so Table 9's comparison to prior test accuracies does not support the near-SOTA or data-leakage claims.","rationale":"The reader's weakest assumption targeted the reuse of the fixed 77-image test set for configuration selection and the fragility of the small, imbalanced test set. My concern is adjacent but more specific: the paper's headline accuracy is not a hold-out test accuracy at all—it is a cross-validation mean, while the fixed test set appears only in the experimental design and in Table 9's comparison. This is more damaging because it means the paper's own evaluation protocol does not produce the number that is compared with prior methods. If the 77-image hold-out accuracy were reported and were close to 91.49%, the near-SOTA claim could be restored; if not, the claim collapses. The leakage accusation against prior work is also weaker than presented, since Table 10 is consistent with split-then-augment as well as augment-then-split, and no direct evidence of train/test image overlap is given. However, this does not change the overall conditional verdict: the paper has a reproducible code repository, honest reporting of augmentation gains, and a clearly described pipeline, so it can be conditionally accepted if the missing hold-out evaluation is added and the prior-work leakage claim is verified. The concern reinforces the reader's CONDITIONAL verdict rather than moving it, hence UNCHANGED.","tokens_in":14352,"tokens_out":9748,"duration_ms":95433,"concrete_test":"Retrain the best configuration (Logistic Regression with the Table 2 hyperparameters) on all 693 original training images and evaluate exactly once on the fixed 77-image stratified hold-out defined in Section 3.5, reporting accuracy and Cohen's Kappa. If this hold-out accuracy falls outside the 95% binomial confidence interval for 77 samples around the 91.49% CV mean (roughly 84–97%), then the Table 9 headline comparison is not supported and the paper's central claim requires revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 9 lists the proposed method's accuracy as 91.49%, but this number is the 10-fold cross-validation mean for Logistic Regression reported in Table 7. Section 3.5 promises that the fixed 77-image hold-out set provides an 'impartial and comparable evaluation' and Section 4.2.1 says results are obtained 'after training ... on a final set of 77 images,' yet no table in the paper reports an accuracy computed on that hold-out set. All reported metrics (Tables 3–8) are cross-validation fold statistics. Consequently, the claim that the proposed method is near-state-of-the-art, and the claim that Bala et al.'s 97.61% and Maqsood et al.'s 98.64% are inflated by test-set augmentation, are not established: a mean over models trained on overlapping 90% subsets is not directly comparable to single-split test accuracies from prior work. The Section 5 conclusion that results are 'assessed on truly unseen data' is therefore unsupported by the evidence presented. A secondary issue: if the hold-out set was in fact used repeatedly to select among configurations, as Section 3.5 and 4.1 suggest, then it is no longer an independent test set, further weakening the generalization claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid pipeline for classifying four classes in the Monkeypox Skin Images Dataset (MSID): a frozen ImageNet-pretrained ResNet-50 extracts deep features, which are then used to train lightweight classifiers (Logistic Regression, SVM, MLP). Three training-condition experiments are compared: original data, SMOTEENN-balanced data, and data augmentation applied only to the training split. The authors report that Logistic Regression on augmented features attains the best mean cross-validation accuracy (91.49%) and use this value to claim near-state-of-the-art performance, while arguing that the higher accuracies reported by Bala et al. (97.61%) and Maqsood et al. (98.64%) are inflated because their test sets were drawn from augmented data. The paper also provides a GitHub link to source code.","tokens_in":14558,"tokens_out":4147,"duration_ms":40169,"significance":"If the evaluation were sound, the paper would make a useful practical contribution: a computationally cheap, interpretable, and auditable pipeline that does not require fine-tuning a deep network, plus a concrete warning about data leakage via test-set augmentation in prior MSID studies. The authors deserve credit for publishing hyperparameter details, making code available, and emphasizing Cohen's Kappa rather than raw accuracy. However, the central quantitative claim is currently not established because the reported headline number is a cross-validation mean, not a hold-out test accuracy, and the fixed 77-image test set appears to have been reused for model selection. These issues materially affect the comparison with prior work and the generalization claims, so the paper needs a substantial revision of its evaluation and reporting before the conclusions can be accepted.","major_comments":[{"comment":"The evaluation protocol does not support the claim of an unbiased final hold-out evaluation. Section 3.5 states that all model configurations are continuously evaluated on the fixed 77-image test set, and Section 4.1 states that the best configuration is selected after inspecting results. Any selection based on that fixed set consumes its information, so the test set is not truly unseen for the selected model. Please report the performance of a single final model on a hold-out set that was never used during development (e.g., a nested cross-validation or a strict three-way split), or present hold-out results with the selection procedure explicitly accounted for.","section":"Section 3.5 and Section 4.1"},{"comment":"The central comparison of the proposed method (91.49%) with Bala et al. (97.61%) and Maqsood et al. (98.64%) is not valid as presented. The 91.49% is the mean accuracy over 10-fold cross-validation folds reported in Table 7, not an accuracy measured on the fixed 77-image hold-out set, while the prior values are single test-set accuracies. Cross-validation means over models trained on overlapping 90% subsets are not directly comparable to a single split, so the comparison cannot support the paper's conclusion that prior results are inflated. Please report the accuracy and confidence intervals of the final selected model on a properly held-out set and use that number, not the CV mean, in Table 9.","section":"Table 9 and Sections 4.2.3/5"},{"comment":"The claim that results are 'assessed on truly unseen data' (Section 5) is not supported by the evidence, because Tables 3-8 report only cross-validation fold statistics (mean +/- std or SE) and no table reports metrics computed on the 77-image final test set. Please add a table with hold-out accuracy, precision, recall, F1, and Kappa, with confidence intervals, for all configurations, and clearly label in the text which numbers are CV estimates and which are hold-out estimates.","section":"Sections 3.5, 4.2, and 5"},{"comment":"The Mann-Whitney U tests are applied to distributions of Cohen's Kappa obtained from the 10 cross-validation folds. These fold-level values are not independent observations: the training sets overlap across folds and the same fixed test set is reused. The independence assumption of the Mann-Whitney U test is therefore violated. Please use a statistical test or resampling procedure that accounts for the dependence structure, or base the tests on independent test-set evaluations.","section":"Section 4.2.1 (and corresponding paragraphs)"}],"minor_comments":[{"comment":"Table 2 lists 'Vertical flipping with 50% probability' in the Data Augmentation row but then states 'Vertical flipping was excluded'; the text in Section 3.2 and Figure 2 also say vertical flipping is excluded. Please resolve this contradiction.","section":"Table 2"},{"comment":"There are several typos and inconsistencies: 'Propossed method' in Table 9, 'SMOTEEN' in the Table 6 heading (should be SMOTEENN), 'EE' in the Table 8 caption (should be SE), and '4.158' in Section 4.2.3 should be '4,158'.","section":"Throughout"},{"comment":"The abstract says the method works 'without the need for augmented data,' but the experiments include a data-augmentation condition and the best reported mean accuracy is from that condition. Please clarify that strong performance can be obtained without augmentation, while acknowledging that augmentation gives a small improvement.","section":"Abstract and Section 4.2.3"},{"comment":"The paper states that 100,352 features are extracted per image from ResNet-50. Please specify the exact layer and pooling strategy (e.g., the 7x7x2048 spatial features before global pooling) so the feature dimensionality is reproducible.","section":"Section 3.3 and Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea and the critique of augmentation leakage in prior work are worth publishing if the evaluation is redone. The main obstacle is that the paper reports cross-validation means as if they were hold-out accuracies and reuses the fixed test set for model selection; these are fixable with a revised protocol and clearer reporting. I would also encourage the authors to provide confidence intervals for the hold-out accuracy, given that the test set contains only 77 images."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's most valuable contribution is its concrete demonstration that Bala et al. and Maqsood et al. built their test sets from augmented data on the Monkeypox Skin Images Dataset, which inflates their reported accuracies. Table 10 lays that out fairly clearly. That critique is real and worth taking seriously.\n\nWhat's new: a frozen ImageNet ResNet-50 feature extractor plus Logistic Regression (or MLP/SVM) gets around 90% cross-validation accuracy without any augmentation or class balancing, which is a cheap, auditable baseline. The paper also honestly reports that augmentation and SMOTEENN produce only modest gains. That is useful empirical evidence.\n\nThe soft spots are real. Table 9 compares '91.49%' against Bala's 97.61% and Maqsood's 98.64%, but that 91.49% is the 10-fold cross-validation mean from Table 7, not an accuracy computed on the fixed 77-image hold-out set. Section 3.5 promises an 'impartial and comparable evaluation' on that hold-out, yet no table reports a single hold-out accuracy. A mean over folds trained on overlapping 90% subsets is not directly comparable to single-split test accuracies from prior work. Even worse, Sections 3.5 and 4.1 say all configurations were continuously evaluated on the same fixed 77-image test set, and that a randomized process selected the best parameters. So that test set is no longer independent, and the conclusion's claim that results are 'assessed on truly unseen data' is not supported by the reported numbers.\n\nThe Mann-Whitney tests on fold distributions are questionable too, since folds are not independent samples. That's minor compared to the missing hold-out result, but it should be fixed. The GitHub link is good, but there are no version pins or a README that lets anyone reproduce the split, so treat the code as a starting point rather than a verified artifact.\n\nOverall, the leakage critique is the real contribution and it deserves a serious referee. The paper needs a revised evaluation section that reports actual hold-out accuracy with confidence intervals, and it should stop claiming the hold-out is untouched if it was used for model selection. With that fixed, it could be a solid reproducibility note.","headline":"The leakage critique is the real contribution; the paper's own 91.49% is a CV mean, not a hold-out accuracy, so the comparison to prior work does not hold as written.","tokens_in":15161,"tokens_out":2404,"would_cite":true,"duration_ms":22666,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen ImageNet-pretrained ResNet-50 used as a fixed feature extractor, followed by Logistic Regression, reaches about 91.5% accuracy on the Monkeypox Skin Images Dataset, and the paper contends that earlier 97–98% state-of-the-art…","keywords":["orthopoxvirus","monkeypox","skin lesion classification","transfer learning","feature extraction","logistic regression","data augmentation","data leakage"],"falsifier":"Re-evaluate the checkpoints released with Bala et al. [6] and Maqsood et al. [27] on the 77 original, non-augmented images used as this paper's test set: if either model still scores near 97-98%, the leakage claim fails, and if accuracy collapses to the ~90% range, it is supported. A complementary test is to re-run this paper's pipeline with nested cross-validation so the fixed test set is touched only once; if Kappa drops materially below 71%, repeated use of the test set inflated the reported results.","tokens_in":14064,"feed_emoji":"🦠","tokens_out":9809,"duration_ms":82472,"temperature":0.7,"pith_summary":"This paper claims that classifying orthopoxvirus skin images does not require training a large deep network or generating synthetic data. A frozen ResNet-50 pretrained on ImageNet, used only as a feature extractor, followed by a simple Logistic Regression classifier, reaches about 90% accuracy and a Cohen's Kappa around 71% on the original Monkeypox Skin Images Dataset, and about 91.5% accuracy when the training data are augmented. The paper further claims that the 97.61% and 98.64% accuracies reported by two earlier studies are inflated because those studies applied data augmentation before splitting, so their test sets contained near-duplicates of training images. If correct, this makes a cheap, auditable pipeline the appropriate baseline for this dataset and casts doubt on reported state-of-the-art numbers.","feed_headline":"One frozen ResNet-50 undercuts inflated mpox benchmarks","feed_subtitle":"Earlier 97–98% accuracy came from testing on augmented images; this pipeline gets ~91% on clean data.","key_machinery":"The mechanism is the hybrid pipeline: a frozen ResNet-50 (trained on ImageNet, last layer removed) converts each 224x224 image into a 100,352-dimensional feature vector, and then a lightweight classifier - Logistic Regression, MLP, or SVM - is trained on those vectors. Evaluation uses stratified 10-fold cross-validation over 90% of the dataset (693 images) plus a fixed, held-out stratified test set of 77 images, with Cohen's Kappa as the primary metric because accuracy is misleading on the imbalanced four classes (normal, measles, chickenpox, monkeypox). Two preprocessing variants are added for comparison: SMOTEENN to balance the training set, and random augmentation (vertical flip and Gaussian blur, applied only to training) to enlarge it. The same fixed test set is used for every configuration, which the paper cites as the guarantee of impartial comparison.","core_discovery":"On the paper's own terms, the central discovery is that deep features extracted once by a frozen ImageNet-pretrained ResNet-50 make orthopox image classification tractable for simple classifiers: Logistic Regression attains the best balance of accuracy (90.00%) and Kappa (71.19%) on the original dataset, and neither SMOTEENN balancing nor six-fold data augmentation changes results by more than a few points. The paper also documents that the two best-performing prior studies, Bala et al. and Maqsood et al., drew their test sets from augmented data (1,738 images) instead of the original dataset, so their reported 97.61% and 98.64% accuracies do not reflect independent generalization; on a clean, stratified 77-image test set the proposed pipeline reaches at most 91.49%. The paper concludes that the apparent performance gap is an artifact of test-set contamination, not a real superiority of complex models.","pith_inferences":["The same augment-then-split leakage the paper documents for Bala et al. and Maqsood et al. likely affects other small medical imaging benchmarks, so those reported numbers should be audited in the same way before being used as baselines.","Because frozen features already saturate performance on this dataset, expensive fine-tuning of large models is probably unnecessary here; a testable extension is to compare fine-tuned ResNet-50 against the frozen-feature pipeline under nested cross-validation.","The fixed test set was reused to select the best configuration, so a natural follow-up is to set aside a second, never-touched test set and check whether the ~71% Kappa survives, or to report a confidence interval for the 77-image accuracy.","For clinical deployment the four-class accuracy matters less than the confusion pattern between monkeypox and the other rash classes; reporting per-class errors would show whether mistakes are clinically safe, which is where this dataset's next evaluation should focus."],"forward_implications":["The two highest published accuracies on the Monkeypox Skin Images Dataset should not be cited as state of the art without first checking whether their test sets were produced by augmentation before splitting; this paper provides the evidence of that leakage.","A linear classifier on frozen, pretrained deep features is a strong and cheap baseline for small medical image datasets, likely beating many bespoke deep models at a fraction of the training cost.","Data augmentation should be applied only after the training/validation/test split, because any accuracy gain from augmenting the full dataset and then splitting is not a real gain.","Balancing and augmentation give diminishing returns: increasing training size by 34% with SMOTEENN or by 600% with augmentation changed Kappa by only a few points, so model capacity is not the bottleneck on this dataset.","The proposed method's generalization claims rest on a single 77-image test set, and point estimates like 91.49% carry wide confidence intervals at that size."],"supporting_citations":[{"why":"Supplies the Monkeypox Skin Images Dataset and the 97.61% accuracy figure used as the main comparison point.","marker":"[6]"},{"why":"Supplies the 98.64% accuracy figure and the hybrid transfer-learning-plus-classifier approach this paper extends and critiques.","marker":"[27]"},{"why":"Defines the ResNet architecture that is used frozen as the feature extractor.","marker":"[21]"},{"why":"Provides the ImageNet dataset on which the frozen weights were pretrained.","marker":"[16]"},{"why":"Introduces the SMOTEENN algorithm used for the balanced-dataset variant of the experiments.","marker":"[26]"},{"why":"Describes the stratified cross-validation and hold-out split protocol used for evaluation.","marker":"[33]"}],"fun_headline_variants":["Frozen ResNet-50 features hit 91% on clean orthopox images","Inflated mpox benchmarks traced to augmented test sets","No augmentation needed: frozen features reach 91% accuracy","Hybrid model exposes overhyped orthopox performance gaps","Simple classifiers on frozen features beat complex deep nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the fixed 77-image test set can be used repeatedly to pick the best model configuration and still count as unseen data; if the test set influenced the choice of classifier or hyperparameters, the reported accuracies overstate generalization.","fun_headline_variants_meta":{"raw":{"variants":["Frozen ResNet-50 features hit 91% on clean orthopox images","Inflated mpox benchmarks traced to augmented test sets","No augmentation needed: frozen features reach 91% accuracy","Hybrid model exposes overhyped orthopox performance gaps","Simple classifiers on frozen features beat complex deep nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1483,"prompt_tokens":877,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":493,"tokens_out":606,"duration_ms":6090,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T06:01:58.879588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-evaluate the checkpoints released with Bala et al. [6] and Maqsood et al. [27] on the 77 original, non-augmented images used as this paper's test set: if either model still scores near 97-98%, the leakage claim fails, and if accuracy collapses to the ~90% range, it is supported. A complementary test is to re-run this paper's pipeline with nested cross-validation so the fixed test set is touched only once; if Kappa drops materially below 71%, repeated use of the test set inflated the reported results.","supporting_citations":[{"cited_title":"Monkeynet: A robust deep convolutional neural network for monkeypox disease detection and classification.Neural Networks, 161:757–775, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Monkeypox Skin Images Dataset and the 97.61% accuracy figure used as the main comparison point."},{"cited_title":"Mox-net: Multi-stage deep hybrid feature fusion and selection framework for monkeypox classification.Expert Systems with Applications, 255:124584, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the 98.64% accuracy figure and the hybrid transfer-learning-plus-classifier approach this paper extends and critiques."},{"cited_title":"Classification of cardiac arrhythmia of 12 lead ecg using combination of smoteenn, xgboost and machine learning algorithms","cited_arxiv_id":null,"evidence_quote":"Introduces the SMOTEENN algorithm used for the balanced-dataset variant of the experiments."},{"cited_title":"Data classification with k-fold cross validation and holdout accuracy estimation methods with 5 different machine learning techniques","cited_arxiv_id":null,"evidence_quote":"Describes the stratified cross-validation and hold-out split protocol used for evaluation."}],"review_version":1}