{"id":"be43a535-30dc-48a9-8db3-07fd9b611e59","arxiv_id":"2501.05991","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A Vision Transformer with CBAM attention achieves 93.46% accuracy on a newly curated 39-class skin lesion dataset, outperforming four other models.","lead":"This paper merges five public image collections into a 39-class skin lesion dataset and benchmarks five deep learning models with and without attention modules. It reports that a Vision Transformer with CBAM attention reaches 93.46% accuracy, and the dataset and code are shared on GitHub.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 93.46% accuracy claim depends on a leakage-free 70/15/15 split of the 39×130 curated dataset; Section 3.1 caps classes but never de-duplicates or groups by patient/source, so near-duplicate images could appear in both training and test sets.","rationale":"I read the paper as a benchmark study whose main empirical claim is the reported 93.46% accuracy of ViT+CBAM on the curated 39-class dataset. The most load-bearing assumption is that the test set is truly independent of the training set. The paper's Section 3.1 explicitly describes a balanced, image-level 70/15/15 split of merged public datasets but never mentions de-duplication or patient/source grouping. Since the source datasets overlap (e.g., HAM10000 appears both directly and inside ISIC 2019, and Dermnet appears in multiple listed Kaggle sources), exact or near-duplicate images across splits is a concrete, plausible failure mode rather than a speculative one. This concern is exactly what the reader identified as the weakest assumption. I do not see an internal inconsistency in the method as described; the issue is an omitted safeguard that is standard for medical imaging benchmarks. Other limitations—single-run evaluation, lack of error bars, selection of the best model on the same test set, and incomplete ViT hyperparameters—are real but secondary; they affect confidence intervals and reproducibility, whereas leakage directly targets the validity of the headline number. I therefore agree with the reader's conditional verdict: the paper should be accepted only after the split is shown to be leakage-free through a concrete deduplication and grouped-split check.","tokens_in":16490,"tokens_out":3946,"duration_ms":41282,"concrete_test":"Download the repository and compute exact (MD5) and perceptual (pHash/dHash) hashes for all images in the curated 39×130 dataset. Flag any near-duplicate pair that crosses the 70/15/15 boundary (e.g., pHash Hamming distance below 10 of 64 bits). Then retrain ViT+CBAM on a deduplicated, source-grouped split, grouping images by originating Kaggle dataset and, where available, patient ID, and report test accuracy. If the accuracy drops materially (more than about 1–2 points) or the top model changes, the 93.46% figure is inflated by leakage; if no cross-split near-duplicates exist and the grouped-split accuracy is unchanged at roughly 93%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that ViT+CBAM achieves 93.46% accuracy on a curated 39-class test set. For that number to be meaningful, the test images must be independent of the training images. Section 3.1 describes merging five public datasets, capping each class at 130 images, and then splitting 70/15/15 at the image level. No de-duplication step is described, and no patient-level or source-level grouping is used. Because the sources overlap—HAM10000 images can appear both through the direct HAM10000 subset and through the ISIC 2019 subset of the merged dataset [10]; Dermnet images can appear in more than one Kaggle collection—the same lesion photograph may exist in both training and test splits, either as an exact duplicate or as a near-duplicate from the same lesion. This would directly inflate the reported 93.46% accuracy and could also affect the relative ranking of models. The GitHub repository is cited, but no commit hash, split seed, or exact train/test file lists are provided, so the split cannot currently be audited from the paper. The class-wise perfect scores for 18 classes, each with only 20 test images, are consistent with small-sample noise, but the leakage issue is the more serious threat to the main claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an attention-guided deep learning framework for classifying 39 types of skin lesions. It curates a dataset by merging five public sources (ISIC 2019, Atlas Dermatology, HAM10000, MSLD 2.0, Dermnet-based collections), caps each class at 130 images, and splits the data 70/15/15 into train/validation/test sets of 90/20/20 images per class. Five architectures (MobileNetV2, Xception, InceptionV3, EfficientNetB1, Vision Transformer) are evaluated in baseline form and with two attention modules (ECA, CBAM). The authors report that ViT+CBAM achieves the best results: 93.46% accuracy, 94% precision, 93% recall, 93% F1-score, and 93.67% specificity. The paper includes class-wise accuracy, confusion matrix, ROC curves, and a GitHub link for code and data.","tokens_in":16758,"tokens_out":4758,"duration_ms":43560,"significance":"If the results are valid, the paper would provide a useful 39-class benchmark for skin lesion classification and evidence that attention mechanisms (especially CBAM) improve a Vision Transformer baseline on such a multi-class task. The dataset integration effort is potentially valuable, and the inclusion of code and dataset links supports reproducibility. However, the headline accuracy and the comparative conclusion are only meaningful if the training/test split is leakage-free and the best model is not selected on the same test set used for evaluation; these issues are the central validity concerns of the paper.","major_comments":[{"comment":"Section 3.1 describes merging five public datasets and capping each class at 130 images, but it does not describe any de-duplication or patient/source-level split. Because HAM10000 images are also part of ISIC 2019, and Dermnet images appear in multiple Kaggle collections, the same lesion photograph (or near-duplicates from the same lesion) can be present in both training and test splits. This would inflate the reported 93.46% accuracy and could change the relative ranking of models. Please specify a de-duplication procedure (e.g., exact-duplicate removal via perceptual hashing, and near-duplicate grouping by lesion or patient identifiers) and provide the exact split (file lists and random seed) so the train/test independence can be audited.","section":"3.1"},{"comment":"The test set of 20 images per class is used both to compare all 15 model configurations and to select the 'best' model (ViT+CBAM) in Table 4. Selecting the best configuration on the same data used for final reporting introduces optimistic bias, so the 93.46% accuracy is not an unbiased estimate of the selected model's generalization performance. The authors should use a validation set for model selection and a separately held-out test set for the final evaluation, or report repeated runs with different seeds and confidence intervals for the comparisons.","section":"4.3, Table 4"},{"comment":"Each class-wise accuracy in Table 5 is computed from only 20 test images. For example, the 100% accuracies for 18 classes correspond to 20/20 correct, and the 50% for Leprosy Lepromatous corresponds to 10/20; these values have very wide binomial confidence intervals, so statements such as 'notable successes with 100% accuracy' are statistically fragile. The overall accuracy of 93.46% (729/780) also lacks a confidence interval. Please report exact counts and confidence intervals, and temper the class-wise claims accordingly.","section":"4.3, Table 5"},{"comment":"The Vision Transformer hyperparameters are not reported: there is no learning rate, batch size, number of epochs, optimizer, patch size, or embedding dimension. The other four models are given such details, so this omission prevents reproducibility of the central ViT+CBAM result. In addition, the GitHub repository is cited without a commit hash or exact train/test file lists, and the split seed is not provided. Please supply complete hyperparameters and a versioned, auditable data split.","section":"3.3, 4.1"}],"minor_comments":[{"comment":"The spatial attention equations use F as their input and denote the output as M_c(F), but the spatial attention module is described as operating on the channel-refined feature F' and should produce M_s(F'). The notation should be corrected to avoid confusion.","section":"3.4.2, Eqs. (4)-(6)"},{"comment":"The class numbering and naming are inconsistent: in Table 3, '34. Psoriasis' appears before '33. Scabies Lyme Disease', and Table 5 lists 'Lerva Migrans' for what is elsewhere 'Larva Migrans'. These should be unified and corrected.","section":"Tables 3 and 5"},{"comment":"Figure 1 lists SGD, Adam, and RectifiedAdam as optimizers, but the text never specifies which optimizer is used for each model or configuration; please clarify.","section":"Figure 1"},{"comment":"The statement 'The overall area under the curve (AUC) score of 0.99' does not specify whether this is macro-average, micro-average, or per-class; Figure 9 would benefit from a clear definition in the caption or text.","section":"5.1"},{"comment":"References [4] and [17] appear to be the same paper (Choudhary et al., Chemometrics and Intelligent Laboratory Systems 2022) cited twice in different contexts; please consolidate.","section":"References [4] and [17]"}],"recommendation":"major_revision","confidential_remarks":"The core contribution—a 39-class merged dataset and a comparison of attention-augmented deep learning models—is potentially useful, but the current manuscript does not establish that the reported 93.46% accuracy is unbiased. The missing de-duplication and the use of the same test set for model selection are risks that likely require re-running experiments or at least a thorough audit of the split. The fix is within the scope of a revision, so major_revision seems appropriate rather than reject. I would also note that the paper's claims would be stronger with a more careful treatment of uncertainty around the small per-class test counts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a benchmark paper, not a methods paper. The new thing is a 39-class merged skin lesion dataset (five public sources, 130 images per class) and a broad comparison of five models with ECA/CBAM. That is genuinely useful to the community, and the misclassification analysis (Leprosy Lepromatous at 50%) is a nice touch. But the headline claim — ViT+CBAM at 93.46% — rests on a split that may leak. Section 3.1 caps classes but never describes de-duplication; the sources overlap (HAM10000 appears inside ISIC 2019, Dermnet appears in more than one Kaggle collection), and the split is image-level. So the same lesion photo could easily be in both train and test. That alone makes the accuracy number unreliable and could shift the model ranking.\n\nOther soft spots are in proportion: single run, no confidence intervals, best model selected from the same test set, and only 20 test images per class (so 18 classes at 100% is mostly noise). The ViT+CBAM integration is described at a high level but not precisely enough to reproduce — no exact placement of CBAM in the ViT pipeline, no commit hash for the GitHub repo. There are also minor copyediting slips (class numbering in Table 3, 'Lerva Migrans' in Table 5). None of these are fatal on their own; together they mean the paper is not yet ready as-is.\n\nWhat it does well: the related work table is a useful map of the field, the 39-class merge is a real contribution, and the paper honestly reports where it fails rather than cherry-picking only success. The limitations section is reasonable.\n\nMy take: the core benchmark idea is fine and worth publishing after revision. A serious referee should insist on: de-duplication (exact and near-duplicate) across the merged sources; a source- or patient-level split; repeated runs with seeds and confidence intervals; and a code/data release with a commit hash. If those are fixed, the 39-class dataset and model comparison will be a useful citation. If not, the 93.46% number should not be taken at face value.\n\nI would send this to peer review — with the expectation of major revision — because the dataset and benchmark are valuable enough to deserve careful scrutiny.","headline":"Useful 39-class skin lesion benchmark, but the headline accuracy is unverified and likely optimistic until de-duplication and split methodology are fixed.","tokens_in":17303,"tokens_out":2738,"would_cite":true,"duration_ms":26245,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding CBAM to a Vision Transformer yields 93.46% accuracy in classifying 39 skin lesion types.","keywords":["skin lesion classification","Vision Transformer","CBAM","attention mechanism","multi-class classification","dataset curation","transfer learning","dermatology imaging"],"falsifier":"Re-run the ViT+CBAM evaluation after removing near-duplicate images and splitting by patient or lesion identity rather than by image; if accuracy drops substantially below 93.46%, the reported test result is inflated by data leakage. A simpler check is to search the training and test sets for identical or near-identical images across the five source collections.","tokens_in":16265,"feed_emoji":"🩺","tokens_out":7228,"duration_ms":63334,"temperature":0.7,"pith_summary":"This paper claims that a Vision Transformer augmented with a convolutional block attention module can classify 39 types of skin lesions from photographs, reaching 93.46% accuracy on a held-out test set. The authors build a balanced dataset by merging five public skin-image collections and capping every class at 130 images, then compare five deep models with and without two attention modules. Their central result is that ViT+CBAM beats the plain ViT, all CNN baselines, and every ECA-augmented variant on accuracy, precision, recall, F1-score, and specificity. If the result holds, attention-guided transformers become a credible tool for broad multi-class dermatology triage rather than the usual small set of lesion types.","feed_headline":"Vision Transformer plus CBAM hits 93.46% on 39 skin lesion types","feed_subtitle":"Merging five public skin-image sets into 39 balanced classes puts attention-guided transformers ahead of all baselines.","key_machinery":"The load-bearing mechanism is CBAM, the Convolutional Block Attention Module, applied after the Vision Transformer encoder. CBAM first computes a channel attention map from average- and max-pooled features through a shared MLP, then a spatial attention map from channel-pooled features through a 7×7 convolution, applying the two maps sequentially to refine the transformer's feature representation. It is the component that separates the proposed model from the plain ViT baseline and produces the claimed gain from 91.79% to 93.46%. The dataset curation — merging five public sources, capping each class at 130 images, and using a 70/15/15 split — is the supporting machinery that makes the 39-class comparison possible.","core_discovery":"The paper's central discovery is an empirical ranking of model configurations on a curated 39-class dataset. Under a 70/15/15 split with 130 images per class, the Vision Transformer with CBAM achieves 93.46% accuracy, 94% precision, 93% recall, 93% F1-score, and 93.67% specificity, outperforming the plain ViT (91.79%) and every other baseline and attention variant tested. The authors attribute this improvement to CBAM's sequential channel and spatial attention, which refines the global features produced by the transformer encoder so the model concentrates on diagnostically relevant regions. The overall ROC AUC is reported as 0.99. The same claim appears in the abstract and the conclusion, so the paper is trying to establish that this particular attention-guided configuration is the strongest among those compared.","pith_inferences":["Because the split is image-level and no de-duplication or patient-level separation is reported, the 93.46% figure is likely optimistic; re-evaluating on a deduplicated or patient-level split would reveal how much of the score comes from near-duplicate images.","The balanced 130-image-per-class design removes natural prevalence, so the accuracy number does not directly transfer to real-world screening where some lesion types are far rarer than others.","The Leprosy Lepromatous confusion with Molluscum Contagiosum suggests the same architecture could gain from clinical metadata or histopathology, as the authors propose in their future-work section, rather than from more image augmentation alone.","The same 'transformer plus convolutional attention' recipe could be tested on other fine-grained medical image tasks where global context and local boundary detail both matter."],"forward_implications":["On the curated 39-class dataset, ViT+CBAM outperforms MobileNetV2, Xception, InceptionV3, EfficientNetB1, and the plain Vision Transformer under the same training protocol.","Adding either ECA or CBAM improves every baseline model relative to the same architecture without attention, so the paper supports attention modules as a general enhancement for skin-lesion classifiers.","Class-wise results show perfect accuracy on 18 of 39 classes and an overall ROC AUC of 0.99, indicating the model separates most classes cleanly.","The released dataset and code let other groups reproduce the 39-class comparison and test new architectures against the same benchmark.","The known weak spot, Leprosy Lepromatous at 50% accuracy, is explicitly attributed to visual similarity with Molluscum Contagiosum, so the paper identifies where image-only classification reaches its limit."],"supporting_citations":[{"why":"Defines the 31-class merged dataset from which 12 curated classes are taken.","marker":"[10]"},{"why":"One of the five source collections; contributes 3 skin-lesion classes to the curated dataset.","marker":"[11]"},{"why":"Source collection merging HAM10000 and MSLD; contributes 7 curated classes.","marker":"[12]"},{"why":"Source collection derived from Dermnet; contributes 16 curated classes.","marker":"[13]"},{"why":"Source collection from Atlas Dermatology; contributes 1 curated class.","marker":"[14]"},{"why":"HAM10000 is the underlying source for part of the curated data and the dataset used by several related baselines.","marker":"[20]"},{"why":"Defines the MSLD 2.0 collection that contributes classes through the merged source.","marker":"[43]"},{"why":"Dermnet dataset, a source for the 23-class collection from which 16 classes are taken.","marker":"[44]"},{"why":"Defines the ECA module used as the comparison attention mechanism.","marker":"[45]"},{"why":"Defines the CBAM module whose integration with ViT is the proposed method.","marker":"[46]"}],"fun_headline_variants":["Attention-guided ViT tops 39 skin lesion types at 93.46%","CBAM refines ViT to 93.46% accuracy across 39 lesion classes","Merged 5 datasets, 39 classes: ViT+CBAM wins at 93.46%","Best of 5 models: ViT with CBAM hits 93.46% on 39 skin lesions","39 skin lesions, one winner: Vision Transformer + CBAM at 93.46%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the 39 merged classes are correctly labelled and that the random split of individual images, rather than patients or unique lesions, does not put the same or near-identical pictures into both training and testing.","fun_headline_variants_meta":{"raw":{"variants":["Attention-guided ViT tops 39 skin lesion types at 93.46%","CBAM refines ViT to 93.46% accuracy across 39 lesion classes","Merged 5 datasets, 39 classes: ViT+CBAM wins at 93.46%","Best of 5 models: ViT with CBAM hits 93.46% on 39 skin lesions","39 skin lesions, one winner: Vision Transformer + CBAM at 93.46%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3356,"prompt_tokens":1028,"completion_tokens":2328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":2205}},"tokens_in":644,"tokens_out":2328,"duration_ms":14517,"temperature":1.0,"reasoning_tokens":2205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:48.306662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ViT+CBAM evaluation after removing near-duplicate images and splitting by patient or lesion identity rather than by image; if accuracy drops substantially below 93.46%, the reported test result is inflated by data leakage. A simpler check is to search the training and test sets for identical or near-identical images across the five source collections.","supporting_citations":[{"cited_title":"Rafay, W","cited_arxiv_id":null,"evidence_quote":"Defines the 31-class merged dataset from which 12 curated classes are taken."},{"cited_title":"Accessed on 24 May 2024","cited_arxiv_id":null,"evidence_quote":"One of the five source collections; contributes 3 skin-lesion classes to the curated dataset."},{"cited_title":"Accessed on 18 May 2024","cited_arxiv_id":null,"evidence_quote":"Source collection merging HAM10000 and MSLD; contributes 7 curated classes."},{"cited_title":"Accessed on 18 May 2024","cited_arxiv_id":null,"evidence_quote":"Source collection derived from Dermnet; contributes 16 curated classes."},{"cited_title":"Accessed on 24 May 2024","cited_arxiv_id":null,"evidence_quote":"Source collection from Atlas Dermatology; contributes 1 curated class."},{"cited_title":"Tschandl, C","cited_arxiv_id":null,"evidence_quote":"HAM10000 is the underlying source for part of the curated data and the dataset used by several related baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MSLD 2.0 collection that contributes classes through the merged source."},{"cited_title":"Accessed on 18 May 2023","cited_arxiv_id":null,"evidence_quote":"Dermnet dataset, a source for the 23-class collection from which 16 classes are taken."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ECA module used as the comparison attention mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CBAM module whose integration with ViT is the proposed method."}],"review_version":1}