{"id":"04db8407-4c45-4cd5-91c3-4425abc7b240","arxiv_id":"2509.04800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"On a new 51-class dataset of everyday skin photos, the Swin Transformer outperforms eight CNN and transformer baselines, reaching about 81 percent accuracy.","lead":"The paper builds a large collection of skin images taken in everyday settings and tests deep learning models to sort them into over 50 disease categories. A Swin Transformer model reached the best accuracy, around 81 percent, which points toward cheap automated skin screening in places without dermatologists.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central dataset claim is internally inconsistent: Table I sums to 24,351 images, not the more than 27,000 stated, and the mobile-acquired provenance and label integrity are unverified; the broken repository link prevents independent checks.","rationale":"The paper's central contribution is the curated dataset, and the Swin-Base performance claim depends entirely on that dataset being as described. The first test is internal consistency, and the paper fails it: Table I sums to 24,351 images rather than the more than 27,000 claimed, and the exclusion of classes described in Section IV-A makes the actual experimental set even smaller. This is not a matter of interpretation; it is a countable discrepancy that undermines the headline dataset-size claim. Beyond the count, the mobile-acquired provenance is asserted without audited evidence, label correctness is not expert-validated, and no de-duplication across web sources is reported, so the reported accuracy and MCC could be inflated by near-duplicate leakage or label noise. These are substantive concerns, but they are correctable: the authors can release the dataset manifest, correct the count, audit capture modality and labels, and rerun on a deduplicated split. The reader's CONDITIONAL verdict already requires exactly this kind of verification, and this stress-test reinforces that requirement rather than moving to a different verdict.","tokens_in":9899,"tokens_out":8938,"duration_ms":73933,"concrete_test":"After repairing the repository link, independently download the dataset and compare per-class file counts with Table I and the claimed >27,000 total; then run an exact and perceptual near-duplicate check (e.g., pHash) between the 70/15/15 splits. If the reconciled total is below 27,000 or duplicate leakage is non-negligible, the authors must report metrics on a corrected, deduplicated split or revise the central claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A states that the curated dataset contains 'more than 27,000 images' across over 50 skin disease categories, and the abstract repeats this as a primary contribution. Summing every category count in Table I yields 24,351 images across 68 listed categories, about 2,650 below the claimed total. Section IV-A then states that categories with too few images were excluded, leaving 51 classes, so the actual training set is even smaller. No explanation reconciles the discrepancy. This matters because the dataset size and novelty ('one of the largest non-dermoscopic collections') are load-bearing: the reported Swin-Base accuracy of 0.808 and MCC of 0.802 are only meaningful on the dataset actually used. The 'mobile-acquired' claim is also unsupported. The only explicitly smartphone-sourced component is PAD-UFES-20; Dermatology Atlas and the Monkeypox repository are web/atlas collections, and no EXIF, capture-source, or device audit is reported. Label correctness is not established by expert validation, and many classes have about 100 samples, so label noise could alter model rankings. No de-duplication across the merged online sources is described, so train/test near-duplicates could inflate the reported figures. The repository URL in the Acknowledgments is malformed, preventing independent verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript curates a non-dermoscopic skin-image dataset from online sources, claims it contains more than 27,000 images across over 50 skin disease categories captured with mobile devices, and evaluates six CNN and three Transformer-based architectures on a 51-class subset. The authors report that Swin-Base achieves the best test-set performance (accuracy 0.808, F1 0.808, MCC 0.802) and use Grad-CAM to visualize the regions driving predictions. The stated goal is to demonstrate that Transformer models, and Swin in particular, outperform CNN baselines on realistic mobile-acquired images, thereby supporting accessible AI-assisted dermatological screening in low-resource settings.","tokens_in":10170,"tokens_out":4720,"duration_ms":42030,"significance":"If the dataset and evaluation claims are supported, the curated public dataset would be a useful resource for non-dermoscopic skin-image research, and the head-to-head comparison of CNNs and Transformers on such images would be informative. The paper also has strengths: it reports MCC, discusses fine-tuning and data-augmentation experiments, and states limitations honestly. However, the central contributions rest on dataset statistics and provenance that are not yet verified, and the architecture comparison is confounded by input resolution and single-split evaluation. These issues are load-bearing rather than cosmetic, so the claims in their current form are not fully established.","major_comments":[{"comment":"The text states that the curated dataset contains 'more than 27,000 images,' but Table I lists 68 categories whose counts sum to 24,351. Section IV-A then states that categories with too few images were excluded, leaving 51 classes, so the actual training pool is smaller still. This discrepancy directly affects the paper's primary contribution of providing 'one of the largest publicly reported collections of non-dermoscopic skin lesion images.' Please reconcile the numbers, report the exact total before and after filtering, list which categories were excluded, and provide final per-class counts and train/validation/test sizes.","section":"Section III-A and Table I"},{"comment":"The claim that the images are 'captured with mobile devices' is not supported by the evidence in the manuscript. Only PAD-UFES-20 is explicitly described as smartphone-acquired; Dermatology Atlas and the Monkeypox repository are web/atlas collections, and no EXIF, device, or capture-source audit is reported. Section V-E later says all images were 'collected from online sources,' which is a different claim. Please provide verifiable provenance per source, or substantially qualify the mobile-acquired characterization throughout the abstract, introduction, and conclusion.","section":"Section III-A and Abstract"},{"comment":"Label correctness and image uniqueness are not established. The sources are unverified online repositories, no expert validation protocol is described, no de-duplication procedure is reported, and the class list contains inconsistent labels such as 'acne' versus 'acne vulgaris.' Because label noise and near-duplicate leakage between training and test sets can alter model rankings, please provide a validation protocol, a deduplication procedure, and a quantitative error analysis. In addition, the repository link in the Acknowledgments is malformed (it contains a space), which prevents independent verification of the dataset and code.","section":"Section III-A and Section V-E"},{"comment":"The evaluation is based on a single data split, one checkpoint per model, and no error bars or significance tests, so the reported differences between architectures may not be stable. Moreover, Table III shows that CNN inputs are 224x224 or 299x299 while Transformer inputs are 384x384, so image resolution is confounded with architecture. The conclusion that Swin-Base outperforms CNN baselines because of self-attention and global feature capture is therefore not supported by the experiment as designed. Please add repeated runs with confidence intervals, control input resolution across architecture families, or otherwise justify that the observed ranking is not an artifact of resolution or split.","section":"Section IV-A, Tables III and IV"}],"minor_comments":[{"comment":"The notation 'Precision w', 'Recall w', and 'F1 w' is used without defining that the weights are the class support proportions; please define the weighted averaging convention explicitly.","section":"Section IV-B"},{"comment":"The GitHub URL is given as 'https://github.com/newaz-aa/skin lesion classification DL', which contains a space and is not a usable link; please provide the correct URL.","section":"Acknowledgments"},{"comment":"The predicted label 'juvenile xanthogranuloma' does not appear in Table I, so it is unclear whether this class is among the 51 evaluated classes or the figure is mislabeled; please correct this inconsistency.","section":"Section V-B, Figure 4b"},{"comment":"The accuracy curves for ConvNeXt and ResNet-50 are plotted as fractions, while the Swin accuracy curve is plotted in percent; please use consistent units and label the axes accordingly.","section":"Section V-A, Figures 1-3"}],"recommendation":"major_revision","confidential_remarks":"The practical contribution of this paper is largely the proposed dataset. If, after revision, the dataset statistics and provenance cannot be fully verified, I would not support acceptance because the central claims and the reported 81% F1 / 80% MCC figures depend on those numbers. The architecture comparison is also weaker than the narrative suggests because of the resolution confound and lack of repeated runs; however, both issues are addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2509.04800. The useful part is the benchmark: a 51-class non-dermoscopic skin lesion dataset, bigger than PAD-UFES-20, and a clean comparison of CNNs versus transformers on it. Swin-Base gets 81% accuracy and MCC, which is plausible for this kind of data. If the dataset were actually public and the numbers held up, it would be a solid resource for mobile skin-screening research.\n\nWhat the paper does well: the training setup is standard and described clearly enough to reproduce—frozen backbones, custom head, early stopping, fixed seed. The limitation section is honest; they note small per-class sample sizes, online-source heterogeneity, and no patient history. Grad-CAM examples look reasonable, though they don't add much beyond showing the model looks at the lesion.\n\nNow the soft spots, and they're not tiny. The abstract and Section III-A claim \"more than 27,000 images.\" Table I, the paper's own summary, sums to 24,351 across the 68 listed categories. Then Section IV-A says low-image categories were excluded, leaving 51 classes, so the actual training set is smaller still. That's a 10% overstatement in the headline number, with no reconciliation. The \"mobile-acquired\" label is also mostly assertion. Only PAD-UFES-20 is explicitly smartphone-sourced; the rest come from web or atlas collections with no EXIF or capture-source audit. Label correctness rests on unverified online sources, and no de-duplication is described, so train/test near-duplicates can't be ruled out. Finally, the repo link in the Acknowledgments is malformed, so independent verification is impossible right now.\n\nThe architecture comparison itself is fine—transformers beating CNNs on this task is no surprise, and the 81% figure is believable. The interpretive claim that Swin wins because of global context is post hoc and untested, but they don't lean on it heavily.\n\nOverall: the dataset idea is worth taking seriously, but the paper as written overstates its size, provenance, and accessibility. Fix the count, verify or soften the mobile-acquired claim, release a working link, and add error bars or multiple splits—then it deserves a real look. As is, I'd send it to peer review with a request for major revision, not desk-reject it: the benchmark is potentially useful and the core method is sound enough to evaluate.\n\nReading group: maybe, as a case study in dataset-claim auditing.","headline":"The proposed 51-class non-dermoscopic benchmark is a useful idea, but the paper's own numbers and broken dataset link undercut the central claims.","tokens_in":10709,"tokens_out":1912,"would_cite":false,"duration_ms":16607,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Swin Transformer beats CNN baselines on phone-photo skin disease classification, hitting an F1 of 81 percent.","keywords":["skin lesion classification","non-dermoscopic images","mobile-acquired images","Swin Transformer","transfer learning","Grad-CAM","medical image analysis","multiclass skin disease dataset"],"falsifier":"Sum the per-category image counts in Table I and compare the result with the claimed total of over 27,000 images; then inspect a random sample for phone-camera provenance and have a dermatologist verify a sample of the labels. If the sum falls far short of the claim, or if a meaningful share of sampled images fails the provenance or label check, the reported 81 percent F1 cannot be taken at face value. An independent check would be to run Swin-Base on a separately collected mobile-acquired skin image dataset and see whether accuracy and MCC remain near 80 percent.","tokens_in":9713,"feed_emoji":"📱","tokens_out":3865,"duration_ms":33711,"temperature":0.7,"pith_summary":"This paper sets out to show that automated skin-disease classification can work on ordinary mobile-phone photos, not just on specialized dermoscopic images. To that end, the authors curate a collection of over 27,000 non-dermoscopic images spanning more than 50 skin disease categories, evaluate a range of CNN and Transformer models on it, and report that the Swin Transformer achieves the best results, with an F1 score of 81 percent and an MCC of 80 percent. The claim matters because a system that works on phone photos could bring preliminary dermatological screening to regions without dermatologists or dermoscopy equipment. The paper also uses Grad-CAM heatmaps to argue that the model focuses on clinically relevant lesion regions rather than background artifacts.","feed_headline":"Swin Transformer tops 51-class skin test on phone photos","feed_subtitle":"A 27,000-image mobile-photo dataset shows transformers beat CNNs for real-world skin disease screening.","key_machinery":"The load-bearing mechanism is the Swin Transformer trained via transfer learning: its hierarchical, shifted-window self-attention, which divides images into patches and computes attention within shifted windows across stages, is what the paper credits for capturing both local lesion detail and global context. The other essential component is the curated non-dermoscopic dataset itself, which supplies the 51-class, real-world mobile-image distribution; both the architecture comparison and the interpretability analysis depend on it.","core_discovery":"The paper's central claim is that Transformer-based architectures, specifically Swin-Base, outperform CNN baselines for multiclass classification of mobile-acquired, non-dermoscopic skin images, reaching 80.8 percent accuracy, 81 percent F1, and 80.2 percent MCC on a 51-class test set. It further claims that this performance rests on a newly curated dataset of more than 27,000 images across over 50 skin disease categories collected from public online sources, described as one of the largest publicly reported non-dermoscopic collections. The authors argue that self-attention enables Swin to capture global contextual features that CNNs miss, and they use Grad-CAM visualizations to show that the models attend to lesion regions rather than background cues.","pith_inferences":["My reading: if the online-sourced labels are noisy, the reported 81 percent F1 is likely an upper bound on real-world performance, because label noise usually inflates apparent agreement on test data drawn from the same noisy source.","My reading: the 'global context' explanation for Swin's edge is consistent with the results but not directly proven; an ablation restricting attention windows, or analysis of attention-map statistics, would test it.","My reading: the dataset's imbalance, with some classes having around 100 images, means per-class reliability varies substantially; deployment should probably be limited to well-represented classes until more data is added.","A testable extension the paper does not run is evaluating the same models on a held-out set of freshly taken phone photos rather than the curated web collection, which would measure true domain transfer to real screening conditions."],"forward_implications":["If the 81 percent F1 and MCC results hold, a phone-photo-based screening tool is within reach for dozens of common skin disease categories, including in settings with no dermatologist or dermoscopy equipment.","Transformer architectures become the preferred starting point for non-dermoscopic skin classification over CNN baselines such as ResNet-50 and EfficientNet-B0.","The released curated dataset gives other researchers a common benchmark for mobile-acquired skin images, reducing the field's reliance on dermoscopic collections.","Frozen-backbone transfer learning with ImageNet weights is sufficient to obtain strong results on this data scale; fine-tuning backbone layers tends to cause overfitting.","Grad-CAM heatmaps can serve as a basic trust check, showing whether a prediction is driven by the lesion itself rather than by background or image artifacts."],"supporting_citations":[{"why":"Supplies the PAD-UFES-20 smartphone-acquired images, a key component of the curated dataset.","marker":"[9]"},{"why":"Provides approximately 500 Monkeypox images included in the curated dataset.","marker":"[17]"},{"why":"Serves as a source for images of rare skin lesions in the curated dataset.","marker":"[18]"},{"why":"Defines the Swin Transformer architecture that achieves the best reported performance.","marker":"[19]"},{"why":"Introduces the Vision Transformer and motivates the global-context advantage attributed to transformer models.","marker":"[8]"},{"why":"Supports the frozen-backbone transfer learning approach and explains why deeper fine-tuning caused overfitting.","marker":"[20]"},{"why":"Represents the dermoscopic ISIC archive that the paper contrasts with its non-dermoscopic dataset.","marker":"[4]"},{"why":"Represents the dermoscopic HAM10000 benchmark that the paper contrasts with its mobile-acquired dataset.","marker":"[5]"}],"fun_headline_variants":["Swin Transformer tops CNN on 51 skin classes from phone photos","New mobile-photo dataset drives 51-class skin lesion AI to 80.8%","AI on phone photos: Swin nets win 51-class skin disease test","27k phone images show Transformers beat CNNs for skin diagnosis","Skin check via smartphone: Swin achieves 80.8% on 51 classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result rests on the curated dataset being what it claims: genuine, correctly labeled mobile-phone photos numbering more than 27,000, even though the images were gathered from unverified online sources and the category counts in the paper's own table do not obviously sum to the stated total.","fun_headline_variants_meta":{"raw":{"variants":["Swin Transformer tops CNN on 51 skin classes from phone photos","New mobile-photo dataset drives 51-class skin lesion AI to 80.8%","AI on phone photos: Swin nets win 51-class skin disease test","27k phone images show Transformers beat CNNs for skin diagnosis","Skin check via smartphone: Swin achieves 80.8% on 51 classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3622,"prompt_tokens":862,"completion_tokens":2760,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":2658}},"tokens_in":478,"tokens_out":2760,"duration_ms":16545,"temperature":1.0,"reasoning_tokens":2658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:26:57.325585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sum the per-category image counts in Table I and compare the result with the claimed total of over 27,000 images; then inspect a random sample for phone-camera provenance and have a dermatologist verify a sample of the labels. If the sum falls far short of the claim, or if a meaningful share of sampled images fails the provenance or label check, the reported 81 percent F1 cannot be taken at face value. An independent check would be to run Swin-Base on a separately collected mobile-acquired skin image dataset and see whether accuracy and MCC remain near 80 percent.","supporting_citations":[{"cited_title":"G., Lima, G","cited_arxiv_id":null,"evidence_quote":"Supplies the PAD-UFES-20 smartphone-acquired images, a key component of the curated dataset."},{"cited_title":"Monkeypox Skin Images Dataset (MSID)","cited_arxiv_id":null,"evidence_quote":"Provides approximately 500 Monkeypox images included in the curated dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as a source for images of rare skin lesions in the curated dataset."},{"cited_title":"E., Cosa-Linan, A., Santhanam, N., Jannesari, M., Maros, M","cited_arxiv_id":null,"evidence_quote":"Supports the frozen-backbone transfer learning approach and explains why deeper fine-tuning caused overfitting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the dermoscopic ISIC archive that the paper contrasts with its non-dermoscopic dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the dermoscopic HAM10000 benchmark that the paper contrasts with its mobile-acquired dataset."}],"review_version":2}