{"id":"b960efc6-0919-4031-91fd-15afdc81e7de","arxiv_id":"2501.08962","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning dermoscopic models on a small clinical subset recovers most clinical performance, but clinical-to-clinical transfer remains poor.","lead":"This paper compares AI skin-cancer models trained on dermoscopic images versus clinical smartphone images and tests whether fine-tuning on a small clinical sample helps. It finds that cross-domain performance is poor, fine-tuning recovers much of the gap, and even two clinical datasets do not transfer well to each other.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's central F1 comparison is not auditable because the PAD test split and DDI class mapping are never specified; the 30%-vs-50% fine-tuning claim and the DDI F1=11.80 results rest on an undefined evaluation protocol.","rationale":"The paper's broad qualitative message, that dermoscopic-trained models transfer poorly to clinical images, that fine-tuning helps, and that clinical-to-clinical transfer is hard, is consistent with prior work and has face validity. However, the strongest and most specific claim is numerical: FineClinic achieves F1=67.18 with 30% of PAD data, nearly matching FClinic's 68.95 with 50%, and all models fail on DDI with F1 around 11.80. Every one of these numbers depends on evaluation details that are absent, in particular the PAD train/test partition and the PAD-to-DDI label mapping. Without those details, the comparison could reflect split artifacts or label mismatches rather than genuine domain effects. The reader's weakest-assumption analysis identified exactly this gap, and I agree with it as the most load-bearing concern. I am not arguing that the authors are wrong; I am arguing that the central evidence as written is insufficient to establish the quantitative conclusion. A simple reproducibility check, releasing splits and code or independently recomputing with fixed seeds and explicit class mappings, would settle whether the concern lands. Therefore the appropriate verdict remains CONDITIONAL: accept only after the protocol is specified and the numbers are verified.","tokens_in":7327,"tokens_out":2702,"duration_ms":29500,"concrete_test":"Reproduce Table 3 with a fully specified protocol: fix a stratified 50/50 PAD split with a documented random seed, hold out the same PAD test partition for FClinic and FineClinic, and define the DDI class mapping by binarizing PAD lesion types into malignant versus benign according to the DDI paper's labels (or state explicitly if DDI images are filtered). Run each configuration over five random seeds and report mean plus/minus standard deviation of F1. If the FClinic versus FineClinic PAD F1 difference shrinks to within the seed variance, or if the DDI F1 changes by more than a few points under a reasonable mapping, then the paper's headline comparison is not supported as stated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in Section 4.2 (Table 3) is that FineClinic, trained on only 30% of PAD-UFES-20 clinical images, reaches PAD F1=67.18, comparable to FClinic's 68.95 with 50% of the data, while FDerm scores 5.22 on PAD and all models score about 11.80 on DDI. These numbers are not auditable because the evaluation protocol is underspecified. Footnote 4 states that PAD has no official split and that 50% of the data was used to train, but it never defines the held-out test partition, whether FClinic and FineClinic are tested on identical PAD subsets, or how class imbalance is handled when forming those subsets. Since FineClinic uses 30% and FClinic uses 50% of the same data, a difference in the test split or in stratification could easily explain the 1.77 F1-point gap. Likewise, DDI has a binary malignant/benign label scheme that does not match PAD's lesion-type classes, yet Table 3 reports DDI F1 without any stated class-mapping or filtering rule. If the DDI evaluation maps labels inconsistently, the 11.80 F1 values lose their meaning. The paper also reports no error bars or multiple-seed variance, and the unusual training description (ADAM with learning rate 1e-1 and a cosine annealing scheduler that decreases every 10 epochs) raises further concern about whether the reported run is representative. The qualitative conclusion that dermoscopic models fail on clinical images and fine-tuning helps is plausible, but the precise quantitative claims cannot be verified from the text alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the domain gap between dermoscopic and clinical skin-lesion images for CNN classification. Using the public ISIC18, PAD-UFES-20, and DDI datasets, the authors train ImageNet-initialized models on dermoscopic data (FDerm), on clinical data (FClinic), and fine-tune dermoscopic-trained models on 30% of the clinical data (FineClinic). The central quantitative claims are that FDerm achieves only 5.22 F1 on PAD, FineClinic reaches 67.18 F1 on PAD with 30% of the training images, which is comparable to FClinic's 68.95 with 50% of the images, and that PAD-trained models transfer poorly to DDI, with F1 around 11.80. The paper also reports class-level accuracy and discusses dataset biases, data imbalance, and the limitations of clinical deployment.","tokens_in":7676,"tokens_out":3816,"duration_ms":40111,"significance":"If the quantitative results were fully auditable, the paper would provide useful evidence on a clinically relevant question: how much dermoscopic pretraining helps in low-resource clinical settings, and how far clinical models generalize across clinical datasets. The qualitative finding that dermoscopic-trained models fail on clinical images and that fine-tuning on a small clinical subset substantially improves performance is plausible and consistent with prior work. Strengths of the manuscript include the use of public datasets, the comparison of several architectures, and the explicit attention to class imbalance and per-class performance. However, the precise numerical claims in Table 3 are currently under-supported because the evaluation protocol is underspecified, no variance estimates are reported, and the class-label mapping between datasets is not defined. These issues prevent the paper, in its present form, from supporting the specific quantitative conclusions it advertises.","major_comments":[{"comment":"The PAD-UFES-20 train/test protocol is never fully specified. Footnote 4 states that PAD has no official split and that 50% of the data was used for training, but it does not define the held-out test partition, state whether the test partition is the same for FDerm, FClinic, and FineClinic, describe how stratification is performed, or specify whether a validation subset is carved out separately. Since FineClinic uses 30% and FClinic uses 50% of the same data, any difference in the test split or in class stratification could explain the 1.77 F1-point gap reported in Table 3. This is load-bearing for the paper's central claim that fine-tuning on fewer clinical images reaches performance comparable to training on more clinical images.","section":"§3.3, footnote 4, and Table 3"},{"comment":"The DDI evaluation is not auditable because the class-label mapping is never described. PAD-UFES-20 and DDI use different label schemes, and the paper reports DDI F1 values without stating how lesion-type classes are mapped to DDI labels, whether some classes are filtered out, or how the metric is averaged. If the mapping is inconsistent, the DDI F1 values in Table 3, including the repeated 11.80 values, lose their meaning. The clinical-to-clinical generalization claim in §4.3 therefore rests on an unspecified evaluation protocol.","section":"§4.2, Table 3, and §4.3"},{"comment":"All results appear to come from single training runs, with no error bars, no multiple seeds, and no measure of run-to-run variance. The training setup also raises concerns: a learning rate of 1e-1 with a cosine annealing scheduler that decreases every 10 epochs is unusual for fine-tuning large CNN models, and no justification or learning-rate sensitivity analysis is provided. Without variance estimates, the central comparison between FClinic's 68.95 and FineClinic's 67.18 cannot be distinguished from noise, especially given the small clinical training sets.","section":"§3.3 and Table 3"},{"comment":"The paper promises to show how to 'combine' data from divergent distributions, but the experiments do not actually train on a combined dataset. The FineClinic setup is sequential: pretraining on ISIC18 followed by fine-tuning on PAD-UFES-20. The conclusion in §5 also refers to assessing 'the combination of clinical and dermoscopic data,' which is a different claim from the fine-tuning protocol actually used. This mismatch between the stated contribution and the experiments should be resolved, either by adding a true combined-training experiment or by reframing the contribution as an analysis of fine-tuning under domain shift.","section":"Abstract, §5, and Table 3"}],"minor_comments":[{"comment":"The column headers are confusing: the table appears to show both 'Test' and 'Val' columns, but the text does not explain how the validation set differs from the test set or why some architectures report a separate validation accuracy value.","section":"Table 1"},{"comment":"There are typographical errors in the table and text: 'FClinc' should be 'FClinic', and the class labels in Table 2 include 'Melanome' and 'Akinc. Keratosis', which should be corrected to 'Melanoma' and 'Actinic Keratosis'.","section":"Table 3"},{"comment":"The text refers to FClinic as being trained on 'complete clinic images' and 'full clinic' data, but footnote 4 says only 50% of PAD-UFES-20 was used for training. This terminology should be clarified so readers do not mistakenly believe the full dataset was used for training.","section":"Footnote 4 and §4.2"},{"comment":"No code, trained model weights, or exact dataset preprocessing steps are provided. Given that the PAD split and DDI class mapping are central to the results, releasing these artifacts or at least a detailed protocol would substantially improve reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant and timely problem, and the qualitative direction is credible. However, the quantitative claims in Table 3 cannot be verified from the text, which is a serious issue for a paper whose main contribution is empirical. I would recommend major revision rather than rejection because the underlying experiments are feasible to fix: the authors need to specify and justify the PAD split, the DDI label mapping, and the metric-averaging choices, and to provide variance estimates. If these details cannot be supplied, the paper would not be suitable for publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper confirms a known result with a wider architecture sweep, and adds a PAD-to-DDI transfer evaluation. The qualitative conclusions are plausible and consistent with prior work, but the headline numbers in Table 3 are not auditable as written.\n\nWhat's actually new: the multi-architecture comparison (ConvNext, DenseNet, ResNet, EfficientNet) across three training regimes, and the clinical-to-clinical transfer test. That's a legitimate extension of Daneshjou et al., and the consistency across architectures is the most useful part. The finding that a dermoscopic-trained model gets F1 5.22 on PAD and that fine-tuning on 30% of clinical data recovers most of the gap is a nice confirmation.\n\nSoft spots, in order of severity. First, the PAD evaluation protocol is underspecified. Footnote 4 says half the data was used for training, but never defines the test split, whether FClinic and FineClinic are evaluated on identical partitions, or how class imbalance is handled. Since FineClinic uses 30% and FClinic 50%, the test sets are likely different, so the 67.18 vs 68.95 comparison is not meaningful without a fixed held-out set. Second, DDI has a binary malignant/benign scheme that doesn't match PAD's lesion classes, yet Table 3 reports DDI F1 with no class-mapping or filtering rule. The 11.80 values are uninterpretable without that. Third, no error bars or multiple seeds, and the training description (Adam at 1e-1, cosine annealing stepping every 10 epochs) is at best misdescribed. Fourth, the abstract promises an analysis of data variation and a method to combine distributions, but no feature analysis appears in the results.\n\nThese are load-bearing for the precise numbers, but not necessarily for the qualitative direction. The paper's own caution about clinical-to-clinical transfer is well placed. Citation pattern is fine; Daneshjou is cited and the extension is positioned honestly.\n\nWho benefits: practitioners who want a quick confirmation that fine-tuning on a small clinical set helps, and researchers working on domain shift in dermatology. It's not a reliable benchmark until the protocol is clarified.\n\nRecommendation: send to peer review with major revisions. The extension is real and the topic matters. Ask for release of splits, code, class mappings, and multiple seeds. If the authors can't provide the split, the quantitative comparison should be dropped.","headline":"Useful confirmation of a known fine-tuning result, but Table 3's key comparison relies on an undocumented PAD split and DDI class mapping.","tokens_in":8296,"tokens_out":3312,"would_cite":false,"duration_ms":30686,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on 30% of clinical data closes most of the dermoscopy-to-clinic gap.","keywords":["skin lesion classification","dermoscopic images","clinical images","transfer learning","fine-tuning","domain shift","dataset bias","class imbalance"],"falsifier":"Reproduce FDerm, FClinic and FineClinic on a single public split of PAD-UFES-20 with one explicit class mapping shared by ISIC18, PAD and DDI, and test all models on the same held-out partition; if FineClinic's clinical F1 no longer lands near FClinic's, or if the PAD-trained model's F1 on DDI departs from roughly 11.80, the central efficiency claim fails.","tokens_in":7118,"feed_emoji":"🩺","tokens_out":6357,"duration_ms":58308,"temperature":0.7,"pith_summary":"Skin-lesion classifiers are usually trained on dermoscopic images, but real users photograph lesions with smartphones. The paper asks whether a model that has never seen clinical photos can be adapted to them with very little clinical data, and answers yes. After training on the dermoscopic ISIC18 data, fine-tuning on only 30% of the clinical PAD-UFES-20 images lifts the clinical F1-score from 5.22 to 67.18, essentially matching the 68.95 F1 of a model trained on 50% of the clinical data. The same experiments show that trained-on-clinical models do not automatically transfer to a different clinical dataset: PAD-trained models score only F1=11.80 on the DDI dataset. The paper reads this as evidence that domain differences, not just data quantity, dominate performance and that small fine-tuning sets can be an efficient remedy.","feed_headline":"30% fine-tune matches full clinical training in skin-lesion AI","feed_subtitle":"F1 climbs from 5.22 to 67.18 with a tiny clinical set, near the 68.95 full-data score.","key_machinery":"The mechanism is transfer learning via fine-tuning: models are first pretrained on ImageNet, then trained on the dermoscopic ISIC18 dataset to produce FDerm, and then a subset is re-trained on clinical PAD-UFES-20 images to produce FineClinic. The comparison table across FDerm, FClinic and FineClinic evaluated on ISIC18, PAD and DDI is the load-bearing instrument, because it isolates the effect of adding a small amount of clinical data to a dermoscopic base. Data augmentation (random flips, rotations, colour jitter, crops and affine transforms) is applied during fine-tuning to counter class imbalance and image noise, which the paper treats as essential for the minority malignant classes.","core_discovery":"On the paper's own terms, the central discovery is an efficiency and a limit. FineClinic—a model initialized on dermoscopic ISIC18 and then fine-tuned on a 30% subset of PAD-UFES-20—reaches an F1-score of 67.18 on clinical PAD data, almost identical to the 68.95 achieved by FClinic, a model trained directly on 50% of the same clinical data, while a purely dermoscopic FDerm model collapses to 5.22 on clinical images. The same experiment also finds that FClinic generalizes poorly to a second clinical dataset, DDI, with F1=11.80, indicating that two 'clinical' collections can be as far apart as dermoscopy and clinic are. These results are averaged across CNN architectures (ConvNext, DenseNet, ResNet, EfficientNet and InceptionV3), and the paper argues they show that combining a large dermoscopic source with a small clinical fine-tuning set is a more data-efficient route to clinical classification than training on clinical data alone, while cautioning that no single clinical dataset yet guarantees transfer to another.","pith_inferences":["A testable extension the paper leaves implicit: sweep the fine-tuning fraction (e.g., 10%, 20%, 30%, 40%, 50%) and plot clinical F1 against it to locate the knee of the curve; if the 30%-equals-50% finding holds, the knee is likely below 30%, which would make data collection even cheaper.","The F1=11.80 cross-clinical result suggests 'clinical' is not one domain; capture conditions such as resolution, distance, illumination and reflectance may dominate over skin-tone diversity. A controlled experiment that varies only the capture device while holding skin-tone distribution fixed would separate these confounds.","The paper's demographic-bias framing and acquisition-shift framing are entangled in its data; one could test the paper's implicit claim that dermoscopic models fail on clinical images largely due to domain shift by fine-tuning on a skin-tone-stratified 30% subset versus a random 30% subset and comparing fairness metrics.","If replicated, the result implies that benchmark design should report both the dermoscopic-to-clinical transfer gain and clinical-to-clinical transfer loss, since either number alone can mislead about real-world readiness."],"forward_implications":["If the claim is right, a deployment recipe for smartphone-based skin-lesion screening is: take a dermoscopic-trained model and fine-tune it on a few hundred carefully chosen clinical images, rather than collecting a large clinical dataset from scratch.","Dermoscopic-only models should not be used directly in clinical workflows: the F1 of 5.22 on PAD shows they effectively fail on uncontrolled phone photos.","Evaluation across multiple clinical datasets is necessary, because a model trained on one clinical set (PAD) can score only 11.80 F1 on another (DDI), so single-dataset clinical numbers are not trustworthy.","Class-imbalance-aware augmentation during fine-tuning matters: without it, the model would simply predict the majority 'nevus' class, hiding poor malignant-class accuracy behind acceptable averages."],"supporting_citations":[{"why":"Supplies the DDI clinical dataset and the prior finding that dermoscopic-trained models underperform on diverse clinical images, which the paper extends.","marker":"[3]"},{"why":"PAD-UFES-20 clinical dataset used for FClinic training and FineClinic fine-tuning; source of the headline 30%-versus-50% comparison.","marker":"[12]"},{"why":"HAM10000, part of the ISIC18 dermoscopic training data for FDerm models.","marker":"[18]"},{"why":"BCN20000, the other component of the ISIC18 dermoscopic training data.","marker":"[1]"},{"why":"ImageNet pretrained weights used to initialize all architectures before dermoscopic or clinical training.","marker":"[4]"},{"why":"ConvNext architecture, the model family that yields the best results in the paper's evaluations.","marker":"[9]"}],"fun_headline_variants":["Fine-tune on 30% clinical matches full training in skin-lesion AI","Small clinical fine-tune beats pure dermoscopy in skin lesion AI","30% clinical data rivals full dataset in skin-lesion classification","Clinical fine-tune equals full training; dermoscopy alone fails","Dermoscopy-then-fine-tune efficient for clinical skin lesion AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on the unstated assumption that the PAD and DDI evaluation splits and the class-label mappings across ISIC18, PAD and DDI are identical across all models; the paper never specifies the held-out test set or the mapping, so if those differ the headline 30%-versus-50% and DDI numbers are not comparable.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tune on 30% clinical matches full training in skin-lesion AI","Small clinical fine-tune beats pure dermoscopy in skin lesion AI","30% clinical data rivals full dataset in skin-lesion classification","Clinical fine-tune equals full training; dermoscopy alone fails","Dermoscopy-then-fine-tune efficient for clinical skin lesion AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1482,"prompt_tokens":1012,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":374}},"tokens_in":628,"tokens_out":470,"duration_ms":4989,"temperature":1.0,"reasoning_tokens":374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:13:28.612640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce FDerm, FClinic and FineClinic on a single public split of PAD-UFES-20 with one explicit class mapping shared by ISIC18, PAD and DDI, and test all models on the same held-out partition; if FineClinic's clinical F1 no longer lands near FClinic's, or if the PAD-trained model's F1 on DDI departs from roughly 11.80, the central efficiency claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DDI clinical dataset and the prior finding that dermoscopic-trained models underperform on diverse clinical images, which the paper extends."},{"cited_title":"Data Brief32(106221), 106221 (Oct 2020)","cited_arxiv_id":null,"evidence_quote":"PAD-UFES-20 clinical dataset used for FClinic training and FineClinic fine-tuning; source of the headline 30%-versus-50% comparison."}],"review_version":1}