REVIEW 4 major objections 6 minor 6 references
Skin Lesion Segmentation and Classification for ISIC 2018 by Combining Deep CNN and Handcrafted Features
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that combining deep CNN features with handcrafted 'hybrid' features improves both skin-lesion segmentation and disease classification, with class-averaged recall of 0.841 and mean overlap of 0.735 on the validation set.
desk verdict A plausible engineering report whose central claim—that hybrid features beat single methods—is never actually tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a two-part fusion. For classification, the 'hybrid feature' vector is formed by concatenating CNN features with 200 handcrafted features computed from the RGB image and the segmented lesion, and this vector is then classified by a multiclass SVM. For segmentation, the fusion is a switching rule: after the U-Net produces a mask on a 224 by 224 image, the lesion area is estimated, and if it is below 4508 pixels the system substitutes the GMM mask, on the heuristic that GMM handles small lesions better and U-Net handles large ones better. The concatenated feature vector and the area threshold are what carry the paper's argument.
What would settle it
Run the same pipeline on an official test set held out from the challenge, or another independent set, and compare the measured class-averaged recall and mean overlap with the reported 0.841 and 0.735; if the independent scores fall substantially lower, the validation-tuned threshold and feature fusion do not generalize.
Extended reading notes
Core claim
The paper claims that a hybrid feature representation, formed by concatenating 200 handcrafted features with features extracted from a trained CNN and classifying with a multiclass SVM, gives stronger discrimination among skin lesion types than either feature type alone. For segmentation, the paper claims that a system switching between a Gaussian mixture model and a deep U-Net according to lesion size improves mask accuracy. On the validation set, the proposed classification system reaches a class-averaged recall of 0.841, and the hybrid segmentation system reaches a mean overlap of 0.735. The authors conclude that hybrid features significantly enhance both segmentation and classification accuracy compared with using a single method.
Load-bearing premise
The load-bearing assumption is that the 4508-pixel switching threshold, chosen on the same 193 validation images used to report final accuracy, will also be right for new dermoscopy images; if that threshold overfits the validation set, both the 0.735 segmentation score and the 0.841 classification score are optimistic.
Editorial extensions
If this is right
- If the hybrid features are as effective as reported, combining CNN features with handcrafted features and an SVM is a viable route to improved multiclass skin-lesion classification without heavy data augmentation or large ensembles.
- The lesion-area switch suggests that segmentation accuracy can be improved by choosing between classical and deep methods by lesion size, rather than committing to a single model for all images.
- Because the handcrafted features are computed from the segmentation mask, any improvement in segmentation carries into classification, making the two tasks coupled in this pipeline.
- The reported validation numbers (0.841 class-averaged recall, 0.735 mean overlap) provide a baseline for future submissions to this challenge, though the paper does not report official test-set scores.
Reading between the lines
- A likely consequence the authors do not state: because both the segmentation switch threshold (4508 pixels) and the reported accuracy are taken from the same 193-image validation set, the 0.735 and 0.841 figures are probably optimistic estimates of performance on new images; an independent test set would likely show lower scores.
- The fixed pixel threshold suggests a testable extension: learn the GMM-versus-U-Net switch as a function of lesion size from data, or replace the hard threshold with a confidence-weighted blend, which could improve robustness to lesion-size distributions not represented in the validation set.
- The same concatenate-and-classify recipe could transfer to other medical image tasks with limited labeled data, where handcrafted morphology features can supplement deep features; this is an extrapolation beyond the paper's dermoscopy setting.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This short report describes the authors' submission to the ISIC 2018 Challenge for Task 1 (lesion segmentation) and Task 3 (disease classification). For Task 1, they combine a 109-layer U-Net with a Gaussian mixture model (GMM) segmenter, switching between the two based on an estimated lesion-area threshold of 4508 pixels (Section 2.2). For Task 3, they concatenate features from two CNNs with 200 handcrafted features and classify with a multiclass SVM (Section 3). The paper reports a mean overlap of 0.735 on the 193-image validation set for Task 1 and a class-averaged recall of 0.841 on the validation set for Task 3, and concludes that hybrid features significantly enhance both tasks compared with single-method alternatives.
Significance. If the hybrid-feature approach were rigorously demonstrated to outperform both deep features alone and handcrafted features alone, it could offer a computationally efficient and practical route to improving dermoscopy analysis. The paper is clearly written at a high level, and the choice to address two challenge tasks in one compact system is sensible. However, the central claim of hybrid-feature superiority is not supported by the evidence presented: no ablation study isolates the contribution of the hybrid combination, no independent test set is used, and the only reported numbers are validation-set scores obtained after tuning a threshold on that same set. These limitations are load-bearing, not cosmetic. The work is a plausible baseline or system description, but as it stands it does not demonstrate the stated conclusion.
major comments (4)
- [Section 2.2 and Section 4.1] The switching threshold of 4508 pixels for choosing between GMM and U-Net is selected using the validation data, and the same validation set is later used to report the final mean overlap of 0.735. This creates a circularity: the reported score is partly a result of fitting the threshold to the evaluation set. An unbiased estimate would require either a separate test set, cross-validation, or a nested validation procedure that does not reuse the reported set for threshold selection.
- [Section 3 and Section 5] The paper's central conclusion that 'Hybrid features significantly enhance the segmentation in Task 1 and classification Task 3 accuracy compared to the use of a single method' is never tested. Section 4 reports only aggregate validation scores (0.735 and 0.841); there is no comparison of the hybrid system against U-Net alone, GMM alone, CNN features alone, handcrafted features alone, or SVM with each feature type separately. Without such an ablation, the superiority claim is an assertion rather than a demonstrated result.
- [Section 4] All performance claims rest on single point estimates with no error bars, confidence intervals, statistical significance tests, or per-class breakdowns. Given that the validation set for Task 1 has only 193 images, the reported 0.735 mean overlap could vary substantially with the choice of validation split; the manuscript should report variance or use repeated cross-validation to establish robustness.
- [Section 3] The description of the classification pipeline is too incomplete for reproducibility: the paper states that 'two convolutional neural networks' were trained but does not specify their architectures, input preprocessing beyond resizing, optimization hyperparameters, training epochs, or the SVM kernel and its hyperparameters. Since the entire Task 3 claim depends on these components, the missing detail prevents an independent check of the reported 0.841 score.
minor comments (6)
- [Abstract] There is a grammatical error in the abstract: 'Our proposed approach is computationally efficient are combines information' should read 'is computationally efficient and combines information.'
- [Section 3] The text says all training examples were resampled to '244 x 224 x 3', while Section 2.1 and Fig. 3 indicate 224 x 224; this appears to be a typo and should be corrected.
- [Section 4.1] The sentence 'The mean overlap score on the testing data is 0.735' refers to the validation set, not test data; the wording should be made consistent with the preceding sentences.
- [References] Reference [2] lacks complete publication information, and reference [3] would benefit from author names and a verified arXiv identifier; the citation format should be made consistent throughout.
- [Figures] Figures 1 and 3 are referenced in the text but their captions are minimal; adding a few sentences explaining the flow in Fig. 3 and the layer details in Fig. 1 would improve readability.
- [Section 4.1] The remark that validation scores 'are not proposed to be made public' seems inconsistent with the fact that the scores are reported in this paper; the authors should clarify their intended data-sharing or publication policy.
Circularity Check
Task 1 segmentation score is a validation-set fit: the 'best' 4508-pixel switch threshold is chosen on the same 193-image validation set later reported as 0.735 mean overlap, so that number is not an independent prediction.
-
fitted input called prediction
[Section 2.2 (threshold selection) and Section 4.1 (Task 1 validation result)]
"We come up with the best threshold that switches to select between a traditional segmentation approach [3] and deep learning method. This switch is based on estimated lesion area. ... If the lesion area of UNet is smaller than 4508 when the image size is 224 x224, we choose the GMMs mask. ... The mean overlap score on the testing data is 0.735."
The 4508-pixel switch threshold is a fitted parameter: it is described as the 'best threshold' for choosing between GMM and UNet masks. The Task 1 score of 0.735 is then reported on the same provided validation dataset (193 samples) used to select that threshold. Thus the reported mean overlap is the result of tuning the method on the evaluation set, not an independent measure of the hybrid system's performance. The 'testing data' label in Section 4.1 is the same validation set, so the 0.735 number is a fit-quality value on the optimization set; no separate test set or nested validation is supplied to break this coupling.
full rationale
The paper is a short system description rather than a derivation, so most of its claims are unsupported rather than circular. The central assertion that hybrid features are superior to single-method features is never tested by an ablation: no CNN-only, handcrafted-only, GMM-only, or UNet-only scores are reported, so that claim is unverified but not definitionally circular. The clearest circular step is in Task 1: the 'best threshold' of 4508 pixels is selected for switching between GMM and UNet, and the same 193-image validation set used for that selection is later used to report the 0.735 mean overlap score. Consequently the Task 1 result is partly forced by fitting the threshold to the evaluation data and is not an unbiased prediction. The Task 3 score of 0.841 is validation-only and not tied in the text to an explicit threshold search, so it does not exhibit the same construction. The citation [3] is from the same group but is used for handcrafted features and is not the load-bearing source of the circularity; this is a normal self-citation rather than an independent uniqueness theorem or ansatz smuggled in by citation. Overall, one reported prediction reduces to its own fitted input, meriting a partial-circularity score of 6.
Assumptions & free parameters
free parameters (1)
- area threshold for GMM/UNet switching =
4508
assumptions (3)
- domain assumption GMM is better for smaller lesions and UNet is better for larger ones.
- domain assumption Handcrafted features computed from the RGB image with respect to the lesion segmentation are informative for classification.
- domain assumption The validation dataset is representative of the unseen test distribution.
Cite this review
Pith. "Pith review of Skin Lesion Segmentation and Classification for ISIC 2018 by Combining Deep CNN and Handcrafted Features." pith.science (2026). https://pith.science/paper/6AN4FFAS
@misc{pith2026190805730,
author = {Pith},
title = {Pith review of: Skin Lesion Segmentation and Classification for ISIC 2018 by Combining Deep CNN and Handcrafted Features},
year = {2026},
howpublished = {\url{https://pith.science/paper/6AN4FFAS}},
note = {Machine review of arXiv:1908.05730}
}
read the original abstract
This short report describes our submission to the ISIC 2018 Challenge in Skin Lesion Analysis Towards Melanoma Detection for Task1 and Task 3. This work has been accomplished by a team of researchers at the University of Dayton Signal and Image Processing Lab. Our proposed approach is computationally efficient are combines information from both deep learning and handcrafted features. For Task3, we form a new type of image features, called hybrid features, which has stronger discrimination ability than single method features. These features are utilized as inputs to a decision-making model that is based on a multiclass Support Vector Machine (SVM) classifier. The proposed technique is evaluated on online validation databases. Our score was 0.841 with SVM classifier on the validation dataset.
Figures
Reference graph
Works this paper leans on
-
[1]
ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection,
The International Skin Imaging Collaboration (ISIC) Website, "ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection," https://challenge2018.isic-archive.com/
work page 2018
-
[2]
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation (2014), arXiv:1411.4038 [cs.CV]
arXiv 2014
-
[3]
3 illustrates our proposed classification approach
Task 3 Methodology Fig. 3 illustrates our proposed classification approach. First, we have trained two convolutional neural networks (CNN) on the available training data for Lesion Diagnosis. The training data consists of 10015 images. All training examples have been resampled to 244 x 224 x 3. We have trained out CNN's without any data augmentations or g...
-
[4]
The validation scores are for our information and are not proposed to be made public
Experimental Results 4.1 Task 1 The results for Task 1 have been obtained using the provided validation dataset. The validation scores are for our information and are not proposed to be made public. We have tested our model on the provided validation data (193 samples in total). The mean overlap score on the testing data is 0.735. 4.2 Task 3 In this Task,...
work page 2018
-
[5]
Conclusion This research proposes a robust system for lesion segmentation and disease from dermoscopy images, which offers the vision of achieving an improved and more accurate classification of lesions from images. Our proposed method is based on the use of hybrid features, which are a combination of handcrafted features and deep learning features. Hybri...
-
[6]
Hardie, Russell C., et al. "Skin Lesion Segmentation and Classification for ISIC 2018 Using Traditional Classifiers with Hand-Crafted Features." arXiv preprint arXiv:1807.07001(2018)
arXiv 2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.