{"id":"74571737-6b05-4b4f-ada5-d43e13d86526","arxiv_id":"1908.08187","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A CNN configuration toolbox is described, and preliminary ISIC experiments show that image resolution and augmentation affect melanoma classification performance while the resizing filter does not.","lead":"The authors describe a spreadsheet-driven software toolbox for configuring convolutional neural networks for skin cancer classification, with preliminary tests on ISIC dermoscopic images. It reports which training choices, resolution, augmentation, and image resizing filter, moved accuracy in melanoma detection and which did not.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Resize-filter conclusion rests on an unquantified single-run comparison: Section 3 reports 'no significant difference' for four filters but gives no numbers, variability, or significance test, so the central 'quantify the impact' claim is not supported on this point.","rationale":"The reader's weakest assumption singles out the absence of repeated trials, error bars, and significance tests, and specifically calls out the resize-filter paragraph. My stress-test lands on the same spot: of the three quantified impact claims in the abstract, the resize-filter claim is the least supported because the actual measurements are omitted entirely. This is not a fatal flaw in the software contribution—the paper is explicitly preliminary and the architecture description is coherent—but it is a genuine evidentiary gap in the central scientific conclusion. The 277x277 vs 227x227 discrepancy adds uncertainty about which experiments were run. Releasing code and data, adding repeated runs with error bars, and reporting the resize-filter numbers would resolve the concern. Since the conditional verdict already requires exactly this kind of strengthening, I do not change the verdict.","tokens_in":7192,"tokens_out":3301,"duration_ms":33609,"concrete_test":"Rerun the three Section 3 conditions (VGG16@227, SC19@227, VGG16@450) on the same preprocessed 12,319-image ISIC corpus, with nearest, bilinear, bicubic, and lanczos filters, using at least five random seeds with identical epochs, batch size, optimizer, and augmentation policy. Report mean and standard deviation for test accuracy, sensitivity, specificity, AUC, and training time per filter. Apply a paired bootstrap or Wilcoxon signed-rank test on the seed-matched AUC values, with effect sizes. If all filter differences are within one standard deviation and not significant, the 'resizing filter can be left to default' conclusion is supported; if any comparison is significant or effect sizes exceed roughly 0.01 AUC, that conclusion must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing component of the abstract is that the experiments 'quantify the impact of image augmentation, image resolution, and rescaling filter.' Augmentation and resolution are backed by numerical outcomes (e.g., SC19 0.803 vs 0.835 AUC; VGG16 0.819 vs 0.862 AUC), but the resizing-filter conclusion in Section 3 is not backed by reported measurements. For three conditions (VGG16 at 227x227, SC19 at 227x227, VGG16 at 450x450) and four filters, the text says 'no significant difference in all the metrics' and recommends nearest, yet no sensitivity, specificity, AUC, or training-time values are given, and no repetition count, standard deviation, or statistical test is reported. If run-to-run stochasticity (weight initialization, data ordering, on-the-fly augmentation sampling) is comparable to or larger than genuine filter differences, a single run per cell will frequently show 'no significant difference' regardless of the truth. There is also a textual inconsistency: earlier experiments are labeled 277x277 while the filter experiment is labeled 227x227, making it unclear whether the same runs are being summarized. The claim that the resizing filter can be left to the default is therefore an absence-of-evidence assertion, not a quantified result. The engineering contribution remains plausible, but this is the weakest link in the headline quantitative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report describes a software toolbox for configuring CNN pipelines for skin cancer classification. The system exposes configuration via a CSV/spreadsheet interface, supports optional UNet-based lesion segmentation, on-the-fly data augmentation (flips, rotations, brightness, saturation, color space), and pluggable Keras classifiers. Preliminary experiments on an ISIC-derived dataset compare a pretrained VGG16 and a custom AlexNet variant (SC19) under different augmentation policies, image resolutions, and resizing filters. The authors report AUC values (e.g., VGG16 0.819 at 277x277 with 48x augmentation versus 0.862 at 450x450), compare favorably with dermatologists on the MClass-D test set, and claim that the resizing filter has no significant impact, recommending nearest-neighbor resampling.","tokens_in":7497,"tokens_out":3400,"duration_ms":31204,"significance":"The toolbox addresses a real need: reproducible, flexible configuration of deep networks for dermatology, and the authors correctly note that prior studies are hard to compare due to different datasets and hyperparameters. The reported architecture—spreadsheet-driven pipeline, CPU-side augmentation, pluggable classifiers, optional segmentation—is sensible and potentially useful to practitioners. The MClass-D comparison with human dermatologists is a strength, and the explicit reporting of sensitivity, specificity, and AUC for several configurations is useful. The quantitative conclusions are preliminary, however, and rest on single training runs without variance or statistical testing; the resize-filter claim is not backed by any reported numbers. This is a systems paper with indicative results, not yet a robust empirical comparison.","major_comments":[{"comment":"The abstract states that the paper quantifies the impact of the rescaling filter, but this paragraph reports 'no significant difference' for four filters under three conditions without giving any accuracy, sensitivity, specificity, AUC, or training-time values, and without repeated runs or a statistical test. Please provide a table of the measured metrics for each filter and condition, or revise the abstract and conclusions to remove the unsupported quantitative claim.","section":"Section 3, 'No impact of image resize filters'"},{"comment":"The resolution is labeled 277x277 in the main experiments but 227x227 in the resize-filter paragraph, and Table 1 lists imgsize 227. This inconsistency makes it unclear whether the resize-filter experiment uses the same resolution as the earlier runs. Please correct the typo and explicitly state the resolutions used in each experiment.","section":"Section 3, 'Experiments at 277x277' vs 'No impact of image resize filters'"},{"comment":"The augmentation, resolution, and architecture comparisons are each based on a single training run without repeated seeds or confidence intervals. For example, the SC19 comparison (0.803 AUC over 7 epochs with hflip rot24 versus 0.835 AUC over 11 epochs with hflip rot4) changes two variables at once. Please add repeated runs or otherwise bound run-to-run variability, and separate the effect of augmentation from the effect of epoch count.","section":"Section 3, all reported comparisons"},{"comment":"The operating point on the MClass-D test set is selected after seeing test results: the threshold is moved from 0.5 to 0.6 to improve specificity. Because this threshold selection is performed on the test set, the reported '0.862 specificity, 0.750 sensitivity' should be labeled as a test-set-tuned operating point, and the default-threshold result should be the primary comparison or justified as a prespecified decision rule.","section":"Section 3, MClass-D evaluation"},{"comment":"The paper does not describe how the 12,319 images are split into training, validation, and test sets for the reported experiments. The 'split' column in Table 1 mentions 'pre' and 'n=100', but the actual splits used for the reported metrics are not specified. Please state the split procedure and sizes, as this is essential for interpreting the results.","section":"Section 3, dataset and split description"}],"minor_comments":[{"comment":"The phrase 'meta leaning frameworks' appears twice; it should be 'meta-learning frameworks'.","section":"Abstract and Section 1"},{"comment":"The sentence 'Our results show no significant difference in the all the metrics' contains a grammatical error; rephrase to 'in all the metrics'.","section":"Section 3, 'No impact of image resize filters'"},{"comment":"'This latest results closely match' should be 'These latest results closely match'.","section":"Section 3"},{"comment":"The color space list is given as 'HVS, LAB, or YCbCr'; the first should be 'HSV' as used in Table 1.","section":"Section 2, Data Augmentation"},{"comment":"The example row 'ISIC-2016 n=100' is not clearly connected to the reported experiments; please clarify what this row represents.","section":"Section 2, Training Input, Table 1"},{"comment":"Figure 1 is not explicitly referenced in the pipeline description in Section 2; please add a reference to the figure.","section":"Section 2, Figure 1"},{"comment":"The text says the software 'will be released as open-source software once out of beta stage'; without a public repository, the experiments are not reproducible from the paper alone. Please provide a code release or state the availability clearly.","section":"Section 1 and Section 4"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads as a work-in-progress technical report. The central systems contribution is plausible, but the abstract's quantitative claims, especially about the resizing filter, outrun the evidence presented. If the authors can supply the missing numbers and variance information, and correct the resolution inconsistency, the paper would be considerably stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest technical report about a CNN configuration toolbox for skin cancer classification. The engineering is plausible, the spreadsheet-driven interface is a real convenience for non-scripting users, and the paper is honest about its preliminary status. The only genuinely new empirical observation—that the choice of resizing filter seems not to matter—is also the weakest part: it is presented as \"no significant difference\" without any numbers, repeats, or statistical test.\n\nWhat the paper does well: the architecture is described clearly (Decorator pattern for augmentation, Factory Method for CNN presets, CSV as the interface), the segmentation module is benchmarked against ISIC 2017 with credible numbers (IoU 0.69 vs 0.76 for the top challenge entry), and the comparison with Esteva and Fujisawa is fair—including the honest admission that transfer learning was necessary because the randomly initialized networks did not converge. The resolution and augmentation comparisons do have reported AUCs (e.g., SC19 improving from 0.803 to 0.835 with a lower augmentation policy; VGG16 moving from 0.819 at 277px to 0.862 at 450px), so those parts of the abstract are backed by numbers.\n\nThe soft spots are real but proportionate. Single training runs with no error bars or repeats mean the reported differences might be noise; this is mitigated somewhat because the effects are described as \"preliminary\" and the paper does not overclaim them. The resize-filter section, however, is the load-bearing example of the \"quantify the impact\" promise, and it delivers no measurements at all. That is absent evidence, not a result. There is also a small textual inconsistency: earlier experiments are called 277x277 while the filter experiment is called 227x227, which makes it unclear whether the same runs are being summarized. The post-hoc threshold change on MClass-D (0.5 to 0.6) is a minor circularity, nudging performance, but the authors present it transparently. Code is not yet released, which limits reproducibility but is not unusual for a beta-stage tool.\n\nAll that said, the central engineering claim—that the toolbox exists and works—holds up. This is the kind of paper that belongs at a workshop or in a systems venue where software descriptions are accepted. It deserves a serious referee, but the referee should insist that the resize-filter conclusion either be removed, downgraded to anecdotal, or backed by actual numbers and a proper error analysis.\n\nFor you: I would not cite this for the quantitative claims, but I would keep it in mind as a software reference once the code is out. Worth a skim if you work in medical imaging tooling; not otherwise.","headline":"A useful preliminary toolbox report whose engineering core is solid, but the headline empirical claim about the resizing filter is supported by no reported numbers, so the 'quantify' framing overreaches.","tokens_in":7998,"tokens_out":1412,"would_cite":false,"duration_ms":16608,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A spreadsheet-configurable CNN toolbox for skin-cancer classification reports that image resolution and augmentation move performance, while the resizing filter does not.","keywords":["skin cancer classification","convolutional neural networks","melanoma detection","data augmentation","image resolution","transfer learning","software toolbox","dermoscopic images"],"falsifier":"Repeat the three resizing-filter comparisons (VGG16 at 227x227, SC19 at 227x227, and VGG16 at 450x450) several times with different random seeds, holding all other settings fixed, and compute a confidence interval for ROC AUC within each filter. If the differences between filters exceed the differences between repeated runs of the same filter, the paper's central claim about filter irrelevance fails.","tokens_in":7001,"feed_emoji":"🔬","tokens_out":10176,"duration_ms":94159,"temperature":0.7,"pith_summary":"This paper presents a software toolbox that lets a user configure a skin-cancer image-classification pipeline from a spreadsheet, then reports what that configuration changes in practice. The authors' central empirical claim is that, for melanoma detection on dermoscopic images, image resolution and the data-augmentation policy are the levers that move detection performance and training time, while the resizing filter (nearest, bilinear, bicubic, or lanczos) makes no significant difference. The practical point is that a non-specialist can systematically explore CNN choices when a new dataset arrives, and can keep the cheapest default resizing filter without hurting results. In their experiments, transfer-learned networks clearly outperformed randomly initialized ones, and raising input resolution from 277x277 to 450x450 pixels raised ROC AUC from 0.819 to 0.862 at a large training-time cost.","feed_headline":"Image-resize filters don't change skin-cancer CNN results","feed_subtitle":"A spreadsheet-driven toolbox finds resolution and augmentation matter more than the resizing filter.","key_machinery":"The load-bearing object is the spreadsheet-driven training pipeline: each row of a CSV file specifies method, dataset, split, optional segmentation, augmentation preset, epochs, batch size, image size, resizing filter, color space, and class weights, and the software turns that row into a concrete network configuration and a training run. The augmentation stage is a chain of modular transforms (horizontal flip, rotation, brightness, saturation) built with the Decorator pattern, and an optional U-shaped convolutional segmentation network masks the lesion before classification. The quantitative conclusions are carried by direct comparisons of configurations that differ in only one controlled variable: resolution, augmentation preset, or resizing filter.","core_discovery":"The paper's central finding is that the configuration levers for skin-cancer CNNs are not equally important. In experiments with a transfer-learned VGG16 and an AlexNet-style network called SC19 on a public dermoscopic dataset of 12,319 images, image resolution and augmentation strength had clear effects: the VGG16 at 450x450 pixels reached 0.862 ROC AUC versus 0.819 at 277x277, at the cost of roughly six times longer training; and halving the SC19 augmentation factor from a 48x preset to an 8x preset improved its results while using one sixth of the compute. The same configurations showed no significant difference across the four tested resizing filters, which the paper takes as evidence that the resizing filter can be left at its nearest-neighbor default. The paper also reports that randomly initialized networks failed to converge or underperformed, leading it to conclude that large-scale natural-image pretraining is necessary at this dataset size.","pith_inferences":["If the resizing-filter result holds more broadly, standardizing the resize step across studies would remove one more source of variance from published skin-cancer CNN comparisons.","The same configuration grid is a natural substrate for automated search: each spreadsheet row is a candidate configuration, so Bayesian optimization or AutoML could treat the toolbox as its action space.","The filter result is demonstrated only for downscaling to 227 or 450 pixels on dermoscopic images; severe downsampling or preprocessing for segmentation could still be filter-sensitive.","Because each configuration was run once, the numeric rankings should be read as directional; a practitioner choosing between two close configurations should repeat each run several times before committing."],"forward_implications":["Practitioners can keep the nearest-neighbor resizing filter as the default and expect no meaningful loss in classification quality, freeing time and memory for other stages.","For a transfer-learned VGG16 on dermoscopic images, increasing input resolution from 277x277 to 450x450 gained about 0.04 ROC AUC while increasing training time by a large factor, so resolution choices should be budgeted explicitly.","Transfer learning from a large natural-image pretraining set appears necessary rather than optional at dataset sizes around twelve thousand images, since randomly initialized networks did not converge.","Augmentation strength should be tuned per architecture: a smaller 8x augmentation preset beat a 48x preset for the SC19 network while using one sixth of the compute.","The spreadsheet interface lets dermatology practitioners explore these trade-offs on a new dataset without writing code, which addresses the reproducibility gap among published skin-cancer CNN studies."],"supporting_citations":[{"why":"Systematic review establishing that published CNN skin-cancer studies are hard to compare; motivates the spreadsheet-driven configuration grid.","marker":"[4]"},{"why":"Shows that strong augmentation and high-resolution images can reach high performance with fewer than 5,000 images; supplies the 48x augmentation preset and the small-data baseline.","marker":"[11]"},{"why":"Large-dataset dermatologist-level classifier used as the state-of-the-art reference point for the paper's ROC AUC results.","marker":"[10]"},{"why":"Defines the U-shaped convolutional architecture used for the optional lesion-segmentation step.","marker":"[16]"},{"why":"Provides the pretrained weights used by the transfer-learned VGG16; the paper argues pretraining is necessary at this data size.","marker":"[9]"},{"why":"Defines VGG16, the main architecture in the resolution, augmentation, and resizing-filter comparisons.","marker":"[17]"},{"why":"Defines the architecture that the custom SC19 network is adapted from.","marker":"[14]"},{"why":"Supplies the 100-image benchmark set on which the 450-pixel model is compared with dermatologist performance.","marker":"[3]"},{"why":"Reports the head-to-head human performance on the same benchmark, giving the comparison values the paper cites.","marker":"[2]"}],"fun_headline_variants":["Resizing filter no impact on skin-cancer CNN accuracy","Skin-cancer CNN tuning: resolution beats resizing filter","Augmentation and resolution trump resize filter in skin-CNNs","Pixel size matters, resizing filter doesn't in skin-cancer CNNs","Resolution and augmentation drive skin-cancer CNN results, not resize filter"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported performance differences reflect genuine effects of the configuration changes rather than run-to-run noise, because each configuration was trained once and the 'no significant difference' statement about resizing filters is not backed by repeated trials or a statistical test.","fun_headline_variants_meta":{"raw":{"variants":["Resizing filter no impact on skin-cancer CNN accuracy","Skin-cancer CNN tuning: resolution beats resizing filter","Augmentation and resolution trump resize filter in skin-CNNs","Pixel size matters, resizing filter doesn't in skin-cancer CNNs","Resolution and augmentation drive skin-cancer CNN results, not resize filter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2515,"prompt_tokens":844,"completion_tokens":1671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":1587}},"tokens_in":460,"tokens_out":1671,"duration_ms":11661,"temperature":1.0,"reasoning_tokens":1587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:54:42.665997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the three resizing-filter comparisons (VGG16 at 227x227, SC19 at 227x227, and VGG16 at 450x450) several times with different random seeds, holding all other settings fixed, and compute a confidence interval for ROC AUC within each filter. If the differences between filters exceed the differences between repeated runs of the same filter, the paper's central claim about filter irrelevance fails.","supporting_citations":[{"cited_title":"Journal of Medical In- ternet Research 20(10), e11936 (Oct 2018)","cited_arxiv_id":null,"evidence_quote":"Systematic review establishing that published CNN skin-cancer studies are hard to compare; motivates the spreadsheet-driven configuration grid."},{"cited_title":"British Journal of Dermatology (Sep 2018)","cited_arxiv_id":null,"evidence_quote":"Shows that strong augmentation and high-resolution images can reach high performance with fewer than 5,000 images; supplies the 48x augmentation preset and the small-data baseline."},{"cited_title":"In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q","cited_arxiv_id":null,"evidence_quote":"Defines the architecture that the custom SC19 network is adapted from."}],"review_version":1}