{"id":"8bae463f-3b7c-46f8-a938-970995c08dea","arxiv_id":"1908.05317","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A new 1,459-image diabetic foot ulcer dataset with visual expert labels enables binary classifiers that reach 90% accuracy for ischaemia and 73% for infection, with ensemble CNNs outperforming handcrafted features.","lead":"This paper introduces a dataset of 1,459 diabetic foot ulcer images labelled by sight for two conditions, ischaemia and infection, and tests machine learning classifiers on them. An ensemble of convolutional neural networks reaches 90% accuracy for ischaemia and 73% for infection, though the labels come from visual inspection alone.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Patch-level evaluation with augmented near-duplicates before splitting likely inflates the headline 90%/73% accuracies; the numbers should be re-estimated with image/patient-level splits.","rationale":"The reader's stated weakest assumption is the visual-only ground truth, which is a real concern and is candidly acknowledged in the paper's own limitations. However, the more immediately decisive issue for the headline numeric claim is the patch-level evaluation protocol: augmented near-duplicates may appear in both training and test folds, and the reported counts are internally inconsistent. This issue would undermine the accuracy estimates even if the labels were clinically confirmed, so it is the single most load-bearing concern. The reader's verdict is already CONDITIONAL, and my analysis reinforces that assessment rather than moving it to a different category; hence UNCHANGED. The proposal for a strict foot-image-level split and non-augmented test set is a concrete, feasible check that would settle whether the headline accuracies survive independent evaluation.","tokens_in":11972,"tokens_out":5736,"duration_ms":60499,"concrete_test":"Re-run the binary classifications with a strict foot-image-level split: assign all original and augmented patches derived from one foot image to exactly one of training, validation, or test, and evaluate only on original (non-augmented) DFU patches in the test set. Then recompute the Ensemble CNN accuracy, MCC, and AUC for ischaemia and infection and compare with Tables 2 and 3. If the accuracy drops by more than about 5 points, or if the ordering against ResNet50 changes, the headline claim is inflated by patch-level leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the evaluation protocol used to produce Tables 2 and 3, not the already-acknowledged visual-only labels. The split in Section 4 is performed at patch level, and the paper does not state that all patches and augmented variants derived from the same foot image are kept in the same fold. Table 1 shows each foot image yields multiple DFU patches, and Figures 3-4 show each patch is then magnified, flipped, rotated, noised, sheared, and translated. If these near-duplicates straddle training and test sets, the model can exploit image-specific shortcuts, so the reported 90% ischaemia and 73% infection accuracies measure recognition of particular ulcers rather than generalizable recognition of the conditions. The patch-count arithmetic strengthens this concern: for ischaemia, Section 4 reports 11,564 training + 1,652 validation + 3,304 test = 16,520 patches, while Table 1 lists only 11,536 total original-plus-augmented ischaemia patches, making it unclear whether test patches are augmented variants of training images. The visual-only ground truth is a second independent issue, but the leakage problem alone is sufficient to make the headline benchmark numbers unreliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a dataset of 1,459 diabetic foot ulcer images with expert visual labels for ischaemia and infection, proposes a Superpixel Color Descriptor (SPCD) for handcrafted feature extraction, and proposes a natural data-augmentation method based on Faster R-CNN localization. The authors benchmark traditional machine learning classifiers and fine-tuned CNNs, including an Ensemble CNN that combines bottleneck features from Inception-V3, ResNet50, and InceptionResNetV2 with an SVM classifier. They report that the Ensemble CNN achieves 90% accuracy for ischaemia classification and 73% for infection classification, outperforming the handcrafted methods.","tokens_in":12223,"tokens_out":4642,"duration_ms":45044,"significance":"If the evaluation is sound, the dataset itself is a valuable contribution: it appears to be the first DFU dataset with labels for ischaemia and infection, and it provides a useful baseline comparison of handcrafted and deep features. The natural data augmentation idea is practical and clearly explained. The paper is also explicitly candid about the limitations of visual-only ground truth, particularly the absence of vascular or blood-test confirmation and the effect of debridement on infection indicators. These strengths are real. However, the headline accuracy figures are currently undermined by the patch-level evaluation protocol and by internal inconsistencies in the reported patch counts, so the quantitative claims cannot be taken at face value until the evaluation is fixed.","major_comments":[{"comment":"The evaluation splits are performed at the patch level, but the paper does not state that all patches and all augmented variants derived from the same source foot image are kept within the same fold. Since the natural augmentation pipeline (Section 3.1) produces magnified, flipped, rotated, noised, sheared, and translated versions of each DFU patch, near-duplicate crops from the same foot can appear in both the training and test sets. This can substantially inflate the reported accuracies (90% ischaemia, 73% infection), because the model can exploit image-specific shortcuts rather than learning generalizable signs of the two conditions. Please re-run the experiments with image-level or patient-level splits, and report the patch-level numbers only as a secondary analysis or as a clearly labeled patch-wise benchmark.","section":"Section 4, Tables 2 and 3; Section 3.1"},{"comment":"The patch-count arithmetic is internally inconsistent. Table 1 lists 1,666 DFU patches and 9,870 augmented patches for ischaemia, giving 11,536 total patches, but Section 4 reports 11,564 training + 1,652 validation + 3,304 test = 16,520 patches. For infection, Table 1 totals 1,666 + 5,892 = 7,558 patches, while Section 4 reports 7,136 + 1,019 + 2,038 = 10,193. Section 4 also refers to 'the 2611 original foot images,' whereas Section 2 describes a dataset of 1,459 images. These discrepancies make the evaluation non-reproducible and prevent a reader from determining the split unit or whether augmented patches leaked into the test set. Please correct the numbers and specify exactly how the split was performed (per image, per patch, or per patient).","section":"Section 4 vs. Table 1 and Section 2"},{"comment":"The ground truth labels are based solely on visual inspection by two clinicians, with disagreement resolved by a senior physician, and no inter-observer agreement measure is reported. The paper itself cites Netten et al. showing low validity of remote DFU assessment by clinicians, and it notes that images were captured after debridement, which removes important infection indicators such as colored exudate. Because the dataset is the paper's central contribution and the accuracy numbers are interpreted as recognition of ischaemia and infection, the absence of a reliability measure is load-bearing. Please report inter-observer agreement (e.g., Cohen's kappa) and temper the abstract and conclusion so that the reported accuracies are clearly framed as predicting the expert visual labels, not a clinical reference standard.","section":"Sections 2, 4.1 (challenge 4), and 5"}],"minor_comments":[{"comment":"The label 'Ist MAG' should be '1st MAG'.","section":"Section 3.1, Figure 3"},{"comment":"There is a missing separator in 'dlab)and'; it should read 'dlab) and'.","section":"Section 3.2, Eq. (2)"},{"comment":"The lines 'return SRed = SRed + 1' and 'return SBlack = SBlack + 1' use 'return' where an assignment is intended; the pseudocode logic should be corrected and clarified.","section":"Algorithm 1"},{"comment":"The text says both '5-fold cross-validation' and a single 70/10/20 split; please clarify how the 5-fold procedure relates to the reported training, validation, and test partitions.","section":"Section 4"},{"comment":"The phrase '785 million patients with diabetes in the US' appears to be a typo; the cited study likely reports 6.7 million ambulatory care cases, not 785 million patients.","section":"Section 1"},{"comment":"The header entries 'Ensemble (CNN)0.903' and similar cells are missing a space between the model name and the value; overall formatting of the tables should be cleaned up.","section":"Tables 2 and 3"},{"comment":"The statement that the dataset 'will be made available upon acceptance of this article' would benefit from a concrete data availability statement with access conditions.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be from a group with prior DFU publications, and the relationship to their earlier DFUNet and localization work should be clarified in the revision. I see no evidence of misconduct; the numeric inconsistencies likely reflect an evolving augmentation pipeline and dataset counts. The dataset contribution is genuinely useful, but the patch-level leakage risk and the arithmetic discrepancies in Section 4 are too severe for the headline numbers to be accepted as stated. The authors should also consider that the abstract's '90% accuracy' claim will be widely quoted, so it must be attached to a clearly described and leakage-free evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nUpfront: the dataset is the real contribution here; the benchmark numbers are not. The new DFU dataset with ischaemia and infection labels—1459 foot images, two expert clinicians, senior adjudication—is genuinely useful and the first of its kind. But the accuracy claims in the abstract (90% ischaemia, 73% infection) are not yet believable, because the evaluation protocol leaks near-duplicate patches across train and test.\n\nWhat the paper does well: it identifies an important clinical gap, delivers a labelled dataset, and is transparent about the limits of visual-only ground truth. The SPCD descriptor is simple but plausible, and the natural-data-augmentation idea—using a detector to localize the ulcer before cropping—is sensible. The authors also list the major limitations themselves, which deserves credit.\n\nThe soft spot is load-bearing. Section 4 describes a patch-level split into 70/10/20, and never says that all patches and augmented variants from the same foot stay in the same fold. Each foot image yields multiple DFU patches, and each patch gets rotations, flips, noise, shear and translation. If any near-duplicates cross the split, the model is memorizing specific ulcers, not learning generalizable features. The paper's own numbers don't reconcile: Table 1 shows 1,666 DFU patches plus 9,870 augmented patches (11,536 total for both conditions), while Section 4 reports 16,520 ischaemia patches in train/validation/test, and claims they come from 2,611 original images when the dataset is 1,459 images. That's an arithmetic inconsistency, not a style issue. Until the authors re-run with image-level or patient-level splits and reconcile the counts, the accuracy figures should be treated as upper bounds.\n\nThe visual-only ground-truth issue is real but minor in context—the authors flag it themselves; it mainly limits what the benchmark can claim about true clinical recognition.\n\nBottom line: this deserves a serious referee. The dataset is valuable, and the evaluation problems are fixable. I would not cite the accuracy numbers as they stand, but I would bring the paper to a reading group as a clear example of data leakage in medical imaging. A revision that re-estimates on foot-level splits and corrects the numbers would be worth taking seriously.","headline":"The dataset is a real contribution, but the evaluation leaks near-duplicates across splits, so the accuracy claims need a re-run before they're believable.","tokens_in":12734,"tokens_out":3911,"would_cite":false,"duration_ms":35024,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new image dataset labels diabetic foot ulcers for ischaemia and infection, and an ensemble of convolutional neural networks is claimed to classify ischaemia at 90% accuracy and infection at 73%.","keywords":["diabetic foot ulcer","ischaemia","infection","deep learning","ensemble CNN","superpixel color descriptor","data augmentation","medical image classification"],"falsifier":"Re-label the same 1,459 images using gold-standard tests — Doppler or ankle-brachial pressure index for ischaemia, and blood tests or wound cultures for infection — and compare the new labels to the clinicians' visual labels; if the two label sets diverge, or if the Ensemble CNN trained on the visual labels performs at chance on the objective labels, the paper's accuracy claims are labelling replication rather than disease recognition.","tokens_in":11804,"feed_emoji":"🦶","tokens_out":9520,"duration_ms":83585,"temperature":0.7,"pith_summary":"This paper introduces the first dataset of diabetic foot ulcer (DFU) images labelled for ischaemia and infection, two conditions that raise the risk of amputation, and asks whether computers can recognise them from ordinary foot photographs. It reports that an ensemble of three convolutional neural networks classifies ischaemia versus non-ischaemia with 90% accuracy and infection versus non-infection with 73% accuracy, outperforming handcrafted feature-based machine learning in both tasks. The authors present these results as a proof of concept for a cost-effective, image-based DFU assessment system that could support remote triage and monitoring in settings where vascular and blood tests are not available. The dataset, with expert visual labels from two clinicians, is offered to the research community as a starting benchmark.","feed_headline":"Ensemble CNN spots foot-ulcer ischaemia with 90% accuracy","feed_subtitle":"First labeled DFU dataset for ischaemia and infection also yields 73% infection accuracy from photos alone.","key_machinery":"The load-bearing object is the new DFU dataset: 1,459 non-standardised foot photographs collected from three cameras and labelled by two consultant physicians (disagreements settled by the senior) for ischaemia (210 present versus 1,249 absent) and infection (628 present versus 831 absent). The method around it has three parts: Natural Data-augmentation, which uses a Faster R-CNN detector to localise the ulcer, crops and magnifies that region, then applies mirroring, rotation, noise, contrast, and shear to multiply training examples; the Superpixel Color Descriptor, which over-segments each patch with SLIC superpixels and counts red and black regions using per-channel thresholds to produce a 10-dimensional feature vector; and the Ensemble CNN, which concatenates the bottleneck features of three pre-trained image classifiers and classifies them with an SVM. The dataset carries the claim's novelty, the ensemble carries its headline numbers.","core_discovery":"The central claim is that ischaemia and bacterial infection in diabetic foot ulcers are detectable from colour photographs by deep learning, and that combining several networks is better than any single approach tested. On the authors' new dataset of 1,459 foot images, the Ensemble CNN — bottleneck features from InceptionV3, ResNet50, and InceptionResNetV2 fed into a support-vector machine — reaches 90.3% accuracy, 0.904 AUC, and an MCC of 0.807 for ischaemia classification, and 72.7% accuracy, 0.731 AUC, and an MCC of 0.454 for infection classification. The ensemble beats every individual CNN and every traditional machine-learning classifier with the handcrafted Superpixel Color Descriptor, which is the paper's proposed new feature. The paper itself notes that infection is harder than ischaemia because the photos were taken after debridement, which removes exudate and other visible infection signs, and because the ground truth was set by visual inspection alone without blood tests or vascular measurements.","pith_inferences":["Implicit in the accuracy gap is that the visual signal for ischaemia (pale or gangrenous tissue, poor reperfusion) is far more legible in ordinary photos than the redness and purulence of infection, which debridement partly removes; a testable extension is to compare post-debridement photos with their pre-debridement counterparts for the same ulcers and see whether the infection gap narrows.","The authors do not report agreement between the two labelling clinicians; measuring Cohen's kappa on a held-out subset would show whether the visual ground truth is stable enough to support the 90%/73% figures, and would give later datasets a labelling protocol to replicate.","Because the same foot images already encode site, area, and depth, the ensemble pipeline could be extended to a single multi-label SINBAD classifier that predicts all five components at once rather than running paired binary tasks.","The Superpixel Color Descriptor throws away colours that are not red or black; adding green/yellow thresholds for purulent exudate, or replacing fixed thresholds with learned colour distributions, is a cheap handcrafted baseline the paper leaves unexplored."],"forward_implications":["The published 90%/73% ensemble baselines become reference points that any later ischaemia or infection recognition method on this dataset will need to beat.","A 90% accuracy for ischaemia suggests automated image screening could plausibly help triage high-risk ulcers in remote or low-resource settings, though not replace clinical tests.","The infection results imply that post-debridement colour photographs alone are a limited medium for infection recognition; progress will likely require pre-debridement imagery or objective blood-based labels.","The localise-then-augment pipeline is not tied to DFU and can be carried over to other medical image tasks where the lesion occupies only a few percent of the frame.","The strong gap between the two tasks argues for collecting standardised images (fixed camera settings and lighting, pre-debridement capture) as a direct route to better sensitivity."],"supporting_citations":[{"why":"Defines the SINBAD classification that gives ischaemia and bacterial infection their clinical meaning.","marker":"[9]"},{"why":"Documents low clinician validity for remote visual DFU assessment, the difficulty this work takes on.","marker":"[10]"},{"why":"Supplies the region-of-interest detection idea behind the paper's Natural Data-augmentation.","marker":"[24]"},{"why":"Provides the Faster R-CNN object detector used to locate DFU regions for cropping.","marker":"[25]"},{"why":"Defines the SLIC superpixel algorithm the Superpixel Color Descriptor is built on.","marker":"[26]"},{"why":"Defines InceptionV3, one of the three CNNs whose bottleneck features form the ensemble.","marker":"[27]"},{"why":"Defines InceptionResNetV2, used as the detector backbone and as a member of the ensemble.","marker":"[28]"},{"why":"Defines ResNet50, the third CNN in the ensemble.","marker":"[29]"}],"fun_headline_variants":["AI ensemble reads foot ulcers: 90% ischaemia accuracy","CNN ensemble flags ischaemia in foot ulcers at 90%","New dataset trains AI to spot ulcer ischaemia and infection","Ensemble CNN bests single nets on foot ulcer ischaemia","AI detects ischaemia in diabetic foot ulcers with 90% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The labels for ischaemia and infection were assigned by two clinicians looking at photographs, with no blood-flow measurements, blood tests, or cultures, so if those visual labels are wrong or noisy the reported accuracies measure how well the model copies the clinicians' visual judgement rather than how well it recognises the true medical conditions.","fun_headline_variants_meta":{"raw":{"variants":["AI ensemble reads foot ulcers: 90% ischaemia accuracy","CNN ensemble flags ischaemia in foot ulcers at 90%","New dataset trains AI to spot ulcer ischaemia and infection","Ensemble CNN bests single nets on foot ulcer ischaemia","AI detects ischaemia in diabetic foot ulcers with 90% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2592,"prompt_tokens":1079,"completion_tokens":1513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":1422}},"tokens_in":695,"tokens_out":1513,"duration_ms":8938,"temperature":1.0,"reasoning_tokens":1422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:16:53.007677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the same 1,459 images using gold-standard tests — Doppler or ankle-brachial pressure index for ischaemia, and blood tests or wound cultures for infection — and compare the new labels to the clinicians' visual labels; if the two label sets diverge, or if the Ensemble CNN trained on the visual labels performs at chance on the objective labels, the paper's accuracy claims are labelling replication rather than disease recognition.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SINBAD classification that gives ischaemia and bacterial infection their clinical meaning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents low clinician validity for remote visual DFU assessment, the difficulty this work takes on."},{"cited_title":"Deep Learning Methods and Applications for Region of Interest Detection in Dermoscopic Images","cited_arxiv_id":"1807.10711","evidence_quote":"Supplies the region-of-interest detection idea behind the paper's Natural Data-augmentation."},{"cited_title":"Achanta, A","cited_arxiv_id":null,"evidence_quote":"Defines the SLIC superpixel algorithm the Superpixel Color Descriptor is built on."},{"cited_title":"Szegedy, V","cited_arxiv_id":null,"evidence_quote":"Defines InceptionV3, one of the three CNNs whose bottleneck features form the ensemble."}],"review_version":1}