Pith. sign in

REVIEW 2 major objections 6 minor 11 references

Do Diabetic Foot Ulcer Segmentation Models Generalize? A Cross-Dataset Benchmark of CNN and Transformer Architectures

T0 review · 2 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Architecture family, not model complexity, drives how well diabetic foot ulcer segmentation models transfer across hospitals.

desk verdict Solid leakage-screened DFU cross-dataset benchmark with a real ranking; the “family not complexity” slogan overreaches the unmatched training recipe. read the letter →

arxiv 2607.02555 v1 pith:5WAPOUOM submitted 2026-06-27 cs.CV eess.IV

Abderrahmane Benfatah This is my paper · ORCID
classification cs.CVeess.IV
keywords DiabeticfootulcerSemanticsegmentationCross-datasetgeneralizationVisionTransformerSegFormerFailureanalysisCNNDicecoefficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep learning models that outline diabetic foot ulcers from photographs look strong when tested on the same hospital data they were trained on, but real clinics need models that work on images from other sites, cameras, and patients. This paper trains three representative models under one leakage-screened protocol on combined FUSeg/AZH data and then evaluates them, without any fine-tuning, on two independent external sets. All three models score well in-domain (Dice roughly 0.80–0.83) yet drop sharply across datasets; the Transformer SegFormer-B2 degrades least on both external sources and produces fewer catastrophic failures than the convolutional baselines, while the heavier DeepLabV3+ generalizes worse than the simpler U-Net. Per-image statistics and Wilcoxon tests confirm the ranking is consistent, not an artifact of a few lucky images. A sympathetic reader cares because high published accuracy does not equal deployment readiness, and the result points to architecture family as a practical lever for cross-hospital robustness.

What carries the argument

The leakage-screened cross-dataset protocol (train on FUSeg/AZH after image-level overlap audit; zero-shot test on DFUC2022 and Medetec) plus per-image Dice failure analysis and paired Wilcoxon tests on 2,160 external images. This isolates architecture-family effects from data leakage and from mean-only summary scores.

What would settle it

Train the same three architectures (or add a second Transformer) with fully matched optimizers, learning rates, and input resolution—including SegFormer at its designed 512×512—and re-evaluate zero-shot on the same two external sets; if the family ranking reverses or disappears, the claim that architecture family (not recipe or capacity) drives generalization is falsified.

Watch

Extended reading notes

Core claim

Under an identical, leakage-screened protocol—training on combined FUSeg/AZH wound data and evaluating without fine-tuning on DFUC2022 and Medetec—SegFormer-B2 generalizes best on both external sets (Dice 0.557 and 0.786) and yields the fewest catastrophic failures on DFUC2022 (31.1% versus 38.5% and 43.0%), while the more complex DeepLabV3+ underperforms the simpler U-Net. The consistent ranking across two independent sources, confirmed by Wilcoxon signed-rank tests (p < 0.001), indicates that architecture family, not model complexity, drives cross-hospital generalization for DFU segmentation.

Load-bearing premise

The three models, trained with family-specific optimizers and learning rates at a fixed 256×256 resolution and without a learning-rate scheduler, are treated as a fair test of architecture family rather than of training recipe or resolution mismatch.

Editorial extensions

If this is right

  • High in-domain Dice above 0.80 does not imply clinical deployment readiness; models can lose roughly a third of that performance at a new hospital.
  • Choosing a Transformer-style architecture over simply adding convolutional complexity is a more reliable route to cross-site DFU segmentation robustness under the paper’s protocol.
  • Validation of wound segmentation tools should report at least two independent external test sets and per-image failure rates, not only in-domain scores.
  • Even the best model still loses substantial accuracy across sources, so domain adaptation and test-time adaptation remain necessary next steps the paper itself flags.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If global self-attention is the real driver, the same family-over-complexity pattern may appear in other medical segmentation tasks that suffer strong site-specific appearance shift (skin lesions, endoscopy, wound photography more broadly).
  • Because failures concentrate on small, occluded, or poorly lit wounds, hybrid systems that keep a Transformer backbone but add explicit size- or lighting-aware post-processing may close more of the remaining gap than architecture swap alone.
  • A controlled ablation that matches optimizer, learning rate, and native resolution across families would be the cleanest follow-up experiment to separate self-attention from training-recipe effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper benchmarks three DFU segmentation architectures—U-Net and DeepLabV3+ (ResNet-34 CNNs) and SegFormer-B2 (Transformer)—under a single leakage-screened protocol: train on combined FUSeg/AZH, evaluate without fine-tuning on DFUC2022 and Medetec. All models reach strong in-domain Dice (0.80–0.83) but degrade on external data; SegFormer-B2 ranks first on both external sets (DFUC2022 Dice 0.557, Medetec 0.786), with fewer catastrophic failures on DFUC2022 (31.1% vs 38.5%/43.0%) and Wilcoxon signed-rank significance (p < 0.001). The authors conclude that architecture family, not model complexity, drives cross-hospital generalization.

Significance. Cross-dataset robustness is a genuine clinical bottleneck for DFU segmentation, and prior work largely stayed in-domain or omitted modern Transformers under leakage control. The dual external test sets, image-level overlap audit, multi-seed DFUC2022 means, per-image failure histograms, and paired Wilcoxon tests are concrete methodological strengths that make the ranking under this protocol credible and reproducible (code released). If the family-over-complexity claim holds after better-matched controls, the result would usefully guide architecture choice for multi-site wound monitoring; even as a carefully documented negative result on deployment readiness, it is a useful contribution to the medical imaging community.

major comments (2)
  1. §3.3–3.4 and Table 1: The central claim that “architecture family, not model complexity, drives” generalization is not isolated by the protocol. CNNs use Adam at 1e-3 while SegFormer uses AdamW at 6e-5; no LR scheduler is used; all models are forced to 256×256 despite SegFormer being designed for 512×512 (§3.6 notes this makes its Dice “conservative”). Only one Transformer is tested against two CNNs of different capacity. DeepLabV3+ underperforming U-Net is offered as evidence against complexity, yet without optimizer-matched, capacity-matched, or resolution-matched ablations the ranking can equally be explained by training recipe or resolution mismatch rather than self-attention. The Wilcoxon tests confirm the ranking under this protocol; they do not establish the causal attribution to family. A matched-recipe ablation (or at least a clear re-framing of the claim as “under standard fami
  2. §3.6 and §4.5: Representativeness of the model set is overstated for a family-level conclusion. A single Transformer (SegFormer-B2) versus two ResNet-34 CNNs cannot support a general “Transformer vs CNN” claim; other Transformers (e.g., Swin-UNet) or foundation models (MedSAM, already cited) may behave differently. The discussion attributes robustness to global self-attention without mechanistic evidence (attention maps, receptive-field controls). Either expand the model set or narrow the claim to “SegFormer-B2 under this protocol” rather than architecture family as such.
minor comments (6)
  1. Table 1 header: “Gap = in-domain Dice − DFUC2022 Dice” is clear, but Medetec gap is not reported; adding it would make the dual-external comparison symmetric.
  2. §3.1 / §3.2: State the exact train/val split sizes after treating FUSeg/AZH as one source, and report the number of unique patients if available, to clarify independence from DFUC2022.
  3. Table 2 / Fig. 2: The Dice < 0.4 failure threshold is reasonable but ad hoc; a short sensitivity note (e.g., at 0.3 and 0.5) would strengthen the failure analysis.
  4. §3.4: Loss is “combined Dice and binary cross-entropy” without weights; specify the combination (sum, equal weight, etc.) for full reproducibility.
  5. Fig. 1 caption: Colour-coding of Dice is helpful; ensure the figure itself is high-resolution enough for the three-row qualitative comparison to be readable in print.
  6. References [10] and [11] are recent SegFormer DFU papers; a one-sentence contrast of their in-domain vs your cross-dataset setting in Related Work would sharpen novelty.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical cross-dataset benchmark with no circular derivation: external Dice/IoU and Wilcoxon tests are measured, not redefined from inputs.

full rationale

This paper is a controlled empirical benchmark, not a first-principles derivation. Models are trained on FUSeg/AZH and evaluated without fine-tuning on independent external sets (DFUC2022, Medetec) after leakage screening; reported Dice/IoU, failure rates (Dice<0.4), and Wilcoxon signed-rank tests are direct measurements on held-out images. No quantity is fitted then re-presented as a prediction of a closely related quantity; no uniqueness theorem or load-bearing premise is imported via self-citation; and architecture-family conclusions are comparative rankings under a stated protocol, not identities by construction. Concerns about unmatched optimizers/LR, fixed 256×256 resolution, or testing only one Transformer affect causal attribution and experimental fairness (correctness risk), not circularity. The evaluation chain is self-contained against external public datasets.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical systems benchmark. Its central ranking rests on standard segmentation metrics, public datasets treated as independent after hash-based overlap removal, and hand-chosen training hyperparameters that differ by architecture family. No new physical entities are introduced. The load-bearing modeling choices are the failure threshold, the train/test source partition after leakage audit, and the claim that the three implementations represent CNN vs Transformer families fairly.

free parameters (6)
  • CNN learning rate (Adam)
    Set to 1e-3 by protocol choice; not swept against the Transformer recipe, so it co-determines the CNN side of the ranking.
  • SegFormer learning rate (AdamW)
    Set to 6e-5 'following common practice'; different optimizer and LR from CNNs confounds pure architecture comparison.
  • Training epochs
    Fixed at 30 with best-validation-Dice checkpoint; may under- or over-train different families unequally.
  • Input resolution
    All models forced to 256×256 for GPU limits; SegFormer originally designed for 512×512, so reported Transformer advantage may be conservative or distorted.
  • Catastrophic failure threshold (Dice < 0.4)
    Hand-chosen cutoff used for failure counts in Table 2; mean Dice ranking is primary, but failure-rate claims depend on this threshold.
  • Loss weighting (Dice + BCE)
    Combined loss used for all models without reported ablation of relative weights.
assumptions (5)
  • domain assumption After MD5 and dHash screening, FUSeg/AZH may be treated as one training source with no meaningful image-level overlap with DFUC2022 or Medetec.
    §3.2; the external-test claim depends on this independence. Visual verification of flagged pairs is stated but not fully enumerated in the paper.
  • domain assumption Dice and IoU on binary wound masks are adequate primary measures of clinical segmentation quality for cross-site comparison.
    Standard in the field; used throughout §3.4 and Tables 1–2 without clinical area-error calibration.
  • ad hoc to paper U-Net and DeepLabV3+ with ResNet-34, versus SegFormer-B2, are representative enough of CNN vs Transformer families to support a family-level conclusion.
    §3.3 and §3.6; only one Transformer and two CNNs are tested, so the family claim extrapolates from a small sample.
  • standard math Wilcoxon signed-rank tests on per-image Dice (one-sided, α=0.05) validly establish ranking significance across models.
    §3.4 and Table 3; standard non-parametric paired test under exchangeability of paired image scores.
  • domain assumption Medetec masks labeled {0,1} can be thresholded at >0 to match conventional binary masks.
    §3.2 note; necessary for fair Medetec evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Diabetic Foot Ulcer Segmentation Models Generalize? A Cross-Dataset Benchmark of CNN and Transformer Architectures." pith.science (2026). https://pith.science/paper/5WAPOUOM

@misc{pith2026260702555,
  author       = {Pith},
  title        = {Pith review of: Do Diabetic Foot Ulcer Segmentation Models Generalize? A Cross-Dataset Benchmark of CNN and Transformer Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5WAPOUOM}},
  note         = {Machine review of arXiv:2607.02555}
}
read the original abstract

Deep learning models for diabetic foot ulcer (DFU) segmentation routinely report high accuracy, but they are almost always trained and tested on the same dataset, leaving their behaviour on data from a different clinical source largely unmeasured. We benchmark three representative segmentation architectures -- U-Net and DeepLabV3+ (convolutional) and SegFormer-B2 (Transformer) -- under an identical, leakage-screened protocol: training on the combined FUSeg/AZH wound data and evaluating, without fine-tuning, on two independent external datasets (DFUC2022 and Medetec). All models achieve strong in-domain performance (Dice 0.80--0.83) but degrade substantially across datasets. The degradation is, however, architecture-dependent: SegFormer-B2 generalizes best on both external sets (DFUC2022 Dice 0.557, Medetec Dice 0.786), outperforming both convolutional models, while the more complex DeepLabV3+ generalizes worse than the simpler U-Net. Per-image failure analysis on 2,160 images across both external test sets confirms that SegFormer-B2 produces the fewest catastrophic failures on DFUC2022 (31.1%), compared with U-Net (38.5%) and DeepLabV3+ (43.0%). The consistent ranking across two independent external sources, confirmed by Wilcoxon signed-rank tests (p < 0.001 on both datasets), indicates that architecture family, not model complexity, drives cross-hospital generalization.

Figures

Figures reproduced from arXiv: 2607.02555 by the authors.

Figure 1
Figure 1. Qualitative segmentation results on DFUC2022 for three representative cases: well-segmented (top), partial (middle), and complete failure (bottom). For each case: input image, ground-truth mask, and predicted mask are shown per model. Dice scores are colour-coded: green (≥ 0.7), orange (0.4–0.7), red (< 0.4). Transformer advantage observed in mean Dice reflects a consistent per-image improvement rather than a small … view at source ↗
Figure 2
Figure 2. Per-image Dice distributions on DFUC2022. SegFormer-B2 shows a rightward shift and reduced left tail relative to both CNN models, corresponding to fewer images below the failure threshold of 0.4 (orange dashed line) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 2 canonical work pages

  1. [1]

    Etiology, Epidemiology, and Disparities in the Burden of Diabetic Foot Ulcers

    Armstrong, D.G., Tan, T.-W., Boulton, A.J.M., Bus, S.A.: Diabetic Foot Ulcers: A Review. Etiology, Epidemiology, and Disparities in the Burden of Diabetic Foot Ulcers. Diabetes Care46(1), 209–221 (2023)

    2023

  2. [2]

    IEEE Access12, 173824–173835 (2024)

    Lucho, S., Naemi, R., Casta˜ neda, B., Treuillet, S.: Can Deep Learning Wound Seg- mentation Algorithms Developed for a Dataset Be Effective for Another Dataset? A Specific Focus on Diabetic Foot Ulcers. IEEE Access12, 173824–173835 (2024)

    2024

  3. [3]

    In: MICCAI, pp

    Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomed- ical Image Segmentation. In: MICCAI, pp. 234–241 (2015) Cross-Dataset DFU Segmentation Benchmark 9

    2015

  4. [4]

    In: ECCV, pp

    Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In: ECCV, pp. 801–818 (2018)

    2018

  5. [5]

    In: NeurIPS, vol

    Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: SegFormer: Sim- ple and Efficient Design for Semantic Segmentation with Transformers. In: NeurIPS, vol. 34, pp. 12077–12090 (2021)

    2021

  6. [6]

    Biomedical Signal Pro- cessing and Control (2024)

    Dhar, M.K., Zhang, T., Patel, Y., Gopalakrishnan, S., Yu, Z.: FUSegNet: A Deep Convolutional Neural Network for Foot Ulcer Segmentation. Biomedical Signal Pro- cessing and Control (2024). arXiv:2305.02961

    arXiv 2024

  7. [7]

    Information15(3), 140 (2024)

    Wang, C., Mahbod, A., Ellinger, I., Galdran, A., Gopalakrishnan, S., Niezgoda, J., Yu, Z.: FUSeg: The Foot Ulcer Segmentation Challenge. Information15(3), 140 (2024)

    2024

  8. [8]

    Nature Communications15, 654 (2024)

    Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment Anything in Medical Images. Nature Communications15, 654 (2024)

    2024

Show all 11 references
  1. [9]

    GitHub repository (2019).https: //github.com/qubvel/segmentation_models.pytorch

    Iakubovskii, P.: Segmentation Models PyTorch. GitHub repository (2019).https: //github.com/qubvel/segmentation_models.pytorch

    2019

  2. [10]

    In: 2025 IEEE 13th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applica- tions (IDAACS), pp

    Kairys, A., Raudonis, V.: Effective SegFormer Ensemble for Diabetic Foot Ul- cer Segmentation. In: 2025 IEEE 13th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applica- tions (IDAACS), pp. 674–678 (2025).https://doi.org...

    2025 doi

  3. [11]

    IEEE Signal Processing Letters 32, 4194–4198 (2025).https://doi.org/10.1109/LSP.2025.3617370

    Kumar, M., Kaur, T., Gandhi, T.K.: UlcerMTL: Multi-Task Learning for Classi- fication and Segmentation of Diabetic Foot Ulcers. IEEE Signal Processing Letters 32, 4194–4198 (2025).https://doi.org/10.1109/LSP.2025.3617370

    2025 doi

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.