Pith. sign in

REVIEW 5 major objections 5 minor 24 references

SegDT: A Diffusion Transformer-Based Segmentation Model for Medical Imaging

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SegDT claims state-of-the-art skin-lesion segmentation with a compact diffusion transformer using rectified flow and 15 inference steps.

desk verdict A useful small-model segmentation pipeline undercut by a false SOTA claim and a missing VAE sanity check. read the letter →

arxiv 2507.15595 v1 pith:YCX5UWX7 submitted 2025-07-21 cs.CV

Salah Eddine Bekhouche , Gaby Maroun , Fadi Dornaika , Abdenour Hadid This is my paper
classification cs.CV
keywords skinlesionsegmentationdiffusiontransformerrectifiedflowlatentmedicalimageISICdatasetefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SegDT is a small transformer-based diffusion model for skin-lesion segmentation that replaces noise prediction with a learned velocity field in latent space, so that masks can be generated in 15 integration steps instead of dozens. The paper claims this design reaches state-of-the-art Dice scores, IoU, pixel accuracy, and specificity on the ISIC 2016, 2017, and 2018 benchmarks while needing only 9.95M parameters and 3.68 GFLOPs, about a tenth of the compute of the strongest U-Net-style competitor it is compared with. If correct, the result matters because diffusion segmenters have been accurate but too slow and large for clinical use; a compact version that runs on low-cost GPUs would make diffusion-based segmentation a realistic real-time option for dermatology. The architecture chains a pretrained 8x image autoencoder, twelve DiT blocks with cross-attention on the image code, and a rectified-flow integrator, ending in a thresholded binarization that turns the decoded image into a mask.

What carries the argument

The load-bearing piece is the rectified-flow velocity field. In the latent space of a pretrained Tiny AutoEncoder for Stable Diffusion (TAESD), the model learns the field $v_\theta(z_t, t, y)$ that points from a noisy mask latent $z_t$ toward the clean latent, and follows it with Euler updates $z_{t-1} = z_t + v_\theta(z_t, t, y)\,\Delta t$ over a $[0,1]$ time interval; learning a velocity instead of the noise is what makes the trajectory nearly straight and 15 steps sufficient. The transformer itself is a DiT-XS with patch size 2 and 12 blocks, each combining adaptive layer normalization modulation, self-attention on image patches, and cross-attention that injects the conditioning image latent $y$; a shared patch embedder processes the mask latent and the image code. Around this core, the pretrained TAESD autoencoder compresses input images and masks 8x per spatial dimension to $4 \times 32 \times 32$ patches, the final linear layer outputs $2C'$ channels of which only the first half is decoded, and a fixed binarization threshold of 0.2 converts the decoded image into the final segmentation.

What would settle it

Take the ISIC ground-truth masks, push them through the TAESD encoder and straight back through the decoder, binarize at the paper's threshold of 0.2, and measure Dice against the originals. If this no-diffusion round trip already loses several points, the reported 91-95% Dice cannot come from the diffusion process as described. A second check is to run the full 15-step pipeline over many random seeds; the paper fixes one seed, so its numbers describe a single noise draw rather than the model's expected performance.

Watch

Extended reading notes

Core claim

The central claim is that a diffusion transformer can reach the highest reported segmentation scores on the three ISIC benchmarks if its denoising is formulated as rectified flow in a pretrained latent space. On ISIC 2016 the model reports the best Dice (94.76%), IoU (91.40%), and accuracy (97.08%); on ISIC 2018 the best Dice (94.51%) and IoU (90.43%); and on ISIC 2017 the paper reports the best Dice (91.70%) and accuracy (95.49%), with the highest specificity on every dataset. The authors attribute this to predicting a velocity field $v_\theta(z,t,y)$ on the latent code of the mask, conditioned by cross-attention on the latent code of the input image, instead of predicting noise at each step, which straightens the reverse trajectory and makes 15 Euler steps match a 35-step baseline. Efficiency is the supporting claim: 9.95M parameters and 3.68 GFLOPs versus 39M parameters and 54 GFLOPs for the main U-Net competitor, with trained weights released publicly.

Load-bearing premise

The load-bearing premise is that the pretrained TAESD autoencoder, trained for Stable Diffusion on natural images, preserves binary mask detail through its 8x encoding and decoding, so that a denoised latent really corresponds to an accurate mask. The paper does not quantify reconstruction error for masks in this latent space, and if boundaries blur under the 8x compression the reported Dice values could not be reproduced.

Editorial extensions

If this is right

  • Diffusion-based segmentation becomes affordable on clinical hardware: 9.95M parameters and 3.68 GFLOPs, roughly a tenth of the compute of the DU-Net+ baseline, while reporting higher Dice on ISIC 2016 and 2018.
  • Rectified flow in latent space cuts inference to 15 Euler steps, about 2 times fewer than the 35-step diffusion baseline, without the paper's claimed quality loss.
  • The model reports the highest specificity on all three datasets (99.44%, 98.74%, 97.43%), which the paper argues is the clinically important direction because it suppresses false positives on healthy tissue.
  • Because the DiT and the VAE are task-agnostic, the same pipeline can be retrained for other segmentation targets; the paper names organ and task generalization as the next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would test the TAESD round trip on masks before trusting the numbers: if encoding and immediate decoding already drop several Dice points, part of the reported score lives or dies in the autoencoder, not the diffusion model.
  • The fixed-seed noise injection means the headline scores are a single realization; reporting variance over seeds would show whether 15-step flow denoising is reliably stable or merely a lucky draw.
  • The binarization threshold (0.2) is tuned on a validation set, so some accuracy may sit in post-processing; sweeping the threshold from 0.1 to 0.5 on the test set would separate generator quality from threshold luck.
  • The table lists GU-Net with a higher ISIC 2017 Dice than SegDT's reported 91.70%, so 'state-of-the-art' appears to depend on which comparators are included; a head-to-head re-evaluation with identical preprocessing would settle which claim survives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript introduces SegDT, a compact diffusion transformer (DiT) with a pretrained TAESD variational autoencoder and rectified-flow sampling, for skin lesion segmentation. The method is evaluated on ISIC 2016, ISIC 2017, and ISIC 2018, reporting Dice, IoU, accuracy, sensitivity, specificity, FLOPs, and parameter counts, and the paper claims state-of-the-art results with 15-step inference. The central claims are that SegDT outperforms prior methods on these benchmarks and is efficient enough for low-cost GPUs.

Significance. Efficient medical image segmentation on resource-constrained hardware is practically relevant, and the combination of a small DiT with rectified flow and a pretrained VAE is a plausible engineering direction. The paper has concrete strengths: the code is publicly released, the evaluation metrics are explicitly defined, the FLOPs/parameter comparisons provide useful context, and the qualitative discussion acknowledges failure cases. However, the significance of the reported contribution is contingent on resolving the internal contradiction in the state-of-the-art claim and on validating the VAE-based latent-space pipeline, neither of which is currently supported in the manuscript.

major comments (5)
  1. [Table 1 / §4.4] The claim that SegDT achieves the highest Dice on ISIC 2017 is directly contradicted by the paper's own Table 1: GU-Net reports Dice 93.94 and IoU 88.98, while SegDT reports Dice 91.70 and IoU 84.70. Section 4.4 nevertheless states that 'In ISIC 2017, SegDT again achieved competitive results, with the highest Dice score (91.70%) and accuracy (95.49%)', and the abstract and conclusion assert state-of-the-art results on the three datasets. This internal contradiction undermines the central experimental claim and must be resolved by correcting the text and either adding the missing comparison or re-framing the contribution as competitive rather than state-of-the-art.
  2. [§3.1–3.2] The training loss is never defined. Section 3.1 refers to 'a loss function' and states that mean squared error is a common metric, but no equation specifies the velocity-matching objective, the distribution over timesteps t, the noise schedule, or the handling of the predicted variance channels described in Section 3.2. Without a precise training objective, the method is not reproducible and the reported results cannot be independently assessed.
  3. [§3 / §3.2] The paper assumes that the pretrained TAESD VAE can encode binary segmentation masks into a 4-channel latent space at H/8 × W/8 and decode denoised latents back into binarizable images, but no reconstruction-fidelity measurement is reported for masks. Because the VAE decoder is not trained or fine-tuned and the training loss is computed entirely in latent space, any decoder distortion is uncorrected. If the autoencoder's own mask reconstruction Dice is below the reported 91–95% range, the described pipeline could not produce the stated results. A quantitative reconstruction check on the ISIC masks is required.
  4. [§3.2 / Fig. 1] The conditioning path is described ambiguously. The Patch Embedder paragraph says that during inference, when ground-truth masks are not available, the model 'processes a randomized tensor for the conditional input', which conflates the input latent and the conditioning latent. It is unclear whether the medical image is encoded by the VAE encoder to produce y, whether y and z are combined by cross-attention or concatenation, and how the fixed-seed random tensor relates to the conditioning image. The data flow in Fig. 1 should be clarified with a precise specification of the input, condition, and output tensors.
  5. [§4.4] The efficiency claim 'SegDT achieves segmentation quality comparable to IDDPM with only 15 inference steps, while IDDPM requires 35 steps' is not supported by any quantitative result in the paper. No table or figure reports Dice or other metrics as a function of inference steps, and no runtime measurements are given. Since fast inference is a central claimed advantage, this comparison needs direct experimental evidence.
minor comments (5)
  1. [§1] The Introduction says 'compared to exiting methods'; this should be 'existing methods'.
  2. [§2.1] The Related Work states that DeepLabV3+ 'incorporates convolutions to capture multi-scale information'; the intended term is likely 'atrous convolutions', which should be stated precisely.
  3. [§4.3] The phrase 'no explicit data enhancement techniques were used' should read 'data augmentation techniques'.
  4. [Fig. 1] The note that the module outputs 2C' but only the first half is passed to the decoder during inference is important but appears only in the figure; it should be explained in the main text where the DiT output is described.
  5. [Table 1] Several entries in Table 1 are missing and marked with hyphens, but the caption does not state that these values were not reported by the respective methods; this should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SegDT is an empirical system paper whose benchmark results are measured outputs, not quantities derived from fitted inputs or self-citations.

full rationale

SegDT is an empirical systems paper: the proposed model is a composition of a pretrained TAESD VAE encoder/decoder, a DiT-XS transformer, and a rectified-flow velocity-field objective. The reported Dice, IoU, ACC, SE, and SP numbers are measured outputs on held-out ISIC test sets, not quantities derived by construction from the model's inputs. The only tuned hyperparameter that touches the reported metrics is the binarization threshold (0.2), selected on a held-out validation set from a range of 0.1 to 0.5; this is standard practice and does not amount to predicting a fitted quantity. The unsupported comparison to IDDPM (35 vs. 15 steps) and the unmeasured reconstruction fidelity of TAESD for binary masks are important correctness and reproducibility risks, but they are not circular steps: no equation in the paper reduces to an input, and no load-bearing claim is justified solely by a self-citation. The limitations noted by the authors (small or irregular lesions) further confirm that the results are empirical rather than forced by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

SegDT is an empirical construction built on pretrained components and standard flow-matching ideas. The central claim depends on assumptions that the VAE latent space is adequate for medical masks, that the conditioning path works as intended, and that the cross-paper comparison is fair. None of these is validated with ablations or controlled experiments in the paper.

free parameters (2)
  • Binarization threshold = 0.2
    Selected on a validation set over the range 0.1 to 0.5 in steps of 0.05 to maximize Dice, Section 4.3. All reported metrics depend on this choice.
  • Inference sampling steps = 15 (mentioned in Section 4.4)
    The paper states 15 steps are used for the IDDPM comparison, but does not specify the step count for the main results; it is a hand-chosen hyperparameter that affects inference speed and quality.
assumptions (4)
  • standard math Euler integration of the learned velocity field (Eq. 1) is a valid approximation of the reverse diffusion trajectory.
    Used without error analysis in Section 3.1. This is a standard numerical integration assumption.
  • domain assumption TAESD's VAE, trained on natural images, preserves the information needed to encode and reconstruct medical segmentation masks after 8x compression.
    Central to the pipeline (Section 3, Section 3.2); no reconstruction error analysis is provided.
  • domain assumption Conditioning on the image latent via cross-attention is correctly implemented and sufficient for semantic segmentation.
    The architecture (Section 3.2, Fig. 2) relies on this; the text ambiguity about the conditional input during inference is never resolved.
  • domain assumption The baseline metrics cited from previous papers were obtained under comparable preprocessing and evaluation protocols.
    No baseline is re-run; differences in resizing, test-time augmentation, and post-processing may bias the SOTA comparison in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SegDT: A Diffusion Transformer-Based Segmentation Model for Medical Imaging." pith.science (2026). https://pith.science/paper/YCX5UWX7

@misc{pith2026250715595,
  author       = {Pith},
  title        = {Pith review of: SegDT: A Diffusion Transformer-Based Segmentation Model for Medical Imaging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YCX5UWX7}},
  note         = {Machine review of arXiv:2507.15595}
}
read the original abstract

Medical image segmentation is crucial for many healthcare tasks, including disease diagnosis and treatment planning. One key area is the segmentation of skin lesions, which is vital for diagnosing skin cancer and monitoring patients. In this context, this paper introduces SegDT, a new segmentation model based on diffusion transformer (DiT). SegDT is designed to work on low-cost hardware and incorporates Rectified Flow, which improves the generation quality at reduced inference steps and maintains the flexibility of standard diffusion models. Our method is evaluated on three benchmarking datasets and compared against several existing works, achieving state-of-the-art results while maintaining fast inference speeds. This makes the proposed model appealing for real-world medical applications. This work advances the performance and capabilities of deep learning models in medical image analysis, enabling faster, more accurate diagnostic tools for healthcare professionals. The code is made publicly available at \href{https://github.com/Bekhouche/SegDT}{GitHub}.

Figures

Figures reproduced from arXiv: 2507.15595 by the authors.

Figure 1
Figure 1. Overview of the SegDT inference architecture for medical image segmen [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Detailed architecture of a single DiT block. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Segmentation results of SegDT on the ISIC datasets. The figure shows [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages

  1. [1]

    In: European conference on computer vision

    Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin- unet: Unet-like pure transformer for medical image segmentation. In: European conference on computer vision. pp. 205–218. Springer (2022)

    2022

  2. [2]

    arXiv preprint arXiv:2102.04306 (2021)

    Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.:Transunet:Transformersmakestrongencodersformedicalimagesegmentation. arXiv preprint arXiv:2102.04306 (2021)

    arXiv 2021

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelli- gence 40(4), 834–848 (2018)

    Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Se- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelli- gence 40(4), 834–848 (2018)

    work page 2018

  4. [4]

    Heliyon10(18) (2024)

    Cheng, D., Gai, J., Yang, B., Mao, Y., Gao, X., Zhang, B., Jing, W., Deng, J., Zhao, F., Mao, N.: Gu-net: Causal relationship-based generative medical image segmentation model. Heliyon10(18) (2024)

    work page 2024

  5. [5]

    arXiv preprint arXiv:1902.03368 (2019)

    Codella, N., Rotemberg, V., Tschandl, P., Celebi, M.E., Dusza, S., Gutman, D., Helba,B.,Kalloo,A.,Liopyris,K., Marchetti,M.,et al.:Skinlesionanalysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 (2019)

    arXiv 2019

  6. [6]

    In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018)

    Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomed- ical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: 2018 IEEE 15th intern...

    2018

  7. [7]

    Bekhouche et al

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is 12 SE. Bekhouche et al. worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    arXiv 2020

  8. [8]

    Biomedical Signal Processing and Control 98, 106674 (2024)

    Feng, Y., Su, J., Zheng, J., Zheng, Y., Zhang, X.: A parallelly contextual convolu- tional transformer for medical image segmentation. Biomedical Signal Processing and Control 98, 106674 (2024)

    work page 2024

Show all 24 references
  1. [9]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Gorade, V., Mittal, S., Jha, D., Bagci, U.: Synergynet: Bridging the gap between discrete and continuous representations for precise medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 7768–7777 (2024)

    2024

  2. [10]

    arXiv preprint arXiv:1605.01397 (2016)

    Gutman, D., Codella, N.C., Celebi, E., Helba, B., Marchetti, M., Mishra, N., Halpern, A.: Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the inter- national skin imaging collaboration (isi...

    2016 arXiv

  3. [11]

    IEEE Access7, 21455– 21467 (2019)

    Jha, D., Riegler, M.A., Johansen, D., Halvorsen, P., Johansen, H.D.: Resunet++: An advanced architecture for medical image segmentation. IEEE Access7, 21455– 21467 (2019)

    2019

  4. [12]

    Signal, Image and Video Processing 19(1), 152 (2025)

    Kaur, R., Ranade, S.K.: Du-net+: a fully convolutional neural network architecture for semantic segmentation of skin lesions. Signal, Image and Video Processing 19(1), 152 (2025)

    2025

  5. [13]

    arXiv preprint arXiv:2401.00722 (2024)

    Lan, L., Cai, P., Jiang, L., Liu, X., Li, Y., Zhang, Y.: Brau-net++: U-shaped hybrid cnn-transformer network for medical image segmentation. arXiv preprint arXiv:2401.00722 (2024)

    2024 arXiv

  6. [14]

    IEEE Transactions on Instru- mentation and Measurement71, 1–15 (2022)

    Lin, A., Chen, B., Xu, J., Zhang, Z., Lu, G., Zhang, D.: Ds-transunet: Dual swin transformer u-net for medical image segmentation. IEEE Transactions on Instru- mentation and Measurement71, 1–15 (2022)

    2022

  7. [15]

    arXiv preprint arXiv:2209.03003 (2022)

    Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 (2022)

    2022 arXiv

  8. [16]

    In: Proceedings of the IEEE/CVF Inter

    Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF Inter. Conf. on Computer Vision. pp. 4195–4205 (2023)

    2023

  9. [17]

    arXiv preprint arXiv:2409.03062 (2024)

    Perera, S., Erzurumlu, Y., Gulati, D., Yilmaz, A.: Mobileunetr: A lightweight end- to-end hybrid vision transformer for efficient medical image segmentation. arXiv preprint arXiv:2409.03062 (2024)

    2024 arXiv

  10. [18]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). pp. 234–241. Springer (2015)

    2015

  11. [19]

    arXiv preprint arXiv:2010.02502 (2020)

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)

    2020 arXiv

  12. [20]

    Advances in neural information pro- cessing systems 30 (2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems 30 (2017)

    2017

  13. [21]

    In: Medical Imaging with Deep Learning

    Wu, J., Fu, R., Fang, H., Zhang, Y., Yang, Y., Xiong, H., Liu, H., Xu, Y.: Med- segdiff: Medical image segmentation with diffusion probabilistic model. In: Medical Imaging with Deep Learning. pp. 1623–1639. PMLR (2024)

    2024

  14. [22]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Wu, J., Ji, W., Fu, H., Xu, M., Jin, Y., Xu, Y.: Medsegdiff-v2: Diffusion-based med- ical image segmentation with transformer. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 6030–6038 (2024)

    2024

  15. [23]

    Biomedical Signal Processing and Control101, 107242 (2025)

    Yang, B., Zhang, R., Peng, H., Guo, C., Luo, X., Wang, J., Long, X.: Slp-net: An efficient lightweight network for segmentation of skin lesions. Biomedical Signal Processing and Control101, 107242 (2025)

    2025

  16. [24]

    IEEE Sensors Journal (2025)

    Yang, Z., Chen, R., Lin, C.: Am-net: A network with attention and multi-scale feature fusion for skin lesion segmentation. IEEE Sensors Journal (2025)

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.