Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

MRGen: Segmentation Data Engine for Underrepresented MRI Modalities

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MRGen claims that a diffusion generator conditioned on text and masks can synthesize MRI for unannotated modalities, lifting segmentation DSC from 6.90 to 66.18 and winning 8 of 10 cross-modality settings.

desk verdict Good dataset and plausible method, but the paired-experiment leakage inflates the headline gains; clean cross-dataset results still support the core idea. read the letter →

arxiv 2412.04106 v3 pith:OERKIYCC submitted 2024-12-04 cs.CV cs.AI

Haoning Wu , Ziheng Zhao , Ya Zhang , Yanfeng Wang , Weidi Xie This is my paper · ORCID
classification cs.CVcs.AI
keywords MRGensyntheticmedicalimagesMRIsegmentationdiffusionmodelsmask-conditionedgenerationtext-guidedunderrepresentedmodalitiescross-modality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that one diffusion-based generator can supply training data for segmentation models on MRI modalities that have no manual masks at all. MRGen is trained in two stages: first on roughly 250,000 unlabeled image-text slices to learn what each MRI modality looks like, then on a smaller set of mask-annotated slices so it can obey an organ mask while generating. Given a templated text prompt and a mask, it synthesizes a realistic image of the requested target modality, and training standard segmentation frameworks on those synthetic images lifts target-domain Dice scores sharply, for example CHAOS T1 to T2-SPIR climbs from 6.90 with source-only training to 66.18 with synthetic data. A sympathetic reader would care because rare but clinically important MRI protocols are exactly those where hand annotation is too costly and scarce to collect.

What carries the argument

The load-bearing object is the two-stage latent diffusion engine $I=\Phi_{\mathrm{MRGen}}(T,M;\Theta,\Theta_c)$, with templated text prompts $T$ and organ masks $M$ as conditions. A VAE maps slices into a latent space; a UNet denoiser is first trained with cross-attention to a biomedical text encoder so the model associates modalities, attributes, regions, and organs with images, and then a mask encoder (initialized from the diffusion UNet and downsampled) is added as a residual into the decoder blocks while the rest stays frozen. An off-the-shelf SAM2-based autofilter checks the mask-faithfulness of each candidate by comparing IoU and confidence scores before synthetic samples enter segmentation training. This arrangement carries the argument because modality knowledge comes from the unlabeled corpus while controllability comes from the frozen diffusion weights plus a small mask-annotated subset, so the two sources of information never compete for the same data.

What would settle it

Take a target MRI protocol that a modality audit shows is absent from the unlabeled pretraining corpus, generate synthetic training data with MRGen, and measure downstream Dice score: if gains over source-only training stay near zero, the claim that MRGen generalizes to genuinely unseen modalities is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that mask-conditioned controllability can be transferred to MRI modalities that never had masks, as long as their appearance was seen during text-guided pretraining. Concretely, MRGen is a latent diffusion model that takes a text prompt (modality name, tissue signal attributes, body region, and organ list) plus a segmentation mask, and returns a 2D MR slice of the requested modality. The two-stage schedule, text pretraining on image-text pairs followed by mask-conditioned finetuning with the diffusion backbone frozen, is what lets the mask controller generalize beyond annotated modalities. In ten cross-modality settings, synthetic MRGen data improves both nnUNet and UMamba, giving the best Dice score in eight experiments; the strongest case is CHAOS T1 to T2-SPIR, where nnUNet's Dice rises from 6.90 to 66.18.

Load-bearing premise

The generator can only synthesize a target modality whose appearance is present somewhere in the unlabeled Internet radiology corpus used for pretraining, and the paper does not verify that its test modalities (T2-SPIR, ADC, T1, T2) actually appear there.

Editorial extensions

If this is right

  • Segmentation models can be trained for target MRI modalities without a single manually annotated target-domain mask, turning a previously zero-shot scenario into a usable one.
  • The benefit transfers across segmentation architectures: both nnUNet and UMamba improve, and MRGen ranks first in 8 of 10 cross-modality experiments.
  • Adding unlabeled target-domain images during training and using SAM2-based autofiltering both further raise Dice scores, so the pipeline degrades gracefully instead of depending on one trick.
  • Synthetic in-domain data can nearly match real annotated data for segmentation, suggesting the engine could also serve privacy-preserving data sharing.
  • Preliminary cross-protocol experiments from CT to MRI indicate the same engine may extend beyond MRI to other underrepresented imaging settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If a target protocol is absent from the unlabeled pretraining corpus, the method probably degrades to source-only performance, so auditing corpus modality coverage should come before deploying MRGen for a new protocol.
  • The proposal effectively converts annotation scarcity into a corpus-coverage problem: the engine's ceiling is set by the breadth of text-image pairs it can pretrain on, not by the number of masks.
  • One could test whether SAM2-based filtering is replaceable by any zero-shot segmenter, or whether the filter itself limits the method to organs SAM2 can recognize.
  • A direct extension is 3D volume synthesis or non-MRI protocols such as CT, where the same text-and-mask conditioning could be retrained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MRGen, a two-stage diffusion-based data engine that generates MRI images conditioned on text prompts and segmentation masks, together with MRGen-DB, a curated dataset of roughly 250,000 2D slices with modality, region, and organ text annotations and about 18,000 mask-annotated slices. The method is evaluated by using MRGen to synthesize target-modality training images from source-domain masks and then training nnUNet and UMamba on the resulting synthetic pairs across five cross-modality dataset pairs. The authors report that MRGen achieves the best DSC in 8 of 10 settings, with the headline example CHAOS T1 to T2-SPIR nnUNet rising from 6.90 to 66.18.

Significance. If the empirical claims survive a corrected evaluation, the paper makes a useful contribution: it introduces a large-scale radiology image-text dataset, a controllable latent diffusion engine for mask-conditioned MRI synthesis, and an automatic filtering pipeline based on SAM2. The release of code, models, and data would enable direct reproducibility and follow-up work. However, the current evaluation contains a train/test leakage in exactly the settings that drive the headline numbers, so the magnitude and ranking of the reported improvements are not yet established.

major comments (4)
  1. [Appendix B.3, Table 7] The synthetic-data statistics state that masks from the entire source-domain dataset (including both training and test sets) are used as conditions to generate target-domain images, with the only exception being MSD-Prostate. For the paired settings CHAOS T1↔T2-SPIR and PanSeg T1↔T2, the source and target test sets are the same subjects imaged under different protocols. Using source test masks to generate synthetic training pairs therefore trains the downstream nnUNet/UMamba on the target test subjects' ground-truth anatomies, inflating the DSC scores in Table 3. In particular, the headline CHAOS T1→T2-SPIR nnUNet result (6.90 to 66.18) and the 'best in 8 of 10' claim rest on these leaked settings. The clean cross-dataset pairs (LiQA→CHAOS, CHAOS→LiQA, MSD-Prostate→PROMISE12, PROMISE12→MSD-Prostate) show smaller but still positive gains, so the central idea may survive, but the paper must re-run the experiments with source test masks excluded from synthetic-data construction for all settings and recompute the summary claims.
  2. [Sec. 4.1 vs Appendix B.3] Section 4.1 states that 'the test data for segmentation is strictly not used for training our generative data engines to prevent information leakage.' This is contradicted by Appendix B.3, which uses source-domain test masks as conditioning inputs for the generated training pairs. The contradiction is not merely a wording issue: in the paired CHAOS and PanSeg settings, the source test masks coincide with the target test subjects' anatomy, so the segmentation model effectively sees the test geometry during training. The protocol must be clarified and fixed so that no test-subject information enters the synthetic training data.
  3. [Tables 2 and 3] All FID and DSC values are reported as point estimates without error bars, multiple seeds, or significance tests. Some settings use very small test sets (e.g., CHAOS has only 4 test volumes for T2-SPIR), so the reported differences may not be statistically reliable. The paper should report mean and standard deviation over at least three training runs of the downstream segmentation models, or provide an equivalent statistical analysis.
  4. [Sec. 3.1, Sec. 4.1, Appendix E] The paper uses the term 'zero-shot' and claims that MRGen generalizes to underrepresented modalities lacking mask annotations. However, the text-guided pretraining stage requires the target modality's appearance to be present in the unlabeled Radiopaedia corpus. The paper does not verify whether T2-SPIR, ADC, T1, and T2 actually occur in the pretraining data, so the method's behavior for truly unseen protocols is untested. Please either provide modality-coverage statistics for MRGen-DB or evaluate on a modality that is absent from the pretraining corpus, and adjust the 'zero-shot' terminology accordingly.
minor comments (6)
  1. [Figure captions] Figure 7 contains the typo 'Groud Truth' and Figure 8 contains 'stuggles'; these should be corrected.
  2. [Table 2] The header 'Source Datset' contains a typo, and the abbreviations CM., MP., PS., LQ., and PR. are only defined in the table caption; please ensure they are introduced in the text as well.
  3. [Table 3] The column layout of Table 3 is difficult to parse, with the DualNorm column separated from the UMamba and nnUNet groups; consider reorganizing the table so that each segmentation framework is clearly nested under its own header.
  4. [Sec. 4.1] DualNorm is evaluated with a customized UNet while all other baselines use nnUNet or UMamba; this makes the DualNorm comparison non-controlled, and the main text should explicitly acknowledge this limitation.
  5. [Table 4] The comparison between SDM-ft and MRGen is not controlled: MRGen uses a higher latent dimension and a different text encoder, so the individual contributions of these components are not isolated.
  6. [Table 5] The ablation on segmentation performance would be easier to interpret if the source-only nnUNet baseline were included in the table for each setting, as it is currently only reported in Table 3.

Circularity Check

1 steps flagged · score 6.0 of 10

Paired-dataset settings leak target test masks into the synthetic training set, inflating the headline DSC gains; cross-dataset results remain independent.

  1. other [Appendix B.3 (Synthetic Data Statistics); Table 3 (Segmentation Results)]
    "we use mask annotations from the entire source-domain dataset (including both training and test sets) as input conditions to generate target-domain images, forming image-mask training pairs. Exceptions include: (i) for the MSD-Prostate [3] dataset, where images of T2 and ADC modalities have already been registered, we restrict inputs to the source-domain training set to prevent data leakage"

    For the CHAOS T1<->T2-SPIR and PanSeg T1<->T2 settings, source and target are the same subjects scanned under different protocols. Taking masks from the entire source-domain dataset therefore injects the target test subjects' ground-truth organ geometry into the synthetic image-mask pairs used to train nnUNet/UMamba. The reported target test DSC, e.g., CHAOS T1-to-T2-SPIR nnUNet rising from 6.90 to 66.18, is not an independent generalization measurement: the 'predicted' test masks were already provided as conditioning inputs during synthetic training. The cross-dataset pairs (LiQA/CHAOS, MSD-Prostate/PROMISE12) are unaffected and show smaller positive gains, so the circular/leaky component is partial rather than total.

full rationale

The method itself is an empirical generation-plus-segmentation pipeline, not a derivation, and its main baselines (nnUNet, UMamba, CycleGAN, UNSB, DualNorm) are external, so there is no self-definitional circularity in the core contribution. The SAM2 autofilter is an external off-the-shelf model, and the autofilter thresholds are ordinary hyperparameters. However, the paper's own Appendix B.3 states that synthetic training pairs are built from masks of the entire source-domain dataset, including test sets, with an exception only for MSD-Prostate. Because CHAOS and PanSeg pair T1/T2 acquisitions of the same subjects, in four of the ten settings the target-domain test subjects' masks are present in the synthetic training data. The two largest headline improvements (CHAOS T1-to-T2-SPIR 6.90-to-66.18 and T2-SPIR-to-T1 0.80-to-58.10) come from these settings, so the 'best in 8/10' claim is substantially inflated. The remaining cross-dataset settings still show consistent positive but smaller gains, indicating the data-engine idea has independent value despite the evaluation flaw. The self-citation to SAT [67] for the train/test split is a normal methodological citation and is not load-bearing for the paper's derivation.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim relies on unstated coverage of target modalities in the pretraining data, on the accuracy of automatically generated text annotations, and on SAM2 filtering quality, in addition to standard diffusion and VAE machinery. The empirical thresholds for filtering and guidance are hand-set and adaptive, so they count as free parameters.

free parameters (4)
  • Autofilter thresholds (tau_IoU, tau_conf, mean tau_IoU, mean tau_conf) = 0.70, 0.80, 0.80, 0.90
    Chosen by hand to select synthetic samples; if no sample meets them, thresholds are relaxed by 0.10, a post hoc adaptive rule that affects downstream segmentation quality.
  • Classifier-free guidance weight w = 7.0
    Set for inference, controls text-conditioning strength, and is not swept in the paper.
  • Number of generated candidates per mask and selection count = 20 candidates, best 2
    Empirically chosen in Section 3.5 and C.2, influences synthetic dataset size and quality.
  • Latent dimension d and KL loss weight gamma = d=16, gamma=1e-4
    Hyperparameters of the autoencoder; the higher latent dim than Stable Diffusion is credited for quality in the ablation.
assumptions (3)
  • domain assumption The Radiopaedia pretraining corpus contains enough examples of each target modality (T2-SPIR, ADC, T1, T2) for the text-guided model to learn their appearance.
    Invoked in Sections 3.4 and 3.5, but no statistics connect the evaluated target modalities to the pretraining distribution; Figure 6 lists modality categories but not which ones are used downstream.
  • domain assumption BiomedCLIP region classification and GPT-4 modality attribute mapping are sufficiently accurate to provide reliable text conditions.
    Stated in Section 3.2 with 95.33% and 91.67% manual verification on small samples (2% and 20%), but the remaining error propagates into generation quality.
  • domain assumption SAM2's segmentation confidence and IoU against the condition mask are valid proxies for generated-image fidelity.
    Used in Section 3.5 and C.2 to filter synthetic samples; no comparison against human evaluation of fidelity is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MRGen: Segmentation Data Engine for Underrepresented MRI Modalities." pith.science (2026). https://pith.science/paper/OERKIYCC

@misc{pith2026241204106,
  author       = {Pith},
  title        = {Pith review of: MRGen: Segmentation Data Engine for Underrepresented MRI Modalities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OERKIYCC}},
  note         = {Machine review of arXiv:2412.04106}
}
read the original abstract

Training medical image segmentation models for rare yet clinically important imaging modalities is challenging due to the scarcity of annotated data, and manual mask annotations can be costly and labor-intensive to acquire. This paper investigates leveraging generative models to synthesize data, for training segmentation models for underrepresented modalities, particularly on annotation-scarce MRI. Concretely, our contributions are threefold: (i) we introduce MRGen-DB, a large-scale radiology image-text dataset comprising extensive samples with rich metadata, including modality labels, attributes, regions, and organs information, with a subset featuring pixel-wise mask annotations; (ii) we present MRGen, a diffusion-based data engine for controllable medical image synthesis, conditioned on text prompts and segmentation masks. MRGen can generate realistic images for diverse MRI modalities lacking mask annotations, facilitating segmentation training in low-source domains; (iii) extensive experiments across multiple modalities demonstrate that MRGen significantly improves segmentation performance on unannotated modalities by providing high-quality synthetic data. We believe that our method bridges a critical gap in medical image analysis, extending segmentation capabilities to scenarios that are challenging to acquire manual annotations. The codes, models, and data will be publicly available at https://haoningwu3639.github.io/MRGen/

Figures

Figures reproduced from arXiv: 2412.04106 by the authors.

Figure 1
Figure 1. Motivations and Overview. Left: The heterogeneity of MRI modalities challenges the generalization of segmentation models. Our proposed data engine, MRGen, overcomes this by controllably synthesizing training data for segmentation models. Right: (a) Prior generative models are restricted to data augmentation for well-annotated modalities; (b) Image translation typically requires registered data pairs (dashed lines), … view at source ↗
Figure 2
Figure 2. Architecture Overview. Developing our MRGen involves three key steps: (a) Train an autoencoder on various images from MRGen-DB dataset; (b) Train a text-guided generative model within the latent space, using image-text pairs across diverse modalities from MRGen-DB, featuring modality, attributes, region, and organs information; (c) Train a mask condition controller jointly on image-text pairs with and without mask a… view at source ↗
Figure 3
Figure 3. Synthetic Data Construction Pipeline. MRGen takes a text prompt and a mask as conditions for controllably generating MR images and employs a pretrained SAM2 model for automatic filtering to guarantee the quality of generated samples. Discussion. Such a two-stage training strategy empowers MRGen to achieve controllable generation across diverse modalities, even for those lacking mask annotations, driven by two key fa… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Qualitative Results of Controllable Generation. We present images from source domains (Ds) and target domains (Dt) for reference. Here, liver, right kidney, left kidney, spleen, prostate, and pancreas are contoured with different colors. Implementation details. All ima…
Figure 5
Figure 5. Figure 5: Qualitative Results of Segmentation towards Unannotated Modalities. Significant imaging differences between source￾domain (Ds) and target-domain (Dt) make segmentation on target domains (Dt) extremely challenging. We visualize liver, right kidney, left kidney, and sple…
Figure 6
Figure 6. Figure 6: Data Statistics of Radiopaedia-MRI. (a) Distribution of slice counts across various modalities in Radiopaedia-MRI; (b) Propor￾tional distribution of slices across different regions in Radiopaedia-MRI. Data without mask annotations. For the image-text pairs from Radiopa…
Figure 7
Figure 7. Figure 7: Qualitative Results of In-domain Generation. Few-shot methods. Specifically, we compare with a few-shot nnUNet [23] (pre-trained on source-domain data and finetuned on 5% target-domain manually annotated data), as well as UniVerSeg [6] with its official implementation …
Figure 8
Figure 8. Figure 8: Failure Cases Analysis. Our proposed MRGen is not without limitations: (a) it may struggle to handle extremely small organ masks; (b) it occasionally produces false-negative samples, such as the unexpected synthesis of kidneys in the given example. E. Limitations & Fut…
Figure 9
Figure 9. Figure 9: The validation of 3D extension feasibility of MRGen on 256 × 256 × 16 volumes. E.2. Future Works Due to limited computational resources, we validate our data engine on 2D slices, with trained segmentation models able to process 3D volumes slice-by-slice. However, our i…
Figure 10
Figure 10. Figure 10: More Qualitative Results of Controllable Generation. We present images from source domains (Ds) and target domains (Dt) for reference. Here, specific organs are contoured with colors: prostate in MSD-Prostate and PROMISE12 datasets, and pancreas in PanSeg dataset, and…
Figure 11
Figure 11. Figure 11: More Qualitative Results on Segmentation towards Unannotated Modalities. Significant imaging differences between source-domain (Ds) and target-domain (Dt) make segmentation on target domains (Dt) extremely challenging. Here, specific organs are highlighted with colors…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Foundation Model Empowered Synesthesia of Machines (SoM): AI-native Intelligent Multi-Modal Sensing-Communication Integration

    eess.SP 2025-06 conditional novelty 4.0 of 10

    The paper proposes a systematic classification and two roadmaps for using foundation models (LLMs and wireless foundation models) to design Synesthesia of Machines systems for 6G, with preliminary case-study evidence ...

Reference graph

Works this paper leans on

73 extracted references · 63 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Totalsegmentator mri: Robust sequence- independent segmentation of multiple anatomic structures in mri

    Tugba Akinci D’Antonoli, Lucas K Berger, Ashraya K In- drakanti, Nathan Vishwanathan, Jakob Weiss, Matthias Jung, Zeynep Berkarda, Alexander Rau, Marco Reisert, Thomas K¨ustner, et al. Totalsegmentator mri: Robust sequence- independent segmentation of multiple anatomic structures in mri. Radiology, 314(2):e241613, 2025. 2, 17, 18

    work page 2025

  3. [3]

    The medical segmentation decathlon.Nature Communications, 13(1):4128, 2022

    Michela Antonelli, Annika Reinke, Spyridon Bakas, Key- van Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon.Nature Communications, 13(1):4128, 2022. 3, 8, 15, 18

    work page 2022

  4. [4]

    A vision–language foundation model for the generation of realistic chest x-ray images

    Christian Bluethgen, Pierre Chambon, Jean-Benoit Del- brouck, Rogier van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and Akshay S Chaud- hari. A vision–language foundation model for the generation of realistic chest x-ray images. Nature Biomedical Engineer- ing, pages 1–13, 2024. 2, 3

    work page 2024

  5. [5]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023. 2

    work page 2023

  6. [6]

    Uni- verseg: Universal medical image segmentation

    Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Uni- verseg: Universal medical image segmentation. In Proceed- ings of the International Conference on Computer Vision , pages 21438–21451, 2023. 1, 17, 18

    work page 2023

  7. [7]

    Towards generaliz- able tumor synthesis

    Qi Chen, Xiaoxi Chen, Haorui Song, Zhiwei Xiong, Alan Yuille, Chen Wei, and Zongwei Zhou. Towards generaliz- able tumor synthesis. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11147– 11158, 2024. 2

    work page 2024

  8. [8]

    Treasure in distribution: a domain randomiza- tion based multi-source domain generalization for 2d medi- cal image segmentation

    Ziyang Chen, Yongsheng Pan, Yiwen Ye, Hengfei Cui, and Yong Xia. Treasure in distribution: a domain randomiza- tion based multi-source domain generalization for 2d medi- cal image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pages 89–99, 2023. 3, 16

    work page 2023

Show all 73 references
  1. [9]

    Conditional diffusion models for semantic 3d brain mri synthesis

    Zolnamar Dorjsembe, Hsing-Kuo Pao, Sodtavilan Odonchimed, and Furen Xiao. Conditional diffusion models for semantic 3d brain mri synthesis. IEEE Journal of Biomedical and Health Informatics, 2024. 2, 3, 16

    2024

  2. [10]

    Segvol: Universal and interactive volumetric medical image segmen- tation

    Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao. Segvol: Universal and interactive volumetric medical image segmen- tation. In Advances in Neural Information Processing Sys- tems, 2024. 2

    2024

  3. [11]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the International Conference on Computer Vision, 2023. 2

    2023

  4. [12]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Proceedings of the International Conference ...

    2024

  5. [13]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 2020. 2

    2020

  6. [14]

    Maisi: Medical ai for synthetic imaging

    Pengfei Guo, Can Zhao, Dong Yang, Ziyue Xu, Vish- wesh Nath, Yucheng Tang, Benjamin Simon, Mason Belue, Stephanie Harmon, Baris Turkbey, et al. Maisi: Medical ai for synthetic imaging. In Winter Conference on Applications of Computer Vision, 2025. 2, 3, 16, 18

    2025

  7. [15]

    Generatect: Text- conditional generation of 3d chest ct volumes

    Ibrahim Ethem Hamamci, Sezgin Er, Anjany Sekuboy- ina, Enis Simsar, Alperen Tezcan, Ayse Gulnihan Sim- sek, Sevval Nil Esirgun, Furkan Almas, Irem Do ˘gan, Muhammed Furkan Dasdelen, et al. Generatect: Text- conditional generation of 3d chest ct volumes. In Proceed- ings of the...

    2024

  8. [16]

    Mrsegmentator: Robust multi-modality segmentation of 40 classes in mri and ct sequences

    Hartmut H ¨antze, Lina Xu, Felix J Dorfner, Leonhard Donle, Daniel Truhn, Hugo Aerts, Mathias Prokop, Bram van Gin- neken, Alessa Hering, Lisa C Adams, et al. Mrsegmentator: Robust multi-modality segmentation of 40 classes in mri and ct sequences. arXiv preprint arXiv:2405.064...

    2024 arXiv

  9. [17]

    Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images

    Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI brainlesion workshop, pages 272–284, 2021. 2

    2021

  10. [18]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In Advances in Neural Information Processing Sys- tems, 2017. 6

    2017

  11. [19]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 6

    2021

  12. [20]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In Advances in Neural Informa- tion Processing Systems, 2020. 2, 13

    2020

  13. [21]

    Video diffu- sion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffu- sion models. In Advances in Neural Information Processing Systems, 2022. 2

    2022

  14. [22]

    Devil is in channels: Contrastive single domain generalization for medical image segmentation

    Shishuai Hu, Zehui Liao, and Yong Xia. Devil is in channels: Contrastive single domain generalization for medical image segmentation. In Medical Image Computing and Computer- Assisted Intervention, pages 14–23, 2023. 3, 16

    2023

  15. [23]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Pe- tersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation. Nature Methods, 18(2):203–211, 2021. 1, 2, 6, 7, 8, 16, 17, 18

    2021

  16. [24]

    Image-to-image translation with conditional adver- 9 sarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- 9 sarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1125– 1134, 2017. 2

    2017

  17. [25]

    Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation

    Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xi- ang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. In Advances in Neural Information Processing Systems ,...

    2022

  18. [26]

    Scaling up gans for text-to-image synthesis

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, 2023. 2

    2023

  19. [27]

    Chaos challenge-combined (ct-mr) healthy abdominal organ seg- mentation

    A Emre Kavur, N Sinem Gezer, Mustafa Barıs ¸, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Duc Duy Pham, Soumick Chatterjee, Philipp Ernst, Savas ¸¨Ozkan, et al. Chaos challenge-combined (ct-mr) healthy abdominal organ seg- mentation. Medical Image Analysis, 2021. 3, 8, 15, 17, 18

    2021

  20. [28]

    Unpaired image-to-image translation via neu- ral schr ¨odinger bridge

    Beomsu Kim, Gihyun Kwon, Kwanyoung Kim, and Jong Chul Ye. Unpaired image-to-image translation via neu- ral schr ¨odinger bridge. In Proceedings of the International Conference on Learning Representations, 2024. 3, 6, 16, 18

    2024

  21. [29]

    Adaptive latent diffusion model for 3d medical image to image translation: Multi- modal magnetic resonance imaging study

    Jonghun Kim and Hyunjin Park. Adaptive latent diffusion model for 3d medical image to image translation: Multi- modal magnetic resonance imaging study. In Winter Con- ference on Applications of Computer Vision, 2024. 3, 16

    2024

  22. [30]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. In Proceedings of the International Conference on Learning Representations, 2014. 5, 13

    2014

  23. [31]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the International Conference on Computer Vision, pages 4015–4026, 2023. 2

    2023

  24. [32]

    Generating synthetic data for medical imaging

    Lennart R Koetzier, Jie Wu, Domenico Mastrodicasa, Aline Lutz, Matthew Chung, W Adam Koszek, Jayanth Pratap, Ak- shay S Chaudhari, Pranav Rajpurkar, Matthew P Lungren, et al. Generating synthetic data for medical imaging. Radi- ology, 312(3):e232471, 2024. 2

    2024

  25. [33]

    Anatomically-controllable medical image gen- eration with segmentation-guided diffusion models

    Nicholas Konz, Yuwen Chen, Haoyu Dong, and Maciej A Mazurowski. Anatomically-controllable medical image gen- eration with segmentation-guided diffusion models. In Med- ical Image Computing and Computer-Assisted Intervention , pages 88–98, 2024. 2

    2024

  26. [34]

    Generative models improve fairness of medical classifiers under distribution shifts

    Ira Ktena, Olivia Wiles, Isabela Albuquerque, Sylvestre- Alvise Rebuffi, Ryutaro Tanno, Abhijit Guha Roy, Shekoofeh Azizi, Danielle Belgrave, Pushmeet Kohli, Tay- lan Cemgil, et al. Generative models improve fairness of medical classifiers under distribution shifts. Nature Med...

    2024

  27. [35]

    Evaluation of prostate segmentation algorithms for mri: the promise12 challenge

    Geert Litjens, Robert Toth, Wendy Van De Ven, Caroline Hoeks, Sjoerd Kerkstra, Bram Van Ginneken, Graham Vin- cent, Gwenael Guillard, Neil Birbeck, Jindang Zhang, et al. Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Medical Image Analysis , 1...

    2014

  28. [36]

    Intelligent grimm - open-ended visual storytelling via latent diffusion models

    Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yan- feng Wang, and Weidi Xie. Intelligent grimm - open-ended visual storytelling via latent diffusion models. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6190–6200, 2024. 2

    2024

  29. [37]

    Merit: Multi-view evidential learning for reliable and interpretable liver fibrosis staging

    Yuanye Liu, Zheyao Gao, Nannan Shi, Fuping Wu, Yuxin Shi, Qingchao Chen, and Xiahai Zhuang. Merit: Multi-view evidential learning for reliable and interpretable liver fibrosis staging. Medical Image Analysis, 2025. 3, 15

    2025

  30. [38]

    Radimagegan–a multi- modal dataset-scale generative ai for medical imaging

    Zelong Liu, Peyton Smith, Alexander Lautin, Jieshen Zhou, Maxwell Yoo, Mikey Sullivan, Haorun Li, Louisa Deyer, Alexander Zhou, Arnold Yang, et al. Radimagegan–a multi- modal dataset-scale generative ai for medical imaging. InIn- ternational Workshop on Applications of Medical...

    2024

  31. [39]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Confer- ence on Learning Representations, 2019. 6

    2019

  32. [40]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15:1–9, 2024. 1, 2, 17

    2024

  33. [41]

    U-mamba: Enhancing long-range dependency for biomedical image segmentation

    Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722, 2024. 2, 6, 7, 16

    2024 arXiv

  34. [42]

    Why rankings of biomedical image analysis competitions should be interpreted with care

    Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew P Bradley, Aaron Carass, et al. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications,...

    2018

  35. [43]

    Multi-modal modality-masked diffusion network for brain mri synthesis with random modality missing

    Xiangxi Meng, Kaicong Sun, Jun Xu, Xuming He, and Ding- gang Shen. Multi-modal modality-masked diffusion network for brain mri synthesis with random modality missing. IEEE Transactions on Medical Imaging, 2024. 2, 3

    2024

  36. [44]

    Multi-source data integration for segmentation of unannotated mri images

    Navapat Nananukul, Hamid Soltanian-Zadeh, and Moham- mad Rostami. Multi-source data integration for segmentation of unannotated mri images. IEEE Journal of Biomedical and Health Informatics, 2024. 2

    2024

  37. [45]

    Causality-inspired single- source domain generalization for medical image segmenta- tion

    Cheng Ouyang, Chen Chen, Surui Li, Zeju Li, Chen Qin, Wenjia Bai, and Daniel Rueckert. Causality-inspired single- source domain generalization for medical image segmenta- tion. IEEE Transactions on Medical Imaging , 42(4):1095– 1106, 2022. 3, 16

    2022

  38. [46]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the International Conference on Computer Vision, pages 4195–4205, 2023. 2

    2023

  39. [47]

    Structure-preserving synthesis: Maskgan for unpaired mr-ct translation

    Vu Minh Hieu Phan, Zhibin Liao, Johan W Verjans, and Minh Son To. Structure-preserving synthesis: Maskgan for unpaired mr-ct translation. InMedical Image Computing and Computer-Assisted Intervention, 2023. 3, 16, 18

    2023

  40. [48]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. In Proceedings of the International Conference on Learning Representations ,

  41. [49]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman 10 R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph F...

    2025

  42. [50]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, 2022. 2, 7, 13

    2022

  43. [51]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vention, 2015. 1, 2, 4, 13

    2015

  44. [52]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2023. 2

    2023

  45. [53]

    Unit-ddpm: Unpaired image translation with denois- ing diffusion probabilistic models

    Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denois- ing diffusion probabilistic models. arXiv preprint arXiv:2104.05358, 2021. 3, 16

    2021 arXiv

  46. [54]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In Proceedings of the Inter- national Conference on Learning Representations, 2020. 2, 6

    2020

  47. [55]

    Rethinking data augmentation for single-source domain generalization in medical image segmentation

    Zixian Su, Kai Yao, Xi Yang, Kaizhu Huang, Qiufeng Wang, and Jie Sun. Rethinking data augmentation for single-source domain generalization in medical image segmentation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 2366–2374, 2023. 3, 16

    2023

  48. [56]

    Self-improving generative foundation model for synthetic medical image generation and clinical applications

    Jinzhuo Wang, Kai Wang, Yunfang Yu, Yuxing Lu, Wen- chao Xiao, Zhuo Sun, Fei Liu, Zixing Zou, Yuanxu Gao, Lei Yang, et al. Self-improving generative foundation model for synthetic medical image generation and clinical applications. Nature Medicine, pages 1–9, 2024. 2, 3, 16

    2024

  49. [57]

    Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. InProceed- ings of the International Conference on Computer Vision , pages 21372–21383, 2023. 3

    2023

  50. [58]

    Megafusion: Extend dif- fusion models towards higher-resolution image generation without further tuning

    Haoning Wu, Shaocheng Shen, Qiang Hu, Xiaoyun Zhang, Ya Zhang, and Yanfeng Wang. Megafusion: Extend dif- fusion models towards higher-resolution image generation without further tuning. In Winter Conference on Applica- tions of Computer Vision, 2025. 2

    2025

  51. [59]

    Freetumor: Advance tumor segmentation via large-scale tu- mor synthesis

    Linshan Wu, Jiaxin Zhuang, Xuefeng Ni, and Hao Chen. Freetumor: Advance tumor segmentation via large-scale tu- mor synthesis. arXiv preprint arXiv:2406.01264, 2024. 2

    2024 arXiv

  52. [60]

    Ad- versarial consistency for single domain generalization in medical image segmentation

    Yanwu Xu, Shaoan Xie, Maxwell Reynolds, Matthew Ragoza, Mingming Gong, and Kayhan Batmanghelich. Ad- versarial consistency for single domain generalization in medical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, 2022. 3, 16

    2022

  53. [61]

    Medm2g: Unifying medical multi-modal genera- tion via cross-guided diffusion with visual invariant

    Chenlu Zhan, Yu Lin, Gaoang Wang, Hongwei Wang, and Jian Wu. Medm2g: Unifying medical multi-modal genera- tion via cross-guided diffusion with visual invariant. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11502–11512, 2024. 2, 16

    2024

  54. [62]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the International Conference on Computer Vision, 2023. 2

    2023

  55. [63]

    A multimodal biomedical foun- dation model trained from fifteen million image–text pairs

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. A multimodal biomedical foun- dation model trained from fifteen million image–text pairs. NEJM AI, 2(1):AIoa2400640, 2025. 3, 5, 7, 13

    2025

  56. [64]

    Knowledge-enhanced visual-language pre- training on chest radiology images.Nature Communications, 14(1):4542, 2023

    Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre- training on chest radiology images.Nature Communications, 14(1):4542, 2023. 3

    2023

  57. [65]

    Self-supervised tumor segmentation with sim2real adaptation

    Xiaoman Zhang, Weidi Xie, Chaoqin Huang, Ya Zhang, Xin Chen, Qi Tian, and Yanfeng Wang. Self-supervised tumor segmentation with sim2real adaptation. IEEE Journal of Biomedical and Health Informatics, 2023. 2

    2023

  58. [66]

    Large- scale multi-center ct and mri segmentation of pancreas with deep learning

    Zheyuan Zhang, Elif Keles, Gorkem Durak, Yavuz Tak- tak, Onkar Susladkar, Vandan Gorade, Debesh Jha, Asli C Ormeci, Alpay Medetalibeyoglu, Lanhong Yao, et al. Large- scale multi-center ct and mri segmentation of pancreas with deep learning. Medical Image Analysis, 2025. 3, 15

    2025

  59. [67]

    One model to rule them all: Towards universal segmentation for medical im- ages with text prompts

    Ziheng Zhao, Yao Zhang, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. One model to rule them all: Towards universal segmentation for medical im- ages with text prompts. arXiv preprint arXiv:2312.17183 ,

    arXiv

  60. [68]

    nn- former: V olumetric medical image segmentation via a 3d transformer

    Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Xiaoguang Han, Lequan Yu, Liansheng Wang, and Yizhou Yu. nn- former: V olumetric medical image segmentation via a 3d transformer. IEEE Transactions on Image Processing, 2023. 2

    2023

  61. [69]

    Generalizable cross-modality medical image segmentation via style augmentation and dual normalization

    Ziqi Zhou, Lei Qi, Xin Yang, Dong Ni, and Yinghuan Shi. Generalizable cross-modality medical image segmentation via style augmentation and dual normalization. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 20856–20865, 2022. 3, 6, 7, 16, 18

    2022

  62. [70]

    Unpaired image-to-image translation using cycle- consistent adversarial networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the In- ternational Conference on Computer Vision , 2017. 2, 3, 6, 16, 18 11 MRGen: Segmentation Data Engine for Un...

    2017

  63. [71]

    Problem Formulation

    Method 3 3.1. Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.2. Dataset Curation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.3. Architecture . ....

  64. [72]

    Experimental Settings

    Experiments 5 4.1. Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.2. Quantitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3. Qualitative...

  65. [73]

    :PanSegT2 𝒟!: MSD-Prostate ADC → 𝒟

    Conclusion 8 A . Preliminaries on Diffusion Models 13 B . Details of MRGen-DB & Synthetic Data 13 B.1. Automatic Annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B.2. Dataset Statistics . . . . . . . . . . . . . . . ....

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.