Pith. sign in

REVIEW 4 major objections 6 minor 46 references

OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single vision-language model claims to run the whole MRI pipeline, from k-space to report text.

desk verdict A coherent but unverified system proposal: the full-stack MRI claim rests on curated figures, not on measurements, so the preprint should not be cited as evidence of capability. read the letter →

arxiv 2508.17524 v1 pith:J4ZG5S7W submitted 2025-08-24 cs.CV cs.AI

Xingxin He , Aurora Rofena , Ruimin Feng , Haozhe Liao , Zhaoye Zhou , Albert Jang , Fang Liu This is my paper
classification cs.CVcs.AI
keywords MRIfoundationmodelvision-languagemulti-taskinstructiontuningimagereconstructionsegmentationabnormalitydetectionradiologyreportgenerationmultimodalpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OmniMRI is one vision-language model trained to handle the full MRI workflow: reconstructing images from undersampled data, segmenting anatomy and pathology, detecting abnormalities, suggesting diagnoses, and generating radiology reports. The authors assemble roughly 19 million MRI slices from 60 public datasets and train the model in four stages—self-supervised vision pretraining, vision-language alignment, multimodal pretraining, and multi-task instruction tuning—so a single set of weights can switch between pixel-level and text-level outputs based on a natural-language prompt. The evidence presented is qualitative: the paper shows example reconstructions, segmentations, detections, diagnostic suggestions, and reports, and its own closing section states that quantitative benchmarking and radiologist-reader validation remain future work. If the claim holds, the fragmented collection of anatomy- and task-specific MRI models could be replaced by one instruction-following system.

What carries the argument

The central object is a unified autoregressive Transformer backbone with multimodal self-attention and a mixture-of-experts feedforward network, into which image tokens from a Swin vision encoder and language tokens from a text encoder are interleaved as a single sequence. A dual-decoder design branches from the backbone: a diffusion-based image decoder produces dense outputs such as reconstructed images and segmentation masks, while a text decoder produces semantic outputs such as bounding boxes, diagnostic suggestions, and reports. The mechanism that lets one model cover the full workflow is the instruction-conditioned token sequence: every task is expressed in the same prompt-plus-image f

What would settle it

Run the trained model on a standard annotated MRI benchmark with ground-truth reconstruction targets, segmentation labels, and lesion boxes, then compare one checkpoint across three tasks: reconstruction from 4x/6x undersampling, tumor segmentation, and lesion detection. If reconstruction PSNR/SSIM, segmentation Dice, or detection average precision falls far below single-task models trained on the same data, the unification claim would be measurably false. A cheaper check: sample a few hundred generated anatomical descriptions and score them against a radiologist's annotations; systematic mid-

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a single autoregressive vision-language Transformer with a diffusion-based image decoder and a text decoder can absorb tasks that are normally built as separate models: reconstructing images from undersampled k-space, segmenting anatomy and pathology, localizing abnormalities with bounding boxes, proposing differential diagnoses, and writing radiology reports. The model treats every task as an instruction-following problem: image tokens and language tokens are interleaved into one sequence, and the answer—whether an image, a mask, a bounding box, or prose—is decoded from the shared representation. Training moves from self-supervised vis

Load-bearing premise

The load-bearing premise is that the automatically generated text descriptions used as training supervision accurately describe the MRI content, because the model's clinical semantics come from those descriptions and the paper checks only a few qualitative examples rather than measuring that accuracy.

Editorial extensions

If this is right

  • A single checkpoint could serve reconstruction, segmentation, detection, diagnosis, and reporting, removing the need to deploy and integrate separate task-specific models.
  • New MRI tasks could be added by reformulating them as instruction-response pairs and fine-tuning the same backbone, without architectural changes.
  • The four-stage training recipe provides a scalable template for using large, partially annotated public MRI corpora to build medical vision-language models.
  • Language-conditioned decoding may expose the model's reasoning for detection and diagnosis in a readable form, not just as a label or mask.
  • The scale of the corpus—over 19 million slices across 60 datasets—suggests the approach could continue to improve with more data, consistent with the behavior of other foundation models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: because no quantitative results are reported, the most direct test of the unification claim is whether one weight set matches specialized baselines on reconstruction fidelity, segmentation Dice, and detection average precision; that benchmark, not additional examples, would settle the claim.
  • The supervision for anatomical structures and tissue-signal descriptors comes from a general-purpose vision-language model, so errors in those generated descriptions likely bound the clinical ceiling of diagnostic suggestions and reports; auditing a random sample against radiologist labels would estimate how much error is baked in.
  • The architecture and training recipe are not MRI-specific, so the same approach could plausibly extend to other tomographic modalities such as CT or ultrasound if paired vision-text data can be generated at similar scale; the paper does not claim this extension.
  • Because detection output is cast as text tokens, detection, diagnosis, and report generation share one token space, which may naturally keep reported findings consistent with detected abnormalities; the paper does not demonstrate that consistency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes OmniMRI, a unified vision-language foundation model intended to cover the full MRI workflow, from undersampled reconstruction and segmentation to abnormality detection, diagnostic suggestion, and report generation. The training corpus is assembled from 60 public datasets and the training paradigm comprises four stages: self-supervised vision pretraining, contrastive vision-language alignment, multimodal autoregressive pretraining, and multi-task instruction tuning. The reported evaluation consists exclusively of qualitative examples in Figures 3 and 4; there are no quantitative metrics, baselines, error bars, or held-out evaluations. The conclusion explicitly states that the current evaluation is qualitative and defers quantitative benchmarking, radiologist validation, and deployment studies to future work.

Significance. If the central capability claim were established — one weight set performing pixel-level reconstruction/segmentation and semantic-level detection/diagnosis/reporting across anatomies and contrasts — this would be a significant contribution to medical imaging foundation models. The scale of the curated corpus (224k volumes, 19M slices) and the multi-stage training recipe are also of potential interest. However, the current manuscript does not substantiate these claims. The absence of any quantitative evaluation, task-specific baselines, or generalization tests means the paper functions as a technical proposal and qualitative showcase rather than a validated foundation model. The additional reliance on Qwen-VL-generated descriptions without radiologist verification raises a further correctness risk that the qualitative evaluation cannot resolve.

major comments (4)
  1. [Section 5, Results] The central claim that OmniMRI 'performs image reconstruction, segmentation, abnormality detection, diagnostic suggestion, and radiology report generation' is supported only by selected qualitative examples. No task has a single quantitative metric: no PSNR/SSIM for reconstruction, no Dice/Jaccard for segmentation, no mAP/precision/recall for detection, no AUROC for diagnostic suggestion, and no BLEU/ROUGE or clinician ratings for report generation. No comparisons to task-specific baselines (e.g., U-Net, Swin UNETR, compressed sensing) or to other vision-language models are provided. The conclusion's own statement that 'our current evaluation focuses on qualitative demonstrations' is an admission that the load-bearing capability claim is unverified. This is not a presentation issue; it is a missing evaluation of the paper's core assertion.
  2. [Section 2.2.2] The paired vision-text data used for vision-language alignment and multimodal pretraining are generated by Qwen-VL, a general-purpose VLM, prompted with the template in Figure 2. The paper states that the mid-level descriptors (visible anatomical structures, tissue signal characteristics) are 'rarely annotated' and are produced by this generative augmentation. There is no validation of these generated descriptions against radiologist annotations, structured reports, or even a random-sample human audit. If Qwen-VL systematically mislabels anatomy or signal characteristics, those errors are propagated into the learned vision-language semantics and cannot be detected by the qualitative figures, which are curated. The manuscript needs either a validation study of the generated text or a demonstration that the multimodal pretraining is robust to this synthetic supervision.
  3. [Sections 1 and 6] The paper claims 'zero-shot generalization' across contrasts, anatomies, and tasks, but no protocol defines what is zero-shot. There is no train/evaluation split, no held-out dataset, and no evidence that the examples in Figures 3–4 were excluded from the training corpus or involve contrasts/anatomies/tasks not seen during training. Without such a protocol, the phrase 'zero-shot' is not operationalized, and the qualitative examples are consistent with memorization or near-duplicate retrieval. This claim must be tested with a defined held-out task set before it can support the paper's generalist framing.
  4. [Sections 3 and 4, Tables 1–2] The architecture and training description lack the details needed for reproducibility or for assessing the validity of the multi-stage recipe. Model size, number of parameters, transformer depth/width, MoE configuration, training steps, batch size, learning rate, and compute are not reported. Table 2 gives only coarse stage-level data ratios (e.g., 0.8 vision-text / 0.2 instruction-response in multimodal pretraining) without actual instance counts or balancing procedures. The prompt templates in Table 1 are representative rather than exhaustive, and the exact instruction sampling scheme is unspecified. These omissions prevent an independent check of whether the described training stages actually contribute to the reported behavior.
minor comments (6)
  1. [Section 2.2.2 heading] The heading 'Paired Vision-T ext Data' contains a typo ('T ext').
  2. [Figure 1C] The labels 'Maksed Modeling' and 'Contrast Recognization' contain typos; also the shaded arrows for positive/negative pairs are not clearly explained in the caption.
  3. [Sections 3.1 and 4.2] The text refers to 'Qwen2.537' (missing a space; presumably Qwen2.5), while reference [37] is the Qwen3 technical report. This is inconsistent and should be corrected.
  4. [Section 2.2.3] The phrase 'detection bounding' appears incomplete; it should be 'detection bounding boxes' or similar.
  5. [Section 5, Segmentation] The sentence 'The consistency of segmentation performance across heterogeneous organs and tissue types of underscores OmniMRI’s ability...' has a grammatical error ('of underscores').
  6. [Figures 3 and 4] The displayed images are small and lack zoomed insets or quantitative annotations; it is difficult to verify claims such as 'sharp boundary definition' or 'suppresses undersampling artifacts' from the printed figures. Cropped regions or error maps would improve legibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are empirical and rest on external data, with acknowledged qualitative evaluation.

full rationale

OmniMRI is an empirical systems paper rather than a formal derivation, so the circularity tests must be applied to its training/evaluation chain. The main candidate is the use of Qwen-VL-generated hierarchical descriptions (Section 2.2.2) as paired vision-text supervision. This is a data-generation strategy, not a self-referential definition: the mid-level descriptors are used as training targets for vision-language alignment and multimodal pretraining, but the paper's downstream claims (reconstruction, segmentation, detection, diagnosis, report generation) are not evaluated against those same Qwen-VL outputs as ground truth. The instruction-response data for the final stage is described as using ground-truth responses from existing datasets, not the Qwen-VL captions. The qualitative demonstrations in Figures 3–4 are therefore not equivalent, by construction, to the input labels. The acknowledged limitation in the Conclusion—that evaluation is qualitative and quantitative benchmarking is deferred—is an evidentiary gap, not circularity. The only self-citation (reference [34] in the Introduction) is contextual and non-load-bearing, and the central architecture builds on external works (Qwen2.5, Swin Transformer, diffusion heads). No equation or fitted parameter is renamed as a prediction. Accordingly, no circular step meets the required evidentiary bar.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two unvalidated pillars: (1) Qwen-VL-generated text is treated as reliable supervision for anatomy and tissue semantics, and (2) the aggregate 60-dataset corpus is assumed sufficient and clean. Neither is verified with any measurement in the paper. No new physical or conceptual entities are introduced.

free parameters (3)
  • Training-stage data mixing ratios = Stage 1: image-only 1.0; Stage 3: vision-text 0.8 vs instruction-response 0.2; Stage 4: instruction-response 1.0 (Table
    Hand-chosen ratios balancing modality and task coverage; no ablation or external target justifies them.
  • Qwen-VL prompt template for text generation = Not specified beyond the hierarchical template in Figure 2
    The template vocabulary and prompt wording determine the distribution of generated supervision; no validation or ablation is reported.
  • Undisclosed model hyperparameters = Not reported (layers, hidden dims, MoE expert count, diffusion steps, learning rates)
    Central to any reproduction but absent from Sections 3 and 4.
assumptions (4)
  • domain assumption Qwen-VL-generated hierarchical descriptions are clinically accurate and consistent enough to serve as training text for anatomy and tissue-signal descriptors
    Invoked in Section 2.2.2; the vision-language alignment and multimodal pretraining stages depend on these labels, and no radiologist validation is described.
  • domain assumption Aggregating 60 public datasets yields a harmonizable corpus whose distribution supports generalization across vendors, field strengths, sequences, anatomies, and pathologies
    Section 2.1; no dataset list, no per-source quality checks, and no leakage analysis across the 60 sources.
  • domain assumption Autoregressive next-token prediction over interleaved image/text tokens plus a diffusion head can jointly learn pixel-level and semantic-level tasks in one backbone
    Sections 4.3-4.4; the paper cites general-domain precedents but provides no MRI-specific evidence that the shared backbone avoids task interference.
  • domain assumption Public dataset annotations (segmentation masks, pathology labels, k-space undersampling targets) are treated as ground truth
    Section 2.2.3; annotation quality varies across 60 sources and is not audited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation." pith.science (2026). https://pith.science/paper/J4ZG5S7W

@misc{pith2026250817524,
  author       = {Pith},
  title        = {Pith review of: OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4ZG5S7W}},
  note         = {Machine review of arXiv:2508.17524}
}
read the original abstract

Magnetic Resonance Imaging (MRI) is indispensable in clinical practice but remains constrained by fragmented, multi-stage workflows encompassing acquisition, reconstruction, segmentation, detection, diagnosis, and reporting. While deep learning has achieved progress in individual tasks, existing approaches are often anatomy- or application-specific and lack generalizability across diverse clinical settings. Moreover, current pipelines rarely integrate imaging data with complementary language information that radiologists rely on in routine practice. Here, we introduce OmniMRI, a unified vision-language foundation model designed to generalize across the entire MRI workflow. OmniMRI is trained on a large-scale, heterogeneous corpus curated from 60 public datasets, over 220,000 MRI volumes and 19 million MRI slices, incorporating image-only data, paired vision-text data, and instruction-response data. Its multi-stage training paradigm, comprising self-supervised vision pretraining, vision-language alignment, multimodal pretraining, and multi-task instruction tuning, progressively equips the model with transferable visual representations, cross-modal reasoning, and robust instruction-following capabilities. Qualitative results demonstrate OmniMRI's ability to perform diverse tasks within a single architecture, including MRI reconstruction, anatomical and pathological segmentation, abnormality detection, diagnostic suggestion, and radiology report generation. These findings highlight OmniMRI's potential to consolidate fragmented pipelines into a scalable, generalist framework, paving the way toward foundation models that unify imaging and clinical language for comprehensive, end-to-end MRI interpretation.

Figures

Figures reproduced from arXiv: 2508.17524 by the authors.

Figure 1
Figure 1. Overview of OmniMRI framework. (A) Large-scale and diverse MRI dataset construction. The training corpus integrates heterogeneous public datasets, covering diverse patient demographics (age, sex, geography), scanner vendors and field strengths, acquisition protocols and sequence types, anatomical regions, and task-specific annotations. (B) Unified vision–language architecture. MRI inputs (2D slices or 3D volumes) an… view at source ↗
Figure 2
Figure 2. Hierarchical text prompt template for MRI volumes. The template defines structured fields that capture progressive levels of clinical semantics, ranging from image modality and acquisition parameters to visible anatomical structures, tissue signal characteristics, pathological findings, and diagnostic impressions. linguistic relationships, while a mixture-of-experts feedforward network improves parameter efficiency … view at source ↗
Figure 3
Figure 3. Qualitative results of OmniMRI on MRI reconstruction and segmentation tasks. (A) Im￾age reconstruction. OmniMRI restores high-fidelity images from undersampled inputs across diverse anatomies, including the brain, breast, prostate, and knee. (B) Image segmentation. OmniMRI delin￾eates anatomical and pathological. Examples include segmentation of enhancing and non-enhancing brain tumors, knee bones and cartilages, pr… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative results of OmniMRI on higher-level tasks. (A) Abnormality detection. The model identifies and localizes brain edema and meniscus tears in the knee, producing bounding boxes. (B) Diagnostic suggestion. OmniMRI generates structured diagnostic reasoning. In th…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 33 canonical work pages

  1. [1]

    MRI from Picture to Proton

    Donald W McRobbie, Elizabeth A Moore, Martin J Graves, and Martin R Prince. MRI from Picture to Proton . Cambridge university press, 2017

    work page 2017

  2. [2]

    Sparse mri: The application of compressed sensing for rapid mr imaging

    Michael Lustig, David Donoho, and John M Pauly. Sparse mri: The application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine , 58(6):1182– 1195, 2007

    work page 2007

  3. [3]

    Statistical shape models for 3d medical image segmentation: a review

    Tobias Heimann and Hans-Peter Meinzer. Statistical shape models for 3d medical image segmentation: a review. Medical image analysis , 13(4):543–563, 2009

    work page 2009

  4. [4]

    Radlex: a new method for indexing online educational materials, 2006

    Curtis P Langlotz. Radlex: a new method for indexing online educational materials, 2006

    work page 2006

  5. [5]

    Toward best practices in radiology reporting

    Charles E Kahn Jr, Curtis P Langlotz, Elizabeth S Burnside, John A Carrino, David S Channin, David M Hovsepian, and Daniel L Rubin. Toward best practices in radiology reporting. Radiology, 252(3):852–856, 2009

    work page 2009

  6. [6]

    Deep learning.nature, 521(7553):436– 444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436– 444, 2015

    2015

  7. [7]

    Advancing mri reconstruction: a systematic review of deep learning and compressed sensing integration

    Mojtaba Safari, Zach Eidex, Chih-Wei Chang, Richard LJ Qiu, and Xiaofeng Yang. Advancing mri reconstruction: a systematic review of deep learning and compressed sensing integration. Biomedical Signal Processing and Control , 111:108291, 2026

    work page 2026

  8. [8]

    Deep-learning methods for parallel magnetic reso- nance imaging reconstruction: A survey of the current approaches, trends, and issues

    Florian Knoll, Kerstin Hammernik, Chi Zhang, Steen Moeller, Thomas Pock, Daniel K Sodickson, and Mehmet Akcakaya. Deep-learning methods for parallel magnetic reso- nance imaging reconstruction: A survey of the current approaches, trends, and issues. IEEE signal processing magazine , 37(1):128–140, 2020

    work page 2020

Show all 46 references
  1. [9]

    A review on deep learning mri reconstruction without fully sampled k-space

    Gushan Zeng, Yi Guo, Jiaying Zhan, Zi Wang, Zongying Lai, Xiaofeng Du, Xiaobo Qu, and Di Guo. A review on deep learning mri reconstruction without fully sampled k-space. BMC Medical Imaging , 21(1):195, 2021

    2021

  2. [10]

    A review of deep learning for brain tumor analysis in mri

    Felix J Dorfner, Jay B Patel, Jayashree Kalpathy-Cramer, Elizabeth R Gerstner, and Christopher P Bridge. A review of deep learning for brain tumor analysis in mri. NPJ Precision Oncology, 9(1):2, 2025. 13

    2025

  3. [11]

    Deep learning for brain mri segmentation: state of the art and future directions

    Zeynettin Akkus, Alfiia Galimzianova, Assaf Hoogi, Daniel L Rubin, and Bradley J Erickson. Deep learning for brain mri segmentation: state of the art and future directions. Journal of digital imaging , 30(4):449–459, 2017

    2017

  4. [12]

    Deep semantic segmentation of natural and medical images: a review

    Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: a review. Artificial intelligence review , 54(1):137–178, 2021

    2021

  5. [13]

    Deep learning approaches for brain tumor detection and classification using mri images (2020 to 2024): a systematic review

    Sara Bouhafra and Hassan El Bahi. Deep learning approaches for brain tumor detection and classification using mri images (2020 to 2024): a systematic review. Journal of Imaging Informatics in Medicine , 38(3):1403–1433, 2025

    2020

  6. [14]

    Applications of deep learning techniques for automated multiple sclerosis detection using magnetic resonance imaging: A review

    Afshin Shoeibi, Marjane Khodatars, Mahboobeh Jafari, Parisa Moridian, Mitra Rezaei, Roohallah Alizadehsani, Fahime Khozeimeh, Juan Manuel Gorriz, J´ onathan Heras, Maryam Panahiazar, et al. Applications of deep learning techniques for automated multiple sclerosis detection usi...

    2021

  7. [15]

    Machine-learning-based disease diagnosis: A comprehensive review

    Md Manjurul Ahsan, Shahana Akter Luna, and Zahed Siddique. Machine-learning-based disease diagnosis: A comprehensive review. In Healthcare, volume 10, page 541. MDPI, 2022

    2022

  8. [16]

    Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on mri

    Maciej A Mazurowski, Mateusz Buda, Ashirbani Saha, and Mustafa R Bashir. Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on mri. Journal of magnetic resonance imaging , 49(4):939–954, 2019

    2019

  9. [17]

    Deep learning applications to breast cancer detection by magnetic resonance imaging: a literature review

    Richard Adam, Kevin Dell’Aquila, Laura Hodges, Takouhie Maldjian, and Tim Q Duong. Deep learning applications to breast cancer detection by magnetic resonance imaging: a literature review. Breast Cancer Research, 25(1):87, 2023

    2023

  10. [18]

    On the opportunities and risks of foundation models

    Rishi Bommasani. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  11. [19]

    Momentum con- trast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum con- trast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729–9738, 2020

    2020

  12. [20]

    A simple frame- work for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR, 2020

    2020

  13. [21]

    Emerging properties in self-supervised vision transform- ers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bo- janowski, and Armand Joulin. Emerging properties in self-supervised vision transform- ers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  14. [22]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000–16009, 2022

    2022

  15. [23]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 , 2021. 14

    2021 arXiv

  16. [24]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  17. [25]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  18. [26]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  19. [27]

    Attention is all you need

    Vaswani Ashish. Attention is all you need. Advances in neural information processing systems, 30:I, 2017

    2017

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  21. [29]

    BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings ...

    2022

  22. [30]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, S...

    2022

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 34892–34916. Curran Associates, Inc., 2023

    2023

  24. [32]

    Stolte, Yunchao Yang, Kang Liu, Kyle B

    Joseph Cox, Peng Liu, Skylar E. Stolte, Yunchao Yang, Kang Liu, Kyle B. See, Huiwen Ju, and Ruogu Fang. Brainsegfounder: Towards 3d foundation models for neuroimage segmentation. Medical Image Analysis, 97:103301, 2024

    2024

  25. [33]

    Sam3d: Segment anything model in volumetric medical images

    Nhat-Tan Bui, Dinh-Hieu Hoang, Minh-Triet Tran, Gianfranco Doretto, Donald Adjeroh, Brijesh Patel, Arabinda Choudhary, and Ngan Le. Sam3d: Segment anything model in volumetric medical images. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI) , pages 1–4, 2024. 15

    2024

  26. [34]

    Few-shot adap- tation of training-free foundation model for 3d medical image segmentation, 2025

    Xingxin He, Yifan Hu, Zhaoye Zhou, Mohamed Jarraya, and Fang Liu. Few-shot adap- tation of training-free foundation model for 3d medical image segmentation, 2025

    2025

  27. [35]

    Vista3d: A unified segmentation foundation model for 3d medical imaging

    Yufan He, Pengfei Guo, Yucheng Tang, Andriy Myronenko, Vishwesh Nath, Ziyue Xu, Dong Yang, Can Zhao, Benjamin Simon, Mason Belue, Stephanie Harmon, Baris Turk- bey, Daguang Xu, and Wenqi Li. Vista3d: A unified segmentation foundation model for 3d medical imaging. In Proceeding...

    2025

  28. [36]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, 2024

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, 2024

    2024

  29. [37]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025

  30. [38]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    2025

  31. [39]

    Blip3- o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3- o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    2025

  32. [40]

    Emerging properties in unified multimodal pretraining, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining, 2025

    2025

  33. [41]

    Ming-omni: A unified multimodal model for perception and generation, 2025

    Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, Furong Xu, Guang- Ming Yao, Jun Zhou, Jingdong Chen, Jianxin Sun, Jiajia Liu, Jianjiang Zhu, Jun Peng, Kaixiang Ji, Kaiyou Song, Kaimeng Re...

    2025

  34. [42]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, October 2021. 16

    2021

  35. [43]

    Diffu- sion models in vision: A survey

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffu- sion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023

    2023

  36. [44]

    Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh

    Yucheng Tang, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2022

  37. [45]

    Roth, and Daguang Xu

    Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R. Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In Alessandro Crimi and Spyridon Bakas, editors, Brainlesion: Glioma, Multiple Sclerosis, Stroke and ...

    2022

  38. [46]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9653–9663, June 2022. 17

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.