Pith. sign in

REVIEW 2 major objections 5 minor 105 references

Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models

T0 review · 2 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that pretraining a 3D medical image encoder with 2D slice images and text descriptions generated by a multimodal large language model improves downstream 3D segmentation and classification beyond image-only self-supervised

desk verdict Solid method with a real data-overlap problem on MSD-Liver; worth refereeing, but the headline claim needs reframing. read the letter →

arxiv 2509.09064 v1 pith:IOGMM4CY submitted 2025-09-11 cs.CV

Qiuhui Chen , Xuancheng Yao , Huping Ye , Yi Hong This is my paper · ORCID
classification cs.CV
keywords 3Dmedicalimageunderstandingvision-languagepretrainingself-supervisedlearningmultimodallargelanguagemodelsoptimaltransportalignmentsegmentationclassificationplane-slice-awaretransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Med3DInsight adds language supervision to 3D medical image pretraining without human annotations. For each 3D volume, it samples one 2D slice, has a multimodal large language model write a text description of that slice, and trains the volume encoder to reconstruct the volume while aligning its features with both the 2D slice and the generated text. The paper reports consistent gains over image-only self-supervised baselines on segmentation datasets spanning CT and MRI, and on two disease classification datasets. The central idea is that language descriptions carry clinical semantics that pixel-level reconstruction and contrastive learning alone miss.

What carries the argument

The Plane-Slice-Aware Transformer (PSAT) is the bridge between the 3D volume and the 2D vision-language space. It uses learnable query tokens with a plane-slice position embedding that encodes which orientation and slice index the 2D image came from; the queries attend to the 3D volume features and produce projected features comparable to 2D image and text embeddings. The second piece is mini-batch Partial Optimal Transport (POT): instead of contrastive push-apart, the loss compares an identity coupling to a transport plan between volume and slice/text features, allowing some features to remain unmatched and thereby absorbing noise in LLM-generated descriptions. A reconstruction loss on the

What would settle it

Compare the actual image volumes used for pretraining with the MSD-Liver evaluation cases, e.g., by matching image hash values or patient IDs. If overlap exists, rerun the MSD-Liver evaluation with a provably disjoint pretraining corpus; if the reported Dice advantage disappears or shrinks, the cross-domain transfer claim for liver segmentation is unsupported.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 3D medical image encoder learns more transferable representations when its features are aligned with 2D slice features and with text descriptions of those slices produced by a 2D multimodal large language model. A learnable query transformer, informed by the slice's plane and position within the volume, projects 3D features into the 2D image-text space. The matching loss is a mini-batch partial optimal transport that tolerates noise in generated text. The authors report consistent improvements over image-only self-supervised baselines on eight or more segmentation datasets and on classification of Alzheimer's and Parkinson's disease, interpreting the gains

Load-bearing premise

The whole generalization story rests on the pretraining corpora being disjoint from downstream evaluation, specifically the claim that MSD-Liver is unseen during pretraining; if the LiTS liver data inside 3DSeg-8 overlaps with MSD-Liver, that transfer claim collapses.

Editorial extensions

If this is right

  • If the central claim is correct, medical image pretraining no longer needs curated image-text pairs: generated 2D descriptions from a general multimodal model can serve as supervision for 3D encoders.
  • The alignment recipe is encoder-agnostic: any 3D encoder and any pre-aligned 2D vision-language encoder pair could be plugged in, so gains should extend beyond the specific backbone tested.
  • Because one slice per volume suffices, pretraining cost stays modest while still bringing the semantic advantage of language.
  • Disease classification improves along with segmentation, suggesting the learned features capture pathology-related cues, not just anatomy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's claim that MSD-Liver is unseen during pretraining is load-bearing; if the LiTS liver data inside 3DSeg-8 overlaps with MSD-Liver volumes, the MSD-Liver result is a domain-familiarity test rather than a cross-domain transfer test.
  • The plane-slice position embedding implicitly teaches the model where a slice lives in a volume; this could be turned into an explicit pretraining signal for slice localization or 3D landmark detection.
  • Partial optimal transport's tolerance to noisy text suggests the method could work with lower-quality text sources such as automated radiology reports or weaker multimodal models, possibly with a different performance trade-off.
  • One-slice-per-volume sampling leaves most anatomical context unused; a multi-slice or volume-level text aggregation might improve performance for pathologies only visible in particular planes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. Med3DInsight proposes a pretraining framework for 3D medical image encoders. For each 3D volume in 3DSeg-8 and M3D, the method samples one 2D slice, generates a text description with GPT-4V, and fine-tunes CLIP on the resulting slice-text pairs. A 3D encoder-decoder (nnFormer) is then trained with two objectives: a plane-slice-aware transformer (PSAT) aligns 3D volume features with CLIP image/text features using a mini-batch partial optimal transport (mPOT) loss, and an L2 reconstruction loss preserves low-level detail. The pre-trained encoder is evaluated by fine-tuning on ten segmentation datasets (Table IV) and two classification datasets (Table V), reporting improvements over six self-supervised baselines and several supervised baselines. The central claimed contribution is that language-guided semantic supervision from 2D MLLMs yields transferable 3D representations that outperform vision-only SSL.

Significance. If the results hold, Med3DInsight is a significant practical step: it provides a scalable, annotation-free way to inject semantic and clinical knowledge into 3D medical encoders, and the PSAT module plus mPOT alignment are plausible mechanisms for bridging 3D volume features with 2D vision-language spaces. The paper includes useful ablations (PSAT components, loss terms, alignment strategy, MLLM choice, slice sampling, preprocessing, data efficiency) and reports consistent gains across CT/MRI, segmentation, and classification. However, the headline 'unseen transfer' claim depends on the pretraining corpora being disjoint from the downstream evaluation datasets; this is questionable for MSD-Liver and, if confirmed, would inflate a portion of the reported average improvement. The mPOT formulation is also underspecified in ways that affect reproducibility. With those two issues resolved, the contribution would be solid.

major comments (2)
  1. [Section IV.B, Table IV(j), Table I] The paper states that MSD-Heart, MSD-Liver, and MSD-Colon are 'unseen during pretraining' and explains that overlapping MSD tasks (Hippocampus, Prostate, Pancreas, Vessel, Spleen) were excluded from evaluation. However, 3DSeg-8 is the Med3D collection [1], whose liver subset is widely understood to be derived from LiTS; Table III lists both MSD-Liver and LiTs with 131 CT scans, and Table I counts 6,684 liver samples in the pretraining set. If LiTS-derived volumes appear in 3DSeg-8, then MSD-Liver is not unseen, and the 88.41 vs. 87.25 Dice gain in Table IV(j) is at least partly a domain-familiarity effect rather than cross-domain transfer. The paper's own enumeration of overlaps omits liver, which is internally inconsistent with the 'unseen' claim. This is load-bearing for the sentence in Section IV-C that Med3DInsight 'consistently outperforms SOTA SSL methods over eight datasets.' The
  2. [Section III-D.1, Eq. (2), Algorithm 1] The mPOT loss is not well-defined as written. The empirical plan \hat\pi_b is a b×b matrix with \hat\pi_ii = 1/b and zero off-diagonal, so it has total mass 1, while \pi^{X,S}_{s,\epsilon,c} is supposed to transport only mass s. A KL divergence between a probability measure and a measure of mass s is undefined unless s=1 or the two arguments are normalized. Algorithm 1 never states how s is set or how the constraint \Pi_s is enforced; the call 'Bregman(s, C, \epsilon)' is not a standard interface for partial OT. The Mahalanobis matrix M is updated with 'Proj' but the feasible set (e.g., PSD, fixed trace, bounded entries) is not given. These details are necessary to reproduce Table VI(c), where mPOT is claimed to outperform contrastive learning.
minor comments (5)
  1. [Section IV-C and Table IV] The text says 'eight datasets' and Figure 3(a) says 'eight different 3D medical image segmentation datasets,' but Table IV reports ten downstream segmentation panels (MM-WHS, CHAOS, OASIS1, MSD-Heart, ADNI, VS, AbdomenCT-1K, LiTs, MSD-Liver, MSD-Colon), and Table IV(a) is titled 'Average over 10 datasets.' The count should be made consistent.
  2. [Section IV-C] The sentence 'All SSL methods employed identical fine-tuning procedures, including pretraining hyperparameters (learning rate, batch size, training epochs) pretrain datasets, and evaluation metrics' is grammatically ambiguous. It should state exactly which pretraining datasets and hyperparameters are shared across baselines and which are method-specific; otherwise the claim of fair comparison is hard to verify.
  3. [Eq. (1)] The double sum is written as \sum_{i=1}^{nX} \sum_{j=1}^{nX}, but the second index should presumably range over the target sample count (nY or nS/nT). This is a notation error that makes the transport plan dimensions unclear.
  4. [Table VI(f)] The row for '1' slice per volume appears to have a formatting issue (the value 92.07 is merged with adjacent columns). Please reformat for readability.
  5. [Figure 3(d)] The modality-gap numbers (e.g., 82.48 vs. 76.07, 15.77 vs. 12.98) are reported without confidence intervals or an explanation of how many random samples/restarts were used; since t-SNE and UMAP projections are stochastic, a small repeated-run analysis would strengthen this visualization claim.

Circularity Check

2 steps flagged · score 6.0 of 10

MSD-Liver is claimed 'unseen' despite liver volumes in pretraining Table I, and the same LiTS data is evaluated twice as MSD-Liver and LiTs, inflating headline gains.

  1. fitted input called prediction [Section IV.B (Downstream Tasks and Corresponding Datasets); Table I]
    "The 3DSeg-8 dataset [1], used for pretraining, includes five tasks that overlap with the MSD benchmark (Hippocampus, Prostate, Pancreas, Vessel, and Spleen). To eliminate domain familiarity and prevent potential data leakage, we exclude these overlapping tasks during evaluation. Instead, we select the Heart, Liver, and Colon tasks from MSD, which are unseen during pretraining, as the downstream evaluation target. This design ensures that performance reflects true cross-domain generalization rather than memorization effects from the pre-training data."

    The pretraining corpus 3DSeg-8 is the Med3D collection, which includes LiTS-derived liver volumes; the paper's own Table I lists Liver (6,684) among pretraining internal organs. MSD-Liver is the same LiTS data (131 CT scans). Therefore the Section IV.B assertion that MSD-Liver is 'unseen during pretraining' is false. The MSD-Liver row of Table IV is an in-distribution test: the reported 88.41 vs. 87.25 Dice advantage over vox2vec can be attributed to memorization/domain familiarity rather than cross-domain transfer. The claimed generalization result for this dataset reduces, by construction, to having included the evaluation volumes in the training distribution.

  2. renaming known result [Table III; Table IV rows (i) and (j)]
    "MSD-Liver [62] CT 3 131 Liver, Kidneys, Spleen ... LiTs [68] CT 2 131 Liver, Liver Tumor"

    MSD-Liver is a re-packaging of the LiTS dataset, and both Table III rows report the identical scan count (131). The paper evaluates 'MSD-Liver' and 'LiTs' as separate datasets in Table IV and then uses both when computing the average Dice/HD95 gain 'over eight datasets'. This double-counts the same 131 volumes, so the headline improvement of over 1% average Dice is not measured over eight independent datasets; it is measured over at most seven, with LiTS contributing twice. The 'eight datasets' claim is thus inflated by renaming the same known dataset and treating it as a distinct result.

full rationale

The paper is an empirical systems paper; its pretraining objective Ltotal = LmPOT + Lres is self-contained and no formal derivation becomes circular by construction. The circularity lies in the evaluation design. Section IV.B asserts that MSD-Liver is 'unseen during pretraining,' but Table I lists Liver among the pretraining internal organs with 6,684 triplets from 3DSeg-8 (Med3D), and MSD-Liver is the LiTS dataset (131 volumes) already present in that corpus. Consequently, the MSD-Liver row in Table IV is not a cross-domain transfer prediction but an in-distribution familiarity/memorization test, so that portion of the headline 'unseen generalization' gain reduces to a training-input overlap. Additionally, Table III lists MSD-Liver and LiTs with the same 131 scans, and Table IV evaluates both as separate datasets, double-counting LiTS in the reported 'over eight datasets' average. These issues are concrete and quoteable; they undermine the central quantitative claim. No load-bearing self-citation, uniqueness import, or ansatz smuggling was found; the methodology itself is not definitionally circular. Overall, partial circularity via fitted/contaminated evaluation gives score 6.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The ledger is dominated by evaluation-setup assumptions and unreported hyperparameters rather than mathematical axioms. The most consequential entry is the disjointness assumption, which is contradicted by known dataset composition. No new physical or ontological entities are postulated; PSAT and position embeddings are architectural components, not invented entities.

free parameters (4)
  • s (transported mass fraction) = not reported
    In Eq. 1 and Algorithm 1, partial OT requires the mass s to transport. The paper never gives a value, and the KL loss in Eq. 2 compares a full-mass diagonal plan with a partial plan, so the meaning of the loss depends on this unreported parameter.
  • epsilon (entropic regularization coefficient) = not reported
    The entropy regularizer in Eq. 1 controls plan smoothness and must be chosen in advance. No value is reported and no sensitivity analysis is provided.
  • Q (number of learnable queries in PSAT) = 300
    Chosen as 'a common setting' in Section IV-A; no ablation over query count is reported, so it is a hand-picked hyperparameter.
  • M (Mahalanobis interaction matrix) = learned during pretraining
    The ground metric matrix M is updated by gradient descent (Algorithm 1), allowing the alignment cost to be fitted to the training data rather than fixed a priori.
assumptions (3)
  • domain assumption GPT-4V captions of 2D slices provide semantically valid supervision for 3D volumes.
    Introduced in Section III.A; the paper relies on MLLM output quality, and its own Table II shows inaccurate captions. POT mitigates but does not remove dependence on caption quality.
  • ad hoc to paper Pretraining datasets (3DSeg-8 and M3D) are disjoint from downstream evaluation datasets, specifically MSD-Liver.
    Stated in Section IV.B as 'MSD-Heart, MSD-Liver, and MSD-Colon are unseen during pretraining.' This appears false because 3DSeg-8 is the Med3D set that includes LiTS liver data, the same source used by MSD-Liver.
  • domain assumption CLIP, after fine-tuning on generated pairs, provides a stable 2D image-text space into which 3D features should be projected.
    Used throughout Section III.C; if the fine-tuned CLIP space does not capture medical semantics, the alignment loss trains toward a noisy target, and the text supervision adds little.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/IOGMM4CY

@misc{pith2026250909064,
  author       = {Pith},
  title        = {Pith review of: Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOGMM4CY}},
  note         = {Machine review of arXiv:2509.09064}
}
read the original abstract

Understanding 3D medical image volumes is critical in the medical field, yet existing 3D medical convolution and transformer-based self-supervised learning (SSL) methods often lack deep semantic comprehension. Recent advancements in multimodal large language models (MLLMs) provide a promising approach to enhance image understanding through text descriptions. To leverage these 2D MLLMs for improved 3D medical image understanding, we propose Med3DInsight, a novel pretraining framework that integrates 3D image encoders with 2D MLLMs via a specially designed plane-slice-aware transformer module. Additionally, our model employs a partial optimal transport based alignment, demonstrating greater tolerance to noise introduced by potential noises in LLM-generated content. Med3DInsight introduces a new paradigm for scalable multimodal 3D medical representation learning without requiring human annotations. Extensive experiments demonstrate our state-of-the-art performance on two downstream tasks, i.e., segmentation and classification, across various public datasets with CT and MRI modalities, outperforming current SSL methods. Med3DInsight can be seamlessly integrated into existing 3D medical image understanding networks, potentially enhancing their performance. Our source code, generated datasets, and pre-trained models will be available at https://github.com/Qybc/Med3DInsight.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

105 extracted references · 17 linked inside Pith

  1. [1]

    A classical technique in SSL involves image restoration to learn visual features by reconstructing corrupted images back to their original state [6], [8], [26], [27]

    Reconstruction-based Methods:Self-supervised learning (SSL) has significantly advanced computer vision by utilizing pretraining tasks that do not require annotations. A classical technique in SSL involves image restoration to learn visual features by reconstructing corrupted images back to their original state [6], [8], [26], [27]. Some of these approache...

  2. [2]

    Describe the image in fewer than 100 words

    Alignment-based Methods:Recently, contrastive learning has emerged as the standard method in SSL [11], [12]. To address the limitations of traditional contrastive learning, such as the requirement for large batch sizes and strong augmentations [29], BYOL [11] and BarlowTwins [12] em- ploy a dual-branch structure to align the embeddings of two augmented im...

  3. [3]

    Alignment Loss:Contrastive learning, as used in CLIP [24], offers an efficient self-learning technique by bring- ing the representations of paired data closer together in the 6 IEEE TRANSACTIONS AND JOURNALS TEMPLA TE embedding space while pushing apart the representations of non-paired data. However, the use of large language models inevitably leads to s...

  4. [4]

    Reconstruction Loss:To enhance visual feature learning with low-level visual semantics, we introduce an auxiliary pretext task, image restoration, which aims to reconstruct the 3D medical volume using latent visual encoders. The training objective for this task is formulated using the L2-norm below: Lres =E (X b) bX i=1 ||xi −ˆxi||2.(3) Input:Initialized ...

  5. [5]

    Total Loss:With a synergy of triplet alignment learning and image reconstruction, the overall pretraining objective of our Med3DInsight framework is represented as follows: Ltotal =L mPOT +L res (4) After pretraining, the resulting 3D image encoder serves as a visual backbone for downstream task-specific fine-tuning. IV. EXPERIMENTS A. Pretraining Setup a...

  6. [6]

    All pretraining experiments are implemented on four NVIDIA GeForce RTX 3090 GPUs

    The training is set for 300 epochs to ensure convergence. All pretraining experiments are implemented on four NVIDIA GeForce RTX 3090 GPUs. B. Downstream Tasks and Corresponding Datasets We evaluate the effectiveness of our method across a diverse set of 3D medical imaging tasks, varying in size, objectives, and modalities. Specifically, we evaluate the r...

  7. [7]

    Multi-Organs or Tumors Segmentation:For the 3D seg- mentation task, we implement this task that includes car- diac structure segmentation (MM-WHS [61] and MSD- Heart [62]), abdominal organ segmentation (MSD-Liver [62], CHAOS [63], and AbdomenCT-1K [64]), brain segmentation (OASIS1 [65] and ADNI [66]), and tumor segmentation (MSD-Colon [62], VS [67], LiTs ...

  8. [8]

    Each MRI scan’s image size is224×224×224, and the voxel spacing is 1.75 mm

    Disease Classification:For Alzheimer’s disease classi- fication, we use the OASIS2 dataset [69], consisting of 312 T1-weighted structural MRI scans collected from 135 subjects, including both AD subjects and healthy volunteers. Each MRI scan’s image size is224×224×224, and the voxel spacing is 1.75 mm. For Parkinson’s disease classification, the PPMI data...

Show all 105 references
  1. [9]

    Our Med3DInsight con- sistently outperforms SOTA SSL methods over eight datasets, surpassing the best baseline by over 1% in average Dice score and 0.7mm in average HD95

    Experimental Results for 3D Segmentation:Table IV showcases the experimental results of multi-organ and sub- structure segmentation across the various datasets, which in- clude 19 different organs or tumors. Our Med3DInsight con- sistently outperforms SOTA SSL methods over eig...

  2. [10]

    The OASIS2 dataset presents the challenge of discerning Alzheimer’s disease, and the PPMI dataset presents the challenge of discerning Parkinson’s disease

    Experimental Results for 3D Classification:Table V re- ports the 3D classification results on the OASIS2 dataset and PPMI dataset. The OASIS2 dataset presents the challenge of discerning Alzheimer’s disease, and the PPMI dataset presents the challenge of discerning Parkinson’s...

  3. [11]

    In Table VI(a),Ex1represents the model without any query transformer or place-slice position embedding

    Effectiveness of the PSAT Model:To demonstrate the effectiveness of each component in PSAT, we conduct an ablation study on the OASIS1 3D segmentation task. In Table VI(a),Ex1represents the model without any query transformer or place-slice position embedding. In this scenario...

  4. [12]

    As depicted in Table VI(b), excluding eitherL mPOT orL res leads to reduced performance

    Loss Ablation:We further investigate the influence of different losses during the pre-training stage. As depicted in Table VI(b), excluding eitherL mPOT orL res leads to reduced performance. However, when combining all components as in our method, we achieve the highest perfor...

  5. [13]

    We report dice scores for contrastive loss and the mini-batch partial optimal transport loss in Ta- ble VI(c)

    Alignment Strategy:We conduct experiments on the OASIS1 test set to examine the transferability of the learned joint embedding space. We report dice scores for contrastive loss and the mini-batch partial optimal transport loss in Ta- ble VI(c). The mPOT loss acquires the highe...

  6. [14]

    Vision/Text Encoder:To assess the versatility of our method, we employ two distinct 2D encoders: the CLIP [24] and the BioMedCLIP [76]. As shown in Table VI(d), the performance differences between these encoders are marginal, underscoring that our method is agnostic to these t...

  7. [15]

    Ablation of different image backbones:As seen in Fig- ure 3(b), we employ two distinct image backbones UNETR [4] and nnFormer [5]. Med3DInsight consistently improves differ- ent backbones across three datasets, indicating it is a compre- hensive representation learning framewo...

  8. [16]

    in which the GPT-4V we use have more accurate descriptions

    Ablation of different multimodal large language models: We conduct a qualitative comparison of the generation quality across three MLLMs, as shown in Table VII, including GPT- 4Vision and Claude, which are general-purpose models, and HuatuoGPT-Vision, which is specifically des...

    1994

  9. [17]

    Different Slice-per-volume Strategies:To validate our design choice, we conduct ablation experiments comparing different slice-per-volume sampling strategies during pretrain- ing. Specifically, we test a setting using only the M3D subset (8,000 volumes) with three slices per v...

  10. [18]

    As shown in Table VI(g), Sliding window achieves slightly higher accuracy for targets with rich edge details, but resize provided faster inference with comparable Dice scores

    Different Preprocess Types:To evaluate the impact of different image preprocessing strategies, we conduct addi- tional experiments comparing two approaches: (1) Isotropic resampling (i.e., resizing) of the entire volume to a fixed 1283 resolution, as we did; (2) Sliding-window...

  11. [19]

    To validate the data efficiency of our Med3DInsight framework, we check the average dice changes with varying amounts of fine-tuning samples

    Data Efficiency:Model pretraining can potentially re- duce the demand for labeled data in downstream tasks. To validate the data efficiency of our Med3DInsight framework, we check the average dice changes with varying amounts of fine-tuning samples. Our method consistently out...

  12. [20]

    multiple hypodense lesions

    Latent Representation of Pretrained Model:We utilize the pretrained visual encoder to extract latent representations from 3D medical volume images of the brain, abdomen, and chest. We then apply the t-SNE algorithm [80] and UMAP [81] for dimensionality reduction and visualizat...

  13. [21]

    Med3d: Transfer learning for 3d medical image analysis,

    S. Chen, K. Ma, and Y . Zheng, “Med3d: Transfer learning for 3d medical image analysis,”arXiv preprint arXiv:1904.00625, 2019

    1904 arXiv

  14. [22]

    M3t: three-dimensional medical image classifier using multi-plane and multi-slice transformer,

    J. Jang and D. Hwang, “M3t: three-dimensional medical image classifier using multi-plane and multi-slice transformer,” inProceedings of the IEEE/CVF conference on CVPR, 2022, pp. 20 718–20 729

    2022

  15. [23]

    Universeg: Universal medical image segmentation,

    V . I. Butoi, J. J. G. Ortiz, T. Ma, M. R. Sabuncu, J. Guttag, and A. V . Dalca, “Universeg: Universal medical image segmentation,” in Proceedings of the IEEE/CVF ICCV, 2023, pp. 21 438–21 451

    2023

  16. [24]

    Unetr: Transformers for 3d medical image segmentation,

    A. Hatamizadeh, Y . Tang, V . Nath, D. Yang, A. Myronenko, B. Land- man, H. R. Roth, and D. Xu, “Unetr: Transformers for 3d medical image segmentation,” inIEEE/CVF WACV, 2022, pp. 574–584

    2022

  17. [25]

    nn- former: V olumetric medical image segmentation via a 3d transformer,

    H.-Y . Zhou, J. Guo, Y . Zhang, X. Han, L. Yu, L. Wang, and Y . Yu, “nn- former: V olumetric medical image segmentation via a 3d transformer,” IEEE Transactions on Image Processing, 2023

    2023

  18. [26]

    Simmim: A simple framework for masked image modeling,

    Z. Xie, Z. Zhang, Y . Cao, Y . Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: A simple framework for masked image modeling,” in Proceedings of the IEEE/CVF CVPR, 2022, pp. 9653–9663

    2022

  19. [27]

    Models genesis: Generic autodidactic models for 3d medical image analysis,

    Z. Zhou, V . Sodha, M. M. Rahman Siddiquee, R. Feng, N. Tajbakhsh, M. B. Gotway, and J. Liang, “Models genesis: Generic autodidactic models for 3d medical image analysis,” inMICCAI 2019, Part IV 22. Springer, 2019, pp. 384–393

    2019

  20. [28]

    Masked au- toencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked au- toencoders are scalable vision learners,” inProceedings of the IEEE/CVF Conference on CVPR, 2022, pp. 16 000–16 009

    2022

  21. [29]

    Masked image modeling advances 3d medical image anal- ysis,

    Z. Chen, D. Agarwal, K. Aggarwal, W. Safta, M. M. Balan, and K. Brown, “Masked image modeling advances 3d medical image anal- ysis,” inProceedings of the IEEE/CVF WACV, 2023, pp. 1970–1980

    2023

  22. [30]

    Self pre- training with masked autoencoders for medical image classification and segmentation,

    L. Zhou, H. Liu, J. Bae, J. He, D. Samaras, and P. Prasanna, “Self pre- training with masked autoencoders for medical image classification and segmentation,” inISBI. IEEE, 2023, pp. 1–6

    2023

  23. [31]

    Bootstrap your own latent-a new approach to self-supervised learning,

    J.-B. Grill, F. Strub, F. Altch ´e, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pireset al., “Bootstrap your own latent-a new approach to self-supervised learning,”Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020

    2020

  24. [32]

    Barlow twins: Self-supervised learning via redundancy reduction,

    J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” inInternational conference on machine learning. PMLR, 2021, pp. 12 310–12 320

    2021

  25. [33]

    Exploring simple siamese representation learning,

    X. Chen and K. He, “Exploring simple siamese representation learning,” inIEEE/CVF CVPR, 2021, pp. 15 750–15 758

    2021

  26. [34]

    A unified visual information preservation framework for self-supervised pre-training in medical image analysis,

    H.-Y . Zhou, C. Lu, C. Chen, S. Yang, and Y . Yu, “A unified visual information preservation framework for self-supervised pre-training in medical image analysis,”IEEE TPAMI, 2023

    2023

  27. [35]

    vox2vec: A framework for self-supervised contrastive learning of voxel- level representations in medical images,

    M. Goncharov, V . Soboleva, A. Kurmukov, M. Pisov, and M. Belyaev, “vox2vec: A framework for self-supervised contrastive learning of voxel- level representations in medical images,” inMICCAI, 2023, pp. 605–614

    2023

  28. [36]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,”arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  29. [37]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynoldset al., “Flamingo: a visual language model for few-shot learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  30. [38]

    Gpt-4v(ision) system card,

    OpenAI, “Gpt-4v(ision) system card,” 2023

    2023

  31. [39]

    Medblip: Bootstrapping language-image pre-training from 3d medical images and texts,

    Q. Chen, X. Hu, Z. Wang, and Y . Hong, “Medblip: Bootstrapping language-image pre-training from 3d medical images and texts,”arXiv preprint arXiv:2305.10799, 2023

    2023 arXiv

  32. [40]

    Gen- erative text-guided 3d vision-language pretraining for unified medical image segmentation,

    Y . Chen, C. Liu, W. Huang, S. Cheng, R. Arcucci, and Z. Xiong, “Gen- erative text-guided 3d vision-language pretraining for unified medical image segmentation,”arXiv preprint arXiv:2306.04811, 2023

    2023 arXiv

  33. [41]

    Performance of multimodal gpt- 4v on usmle with image: Potential for imaging diagnostic support with explanations,

    Z. Yang, Z. Yao, M. Tasmin, P. Vashisht, W. S. Jang, F. Ouyang, B. Wang, D. Berlowitz, and H. Yu, “Performance of multimodal gpt- 4v on usmle with image: Potential for imaging diagnostic support with explanations,”medRxiv, pp. 2023–10, 2023

    2023

  34. [42]

    Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis,

    C. Wu, J. Lei, Q. Zheng, W. Zhao, W. Lin, X. Zhanget al., “Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis,”arXiv preprint arXiv:2310.09909, 2023

    2023 arXiv

  35. [43]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in ECCV. Springer, 2020, pp. 213–229

    2020

  36. [44]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askellet al., “Learning transferable visual models from natural language supervision,” inICML, 2021, pp. 8748–8763

    2021

  37. [45]

    Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding,

    L. Xue, M. Gao, C. Xing, R. Mart ´ın-Mart´ın, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding,” inPro- ceedings of the IEEE/CVF Conference on CVPR, 2023, pp. 1179–1189

    2023

  38. [46]

    Masked feature prediction for self-supervised visual pre-training,

    C. Wei, H. Fan, S. Xie, C.-Y . Wu, A. Yuille, and C. Feichtenhofer, “Masked feature prediction for self-supervised visual pre-training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 668–14 678

    2022

  39. [47]

    Unsupervised repre- sentation learning by predicting image rotations,

    S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised repre- sentation learning by predicting image rotations,”arXiv preprint arXiv:1803.07728, 2018

    2018 arXiv

  40. [48]

    Self-supervised learning for medical image analysis using image context restoration,

    L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, and D. Rueck- ert, “Self-supervised learning for medical image analysis using image context restoration,”Medical image analysis, vol. 58, p. 101539, 2019

    2019

  41. [49]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” inProceedings of the IEEE/CVF Conference on CVPR, 2020, pp. 9729–9738

    2020

  42. [50]

    Contrastive learning of medical visual representations from paired images and text,

    Y . Zhang, H. Jiang, Y . Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” inMachine Learning for Healthcare Conference. PMLR, 2022, pp. 2–25

    2022

  43. [51]

    Medclip: Contrastive learning from unpaired medical images and text,

    Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Contrastive learning from unpaired medical images and text,”arXiv preprint arXiv:2210.10163, 2022

    2022 arXiv

  44. [52]

    Gloria: A multimodal global-local representation learning framework for label- efficient medical image recognition,

    S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, “Gloria: A multimodal global-local representation learning framework for label- efficient medical image recognition,” inIEEE/CVF ICCV, 2021, pp. 3942–3951

    2021

  45. [53]

    Uniseg: A prompt- driven universal segmentation model as well as a strong representation learner,

    Y . Ye, Y . Xie, J. Zhang, Z. Chen, and Y . Xia, “Uniseg: A prompt- driven universal segmentation model as well as a strong representation learner,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 508–518

    2023

  46. [54]

    Towards generalist foundation model for radiology,

    C. Wu, X. Zhang, Y . Zhang, Y . Wang, and W. Xie, “Towards generalist foundation model for radiology,”arXiv:2308.02463, 2023

    2023 arXiv

  47. [55]

    M3d: Advancing 3d medical image analysis with multi-modal large language models,

    F. Bai, Y . Du, T. Huang, M. Q.-H. Meng, and B. Zhao, “M3d: Advancing 3d medical image analysis with multi-modal large language models,” arXiv preprint arXiv:2404.00578, 2024

    2024 arXiv

  48. [56]

    Utilizing synthetic data for medical vision-language pre-training: Bypassing the need for real images,

    C. Liu, A. Shah, W. Bai, and R. Arcucci, “Utilizing synthetic data for medical vision-language pre-training: Bypassing the need for real images,”arXiv preprint arXiv:2310.07027, 2023

    2023 arXiv

  49. [57]

    Roentgen: Vision-language foundation model for chest x-ray generation,

    P. Chambon, C. Bluethgen, J.-B. Delbrouck, R. Van der Sluijs, M. Połacin, J. M. Z. Chaves, T. M. Abraham, S. Purohit, C. P. Langlotz, and A. Chaudhari, “Roentgen: Vision-language foundation model for chest x-ray generation,”arXiv preprint arXiv:2211.12737, 2022

    2022 arXiv

  50. [58]

    Medimp: 3d medical images with 14 IEEE TRANSACTIONS AND JOURNALS TEMPLA TE clinical prompts from limited tabular data for renal transplantation,

    L. Milecki, V . Kalogeiton, S. Bodard, D. Anglicheau, J.-M. Correas, M.-O. Timsit, and M. Vakalopoulou, “Medimp: 3d medical images with 14 IEEE TRANSACTIONS AND JOURNALS TEMPLA TE clinical prompts from limited tabular data for renal transplantation,” in Medical Imaging with De...

    2023

  51. [59]

    Joint distribution optimal transportation for domain adaptation,

    N. Courty, R. Flamary, A. Habrard, and A. Rakotomamonjy, “Joint distribution optimal transportation for domain adaptation,”Advances in neural information processing systems, vol. 30, 2017

    2017

  52. [60]

    Disentangle then calibrate with gradient guidance: A unified framework for common and rare disease diagnosis,

    Y . Chen, X. Guo, Y . Xia, and Y . Yuan, “Disentangle then calibrate with gradient guidance: A unified framework for common and rare disease diagnosis,”IEEE Transactions on Medical Imaging, 2024

    2024

  53. [61]

    Spatial and modal optimal transport for fast cross-modal mri reconstruction,

    Q. Wang, Z. Wen, J. Shi, Q. Wang, D. Shen, and S. Ying, “Spatial and modal optimal transport for fast cross-modal mri reconstruction,”IEEE Transactions on Medical Imaging, 2024

    2024

  54. [62]

    Uniter: Universal image-text representation learning,

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” inECCV. Springer, 2020, pp. 104–120

    2020

  55. [63]

    Weakly-supervised temporal ac- tion alignment driven by unbalanced spectral fused gromov-wasserstein distance,

    D. Luo, Y . Wang, A. Yue, and H. Xu, “Weakly-supervised temporal ac- tion alignment driven by unbalanced spectral fused gromov-wasserstein distance,” in30th ACM International Conference on Multimedia, 2022, pp. 728–739

    2022

  56. [64]

    Self-supervised video summarization guided by semantic inverse optimal transport,

    Y . Wang, H. Xu, and D. Luo, “Self-supervised video summarization guided by semantic inverse optimal transport,” inProceedings of 31st ACM International Conference on Multimedia, 2023, pp. 6611–6622

    2023

  57. [65]

    Partial unbalanced feature transport for cross-modality cardiac image segmentation,

    S. Dong, Z. Pan, Y . Fu, D. Xu, K. Shi, Q. Yang, Y . Shi, and C. Zhuo, “Partial unbalanced feature transport for cross-modality cardiac image segmentation,”IEEE Transactions on Medical Imaging, 2023

    2023

  58. [66]

    Prototype correlation matching and class-relation reasoning for few-shot medical image segmentation,

    Y . Zhang, H. Li, Y . Gao, H. Duan, Y . Huang, and Y . Zheng, “Prototype correlation matching and class-relation reasoning for few-shot medical image segmentation,”IEEE Transactions on Medical Imaging, 2024

    2024

  59. [67]

    Learning to match via inverse optimal transport,

    R. Li, X. Ye, H. Zhou, and H. Zha, “Learning to match via inverse optimal transport,”Journal of machine learning research, vol. 20, no. 80, pp. 1–37, 2019

    2019

  60. [68]

    Inverse optimal transport,

    A. M. Stuart and M.-T. Wolfram, “Inverse optimal transport,”SIAM Journal on Applied Mathematics, vol. 80, no. 1, pp. 599–619, 2020

    2020

  61. [69]

    Data efficient language-supervised zero-shot recognition with optimal transport distillation,

    B. Wu, R. Cheng, P. Zhang, T. Gao, P. Vajda, and J. E. Gonzalez, “Data efficient language-supervised zero-shot recognition with optimal transport distillation,”arXiv preprint arXiv:2112.09445, 2021

    2021 arXiv

  62. [70]

    A computational fluid mechanics solution to the monge-kantorovich mass transfer problem,

    J.-D. Benamou and Y . Brenier, “A computational fluid mechanics solution to the monge-kantorovich mass transfer problem,”Numerische Mathematik, vol. 84, no. 3, pp. 375–393, 2000

    2000

  63. [71]

    Sinkhorn distances: Lightspeed computation of optimal transport,

    M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,”Advances in neural information processing systems, vol. 26, 2013

    2013

  64. [72]

    Revisiting deep audio-text retrieval through the lens of transportation,

    M. Luong, K. Nguyen, N. Ho, R. Haf, D. Phung, and L. Qu, “Revisiting deep audio-text retrieval through the lens of transportation,”arXiv preprint arXiv:2405.10084, 2024

    2024 arXiv

  65. [73]

    The ma- halanobis distance,

    R. De Maesschalck, D. Jouan-Rimbaud, and D. L. Massart, “The ma- halanobis distance,”Chemometrics and intelligent laboratory systems, vol. 50, no. 1, pp. 1–18, 2000

    2000

  66. [74]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  67. [75]

    Computational optimal transport: With applications to data science,

    G. Peyr ´e, M. Cuturiet al., “Computational optimal transport: With applications to data science,”Foundations and Trends® in Machine Learning, vol. 11, no. 5-6, pp. 355–607, 2019

    2019

  68. [76]

    Scaling algorithms for unbalanced optimal transport problems,

    L. Chizat, G. Peyr ´e, B. Schmitzer, and F.-X. Vialard, “Scaling algorithms for unbalanced optimal transport problems,”Mathematics of Computa- tion, vol. 87, no. 314, pp. 2563–2609, 2018

    2018

  69. [77]

    Iterative bregman projections for regularized transportation problems,

    J.-D. Benamou, G. Carlier, M. Cuturi, L. Nenna, and G. Peyr ´e, “Iterative bregman projections for regularized transportation problems,”SIAM Journal on Scientific Computing, vol. 37, no. 2, pp. A1111–A1138, 2015

    2015

  70. [78]

    Ground metric learning,

    M. Cuturi and D. Avis, “Ground metric learning,”The Journal of Machine Learning Research, vol. 15, no. 1, pp. 533–564, 2014

    2014

  71. [79]

    Euclidean distance mapping,

    P.-E. Danielsson, “Euclidean distance mapping,”Computer Graphics and image processing, vol. 14, no. 3, pp. 227–248, 1980

    1980

  72. [80]

    A swiss army knife for minimax optimal transport,

    S. Dhouib, I. Redko, T. Kerdoncuff, R. Emonet, and M. Sebban, “A swiss army knife for minimax optimal transport,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 2504–2513

    2020

  73. [81]

    Multi-scale patch and multi-modality atlases for whole heart segmentation of mri,

    X. Zhuang and J. Shen, “Multi-scale patch and multi-modality atlases for whole heart segmentation of mri,”Medical image analysis, vol. 31, pp. 77–87, 2016

    2016

  74. [82]

    The medical segmentation decathlon,

    M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menzeet al., “The medical segmentation decathlon,”Nature communications, vol. 13, no. 1, p. 4128, 2022

    2022

  75. [83]

    Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation,

    A. E. Kavur, N. S. Gezer, M. Barıs ¸, S. Aslan, P.-H. Conze, V . Groza et al., “Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation,”Medical Image Analysis, vol. 69, p. 101950, 2021

    2021

  76. [84]

    Abdomenct-1k: Is abdominal organ segmen- tation a solved problem?

    J. Ma, Y . Zhang, S. Gu, C. Zhu, C. Ge, Y . Zhang, X. An, C. Wang, Q. Wang, X. Liuet al., “Abdomenct-1k: Is abdominal organ segmen- tation a solved problem?”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6695–6714, 2021

    2021

  77. [85]

    Open access series of imaging studies (oasis): cross- sectional mri data in young, middle aged, nondemented, and demented older adults,

    D. S. Marcus, T. H. Wang, J. Parker, J. G. Csernansky, J. C. Morris, and R. L. Buckner, “Open access series of imaging studies (oasis): cross- sectional mri data in young, middle aged, nondemented, and demented older adults,”Journal of cognitive neuroscience, vol. 19, no. 9, p...

    2007

  78. [86]

    The alzheimer’s disease neuroimaging initiative,

    S. G. Mueller, M. W. Weiner, L. J. Thal, R. C. Petersen, C. Jack, W. Jagust, J. Q. Trojanowski, A. W. Toga, and L. Beckett, “The alzheimer’s disease neuroimaging initiative,”Neuroimaging Clinics of North America, vol. 15, no. 4, p. 869, 2005

    2005

  79. [87]

    Segmentation of vestibular schwannoma from mri, an open annotated dataset and baseline algorithm,

    J. Shapey, A. Kujawa, R. Dorent, G. Wang, A. Dimitriadis, D. Gr- ishchuk, I. Paddick, N. Kitchen, R. Bradfordet al., “Segmentation of vestibular schwannoma from mri, an open annotated dataset and baseline algorithm,”Scientific Data, vol. 8, no. 1, p. 286, 2021

    2021

  80. [88]

    The liver tumor segmentation benchmark (lits),

    P. Bilic, P. Christ, H. B. Li, E. V orontsov, A. Ben-Cohen, G. Kaissis, A. Szeskin, C. Jacobset al., “The liver tumor segmentation benchmark (lits),”Medical Image Analysis, vol. 84, p. 102680, 2023

    2023

  81. [89]

    Open access series of imaging studies: longitudinal mri data in nondemented and demented older adults,

    D. S. Marcus, A. F. Fotenos, J. G. Csernansky, J. C. Morris, and R. L. Buckner, “Open access series of imaging studies: longitudinal mri data in nondemented and demented older adults,”Journal of cognitive neuroscience, vol. 22, no. 12, pp. 2677–2684, 2010

    2010

  82. [90]

    The parkinson progression marker initiative (ppmi),

    K. Marek, D. Jennings, S. Lasch, A. Siderowf, C. Tanner, T. Simuni, C. Coffey, K. Kieburtz, E. Flagg, S. Chowdhuryet al., “The parkinson progression marker initiative (ppmi),”Progress in neurobiology, vol. 95, no. 4, pp. 629–635, 2011

    2011

  83. [91]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inMedical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III

    2015

  84. [92]

    Springer, 2015, pp. 234–241

    2015

  85. [93]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,

    F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,”Nature methods, vol. 18, no. 2, pp. 203–211, 2021

    2021

  86. [94]

    Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images,

    A. Hatamizadeh, V . Nath, Y . Tang, D. Yang, H. R. Roth, and D. Xu, “Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images,” inInternational MICCAI brainlesion workshop. Springer, 2021, pp. 272–284

    2021

  87. [95]

    Contrastive learning of relative position regression for one-shot ob- ject localization in 3d medical images,

    W. Lei, W. Xu, R. Gu, H. Fu, S. Zhang, S. Zhang, and G. Wang, “Contrastive learning of relative position regression for one-shot ob- ject localization in 3d medical images,” inMedical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference...

    2021

  88. [96]

    V oco: A simple-yet-effective volume contrastive learning framework for 3d medical image analysis,

    L. Wu, J. Zhuang, and H. Chen, “V oco: A simple-yet-effective volume contrastive learning framework for 3d medical image analysis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 873–22 882

    2024

  89. [97]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs,

    S. Zhang, Y . Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluriet al., “Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs,”arXiv preprint arXiv:2303.00915, 2023

    2023 arXiv

  90. [98]

    The claude 3 model family: Opus, sonnet, haiku

    “The claude 3 model family: Opus, sonnet, haiku.” [Online]. Available: https://api.semanticscholar.org/CorpusID:268232499

  91. [99]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale,

    J. Chen, R. Ouyang, A. Gao, S. Chen, G. H. Chen, X. Wang, R. Zhang, Z. Cai, K. Ji, G. Yu, X. Wan, and B. Wang, “Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale,”

  92. [101]

    Monai: An open-source frame- work for deep learning in healthcare,

    M. J. Cardoso, W. Li, R. Brown, N. Ma, E. Kerfoot, Y . Wang, B. Murrey, A. Myronenko, C. Zhao, D. Yanget al., “Monai: An open-source frame- work for deep learning in healthcare,”arXiv preprint arXiv:2211.02701, 2022

    2022 arXiv

  93. [102]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.”Journal of machine learning research, vol. 9, no. 11, 2008

    2008

  94. [103]

    Dimensionality reduction for visualizing single-cell data using umap,

    E. Becht, L. McInnes, J. Healy, C.-A. Dutertre, I. W. Kwok, L. G. Ng, F. Ginhoux, and E. W. Newell, “Dimensionality reduction for visualizing single-cell data using umap,”Nature biotechnology, vol. 37, no. 1, pp. 38–44, 2019

    2019

  95. [104]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,

    V . W. Liang, Y . Zhang, Y . Kwon, S. Yeung, and J. Y . Zou, “Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 17 612–17 625, 2022

    2022

  96. [105]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  97. [2024]

    Available: https://arxiv.org/abs/2406.19280

    [Online]. Available: https://arxiv.org/abs/2406.19280

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.