Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models

Asbj{\o}rn Munk; Jakob Krogh Petersen; Mads Nielsen; Valdemar Licht

REVIEW 4 major objections 5 minor 24 references

Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CLIP-style alignment of 3D brain MRI and tabular data is feasible with only 62 MRI scans, using a domain-specific 3D foundation model encoder and cross-batch embedding accumulation to stabilize the contrastive loss.

desk verdict The embedding-accumulation trick and native-3D CLIP framing are useful, but the scan-level split undermines the headline '62 scans' claim until a patient-level split is shown. read the letter →

arxiv 2501.14051 v1 pith:VKMPMZFB submitted 2025-01-23 cs.CV cs.AIcs.LG

Jakob Krogh Petersen , Valdemar Licht , Mads Nielsen , Asbj{o}rn Munk This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords CLIPmulti-modalalignment3DMRItabulardatafoundationmodelscontrastivelearningzero-shotclassificationembeddingaccumulation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most CLIP-style alignment assumes abundant data, but 3D medical imaging rarely has it. This paper claims that modality alignment between 3D brain MRI and tabular clinical data is feasible with only 62 training scans, provided the vision encoder is a domain-specific 3D foundation model and the CLIP loss is computed with an embedding accumulation strategy that supplies many more negative pairs than a single 3D batch can hold. The tabular data is converted into natural-language sentences and encoded with BERT, so text supervises the MRI embeddings. The authors show the result by zero-shot classification of brain lesion location: the best configuration reaches an average AUC of 0.72, while zero-shot image retrieval remains partially unresolved. If the claim holds, it lowers the data barrier for jointly embedding medical images and structured records.

What carries the argument

Embedding accumulation: for N batches, all embeddings are computed in a gradient-free pass, then for each batch j the CLIP loss is computed against the concatenated negative embeddings from all other batches while gradients flow only through batch j, so the effective contrastive batch is N times B. This works together with two pretrained encoders: a 3D Swin-Transformer (or MedNeXt/ResNet) initialized from AMAES self-supervised pretraining on BRAINS-45K, and a pretrained BERT for the textified tabular records, so the alignment step does not have to learn representations from scratch.

What would settle it

Take the same 77 scans and split strictly by patient identity, ensuring no patient contributes scans to both training and test; retrain the best Swin-T configuration and measure zero-shot AUC. If average AUC falls toward random, the small-sample alignment result is explained by patient identity leakage rather than modality alignment.

Watch

Extended reading notes

Core claim

The paper demonstrates that a shared embedding space between 3D brain MRI and tabular clinical data can be learned from just 62 training scans when the encoders are pretrained: the MRI encoder is a 3D foundation model pretrained on large-scale brain MRI data, and the tabular data is rendered as natural-language sentences and encoded by BERT. The enabling technical object is an embedding accumulation trick that gives the CLIP loss an effective batch size of N times B while storing only B-sized batches, so enough negative pairs exist to keep training stable in 3D at small sample size. On zero-shot classification of lesion location, the best model reaches an average AUC of 0.72 (Swin-T, effective batch 128), well above random, while zero-shot image retrieval remains mixed; Swin-T reaches mAP 0.34 versus 0.21 for random. The paper frames this as the first application of CLIP training to brain MRI in native 3D.

Load-bearing premise

The results assume that splitting the 77 scans 80/20 at scan level, with some patients contributing up to eight treatment courses, keeps the evaluation free of patient overlap between training and test; if the same patient appears on both sides, the zero-shot AUCs could reflect patient identity memorization rather than modality alignment.

Editorial extensions

If this is right

A CLIP-style objective can be applied to native 3D medical imaging with dataset sizes typical of clinical studies, not only to web-scale caption collections.
Pretrained, domain-specific encoders are load-bearing for small-sample alignment; Table 1 shows training is not possible without them.
Tabular clinical data can act as the textual modality by being converted into templated sentences, meaning the same pipeline extends to free-text reports and semi-structured records.
Zero-shot classification of lesion location from MRI alone is achievable, with average AUC 0.72 for the best configuration, while zero-shot image retrieval is not yet solved.
Scaling negative pairs through embedding accumulation improves stability and performance, for example Swin-T's average AUC rises from 0.64 without accumulation to 0.72 with N=16.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We infer that the reported positive result is sensitive to the scan-level split: with 77 scans from 47 patients and some patients having up to eight treatment courses, an 80/20 split by scan rather than by patient can let the same patient appear on both sides, so part of the apparent alignment may be patient-identity memorization. Re-splitting by patient is a direct test.
We infer that embedding accumulation is not specific to MRI: any modality where memory limits batch size, such as video, whole-slide pathology, or larger 3D volumes, could benefit from the same effective-batch enlargement.
We infer that the template-sentence encoding of tabular data leaves headroom; replacing fixed templates with free-text clinical narratives could test whether richer language supervision improves the joint embedding space.
We infer that the different ranking of MedNeXt and ResNet between classification and retrieval indicates the small-sample joint space is not uniformly robust, so task-specific evaluation remains necessary.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

This is a likeable paper with a real problem. The genuinely new bits: it is the first native-3D CLIP training on brain MRI, and the embedding accumulation across batches is a practical way to scale negatives when 3D patches cap batch size. The ablations are clean, code and weights are released, and the limitations paragraph is honest. I believe the authors believe their own numbers.

But the headline claim—that modality alignment is feasible with 62 MRI scans—rests on an evaluation split that does not rule out patient identity memorization. The dataset has 77 scans from 47 patients, some with up to eight treatment courses, and Section 3.1 says an 80/20 split on the scan level. That means the same patient can appear in both train and test. Since the training text includes that patient's lesion locations, the model can learn a mapping from patient-specific anatomy to regions without learning anything about MRI-text alignment. The AUCs in Table 3 could therefore be inflated. This is not a minor nit; it is the load-bearing evidence for the paper's central promise.

The paper does some other things that soften the blow. The retrieval results are inconsistent—ResNet below random on mAP—which suggests the shared space is fragile. There are no supervised or unimodal baselines, so it is hard to know what gain the alignment itself provides. The 'instrumental' claim about the domain-specific encoder is also stronger than the evidence: Table 1 shows a +0.05 AUC boost, not a dramatic effect.

Still, I would not desk-reject this. The idea is plausible, the method is simple, and the flaw is fixable. A patient-level split (or better, treated-vs-untreated scans within the same patient) and a supervised baseline would tell us whether the '62 scans' claim holds. The embedding accumulation trick and the ablations are worth preserving regardless.

For you: worth a serious referee, but only with the expectation that the evaluation is redone. For the authors: resubmit with a proper split and modestly reworded claims.

Referee Report

4 major / 5 minor

Summary. The paper proposes a CLIP-style contrastive alignment method for 3D brain MRI volumes and tabular patient data (age, gender, and lesion locations converted into natural-language sentences). The image encoder is a 3D encoder pretrained with the authors' AMAES framework on the public BRAINS-45K dataset, and the text encoder is a pretrained BERT model. To cope with the memory cost of 3D volumes, the authors introduce an embedding accumulation strategy that computes the CLIP loss with negatives from multiple batches while backpropagating only through the current batch. They evaluate the method on zero-shot classification of brain lesion location and zero-shot image retrieval on the BrainTR-GammaKnife dataset, training on 62 MRI scans. The reported zero-shot classification AUCs are 0.65-0.72 across three backbone architectures, above the random baseline, while retrieval results are mixed, with one backbone below random on mAP and MRR.

Significance. If the results hold, the paper would make a useful empirical contribution: it would show that CLIP-style alignment between 3D medical volumes and tabular data is feasible with very small training sets when both encoders are pretrained on domain-specific data. The paper also releases code and model checkpoints, and it includes ablations of the foundation-model encoders and of the accumulation frequency, which are useful for the community. The central claim, however, rests on an evaluation split that may allow patient identity leakage, and the retrieval results show unexplained instability. The absence of unimodal or supervised baselines makes it difficult to attribute the classification performance to genuine cross-modal alignment. These issues need to be resolved before the headline claim can be accepted.

major comments (4)

[§3.1 Data] The 80/20 scan-level split is not a patient-level split: the 77 scans come from 47 patients, with some patients contributing up to eight treatment-course scans. The same patient can therefore appear in both the training and test sets. Because each training caption names the lesion locations and the MRI encoder can learn patient-specific anatomy, the model could memorize a patient-to-lesion-location mapping and achieve high zero-shot classification AUC in Table 3 without learning generalizable cross-modal alignment. The limitations paragraph at the end of §4.1 acknowledges missing external evaluation but does not address this identity-leakage path. The authors should re-run the evaluation with a patient-level split (or otherwise show that overlapping patients do not drive the results); until then, the claim that alignment is feasible with 62 MRI scans is not supported.
[Table 4 and §4.1 Discussion] The retrieval results are internally inconsistent: the ResNet encoder achieves MMR 0.18 and mAP 0.19, both below the random baseline, while Swin-T and MedNeXt beat random. The paper notes this only in passing ('interestingly Resnet does not'). A below-random retrieval performance for one of the three encoders is a sign of a collapsed or mis-scaled embedding space, and it undercuts the general claim of meaningful alignment. The authors should analyze why this occurs (for example, by inspecting embedding norms, per-label retrieval errors, or the effect of the accumulation setup) and report a concrete diagnosis rather than an anecdotal remark.
[§2.1.3 and §3.3 Evaluation Tasks] The paper lacks unimodal and supervised baselines that would isolate the contribution of cross-modal alignment. In Table 1, the configuration with a random image encoder and a pretrained text encoder reaches an average AUC of 0.67, which is far above chance; the paper does not explain why a non-aligned random image encoder achieves this, nor does it provide a text-only baseline that predicts lesion locations from language priors or class frequencies. Without such baselines, the claim that the image-text alignment, rather than the language model or label distribution, drives the zero-shot classification results is not established.
[§3.3 Evaluation Tasks] The zero-shot classification evaluation is multi-label, since each patient can have lesions in multiple regions, but the one-versus-rest AUC computation is described only briefly. The paper does not specify how samples with multiple positive labels are handled in the OvR averaging, or whether the reported AUC is macro-averaged over labels. This matters for interpreting the per-region AUCs in Table 3, especially for regions with high co-occurrence. Please clarify the exact evaluation protocol.

minor comments (5)

[Table 4 and §4] The text and Table 4 use 'MMR' for mean reciprocal rank; the standard abbreviation is 'MRR'.
[Tables 1 and 2] There are formatting artifacts in the tables, such as '0 .50', '0 .87±0.1', and '0 .64±0.01'. These should be corrected to standard decimal notation.
[§2.2] The pseudocode in Figure 2 is helpful, but the sentence 'Computing the loss and backpropagating at each iteration of j results in the effective computation of the CLIP loss with a batch size N·B while only tracking the gradient batches with size B' is slightly ambiguous: the loss is accumulated over all j before a single optimizer step, so the gradient is computed with respect to a sum of N losses. Consider rephrasing to clarify the exact gradient computation.
[§3.1 Data] The description of the region-label subset says 'we define a subset of the dataset containing 222 lesions', but it is not stated how the original 244 lesions relate to this subset and whether the excluded lesions are from patients or regions outside the five chosen labels. Please clarify.
[§1 Introduction] The t-SNE visualization in Figure 1 is presented without a detailed explanation of how the test set was augmented to increase sample size; the augmentation procedure should be described or referenced here.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pretrained encoder is externally self-supervised and ablated; the 62-scan alignment claim is not forced by construction.

full rationale

The paper's derivation chain is self-contained with respect to the downstream labels. The image encoder is pretrained with the AMAES framework on BRAINS-45K ([5]), which is self-supervised on public brain MRI volumes and does not use the BrainTR-GammaKnife lesion-location labels or the tabular attributes used during alignment, so the alignment result is not forced by that pretraining. Table 1 explicitly ablates the pretrained encoders, showing that the vision foundation model provides a modest AUC boost and that alignment training is not impossible without it; this makes the self-citation non-load-bearing. The embedding accumulation method in Figure 2 is presented as an effective batch-size scaling technique with detached cached embeddings, is described in pseudocode, and is ablated in Table 2; it is not defined in terms of the evaluation metric. The zero-shot classification and retrieval tasks use standard CLIP-style cosine-similarity prompts. The tabular-to-text templates and the evaluation prompts are semantically related but not identical, and no test labels are fitted. The scan-level 80/20 split in Section 3.1 may allow patient overlap between training and test sets, but that is a leakage or evaluation-validity concern, not a circularity step, and it does not reduce the derivation to its inputs. The limitations paragraph in Section 4.1 explicitly acknowledges the single-dataset evaluation and missing out-of-distribution assessment, further supporting that the authors do not claim external validation beyond what is tested. No load-bearing argument reduces to a self-citation as its sole justification.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method leans on four assumptions: transfer from AMAES/BRAINS-45K pretraining, the fidelity of templated text, the validity of detached cross-batch accumulation, and the absence of patient leakage in a scan-level split. The tuned temperature and accumulation frequency are free choices that affect the contrastive objective, and the scan-level split assumption is the most fragile.

free parameters (3)

CLIP temperature tau = 1.351
Tuned on validation splits; controls the sharpness of the contrastive softmax and is stated to be critical for stable training (Section 3.2).
Embedding accumulation frequency N = 8 (MedNeXt, Resnet), 16 (Swin-T)
Selected per backbone on validation (Table 2); scales the effective batch size and the number of negative pairs.
Vision/text learning-rate ratio = 10x (vision higher)
Hand-chosen stability mechanism described in Section 2.3; no sensitivity analysis is reported.

assumptions (4)

domain assumption AMAES/BRAINS-45K pretraining transfers to the GammaKnife T1-Gadolinium MRI domain
The image encoder is initialized from weights pretrained on public brain MRI, and the paper assumes this representation is a better starting point than random init; Table 1 supports a modest +0.05 AUC boost, but no domain-gap analysis is given.
domain assumption Templated natural-language sentences preserve the tabular information needed for alignment
Tabular attributes (age, gender, lesion location) are converted to fixed templates and encoded with BERT; if the template loses or distorts label information, the alignment target is corrupted.
domain assumption Detached cross-batch embedding accumulation approximates the true large-batch CLIP loss
The pseudocode detaches embeddings from earlier batches, so gradients are computed against stale negatives; the paper assumes this still yields a valid contrastive signal.
domain assumption Scan-level rather than patient-level splitting does not introduce leakage
With 77 scans from 47 patients and up to 8 treatment courses per patient, an 80/20 scan-level split can place the same patient in both training and test; the paper does not justify why this is valid.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models." pith.science (2026). https://pith.science/paper/VKMPMZFB

@misc{pith2026250114051,
  author       = {Pith},
  title        = {Pith review of: Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKMPMZFB}},
  note         = {Machine review of arXiv:2501.14051}
}

read the original abstract

Multi-modal models require aligned, shared embedding spaces. However, common CLIP-based approaches need large amounts of samples and do not natively support 3D or tabular data, both of which are crucial in the medical domain. To address these issues, we revisit CLIP-style alignment by training a domain-specific 3D foundation model as an image encoder and demonstrate that modality alignment is feasible with only 62 MRI scans. Our approach is enabled by a simple embedding accumulation strategy required for training in 3D, which scales the amount of negative pairs across batches in order to stabilize training. We perform a thorough evaluation of various design choices, including the choice of backbone and loss functions, and evaluate the proposed methodology on zero-shot classification and image-retrieval tasks. While zero-shot image-retrieval remains challenging, zero-shot classification results demonstrate that the proposed approach can meaningfully align the representations of 3D MRI with tabular data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

[1]

Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models

INTRODUCTION In the pursuit of general purpose multi-modal medical mod- els, it has become of great interest to learn aligned, shared embedding spaces. The dominant method for learning joint- embedding spaces is the contrastive language-image pre- training (CLIP) objective [1]. By leveraging 100 million captioned images, CLIP aligns textual and image repr...

work page Pith review arXiv 2025
[2]

In the following, we directly ablate the result of each design decision

METHOD Starting from the standard CLIP-setup [1], we revisit four core design decisions in CLIP-style training: Encoding 3D MRI (§2.1.1), encoding the tabular data (§2.1.2), calculat- ing the CLIP-loss across batches (§2.2) and obtaining stable training caused by the 3D input (§2.3). In the following, we directly ablate the result of each design decision....

work page 2000
[3]

The dataset is created to study the recurrence of brain tumors after Gamma-Knife Radiother- apy

EXPERIMENTAL SETUP We validate the effectiveness of our embedding alignment on two challenging tasks: zero-shot classification and image- retrieval of tumor location from brain MRI using the Brain- TR-GammaKnife dataset [16]. The dataset is created to study the recurrence of brain tumors after Gamma-Knife Radiother- apy. We use this dataset since it conta...

work page 1923
[4]

We measure the area under the receiving operating characteristic curve (AUC)

RESULTS & DISCUSSION Zero-shot classification . We measure the area under the receiving operating characteristic curve (AUC). Owing to the multiclass nature of the problem, we employ the one-versus- rest (OvR) strategy, where we treat each sample as a binary classification task. Thus, we measure the probability of the model predicting the ground truth lab...

work page
[5]

CONCLUSION In this study, we explore CLIP-alignment using a small domain-specific dataset using domain-specific foundation model encoders. We show that tabular data transformed into natural language can supervise 3D vision tasks and that a joint embedding space can yield meaningful results on non- trivial downstream tasks in a zero-shot setting. Enabled b...

work page
[6]

Ethical ap- proval was not required as confirmed by the license attached with the open access data

COMPLIANCE WITH ETHICAL STANDARDS This research study was conducted retrospectively using hu- man subject data made available in open access. Ethical ap- proval was not required as confirmed by the license attached with the open access data

work page
[7]

ACKNOWLEDGEMENTS Thanks to Jakob Ambsdorf for his comments on the manuscript. This work was supported by Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (grant num- ber NNF21SA0069429) and Villum Fonden (grant number 40516), and Pioneer Centre for AI, Danish National Research Foundation, grant number P1

work page
[8]

Further, we used initial temperature τ = 1.351 in the CLIP-loss and an embedding space with 512 dimensions

and for the Swin-T the optimal batch size was 128 ( N = 16). Further, we used initial temperature τ = 1.351 in the CLIP-loss and an embedding space with 512 dimensions. All experiments were run on single Titan RTX GPUs with 24 GB VRAM. Table 3. Results for zero-shot classification of lesion location for all models averaged over five folds. Metric is avera...

work page

Show all 24 references

[9]

Learning transferable vi- sual models from natural language supervision,

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning transferable vi- sual models from natural language supervision,” 2021

2021
[10]

Contrastive learning of medical visual representations from paired images and text,

Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christo- pher D. Manning, and Curtis P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” 2022

2022
[11]

Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding,

Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng, “Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding,” arXiv preprint arXiv:2302.12971, 2023

2023 arXiv
[12]

Visu- alizing data using t-sne,

Laurens van der Maaten and Geoffrey Hinton, “Visu- alizing data using t-sne,” Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579–2605, 2008

2008
[13]

Amaes: Augmented masked au- toencoder pretraining on public brain mri data for 3d- native segmentation,

Asbjørn Munk, Jakob Ambsdorf, Sebastian Llambias, and Mads Nielsen, “Amaes: Augmented masked au- toencoder pretraining on public brain mri data for 3d- native segmentation,” Advancing Data Solutions in Medical Imaging AI. MICCAI 2024., 2024

2024
[14]

Swin transformer: Hierarchical vision transformer us- ing shifted windows,

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo, “Swin transformer: Hierarchical vision transformer us- ing shifted windows,” 2021

2021
[15]

Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images,

Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger Roth, and Daguang Xu, “Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images,” 2022

2022
[16]

Mednext: Transformer- driven scaling of convnets for medical image segmenta- tion,

Saikat Roy, Gregor Koehler, Constantin Ulrich, Michael Baumgartner, Jens Petersen, Fabian Isensee, Paul F. Jaeger, and Klaus Maier-Hein, “Mednext: Transformer- driven scaling of convnets for medical image segmenta- tion,” 2023

2023
[17]

Deep residual learning for image recognition,

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” 2015

2015
[18]

An introduction to con- volutional neural networks,

Keiron O’Shea and Ryan Nash, “An introduction to con- volutional neural networks,” 2015

2015
[19]

nnu-net revisited: A call for rigor- ous validation in 3d medical image segmentation,

Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F. Jaeger, “nnu-net revisited: A call for rigor- ous validation in 3d medical image segmentation,” 2024

2024
[20]

Yucca: A deep learning framework for medical image analysis,

Sebastian Nørgaard Llambias, Julia Machnio, Asbjørn Munk, Jakob Ambsdorf, Mads Nielsen, and Mostafa Mehdipour Ghazi, “Yucca: A deep learning framework for medical image analysis,” arXiv preprint arXiv:2407.19888, 2024

2024 arXiv
[21]

Heterogeneous learning for brain lesion segmentation, detection, and classifica- tion,

Sebastian Nørgaard Llambias, Mads Nielsen, and Mostafa Mehdipour Ghazi, “Heterogeneous learning for brain lesion segmentation, detection, and classifica- tion,” in Proceedings of the 5th Northern Lights Deep Learning Conference (NLDL) . 2024, Proceedings of Machine Learning Re...

2024
[22]

Bert: Pre-training of deep bidirec- tional transformers for language understanding,

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre-training of deep bidirec- tional transformers for language understanding,” 2019

2019
[23]

Learning trans- ferable visual models from natural language supervi- sion,

Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning trans- ferable visual models from natural language supervi- sion,” in ICML, 2021

2021
[24]

A brain mri dataset and baseline evaluations for tumor recurrence prediction after gamma knife radiotherapy,

Yibin Wang, William Neil Duggar, David Michael Ca- ballero, Toms Vengaloor Thomas, Neha Adari Eswara, Kumar Mundra, and Haifeng Wang, “A brain mri dataset and baseline evaluations for tumor recurrence prediction after gamma knife radiotherapy,” 2023

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models

INTRODUCTION In the pursuit of general purpose multi-modal medical mod- els, it has become of great interest to learn aligned, shared embedding spaces. The dominant method for learning joint- embedding spaces is the contrastive language-image pre- training (CLIP) objective [1]. By leveraging 100 million captioned images, CLIP aligns textual and image repr...

work page Pith review arXiv 2025

[2] [2]

In the following, we directly ablate the result of each design decision

METHOD Starting from the standard CLIP-setup [1], we revisit four core design decisions in CLIP-style training: Encoding 3D MRI (§2.1.1), encoding the tabular data (§2.1.2), calculat- ing the CLIP-loss across batches (§2.2) and obtaining stable training caused by the 3D input (§2.3). In the following, we directly ablate the result of each design decision....

work page 2000

[3] [3]

The dataset is created to study the recurrence of brain tumors after Gamma-Knife Radiother- apy

EXPERIMENTAL SETUP We validate the effectiveness of our embedding alignment on two challenging tasks: zero-shot classification and image- retrieval of tumor location from brain MRI using the Brain- TR-GammaKnife dataset [16]. The dataset is created to study the recurrence of brain tumors after Gamma-Knife Radiother- apy. We use this dataset since it conta...

work page 1923

[4] [4]

We measure the area under the receiving operating characteristic curve (AUC)

RESULTS & DISCUSSION Zero-shot classification . We measure the area under the receiving operating characteristic curve (AUC). Owing to the multiclass nature of the problem, we employ the one-versus- rest (OvR) strategy, where we treat each sample as a binary classification task. Thus, we measure the probability of the model predicting the ground truth lab...

work page

[5] [5]

CONCLUSION In this study, we explore CLIP-alignment using a small domain-specific dataset using domain-specific foundation model encoders. We show that tabular data transformed into natural language can supervise 3D vision tasks and that a joint embedding space can yield meaningful results on non- trivial downstream tasks in a zero-shot setting. Enabled b...

work page

[6] [6]

Ethical ap- proval was not required as confirmed by the license attached with the open access data

COMPLIANCE WITH ETHICAL STANDARDS This research study was conducted retrospectively using hu- man subject data made available in open access. Ethical ap- proval was not required as confirmed by the license attached with the open access data

work page

[7] [7]

ACKNOWLEDGEMENTS Thanks to Jakob Ambsdorf for his comments on the manuscript. This work was supported by Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (grant num- ber NNF21SA0069429) and Villum Fonden (grant number 40516), and Pioneer Centre for AI, Danish National Research Foundation, grant number P1

work page

[8] [8]

Further, we used initial temperature τ = 1.351 in the CLIP-loss and an embedding space with 512 dimensions

and for the Swin-T the optimal batch size was 128 ( N = 16). Further, we used initial temperature τ = 1.351 in the CLIP-loss and an embedding space with 512 dimensions. All experiments were run on single Titan RTX GPUs with 24 GB VRAM. Table 3. Results for zero-shot classification of lesion location for all models averaged over five folds. Metric is avera...

work page