REVIEW 5 major objections 4 minor 1 cited by
MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis
T0 review · 5 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper claims that adapting DINOv2 for multi-modal MRI—modality-aware patch embeddings, whole-sequence masking, and semi-supervised labels—yields external-test MCC 0.60 for glioma subtypes, 11.1% above supervised baselines.
desk verdict Useful multi-modal DINOv2 adaptation with honest ablations, but the abstract cherry-picks the supervised variant's number and the baselines are too weak to support 'state-of-the-art.' read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a modified patch embedding: z_{i,m} = z_p + z_i + z_m, where z_p is the pre-trained patch projection, z_i is the pre-trained positional embedding applied separately within each modality, and z_m is a learnable modality token that identifies which MRI sequence a patch came from. This lets the frozen backbone see multi-modal data as interleaved single-modality images rather than as one stitched image. Two further mechanisms carry the method: full modality masking, which drops every patch of one sequence in the student input and forces the student to reconstruct the teacher's features from the remaining sequences, thereby learning cross-modality relationships; and a sem
What would settle it
Run an external evaluation in which every patient is classified from the full 3D volume rather than a single 96x96 axial middle slice, with molecular subtype labels as ground truth; if patient-level MCC does not track the reported slice-level 0.60, the gains are slice-level and may not transfer clinically.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the gap between a uni-modal natural-image foundation model and multi-modal medical imaging is not architectural but representational. By replacing the single set of positional embeddings with per-modality positional embeddings plus a learnable modality embedding for each MRI sequence, the same pre-trained DINOv2 transformer can consume patches from several modalities without corrupting spatial or modality identity. Full-modality masking—dropping all patches of one sequence from the student while the teacher sees all—turns the masked-patch objective into a cross-modality prediction task, which is what gives robustness to missing sequences. Final
Load-bearing premise
That one 96x96 axial slice through the tumor's middle captures enough molecular-subtype information to classify a patient; if subtype-defining features live in other slices or in volumetric texture, the external-test gain may not represent patient-level diagnostic accuracy.
Editorial extensions
If this is right
- If correct, any DINOv2 checkpoint can be adapted to a multi-modal medical task by changing the embedding head and masking schedule, without altering the transformer backbone.
- Semi-supervised pre-training on unlabeled MRI archives should become a standard first step for classification tasks with few labels: the authors report it beats fully supervised fine-tuning.
- The full-modality masking result implies models can be deployed in clinics where one sequence is missing, with a smaller accuracy drop than models trained on complete inputs only.
- The ablation ordering implies future multi-modal adaptations should keep per-modality position encoding and modality tokens, since naive modality concatenation or RGB stacking actively hurts performance.
Reading between the lines
- The single-slice evaluation design suggests the method's true ceiling may be higher under volumetric aggregation: a 3D or multi-slice version could combine the cross-modality robustness with spatial context, a direction the paper names but does not test.
- A stratified missing-modality analysis by which sequence is absent would likely show that robustness is strongest when the missing sequence is redundant; if the most diagnostically informative sequence is missing, performance may not match the average reported here.
- Because the semi-supervised gain is uneven across classes and worsens the rare oligodendroglioma F1 on the external set, class-balanced pseudo-labeling or oversampling of rare subtypes is a natural extension.
- The modality-embedding idea is not MRI-specific; it should transfer to any multi-image modality stack (CT/PET, multiparametric imaging) and to settings where scans arrive at different times.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MM-DINOv2, an adaptation of the DINOv2 vision foundation model to multi-modal MRI. Three modifications are introduced: modality-specific patch embeddings (with per-modality positional embeddings), a full-modality masking strategy during the masked-image-modeling objective, and a semi-supervised extension that combines the DINOv2 image-level objective with available class labels. The method is evaluated on glioma subtype classification from four MRI sequences, using a large multi-institutional training set and an external TCGA test set. The paper reports an external MCC of 0.60 and a +11.1% improvement over a ResNet34 baseline, and claims robustness to missing MRI sequences.
Significance. If the reported results are representative, the core architectural idea of adding modality embeddings and full-modality masking to a frozen-pretrained DINOv2 is a useful and practical contribution for multi-modal medical imaging, and the external validation on TCGA is a strength. The paper also makes code publicly available, which supports reproducibility. However, several load-bearing claims are not supported by the reported numbers as written: the headline 0.60 external MCC belongs to the supervised variant, not the semi-supervised method that is the paper's main contribution, and the semi-supervised variant's external MCC is actually lower (0.57). The 'state-of-the-art' claim is also not backed by any current SOTA comparator. These issues need to be resolved before the paper's central claims can be accepted.
major comments (5)
- [Abstract; §4.1, Table 1] The abstract's headline claim, 'our method achieves an MCC of 0.6 ... surpassing state-of-the-art supervised approaches by +11.1%,' is not supported by Table 1 for the method emphasized in the paper. The 0.60 external MCC is the fully supervised MM-DINOv2 row; the semi-supervised MM-DINOv2 row, which includes the paper's third contribution, reports 0.57. The +11.1% is exactly (0.60−0.54)/0.54 against ResNet34. If 'our method' refers to the semi-supervised framework, the relative gain is +5.6%, not +11.1%, and this gap is not statistically established. Please correct the abstract or explicitly identify which configuration is being reported.
- [§4.1, Table 1; §5] The claim that semi-supervised learning improves performance is contradicted by the external results. Table 1 shows that semi-supervised MM-DINOv2 improves internal MCC from 0.68 to 0.74 but decreases external MCC from 0.60 to 0.57. The Conclusion states that 'semi-supervised learning ... outperformed supervised training' without acknowledging the external drop. This is a load-bearing issue for contribution 3. The authors should report the external numbers and discuss possible reasons (e.g., domain shift, class imbalance) or temper the claim.
- [§4.1, Table 1] 'Surpassing state-of-the-art supervised approaches' is not established. The only supervised comparators are ResNet34 and two RGB-DINOv2 variants. No recent glioma-subtype classification method (e.g., the multi-task method in [28] or the imbalance-aware method in [24]) is included. Without these baselines, the claim of state-of-the-art performance is unjustified. Either add representative SOTA baselines evaluated on the same external set, or reframe the claim as an improvement over the selected baselines.
- [§4.1–§4.3, Tables 1–3] No uncertainty quantification is provided. The paper uses a single data split and gives no confidence intervals, error bars, or significance tests for the external test set of 214 patients. The differences between key numbers (e.g., 0.60 vs 0.57) are small enough that they may be within sampling noise. Reporting multiple seeds or bootstrap confidence intervals is necessary to support the comparative claims.
- [§4.1; §3.5] The comparison between supervised and semi-supervised settings is not apples-to-apples. Section 4.1 states that supervised training uses full fine-tuning, while the semi-supervised setting uses 'pre-training, followed by linear evaluation.' This protocol difference alone can explain part of the performance gap. Additionally, evaluation in Section 3.5 uses a single 96×96 axial middle slice, whereas tumor subtype-relevant heterogeneity is often volumetric. The authors should justify the single-slice evaluation or provide a patient-level aggregated evaluation, otherwise the reported MCC may not reflect diagnostic accuracy in practice.
minor comments (4)
- [Section 3.1] The sentence 'These positional embeddings are learned during the DINOv2 pre-training and continuously optimized during the pre-training' contains a redundant phrase. Also, it should be clarified whether these embeddings are frozen or fine-tuned in the proposed adaptation.
- [Section 4.2] The missing-sequence evaluation is described as 'one MRI sequence is randomly masked.' It is unclear whether this is a single random mask per test sample, whether results are averaged over multiple masks, and how the masked modality is handled at the input (e.g., zero-filled). Please provide these details.
- [Table 1 caption] The caption mentions 'continuous pre-training' but the text in Section 4.1 refers to 'pre-training, followed by linear evaluation.' This inconsistency should be resolved so the reader knows exactly which training/evaluation protocol each table row uses.
- [Section 3.5] The dataset description states that all images provide all four sequences, but the training set includes both labeled and unlabeled subjects. Please clarify how the unlabeled subset is defined and whether the same missing-sequence simulation is applied during training.
Circularity Check
No circularity: external test set is held out; no prediction reduces to a fitted input or self-citation.
full rationale
The paper's derivation chain is self-contained with respect to circularity. The central results (Table 1) are evaluated on an external TCGA test set that is explicitly held out from all training and validation (Section 3.5: 'We hold out the TCGA dataset [3] for external testing and randomly split the remaining datasets in 70/10/20% for training, validation, and internal testing'). No parameter is fitted to the external test set, and the design choices (modality embeddings, full-modality masking, semi-supervised loss) are architectural and trained with standard supervised/self-supervised objectives. The semi-supervised extension is based on Fini et al. [9], an external method, not on the authors' own prior work. The only self-citation is [24], a related-work reference on imbalance-aware losses; it is not load-bearing for the method or the headline result. The abstract's '+11.1%' arithmetic is derived from Table 1 by comparing the supervised MM-DINOv2 external MCC (0.60) to the ResNet34 baseline (0.54); this is a reporting inconsistency about which configuration constitutes 'our method', not a circular reduction. Similarly, the absence of contemporary SOTA baselines and confidence intervals is a correctness/rigor concern, not a circularity concern. Overall, the derivation does not define its predictions in terms of its inputs, does not fit a parameter and then rename it as a prediction, and does not import a uniqueness result via self-citation. Therefore, no significant circularity is present.
Assumptions & free parameters
free parameters (4)
- semi_supervised_loss_weight =
2.0
- softmax_temperature =
0.1
- label_smoothing =
0.1
- tumor_pixel_threshold =
500
assumptions (3)
- domain assumption DINOv2's pre-trained representations from natural images transfer to medical MRI slices.
- domain assumption The teacher network's prototype scores provide reliable pseudo-labels for unlabeled data.
- domain assumption A single axial slice of the tumor is sufficient to classify glioma subtype.
Cite this review
Pith. "Pith review of MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis." pith.science (2026). https://pith.science/paper/VQBWHXFT
@misc{pith2026250906617,
author = {Pith},
title = {Pith review of: MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/VQBWHXFT}},
note = {Machine review of arXiv:2509.06617}
}
read the original abstract
Vision foundation models like DINOv2 demonstrate remarkable potential in medical imaging despite their origin in natural image domains. However, their design inherently works best for uni-modal image analysis, limiting their effectiveness for multi-modal imaging tasks that are common in many medical fields, such as neurology and oncology. While supervised models perform well in this setting, they fail to leverage unlabeled datasets and struggle with missing modalities, a frequent challenge in clinical settings. To bridge these gaps, we introduce MM-DINOv2, a novel and efficient framework that adapts the pre-trained vision foundation model DINOv2 for multi-modal medical imaging. Our approach incorporates multi-modal patch embeddings, enabling vision foundation models to effectively process multi-modal imaging data. To address missing modalities, we employ full-modality masking, which encourages the model to learn robust cross-modality relationships. Furthermore, we leverage semi-supervised learning to harness large unlabeled datasets, enhancing both the accuracy and reliability of medical predictions. Applied to glioma subtype classification from multi-sequence brain MRI, our method achieves a Matthews Correlation Coefficient (MCC) of 0.6 on an external test set, surpassing state-of-the-art supervised approaches by +11.1%. Our work establishes a scalable and robust solution for multi-modal medical imaging tasks, leveraging powerful vision foundation models pre-trained on natural images while addressing real-world clinical challenges such as missing data and limited annotations.
Figures
Forward citations
Cited by 1 Pith paper
-
Improving Medical Image Generative Models with Fr\'echet Distance Loss
Adding a Fréchet distance loss during generative model finetuning improves realism of synthetic medical images and downstream tumor segmentation.
Reference graph
Works this paper leans on
-
[28]
Neuro-Oncology25(2), 279–289 (feb 2023)
van der Voort, S.R., Incekara, F., Wijnenga, M.M.J., Kapsas, G., Gahrmann, R., et al.: Combined molecular subtyping, grading, and segmentation of glioma using multi-task deep learning. Neuro-Oncology25(2), 279–289 (feb 2023)
work page 2023
-
[24]
In: Medical Imaging with Deep Learning (feb 2024)
Scholz, D., Erdur, A.C., Buchner, J.A., Peeken, J.C., Rueckert, D., et al.: Imbalance-aware loss functions improve medical image classification. In: Medical Imaging with Deep Learning (feb 2024)
work page 2024
-
[1]
Baharoon, M., Qureshi, W., Ouyang, J., Xu, Y., Aljouie, A., et al.: Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Ex- perimental Study of DINOv2 on Radiology Benchmarks (sep 2024)
work page 2024
-
[2]
Baid, U., Ghodasara, S., Mohan, S., Bilello, M., Calabrese, E., et al.: The RSNA- ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Ra- diogenomic Classification (sep 2021)
work page 2021
-
[3]
Scientific Data4(1), 170117 (sep 2017)
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., et al.: Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific Data4(1), 170117 (sep 2017)
work page 2017
-
[4]
Scientific Data9(1), 453 (jul 2022)
Bakas, S., Sako, C., Akbari, H., Bilello, M., Sotiras, A., et al.: The University of Pennsylvania glioblastoma (UPenn-GBM) cohort: Advanced MRI, clinical, ge- nomics, & radiomics. Scientific Data9(1), 453 (jul 2022)
work page 2022
-
[5]
Bannur,S.,Bouzid,K.,Castro,D.C.,Schwaighofer,A.,Thieme,A.,etal.:MAIRA- 2: Grounded Radiology Report Generation (sep 2024)
work page 2024
-
[6]
Radiology: Artificial Intelligence4(6), e220058 (nov 2022)
Calabrese, E., Villanueva-Meyer, J.E., Rudie, J.D., Rauschecker, A.M., Baid, U., et al.: The University of California San Francisco Preoperative Diffuse Glioma MRI Dataset. Radiology: Artificial Intelligence4(6), e220058 (nov 2022)
work page 2022
Show all 30 references
-
[7]
Neuro-Oncology24(4), 639–652 (apr 2022)
Cluceru, J., Interian, Y., Phillips, J.J., Molinaro, A.M., Luks, T.L., et al.: Improv- ing the noninvasive classification of glioma genetic subtype with deep learning and diffusion-weighted imaging. Neuro-Oncology24(4), 639–652 (apr 2022)
2022
-
[8]
In: International Conference on Learning Representations (oct 2020) 10 D
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., et al.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In: International Conference on Learning Representations (oct 2020) 10 D. Scholz et al
2020
-
[9]
In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Fini, E., Astolfi, P., Alahari, K., Alameda-Pineda, X., Mairal, J., et al.: Semi- Supervised Learning Made Simple With Self-Supervised Clustering. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3187–3197 (2023)
2023
-
[10]
European Radiology (sep 2023)
Foltyn-Dumitru, M., Schell, M., Rastogi, A., Sahm, F., Kessler, T., et al.: Impact of signal intensity normalization of MRI on the generalizability of radiomic-based prediction of molecular glioma subtypes. European Radiology (sep 2023)
2023
-
[11]
BMC Medical Imaging20(1), 87 (jul 2020)
Ge, C., Gu, I.Y.H., Jakola, A.S., Yang, J.: Deep semi-supervised learning for brain tumor classification. BMC Medical Imaging20(1), 87 (jul 2020)
2020
-
[12]
Scientific Data5(1), 180158 (aug 2018)
Gusev, Y., Bhuvaneshwar, K., Song, L., Zenklusen, J.C., Fine, H., et al.: The REMBRANDTstudy,alargecollectionofgenomicdatafrombraincancerpatients. Scientific Data5(1), 180158 (aug 2018)
2018
-
[13]
arXiv:1512.03385 [cs] (dec 2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs] (dec 2015)
2015 arXiv
-
[14]
Huang, Y., Zou, J., Meng, L., Yue, X., Zhao, Q., et al.: Comparative Analysis of ImageNet Pre-Trained Deep Learning Models and DINOv2 in Medical Imaging Classification (feb 2024)
2024
-
[15]
Jiang, D., Liu, Y., Liu, S., Zhao, J., Zhang, H., et al.: From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models (mar 2024)
2024
-
[16]
Nature Methods21(2), 195–212 (feb 2024)
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M.D., Buettner, F., et al.: Metrics reloaded: Recommendations for image analysis validation. Nature Methods21(2), 195–212 (feb 2024)
2024
-
[17]
Maniparambil, M., Akshulakov, R., Djilali, Y.A.D., Narayan, S., Singh, A., et al.: From Unimodal to Multimodal: Scaling up Projectors to Align Modalities (sep 2024)
2024
-
[18]
IEEE trans- actions on medical imaging34(10), 1993–2024 (oct 2015)
Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., et al.: The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE trans- actions on medical imaging34(10), 1993–2024 (oct 2015)
1993
-
[19]
Müller-Franzes,G.,Khader,F.,Siepmann,R.,Han,T.,Kather,J.N.,etal.:Medical Slice Transformer: Improved Diagnosis and Explainability on 3D Medical Images with DINOv2 (nov 2024)
2024
-
[20]
Transactions on Machine Learning Research (jul 2023)
Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., et al.: DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research (jul 2023)
2023
-
[21]
Scientific Reports13(1), 18911 (2023)
Pemberton, H.G., Wu, J., Kommers, I., Müller, D.M., Hu, Y., Goodkin, O., Vos, S.B., Bisdas, S., Robe, P.A., Ardon, H., et al.: Multi-class glioma segmentation on real-world data with missing mri sequences: comparison of three deep learning algorithms. Scientific Reports13(1), ...
2023
-
[22]
Nature Machine Intelligence7(1), 119–130 (jan 2025)
Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., et al.: Ex- ploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence7(1), 119–130 (jan 2025)
2025
-
[23]
Human Brain Mapping31(5), 798– 819 (dec 2009)
Rohlfing, T., Zahr, N.M., Sullivan, E.V., Pfefferbaum, A.: The SRI24 multichannel atlas of normal adult human brain structure. Human Brain Mapping31(5), 798– 819 (dec 2009)
2009
-
[25]
Song, X., Xu, X., Yan, P.: General Purpose Image Encoder DINOv2 for Medical Image Registration (Feb 2024) Multi-Modal DINOv2 for Medical Image Analysis 11
2024
-
[26]
Scientific Data9(1), 768 (dec 2022)
Suter,Y., Knecht, U., Valenzuela, W., Notter, M.,Hewer, E.,et al.: The LUMIERE dataset: Longitudinal Glioblastoma MRI with expert RANO evaluation. Scientific Data9(1), 768 (dec 2022)
2022
-
[27]
Data in Brief37, 107191 (aug 2021)
van der Voort, S.R., Incekara, F., Wijnenga, M.M.J., Kapsas, G., Gahrmann, R., et al.: The Erasmus Glioma Database (EGD): Structural MRI scans, WHO 2016 subtypes, and segmentations of 774 patients with glioma. Data in Brief37, 107191 (aug 2021)
2016
-
[29]
Neuro-Oncology23(8), 1215 (aug 2021)
Wen, P.Y., Packer, R.J.: The 2021 WHO Classification of Tumors of the Central Nervous System: Clinical implications. Neuro-Oncology23(8), 1215 (aug 2021)
2021
-
[30]
In: International Conference on Learning Representations (oct 2021)
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., et al.: Image BERT Pre-training with Online Tokenizer. In: International Conference on Learning Representations (oct 2021)
2021
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.