{"id":"30d3d933-a9b8-42ea-9458-c65d3da24857","arxiv_id":"2509.06617","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MM-DINOv2, which adds modality embeddings, per-modality positional embeddings, full-modality masking, and semi-supervised training to DINOv2, improved external-test glioma subtype classification to MCC 0.60 versus 0.54 for ResNet34.","lead":"This paper adapts the DINOv2 vision model to combine four types of MRI brain scans for glioma subtyping, adding modality-specific embeddings and a masking technique for missing scans. It reports an external-test Matthews correlation of 0.60, outperforming its ResNet and naive DINOv2 baselines, which matters because clinical scans often lack one or more sequences.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline external MCC 0.60/+11.1% rests on the supervised variant and a single baseline, not the full semi-supervised method, with no error bars or SOTA comparison.","rationale":"The strongest claim is the external MCC 0.60 with a +11.1% improvement over supervised approaches. The most load-bearing issue is that this number corresponds to the supervised MM-DINOv2 row in Table 1, not to the full semi-supervised pipeline that the paper introduces and evaluates in the ablation study. The abstract's 'our method' is therefore ambiguous or misleading: the semi-supervised row gives 0.57 externally, and the +11.1% computation is only valid against ResNet34's 0.54. The paper also does not compare against actual recent glioma-subtype classifiers, instead calling ResNet34 a 'strong baseline,' and reports no uncertainty quantification, so the 0.06 MCC gap may not be statistically meaningful. The architectural ideas—modality-specific embeddings, full-modality masking, and joint SSL—are reasonable and code is released, but the central quantitative claim is not yet supported. This is a more direct threat than the reader's slice-level concern, because even granting the 2D-slice setup, the claimed advantage is tied to a specific model-configuration choice and a weak baseline. The concrete test—bootstrap CIs plus an additional SOTA comparison—would settle whether the claim survives. Since the reader already gave CONDITIONAL, this concern does not change the verdict but reinforces the conditions.","tokens_in":8030,"tokens_out":5847,"duration_ms":60939,"concrete_test":"Reproduce the external TCGA evaluation for both Table 1 rows using 10 random seeds and bootstrap the 214 patients (1000 resamples), reporting the mean and 95% CI for the MCC difference between each MM-DINOv2 configuration and ResNet34. In addition, rerun the comparison with at least one published glioma-subtype classifier under the same 96×96 middle-slice protocol. If the semi-supervised MM-DINOv2's external MCC is not significantly above ResNet34, or if the 0.60 result does not correspond to the full proposed method, the abstract's +11.1% claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result—external MCC 0.60, +11.1% over 'state-of-the-art supervised approaches'—is not supported by the paper's own Table 1 for the method actually proposed. In §4.1, Table 1, the supervised MM-DINOv2 row reports external MCC 0.60, but the full semi-supervised MM-DINOv2 row—which includes the paper's third contribution and is the setting used for the ablations in Table 3—reports 0.57. The +11.1% is exactly (0.60−0.54)/0.54, where 0.54 is the ResNet34 baseline; thus the abstract claims the supervised variant's number while presenting the semi-supervised method as the contribution. The semi-supervised variant's relative gain is only +5.6%. Furthermore, the baseline set is only ResNet34 and two DINOv2 variants; no current glioma-subtype SOTA method (e.g., multi-task or imbalance-aware classifiers) is included, so 'surpassing state-of-the-art' is not established. No confidence intervals, random seeds, or significance tests are reported for the 214-patient external set, so the 0.06 MCC gap could be within sampling noise. The load-bearing assumption is that 'our method' in the abstract refers to the supervised MM-DINOv2 configuration and that ResNet34 is an acceptable SOTA proxy; if 'our method' instead includes semi-supervised learning, the headline advantage disappears.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MM-DINOv2, an adaptation of the DINOv2 vision foundation model to multi-modal MRI. Three modifications are introduced: modality-specific patch embeddings (with per-modality positional embeddings), a full-modality masking strategy during the masked-image-modeling objective, and a semi-supervised extension that combines the DINOv2 image-level objective with available class labels. The method is evaluated on glioma subtype classification from four MRI sequences, using a large multi-institutional training set and an external TCGA test set. The paper reports an external MCC of 0.60 and a +11.1% improvement over a ResNet34 baseline, and claims robustness to missing MRI sequences.","tokens_in":8412,"tokens_out":4773,"duration_ms":46661,"significance":"If the reported results are representative, the core architectural idea of adding modality embeddings and full-modality masking to a frozen-pretrained DINOv2 is a useful and practical contribution for multi-modal medical imaging, and the external validation on TCGA is a strength. The paper also makes code publicly available, which supports reproducibility. However, several load-bearing claims are not supported by the reported numbers as written: the headline 0.60 external MCC belongs to the supervised variant, not the semi-supervised method that is the paper's main contribution, and the semi-supervised variant's external MCC is actually lower (0.57). The 'state-of-the-art' claim is also not backed by any current SOTA comparator. These issues need to be resolved before the paper's central claims can be accepted.","major_comments":[{"comment":"The abstract's headline claim, 'our method achieves an MCC of 0.6 ... surpassing state-of-the-art supervised approaches by +11.1%,' is not supported by Table 1 for the method emphasized in the paper. The 0.60 external MCC is the fully supervised MM-DINOv2 row; the semi-supervised MM-DINOv2 row, which includes the paper's third contribution, reports 0.57. The +11.1% is exactly (0.60−0.54)/0.54 against ResNet34. If 'our method' refers to the semi-supervised framework, the relative gain is +5.6%, not +11.1%, and this gap is not statistically established. Please correct the abstract or explicitly identify which configuration is being reported.","section":"Abstract; §4.1, Table 1"},{"comment":"The claim that semi-supervised learning improves performance is contradicted by the external results. Table 1 shows that semi-supervised MM-DINOv2 improves internal MCC from 0.68 to 0.74 but decreases external MCC from 0.60 to 0.57. The Conclusion states that 'semi-supervised learning ... outperformed supervised training' without acknowledging the external drop. This is a load-bearing issue for contribution 3. The authors should report the external numbers and discuss possible reasons (e.g., domain shift, class imbalance) or temper the claim.","section":"§4.1, Table 1; §5"},{"comment":"'Surpassing state-of-the-art supervised approaches' is not established. The only supervised comparators are ResNet34 and two RGB-DINOv2 variants. No recent glioma-subtype classification method (e.g., the multi-task method in [28] or the imbalance-aware method in [24]) is included. Without these baselines, the claim of state-of-the-art performance is unjustified. Either add representative SOTA baselines evaluated on the same external set, or reframe the claim as an improvement over the selected baselines.","section":"§4.1, Table 1"},{"comment":"No uncertainty quantification is provided. The paper uses a single data split and gives no confidence intervals, error bars, or significance tests for the external test set of 214 patients. The differences between key numbers (e.g., 0.60 vs 0.57) are small enough that they may be within sampling noise. Reporting multiple seeds or bootstrap confidence intervals is necessary to support the comparative claims.","section":"§4.1–§4.3, Tables 1–3"},{"comment":"The comparison between supervised and semi-supervised settings is not apples-to-apples. Section 4.1 states that supervised training uses full fine-tuning, while the semi-supervised setting uses 'pre-training, followed by linear evaluation.' This protocol difference alone can explain part of the performance gap. Additionally, evaluation in Section 3.5 uses a single 96×96 axial middle slice, whereas tumor subtype-relevant heterogeneity is often volumetric. The authors should justify the single-slice evaluation or provide a patient-level aggregated evaluation, otherwise the reported MCC may not reflect diagnostic accuracy in practice.","section":"§4.1; §3.5"}],"minor_comments":[{"comment":"The sentence 'These positional embeddings are learned during the DINOv2 pre-training and continuously optimized during the pre-training' contains a redundant phrase. Also, it should be clarified whether these embeddings are frozen or fine-tuned in the proposed adaptation.","section":"Section 3.1"},{"comment":"The missing-sequence evaluation is described as 'one MRI sequence is randomly masked.' It is unclear whether this is a single random mask per test sample, whether results are averaged over multiple masks, and how the masked modality is handled at the input (e.g., zero-filled). Please provide these details.","section":"Section 4.2"},{"comment":"The caption mentions 'continuous pre-training' but the text in Section 4.1 refers to 'pre-training, followed by linear evaluation.' This inconsistency should be resolved so the reader knows exactly which training/evaluation protocol each table row uses.","section":"Table 1 caption"},{"comment":"The dataset description states that all images provide all four sequences, but the training set includes both labeled and unlabeled subjects. Please clarify how the unlabeled subset is defined and whether the same missing-sequence simulation is applied during training.","section":"Section 3.5"}],"recommendation":"major_revision","confidential_remarks":"The core architectural idea—adding modality embeddings and full-modality masking to DINOv2—is potentially useful, and the external TCGA evaluation is a positive feature. However, the abstract's headline number is drawn from the supervised variant rather than the proposed semi-supervised variant, the SOTA claim lacks supporting baselines, and the semi-supervised external result is internally contradictory. These issues are fixable within the scope of the paper, but they require more than minor editing. I would encourage the editor to ask for a revised version with corrected reporting, proper baselines, and uncertainty quantification, and to re-evaluate then."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the paper has a genuinely useful idea—add modality embeddings and modality-level masking to DINOv2 so it can consume multi-sequence MRI and keep working when a sequence is missing—and the external test on TCGA is a real step. But the abstract oversells it: the 0.60 MCC it quotes is from the purely supervised variant, and the semi-supervised variant that the paper actually promotes gets 0.57. The +11.1% is computed against ResNet34, which is not current state of the art for glioma subtyping. That gap between headline and table is the thing to fix.\n\nWhat's new is the combination, not a new learning principle. The multi-modal patch embedding is simple and sensible: instead of stacking sequences as RGB channels or concatenating tokens with shared positional embeddings, they add a learnable modality vector and apply positional embeddings per modality. The full-modality masking is a straightforward way to force reliance on cross-modality redundancy. The ablation in Table 3 shows each component helping monotonically, which is the right kind of evidence. Releasing code is a plus.\n\nThe soft spots are claim discipline and evaluation breadth. The abstract picks the best-looking row; the semi-supervised variant, which is the third stated contribution, is weaker externally. No confidence intervals or multiple seeds for the 214-patient external set, so the 0.03–0.06 MCC differences could be noise. The baseline set is thin: ResNet34 and two DINOv2 configurations, not the current published methods for glioma subtype classification, so 'surpassing state-of-the-art' is not established. Also, evaluation uses a single 96x96 axial slice through the tumor. The paper doesn't justify why middle-slice appearance suffices for subtype, which is clinically questionable.\n\nFor a reader working on foundation models for medical imaging, the recipe is worth knowing and the ablations are clean. For a reader focused on clinical deployment, the evaluation is too coarse. The paper deserves a serious referee, not a desk reject, but it should be revised so the claims match the actual rows and the baselines include contemporary methods. Send it to review with that condition.","headline":"Useful multi-modal DINOv2 adaptation with honest ablations, but the abstract cherry-picks the supervised variant's number and the baselines are too weak to support 'state-of-the-art.'","tokens_in":8851,"tokens_out":3285,"would_cite":true,"duration_ms":33019,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adapting DINOv2 for multi-modal MRI—modality-aware patch embeddings, whole-sequence masking, and semi-supervised labels—yields external-test MCC 0.60 for glioma subtypes, 11.1% above supervised baselines.","keywords":["multi-modal medical imaging","DINOv2","glioma subtype classification","MRI","semi-supervised learning","missing modality robustness","vision foundation model","masked image modeling"],"falsifier":"Run an external evaluation in which every patient is classified from the full 3D volume rather than a single 96x96 axial middle slice, with molecular subtype labels as ground truth; if patient-level MCC does not track the reported slice-level 0.60, the gains are slice-level and may not transfer clinically.","tokens_in":7976,"feed_emoji":"🧠","tokens_out":7339,"duration_ms":73920,"temperature":0.7,"pith_summary":"The paper tries to show that a vision foundation model pre-trained on natural images can be made to work for multi-modal medical imaging—where the same patient is imaged with several MRI sequences—by changing how patches are embedded, how masking is done, and how labels enter training. Applied to glioma subtype classification from four MRI sequences, the adapted model reaches a Matthews Correlation Coefficient of 0.60 on an external test set, which the authors place 11.1% above the supervised baselines they compare against. The practical stake is clinical: MRI sequences are often missing in routine care, labels are scarce, and large unlabeled scan archives exist but are unused by supervised methods. The paper's central wager is that all three problems can be handled inside one DINOv2-style pre-training loop rather than through a task-specific architecture.","feed_headline":"Glioma MRI model beats supervised baselines by 11.1%","feed_subtitle":"Modality-aware patches and whole-sequence masking keep the model accurate when one MRI scan is missing.","key_machinery":"The central mechanism is a modified patch embedding: z_{i,m} = z_p + z_i + z_m, where z_p is the pre-trained patch projection, z_i is the pre-trained positional embedding applied separately within each modality, and z_m is a learnable modality token that identifies which MRI sequence a patch came from. This lets the frozen backbone see multi-modal data as interleaved single-modality images rather than as one stitched image. Two further mechanisms carry the method: full modality masking, which drops every patch of one sequence in the student input and forces the student to reconstruct the teacher's features from the remaining sequences, thereby learning cross-modality relationships; and a sem","core_discovery":"On the paper's own terms, the discovery is that the gap between a uni-modal natural-image foundation model and multi-modal medical imaging is not architectural but representational. By replacing the single set of positional embeddings with per-modality positional embeddings plus a learnable modality embedding for each MRI sequence, the same pre-trained DINOv2 transformer can consume patches from several modalities without corrupting spatial or modality identity. Full-modality masking—dropping all patches of one sequence from the student while the teacher sees all—turns the masked-patch objective into a cross-modality prediction task, which is what gives robustness to missing sequences. Final","pith_inferences":["The single-slice evaluation design suggests the method's true ceiling may be higher under volumetric aggregation: a 3D or multi-slice version could combine the cross-modality robustness with spatial context, a direction the paper names but does not test.","A stratified missing-modality analysis by which sequence is absent would likely show that robustness is strongest when the missing sequence is redundant; if the most diagnostically informative sequence is missing, performance may not match the average reported here.","Because the semi-supervised gain is uneven across classes and worsens the rare oligodendroglioma F1 on the external set, class-balanced pseudo-labeling or oversampling of rare subtypes is a natural extension.","The modality-embedding idea is not MRI-specific; it should transfer to any multi-image modality stack (CT/PET, multiparametric imaging) and to settings where scans arrive at different times."],"forward_implications":["If correct, any DINOv2 checkpoint can be adapted to a multi-modal medical task by changing the embedding head and masking schedule, without altering the transformer backbone.","Semi-supervised pre-training on unlabeled MRI archives should become a standard first step for classification tasks with few labels: the authors report it beats fully supervised fine-tuning.","The full-modality masking result implies models can be deployed in clinics where one sequence is missing, with a smaller accuracy drop than models trained on complete inputs only.","The ablation ordering implies future multi-modal adaptations should keep per-modality position encoding and modality tokens, since naive modality concatenation or RGB stacking actively hurts performance."],"supporting_citations":[{"why":"Supplies the pre-trained DINOv2 vision transformer that the paper adapts; its student-teacher setup and patch- and image-level objectives are the starting point for all modifications.","marker":"[20]"},{"why":"Defines the ViT architecture with patch and positional embeddings that the multi-modal patch embedding modifies.","marker":"[8]"},{"why":"Provides the patch-level masked-image modeling objective that full modality masking extends.","marker":"[30]"},{"why":"Supplies the semi-supervised clustering loss whose teacher pseudo-labels the paper replaces with real labels when available.","marker":"[9]"},{"why":"The RGB-channel DINOv2 baseline that treats three MRI sequences as an RGB image; the paper must beat it to show its multi-modal embedding matters.","marker":"[14]"},{"why":"The large public multi-institutional MRI dataset used as unlabeled data in semi-supervised training.","marker":"[2]"},{"why":"Documents the clinical prevalence of missing MRI sequences, motivating the full-modality masking objective.","marker":"[21]"},{"why":"A strong supervised convolutional baseline whose external MCC the paper directly compares against in the results table.","marker":"[13]"}],"fun_headline_variants":["Foundation model adapts to multi-modal MRI with 11% gain","MM-DINOv2: Multi-modal MRI beats supervised by 11.1%","Missing MRI? New model stays accurate with masking trick","Per-modality embeddings make DINOv2 multimodal","Semi-supervised multimodal MRI model tops SOTA"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"That one 96x96 axial slice through the tumor's middle captures enough molecular-subtype information to classify a patient; if subtype-defining features live in other slices or in volumetric texture, the external-test gain may not represent patient-level diagnostic accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Foundation model adapts to multi-modal MRI with 11% gain","MM-DINOv2: Multi-modal MRI beats supervised by 11.1%","Missing MRI? New model stays accurate with masking trick","Per-modality embeddings make DINOv2 multimodal","Semi-supervised multimodal MRI model tops SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000569,"raw_usage":{"total_tokens":2539,"prompt_tokens":765,"completion_tokens":1774,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1688}},"tokens_in":509,"tokens_out":1774,"duration_ms":12496,"temperature":1.0,"reasoning_tokens":1688,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:21:47.360312+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an external evaluation in which every patient is classified from the full 3D volume rather than a single 96x96 axial middle slice, with molecular subtype labels as ground truth; if patient-level MCC does not track the reported slice-level 0.60, the gains are slice-level and may not transfer clinically.","supporting_citations":[{"cited_title":"Transactions on Machine Learning Research (jul 2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained DINOv2 vision transformer that the paper adapts; its student-teacher setup and patch- and image-level objectives are the starting point for all modifications."},{"cited_title":"In: International Conference on Learning Representations (oct 2020) 10 D","cited_arxiv_id":null,"evidence_quote":"Defines the ViT architecture with patch and positional embeddings that the multi-modal patch embedding modifies."},{"cited_title":"In: International Conference on Learning Representations (oct 2021)","cited_arxiv_id":null,"evidence_quote":"Provides the patch-level masked-image modeling objective that full modality masking extends."},{"cited_title":"In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the semi-supervised clustering loss whose teacher pseudo-labels the paper replaces with real labels when available."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The RGB-channel DINOv2 baseline that treats three MRI sequences as an RGB image; the paper must beat it to show its multi-modal embedding matters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The large public multi-institutional MRI dataset used as unlabeled data in semi-supervised training."},{"cited_title":"Scientific Reports13(1), 18911 (2023)","cited_arxiv_id":null,"evidence_quote":"Documents the clinical prevalence of missing MRI sequences, motivating the full-modality masking objective."}],"review_version":1}