{"id":"b260297e-88b7-4f41-8d7c-81b9b262c916","arxiv_id":"2501.14051","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A domain-specific pretrained 3D encoder plus cross-batch embedding accumulation makes CLIP-style alignment of brain MRI and tabular data feasible with 62 scans.","lead":"This paper shows that CLIP-style alignment between 3D brain MRI scans and tabular patient data can be trained with only 62 MRI scans, provided the image encoder is first pretrained on a large public brain MRI dataset and negatives are accumulated across batches. The result matters because it suggests medical multimodal models may be buildable for small, rare datasets instead of requiring millions of captioned images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scan-level split in §3.1 allows patient overlap between train and test, so the '62 scans' alignment claim may rest on identity memorization rather than modality alignment.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the scan-level split in §3.1 permits the same patient to appear in both training and test sets, allowing the model to exploit patient identity rather than modality alignment. This is the most direct threat to the central claim because the reported zero-shot AUC is the primary evidence for 'alignment feasible with only 62 MRI scans.' A patient-level split is the decisive check. I considered whether overlap between the BRAINS-45K pretraining data and the evaluation dataset is an even larger leak, but the text labels paired with same-patient scans during CLIP training create a clearer path to inflating the specific zero-shot classification numbers. Since the reader's verdict is already REJECT and this concern supports that verdict, no change is needed.","tokens_in":6451,"tokens_out":5886,"duration_ms":53633,"concrete_test":"Group the 77 scans by patient ID; perform an 80/20 split at the patient level (e.g., 38 patients train, 9 patients test) so no patient appears in both sets; retrain with the same hyperparameters and report the five-fold zero-shot AUC from Table 3. If the average AUC drops materially (e.g., by more than 0.1 or to near chance on parietal/temporal), the original scan-level result was inflated by patient identity leakage. Also report the number of test patients and the distribution of scans per patient to confirm the split is balanced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CLIP-style alignment is feasible with only 62 MRI scans depends on the zero-shot classification results in Table 3. The evaluation split described in §3.1 is an 80/20 split on the scan level, but the 77 scans come from 47 patients, with some patients contributing up to eight treatment-course scans. An 80/20 scan-level split can therefore place the same patient in both the training and test sets. Because each training text includes that patient's lesion locations and the image encoder can recognize patient-specific anatomy, the model can memorize a patient-to-lesion-location mapping. At test, recognizing the patient from the MRI is sufficient to rank the correct region sentence highly, inflating the reported AUC without requiring any genuine alignment of 3D MRI and tabular representations. The paper's limitations paragraph acknowledges missing external evaluation but does not address this identity-leakage path. Until a patient-level split is shown to preserve the result, the headline claim 'alignment is feasible with only 62 MRI scans' is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a CLIP-style contrastive alignment method for 3D brain MRI volumes and tabular patient data (age, gender, and lesion locations converted into natural-language sentences). The image encoder is a 3D encoder pretrained with the authors' AMAES framework on the public BRAINS-45K dataset, and the text encoder is a pretrained BERT model. To cope with the memory cost of 3D volumes, the authors introduce an embedding accumulation strategy that computes the CLIP loss with negatives from multiple batches while backpropagating only through the current batch. They evaluate the method on zero-shot classification of brain lesion location and zero-shot image retrieval on the BrainTR-GammaKnife dataset, training on 62 MRI scans. The reported zero-shot classification AUCs are 0.65-0.72 across three backbone architectures, above the random baseline, while retrieval results are mixed, with one backbone below random on mAP and MRR.","tokens_in":6622,"tokens_out":3322,"duration_ms":30192,"significance":"If the results hold, the paper would make a useful empirical contribution: it would show that CLIP-style alignment between 3D medical volumes and tabular data is feasible with very small training sets when both encoders are pretrained on domain-specific data. The paper also releases code and model checkpoints, and it includes ablations of the foundation-model encoders and of the accumulation frequency, which are useful for the community. The central claim, however, rests on an evaluation split that may allow patient identity leakage, and the retrieval results show unexplained instability. The absence of unimodal or supervised baselines makes it difficult to attribute the classification performance to genuine cross-modal alignment. These issues need to be resolved before the headline claim can be accepted.","major_comments":[{"comment":"The 80/20 scan-level split is not a patient-level split: the 77 scans come from 47 patients, with some patients contributing up to eight treatment-course scans. The same patient can therefore appear in both the training and test sets. Because each training caption names the lesion locations and the MRI encoder can learn patient-specific anatomy, the model could memorize a patient-to-lesion-location mapping and achieve high zero-shot classification AUC in Table 3 without learning generalizable cross-modal alignment. The limitations paragraph at the end of §4.1 acknowledges missing external evaluation but does not address this identity-leakage path. The authors should re-run the evaluation with a patient-level split (or otherwise show that overlapping patients do not drive the results); until then, the claim that alignment is feasible with 62 MRI scans is not supported.","section":"§3.1 Data"},{"comment":"The retrieval results are internally inconsistent: the ResNet encoder achieves MMR 0.18 and mAP 0.19, both below the random baseline, while Swin-T and MedNeXt beat random. The paper notes this only in passing ('interestingly Resnet does not'). A below-random retrieval performance for one of the three encoders is a sign of a collapsed or mis-scaled embedding space, and it undercuts the general claim of meaningful alignment. The authors should analyze why this occurs (for example, by inspecting embedding norms, per-label retrieval errors, or the effect of the accumulation setup) and report a concrete diagnosis rather than an anecdotal remark.","section":"Table 4 and §4.1 Discussion"},{"comment":"The paper lacks unimodal and supervised baselines that would isolate the contribution of cross-modal alignment. In Table 1, the configuration with a random image encoder and a pretrained text encoder reaches an average AUC of 0.67, which is far above chance; the paper does not explain why a non-aligned random image encoder achieves this, nor does it provide a text-only baseline that predicts lesion locations from language priors or class frequencies. Without such baselines, the claim that the image-text alignment, rather than the language model or label distribution, drives the zero-shot classification results is not established.","section":"§2.1.3 and §3.3 Evaluation Tasks"},{"comment":"The zero-shot classification evaluation is multi-label, since each patient can have lesions in multiple regions, but the one-versus-rest AUC computation is described only briefly. The paper does not specify how samples with multiple positive labels are handled in the OvR averaging, or whether the reported AUC is macro-averaged over labels. This matters for interpreting the per-region AUCs in Table 3, especially for regions with high co-occurrence. Please clarify the exact evaluation protocol.","section":"§3.3 Evaluation Tasks"}],"minor_comments":[{"comment":"The text and Table 4 use 'MMR' for mean reciprocal rank; the standard abbreviation is 'MRR'.","section":"Table 4 and §4"},{"comment":"There are formatting artifacts in the tables, such as '0 .50', '0 .87±0.1', and '0 .64±0.01'. These should be corrected to standard decimal notation.","section":"Tables 1 and 2"},{"comment":"The pseudocode in Figure 2 is helpful, but the sentence 'Computing the loss and backpropagating at each iteration of j results in the effective computation of the CLIP loss with a batch size N·B while only tracking the gradient batches with size B' is slightly ambiguous: the loss is accumulated over all j before a single optimizer step, so the gradient is computed with respect to a sum of N losses. Consider rephrasing to clarify the exact gradient computation.","section":"§2.2"},{"comment":"The description of the region-label subset says 'we define a subset of the dataset containing 222 lesions', but it is not stated how the original 244 lesions relate to this subset and whether the excluded lesions are from patients or regions outside the five chosen labels. Please clarify.","section":"§3.1 Data"},{"comment":"The t-SNE visualization in Figure 1 is presented without a detailed explanation of how the test set was augmented to increase sample size; the augmentation procedure should be described or referenced here.","section":"§1 Introduction"}],"recommendation":"major_revision","confidential_remarks":"The patient-level split issue is the most serious concern and is load-bearing for the paper's central claim. I do not see evidence of bad faith; the scan-level split appears to be a methodological oversight. The paper is well suited to a workshop or short conference format, but for a journal the leakage concern and the missing baselines must be addressed. The release of code and checkpoints is a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a likeable paper with a real problem. The genuinely new bits: it is the first native-3D CLIP training on brain MRI, and the embedding accumulation across batches is a practical way to scale negatives when 3D patches cap batch size. The ablations are clean, code and weights are released, and the limitations paragraph is honest. I believe the authors believe their own numbers.\n\nBut the headline claim—that modality alignment is feasible with 62 MRI scans—rests on an evaluation split that does not rule out patient identity memorization. The dataset has 77 scans from 47 patients, some with up to eight treatment courses, and Section 3.1 says an 80/20 split on the scan level. That means the same patient can appear in both train and test. Since the training text includes that patient's lesion locations, the model can learn a mapping from patient-specific anatomy to regions without learning anything about MRI-text alignment. The AUCs in Table 3 could therefore be inflated. This is not a minor nit; it is the load-bearing evidence for the paper's central promise.\n\nThe paper does some other things that soften the blow. The retrieval results are inconsistent—ResNet below random on mAP—which suggests the shared space is fragile. There are no supervised or unimodal baselines, so it is hard to know what gain the alignment itself provides. The 'instrumental' claim about the domain-specific encoder is also stronger than the evidence: Table 1 shows a +0.05 AUC boost, not a dramatic effect.\n\nStill, I would not desk-reject this. The idea is plausible, the method is simple, and the flaw is fixable. A patient-level split (or better, treated-vs-untreated scans within the same patient) and a supervised baseline would tell us whether the '62 scans' claim holds. The embedding accumulation trick and the ablations are worth preserving regardless.\n\nFor you: worth a serious referee, but only with the expectation that the evaluation is redone. For the authors: resubmit with a proper split and modestly reworded claims.","headline":"The embedding-accumulation trick and native-3D CLIP framing are useful, but the scan-level split undermines the headline '62 scans' claim until a patient-level split is shown.","tokens_in":7192,"tokens_out":2228,"would_cite":false,"duration_ms":19066,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP-style alignment of 3D brain MRI and tabular data is feasible with only 62 MRI scans, using a domain-specific 3D foundation model encoder and cross-batch embedding accumulation to stabilize the contrastive loss.","keywords":["CLIP","multi-modal alignment","3D MRI","tabular data","foundation models","contrastive learning","zero-shot classification","embedding accumulation"],"falsifier":"Take the same 77 scans and split strictly by patient identity, ensuring no patient contributes scans to both training and test; retrain the best Swin-T configuration and measure zero-shot AUC. If average AUC falls toward random, the small-sample alignment result is explained by patient identity leakage rather than modality alignment.","tokens_in":6240,"feed_emoji":"🧠","tokens_out":5716,"duration_ms":45896,"temperature":0.7,"pith_summary":"Most CLIP-style alignment assumes abundant data, but 3D medical imaging rarely has it. This paper claims that modality alignment between 3D brain MRI and tabular clinical data is feasible with only 62 training scans, provided the vision encoder is a domain-specific 3D foundation model and the CLIP loss is computed with an embedding accumulation strategy that supplies many more negative pairs than a single 3D batch can hold. The tabular data is converted into natural-language sentences and encoded with BERT, so text supervises the MRI embeddings. The authors show the result by zero-shot classification of brain lesion location: the best configuration reaches an average AUC of 0.72, while zero-shot image retrieval remains partially unresolved. If the claim holds, it lowers the data barrier for jointly embedding medical images and structured records.","feed_headline":"CLIP-style alignment works with just 62 MRI scans","feed_subtitle":"A 3D brain MRI encoder plus cross-batch negative pairs turns templated tabular data into zero-shot lesion classification.","key_machinery":"Embedding accumulation: for N batches, all embeddings are computed in a gradient-free pass, then for each batch j the CLIP loss is computed against the concatenated negative embeddings from all other batches while gradients flow only through batch j, so the effective contrastive batch is N times B. This works together with two pretrained encoders: a 3D Swin-Transformer (or MedNeXt/ResNet) initialized from AMAES self-supervised pretraining on BRAINS-45K, and a pretrained BERT for the textified tabular records, so the alignment step does not have to learn representations from scratch.","core_discovery":"The paper demonstrates that a shared embedding space between 3D brain MRI and tabular clinical data can be learned from just 62 training scans when the encoders are pretrained: the MRI encoder is a 3D foundation model pretrained on large-scale brain MRI data, and the tabular data is rendered as natural-language sentences and encoded by BERT. The enabling technical object is an embedding accumulation trick that gives the CLIP loss an effective batch size of N times B while storing only B-sized batches, so enough negative pairs exist to keep training stable in 3D at small sample size. On zero-shot classification of lesion location, the best model reaches an average AUC of 0.72 (Swin-T, effective batch 128), well above random, while zero-shot image retrieval remains mixed; Swin-T reaches mAP 0.34 versus 0.21 for random. The paper frames this as the first application of CLIP training to brain MRI in native 3D.","pith_inferences":["We infer that the reported positive result is sensitive to the scan-level split: with 77 scans from 47 patients and some patients having up to eight treatment courses, an 80/20 split by scan rather than by patient can let the same patient appear on both sides, so part of the apparent alignment may be patient-identity memorization. Re-splitting by patient is a direct test.","We infer that embedding accumulation is not specific to MRI: any modality where memory limits batch size, such as video, whole-slide pathology, or larger 3D volumes, could benefit from the same effective-batch enlargement.","We infer that the template-sentence encoding of tabular data leaves headroom; replacing fixed templates with free-text clinical narratives could test whether richer language supervision improves the joint embedding space.","We infer that the different ranking of MedNeXt and ResNet between classification and retrieval indicates the small-sample joint space is not uniformly robust, so task-specific evaluation remains necessary."],"forward_implications":["A CLIP-style objective can be applied to native 3D medical imaging with dataset sizes typical of clinical studies, not only to web-scale caption collections.","Pretrained, domain-specific encoders are load-bearing for small-sample alignment; Table 1 shows training is not possible without them.","Tabular clinical data can act as the textual modality by being converted into templated sentences, meaning the same pipeline extends to free-text reports and semi-structured records.","Zero-shot classification of lesion location from MRI alone is achievable, with average AUC 0.72 for the best configuration, while zero-shot image retrieval is not yet solved.","Scaling negative pairs through embedding accumulation improves stability and performance, for example Swin-T's average AUC rises from 0.64 without accumulation to 0.72 with N=16."],"supporting_citations":[{"why":"Supplies the CLIP contrastive objective that the paper revisits and adapts to 3D.","marker":"[1]"},{"why":"Shows that domain-specific medical contrastive pretraining outperforms natural-image CLIP, motivating the use of a medical image encoder.","marker":"[2]"},{"why":"Provides the AMAES masked-autoencoder pretraining framework and the BRAINS-45K dataset used to train the 3D image encoders.","marker":"[5]"},{"why":"Supplies the pretrained BERT text encoder that embeds the templated tabular sentences.","marker":"[14]"},{"why":"The reference the embedding-accumulation implementation is adapted from, enabling the cross-batch negative scaling.","marker":"[15]"},{"why":"Provides the BrainTR-GammaKnife dataset of 77 MRI scans with tabular metadata and lesion annotations used for training and zero-shot evaluation.","marker":"[16]"}],"fun_headline_variants":["62 MRI scans align 3D images with tabular data","Embedding accumulation scales CLIP negatives for 3D","Zero-shot MRI classification from only 62 scans","Swin-T CLIP hits AUC 0.72 with 62 scans","Domain-specific pretraining makes CLIP data-efficient"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that splitting the 77 scans 80/20 at scan level, with some patients contributing up to eight treatment courses, keeps the evaluation free of patient overlap between training and test; if the same patient appears on both sides, the zero-shot AUCs could reflect patient identity memorization rather than modality alignment.","fun_headline_variants_meta":{"raw":{"variants":["62 MRI scans align 3D images with tabular data","Embedding accumulation scales CLIP negatives for 3D","Zero-shot MRI classification from only 62 scans","Swin-T CLIP hits AUC 0.72 with 62 scans","Domain-specific pretraining makes CLIP data-efficient"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000849,"raw_usage":{"total_tokens":3678,"prompt_tokens":912,"completion_tokens":2766,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":2684}},"tokens_in":528,"tokens_out":2766,"duration_ms":19537,"temperature":1.0,"reasoning_tokens":2684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:25:43.816365+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 77 scans and split strictly by patient identity, ensuring no patient contributes scans to both training and test; retrain the best Swin-T configuration and measure zero-shot AUC. If average AUC falls toward random, the small-sample alignment result is explained by patient identity leakage rather than modality alignment.","supporting_citations":[{"cited_title":"Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models","cited_arxiv_id":"2501.14051","evidence_quote":"Supplies the CLIP contrastive objective that the paper revisits and adapts to 3D."},{"cited_title":"In the following, we directly ablate the result of each design decision","cited_arxiv_id":null,"evidence_quote":"Shows that domain-specific medical contrastive pretraining outperforms natural-image CLIP, motivating the use of a medical image encoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AMAES masked-autoencoder pretraining framework and the BRAINS-45K dataset used to train the 3D image encoders."},{"cited_title":"Mednext: Transformer- driven scaling of convnets for medical image segmenta- tion,","cited_arxiv_id":null,"evidence_quote":"Provides the BrainTR-GammaKnife dataset of 77 MRI scans with tabular metadata and lesion annotations used for training and zero-shot evaluation."}],"review_version":1}