{"id":"a2b57ce5-c576-4b27-9ce0-8b65016d888b","arxiv_id":"2509.09064","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 3D encoder pretrained with GPT-4V slice captions and partial optimal transport alignment beats vision-only SSL baselines on several medical tasks, but a key evaluation dataset may overlap with pretraining.","lead":"Med3DInsight trains a 3D medical image encoder using 2D slice captions generated by GPT-4V, then reports gains over self-supervised baselines on CT and MRI segmentation and classification tasks. The main caveat is that one evaluation dataset, MSD-Liver, appears to overlap with the pretraining data, so the claimed unseen transfer is not fully clean.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'unseen' claim for MSD-Liver is contradicted by 3DSeg-8's known LiTS composition; the central generalization gain may be inflated.","rationale":"The reader's weakest_assumption is precisely the load-bearing concern I identify: the claim that MSD-Heart, MSD-Liver, and MSD-Colon are 'unseen during pretraining' (Section IV.B) is contradicted by the likely composition of 3DSeg-8, which is the Med3D collection [1] and is known to include LiTS-derived liver volumes. Since MSD-Liver is the same LiTS data, the model has been pretrained on the exact evaluation volumes, making that row a memorization test rather than a cross-domain transfer test. The paper's own list of overlapping tasks (Hippocampus, Prostate, Pancreas, Vessel, Spleen) omits liver, which is itself a red flag. This concern is central because the headline quantitative claim (over 1% average Dice and 0.7mm HD95 improvement) includes the contaminated MSD-Liver row. I do not think this invalidates the entire method; the architecture is interesting, the ablations are reasonable, and the code/data release is a plus. But the evaluation's validity must be restored by removing any overlapping pretraining volumes and re-running the downstream tasks. The reader's CONDITIONAL verdict is therefore appropriate, and I would not change it.","tokens_in":25859,"tokens_out":7129,"duration_ms":68905,"concrete_test":"Inspect the Med3D/3DSeg-8 dataset manifest to confirm whether liver volumes originate from LiTS (and whether any colon volumes overlap with MSD-Colon). Then pretrain Med3DInsight from scratch on the same triplets but with all LiTS-derived (and any overlapping colon) volumes removed, and fine-tune on MSD-Liver/MSD-Colon with the same protocol. Compare Dice/HD95 to the original model and to vox2vec. If the advantage on these rows disappears or the average over the remaining datasets falls below the claimed 1% / 0.7mm, the headline generalization claim requires revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section IV-C: 'consistently outperforms SOTA SSL methods over eight datasets') depends on the assertion in Section IV.B that 'MSD-Heart, MSD-Liver, and MSD-Colon are unseen during pretraining.' That assertion is the load-bearing condition. However, the pretraining corpus 3DSeg-8 is the Med3D collection [1], which is known to assemble its liver subset from LiTS. MSD-Liver is the same LiTS dataset (131 CT scans). If this overlap exists, then the MSD-Liver row in Table IV(j) is not a cross-domain transfer test but a domain-familiarity test; the model has seen the exact liver volumes during pretraining, so its 88.41 vs. vox2vec's 87.25 Dice gain is at least partly memorization. The paper's own enumeration of overlaps between 3DSeg-8 and MSD (Hippocampus, Prostate, Pancreas, Vessel, Spleen) omits Liver, and possibly Colon; this inconsistency strengthens the concern. This does not prove the method is useless, but it directly undermines the 'unseen transfer' interpretation and the quantitative headline. The fix is straightforward: verify the composition of 3DSeg-8 and re-evaluate on a checkpoint trained without LiTS-derived volumes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Med3DInsight proposes a pretraining framework for 3D medical image encoders. For each 3D volume in 3DSeg-8 and M3D, the method samples one 2D slice, generates a text description with GPT-4V, and fine-tunes CLIP on the resulting slice-text pairs. A 3D encoder-decoder (nnFormer) is then trained with two objectives: a plane-slice-aware transformer (PSAT) aligns 3D volume features with CLIP image/text features using a mini-batch partial optimal transport (mPOT) loss, and an L2 reconstruction loss preserves low-level detail. The pre-trained encoder is evaluated by fine-tuning on ten segmentation datasets (Table IV) and two classification datasets (Table V), reporting improvements over six self-supervised baselines and several supervised baselines. The central claimed contribution is that language-guided semantic supervision from 2D MLLMs yields transferable 3D representations that outperform vision-only SSL.","tokens_in":26224,"tokens_out":4153,"duration_ms":44669,"significance":"If the results hold, Med3DInsight is a significant practical step: it provides a scalable, annotation-free way to inject semantic and clinical knowledge into 3D medical encoders, and the PSAT module plus mPOT alignment are plausible mechanisms for bridging 3D volume features with 2D vision-language spaces. The paper includes useful ablations (PSAT components, loss terms, alignment strategy, MLLM choice, slice sampling, preprocessing, data efficiency) and reports consistent gains across CT/MRI, segmentation, and classification. However, the headline 'unseen transfer' claim depends on the pretraining corpora being disjoint from the downstream evaluation datasets; this is questionable for MSD-Liver and, if confirmed, would inflate a portion of the reported average improvement. The mPOT formulation is also underspecified in ways that affect reproducibility. With those two issues resolved, the contribution would be solid.","major_comments":[{"comment":"The paper states that MSD-Heart, MSD-Liver, and MSD-Colon are 'unseen during pretraining' and explains that overlapping MSD tasks (Hippocampus, Prostate, Pancreas, Vessel, Spleen) were excluded from evaluation. However, 3DSeg-8 is the Med3D collection [1], whose liver subset is widely understood to be derived from LiTS; Table III lists both MSD-Liver and LiTs with 131 CT scans, and Table I counts 6,684 liver samples in the pretraining set. If LiTS-derived volumes appear in 3DSeg-8, then MSD-Liver is not unseen, and the 88.41 vs. 87.25 Dice gain in Table IV(j) is at least partly a domain-familiarity effect rather than cross-domain transfer. The paper's own enumeration of overlaps omits liver, which is internally inconsistent with the 'unseen' claim. This is load-bearing for the sentence in Section IV-C that Med3DInsight 'consistently outperforms SOTA SSL methods over eight datasets.' The","section":"Section IV.B, Table IV(j), Table I"},{"comment":"The mPOT loss is not well-defined as written. The empirical plan \\hat\\pi_b is a b×b matrix with \\hat\\pi_ii = 1/b and zero off-diagonal, so it has total mass 1, while \\pi^{X,S}_{s,\\epsilon,c} is supposed to transport only mass s. A KL divergence between a probability measure and a measure of mass s is undefined unless s=1 or the two arguments are normalized. Algorithm 1 never states how s is set or how the constraint \\Pi_s is enforced; the call 'Bregman(s, C, \\epsilon)' is not a standard interface for partial OT. The Mahalanobis matrix M is updated with 'Proj' but the feasible set (e.g., PSD, fixed trace, bounded entries) is not given. These details are necessary to reproduce Table VI(c), where mPOT is claimed to outperform contrastive learning.","section":"Section III-D.1, Eq. (2), Algorithm 1"}],"minor_comments":[{"comment":"The text says 'eight datasets' and Figure 3(a) says 'eight different 3D medical image segmentation datasets,' but Table IV reports ten downstream segmentation panels (MM-WHS, CHAOS, OASIS1, MSD-Heart, ADNI, VS, AbdomenCT-1K, LiTs, MSD-Liver, MSD-Colon), and Table IV(a) is titled 'Average over 10 datasets.' The count should be made consistent.","section":"Section IV-C and Table IV"},{"comment":"The sentence 'All SSL methods employed identical fine-tuning procedures, including pretraining hyperparameters (learning rate, batch size, training epochs) pretrain datasets, and evaluation metrics' is grammatically ambiguous. It should state exactly which pretraining datasets and hyperparameters are shared across baselines and which are method-specific; otherwise the claim of fair comparison is hard to verify.","section":"Section IV-C"},{"comment":"The double sum is written as \\sum_{i=1}^{nX} \\sum_{j=1}^{nX}, but the second index should presumably range over the target sample count (nY or nS/nT). This is a notation error that makes the transport plan dimensions unclear.","section":"Eq. (1)"},{"comment":"The row for '1' slice per volume appears to have a formatting issue (the value 92.07 is merged with adjacent columns). Please reformat for readability.","section":"Table VI(f)"},{"comment":"The modality-gap numbers (e.g., 82.48 vs. 76.07, 15.77 vs. 12.98) are reported without confidence intervals or an explanation of how many random samples/restarts were used; since t-SNE and UMAP projections are stochastic, a small repeated-run analysis would strengthen this visualization claim.","section":"Figure 3(d)"}],"recommendation":"major_revision","confidential_remarks":"The key decision point is dataset overlap. The authors must verify the composition of 3DSeg-8 and either re-run pretraining without LiTS-derived liver volumes or remove the MSD-Liver claim. This is fixable but cannot be patched with text alone. Second, the mPOT equation needs a rigorous definition and the algorithm needs the missing constraint details. If both are addressed, the paper is likely publishable; if the overlap is confirmed and no re-run is provided, the headline average Dice improvement should be substantially discounted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things stand out. First, the method is a real step in a useful direction: using 2D MLLMs to create annotation-free text supervision for 3D medical encoders. The PSAT module, one-slice-per-volume sampling, and partial-OT alignment are concrete and sensible design choices, and the ablation study gives them genuine support. Second, the central 'unseen transfer' claim has a hole: the pretraining corpus 3DSeg-8 is the Med3D collection, which includes LiTS liver volumes, and MSD-Liver is drawn from the same LiTS source. The paper's own enumeration of overlaps omits Liver (and possibly Colon). So the MSD-Liver row in Table IV is at least partly a memorization test, not a cross-domain one. That inflates the reported average gain.\n\nThat said, the bulk of the evidence is not dependent on that one dataset. The method wins on CHAOS, OASIS1, MSD-Heart, ADNI, VS, AbdomenCT-1K, LiTs, and MSD-Colon, with gains that are consistent and often large. The ablations show each component matters, and the method helps different 3D backbones. So the core idea is likely sound and worth building on.\n\nThe soft spots beyond the data leak: no comparison to GTGM, MEDIMP, or M3D, which are the most direct vision-language pretraining baselines; the mPOT loss leaves s and epsilon unspecified, which hurts reproducibility; and fine-tuning CLIP on the generated triplets is a reasonable step but a confounding factor. None of these is fatal alone, but they need to be addressed.\n\nThe paper deserves a serious referee. It is not a desk reject. The reviewer should push for a re-run without pretraining on LiTS-derived volumes, explicit reporting of POT hyperparameters, and adding VLP baselines. If those are fixed, the result is a solid subfield contribution.","headline":"Solid method with a real data-overlap problem on MSD-Liver; worth refereeing, but the headline claim needs reframing.","tokens_in":26682,"tokens_out":2603,"would_cite":false,"duration_ms":27261,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pretraining a 3D medical image encoder with 2D slice images and text descriptions generated by a multimodal large language model improves downstream 3D segmentation and classification beyond image-only self-supervised","keywords":["3D medical image understanding","vision-language pretraining","self-supervised learning","multimodal large language models","optimal transport alignment","medical image segmentation","medical image classification","plane-slice-aware transformer"],"falsifier":"Compare the actual image volumes used for pretraining with the MSD-Liver evaluation cases, e.g., by matching image hash values or patient IDs. If overlap exists, rerun the MSD-Liver evaluation with a provably disjoint pretraining corpus; if the reported Dice advantage disappears or shrinks, the cross-domain transfer claim for liver segmentation is unsupported.","tokens_in":25755,"feed_emoji":"🩻","tokens_out":5402,"duration_ms":53549,"temperature":0.7,"pith_summary":"Med3DInsight adds language supervision to 3D medical image pretraining without human annotations. For each 3D volume, it samples one 2D slice, has a multimodal large language model write a text description of that slice, and trains the volume encoder to reconstruct the volume while aligning its features with both the 2D slice and the generated text. The paper reports consistent gains over image-only self-supervised baselines on segmentation datasets spanning CT and MRI, and on two disease classification datasets. The central idea is that language descriptions carry clinical semantics that pixel-level reconstruction and contrastive learning alone miss.","feed_headline":"Text-aided pretraining beats image-only models on 3D medical scans","feed_subtitle":"Aligned to 2D slice descriptions, a 3D encoder outperforms image-only self-supervised baselines on segmentation and classification.","key_machinery":"The Plane-Slice-Aware Transformer (PSAT) is the bridge between the 3D volume and the 2D vision-language space. It uses learnable query tokens with a plane-slice position embedding that encodes which orientation and slice index the 2D image came from; the queries attend to the 3D volume features and produce projected features comparable to 2D image and text embeddings. The second piece is mini-batch Partial Optimal Transport (POT): instead of contrastive push-apart, the loss compares an identity coupling to a transport plan between volume and slice/text features, allowing some features to remain unmatched and thereby absorbing noise in LLM-generated descriptions. A reconstruction loss on the","core_discovery":"The paper's central claim is that a 3D medical image encoder learns more transferable representations when its features are aligned with 2D slice features and with text descriptions of those slices produced by a 2D multimodal large language model. A learnable query transformer, informed by the slice's plane and position within the volume, projects 3D features into the 2D image-text space. The matching loss is a mini-batch partial optimal transport that tolerates noise in generated text. The authors report consistent improvements over image-only self-supervised baselines on eight or more segmentation datasets and on classification of Alzheimer's and Parkinson's disease, interpreting the gains","pith_inferences":["The paper's claim that MSD-Liver is unseen during pretraining is load-bearing; if the LiTS liver data inside 3DSeg-8 overlaps with MSD-Liver volumes, the MSD-Liver result is a domain-familiarity test rather than a cross-domain transfer test.","The plane-slice position embedding implicitly teaches the model where a slice lives in a volume; this could be turned into an explicit pretraining signal for slice localization or 3D landmark detection.","Partial optimal transport's tolerance to noisy text suggests the method could work with lower-quality text sources such as automated radiology reports or weaker multimodal models, possibly with a different performance trade-off.","One-slice-per-volume sampling leaves most anatomical context unused; a multi-slice or volume-level text aggregation might improve performance for pathologies only visible in particular planes."],"forward_implications":["If the central claim is correct, medical image pretraining no longer needs curated image-text pairs: generated 2D descriptions from a general multimodal model can serve as supervision for 3D encoders.","The alignment recipe is encoder-agnostic: any 3D encoder and any pre-aligned 2D vision-language encoder pair could be plugged in, so gains should extend beyond the specific backbone tested.","Because one slice per volume suffices, pretraining cost stays modest while still bringing the semantic advantage of language.","Disease classification improves along with segmentation, suggesting the learned features capture pathology-related cues, not just anatomy."],"fun_headline_variants":["Text-aided pretraining beats image-only on 3D medical scans","2D LLM text helps 3D encoders segment and classify better","Pretraining with generated text improves 3D medical imaging","Aligning 3D scans to 2D text boosts medical AI accuracy","Med3DInsight leverages LLM captions for stronger 3D features"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole generalization story rests on the pretraining corpora being disjoint from downstream evaluation, specifically the claim that MSD-Liver is unseen during pretraining; if the LiTS liver data inside 3DSeg-8 overlaps with MSD-Liver, that transfer claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Text-aided pretraining beats image-only on 3D medical scans","2D LLM text helps 3D encoders segment and classify better","Pretraining with generated text improves 3D medical imaging","Aligning 3D scans to 2D text boosts medical AI accuracy","Med3DInsight leverages LLM captions for stronger 3D features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1390,"prompt_tokens":768,"completion_tokens":622,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":512,"tokens_out":622,"duration_ms":6664,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:44:03.553456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the actual image volumes used for pretraining with the MSD-Liver evaluation cases, e.g., by matching image hash values or patient IDs. If overlap exists, rerun the MSD-Liver evaluation with a provably disjoint pretraining corpus; if the reported Dice advantage disappears or shrinks, the cross-domain transfer claim for liver segmentation is unsupported.","supporting_citations":[],"review_version":1}