{"id":"17192776-a357-4872-a35f-f81c426f92e3","arxiv_id":"2608.12689","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A 4-billion-parameter vision-language model with a shared 3D MRI encoder and 4D rotary position encoding reports BERTScore 0.856 for mpMRI report generation and 0.912 multiple-choice accuracy on its internal dataset.","lead":"The paper introduces Mr3D-VL, a 4-billion-parameter AI model that reads several 3D brain MRI scan types together and answers questions or writes radiology reports in natural language. It reports large gains over existing medical and general vision-language models, but the evaluation setup makes those gains hard to verify.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on an evaluation where the test set is drawn from the same LLM-generated SFT distribution and baselines are zero-shot and 2D-sliced; reported margins may reflect distribution familiarity rather than capability.","rationale":"Read in good faith, Mr3D-VL is a coherent engineering contribution: the shared encoder, 4D positional encoding, multi-resolution injection, and three-stage pre-training are described in detail, and the appendix reports consistent ablation improvements (e.g., Figures .8, .11, .12). The problem is not architectural incoherence; it is that the evidence for the headline claim is not capable of supporting it. §5.1 explicitly draws the evaluation from the same LLM-generated SFT distribution used for training, and §5.3 compares against zero-shot baselines, including Hulu-Med evaluated via 2D slicing despite native 3D support. A model fine-tuned on a distribution will naturally score far higher on that distribution than models seeing it for the first time; this explains the reported margins without requiring any genuine capability advantage. The paper also omits code, data, weights, and confidence intervals, so the claim is not independently reproducible. The reader's weakest assumption—that the evaluation protocol measures real capability—identifies the same load-bearing flaw, and my read does not change the REJECT verdict. An external, human-scored comparison with all baselines fine-tuned would settle the question.","tokens_in":20023,"tokens_out":5180,"duration_ms":53224,"concrete_test":"Re-run the §5.3 comparison on an external mpMRI corpus with expert-written reports and expert-authored QA (e.g., BraTS 2021/2023 augmented with radiologist reports, or an institutional set of ≥200 cases) using patient-level splits. Fine-tune every baseline—Hulu-Med in native 3D mode, Qwen3.5 and Lingshu in their native multimodal protocols—on the same SFT data used for Mr3D-VL, and score open-ended answers with blinded radiologist judgments alongside the QWen3.5-27B judge. Report per-metric means and confidence intervals. If Mr3D-VL no longer significantly exceeds the best fine-tuned baseline, the SOTA claim fails; if it retains the margin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('Mr3D-VL surpassed all competing models', §5.3) depends on the evaluation protocol isolating real 3D mpMRI capability. It does not. §5.1 states that test data 'primarily originated from the fine-tuning dataset constructed during the post-training phase'—i.e., the same LLM-generated SFT pipeline used to train Mr3D-VL (Table 2: ~100k open-ended QA, ~100k multiple-choice, ~89k report-generation samples). Baselines are not fine-tuned on that distribution; Hulu-Med, which natively supports 3D volumes, is evaluated through 2D slicing; and open-ended accuracy is judged by QWen3.5-27B rather than human experts. Under this protocol, Mr3D-VL's large margins (BERTScore 0.856 vs 0.733 for the strongest zero-shot baseline, open-ended accuracy 0.713, multiple-choice accuracy 0.912) are exactly what would be expected from a model SFT-tuned to the eval distribution confronting zero-shot baselines. The paper provides no external or human-validated test set, no confidence intervals, and no code/data/weights, so the reported SOTA cannot be attributed to the architecture rather than to evaluation circularity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mr3D-VL, a 4-billion-parameter vision-language foundation model for multiparametric 3D brain MRI. The system combines a shared MedNeXt 3D encoder pretrained with DINOv2-style self-supervision, a 4D rotary positional encoding scheme, and multi-resolution visual feature injection into a Qwen3-4B language model. Training uses a private dataset of 103,605 mpMRI cases (460,541 images), augmented by an LLM-based pipeline that rewrites radiology reports and synthesizes question-answer pairs and multiple-choice questions. The authors report results on report generation, open-ended VQA, and multiple-choice VQA, claiming state-of-the-art performance, e.g., BERTScore 0.856 for report generation, open-ended QA accuracy 0.713, and multiple-choice accuracy 0.912.","tokens_in":20275,"tokens_out":5013,"duration_ms":54400,"significance":"If the reported results were supported by a sound evaluation, the paper would make a useful contribution: it addresses a real gap in 3D mpMRI vision-language modeling, and the architectural components (shared modality-agnostic encoder, 4D-RoPE, multi-resolution feature injection, DINOv2 pretraining) are well motivated and individually ablated in the appendix. The scale of the private dataset and the efficiency analysis (Table 3) are also assets. However, the central claim that Mr3D-VL 'surpassed all competing models' is not established by the evidence: the test labels are generated by the same LLM pipeline used for training, baselines are evaluated in a disadvantaged protocol, and the accuracy numbers lack external or human validation and statistical confidence. The current evaluation is therefore insufficient to support the paper's headline conclusions.","major_comments":[{"comment":"The evaluation is circular with respect to the claimed state-of-the-art results. The test data 'primarily originated from the fine-tuning dataset constructed during the post-training phase' (§5.1), and that dataset was produced by the same LLM-based generation pipeline that created the SFT training data (§4.2.1, Table 2). Even though the split is by patient, the test labels are machine-generated and drawn from the same distribution on which the model was fine-tuned. Consequently, the large margins in §5.3 (e.g., BERTScore 0.856 vs. 0.733 for the strongest zero-shot baseline) may reflect distribution familiarity rather than genuine 3D mpMRI capability. An external benchmark with expert-written reports and human-verified QA, or at least a held-out set whose labels were not produced by the training pipeline, is required before any superiority claim can be accepted.","section":"§5.1, Table 2"},{"comment":"The baseline comparison is not fair. Lingshu and the Qwen3.5 series are evaluated by slicing 3D volumes into 2D image stacks, and HuluMed—which the paper itself states supports volumetric data—is also 'uniformly adopted' into the multi-image 2D-slicing protocol despite native 3D support. This disadvantages 3D-capable baselines. Moreover, no baseline is fine-tuned on the mpMRI SFT data, so the comparison is effectively zero-shot baselines versus a model fine-tuned on the test distribution. The claim in §5.3 that Mr3D-VL 'surpassed all competing models' is therefore not a controlled comparison; fair baselines should be fine-tuned on the same data and, for HuluMed, evaluated in its native 3D mode.","section":"§5.1, Tables 4–6"},{"comment":"The accuracy numbers are not statistically or clinically validated. Open-ended QA accuracy is judged by QWen3.5-27B acting as an 'external expert judge', but this model is itself one of the baselines in Table 4, and no human agreement, calibration, or error analysis is reported. Multiple-choice accuracy is derived by post-processing, but no confidence intervals, significance tests, or per-category breakdowns are provided for any metric in Tables 4 and 5. Without such validation, the claimed improvements (e.g., 0.713 open-ended accuracy and 0.912 multiple-choice accuracy) cannot be distinguished from noise or judge bias.","section":"§5.3, Figure 4"},{"comment":"The reliability of the LLM-generated labels is explicitly acknowledged as a limitation in the paper itself: Future Work calls for 'Automated Data Quality Control' to 'remove samples with semantic drift or hallucinations' and for a 'Post-Training Paradigm Shift' to incorporate human feedback. These statements indicate that the training and test labels have not been human-validated. Since the reported evaluation depends entirely on these labels, the manuscript should either provide evidence of label quality (e.g., human review of a sample, agreement statistics) or temper the SOTA claim accordingly.","section":"§4.2.1, §5.4 Future Work"}],"minor_comments":[{"comment":"There are numerous typographical and OCR-style artifacts throughout, including 'corss modality', 'ia', 'ns', and inconsistent terminology between 'M3RoPE' and 'M4RoPE' in §3; these should be cleaned up before resubmission.","section":"§1, §3"},{"comment":"The data counts in Tables 1 and 2 lack clear units; the text refers to 'approximately 1 billion tokens' for pre-training, but the table entries (e.g., 2,711,712) appear to be sample counts. Please clarify whether these are cases, samples, or tokens, and state the exact train/test split sizes.","section":"Table 1, Table 2"},{"comment":"The claim that QWen3.5-27B shows a 'weaker positive correlation' than Mr3D-VL between ROUGE-L and BERTScore is made without reporting correlation coefficients; please provide the quantitative values and, ideally, confidence intervals.","section":"§5.4, Figure 6"},{"comment":"Figure cross-references appear inconsistent: Figure .11 is referenced for both the multi-resolution token injection experiment and the DINO-vs-CLIP pretraining experiment, while Figure .12 is also referenced for DINO-vs-CLIP. Please renumber the figures and correct the references.","section":"Appendix, Figures .11–.12"},{"comment":"The relative improvement claims are inconsistent with the tables: for example, BERTScore 0.856 vs. 0.733 is about 17% relative improvement, not 'over 20%', and open-ended BLEU-4 0.525 vs. 0.112 is far more than 30%. Please report absolute differences or define the relative-change formula explicitly.","section":"§5.3"}],"recommendation":"reject","confidential_remarks":"I concur with the reader's assessment: the central evaluation is circular, and the baseline protocol is biased in the proposed model's favor. The architectural ideas may have merit, but the current evidence does not support the claimed state-of-the-art performance. A major redesign of the evaluation—using external human-validated labels, fair fine-tuned baselines, and native 3D inference for 3D-capable baselines—would be needed before the paper could be considered for publication. I also note that several references have dates in 2025/2026 and one reference is the manuscript's own arXiv ID; the reference list should be checked for accuracy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is a real engineering effort with a couple of genuinely new pieces, but the headline numbers don't mean what the paper says they mean. The test set is drawn from the same LLM-generated SFT pipeline used to train the model, and the baselines are zero-shot and evaluated in a mode that disadvantages them. That's enough to sink the comparison.\n\nWhat's actually new: the specific combination of a shared MedNext encoder pre-trained with DINOv2, 4D rotary position embeddings that extend MRoPE with a depth axis, and DeepStack-style multi-resolution injection into a Qwen3-4B LLM. I don't know of another mpMRI VLM doing all three. The data generation pipeline—parsing reports, using segmentation/detection outputs to ground text, generating QA pairs—is substantial, and the efficiency table (FLOPs and memory versus 4B/7B baselines) is a useful datapoint. The appendix ablations, while informal, give some support for the 4D RoPE, shared encoder, and multi-resolution injection choices.\n\nWhere it falls apart: the evaluation protocol. Section 5.1 says test data 'primarily originated from the fine-tuning dataset constructed during the post-training phase.' In other words, the model is tested on more of its own training distribution. Baselines like Qwen3.5 and Lingshu are not fine-tuned on that data, and Hulu-Med, which natively supports 3D, is run in 2D-slicing mode. That makes the margins (BERTScore 0.856 vs 0.733 best baseline; open-ended accuracy 0.713) exactly what you'd predict from SFT on the eval distribution versus zero-shot. No confidence intervals, no significance tests, no external or human-validated test set. Open-ended answers are judged by Qwen3.5-27B rather than clinicians. None of this supports 'surpassed all competing models.'\n\nThere are smaller issues too: the paper claims four key innovations but at least two (LLM-driven data generation, shared encoder) are variations on existing ideas; the writing has typos and the 4D-RoPE section is a bit muddled about time vs depth axes. But those are minor next to the evaluation.\n\nI'd like to see this with an external clinical dataset, fine-tuned baselines, and human or at least radiologist-validated ground truth. As it stands, the architecture deserves attention but the empirical claims don't.\n\nFor peer review: send it out, but the reviewers should hold it to a fair comparison. It's not a desk reject; it's a paper that needs a serious revision. I'd bring it to reading group as a cautionary example.\n\nBest.","headline":"A plausible architecture with a couple of genuinely new pieces, but the evaluation protocol is circular enough that the headline SOTA claim does not hold.","tokens_in":20816,"tokens_out":2192,"would_cite":false,"duration_ms":20966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 4B vision-language model built for volumetric mpMRI outperforms 2D-slicing baselines on brain MRI report generation and question answering.","keywords":["multiparametric MRI","3D vision-language model","medical report generation","visual question answering","rotary position embedding","self-supervised pretraining","brain MRI","multimodal learning"],"falsifier":"Re-run the head-to-head with baselines fine-tuned on the same SFT data and score a held-out set of real clinical mpMRI reports by blinded radiologists; if the fine-tuned baselines match Mr3D-VL or the expert accuracy on open-ended questions falls well below 0.713, the central claim collapses.","tokens_in":19799,"feed_emoji":"🧠","tokens_out":6993,"duration_ms":65505,"temperature":0.7,"pith_summary":"The paper sets out to show that a vision-language model designed for native volumetric processing can serve as a generalist assistant for multiparametric brain MRI, handling several imaging sequences at once and answering clinical questions in natural language. It claims that Mr3D-VL, a 4-billion-parameter model, surpasses existing general-purpose and medical vision-language models on mpMRI report generation and visual question answering, with BERTScore 0.856, open-ended QA accuracy 0.713, and multiple-choice accuracy 0.912. The result would matter because current models either flatten 3D volumes into 2D slices, losing spatial relationships and inflating token counts, or handle only a single modality, whereas real neuro-oncology decisions require combining sequences such as T1, T2, FLAIR, and DWI. If the claim holds, hospitals could query a model about tumor location, signal behavior, and cross-modality findings and receive interpretable, report-quality text.","feed_headline":"Volumetric 4B model beats 2D-slice baselines on brain MRI reports","feed_subtitle":"It reports BERTScore 0.856 on report generation and 0.912 on multiple-choice MRI questions.","key_machinery":"The load-bearing object is the 4D rotational positional embedding (4D-RoPE), which extends rotary position encoding from the usual time-height-width axes to include depth, so that a 3D volume keeps its native geometry inside the LLM instead of being flattened into slice sequences. Around this sit two further mechanisms: a shared modality-agnostic 3D encoder (a MedNext-style CNN pretrained with DINOv2-style self-supervision) that lets one set of weights handle every MRI sequence without modality-specific branches, and a multi-resolution vision token injection strategy, inspired by DeepStack, that draws low-, medium-, and high-resolution feature maps from the encoder and feeds them into the projection ViT and then into multiple early layers of the language model. Together these give the model positional awareness in volumetric space, cross-modal feature sharing, and simultaneous access to global anatomy and fine lesion detail.","core_discovery":"On its own terms, the paper's central discovery is that a comparatively small model can dominate much larger and domain-specific predecessors on 3D mpMRI language tasks when the architecture respects volumetric structure. Mr3D-VL shares one modality-agnostic 3D encoder across all MRI sequences, pretrains it with self-supervised DINOv2-style learning, positions tokens in a four-dimensional coordinate system of time, depth, width, and height via rotational embeddings, and injects low-, medium-, and high-resolution visual features into the LLM decoder. Across the paper's benchmarks the model reports a BERTScore of 0.856, METEOR 0.496, and CIDEr 0.655 for report generation, BLEU-4 0.525 and BERTScore 0.794 for open-ended QA, and accuracy of 0.713 and 0.912 on open-ended and multiple-choice questions, all stated as surpassing the tested 4B/7B/30B rivals while using far fewer FLOPs and less memory.","pith_inferences":["A direct way to separate architectural gains from data advantages is to fine-tune the baselines on the same SFT set; if they close the gap, the difference is largely the dataset rather than 4D-RoPE.","The open-ended accuracy judge is another LLM (Qwen3.5-27B); a small-sample human radiologist rating study would show whether the 0.713 figure reflects expert agreement.","The test split is drawn from the same LLM-generated corpus as training, so a prospective evaluation on real clinical reports with expert ground truth would be the stronger demonstration of clinical utility."],"forward_implications":["If the reported numbers are right, a 4B model can generate clinically structured radiology reports from multiple 3D MRI sequences, making deployment on modest hospital hardware feasible.","Native 4D positioning removes the need to slice volumes, which directly cuts token counts and GPU memory, and the paper reports roughly 96% lower FLOPs than a 4B generalist under five-modality input.","Because a single shared encoder handles all sequences, new MRI modalities can be added without per-modality branches; the paper presents this as a path to a broader mpMRI foundation model.","Stable BERTScore and METEOR as the number of input modalities grows suggests native multimodal training avoids the semantic degradation the paper observes in medical baselines.","If the approach generalizes beyond brain MRI, the same architecture could be retrained for prostate, liver, and spine mpMRI, which the paper explicitly lists as next steps."],"supporting_citations":[{"why":"Hulu-Med, the 3D medical VLM baseline whose slice-based treatment of volumes the paper argues is inadequate.","marker":"[15]"},{"why":"Lingshu, the domain-specific medical baseline that Mr3D-VL must beat on reports and QA.","marker":"[43]"},{"why":"Qwen3-VL/Qwen3.5 series, the general-purpose baselines and the source of the DeepStack-style inspiration.","marker":"[2]"},{"why":"Qwen2-VL's multimodal RoPE, the positional encoding scheme that 4D-RoPE extends to include depth.","marker":"[48]"},{"why":"MedNext, the CNN backbone chosen for the shared 3D visual encoder.","marker":"[34]"},{"why":"DINOv2-style self-supervised pretraining that removes dependence on paired image-text data for encoder pretraining.","marker":"[30]"},{"why":"DeepStack, the deep visual token injection strategy the multi-resolution implantation builds on.","marker":"[26]"},{"why":"mpLLM, the earlier multimodal 3D MRI approach whose feature pooling and modality-specific experts motivate the shared-encoder design.","marker":"[47]"}],"fun_headline_variants":["4B Mr3D-VL beats 30B models on 3D brain MRI tasks","Volumetric 4B MRI model outranks 30B rivals","Mr3D-VL: 4B model tops 3D MRI benchmarks","3D-aware 4B model for MRI language tasks beats 30B","Mr3D-VL: 4B parameters, 30B-class performance on 3D MRI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation measures real mpMRI capability: test cases come from the same LLM-generated dataset used for training, the rival models are not fine-tuned on that data, and open-ended answers are judged by an LLM rather than clinicians; if that protocol is not a fair measure, the reported superiority does not follow.","fun_headline_variants_meta":{"raw":{"variants":["4B Mr3D-VL beats 30B models on 3D brain MRI tasks","Volumetric 4B MRI model outranks 30B rivals","Mr3D-VL: 4B model tops 3D MRI benchmarks","3D-aware 4B model for MRI language tasks beats 30B","Mr3D-VL: 4B parameters, 30B-class performance on 3D MRI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001227,"raw_usage":{"total_tokens":5091,"prompt_tokens":1044,"completion_tokens":4047,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":3935}},"tokens_in":660,"tokens_out":4047,"duration_ms":26940,"temperature":1.0,"reasoning_tokens":3935,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:32:05.070080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the head-to-head with baselines fine-tuned on the same SFT data and score a held-out set of real clinical mpMRI reports by blinded radiologists; if the fine-tuned baselines match Mr3D-VL or the expert accuracy on open-ended questions falls well below 0.713, the central claim collapses.","supporting_citations":[{"cited_title":"2024 , archivePrefix=","cited_arxiv_id":null,"evidence_quote":"Hulu-Med, the 3D medical VLM baseline whose slice-based treatment of volumes the paper argues is inadequate."},{"cited_title":"2024 , eprint=","cited_arxiv_id":null,"evidence_quote":"MedNext, the CNN backbone chosen for the shared 3D visual encoder."}],"review_version":1}