{"id":"7e5d0820-d7b8-4494-a9a9-815f85978b21","arxiv_id":"2608.13309","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new expert-verified benchmark of 3,920 questions on longitudinal brain MRIs shows vision-language models can order scans but cannot reliably judge change direction or volume.","lead":"This paper builds a large test for AI systems that read brain MRI scans taken over time, with questions checked by radiologists. It finds current models can put scans in order but often miss whether a lesion grew, shrank, or changed volume.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's reference answers are grounded in clinical metadata (RANO, MGMT/IDH, tumor volumes) that GPT-5 received but the evaluated models did not; the reported failure on change-type recognition may reflect missing input information rather than model reasoning ability.","rationale":"I read the paper as making two claims: (1) the benchmark is a valid instrument for longitudinal MRI reasoning; (2) current VLMs fail specifically at change-type recognition while handling temporal ordering. Claim (2) depends on claim (1), and the weakest link is the relation between the information used to construct the reference labels and the information given to the models. Section 2.3 explicitly lists RANO scores, MGMT/IDH status, and tumor volumes as inputs to GPT-5, and Section 4 never states that these are passed to the evaluated models. A model cannot be expected to answer a question whose correct response is defined by a molecular marker or tumor volume measurement that is absent from its inputs. This is not a disagreement with the field's consensus; it is an internal inconsistency between the label-generation protocol and the evaluation protocol. The reader's verdict already flags ground-truth validity and asks for a human baseline; my concern is more specific, pointing to a documented input asymmetry that would directly explain the low Sign Acc and Trend-F1 numbers. The proposed A/B test is cheap and decisive: if adding the metadata closes the gap, the paper's headline conclusion about model failure is not supported by the current experiments.","tokens_in":8663,"tokens_out":7487,"duration_ms":81032,"concrete_test":"On a stratified random sample of 200 QA pairs from the Disease Progression and Change Localization Over Time tasks, run the same zero-shot evaluation twice: once with the current image-only prompt, and once with the clinical metadata used by GPT-5 (RANO scores, tumor volumes, MGMT/IDH status, interval times) appended to the prompt. If Trend-F1 and Sign Accuracy improve substantially in the metadata condition, the original failure is confounded by missing input; if they stay low, the failure reflects model reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 states that GPT-5 was prompted with 'clinical annotations where available (RANO scores, MGMT/IDH status, tumor volumes)' plus cohort identity and temporal metadata, while Section 4 reports zero-shot evaluation using 'multi-sequence, multi-view MRI inputs' only. If the correct answers to Disease Progression and Change Localization questions depend on these annotations, then the low Trend-F1 and Sign Acc values in Table 1 are an expected consequence of withholding information from the models, not evidence of a specific deficiency in change-type recognition. The absence of a human radiologist baseline and inter-rater agreement (Section 2.3) makes this impossible to calibrate: radiologists were verifying GPT-5's answers with access to the same metadata, so the benchmark may measure alignment with privileged information rather than visual-temporal reasoning. The central claim in Section 4 therefore rests on an unverified assumption that the reference labels are recoverable from the images alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Time-Aware Multi-View MRI Benchmark, a longitudinal MRI reasoning benchmark with 3,920 expert-verified QA pairs derived from 890 patients across seven public cohorts and five task categories (Temporal Reasoning, Disease Progression, Structured Localization Guidance, Temporal Sequence Ordering, and Change Localization Over Time). Questions are generated by GPT-5 from longitudinal multi-sequence MRI inputs plus clinical metadata, then independently verified by two board-certified radiologists, with a 72% acceptance rate. The paper evaluates 16 vision-language models zero-shot on the benchmark, reporting final accuracy, a Reasoning Score, and a hand-weighted Time-Aware Composite (TAC) metric. The main findings are that models achieve moderate temporal alignment (TAC 0.57-0.80) but weak Trend-F1 (0.19-0.63) and Sign Accuracy (0.42-0.74), leading the authors to conclude that current VLMs handle temporal ordering reliably but consistently fail on change-type recognition. A secondary agentic-workflow experiment on a glioblastoma subset reports that multi-view inputs improve progression localization but degrade temporal ordering in smaller open-source models, and that all tested models score below 16% on change segmentation/quantification.","tokens_in":8899,"tokens_out":4954,"duration_ms":52519,"significance":"If the benchmark's validity holds, it would be a valuable and reusable resource: it is the first longitudinal, multi-view MRI QA benchmark of its kind, covers diverse pathologies (glioblastoma, brain metastases, neurodegeneration, vestibular schwannoma), and the authors release code, evaluation splits, and the dataset, which is a substantive community contribution. The falsifiable claim that current VLMs fail at change-direction recognition and volumetric quantification, despite strong temporal ordering, is clinically relevant and would be an important negative result for deployment decisions. However, the significance of these findings is contingent on the benchmark's ground truth being recoverable from the images the models actually see and on the evaluation metrics being statistically grounded. As written, the paper lacks a human radiologist baseline, inter-rater agreement statistics, confidence intervals, and a treatment of the metadata asymmetry between question generation and model evaluation. These gaps do not necessarily invalidate the benchmark, but they currently prevent the central 'systematic failure' conclusion from being fully supported.","major_comments":[{"comment":"The reference answers are generated by GPT-5 prompted with 'clinical annotations where available (RANO scores, MGMT/IDH status, tumor volumes)' in Section 2.3, while the zero-shot evaluation in Section 4 uses 'multi-sequence, multi-view MRI inputs' only. If any Disease Progression or Change Localization questions depend on those annotations, the low Trend-F1 and Sign Accuracy values in Table 1 are an expected consequence of withholding information from the models, not evidence of a specific deficiency in change-type recognition. Please provide an ablation that either generates reference answers without metadata or supplies the same metadata to the evaluated models, and show that the conclusions are unchanged. Without this, the central claim in Section 4 that current VLMs 'consistently fail on change-type recognition' is not established.","section":"Section 2.3 and Section 4"},{"comment":"The validation pipeline reports a 72% acceptance rate and dual radiologist approval, but it does not report inter-rater agreement (e.g., Cohen's kappa) or a human radiologist baseline on the actual image-only inputs. The two radiologists were verifying GPT-5's candidates with access to the same clinical metadata, so their approval does not calibrate whether the correct answers are recoverable from images alone. Please report inter-rater agreement, include a human baseline on a representative sample using the same inputs as the models, and analyze the persistent disagreements that were discarded (for example, whether they are concentrated in clinically ambiguous cases that matter most).","section":"Section 2.3"},{"comment":"The TAC metric is defined as TAC = 0.5*TEDS + 0.2*Trend-F1 + 0.2*SignAcc + 0.1*Coverage with no justification, no sensitivity analysis, and no formal definitions of its four components. The paper's central claims about relative model capability are expressed through TAC and its components, so load-bearing conclusions rest on this hand-weighted composite. Please provide formal definitions of TEDS, Trend-F1, Sign Accuracy, and Coverage, test the sensitivity of the conclusions to the choice of weights, and report confidence intervals or bootstrap errors for the point estimates in Tables 1 and 2, which currently have none.","section":"Section 3, Eq. (1)"},{"comment":"The multi-view ablation is performed on a single cohort subset (UCSF-GBM, 1,192 samples) with six models and no statistical error bars. The conclusion that 'multi-view inputs improve spatial localization yet degrade temporal reasoning in compact architectures' rests on per-model deltas such as Qwen3-VL-8B: -8.0 pp and MedGemma-4B: -5.8 pp, which could easily fall within prompt or sampling noise. Please provide repeated-run or bootstrap intervals, and ideally evaluate on at least one additional cohort, before generalizing this result.","section":"Section 4, Table 2"}],"minor_comments":[{"comment":"The Chronology metric reported in Table 1 is mentioned in Section 3 but never defined; please specify how it is computed and how it differs from Temporal Sequence Ordering accuracy.","section":"Section 3 and Table 1"},{"comment":"The column headers 'Global Ch., Seg. Ch., Quant., Temp. Ord., Prog. Loc.' are cryptic; please expand them to full task names in the table or a footnote.","section":"Table 2"},{"comment":"The Reasoning Score is said to be 'computed following LlamaV-o1 [21]' with no details of how reasoning consistency, temporal alignment, and factual accuracy are scored for these new MRI tasks; please provide the rubric or a reference to an appendix.","section":"Section 2.3"},{"comment":"The normalization description 'p1 top 99.5' and 'p 2 top 98' is ambiguous; clarify whether these are percentile clip bounds and whether the adaptive ceiling extension was applied consistently across all cohorts and sequences.","section":"Section 2.1"},{"comment":"The paper states there are 3,920 QA pairs from 890 patients over 3,200 timepoints but does not state whether multiple QA pairs from the same patient are kept in the same data split; please clarify the patient-level split strategy, as leakage across timepoints of the same patient could inflate scores.","section":"Section 2.3"},{"comment":"Figure 2 is dense and hard to read at its current resolution; consider splitting it into separate panels or providing a higher-resolution version.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's main strength is the released benchmark and the breadth of models evaluated. The overriding risk is the metadata asymmetry between GPT-5's question generation (which had access to RANO, MGMT/IDH, and tumor volumes) and the zero-shot model evaluation (which did not). If the reference answers are not recoverable from images alone, the benchmark measures alignment with privileged information rather than visual-temporal reasoning. This is fixable with ablations and a human baseline, but without those additions the headline conclusion is not yet supported. I also note that the Reasoning Score is adopted from LlamaV-o1 [21], which shares authors with this paper; the manuscript should state this relationship explicitly for transparency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is the first benchmark I've seen that combines longitudinal multi-timepoint MRI, multi-view planes, and structured localization guidance in one evaluation. The five-task structure is a genuine advance over MedAtlas and OmniMRI, which cover only parts of this space. The dataset itself is documented well: seven cohorts, 3,920 expert-verified QA pairs, 3,200+ timepoints, with a proper preprocessing pipeline (ANTs registration, N4 bias correction, tumor-aware slice extraction). The evaluation across 16 VLMs is thorough, and the main finding—models can order scans but struggle to characterize change direction (Trend-F1 0.19–0.63, Sign Acc 0.42–0.74) while chronology scores are high—makes a coherent, if troubling, story.\n\nNow the soft spot, and it is a real one. The GPT-5 generation prompt included clinical annotations—RANO scores, MGMT/IDH status, tumor volumes—that the evaluated models never received. If the correct answers to 'disease progression' or 'change localization' questions rely on those annotations, the reported failure on change-type recognition is partly an artifact of information asymmetry, not a pure measure of visual-temporal reasoning. The paper needs at least a human radiologist baseline on the same QA pairs, an ablation varying whether GPT-5 or the models receive the metadata, and inter-rater agreement for the radiologist verification. Without these, the 'systematically fail on the clinically most critical capability' claim is not fully calibrated.\n\nSmaller issues: no confidence intervals or significance tests, so differences between models (e.g., InternVL3.5 vs Gemini-3-Pro) are just point estimates. The axial-only versus multi-view ablation is confined to the UCSF-GBM subset and a separate agentic setup; it would strengthen the paper to report this in the main evaluation. The TAC weights are arbitrary, but since the components are reported separately, that is not a problem. The citation pattern is clean; the only self-referential element is the Reasoning Score following LlamaV-o1, which is minor.\n\nBottom line: this is a solid, valuable benchmark that deserves peer review. I would want revisions addressing the metadata privilege and adding a human baseline before treating the failure-mode conclusions as settled, but the resource itself is reusable now and will likely become a standard evaluation axis for medical VLMs.","headline":"Useful new longitudinal multi-view MRI benchmark, but the GPT-5 metadata privilege means the 'systematic failure on change-type recognition' claim needs a human baseline and an information-asymmetry ablation before it is fully calibrated.","tokens_in":9431,"tokens_out":2315,"would_cite":true,"duration_ms":23949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current vision-language models can order longitudinal MRI scans but struggle to recognize what type of change occurred, this paper claims.","keywords":["longitudinal MRI","vision-language model","medical visual question answering","disease progression","temporal reasoning","multi-view MRI","benchmark","change localization"],"falsifier":"Have a fresh panel of radiologists, blinded to the benchmark's gold labels, independently re-answer a random subsample of the 3,920 questions, and separately score the questions that were discarded during dual review; if the panel disagrees with the gold labels on a substantial fraction, or if the discarded cases are the ambiguous growth-versus-pseudoprogression ones, then the benchmark's validity—and the conclusion that models fail on change-type recognition—is measuring the generator's blind spots rather than clinical competence.","tokens_in":8474,"feed_emoji":"🧠","tokens_out":9113,"duration_ms":92041,"temperature":0.7,"pith_summary":"This paper claims that the standard way of testing vision-language models in radiology—one image, one question—skips the part of the job that matters most: comparing scans across time and across anatomical planes. To close that gap, it introduces the Time-Aware Multi-View MRI Benchmark, a set of 3,920 expert-verified question-answer pairs drawn from 890 patients and more than 3,200 longitudinal MRI timepoints across seven cohorts and four disease groups. When the authors run 16 current vision-language models on this benchmark, they find that models can usually reconstruct the chronological order of scans but systematically fail to recognize what type of change occurred, such as growth versus regression or volumetric change. The paper's claim is that this failure is not an artifact of the test, because every retained question was independently approved by two radiologists. If the benchmark is valid, it gives the field a reusable instrument for measuring whether a model can actually follow a disease over time.","feed_headline":"New MRI benchmark: AI can order scans but can't say what changed","feed_subtitle":"3,920 radiologist-verified questions over 3,200 timepoints test whether models actually track disease progression.","key_machinery":"The load-bearing instrument is the benchmark itself: 3,920 question-answer pairs in open-ended, multiple-choice, and binary formats, built by generating candidate questions from longitudinal multi-view MRI inputs and retaining only cases that two board-certified radiologists independently approved, a 72 percent acceptance decision. The paper's named evaluation object is the Time-Aware Composite (TAC) metric, defined as $TAC = 0.5 \\times \\text{TEDS} + 0.2 \\times \\text{Trend-F1} + 0.2 \\times \\text{SignAcc} + 0.1 \\times \\text{Coverage}$, where TEDS scores how well a model's predicted change sequence aligns with the reference sequence, Trend-F1 scores progression-versus-regression detection, SignAcc scores change-direction correctness, and Coverage scores completeness of interval reasoning. TAC is what turns raw answer accuracy into a claim about temporal coherence, and the five task categories are what let the authors separate simple chronology from genuine progression understanding.","core_discovery":"The central discovery, on the paper's own terms, is a capability split: temporal ordering is tractable for current vision-language models, while change-type recognition is not. Across the 16 evaluated models, Time-Aware Composite scores are moderate (0.57–0.80) and chronology scores are high, but Trend-F1 and sign accuracy are low, meaning models identify that something changed less reliably than they identify when it changed. The benchmark also produces a second finding: feeding axial, coronal, and sagittal views together improves spatial localization in a two-stage agent workflow, but the extra views degrade temporal ordering in smaller open-source models. The paper interprets this as evidence that current 2D-input architectures lack the geometric and cross-timepoint machinery needed for longitudinal MRI reasoning, and that domain specialization alone does not fix it.","pith_inferences":["A human-radiologist baseline on the same 3,920 questions is not reported; without it, a 35 percent final-accuracy figure has no calibration for what expert-level performance looks like on this hard test.","Because 28 percent of generated candidates were rejected or discarded, the benchmark may under-represent the clinically ambiguous cases—precisely the ones where progression decisions are hardest.","High chronology scores could partly reflect side information such as image order, filenames, or metadata rather than visible disease features; a version that strips those cues would test whether models truly see progression.","If the benchmark's candidate-generation step carries a bias that survives radiologist review, the reported failure pattern measures the generator's blind spots; comparing the rejected candidates' content against the retained set would reveal this."],"forward_implications":["Zero-shot performance on this benchmark becomes a reusable baseline: later work can compare longitudinal MRI models on identical questions and splits.","Current vision-language models are not ready to drive clinical progression decisions such as growth-versus-regression or volumetric measurement, even when they localize the right region.","Adding more anatomical views is not universally helpful; compact architectures need adaptive view selection to avoid degrading temporal reasoning.","The gap between strong localization and weak quantification points future work toward explicit 3D geometric priors and cross-timepoint attention rather than larger versions of the same 2D models."],"supporting_citations":[{"why":"Registers serial scans to baseline so that later timepoints align anatomically for interval comparison.","marker":"[2]"},{"why":"Provides the longitudinal brain-metastasis cohort with serial post-treatment scans.","marker":"[7]"},{"why":"Provides the post-surgical glioma cohort used for the multi-view ablation study.","marker":"[9]"},{"why":"Provides a glioblastoma cohort with molecular ground truth that grounds generated questions.","marker":"[10]"},{"why":"Provides the Alzheimer's disease cohort used for neurodegeneration progression cases.","marker":"[15]"},{"why":"Provides the vestibular schwannoma cohort with multi-year follow-up imaging.","marker":"[16]"},{"why":"Provides the aging and dementia cohort for neurodegeneration cases.","marker":"[17]"},{"why":"Provides expert RANO annotations used to ground disease-progression questions.","marker":"[20]"},{"why":"Defines the reasoning score used to grade model outputs beyond final-answer accuracy.","marker":"[21]"},{"why":"Applies bias-field correction in the preprocessing pipeline before slice extraction.","marker":"[22]"}],"fun_headline_variants":["AI MRI: good at ordering scans, bad at spotting changes","Benchmark shows AI can't tell what changed in MRI series","Temporal order yes, change type no: MRI AI benchmark","MRI AI fails change recognition despite temporal skill","Multi-view aids localization but harms temporal reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the expert-approved gold answers are clinically correct for the questions asked, so that a model missing them is actually failing at longitudinal MRI reasoning.","fun_headline_variants_meta":{"raw":{"variants":["AI MRI: good at ordering scans, bad at spotting changes","Benchmark shows AI can't tell what changed in MRI series","Temporal order yes, change type no: MRI AI benchmark","MRI AI fails change recognition despite temporal skill","Multi-view aids localization but harms temporal reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1382,"prompt_tokens":964,"completion_tokens":418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":580,"tokens_out":418,"duration_ms":5151,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:58:57.953708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh panel of radiologists, blinded to the benchmark's gold labels, independently re-answer a random subsample of the 3,920 questions, and separately score the questions that were discarded during dual review; if the panel disagrees with the gold labels on a substantial fraction, or if the discarded cases are the ambiguous growth-versus-pseudoprogression ones, then the benchmark's validity—and the conclusion that models fail on change-type recognition—is measuring the generator's blind spots rather than clinical competence.","supporting_citations":[{"cited_title":"NeuroImage (2011)","cited_arxiv_id":null,"evidence_quote":"Registers serial scans to baseline so that later timepoints align anatomically for interval comparison."},{"cited_title":"Al Ghallabi et al","cited_arxiv_id":null,"evidence_quote":"Provides the longitudinal brain-metastasis cohort with serial post-treatment scans."},{"cited_title":"Radiology: Artificial Intelligence (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the post-surgical glioma cohort used for the multi-view ablation study."},{"cited_title":"Scientific Data (2026)","cited_arxiv_id":null,"evidence_quote":"Provides a glioblastoma cohort with molecular ground truth that grounds generated questions."},{"cited_title":"Journal of Magnetic Res- onance Imaging (2008)","cited_arxiv_id":null,"evidence_quote":"Provides the Alzheimer's disease cohort used for neurodegeneration progression cases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the vestibular schwannoma cohort with multi-year follow-up imaging."},{"cited_title":"Journal of Cognitive Neuroscience (2010)","cited_arxiv_id":null,"evidence_quote":"Provides the aging and dementia cohort for neurodegeneration cases."},{"cited_title":"Scientific Data (2022)","cited_arxiv_id":null,"evidence_quote":"Provides expert RANO annotations used to ground disease-progression questions."},{"cited_title":"IEEE Transactions on Medical Imaging (2010)","cited_arxiv_id":null,"evidence_quote":"Applies bias-field correction in the preprocessing pipeline before slice extraction."}],"review_version":1}