{"id":"e8f585f0-3c9a-408d-8701-7f8575a8e52f","arxiv_id":"2411.15490","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PIRTA generates acute ischemic stroke radiology reports by retrieving similar 3D DWI/ADC images and augmenting LLM generation with their paired expert reports, improving ischemic-territory accuracy over direct image-to-text baselines.","lead":"A stroke-report pipeline that skips image-text alignment by retrieving similar 3D brain MRIs and recycling their clinician-written reports to ground a language model. The approach reports higher ischemic-territory accuracy than 2D multimodal baselines across internal and external datasets, positioning retrieval-augmented generation as a practical route for 3D medical report factuality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparison against GPT-4o/LLaVA-Med is confounded by 3D-vs-2D input and LLM instruction-tuning on the structured template; no ablation isolates whether retrieval-augmentation itself improves factuality.","rationale":"The paper is coherent and provides useful external validation (BRMH and ISLES), and its explicit discussion of the low-similarity failure case is honest. The reader's weakest_assumption is retrieval-faithfulness in the 3D ViT embedding space; that is a legitimate concern and is partially validated by the failure case. However, I judge the more load-bearing issue to be the lack of an ablation isolating the retrieval contribution from input dimensionality and LLM fine-tuning. This is why I say partial agreement: the reader's rationale already noted the baseline confound, but did not make it the central weakest point. If the three-way ablation shows PIRTA beating no-retrieval and scrambled-retrieval conditions, the claim is secured; if not, the paper should be recharacterized as showing that 3D input plus a template-tuned LLM beats 2D baselines. The absence of error bars and the use of the same four territory labels for SFT and evaluation further justify the conditional verdict, but they are secondary. A CONDITIONAL verdict is appropriate because the concern is concrete and addressable with existing code and data.","tokens_in":17517,"tokens_out":10142,"duration_ms":77297,"concrete_test":"Run a three-way ablation on the internal SNUH+SNUBH test set with the same fine-tuned 3D ViT and LoRA-tuned LLaMA-3-8B: (A) PIRTA with real top-5 retrieved reports; (B) the same LLM with no retrieved reports (direct 3D-conditioned generation); (C) the same LLM with top-5 reports randomly shuffled from the database. Calculate ischemic-territory Acc@1 with 95% confidence intervals. If (A) is not significantly better than (B) and (C), the claim that text-domain augmentation improves factuality over direct image-to-text generation is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that retrieval-grounded generation (PIRTA) improves factuality over direct image-to-text baselines. The comparison in Figure 3 changes three variables at once: (1) input modality (full 3D DWI+ADC via the 3D ViT vs a single 2D slice for GPT-4o/LLaVA-Med), (2) LLM preparation (LoRA instruction-tuning on the exact structured template vs one-example prompting), and (3) retrieved context (top-5 clinician reports vs no retrieval). No ablation holds the 3D encoder and the tuned LLM fixed and toggles only the retrieved reports, so the reported advantage cannot be attributed to retrieval-grounded generation. The paper's own Table 5 second example shows that when top-5 similarities are low (0.53 to 0.46), the generation is wrong — which confirms factuality is inherited from retrieval, but also shows the failure mode is known and acknowledged. The reader identified retrieval-faithfulness as the weakest assumption; my concern overlaps but is more specific: even if retrieval were perfect, the paper has not shown that the same fine-tuned 3D LLM plus template would not already match PIRTA without any retrieved text. The complexity argument (Eqs. 8-9) does not establish factual superiority; it only bounds hypothesis-space size. External validation is a genuine strength, but it does not resolve the within-pipeline confound. Therefore, the central claim is under-supported until the retrieval contribution is isolated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PIRTA, a retrieval-augmented generation framework for 3D DWI/ADC brain MRI reports. A 3D ViT is pretrained with masked autoencoding on a large unlabeled dataset, supervised fine-tuned to classify four ischemic-territory/normal labels, and then used to retrieve top-k similar images from a paired image-report database. The paired clinician-authored reports are provided as in-context text to a LoRA-tuned LLaMA-3-8B-Instruct, which generates a structured radiology report. Experiments on an internal set (SNUH+SNUBH) and external sets (BRMH, ISLES) report high retrieval mAP/Acc@k and higher ischemic-territory Acc@1 than GPT-4o and LLaVA-Med baselines, which receive only a single 2D slice and one in-context example. The paper argues that by avoiding explicit image-text alignment and instead retrieving paired reports, PIRTA improves factual accuracy of generated reports.","tokens_in":17735,"tokens_out":3469,"duration_ms":33196,"significance":"If the central claim holds, the work is a useful step toward clinically grounded report generation from volumetric brain MRI, with a practical recipe (MAE pretraining + supervised fine-tuning on a small label set + nearest-neighbor report retrieval + LLM augmentation) that avoids expensive cross-modal alignment. Strengths of the manuscript include the use of two external datasets (BRMH and ISLES), a large unlabeled pretraining corpus (UK Biobank), and the release of source code. The retrieval results in Table 4, especially the improvement from large-scale MAE pretraining, are internally consistent and are a solid empirical contribution. However, the paper's headline claim—that retrieval-grounded generation improves report factuality over direct image-to-text models—is currently under-supported because the comparison to GPT-4o and LLaVA-Med changes several variables at once, and the retrieval evaluation is tied to the same four labels used to fine-tune the encoder.","major_comments":[{"comment":"The central empirical comparison is confounded. PIRTA receives full 3D DWI+ADC volumes, five retrieved clinician-authored reports, a LoRA-tuned LLaMA-3 that has been instruction-tuned exactly on the structured template, and additional clinical context fields; GPT-4o and LLaVA-Med receive a single 2D axial slice, one in-context example of the desired format, and no retrieval or clinical context. The reported advantage in ischemic-territory Acc@1 could therefore be due to the 3D input, the LLM instruction-tuning, the extra clinical variables, or the retrieved reports rather than to retrieval-grounded generation per se. The manuscript should include an ablation that holds the 3D encoder and the LoRA-tuned LLM fixed and toggles only the retrieved text (e.g., PIRTA without retrieved reports, PIRTA with randomly sampled retrieved reports, or both). Without such an ablation, the title claim that retrieval-grounded generation improves factuality is not established.","section":"Section 3.1.2 and Figure 3"},{"comment":"The complexity argument does not support the factual-superiority claim. Equation (9) reduces to C(H_image) + C(H_text) >= C(H_image), which is trivially true because C(H_text) is non-negative. This bound concerns the size of a hypothesis space and says nothing about whether retrieved reports will be clinically relevant or whether generated reports will be factual. The discussion in Section 2.2 (\"The retrieval of relevant text information from the paired database ensures factual accuracy\") asserts rather than demonstrates the link between retrieval relevance and report factuality. The paper needs a quantitative analysis connecting retrieval quality (e.g., top-5 agreement or similarity score) to generation accuracy, or a separate experiment that corrupts the retrieved context to show its causal effect.","section":"Section 2.2, Eqs. (8)-(9)"},{"comment":"The retrieval evaluation is circular with respect to the generation target. The image encoder is supervised fine-tuned to classify exactly the four categories (normal, anterior, deep gray, posterior) that define relevance in Table 4, and the generated-report evaluation in Figure 3 uses the same four-category ischemic-territory accuracy. Thus the high mAP@1 and Acc@1 values in Table 4 demonstrate that the encoder can recognize the four training labels on internal and external data; they do not independently establish that the top-k retrieved reports contain the clinically relevant findings for the query image. The authors should either evaluate retrieval with a clinician-defined relevance judgment that goes beyond the four territory labels, or explicitly acknowledge that Table 4 measures a recognition performance that is partially enforced by the SFT stage.","section":"Table 4 and Section 3.1.1"},{"comment":"The paper's own failure case shows that when top-5 similarity scores are low (0.53 to 0.46), the generated report is wrong. This is an honest and informative observation, but the manuscript does not quantify how often such low-confidence retrievals occur or how generation accuracy varies with retrieval similarity. Given the paper's central thesis that factuality is inherited from retrieval, the authors should report generation Acc@1 stratified by retrieval similarity (e.g., bins of top-1 or top-5 similarity) or by retrieval success/failure on the internal and external test sets. This would also provide a clinically useful operating characteristic for PIRTA and would prevent the reader from having to rely on a single qualitative example.","section":"Section 3.2.2 / Table 5 and Discussion"}],"minor_comments":[{"comment":"The handling of the MAE mask ratio is unclear: the text says \"a random mask is applied to remove (1-q) of the patches\" and later sets \"mask ratio q of 0.25,\" which would mean 75% of patches are removed; typically a 0.25 mask ratio means 25% are removed. Please clarify the intended keep/remove proportion.","section":"Section 2.3.2"},{"comment":"No details are provided on how the \"center of the stroke lesion\" slice was identified for GPT-4o and LLaVA-Med; if lesion localization was used to choose the input slice, the baseline setup should be described precisely, and the authors should discuss whether this favors or disadvantages the baselines.","section":"Section 3.1.2"},{"comment":"The figure and tables report point estimates without confidence intervals, error bars, or significance tests. Given the small numbers in some external classes (e.g., ISLES deep-gray n=52 in Table 2), the authors should report variance or at least a statistical comparison for the main Acc@1 differences.","section":"Figure 3 and Table 6"},{"comment":"The text in Section 2.3.1 says the ISLES 2022 dataset includes 211 subjects, but Table 2 lists 206 ISLES test samples; please reconcile this discrepancy.","section":"Table 2"},{"comment":"Several references are incomplete or formatted inconsistently (e.g., Ram et al., Borgeaud et al., and Karpukhin et al. contain ellipses in the author lists, and some entries have inconsistent venue naming). Please ensure all references are complete and uniform.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core architecture is reasonable and the external validation is a genuine strength. However, the headline comparison currently bundles 3D input, LLM fine-tuning, clinical context, and retrieval into a single treatment, so the specific contribution of retrieval is not isolated. This is fixable within the manuscript's scope: an ablation that toggles the retrieved reports while keeping the 3D encoder and tuned LLM fixed would directly address the confound. I also recommend that the authors add a quantitative link between retrieval confidence and generation accuracy, which would strengthen the central retrieval-inheritance thesis. If the authors are unwilling or unable to add these experiments, I would suggest reframing the central claim from \"retrieval improves factuality\" to \"a retrieval-augmented 3D MR-to-report pipeline outperforms 2D-slice VLMs,\" which is supported by the current evidence. The paper fits the journal's scope and has potential, but the current evidence is not sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the idea of replacing cross-modal alignment with in-domain retrieval is the most interesting part of this paper, and the external validation is a real plus. But the main comparison is confounded, so the paper's central claim is not yet proven.\n\nHere's what I think is genuinely new. PIRTA trains a 3D ViT (MAE-pretrained on UK Biobank, then fine-tuned on territory classes) and retrieves similar DWI/ADC volumes by cosine similarity. The paired reports from the top-5 retrievals are then fed as context to a LoRA-tuned LLaMA-3 to generate the structured report. It's a clean way to avoid image-text alignment, and the code is released. The external tests on BRMH and ISLES are a strength. The authors also acknowledge in the Discussion that their method fails when retrieval similarity is low, and Table 5's second example shows exactly that. That's honest.\n\nNow the soft spots. The comparison against GPT-4o and LLaVA-Med changes three things at once: full 3D volumes versus a single 2D slice, top-5 retrieved reports versus no retrieval, and instruction-tuned LLM versus one-shot prompting. Without an ablation that holds the encoder and LLM fixed and toggles only the retrieved context, you can't attribute the reported gains to retrieval augmentation. I agree with the stress-test here. Second, the retrieval encoder is fine-tuned on the same four territory labels that define retrieval relevance and the evaluation metric. The external mAP numbers (71%, 70%) suggest the model is doing something real, but the metric is partly measuring the model's agreement with its own training labels. Third, no confidence intervals or significance tests. The internal retrieval difference (94% vs 81%) is probably robust, but the report-generation bar charts in Figure 3 could easily be noise.\n\nThe complexity argument in Eqs. 8-9 doesn't help; bounding a hypothesis space doesn't establish factual superiority. Treat it as motivation, not evidence. Also minor: earlier retrieval-based medical report generation isn't cited, and the ISLES labeling is under-specified.\n\nWho is this for? Anyone working on RAG for volumetric medical imaging, and anyone who wants a concrete example of why ablations matter in retrieval-augmented generation. It deserves a serious referee. I'd send it back with a request for the retrieval ablation, a 3D-capable VLM baseline, and evaluation on labels not used in SFT.","headline":"A genuinely useful retrieval-based framing for 3D MRI report generation, with a confounded baseline comparison and metric circularity that leave the central claim under-supported.","tokens_in":18435,"tokens_out":3841,"would_cite":false,"duration_ms":34633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieving similar 3D DWI/ADC volumes and grounding an LLM on their paired reports yields more factually accurate ischemic-stroke radiology reports than direct image-to-text generation with GPT-4o or LLaVA-Med.","keywords":["retrieval-augmented generation","3D brain MRI","DWI/ADC","acute ischemic stroke","radiology report generation","masked autoencoder","vision transformer","ischemic territory classification"],"falsifier":"Compare PIRTA's generated-report territory accuracy with a retrieval-only oracle that outputs the territory label of the top-1 retrieved image's paired report; if generated-report accuracy never exceeds the oracle, the LLM adds no independent factual information and report factuality is exactly retrieval accuracy. A direct test would use rare sites such as isolated amygdala infarction, where similarity scores drop, and check whether every generation error coincides with a retrieval error.","tokens_in":17197,"feed_emoji":"🧠","tokens_out":9305,"duration_ms":75607,"temperature":0.7,"pith_summary":"PIRTA aims to establish that factually reliable radiology reports for 3D brain MRI can be generated by retrieving clinically similar DWI/ADC volumes and reusing their paired clinician-authored reports, rather than by learning to map images directly to text. The paper's argument is that the hard part of image-to-text generation is explicit cross-modal alignment, so it avoids that step entirely: only a 3D vision encoder is trained, and the retrieved reports supply the clinical text that grounds an LLM. On internal and external datasets the framework reaches image-retrieval mAP@1 of 94.04% internally and 71.21% and 70.87% on two external cohorts, with ischemic-territory accuracy in generated reports consistently higher than GPT-4o and LLaVA-Med given 2D slices. If true, this means report factuality for 3D medical imaging can be improved with retrieval infrastructure and paired historical reports instead of larger multimodal alignment models.","feed_headline":"For 3D brain MRI, retrieval beats direct image-to-text generation","feed_subtitle":"PIRTA grounds an LLM on paired reports of similar DWI/ADC scans, lifting ischemic-territory accuracy.","key_machinery":"The central object is PIRTA, a paired image-domain retrieval and text-domain augmentation pipeline. Its engine is a 3D Vision Transformer image encoder: stage one pretrains it as a masked autoencoder on two-channel DWI+ADC volumes, and stage two fine-tunes it with a linear classifier over four classes (anterior circulation, deep gray matter, posterior circulation, normal). During inference the encoder is frozen, cosine similarity over its embeddings retrieves the top m=5 database images, and their paired structured findings, similarity scores, and patient registry data are fed as instruction context to a LoRA-tuned LLaMA-3-8B-Instruct that outputs clinical presentation, NIHSS, past history, findings, and impression. This machinery carries the argument because it replaces cross-modal alignment with image-only representation learning plus reuse of verified text.","core_discovery":"The central discovery is that the cross-modal mapping problem in radiology-report generation can be recast as an in-domain retrieval problem. Instead of learning a text encoder and aligning image and text representations in a shared space, PIRTA trains only a 3D ViT image encoder and retrieves the database image that minimizes the cosine distance between query and database embeddings; the paired report of each retrieved image then augments LLM generation. The paper shows that a 3D ViT pretrained with masked autoencoding on over 38,000 unlabeled DWI/ADC volumes and fine-tuned to classify four ischemic territories produces retrieval that transfers to external institutions and to the public ISLES benchmark, and that grounding LLaMA-3-8B-Instruct on the top-5 retrieved reports yields higher ischemic-territory accuracy than direct generation by GPT-4o or LLaVA-Med. The paper frames this as a complexity reduction from learning $C(\\mathcal{H}_{\\text{image}}) + C(\\mathcal{H}_{\\text{text}})$ to learning $C(\\mathcal{H}_{\\text{image}})$ alone.","pith_inferences":["Implicit upper bound: generated-report factuality cannot exceed retrieval quality, so the framework's benefit over direct generation should be measured against an oracle that simply copies the top-1 retrieved report's territory label.","The printed similarity scores could serve as an uncertainty flag: low top-5 scores (the failure case shows 0.53 to 0.46) mark cases that should route to human review rather than autonomous sign-off.","The same recipe—masked-autoencoder pretraining, territory fine-tuning, cosine retrieval, paired-report grounding—should transfer to other 3D modalities such as CT or to pathology slides whenever a paired verified-report database exists.","Because the encoder is fine-tuned on the same four territory labels used to judge retrieval relevance, an independent evaluation should test retrieval against richer clinical ground truth such as laterality, lesion size, or multi-territory involvement."],"forward_implications":["Direct image-to-text alignment is not required for factual 3D MRI report generation; a paired image-report database can substitute for a learned cross-modal mapping.","Pretraining scale is a first-order driver: adding 38,532 unlabeled UK Biobank volumes raised retrieval mAP@1 from 80.73% to 94.04% internally, from 56.72% to 71.21% on BRMH, and from 38.83% to 70.87% on ISLES.","Retrieval-grounded generation transfers across institutions and scanner vendors, so a system built this way can be externally validated without retraining the text side.","The output includes the top-5 retrieved reports and their similarity scores, giving clinicians auditable evidence for each generated finding."],"supporting_citations":[{"why":"Supplies the 38,532-subject unlabeled UK Biobank DWI/ADC set used for MAE pretraining.","marker":"Alfaro-Almagro et al. 2018"},{"why":"Supplies the masked autoencoder pretraining method used for the 3D ViT image encoder.","marker":"He et al. 2022"},{"why":"Supplies the Vision Transformer architecture that the 3D image encoder is built on.","marker":"Dosovitskiy et al. 2020"},{"why":"Supplies LoRA, used to instruction-tune LLaMA-3-8B-Instruct efficiently.","marker":"Hu et al. 2021"},{"why":"Supplies the retrieval-augmented generation framework that PIRTA adapts to 3D MRI reports.","marker":"Lewis et al. 2020"},{"why":"Provides LLaVA-Med, the medical vision-language baseline compared against PIRTA.","marker":"Li et al. 2024"},{"why":"Provides the ISLES 2022 public benchmark used for external validation.","marker":"Hernandez Petzsche et al. 2022"}],"fun_headline_variants":["Retrieval beats direct image-to-text for 3D brain MRI","PIRTA retrieves similar scans to ground LLM reports, lifting accuracy","Grounding brain MRI reports on similar retrieved scans outperforms direct generation","Retrieval-grounded generation improves factual 3D MRI reports","For 3D MRI, retrieved reports beat learned image-text alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything depends on the assumption that images the encoder finds similar truly share the same ischemic findings, so the retrieved paired reports are the right text to ground the generated report.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval beats direct image-to-text for 3D brain MRI","PIRTA retrieves similar scans to ground LLM reports, lifting accuracy","Grounding brain MRI reports on similar retrieved scans outperforms direct generation","Retrieval-grounded generation improves factual 3D MRI reports","For 3D MRI, retrieved reports beat learned image-text alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2784,"prompt_tokens":1032,"completion_tokens":1752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1658}},"tokens_in":648,"tokens_out":1752,"duration_ms":12500,"temperature":1.0,"reasoning_tokens":1658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:14:12.663428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare PIRTA's generated-report territory accuracy with a retrieval-only oracle that outputs the territory label of the top-1 retrieved image's paired report; if generated-report accuracy never exceeds the oracle, the LLM adds no independent factual information and report factuality is exactly retrieval accuracy. A direct test would use rare sites such as isolated amygdala infarction, where similarity scores drop, and check whether every generation error coincides with a retrieval error.","supporting_citations":[{"cited_title":", author Jenkinson, M","cited_arxiv_id":null,"evidence_quote":"Supplies the 38,532-subject unlabeled UK Biobank DWI/ADC set used for MAE pretraining."},{"cited_title":", author de la Rosa, E","cited_arxiv_id":null,"evidence_quote":"Provides the ISLES 2022 public benchmark used for external validation."}],"review_version":1}