{"id":"35190142-3438-4fdd-8aea-018533f2aec9","arxiv_id":"2508.09225","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A LoRA-tuned MedGemma VLM generates mammography reports on the public DMID dataset, with ROUGE-L 0.5691, METEOR 0.6152, CIDEr 0.5818, and BI-RADS accuracy 0.5582.","lead":"AMRG fine-tunes the medical vision-language model MedGemma with LoRA adapters to generate narrative mammography reports from images, reporting ROUGE-L 0.5691, METEOR 0.6152, CIDEr 0.5818, and BI-RADS accuracy 0.5582 on the public DMID dataset. It is positioned as the first reproducible public benchmark for mammography report generation, though it does not release code and its test set has only 52 cases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline numbers are not statistically grounded: the 52-case test set yields no confidence intervals, LoRA hyperparameters appear to be selected on that test set, and several clinical accuracies imply a 51-case denominator.","rationale":"The reader's weakest assumption focuses on the unspecified BI-RADS label extraction, which is a genuine reproducibility gap. I see the more load-bearing problem as the statistical and split validity of the entire evaluation: the comparison is based on a single 52-case test set with no confidence intervals, and the hyperparameters for the backbone comparison appear to be selected from the test-set results in Table 1. Even a correct label extractor would not fix the absence of uncertainty quantification or the possibility of test-set selection bias. The denominator observation (many accuracies equal counts out of 51) makes the split inconsistency concrete and checkable. This does not amount to an accusation of fabrication; the method itself is standard LoRA fine-tuning and the qualitative examples are useful. The paper should remain CONDITIONAL: the authors need to release evaluation code/predictions, report bootstrap intervals or multiple seeds, specify validation-based hyperparameter selection, and clarify the evaluation split and label extraction rule.","tokens_in":16076,"tokens_out":12353,"duration_ms":130663,"concrete_test":"Compute paired bootstrap 95% confidence intervals for the MedGemma-vs-Qwen differences in ROUGE-L, METEOR, and BI-RADS accuracy over the 52 test reports, and verify that each BI-RADS and density accuracy resolves to a count out of 52 (not 51). If any interval includes zero, or if any accuracy value is an exact multiple of 1/51, the reported superiority claim is not established and the headline metrics must be recomputed on the correct split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AMRG 'outperforms' general-purpose VLMs is not supported by the reported evaluation. Section 3.1 fixes the test set at 52 cases, yet Section 4.2 says the LoRA hyperparameters (r = 32, alpha = 16) were 'determined' from the Table 1 ablation, and Table 1 appears to be reported on the test split; no validation-based hyperparameter selection is described. All five backbone comparisons therefore use hyperparameters selected for MedGemma on the evaluation set itself, which inflates MedGemma's apparent advantage. With only 52 cases, the headline differences are within plausible noise: ROUGE-L 0.5691 vs 0.5634 is a 0.006 gap, and the BI-RADS gap (0.5582 vs 0.4510) corresponds to roughly 5-6 cases; no confidence intervals, multiple seeds, or significance tests are provided. This is compounded by an internal split inconsistency: most clinical accuracies in Tables 1-2 are exact multiples of 1/51 (e.g., 0.4510 = 23/51, 0.3529 = 18/51) although Section 3.1 and Table A.1 specify a 52-case test set, suggesting the reported clinical metrics may have been computed on a different split than the one described. The missing BI-RADS extraction rule is a separate reproducibility gap, but even with a perfect extractor, the current numbers cannot support the claimed hierarchy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AMRG, a mammography report generation framework built by applying LoRA fine-tuning to the MedGemma-4B-it vision-language model and evaluating it on the public DMID dataset. The authors report a systematic LoRA hyperparameter ablation, a comparison against four other backbones (Qwen2.5-VL-7B, Phi-3.5-Vision, CLIP+GPT2, MedCLIP+GPT2) under a supposed unified tuning protocol, and both standard NLP metrics (BLEU, ROUGE, METEOR, CIDEr, word-level F1) and clinical metrics (BI-RADS accuracy, breast density accuracy). The central claim is that AMRG is the first end-to-end framework for free-text mammography report generation and outperforms general-purpose VLMs, with headline scores of ROUGE-L 0.5691, METEOR 0.6152, CIDEr 0.5818, and BI-RADS accuracy 0.5582. The manuscript also includes image preprocessing steps, a description of the LoRA adaptation, qualitative examples, and a limitations section.","tokens_in":16331,"tokens_out":7734,"duration_ms":75192,"significance":"If the experimental claims hold, a lightly tuned 4B medical-domain VLM would provide a useful and reusable baseline for a publicly available mammography report dataset, and the systematic comparison of medical versus general-purpose backbones under one PEFT protocol would be informative. The paper has genuine strengths: it uses a public dataset (DMID), gives a detailed preprocessing pipeline, and demonstrates that fine-tuning is necessary because the frozen MedGemma baseline scores near zero. However, the current evaluation is not statistically grounded: there are no confidence intervals or significance tests, the clinical label extraction rule is unspecified, and several reported clinical accuracies appear to use a different denominator from the stated test split. These issues are load-bearing for the headline 'outperforms' claim, so the contribution is best viewed as a promising draft rather than an established benchmark.","major_comments":[{"comment":"The test split is fixed at 52 cases in Section 3.1 and Table A.1, but many clinical accuracies in Tables 1 and 2 are exact multiples of 1/51, including 0.4510 = 23/51, 0.3529 = 18/51, 0.1176 = 6/51, and 0.4902 = 25/51. This suggests the clinical metrics were computed on a 51-case denominator rather than the described 52-case test set. The authors must report the exact denominator for every clinical metric and reconcile this discrepancy, because a one-case difference is about 2% of the test set and is material to the BI-RADS accuracy gap between systems.","section":"Section 3.1, Tables 1-2, Table A.1"},{"comment":"The hyperparameters used for all backbone comparisons were selected from Table 1, but Section 4.1 never states whether Table 1 is computed on the validation or test split. If Table 1 is test-set performance, then choosing (r = 32, alpha = 16) on the basis of that table and then reporting Table 2 on the same test set means every model is evaluated with hyperparameters tuned to the evaluation set; this biases the comparison in favor of MedGemma and invalidates the 'outperforms' claim. The authors should either use a held-out validation split for hyperparameter selection and state this explicitly, or report a corrected protocol that avoids test-set contamination.","section":"Sections 4.1 and 4.2"},{"comment":"The clinical metrics are not reproducible as specified. The text only states that accuracy is 'the proportion of exact matches between predicted and ground-truth labels' and does not define how a BI-RADS or breast density label is extracted from a generated free-text report. If extraction is keyword-based, the reported clinical accuracy may measure phrasing similarity rather than clinical classification. The authors should specify the extraction procedure (rule-based parser, prompted LLM, or classifier), validate that extractor, and report its agreement with the ground-truth labels.","section":"Section 4.1"},{"comment":"All conclusions rest on single point estimates with no confidence intervals, significance tests, or multiple seeds. With 52 test cases, the headline ROUGE-L gap (0.5691 vs 0.5634) and the BI-RADS gap (0.5582 vs 0.4510, roughly 5-6 cases) are within plausible sampling noise. The authors should report bootstrap or exact binomial confidence intervals for each metric, and ideally multiple seeds or a paired test for the main system comparisons.","section":"Tables 1-2"},{"comment":"The abstract and qualitative section emphasize multi-view reasoning, but the described method does not explain how multiple mammographic views are combined. Preprocessing in Section 3.1 is defined per image, and Eqs. (3)-(4) condition generation on a single image I. The authors should specify how CC/MLO views are fed to the model (for example, concatenation, separate vision tokens, or per-view decoding) and confirm whether the quantitative evaluation actually uses multiple views per case.","section":"Section 3.1 and Eq. (3)"}],"minor_comments":[{"comment":"The prose says that (r = 32, alpha = 16) achieves ROUGE-L 0.52, METEOR 0.5194, and CIDEr 0.5336, but Table 1 reports 0.5691, 0.6152, and 0.5818 for that configuration; please correct the mismatch.","section":"Section 4.1, text after Table 1"},{"comment":"The heading 'Casual LM Loss' should be 'Causal LM Loss.'","section":"Section 3.3 heading"},{"comment":"The word 'resurts' appears in the LoRA analysis paragraph and should be corrected to 'results.'","section":"Section 5.2"},{"comment":"The claims 'first end-to-end framework' and 'first reproducible benchmark' are stronger than the evidence and related-work discussion support; Yalunin et al. already address free-text mammography report generation, and no code or trained weights are released. Please soften or justify these priority claims.","section":"Abstract and Section 1"},{"comment":"The limitations section appropriately acknowledges that surface-level NLP metrics do not fully capture clinical correctness, but the abstract and conclusion still describe the clinical metrics as evidence of strong performance; please align the claims with this acknowledged limitation.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper addresses a real gap and the basic recipe is sensible, but the evaluation section needs substantial work before the claims can be accepted. The denominator inconsistency and the missing label-extraction rule are the most serious technical issues, and the priority claims about being 'first' should be checked carefully during revision. I recommend major revision rather than rejection, because the core framework is straightforward and the reported evaluation can be fixed within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main thing to know: this is a benchmark paper, not a method paper. The genuinely useful contribution is pairing DMID's public mammograms with a systematic comparison of five VLMs under one LoRA protocol. That gives the community a starting point and a baseline. The LoRA fine-tuning itself is off-the-shelf.\n\nWhat the paper does well: it identifies a real gap—mammography report generation had only private-dataset work—and it is upfront about DMID's imbalance and small size. The preprocessing is described in enough detail to reproduce (Otsu, flipping, CLAHE). The qualitative examples are actually informative.\n\nSoft spots, in order of severity:\n\n1. The headline numbers are not statistically grounded. The test set is 52 cases; there are no CIs or significance tests, and several differences (ROUGE-L 0.5691 vs 0.5634; BI-RADS 0.5582 vs 0.4510, roughly 5-6 cases) could be noise.\n\n2. The hyperparameters appear to be chosen on the test set. Section 4.2 says the optimal (r=32, alpha=16) were 'determined' from Table 1, and Table 1 is reported on the DMID dataset without specifying a validation split. If so, the comparison across backbones is biased toward MedGemma. This is the deepest flaw.\n\n3. The split inconsistency the stress-test flags is real and needs explanation: several clinical accuracies (0.4510, 0.3529, 0.1176) are exact multiples of 1/51, while the stated test size is 52. Either the test set is 51 cases or the text/table is wrong; the paper can't stand without resolving this.\n\n4. The BI-RADS/density extraction procedure is not described. 'Exact match' between predicted and ground-truth labels requires knowing how free text becomes a label. This is a reproducibility blocker.\n\n5. The text in Section 4.1 does not match Table 1 (e.g., says METEOR 0.5194 and CIDEr 0.5336 while the table lists 0.6152 and 0.5818). A typo at best; makes you wonder about quality control.\n\nNone of these suggest fabrication. The citation pattern is reasonable and the work is a serious empirical effort. But the current write-up does not support the claimed hierarchy. I would send it to review because the benchmark is worth having; the authors need to fix the split, report error bars or multiple seeds, release the evaluation script, and clarify the label extraction.","headline":"A useful public benchmark for mammography report generation, undermined by loose statistics and test-set hyperparameter selection that currently make the headline comparisons unproven.","tokens_in":16910,"tokens_out":2581,"would_cite":true,"duration_ms":24442,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A LoRA-tuned MedGemma generates narrative mammography reports and beats larger general-purpose VLMs on a public benchmark.","keywords":["automatic mammography report generation","vision-language models","low-rank adaptation","MedGemma","BI-RADS","DMID dataset","parameter-efficient fine-tuning","radiology report generation"],"falsifier":"Take the same 52 test reports, replace the unspecified label-extraction step with two explicit procedures, an exact keyword match for BI-RADS phrases and a separately trained BI-RADS classifier, and recompute accuracy; if the two results diverge from 0.5582 by more than a few cases, the clinical metric is not a stable property of the model.","tokens_in":15823,"feed_emoji":"🩻","tokens_out":11477,"duration_ms":97227,"temperature":0.7,"pith_summary":"This paper claims that narrative mammography reports can be generated end-to-end from paired mammograms and diagnostic reports by a domain-specialized vision-language model tuned with low-rank adapters, and that this beats both larger general-purpose VLMs and modular encoder-decoder baselines under one fixed protocol. The authors build AMRG on MedGemma-4B-it, insert LoRA adapters into all linear layers, and train on the public DMID dataset with a causal language-modeling loss. On the 52-case test split they report ROUGE-L 0.5691, METEOR 0.6152, CIDEr 0.5818, and BI-RADS accuracy 0.5582, with the configuration $r=32$, $\\alpha=16$ working best. The paper also positions AMRG as the first reproducible benchmark for mammography report generation on a public paired image-report dataset. If the claim holds, the contribution is a small, reproducible recipe for adapting a medical VLM to a low-resource imaging domain without full fine-tuning.","feed_headline":"LoRA-tuned MedGemma beats larger VLMs on mammography reports","feed_subtitle":"A 4B medical VLM with lightweight adapters writes narrative reports that a 7B generalist cannot match, on a public benchmark.","key_machinery":"The load-bearing mechanism is the AMRG pipeline: MedGemma-4B-it with Low-Rank Adaptation (LoRA) inserted into every linear projection layer, including attention query/key/value/output projections, feed-forward layers, and gated MLP projections, while all pretrained weights stay frozen. The LoRA update is $\\Delta W = \\alpha AB$ with rank $r$ and scaling factor $\\alpha$; only the LoRA matrices, the embedding layer, and the language-model head are trained. A causal language-modeling loss conditions report generation on the image and instruction prompt, and a preprocessing chain of Otsu thresholding, left-right flip for laterality, and CLAHE contrast enhancement standardizes multiview mammograms into 512$\\times$512 inputs before they reach the model.","core_discovery":"The central claim is that domain specialization beats scale for this task. MedGemma-4B-it, a 4B medical instruction-tuned VLM, fine-tuned with LoRA, outperforms Qwen2.5-VL-7B on ROUGE-L, METEOR, CIDEr, word-level F1, and BI-RADS accuracy, while only narrowly trailing on BLEU-1 and ROUGE-2; the modular CLIP+GPT2 and MedCLIP+GPT2 pipelines lag substantially, and Phi-3.5-Vision performs worst. The paper interprets this hierarchy as evidence that medical pretraining and instruction tuning, combined with moderate LoRA capacity, supply clinical priors that generic visual-linguistic alignment lacks. It also claims this is the first end-to-end framework for free-text mammography report generation on a public dataset, with low scores from the frozen backbone showing that the LoRA adaptation is doing the work.","pith_inferences":["Because the whole comparison runs on one public dataset with one tuning protocol, a natural next test is to apply the same LoRA recipe to another small paired imaging domain, such as ultrasound or MRI, to see whether the domain-specialized backbone keeps its advantage; the paper does not run that experiment.","The unstated BI-RADS extraction rule means the clinical accuracy numbers function as report-similarity scores until the rule is published; a separately trained radiology text labeler would make the metric reproducible.","The paper's preprocessing downsamples mammograms to 512$\\times$512 even though the authors list high-resolution visual cues as a core difficulty, so feeding multi-view images at native resolution is a testable extension that might change both generation quality and the BI-RADS numbers.","The large gap between the frozen and adapted baselines suggests the LoRA adapters encode most of the mammography-specific vocabulary, implying that the same backbone may need retraining rather than prompting when the target reporting style changes."],"forward_implications":["A 4B domain-specialized VLM fine-tuned with LoRA on just 407 training cases can generate narrative reports that beat 7B and 4B general-purpose VLMs and modular CLIP+GPT2 baselines under the same tuning protocol.","The best configuration, rank 32 with scaling factor 16 and temperature 0.1, reaches ROUGE-L 0.5691, METEOR 0.6152, CIDEr 0.5818, and BI-RADS accuracy 0.5582 on the 52-case test set.","Raising the LoRA rank to 64 degrades both language and clinical metrics, indicating that overfitting, not capacity, limits performance on the small DMID dataset.","The frozen MedGemma baseline scores far below the adapted model on every metric, so most task ability comes from the LoRA adaptation rather than from the pretrained model alone.","Because DMID is public and the split is fixed, these numbers form a baseline that later work on mammography report generation can compare against directly."],"supporting_citations":[{"why":"Supplies the MedGemma-4B-it backbone, the domain-specialized instruction-tuned VLM that AMRG adapts.","marker":"[7]"},{"why":"Defines Low-Rank Adaptation, the parameter-efficient method the whole tuning strategy rests on.","marker":"[8]"},{"why":"Provides the paired high-resolution mammograms and narrative reports that form the training, validation, and test data.","marker":"[13]"},{"why":"Supplies the ROUGE-L metric reported as the paper's headline language-generation score.","marker":"[10]"},{"why":"Supplies the METEOR metric used to measure fluency and lexical diversity.","marker":"[11]"},{"why":"Supplies the CIDEr metric used to measure consensus-based content relevance.","marker":"[12]"},{"why":"Supplies the MedCLIP encoder used in the modular MedCLIP+GPT2 baseline.","marker":"[15]"},{"why":"Supplies the Qwen2.5-VL-7B general-purpose backbone that is the strongest generalist comparison.","marker":"[37]"},{"why":"Supplies the Phi-3.5-Vision compact generalist baseline in the backbone comparison.","marker":"[38]"},{"why":"Supplies the CLIP image encoder used in the modular CLIP+GPT2 baseline.","marker":"[39]"}],"fun_headline_variants":["Domain-specialized 4B VLM outdoes 7B generalist on mammography","LoRA-tuned MedGemma beats 7B Qwen on mammography reports","First end-to-end VLM for mammography reports, beats larger models","LoRA fine-tuning makes 4B MedGemma outperform 7B on mammography"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported clinical accuracy numbers stand or fall on the unstated rule that converts each generated free-text report into a BI-RADS label, so the 0.5582 figure could be measuring how closely the wording matches rather than true diagnostic classification.","fun_headline_variants_meta":{"raw":{"variants":["Domain-specialized 4B VLM outdoes 7B generalist on mammography","LoRA-tuned MedGemma beats 7B Qwen on mammography reports","First end-to-end VLM for mammography reports, beats larger models","LoRA fine-tuning makes 4B MedGemma outperform 7B on mammography"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":3061,"prompt_tokens":1019,"completion_tokens":2042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":1951}},"tokens_in":635,"tokens_out":2042,"duration_ms":14234,"temperature":1.0,"reasoning_tokens":1951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:33:45.070378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 52 test reports, replace the unspecified label-extraction step with two explicit procedures, an exact keyword match for BI-RADS phrases and a separately trained BI-RADS classifier, and recompute accuracy; if the two results diverge from 0.5582 by more than a few cases, the clinical metric is not a stable property of the model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the paired high-resolution mammograms and narrative reports that form the training, validation, and test data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MedCLIP encoder used in the modular MedCLIP+GPT2 baseline."},{"cited_title":"Radford, J","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP image encoder used in the modular CLIP+GPT2 baseline."}],"review_version":2}