{"id":"a96f8445-7d9f-4b49-a5cb-cad7497943a3","arxiv_id":"2505.18503","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A3Tune aligns the visual attention of medical LVLMs to prompt-relevant regions via SAM and BioMedCLIP weak labels plus a Mixture-of-Experts over LoRA, improving VQA and report generation accuracy.","lead":"This paper introduces A3Tune, a fine-tuning method that steers medical vision-language models to focus their visual attention on clinically relevant image regions using automatically generated weak labels, and reports consistent gains on medical VQA and report generation benchmarks. It matters because it offers a way to reduce hallucinations in medical image question answering without requiring manual region annotations at inference time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol is the weak link: hyperparameters appear selected on test folds with no validation split or error bars, so the headline 'outperforms SOTA' is not yet established; the weak-label assumption is secondary.","rationale":"The reader's verdict is CONDITIONAL and their rationale already includes 'hyperparameters are tuned per dataset without a clear validation split' and 'reported results lack error bars.' My stress-test elevates this to the single most load-bearing concern. The central claim is comparative ('outperforms state-of-the-art baselines'), so any threat to the validity of the comparison is more direct than the weak-label assumption: even if weak labels are noisy (a concern the authors acknowledge and partially bound via the GT experiment in Figure 4), the method could still improve performance through regularization or MoE; but if the evaluation leaks test information into hyperparameter selection, the headline numbers are not trustworthy at all. The manuscript never describes a validation split; Appendix A gives only train/test partitions, and Appendix C/Figures 5, 7, 8 tune λ, K, R, B on benchmark performance (SLAKE) without stating that this is a validation fold. With point estimates only and no repeated seeds, differences like +1.71 (SLAKE Open) and +3.60 (VQA-RAD Open) are not statistically assessable. The concrete test above would settle whether the advantage persists under a clean protocol. Because the flaw is methodological and fixable, and the method has a coherent design with ablations and available code, I would keep the reader's CONDITIONAL verdict rather than escalate to REJECT; hence UNCHANGED.","tokens_in":21708,"tokens_out":12460,"duration_ms":105045,"concrete_test":"Re-run the main VQA comparison (SLAKE and VQA-RAD) with an 80/10/10 train/validation/test split. Select λ, epochs, K, R, and B on the validation split only; report mean ± std over at least 3 seeds for A3Tune and LLaVA-Med+LoRA. If the validation-selected A3Tune does not beat LoRA on the held-out test split by more than the seed noise, the 'outperforms SOTA' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: A3Tune 'outperforms state-of-the-art baselines.' The most load-bearing condition is therefore that the reported benchmark numbers are honest estimates. The paper does not support this: Appendix A defines only train/test splits; Appendix C (Table 7) sets λ and epochs per dataset, and Figures 5, 7, and 8 select K, R, B, and λ based on test-set performance (e.g., 'performance peaks at K=4' on SLAKE), with no described held-out validation split. If hyperparameters are chosen on the test folds, then Tables 1, 3, 4, and 5 are optimistically biased and the headline advantage over LoRA may not generalize. No error bars, confidence intervals, or multiple-seed results are reported, so even a small number of point-score gains (e.g., +1.71 on SLAKE Open) cannot be distinguished from noise. The reader's weak-label concern is real but second-order: the paper's own GT-upper-bound experiment (Figure 4) shows weak labels are better than none and worse than GT, partially bounding the noise. The validation leakage, by contrast, affects every reported improvement and the central claim itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"A3Tune is a fine-tuning framework for medical vision-language models that generates prompt-aware weak segmentation labels with SAM and BioMedCLIP, selects a small set of \"visually-critical\" attention heads, and trains an attention-alignment loss jointly with the language-modeling objective. It augments LoRA with two Mixture-of-Experts modules, Q-MoE and K-MoE, to adapt attention computations per prompt and per visual token. The paper reports accuracy gains over LoRA and decoding-based baselines on five VQA benchmarks and two report-generation benchmarks, supported by ablations, attention-map visualizations, and a ground-truth upper-bound experiment.","tokens_in":21997,"tokens_out":6463,"duration_ms":55183,"significance":"If the reported gains are reliable, A3Tune offers a practical, inference-time-free recipe for reducing attention bias in Med-LVLMs without requiring manual ROI annotations at test time. The paper's strengths include evaluation on three backbones (LLaVA-Med, LLaVA-Med-1.5, and HuatuoGPT-Vision-7B), an ablation showing that the attention-alignment loss alone (without MoE) still improves over the LoRA baseline, a ground-truth upper-bound experiment in Figure 4, and a public code link. The main weakness is that the evaluation protocol does not rule out test-set overfitting through hyperparameter selection, so the headline empirical claim is not yet established as reported.","major_comments":[{"comment":"The paper does not describe a held-out validation split. Appendix A (Table 6) defines only train/test partitions, while Appendix C (Table 7) reports dataset-specific lambda and epoch values, and Figures 5(a), 5(b), 7(b), and 8 select K, R, B, and lambda by choosing the point at which \"performance peaks\" on SLAKE. If these hyperparameters are chosen on the test folds, then Tables 1, 3, 4, and 5 are optimistically biased, and the reported advantage over LLaVA-Med+LoRA (e.g., +1.71 on SLAKE Open in Table 1) may not generalize. Please specify a clear validation protocol or nested cross-validation, and report multiple-seed results with standard deviations or confidence intervals; without these, the point-score differences cannot be distinguished from noise.","section":"Appendix A, Appendix C, Figures 5, 7, 8"},{"comment":"The weak-label generation is specified inconsistently. Eq. (3) selects segments whose cosine similarity exceeds an adaptive threshold tau_K, but the text and Figure 5(a) describe K as the number of segments chosen by ranking similarity, and Section 5.4.1(2) states \"we select K segments that are most similar to the text prompt.\" Please define whether K is a count or a threshold, and state how tau_K is set; this is needed to reproduce Figure 5(a) and the main experiments.","section":"Eq. (3) and Section 5.4.1(2)"},{"comment":"The attention metrics in Table 2 are not an independent validation of attention quality, because the training objective Eq. (10) directly maximizes attention inside the weak-label masks that are also used to compute Coverage and Intensity. The VQA and report-generation gains on external benchmarks are not circular, and Figure 4's ground-truth upper-bound experiment partially addresses label quality, but the Coverage/Intensity comparison should be interpreted only as a check that training moved attention as intended. A stronger test would be to evaluate the attention metrics on held-out images or on cases where the weak labels are manually scored for correctness.","section":"Table 2 and Eq. (10)"}],"minor_comments":[{"comment":"The phrase \"a A3MOE module\" should be \"an A3MOE module\" in both the abstract and the introduction.","section":"Abstract and Section 1"},{"comment":"The text in Section 5.3 says A3Tune outperforms all baselines across both datasets and almost all metrics; this is accurate because on MIMIC-CXR CheXbert the PAI baseline scores 25.78 while A3Tune scores 24.93, but the exception should be stated explicitly rather than left to the reader to infer from the table.","section":"Table 3, MIMIC-CXR CheXbert row"},{"comment":"The routers in Q-MoE and K-MoE are trained without a load-balancing or auxiliary loss, and Figure 7(b) shows unstable returns for larger B; a brief discussion of expert utilization or collapse would strengthen the paper.","section":"Equations (4)-(7)"},{"comment":"The notation M^v_lh[c] in Eq. (8) is not defined; M^v_lh was introduced as a submatrix, so the indexing convention for individual tokens should be stated.","section":"Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The validation-leakage concern is the key obstacle to accepting the empirical claims. The paper's ablations and ground-truth upper-bound experiment suggest the method has substance, so I do not recommend rejection; however, the final numbers cannot be trusted as reported until a held-out validation protocol and variance estimates are provided. The inconsistency in defining K also needs to be resolved for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is a fine-tuning recipe: SAM produces candidate segments, BioMedCLIP filters them by prompt similarity, and then the method selectively tunes the most visually-critical attention heads using a Mixture-of-Experts over LoRA parameters. That combination is new in medical LVLMs, and the ablation in Figure 7 cleanly separates the attention-alignment loss from the MoE contribution, showing the loss alone helps over LoRA. The ground-truth upper-bound experiment in Figure 4 is a good sanity check: weak labels beat no labels, GT beats weak labels, which partially bounds the noise in the weak-label pipeline. The gains are consistent across five VQA and two report-generation benchmarks, with a second backbone (HuatuoGPT-Vision) confirming the method transfers. That is a solid empirical foundation in principle.\n\nThe soft spot is not the weak-label assumption; it's the evaluation protocol. The headline claim is 'outperforms state-of-the-art baselines,' but the reported numbers look selected on the test folds. Appendix C and Figures 5, 7, and 8 set K, R, B, and lambda based on test performance with no held-out validation split, and there are no error bars, confidence intervals, or multiple-seed results. A +1.71 gain on SLAKE Open cannot be distinguished from noise. The abstract overclaims as stated: on MIMIC-CXR CheXbert, PAI scores 25.78 vs A3Tune's 24.93, so 'outperforms' is not true on every metric. The weak-label issue is real but second-order, because the GT experiment bounds it and the evaluations are on external benchmarks.\n\nThis is a paper for people working on attention-tuning and hallucination mitigation in medical LVLMs. The method is practical, requires no inference-time annotation, and the central idea is worth engaging. It deserves a serious referee, but the authors need to add a proper validation split, report variance over multiple seeds, and soften the SOTA claim. I'd send it to peer review with the expectation of major revision.","headline":"Useful fine-tuning recipe with a clean ablation, but test-set hyperparameter selection undermines the headline claim.","tokens_in":22521,"tokens_out":2677,"would_cite":false,"duration_ms":21059,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning redirects medical AI attention to the region that actually matters","keywords":["medical vision-language models","attention alignment","hallucination mitigation","weak supervision","visual grounding","mixture-of-experts","parameter-efficient fine-tuning","medical visual question answering"],"falsifier":"Construct a medical VQA set where the prompt describes a region that BioMedCLIP fails to associate with the image segments but a clinician would identify, then run A3Tune and check whether attention coverage on the true region decreases or the answers become more hallucinated; a drop relative to plain LoRA fine-tuning on that set would indicate the weak labels are not reliable enough as supervision.","tokens_in":21511,"feed_emoji":"🩻","tokens_out":1429,"duration_ms":13667,"temperature":0.7,"pith_summary":"This paper proposes A3Tune, a fine-tuning framework that fixes a specific failure mode of medical vision-language models: their attention on an image is often spread over irrelevant background, so they hallucinate answers even when the right region is visible. The claim is that by generating prompt-aware weak region labels automatically from SAM and BioMedCLIP, then fine-tuning only the most visually-critical attention heads with a mixture-of-experts router, the model learns to focus on diagnostically relevant regions and answers more accurately. If this works, it would give a practical, annotation-free recipe for improving visual grounding in medical VQA and report generation, without needing inference-time interventions or ground-truth regions of interest.","feed_headline":"Training redirects medical AI's gaze to the right region","feed_subtitle":"A label-free fine-tuning recipe aligns attention maps with clinical regions and beats LoRA plus decoding fixes.","key_machinery":"The central object is the averaged visual attention map $\\bar{M}^v$ restricted to visual tokens, and the mechanism is the attention alignment loss $\\mathcal{L}_{\\text{align}} = \\sum_{s\\in S} (1 - \\frac{\\sum_{c\\in s} \\bar{M}^v_c}{\\sum_{c'} \\bar{M}^v_{c'}})^2$, which encourages higher attention inside each prompt-aware weak segment $s$. The argument is carried by three cooperating components: prompt-aware weak labels generated by SAM plus BioMedCLIP similarity filtering, selective tuning of the top-R visually-critical heads measured by visual attention ratio, and two Mixture-of-Experts modules (Q-MoE and K-MoE) that let the LoRA query and key updates adapt per prompt and per visual token, respectively.","core_discovery":"The paper's central claim is that attention misalignment in medical vision-language models can be corrected at fine-tuning time, rather than at inference time, using only automatically generated weak labels. A3Tune first produces candidate segments with SAM, filters them by cosine similarity between BioMedCLIP segment embeddings and the text prompt, and then uses a mask-based energy loss to pull the averaged visual attention map toward those segments. The paper also argues that modifying all attention heads is risky, so it selects the top-R heads by visual attention ratio, and that static parameter sharing is insufficient, so it wraps the LoRA query and key updates in a Q-MoE and a token-level sparse K-MoE. Experiments across five VQA and two report generation benchmarks, with LLaVA-Med, LLaVA-Med-1.5, and HuatuoGPT-Vision-7B, show consistent gains over LoRA fine-tuning and over decoding-based hallucination mitigation baselines, together with higher coverage and intensity of attention on ground-truth regions in SLAKE.","pith_inferences":["A natural extension is to apply the same alignment recipe to general-domain LVLMs, since the weak-label generation steps are not inherently medical.","One could test whether the improvement is driven mainly by the classifier-like separation of relevant segments or by the MoE routing; ablations already hint K-MoE matters most, but a direct comparison of alignment loss strength with and without MoE would clarify the mechanism.","The authors' own limitation regarding noisy weak labels implies an obvious next experiment: replace BioMedCLIP-based filtering with a stronger or prompt-conditioned segmenter, and measure whether the gap between weak and ground-truth labels shrinks.","The coverage and intensity metrics, being patch-level, could be refined to pixel-level evaluation if higher-resolution attention maps ever become available."],"forward_implications":["Medical VQA and report generation systems can gain accuracy and interpretability without any manual region annotations at inference time.","The alignment objective can serve as a drop-in regularization on top of existing parameter-efficient fine-tuning, extending to newer and stronger Med-LVLM backbones.","Attention-based evaluation metrics such as coverage and intensity can expose interpretability improvements that accuracy numbers alone miss.","The framework suggests that attention biases are a trainable property of Med-LVLMs, not only a decoding-time artifact.","If attention becomes more grounded, hallucination rates on region-sensitive questions about anatomy and abnormalities should drop correspondingly."],"supporting_citations":[{"why":"Supplies SAM, the zero-shot segmenter that generates candidate segments for the weak labels.","marker":"Kirillov et al., 2023"},{"why":"Supplies BioMedCLIP, whose vision and text encoders are used to compute prompt-aware cosine similarity and filter segments.","marker":"Zhang et al., 2023"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning base that A3Tune modifies with its MoE modules.","marker":"Hu et al., 2021"},{"why":"Supplies the mask-based cross-attention energy function that motivates the attention alignment loss.","marker":"Chen et al., 2024c"},{"why":"Provides ControlMLLM, the inference-time RoI-guided baseline that A3Tune aims to improve upon.","marker":"Wu et al., 2024"},{"why":"Supplies LLaVA-Med, the primary Med-LVLM backbone on which the main experiments are run.","marker":"Li et al., 2024"},{"why":"Supplies the SLAKE dataset with ground-truth RoIs used for attention evaluation and the weak-label quality study.","marker":"Liu et al., 2021"}],"fun_headline_variants":["Auto alignment tuning sharpens medical AI focus","Training-time attention fix for medical vision-LMs","SAM-guided weak labels align medical AI attention","A3Tune: Train medical AI to focus on clinical regions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the cosine similarity between a BioMedCLIP segment embedding and the text prompt reliably identifies the clinically correct region; if those zero-shot weak labels are noisy or miss the true region, the alignment loss can pull attention toward irrelevant areas and hurt performance.","fun_headline_variants_meta":{"raw":{"variants":["Auto alignment tuning sharpens medical AI focus","Training-time attention fix for medical vision-LMs","SAM-guided weak labels align medical AI attention","A3Tune: Train medical AI to focus on clinical regions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2765,"prompt_tokens":912,"completion_tokens":1853,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1792}},"tokens_in":528,"tokens_out":1853,"duration_ms":13285,"temperature":1.0,"reasoning_tokens":1792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:29:38.169252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a medical VQA set where the prompt describes a region that BioMedCLIP fails to associate with the image segments but a clinician would identify, then run A3Tune and check whether attention coverage on the true region decreases or the answers become more hallucinated; a drop relative to plain LoRA fine-tuning on that set would indicate the weak labels are not reliable enough as supervision.","supporting_citations":[],"review_version":1}