{"id":"f70afa77-0671-4e92-9a8f-ac7a01d2a162","arxiv_id":"2506.10633","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Anatomy-token prompt tuning with Gaussian location priors improves phrase grounding in a chest X-ray latent diffusion model, beating prior models on MS-CXR-loc and VinDr-CXR.","lead":"This paper fine-tunes a chest X-ray diffusion model so it can better point to the body part named in a radiology report, improving phrase grounding. It uses weak location hints from reports and a small set of anatomical boxes, and reports better localization than prior models on two benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fine-tuning set and the MS-CXR-loc evaluation set are both drawn from MIMIC-CXR, yet the paper never reports their overlap; if any evaluation images were used for fine-tuning, the SOTA claim is not established.","rationale":"The paper's headline contribution is empirical: fine-tuning anatomy token embeddings yields SOTA phrase grounding on MS-CXR and strong OOD results on VinDr-CXR. For the MS-CXR claim to hold, the evaluation must not be contaminated by the fine-tuning set. The method description in §3.2 and Appendix A shows both the fine-tuning set and MS-CXR are built from MIMIC-CXR, yet the paper only quantifies overlap with Chest Imagenome (28 images), not with MS-CXR. This omission is conspicuous because the authors clearly considered overlap when auditing the Gaussian LUT. The concern is not that the authors acted improperly; it is that a standard benchmark comparison requires this disclosure. Because only token embeddings are updated, contamination would not manifest as per-image memorization, but it would still let the shared location embedding be optimized on the test images, inflating the reported margins (e.g., +7.5 mIoU points over frozen LDM in Table 1). The suggested check is decisive and cheap. The reader's weakest assumption (fixed Gaussians) is related but distinct; I partially agree with the reader, and the same conditional verdict is appropriate. The OOD VinDr-CXR result also deserves scrutiny because synthetic prompts are generated from the same LUT (Appendix B), but the overlap check is the more direct threat to the main claim. If the overlap is zero, the central claim stands; if non-zero, the results need re-evaluation.","tokens_in":15598,"tokens_out":12206,"duration_ms":154731,"concrete_test":"Using the MIMIC-CXR study/series identifiers, compute the set intersection between the 6,480 fine-tuning samples and the MS-CXR-loc evaluation set. Report the number of overlapping images (and, ideally, overlapping patients/studies). If the intersection is non-empty, retrain the model with those images excluded and recompute Table 1. If average CNR/mIoU drops by more than the bootstrap confidence interval width (roughly ±0.05 CNR, ±1-2% mIoU in Table 1), the SOTA claim is not supported; if the intersection is empty, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 and Appendix A state that the fine-tuning set comprises 6,480 samples selected from the original ~70k MIMIC-CXR training set, while the primary evaluation is MS-CXR-loc, a filtered subset of MS-CXR, which is itself built from MIMIC-CXR. The paper discloses the overlap between the fine-tuning set and Chest Imagenome (28 images) but never discloses the overlap between the fine-tuning set and the MS-CXR-loc test images. This matters because the central claim (Table 1: LDM (ours) average CNR 1.37, mIoU 29.5 vs frozen LDM 0.93/22.0) is a comparison on images that may have been used to update the anatomy token embeddings. Even though the U-Net and text encoder are frozen, the gradient from a test image directly updates the shared embedding for the location token; if that image is in the fine-tuning set, the embedding is partially optimized for the exact evaluation images. The paper's silence on this overlap, despite explicitly auditing overlap with Chest Imagenome, is a missing support for the SOTA claim. The concern is not that the authors are dishonest; it is a standard train/test leakage check that the evaluation protocol does not rule out.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a parameter-efficient fine-tuning method to improve the alignment between text prompts and anatomical regions in a pre-trained chest X-ray latent diffusion model (LDM). The authors first show that the model's cross-attention activations are diffuse when prompted with either free-text report sentences or structured '{location} {pathology}' prompts (Figure 1). They then curate a fine-tuning set of 6,480 MIMIC-CXR samples by extracting anatomy entities with RadGraph-XL, standardizing prompts, and mapping each of 27 anatomical locations to a fixed 2D Gaussian derived from Chest Imagenome bounding-box statistics. Only the anatomy/pathology token embeddings (a codebook of 46 x 1024 entries) are updated, while the VAE, CLIP text encoder, and U-Net remain frozen; the objective combines a token diversity loss with a localization loss whose target is a masked, detached copy of the model's own activations plus the Gaussian prior (Eqs. 2-4). The fine-tuned model is evaluated on a filtered subset of MS-CXR (MS-CXR-loc, 1,043 samples) and on VinDr-CXR with synthetic prompts, reporting average CNR of 1.37 vs 0.93 and mIoU of 29.5 vs 22.0 for the fine-tuned vs frozen LDM on MS-CXR-loc, with bootstrap confidence intervals. The paper claims state-of-the-art phrase grounding on both benchmarks and provides ablations, an oracle upper bound, a prompt-format analysis, and a limitations section.","tokens_in":15925,"tokens_out":17595,"duration_ms":184116,"significance":"If the claims hold, the paper offers a practical way to repurpose a generative model for weakly supervised phrase grounding with very few trainable parameters, and the evaluation design is more thorough than is typical: it includes an out-of-distribution dataset, a prompt-format analysis (Appendix D), an oracle upper bound, a MAIRA-2 comparison, and an honest limitations section. The empirical results are also consistent: across most classes the fine-tuned LDM's bootstrap intervals do not overlap those of the frozen LDM (e.g., Pneumothorax CNR from -0.07 to 1.38 on MS-CXR-loc, Table 1), and the gains transfer to VinDr-CXR (Table 3). The data-curation pipeline combining RadGraph-XL entity extraction with Chest Imagenome-derived Gaussian priors is a useful methodological contribution, and the commitment to release code is welcome.","major_comments":[{"comment":"The central state-of-the-art claim requires an explicit audit of the overlap between the fine-tuning set and the evaluation set. The 6,480-sample fine-tuning set is drawn from the ~70k samples used to train the LDM (Section 3.2, Appendix A), and MS-CXR-loc is also a subset of MIMIC-CXR (Section 4.1), yet the paper only reports the overlap with Chest Imagenome (28 images) and never the overlap with MS-CXR-loc. Because the objective in Eq. 3 updates the shared anatomy token embeddings from every fine-tuning sample, any MS-CXR-loc image present in the fine-tuning set would partially optimize the embeddings for the exact evaluation images and could inflate the results in Table 1. Please report the image-level and subject-level overlap between the fine-tuning set and MS-CXR-loc (and VinDr-CXR for completeness), and, if the overlap is non-zero, re-run the evaluation on the non-overlapping subset. This is a standard leakage check and it is load-bearing for the abstract's SOTA claim.","section":"Section 3.2, Section 4.1, Table 1"},{"comment":"The supervision target in Eq. (3) is dominated by the model's own detached activations, so the external anatomical prior plays a smaller role than the paper's framing suggests. With the stated threshold epsilon = 1e-5, the binary mask 1_{T>epsilon} is nearly all-ones for broad locations such as left/right lung, giving trg approximately equal to sg(A_sp) + 0.1*T; the localization loss then mostly sharpens the model's existing attention, with the Gaussian acting as a weak additive bias. This is consistent with Table 2, where the alpha = 0 ablation (no Gaussian term at all) already accounts for most of the improvement over the frozen LDM (CNR 1.30 vs 1.37, mIoU 28.5 vs 29.5). The authors should (i) report the average fraction of active mask entries per location, and (ii) add an ablation with the target set to the pure Gaussian (trg = T) to isolate the contribution of the anatomical prior. As written, the claim that the method steers attention 'towards the anatomical area' overstates the role of the external, anatomy-grounded signal.","section":"Section 3.3, Eq. (3), Table 2"},{"comment":"The abstract's claim of a 'new state-of-the-art on a standard benchmark dataset (MS-CXR)' is stronger than what the experiments support. The evaluation is performed on MS-CXR-loc, a filtered subset of MS-CXR that discards pairs for which RadGraph-XL predicts no anatomy entity, under standardized '{location} {pathology}' prompts rather than the original MS-CXR sentences. Moreover, on the average CNR metric the proposed model ties with BioViL-T on MS-CXR-loc (both 1.37), so the statement in Section 4.6 that the method outperforms contrastive baselines 'by a significant margin' should be qualified. Please reword the claims to name MS-CXR-loc and the prompt transformation, and ideally report results on the unfiltered MS-CXR set with original prompts so that comparison with the published BioViL/BioViL-T numbers is direct; the prompt-format analysis in Appendix D mitigates but does not eliminate this comparability issue.","section":"Abstract, Section 4.1, Table 1"}],"minor_comments":[{"comment":"The Pneumothorax confidence interval on VinDr-CXR reads '[5,61, 23.3]', which appears to be a typo for '[5.61, 23.3]', and the Lung Opacity row reports a point estimate of 32.2 equal to the lower CI bound '[32.2, 41.4]', which also appears erroneous.","section":"Table 4"},{"comment":"The per-pathology counts in Figure 5 sum to 7,298 (including the Edema and Cardiomegaly counts given in the caption), but the text states that the fine-tuning set contains 6,480 samples with each sample assigned to exactly one pathology; these numbers should be reconciled.","section":"Figure 5, Section 3.2"},{"comment":"The codebook size is given as 46 entries for 27 locations, 8 pathologies, and 4 special tokens, which accounts for only 39 entries; please clarify what the remaining entries are (e.g., sub-token splits such as 'bibasilar' or 'costophrenic').","section":"Section 3.3"},{"comment":"Please state the normalization of the Gaussian T (e.g., peak value 1 versus unit integral) and specify that the mask 1_{T>epsilon} is applied elementwise; with epsilon = 1e-5 the mask is close to all-ones for large anatomical structures, which is relevant for interpreting the target.","section":"Eq. (3), Section 3.3"},{"comment":"The synthetic VinDr-CXR prompts are generated by matching ground-truth boxes to the same lookup table that provides the supervision signals, so the OOD evaluation only exercises the 27 LUT locations; a sentence acknowledging this would clarify the scope of the OOD claim.","section":"Appendix B"},{"comment":"The results correspond to a single run; for classes with small gaps (e.g., Cardiomegaly CNR 1.13 for the fine-tuned LDM versus 1.16 for the frozen LDM on MS-CXR-loc), it would be useful to report run-to-run variability or at least acknowledge explicitly that the fine-tuned model does not improve every class.","section":"Section 4.5, Table 1"},{"comment":"The absence of an Edema row in Table 3 is only explained in the Appendix C caption; please note this in the main text or in the Table 3 caption as well.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"I support a major revision. The empirical work is competently executed and honestly reported, but the two load-bearing issues (the MIMIC-CXR train/test overlap audit and the self-referential nature of Eq. 3) need to be addressed before the SOTA claim can be taken at face value. I do not suspect intentional leakage; the paper's own asymmetry, in which the Chest Imagenome overlap is audited but the MS-CXR-loc overlap is not, makes the omission visible and easy to fix. The paper is within scope for a vision-language or medical imaging venue, and with the requested clarifications it should be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you'll want to know about: Vilouras et al. fine-tune only the anatomy token embeddings of a chest X-ray LDM using a weak supervision signal built from RadGraph-XL entity extraction and Gaussian targets computed from Chest Imagenome boxes. That combination is new, and it mostly works. On MS-CXR-loc, average mIoU improves from 22.0 (frozen LDM) to 29.5, and CNR from 0.93 to 1.37; on VinDr-CXR the gains are even clearer. The method updates only a 46x1024 embedding codebook, and the paper includes ablations for alpha and the diversity loss, plus an oracle and MAIRA-2 in the appendix. That's a serious, within-subfield contribution.\n\nThe soft spots are real but fixable. The stress-test note is on target: the fine-tuning set and MS-CXR-loc are both drawn from MIMIC-CXR, and the paper never reports their overlap. Since a test image's gradient would directly update the shared embedding for the location token, that's a standard leakage check the protocol doesn't rule out. The paper audits overlap with Chest Imagenome (28 images) but is silent on MS-CXR; that silence is the main reason I don't take the SOTA claim at face value yet. One-line fix: report the overlap or use a disjoint split.\n\nTwo smaller issues. The abstract says SOTA on MS-CXR, but the experiments use the filtered MS-CXR-loc, and even there the average CNR ties BioViL-T (1.37 to 1.37); the SOTA claim holds for mIoU only. The abstract should be precise. Also, Eq. 3's target mixes the Gaussian with the model's own detached activations. The stop-gradient makes it less circular than it looks, but the contribution of sg(A_sp) is not ablated, so the reader can't tell how much of the gain comes from self-alignment versus the Gaussian prior. A quick ablation would settle it.\n\nOverall, this deserves a serious referee. With the overlap disclosure and a sharper abstract, I'd expect it to pass. The coarse Gaussian mapping for terms like 'lingular' is a limitation, but the paper flags it and the VinDr-CXR results suggest the method is robust.","headline":"A useful prompt-tuning recipe for medical phrase grounding, with an undisclosed train/test overlap that should be fixed before the SOTA claim is trusted.","tokens_in":16444,"tokens_out":4226,"would_cite":true,"duration_ms":45830,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuning only the anatomy token embeddings of a pre-trained chest X-ray latent diffusion model, guided by a weak Gaussian supervision signal derived from radiology reports and a small anatomy-box dataset, yields…","keywords":["phrase grounding","latent diffusion models","chest X-ray","weakly supervised learning","prompt tuning","cross-attention","out-of-distribution generalization"],"falsifier":"Compare per-location phrase-grounding gains (fine-tuned minus frozen LDM) between anatomical terms that have direct Chest Imagenome box statistics and the mapped terms ('lingular', 'base', 'pleural'); if the mapped terms show no improvement, or the fine-tuned attention simply follows the fixed Gaussian even when the ground-truth box is elsewhere, then the method's apparent success is driven by the prior rather than by prompt tuning.","tokens_in":15439,"feed_emoji":"🩻","tokens_out":6578,"duration_ms":66788,"temperature":0.7,"pith_summary":"The paper sets out to show that a pre-trained chest X-ray latent diffusion model can be repurposed for phrase grounding without retraining its image or text encoders. The authors argue that the model's cross-attention maps leak diffusely over the image, and that neither free-text radiology prompts nor a simplified '{location} {pathology}' format fix this. Their remedy is to fine-tune only the anatomy token embeddings in the prompt, using a weak supervision signal: a per-location 2D Gaussian built from a small set of anatomy bounding boxes, combined with a diversity loss that separates token-level attention. On the MS-CXR phrase-grounding benchmark the fine-tuned model sets a new state of the art, and it also outperforms contrastive baselines on the out-of-distribution VinDr-CXR test set. The claim matters because it suggests frozen diffusion models can be converted into localization tools with minimal data and parameters, using only unstructured radiology reports as supervision.","feed_headline":"Fine-tuning anatomy tokens fixes chest X-ray phrase grounding","feed_subtitle":"Tuning only anatomy token embeddings makes cross-attention land on the right lung zone, beating MS-CXR and VinDr-CXR baselines.","key_machinery":"The load-bearing object is the stack of cross-attention activation maps $A \\in \\mathbb{R}^{T\\times L\\times D\\times S}$ extracted from 30 diffusion timesteps and 4 U-Net layers (one bottleneck, three decoder) of the frozen LDM. The maps are averaged over the token dimension, reshaped into spatial feature maps, and compared against a fixed per-location 2D Gaussian lookup table (LUT) whose means and variances come from Chest Imagenome bounding boxes. The supervision target is a linear mix of the stopped-gradient attention map and the Gaussian, $\\mathrm{trg} = \\mathbf{1}_{T>\\epsilon}\\cdot\\mathrm{sg}(A_{\\mathrm{sp}}) + \\alpha T$, and the objective is $\\mathcal{L} = \\mathcal{L}_{\\mathrm{div}} + \\mathcal{L}_{\\mathrm{loc}}$, where $\\mathcal{L}_{\\mathrm{loc}}$ is a cosine localization loss against this target and $\\mathcal{L}_{\\mathrm{div}}$ penalizes pairwise cosine overlap between $\\ell_2$-normalized token attention maps. The only trainable parameters are 46 token embeddings (locations, pathologies, and special tokens), each 1024-dimensional; the VAE, CLIP text encoder, and denoising U-Net stay frozen.","core_discovery":"The central discovery is that a frozen text-to-image latent diffusion model, whose cross-attention activations normally bleed across the whole chest X-ray, can be steered to the correct anatomy by updating only a small codebook of location-token embeddings. Fine-tuning uses a curated set of 6,480 image-report pairs: RadGraph-XL extracts anatomical entities, a lookup table converts each entity into a fixed 2D Gaussian target derived from Chest Imagenome bounding-box statistics, and the training objective pushes the model's averaged cross-attention maps toward a dynamically mixed target while penalizing overlap between token-level attention maps. After tuning, the cross-attention heatmaps themselves serve as phrase-grounding predictions, achieving higher CNR and mIoU than BioViL, BioViL-T and the frozen LDM on MS-CXR-loc, and similar gains on VinDr-CXR. The authors interpret this as evidence that token-level prompt tuning, rather than architectural change or full fine-tuning, is sufficient to align clinical language with image regions.","pith_inferences":["The recipe is not tied to chest X-rays: any frozen text-to-image diffusion model with exposed token embeddings could in principle be steered with the same two losses, provided a spatial prior (even a crude one) can be written down for each concept of interest.","Because the supervision signal is the Gaussian prior itself, per-location performance should track the quality of the LUT; locations that had to be mapped from a neighbouring zone ('lingular', 'base', 'pleural') are a natural pressure test for whether the gains are real alignment or prior-fitting.","The ablation in which removing the diversity loss slightly improves average mIoU, driven by pneumothorax, suggests a pathology-aware weighting of $\\mathcal{L}_{\\mathrm{div}}$ (perhaps down-weighting it for pathologies intrinsically tied to one anatomy) could improve results further; the paper leaves this open.","If the approach scales to larger report corpora, the same coarse supervision could generate training signal for many more fine-grained anatomical terms than the 27 used here, effectively bootstrapping a localization model from text alone plus a small anatomy atlas."],"forward_implications":["A diffusion model's cross-attention can be used directly as a phrase-grounding heatmap: the fine-tuned LDM beats contrastive baselines (BioViL, BioViL-T) without any task-specific detection head.","Because only a (46, 1024) embedding codebook is updated, fine-tuning is cheap (about 3 hours on a single RTX 3090) and needs only about 6,500 curated image-report pairs.","The localization gains transfer across hospitals: VinDr-CXR, collected with different scanners and demographics, also shows higher CNR/mIoU than the baselines.","Standardizing free-text reports to '{location} {pathology}' prompts preserves average grounding performance, so the reported gains come from the alignment tuning rather than from prompt simplification.","Adding more anatomical locations or pathologies only requires adding an entry to the Gaussian lookup table and a corresponding token embedding, so the framework extends without architectural changes."],"supporting_citations":[{"why":"Supplies the latent diffusion architecture whose cross-attention is repurposed for phrase grounding.","marker":"(Rombach et al., 2022)"},{"why":"Provides the pre-trained frozen LDM checkpoint used as the backbone for fine-tuning.","marker":"(Pinaya et al., 2022)"},{"why":"Supplies the method for extracting phrase-grounding heatmaps from intermediate cross-attention layers and diffusion timesteps.","marker":"(Vilouras et al., 2024)"},{"why":"Contributes the MS-CXR benchmark, the BioViL baseline, and the CNR/mIoU evaluation protocol.","marker":"(Boecking et al., 2022)"},{"why":"RadGraph-XL is the clinical entity recognizer that identifies anatomical location mentions ('ANAT-DP') in free-text reports.","marker":"(Delbrouck et al., 2024)"},{"why":"Chest Imagenome gold-standard bounding boxes are used to compute the Gaussian lookup-table parameters.","marker":"(Wu et al., 2021)"},{"why":"VinDr-CXR provides the out-of-distribution test set with ground-truth bounding boxes and synthetic prompts.","marker":"(Nguyen et al., 2022)"},{"why":"BioViL-T is a strong contrastive baseline that the fine-tuned LDM outperforms on average in both benchmarks.","marker":"(Bannur et al., 2023)"}],"fun_headline_variants":["Token-only tuning aligns chest X-ray language and image","Anatomy tokens alone fix diffusion phrase grounding","Fine-tuning a few embeddings repairs chest X-ray grounding","Location-token tuning beats BioViL on chest X-ray grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a fixed Gaussian per anatomical term, averaged from Chest Imagenome bounding boxes (with 'lingular' mapped to 'left mid lung zone' and 'base' to 'lower lung zone'), is a correct-enough spatial prior for that term across every X-ray in the fine-tuning and evaluation sets, so that pushing attention toward the Gaussian teaches the right location rather than a systematically wrong one.","fun_headline_variants_meta":{"raw":{"variants":["Token-only tuning aligns chest X-ray language and image","Anatomy tokens alone fix diffusion phrase grounding","Fine-tuning a few embeddings repairs chest X-ray grounding","Location-token tuning beats BioViL on chest X-ray grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000389,"raw_usage":{"total_tokens":2056,"prompt_tokens":955,"completion_tokens":1101,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1037}},"tokens_in":571,"tokens_out":1101,"duration_ms":11058,"temperature":1.0,"reasoning_tokens":1037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:21:35.695331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare per-location phrase-grounding gains (fine-tuned minus frozen LDM) between anatomical terms that have direct Chest Imagenome box statistics and the mapped terms ('lingular', 'base', 'pleural'); if the mapped terms show no improvement, or the fine-tuned attention simply follows the fixed Gaussian even when the ground-truth box is elsewhere, then the method's apparent success is driven by the prior rather than by prompt tuning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained frozen LDM checkpoint used as the backbone for fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the method for extracting phrase-grounding heatmaps from intermediate cross-attention layers and diffusion timesteps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the MS-CXR benchmark, the BioViL baseline, and the CNR/mIoU evaluation protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VinDr-CXR provides the out-of-distribution test set with ground-truth bounding boxes and synthetic prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BioViL-T is a strong contrastive baseline that the fine-tuned LDM outperforms on average in both benchmarks."}],"review_version":1}