Pith. sign in

REVIEW 4 major objections 4 minor 68 references

A new training scheme aligns chest X-rays and radiology reports through shared pathology concepts, making generated findings track the actual image rather than language habits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:49 UTC pith:C2YBETUN

load-bearing objection A credible-sounding RRG method whose main claimed mechanism (MEM forcing local patch semantics) is not actually demonstrated; worth a referee but not a clean accept. the 4 major comments →

arxiv 2608.00279 v1 pith:C2YBETUN submitted 2026-07-31 eess.IV cs.CV

Learning to See Locally and Align Clinically with Pathology Semantics for Radiology Report Generation

Xuan Cuong Ngo This is my paper
classification eess.IV cs.CV
keywords radiology report generationchest X-rayvision-language alignmentpathology prototypesmasked evidence modelingcontrastive learningoptimal transportclinical faithfulness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the usual ways of tying chest X-ray images to their reports—autoregressive report supervision and contrastive learning—leave image-text alignment too loose, so decoders fall back on learned language priors and write plausible reports that ignore visible evidence. It proposes PALM, which replaces pairwise contrastive matching with alignment through a set of shared pathology prototypes, so reports that describe overlapping findings are pulled toward the same clinical concepts instead of being pushed apart. A second component, masked evidence modeling, trains the visual encoder to register the semantic loss when local image regions are removed, in order to sharpen patch-level sensitivity to radiographic cues. If these claims hold, the framework offers a route to radiology reports that are clinically specific and grounded in the image rather than statistically typical.

Core claim

PALM is a two-part training framework. In the alignment module, image and report embeddings are softly assigned to K learnable pathology prototypes via an optimal-transport plan, initialized from standard chest-radiology finding labels; a cross-entropy loss enforces consistency between the image-to-prototype and text-to-prototype distributions, permitting compatible but unpaired cases to share concepts. In the masked-evidence module, a frozen teacher encoder reads a masked image and produces the target semantics, while a student encoder sees the full image and must predict that target from only the visible patch tokens. The paper reports that this combination, followed by autoregressive repo

What carries the argument

The central mechanism is a set of shared pathology prototypes: learnable anchor vectors in a common vision-language space that stand for clinical concepts such as opacity or cardiomegaly. A balanced optimal-transport assignment softly matches every image and report in a batch to these prototypes, and consistency losses align the two modalities through the same prototype distribution, avoiding the false-negative problem of instance-level contrastive learning. The complementary mechanism is masked evidence modeling, a teacher-student self-distillation in which the student must reconstruct a frozen teacher's representation of a masked image from the remaining visible patch tokens, which is inte

Load-bearing premise

The paper assumes that solving the masked-evidence self-distillation task forces the student's patch tokens to encode local radiographic cues; it never verifies this at the patch level, and if the task is solved from coarse global image statistics, the local-evidence mechanism collapses even though benchmark scores may stay high.

What would settle it

A concrete test: take the trained student encoder and train a linear probe on its patch tokens to predict local findings region by region; if probe accuracy is no higher than for an encoder trained without masked evidence modeling, the claimed local-evidence mechanism is not doing the work. A complementary check: swap the visible patch-token input to the prediction decoder for a single global pooling of the image; if report-generation performance does not change, the objective can be solved without patch-level local semantics.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, image-report alignment in radiology should be organized around shared clinical concepts rather than strict image-report pairing, avoiding the counterproductive separation of compatible cases.
  • Encoders trained with masked evidence modeling should supply decoders with patch tokens that carry local radiographic content, reducing reliance on pretrained language priors.
  • The reported gains—best published BLEU-4, entity-relation F1, and clinical F1 on the largest public chest X-ray benchmark—imply the combination is compatible with existing autoregressive report generators and needs no retrieval or multi-view data.
  • Region-masking results suggest a practical evaluation protocol: if a report generator is truly evidence-sensitive, its quality should drop more when anatomical regions are masked than when background is masked.
  • Ablation shows the two components are complementary, with the full combination yielding the largest improvements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not tested in the paper: adapt the prototype set online or initialize it from a richer clinical ontology, since the fixed label-set initialization is acknowledged by the authors as a limit for rare findings.
  • If the local-token mechanism is real, a patch-level probe—classifying local findings from individual student tokens—should show a measurable advantage over an encoder trained without masked evidence modeling; the paper does not run that probe.
  • The prototype-alignment idea should transfer to other modalities where unpaired samples share compatible semantics, such as pathology slide-caption or CT-report learning, because the false-negative objection applies there too.
  • The counterfactual masking analysis points to a cheap general audit: comparing report-quality drops under clinically meaningful versus background masking could serve as a visual-grounding test for any radiology report model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PALM, a framework for chest X-ray report generation that adds two components to a DINOv2 image encoder and DistilGPT2 decoder: (1) Masked Evidence Modeling (MEM), a self-distillation objective intended to make patch tokens encode local radiographic evidence, and (2) Pathology-Aware Alignment (PAA), which aligns image and text embeddings to shared learnable pathology prototypes via Sinkhorn-based soft assignments. The authors claim state-of-the-art results on MIMIC-CXR, MIMIC-ABN, and IU X-ray across NLG and clinical efficacy metrics, and argue that the improvements reflect greater sensitivity to visual evidence rather than language priors. The central claim is that prototype-based alignment plus masked evidence modeling yields reports that are more faithful to the input image.

Significance. If the reported gains are real and the mechanisms are as stated, the paper would make a useful contribution: it identifies a known failure mode of contrastive learning in radiology — treating unpaired but clinically compatible reports as negatives — and offers a prototype-based alternative that is intuitive and plausible. The two-module design is clean and the experiments span three datasets with many recent baselines. The MEM objective is a reasonable self-supervised idea, and the paper explicitly targets the important issue of visual-evidence grounding. However, the current evidence does not establish the mechanism-level claims, and several reporting gaps prevent independent verification. The paper's own limitations section acknowledges the CheXpert-label-bound prototype set, which is relevant to the circularity concern below. With additional analyses and transparent reporting, this could become a solid contribution.

major comments (4)
  1. [Sec. 3.1, Eqs. (1)-(5)] The load-bearing inference that MEM forces the student to 'learn patch tokens that encode meaningful local information' is not demonstrated. The student encoder receives the full, unmasked image (Eq. 2); masking is applied only by selecting visible tokens afterward (Eq. 3). With a global ViT/DINOv2 backbone, visible tokens already contain information from masked patches via self-attention, so a lightweight decoder D could predict the teacher's masked-image representation from global/contextual statistics rather than from localized patch semantics. No patch-level probe, no attention/saliency analysis, and no control with a locally constrained encoder is provided. The sentence 'To solve this task well, the student must learn patch tokens that encode meaningful local information' is therefore a non-sequitur. This directly undermines Contribution 2 and the 'faithful to visual evidence' inter
  2. [Table 4 vs Tables 1 and 3] The counterfactual region-masking analysis is internally inconsistent. In Table 4, the unmasked PALM row reports RadGraph F1 = 0.330 and clinical F1 = 0.541, whereas Table 1 reports RG = 0.347 for PALM on MIMIC-CXR and Table 3 reports F1 = 0.545 for the full model. Similarly, MLRG unmasked in Table 4 gives RG = 0.252, but Table 1 gives RG = 0.291 for MLRG. If Table 4 uses a different evaluation subset or a different decoding protocol, that must be stated explicitly; otherwise the unmasked rows serve as the necessary baseline for the masking effect, and their mismatch with the main experimental tables makes the robustness comparison uninterpretable. The claim in Sec. 4.5 that 'PALM changes more clearly when clinically relevant regions are removed' depends on this baseline being commensurable.
  3. [Sec. 4.1 and Eqs. (9)-(13)] The experimental section does not provide error bars or significance tests for any reported metric, so the 'strong and consistent performance' claim in Sec. 4.2 is not statistically supported, especially when the margins over the runner-up are small (e.g., IU X-ray ROUGE-L +0.007). In addition, core hyperparameters are unpublished: K (number of prototypes), tau and epsilon in Eqs. (9)-(11), the MEM mask ratio and sampling scheme, the training stage lengths, and the loss weights for L_align and L_gen. Without these numbers, the results cannot be reproduced or compared fairly. Please include a complete hyperparameter table and report means/variance over at least three seeds or a significance test for the main comparisons.
  4. [Sec. 3.3, Sec. 4.1] There is a circularity risk between the prototype initialization/evaluation and the clinical metric. The prototypes are initialized by encoding CheXpert label names with CXR-BERT, and the text encoder is then fine-tuned jointly. The clinical efficacy metric in Sec. 4.1 also uses CheXpert labels to score the generated reports. Thus the 'clinically meaningful' bridge and the primary clinical evaluation share the same label source, and the reported F1/precision improvements (Table 2) could partly reflect the model aligning to CheXpert label semantics rather than to genuine radiographic evidence. The paper's own limitations section admits the prototype set is limited to predefined CheXpert labels. Please provide evidence that the alignment transfers beyond CheXpert, e.g., by evaluating on an independent label set, using expert annotation, or at least showing that the prototype set does not c
minor comments (4)
  1. [Sec. 4.1] The text says 'Following [30], we evaluate SCOPE using...' — 'SCOPE' appears to be a typo; it should be 'PALM' or 'our method'.
  2. [References] Reference [10] (Cuturi, Sinkhorn distances) is listed in the method but the citation for optimal transport is [41]; also reference [10] is not cited in the text. Please unify the citation style for the Sinkhorn algorithm.
  3. [Table 1] In the IU X-ray block, PALM's BLEU-1 is 0.483, which is not bold and is lower than MA's 0.501. The text says PALM 'achieves the best performance on most metrics' and 'some prior methods obtain slightly higher BLEU-1' — this is accurate, but the reader may be confused why the best score is not bolded; consider adding a footnote or explicit note.
  4. [Sec. 2] The phrase 'LLaV A-Med' contains an extra space; it should be 'LLaVA-Med'. Similarly, 'LLaV A-Rad' should be 'LLaVA-Rad'.

Circularity Check

0 steps flagged

No significant circularity: core derivation is self-contained, with an unverified but non-circular MEM mechanism and a non-reductive CheXpert-label overlap.

full rationale

PALM's derivation chain is not circular. MEM (Sec. 3.1, Eqs. 1-5) is a teacher-student self-distillation objective with a frozen teacher and no downstream-label supervision; the claim that it forces patch tokens to encode local evidence is an unproven architectural inference, not a definitional reduction. The pathology-aware alignment (Sec. 3.3) uses learnable prototypes initialized by encoding CheXpert label names through an external CXR-BERT encoder; the final prototypes and report decoder are trained with cross-entropy, not by optimizing the CheXpert F1 metric. The fact that the CheXpert label set is used both for prototype initialization and for clinical efficacy evaluation (Sec. 4.1) is an inductive-bias/evaluation overlap, but it does not make the result equivalent to the input by construction: the model must still detect each observation from images and generate free text, as verified in Tables 1-3 against external baselines. The region-masking experiment (Table 4) is a post-hoc sensitivity check, not a fitted prediction. There are no self-citations, imported uniqueness theorems, or renamed known results carrying the argument. The Limitations section appropriately flags the CheXpert-label coverage limit. The main weakness, that the MEM local-evidence mechanism is asserted without patch-level probes or a locally-constrained encoder control, is a correctness/verification gap, not a circularity, and does not affect the independent benchmark comparisons.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The method imports several assumptions it does not pay for: pretrained DINOv2/CXR-BERT features are suitable anchors; CheXpert's 14 labels are the right disease scaffold (the paper's Limitations concedes this); and the two causal mechanisms — masked-image feature prediction forces local patch semantics, and separating unpaired compatible reports causes hallucination — are asserted rather than probed. The unpublished hyperparameters (K, tau, epsilon, mask ratio, loss weights) are fitted by the authors to make the pipeline work and are required to reproduce the central claims.

free parameters (5)
  • K — number of pathology prototypes = not reported (implied 14, matching CheXpert)
    Structural capacity of the alignment module (Sec. 3.3, Eq. 9); never stated in the paper; with batch size B=8 it fixes the OT problem size and the granularity of pathology clustering.
  • tau — similarity/assignment temperature = not reported
    Used in C_ij = -z_i^T p_j / tau and in the softmax prototype similarities S_I-P, S_T-P (Sec. 3.3); controls assignment sharpness and is central to PAA behavior.
  • epsilon — Sinkhorn entropy regularizer = not reported
    Eq. 10; controls smoothness of the transport plans; determines whether assignments are near-uniform or peaked, yet no value or schedule is given.
  • MEM mask ratio and sampling scheme = not reported
    The fraction and distribution of masked patches in Sec. 3.1 determine the difficulty and effect of the masked-evidence objective; not specified.
  • Loss weights for L_align and L_gen = not reported (presumed equal weight 1)
    The paper never states how L_align (Eq. 13) and L_gen (Eq. 16) are combined, so the actual training objective is underdetermined.
axioms (5)
  • standard math Entropy-regularized OT (Sinkhorn-Knopp) yields a well-defined soft assignment that resists collapse to dominant prototypes.
    Invoked in Sec. 3.3 Eq. 10 to produce T_I-P and T_T-P; the solver's convergence and cooling behavior are taken as given.
  • domain assumption Predicting the frozen teacher's masked-image representation from visible student tokens forces patch tokens to encode local radiographic evidence.
    Sec. 3.1 states this as the MEM mechanism ('To solve this task well, the student must learn patch tokens that encode meaningful local information') but no patch-level probe or diagnostic verifies it; the objective could be satisfied via global feature statistics.
  • domain assumption CheXpert's label set is a clinically meaningful scaffold for CXR pathology and a valid seed for the prototypes.
    Prototypes are initialized from CheXpert names via CXR-BERT (Sec. 3.3); the paper's own Limitations (Sec. 5) concedes this misses rare or fine-grained findings, so 'pathology-aware' is bounded by CheXpert's categories.
  • domain assumption Contrastive negative separation of unpaired reports that share pathologies is a major cause of RRG hallucination and the modality gap.
    The motivating premise of Secs. 1 and 3.3; cited to Chowers et al. [8] for the modality gap, but the causal link to hallucinated findings is not demonstrated by an experiment that varies contrastive strength.
  • domain assumption Pretrained DINOv2 (Rad-DINO) and CXR-BERT provide suitable backbones whose feature spaces are a reasonable starting point for alignment.
    Used as the frozen teacher (Sec. 3.1) and text anchor (Sec. 3.3); the paper does not study sensitivity to these choices, so results are conditional on them.
invented entities (1)
  • Learned pathology prototypes P (K vectors in the shared embedding space) no independent evidence
    purpose: Shared semantic anchors that align image and text [CLS] embeddings via Sinkhorn assignments, replacing hard-negative instance separation with soft prototype-level alignment.
    The prototypes are central to the PAA contribution but have no falsifiable handle outside this paper: prototype-label agreement, interpretability, and seed variability are never measured. Clinical meaningfulness is asserted from the CheXpert-name initialization; the Limitations section concedes the scaffold is incomplete. The prototypes may be a useful clustering regularizer even without clinical content, but the claimed 'clinically meaningful bridge' is unvalidated.

pith-pipeline@v1.3.0-alltime-deepseek · 14198 in / 19988 out tokens · 189571 ms · 2026-08-04T00:49:54.112021+00:00 · methodology

0 comments
read the original abstract

Recent radiology-adapted vision-language models have achieved strong performance on standard report generation benchmarks, yet their robustness and generalization remain constrained by imperfect alignment and correlation between visual and textual features. Existing methods connect image and text either implicitly through autoregressive report supervision or explicitly through contrastive learning. However, autoregressive supervision alone is insufficient to establish reliable image-text alignment, while contrastive learning can push apart unpaired reports that describe related pathologies simply because they are not paired with the same image. This is problematic in radiology, where different reports may share compatible pathology semantics rather than being true negatives. As a result, the learned representation may fail to organize images and reports around shared pathology concepts, causing the decoder to rely on pretrained language priors and generate clinically plausible reports that are not fully supported by radiographic evidence. To address this issue, we propose PALM, a pathology-aware alignment framework for radiology report generation. Instead of directly matching each image-report pair while separating all others, PALM aligns visual and textual features through shared pathology prototypes. These prototypes provide a clinically meaningful bridge between radiographic evidence and textual findings, allowing cases with similar pathology semantics to move toward common concepts without separating compatible cases. In addition, we introduce Masked Evidence Modeling to strengthen the image encoder sensitivity to local radiographic evidence by learning semantic changes caused by masked image regions. Experiments on MIMIC-CXR, IU X-Ray, and MIMIC-ABN show that PALM consistently improves both report generation and abnormality-focused robustness.

Figures

Figures reproduced from arXiv: 2608.00279 by Xuan Cuong Ngo.

Figure 1
Figure 1. Figure 1: Model responses under visual information loss and abnormal findings. When lung regions are masked, existing MVLM-based RRG models still describe lung-related findings de￾spite the removal of visual evidence. On an abnormal case, they produce plausible but incomplete reports that miss the true finding, whereas our report is more consistent with the ground truth. We argue that this limitation is partly cause… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of PALM. The framework first fine-tunes the visual encoder with Masked Evidence Modeling, which encourages image tokens to encode local radiographic evidence under random masking. It then performs multimodal alignment through the proposed Pathology-Aware Alignment module, where visual and textual representations are aligned via shared pathology prototypes. The learned features are finally used, to… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of generated reports. MLRG pro [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 7 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Improving patient safety: avoiding unread imaging exams in the national va en- terprise electronic health record.Journal of digital imaging, 30(3):309–313, 2017

    Sarah Bastawrous and Benjamin Carney. Improving patient safety: avoiding unread imaging exams in the national va en- terprise electronic health record.Journal of digital imaging, 30(3):309–313, 2017. 1

    2017

  3. [3]

    Making the most of text semantics to improve biomedical vision–language processing

    Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision–language processing. InEuro- pean conference on computer vision, pages 1–21. Springer,

  4. [4]

    Generating radiology reports via memory-driven transformer

    Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. InProceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1439– 1449, 2020. 6, 7

    2020

  5. [5]

    Generating radiology reports via memory-driven transformer

    Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. InEMNLP, pages 1439–1449, 2020. 3

    2020

  6. [6]

    Cross-modal memory networks for radiology report genera- tion

    Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. Cross-modal memory networks for radiology report genera- tion. InACL, pages 5904–5914, 2021. 6, 7

    2021

  7. [7]

    Cross-modal memory networks for radiology report genera- tion

    Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. Cross-modal memory networks for radiology report genera- tion. InACL, pages 5904–5914, 2021. 3

    2021

  8. [8]

    Is the modality gap a bug or a feature? a robustness perspective

    Rhea Chowers, Oshri Naparstek, Udi Barzelay, and Yair Weiss. Is the modality gap a bug or a feature? a robustness perspective. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 30288– 30298, 2026. 2

    2026

  9. [9]

    Meshed-memory transformer for image caption- ing

    Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image caption- ing. InCVPR, pages 10578–10587, 2020. 3

    2020

  10. [10]

    Sinkhorn distances: Lightspeed computation of optimal transport.Advances in neural information pro- cessing systems, 26, 2013

    Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport.Advances in neural information pro- cessing systems, 26, 2013. 5

    2013

  11. [11]

    Preparing a collection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics As- sociation, 23(2):304–310, 2015

    Dina Demner-Fushman, Marc D Kohli, Marc B Rosen- man, Sonya E Shooshan, Laritza Rodriguez, Sameer An- tani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics As- sociation, 23(2):304–310, 2015. 6

    2015

  12. [12]

    Huang, X

    Z. Huang, X. Zhang, and S. Zhang. Kiut: Knowledge-injected u-transformer for radiology report generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19809–19818, 2023. 6, 7

    2023

  13. [13]

    Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Sil- viana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. InProceedings of the AAAI conference on artificial intelligence, pages 590–597, 2019. 1, 5, 6

    2019

  14. [14]

    Radgraph: Extracting clinical entities and relations from radiology re- ports.arXiv preprint arXiv:2106.14463, 2021

    Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. Radgraph: Extracting clinical entities and relations from radiology re- ports.arXiv preprint arXiv:2106.14463, 2021. 6

    Pith/arXiv arXiv 2021

  15. [15]

    On the automatic generation of medical imaging reports

    Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports. InACL, pages 2577– 2586, 2018. 2

    2018

  16. [16]

    Show, describe and conclude: On exploiting the structure information of chest x-ray reports

    Baoyu Jing, Zeya Wang, and Eric Xing. Show, describe and conclude: On exploiting the structure information of chest x-ray reports. InACL, pages 6570–6580, 2019. 2

    2019

  17. [17]

    Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.arXiv preprint arXiv:1901.07042, 2019

    Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.arXiv preprint arXiv:1901.07042, 2019. 6

    Pith/arXiv arXiv 1901

  18. [18]

    Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. Mimic-cxr database (version 2.0.0).PhysioNet, 2019. 1

    2019

  19. [19]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564,

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564,

  20. [20]

    Dynamic graph enhanced contrastive learning for chest X-ray report generation

    Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xi- aodan Liang, and Xiaojun Chang. Dynamic graph enhanced contrastive learning for chest X-ray report generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3334–3343, 2023. 3

    2023

  21. [21]

    Con- trastive learning with counterfactual explanations for radiol- ogy report generation, 2024

    Mingjie Li, Haokun Lin, Liang Qiu, Xiaodan Liang, Ling Chen, Abdulmotaleb Elsaddik, and Xiaojun Chang. Con- trastive learning with counterfactual explanations for radiol- ogy report generation, 2024. 6, 7

    2024

  22. [22]

    Yuan Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. Hy- brid retrieval-generation reinforced agent for medical image report generation.NIPS, 31, 2018. 2

    2018

  23. [24]

    Bootstrapping large language models for radiology report generation

    Chang Liu, Yuanhe Tian, Weidong Chen, Yan Song, and Yongdong Zhang. Bootstrapping large language models for radiology report generation. InProceedings of the AAAI Conference on Artificial Intelligence, pages 18635–18643,

  24. [25]

    Competence-based mul- timodal curriculum learning for medical report generation

    Fenglin Liu, Shen Ge, and Xian Wu. Competence-based mul- timodal curriculum learning for medical report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 3001–3012, 2021. 6, 7

    2021

  25. [26]

    Exploring and distilling posterior and prior knowledge for 9 radiology report generation

    Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. Exploring and distilling posterior and prior knowledge for 9 radiology report generation. InCVPR, pages 13753–13762,

  26. [27]

    Contrastive attention for automatic chest x-ray report generation

    Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. Contrastive attention for automatic chest x-ray report generation. InACL-IJCNLP, pages 269–280,

  27. [28]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 1, 3

    2023

  28. [29]

    Struc- tural entities extraction and patient indications incorporation for chest x-ray report generation

    Kang Liu, Zhuoqi Ma, Xiaolu Kang, Zhusi Zhong, Zhicheng Jiao, Grayson Baird, Harrison Bai, and Qiguang Miao. Struc- tural entities extraction and patient indications incorporation for chest x-ray report generation. InMedical Image Com- puting and Computer Assisted Intervention – MICCAI 2024, pages 433–443, Cham, 2024. Springer Nature Switzerland. 1, 6, 7

    2024

  29. [30]

    Enhanced contrastive learning with multi-view longitudinal data for chest x-ray report generation

    Kang Liu, Zhuoqi Ma, Xiaolu Kang, Yunan Li, Kun Xie, Zhicheng Jiao, and Qiguang Miao. Enhanced contrastive learning with multi-view longitudinal data for chest x-ray report generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10348–10359,

  30. [31]

    In-context learning for zero-shot medical re- port generation

    Rui Liu, Mingjie Li, Shen Zhao, Ling Chen, Xiaojun Chang, and Lina Yao. In-context learning for zero-shot medical re- port generation. InProceedings of the 32nd ACM Interna- tional Conference on Multimedia, pages 8721–8730, 2024. 6, 7

    2024

  31. [32]

    In-context learning for zero-shot medical re- port generation

    Rui Liu, Mingjie Li, Shen Zhao, Ling Chen, Xiaojun Chang, and Lina Yao. In-context learning for zero-shot medical re- port generation. InProceedings of the 32nd ACM Interna- tional Conference on Multimedia, pages 8721–8730, 2024. 3

    2024

  32. [33]

    In-context learning for zero-shot medical re- port generation

    Rui Liu, Mingjie Li, Shen Zhao, Ling Chen, Xiaojun Chang, and Lina Yao. In-context learning for zero-shot medical re- port generation. InACM Multimedia 2024, 2024. 6, 7

    2024

  33. [34]

    From observation to concept: A flexible multi-view paradigm for medical report generation

    Zhizhe Liu, Zhenfeng Zhu, Shuai Zheng, Yawei Zhao, Kun- lun He, and Yao Zhao. From observation to concept: A flexible multi-view paradigm for medical report generation. IEEE Transactions on Multimedia, 26:5987–5995, 2024. 3, 6, 7

    2024

  34. [35]

    Improving factual completeness and con- sistency of image-to-text radiology report generation

    Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky. Improving factual completeness and con- sistency of image-to-text radiology report generation. InPro- ceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5288–5304, 2021. 1

    2021

  35. [36]

    Learning visual-semantic embeddings for re- porting abnormal findings on chest x-rays.arXiv preprint arXiv:2010.02467, 2020

    Jianmo Ni, Chun-Nan Hsu, Amilcare Gentili, and Julian McAuley. Learning visual-semantic embeddings for re- porting abnormal findings on chest x-rays.arXiv preprint arXiv:2010.02467, 2020. 6

    Pith/arXiv arXiv 2010

  36. [37]

    Im- proving chest x-ray report generation by leveraging warm- starting.arXiv preprint arXiv:2201.09405, 2022

    Aaron Nicolson, Jason Dowling, and Bevan Koopman. Im- proving chest x-ray report generation by leveraging warm- starting.arXiv preprint arXiv:2201.09405, 2022. 3

    Pith/arXiv arXiv 2022

  37. [38]

    Lon- gitudinal data and a semantic similarity reward for chest x-ray report generation, 2023

    Aaron Nicolson, Jason Dowling, and Bevan Koopman. Lon- gitudinal data and a semantic similarity reward for chest x-ray report generation, 2023. 3, 6, 7

    2023

  38. [39]

    Dart: Disease-aware image-text alignment and self-correcting re-alignment for trustworthy radiology report generation

    Sang-Jun Park, Keun-Soo Heo, Dong-Hee Shin, Young-Han Son, Ji-Hye Oh, and Tae-Eui Kam. Dart: Disease-aware image-text alignment and self-correcting re-alignment for trustworthy radiology report generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 15580–15589, 2025. 1, 6, 7

    2025

  39. [40]

    Rad-dino: Exploring scalable medical image encoders beyond text supervision; 2024

    F P ´erez-Garc´ıa, H Sharma, S Bond-Taylor, K Bouzid, V Salvatelli, M Ilse, et al. Rad-dino: Exploring scalable medical image encoders beyond text supervision; 2024. 6

    2024

  40. [41]

    Now Founda- tions and Trends, 2019

    Gabriel Peyr ´e and Marco Cuturi.Computational optimal transport: With applications to data science. Now Founda- tions and Trends, 2019. 5

    2019

  41. [42]

    Im- proving radiology report generation systems by removing hallucinated references to non-existent priors

    Vignav Ramesh, Nathan A Chi, and Pranav Rajpurkar. Im- proving radiology report generation systems by removing hallucinated references to non-existent priors. InMachine Learning for Health, pages 456–473. PMLR, 2022. 1

    2022

  42. [43]

    Rosenow III

    Suhail Raoof, David Feigin, Arthur Sung, Sabiha Raoof, La- vanya Irugulpati, and Edward C. Rosenow III. Interpretation of plain chest roentgenogram.Chest, 141(2):545–558, 2012. 1

    2012

  43. [44]

    Radiologist shortage leaves patient care at risk, warns royal college.BMJ, 359, 2017

    Abi Rimmer. Radiologist shortage leaves patient care at risk, warns royal college.BMJ, 359, 2017. 1

    2017

  44. [45]

    Rosenkrantz, Danny R

    Andrew B. Rosenkrantz, Danny R. Hughes, and Richard Duszak Jr. The us radiologist workforce: an analysis of temporal and geographic variation by using large national datasets.Radiology, 279(1):175–184, 2016. 1

    2016

  45. [46]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019. 5

    Pith/arXiv arXiv 1910

  46. [47]

    Automatic radiology reports generation via memory align- ment network

    Hongyu Shen, Mingtao Pei, Juncai Liu, and Zhaoxing Tian. Automatic radiology reports generation via memory align- ment network. InProceedings of the AAAI Conference on Artificial Intelligence, pages 4776–4783, 2024. 3, 6, 7

    2024

  47. [48]

    Automatic radiology reports generation via memory align- ment network

    Hongyu Shen, Mingtao Pei, Juncai Liu, and Zhaoxing Tian. Automatic radiology reports generation via memory align- ment network. InProceedings of the AAAI Conference on Artificial Intelligence, pages 4776–4783, 2024. 6, 7

    2024

  48. [49]

    Cwcd: Category-wise contrastive decoding for structured medical report generation

    Shantam Srivastava, Mahesh Bhosale, David Doermann, and Mingchen Gao. Cwcd: Category-wise contrastive decoding for structured medical report generation. InMedical Imaging with Deep Learning, 2026. 1

    2026

  49. [50]

    Interactive and explainable region-guided radi- ology report generation

    Tim Tanida, Philip M¨ uller, Georgios Kaissis, and Daniel Rueckert. Interactive and explainable region-guided radi- ology report generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7433–7442, 2023. 3

    2023

  50. [51]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.NIPS, 30, 2017. 3

    2017

  51. [52]

    Show and tell: A neural image caption generator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, pages 3156–3164, 2015. 3

    2015

  52. [53]

    Multi-granularity cross-modal align- ment for generalized medical visual representation learning

    Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanab- huti, and Lequan Yu. Multi-granularity cross-modal align- ment for generalized medical visual representation learning. InAdvances in Neural Information Processing Systems, pages 33536–33549, 2022. 3 10

    2022

  53. [54]

    Hergen: Ele- vating radiology report generation with longitudinal data

    Fuying Wang, Shenghui Du, and Lequan Yu. Hergen: Ele- vating radiology report generation with longitudinal data. In European Conference on Computer Vision. Springer, 2024. 3, 6, 7

    2024

  54. [55]

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. Tienet: Text-image embedding net- work for common thorax disease classification and reporting in chest x-rays. InCVPR, pages 9049–9058, 2018. 2

    2018

  55. [56]

    A medical semantic-assisted transformer for ra- diographic report generation

    Zhanyu Wang, Mingkang Tang, Lei Wang, Xiu Li, and Lup- ing Zhou. A medical semantic-assisted transformer for ra- diographic report generation. InInternational Conference on Medical Image Computing and Computer-Assisted Interven- tion, pages 655–664. Springer, 2022. 3, 6, 7

    2022

  56. [57]

    Metransformer: Radiology report generation by transformer with multiple learnable expert tokens

    Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. Metransformer: Radiology report generation by transformer with multiple learnable expert tokens. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11558–11567, 2023. 6, 7

    2023

  57. [58]

    Medklip: Medical knowledge enhanced language- image pre-training for x-ray diagnosis

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language- image pre-training for x-ray diagnosis. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 21372–21383, 2023. 3

    2023

  58. [59]

    Chest imagenome dataset for clinical reasoning.arXiv preprint arXiv:2108.00316, 2021

    Joy T Wu, Nkechinyere N Agu, Ismini Lourentzou, Arjun Sharma, Joseph A Paguio, Jasper S Yao, Edward C Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. Chest imagenome dataset for clinical reasoning.arXiv preprint arXiv:2108.00316, 2021. 8

    Pith/arXiv arXiv 2021

  59. [60]

    Show, attend and tell: Neural image caption genera- tion with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption genera- tion with visual attention. InICML, pages 2048–2057, 2015. 3

    2048

  60. [61]

    Multimodal recurrent model with attention for automated radiology report generation

    Yuan Xue, Tao Xu, L Rodney Long, Zhiyun Xue, Sameer Antani, George R Thoma, and Xiaolei Huang. Multimodal recurrent model with attention for automated radiology report generation. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 457–

  61. [62]

    Kuo, Subathra Adithan, Eduardo Pontes Reis, Stephen Kwak, Vasantha Ku- mar Venugopal, Chloe O’Connell, Agustina Saenz, Pranav Rajpurkar, and Michael Moor

    Benjamin Yan, Ruochen Liu, David E. Kuo, Subathra Adithan, Eduardo Pontes Reis, Stephen Kwak, Vasantha Ku- mar Venugopal, Chloe O’Connell, Agustina Saenz, Pranav Rajpurkar, and Michael Moor. Style-aware radiology re- port generation with radgraph and few-shot prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14676–...

    2023

  62. [63]

    Automatic gener- ation of medical imaging diagnostic report with hierarchical recurrent neural network

    Changchang Yin, Buyue Qian, Jishang Wei, Xiaoyu Li, Xi- anli Zhang, Yang Li, and Qinghua Zheng. Automatic gener- ation of medical imaging diagnostic report with hierarchical recurrent neural network. InProceedings of the IEEE Interna- tional Conference on Data Mining (ICDM), pages 728–737. IEEE, 2019. 2

    2019

  63. [64]

    Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation

    Di You, Fenglin Liu, Shen Ge, Xiaoxia Xie, Jing Zhang, and Xian Wu. Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation. InMICCAI, pages 72–82. Springer, 2021. 3

    2021

  64. [65]

    Image captioning with semantic attention

    Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, pages 4651–4659, 2016. 3

    2016

  65. [66]

    A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings.Nature Communications, 16 (1):3108, 2025

    Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, et al. A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings.Nature Communications, 16 (1):3108, 2025. 1, 3, 6, 7

    2025

  66. [67]

    Radflag: A black-box hallucination detec- tion method for medical vision language models

    S Zhang, S Sambara, O Banerjee, J Acosta, LJ Fahrner, and P Rajpurkar. Radflag: A black-box hallucination detec- tion method for medical vision language models. arxiv 2024. arXiv preprint arXiv:2411.00299. 1

    Pith/arXiv arXiv 2024

  67. [68]

    Knowledge-enhanced visual-language pre- training on chest radiology images.Nature Communications, 14(1):4542, 2023

    Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre- training on chest radiology images.Nature Communications, 14(1):4542, 2023. 3

    2023

  68. [69]

    When radiology report generation meets knowledge graph

    Yixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu, Alan Yuille, and Daguang Xu. When radiology report generation meets knowledge graph. InAAAI, pages 12910–12917, 2020. 2, 3 11

    2020