Pith. sign in

REVIEW 5 major objections 5 minor 61 references

Anatomy-Guided Radiology Report Generation with Pathology-Aware Regional Prompts

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that explicitly linking detected lesions to anatomical regions through pathology-aware regional prompts makes chest X-ray report generation more clinically accurate.

desk verdict A competent incremental paper whose clinical-superiority claim outruns its evidence; the method is novel enough and the ablation strong enough to justify review, but the evaluation needs to account for annotation circularity and the expert results don't show clear superiority. read the letter →

arxiv 2411.10789 v1 pith:T7QLQ3IL submitted 2024-11-16 cs.CV

Yijian Gao , Dominic Marshall , Xiaodan Xing , Junzhi Ning , Giorgos Papanastasiou , Guang Yang , Matthieu Komorowski This is my paper · ORCID
classification cs.CV
keywords radiologyreportgenerationchestX-rayanatomicalregiondetectionmulti-labellesionpathology-awareregionalpromptsprompt-guideddecodingclinicalefficacyevaluationMIMIC-CXR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a chest X-ray report generator works better when it is explicitly told, before it writes, which pathology sits in which anatomical region. The proposed system detects 29 anatomical regions, detects multiple lesions per bounding box, and converts the two into a 29-token pathology-aware regional prompt that guides a BERT decoder. This mimics how a radiologist first scans each region, notes findings, and then composes an impression. Against prior state-of-the-art models on MIMIC-CXR, the system reports gains on most natural-language and clinical-efficacy metrics, and anonymised clinician review rates its reports as concise with a low rate of dangerous errors.

What carries the argument

The load-bearing object is the pathology-aware regional prompt: a vector of 29 tokens, one per anatomical region, where each token names the lesion assigned to that region or '[NEG]' if none is found. It is generated by a rule-based mapping from lesion bounding boxes to the 29 anatomical regions via IoU overlap, with only the highest-overlap box kept and with class reduction that removes the redundant 'lung opacity' root whenever a child node is present. A second mechanism is the label-squeeze loss in the multi-label lesion detector, which replaces one-class-per-box labels with a multi-hot class vector per box. Together these provide the decoder with explicit, localized diagnostic guidance on top of the anatomy-level visual features $V_a \in \mathbb{R}^{29 \times 1024}$.

What would settle it

A reader could take one hundred held-out MIMIC-CXR images, have radiologists mark every lesion and its region directly from the images, and then compare both the detector's prompt tokens and the generated report sentences to those marks; if report sentences frequently describe a lesion where the radiologist mark is negative, or miss lesions the radiologist marks positive more often than a baseline does, the claim that regional prompts encode genuine visual diagnosis is refuted.

Watch

Extended reading notes

Core claim

The paper's claim is that joint, spatially linked anatomy and pathology information is the missing ingredient in radiology report generation. It replaces fixed patch-level features with anatomy-level features from Faster R-CNN region proposals over 29 chest regions, and adds a YOLOv5-based multi-label lesion detector whose loss permits several diseases in one bounding box. Predicted lesions are assigned to the anatomical region with the highest IoU overlap, and a rule-based label strategy—drop 'lung opacity' as root, keep the more specific second-level node, fall back to '[NEG]'—builds a 29-token prompt that is fed to a BERT decoder along with region features. In Table II the method leads on BLEU-1, ROUGE-L, precision, and F1-score (0.394, 0.302, 0.509, 0.470) and in expert evaluation it scores best on brevity (0.01) and ties for lowest danger (0.03). The paper interprets this as evidence that emulating the radiologist's anatomy-first diagnostic search improves clinical accuracy.

Load-bearing premise

That the automatically constructed Chest ImaGenome annotations—29 region boxes plus lesion labels—are accurate enough to supervise the detectors, to build the prompt tokens, and to support the automated evaluation; if they are noisy or biased, the detectors, prompts, and clinical-efficacy scores all inherit the same bias.

Editorial extensions

If this is right

  • The intermediate detections become an audit trail: each generated report sentence can in principle be traced to a detected lesion in a specific region, which would let clinicians spot and correct errors.
  • Multi-label association per bounding box models co-occurring findings like 'lung opacity' alongside 'pleural effusion' without forcing a single label choice, a cleaner fit for chest X-rays.
  • Adding prompt guidance through a rule-based label strategy improves clinical efficacy without adding trainable parameters to the decoder.
  • Seeing all 29 region tokens at once should reduce laterality errors and self-contradictory statements that occur when each region is described by a separate sentence.
  • The reported results suggest that structured regional prompts can beat much larger vision-language models on this dataset, at a fraction of the parameter count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the prompt tokens and the CheXbert clinical-efficacy labels both come from the same MIMIC-CXR/Chest ImaGenome annotation stream, the reported F1 gains may partly measure consistency with that dataset's labeling style rather than independent visual truth; an external expert-adjudicated test set would separate the two.
  • The rule that always prefers second-level lesion tokens over rarer third-level subtypes keeps prompts reliable but may systematically omit precise diagnoses; a variant that samples or includes third-level tokens when detector confidence is high could recover some of that detail.
  • The two-detector-plus-prompt design is not chest-specific: any imaging modality with a region atlas (e.g., CT body-region or mammography reporting) could reuse the same IoU-based prompt construction, though the paper does not claim this.
  • The comparison with LLM baselines leaves open whether prompt structure or decoder capacity drives the gap; a controlled experiment varying only the decoder size would answer that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes an anatomy-guided radiology report generation system. A Faster R-CNN anatomical region detector extracts features from 29 chest regions, and a modified multi-label YOLOv5 lesion detector identifies up to 21 pathologies. Detected lesions are assigned to anatomical regions via IoU, generating 29 pathology-aware regional prompt tokens that prefix a BERT-style decoder. The pipeline is trained in two stages on MIMIC-CXR-JPG with Chest ImaGenome scene graphs. The authors report state-of-the-art or competitive results on most NLG and clinical efficacy metrics (Table II), an expert evaluation (Table III), an ablation study (Table IV), and object-detector performance (Tables V-VI).

Significance. If the central claim holds, the combination of anatomy-level visual features with pathology-aware regional prompts is a meaningful step toward explainable and clinically relevant report generation. The architecture is well motivated by radiological workflow, and the internal ablation consistently favors the full prompt system over the no-prompt baseline. The paper also reports detector-level results and qualitative localizations, which are useful for transparency. However, the clinical-efficacy claim is conditional on breaking the circular dependency between the scene-graph annotations, the CheXbert evaluation labels, and the reference reports; the current evidence does not establish that the gains reflect visual diagnosis rather than reproduction of report-derived label statistics.

major comments (5)
  1. [§IV.A, §III.C.2, §III.D.1, §IV.C.1] The clinical-efficacy claim rests on annotations that are not independent of the evaluation target. Chest ImaGenome scene graphs are automatically constructed from MIMIC-CXR reports (§IV.A), and these scene graphs supply both the lesion-detector training labels (§III.C.2) and the ground-truth prompt labels (§III.D.1). The CheXbert CE labels used as the automated outcome (§IV.C.1) are extracted from the same MIMIC-CXR report stream. The reported Precision/F1 gains could therefore reflect the model learning to reproduce report-derived annotation patterns rather than to diagnose image content. Please provide a concrete test of visual diagnosis: for example, evaluate on a subset with independent radiologist labels, or replace CheXbert with an external labeler on a different corpus, and report whether the PARP advantage persists. A prompt-scrambling control and an oracle-prompt upper bound would also help separate prompt information from decoder memorization.
  2. [§IV.D, Table II] The headline comparison is not statistically grounded. Most rows in Table II are taken from the original publications rather than re-evaluated under a common protocol; only RGRG*, CheXagent*, and PromptMRG* carry the asterisk denoting re-implementation. Several rows are missing key metrics (e.g., XrayGPT Precision/Recall, MedDr BLEU-2/3 and Precision/Recall/F1), and no confidence intervals or significance tests are reported. The differences that drive the 'outperforms SOTA' claim are sometimes within one or two thousandths (BLEU-2 0.251 vs 0.250; BLEU-4 tie at 0.126). Please re-evaluate all baselines under identical train/test conditions, or at minimum provide paired bootstrap confidence intervals and significance tests for the re-implemented subset, and clearly label missing entries.
  3. [§IV.C.2, Table III] The formal expert evaluation does not support the claim of clinical superiority. It was performed by two of the paper's authors (M. Komorowski and D. Marshall) on only 100 random samples, with no inter-rater reliability or significance testing reported. In Table III, Ours is not the best on Accuracy (3.51 vs. CheXagent's 3.55) or Rubric (2.26 vs. PromptMRG's 2.32); it only leads on Brevity and ties on Danger. Independent, blinded radiologist assessment on a larger sample, with agreement statistics, is needed before 'formal expert evaluations affirming its potential' can be claimed.
  4. [§IV.G, Table IV] The ablation is confounded by a training/inference mismatch. The +Text Prompt condition receives free-form textual prompts only at inference, because they have variable length and are not used during training, whereas the +PARP condition uses its prompts in both training and inference. The resulting drop in all metrics (e.g., BLEU-1 from 0.394 to 0.261) may reflect distribution shift rather than the intrinsic superiority of structured token prompts. Please compare against a fixed-length token-prompt baseline that is trained and tested symmetrically, and include a condition with randomly assigned or empty prompts to isolate the contribution of prompt content.
  5. [§IV.H.2, Table VI] The lesion detector's low recall for the very nodes prioritized in prompt construction undermines the assumed mechanism. Average recall is 0.285 and mAP@0.5 is 0.345; airspace opacity (recall 0.061) and consolidation (recall 0.075) are second-level nodes that, by §III.D.1, would be selected as the prompt token when present. The paper's statement that 'this challenge is unlikely to adversely affect report generation' is not quantified. Please report the fidelity of the constructed inference prompts against ground-truth scene graphs (prompt-level precision/recall per region) and show how prompt errors propagate to CE metrics.
minor comments (5)
  1. [§III.E, §IV.J] BERT-base has approximately 110 million parameters, not 'around 2B'; please correct this and clarify what 'configured in decoder mode' means for a bidirectional encoder model, including how causal masking is applied.
  2. [§III.D.2] If [NEG] tokens are used at inference but ground-truth prompts never contain [NEG], the decoder has never learned to handle them; please state whether [NEG] appears in the training prompt labels and, if not, explain how the model is expected to respond to it.
  3. [§IV.G, Table IV] The phrase 'F1-score increases by 8.9%' should be expressed as 8.9 percentage points (or a relative improvement of about 23%) to avoid ambiguity.
  4. [Table II] The rows for AdaMatch-Cyclic and MedDr are missing several metrics; please report the missing values or explicitly state why they are unavailable, since the 'most metrics' claim depends on complete comparisons.
  5. [§IV.A] Please clarify whether 'the official dataset split' refers to the MIMIC-CXR split or the Chest ImaGenome split, as the sample counts in the paper are not immediately matched to a named reference split.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the prompt labels are predicted from images at inference and the central comparison is against external baselines; shared report-derived annotation is a validity caveat, not a by-construction reduction.

full rationale

The claimed derivation chain is not circular by construction. Chest ImaGenome supplies bounding-box and lesion labels (Section IV.A), and the multi-label lesion detector is trained on those labels with image input only; at inference the pathology-aware regional prompt is built from detector outputs and anatomical-region overlap (Section III.D.2), not from the reference report. The decoder is trained with ground-truth prompt tokens prefixed to reference reports (Section III.E), which is teacher forcing of a latent summary, but the reported test-time generation uses predicted prompts on held-out images and is evaluated against reference reports and independent baselines. The fact that Chest ImaGenome scene graphs and CheXbert CE labels both derive from MIMIC-CXR reports is a legitimate benchmark-validity concern about shared annotation sources, and the expert evaluation is conducted by two of the authors, but neither makes any model output equal to its input by definition. No fitted parameter is renamed as a prediction, no uniqueness claim is imported from overlapping-author work, and no ansatz is smuggled in via citation. The finding is therefore no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The reported performance depends on several hand-set thresholds and filtering choices, on the fidelity of automatically constructed Chest ImaGenome scene graphs, and on the validity of CheXbert labels as clinical ground truth. No new physical or conceptual entities are introduced beyond the method's prompt tokens, which are a design component rather than an independent postulated entity.

free parameters (6)
  • lesion class frequency cutoff = 0.5% of training data
    Tail classes below 0.5% of training data are removed from the 42-class Chest ImaGenome label set, changing the detector and prompt vocabulary (Section III-C.2).
  • negative image filter fraction = 12%
    During lesion detector training, all but 12% of negative images are filtered following YOLOv5 docs (Section IV-A). This changes the prior over negative examples.
  • lesion-to-region IoU threshold = 0.4
    A lesion bbox is assigned to an anatomical region only if IoU exceeds 0.4 (Sections III-D.2 and IV-B.3). This determines which prompts are set to [NEG].
  • lesion detector confidence and NMS thresholds = conf 0.35, NMS IoU 0.45
    Set during inference of YOLOv5; these thresholds control which lesion bboxes become prompt tokens (Section IV-B.2).
  • lesion detector loss weights = lambda_cls=0.5, lambda_obj=1.0, lambda_box=0.05
    Hand-set weights in the multi-label YOLO loss (Equation 3), affecting detection behavior that feeds the prompts.
  • number of anatomical regions = 29
    The anatomical region detector outputs 29 fixed proposals per image (Equation 1), a design choice inherited from Chest ImaGenome that determines prompt length.
assumptions (5)
  • domain assumption Chest ImaGenome scene graph annotations are accurate and complete for 29 anatomical regions and 21 lesion classes.
    Both detectors and prompt labels are trained from these automatically constructed annotations (Sections III-B, III-C, III-D).
  • domain assumption IoU overlap between lesion bboxes and anatomical region bboxes is a valid way to localize pathologies to regions.
    Pathology-aware regional prompts are built by assigning each lesion to the anatomical region with highest IoU above 0.4 (Section III-D.2).
  • domain assumption CheXbert-based CE metrics on 14 observations are valid proxies for clinical accuracy.
    Clinical efficacy is measured by precision, recall, and F1 from CheXbert labels (Section IV-C1).
  • domain assumption The class reduction and rule-based prompt construction retain clinically sufficient information after removing the 'lung opacity' root and tail classes.
    The prompt design intentionally drops the most frequent class and keeps one token per region to reduce imbalance (Sections III-C.2 and III-D.1).
  • domain assumption The MIMIC-CXR reference reports used as targets are reliable ground truth for report quality.
    Training and CE evaluation use reference reports and derived labels (Section IV-A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Anatomy-Guided Radiology Report Generation with Pathology-Aware Regional Prompts." pith.science (2026). https://pith.science/paper/T7QLQ3IL

@misc{pith2026241110789,
  author       = {Pith},
  title        = {Pith review of: Anatomy-Guided Radiology Report Generation with Pathology-Aware Regional Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T7QLQ3IL}},
  note         = {Machine review of arXiv:2411.10789}
}
read the original abstract

Radiology reporting generative AI holds significant potential to alleviate clinical workloads and streamline medical care. However, achieving high clinical accuracy is challenging, as radiological images often feature subtle lesions and intricate structures. Existing systems often fall short, largely due to their reliance on fixed size, patch-level image features and insufficient incorporation of pathological information. This can result in the neglect of such subtle patterns and inconsistent descriptions of crucial pathologies. To address these challenges, we propose an innovative approach that leverages pathology-aware regional prompts to explicitly integrate anatomical and pathological information of various scales, significantly enhancing the precision and clinical relevance of generated reports. We develop an anatomical region detector that extracts features from distinct anatomical areas, coupled with a novel multi-label lesion detector that identifies global pathologies. Our approach emulates the diagnostic process of radiologists, producing clinically accurate reports with comprehensive diagnostic capabilities. Experimental results show that our model outperforms previous state-of-the-art methods on most natural language generation and clinical efficacy metrics, with formal expert evaluations affirming its potential to enhance radiology practice.

Figures

Figures reproduced from arXiv: 2411.10789 by the authors.

Figure 1
Figure 1. Pipeline of the proposed system. Initially, the anatomical region detector identifies and extracts visual features from 29 regions, with the first six shown in the figure. Simultaneously, the multi-label lesion detector identifies global pathologies, assigning multiple lesions to a single bbox. Pathology-aware regional prompts are generated by mapping lesion bboxes to corresponding anatomical regions based on overla… view at source ↗
Figure 2
Figure 2. Distribution of the lesion classes showing only the first three and last three classes. Original (42 classes); Modified (21 classes). 2, the dataset suffers from a highly imbalanced distribution, which can significantly impair computer vision tasks [48]. To address the long-tail effect and prioritize crucial lesion classes, we initially eliminate tail classes that constitute less than 0.5% of the training data. Furt… view at source ↗
Figure 3
Figure 3. Parent-child relationship in the Chest ImaGenome dataset [47], with ’Lung opacity’ as the root node. It branches into second-level nodes like ’pleural effusion’ and ’lung lesion,’ which further split into third-level nodes. The proportion of each node among the 42 lesion classes is indicated in brackets. D. Pathology-Aware Regional Prompts To emulate the working pattern of radiologists and explicitly guide the repor… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Examples of generated reports by our model and RGRG [40]. Green font indicates descriptions consistent with the reference report, while red font denotes incorrect descriptions of negative or unmentioned pathologies. Green highlights confirm correct pathology locations,…
Figure 5
Figure 5. Figure 5: Examples of the ablation study. Green font indicates descriptions consistent with the reference report, while red font denotes incorrect descriptions of negative or unmentioned pathologies. Green highlights confirm correct pathology locations, and red highlights point …
Figure 6
Figure 6. Figure 6: Examples of detection results and generated reports. Each image includes detections across 29 anatomical regions, with only the first 6 shown for clarity. Lesion bounding boxes are labeled with detected lesions and their confidence scores, with some labels hidden in th…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 36 canonical work pages

  1. [1]

    Ai-based radiodiagnosis using chest x-rays: A review,

    Y . Akhter, R. Singh, and M. Vatsa, “Ai-based radiodiagnosis using chest x-rays: A review,” Front. Big Data, vol. 6, p. 1120989, 2023

    work page 2023

  2. [2]

    Radiologist shortage leaves patient care at risk, warns royal college,

    A. Rimmer, “Radiologist shortage leaves patient care at risk, warns royal college,” BMJ-BRIT. MED. J., vol. 359, 2017

    2017

  3. [3]

    Show, attend and tell: Neural image caption generation with visual attention,

    K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in Proc. Int. Conf. Mach. Learn., 2015, pp. 2048– 2057

    work page 2015

  4. [4]

    Knowing when to look: Adaptive attention via a visual sentinel for image captioning,

    J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2017, pp. 375–383

    work page 2017

  5. [5]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in Neural Inf. Process. Syst., vol. 25, 2012

    work page 2012

  6. [6]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Inf. Process. Syst. , vol. 30, 2017

    work page 2017

  7. [7]

    A survey on deep learning and explainability for automatic report generation from medical images,

    P. Messina, P. Pino, D. Parra, A. Soto, C. Besa, S. Uribe, M. And ´ıa, C. Tejos, C. Prieto, and D. Capurro, “A survey on deep learning and explainability for automatic report generation from medical images,” ACM Comput. Surv., vol. 54, no. 10s, 2022

    work page 2022

  8. [8]

    Foundation model for advancing healthcare: Challenges, opportunities, and future directions,

    Y . He, F. Huang, X. Jiang, Y . Nie, M. Wang, J. Wang, and H. Chen, “Foundation model for advancing healthcare: Challenges, opportunities, and future directions,” arXiv:2404.03264, 2024

    arXiv 2024

Show all 61 references
  1. [9]

    Promptmrg: Diagnosis-driven prompts for medical report generation,

    H. Jin, H. Che, Y . Lin, and H. Chen, “Promptmrg: Diagnosis-driven prompts for medical report generation,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 3, 2024, pp. 2607–2615

    2024

  2. [10]

    Camanet: Class activation map guided attention network for radiology report generation,

    J. Wang, A. Bhalerao, T. Yin, S. See, and Y . He, “Camanet: Class activation map guided attention network for radiology report generation,” IEEE J. Biomed. Health Inform. , vol. 28, no. 4, pp. 2199–2210, 2024

    2024

  3. [11]

    Generating radiology re- ports via memory-driven transformer,

    Z. Chen, Y . Song, T.-H. Chang, and X. Wan, “Generating radiology re- ports via memory-driven transformer,” in Proc.Conf. Empirical Methods Natural Lang. Process., Nov. 2020

    2020

  4. [12]

    Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

    P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Comput. Surv. , vol. 55, no. 9, 2023

    2023

  5. [13]

    Can prompt learning benefit radiology report generation?

    J. Wang, L. Zhu, A. Bhalerao, and Y . He, “Can prompt learning benefit radiology report generation?” arXiv:2308.16269, 2023

    2023 arXiv

  6. [14]

    Advances in deep learning for tuberculosis screening using chest x-rays: the last 5 years review,

    K. Santosh, S. Allu, S. Rajaraman, and S. Antani, “Advances in deep learning for tuberculosis screening using chest x-rays: the last 5 years review,” J. Med. Syst. , vol. 46, no. 11, p. 82, 2022

    2022

  7. [15]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” inProc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017, pp. 4700–4708

    2017

  8. [16]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016, pp. 770–778

    2016

  9. [17]

    TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation,

    X. Yi, Y . Fu, R. Liu, H. Zhang, and R. Hua, “TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation,” IEEE J. Biomed. Health Inform. , vol. 28, no. 4, pp. 2152–2162, 2024

    2024

  10. [18]

    Memory Guided Transformer With Spatio-Semantic Visual Extractor for Medical Report Generation,

    P. Divya, Y . Sravani, C. Vishnu, C. K. Mohan, and Y . W. Chen, “Memory Guided Transformer With Spatio-Semantic Visual Extractor for Medical Report Generation,” IEEE J. Biomed. Health Inform. , vol. 28, no. 5, pp. 3079–3089, 2024

    2024

  11. [19]

    Eye Gaze Guided Cross-Modal Alignment Network for Radiology Report Generation,

    P. Peng, W. Fan, Y . Shen, W. Liu, X. Yang, Q. Zhang, X. Wei, and D. Zhou, “Eye Gaze Guided Cross-Modal Alignment Network for Radiology Report Generation,” IEEE J. Biomed. Health Inform. , pp. 1–14, 2024

    2024

  12. [20]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE Int. Conf. Comp. Vis., 2021, pp. 10 012–10 022

    2021

  13. [21]

    Cgftrans: Cross- modal global feature fusion transformer for medical report generation,

    L. Xu, Q. Tang, B. Zheng, J. Lv, W. Li, and X. Zeng, “Cgftrans: Cross- modal global feature fusion transformer for medical report generation,” IEEE J. Biomed. Health Inform. , vol. 28, no. 9, pp. 5600–5612, 2024

    2024

  14. [22]

    S3-Net: A Self- Supervised Dual-Stream Network for Radiology Report Generation,

    R. Pan, R. Ran, W. Hu, W. Zhang, Q. Qin, and S. Cui, “S3-Net: A Self- Supervised Dual-Stream Network for Radiology Report Generation,” IEEE J. Biomed. Health Inform. , vol. 28, no. 3, pp. 1448–1459, 2024

    2024

  15. [23]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” axXiv:2010.11929, 2021

    2010 arXiv

  16. [24]

    Xraygpt: Chest radiographs summarization using medical vision-language models,

    O. Thawkar, A. Shaker, S. S. Mullappilly, H. Cholakkal, R. M. Anwer, S. Khan, J. Laaksonen, and F. S. Khan, “Xraygpt: Chest radiographs summarization using medical vision-language models,” arXiv:2306.07971, 2023

    2023 arXiv

  17. [25]

    Metransformer: Radiology report generation by transformer with multiple learnable expert tokens,

    Z. Wang, L. Liu, L. Wang, and L. Zhou, “Metransformer: Radiology report generation by transformer with multiple learnable expert tokens,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , June 2023, pp. 11 558– 11 567

    2023

  18. [26]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  19. [27]

    Empirical eval- uation of gated recurrent neural networks on sequence modeling,

    J. Chung, C. Gulcehre, K. Cho, and Y . Bengio, “Empirical eval- uation of gated recurrent neural networks on sequence modeling,” arXiv:1412.3555, 2014

    2014 arXiv

  20. [28]

    Clinically accurate chest x-ray report generation,

    G. Liu, T.-M. H. Hsu, M. McDermott, W. Boag, W.-H. Weng, P. Szolovits, and M. Ghassemi, “Clinically accurate chest x-ray report generation,” in Proc. Mach. Learn. Healthcare Conf., 2019, pp. 249–269. 12 IEEE TRANSACTIONS AND JOURNALS TEMPLATE

    2019

  21. [29]

    When radiology report generation meets knowledge graph,

    Y . Zhang, X. Wang, Z. Xu, Q. Yu, A. Yuille, and D. Xu, “When radiology report generation meets knowledge graph,” in Proc. AAAI Conf. Artif. Intell. , vol. 34, no. 07, 2020, pp. 12 910–12 917

    2020

  22. [30]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics., 2019, pp. 4171– 4186

    2019

  23. [31]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  24. [32]

    Generating radiology reports via memory-driven transformer,

    Z. Chen, Y . Shen, Y . Song, and X. Wan, “Generating radiology reports via memory-driven transformer,” in Proc. 59th Annu. Meeting Assoc. Comput. Linguistics 11th Int. Joint Conf. Natural Lang. Process. , 2021

    2021

  25. [33]

    Clinical-bert: Vision-language pre-training for radiograph diagnosis and reports generation,

    B. Yan and M. Pei, “Clinical-bert: Vision-language pre-training for radiograph diagnosis and reports generation,” in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 3, 2022, pp. 2982–2990

    2022

  26. [34]

    On the difficulty of training recurrent neural networks,

    R. Pascanu, T. Mikolov, and Y . Bengio, “On the difficulty of training recurrent neural networks,” in Proc. Int. Conf. Mach. Learn. , 2013, pp. 1310–1318

    2013

  27. [35]

    Chexagent: Towards a foundation model for chest x-ray interpretation,

    Z. Chen, M. Varma, J.-B. Delbrouck, M. Paschali, L. Blankemeier, D. V . Veen, J. M. J. Valanarasu, A. Youssef, J. P. Cohen, E. P. Reis, E. B. Tsai, A. Johnston, C. Olsen, T. M. Abraham, S. Gatidis, A. S. Chaudhari, and C. Langlotz, “Chexagent: Towards a foundation model for ch...

    2024 arXiv

  28. [36]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” arXiv:2310.06825, 2023

    arXiv

  29. [37]

    Eva-clip: Improved training techniques for clip at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “Eva-clip: Improved training techniques for clip at scale,” arXiv:2303.15389, 2023

    2023 arXiv

  30. [38]

    Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning,

    S. He, Y . Nie, Z. Chen, Z. Cai, H. Wang, S. Yang, and H. Chen, “Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning,” arXiv:2404.15127, 2024

    2024 arXiv

  31. [39]

    Bootstrapping large language models for radiology report generation,

    C. Liu, Y . Tian, W. Chen, Y . Song, and Y . Zhang, “Bootstrapping large language models for radiology report generation,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 17, 2024, pp. 18 635–18 643

    2024

  32. [40]

    Interactive and explainable region-guided radiology report generation,

    T. Tanida, P. M ¨uller, G. Kaissis, and D. Rueckert, “Interactive and explainable region-guided radiology report generation,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2023, pp. 7433–7442

    2023

  33. [41]

    Finding-aware anatomical tokens for chest x-ray automated reporting,

    F. D. Serra, C. Wang, F. Deligianni, J. Dalton, and A. Q. O’Neil, “Finding-aware anatomical tokens for chest x-ray automated reporting,” arXiv:2308.15961, 2023

    2023 arXiv

  34. [42]

    Scene graph aided radiology report generation,

    J. Wang, L. Zhu, A. Bhalerao, and Y . He, “Scene graph aided radiology report generation,” arXiv:2403.05687, 2024

    2024 arXiv

  35. [43]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in Neural Inf. Process. Syst., vol. 28, 2015

    2015

  36. [44]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  37. [45]

    Medical image understanding with pretrained vision language models: A comprehensive study,

    Z. Qin, H. H. Yi, Q. Lao, and K. Li, “Medical image understanding with pretrained vision language models: A comprehensive study,” inProc. Int. Conf. Learn. Representations , 2022

    2022

  38. [46]

    YOLOv5: A state-of-the-art real-time object detection sys- tem,

    Ultralytics, “YOLOv5: A state-of-the-art real-time object detection sys- tem,” https://docs.ultralytics.com, 2021

    2021

  39. [47]

    Chest imagenome dataset,

    J. Wu, N. Agu, I. Lourentzou, A. Sharma, J. Paguio, J. S. Yao, E. C. Dee, W. Mitchell, S. Kashyap, A. Giovannini et al., “Chest imagenome dataset,” Physio Net, 2021

    2021

  40. [48]

    A survey on long-tailed visual recognition,

    L. Yang, H. Jiang, Q. Song, and J. Guo, “A survey on long-tailed visual recognition,” Int. J. Comput. Vision , vol. 130, no. 7, pp. 1837–1872, 2022

    2022

  41. [49]

    Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports,

    A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng, “Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports,” Sci. Data., vol. 6, no. 1, p. 317, 2019

    2019

  42. [50]

    Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs,

    A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y . Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng, “Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs,” arXiv:1901.07042, 2019

    1901 arXiv

  43. [51]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv:1711.05101, 2017

    2017 arXiv

  44. [52]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proc. 40th Annu. Meeting Assoc. Comput. Linguistics , 2002, pp. 311–318

    2002

  45. [53]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proc. 6th Workshop Statist. Mach. Transl. , 2005, pp. 65–72

    2005

  46. [54]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Proc. Text Summarization Branches Out , 2004, pp. 74–81

    2004

  47. [55]

    Combining automatic labelers and expert annotations for accurate radiology report labeling using bert,

    A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y . Ng, and M. P. Lungren, “Combining automatic labelers and expert annotations for accurate radiology report labeling using bert,” Proc. Conf. Empirical Methods Natural Lang. Process., 2020

    2020

  48. [56]

    Advancing multimodal medical capabilities of gemini,

    L. Yang, S. Xu, A. Sellergren, T. Kohlberger, Y . Zhou, I. Ktena, A. Kiraly, F. Ahmed, F. Hormozdiari, T. Jaroensri et al. , “Advancing multimodal medical capabilities of gemini,” arXiv:2405.03162, 2024

    2024 arXiv

  49. [57]

    Cxr-agent: Vision-language models for chest x-ray interpre- tation with uncertainty aware radiology reporting,

    N. Sharma, “Cxr-agent: Vision-language models for chest x-ray interpre- tation with uncertainty aware radiology reporting,” arXiv:2407.08811, 2024

    2024 arXiv

  50. [58]

    Fine-grained image-text alignment in medical imaging enables explainable cyclic image-report generation,

    W. Chen, L. Shen, J. Lin, J. Luo, X. Li, and Y . Yuan, “Fine-grained image-text alignment in medical imaging enables explainable cyclic image-report generation,” arXiv:2312.08078, 2024

    2024 arXiv

  51. [59]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm,

    M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin, “Free dolly: Introducing the world’s first truly open instruction-tuned llm,” Company Blog of Databricks , 2023

    2023

  52. [60]

    Medimageinsight: An open-source embedding model for general domain medical imaging,

    N. C. F. Codella, Y . Jin, S. Jain, Y . Gu et al. , “Medimageinsight: An open-source embedding model for general domain medical imaging,” arXiv:2410.06542, 2024

    2024 arXiv

  53. [61]

    Foundation models for generalist medical artificial intelligence,

    M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,” Nature, vol. 616, no. 7956, pp. 259–265, 2023

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.