RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment

Difei Gu; Dimitris Metaxas; Mu Zhou; Yang Zhou; Yunhe Gao

REVIEW 4 major objections 5 minor 1 cited by

RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RadAlign aligns X-ray images with diagnostic concepts first, then lets a frozen language model write the report; it reports an AUC of 0.885 on MIMIC-CXR and a GREEN score of 0.678, above the previous best 0.634.

desk verdict Plausible concept-aligned VLM + RAG for radiology reports, with strong classification but an unresolved patient-leakage question that could deflate the headline GREEN advantage. read the letter →

arxiv 2501.07525 v2 pith:FKNDSQXY submitted 2025-01-13 cs.CV cs.AIcs.LG

Difei Gu , Yunhe Gao , Yang Zhou , Mu Zhou , Dimitris Metaxas This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords radiologyreportgenerationvision-languagemodelconceptalignmentchestX-rayclassificationretrieval-augmentedexplainableAILLMpromptingmedicalimaging

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RadAlign is a chest X-ray system that claims disease classification and report writing should not be separate tasks. It first aligns image features with a small set of LLM-generated diagnostic concepts—heart size, opacity, borders, and similar criteria—through cross-attention and a contrastive loss, making the visual model itself an explainable classifier. That classifier's concept tokens then prompt a frozen LLM, with similar historical reports retrieved to ground the output, so language generation is used for phrasing rather than diagnosis. The paper reports that this arrangement reaches an average AUC of 0.885 on MIMIC-CXR, 0.923 on the unseen OpenI dataset, and a GREEN report score of 0.678 versus 0.634 for prior systems, without any language-model fine-tuning.

What carries the argument

The central object is a set of $K$ learnable visual concept tokens $z \in \mathbb{R}^{K \times d}$ inserted into the vision encoder and refined by cross-attention: $\hat{z} = \mathrm{cross\text{-}attention}(z, V(x), V(x))$. Each token is trained, through a domain-specific contrastive loss, to match the text embedding of one diagnostic criterion (e.g., heart size, opacity, borders), so the token becomes an interpretable anchor rather than a free latent. The similarities $\mathrm{sim}(\hat{z}_i, e_i)$ between tokens and criterion anchors are fed to a linear classifier, and the same $\hat{z}_i$ vectors index a retrieval database of training reports. That triple role—classify, retrieve, prompt—is what lets one aligned representation carry both the classification and the report generation.

What would settle it

Compute the MIMIC-CXR patient IDs in the training retrieval database and in the test set; if any overlap exists, remove those entries, rerun report generation, and see whether the GREEN score stays at 0.678. A second check is to re-run with the GPT-4-derived criteria replaced by rule-based criteria and compare both AUC and GREEN.

Watch

Extended reading notes

Core claim

The paper claims that a vision-language model trained to align visual features with expert diagnostic criteria can do two jobs at once: give state-of-the-art disease labels and supply the factual content of a radiology report. On MIMIC-CXR, RadAlign's average AUC is 0.885, above the best compared baseline PCAM's 0.861; on OpenI, it is 0.923 versus 0.836 with no fine-tuning on that dataset. For report generation, the same model reaches a GREEN score of 0.678 with GPT-4o, compared with 0.634 for the previous best R2GenCMN and ChatCAD. Because the LLM only receives recognized concepts, predicted labels, and retrieved reports, the paper argues that hallucinations are reduced without training a medical language model; the reports are grounded in what the alignment step has actually found.

Load-bearing premise

The method's reported report-quality gain assumes the retrieval database is built only from patients who are not in the test set; if a test patient's earlier study is stored in the database, retrieval could hand the LLM a near-reference report to paraphrase and inflate the GREEN score.

Editorial extensions

If this is right

A frozen general-purpose LLM can write clinically grounded radiology reports if the factual findings are decided beforehand by the concept-aligned vision model; the paper shows upgrading from GPT-4o-mini to GPT-4o raises RadAlign's GREEN score from 0.648 to 0.678 while the non-aligned ChatCAD baseline stays at 0.634.
Classification and reporting share the same visual features, so the model is inspectable: attention heatmaps for the five diseases highlight distinct anatomical regions, meaning the classifier is explainable and the report generator's content is already justified.
The concept bottleneck transfers across institutions: without any fine-tuning on OpenI, RadAlign maintains an average AUC of 0.923 versus 0.836 for a MIMIC-CXR-finetuned baseline, indicating the learned concepts are not dataset-specific.
The design point of $K=14$ concept anchors and $K=7$ retrieved reports is empirical: adding more anchors lowers all metrics, suggesting that a small, well-chosen concept set is preferable to a larger one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A reader could test whether the reported GREEN gain comes from concept prompting or from report retrieval by ablating retrieval entirely; if the score drops sharply, a large share of the 0.678 is likely stylistic grounding from similar cases rather than from the alignment itself.
If the concept alignment is doing the real work, then corrupting or removing a single diagnostic criterion anchor should degrade both the classification AUC and the GREEN score together; that paired movement is a testable signature of the paper's mechanism.
The same architecture should transfer to other structured reporting tasks, such as pathology or ECG interpretation, wherever an LLM can enumerate diagnostic criteria and a vision model can anchor to them, though the paper does not demonstrate this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: RadAlign is a slick, readable pipeline that combines concept-level VLM alignment, LLM prompting, and retrieval-augmented generation for chest X-ray reporting. The classification numbers are strong; the GREEN score advantage is plausible but the paper as written does not prove it. The stress-test flag is real: Section 3.4 constructs the retrieval database Q from the training set and never states the MIMIC-CXR split is patient-disjoint. MIMIC-CXR has multiple studies per patient, so if the split is by study, the retriever can hand the LLM a near-reference report from the same patient, inflating the 0.678 vs 0.634. That would make the RAG contribution look larger than it is. This is a load-bearing issue for the report-generation half of the central claim.

The concept alignment itself is a legitimate extension of the authors' earlier MICCAI work: learnable visual concept tokens anchored to LLM-derived diagnostic criteria, then used for classification and to seed the LLM prompt. That part is clean and the OpenI AUC jump (0.923 vs 0.836) is the most convincing result. The paper also deserves credit for testing several LLMs and showing attention maps.

Soft spots beyond the split: no no-retrieval ablation in Table 1, so you can't separate the RAG contribution from the concept alignment. The hyperparameter selection (K=14 anchors, TopK=7 reports) is done on what appear to be the same test metrics reported as final, with no validation split described. No error bars or significance tests. Those are evaluation gaps, not necessarily fatal—if the authors confirm a patient-disjoint split and add the no-retrieval baseline, the story holds. The GPT-4-derived criteria are an interesting form of knowledge grounding, but I'd like to see sensitivity to the criteria wording.

I'd accept this for peer review. It's a solid medical-imaging paper that needs a rigorous referee to check the split and ask for the missing ablations. I'd take it to our reading group for the evaluation-discussion value. Not sure I'd cite it until the split question is settled.

Referee Report

4 major / 5 minor

Summary. RadAlign proposes a framework that couples a concept-aligned vision-language model with LLM-based prompting for chest radiograph interpretation. The model first uses GPT-4 to derive K diagnostic criteria from MIMIC-CXR reports, then trains K visual concept tokens by contrasting them against text embeddings of those criteria. Disease classification is performed as a linear combination of per-criterion similarities, and report generation is done by prompting an LLM with the predicted concepts, disease labels, and several retrieved training reports. The paper reports average AUCs of 0.885 on MIMIC-CXR and 0.923 on OpenI, and a GREEN score of 0.678 for report generation versus a 0.634 baseline. The central claim is that aligning visual features with medical diagnostic criteria yields both accurate classification and clinically grounded report generation without extensive language-model training.

Significance. If the stated results hold, RadAlign would be a meaningful step toward interpretable, retrieval-grounded radiology report generation: it combines an explainable concept bottleneck with a frozen LLM, avoiding expensive report-generation training, and it reports strong out-of-domain classification on OpenI. The code release and the use of GREEN, a clinically oriented evaluation metric, are positives. The interpretability evidence in Fig. 2 is also a useful addition. However, the report-generation half of the claim rests on evaluation choices that are not fully documented, and the absence of a no-retrieval ablation leaves the contribution of the RAG component unclear. The classification results are more robust to these concerns, but the headline GREEN comparison needs additional verification.

major comments (4)

[Section 3.4, Eq. (5)-(6); Table 1] The retrieval database Q is built from the training set D, and the paper never states that the MIMIC-CXR train/test split is patient-disjoint. MIMIC-CXR contains multiple studies per patient; if the split is at the study level rather than the patient level, then a test patient's earlier or later reports can appear in Q. Since the LLM prompt includes retrieved reports, the model could paraphrase a near-reference report from the same patient, inflating the reported GREEN score of 0.678 versus 0.634. The authors should explicitly state the split protocol and, if it is not already patient-disjoint, rerun the evaluation with same-patient reports excluded from retrieval.
[Section 4.2, Fig. 2] The hyperparameters K=14 concept anchors and TopK=7 retrieved reports are selected from plots of the same metrics that are later reported as final results, yet no validation split is described anywhere in Section 4.1 or Section 4.2. Selecting K and TopK on test data makes the reported numbers optimistic and prevents a clean assessment of the method's true generalization. The authors should introduce a validation split for hyperparameter selection and report final results on a held-out test set only.
[Table 1, Section 4.2] There is no ablation comparing RadAlign with and without the retrieval-augmented generation component. Because the RAG mechanism is one of the paper's three main contributions and the headline GREEN score depends on it, the reader cannot tell how much of the 0.678 comes from concept alignment versus retrieval. An ablation with no retrieval, and ideally with randomly retrieved reports as a control, is needed to quantify the RAG contribution and to rule out that retrieval is simply copying style rather than adding diagnostic grounding.
[Table 1 and Section 4.2] No confidence intervals or repeated-run statistics are reported for the GREEN scores. The comparisons are between 0.678 and 0.634, and between 0.648 and 0.646 for different LLMs; without measures of variance, it is unclear whether these differences are meaningful, especially since LLM decoding is stochastic and the GREEN metric itself uses an LLM judge. The authors should report variance over multiple runs or at least provide a significance test for the main comparisons.

minor comments (5)

[Section 4.1] The phrase '40 epoches' should read '40 epochs'.
[Table 2(b)] The OpenI row for BioViL is identical to the CLIP row for all five diseases; this is likely a copy-paste error and should be corrected, since it makes the baseline comparison confusing.
[Fig. 2] The x-axis labels in the ablation plots are unevenly spaced (7, 11, 14, 20) and the axis for the GREEN plot starts at 0.625; displaying the actual tick values and a consistent scale would make the peaks easier to interpret.
[Section 3.1] The prompt given to GPT-4 for deriving diagnostic criteria is only described as 'I will provide a list of medical reports, can you help me identify common structural patterns in these reports?' in Fig. 1 rather than being fully specified in the text; the exact prompt and the number of criteria per disease should be included for reproducibility.
[Section 3.2, Eq. (2)] The contrastive loss in Eq. (2) is written with a single positive example, but the text says there are multiple descriptions per criterion; it would be clearer to denote the positive set explicitly or to explain how multiple positives are handled.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; classification is supervised and RAG retrieval is explicitly from training data, though the omitted patient-disjoint split is a leakage risk.

full rationale

The paper's classification and report-generation claims are not circular by construction. The concept-alignment loss (Eq. 2) trains visual tokens against text embeddings of diagnostic criteria, and the classifier (Eq. 3) adds a supervised cross-entropy term (Eq. 4) on disease labels; the reported AUCs therefore measure generalization to held-out labels, not a restatement of the training input. The diagnostic criteria in Section 3.1 are mined from the training reports D, not from test reports, so their use as anchors is a standard training-signal extraction. The RAG component (Section 3.4) is explicitly constructed from training images: Q = {(z_i, P_i)}_{i=1}^{|D|}, and retrieval is performed for x not in D. By the paper's own equations, the retrieved reports come from the training database, not from the test image itself. The one legitimate concern is that the paper never states whether the MIMIC-CXR train/test split is patient-disjoint. MIMIC-CXR contains multiple studies per patient, so if the split is by study, a test patient's earlier reports could sit in Q and the LLM prompt (Fig. 1) could paraphrase a near-reference report, inflating the GREEN score. This is a data-leakage and reproducibility risk, not a by-construction circularity: the equations do not force the test report into the inputs, and under a patient-level split the concern vanishes. The self-citation [8] is used only as general motivation that vision-language alignment is important; it is not invoked as a uniqueness theorem or as the sole support for the method. Overall, the derivation is self-contained and the central claims have independent supervised content.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; the visual concept tokens are learned parameters. The main burdens are the GPT-4 knowledge input and the unstated evaluation split.

free parameters (5)

Number of diagnostic criteria K = 14
Selected by ablation on AUC/F1/GREEN (Fig. 2); affects classifier and prompt.
Number of retrieved reports TopK = 7
Selected by ablation on GREEN score (Fig. 2d); affects report generation.
Linear classification weights W = learned
Trained with cross-entropy loss to map concept similarities to disease labels.
Contrastive temperature tau = not stated
Temperature in anchor contrastive loss (Eq. 2); value not reported.
Learning rate schedule = [1e-3, 1e-4] over 40 epochs
Standard hyperparameter setting for training the visual encoder.

assumptions (4)

ad hoc to paper GPT-4 derived diagnostic criteria are complete and clinically accurate for the five diseases.
Section 3.1: criteria are extracted by prompting GPT-4 on training reports; no validation against expert labels.
domain assumption The MIMIC-CXR train/test split is patient-disjoint and retrieval database contains no test patient reports.
Section 4.1 does not state the split; if false, retrieval can copy reference reports and inflate GREEN.
domain assumption Similarity between visual concept tokens is a reliable proxy for clinical report similarity.
Section 3.4: retrieval selects cases by concept-token similarity; no evaluation of retrieval quality.
domain assumption MIMIC-CXR and OpenI labels and reports are accurate ground truth.
Standard in the field; labels are derived from public datasets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment." pith.science (2026). https://pith.science/paper/FKNDSQXY

@misc{pith2026250107525,
  author       = {Pith},
  title        = {Pith review of: RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FKNDSQXY}},
  note         = {Machine review of arXiv:2501.07525}
}

read the original abstract

Automated chest radiographs interpretation requires both accurate disease classification and detailed radiology report generation, presenting a significant challenge in the clinical workflow. Current approaches either focus on classification accuracy at the expense of interpretability or generate detailed but potentially unreliable reports through image captioning techniques. In this study, we present RadAlign, a novel framework that combines the predictive accuracy of vision-language models (VLMs) with the reasoning capabilities of large language models (LLMs). Inspired by the radiologist's workflow, RadAlign first employs a specialized VLM to align visual features with key medical concepts, achieving superior disease classification with an average AUC of 0.885 across multiple diseases. These recognized medical conditions, represented as text-based concepts in the aligned visual-language space, are then used to prompt LLM-based report generation. Enhanced by a retrieval-augmented generation mechanism that grounds outputs in similar historical cases, RadAlign delivers superior report quality with a GREEN score of 0.678, outperforming state-of-the-art methods' 0.634. Our framework maintains strong clinical interpretability while reducing hallucinations, advancing automated medical imaging and report analysis through integrated predictive and generative AI. Code is available at https://github.com/difeigu/RadAlign.

Figures

Figures reproduced from arXiv: 2501.07525 by the authors.

**Figure 1.** Overview of the RadAlign framework. Our unified VLM predicts three key components: 1. Diagnostic criteria and associated concepts, formulated by an LLM based on historical reports, facilitate learning of image-concept alignment; 2. Disease prediction is derived from this alignment, enabling an explainable classifier; 3. Augmented historical reports are retrieved using learned visual concept tokens. These three comp… view at source ↗

**Figure 2.** Top: Ablation studies. (a-c) show classification performance of different number of concept anchors. (d) illustrates GREEN-score performance when varying the number of retrieved similar reports K. Bottom: Visualization. Attention map of concept tokens for different disease classes (AT, CM, CD, ED, PE), with warmer colors indicating higher attention scores. Ablation Studies. Our experiments identified optimal parame… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

BUSTR: Descriptor-Aware Vision-Language Learning for Breast Ultrasound Report Generation
cs.CV 2025-11 conditional novelty 5.0 of 10

BUSTR combines a descriptor-predicting vision encoder with a frozen language model to generate breast-ultrasound reports without paired image–report data, improving NLG and clinical-efficacy metrics over five baseline...

Reference graph

Works this paper leans on

33 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)

work page 2018
[2]

In: 2020 IEEE 6th international conference on computer and communications (ICCC)

Asif, S., Wenhui, Y., Jin, H., Jinhai, S.: Classification of covid-19 from chest x-ray images using deep convolutional neural network. In: 2020 IEEE 6th international conference on computer and communications (ICCC). pp. 426–433. IEEE (2020)

work page 2020
[3]

In: European conference on computer vision

Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: European conference on computer vision. pp. 1–21. Springer (2022)

2022
[4]

arXiv preprint arXiv:2204.13258 (2022)

Chen, Z., Shen, Y., Song, Y., Wan, X.: Cross-modal memory networks for radiology report generation. arXiv preprint arXiv:2204.13258 (2022)

arXiv 2022
[5]

arXiv preprint arXiv:2010.16056 (2020)

Chen, Z., Song, Y., Chang, T.H., Wan, X.: Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16056 (2020)

arXiv 2020
[6]

Journal of the American Medical Informatics Association23(2), 304–310 (2016)

Demner-Fushman, D., Antani, S., Simpson, M., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association23(2), 304–310 (2016)

work page 2016
[7]

arXiv preprint arXiv:2010.11929 (2020)

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

arXiv 2020
[8]

In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention

Gao, Y., Gu, D., Zhou, M., Metaxas, D.: Aligning human knowledge with visual concepts towards explainable medical image classification. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 46–56. Springer (2024)

2024

Show all 33 references

[9]

He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

2016
[10]

Hodler, J., Kubik-Huch, R.A., von Schulthess, G.K.: Diseases of the chest, breast, heart and vessels 2019-2022: diagnostic and interventional imaging (2019) 10 Difei Gu, Yunhe Gao, Yang Zhou, Mu Zhou, Dimitris Metaxas

2019
[11]

arXiv preprint arXiv:2004.12274 (2020)

Jing, B., Wang, Z., Xing, E.: Show, describe and conclude: On exploiting the struc- ture information of chest x-ray reports. arXiv preprint arXiv:2004.12274 (2020)

2020 arXiv
[12]

Scientific data 6(1), 317 (2019)

Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6(1), 317 (2019)

2019
[13]

In: International conference on machine learning

Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

2023
[14]

Advances in neural information processing systems 31 (2018)

Li, Y., Liang, X., Hu, Z., Xing, E.P.: Hybrid retrieval-generation reinforced agent for medical image report generation. Advances in neural information processing systems 31 (2018)

2018
[15]

In: Text Summarization Branches Out

Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. Association for Computational Linguis- tics, Barcelona, Spain (Jul 2004),https://aclanthology.org/W04-1013

2004
[16]

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y.: Exploring and distilling posterior and prior knowledge for radiology report generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13753–13762 (2021)

2021
[17]

Academic radiology 25(11), 1472–1480 (2018)

McBee, M.P., Awan, O.A., Colucci, A.T., Ghobadi, C.W., Kadom, N., Kansagra, A.P., Tridandapani, S., Auffermann, W.F.: Deep learning in radiology. Academic radiology 25(11), 1472–1480 (2018)

2018
[18]

Computer Methods and Programs in Biomedicine 226, 107141 (2022)

Okolo, G.I., Katsigiannis, S., Ramzan, N.: Ievit: An enhanced vision transformer architecture for chest x-ray image classification. Computer Methods and Programs in Biomedicine 226, 107141 (2022)

2022
[19]

arXiv preprint arXiv:2405.03595 (2024)

Ostmeier, S., Xu, J., Chen, Z., Varma, M., Blankemeier, L., Bluethgen, C., Michal- son, A.E., Moseley, M., Langlotz, C., Chaudhari, A.S., et al.: Green: Generative radiology report evaluation and error notation. arXiv preprint arXiv:2405.03595 (2024)

2024 arXiv
[20]

In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics

Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)

2002
[21]

In: Findings of the Association for Computational Linguistics: ACL 2022

Qin, H., Song, Y.: Reinforced cross-modal alignment for radiology report genera- tion. In: Findings of the Association for Computational Linguistics: ACL 2022. pp. 448–458 (2022)

2022
[22]

In: ICML

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)

2021
[23]

Ramesh, V., Chi, N.A., Rajpurkar, P.: Improving radiology report generation sys- temsbyremovinghallucinatedreferencestonon-existentpriors.In:MachineLearn- ing for Health. pp. 456–473. PMLR (2022)

2022
[24]

Medical Image Analysis p

Reale-Nosei, G., Amador-Domínguez, E., Serrano, E.: From vision to text: A com- prehensive review of natural image captioning in medical diagnosis and radiology report generation. Medical Image Analysis p. 103264 (2024)

2024
[25]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3156–3164 (2015)

2015
[26]

arXiv preprint arXiv:2302.07257 (2023)

Wang, S., Zhao, Z., Ouyang, X., Wang, Q., Shen, D.: Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257 (2023)

2023 arXiv
[27]

arXiv preprint arXiv:2210.10163 (2022) RadAlign: Advancing Radiology Report Generation 11

Wang, Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from un- paired medical images and text. arXiv preprint arXiv:2210.10163 (2022) RadAlign: Advancing Radiology Report Generation 11

2022 arXiv
[28]

arXiv preprint arXiv:2301.02228 (2023)

Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W.: Medklip: Medical knowledge en- hanced language-image pre-training in radiology. arXiv preprint arXiv:2301.02228 (2023)

2023 arXiv
[29]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., Yatskar, M.: Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19...

2023
[30]

arXiv preprint arXiv:2005.14480 (2020)

Ye, W., Yao, J., Xue, H., Li, Y.: Weakly supervised lesion localization with probabilistic-cam pooling. arXiv preprint arXiv:2005.14480 (2020)

2020 arXiv
[31]

arXiv preprint arXiv:2303.00915 (2023)

Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al.: Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915 (2023)

2023 arXiv
[32]

arXiv preprint arXiv:1904.09675 (2019)

Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

2019 arXiv
[33]

arXiv preprint arXiv:2401.02458 (2024)

Zhang, Y., Gao, J., Tan, Z., Zhou, L., Ding, K., Zhou, M., Zhang, S., Wang, D.: Data-centric foundation models in computational healthcare: A survey. arXiv preprint arXiv:2401.02458 (2024)

2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)

work page 2018

[2] [2]

In: 2020 IEEE 6th international conference on computer and communications (ICCC)

Asif, S., Wenhui, Y., Jin, H., Jinhai, S.: Classification of covid-19 from chest x-ray images using deep convolutional neural network. In: 2020 IEEE 6th international conference on computer and communications (ICCC). pp. 426–433. IEEE (2020)

work page 2020

[3] [3]

In: European conference on computer vision

Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: European conference on computer vision. pp. 1–21. Springer (2022)

2022

[4] [4]

arXiv preprint arXiv:2204.13258 (2022)

Chen, Z., Shen, Y., Song, Y., Wan, X.: Cross-modal memory networks for radiology report generation. arXiv preprint arXiv:2204.13258 (2022)

arXiv 2022

[5] [5]

arXiv preprint arXiv:2010.16056 (2020)

Chen, Z., Song, Y., Chang, T.H., Wan, X.: Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16056 (2020)

arXiv 2020

[6] [6]

Journal of the American Medical Informatics Association23(2), 304–310 (2016)

Demner-Fushman, D., Antani, S., Simpson, M., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association23(2), 304–310 (2016)

work page 2016

[7] [7]

arXiv preprint arXiv:2010.11929 (2020)

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

arXiv 2020

[8] [8]

In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention

Gao, Y., Gu, D., Zhou, M., Metaxas, D.: Aligning human knowledge with visual concepts towards explainable medical image classification. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 46–56. Springer (2024)

2024