Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Generated radiology reports, inserted as extra text context, improve a chest X-ray VQA model's answers on both single-image and image-difference questions, achieving state-of-the-art results on the Medical-Diff-VQA benchmark.

desk verdict Useful, honest empirical paper: predicted reports help CXR VQA on Medical-Diff-VQA, but the gain is small, benchmark-derived, and needs variance and generalization checks before trusting. read the letter →

arxiv 2505.16624 v1 pith:5ALB42SU submitted 2025-05-22 cs.CV cs.CL

Francesco Dalla Serra , Patrick Schrempf , Chaoyang Wang , Zaiqiao Meng , Fani Deligianni , Alison Q. O'Neil This is my paper
classification cs.CVcs.CL
keywords visualquestionansweringchestx-rayradiologyreportgenerationchain-of-thoughtreasoningimage-differenceVQAvision-languagemodelMedical-Diff-VQAanatomicaltokens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a chest X-ray visual question answering (VQA) system answers more accurately when it first generates a radiology report for the image and then reads that report alongside the question, a form of chain-of-thought grounding. The authors propose a two-stage pipeline: a Report Generator writes the findings and impression sections, and an Answer Generator receives those sections as extra text evidence before producing the answer. On the Medical-Diff-VQA dataset, the grounded model beats the same model without the report on every reported metric, including difference-question BLEU-4 (0.551 vs 0.525) and open-ended accuracy (0.523 vs 0.509). The claim matters because it offers a simple, architecture-agnostic way to inject clinical reasoning into CXR VQA: use the report as an intermediate reasoning product rather than only as a pre-training target.

What carries the argument

The work is carried by a shared vision-language architecture in two roles. Visual input is encoded as finding-aware anatomical tokens: a two-stage object detector localises 36 anatomical regions and scores 71 findings, producing a token per region that is fed to both modules. A Longitudinal Projection Module aligns and concatenates current and prior tokens through a residual MLP, giving the model a joint representation for image-difference questions. The language model is an encoder-decoder Transformer (three layers, eight heads, 512 hidden units, 68M trainable parameters). The Report Generator uses two prompts to produce findings and impression sections separately; the Answer Generator, initialised from the report generator's weights, takes the predicted findings and impression concatenated with the question (and candidate answers for multiple choice) as its text input. The predicted report is the load-bearing mechanism: it carries the facts the answer must draw on, and its quality determines the ceiling of the whole pipeline.

What would settle it

Run the same RG-AG model on a held-out question set whose answers are deliberately not mentioned in the predicted report (for example, ask about a finding the report omits, or use clinician-written questions outside the 18 templates); if accuracy stays at or above the AG-without-report baseline, the report is not doing the causal work, whereas a drop would confirm error propagation through the two-stage pipeline.

Watch

Extended reading notes

Core claim

The central discovery is that a predicted radiology report, even when imperfect, acts as effective intermediate evidence for answer generation. Concretely, the Report Generator-Answer Generator (RG-AG) model outperforms the answer-generator-only baseline across all metrics: difference questions improve on BLEU-1 to CIDEr (e.g., BLEU-4 0.551 vs 0.525), open-ended exact-match accuracy rises from 0.509 to 0.523, closed yes/no accuracy from 0.865 to 0.871, and overall accuracy from 0.683 to 0.693. The ablation shows the findings section contributes more than the impression section, but the two together work best; replacing predicted reports with ground-truth expert reports lifts accuracy from 0.693 to 0.751, identifying report quality as the main driver. Removing the image and using only the report text drops accuracy to 0.630, so the visual input remains necessary. The authors frame the report as a chain-of-thought product that makes the evidence behind the answer explicit, and they position RG-AG as the first demonstration that grounding medical VQA with generated reports helps both single-image and image-difference questions.

Load-bearing premise

The whole improvement rests on the generated report already containing the information needed to answer the question; on this dataset that holds by construction because the questions are derived from the ground-truth reports, but for clinical questions that do not match the report template the grounding may add nothing or even inject errors.

Editorial extensions

If this is right

  • Adding a predicted report as input improves the same answer generator on every metric, so report grounding can be adopted without redesigning the answer-generation head.
  • Gains are strongest on open-ended and difference-type questions, where the answer is not a single word and the report supplies the needed vocabulary.
  • Because ground-truth reports raise accuracy from 0.693 to 0.751, better report generation should translate directly into better VQA answers.
  • The finding section matters more than the impression section, but combining both helps most, suggesting the model benefits from both granular observations and their synthesis.
  • Visual input is still required: conditioning on report text alone drops accuracy to 0.630, so the report complements rather than replaces image features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Medical-Diff-VQA questions are automatically built from ground-truth reports, the measured gain may be inflated by the report containing the answer's wording; a fair stress test would evaluate RG-AG on clinician-written questions not derived from report templates.
  • The dependence on report quality suggests a natural extension: route the answer generator to use only the report sections whose content is relevant to the question, or verify that the predicted report contains the answer span before trusting the grounded answer.
  • The two-stage design could be inverted as a model-diagnostic tool: when the predicted report and the answer disagree, the failure is attributable to either report generation or answer generation, offering a clean error-analysis split for future CXR VQA systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a two-stage Chest X-ray Visual Question Answering (VQA) pipeline, RG-AG. A Report Generator (RG) first produces the findings and impression sections of a radiology report from anatomical tokens extracted from one or two CXRs; an Answer Generator (AG), initialized from the RG weights, then answers single-image or image-difference questions, optionally using the predicted report as additional textual evidence. The method is evaluated on Medical-Diff-VQA, covering both free-form (open) and yes/no (closed) questions as well as difference-type questions. The authors report state-of-the-art results and present ablations with and without the report, with findings vs. impression sections, with ground-truth reports as an upper bound, and with text-only input.

Significance. If the claimed improvements are reliable, the paper makes a useful step toward grounding medical VQA in structured radiology evidence. The experimental design is clean: the ablations bracket the contribution of the predicted report, and the use of ground-truth reports as an upper bound is informative. The paper is also candid about failure modes and dataset limitations, and the qualitative analysis of error propagation is valuable. However, the magnitude of the grounding benefit is small and statistically unverified, and the benchmark is derived from the very reports used to train the report generator, which limits the external validity of the state-of-the-art claim. These issues need to be addressed before the central claim can be accepted as demonstrated.

major comments (3)
  1. [Section V-B / Table IV / Section IV-B] The central claim that grounding with a predicted report improves answers rests on small differences: overall accuracy increases from 0.683 (AG) to 0.693 (RG-AG), open-ended accuracy from 0.509 to 0.523, and diff BLEU-4 from 0.525 to 0.551. The paper states that each experiment was repeated three times with different random seeds and the average is reported, but no variance or statistical significance is given. Without error bars or a significance test, the 0.010 overall accuracy gain could be within seed noise. Please report per-seed results with standard deviations, or provide paired significance tests over the test set, to establish that the grounding improvement is not due to random variation.
  2. [Section IV-A / Section VI] The Medical-Diff-VQA question-answer pairs are semi-automatically derived from the ground-truth MIMIC-CXR reports (Section IV-A), and the RG module is trained on the same MIMIC-CXR reports. Consequently, the predicted reports are strongly biased to contain the exact keywords and attributes that constitute the answers, allowing the AG to perform near-extractive QA. The paper acknowledges in Section VI that questions not explicitly addressed in the reports may limit or negate the benefit of grounding. This means the empirical gain is demonstrated only for a distribution that is generated from report text; the abstract and conclusion claim a more general enhancement. To support the general claim, the authors should either evaluate on questions that are not derivable from report language, or explicitly restrict the claim to report-derived question distributions.
  3. [Section V-A / Tables II and III] The 'state-of-the-art' attribution is not solely due to grounding. The AG baseline without the predicted report already surpasses PLURAL on several diff-VQA metrics (BLEU-4 0.525 vs. 0.520; CIDEr 2.102 vs. 1.832) and is only 0.008 below PLURAL's closed-question accuracy (0.865 vs. 0.873). The incremental gain from the predicted report is therefore small relative to the architectural choices and report-generation pretraining. Please disentangle the contribution of grounding from the contribution of the base architecture and pretraining, and temper the abstract's phrasing accordingly.
minor comments (6)
  1. [Abstract] The phrase 'both single-image image-difference questions' appears to be missing a conjunction; it should likely read 'both single-image and image-difference questions.'
  2. [Section III-C / Eq. (4)] The prompt template in Eq. (4) uses '[RG]{ind}[Q]Inst_f', but the special token [Q] and the exact separation between the indication field and the instruction are not fully specified. A short complete input example would improve reproducibility.
  3. [Section III-D / Eq. (6)] The notation {rr} is defined as the concatenation of F and I from Eq. (4), but it is not stated whether the two sections are separated by a delimiter. Please clarify the exact input format.
  4. [Section V-C / Figure 5] The row descriptions in the text (e.g., 'report inconsistency leading to incorrect answer' for row 4) should be aligned with the row labels in the figure caption to avoid ambiguity.
  5. [Table IV] For the text-only row ('- F + I'), the visual input is marked as none; please clarify how the model handles missing visual tokens (e.g., all-zero vectors) and state explicitly that this row is a lower-bound condition.
  6. [Section IV-B] A sentence describing hyperparameter selection (e.g., whether any regularization or warmup was used) would strengthen reproducibility, given that only the learning rate schedule is currently reported.

Circularity Check

1 steps flagged · score 4.0 of 10

The reported benefit of grounding with generated reports is largely an artifact of Medical-Diff-VQA's report-derived question construction; the paper's own limitation section concedes the effect may vanish for questions not addressed in the reports.

  1. self definitional [Section IV-A (Datasets), Section III-D / Eq. (6), and Section VI (Limitations)]
    "The question-answer pairs in this dataset are derived from the free-text radiology reports from MIMIC-CXR [15, 16, 9]. ... where {rr} refers to the predicted radiology report for the given scan, obtained by concatenating the findings (F) and impression sections (I) from Eq. 4. ... Pairs were derived semi-automatically from the ground truth radiology reports, therefore, the style and content of information contained within these reports are highly relevant for answering the questions."

    Medical-Diff-VQA answers are extracted from the same MIMIC-CXR ground-truth reports that the Report Generator is trained to imitate (Section IV-A/IV-B). The Answer Generator is fed the predicted findings+impression as {rr} (Eq. 6). Whenever the predicted report approximates the ground-truth report, the answer tokens (disease keywords, locations, types, levels) are already present in the input text. The claimed improvement of RG-AG over AG (e.g., open accuracy 0.523 vs 0.509; diff BLEU-4 0.551 vs 0.525) is therefore largely a measure of how well the RG reproduces the answer-bearing report, not an independent demonstration that report grounding improves VQA reasoning.

full rationale

The two-stage pipeline itself is not logically circular: the report generator is trained on MIMIC-CXR reports and the answer generator is evaluated on predicted reports without fitting the answer labels into the report. However, the central empirical claim—that adding a predicted report improves VQA accuracy—is confounded by the benchmark construction: Medical-Diff-VQA question-answer pairs are semi-automatically generated from the ground-truth radiology reports (Section IV-A), and the RG is trained on those same reports. Consequently, the predicted report is a noisy surrogate for the exact text from which the answer was derived, making the AG's task substantially extractive. The paper's own limitations section states that for question types not addressed in the reports, the benefit may be limited or negated, which is the definitional signature of this benchmark-specific circularity. No load-bearing self-citation or uniqueness-theorem smuggling was found; the architectural citations ([5], [6]) are independent component works, and the RG-AG ablation with ground-truth reports (accuracy 0.751 vs 0.693) shows report quality matters. The finding is therefore a partial, acknowledged evaluation circularity rather than a fully forced derivation, supporting a moderate score of 4.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is an empirical deep learning paper, so there are no physical free parameters or invented entities. The central claim rests on domain assumptions: that the Medical-Diff-VQA benchmark is a valid measure of CXR VQA performance, that the anatomical tokens from [6] preserve sufficient visual information, and that the training distribution (MIMIC-CXR reports) matches the evaluation distribution. The paper is transparent about the first assumption in Section VI. No code release means the reproducibility of these assumptions cannot be independently checked.

assumptions (3)
  • domain assumption Medical-Diff-VQA question-answer pairs are a valid proxy for clinical CXR VQA performance.
    The authors construct and evaluate on this benchmark, and even note in Section VI that questions are limited to 18 templates and are derived from radiology reports, which may limit generalization to real clinical questions.
  • domain assumption The finding-aware anatomical tokens from the Faster R-CNN detector carry enough visual information for both report and answer generation.
    These tokens are the only visual representation used; the design is borrowed from [6] and not independently validated here. If the tokens miss findings, both the report and the answer will be wrong.
  • domain assumption The predicted reports used to train the answer generator are similar in distribution to the ground-truth reports from which the questions were derived.
    The report generator is trained on MIMIC-CXR reports, the same source that was used to create the VQA dataset, so the predicted reports are likely to contain answer keywords. This is what makes the grounding effective, but it is an assumption about distribution overlap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports." pith.science (2026). https://pith.science/paper/5ALB42SU

@misc{pith2026250516624,
  author       = {Pith},
  title        = {Pith review of: Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ALB42SU}},
  note         = {Machine review of arXiv:2505.16624}
}
read the original abstract

We present a novel approach to Chest X-ray (CXR) Visual Question Answering (VQA), addressing both single-image image-difference questions. Single-image questions focus on abnormalities within a specific CXR ("What abnormalities are seen in image X?"), while image-difference questions compare two longitudinal CXRs acquired at different time points ("What are the differences between image X and Y?"). We further explore how the integration of radiology reports can enhance the performance of VQA models. While previous approaches have demonstrated the utility of radiology reports during the pre-training phase, we extend this idea by showing that the reports can also be leveraged as additional input to improve the VQA model's predicted answers. First, we propose a unified method that handles both types of questions and auto-regressively generates the answers. For single-image questions, the model is provided with a single CXR. For image-difference questions, the model is provided with two CXRs from the same patient, captured at different time points, enabling the model to detect and describe temporal changes. Taking inspiration from 'Chain-of-Thought reasoning', we demonstrate that performance on the CXR VQA task can be improved by grounding the answer generator module with a radiology report predicted for the same CXR. In our approach, the VQA model is divided into two steps: i) Report Generation (RG) and ii) Answer Generation (AG). Our results demonstrate that incorporating predicted radiology reports as evidence to the AG model enhances performance on both single-image and image-difference questions, achieving state-of-the-art results on the Medical-Diff-VQA dataset.

Figures

Figures reproduced from arXiv: 2505.16624 by the authors.

Figure 1
Figure 1. Overview of the Report Generator–Answer Generator (RG-AG) pipeline: (1) the Report Generator first produces a radiology report based on the given Chest X-ray (or a pair of images in the case of a follow-up study), along with the instruction and the indication field. The report consists of the ‘finding’ and ‘impression’ sections, which are generated independently based on the specific instructions received by the RG … view at source ↗
Figure 2
Figure 2. 1) Longitudinal Projection Module: The LPM is responsible for projecting the current and prior CXR scans into a joint representation. The input visual components of the LPM correspond to the finding-aware anatomical tokens from the current and for image-difference types of questions, the prior scan is also provided. These are denoted as Vc = {⃗vc,n} N n=1 and Vp = {⃗vp,n} N n=1, respectively. Whenever we do not inte… view at source ↗
Figure 2
Figure 2. The model architecture of the Report Generator and Answer Generator. This diagram illustrates on the left the Visual Anatomical Token Extractor – responsible for extracting the visual tokens from CXRs. This component is trained independently of the Vision-Language Model. On the right, the Vision-Language Model architecture is responsible for generating the radiology report or the answer. The diagram shows how visual… view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 4
Figure 4. Figure 4: We compare the quality of our predicted answers without the predicted CXR radiology report (AG model) and with it (our RG-AG model). For each question (Q), we highlight the correct parts of the answer (A) in green and the errors in red. Similarly, in the predicted radi…
Figure 5
Figure 5. Figure 5: We present borderline and failure cases of our RG-AG model, with explanatory comments in the rightmost column to describe the associated errors. For each question (Q), we highlight the correct parts of the answer (A) in green and the errors in red. Similarly, in the pr…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence

    cs.CV 2026-03 reject novelty 5.0 of 10

    MEDIC-AD adds anomaly-aware and difference tokens to a medical VLM, claiming SOTA lesion detection, temporal tracking, and visual grounding; the zero-shot claim is undermined by likely train/test overlap.

Reference graph

Works this paper leans on

48 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [4]

    Pretraining Vision- Language Model for Difference Visual Question An- swering in Longitudinal Chest X-rays

    Yeongjae Cho, Taehee Kim, Heejun Shin, Sung- zoon Cho, and Dongmyung Shin. “Pretraining Vision- Language Model for Difference Visual Question An- swering in Longitudinal Chest X-rays”. In: Medical Imaging with Deep Learning . 2024

    work page 2024

  2. [1]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments”. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion. 2005, pp. 65–72

    work page 2005

  3. [2]

    Current status of burnout in Canadian radiology

    Daniel J Cao, Casey Hurrell, and Michael N Patlas. “Current status of burnout in Canadian radiology”. In: Canadian Association of Radiologists Journal 74.1 (2023), pp. 37–43

    work page 2023

  4. [3]

    A national review of radiol- ogy reporting within the NHS in England

    Care Quality Commission. “A national review of radiol- ogy reporting within the NHS in England”. In: (2018), pp. 1–26

    work page 2018

  5. [5]

    Control- lable Chest X-Ray Report Generation from Longitudi- nal Representations

    Francesco Dalla Serra, Chaoyang Wang, Fani Deli- gianni, Jeff Dalton, and Alison O’Neil. “Control- lable Chest X-Ray Report Generation from Longitudi- nal Representations”. In: Findings of the Association for Computational Linguistics: EMNLP 2023 . Ed. by Houda Bouamor, Juan Pino, and Kalika Bali. Singapore: Association for Computational Linguistics, Dec....

    work page 2023

  6. [6]

    Finding- Aware Anatomical Tokens for Chest X-Ray Automated Reporting

    Francesco Dalla Serra, Chaoyang Wang, Fani Deli- gianni, Jeffrey Dalton, and Alison Q. O’Neil. “Finding- Aware Anatomical Tokens for Chest X-Ray Automated Reporting”. In: Machine Learning in Medical Imaging . Ed. by Xiaohuan Cao, Xuanang Xu, Islem Rekik, Zhiming Cui, and Xi Ouyang. Cham: Springer Nature Switzerland, 2024, pp. 413–423

    work page 2024

  7. [7]

    Overview of the RadSum23 Shared Task on Multi-modal and Multi-anatomical Ra- diology Report Summarization

    Jean-Benoit Delbrouck, Maya Varma, Pierre Cham- bon, and Curtis Langlotz. “Overview of the RadSum23 Shared Task on Multi-modal and Multi-anatomical Ra- diology Report Summarization”. In: The 22nd Work- shop on Biomedical Natural Language Processing and BioNLP Shared Tasks . Ed. by Dina Demner-fushman, Sophia Ananiadou, and Kevin Cohen. Toronto, Canada: As...

    work page 2023

  8. [8]

    Multiple meta-model quantifying for medical visual question answering

    Tuong Do, Binh X Nguyen, Erman Tjiputra, Minh Tran, Quang D Tran, and Anh Nguyen. “Multiple meta-model quantifying for medical visual question answering”. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Con- ference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24 . Springer. 2021, pp. 64– 74

    work page 2021

Show all 48 references
  1. [9]

    PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals

    Ary L Goldberger, Luis AN Amaral, Leon Glass, Jef- frey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. “PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic...

    2000

  2. [10]

    PathVQA: 30000+ Questions for Medical Visual Question Answering

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. PathVQA: 30000+ Questions for Medical Visual Question Answering . 2020

    2020

  3. [11]

    Medical-Diff-VQA: A Large-Scale Medical Dataset for Difference Visual Question An- swering on Chest X-Ray Images

    Xinyue Hu et al. “Medical-Diff-VQA: A Large-Scale Medical Dataset for Difference Visual Question An- swering on Chest X-Ray Images”. In: ()

  4. [12]

    Medical-Diff-VQA: A Large-Scale Medical Dataset for Difference Visual Question Answering on Chest X-Ray Images

    Xinyue Hu, L Gu, Q An, M Zhang, L Liu, K Kobayashi, T Harada, R Summers, and Y Zhu. Medical-Diff-VQA: A Large-Scale Medical Dataset for Difference Visual Question Answering on Chest X-Ray Images . 2023. 10

    2023

  5. [13]

    Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering

    Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M Summers, and Yingying Zhu. “Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering”. In: Proceedings o...

    2023

  6. [14]

    OVQA: A clinically generated visual question answering dataset

    Yefan Huang, Xiaoli Wang, Feiyan Liu, and Guofeng Huang. “OVQA: A clinically generated visual question answering dataset”. In: Proceedings of the 45th Interna- tional ACM SIGIR Conference on Research and Devel- opment in Information Retrieval . 2022, pp. 2924–2938

    2022

  7. [15]

    MIMIC- CXR, a de-identified publicly available database of chest radiographs with free-text reports

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Roger G Mark, and Steven Horng. “MIMIC- CXR, a de-identified publicly available database of chest radiographs with free-text reports”. In: Scientific Data 6.1 (2019)

    2019

  8. [16]

    MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs

    Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yi- fan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. “MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs”. In: arXiv preprint arXiv...

    2019 arXiv

  9. [17]

    MMBERT: Mul- timodal BERT Pretraining for Improved Medical VQA

    Yash Khare, Viraj Bagal, Minesh Mathew, Adithi Devi, U Deva Priyakumar, and CV Jawahar. MMBERT: Mul- timodal BERT Pretraining for Improved Medical VQA . 2021

    2021

  10. [18]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. “Adam: A method for stochastic optimization”. In: arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  11. [19]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. “Large language models are zero-shot reasoners”. In: Advances in neural information processing systems 35 (2022), pp. 22199– 22213

    2022

  12. [20]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. “A dataset of clinically generated visual questions and answers about radiology images”. In: Scientific Data 5.1 (Nov. 2018)

    2018

  13. [21]

    Rouge: A package for automatic eval- uation of summaries

    Chin-Yew Lin. “Rouge: A package for automatic eval- uation of summaries”. In: Text summarization branches out. 2004, pp. 74–81

    2004

  14. [22]

    Medical visual question answering: A survey

    Zhihong Lin, Donghao Zhang, Qingyi Tao, Danli Shi, Gholamreza Haffari, Qi Wu, Mingguang He, and Zongyuan Ge. “Medical visual question answering: A survey”. In: Artificial Intelligence in Medicine 143 (2023), p. 102611

    2023

  15. [23]

    SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Ques- tion Answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Ques- tion Answering. 2021

    2021

  16. [24]

    ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing

    Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. “ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing”. In: BioNLP Workshop and Shared Task. 2019, pp. 319–327

    2019

  17. [25]

    Over- coming data limitation in medical visual question an- swering

    Binh D Nguyen, Thanh-Toan Do, Binh X Nguyen, Tuong Do, Erman Tjiputra, and Quang D Tran. “Over- coming data limitation in medical visual question an- swering”. In: Medical Image Computing and Com- puter Assisted Intervention–MICCAI 2019: 22nd In- ternational Conference, Shenzh...

    2019

  18. [26]

    GPT-4 Technical Report

    OpenAI. “GPT-4 Technical Report”. In: ArXiv abs/2303.08774 (2023)

    2023 arXiv

  19. [27]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. “Bleu: a method for automatic evaluation of machine translation”. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002, pp. 311–318

    2002

  20. [28]

    Describing and Localizing Multiple Changes With Transformers

    Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, and Yutaka Satoh. “Describing and Localizing Multiple Changes With Transformers”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . Oct. 2021, pp. 1971–1980

    2021

  21. [29]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. “Language models are unsupervised multitask learners”. In: OpenAI blog 1.8 (2019), p. 9

    2019

  22. [30]

    CGMVQA: A New Classification and Generative Model for Medical Visual Question An- swering

    F. Ren and Y . Zhou. “CGMVQA: A New Classification and Generative Model for Medical Visual Question An- swering”. In: IEEE Access 8 (2020), pp. 50626–50636

    2020

  23. [31]

    Faster R-CNN: Towards real-time object detec- tion with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. “Faster R-CNN: Towards real-time object detec- tion with region proposal networks”. In: Advances in neural information processing systems 28 (2015)

    2015

  24. [32]

    Radiologist shortage leaves patient care at risk, warns royal college

    Abi Rimmer. “Radiologist shortage leaves patient care at risk, warns royal college”. In: BMJ: British Medical Journal (Online) 359 (2017)

    2017

  25. [33]

    MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain

    Dhruv Sharma, Sanjay Purushotham, and Chandan K Reddy. “MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain”. In: Scientific Reports 11.1 (2021), p. 19826

    2021

  26. [34]

    Rising use of diagnostic medical imaging in a large integrated health system

    Rebecca Smith-Bindman, Diana L Miglioretti, and Eric B Larson. “Rising use of diagnostic medical imaging in a large integrated health system”. In: Health affairs 27.6 (2008), pp. 1491–1502

    2008

  27. [35]

    LLaMA: Open and Effi- cient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Bap- tiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. “LLaMA: Open and Effi- cient Foundation L...

    2023 arXiv

  28. [36]

    Open-ended medical visual question answering through prefix tuning of language models

    Tom Van Sonsbeek, Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees GM Snoek, and Marcel Wor- ring. “Open-ended medical visual question answering through prefix tuning of language models”. In: In- ternational Conference on Medical Image Computing and Computer-Assisted Interv...

    2023

  29. [37]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is all you need”. In: Advances in neural information processing systems 30 (2017)

    2017

  30. [38]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. “Cider: Consensus-based image description evaluation”. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2015, pp. 4566–4575

    2015

  31. [39]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. “Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework”. In: International Con- ference on Machine Le...

    2022

  32. [40]

    Chatcad: Interactive computer- aided diagnosis on medical image using large language models

    Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. “Chatcad: Interactive computer- aided diagnosis on medical image using large language models”. In: arXiv preprint arXiv:2302.07257 (2023)

    2023 arXiv

  33. [41]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. “Self-Consistency Improves Chain of Thought Reasoning in Language Models”. In: The Eleventh International Conference on Learning Repre- sentations. 2023

    2023

  34. [42]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. “Chain-of-thought prompting elicits reasoning in large language models”. In: Advances in neural information processing systems 35 (2022), pp. 24824–24837

    2022

  35. [43]

    Chest ImaGenome Dataset for Clinical Reasoning

    Joy T Wu, Nkechinyere Nneka Agu, Ismini Lourentzou, Arjun Sharma, Joseph Alexander Paguio, Jasper Seth Yao, Edward Christopher Dee, William G Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. “Chest ImaGenome Dataset for Clinical Reasoning”. In: NeurIPS: Datasets and Ben...

    2021

  36. [44]

    Image dif- ference captioning with pre-training and contrastive learning

    Linli Yao, Weiying Wang, and Qin Jin. “Image dif- ference captioning with pre-training and contrastive learning”. In: Proceedings of the AAAI Conference on Artificial Intelligence. V ol. 36. 3. 2022, pp. 3108–3116

    2022

  37. [45]

    Hetero- geneity and predictors of the effects of AI assistance on radiologists

    Feiyang Yu, Alex Moehring, Oishi Banerjee, Tobias Salz, Nikhil Agarwal, and Pranav Rajpurkar. “Hetero- geneity and predictors of the effects of AI assistance on radiologists”. In: Nature Medicine 30.3 (2024), pp. 837– 849

    2024

  38. [46]

    Pmc- vqa: Visual instruction tuning for medical visual ques- tion answering

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. “Pmc- vqa: Visual instruction tuning for medical visual ques- tion answering”. In: arXiv preprint arXiv:2305.10415 (2023)

    2023 arXiv

  39. [47]

    Multimodal Chain- of-Thought Reasoning in Language Models

    Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. “Multimodal Chain- of-Thought Reasoning in Language Models”. In: Trans- actions on Machine Learning Research (2024)

    2024

  40. [48]

    Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

    Denny Zhou, Nathanael Sch ¨arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”. In: The Eleventh Interna- tional Conference on...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.