Pith. sign in

REVIEW 4 major objections 4 minor 48 references

Adapting Lightweight Vision Language Models for Radiological Visual Question Answering

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a compact 3B vision-language model, fine-tuned with curated synthetic data, can be competitive with much larger models on radiological visual question answering.

desk verdict The claimed 'competitive performance' is contradicted by the paper's own Table 1, and the appendix numbers don't line up—useful recipe and data, but not ready for peer review. read the letter →

arxiv 2506.14451 v1 pith:HXTWSBGK submitted 2025-06-17 cs.CV cs.AI

Aditya Shourya , Michel Dumontier , Chang Sun This is my paper
classification cs.CVcs.AI
keywords radiologicalvisualquestionansweringvision-languagemodelsparameter-efficientfine-tuningLoRAPaliGemmasyntheticQAgenerationsaliencyanalysismedicalVQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a 3-billion-parameter vision-language model, when fine-tuned with a deliberately cheap recipe, can be competitive on radiological visual question answering even though state-of-the-art medical VLMs are two to three times larger. The recipe is concrete: use LLaMA-8B to synthesize question-answer pairs from image captions in ROCO and MedPix, anneal the corpus toward high-quality MedPix cases, fine-tune the projection head on SLAKE first, then apply LoRA to the whole model. The same work introduces a saliency diagnostic that highlights which image patches support a generated answer, so clinicians can audit failure modes. If the claim holds, it matters because it would lower the compute and data bar for building specialty medical QA systems without sacrificing much accuracy. In the paper's own tables the 3B model trails LLaVA-Med by 15 to 25 accuracy points on the shared benchmarks, so the intended claim is competitiveness relative to size, training cost, and data, not parity.

What carries the argument

The load-bearing object is PaliGemma-mix-448, a 3B VLM that pairs a 400M-parameter SigLIP vision tower with a 2B Gemma language model through a single linear projection. The paper's training machinery has four parts: synthetic QA generation with LLaMA-8B from ROCO and MedPix captions; annealing, which repeatedly mixes in the smaller, higher-quality MedPix set to enrich the larger corpus; a first fine-tuning stage that trains only the projection head on SLAKE to align visual features with anatomical vocabulary; and a second stage that applies LoRA low-rank adapters to the vision and language attention heads. For evaluation, the machinery is a GPT-4 judge for open-ended answers plus raw and rollout attention maps used as a saliency diagnostic. The two-stage schedule and annealing together are what carry the claim that small models can be specialized cheaply.

What would settle it

Check the released ROCO-QA-Train and ROCO-QA test splits for overlapping images or captions; if a meaningful fraction of test items also appeared in the generated training set, re-score the model on a non-overlapping split and compare it again with LLaVA-Med under identical conditions. A large accuracy drop after removing overlap would show the reported gains came from memorization rather than radiological reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that model scale is not the only route to radiological reasoning: a 3B PaliGemma-mix-448 VLM, adapted through a two-stage pipeline of projection-head alignment on SLAKE followed by LoRA fine-tuning on ROCO, MedPix, and PMC-VQA, achieves competitive accuracy on both open- and closed-ended radiology questions. The authors present this as a reassessment of scaling trends in medical VQA, arguing that small models with curated synthetic data can approach the behavior of much larger systems. They support the claim with accuracy comparisons against LLaVA-Med across four datasets, with the compact model reaching 79% on SLAKE closed-ended QA and 33-41% on the other benchmarks while LLaVA-Med scores 56-58% there. The paper also claims that its attention-rollout saliency tool can reveal whether the model is keying on clinically relevant image regions, framing interpretability as part of the evaluation, not an afterthought.

Load-bearing premise

The evaluation data are assumed to be independent of the data used to create the training question-answer pairs, but the paper does not describe train/test splits and the QA pairs were generated from the same caption corpora that later appear in evaluation.

Editorial extensions

If this is right

  • If the central claim is right, a single NVIDIA H100 can produce a radiology VQA model in roughly three days, which makes model development feasible for teams without large GPU clusters.
  • Synthetic question-answer pairs generated from captions, rather than expert-written questions, can provide enough supervision for domain adaptation, easing the data bottleneck.
  • Two-stage fine-tuning with a frozen projection-head stage followed by LoRA gives faster convergence than single-stage tuning, so small models can be trained with fewer epochs.
  • Attention-rollout saliency can be used to spot answers that are generated from irrelevant image regions, supplying a lightweight auditing step before a model is trusted on new cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an editorial check: because the QA pairs used for training were generated from the same ROCO and MedPix caption corpora used in evaluation, and the paper does not describe train/test splits, the reported accuracies could be inflated if any generated training pair overlaps a test sample; a held-out evaluation would settle this.
  • The open-ended scores also depend on GPT-4 as the judge, so a judge that rewards fluent or verbose wording could raise accuracy beyond what strict factual matching would give; scoring the same generations on a factual entailment rubric would be a useful comparison.
  • The recipe is largely model-agnostic, so the same data-generation, annealing, and two-stage fine-tuning pattern could be tested on other small VLMs or other imaging domains such as pathology, dermatology, or retinal imaging.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a pipeline for adapting a 3B-parameter PaliGemma vision-language model to radiological visual question answering (VQA). The pipeline comprises synthetic QA generation from radiological captions (using LLaMA-8B), dataset annealing with MedPix, and two-stage fine-tuning (projection-head training on SLAKE followed by LoRA fine-tuning on ROCO, MedPix, and PMC-VQA). The authors report accuracy numbers on SLAKE, PMC-VQA, ROCO, and an annealed ROCO+MedPix set, and claim that this compact model achieves competitive performance approaching LLaVA-Med. They also introduce a saliency-based diagnostic tool for cross-modal interpretability and perform manual verification on organ-level pathologies.

Significance. If the reported results were reliable, the paper would make a modest but useful contribution: it demonstrates a compute-efficient recipe for specializing a small VLM to a medical domain and releases code and synthetic QA data. The proposed saliency tool is a practical addition, even if it is not deeply validated. However, the significance is substantially undercut by severe internal inconsistencies between the main tables and the appendix, an incompletely specified evaluation protocol, and an unvalidated GPT-4 judge. The headline claim of 'competitive performance' is not supported by the paper's own numbers, which show 15–25 percentage-point gaps against LLaVA-Med on the primary benchmarks.

major comments (4)
  1. [Table 1 vs. Appendix Tables 4 and 5] The accuracy numbers are inconsistent between the main text and the appendix for the same configurations. For example, ROCO+MedPix with Stage 1 fine-tuning is reported as 41.48±3.90 in Table 1 but only 74/200 = 37.0% in Table 5; SLAKE with Stage 1 is 79.00±2.75 in Table 1 but 0.26 (26%) in Table 5; and ROCO without Stage 1 is 32.25±3.60 in Table 1 but 63/200 = 31.5% in Table 4. These discrepancies are not minor rounding differences—they change the conclusions and make the reported results unreliable.
  2. [§5.3 and §4.2] The evaluation protocol does not specify how the test samples (reported as 'a sample of 200' in Table 1) are drawn, nor whether the evaluated instances are disjoint from the fine-tuning data. In §4.2, synthetic QA pairs are generated from ROCO and MedPix captions, the same corpora that are later used for fine-tuning (§5.1). Without explicit train/test splits, any overlap between the training data and the 200-item evaluation samples would inflate the reported accuracies. This issue must be clarified and resolved for the results to be interpretable.
  3. [§5.3 and Appendix Figure 10] The GPT-4 judge used for open-ended QA evaluation is not validated. The paper provides no evidence that GPT-4's binary factuality judgments agree with human expert assessments, and the supplied prompt in Figure 10 calls 'ollama.chat(model='gpt-4', ...)', which is not a valid OpenAI API interface and suggests the judge may not actually be GPT-4. The open-ended accuracy numbers (e.g., on ROCO and ROCO+MedPix) therefore cannot be trusted as measurements of factual correctness.
  4. [§1 and Table 1] The central claim that the 3B model 'can achieve competitive performance' and 'approaching the performance of much larger models like LLaVA-Med' is contradicted by the paper's own results. Table 1 shows gaps of 7.5 points on SLAKE, 25.3 points on PMC-VQA, 22.6 points on ROCO, and 15.2 points on ROCO+MedPix. These differences are several standard errors from the reported means, so the claim as stated is not supported by the data. The wording should be revised to reflect that the model is 'promising given its small size' rather than 'competitive' with LLaVA-Med.
minor comments (4)
  1. [§5.2, Eq. (1)] The scaling-law analysis around Equation (1) is presented without any fitted parameter values, goodness-of-fit measures, or statistical tests. The qualitative discussion of differing scaling exponents would be more convincing if the fitting procedure and uncertainties were reported.
  2. [Abstract and §1] The abstract contains a duplicated phrase 'evaluation evaluation efforts' and the introduction uses inconsistent spacing in 'LLaV A-Med'; these should be corrected.
  3. [§5.4] The manual verification in Table 2 is based on a single inference run per case, and the paper acknowledges that it was conducted without expert involvement. For a tool intended for expert use, some measure of inter-annotator reliability or a structured sampling procedure would strengthen the claim.
  4. [§3.1] The vision tower is described as having 'approximately 400M parameters' and later as 'pretrained with a sigmoid contrastive loss and comprising 400M parameters'; the repetition and inconsistent phrasing should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical fine-tuning study whose only formal model, Eq. 1, is quoted from prior literature and is not used to predict any reported result.

full rationale

The paper does not contain a derivation chain whose outputs reduce to its inputs. Table 1 reports measured fine-tuning accuracies against LLaVA-Med, and the only formal expression, Eq. (1), is presented as an empirical loss model from prior scaling-law work; the authors state that scaling exponents differ by template type, but they do not fit Eq. (1) and use it to produce any accuracy reported in the paper. The QA pairs are generated from ROCO and MedPix captions and evaluation is conducted on the resulting ROCO and ROCO+MedPix sets, which creates a legitimate train/test contamination concern, and the 15–25 point gaps in Table 1 contradict the paper's 'competitive' claim; however, these are evaluation validity and correctness issues rather than circular reasoning by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameters renamed as predictions, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claim depends on several unvalidated evaluation choices and domain assumptions. There are no invented physical or model entities.

free parameters (5)
  • grade penalty for non-robust predictions = 1 point
    In Section 5.3, predictions that differ across 3 of 5 stochastic generations are penalized by one accuracy point. The choice is ad hoc and can shift results.
  • evaluation sample size = 200
    All accuracy results are computed on a sample of 200 test instances per dataset, which limits statistical power and may not reflect the full distribution.
  • number of stochastic generations = 5
    Used in closed-ended evaluation to determine robustness; no justification for the threshold of 3 dissimilar answers.
  • learning rate = 1e-5
    Constant learning rate used in both fine-tuning stages; chosen without reported tuning.
  • stage-1 epochs = 5
    Projection head trained for 5 epochs on SLAKE; no ablation on epoch count.
assumptions (4)
  • domain assumption GPT-4 provides a valid 0/1 factual-correctness judgment for open-ended radiology answers.
    Used in Section 5.3; no validation against human radiologists or inter-rater agreement is reported.
  • ad hoc to paper Synthetic QA pairs generated by LLaMA-8B from image captions are clinically relevant and correctly formatted after manual filtering.
    The filtering in Section 4.3 is manual and subjective, and the generation prompts (Appendix A) are not shown to avoid hallucinations.
  • domain assumption PaliGemma-mix-448 is a representative off-the-shelf 3B VLM whose base capabilities are well understood.
    The choice is justified by training-data transparency, but no comparison with other 3B VLMs is made.
  • domain assumption ROCO v2.0 and MedPix v2.0 are suitable proxies for radiological VQA and their captions accurately describe images.
    These datasets are used for both QA generation and evaluation; any systematic caption errors propagate into the results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adapting Lightweight Vision Language Models for Radiological Visual Question Answering." pith.science (2026). https://pith.science/paper/HXTWSBGK

@misc{pith2026250614451,
  author       = {Pith},
  title        = {Pith review of: Adapting Lightweight Vision Language Models for Radiological Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXTWSBGK}},
  note         = {Machine review of arXiv:2506.14451}
}
read the original abstract

Recent advancements in vision-language systems have improved the accuracy of Radiological Visual Question Answering (VQA) Models. However, some challenges remain across each stage of model development: limited expert-labeled images hinders data procurement at scale; the intricate and nuanced patterns of radiological images make modeling inherently difficult; and the lack of evaluation evaluation efforts makes it difficult to identify cases where the model might be ill-conditioned. In this study, we fine-tune a lightweight 3B parameter vision-language model for Radiological VQA, demonstrating that small models, when appropriately tuned with curated data, can achieve robust performance across both open- and closed-ended questions. We propose a cost-effective training pipeline from synthetic question-answer pair generation to multi-stage fine-tuning on specialised radiological domain-targeted datasets (e.g., ROCO v2.0, MedPix v2.0). Our results show that despite operating at a fraction of the scale of state-of-the-art models such as LLaVA-Med, our model achieves promising performance given its small parameter size and the limited scale of training data. We introduce a lightweight saliency-based diagnostic tool that enables domain experts to inspect VQA model performance and identify ill-conditioned failure modes through saliency analysis.

Figures

Figures reproduced from arXiv: 2506.14451 by the authors.

Figure 1
Figure 1. Our and PaliGemma [9] Vision Language Model Architecture [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Training Recipe Overview 4 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distributions across four used datasets. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Filtering and curation pipeline. Medical VQA tasks demand not only visual understanding but also clinical reasoning, which general￾purpose VLMs often lack. To address this, we prioritize datasets where questions are grounded in patient context and, where possible, link…
Figure 5
Figure 5. Figure 5: Fine Tuning Evaluation Loss 5 Experiments and Evaluation 5.1 Experiment Setting All experiments including fine-tuning and evaluation were conducted using a single NVIDIA H100 GPU. With adequate allocation, ROCO and MedPix and Roco + Medpix datasets were fine-tuned in a…
Figure 6
Figure 6. Figure 6: LLM-based Evaluation Examples Closed-ended QA Evaluation: For multiple-choice question answering (MCQA) such as PMC-VQA [12], we measure model accuracy across five stochastic generations per test instance. Inspired by [15], we define a prediction as non-robust if the m…
Figure 7
Figure 7. Figure 7: An example illustrates saliency analysis with Raw Attention and Attention Rollout for a [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Generate Case Based Questions Prompt def generate_qapairs_lit(caption,case_topic): prompt = f""" Generate a relevant,short and insightful literature-based question for the given case topic. Treat the caption as ground truth: {caption}, with associated literature: {case…
Figure 9
Figure 9. Figure 9: Generate Literature Based Questions Prompt [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Evaluation prompt for GPT-4 as a judge. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: A Patient suffering from lesion on their right kidney [Notice High Rollout Saliency in the [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: An X-ray of a patient with Bone Fracture[Notice High Saliency on the fractured region] [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: A patient suffering from Leukemia [In such examples the authors of the study refrain from [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 21 canonical work pages

  1. [1]

    Ai agriculture: Boost yields with yolo11

    Haziqa Sajid. Ai agriculture: Boost yields with yolo11. https://www.ultralytics.com/ blog/the-changing-landscape-of-ai-in-agriculture , 2024. Accessed: 2025-04-30

    work page 2024

  2. [2]

    Video alarms for 24/7 security & monitoring

    Verkada Inc. Video alarms for 24/7 security & monitoring. https://www.verkada.com/ alarms/video-alarms/. Accessed: 2025-04-30

    work page 2025

  3. [3]

    Babydoctor

    Vir Chau Markus Zhang. Babydoctor. https://github.com/photomz/BabyDoctor, 2023. GitHub

    work page 2023

  4. [4]

    Tom van Sonsbeek, Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees G. M. Snoek, and Marcel Worring. Open-ended medical visual question answering through prefix tuning of language models, 2023. URL https://arxiv.org/abs/2303.05977

    arXiv 2023

  5. [5]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tomas Tu, et al. Large language models encode clinical knowledge. Nature, 620:172–180, 2023. doi: 10.1038/s41586-023-06291-2. URL https: //doi.org/10.1038/s41586-023-06291-2

    doi:10.1038/s41586-023-06291-2 2023

  6. [6]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day, 2023. URL https://arxiv.org/abs/2306.00890

    arXiv 2023

  7. [7]

    All you may need for VQA are image captions

    Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for VQA are image captions. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...

    doi:10.18653/v1/2022.naacl-main.142 2022

  8. [8]

    Pre-training image-language transform- ers for open-vocabulary tasks, 2022

    AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Pre-training image-language transform- ers for open-vocabulary tasks, 2022. URL https://arxiv.org/abs/2209.04372

    arXiv 2022

Show all 48 references
  1. [9]

    PaliGemma: A versatile 3B VLM for transfer

    Lucas Beyer*, Andreas Steiner*, André Susano Pinto*, Alexander Kolesnikov*, Xiao Wang*, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, N...

    2024 arXiv

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  3. [11]

    Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering, 2021

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering, 2021. URL https: //arxiv.org/abs/2102.09542

    2021 arXiv

  4. [12]

    Pmc-vqa dataset, 2023

    Xiaoman Zhang et al. Pmc-vqa dataset, 2023. URL https://huggingface.co/datasets/ xmcmic/PMC-VQA. Accessed: 2025-02-08

    2023

  5. [13]

    Friedrich

    Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M. Friedrich. Ra- diology objects in context (roco): A multimodal image dataset. Technical report, University of Applied Sciences and Arts Dortmund, TU Dortmund University, University of Duisburg- Essen, 2...

    2018

  6. [14]

    Medpix 2.0: A comprehensive multimodal biomedical dataset for advanced ai applications,

    Irene Siragusa, Salvatore Contino, Massimo La Ciura, Rosario Alicata, and Roberto Pirrone. Medpix 2.0: A comprehensive multimodal biomedical dataset for advanced ai applications,

  7. [15]

    Pmc-vqa: Visual instruction tuning for medical visual question answering, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering, 2024. URL https://arxiv.org/abs/2305.10415

    2024 arXiv

  8. [16]

    ChatGPT: A Large Language Model

    OpenAI. ChatGPT: A Large Language Model. Online, 2024. Available at: https://openai. com/chatgpt [Accessed YEAR MONTH DAY]

    2024

  9. [17]

    Healthsearchqa

    katielink. Healthsearchqa. https://huggingface.co/datasets/katielink/ healthsearchqa, 2023. Accessed: 2025-05-01

    2023

  10. [18]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent...

    2022 arXiv

  11. [19]

    PubMed Central (PMC), 2024

    National Library of Medicine. PubMed Central (PMC), 2024. URL https://www.ncbi.nlm. nih.gov/pmc/. Accessed: 2024-02-08

    2024

  12. [20]

    Nvidia a100 tensor core gpu, 2020

    NVIDIA Corporation. Nvidia a100 tensor core gpu, 2020. URL https://www.nvidia.com/ en-us/data-center/a100/. Accessed: 2025-03-30

    2020

  13. [21]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  14. [22]

    Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahmoud Khademi, Lev Kurilenko, James R

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishr...

    2024

  15. [23]

    Show and tell: A neural image caption generator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. InProceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2015

    2015

  16. [24]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014

    2014

  17. [25]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computati...

    2018 doi

  18. [26]

    Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning

    Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Re...

    2021

  19. [27]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023. URL https://arxiv.org/abs/2303.15343

    2023 arXiv

  20. [28]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, 2018

    Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, 2018. URL https://arxiv.org/abs/ 1808.06226

    2018 arXiv

  21. [29]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  22. [30]

    Arnab Kumar Mondal, Arnab Bhattacharjee, Parag Singla, and A. P. Prathosh. xvitcos: Ex- plainable vision transformer based covid-19 screening using radiography. IEEE Journal of Translational Engineering in Health and Medicine, 10:1–10, 2022. ISSN 2168-2372. doi: 10.1109/JTEHM....

    2022

  23. [31]

    Lvlm-interpret: An interpretability tool for large vision-language models, 2024

    Gabriela Ben Melech Stan, Estelle Aflalo, Raanan Yehezkel Rohekar, Anahita Bhiwandiwalla, Shao-Yen Tseng, Matthew Lyle Olson, Yaniv Gurwicz, Chenfei Wu, Nan Duan, and Vasudev Lal. Lvlm-interpret: An interpretability tool for large vision-language models, 2024. URL https://arxi...

    2024 arXiv

  24. [32]

    Searching for unintended biases with saliency

    Astrid Bertrand, Adam Pearce, and Nithum Thain. Searching for unintended biases with saliency. PAIR Explorables, 2022. https://pair.withgoogle.com/explorables/saliency/. 13

    2022

  25. [33]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv. org/abs/1706.03762

    2023 arXiv

  26. [34]

    Transformer interpretability beyond attention visualization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 782–791, June 2021. doi: 10.1109/CVPR46437.2021.00084

    2021

  27. [35]

    Explainability for vision transformers

    Jacob Gildenblat. Explainability for vision transformers. https://github.com/jacobgil/ vit-explain, 2021. Accessed: 2025-02-03

    2021

  28. [36]

    MedPix: Free Online Medical Image Database

    National Library of Medicine. MedPix: Free Online Medical Image Database. Online, 2024. Available at: https://medpix.nlm.nih.gov/home [Accessed YEAR MONTH DAY]

    2024

  29. [37]

    Lin et al

    W. Lin et al. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In H. Greenspan, editor,Medical Image Computing and Computer Assisted Intervention, pages 525–536. Springer, 2023

    2023

  30. [38]

    Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions, 2023

    Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions, 2023. URL https://arxiv.org/abs/2303.06594

    2023 arXiv

  31. [39]

    Language models are few-shot learners

    Tom B Brown, Ben Mann, Nick Ryder, Melvin Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Nicholas Shinn, Donald Ziegler, Jeffrey Wu, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  32. [40]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  33. [41]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  34. [42]

    Curriculum learning for data-efficient vision-language alignment, 2022

    Tejas Srinivasan, Xiang Ren, and Jesse Thomason. Curriculum learning for data-efficient vision-language alignment, 2022. URL https://arxiv.org/abs/2207.14525

    2022 arXiv

  35. [43]

    PMC Open Access Subset

    National Library of Medicine. PMC Open Access Subset. Online, 2003. Bethesda (MD): National Library of Medicine. [cited YEAR MONTH DAY]. Available from:https://pmc. ncbi.nlm.nih.gov/tools/openftlist/

    2003

  36. [44]

    When scaling meets llm finetuning: The effect of data, model and finetuning method

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. InProceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id= 5HCnKDeTws

    2024

  37. [45]

    Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs, 2019. URL https://ar...

    2019 arXiv

  38. [46]

    Bleu: A method for auto- matic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics, ...

    2002

  39. [47]

    "" Figure 9: Generate Literature Based Questions Prompt B Evaluation and Saliency Diagnostics def evaluate_generation(generation,ground): prompt = f

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/W04-1013. 14 Appendix A Generating Question Answe...

    2004

  40. [2024]

    URL https://arxiv.org/abs/2407.02994. 11

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.