On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI

David Restrepo; Enzo Ferrante; Ira Ktena; Maria Vakalopoulou; Stergios Christodoulidis

REVIEW 4 major objections 3 minor 27 references

On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Medical AI models lean on text, not images, when diagnosing, this paper shows.

desk verdict Useful diagnostic protocol, but the headline claim about VLMs 'overlooking visual cues' is undercut by a text-as-answer confound that needs a stratified re-analysis. read the letter →

arxiv 2508.00171 v1 pith:PO4MVVMV submitted 2025-07-31 cs.CV cs.CL

David Restrepo , Ira Ktena , Maria Vakalopoulou , Stergios Christodoulidis , Enzo Ferrante This is my paper · ORCID

classification cs.CVcs.CL

keywords vision-languagemodelsmodalitybiastextualcalibrationmedicalimagingselectiveshiftingchestX-rayglaucomadetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that medical vision-language models, when asked to read an X-ray or eye scan together with a clinical note, lean on the text so heavily that contradictory images barely matter. The authors introduce Selective Modality Shifting (SMS), a perturbation method that swaps either the image or the text with one from an opposite-label sample to measure which modality drives the prediction. Across six open VLMs and two medical datasets, text swaps flip many correct answers to wrong ones, sometimes driving accuracy below random, while image swaps cause only small changes. The authors argue this reveals a fundamental over-reliance on textual priors, and that miscalibration accompanies these shortcut predictions. If true, this matters because such models could give confident, wrong clinical answers when a report and image disagree.

What carries the argument

Selective Modality Shifting (SMS) is the central mechanism: take a binary classification sample (I, T, y), build two counterfactual samples by replacing only the text (I, T′, y) or only the image (I′, T, y) with a component drawn from an opposite-label sample, and compare predictions, accuracy, Negative Flip Rate, and calibration across the three conditions. The attention analysis decomposes generated-token attention into image and text token components to show that text attention changes as decoding proceeds while image attention stays flat. The calibration evaluation extracts the softmax probability over the 'yes' and 'no' first tokens and computes Expected Calibration Error under each shift.

What would settle it

Retest the same six models on the same two datasets with reports masked to remove any phrase that states or clearly implies the diagnosis (for example, replacing 'Small right pneumothorax is new' with 'The patient is status post right thoracentesis'); if text-swap flips largely disappear, the reported text bias is an artifact of label leakage in the text rather than a genuine disregard for images.

Watch

Extended reading notes

Core claim

The paper's central claim is that current medical VLMs exhibit a marked dependency on the text modality, which persists even when visual information contradicts it. Using SMS, the authors show that swapping the text of a positive sample with text from a negative sample often flips the model's prediction, while swapping the image does not; on MIMIC-CXR, models like Qwen-2 VL and LLaVA 1.5 drop by over 20 points and show Negative Flip Rates above 0.60 under text shift. The same pattern appears, in milder form, in medically fine-tuned models such as Med-LLaVA and Med-Gemma. The authors further report that the misplaced confidence is not merely inaccurate but poorly calibrated: under text shift, calibration curves invert for some models, meaning high-confidence errors coincide with modality-conflicting inputs.

Load-bearing premise

The experimental comparison assumes the clinical report text is a neutral information channel, but in these datasets the report often states the diagnosis outright, so text-swap results may reflect the model simply following a clear textual answer rather than ignoring the image.

Editorial extensions

If this is right

On MIMIC-CXR, text shifting makes several models perform worse than random guessing, with Negative Flip Rates above 0.60 for Qwen-2 VL and LLaVA 1.5.
Calibration degrades under text shift, with some models showing inverted calibration curves, implying that overconfident errors are a secondary failure mode.
Attention analysis shows image token attention remains stable across output tokens while text token attention varies, consistent with text dominating generation.
The findings extend to both generalist models (LLaVA 1.5, Qwen-2 VL, Llama 3.2, Janus-Pro) and medical fine-tuned models (Med-LLaVA, Med-Gemma), so the problem is not cured by domain fine-tuning.
Performance with text alone often remains functional, while image-alone performance degrades severely, further confirming minimal reliance on visual features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the radiology reports in MIMIC-CXR contain the same conclusion that the label encodes, the SMS text-swap result conflates 'the model follows the text' with 'the model ignores the image'; masking diagnostic phrases would separate the two causes.
The same perturbation logic could be turned into a training signal: penalizing prediction deltas under text swaps, or rewarding invariance to text swaps, might reduce the shortcut reliance the paper diagnoses.
The first-token calibration proxy assumes the probabilities of 'yes' and 'no' capture the model's confidence in its final free-form answer; using full-sequence logprobs would give an independent check.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

David,

Quick read of Restrepo et al. (2508.00171). The SMS protocol is worth knowing: systematically swap text or image between opposing-label pairs, then measure accuracy, NFR, and ECE across six VLMs on MIMIC-CXR and FairVLMed. The combination of modality-shift sensitivity with calibration is a genuinely useful addition, and the benchmarking of generalist vs. medical fine-tuned models is solid. Methods are transparent, code is public, and the attention inspection, while qualitative, adds some mechanism.

The soft spot is the one the reader flagged: the text inputs often contain the diagnosis. In MIMIC-CXR, the report is the full final radiology text, and the prompt in Figure 4 literally says "Small right pneumothorax is new" and "Opacification is probably atelectasis." When you swap that text with a report from the opposite class, any model that reads the report will flip—regardless of whether it looked at the image. So the high NFR under text shift is largely measuring "the model follows the report," not "the model ignores the image." The authors acknowledge reports are not always conclusive, but they never stratify on whether the text alone determines the label. That step is needed before saying these models "disregard critical visual cues."

FairVLMed is less obviously confounded, so the claimed text bias may still hold there, but the same issue could partly apply. The absence of confidence intervals or significance tests also makes the model rankings shakier than the text suggests.

The central conclusion—that these models lean heavily on text—probably survives a re-analysis, but the stronger claim about "overlooking visual cues" is not yet established. Limiting the claim to "sensitivity to textual descriptions" and stratifying by text conclusiveness would make this a clean, useful paper. As is, it is a good diagnostic toolkit with an overinterpreted headline.

Recommendation: yes, send it to peer review, but with a required revision that separates conclusive from inconclusive reports and adds error bars. I'd bring it to reading group; I probably would not cite the headline result, but I might cite the SMS protocol if it gets refined.

Best, [Your name]

Referee Report

4 major / 3 minor

Summary. The paper introduces Selective Modality Shifting (SMS), a perturbation-based method to quantify how much medical vision-language models (VLMs) rely on text versus image inputs in binary classification. The authors swap the text or the image of a sample with a component from a sample of the opposite label and measure the change in accuracy, F1, negative flip rate, and calibration. They apply SMS to six open VLMs (four generalist and two medically fine-tuned) on two datasets: MIMIC-CXR (chest X-ray with full radiology reports) and FairVLMed (scanning laser ophthalmoscopy with clinical notes). They report that text swaps cause large performance drops and miscalibration, while image swaps cause little change, concluding that the models are strongly biased toward text and often ignore visual evidence. They also provide a qualitative attention analysis on one example. The central claim is that these VLMs exhibit a marked dependency on text input that persists even when visual cues contradict it.

Significance. If the central claim holds, the findings are practically important: medical VLMs that ignore images in favor of text could make unsafe predictions when the text is misleading, particularly in zero-shot deployment. The proposed SMS is a transparent, perturbation-based evaluation protocol that is easy to reproduce, and the authors commit to releasing code. The paper covers a useful model zoo and two different imaging modalities, and the inclusion of calibration analysis adds a dimension often missing in modality-bias studies. However, the strength of the claim is currently weakened by a design confound (the text often contains the ground-truth diagnosis), so the significance is conditional on whether the analysis can be re-run on subsets where the text does not determine the label.

major comments (4)

[Section 3.1 and Figure 4] The central claim that models 'overlook critical visual cues' under text shift is confounded by the fact that the text input is the full final radiology report, which often explicitly states the diagnosis (e.g., 'Small right pneumothorax is new' in Figure 4). For a binary 'any condition' task, swapping in a report from the opposite class injects an explicit counter-signal that would flip any model that can read, regardless of whether it uses the image. The aggregate NFR and accuracy drops under Text Shift therefore conflate 'the model follows the report' with 'the model is biased against images.' The authors acknowledge in Section 3.1 that reports are not always conclusive and may require image analysis, but they do not stratify results on this property. To support the claim of text bias 'even when visual cues contradict it,' the paper must report metrics on the subset of samples where the text alone does not determine the label (e.g., where the report is inconclusive or where the image and report disagree). Without such stratification, the main conclusion is not established.
[Section 4 and Figure 2] The 'Only Text' and 'Only Image' ablations are also affected by the text-as-ground-truth confound. High accuracy with text alone and low accuracy with image alone is exactly what one would expect if the text explicitly contains the answer; it does not demonstrate that the model preferentially exploits text in the multimodal setting. A more informative ablation would compare text-only performance on reports that state the diagnosis versus reports that are inconclusive, and image-only performance on the same samples. The current comparison does not separate the model's ability to read the report from its tendency to discount visual information.
[Section 2.2] The interpretation of image-swap insensitivity as 'the model may be underutilizing image information' is under-specified. If the text already carries the label, then a model that correctly uses both modalities would still be insensitive to an image swap, because the text still points to the same answer. Conversely, if the swapped image comes from the opposite class but the text is diagnostic, a text-following model would also be insensitive to the image swap. Thus, the image-swap results in Figure 2 cannot by themselves distinguish 'the model ignores images' from 'the text is sufficient for the task.' The method needs an additional control, such as measuring image-shift sensitivity on samples where the report is inconclusive.
[Figures 2 and 3] No confidence intervals, error bars, or statistical tests are reported for any of the accuracy, NFR, or ECE differences. Given the large number of models and conditions, some differences may be within sampling noise. This is especially important because the confound described above biases the aggregate comparisons; providing bootstrap confidence intervals or a paired test over samples would help the reader assess the robustness of the reported gaps (e.g., the Qwen-2 VL NFR of 0.75 under text shift).

minor comments (3)

[Section 4] The attention analysis is presented on a single exemplar (Figure 4) and the text claims this 'reflects the tendency observed in our experiments.' Since the method is quantitative (Section 2.3), the paper should report the average attention split between image and text tokens across a larger set of samples, with a variance measure, instead of one illustrative case.
[Figure 1] The caption of Figure 1 is repetitive and the figure itself mixes three different panels with inconsistent labels (e.g., 'VLM' appears multiple times, and part (b) is referenced as 'Output after Selective Modality Shifting' but combines text swap and calibration). Please redraw for clarity.
[Section 3.2] The sentence 'The first four are primarily trained on natural-image tasks' is followed by 'whereas the latter two incorporate clinical/biomedical training data.' It would help to clarify that Med-LLaVA and MedGemma are the two medical models, and to specify versions (e.g., Med-LLaVA version) since the field evolves quickly.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: SMS is a transparent, parameter-free perturbation measurement; the two self-citations ([17], [18]) are background-only and non-load-bearing. The text-contains-diagnosis confound (acknowledged in Sec 3.1) weakens the 'contradict' framing but is a validity issue, not a circular derivation.

full rationale

The central claim (marked text dependency) is measured, not derived: SMS defines text reliance as the change in a zero-shot VLM's binary output when the paired report is swapped with one from the opposite class (Sec 2.2), and the reported quantities (accuracy, F1, NFR in Fig. 2, ECE in Fig. 3) are empirical measurements under that definition. No parameter is fitted to a subset of the data and then reported as a prediction, and no result is obtained from a quantity it is supposed to predict. The two self-citations, [17] and [18] (Restrepo et al.), appear only in the introduction as background on LLM limitations in ophthalmology; neither is load-bearing for SMS, the datasets, or the headline conclusions, so they do not raise the circularity score beyond the null band. The only mildly self-definitional element is the interpretive frame: 'reliance' is operationalized as swap-sensitivity (Sec 2.2: 'We hypothesize that extreme sensitivity to either text or image swapping implies over-reliance on that modality'), so the later statement that high NFR values 'confirm that models heavily depend on textual input' (Sec 4) partly restates the metric's definition. Because the NFR magnitudes are empirical and could have been low, the central measurement retains independent content, and the operationalization is stated transparently rather than hidden. The more serious concern is a confound, not circularity: MIMIC-CXR text is the full final report, which, as the Figure 4 prompt shows, explicitly states findings such as 'Small right pneumothorax is new following right thoracentesis,' so a text swap injects a label-carrying counter-signal; the authors acknowledge that 'medical records ... do not always provide a clear diagnosis' (Sec 3.1) but do not stratify by report conclusiveness, leaving the strong claim that text bias persists 'even when visual cues contradict it' under-supported. That is a validity limitation of the perturbation test, not a circular reduction. The calibration and attention analyses are separate, parameter-free measurements that do not reduce to the SMS definition. The paper is self-contained against external benchmarks (MIMIC-CXR, FairVLMed, prior SOTA as reference [24]).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about how to interpret perturbation outcomes and attention. No free parameters are fitted, and no new entities are introduced. The most fragile assumption is that the text report acts as an ordinary modality rather than a direct source of the label.

assumptions (4)

domain assumption First-token logits for 'yes' and 'no' are a reliable proxy for the model's full answer.
Section 2.4 states this without independent validation, and it is used for all calibration measurements.
domain assumption Attention scores assigned to image and text tokens reflect the model's modality reliance.
Section 2.3 adapts this from prior work, but the qualitative attention analysis is limited to one example.
ad hoc to paper The clinical report text is a complementary modality rather than the ground-truth answer.
Section 3.1 and Figure 4 show the full radiology report in the prompt, which often contains explicit findings. The conclusion about 'bias' requires treating this report text as a neutral input.
domain assumption Swapping a modality with a sample of the opposing label and measuring performance change isolates reliance on that modality.
This is the standard counterfactual interpretation, but it is valid only when the unswapped modality does not already contain the answer.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI." pith.science (2026). https://pith.science/paper/PO4MVVMV

@misc{pith2026250800171,
  author       = {Pith},
  title        = {Pith review of: On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PO4MVVMV}},
  note         = {Machine review of arXiv:2508.00171}
}

read the original abstract

Clinical decision-making relies on the integrated analysis of medical images and the associated clinical reports. While Vision-Language Models (VLMs) can offer a unified framework for such tasks, they can exhibit strong biases toward one modality, frequently overlooking critical visual cues in favor of textual information. In this work, we introduce Selective Modality Shifting (SMS), a perturbation-based approach to quantify a model's reliance on each modality in binary classification tasks. By systematically swapping images or text between samples with opposing labels, we expose modality-specific biases. We assess six open-source VLMs-four generalist models and two fine-tuned for medical data-on two medical imaging datasets with distinct modalities: MIMIC-CXR (chest X-ray) and FairVLMed (scanning laser ophthalmoscopy). By assessing model performance and the calibration of every model in both unperturbed and perturbed settings, we reveal a marked dependency on text input, which persists despite the presence of complementary visual information. We also perform a qualitative attention-based analysis which further confirms that image content is often overshadowed by text details. Our findings highlight the importance of designing and evaluating multimodal medical models that genuinely integrate visual and textual cues, rather than relying on single-modality signals.

Figures

Figures reproduced from arXiv: 2508.00171 by the authors.

**Figure 1.** Example of Selective Modality Shifting (SMS) and calibration evaluation on chest X-ray: (a) Original inputs yield correct "No condition" output. (b) Text swap induces false "Pneumonia" prediction, unlike image swap—indicating strong textual bias. (c) Calibration curves are computed over the probabilities of the tokens "yes" and "no". perturbed sample (I, T′ , y). The model’s prediction for this new sample is yˆ (T ′… view at source ↗

**Figure 2.** Zero-shot classification results on (a) MIMIC-CXR and (b) FairVLMed under the Selective Modality Shifting framework. Panels (c) and (d) show the Negative Flip Rate (NFR), measuring how often correct predictions flip to incorrect ones after a perturbation. Janus-Pro does not support text-only input, so the corresponding ablation is omitted. Baseline results from a prior SOTA model are included for reference [24]. c)… view at source ↗

**Figure 3.** (a) ECE values across shift conditions on MIMIC-CXR and (b) FairVLMed. (c,d) Calibration curves for all models. A strong misalignment between predicted probabilities and actual frequencies appears under text shifts, reflecting overconfidence. ical decision-making. Through experiments on MIMIC-CXR and FairVLMed, we observed that performance dropped significantly when textual notes were swapped, whereas image perturb… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Visualization of token-level attention during the generation of an X-ray classification output. The text input prompt is color-coded: violet for dataset metadata, green for clinical notes, and red for model instructions. Each panel shows how the model distributes atte…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 12 canonical work pages

[1]

Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks

Ambsdorf, J.: Benchmarking faithfulness: Towards accurate natural language ex- planations in vision-language tasks. arXiv preprint arXiv:2304.08174 (2023)

work page Pith review arXiv 2023
[2]

arXiv preprint arXiv:2305.18029 (2023)

Atanasova, P., Camburu, O.M., Lioma, C., Lukasiewicz, T., Simonsen, J.G., Au- genstein, I.: Faithfulness tests for natural language explanations. arXiv preprint arXiv:2305.18029 (2023)

arXiv 2023
[3]

In: Proceedings of the IEEE/CVF international conference on computer vision

Chefer, H., Gur, S., Wolf, L.: Generic attention-model explainability for interpret- ing bi-modal and encoder-decoder transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 397–406 (2021) 10 D. Restrepo et al

work page 2021
[4]

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visu- alization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 782–791 (2021)

work page 2021
[5]

arXiv preprint arXiv:2501.17811 (2025)

Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)

arXiv 2025
[6]

https://huggingface.co/collections/ google/medgemma-release-680aade845f90bec6a3f60c4 (2025), accessed: [Insert Date Accessed, e.g., 2025-05-20]

Google: Medgemma hugging face. https://huggingface.co/collections/ google/medgemma-release-680aade845f90bec6a3f60c4 (2025), accessed: [Insert Date Accessed, e.g., 2025-05-20]

2025
[7]

arXiv preprint arXiv:2407.21783 (2024)

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024
[8]

Scientific data 6(1), 317 (2019)

Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6(1), 317 (2019)

2019

Show all 27 references

[9]

npj Digital Medicine 5(1), 171 (2022)

Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., Luo, Y.: Multimodal machine learning in precision health: A scoping review. npj Digital Medicine 5(1), 171 (2022)

2022
[10]

Advances in Neural Information Processing Systems36, 28541–28564 (2023)

Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems36, 28541–28564 (2023)

2023
[11]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26296–26306 (2024)

2024
[12]

Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023)

2023
[13]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Luo, Y., Shi, M., Khan, M.O., Afzal, M.M., Huang, H., Yuan, S., Tian, Y., Song, L., Kouhana, A., Elze, T., et al.: Fairclip: Harnessing fairness in vision-language learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12289–12301 (2024)

2024
[14]

arXiv preprint arXiv:2212.08158 (2022)

Parcalabescu, L., Frank, A.: Mm-shap: A performance-agnostic metric for mea- suring multimodal contributions in vision and language models & tasks. arXiv preprint arXiv:2212.08158 (2022)

2022 arXiv
[15]

Parcalabescu, L., Frank, A.: Do vision & language decoders use images and text equally? how self-consistent are their explanations? arXiv preprint arXiv:2404.18624 (2024)

2024 arXiv
[16]

In: International conference on artificial neural networks

Posocco, N., Bonnefoy, A.: Estimating expected calibration errors. In: International conference on artificial neural networks. pp. 139–150. Springer (2021)

2021
[17]

In: 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI)

Restrepo, D., Nakayama, L.F., Dychiao, R.G., Wu, C., McCoy, L.G., Artiaga, J.C., Cobanaj, M., Matos, J., Gallifant, J., Bitterman, D.S., et al.: Seeing beyond borders: Evaluating llms in multilingual ophthalmological question answering. In: 2024 IEEE 12th International Confere...

2024
[18]

In: Proceedings of the AAAI Conference on Artificial Intelligence

Restrepo, D., Wu, C., Tang, Z., Shuai, Z., Phan, T.N.M., Ding, J.E., Dao, C.T., Gallifant, J., Dychiao, R.G., Artiaga, J.C., et al.: Multi-ophthalingua: A multilin- gual benchmark for assessing and debiasing llm ophthalmological qa in lmics. In: Proceedings of the AAAI Confere...

2025
[19]

arXiv preprint arXiv:2404.03118 (2024)

Stan, G.B.M., Aflalo, E., Rohekar, R.Y., Bhiwandiwalla, A., Tseng, S.Y., Olson, M.L., Gurwicz, Y., Wu, C., Duan, N., Lal, V.: Lvlm-interpret: An interpretability tool for large vision-language models. arXiv preprint arXiv:2404.03118 (2024)

2024 arXiv
[20]

Advances in Neural Information Processing Systems36, 74952–74965 (2023)

Turpin, M., Michael, J., Perez, E., Bowman, S.: Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems36, 74952–74965 (2023)

2023
[21]

arXiv preprint arXiv:1906.04284 (2019)

Vig,J.,Belinkov,Y.:Analyzingthestructureofattentioninatransformerlanguage model. arXiv preprint arXiv:1906.04284 (2019)

2019 arXiv
[22]

arXiv preprint arXiv:2409.12191 (2024)

Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

2024 arXiv
[23]

arXiv preprint arXiv:1809.02805 (2018)

Wu, J., Mooney, R.J.: Faithful multimodal explanation for visual question answer- ing. arXiv preprint arXiv:1809.02805 (2018)

2018 arXiv
[24]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Xia, P., Zhu, K., Li, H., Zhu, H., Li, Y., Li, G., Zhang, L., Yao, H.: RULE: Reliable multimodal RAG for factuality in medical vision language models. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language ...

2024
[25]

In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Yan, S., Xiong, Y., Kundu, K., Yang, S., Deng, S., Wang, M., Xia, W., Soatto, S.: Positive-congruent training: Towards regression-free model updates. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14299–14308 (2021)

2021
[26]

Communications medicine 4(1), 176 (2024)

Yang, Y., Liu, X., Jin, Q., Huang, F., Lu, Z.: Unmasking and quantifying racial bias of large language models in medical report generation. Communications medicine 4(1), 176 (2024)

2024
[27]

arXiv preprint arXiv:2403.05262 (2024)

Zhang, Y.F., Yu, W., Wen, Q., Wang, X., Zhang, Z., Wang, L., Jin, R., Tan, T.: Debiasing multimodal large language models. arXiv preprint arXiv:2403.05262 (2024)

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks

Ambsdorf, J.: Benchmarking faithfulness: Towards accurate natural language ex- planations in vision-language tasks. arXiv preprint arXiv:2304.08174 (2023)

work page Pith review arXiv 2023

[2] [2]

arXiv preprint arXiv:2305.18029 (2023)

Atanasova, P., Camburu, O.M., Lioma, C., Lukasiewicz, T., Simonsen, J.G., Au- genstein, I.: Faithfulness tests for natural language explanations. arXiv preprint arXiv:2305.18029 (2023)

arXiv 2023

[3] [3]

In: Proceedings of the IEEE/CVF international conference on computer vision

Chefer, H., Gur, S., Wolf, L.: Generic attention-model explainability for interpret- ing bi-modal and encoder-decoder transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 397–406 (2021) 10 D. Restrepo et al

work page 2021

[4] [4]

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visu- alization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 782–791 (2021)

work page 2021

[5] [5]

arXiv preprint arXiv:2501.17811 (2025)

Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)

arXiv 2025

[6] [6]

https://huggingface.co/collections/ google/medgemma-release-680aade845f90bec6a3f60c4 (2025), accessed: [Insert Date Accessed, e.g., 2025-05-20]

Google: Medgemma hugging face. https://huggingface.co/collections/ google/medgemma-release-680aade845f90bec6a3f60c4 (2025), accessed: [Insert Date Accessed, e.g., 2025-05-20]

2025

[7] [7]

arXiv preprint arXiv:2407.21783 (2024)

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

arXiv 2024

[8] [8]

Scientific data 6(1), 317 (2019)

Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6(1), 317 (2019)

2019