FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering

Brisa Maneechotesuwan; Chengyue Huang; Shivang Chopra; Zsolt Kira

REVIEW 4 major objections 5 minor 69 references

FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces FRAMES-VQA, a benchmark that arranges ten VQA datasets into in-distribution, near, and far out-of-distribution sets across visual, question, answer, multi-modal, and adversarial shifts, and shows that robust…

desk verdict A useful VQA robustness testbed assembled from existing data, but the paper's headline ranking claims (FTP best far OOD, SPD best overall) do not survive the paper's own supplementary results across backbones and full fine-tuning. read the letter →

arxiv 2505.21755 v2 pith:PMULYQ76 submitted 2025-05-27 cs.CV cs.AIcs.CLcs.LG

Chengyue Huang , Brisa Maneechotesuwan , Shivang Chopra , Zsolt Kira This is my paper

classification cs.CVcs.AIcs.CLcs.LG

keywords visualquestionansweringrobustfine-tuningout-of-distributiongeneralizationmulti-modaldistributionshiftMahalanobisdistancemodalityimportancevision-languagemodelsbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FRAMES-VQA is a benchmark for testing fine-tuning robustness in visual question answering when the data moves away from the training set along one or several modalities at once. The paper assembles nine out-of-distribution datasets around VQAv2, covering visual, question, answer, multi-modal, and adversarial shifts, plus three far-domain sets that require reading text, handling low-quality user images, or using outside knowledge, and measures each shift with Mahalanobis distance on image, question, and joint embeddings. The central finding is that in VQA the usual robustness story from image classification does not hold: plain fine-tuning improves out-of-distribution accuracy rather than degrading it, weight interpolation (WiSE-FT) hurts on every axis, and the best method depends on how far the shift is. SPD, which selectively regularizes layers, leads on in-distribution, near, and average out-of-distribution accuracy, while FTP, which underfits the training distribution, is best on the three far-shift datasets. Attention analysis adds a mechanism: fine-tuning amplifies question-shift influence on joint representations, and out-of-distribution samples draw question tokens toward image attention.

What carries the argument

Three components carry the argument. The first is the benchmark taxonomy: ten datasets sorted into one in-distribution set (VQAv2) and nine out-of-distribution sets, labeled near or far by qualitative shift type, with each set's shift quantified by the Mahalanobis distance of its samples from the VQAv2 training distribution in image, question, and joint embedding spaces. The second is Modality Importance (MI), defined for a token as the ratio of its attention to question tokens over its attention to image tokens, which tracks how fine-tuning redistributes cross-modal attention as shift grows. The third is the controlled comparison of fine-tuning methods, all LoRA-adapted on the same PaliGemma backbone, which turns the benchmark into a ranking of regularization strategies rather than architectures.

What would settle it

Re-run the Table 3 comparison with each method's regularization strength (FTP's gradient-annealing factor, SPD's weight decay, WiSE-FT's interpolation coefficient) selected on an out-of-distribution validation split instead of by VQAv2 in-distribution accuracy, and check whether SPD still leads on near and average OOD and FTP still leads on far OOD; a reversal would show the benchmark's headline rankings come from model selection rather than method robustness.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a multi-modal shift benchmark with measured distances produces a different and finer-grained picture of robust fine-tuning than unimodal image benchmarks. Fine-tuning PaliGemma-3B with LoRA on VQAv2, the paper reports no degradation in out-of-distribution accuracy relative to zero-shot for vanilla fine-tuning, which it explains by the fine-tuning objective matching the pre-training objective; WiSE-FT's interpolation of zero-shot and fine-tuned weights, by contrast, loses robustness across all shifts. The ranking divides by shift distance: SPD, which selects for strong regularization only the layers whose updates consistently reduce the loss, achieves the highest in-distribution, near-OOD, and average-OOD accuracy, while FTP, whose projection constraint is non-decreasing and underfits the in-distribution data, achieves the highest accuracy on all three far-OOD datasets (TextVQA, VizWiz, OK-VQA).

Load-bearing premise

The whole ranking assumes that picking each method's regularization hyperparameter by best in-distribution validation accuracy gives a fair and representative checkpoint for comparing out-of-distribution behavior; if the optimal hyperparameters for OOD robustness differ from those for in-distribution accuracy, the reported SPD and FTP leads could be artifacts of that selection rule.

Editorial extensions

If this is right

Robustness results from image classification do not carry over to VQA: because fine-tuning for this task retains the pre-training objective, plain fine-tuning can improve out-of-distribution accuracy, so multi-modal robustness needs its own benchmarks.
SPD is the method to choose when deployment includes in-distribution or near-shift data, since it reaches the highest in-distribution, near-OOD, and average-OOD accuracy in the main PaliGemma-LoRA experiments.
For far-domain deployment, reading text in images, handling user-generated photos, or answering knowledge-requiring questions, FTP's stronger non-decreasing projection regularization gives the best accuracy even though it underfits the in-distribution data.
WiSE-FT should not be expected to transfer from image classification to VQA: in this setting its weight interpolation reduces robustness across all shift types.
Because fine-tuning amplifies the correlation between question shifts and joint shifts, and question-to-image attention rises on out-of-distribution samples, future robust fine-tuning for VQA should target language-side shifts and cross-modal attention, not only visual shifts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence follows from the paper's model-selection protocol: re-choosing each method's hyperparameters on out-of-distribution validation accuracy rather than in-distribution accuracy could reorder SPD and FTP, which would pinpoint whether the ranking reflects method robustness or checkpoint choice.
The attention-shift finding suggests a direct intervention: adding a regularization that penalizes the rise in question-to-image attention on high-shift samples during fine-tuning could reproduce the intra-modality dominance the paper associates with robustness.
Since question shift dominates joint shift after fine-tuning, regularizing the language tower or question embeddings specifically may beat global layer-wise methods at equal in-distribution accuracy; this follows from the paper's correlation analysis but is not tested there.
The near/far split is really a continuous distance axis, so a practical extension would be to weight training samples by their Mahalanobis shift score, directly optimizing the negative shift-performance correlation the paper reports.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful part of this paper is the assembly: ten existing VQA datasets organized into ID, near OOD, and far OOD groups, with shift distances computed via Mahalanobis embeddings and a clean comparison of robust fine-tuning methods on PaliGemma-LoRA. The code is public and the evaluation protocol is reproducible. That is a real service to the community, and the shift/attention analyses (question-joint correlation rising after fine-tuning, intra-modality attention differences) are suggestive directions for future work.

The soft spot is the central ranking claim. The paper says FTP is best on far OOD and SPD best on ID/near/overall OOD, and frames FRAMES-VQA as a benchmark that separates methods by shift type. But the supplementary results contradict that. With LLaVA-LoRA, FTP's far OOD average is 26.21 vs SPD's 29.98. With full fine-tuning on PaliGemma, the FTP-SPD far OOD gap is 0.09 points, within seed noise, and FTP's overall OOD average actually exceeds SPD's. The sign of the FTP-SPD gap flips across three configurations. So the headline finding is a property of the PaliGemma-LoRA setup, not of the benchmark itself. Combined with no error bars, single runs, and model selection by ID validation accuracy, the specific method rankings should be labeled as observations from one configuration rather than general conclusions.

That said, the benchmark still deserves to exist. The ID/near/far taxonomy and the public evaluation harness are usable by other groups. The main fix is to soften the claims, add multiple seeds, and report sensitivity to the Mahalanobis threshold and hyperparameter choices. With those revisions, this becomes a solid resource for the robust fine-tuning community.

I would send it to peer review: the benchmark is substantial, the code is available, and the issues are correctable rather than load-bearing. But a serious referee should require the additional experiments before publication.

Referee Report

4 major / 5 minor

Summary. The paper introduces FRAMES-VQA, a benchmark for evaluating fine-tuning robustness in multimodal VQA. Ten existing VQA datasets are categorized into ID (VQAv2), six near-OOD, and three far-OOD sets, with shifts quantified by Mahalanobis distance on unimodal (image, question) and joint embeddings. On a PaliGemma-3B model fine-tuned with LoRA, the authors compare several fine-tuning strategies and report that SPD achieves the best ID, near-OOD, and overall-OOD performance, while FTP excels on far-OOD datasets; they also report that vanilla fine-tuning improves OOD performance over zero-shot in this setting. The paper further analyzes correlations between unimodal and multimodal shifts and modality-attention importance, concluding that robust methods show lower unimodal/multimodal correlation and higher intra-modality attention. The code is released.

Significance. If the ranking claims were stable, FRAMES-VQA would be a valuable resource: it is the first to systematically distinguish near and far OOD along multiple modalities in VQA, and it provides a concrete demonstration that fine-tuning robustness method rankings can depend on shift distance. The paper's strengths are the thoughtful dataset curation, the use of multiple embedding types for shift quantification, cross-checking with MMD, and the extensive supplementary experiments including GQA and a second backbone. However, the central conclusions currently rest on a single backbone/adaptation configuration with no error bars, and the paper's own supplementary tables show that the FTP-vs-SPD ranking flips under LLaVA-LoRA and nearly disappears under full fine-tuning. This makes the significance conditional on additional validation.

major comments (4)

[§4.2, Tab. 3, Supp. Tab. 8] The headline ranking is not stable across the paper's own configurations. In Tab. 3 (PaliGemma-LoRA), FTP's far-OOD average is 41.95 vs SPD's 38.91, supporting 'FTP best far OOD.' But in Supp. Tab. 8(a) (LLaVA-7B-LoRA, 10% VQAv2), FTP's far-OOD average is 26.21 vs SPD's 29.98, so SPD is better; in Supp. Tab. 8(b) (PaliGemma full fine-tuning), FTP's far-OOD average is 32.55 vs SPD's 32.46, a near tie, and FTP's overall OOD average (53.74) exceeds SPD's (53.43), contradicting 'SPD best overall OOD.' The authors state these results are 'consistent,' but the sign and magnitude of the FTP-SPD far-OOD gap vary from +3.04 to -3.77 to +0.09 across the three tables. Since Tab. 3 reports a single configuration without variance or multiple seeds, the paper's central claim that FRAMES-VQA distinguishes robust fine-tuning methods by shift type and distance is not established beyond the specific PaliGemma-LoRA setup. Please either restrict the conclusions to that configuration or provide a stability analysis (e.g., rank correlations across backbones and seeds) and adjust the claims accordingly.
[Supp. Sec. 8] The model-selection protocol is load-bearing but potentially unfair for OOD comparison. The supplement states: 'The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken.' If OOD performance had been the selection criterion, the optimal hyperparameters for methods like FTP (which trades ID for far-OOD performance) might differ, and the Tab. 3 rankings could change. Because the benchmark's purpose is to evaluate OOD robustness, please justify this ID-only selection choice, report sensitivity of the rankings to the selection criterion, or provide results for a small grid of regularization strengths.
[§4.2, Tab. 3] No variance estimates are reported for any metric. All differences between methods (e.g., FTP 41.95 vs SPD 38.91 on far OOD, or SPD 87.39 vs Vanilla 86.29 on ID) are single runs. Without multiple seeds or confidence intervals, the 'best' designations may reflect training noise rather than true differences. Please report standard deviations over at least three seeds for the main configuration, or, if that is computationally infeasible, state the limitation and provide an effect-size argument that the observed gaps exceed expected noise.
[§5.2, §6.1, Tab. 5] The shift-correlation and modality-importance analyses, which are presented as key contributions, lack statistical support. Claims such as 'more robust fine-tuning methods exhibit low correlation between uni-modal and multimodal shifts' (Sec. 5.2) and 'question-to-image attention rises for OOD samples' (Sec. 6.3) are based on Pearson correlations over only 9 datasets and on a Mahalanobis threshold of 60 chosen as the 'relative median' without sensitivity analysis. With n=9, the correlations are noisy, and the threshold choice could affect the ID/OOD split in Tab. 5. Please add confidence intervals or permutation tests for the correlations, and show that the qualitative conclusions are robust to reasonable threshold choices.

minor comments (5)

[§4.2] The text 'FTP [46] underfits the ID dataset' cites reference [46], which is TPGM; FTP is reference [47]. Please correct the citation.
[§4.1, Eq. (2)] Equation (2) is typeset in a garbled way; the bi-level optimization should be written with explicit inner and outer objective functions and clear constraints. The same issue affects Eq. (4) and surrounding text.
[Supp. Tab. 8] The column headers 'VQAv2 valNear OOD Avg.Far OOD Avg.OOD Avg.' are missing spaces, making the table hard to read. Also, note explicitly in the main text that Tab. 8 uses only 10% of VQAv2, unlike Tab. 3, so readers can compare directly.
[§3.1] The text says 'We choose this as our ID datasets'; please fix the grammar to 'ID dataset.' Additionally, consider clarifying that the far-OOD datasets (TextVQA, VizWiz, OK-VQA) are not derived from VQAv2, unlike the near-OOD datasets.
[§1 and §4.2] The paper evaluates FTP [47] and SPD [48], both of which are from the same research group as the authors, without an explicit disclosure in the main text. Please add a statement of competing interests or a note that these methods are the authors' own work, for transparency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark's headline results are direct empirical evaluations on public datasets, and the self-citations are method attribution rather than load-bearing assumptions.

full rationale

FRAMES-VQA is an empirical benchmark paper. Its derivation chain is: assemble public VQA datasets, categorize them by shift type, fine-tune PaliGemma/LLaVA with published robust-fine-tuning baselines, measure accuracy on held-out test splits, compute Mahalanobis distances from model embeddings, and correlate these quantities with performance. None of these steps fits a parameter to the target result and then relabels that fit as a prediction. The headline rankings (SPD best on ID/near/overall OOD and FTP best on far OOD in the main PaliGemma-LoRA table) are direct measurements on public benchmarks, not outputs of a fitted model. Self-citations to [25], [47], and [48] attribute the preliminary version of the benchmark and the compared methods; they do not provide a uniqueness theorem or an assumption that entails the ranking. The supplementary's ID-validation-based hyperparameter selection, and the differing FTP/SPD ordering under LLaVA-LoRA and full fine-tuning, are legitimate stability and model-selection concerns, but they are correctness risks rather than circularity: no claim reduces by construction to its inputs. The paper is therefore self-contained against external benchmarks and receives a circularity score of 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on hyperparameter choices, on embedding-based shift measurement assumptions, and on a data-dependent threshold. No invented entities are introduced; all datasets are pre-existing.

free parameters (5)

ID/OOD Mahalanobis threshold = 60
Used in Sec. 6.1 to label ID vs OOD samples for the modality-importance analysis; chosen as the relative median of observed shift scores.
WiSE-FT interpolation alpha = 0.5
Interpolation coefficient in Eq. 3, selected by validation; determines WiSE-FT's poor ranking.
FTP gradient annealing factor kappa = 0
Reported best configuration in Supp. Tab. 6; FTP's far-OOD advantage is tied to this minimal-regularization setting.
SPD weight decay = 0.5
Strong penalty in SPD's layer selection (Eq. 8); key to its ID and near-OOD performance.
LoRA rank and target modules = r=8, Wq/Wk/Wv/Wo
All main results use LoRA rank 8 on attention weights; the benchmark scope is this parameter-efficient setting.

assumptions (5)

domain assumption Mean-pooled final-layer token embeddings faithfully represent image, question, and joint distributions.
Sec. 3.2: all Mahalanobis scores and shift/performance correlations are computed from these embeddings.
domain assumption The VQAv2-train empirical covariance matrix can be inverted to compute Mahalanobis distances in high-dimensional feature spaces.
Eq. 1; no regularization or pseudo-inverse details are given for the covariance estimate.
domain assumption Selecting checkpoints by best ID validation accuracy is a fair model-selection rule for comparing OOD robustness.
Supp. Sec. 8 states all methods are chosen this way; it can favor adapters that overfit ID.
domain assumption Attention-weight ratios measure modality importance and shortcut behavior.
Sec. 6.1 Eqs. 9-10; no evidence that attention correlates causally with model decisions.
domain assumption PaliGemma-3B with LoRA is representative enough for the benchmark's method rankings.
Main experiments use one backbone and LoRA; supplementary LLaVA results show rankings can differ.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering." pith.science (2026). https://pith.science/paper/PMULYQ76

@misc{pith2026250521755,
  author       = {Pith},
  title        = {Pith review of: FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMULYQ76}},
  note         = {Machine review of arXiv:2505.21755}
}

read the original abstract

Visual question answering (VQA) systems face significant challenges when adapting to real-world data shifts, especially in multi-modal contexts. While robust fine-tuning strategies are essential for maintaining performance across in-distribution (ID) and out-of-distribution (OOD) scenarios, current evaluation settings are primarily unimodal or particular to some types of OOD, offering limited insight into the complexities of multi-modal contexts. In this work, we propose a new benchmark FRAMES-VQA (Fine-Tuning Robustness across Multi-Modal Shifts in VQA) for evaluating robust fine-tuning for VQA tasks. We utilize ten existing VQA benchmarks, including VQAv2, IV-VQA, VQA-CP, OK-VQA and others, and categorize them into ID, near and far OOD datasets covering uni-modal, multi-modal and adversarial distribution shifts. We first conduct a comprehensive comparison of existing robust fine-tuning methods. We then quantify the distribution shifts by calculating the Mahalanobis distance using uni-modal and multi-modal embeddings extracted from various models. Further, we perform an extensive analysis to explore the interactions between uni- and multi-modal shifts as well as modality importance for ID and OOD samples. These analyses offer valuable guidance on developing more robust fine-tuning methods to handle multi-modal distribution shifts. The code is available at https://github.com/chengyuehuang511/FRAMES-VQA .

Figures

Figures reproduced from arXiv: 2505.21755 by the authors.

**Figure 1.** ID and OOD VQA datasets with Uni-modal (Vision, Question, Answer), Multi-modal, Adversarial and Far Distribution Shifts. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Overview of Robust Fine-Tuning Methods. The flame icon represents tunable layers, while the snowflake icon represents frozen [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Mahalanobis Distance Heatmap of Different Datasets, Embeddings and Fine-Tuning Methods. Darker blue represents higher [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Variation of MIv and MIq w.r.t. shift score under vanilla FT, FTP and SPD across VQAv2 and VQA-CE. The blue and orange bars represent MIv and MIq respectively. The red dotted line marks a reference MI of 1 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Heatmap of correlation between uni-modal and multi [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Sampling region in histogram Under V, Q and V+Q, we sample 50 instances from 4 regions as shown in [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Comparison set of ID and OOD images in terms of object categories and compositions (e.g., # objects) [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Comparison of ID and OOD images but with similar objects. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Comparison set of ID and OOD joint samples. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: Histogram for Vanilla FT Visual Shifts: We depict the [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

**Figure 11.** Figure 11: Histogram for Vanilla FT Question Shifts: We depict the [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

**Figure 12.** Figure 12: Histogram for Vanilla FT V+Q Shifts : We depict the [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

**Figure 13.** Figure 13: Variation of MIv and MIq w.r.t. shift score under PT, FT, LP, LP-FT, FTP and SPD across all ID and near OOD datasets. The blue and orange bars represent MIv and MIq respectively. The red dotted line marks a reference MI of 1. 9 [PITH_FULL_IMAGE:figures/full_fig_p019_…

**Figure 14.** Figure 14: Variation of MIv and MIq w.r.t. shift score under PT, FT, LP, LP-FT, FTP and SPD across all far OOD datasets. The blue and orange bars represent MIv and MIq respectively. The red dotted line marks a reference MI of 1. 10 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 46 canonical work pages

[1]

To- wards Causal VQA: Revealing and Reducing Spurious Cor- relations by Invariant and Covariant Semantic Editing

Vedika Agarwal, Rakshith Shetty, and Mario Fritz. To- wards Causal VQA: Revealing and Reducing Spurious Cor- relations by Invariant and Covariant Semantic Editing. In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 9687–9695, Seattle, W A, USA, 2020. IEEE. 1, 2, 3

work page 2020
[2]

Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering, 2018

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Anirud- dha Kembhavi. Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering, 2018. arXiv:1712.00377 [cs]. 1, 2, 3

arXiv 2018
[3]

Reassessing Evaluation Practices in Visual Ques- tion Answering: A Case Study on Out-of-Distribution Gen- eralization, 2023

Aishwarya Agrawal, Ivana Kaji ´c, Emanuele Bugliarello, El- naz Davoodi, Anita Gergely, Phil Blunsom, and Aida Ne- matzadeh. Reassessing Evaluation Practices in Visual Ques- tion Answering: A Case Study on Out-of-Distribution Gen- eralization, 2023. arXiv:2205.12191 [cs]. 2

arXiv 2023
[4]

PaliGemma: A versatile 3B VLM for transfer, 2024

Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisensch- los, Rishabh Kabra, Matthi...

arXiv 2024
[5]

Cunningham

Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. Lora learns less and forgets less, 2024. 5

work page 2024
[6]

VizWiz: nearly real-time answers to visual questions

Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Lit- tle, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. VizWiz: nearly real-time answers to visual questions. 3

work page
[7]

Behind the scene: Revealing the secrets of pre-trained vision-and-language models, 2020

Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and-language models, 2020. 7

work page 2020
[8]

Benchmarking robustness of adaptation methods on pre-trained vision-language models, 2023

Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and V olker Tresp. Benchmarking robustness of adaptation methods on pre-trained vision-language models, 2023. 2

work page 2023

Show all 69 references

[9]

Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answer- ing, 2021

Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answer- ing, 2021. arXiv:2104.03149 [cs]. 1, 2, 3

2021 arXiv
[10]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 1, 2

2009
[11]

Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019. 4

2019
[12]

Xin Dong, Junfeng Guo, Ang Li, Wei-Te Ting, Cong Liu, and H. T. Kung. Neural mean discrepancy for efficient out- of-distribution detection, 2022. 2

2022
[13]

An image is worth 16x16 words: Transformers for image recognition at scale, 2021

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

2021
[14]

Ex- ploring the limits of out-of-distribution detection, 2021

Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Ex- ploring the limits of out-of-distribution detection, 2021. 2

2021
[15]

VQA-LOL: Visual Question Answering under the Lens of Logic, 2020

Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. VQA-LOL: Visual Question Answering under the Lens of Logic, 2020. arXiv:2002.08325 [cs]. 1, 2

2020 arXiv
[16]

paligemma-3b-pt-224.https : / / huggingface

Google. paligemma-3b-pt-224.https : / / huggingface . co / google / paligemma - 3b - pt-224, 2024. 4, 6, 8, 1

2024
[17]

Hospedales, and Massimiliano Pontil

Henry Gouk, Timothy M. Hospedales, and Massimiliano Pontil. Distance-Based Regularisation of Deep Networks for Fine-Tuning, 2021. arXiv:2002.08253 [cs, stat]. 2

2021 arXiv
[18]

Finetune like you pretrain: Im- proved finetuning of zero-shot vision models

Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Im- proved finetuning of zero-shot vision models. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19338–19347, 2023. 5

2023
[19]

Making the V in VQA Matter: Ele- vating the Role of Image Understanding in Visual Question Answering, 2017

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA Matter: Ele- vating the Role of Image Understanding in Visual Question Answering, 2017. arXiv:1612.00837 [cs]. 2, 3

2017 arXiv
[20]

Rasch, Bern- hard Scholkopf, and Alexander J

Arthur Gretton, Karsten Borgwardt, Malte J. Rasch, Bern- hard Scholkopf, and Alexander J. Smola. A kernel method for the two-sample problem, 2008. 3, 2

2008
[21]

The many faces of robust- ness: A critical analysis of out-of-distribution generalization

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. InProceedings of the IEEE/CVF Internationa...

2021
[22]

Natural adversarial examples

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 15262–15271, 2021. 1, 2

2021
[23]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models,
[24]

Llm-adapters: An adapter family for parameter- efficient fine-tuning of large language models, 2023

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. Llm-adapters: An adapter family for parameter- efficient fine-tuning of large language models, 2023. 5

2023
[25]

Directional gradient pro- jection for robust fine-tuning of foundation models, 2025

Chengyue Huang, Junjiao Tian, Brisa Maneechotesuwan, Shivang Chopra, and Zsolt Kira. Directional gradient pro- jection for robust fine-tuning of foundation models, 2025. 1

2025
[26]

Hudson and Christopher D

Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. 2

2019
[27]

Roses are red, violets are blue

Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are blue... but should vqa expect them to?, 2021. 2 9

2021
[28]

Fine-Tuning can Distort Pre- trained Features and Underperform Out-of-Distribution,

Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-Tuning can Distort Pre- trained Features and Underperform Out-of-Distribution,
[29]

Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Sil- vio Savarese, and Steven C. H. Hoi. Lavis: A library for language-vision intelligence, 2022. 1

2022
[30]

A Closer Look at the Robustness of Vision-and-Language Pre-trained Models,

Linjie Li, Zhe Gan, and Jingjing Liu. A Closer Look at the Robustness of Vision-and-Language Pre-trained Models,
[31]

Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models, 2021

Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models, 2021. arXiv:2106.00245 [cs]. 1, 2

2021 arXiv
[32]

Ex- plicit Inductive Bias for Transfer Learning with Convolu- tional Networks, 2018

Xuhong Li, Yves Grandvalet, and Franck Davoine. Ex- plicit Inductive Bias for Transfer Learning with Convolu- tional Networks, 2018. arXiv:1802.01483 [cs]. 2, 4

2018 arXiv
[33]

arXiv:2012.08673 [cs]. 2

2012 arXiv
[34]

Robust Visual Ques- tion Answering: Datasets, Methods, and Future Challenges,

Jie Ma, Pinghui Wang, Dechen Kong, Zewei Wang, Jun Liu, Hongbin Pei, and Junzhou Zhao. Robust Visual Ques- tion Answering: Datasets, Methods, and Future Challenges,
[35]

Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. 3

2019
[36]

Visual instruction tuning, 2023

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 5, 1

2023
[37]

Maximum mean discrep- ancy for generalization in the presence of distribution and missingness shift, 2022

Liwen Ouyang and Aaron Key. Maximum mean discrep- ancy for generalization in the presence of distribution and missingness shift, 2022. 2

2022
[38]

Moment matching for multi-source domain adaptation

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 1406–1415,
[39]

Learning Transferable Vi- sual Models From Natural Language Supervision, 2021

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Vi- sual Models From Natural Language Supervision, 2021. arXiv:2103.0...

2021 arXiv
[40]

Generalized out-of-distribution detection and be- yond in vision language model era: A survey, 2024

Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, Ziwei Liu, Toshihiko Yamasaki, and Kiyoharu Aizawa. Generalized out-of-distribution detection and be- yond in vision language model era: A survey, 2024. 2

2024
[41]

Cycle-Consistency for Robust Visual Question Answering,

Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-Consistency for Robust Visual Question Answering,
[42]

Human-Adversarial Visual Question Answer- ing, 2021

Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Al- berto Lopez Magana, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Human-Adversarial Visual Question Answer- ing, 2021. arXiv:2106.02280 [cs]. 1, 2, 3

2021 arXiv
[43]

Benchmarking out-of- distribution detection in visual question answering, 2024

Xiangxi Shi and Stefan Lee. Benchmarking out-of- distribution detection in visual question answering, 2024. 3

2024
[44]

Do imagenet classifiers generalize to im- agenet? InInternational Conference on Machine Learning, pages 5389–5400

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? InInternational Conference on Machine Learning, pages 5389–5400. PMLR, 2019. 1, 2

2019
[45]

Towards VQA Models That Can Read, 2019

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read, 2019. arXiv:1904.08920 [cs]. 3

2019 arXiv
[46]

Trainable Projected Gradi- ent Method for Robust Fine-tuning, 2023

Junjiao Tian, Xiaoliang Dai, Chih-Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable Projected Gradi- ent Method for Robust Fine-tuning, 2023. arXiv:2303.10720 [cs]. 2, 4, 5

2023 arXiv
[47]

Fast Trainable Projection for Robust Fine-Tuning,

Junjiao Tian, Yen-Cheng Liu, James Seale Smith, and Zsolt Kira. Fast Trainable Projection for Robust Fine-Tuning,
[48]

Rethinking weight decay for robust fine-tuning of foundation models,

Junjiao Tian, Chengyue Huang, and Zsolt Kira. Rethinking weight decay for robust fine-tuning of foundation models,
[49]

Language Prior Is Not the Only Short- cut: A Benchmark for Shortcut Learning in VQA, 2022

Qingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin, Yuanxin Liu, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. Language Prior Is Not the Only Short- cut: A Benchmark for Shortcut Learning in VQA, 2022. arXiv:2210.04692 [cs]. 1, 2

2022 arXiv
[50]

Learning robust global representations by penalizing local predictive power.Advances in Neural Information Pro- cessing Systems, 32, 2019

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power.Advances in Neural Information Pro- cessing Systems, 32, 2019. 1, 2

2019
[51]

Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Si- mon Kohl, Taylan Cemgil, S

Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Si- mon Kohl, Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive training for improved out-of- distribut...

2020
[52]

Robust fine-tuning of zero-shot models, 2022

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models, 2022. arXiv:2109.01903 [cs]. 1, 2, 4, 5, 6

2022 arXiv
[53]

Domain-robust vqa with di- verse datasets and methods but no target labels, 2021

Mingda Zhang, Tristan Maidment, Ahmad Diab, Adriana Kovashka, and Rebecca Hwa. Domain-robust vqa with di- verse datasets and methods but no target labels, 2021. 2

2021
[54]

Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023. 5 10 FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering Supp...

2023
[55]

VQA- GEN: A Visual Question Answering Benchmark for Domain Generalization, 2023

Suraj Jyothi Unni, Raha Moraffah, and Huan Liu. VQA- GEN: A Visual Question Answering Benchmark for Domain Generalization, 2023. arXiv:2311.00807 [cs]. 1, 2

2023 arXiv
[61]

We use the LA VIS [29] public repository to fine-tune all methods

Training Details We use the model pretrained with224∗224input images and128token input/output text sequences and fine-tune with the precision of bfloat16. We use the LA VIS [29] public repository to fine-tune all methods. Standard hyper- parameters are used for all: learning r...
[62]

Histograms of Shift Scores We display the histograms for the Mahalanobis score dis- tribution between test datasets with the ID set. Fig. 10, 11 and 12 are visual, question and joint shifts from vanilla FT repectively. The histograms show that under Vanilla FT, visual shifts a...
[63]

7 shows the correlation between shift and performance for different embeddings under different fine-tuning meth- ods

Correlation between Shift & Performance Tab. 7 shows the correlation between shift and performance for different embeddings under different fine-tuning meth- ods. Overall, visual and joint shifts exhibit the strongest correlation with performance across all types of methods. T...
[64]

5 shows the heatmap of the correlation between uni- modal and multi-modal shifts per dataset

Correlation between Uni- & Multi-Modal Shifts per Dataset Fig. 5 shows the heatmap of the correlation between uni- modal and multi-modal shifts per dataset. Question-joint shift correlations are higher than image-joint shift corre- lations across all VQA datasets and fine-tuni...
[65]

13 and 14 show the variation of MIv and MIq w.r.t

Modality Importance of different Datasets and Fine-Tuning Methods Fig. 13 and 14 show the variation of MIv and MIq w.r.t. shift score under all datasets and fine-tuning methods. Over- all, intra-modality attention is more dominant than inter- modality attention. There is a str...
[66]

8, including LLaV A- 7B [33] with LoRA and PaliGemma-3B with full fine- tuning

Additional Results using Full Fine-Tuning and LLaV A We present additional results in Tab. 8, including LLaV A- 7B [33] with LoRA and PaliGemma-3B with full fine- tuning. These results are consistent with PaliGemma with 1 (1) Question-Joint shift correlation heatmap (2) Image-...
[67]

The only exception, GQA-OOD [27] (based on GQA [26]), has only answer shifts

Fine-Tuning Results on GQA We use VQAv2 as the ID dataset since most OOD VQA datasets, covering various shifts, are built on it. The only exception, GQA-OOD [27] (based on GQA [26]), has only answer shifts. To further validate our findings, we fine-tune PaliGemma-3B on GQA as ...
[68]

We further compare shifts using Maximum Mean Discrepancy (MMD) [12, 20, 37] with RBF kernel in Tab

Quantifying Shifts using Maximum Mean Discrepancy Mahalanobis distance is a dominant metric for measuring distribution shifts [36]. We further compare shifts using Maximum Mean Discrepancy (MMD) [12, 20, 37] with RBF kernel in Tab. 10. We observe similar trends as with Mahalan...
[69]

This also serves as a veri- fication of the reliability in quantifying shifts via feature- based representations

Qualitative Analysis: Inspect via Sampling In order to investigate the types of ID and OOD samples under different modalities, we perform sampling on the var- ious regions of the histogram to inspect how the model rep- resents ID/OOD embeddings. This also serves as a veri- fic...
[2019]

arXiv:1902.05660 [cs]. 1, 2, 3

1902 arXiv
[2021]

5, 6, 8, 1

arXiv:2106.09685 [cs]. 5, 6, 8, 1

arXiv
[2022]

2, 4, 6, 8, 1

arXiv:2202.10054 [cs]. 2, 4, 6, 8, 1

arXiv
[2023]

2, 4, 5, 6, 8, 1

arXiv:2310.19182 [cs]. 2, 4, 5, 6, 8, 1

arXiv
[2024]

arXiv:2307.11471 [cs]. 2

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

To- wards Causal VQA: Revealing and Reducing Spurious Cor- relations by Invariant and Covariant Semantic Editing

Vedika Agarwal, Rakshith Shetty, and Mario Fritz. To- wards Causal VQA: Revealing and Reducing Spurious Cor- relations by Invariant and Covariant Semantic Editing. In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 9687–9695, Seattle, W A, USA, 2020. IEEE. 1, 2, 3

work page 2020

[2] [2]

Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering, 2018

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Anirud- dha Kembhavi. Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering, 2018. arXiv:1712.00377 [cs]. 1, 2, 3

arXiv 2018

[3] [3]

Reassessing Evaluation Practices in Visual Ques- tion Answering: A Case Study on Out-of-Distribution Gen- eralization, 2023

Aishwarya Agrawal, Ivana Kaji ´c, Emanuele Bugliarello, El- naz Davoodi, Anita Gergely, Phil Blunsom, and Aida Ne- matzadeh. Reassessing Evaluation Practices in Visual Ques- tion Answering: A Case Study on Out-of-Distribution Gen- eralization, 2023. arXiv:2205.12191 [cs]. 2

arXiv 2023

[4] [4]

PaliGemma: A versatile 3B VLM for transfer, 2024

Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisensch- los, Rishabh Kabra, Matthi...

arXiv 2024

[5] [5]

Cunningham

Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. Lora learns less and forgets less, 2024. 5

work page 2024

[6] [6]

VizWiz: nearly real-time answers to visual questions

Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Lit- tle, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. VizWiz: nearly real-time answers to visual questions. 3

work page

[7] [7]

Behind the scene: Revealing the secrets of pre-trained vision-and-language models, 2020

Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and-language models, 2020. 7

work page 2020

[8] [8]

Benchmarking robustness of adaptation methods on pre-trained vision-language models, 2023

Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and V olker Tresp. Benchmarking robustness of adaptation methods on pre-trained vision-language models, 2023. 2

work page 2023