Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?

Anthony Z. Liu; Bill Howe; Bingbing Wen; Chung Peng Lee; Dora Zhao; Shangbin Feng; Yiwei Yang; Yulia Tsvetkov

REVIEW 4 major objections 5 minor 20 references

Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper builds a benchmark of 124 naturally occurring visual shortcuts and shows that fine-tuning on synthetic spurious examples transfers to unseen spurious correlations, while revealing a robustness-accuracy trade-off.

desk verdict A genuinely useful new benchmark for LVLM spurious correlations, but the headline fine-tuning claim needs a missing control before it is believable. read the letter →

arxiv 2506.18322 v1 pith:E62S6U76 submitted 2025-06-23 cs.CV cs.LG

Yiwei Yang , Chung Peng Lee , Shangbin Feng , Dora Zhao , Bingbing Wen , Anthony Z. Liu , Yulia Tsvetkov , Bill Howe This is my paper · ORCID

classification cs.CVcs.LG

keywords spuriouscorrelationlargevision-languagemodelsvisualquestionansweringshortcutlearningsyntheticcounterfactualimagesfine-tuninggeneralizationrobustness-accuracytrade-offbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpuriVerse is a benchmark of 124 spurious correlations mined from real visual-question-answering data, each with one real image and ten synthetic images engineered to contain or omit the spurious cue. On these naturally occurring shortcuts, fifteen large vision-language models all score below random guessing on the real anchors, with the best at 37.9% against a 40.7% random baseline. The paper's central experimental claim is that fine-tuning on synthetic images that emphasize diverse spurious patterns transfers to spurious correlations never seen during training: accuracy on held-out anchors rises to 80.0% for Llama-3.2-11B and 78.4% for Qwen2.5-VL-7B. This transfer comes with a cost, because the same fine-tuning lowers accuracy on ordinary non-spurious samples, suggesting models rely on shortcuts for part of their normal performance. The broader point is that shortcut reliance in pretrained multimodal models can be studied, measured, and partly trained away with a single diverse benchmark.

What carries the argument

The load-bearing object is the paired counterfactual image group. For each error sample, the pipeline generates two scene descriptions from the question and correct answer—one containing the candidate spurious attribute and one without it—and renders ten Stable Diffusion images per description. Selection into SpuriVerse requires that at least one of three reference LVLMs (GPT-4o, Gemini 2.0 Flash, or Qwen-VL-Max) shows an accuracy gap of at least 30 percentage points between the core group and the spurious group. That core-versus-spurious gap is the operational definition of spurious reliance; the same synthetic spurious groups are then used as the fine-tuning data that produces the transfer result.

What would settle it

Re-measure core-versus-spurious accuracy using image pairs matched on style, seed, layout, and rendering quality, then see whether the 30-percentage-point gaps that define SpuriVerse's anchors persist; if the gaps collapse, the benchmark is capturing generator artifacts instead of spurious reliance. Alternatively, have human annotators label which images contain the spurious feature and check whether models' errors track human judgments better than the core-group contrast does.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that state-of-the-art LVLMs systematically fail on real-world VQA questions whenever a dominant but unreliable visual feature points to the wrong answer, and that this failure is not fixed by prompting. The operational discovery is the transfer result: fine-tuning Llama-3.2-11B-vision-instruct on synthetic spurious-group images raises held-out anchor accuracy from 41.60% to 80.00%, and Qwen2.5-VL-7B-instruct from 35.20% to 78.40%, where the held-out set contains spurious correlation types absent from fine-tuning. The same training raises accuracy on held-out synthetic spurious groups to 79.12% and 75.60%. Prompting strategies help only modestly, with the best spurious-aware prompt reaching 51.61% anchors for Qwen2.5-VL-7B, barely above random. The paper interprets these results as evidence that models can learn a general meta-skill of ignoring dominant-but-unreliable cues and attending to overall scene context, while noting a systematic trade-off: accuracy on non-spurious samples falls by roughly 14 to 17 percentage points when models are fine-tuned on spurious groups alone.

Load-bearing premise

The benchmark's validity rests on the assumption that the synthetic core and spurious images differ only in the targeted spurious attribute; if other visual differences (style, quality, composition) leak through, the accuracy gap between groups measures those confounds rather than the model's reliance on the spurious cue.

Editorial extensions

If this is right

Any LVLM can be scored on SpuriVerse's 124 anchors to expose shortcut reliance, because the gap between anchor and non-spurious accuracy isolates how much performance comes from spurious cues.
Fine-tuning on the synthetic spurious groups, not the original anchors, is what produces large gains on unseen spurious correlations, so the synthetic counterfactual images are the active ingredient.
Prompt-level fixes such as chain-of-thought or explicit spurious-awareness are insufficient; only training updates materially change shortcut behavior.
A model that avoids spurious cues pays a measurable accuracy penalty on ordinary samples, implying shortcut use contributes to normal-case performance.
The robustness-accuracy trade-off grows with the proportion of spurious training samples, so the diversity of spurious correlations in the training mix controls how much generalization transfers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the transfer result generalizes beyond these two open-weight models, spurious reliance can be treated as a general attentional failure mode rather than 124 independent memorized correlations, and one diverse counterfactual dataset might serve as a general robustness intervention for other VLMs.
Because the benchmark is seeded from GPT-4o's errors, SpuriVerse is likely biased toward shortcuts that trip GPT-4o; models with different failure modes may need their own error-driven construction, and a fair cross-model comparison would require errors sourced from each model.
A direct test of the meta-skill interpretation would be to fine-tune on spurious groups from only half the categories and check whether held-out gains concentrate in those categories or spread across all six; the paper does not report per-category held-out results.
One testable extension is to compare fine-tuning on synthetic spurious groups with fine-tuning on real anchors matched for size; if synthetic images are the key driver, the transfer should persist even when the real anchors are withheld from training.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about 2506.18322: the benchmark is a real contribution, and the headline fine-tuning result is not yet supported by the controls in the paper. The 40.70% 'random' baseline is also mislabeled, though that is fixable.

What is new: the authors build SpuriVerse bottom-up from GPT-4o's errors on real VQA benchmarks, ask an LLM to attribute errors to spurious features, have humans refine those, then use Stable Diffusion counterfactuals to keep only samples where a model's accuracy drops by at least 30% when the spurious feature is present. That gives 124 distinct spurious correlation types with real anchors plus synthetic counterparts. It is a more realistic and diverse instrument than Waterbirds or CelebA. Evaluating 15 LVLMs, they find all score low; prompting (CoT, spurious-aware) does not fix it; and fine-tuning on a diverse set of spurious examples improves held-out performance, with a clear trade-off on non-spurious data. They report 5 seeds and full hyperparameters.

The soft spots, in proportion:

- The 'random guess' claim is wrong as stated. With 4-choice questions, uniform random is 25%, not 40.70%. The 40.70% looks like a majority-class baseline. Several models exceed 25%, so 'all models are worse than random' is false under uniform random; it only holds against majority class. Easy to fix, but the current wording misleads.

- The 78.40% meta-skill claim is missing the right control. Fine-tuning on the spurious group is compared to fine-tuning on the non-spurious set, which is real images from the source benchmarks. That confounds two variables: the spurious feature and the synthetic-image distribution. The authors already created the matched control—the core group images from Step 4—but did not fine-tune on them. If core-group fine-tuning also gets near 78%, the 'spurious emphasis' story collapses into a generic synthetic-data effect. This is the experiment I would want before believing the abstract's 'learn to avoid shortcuts' framing.

- The held-out anchor test set is 25 items, so 78.40% vs 59.20% is about five correct answers. The spurious-group test set (250 items) is more stable, but the headline anchor number is noisy.

- The benchmark is partly self-selecting: samples are kept only when some model shows a 30% gap, so 'models fail on SpuriVerse' is partly by construction. Acceptable for a benchmark, but the paper should say so plainly rather than presenting it as an independent finding.

The authors do not release the dataset in the preprint, which hurts reproducibility. Fixing that and adding the core-group control would make the fine-tuning claim credible.

Who it is for: people building or testing LVLM robustness, and anyone working on shortcut learning in multimodal settings. I would send it to review with a request for the control and baseline correction. It is not ready as-is, but it is worth one serious round.

Referee Report

4 major / 5 minor

Summary. The paper introduces SpuriVerse, a benchmark of 124 spurious-correlation types extracted from real-world VQA benchmarks by starting from GPT-4o errors, filtering through LVLM/human verification, and generating paired Stable Diffusion counterfactual images with and without the suspected spurious attribute. Each type contributes 1 real anchor and 10 synthetic spurious-group images, yielding 1,364 multiple-choice questions. The authors evaluate 15 open- and closed-source LVLMs, report that even the best model scores below their stated random baseline, test prompting strategies, and fine-tune Llama-3.2-11B and Qwen2.5-VL-7B on anchors, spurious groups, non-spurious samples, and a mixed set. The headline claim is that fine-tuning on synthetic examples that emphasize spurious correlations raises held-out anchor accuracy to 78.40% on previously unseen spurious-correlation types, suggesting a learnable meta-skill to ignore shortcuts, at the cost of a trade-off on non-spurious samples.

Significance. If the central claims hold, SpuriVerse would be a useful complement to existing spurious-correlation benchmarks because it targets diverse, naturally occurring correlations in general VQA settings rather than a single contrived task, and the cross-type fine-tuning result would be an interesting and falsifiable finding. The paper also has clear strengths: the curation pipeline combines model-error mining, human verification, and synthetic counterfactual validation; the evaluation spans 15 models including closed-weight frontier systems; and the appendix gives substantial reproducibility detail on prompts, hyperparameters, and compute. However, the current evidence does not yet support the headline generalizations. The reported 'random' baseline is inconsistent with the 4-choice format, the benchmark selection criterion partly builds in the failure it later reports, the headline fine-tuning comparison lacks a matched synthetic control, and the main fine-tuning experiment does not document a type-level train/test split. These issues are addressable, so the manuscript merits major revision rather than rejection.

major comments (4)

[Section 3.1, Table 2] The 'Random' row reports 40.70% in all three columns, but SpuriVerse uses four-option multiple-choice questions, for which uniform random guessing gives 25%. The paper never defines how the 40.70% baseline is computed. If it is a majority-class or answer-frequency baseline, calling it 'random guess' is misleading, and the repeated claim that all LVLMs are worse than random (Abstract and Section 3.1) is unsupported; if it is intended as uniform random, the number is wrong. This matters because several model accuracies (e.g., 37.90%) are above 25% but below 40.70%, so the baseline choice changes the qualitative conclusion. Please specify the baseline construction and revise the claims accordingly.
[Section 2.2, Step 5] The benchmark is constructed by retaining samples only when max over {GPT-4o, Gemini 2.0 Flash, Qwen-VL-Max} of [Acc(core) - Acc(spurious)] is at least 30%. The later observation that models perform poorly on the spurious group is therefore partly a restatement of the selection rule: every retained sample is guaranteed to trip at least one of the three selector models. Evaluating other models on these samples is still informative, but the manuscript should state this explicitly and should report sensitivity of the main conclusions to the 30% threshold and to the choice of the error-source model. In addition, because the core and spurious groups are generated from different scene descriptions, the 30% gap may reflect uncontrolled differences in image composition or visual complexity rather than the targeted spurious attribute alone.
[Section 3.3, Table 4] The headline fine-tuning result lacks a matched control. The 'non-spurious set' is sampled randomly from the source benchmarks and differs from the spurious group in both the presence of the spurious attribute and the synthetic/stylized image distribution, including the repeated question and instruction template shared by all synthetic group samples. The core groups described in Section 2.2, Steps 3-4 are the natural matched control: they use the same Stable Diffusion pipeline, the same scene-description prompts, and the same questions, with only the spurious attribute removed. The paper should fine-tune on the core groups with identical splits and hyperparameters. Without this control, the 78.40% result does not establish that the model learned to suppress spurious cues; it could reflect generic adaptation to the synthetic VQA distribution.
[Section 3.3 and Appendix B.4] The claim that fine-tuning generalizes to previously unseen spurious correlations requires that the train/test split be made by spurious-correlation type, so that no test anchor has its corresponding 10 spurious-group images in the training set. Section 3.4 explicitly states that splits are performed by type 'to ensure we are measuring generalization to unseen cases,' but the main fine-tuning setup in Section 3.3 and Appendix B.4 only says the anchor set and spurious groups are divided 70/10/20, without specifying the splitting unit. If the split is sample-level, a test anchor can share its spurious-correlation type with spurious-group images used for training, leaking the exact pattern the paper claims is unseen. Please specify the splitting unit and, if sample-level splitting was used, rerun the experiment with type-level splits.

minor comments (5)

[Abstract and Table 2] The Abstract says the best model achieves only 37.1% accuracy, but Table 2 reports the best anchor accuracy as 37.90% (llama-3.2-11b and qwen-2.5-32b); these numbers should be reconciled.
[Figure 3b and Table 2] There are typos and naming inconsistencies: 'Contextural cues' should be 'Contextual cues,' and model names such as 'qwen-2.5-7b' are inconsistent with the 'Qwen2.5-vl-7b-instruct' used elsewhere.
[Section 1 and Appendix B.2] Minor wording and typographical issues include 'No prompt-based methods such as as Chain-of-Thought' (duplicated 'as') and 'sytem prompt' in Appendix B.2.
[Appendix A.2] The annotation interface description contains 'clickong' instead of 'clicking,' and the appendix does not report inter-annotator agreement despite asserting that human annotations were done by two contributors.
[Section 3.3] The sentence 'Its accuracy drops 66.48% when finetuned on anchors' should read 'drops to 66.48%,' since the table shows a drop from 73.44% to 66.48%.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SpuriVerse's selection rule is construct validation, and the fine-tuning meta-skill claim is not forced by the inclusion criterion.

full rationale

The claimed derivation chain in this paper is a benchmark-construction pipeline followed by an intervention study, not a prediction derived from fitted inputs. Step 5 of the curation pipeline retains samples only when at least one of GPT-4o, Gemini 2.0 Flash, or Qwen-VL-Max shows a core-versus-spurious accuracy gap of at least 30%. This is an operational definition and validation of 'spurious correlation' for the benchmark, rather than a hidden circular reduction: the paper's headline finding of low model accuracy is measured on the original anchor images for fifteen models, most of which were not used in the selection rule, and the selection metric (core-group minus spurious-group accuracy) is not the same quantity as the reported anchor accuracy. GPT-4o's low anchor accuracy is unsurprising because anchors are seeded from GPT-4o errors, but the paper does not base its central contribution on that single fact; the cross-model ranking and the held-out fine-tuning results are empirically determined and are not guaranteed by any equation in the paper. The fine-tuning comparison does contain a genuine confound: the spurious-group images are synthetic and share a generation pipeline, while the 'non-spurious' control is sampled from real source benchmarks, so the improvement to 78.40% does not cleanly isolate the spurious attribute as the active training signal. That is an experimental-design and correctness risk, not a circularity, and the paper's own core-group images would be the natural matched control. Appendix C also candidly acknowledges the GPT-4o-centric curation bias. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in through citation. The central generalization claim therefore retains independent content, and no circular step meeting the quoted-evidence bar is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

There are no fitted model parameters; the free parameters are hand-chosen curation constants. The axioms are the operational and transfer assumptions that make the benchmark interpretable; the transfer assumption is the most consequential for the fine-tuning claim.

free parameters (4)

Epsilon gap threshold = 0.30
Step 5 keeps a sample only when max over three models of core-minus-spurious accuracy is at least 30%; this choice determines which correlation types enter the benchmark.
Synthetic images per group = 10
Each anchor gets 10 Stable Diffusion images per group; this fixed group size affects the precision of the accuracy-gap estimate.
Top-k spurious attributes = 2
The Step 2 prompt asks GPT-4o for the top two spurious attributes per error, bounding the hypothesis space for the generated counterfactuals.
Error-source model = GPT-4o
The initial error pool and spurious attributions come from GPT-4o only, so SpuriVerse is limited to spurious correlations that fool this specific model.

assumptions (4)

domain assumption A >=30% accuracy gap between the core and spurious groups implies the spurious feature drives model error.
Step 5 uses this threshold as the operational definition of a valid spurious sample without a statistical significance test.
domain assumption Stable Diffusion counterfactual pairs differ only in the targeted spurious feature.
Human verification checks that images match descriptions, but no automated control ensures no other visual properties change between groups (Section 2.2, Step 4).
domain assumption GPT-4o can reliably introspect its own errors and name the spurious attribute.
Step 2 relies on GPT-4o's self-diagnosis, with human annotators as a partial check; systematic blind spots in GPT-4o could bias the benchmark.
domain assumption Fine-tuning on synthetic images teaches a generalizable skill for real anchor images.
Section 3.3 assumes that improvements on synthetic spurious groups transfer to real held-out anchors, which is the paper's main generalization claim.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?." pith.science (2026). https://pith.science/paper/E62S6U76

@misc{pith2026250618322,
  author       = {Pith},
  title        = {Pith review of: Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E62S6U76}},
  note         = {Machine review of arXiv:2506.18322}
}

read the original abstract

Finetuning can cause spurious correlations to arise between non-essential features and the target labels, but benchmarks to study these effects involve contrived settings and narrow tasks. In contrast, we consider spurious correlations in multi-modal Large Vision Language Models (LVLMs) pretrained on extensive and diverse datasets without explicit task supervision. We develop a benchmark by sourcing GPT-4o errors on real-world visual-question-answering (VQA) benchmarks, then curating a subset through LVLM-human annotation and synthetic counterfactual evaluation to identify errors caused by spurious correlations. This process yields SpuriVerse, a novel benchmark comprised of 124 distinct types of spurious correlations extracted from real-world datasets, each containing 1 realistic and 10 synthetic VQA samples for a total of 1364 multiple choice questions. We evaluate 15 open and closed-source LVLMs on SpuriVerse, finding that even state-of-the-art closed-source models struggle significantly, achieving at best only 37.1% accuracy. Fine-tuning on synthetic examples that emphasize the spurious correlation improves performance to 78.40%, suggesting that training on diverse spurious patterns generalizes to unseen situations: models appear to learn to avoid "shortcuts" and attend to the overall image context.

Figures

Figures reproduced from arXiv: 2506.18322 by the authors.

**Figure 1.** SpuriVerse consists of 124 distinct spurious correlations, each of which is comprised of 1 original example and 10 synthetic images. Both original and synthetic images share the same multiple-choice question. Each image contains a feature that is spuriously correlated with one of the choices, causing the model to make an error. and without spurious features. These findings suggest a relationship between spurious fea… view at source ↗

**Figure 2.** Curating SpuriVerse consists of 5 steps: (1) Derive errors from GPT-4o on a set of multi-modal multiple choice question benchmarks. (2) Two-stage Verification: (a) Prompt GPT-4o to identify errors attributable to spurious correlation and report the candidate spurious feature, and (b) Human verify the attribution and refine the spurious features if necessary. (3) Prompt GPT-4o to generate two scene descriptions based… view at source ↗

**Figure 3.** Categories of spurious correlations. Visual predominance and Object co-occurrence make up 37.1%, 33.1% of SpuriVerse, respectively. Only 2 spurious correlations fall under Spatial relationships. LVLMs tend to do better on Visual resemblance and Spatial relationships, achieving more than 50% accuracy, and achieve less than 50% accuracy on all other categories. 3 Experiments In this section, we first present a compreh… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: shows that, as the number of spurious correlations increases, both Llama-3.2-11B-visioninstruct and Qwen2.5-vl-7b-instruct’s accuracies increase on both the anchor set and the spurious groups, suggesting that diversity of spurious correlation can indeed improve genera…

**Figure 5.** Figure 5: Annotation interface for refining spurious attributes and image descriptions. The curation pipeline consists of 5 steps: Step 1 displays the image and question from the error set. Step 2 allows annotators to view extracted spurious attributes by clicking on From store …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 5 canonical work pages

[3]

Towards mitigating more challenging spurious correlations: A benchmark & new datasets

Siddharth Joshi, Yu Yang, Yihao Xue, Wenhan Yang, and Baharan Mirzasoleiman. Towards mitigating more challenging spurious correlations: A benchmark & new datasets. arXiv preprint arXiv:2306.11957,

arXiv
[8]

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach

URL https://arxiv.org/abs/2303.08774. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,

arXiv
[9]

Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731,

arXiv 1911
[10]

a model with good accuracy on SpuriVerse can be free from all 18 potential spurious correlation attacks,

As a toy example, suppose Sanchor ={s1,s 2,s 3,s 4,s 5}. Let {s1,s 2,s 3,s 4} be the training split at first, and each of them comes from a distinct benchmark we use. Let the second split between spurious and non-spurious yieldsStrain, spurious ={s1,s 2},Strain, non-spurious ={s3,s 4} withr = 50%. Then, the next step will yield a training set of 40 sample...

work page 2022
[11]

InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Models

Bingbing Wen, Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Bill Howe, and Lijuan Wang. Infovisdial: An informative visual dialogue dataset by bridging large multimodal and language models. arXiv preprint arXiv:2312.13503,

work page Pith review arXiv
[12]

A broad-coverage challenge corpus for sentence understanding through inference

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long Papers), pag...

work page 2018
[13]

doi: 10.18653/v1/N18-1101

Association for Computational Linguistics. doi: 10.18653/v1/N18-1101. URL https://aclanthology. org/N18-1101/. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Brew, Alexis Nikolov, Sudhanshu Goyal, Yoann Dreyer, Julien Chaumond, and Alexander M. Rush. Tra...

doi:10.18653/v1/n18-1101
[14]

doi: 10.18653/v1/2020.emnlp-demos.6

Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6. Yiwei Yang, Anthony Z Liu, Robert Wolfe, Aylin Caliskan, and Bill Howe. Label-efficient group robustness via out-of-distribution concept curation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

doi:10.18653/v1/2020.emnlp-demos.6 2020

Show all 20 references

[15]

Mmmg: a comprehensive and reliable evaluation suite for multitask multimodal generation

Jihan Yao, Yushi Hu, Yujie Yi, Bin Han, Shangbin Feng, Guang Yang, Bingbing Wen, Ranjay Krishna, Lucy Lu Wang, Yulia Tsvetkov, et al. Mmmg: a comprehensive and reliable evaluation suite for multitask multimodal generation. arXiv preprint arXiv:2505.17613,

arXiv
[17]

Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations

Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.01517,

arXiv
[18]

In step 5, annotators can take a peek at models’ evaluations on one image for spurious group and one image for core group

Annotators can go back to step 3 and refine the descriptions if the images generated are not faithful. In step 5, annotators can take a peek at models’ evaluations on one image for spurious group and one image for core group. (a) Step 1 (b) Steps 2-5 Figure 5: Annotation inter...
[300]

gpt-4o-2024-08-06

All the open-sourced models are 4-bit quantized during evaluation. Versions For GPT-4o, we used version “gpt-4o-2024-08-06”. For GPT-4o-mini, we used version “gpt-4o-mini-2024-07-18”. For Claude 3.7 Sonnet, we used version “claude-3-7-sonnet-20250219”. B.4 Finetuning details W...

2024
[2015]

Spawrious: A benchmark for fine control of spurious correlation biases

Aengus Lynch, Gbètondji JS Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A benchmark for fine control of spurious correlation biases. arXiv preprint arXiv:2303.05470,

arXiv
[2018]

Naturalbench: Evaluating vision-language models on natural adversarial samples

Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision-language models on natural adversarial samples. arXiv preprint arXiv:2410.14669, 2024a. Bohao Li,...

arXiv
[2020]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

arXiv
[2021]

Feature-wise bias amplification

Klas Leino, Emily Black, Matt Fredrikson, Shayak Sen, and Anupam Datta. Feature-wise bias amplification. arXiv preprint arXiv:1812.08999,

arXiv
[2022]

Salient imagenet: How to discover spurious features in deep learning? arXiv preprint arXiv:2110.04301,

Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? arXiv preprint arXiv:2110.04301,

arXiv
[2023]

Last layer re-training is sufficient for robustness to spurious correlations

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937,

arXiv
[2024]

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz

URL https://www.anthropic.com. Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893,

1907 arXiv
[2025]

Mm-spubench: Towards better understanding of spurious biases in multimodal llms

11 Wenqian Ye, Guangtao Zheng, Yunsheng Ma, Xu Cao, Bolin Lai, James M Rehg, and Aidong Zhang. Mm-spubench: Towards better understanding of spurious biases in multimodal llms. arXiv preprint arXiv:2406.17126,

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [3]

Towards mitigating more challenging spurious correlations: A benchmark & new datasets

Siddharth Joshi, Yu Yang, Yihao Xue, Wenhan Yang, and Baharan Mirzasoleiman. Towards mitigating more challenging spurious correlations: A benchmark & new datasets. arXiv preprint arXiv:2306.11957,

arXiv

[2] [8]

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach

URL https://arxiv.org/abs/2303.08774. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,

arXiv

[3] [9]

Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731,

arXiv 1911

[4] [10]

a model with good accuracy on SpuriVerse can be free from all 18 potential spurious correlation attacks,

As a toy example, suppose Sanchor ={s1,s 2,s 3,s 4,s 5}. Let {s1,s 2,s 3,s 4} be the training split at first, and each of them comes from a distinct benchmark we use. Let the second split between spurious and non-spurious yieldsStrain, spurious ={s1,s 2},Strain, non-spurious ={s3,s 4} withr = 50%. Then, the next step will yield a training set of 40 sample...

work page 2022

[5] [11]

InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Models

Bingbing Wen, Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Bill Howe, and Lijuan Wang. Infovisdial: An informative visual dialogue dataset by bridging large multimodal and language models. arXiv preprint arXiv:2312.13503,

work page Pith review arXiv

[6] [12]

A broad-coverage challenge corpus for sentence understanding through inference

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long Papers), pag...

work page 2018

[7] [13]

doi: 10.18653/v1/N18-1101

Association for Computational Linguistics. doi: 10.18653/v1/N18-1101. URL https://aclanthology. org/N18-1101/. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Brew, Alexis Nikolov, Sudhanshu Goyal, Yoann Dreyer, Julien Chaumond, and Alexander M. Rush. Tra...

doi:10.18653/v1/n18-1101

[8] [14]

doi: 10.18653/v1/2020.emnlp-demos.6

Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6. Yiwei Yang, Anthony Z Liu, Robert Wolfe, Aylin Caliskan, and Bill Howe. Label-efficient group robustness via out-of-distribution concept curation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

doi:10.18653/v1/2020.emnlp-demos.6 2020