Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Changing only the scene behind a person shifts what vision-language models say about them, from offered salary to toxic completions; this paper builds the 60,000-image counterfactual dataset that demonstrates it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Cultural Counterfactuals — same person placed in different cultural contexts — shows that LVLMs vary salary, rent, and character judgments with the depicted religion, nationality, and income level.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A solid, large-scale counterfactual dataset for cultural-context bias in LVLMs, but the 'same person' premise is under-verified; the bias attribution is conditional on identity-preservation checks. the 3 major comments →

arxiv 2603.02370 v2 pith:JBZBL5KZ submitted 2026-03-02 cs.CV

Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples

Phillip Howard , Xin Su , Kathleen C. Fraser This is my paper
classification cs.CV
keywords cultural biascounterfactual imageslarge vision-language modelssocioeconomic statusreligious contextnationality stereotypesbias evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cultural Counterfactuals aims to prove that large vision-language models carry cultural stereotypes triggered by the scene behind a person, not by the person's appearance: same face, pose, and clothing, different religious, national, or socioeconomic backdrop. The authors construct nearly 60,000 counterfactual images — sets in which one synthetic person is digitally placed into real photographs of six places of worship, eight country scenes, and three income settings — and ask five open-source vision-language models for salary, rent, keyword, refusal, and sentence-completion responses. Across 9.07 million generations, they find statistically significant output shifts: salaries rise or fall by thousands of dollars depending on depicted country; mosque backdrops trigger classification refusals in some models; high-toxicity completions cluster on specific contexts; and stereotype topics such as "manufacturing weapons of mass destruction" co-occur with mosque contexts, amplified when the person looks Middle Eastern. The point is that cultural context alone moves an LVLM's judgment, so bias evaluation that only looks at race, gender, or age misses a whole axis of bias.

Core claim

The discovery is a method and a measurement: cultural bias in vision-language models can be isolated by generating counterfactual image sets — the same individual placed in different real cultural settings — and then comparing outputs within each set. The central finding is that LVLM outputs are systematically conditioned by the depicted cultural context. For example, in one model's salary estimates, mean offers differ by close to $5,000 above and below the set mean depending on the country, with higher offers for Germany, France, the US, and China and lower for Brazil, India, Morocco, and South Africa. For socioeconomic contexts, several models pay people depicted in high-income scenes the

What carries the argument

The central object is the counterfactual set: a group of images showing the same synthetic person inserted into every context within a cultural dimension (six religions, eight countries, three income levels). To build it, the authors start from real photographs with human-annotated cultural labels, generate 7,200 person images with diverse combinations of race, age, and gender, and use a latent image-editing model with the instruction "put the person in the scene" to merge each person into each context. An iterative pipeline filters failures by visual similarity to both source images and by re-checking, with a separate vision-language model after removing the foreground person, that the cont

Load-bearing premise

The load-bearing assumption is that the person is truly the same across contexts — identical facial identity, pose, clothing, and lighting, with only the background changing. The construction pipeline checks visual similarity to the source person and context images and re-checks context recognizability, but it never directly verifies identity, and the paper itself notes that inserted figures can be incongruent in context (for example, women with bare heads and shoulders insid

What would settle it

Run independent face-recognition verification comparing each counterfactual person image against the original person image, stratified by context. If verification match rates differ systematically across contexts — for example, if women placed in mosques are altered more than men in mosques — the inference that output differences measure background rather than person fails. Equally, ablating the context cue (blurring or replacing only the background while keeping the person untouched) should eliminate the output differences if the effect is truly conditional on the depicted context.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Hiring and housing decisions generated by vision-language models are context-dependent: the same person can be offered materially different salaries and rent solely because of the cultural scene behind them, with implications for any automated screening use.
  • Refusal behavior is itself biased: some models refuse to answer requests only when a mosque is depicted in the background, and arrest-prompt refusals rise with socioeconomic status, so safety guardrails can encode stereotypes.
  • Toxic generations are skewed by context: high-toxicity completions cluster on particular religious or income contexts rather than being random, meaning content moderation and toxicity audits should control for background cues.
  • Cultural context and appearance interact: stereotype-consistent combinations (mosque background plus Middle Eastern person) amplify harmful completions, while even mismatched combinations are shifted by context alone.
  • Cultural awareness is a precondition for observable context bias: models that cannot recognize the context do not exhibit context-conditioned output differences, so bias benchmarks should measure awareness alongside bias.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: because identity was not directly verified, some of the measured context shift may be appearance drift — clothing, pose, or artifacts introduced by the image-editing pipeline. Adding an independent face-identity check would tell which portion of the effect is truly contextual.
  • Inference: the awareness/bias pairing suggests an inverse relationship — improving a model's cultural recognition may initially increase measured stereotype bias before safety training catches up; evaluating models in the awareness-rich regime is therefore the honest stress test.
  • Inference: the same counterfactual protocol could be pointed at other protected or incidental cues (disability markers, language scripts on signs, urban versus rural scenes) and at interventions (context-robust prompting, refusal calibration), making the dataset a general harness for context bias.
  • Inference: the mosque-specific refusal pattern implies that safety guardrails fire on the depicted place of worship itself; monitoring refusal rates by context is a cheap, high-signal audit that should be part of deployment evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Cultural Counterfactuals, a synthetic dataset of nearly 60k images organized into counterfactual sets that place the same generated person into different real photographic contexts for religion, nationality, and socioeconomic status. The authors evaluate five open-source LVLMs with prompts eliciting context classification, salary/rent numbers, five keywords, sentence completions, and refusal behavior, then analyze sensitivity, within-set numerical deviations, toxicity, and topical content. The central claim is that LVLM outputs vary systematically with the depicted cultural context, revealing cultural biases not visible from the person's appearance alone. The dataset and code are released publicly.

Significance. If the counterfactual manipulation is valid, this is a valuable and timely contribution: it targets under-studied bias dimensions, uses real context photographs rather than fully synthetic scenes, includes intersectional person attributes, and evaluates 9.07M outputs from five open models. The multi-stage filtering, regeneration loop, and human validation are concrete steps toward dataset quality. The main scientific value is the counterfactual design, which promises controlled measurement of context-conditioned behavior; the public release of the dataset and evaluation code would allow the community to reuse and stress-test the resource. However, the causal interpretation depends on a load-bearing assumption—that only the cultural background changes across images in a set—and the current evidence for that assumption is incomplete.

major comments (3)
  1. [§3.2, Appendix A.4, Impact Statement] Identity preservation across contexts is not verified. The only quantitative check is CLIP image similarity ≥0.85 between the generated image and the person-only source. CLIP similarity is a global semantic similarity and does not establish that facial identity, clothing, pose, or lighting are held constant. The human validation in Appendix A.4 verifies age, race, and gender, but not identity, attire, pose, or other presentational attributes that can directly affect the salary, rent, and keyword prompts. The Impact Statement itself concedes systematic appearance changes, e.g., 'Middle Eastern' people in hijabs/keffiyehs and incongruities like bare-headed women inside a mosque. Because the central claim in Section 1 requires that output differences be caused solely by the cultural background, the observed variation may be confounded with attire or other appearance changes. The authors sho
  2. [§3.1, §3.2, Table 1] The cultural-awareness filtering is performed with Qwen2.5-VL-32B-Instruct, and Qwen2.5-VL-7B-Instruct is also the best-performing evaluated model on context classification. Specifically, both source image filtering and post-merge Context Detectability Filtering use Qwen2.5-VL-32B to decide whether a context is recognizable, so the dataset is curated to be legible to the Qwen model family. Qwen2.5-VL-7B's high accuracy and high seed stability are therefore partly by construction. This does not invalidate the bias measurements, which use model outputs rather than the selected labels, but it weakens cross-model comparisons and the paper's general claim about 'popular LVLMs.' The authors should either use human labels or an independent oracle for the filtering, or report context-awareness results separately on unfiltered image sets and on per-model recognizable subsets.
  3. [§4.2.5, Appendix F, §4.2.6] Several headline bias findings are driven by a small number of high-toxicity examples from specific model–prompt–context combinations, especially Molmo-7B on Arrest/Bad Influence. The qualitative topic examples (e.g., 'manufacturing weapons of mass destruction' for Mosque contexts) are compelling but are based on a subset of generations selected for high MaxToxicity, and the paper does not report confidence intervals or multiple-comparison controls for the topic-frequency skewness. Given that the paper draws conclusions such as 'the frequency of this particular bias is stronger when race and cultural attributes intersect,' the authors should provide uncertainty estimates for these proportions or at least state clearly that the topic-model section is illustrative rather than a quantitative claim.
minor comments (6)
  1. [§1] Typo: 'pubilcly' should be 'publicly'.
  2. [Table 1, Sections 4.2.2–4.2.4] Model names are inconsistent: 'LLaV A-v1.6' vs 'LLaVA-v1.6' vs 'llava-v1.6' vs 'Llava'; 'Molmo-7B' vs 'Molmo-7B-D-0924'. Please standardize.
  3. [§4.2.4] The text says 'both Llama and Gemma show a different trend,' but no Llama model is evaluated; this should be 'Llava and Gemma.'
  4. [Appendix D, Figures 13–14] The captions note that Gemma data for Religion and Socioeconomic dimensions are unavailable, but the reason is not explained. Please state whether this is due to refusal rates, data loss, or a filtering decision.
  5. [Appendix C.2] The leakage-removal procedure removes label names and 'direct synonyms,' but the exact synonym lists are not provided. Please include them for reproducibility, especially because leakage rates up to 6% remain.
  6. [§4.2.3, Appendix C.4] For low-awareness models, the correctness-filtered subsets are extremely small (e.g., Molmo-7B has N=1 majority-correct religious set and no majority-correct nationality sets). The paper acknowledges this, but the figure still plots these points; it would be clearer to omit or threshold them.

Circularity Check

1 steps flagged

Partial circularity in the cultural-awareness measurement via Qwen-based context filtering; main bias estimates remain model outputs and are not circular.

specific steps
  1. fitted input called prediction [Section 3.2 (Context Detectability Filtering) and Table 1]
    "to avoid any potential influence that the depicted person could have on the context classification results, we first utilize RMBG-2.0 ... to remove the foreground (i.e., the person) from the counterfactual image and then prompt Qwen2.5-VL-32B-Instruct to classify the cultural context. Images which have incorrectly classified contexts are then discarded."

    The Context Detectability Filter defines the surviving dataset as images whose cultural context Qwen2.5-VL-32B can classify correctly. The cultural-awareness result (Table 1) then reports classification accuracy on this same filtered set; Qwen2.5-VL-7B, from the same model family, is measured on a set constructed to be legible to Qwen2.5-VL-32B. Thus the high Qwen accuracy is partly a self-fulfilling selection effect rather than independent evidence. However, the main bias findings (salary/rent deviations, toxicity, topics) are LVLM outputs and are not derived from the filter labels, so the central claim retains substantial independent content.

full rationale

The paper's central bias measurements—salary/rent deviations, MaxToxicity, keyword shifts, topic over-representation—are model outputs compared across counterfactual sets, not parameters fitted to the dataset labels. Equation (1) is a definition of within-set mean deviation, not a reduction. The most notable circularity-adjacent issue is the use of Qwen2.5-VL-32B in context-detectability filtering: the evaluation set is selected for Qwen legibility, and Qwen2.5-VL-7B then tops the cultural-awareness table. This is a benchmark-selection effect that partially inflates one supporting result, but it does not by construction determine the salary/rent or toxicity findings, which are the paper's main evidence of cultural bias. The Impact Statement honestly concedes the deeper validity threat: FLUX.1-Kontext-dev may conflate culture, race, and appearance (e.g., hijabs, turbans, bare-headed women in mosques), so the assumption that only cultural context varies is imperfect. That is a confound and an unverified identity-preservation premise (CLIP similarity >= 0.85 is not identity verification), but it is a validity limitation, not a circular derivation. Self-citations to Howard et al. (2024, 2025) supply methodological priors (people-image generation, MaxToxicity, keyword prompts) but do not carry the argument alone. Overall, no central result reduces to its inputs by construction; the partial circularity is confined to the cultural-awareness measure.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on a chain of curation and normative assumptions: source dataset gold labels, Qwen-based context filtering, identity preservation in the fused images, the assumption that context cues transfer to the person, and the premise that salary/rent should be context-invariant. Most are stated, and several are conceded as limitations in the Impact Statement, but they are load-bearing rather than decorative.

free parameters (4)
  • CLIP similarity thresholds = 0.75 (context), 0.85 (person)
    Hand-chosen in Section 3.2 to filter fusion failures; directly determines which counterfactual images survive into the dataset and hence what the models see.
  • Guidance scale for FLUX.1-Kontext-dev = 2.5 initial; sampled from [1,4] for regeneration
    Controls the balance between preserving the person and the context image; chosen by manual inspection, affects image realism and identity preservation.
  • Keyword-union aggregation rule = retain if appears in ≥1 of 3 seeds
    Chosen in Appendix C.2 for the sensitivity analysis; affects the Jaccard-based context sensitivity values.
  • Label leakage removal terms = context label + direct synonyms
    Label-specific leakage removal defined in Appendix C.2; different choices would change the measured keyword sensitivity.
axioms (5)
  • domain assumption The gold labels in the source datasets (Google Landmarks, VIPPGeo, Dollar Street) correctly identify the intended cultural context.
    Used in Appendix A.2 to select context images; if labels are wrong, the counterfactual sets do not depict the claimed cultural differences.
  • ad hoc to paper Qwen2.5-VL-32B's context classifications are a valid oracle for whether a context is recognizable in an image.
    The pipeline filters source images and counterfactual images based on Qwen2.5-VL-32B's classifications (Sections 3.1, 3.2); this both selects the dataset and, because Qwen2.5-VL-7B is an evaluated model, biases the awareness comparison.
  • domain assumption The generated counterfactual images preserve the identity of the person, so output differences are attributable to context rather than appearance changes.
    The paper relies on CLIP similarity >0.85 to the source person image to ensure the person appears, but identity (pose, clothing, facial features) is not directly measured; the Impact Statement concedes incongruent depictions such as women with bare heads in a mosque.
  • domain assumption If an image depicts an individual in a religious, national, or socioeconomic context, the LVLM will associate that cultural trait with the individual.
    Explicitly stated in the Impact Statement as a 'core assumption'; the paper notes it 'may not always be valid or even desirable'.
  • domain assumption For the salary and rent prompts, the ideal response should not vary with image context; any deviation indicates bias.
    This normative premise underlies Eq. (1) in Section 4.1.3; a model might reasonably adjust salary for the depicted country's cost of living, which would be counted as bias.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples." pith.science (2026). https://pith.science/paper/JBZBL5KZ

@misc{pith2026260302370,
  author       = {Pith},
  title        = {Pith review of: Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JBZBL5KZ}},
  note         = {Machine review of arXiv:2603.02370}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Vision-Language Models (LVLMs) have grown increasingly powerful in recent years, but can also exhibit harmful biases. Prior studies investigating such biases have primarily focused on demographic traits related to the visual characteristics of a person depicted in an image, such as their race or gender. This has left biases related to cultural differences (e.g., religion, socioeconomic status), which cannot be readily discerned from an individual's appearance alone, relatively understudied. A key challenge in measuring cultural biases is that determining which group an individual belongs to often depends upon cultural context cues in images, and datasets annotated with cultural context cues are lacking. To address this gap, we introduce Cultural Counterfactuals: a high-quality synthetic dataset containing nearly 60k counterfactual images for measuring cultural biases related to religion, nationality, and socioeconomic status. To ensure that cultural contexts are accurately depicted, we generate our dataset using an image-editing model to place people of different demographics into real cultural context images. This enables the construction of counterfactual image sets which depict the same person in multiple different contexts, allowing for precise measurement of the impact that cultural context differences have on LVLM outputs. We demonstrate the utility of Cultural Counterfactuals for quantifying cultural biases in popular LVLMs.

Figures

Figures reproduced from arXiv: 2603.02370 by Kathleen C. Fraser, Phillip Howard, Xin Su.

Figure 1
Figure 1. Figure 1: A counterfactual set depicting the same person in different socioeconomic contexts. See Figures 4, 5, and 6 for more examples. to religion, nationality, and socioeconomic status. Images in Cultural Counterfactuals are organized into counterfactual sets, where images in each set depict the same person in different cultural contexts (e.g., Figures 1, 4, 5, 6). This enables precise measurement of the impact t… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of our counterfactual generation approach. A source cultural context and person image are concatenated (a) and input to FLUX.1-Kontext with the prompt “Put the person in the scene”, resulting in a counterfactual image merging the two source images (b). images. These images were acquired from three existing datasets with labels for religion, nationality, and socioeco￾nomic cultural contexts (se… view at source ↗
Figure 3
Figure 3. Figure 3: Qwen2.5-VL Salary Deviation by Nationality 4.2.5. MAXTOXICITY [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example of a counterfactual set from our dataset depicting the same subject in six different religious contexts. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example of a counterfactual set from our dataset depicting the same subject in three different socioeconomic contexts. (a) Brazil (b) China (c) France (d) Germany (e) India (f) Morocco (g) South Africa (h) United States [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of a counterfactual set from our dataset depicting the same subject in eight different nationality contexts. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Example of a failed counterfactual image generation where the source context image does not appear in the generated counterfactual image [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: illustrates a case where FLUX.1-Kontext-dev failed to successfully merge the source context and people images. In this example, the generated counterfactual image only depicts the source context image without the person. This was identified based on the CLIP image similarity between the source person image and the counterfactual image, which was 0.716. Since this falls below our minimum required threshold … view at source ↗
Figure 9
Figure 9. Figure 9: Example of a failed counterfactual image generation where the cultural context is not recognizable in the counterfactual image. B. Methodology Here, we provide additional information on the prompts, models, and other methodological details of our experiments. B.1. Models We evaluate five popular open-source LVLMs, summarized in [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Bias evidence map across three cultural dimensions. The x-axis is majority-vote context classification accuracy (cultural awareness), and the y-axis is context sensitivity (1 − mean pairwise Jaccard). Marker size encodes seed stability (mean Jaccard overlap across the three keyword generations for the same image). Dashed vertical lines indicate random baselines for each label set size. Correctness-filtere… view at source ↗
Figure 11
Figure 11. Figure 11: Sensitivity under stricter correctness filters. We recompute context sensitivity on (i) all complete sets, (ii) sets where every image is majority-correctly classified, and (iii) sets where every image is unanimously correctly classified. Marker styles indicate when N is small, highlighting slices where filtered sensitivity should be interpreted cautiously. D.2. Context-Specific Refusal Patterns The avera… view at source ↗
Figure 12
Figure 12. Figure 12: Refusal rates for Classification prompt [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Refusal rates for Arrest prompt. (Note that Gemma data for the Religion and Socioeconomic dimensions are unavailable.) [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Refusal rates for Bad Influence prompt. (Note that Gemma data for the Religion and Socioeconomic dimensions are unavailable.) 24 [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Salary deviations by nationality context 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Salary deviations by socioeconomic context 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Salary deviations by religion context 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Rent deviations by nationality context 28 [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Rent deviations by socioeconomic context 29 [PITH_FULL_IMAGE:figures/full_fig_p029_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Rent deviations by religion context 30 [PITH_FULL_IMAGE:figures/full_fig_p030_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Proportional representation of context groups which exceeded the 95th percentile of MaxToxicity by LVLM. “long-tail” behavior where the occurrence of toxicity varies significantly across social groups at higher percentiles of the MaxToxicity distribution. Motivated by these findings, we report the 95th percentile of MaxToxicity, which characterizes rarer cases of large toxicity differences across contexts… view at source ↗
Figure 22
Figure 22. Figure 22: Case study on the “manufacturing weapons of mass destruction” topic for the Arrest prompt. When a Mosque is depicted, 45.9% of the responses assigned to this topic are for images depicting a Middle Eastern person, while only 9% depict a Latino person. However, in cases where a Latino individual is depicted in the image, 59.5% of the response assigned to this topic have a Mosque context. most ∼ 0.06 in P(h… view at source ↗
Figure 23
Figure 23. Figure 23: Lexicon coverage by model, dimension, and keyword prompt (Prompt A vs. Prompt B; Appendix B.2). SCM coverage counts matches to any SCM subdimension (sociability, morality, ability, agency, status), while VAD coverage counts matches to NRC-VAD terms. Grey cells indicate slices where the corresponding prompt/model data is unavailable. Cross-dimension comparison. Across models, religion yields substantially … view at source ↗
Figure 24
Figure 24. Figure 24: SCM projections by religion context label. Each cell is the fraction of matched unique keyword types associated with high warmth or high competence under the SCM lexicon (higher is more “warm”/“competent” language). 37 [PITH_FULL_IMAGE:figures/full_fig_p037_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: NRC-VAD extremes by religion context label. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: NRC-VAD low/negative extremes by religion context label. Christian church Mosque Synagogue Shinto shrine Hindu temple Valence: context-of-min Buddhist temple Prompt A Prompt B Christian church Mosque Synagogue Shinto shrine Hindu temple Arousal: context-of-max Buddhist temple Qwen2.5-VL Gemma-3-12b InternVL3-8B LLaVA-v1.6 Molmo-7B Christian church Mosque Synagogue Shinto shrine Hindu temple Buddhist templ… view at source ↗
Figure 27
Figure 27. Figure 27: Within-set NRC-VAD “context-of-extreme” winner rates for religion. Each cell shows how often a context label is the winner within a counterfactual set for (top) minimum valence and (middle/bottom) maximum arousal/dominance, aggregated over sets. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Within-set context-of-extreme winners (Nationality). Winner rates for argmin valence and argmax arousal/dominance; all sets vs. majority-correct sets. low income middle income high income Valence: context-of-min Prompt A Prompt B low income middle income high income Arousal: context-of-max Qwen2.5-VL Gemma-3-12b InternVL3-8B LLaVA-v1.6 Molmo-7B low income middle income high income Dominance: context-of-ma… view at source ↗
Figure 29
Figure 29. Figure 29: Within-set context-of-extreme winners (Socioeconomic). Winner rates for argmin valence and argmax arousal/dominance; all sets vs. majority-correct sets. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: SCM by context (Nationality). Fraction of matched unique keyword types with high warmth/competence. low income middle income high income Warmth: P(high) Prompt A Prompt B Qwen2.5-VL Gemma-3-12b InternVL3-8B LLaVA-v1.6 Molmo-7B low income middle income Competence: P(high) high income Qwen2.5-VL Gemma-3-12b InternVL3-8B LLaVA-v1.6 Molmo-7B 0.0 0.5 1.0 P(high) SCM by context Socioeconomic [PITH_FULL_IMAGE:f… view at source ↗
Figure 31
Figure 31. Figure 31: SCM by context (Socioeconomic). Fraction of matched unique keyword types with high warmth/competence. 41 [PITH_FULL_IMAGE:figures/full_fig_p041_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: NRC-VAD by context (Nationality; high/positive). Fraction of matched unique keyword types with v > 0.5 and a, d > 0.75. 42 [PITH_FULL_IMAGE:figures/full_fig_p042_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: NRC-VAD by context (Socioeconomic; high/positive). Fraction of matched unique keyword types with v > 0.5 and a, d > 0.75. 43 [PITH_FULL_IMAGE:figures/full_fig_p043_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: NRC-VAD by context (Nationality; low/negative). Fraction of matched unique keyword types with v < −0.5 and a, d < −0.75. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: NRC-VAD by context (Socioeconomic; low/negative). Fraction of matched unique keyword types with v < −0.5 and a, d < −0.75. 0.0 0.2 0.4 0.6 0.8 1.0 Classification accuracy 0 1 2 3 4 Median |log-odds| (q < 0.05) random Religion 0.0 0.2 0.4 0.6 0.8 1.0 Classification accuracy random Nationality 0.0 0.2 0.4 0.6 0.8 1.0 Classification accuracy random Socioeconomic Qwen2.5-VL Gemma-3-12b InternVL3-8B LLaVA-v1.6… view at source ↗
Figure 36
Figure 36. Figure 36: Differential keyword evidence map across dimensions. The x-axis is majority-vote context classification accuracy, and the y-axis summarizes the magnitude of significant differential keyword associations (median |log-odds| at q < 0.05). Error bars show the 5th–95th percentile of |log-odds| across significant (context-label, term) pairs; marker size indicates the number of significant pairs. 45 [PITH_FULL_… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Cross-Cultural Value Attribution in Large Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    LVLMs invert the human SES–Authority value link and override cultural context for Middle Eastern faces across 4.8M generations and nine models.

  2. Cross-Cultural Value Attribution in Large Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 5.0

    LVLMs exhibit sensitivity to depicted cultural contexts when generating value judgments, diagnosed via Moral Foundations Theory, lexical analysis, and counterfactual image sets.

Reference graph

Works this paper leans on

38 extracted references · 8 linked inside Pith · cited by 1 Pith paper

  1. [1]

    M., Wang, J., Tondi, B., and Barni, M

    Alamayreh, O., Dimitri, G. M., Wang, J., Tondi, B., and Barni, M. Which country is this picture from? new data and methods for dnn-based country recognition. arXiv preprint arXiv:2209.02429, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    D., de la Torre, C., and Musumeci, J

    Benjamin, J. D., de la Torre, C., and Musumeci, J. Controlling the incentive problems in real estate leasing. The Journal of Real Estate Finance and Economics, 10 0 (2): 0 177--191, 1995

    1995

  3. [3]

    Black Forest Labs . Flux. https://github.com/black-forest-labs/flux, 2024

    2024

  4. [4]

    Flux.1 kontext: Flow matching for in-context image generation and editing in latent space, 2025

    Black Forest Labs , Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dockhorn, T., English, J., English, Z., Esser, P., Kulal, S., Lacey, K., Levi, Y., Li, C., Lorenz, D., Müller, J., Podell, D., Rombach, R., Saini, H., Sauer, A., and Smith, L. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space, 2025. ...

    Pith/arXiv arXiv 2025

  5. [5]

    L., Barocas, S., Daum \'e III, H., and Wallach, H

    Blodgett, S. L., Barocas, S., Daum \'e III, H., and Wallach, H. Language (technology) is power: A critical survey of ``bias'' in NLP . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 5454--5476, 2020

    2020

  6. [6]

    Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Cahyawijaya, S., Lovenia, H., Moniz, J. R. A., Wong, T. H., Farhansyah, M. R., Maung, T. T., Hudi, F., Anugraha, D., Habibi, M. R. S., Qorib, M. R., Agarwal, A., Imperial, J. M., Patel, H. L., Feliren, V., Nasution, B. I., Rufino, M. A., Winata, G. I., Rajagede, R. A., Catalan, C. R., Imam, M. F. M., Pattnayak, P., Pranida, S. Z., Pratama, K., Bangera, Y....

    2025

  8. [8]

    A stereotype content analysis on color-related social bias in large vision language models

    Choi, J., Kim, M., Hong, Y., and Kim, B. A stereotype content analysis on color-related social bias in large vision language models. arXiv preprint arXiv:2505.20901, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    T., Cuddy, A

    Fiske, S. T., Cuddy, A. J. C., Glick, P., and Xu, J. A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology, 82 0 (6): 0 878--902, 2002

    2002

  10. [10]

    Fraser, K. C. and Kiritchenko, S. Examining gender and racial bias in large vision--language models using a novel dataset of parallel images. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 690--713, 2024

    2024

  11. [11]

    The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world

    Gaviria Rojas, W., Diamos, S., Kini, K., Kanter, D., Janapa Reddi, V., and Coleman, C. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. Advances in Neural Information Processing Systems, 35: 0 12979--12990, 2022

    2022

  12. [12]

    Revealing and reducing gender biases in vision and language assistants ( VLAs )

    Girrbach, L., Alaniz, S., Huang, Y., Darrell, T., and Akata, Z. Revealing and reducing gender biases in vision and language assistants ( VLAs ). In The Thirteenth International Conference on Learning Representations, 2025

    2025

  13. [13]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure

    Grootendorst, M. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794, 2022

    Pith/arXiv arXiv 2022

  14. [14]

    M., Abrantes, F

    Hall, S. M., Abrantes, F. G., Zhu, H., Sodunke, G., Shtedritski, A., and Kirk, H. R. VisoGender : A dataset for benchmarking gender bias in image-text pronoun resolution. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=BNwsJ4bFsc

    2023

  15. [15]

    I., Knott, M., Camerer, C

    Hausladen, C. I., Knott, M., Camerer, C. F., and Perona, P. Social perception of faces in a vision-language model. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 639--659, 2025

    2025

  16. [16]

    L., Bhiwandiwalla, A., and Lal, V

    Howard, P., Madasu, A., Le, T., Moreno, G. L., Bhiwandiwalla, A., and Lal, V. Socialcounterfactuals: Probing and mitigating intersectional social biases in vision-language models with counterfactual examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11975--11985, 2024

    2024

  17. [17]

    C., Bhiwandiwalla, A., and Kiritchenko, S

    Howard, P., Fraser, K. C., Bhiwandiwalla, A., and Kiritchenko, S. Uncovering bias in large vision-language models at scale with counterfactuals. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 5946--5991, 2025

    2025

  18. [18]

    and De Melo, G

    Janghorbani, S. and De Melo, G. Multi-modal bias: Introducing a framework for stereotypical bias assessment beyond gender and race in vision--language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp.\ 1725--1735, 2023

    2023

  19. [19]

    Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L

    Lees, A., Tran, V. Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp.\ 3197--3207, 2022

    2022

  20. [20]

    Mohammad, S. M. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of The Annual Conference of the Association for Computational Linguistics (ACL), Melbourne, Australia, 2018

    2018

  21. [21]

    Mohammad, S. M. NRC VAD Lexicon v2: Norms for Valence, Arousal, and Dominance for over 55k English Terms . arXiv preprint arXiv:2503.23547, 2025. URL https://arxiv.org/abs/2503.23547

    Pith/arXiv arXiv 2025

  22. [22]

    L., Colaresi, M

    Monroe, B. L., Colaresi, M. P., and Quinn, K. M. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16 0 (4): 0 372--403, 2008

    2008

  23. [23]

    R., and Raza, S

    Narayanan, A., Khazaie, V. R., and Raza, S. Bias in the picture: B enchmarking VLMs with social-cue news images and LLM -as-judge assessment. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling, 2025

    2025

  24. [24]

    SB-Bench : S tereotype bias benchmark for large multimodal models

    Narnaware, V., Vayani, A., Gupta, R., Swetha, S., and Shah, M. SB-Bench : S tereotype bias benchmark for large multimodal models. arXiv preprint arXiv:2502.08779, 2025

    arXiv 2025

  25. [25]

    Biases in large language models: origins, inventory, and discussion

    Navigli, R., Conia, S., and Ross, B. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15 0 (2): 0 1--21, 2023

    2023

  26. [26]

    Nicolas, G., Bai, X., and Fiske, S. T. Comprehensive stereotype content dictionaries using a semi-automated method. European Journal of Social Psychology, 51 0 (1): 0 178--196, 2021

    2021

  27. [27]

    G., Song, I., Oh, A., and Augenstein, I

    Pawar, S., Park, J., Jin, J., Arora, A., Myung, J., Yadav, S., Haznitrama, F. G., Song, I., Oh, A., and Augenstein, I. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, pp.\ 1--96, 2025

    2025

  28. [28]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PmLR, 2021

    2021

  29. [29]

    Optimization of Rank Losses for Image Retrieval

    Ramzi, E., Audebert, N., Rambour, C., Araujo, A., Bitot, X., and Thome, N. Optimization of Rank Losses for Image Retrieval . In In submission to: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  30. [30]

    and Gurevych, I

    Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  31. [31]

    A multi-dimensional study on bias in vision-language models

    Ruggeri, G., Nozza, D., et al. A multi-dimensional study on bias in vision-language models. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023

    2023

  32. [32]

    VLBiasBench : A comprehensive benchmark for evaluating bias in large vision-language model

    Wang, S., Cao, X., Zhang, J., Yuan, Z., Shan, S., Chen, X., and Gao, W. VLBiasBench : A comprehensive benchmark for evaluating bias in large vision-language model. arXiv preprint arXiv:2406.14194, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval

    Weyand, T., Araujo, A., Cao, B., and Sim, J. Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval . In Proc. CVPR, 2020

    2020

  34. [34]

    Evaluating fairness in large vision-language models across diverse demographic attributes and prompts

    Wu, X., Wang, Y., Wu, H.-T., Tao, Z., and Fang, Y. Evaluating fairness in large vision-language models across diverse demographic attributes and prompts. arXiv preprint arXiv:2406.17974, 2024

    arXiv 2024

  35. [35]

    GenderBias-VL : Benchmarking gender bias in vision language models via counterfactual probing

    Xiao, Y., Liu, X., Cheng, Q., Yin, Z., Liang, S., Li, J., Shao, J., Liu, A., and Tao, D. GenderBias-VL : Benchmarking gender bias in vision language models via counterfactual probing. International Journal of Computer Vision, 133 0 (12): 0 8332--8355, 2025

    2025

  36. [36]

    Bilateral reference for high-resolution dichotomous image segmentation

    Zheng, P., Gao, D., Fan, D.-P., Liu, L., Laaksonen, J., Ouyang, W., and Sebe, N. Bilateral reference for high-resolution dichotomous image segmentation. CAAI Artificial Intelligence Research, 2024

    2024

  37. [37]

    VLS tereo S et: A study of stereotypical bias in pre-trained vision-language models

    Zhou, K., Lai, E., and Jiang, J. VLS tereo S et: A study of stereotypical bias in pre-trained vision-language models. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.\ 527--538, Online onl...

    doi:10.18653/v1/2022.aacl-main.40 2022

  38. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.