Pith. sign in

REVIEW 2 major objections 6 minor 27 references

Obscured but Not Erased: Evaluating Nationality Bias in LLMs via Name-Based Bias Benchmarks

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing nationality labels with culturally indicative names lowers but does not eliminate stereotyped answers in LLMs, and smaller models retain more of the bias.

desk verdict A solid, incremental name-based extension of BBQ with a useful error-retention metric; the missing name-recognition validation and unreleased name lists are the real soft spots, not the core finding. read the letter →

arxiv 2507.16989 v1 pith:4BDIA2IK submitted 2025-07-22 cs.CL

Giulio Pelosio , Devesh Batra , Noémie Bovey , Robert Hankache , Cristovao Iglesias , Greig Cowan , Raad Khraishi This is my paper
classification cs.CL
keywords LLMbiasnationalitystereotypesculturalnameproxiesBBQbenchmarkscoreerrorretentionratiomodelcapacityambiguouscontext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models continue to answer with nationality stereotypes even when the prompt contains no country label, only a culturally suggestive first name. The paper builds a name-based variant of the Bias Benchmark for QA (BBQ) nationality subset, replacing each of 68 nationalities with ten curated first names, and tests six commercial models from OpenAI, Google, and Anthropic. Across all models, substituting names for nationality labels raises accuracy and lowers measured bias, but positive stereotype-aligned bias remains in ambiguous contexts. Smaller models are less accurate, more biased, and retain more of their errors after the substitution. The result matters because real deployments often see names, not demographic labels, in settings like hiring, credit, and healthcare.

What carries the argument

The load-bearing machinery is a re-engineered dataset and a ratio. The dataset replaces explicit nationality references in BBQ prompts, such as 'the Indian family,' with culturally indicative first names such as 'Santhosh's family,' using ten gender-balanced names per nationality and three random seeds, with GPT-4o used to keep gender agreement in the substitution. The error retention ratio, $\varepsilon_c = \frac{1 - \alpha_{c,\mathrm{nmd}}}{1 - \alpha_{c,\mathrm{ntd}}}$ for context type $c$, quantifies the fraction of nationality-prompt errors that survive the substitution. Together with the ambiguous-context bias score, this separates whether a model stops stereotyping from whether it simply becomes more accurate.

What would settle it

A rerun with the name-to-nationality mapping randomly shuffled would settle the mechanism: if bias scores stay elevated under mismatched names, the residual bias is driven by name familiarity or other confounds rather than by the specific cultural associations the paper attributes to names. A separate annotation study measuring how strongly each chosen name is associated with its intended nationality would test the proxy assumption directly.

Watch

Extended reading notes

Core claim

The central discovery is that culturally indicative names act as partial proxies for explicit nationality labels in LLM stereotype behavior. Across every model tested, replacing labels such as 'Italian' or 'Iraqi' with names such as 'Paolo' or 'Hussein' improved accuracy and lowered the bias score, yet positive stereotype-aligned bias persisted in ambiguous contexts for all models. The smallest Anthropic model, Claude Haiku, still showed an ambiguous-context bias of 0.091, against 0.035 for Claude Sonnet, and the error retention ratio showed that GPT-4o-mini keeps 76% of its nationality-prompt errors after substitution while GPT-4o keeps 68%. The paper also reports an asymmetry in how models fail: although larger models are less biased overall, when they do answer wrongly in ambiguous contexts their wrong answers are stereotype-aligned more often than those of smaller models, as shown by the unscaled bias scores.

Load-bearing premise

The measurement rests on the assumption that the ten hand-picked names per nationality are recognized by the models as reliable signals of that nationality and are otherwise no harder than the labels they replace; if name recognition or cross-cultural name ambiguity varies, the residual bias and error-retention values could be artifacts.

Editorial extensions

If this is right

  • Name-only inputs are not a safe substitute for demographic-blind design: the benchmark shows residual bias in ambiguous contexts for every model, so applications that see names should assume stereotype risk.
  • Smaller models need separate bias evaluation before deployment; the accuracy and error-retention gaps between large and small variants of the same provider are large enough to change risk profiles.
  • Disambiguating context is a strong mitigation: with clear evidence, bias scores drop near zero, so high-stakes systems should be designed to elicit or require disambiguating information rather than answer from ambiguous prompts.
  • The error retention ratio gives a standard, provider-comparable measure of how much stereotype-driven error survives proxy substitution, and could be reported alongside accuracy and bias scores in future audits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The unscaled-bias result implies a safety trade-off the paper leaves implicit: larger models are correct more often, but their remaining errors are more reliably stereotype-aligned, which may be the worst kind of error in decisions that affect individuals.
  • Because the name lists draw on historically common names, the benchmark may understate bias for transnational or ambiguous names; extending the list to include names that cross cultural boundaries would test how quickly residual bias grows.
  • The same substitution machinery could map onto BBQ's other categories, such as race, religion, and gender, and onto non-English prompts, offering a direct test of whether 'obscured but not erased' is a general property of proxy cues.
  • A stress-test extension would vary the number of names per nationality and measure how bias and error retention change, isolating name-list diversity as a factor in benchmark sensitivity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces a name-based variant of the BBQ nationality subset, replacing explicit nationality labels with curated culturally indicative names. It evaluates six LLMs (GPT-4o, GPT-4o-mini, Claude Sonnet, Claude Haiku, Gemini-1.5-pro, Gemini-1.5-flash) under ambiguous and disambiguated contexts and reports accuracy, scaled and unscaled bias scores, and a newly proposed Error Retention Ratio (ERR). The central findings are that name substitution reduces but does not eliminate stereotype-aligned responses; larger models are generally more accurate and show lower scaled bias scores; and larger models retain a smaller share of errors after substitution, although when they do err in ambiguous contexts they default to stereotypes more often than smaller models do.

Significance. If the name-based proxy is valid, the paper makes a useful contribution: it operationalizes a realistic proxy-bias scenario, provides a current cross-provider comparison, and introduces the ERR metric. The work is empirical, and bias scores and ERR are computed directly from model outputs against fixed templates, with no fitted parameters. The three-seed repetition and reporting of standard deviations are also strengths. However, the benchmark's validity hinges on the name-to-nationality mapping, which is currently unvalidated and not released, and there is an internal inconsistency in the definition of the unscaled bias score. These issues affect the central claims but appear addressable with additional validation, clarification, and data release.

major comments (2)
  1. [§4.1 and Table 10] The paper provides no evidence that the 10 hand-picked names per nationality are recognized by models (or humans) as proxies for the intended nationalities. The Limitations section explicitly concedes that names 'transcend cultural boundaries' and that the limited set 'may inadvertently magnify or underestimate biases,' but no association check is reported: there are no human ratings, no forced-choice name-to-nationality probe, and no name-frequency or recognizability analysis. Moreover, Table 10 lists only 'Source link' placeholders rather than the actual names, so the intervention cannot be inspected or replicated. Because the core claims—accuracy gains, residual bias, and ERR differences between small and large models—are all defined relative to this substitution, unrecognizable or ambiguous names could produce the observed patterns as artifacts. Please add a name-to-nationality validation (e.g., human or model forced-choice probes) and release the full name lists and sources.
  2. [§4.3.2, §5.3, and Table 5] The definition of the unscaled ambiguous bias score is internally inconsistent. Equation (1) defines βdis as (2·nbias − nnon-UNK)/nnon-UNK, which ranges from −1 to +1 and equals 2p−1, where p is the proportion of stereotype-aligned answers among non-Unknown responses. The text, however, states that 'the unscaled bias score represents the raw proportion of stereotype-aligned responses among all incorrect answers,' and Section 5.3 calls it 'the percentage of times a model incorrectly chooses a stereotype-aligned target rather than Unknown.' These are different quantities: the raw proportion is p, not 2p−1. Table 5 reports values such as 0.676 for GPT-4o, which is not a proportion; the corresponding p would be 0.838. Additionally, for GPT-4o the scaled ambiguous bias (0.035) and accuracy (0.950) in Tables 1 and 3 imply βdis = 0.70, while Table 5 reports 0.676. Please clarify the metric, recompute or relabel the reported values, and ensure the 'larger models default to stereotypes more often when they err' claim is based on a correctly defined and consistently reported quantity.
minor comments (6)
  1. [§4.1.1] Using GPT-4o to label the gender of answer choices introduces a potential circularity, since GPT-4o is itself one of the evaluated models; if its gender judgments contain nationality-correlated errors, the name substitution could be systematically distorted. Please report the accuracy of this labeling step on a sample or use an independent annotator.
  2. [Throughout] The word 'disambiguous' is used repeatedly (e.g., Section 5.1 heading and Figure captions); it should be 'disambiguated.'
  3. [Table 7 and Section 5] Table 7 lists GPT-4 Turbo among the evaluated models, but no experimental results for this model appear in any results table or figure; either include its results or remove it from the model list.
  4. [Table 10] Table 10 shows 'Source link' placeholders rather than actual names or accessible URLs; the appendix should provide the complete name-to-nationality mapping and working source references.
  5. [Limitations] The Limitations note that BBQ may have been used to train some of the models; since contamination could affect absolute bias and accuracy levels, please discuss the expected direction of this effect or report results on a held-out subset.
  6. [Tables 1–6] The paper reports means and standard deviations but no significance tests; for claims comparing small and large models (e.g., Gemini-1.5-flash vs Gemini-1.5-pro in Table 4), a paired test across seeds or templates would help assess whether the differences are reliable.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmark analysis; no circular derivation found.

full rationale

The paper is an empirical measurement study rather than a derivation chain. It takes the external BBQ nationality templates, performs deterministic string substitution of nationality labels with curator-selected name lists, queries fixed LLM APIs, and then computes accuracy, bias scores, and the Error Retention Ratio from raw outputs. No parameter is fitted to a subset of the data and then renamed as a prediction; the bias score formulas in Equations (1)-(2) are inherited from Parrish et al. and defined directly on model outputs, and the ERR in Section 4.3.3 is a ratio of measured error rates. The name lists are sourced from public census and onomastic references, and the use of GPT-4o to label gender for grammatical substitution does not define the outcome metrics or inject the paper's conclusions. The cited BBQ dataset is an external benchmark with crowdworker validation, and the paper's limitations explicitly acknowledge name ambiguity and possible training-data contamination; these are internal-validity threats, not circular steps. No load-bearing claim reduces to its own input by construction, so there is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims depend on the quality of the name lists and the assumption that names work as proxies; these are experimental choices rather than fitted parameters, and no new theoretical entities are introduced. The Error Retention Ratio is a derived metric, not an entity.

free parameters (3)
  • Name lists per nationality (10 names, 5 female, 5 male)
    Curated by the authors from census, Wikipedia, and Forebears; no validation that the names are recognized or unique to the nationality. Directly affects bias magnitudes and accuracy on the name dataset.
  • Inference temperature = 1
    Chosen to promote diversity in completions; affects the variance and point estimates of accuracy and bias scores.
  • Random seeds = 42 as global, then three sub-seeds from randint(1,10000)
    Arbitrary but fixed for reproducibility; the specific seeds determine the name substitutions and sampling.
assumptions (4)
  • domain assumption BBQ templates are a valid instrument for nationality bias
    The paper relies on the BBQ dataset's 25 nationality templates as representative of harmful stereotypes; standard in the field.
  • domain assumption Culturally indicative names act as demographic proxies
    The entire name-based benchmark assumes that names trigger or reveal the same stereotype associations as explicit nationality labels.
  • ad hoc to paper The 10 selected names per nationality are representative
    No evidence is provided that these names are the most recognized or least ambiguous; the authors acknowledge this in the Limitations.
  • ad hoc to paper GPT-4o provides accurate gender labels during augmentation
    GPT-4o is used to label answer choices as male, female, or neutral; labeling errors would propagate into name substitutions for all models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Obscured but Not Erased: Evaluating Nationality Bias in LLMs via Name-Based Bias Benchmarks." pith.science (2026). https://pith.science/paper/4BDIA2IK

@misc{pith2026250716989,
  author       = {Pith},
  title        = {Pith review of: Obscured but Not Erased: Evaluating Nationality Bias in LLMs via Name-Based Bias Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4BDIA2IK}},
  note         = {Machine review of arXiv:2507.16989}
}
read the original abstract

Large Language Models (LLMs) can exhibit latent biases towards specific nationalities even when explicit demographic markers are not present. In this work, we introduce a novel name-based benchmarking approach derived from the Bias Benchmark for QA (BBQ) dataset to investigate the impact of substituting explicit nationality labels with culturally indicative names, a scenario more reflective of real-world LLM applications. Our novel approach examines how this substitution affects both bias magnitude and accuracy across a spectrum of LLMs from industry leaders such as OpenAI, Google, and Anthropic. Our experiments show that small models are less accurate and exhibit more bias compared to their larger counterparts. For instance, on our name-based dataset and in the ambiguous context (where the correct choice is not revealed), Claude Haiku exhibited the worst stereotypical bias scores of 9%, compared to only 3.5% for its larger counterpart, Claude Sonnet, where the latter also outperformed it by 117.7% in accuracy. Additionally, we find that small models retain a larger portion of existing errors in these ambiguous contexts. For example, after substituting names for explicit nationality references, GPT-4o retains 68% of the error rate versus 76% for GPT-4o-mini, with similar findings for other model providers, in the ambiguous context. Our research highlights the stubborn resilience of biases in LLMs, underscoring their profound implications for the development and deployment of AI systems in diverse, global contexts.

Figures

Figures reproduced from arXiv: 2507.16989 by the authors.

Figure 1
Figure 1. An illustrative prompt from the original BBQ [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Example illustrating how explicit national [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average model accuracy comparison across [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average model bias score comparison across [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 7
Figure 7. Figure 7: Ambiguous vs. Disambiguated Accuracy for [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Ambiguous vs. Disambiguated Bias Scores for each model. Circles denote the Name dataset, while squares denote the Nationality dataset. The x-axis shows the bias score in ambiguous prompts, and the y-axis shows the bias score in disambiguated prompts. Metrics reported a…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 8 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    culture

    Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Alham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling" culture" in llms: A survey. arXiv preprint arXiv:2403.15412

    arXiv 2024

  4. [4]

    Khaled AlNuaimi, Gautier Marti, Mathieu Ravaut, Abdulla AlKetbi, Andreas Henschel, and Raed Jaradat. 2024. Enriching datasets with demographics through large language models: What's in a name? arXiv preprint arXiv:2409.11491

    arXiv 2024

  5. [5]

    Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. 2024. Measuring implicit bias in explicitly unbiased large language models. arXiv preprint arXiv:2402.04105

    arXiv 2024

  6. [6]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  7. [7]

    Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. Measuring political bias in large language models: What is said and how it is said. arXiv preprint arXiv:2403.18932

    arXiv 2024

  8. [8]

    Yucong Duan. 2024. The large language model (llm) bias evaluation (age bias). DIKWP Research Group International Standard Evaluation. DOI, 10

    work page 2024

Show all 27 references
  1. [9]

    Jasmina S Ernst, Sascha Marton, Jannik Brinkmann, Eduardo Vellasques, Damien Foucard, Martin Kraemer, and Marian Lambert. 2023. Bias mitigation for large language models using adversarial learning. In CEUR Workshop Proceedings, volume 3523, pages 1--14. RWTH Aachen

    2023

  2. [10]

    Duanyu Feng, Yongfu Dai, Jimin Huang, Yifang Zhang, Qianqian Xie, Weiguang Han, Zhengyu Chen, Alejandro Lopez-Lira, and Hao Wang. 2023. Empowering many, biasing a few: Generalist credit scoring through large language models. arXiv preprint arXiv:2310.00566

    2023 arXiv

  3. [11]

    Chengguang Gan, Qinghao Zhang, and Tatsunori Mori. 2024. Application of llm agents in recruitment: A novel framework for resume screening. arXiv preprint arXiv:2401.08315

    2024 arXiv

  4. [12]

    Rem Hida, Masahiro Kaneko, and Naoaki Okazaki. 2024. Social bias evaluation for large language models requires prompt variations. arXiv preprint arXiv:2407.03129

    2024 arXiv

  5. [13]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  6. [14]

    Mahammed Kamruzzaman and Gene Louis Kim. 2024. Exploring changes in nation perception with nationality-assigned personas in llms. arXiv preprint arXiv:2406.13993

    2024 arXiv

  7. [15]

    Zhao Liu. 2024. Evaluating and mitigating social bias for large language models in open-ended settings. arXiv preprint arXiv:2412.06134

    2024

  8. [16]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  9. [17]

    Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. 2020. Bias in word embeddings. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 446--457

    2020

  10. [18]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. 2021. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193

    2021 arXiv

  11. [19]

    Chahat Raj, Anjishnu Mukherjee, Aylin Caliskan, Antonios Anastasopoulos, and Ziwei Zhu. 2024. Breaking bias, building bridges: Evaluation and mitigation of social biases in llms via contact hypothesis. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volum...

    2024

  12. [20]

    Aishik Rakshit, Smriti Singh, Shuvam Keshari, Arijit Ghosh Chowdhury, Vinija Jain, and Aman Chadha. 2024. From prejudice to parity: A new approach to debiasing large language model word embeddings. arXiv preprint arXiv:2402.11512

    2024 arXiv

  13. [21]

    Yuval Reif and Roy Schwartz. 2024. Beyond performance: Quantifying and mitigating label bias in llms. arXiv preprint arXiv:2405.02743

    2024 arXiv

  14. [22]

    Lauren Rhue, Sofie Goethals, and Arun Sundararajan. 2024. Evaluating llms for gender disparities in notable persons. arXiv preprint arXiv:2403.09148

    2024 arXiv

  15. [23]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  16. [24]

    Ehsan Ullah, Anil Parwani, Mirza Mansoor Baig, and Rajendra Singh. 2024. Challenges and barriers of using large language models (llm) such as chatgpt for diagnostic medicine with a focus on digital pathology--a recent scoping review. Diagnostic pathology, 19(1):43

    2024

  17. [25]

    kelly is a warm person, joseph is a role model

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. " kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters. arXiv preprint arXiv:2310.09219

    2023 arXiv

  18. [26]

    Zhiwen You, HaeJin Lee, Shubhanshu Mishra, Sullam Jeoung, Apratim Mishra, Jinseok Kim, and Jana Diesner. 2024. Beyond binary gender labels: Revealing gender biases in llms through gender-neutral name predictions. arXiv preprint arXiv:2407.05271

    2024 arXiv

  19. [27]

    Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity. arXiv preprint arXiv:2301.12867

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.