Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that LLMs systematically prefer US and UK interpretations of contested historical events over Soviet and Chinese ones, with GPT-4o-mini choosing the US position in up to 81% of pairwise comparisons.

desk verdict Useful new dataset, plausible bias direction, but the headline preference magnitudes are confounded by unvalidated differences in viewpoint text quality. read the letter →

arxiv 2506.06751 v2 pith:7CHGR3TL submitted 2025-06-07 cs.CL

Mikhail Salnikov , Dmitrii Korzh , Ivan Lazichny , Elvir Karimov , Artyom Iudin , Ivan Oseledets , Oleg Y. Rogov , Natalia Loukachevitch
show 2 more authors
Alexander Panchenko Elena Tutubalina
This is my paper
classification cs.CL
keywords geopoliticalbiaslargelanguagemodelsnationalnarrativeshistoricaleventsdebiasingpromptsviewpointevaluationmodelmeasurement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that major large language models systematically prefer US and UK national interpretations of contested historical events over Soviet and Chinese ones, and that this bias survives a simple 'be unbiased' instruction. The authors build a dataset of 109 events with neutral descriptions and paired, conflicting viewpoints from the USA, UK, USSR, and China, then ask four models—GPT-4o-mini, Qwen2.5 72B, Llama-4-Maverick, and GigaChat-Max—to declare which position is correct. GPT-4o-mini picks the US position in up to 81% of US-China comparisons, and US-aligned preferences appear across most model-event pairs. Swapping the country labels attached to the viewpoints makes models hesitate and often declare both positions wrong, while a 'Chinese patriot' persona reliably flips preferences toward China. The paper offers the dataset and framework as a tool for studying geopolitical bias in AI systems.

What carries the argument

The load-bearing mechanism is a pairwise viewpoint-comparison test: for each of 109 historical events, the dataset pairs a neutral description with two country-specific positional statements, and the model must pick one of four structured answers in JSON. The framework's controls—mentioning participants, substituting their labels, and adding a persona prompt—allow the authors to separate content-driven from attribution-driven bias. The 'Substituted Participants' condition, which swaps the country names attached to fixed texts, is what reveals that models track the alignment between name and content, not just the names themselves.

What would settle it

Have neutral raters, blind to country labels, rank the paired viewpoints for persuasiveness, length, and assertion strength; then check whether the LLMs' preferred viewpoint correlates with the raters' top-ranked text. Alternatively, re-run the full comparison on viewpoint pairs matched for length and assertiveness—if the US preference disappears, the bias is stylistic.

Watch

Extended reading notes

Core claim

The central claim is that LLMs exhibit a measurable, systematic geopolitical bias: when presented with two rival national interpretations of the same historical event, they favor the US (and often UK) narrative as 'correct' far more often than Soviet or Chinese ones. The bias is demonstrated through a structured pairwise-choice protocol in which models choose between Position A, Position B, 'both incorrect', and 'both equal', with responses forced into JSON for reliable counting. A one-line debiasing prompt asking the model to avoid stereotypes changes little for the most biased models. Explicitly naming the countries in the prompt sometimes strengthens the bias and sometimes induces caution, depending on the model; swapping the country labels causes a sharp rise in 'both incorrect' answers, suggesting models are sensitive to a mismatch between content and attribution. Instructing the model to adopt a 'Chinese patriot' persona flips the preference toward China almost universally, showing the default US preference is not a hard constraint but a prior that explicit framing can override.

Load-bearing premise

The hand-written national viewpoints handed to each model are assumed to be equally persuasive and historically balanced; if US-authored statements are longer, more assertive, or closer to the source phrasing, the measured 'bias' could be a style effect rather than a country effect.

Editorial extensions

If this is right

  • Models used for education, search, or policy briefing will present US- and UK-centered interpretations of contested history as the correct ones, marginalizing Soviet and Chinese perspectives.
  • Asking a model to 'be unbiased' is not an effective correction; the paper finds shifts of only a few percentage points for the most biased models.
  • Models can adopt an instructed national persona (e.g., 'Chinese patriot') and reliably endorse that country's position, so the default Western-leaning prior is an unspoken default, not an inability to represent other views.
  • The 109-event dataset and JSON-forced evaluation protocol give other researchers a direct way to measure geopolitical bias in new or updated models.
  • The mismatch detection seen under swapped labels suggests models possess some internal consistency check between historical content and the country it is attributed to, which future debiasing could exploit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The sharp rise in 'both incorrect' when country labels are swapped implies that the models' preference is partly a reasoning effect: they can detect when a familiar US-framed argument is mislabeled as Chinese. A natural next test is whether models trained on more balanced multilingual corpora show a smaller swap-confusion effect.
  • The one-position ablation ordering (USSR ≤ China ≤ UK ≤ US in how often a viewpoint is judged correct) suggests a credibility hierarchy that may reflect the volume and editorial slant of English Wikipedia coverage; this is the paper's own data point but the interpretation is ours.
  • A direct test of the stylistic-confound hypothesis: generate viewpoint pairs matched for word count, assertiveness, and source phrasing, then re-run the pairwise comparison. If the US preference shrinks, the bias is partly a writing-style artifact rather than a pure country preference.
  • The language-invariance finding (English, Chinese, Russian, French all give similar results) predicts that the bias operates at the semantic/instruction level, so multilingual debiasing cannot be achieved by translation alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces a dataset of 109 historical events, each with a neutral description and two contrasting national viewpoints (USA, UK, USSR, China), and prompts four LLMs (GPT-4o-mini, Qwen2.5 72B, Llama-4-Maverick, GigaChat-Max) to decide which viewpoint is correct, with additional options for 'both incorrect' and 'both equal'. Experiments include a baseline, a simple debiasing prompt, explicit participant mention, participant label substitution, a 'Chinese patriot' persona, and multilingual variants. The authors report systematic pro-USA/UK preferences (e.g., GPT-4o-mini choosing the US position in up to 81% of USA-China cases), limited effectiveness of the debiasing prompt, strong instruction-following under the patriot prompt, and label-swap sensitivity. The dataset and code are released.

Significance. If the central claim holds, this is a useful contribution to a relatively underexplored area: it extends political-bias evaluation from domestic politics to international historical narratives, includes models from multiple countries (US, China, Russia), and provides a publicly available dataset and a structured JSON-output framework that facilitates reproducibility. The debiasing null result, although based on a single prompt, is consistent with prior work on the limited effect of simple instruction-based debiasing. The multilingual extension is a strength. However, the core measurement rests on an unvalidated assumption about the symmetry of the hand-written viewpoint texts, and the absence of statistical inference plus inconsistent numbers between the main text and appendix currently preclude interpreting the magnitude of the reported preferences as stable evidence of geopolitical bias.

major comments (3)
  1. [Section 3, Appendix A] The central measurement assumes that the two national viewpoint texts for each event are interchangeable in style, specificity, and factual defensibility, so that a model's disproportionate selection of one viewpoint uniquely indicates country bias. This assumption is not validated. The Greek Civil War example in Appendix A visibly gives the USA viewpoint more concrete detail (e.g., explicit mention of the Truman Doctrine) than the USSR viewpoint, which is generic. No human rating, style-matching, or adjudication of historical accuracy is provided. The Substituted Participants control (Section 4.2, Tables 2 and 5) swaps labels but keeps the same texts; the resulting large shift to 'Both incorrect' shows that models detect label-content mismatches, but it does not establish that in the consistent baseline the model is insensitive to unmatched text quality. A style-balanced rerun, or at least a human evaluation of the paired texts, is needed to support the claim that the observed preferences are geopolitical rather than stylistic.
  2. [Section 4, Tables 2 and 5] The numbers for the USA-China condition differ between the main text and the appendix, and no statistical precision is reported. For GPT-4O-MINI, the baseline USA preference is 81.0% in Table 2 but 78.6% in Table 5; the debias condition is 83.3% in Table 2 but 85.7% in Table 5; the substituted-participant condition is 26.2/31.0/42.9 in Table 2 but 21.4/28.6/50.0 in Table 5. These are not roundings of the same values. In addition, all results in Tables 2-13 are reported as percentages without confidence intervals, error bars, or significance tests, even though each event was tested three times with different temperatures, so per-item variance is available. Given that the USA-China condition has only 14 events, a difference of 81% vs. 78.6% can correspond to a single response. The manuscript should reconcile the tables and provide per-item counts or variance measures.
  3. [Section 4.1, Table 2] The claim that debiasing had 'minimal changes (below ± 2%)' for GIGACHAT-MAX and GPT-4O-MINI is not supported by the reported numbers. In Table 2, GIGACHAT-MAX's USA preference changes from 71.4% to 66.7% (a 4.7-point drop) and GPT-4O-MINI changes from 81.0% to 83.3% (a 2.3-point rise). In Table 5, GIGACHAT-MAX changes from 71.4% to 64.3% (7.1 points). Even if these shifts are not statistically meaningful, the statement as written is factually inconsistent with the tables. This matters because the conclusion that simple debiasing is ineffective is a headline contribution and needs to be based on accurately reported numbers and, ideally, a test of whether the changes exceed sampling noise.
minor comments (6)
  1. [Abstract] The abstract contains a typo: 'though an analysis' should be 'through an analysis'.
  2. [Figure 2 caption] The caption spells one model as 'OWEN2.5'; elsewhere it is 'QWEN2.5 72B'.
  3. [Section 4] The phrase 'the Instructor was used' is unclear; it is not defined what 'Instructor' refers to (possibly a prompting tool or a typo for 'instruction').
  4. [Appendix B.1] The bullet point stating that GPT-4o-mini 'generally shows more balanced responses compared to others' seems to contradict the main-text characterization of GPT-4o-mini as exhibiting explicit US bias; the statement should either be qualified or reconciled with the tables.
  5. [Table 4] The 'Positions' column lists codes such as SS-1, CW-1, ChUK-1 without explaining what they denote; a brief description would help readers navigate the dataset.
  6. [Limitations] The Limitations section says 'Besides, our study is limited...' shortly after a numbered list of limitations, which is redundant and informal; the two paragraphs on dataset scope could be merged.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity in the central bias measurement; one non-load-bearing 'Chinese patriot' condition is a prompt-following tautology.

  1. self definitional [Section 4.2 (Table 3) and Appendix B.2]
    "In addition to debiasing prompts and participant label manipulations, we prompted the LLM to act as a Chinese Patriot to evaluate geopolitical biases, with all results presented in Tab. 3. We see that all LLMs follow patriot instructions, favoring China in almost all pairwise comparisons."

    The 'Chinese patriot' condition instructs the model to 'answer the question based on the Chinese perspective'; therefore any model that complies will favor the Chinese position by construction. Reporting this as evidence of 'bias' is a manipulation check, not a measurement of an underlying predisposition, because the outcome is entailed by the input prompt. The central USA-bias claim does not rest on this condition; it comes from the baseline prompt, where no national perspective is imposed, so this step is non-load-bearing.

full rationale

The paper's central result is a direct behavioral measurement: fresh LLM responses to unseen paired viewpoint prompts are counted, with no parameter fitted from those responses and then presented as a prediction. There is no self-citation chain, no imported uniqueness theorem, and no fitted input renamed as a prediction. The 'Substituted Participants' and 'Mention Participants' manipulations are genuine experimental controls, and the debiasing-prompt result is a straightforward intervention outcome. The main weaknesses—hand-written viewpoints of possibly unequal style or specificity, Wikipedia sourcing, and possible position-order confounds—concern construct validity and confound control, not circularity. The only by-construction element is the 'Chinese patriot' condition, which is an instruction-following tautology and is not load-bearing for the central claim.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claim rests on the assumption that the hand-crafted viewpoints and neutral event descriptions are balanced across countries and that model choices reflect bias rather than text style. No statistical validation or style controls are provided.

assumptions (2)
  • domain assumption The multiple-choice evaluation response (selecting a viewpoint) is a valid, direct measure of geopolitical bias.
    The paper equates the model's choice with bias, without testing whether the choices track persuasion, informativeness, or wording. Used throughout Section 4.
  • ad hoc to paper Neutral event descriptions drawn from Wikipedia are a neutral reference point.
    The authors use Wikipedia as the ground truth for event descriptions, but later acknowledge in Limitations that this 'may carry biases' and 'Korean War is described through US/UK lenses'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models." pith.science (2026). https://pith.science/paper/7CHGR3TL

@misc{pith2026250606751,
  author       = {Pith},
  title        = {Pith review of: Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CHGR3TL}},
  note         = {Machine review of arXiv:2506.06751}
}
read the original abstract

This paper evaluates geopolitical biases in LLMs with respect to various countries though an analysis of their interpretation of historical events with conflicting national perspectives (USA, UK, USSR, and China). We introduce a novel dataset with neutral event descriptions and contrasting viewpoints from different countries. Our findings show significant geopolitical biases, with models favoring specific national narratives. Additionally, simple debiasing prompts had a limited effect in reducing these biases. Experiments with manipulated participant labels reveal models' sensitivity to attribution, sometimes amplifying biases or recognizing inconsistencies, especially with swapped labels. This work highlights national narrative biases in LLMs, challenges the effectiveness of simple debiasing methods, and offers a framework and dataset for future geopolitical bias research.

Figures

Figures reproduced from arXiv: 2506.06751 by the authors.

Figure 1
Figure 1. Example prompt structure for instructing LLM [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of LLM viewpoint selection for historical events by country pairs. Charts represent country pairs (e.g., UK-USA). Rings denote LLMs: GPT-4O-MINI, OWEN2.5, LLAMA-4-MAVERICK, GIGACHAT￾MAX. Segments correspond to viewpoint selection frequency (e.g., blue for China, green for USA), for example, GPT-4O-MINI (outermost rings) demonstrated explicit US bias. pants. The main idea here is to check if the models h… view at source ↗
Figure 4
Figure 4. Probability to change opinion about the coun [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: Probability to change opinion about a country [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 5
Figure 5. Figure 5: Probability to change opinion about the coun [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Probability to change opinion about the coun [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Probability to change opinion about the coun [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Probability to change opinion about the coun [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Example prompt structure for instructing LLM [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Estimating the Geopolitical Preferences of Large Language Models from United Nations Voting Data

    cs.CY 2026-07 conditional novelty 6.0 of 10

    On a UN-vote ideal-point scale, GPT-5, Claude Sonnet, and Gemini are closer to Russia than to the US among P5 states, DeepSeek is closest to France, and all four are farthest from the US.

Reference graph

Works this paper leans on

21 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Paul J Bolt and Sharyl N Cross. 2018. China, Russia, and twenty-first century global geopolitics. Oxford University Press

    work page 2018

  5. [5]

    Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arXiv preprint arXiv:2303.17466

    arXiv 2023

  6. [6]

    Jason A Edwards. 2012. An exceptional debate: The championing of and challenge to american exceptionalism. Rhetoric & Public Affairs, 15(2):351--367

    work page 2012

  7. [7]

    Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. 2024. On the relationship between truth and political bias in language models. arXiv preprint arXiv:2409.05283

    arXiv 2024

  8. [8]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097--1179

    2024

Show all 21 references
  1. [9]

    Daniel Kokotajlo, Eli Lifland, Thomas Larsen, Romeo Dean, and Scott Alexander. 2025. https://ai-2027.com Ai 2027: A scenario for the impact of superhuman ai . https://ai-2027.com. Accessed: 2025-04-24

    2025

  2. [10]

    Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pages 12--24

    2023

  3. [11]

    Bryan Li, Samar Haider, and Chris Callison-Burch. 2024. This land is your, my land: Evaluating geopolitical bias in language models through territorial disputes. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistic...

    2024

  4. [12]

    Luyang Lin, Lingzhi Wang, Jinsong Guo, and Kam - Fai Wong. 2025. https://aclanthology.org/2025.coling-main.709/ Investigating bias in llm-based bias detection: Disparities between llms and human perception . In Proceedings of the 31st International Conference on Computational ...

    2025

  5. [13]

    Riccardo Lunardi, David La Barbera, and Kevin Roitero. 2024. The elusiveness of detecting political bias in language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 3922--3926

    2024

  6. [14]

    Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2024. More human than human: measuring chatgpt political bias. Public Choice, 198(1):3--23

    2024

  7. [15]

    Yujin Potter, Shiyang Lai, Junsol Kim, James Evans, and Dawn Song. 2024. Hidden persuaders: Llms' political leaning and their influence on voters. arXiv preprint arXiv:2410.24190

    2024 arXiv

  8. [16]

    Jirui Qi, Raquel Fern \'a ndez, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. arXiv preprint arXiv:2310.10378

    2023 arXiv

  9. [17]

    o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \"u tze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. arXiv preprint arXiv:2402.16786

    2024 arXiv

  10. [18]

    Yan Tao, Olga Viberg, Ryan S Baker, and Ren \'e F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS nexus, 3(9):pgae346

    2024

  11. [19]

    Himanshu Thakur, Atishay Jain, Praneetha Vaddamanu, Paul Pu Liang, and Louis-Philippe Morency. 2023. https://doi.org/10.18653/v1/2023.acl-short.30 Language models get a gender makeover: Mitigating gender bias with few-shot data interventions . In Proceedings of the 61st Annual...

    2023 doi

  12. [20]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  13. [21]

    Franklin M Zaromb, James H Liu, Dario P \'a ez, Katja Hanke, Adam L Putnam, and Henry L Roediger III. 2018. We made history: Citizens of 35 countries overestimate their nation's role in world history. Journal of Applied Research in Memory and Cognition, 7(4):521

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.