REVIEW 3 major objections 5 minor 1 cited by
POW: Political Overton Windows of Large Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Using extreme political personas, this paper maps each LLM's Overton Window — coverage ranges from about 1% (DeepSeek r1) to 67% (Gemini 2.0 Flash), with most models unable to voice authoritarian-left or liberal-right views.
desk verdict Overton Window framing is a real step beyond point-estimate political audits, but the headline area numbers rest on an undocumented algorithm and a persona-compliance confound the paper itself owns. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Overton Window itself, borrowed from political theory and redefined here as the set of Political Compass Test positions an LLM will espouse when asked. It is measured by PRISM-style probing: each model writes essays on all 62 PCT propositions under eight extreme personas (e.g., 'Economic Right-Wing Authoritarian'), an AI assessor scores each essay on a five-point agree/disagree scale (validated at 90.3% binary agreement against human labels), and the scores are plotted as the window's area as a percentage of the full compass plane. The persona essays do the load-bearing work: a model that stays in character reveals willingness; a model that drifts toward its default
What would settle it
Re-probe the most restrictive models (e.g., DeepSeek r1 at ~1% coverage) with the same extreme positions reworded as neutral, non-political tasks — e.g., 'write a debate brief arguing that authority should organize the economy' — using several phrasings per proposition and varied temperatures. If a model's window jumps from ~1% to tens of percent, the original windows describe the probe rather than the model; if it stays near ~1%, the boundary is real. A second check: have human raters score the same extreme-persona essays the AI assessor scored, to test whether the window's edges are an artif
Extended reading notes
Core claim
The paper's central claim is that each LLM has a measurable political Overton Window — the region of the economic left-right and social authoritarian-libertarian plane in which it will produce essays matching an assigned extreme persona — and that these windows vary enormously across the 27 models tested. Most models default to the lower-left (left, liberal) quadrant, confirming prior point-estimate results. But under persona pressure, most cannot or will not voice authoritarian-left or liberal-right arguments: several DeepSeek models cover barely 1% of the space while Gemini 2.0 Flash covers 67%. The authors interpret drift out of persona as evidence of ideological boundaries set by trainin
Load-bearing premise
The entire window measurement assumes that a model can faithfully adopt an assigned extreme persona whenever its training and alignment do not block that view — so a model that drifts back to its default is displaying a political boundary, not a quirk of instruction following or prompt format.
Editorial extensions
If this is right
- A single ideological point estimate (e.g., 'left-libertarian') is insufficient to audit an LLM; the area and shape of its Overton Window must be reported instead.
- Models with very small windows (e.g., DeepSeek r1 at ~1%) will fail users who ask for arguments from a wide range of political perspectives, which matters for debate, tutoring, and journalism applications.
- Window coverage tends to grow across successive model versions from most providers, while default policy simultaneously shifts further left and liberal.
- A model's refusal to voice a position is not symmetric: most models restrict authoritarian-left and liberal-right views specifically, implying provider-side alignment choices rather than generic safety filtering.
- The window metric offers a concrete, comparable audit output that developers and regulators could use to disclose the normative boundaries of deployed models.
Reading between the lines
- If the persona-probe interpretation is right, a testable corollary is that further fine-tuning or prompt variation could expand a 1% window substantially; that prediction is not tested in this paper.
- The measurement conflates 'unwilling' with 'unable': a model might possess authoritarian-left knowledge but be aligned to refuse, or might genuinely lack the capability; distinguishing the two would change the policy response from censorship to coverage gap.
- Because the assessor is itself an LLM, the window boundaries inherit any ideological bias of the assessor; a human-rated subsample focused on the extreme personas would sharpen the boundary estimates.
- One comparison in this paper is internally confounded: GPT-5-mini was evaluated at temperature 1.0 because that is its API minimum, while every other model ran at 0.0, so its 62.5% window may partly reflect sampling noise.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper adapts the Overton Window concept to large language models, arguing that point-estimate political audits miss the boundaries of what models are willing to espouse. Using the PRISM methodology, the authors prompt 28 models with 62 Political Compass Test propositions under eight extreme ideological personas (plus a default condition), have GPT-3.5 Turbo rate the resulting essays, and compute a 2D 'window' for each model. They report default political positions and window areas ranging from 0.3% to 67.5%, concluding that most models cannot articulate authoritarian-left or liberal-right perspectives and that provider choices strongly shape expressible ideological space.
Significance. If the measurement is valid, the contribution is significant: it moves LLM political-bias auditing from a single point to a boundary, with practical implications for transparency and pluralism. The paper has tangible strengths: it releases code and data, evaluates a broad set of open and proprietary models, uses deterministic settings, and validates the AI assessor against a human gold set (90.3% agreement). However, the central quantitative claim currently rests on an undocumented aggregation procedure and on an interpretive assumption that persona non-compliance reveals ideological boundaries rather than generic instruction-following or refusal behavior. These issues are load-bearing, so the significance is conditional on a major revision.
major comments (3)
- [Section 3 / Table 1] The procedure that converts eight persona probes plus a default into a 2D window area is never described. The manuscript does not state how each persona's 62 essay ratings are aggregated into Economic/Social coordinates, how refusals are scored, what geometric shape is fitted to the resulting points, or how the area percentage is computed. The Fig. 2 caption gives only thresholds (±7.5, ±1.5) for 'extreme' and 'centre,' not the full algorithm. Without this pipeline, the headline 0.3%–67.5% figures and the quadrant-level claims in Section 4 cannot be independently verified or reproduced.
- [Section 5.1 / Section 3] The construct validity of the window depends on the assumption that an LLM instructed to write from an extreme persona will do so unless training or alignment precludes it. The Limitations section explicitly states that persona non-faithfulness 'was exactly the point of this study.' But the same observations are equally consistent with generic instruction-following or safety/refusal training that moderates extreme role-play regardless of political direction. No control condition—for example, prompting the strongest possible argument for the same positions without the persona frame, or explicitly permitting extreme speech—is provided. The conclusion that models are 'incapable of articulating perspectives' (Section 5) is therefore underdetermined: the measurement may describe the probe's interaction with refusal behavior rather than a latent ideological boundary.
- [Section 4 / Table 1 / Limitations] The results report one essay per persona-proposition at temperature 0.0 (with gpt-5-mini at 1.0), and no variance or sensitivity analysis is presented. The paper acknowledges in Section 5.1 that additional sampling could capture variance, but the Results section still treats the area values and default positions as robust, including differences such as llama2 1.7% vs. llama3 14.8% and qwen:7b 12.2% vs. qwen:32b 51.8%. Since the central claim is a cross-model ranking of window sizes, the lack of error bars, repeated runs, or prompt-variation checks makes it impossible to distinguish real differences from measurement noise.
minor comments (5)
- [Abstract / Section 3 / Table 1] The abstract says 28 models, while Section 3 says 'twenty-seven models' and Section 4 says '7 out of 27'; Table 1 lists 28 entries. Please reconcile these numbers.
- [Figure 2 caption] The phrase 'Extreme is considered greater/lesser than +7.5/-7.5, and centre is greater/lesser than -1.5/+1.5' is grammatically unclear and does not specify which axis the thresholds apply to. A precise definition of the coordinate space is needed.
- [Section 3] The text reports 'over 17,000 essays.' With 28 models × 62 propositions × 9 conditions (8 personas + default), the expected count is 15,624. Please clarify the discrepancy, including whether some conditions were repeated or whether the assessor generated additional essays.
- [Section 3 / Table 1] The handling of refusals is described only as 'additional handling for refusals to respond.' The scoring of refusals directly affects the coordinates and area; a precise description should be included.
- [Table 1] Area values are reported with inconsistent precision (0.3, 51.8, 67.5) and the default-position labels ('Cent.', 'Left.', 'Auth.', 'Lib.') could be defined in a note, as could the ranges for each category.
Circularity Check
Overton Window is defined by persona compliance, and PRISM's validity is self-cited; central empirical findings remain independent.
-
self definitional
[Section 5.1 (Limitations), following Section 3 (Method)]
"Indeed, the fact that the LLM was not faithful to the personas was exactly the point of this study, which is how we were able to map the space of what LLMs considered acceptable i.e., the methodology probed what views the LLMs were willing to espouse when told explicitly to take on extreme positions."
The Overton Window is operationally defined in Section 3 by whether the model complies with an assigned extreme persona; the set of compliant personas is used to construct the window. Section 5.1 then treats non-compliance as the evidence of an ideological boundary ('exactly the point of this study'). Thus the conclusion that a model is 'incapable of articulating' a view is a restatement of the measurement (persona non-compliance) rather than an independent derivation. The construct and the measurement are defined by the same behavioral observation, so the claimed boundary is equivalent to the operational criterion by construction.
-
self citation load bearing
[Section 3 (Method), first paragraph]
"To audit political bias and expressive boundaries in large language models (LLMs), we applied the Preference Revelation through Indirect Stimulus Methodology (PRISM) (Azzopardi and Moshfeghi, 2024). PRISM was designed to probe models’ normative boundaries using indirect, task-driven elicitation rather than direct questioning... A key benefit of PRISM is that it leads to greater compliance and fewer refusals, leading to more accurate approximations of the political stances of the models."
The central measurement's validity rests on the claim that PRISM elicits latent political stances with 'greater compliance and fewer refusals, leading to more accurate approximations.' This claim is supported only by citation to the authors' own prior work (Azzopardi and Moshfeghi, 2024). The present paper validates only the AI assessor against human labels, not PRISM's core assumption that non-compliance indicates a model's normative boundary rather than task effects such as instruction-following or refusal training. The load-bearing premise thus reduces to a self-citation that is not independently verified in this paper.
full rationale
The paper's main quantitative output—the Overton Window area for each model—is a direct summary of measured compliance with extreme persona prompts, not a fitted parameter that is then 'predicted' from itself. There is no equation in which an output variable is constructed from the target variable. The 1%–67% area differences and the quadrant asymmetries are empirical observations and would remain informative even without the Overton Window interpretation. However, two circularity-adjacent features warrant a moderate score. First, the construct 'Overton Window' is operationally defined by persona faithfulness, and Section 5.1 explicitly identifies persona non-faithfulness as 'exactly the point' and as the evidence for the boundary; the conclusion that models are 'incapable of articulating' authoritarian-left or liberal-right views is thus a re-description of the measurement rather than an independent derivation. This is a self-definitional conflation of construct and measure, though it does not make the area numbers themselves circular. Second, the central premise that PRISM 'leads to greater compliance and fewer refusals, leading to more accurate approximations of the political stances' is justified by citation to the authors' own prior work, and the current paper does not independently validate that premise. That is load-bearing self-citation. A control condition (e.g., explicit permission to argue as strongly as possible) would be needed to dissociate ideological willingness from instruction-following/refusal behavior; its absence is a correctness risk, not circularity itself. Overall score 4.
Assumptions & free parameters
free parameters (3)
- Extreme/centre thresholds =
±7.5 extreme; ±1.5 centre
- Eight ideological persona positions =
Not enumerated in full (examples: 'Economic Left-Wing Authoritarian', 'Authoritarian', 'Economic Right-Wing Authoritaria
- Temperature =
0.0 (1.0 for GPT-5-mini)
assumptions (4)
- domain assumption LLMs are aware of the Political Compass Test and can faithfully adopt extreme ideological personas unless alignment precludes them from doing so
- domain assumption GPT-3.5-Turbo essay ratings are a valid measure of political stance
- ad hoc to paper The area computed from eight persona probes plus the default position represents the model's full voluntary political space
- domain assumption The two Political Compass Test axes adequately represent political ideology
invented entities (1)
-
LLM-specific Overton Window
Cite this review
Pith. "Pith review of POW: Political Overton Windows of Large Language Models." pith.science (2026). https://pith.science/paper/25IUBCDK
@misc{pith2026250908853,
author = {Pith},
title = {Pith review of: POW: Political Overton Windows of Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/25IUBCDK}},
note = {Machine review of arXiv:2509.08853}
}
read the original abstract
Political bias in Large Language Models (LLMs) presents a growing concern for the responsible deployment of AI systems. Traditional audits often attempt to locate a model's political position as a point estimate, masking the broader set of ideological boundaries that shape what a model is willing or unwilling to say. In this paper, we draw upon the concept of the Overton Window as a framework for mapping these boundaries: the range of political views that a given LLM will espouse, remain neutral on, or refuse to endorse. To uncover these windows, we applied an auditing-based methodology, called PRISM, that probes LLMs through task-driven prompts designed to elicit political stances indirectly. Using the Political Compass Test, we evaluated twenty-eight LLMs from eight providers to reveal their distinct Overton Windows. While many models default to economically left and socially liberal positions, we show that their willingness to express or reject certain positions varies considerably, where DeepSeek models tend to be very restrictive in what they will discuss and Gemini models tend to be most expansive. Our findings demonstrate that Overton Windows offer a richer, more nuanced view of political bias in LLMs and provide a new lens for auditing their normative boundaries.
Figures
Forward citations
Cited by 1 Pith paper
-
Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies
Across 1,394 paired biographies, four LLM judges rated AI-written Grokipedia as less neutral than Wikipedia, with Grokipedia favoring economically right-wing politicians and Wikipedia favoring socially liberal ones.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. https://doi.org/10.1017/pan.2023.2 Out of one, many: Using language models to simulate human samples . Political Analysis, 31(3):337–351
-
[4]
Arnav Arora, Lucie-aim \'e e Kaffee, and Isabelle Augenstein. 2023. https://doi.org/10.18653/v1/2023.c3nlp-1.12 Probing pre-trained language models for cross-cultural differences in values . In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114--130, Dubrovnik, Croatia. Association for Computational Linguistics
-
[5]
Leif Azzopardi and Yashar Moshfeghi. 2024. http://arxiv.org/abs/2410.18906 Prism: A methodology for auditing biases in large language models . arXiv preprint arXiv:2410.18906
work page Pith review arXiv 2024
-
[6]
Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.acl-long.600 Measuring political bias in large language models: What is said and how it is said . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142--11159, Bangkok, Thailand. Associatio...
-
[7]
Maarten Buyl, Alexander Rogiers, Sander Noels, Iris Dominguez-Catena, Edith Heiter, Raphael Romero, Iman Johary, Alexandru-Cristian Mara, Jefrey Lijffijt, and Tijl De Bie. 2024. https://arxiv.org/abs/2410.18417 Large language models reflect the ideology of their creators . arXiv preprint arXiv:2410.18417
arXiv 2024
-
[8]
Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. 2022. https://doi.org/10.18653/v1/2022.acl-long.102 Spurious correlations in reference-free evaluation of text generation . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1443--1454, Dublin, Ireland. Association for Computational...
Show all 24 references
-
[9]
European Parliament . 2023. https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence Eu ai act: First regulation on artificial intelligence . Accessed: 2024-09-22
2023
-
[10]
David Exler, Mark Schutera, Markus Reischl, and Luca Rettenberger. 2025. http://arxiv.org/abs/2505.04393 Large means left: Political bias in large language models increases with their number of parameters
2025 arXiv
-
[11]
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.acl-long.656 From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models . In Proceedings of the 61st An...
2023 doi
-
[12]
Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768
2023 arXiv
-
[13]
Joseph G. Lehman. 2010. https://www.mackinac.org/7504 An introduction to the overton window of political possibility . Mackinac Center for Public Policy
2010
-
[14]
Jakob Mokander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. 2023. https://doi.org/10.1007/s43681-023-00289-2 Auditing large language models: a three-layered approach . AI and ethics, abs/2302.08500
2023 arXiv
-
[15]
Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2024. More human than human: measuring chatgpt political bias. Public Choice, 198:3--23
2024
-
[16]
Devon Myers, Rami Mohawesh, Venkata Ishwarya Chellaboina, Anantha Lakshmi Sathvik, Praveen Venkatesh, Yi-Hui Ho, Hanna Henshaw, Muna Alhawawreh, David Berdik, and Yaser Jararweh. 2024. https://doi.org/10.1007/s10586-023-04203-7 Foundation and large language models: fundamental...
2024 doi
-
[17]
Tai-Quan Peng, Kaiqi Yang, Sanguk Lee, Hang Li, Yucheng Chu, Yuping Lin, and Hui Liu. 2025. http://arxiv.org/abs/2412.16746 Beyond partisan leaning: A comparative analysis of political bias in large language models
2025
-
[18]
Luca Rettenberger, Markus Reischl, and Mark Schutera. 2025. https://doi.org/10.1007/s42001-025-00376-w Assessing political bias in large language models . Journal of Computational Social Science, 8(2):42--
2025 doi
-
[19]
o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \
Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \"u tze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. arXiv preprint arXiv:2402.16786
2024 arXiv
-
[20]
David Rozado. 2023. https://doi.org/10.3390/socsci12030148 The political biases of chatgpt . Social Sciences, 12(3)
2023 doi
-
[21]
David Rozado. 2024. https://doi.org/10.1371/journal.pone.0306621 The political preferences of llms . PLoS ONE, 19(7):e0306621
2024 doi
-
[22]
David Rozado. 2025. http://arxiv.org/abs/2503.10649 Measuring political preferences in ai systems: An integrative approach . arXiv preprint arXiv:2503.10649
2025 arXiv
-
[23]
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. http://arxiv.org/abs/2305.04388 Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting
2023 arXiv
-
[24]
Dustin Wright, Arnav Arora, Nadav Borenstein, Srishti Yadav, Serge Belongie, and Isabelle Augenstein. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.995 LLM tropes: Revealing fine-grained values and opinions in large language models . In Findings of the Association for ...
2024 doi
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.