Political Plasticity: An Analysis of Ideological Adaptability in Large Language Models
Pith reviewed 2026-05-12 01:47 UTC · model grok-4.3
The pith
Newer and larger LLMs adapt their political responses to user context while smaller and older models do not.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that political plasticity increases with model scale and recency. Newer frontier models reliably alter their answers to the 200-question set when given user prompts containing few-shot examples, especially on economic freedom items, producing shifts in the expected direction. Older and smaller models exhibit limited or unstable changes. System prompts fail to induce comparable shifts, and inverting the sense of the questions produces counter-intuitive results in most models, indicating possible recognition of question formats rather than engagement with content.
What carries the argument
A 200-question questionnaire spanning economic and personal freedom axes, paired with few-shot user prompts to induce and measure directional shifts in model answers.
If this is right
- User prompts with examples can steer larger models toward specific positions on economic freedom questions.
- System-level instructions alone do not produce measurable ideological shifts in LLMs.
- Political plasticity appears more stable and directionally consistent in newer models than in older ones.
- Inverting question wording triggers unexpected answer changes, suggesting models may recognize questionnaire formats.
- The same prompt experiments yield subtle differences in adaptability when conducted in languages other than English.
Where Pith is reading between the lines
- If plasticity scales with model size, real-world applications could steer large models more easily than small ones toward desired political framings.
- The results raise the possibility that recent training data encodes political topics with less rigidity than earlier data.
- A direct follow-up test could measure whether plasticity tracks parameter count or training recency more closely.
- The same questionnaire method might reveal analogous adaptability patterns in non-political domains such as moral or factual reasoning.
Load-bearing premise
That changes in answers to the fixed set of 200 questions reflect an underlying ideological position rather than surface pattern matching or training data artifacts.
What would settle it
If frontier models produce unchanged political scores on the 200 questions even after receiving explicit few-shot user examples that clearly advocate a different ideology, the claim of reliable plasticity would not hold.
Figures
read the original abstract
Since the advent of Large Language Models (LLMs), a significant area of research has focused on their intrinsic biases, particularly in political discourse. This study investigates a different but related concept, "political plasticity", which is defined as the capacity of models to adapt their responses based on the user supplied context. To analyze this, a testing framework was developed using an expanded corpus of 200 politically-oriented questions across economic and personal freedom axes, based on a prior framework by Lester (1996). The study explored several methods to induce political bias, including simplified and topic-based system prompts, as well as user prompts with few-shot examples. The results show that while system prompts were largely ineffective, user prompts successfully elicited significant ideological shifts, particularly along the Economic Freedom axis in larger and newer models. Through a validation experiment, we examined whether models answer questionnaires by recognizing the underlying question format. Inverting the sense of the questions revealed unexpected, counter-intuitive shifts in most models, suggesting potential data leakage. Finally, we also analyzed how model plasticity varies when the experiment is conducted in different languages. The results reveal subtle yet notable shifts across each of the analyzed languages. Overall, our results indicate that small and older LLMs exhibit limited or unstable political plasticity, whereas newer frontier models display reliable, expected adaptability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript defines 'political plasticity' as LLMs' capacity to adapt responses to a fixed corpus of 200 politically-oriented questions (expanded from Lester 1996) across economic and personal freedom axes when given user context. It tests induction via simplified/topic-based system prompts (largely ineffective) versus user few-shot prompts (effective, especially on Economic Freedom in larger/newer models). A validation experiment inverts question sense and reports counter-intuitive shifts in most models, interpreted as data leakage. Multilingual tests show subtle shifts. The central claim is that small/older LLMs exhibit limited or unstable plasticity while newer frontier models display reliable, expected adaptability.
Significance. If the reported shifts genuinely reflect ideological adaptability rather than memorization or pattern-matching artifacts, the work would provide a useful empirical benchmark for prompt-induced bias in LLMs and highlight generational differences relevant to alignment and safety. The use of an external, established questionnaire framework supplies a falsifiable test, but the current absence of quantitative controls and disambiguation experiments limits the strength of any broader claims about model behavior.
major comments (3)
- [Validation experiment] Validation experiment: the interpretation of counter-intuitive shifts under question inversion as evidence of data leakage is load-bearing for the plasticity metric, yet no quantitative details are supplied on shift magnitudes, per-model sample sizes, variance, or controls for prompt length/ordering. If leakage dominates, observed differences between small/older and frontier models could reflect memorization capacity rather than adaptability.
- [Results] Results on prompt effectiveness: the directional claim that user few-shot prompts induce significant ideological shifts (particularly Economic Freedom) while system prompts do not, with reliable effects only in newer models, is presented without error bars, statistical tests, or model-by-model sample sizes. This makes it impossible to assess whether the reported distinction between model classes is robust.
- [Methodology] Methodology and weakest assumption: the framework treats responses to the fixed 200-question set as indicators of underlying ideological position, but the inversion results directly challenge this without follow-up experiments (e.g., paraphrased controls or non-political baselines) to separate plasticity from surface artifacts or training-data effects.
minor comments (2)
- [Abstract] The abstract states 'subtle yet notable shifts' across languages but supplies no list of languages tested, quantitative measures, or per-axis breakdowns.
- [Methodology] Notation for the two axes (economic vs. personal freedom) and how plasticity is scored (e.g., direction and magnitude of shift) should be defined explicitly in the main text rather than left implicit.
Simulated Author's Rebuttal
We thank the referee for the detailed and constructive report. The comments highlight important areas for strengthening the empirical presentation and clarifying the interpretation of the validation results. We address each major comment below and indicate the revisions planned for the next version of the manuscript.
read point-by-point responses
-
Referee: Validation experiment: the interpretation of counter-intuitive shifts under question inversion as evidence of data leakage is load-bearing for the plasticity metric, yet no quantitative details are supplied on shift magnitudes, per-model sample sizes, variance, or controls for prompt length/ordering. If leakage dominates, observed differences between small/older and frontier models could reflect memorization capacity rather than adaptability.
Authors: We agree that the current manuscript provides insufficient quantitative detail on the inversion results. In the revised version we will add a dedicated table reporting per-model mean shift magnitudes (with standard deviations), effective sample sizes after filtering, and any observed effects of prompt length or ordering. We will also include a brief analysis comparing shift sizes between the main prompt conditions and the inverted condition. On the interpretation, the counter-intuitive direction of shifts in the majority of models is difficult to explain by semantic understanding alone and is therefore presented as suggestive of data leakage or format memorization. We acknowledge, however, that this does not rule out memorization capacity as a partial explanation for generational differences; the revised text will explicitly note this possibility and will separate the leakage interpretation from the primary claim that frontier models exhibit more reliable prompt-driven adaptation when the question sense is preserved. revision: partial
-
Referee: Results on prompt effectiveness: the directional claim that user few-shot prompts induce significant ideological shifts (particularly Economic Freedom) while system prompts do not, with reliable effects only in newer models, is presented without error bars, statistical tests, or model-by-model sample sizes. This makes it impossible to assess whether the reported distinction between model classes is robust.
Authors: We accept that the absence of error bars, statistical tests, and explicit per-model sample sizes weakens the current presentation. The revised manuscript will include error bars on all bar plots and line charts, report the number of valid responses per model and condition, and add statistical comparisons (paired t-tests or Wilcoxon tests within models, and between-group tests across model-size cohorts) for the key contrasts between user few-shot and system-prompt conditions. These additions will allow readers to evaluate the robustness of the claimed generational difference in plasticity. revision: yes
-
Referee: Methodology and weakest assumption: the framework treats responses to the fixed 200-question set as indicators of underlying ideological position, but the inversion results directly challenge this without follow-up experiments (e.g., paraphrased controls or non-political baselines) to separate plasticity from surface artifacts or training-data effects.
Authors: The inversion experiment was included precisely because we recognized that the fixed-question format could be vulnerable to surface or memorization effects. The observed counter-intuitive shifts are reported as evidence that the assumption does not hold uniformly across models. We did not conduct additional paraphrased-question or non-political baseline runs in the original study. In the revision we will expand the limitations and future-work sections to discuss this gap explicitly and to outline how paraphrased controls could be used to isolate semantic plasticity from format recognition. We maintain that the differential pattern—limited and unstable adaptation in smaller/older models versus consistent, directionally expected adaptation in frontier models—still provides useful comparative evidence even while acknowledging that some portion of the observed shifts may be artifactual. revision: partial
Circularity Check
No circularity: purely empirical measurement on external questionnaire framework
full rationale
The paper defines political plasticity operationally as the capacity of LLMs to adapt responses to supplied user context and measures it directly by recording shifts in answers to a fixed corpus of 200 questions drawn from the external Lester (1996) framework. No equations, fitted parameters, or derived quantities are introduced that reduce the reported adaptability scores to quantities defined by the authors' own choices. System-prompt and few-shot experiments are presented as independent interventions whose effects are observed rather than presupposed. The inversion validation is reported as a separate empirical finding (potential data leakage) rather than a load-bearing premise that circularly justifies the plasticity metric. No self-citations appear in the load-bearing steps, and the distinction between small/older versus frontier models rests on observed differences against the external benchmark, making the derivation self-contained.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The Lester (1996) political questionnaire provides a valid, language-independent measure of positions along economic and personal freedom axes.
invented entities (1)
-
political plasticity
no independent evidence
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
plasticity... capacity of models to adapt their responses based on the user supplied context... 200 politically-oriented questions across economic and personal freedom axes... user prompts successfully elicited significant ideological shifts... inverting the sense of the questions revealed unexpected, counter-intuitive shifts... suggesting potential data leakage
-
IndisputableMonolith/Foundation/AlphaCoordinateFixation.leanJ_uniquely_calibrated_via_higher_derivative unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
small and older LLMs exhibit limited or unstable political plasticity, whereas newer frontier models display reliable, expected adaptability
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
-
[1]
Journal of social philosophy , volume=
The political compass and why libertarianism is not right-wing , author=. Journal of social philosophy , volume=
-
[2]
Improving language understanding by generative pre-training , author=. 2018 , publisher=
work page 2018
-
[3]
arXiv preprint arXiv:2408.15266 , year=
People over trust AI-generated medical responses and view them to be as valid as doctors, despite low accuracy , author=. arXiv preprint arXiv:2408.15266 , year=
-
[4]
arXiv preprint arXiv:2403.18932 , year =
Measuring political bias in large language models: What is said and how it is said , author=. arXiv preprint arXiv:2403.18932 , year=
-
[5]
The political preferences of LLMs , author=. PloS one , volume=. 2024 , publisher=
work page 2024
-
[6]
arXiv preprint arXiv:2410.24190 , year=
Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters , author=. arXiv preprint arXiv:2410.24190 , year=
-
[7]
Feng, Shangbin and Park, Chan Young and Liu, Yuhan and Tsvetkov, Yulia. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.656
-
[8]
International Conference on Machine Learning , pages=
Whose opinions do language models reflect? , author=. International Conference on Machine Learning , pages=. 2023 , organization=
work page 2023
-
[9]
arXiv preprint arXiv:2410.09978 , year=
When Neutral Summaries are not that Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries , author=. arXiv preprint arXiv:2410.09978 , year=
-
[10]
The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation , author=. arXiv preprint arXiv:2301.01768 , year=
-
[11]
A Closer Look at System Prompt Robustness , 2025
A Closer Look at System Prompt Robustness , author=. arXiv preprint arXiv:2502.12197 , year=
-
[12]
arXiv preprint arXiv:2508.16013 , year=
Political Ideology Shifts in Large Language Models , author=. arXiv preprint arXiv:2508.16013 , year=
-
[13]
Journal of the American Statistical Association , number=
On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization , author=. Journal of the American Statistical Association , number=. 2025 , publisher=
work page 2025
-
[14]
Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume=
GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy , author=. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume=
-
[15]
Comparing the persuasiveness of role-playing large language models and human experts on polarized US political issues , author=. OSF Preprints , volume=
-
[16]
The Fourth Blogpost Track at ICLR 2025 , year=
LLMs' Potential Influences on Our Democracy: Challenges and Opportunities , author=. The Fourth Blogpost Track at ICLR 2025 , year=
work page 2025
-
[17]
arXiv preprint arXiv:2412.12175 , year=
Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning , author=. arXiv preprint arXiv:2412.12175 , year=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.