Pith. sign in

REVIEW 3 major objections 5 minor 47 references

Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that large language models systematically rate left-leaning politicians more positively and right-leaning ones more negatively in target-oriented sentiment classification, across every model and language tested.

desk verdict A genuinely large-scale measurement of name-dependent sentiment variability in LLMs, but the main 'bias' claim rests on an unvalidated invariance assumption and is not yet established. read the letter →

arxiv 2505.19776 v1 pith:O3EDJVYL submitted 2025-05-26 cs.CL cs.AI

Akram Elbouanani , Evan Dufraisse , Adrian Popescu This is my paper
classification cs.CLcs.AI
keywords politicalbiaslargelanguagemodelstarget-orientedsentimentclassificationanalysisentity-levelmultilingualevaluationmeasurementmitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models carry a systematic political bias: when asked to judge the sentiment of a sentence toward a named politician, they rate left, center-left, and centrist politicians more positively and right and far-right politicians more negatively, in every model and language tested. The authors build this case with a large controlled experiment, 450 political sentences, 1,319 politicians, seven models, six languages, and about 25 million predictions, and a new inconsistency metric that measures how much the predicted sentiment changes when only the politician's name changes. They also show the bias mostly disappears when real names are swapped for fictional but plausible ones, indicating that political attributes, not demographics or fame, drive the effect. If the finding holds, it matters for any real-world deployment of LLMs in politically sensitive tasks such as content moderation or media coverage analysis.

What carries the argument

The load-bearing instrument is an entropy-based inconsistency metric. For a fixed sentence, the model's sentiment predictions are collected across all 1,319 politician names; the entropy of that label distribution, averaged over the 450 sentences, measures how much the predicted sentiment varies when only the target name changes. An unbiased model would give constant predictions and zero entropy, so IC equals zero. The same setup is run with a control group of fictional names, and the drop in inconsistency after name replacement is what isolates the political component of the bias.

What would settle it

Run the same 450 sentences through the same models with a matched group of fictional names whose assigned party labels are randomly permuted; if the political-alignment gradient in sentiment persists under a permutation that severs name from alignment, then the measured effect is not actually carried by political affiliation. Alternatively, have human raters score the same name-swapped sentences; if humans show the same left-right gradient, the 'bias' is partly a property of the sentence content rather than of the models.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that target-oriented sentiment classification by LLMs is systematically skewed by the political alignment of the target entity: sentiment predictions become more positive for politicians on the left, center-left, and center, and more negative for those on the right and far right, with the effect present across all seven tested models and all six tested languages. The bias is stronger in English, French, and Spanish than in Russian, Arabic, and Chinese, and stronger in larger models than in their smaller counterparts. A two-dimensional political compass analysis shows the most positive sentiment goes to left-libertarian figures and the most negative to right-authoritarian ones. The paper further reports that replacing politician names with fictional names that preserve gender, birth year, and country largely removes the alignment-driven component of the bias, which the authors take as evidence that political attributes are the primary source.

Load-bearing premise

The measurement treats any change in predicted sentiment when only the politician's name changes as bias, but the sentences contain normative and evaluative content where different politicians could legitimately evoke different sentiments, and the fictional-name control only matches gender, birth year, and country, not fame or prior associations.

Editorial extensions

If this is right

  • Politically sensitive NLP applications such as content moderation, stance detection, and media-coverage analysis inherit this bias when built on LLMs, so their outputs about right-leaning figures will skew negative.
  • Larger models do not cure the bias; they intensify it, so scaling alone is not a mitigation strategy.
  • The bias is not confined to English: it appears in all six tested languages, though Western languages show stronger effects.
  • Replacing politician names with non-political fictional names offers a cheap, model-agnostic partial mitigation, cutting inconsistency while slightly improving accuracy.
  • Models encode structured knowledge about individual politicians: sentiment patterns cluster politicians with similar ideologies and contexts, which the paper reads as evidence of internal entity representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: the same inconsistency metric could serve as a general probe for bias toward other entity classes such as companies, ethnic groups, or public institutions, wherever a sentence can be re-issued with a substitution.
  • If the left-leaning bias reflects training-data frequency and narrative valence rather than intent, then model updates that rebalance political coverage could shift the measured bias, making periodic re-auditing necessary.
  • A testable consequence beyond the paper: the entity-similarity matrix should predict transfer of sentiment between politicians in unseen sentences; one could hold out a subset of politicians and check whether sentiment patterns generalize along the similarity structure.
  • The finding that fictional names still carry demographic residue, with female names rated higher and Russian names lower, suggests a second-order bias that the paper measures but does not fully separate from political alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a new framework for measuring political bias in large language models (LLMs) by using target-oriented sentiment classification (TSC). The authors insert 1,319 politician names into 450 political sentences (with male/female variants), evaluate seven models in six languages, and define an entropy-based inconsistency metric (Eq. 2) that treats any variation in predicted sentiment across target entities as bias. They report a systematic pattern where left, center-left, and centrist politicians receive more positive sentiment and right and far-right politicians receive more negative sentiment, across all tested models and languages. They also show that replacing real politician names with GPT-4-generated fictional names reduces inconsistency, which they interpret as evidence that political attributes are the primary driver of the observed biases. The central claim is that LLMs exhibit robust, systematic political bias in a downstream task, with language, model-size, and individual-entity analyses.

Significance. If the central claim survives scrutiny, the paper offers a scalable and statistically powerful alternative to questionnaire- and generation-based bias measurements. Its strengths include a large dataset (nearly 25 million prompt-level predictions), a clear and simple inconsistency metric, multilingual coverage, multiple model families, and a concrete mitigation attempt. The paper also promises to release code and data, which would aid reproducibility. However, the validity of the bias measurement rests entirely on the entity-invariance assumption of Eq. (2), which is not empirically validated, and the control group confounds political alignment with fame and salience. These issues make the central claim underdetermined in the current form.

major comments (3)
  1. [Section 3.1, Eq. (2)] The inconsistency metric assumes that an unbiased LLM should produce the same sentiment prediction for a sentence regardless of the target entity, so any entropy in sentiment across entities is treated as bias. This entity-invariance assumption is never validated. The paper's own results in Section H show that the strongest negative sentiment signals come from notorious historical figures (Hitler -0.81, Himmler -0.55, Stalin -0.34); for these entities, a negative sentiment is a reasonable response independent of the sentence frame. Since the entity set is sampled by frequency of news mentions, far-right and far-left alignments contain a disproportionate share of such high-salience, valence-laden figures. The gradient in Figure 2 could therefore reflect real-world associations rather than an LLM-specific political bias. The paper needs a human annotation baseline or a control for entity salience to separate these explanations; without it, the central claim that LLMs are systematically biased by political alignment is not established.
  2. [Section C and Figure 7] The control group replaces real politician names with GPT-4-generated fictional names that preserve gender, birth year, and country of origin. This manipulation simultaneously removes fame, recognition, and all real-world biographical associations, not just political alignment. The observed reduction in inconsistency after replacement therefore does not isolate political alignment as the causal factor. For example, a famous politician like Angela Merkel is replaced by a completely unknown fictional person, so any decrease in sentiment variability could be due to the removal of salience rather than the removal of political attributes. The Limitations section acknowledges demographic confounds but does not address fame. The authors should include an additional control condition using real non-political celebrities or public figures matched by fame, or otherwise demonstrate that fame does not explain the alignment gradient.
  3. [Appendix J] The Mann-Whitney U tests reported in Appendix J seem to treat individual prediction outcomes as independent observations. With 593,550 prompts per model-language combination (Section 3.4), the tests are extremely high-powered and will reject the null hypothesis for trivially small effect sizes. Moreover, observations are not independent: each entity contributes 450 sentiment predictions, and each sentence is evaluated with all 1,319 entities. A proper statistical analysis should account for this clustering, for example by using entity-level or sentence-level random effects, or by bootstrapping at the entity level. Without such an adjustment, the claim of 'statistically robust analysis at different granularity levels' is not supported.
minor comments (5)
  1. [Appendix D] The definition section begins with a duplicated line: "We define the eight political alignments used in the analysis below: We define the eight political alignments used in the analysis below:". Please remove the repetition.
  2. [Appendix J (Tables 6-12)] The term "Mann-Withney" is a typo for "Mann-Whitney" in several table captions and the surrounding text.
  3. [Algorithm 1] The 'round away from zero' rule in the alignment computation is ad hoc and can flip a party's alignment label based on the averaged score. Since political alignment is the main independent variable, the sensitivity of the results to this rounding choice should be discussed or tested.
  4. [Figure 2] Panels (a) and (b) of Figure 2 both have identical y-axis labels, and the caption could be clearer about which panel corresponds to languages and which to models. The reader should not have to infer this from the main text.
  5. [Section 4] The statement that "we can safely assume they are similar within the same family of models" about training corpora is speculative; since training data are not public for all models, this should be phrased as an assumption, and the possibility that other factors (e.g., alignment tuning, data mixture) explain model-size differences should be acknowledged.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the political-bias conclusion is a direct aggregate of LLM sentiment outputs across external entity labels, with no fitted parameter or self-citation chain reducing the result to its inputs.

full rationale

The paper's measurement chain is transparent and self-contained: it takes 450 sentences from the external MAD-TSC dataset, replaces the target with 1,319 politician names whose alignments come from Wikidata and ParlGov, prompts seven LLMs for sentiment, and then computes per-alignment average sentiment and Mann-Whitney U tests. The bias conclusion in Figure 2 is a direct aggregation of raw sentiment predictions, not an output of the inconsistency metric; the metric in Eq. 2 is introduced only to quantify prediction variability and is not used as the measure of political bias. The control group with GPT-4-generated fictional names is an experimental manipulation, not a derivation, and the paper's Limitations section explicitly acknowledges residual demographic confounds and data representativeness limits. The only self-citations are to MAD-TSC (Dufraisse et al., 2023) and a related media-bias paper (Dufraisse et al., 2024); these are external, published resources used as data and related work, not as authority for the paper's own conclusion. No equation reduces to another by construction, and no fitted parameter is renamed as a prediction. The entity-invariance assumption is a substantive interpretive choice, but it is not circular: the paper does not define 'political bias' as 'sentiment variability'; it measures sentiment differences across externally labeled alignments.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. The primary assumptions are the normative baseline of entity-invariance, the reliability of external political labels, and the adequacy of the control group in isolating political from non-political attributes. The free parameters are choices in the data construction pipeline that could affect the measured bias.

free parameters (4)
  • Hierarchical sampling k1, k2, k3, k4 = Not specified in paper
    These control how many politicians are sampled per country and alignment. The values are chosen by hand but not reported, affecting the final entity balance.
  • Alignment mapping scores = Far-Left=-3, Left-Wing=-2, Center-Left=-1, Center=0, Center-Right=1, Right-Wing=2, Far-Right=3
    Arbitrary integer scores used in Algorithm 1 to average multiple party alignments. The choice of linear spacing and round-away-from-zero rule is ad hoc.
  • Round-away-from-zero in alignment averaging = Yes
    Algorithm 1 rounds the average alignment away from zero, which is a deliberate choice to avoid centrist bias but has no empirical justification.
  • Sentence selection thresholds = Manual decision diagram
    The decision diagram (Appendix E) is a manual, non-quantified filter for which sentences are 'general enough' for entity replacement.
assumptions (5)
  • domain assumption An unbiased LLM should give the same sentiment for a given sentence regardless of the target entity.
    Section 3.1 defines inconsistency as deviation from this entity-invariance. This is the normative baseline against which all bias is measured, but it is not empirically validated.
  • domain assumption Political alignment labels from Wikidata and ParlGov are accurate for all 1319 politicians.
    Alignments are derived from party-level data and averaged, which can misclassify politicians with complex or changing affiliations. This is used to aggregate sentiment by alignment.
  • domain assumption The 450 selected sentences are semantically general enough that replacing the target entity does not change the sentence's factual or counterfactual meaning.
    Appendix E describes a manual filtering process, but the criteria are subjective. If a sentence contains hidden entity-specific implications, the sentiment variability may reflect factual mismatch rather than bias.
  • domain assumption The fictional-name control group preserves all non-political attributes (gender, birth year, country) and removes political ones.
    Fictional names are generated by GPT-4 and may carry residual demographic or cultural connotations that differ from the original politicians, confounding the isolation of political attributes.
  • domain assumption Models within the same family (Qwen-7B vs 72B, Llama-8B vs 70B) differ primarily in scale, with similar training data.
    The paper claims 'we can safely assume they are similar within the same family' to attribute observed differences to model size, but training data and other factors may differ.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification." pith.science (2026). https://pith.science/paper/O3EDJVYL

@misc{pith2026250519776,
  author       = {Pith},
  title        = {Pith review of: Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O3EDJVYL}},
  note         = {Machine review of arXiv:2505.19776}
}
read the original abstract

Political biases encoded by LLMs might have detrimental effects on downstream applications. Existing bias analysis methods rely on small-size intermediate tasks (questionnaire answering or political content generation) and rely on the LLMs themselves for analysis, thus propagating bias. We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence. We define an entropy-based inconsistency metric to encode this prediction variability. We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages. We observe inconsistencies in all tested combinations and aggregate them in a statistically robust analysis at different granularity levels. We observe positive and negative bias toward left and far-right politicians and positive correlations between politicians with similar alignment. Bias intensity is higher for Western languages than for others. Larger models exhibit stronger and more consistent biases and reduce discrepancies between similar languages. We partially mitigate LLM unreliability in target-oriented sentiment classification (TSC) by replacing politician names with fictional but plausible counterparts.

Figures

Figures reproduced from arXiv: 2505.19776 by the authors.

Figure 1
Figure 1. Sentiment-prediction based analysis of LLM [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Average sentiment scores for languages (aggregated across all models) and for models (aggregated [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Boxplots depicting average sentiment scores [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Jaccard similarity index between the sentiment predictions in the tested languages obtained with Qwen-7B, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Political compasses showing sentiment bias [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Average sentiment scores for languages and for models using a set of fake names reflecting non-political [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Alignment distribution among entities ex [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Countries distribution among entities extracted from the CC-News corpus (in blue) and among entities [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Prompt used for fake name generation. Place [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Sentence selection diagram. This decision diagram outlines the process of either rejecting or selecting [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Steps for translating a sentence while ensuring gender adaptability. The process begins by creating two [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Prompts used for the experiment across six languages: English, French, and Spanish (first row), and [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 16 canonical work pages

  1. [1]

    Marwa Abdulhai, Gregory Serapio-Garc \'i a , Clement Crepy, Daria Valter, John Canny, and Natasha Jaques. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.982 Moral Foundations of Large Language Models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 17737--17752, Miami, Florida, USA. Association for Com...

    doi:10.18653/v1/2024.emnlp-main.982 2024

  2. [2]

    Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, and Monojit Choudhury. 2024. Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language We Prompt Them in. In Proceedings of the 2024 Joint International Conference on Computational Linguistics , Language Resources and Evaluation ( LREC-COLING 2024) , pages 6330--6340, Torino, Italia. ELRA and ICCL

    work page 2024

  3. [3]

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (demonstrations), pages 54--59

    work page 2019

  4. [4]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. https://doi.org/10.18653/v1/2024.acl-long.671 Investigating Cultural Alignment of Large Language Models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages 12404--12422, Bangkok, Thailand. Association for Co...

    doi:10.18653/v1/2024.acl-long.671 2024

  5. [5]

    Jiafu An, Difang Huang, Chen Lin, and Mingzhu Tai. 2025. https://doi.org/10.1093/pnasnexus/pgaf089 Measuring gender and racial biases in large language models: Intersectional evidence from automated resume evaluation . PNAS Nexus, 4(3):pgaf089

    doi:10.1093/pnasnexus/pgaf089 2025

  6. [6]

    Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.acl-long.600 Measuring Political Bias in Large Language Models : What Is Said and How It Is Said . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages 11142--11159, Bangkok, Thailand. Associ...

    doi:10.18653/v1/2024.acl-long.600 2024

  7. [7]

    Martin Juan Jos \'e Bucher and Marco Martini. 2024. Fine-tuned'small'llms (still) significantly outperform zero-shot generative ai models in text classification. arXiv preprint arXiv:2406.08660

    arXiv 2024

  8. [8]

    Maarten Buyl, Alexander Rogiers, Sander Noels, Iris Dominguez-Catena , Edith Heiter, Raphael Romero, Iman Johary, Alexandru-Cristian Mara, Jefrey Lijffijt, and Tijl De Bie. 2024. https://doi.org/10.48550/arXiv.2410.18417 Large Language Models Reflect the Ideology of their Creators . Preprint, arXiv:2410.18417

    arXiv doi:10.48550/arxiv.2410.18417 2024

Show all 47 references
  1. [9]

    Ke-Li Chiu, Annie Collins, and Rohan Alexander. 2022. https://doi.org/10.48550/arXiv.2103.12407 Detecting Hate Speech with GPT-3 . Preprint, arXiv:2103.12407

    2022 doi arXiv

  2. [10]

    Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. https://doi.org/10.1162/tacl_a_00460 Multilingual autoregressive entity linking . Transactions of the Associati...

    2022 doi

  3. [11]

    Holger D \"o ring and Philip Manow. 2012. Parliament and government composition database (parlgov). An infrastructure for empirical information on parties, elections and governments in modern democracies. Version, 12(10)

    2012

  4. [12]

    Evan Dufraisse, Adrian Popescu, Julien Tourille, Armelle Brun, and Jerome Deshayes. 2023. https://doi.org/10.18653/v1/2023.acl-long.461 MAD - TSC : A multilingual aligned news dataset for target-dependent sentiment classification . In Proceedings of the 61st Annual Meeting of ...

    2023 doi

  5. [13]

    Evan Dufraisse, Adrian Popescu, Julien Tourille, Armelle Brun, and Olivier Hamon. 2024. https://arxiv.org/abs/2408.11174 Combining objective and subjective perspectives for political news understanding . Preprint, arXiv:2408.11174

    2024 arXiv

  6. [14]

    Hans J Eysenck. 2018. The psychology of politics. Routledge

    2018

  7. [15]

    Fabian Falck, Julian Marstaller, Niklas Stoehr, S \"o ren Maucher, Jeana Ren, Andreas Thalhammer, Achim Rettinger, and Rudi Studer. 2020. https://doi.org/10.1002/poi3.222 Measuring Proximity Between Newspapers and Political Parties : The Sentiment Political Compass . Policy & ...

    2020 doi

  8. [16]

    Lucas Gover. 2023. Political bias in large language models. The Commons: Puget Sound Journal of Politics, 4(1):2

    2023

  9. [17]

    Katharina H \"a mmerl, Bjoern Deiseroth, Patrick Schramowski, Jind r ich Libovick \'y , Constantin Rothkopf, Alexander Fraser, and Kristian Kersting. 2023. https://doi.org/10.18653/v1/2023.findings-acl.134 Speaking Multiple Languages Affects the Moral Bias of Language Models ....

    2023 doi

  10. [18]

    Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. https://doi.org/10.48550/arXiv.2301.01768 The political ideology of conversational AI : Converging evidence on ChatGPT 's pro-environmental, left-libertarian orientation . Preprint, arXiv:2301.01768

    2023 doi arXiv

  11. [19]

    Andrew Heywood. 2021. Political ideologies: An introduction. Bloomsbury Publishing

    2021

  12. [20]

    Yi-Li Hsu, Jui-Ning Chen, Yang Fan Chiang, Shang-Chien Liu, Aiping Xiong, and Lun-Wei Ku. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.137 Enhancing perception: Refining explanations of news claims with LLM conversations . In Findings of the Association for Computatio...

    2024 doi

  13. [21]

    Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.7 Reducing Sentiment Bias in Language Models via Counterfactual Evaluation . In Findings of t...

    2020 doi

  14. [22]

    Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022. CommunityLM : Probing Partisan Worldviews from Language Models . In Proceedings of the 29th International Conference on Computational Linguistics , pages 6818--6826, Gyeongju, Republic of Korea. International Committe...

    2022

  15. [23]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  16. [24]

    Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. 2024. Llm-mod: Can large language models assist content moderation? In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1--8

    2024

  17. [25]

    Alejandro Lopez-Lira and Yuehua Tang. 2023. https://doi.org/10.2139/ssrn.4412788 Can ChatGPT Forecast Stock Price Movements ? Return Predictability and Large Language Models . Preprint, Social Science Research Network:4412788

    2023 doi

  18. [26]

    Chenyang Lyu, Minghao Wu, and Alham Aji. 2024. https://doi.org/10.18653/v1/2024.knowllm-1.10 Beyond Probabilities : Unveiling the Misalignment in Evaluating Large Language Models . In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models ( KnowLLM 2024) , pa...

    2024 doi

  19. [27]

    Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R Trippas, J Shane Culpepper, and Alistair Moffat. 2020. Cc-news-en: A large english news corpus. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3077--3084

    2020

  20. [28]

    Robert W. McGee. 2023. https://doi.org/10.2139/ssrn.4359405 Is Chat Gpt Biased Against Conservatives ? An Empirical Study . Preprint, Social Science Research Network:4359405

    2023 doi

  21. [29]

    Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2024. https://doi.org/10.1007/s11127-023-01097-2 More human than human: Measuring ChatGPT political bias . Public Choice, 198(1):3--23

    2024 doi

  22. [30]

    Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. https://doi.org/10.18653/v1/2024.acl-long.862 Having Beer after Prayer ? Measuring Cultural Bias in Large Language Models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( ...

    2024 doi

  23. [31]

    Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.146 The Shifted and The Overlooked : A Task-oriented Investigation of User-GPT Interactions . In Proc...

    2023 doi

  24. [32]

    Thomas Pellissier Tanon, Denny Vrande c i \'c , Sebastian Schaffert, Thomas Steiner, and Lydia Pintscher. 2016. From freebase to wikidata: The great migration. In Proceedings of the 25th international conference on world wide web, pages 1419--1428

    2016

  25. [33]

    Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819

    2024 arXiv

  26. [34]

    o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch \"u tze, and Dirk Hovy. 2024. Political Compass or Spinning Arrow ? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models . https://arxiv.org/abs/...

    2024 arXiv

  27. [35]

    David Rozado. 2024. https://doi.org/10.1371/journal.pone.0306621 The political preferences of LLMs . PLOS ONE, 19(7):e0306621

    2024 doi

  28. [36]

    Abel Salinas, Parth Shah, Yuzhong Huang, Robert McCormack, and Fred Morstatter. 2023. The unequal opportunities of large language models: Examining demographic biases in job recommendations by chatgpt and llama. In Proceedings of the 3rd ACM Conference on Equity and Access in ...

    2023

  29. [37]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose Opinions Do Language Models Reflect ? In Proceedings of the 40th International Conference on Machine Learning , pages 29971--30004. PMLR

    2023

  30. [38]

    Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. 2024. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36

    2024

  31. [39]

    Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. https://doi.org/10.18653/v1/2021.acl-long.330 Societal Biases in Language Generation : Progress and Challenges . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and th...

    2021 doi

  32. [40]

    Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. https://doi.org/10.18653/v1/D19-1339 The Woman Worked as a Babysitter : On Biases in Language Generation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the ...

    2019 doi

  33. [41]

    Ungless, Bj \"o rn Ross, and Vaishak Belle

    Eddie L. Ungless, Bj \"o rn Ross, and Vaishak Belle. 2023. https://doi.org/10.1177/08944393231152946 Potential Pitfalls With Automatic Sentiment Analysis : The Example of Queerphobic Bias . Social Science Computer Review, 41(6):2211--2229

    2023 doi

  34. [42]

    Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, AiTi Aw, and Nancy Chen. 2024 a . https://doi.org/10.18653/v1/2024.naacl-long.22 SeaEval for Multilingual Foundation Models : From Cross-Lingual Alignment to Cultural Reasoning . In Proceedings of the 2024 Conference...

    2024 doi

  35. [43]

    Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel , Paul R \"o ttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.441 `` My Answer is C '': First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned La...

    2024 doi

  36. [44]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. https://arxiv.org/abs/2303.17564 Bloomberggpt: A large language model for finance . Preprint, arXiv:2303.17564

    2023 arXiv

  37. [45]

    Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Yuqi Ye, and Hanwen Gu. 2024. A survey on multilingual large language models: Corpora, alignment, and bias. arXiv preprint arXiv:2404.00929

    2024 arXiv

  38. [46]

    Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language models. arXiv preprint arXiv:2205.12247

    2022 arXiv

  39. [47]

    Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.246 Sentiment Analysis in the Era of Large Language Models : A Reality Check . In Findings of the Association for Computational Linguistics : NAACL 2024 , pages...

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.