{"id":"e01d1c68-2836-466e-aeb5-df81130d46c9","arxiv_id":"2505.19776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs show systematic target-dependent sentiment inconsistency that is politically biased: left and center politicians rated more positively, far-right politicians more negatively, with stronger effects in larger models and Western languages.","lead":"Researchers measured whether seven large language models give consistently different sentiment ratings for the same political sentence when the politician's name is swapped, and found systematic left-leaning and anti-far-right biases across six languages. The paper offers a new inconsistency metric that could be used to audit LLMs for political bias in downstream tasks like content moderation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sentiment variability across target entities is treated as bias, but the metric likely captures legitimate entity-specific sentiment; the control group cannot separate political alignment from fame, so the central bias claim is not yet established.","rationale":"The reader's weakest_assumption identifies the entity-invariance assumption behind the inconsistency metric; I agree that this is the most load-bearing point. My reading of the appendix strengthens the concern: the entity-level results in Section H show that the largest sentiment deviations are associated with historical dictators and universally admired figures, not with rank-and-file politicians. Because the entity set is deliberately composed of frequently mentioned politicians, the extreme alignments are populated by high-salience, high-valence individuals, making it difficult to attribute the Figure 2 gradient to political alignment rather than to general knowledge about those individuals. The control group is meant to address this, but replacing real names with fictional ones removes not only political affiliation but also fame and biographical associations, so the reduced inconsistency in the control condition is overinterpreted as evidence that politics is the primary driver. A human-rating baseline directly tests whether the measured variability is bias or reasonable sentiment; excluding notorious outliers tests whether the aggregate gradient is robust. Both checks are feasible on the existing dataset. The conditional verdict is appropriate: the empirical pattern is real, but its interpretation as political bias requires the entity-invariance assumption to hold, and the paper does not yet provide evidence for that assumption.","tokens_in":35623,"tokens_out":5596,"duration_ms":68797,"concrete_test":"Select a stratified subsample of 300 (sentence, politician) pairs spanning all eight alignments and the full sentiment range, and collect human sentiment ratings using the same 3-class scale under the same sentence frames. If human raters reproduce the same alignment gradient (more positive for left/center, more negative for right/far-right), then the variability measured by Eq. 2 is not LLM-specific bias but legitimate entity-dependent sentiment, and the paper's entity-invariance assumption fails. If human ratings are flat across alignments while LLM predictions show the gradient, the bias claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is entity invariance: Eq. 2 defines inconsistency as any entropy in sentiment predictions when the target entity changes, and the paper treats this variability as political bias. This assumption is never validated, and it is directly contradicted by the paper's own appendix: the strongest signals come from notorious historical figures (Hitler -0.81, Himmler -0.55, Stalin -0.34 in Section H), entities for which a negative sentiment is a reasonable response regardless of the sentence frame. Because the entity set is sampled from frequent news mentions, far-right and far-left alignments contain a disproportionate number of such high-salience, valence-laden figures, so the observed left-positive/right-negative gradient in Figure 2 could reflect real-world associations rather than LLM political bias. The control group (Section C) replaces real names with GPT-4-generated fictional names that preserve gender, birth year, and country, but that manipulation simultaneously removes fame, recognition, and all real-world biographical associations, so the reduction in inconsistency after replacement does not isolate political alignment as the causal factor. Without a human baseline or an explicit control for entity salience, the central claim that LLMs are systematically biased by political alignment remains underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new framework for measuring political bias in large language models (LLMs) by using target-oriented sentiment classification (TSC). The authors insert 1,319 politician names into 450 political sentences (with male/female variants), evaluate seven models in six languages, and define an entropy-based inconsistency metric (Eq. 2) that treats any variation in predicted sentiment across target entities as bias. They report a systematic pattern where left, center-left, and centrist politicians receive more positive sentiment and right and far-right politicians receive more negative sentiment, across all tested models and languages. They also show that replacing real politician names with GPT-4-generated fictional names reduces inconsistency, which they interpret as evidence that political attributes are the primary driver of the observed biases. The central claim is that LLMs exhibit robust, systematic political bias in a downstream task, with language, model-size, and individual-entity analyses.","tokens_in":35870,"tokens_out":2602,"duration_ms":31493,"significance":"If the central claim survives scrutiny, the paper offers a scalable and statistically powerful alternative to questionnaire- and generation-based bias measurements. Its strengths include a large dataset (nearly 25 million prompt-level predictions), a clear and simple inconsistency metric, multilingual coverage, multiple model families, and a concrete mitigation attempt. The paper also promises to release code and data, which would aid reproducibility. However, the validity of the bias measurement rests entirely on the entity-invariance assumption of Eq. (2), which is not empirically validated, and the control group confounds political alignment with fame and salience. These issues make the central claim underdetermined in the current form.","major_comments":[{"comment":"The inconsistency metric assumes that an unbiased LLM should produce the same sentiment prediction for a sentence regardless of the target entity, so any entropy in sentiment across entities is treated as bias. This entity-invariance assumption is never validated. The paper's own results in Section H show that the strongest negative sentiment signals come from notorious historical figures (Hitler -0.81, Himmler -0.55, Stalin -0.34); for these entities, a negative sentiment is a reasonable response independent of the sentence frame. Since the entity set is sampled by frequency of news mentions, far-right and far-left alignments contain a disproportionate share of such high-salience, valence-laden figures. The gradient in Figure 2 could therefore reflect real-world associations rather than an LLM-specific political bias. The paper needs a human annotation baseline or a control for entity salience to separate these explanations; without it, the central claim that LLMs are systematically biased by political alignment is not established.","section":"Section 3.1, Eq. (2)"},{"comment":"The control group replaces real politician names with GPT-4-generated fictional names that preserve gender, birth year, and country of origin. This manipulation simultaneously removes fame, recognition, and all real-world biographical associations, not just political alignment. The observed reduction in inconsistency after replacement therefore does not isolate political alignment as the causal factor. For example, a famous politician like Angela Merkel is replaced by a completely unknown fictional person, so any decrease in sentiment variability could be due to the removal of salience rather than the removal of political attributes. The Limitations section acknowledges demographic confounds but does not address fame. The authors should include an additional control condition using real non-political celebrities or public figures matched by fame, or otherwise demonstrate that fame does not explain the alignment gradient.","section":"Section C and Figure 7"},{"comment":"The Mann-Whitney U tests reported in Appendix J seem to treat individual prediction outcomes as independent observations. With 593,550 prompts per model-language combination (Section 3.4), the tests are extremely high-powered and will reject the null hypothesis for trivially small effect sizes. Moreover, observations are not independent: each entity contributes 450 sentiment predictions, and each sentence is evaluated with all 1,319 entities. A proper statistical analysis should account for this clustering, for example by using entity-level or sentence-level random effects, or by bootstrapping at the entity level. Without such an adjustment, the claim of 'statistically robust analysis at different granularity levels' is not supported.","section":"Appendix J"}],"minor_comments":[{"comment":"The definition section begins with a duplicated line: \"We define the eight political alignments used in the analysis below: We define the eight political alignments used in the analysis below:\". Please remove the repetition.","section":"Appendix D"},{"comment":"The term \"Mann-Withney\" is a typo for \"Mann-Whitney\" in several table captions and the surrounding text.","section":"Appendix J (Tables 6-12)"},{"comment":"The 'round away from zero' rule in the alignment computation is ad hoc and can flip a party's alignment label based on the averaged score. Since political alignment is the main independent variable, the sensitivity of the results to this rounding choice should be discussed or tested.","section":"Algorithm 1"},{"comment":"Panels (a) and (b) of Figure 2 both have identical y-axis labels, and the caption could be clearer about which panel corresponds to languages and which to models. The reader should not have to infer this from the main text.","section":"Figure 2"},{"comment":"The statement that \"we can safely assume they are similar within the same family of models\" about training corpora is speculative; since training data are not public for all models, this should be phrased as an assumption, and the possibility that other factors (e.g., alignment tuning, data mixture) explain model-size differences should be acknowledged.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper tackles a timely problem and provides a large-scale, multilingual dataset and a clear methodology. The main risk is that the entity-invariance assumption conflates political bias with legitimate entity-specific sentiment. If the authors can add a human baseline or an appropriate salience control, the contribution would be substantially strengthened. I would also recommend that the editor ask for a revised statistical analysis that accounts for the hierarchical structure of the data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a useful, very large measurement exercise, and the entropy-based inconsistency metric is a new lens for auditing LLM behavior. But the central claim—that sentiment differences across politician names measure political bias—relies on an assumption the paper never validates: that an unbiased model should give the same sentiment for any entity in the same sentence. That assumption is false for many of their own sentences, and the control group doesn't fix it. So read the paper as a well-executed descriptive study, not a conclusive bias estimate.\n\nWhat's actually new: they replace names in 450 TSC sentences, use 1,319 politicians, seven models, six languages, and nearly 25 million predictions. The scale alone is a step up from questionnaire-based studies. The inconsistency metric is simple and model-agnostic, and aggregating at alignment, compass, and entity levels gives a rich picture. The control group with fictional names is a reasonable idea, and the finding that replacing real names reduces inconsistency is real. The Mann-Whitney U testing is thorough.\n\nWhere it's soft: the entity-invariance assumption is load-bearing. Many sentences are about events or actions where different politicians legitimately evoke different sentiment. The paper's own appendix shows Hitler, Himmler, and Stalin with the largest negative scores; that's not bias, that's accurate sentiment. Because the entity set under-samples low-salience politicians in extreme alignments and over-samples Western media figures, the aggregate left-positive/right-negative gradient in Figure 2 could reflect real-world associations rather than model bias.\n\nThe control group is supposed to address this, but it swaps in GPT-4-generated fictional names with the same gender, birth year, and country—which simultaneously removes fame, recognition, and all biographical associations. So the reduction in inconsistency after replacement doesn't isolate political alignment. The residual gender and nationality effects they report in Appendix I are themselves confounded with alignment. The paper acknowledges Western-media dominance in Limitations, but doesn't discuss the fame confound directly, and the invariance assumption is asserted rather than tested.\n\nBottom line: the measurement infrastructure is worth engaging with, and the descriptive patterns are informative, but the central bias claim is underdetermined. A serious referee should ask for a human baseline or an explicit control for entity salience, and ideally release the code and data so the 25-million-point result is checkable. I'd send it to review.","headline":"A genuinely large-scale measurement of name-dependent sentiment variability in LLMs, but the main 'bias' claim rests on an unvalidated invariance assumption and is not yet established.","tokens_in":36371,"tokens_out":2046,"would_cite":false,"duration_ms":24137,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that large language models systematically rate left-leaning politicians more positively and right-leaning ones more negatively in target-oriented sentiment classification, across every model and language tested.","keywords":["political bias","large language models","target-oriented sentiment classification","sentiment analysis","entity-level bias","multilingual evaluation","bias measurement","bias mitigation"],"falsifier":"Run the same 450 sentences through the same models with a matched group of fictional names whose assigned party labels are randomly permuted; if the political-alignment gradient in sentiment persists under a permutation that severs name from alignment, then the measured effect is not actually carried by political affiliation. Alternatively, have human raters score the same name-swapped sentences; if humans show the same left-right gradient, the 'bias' is partly a property of the sentence content rather than of the models.","tokens_in":1648,"feed_emoji":"🗳️","tokens_out":3018,"duration_ms":76144,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models carry a systematic political bias: when asked to judge the sentiment of a sentence toward a named politician, they rate left, center-left, and centrist politicians more positively and right and far-right politicians more negatively, in every model and language tested. The authors build this case with a large controlled experiment, 450 political sentences, 1,319 politicians, seven models, six languages, and about 25 million predictions, and a new inconsistency metric that measures how much the predicted sentiment changes when only the politician's name changes. They also show the bias mostly disappears when real names are swapped for fictional but plausible ones, indicating that political attributes, not demographics or fame, drive the effect. If the finding holds, it matters for any real-world deployment of LLMs in politically sensitive tasks such as content moderation or media coverage analysis.","feed_headline":"LLMs favor left-leaning politicians in every tested model","feed_subtitle":"25 million sentiment predictions across seven models and six languages show consistent bias, stronger in larger models.","key_machinery":"The load-bearing instrument is an entropy-based inconsistency metric. For a fixed sentence, the model's sentiment predictions are collected across all 1,319 politician names; the entropy of that label distribution, averaged over the 450 sentences, measures how much the predicted sentiment varies when only the target name changes. An unbiased model would give constant predictions and zero entropy, so IC equals zero. The same setup is run with a control group of fictional names, and the drop in inconsistency after name replacement is what isolates the political component of the bias.","core_discovery":"On the paper's own terms, the central discovery is that target-oriented sentiment classification by LLMs is systematically skewed by the political alignment of the target entity: sentiment predictions become more positive for politicians on the left, center-left, and center, and more negative for those on the right and far right, with the effect present across all seven tested models and all six tested languages. The bias is stronger in English, French, and Spanish than in Russian, Arabic, and Chinese, and stronger in larger models than in their smaller counterparts. A two-dimensional political compass analysis shows the most positive sentiment goes to left-libertarian figures and the most negative to right-authoritarian ones. The paper further reports that replacing politician names with fictional names that preserve gender, birth year, and country largely removes the alignment-driven component of the bias, which the authors take as evidence that political attributes are the primary source.","pith_inferences":["An implication the authors leave implicit: the same inconsistency metric could serve as a general probe for bias toward other entity classes such as companies, ethnic groups, or public institutions, wherever a sentence can be re-issued with a substitution.","If the left-leaning bias reflects training-data frequency and narrative valence rather than intent, then model updates that rebalance political coverage could shift the measured bias, making periodic re-auditing necessary.","A testable consequence beyond the paper: the entity-similarity matrix should predict transfer of sentiment between politicians in unseen sentences; one could hold out a subset of politicians and check whether sentiment patterns generalize along the similarity structure.","The finding that fictional names still carry demographic residue, with female names rated higher and Russian names lower, suggests a second-order bias that the paper measures but does not fully separate from political alignment."],"forward_implications":["Politically sensitive NLP applications such as content moderation, stance detection, and media-coverage analysis inherit this bias when built on LLMs, so their outputs about right-leaning figures will skew negative.","Larger models do not cure the bias; they intensify it, so scaling alone is not a mitigation strategy.","The bias is not confined to English: it appears in all six tested languages, though Western languages show stronger effects.","Replacing politician names with non-political fictional names offers a cheap, model-agnostic partial mitigation, cutting inconsistency while slightly improving accuracy.","Models encode structured knowledge about individual politicians: sentiment patterns cluster politicians with similar ideologies and contexts, which the paper reads as evidence of internal entity representations."],"supporting_citations":[{"why":"Supplies the 450-sentence MAD-TSC dataset used for controlled entity replacement and sentiment prediction.","marker":"Dufraisse et al., 2023"},{"why":"Provides the ParlGov party-position data used to map politicians onto the political compass.","marker":"Döring and Manow, 2012"},{"why":"Provides the CC-News corpus used to select frequently mentioned politicians via entity linking.","marker":"Mackenzie et al., 2020"},{"why":"Supplies structured Wikidata attributes (party, birth year, gender, country) for entity construction.","marker":"Pellissier Tanon et al., 2016"},{"why":"Prior evidence of ChatGPT's left-libertarian orientation, which the paper's political compass results corroborate.","marker":"Hartmann et al., 2023"},{"why":"Prior measurement of ChatGPT political bias that this paper's findings align with and extend.","marker":"Motoki et al., 2024"},{"why":"Establishes that larger LLMs improve TSC accuracy, supporting the paper's model-size comparison.","marker":"Zhang et al., 2024"}],"fun_headline_variants":["LLM sentiment skews left in 7 models, 6 languages","Political bias in LLM sentiment spans all tested models","Larger LLMs show stronger political bias in sentiment","LLMs rate left politicians higher across all models","Sentiment bias favoring left found in all tested LLMs"],"cache_read_input_tokens":38528,"weakest_assumption_plain":"The measurement treats any change in predicted sentiment when only the politician's name changes as bias, but the sentences contain normative and evaluative content where different politicians could legitimately evoke different sentiments, and the fictional-name control only matches gender, birth year, and country, not fame or prior associations.","fun_headline_variants_meta":{"raw":{"variants":["LLM sentiment skews left in 7 models, 6 languages","Political bias in LLM sentiment spans all tested models","Larger LLMs show stronger political bias in sentiment","LLMs rate left politicians higher across all models","Sentiment bias favoring left found in all tested LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2581,"prompt_tokens":896,"completion_tokens":1685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1605}},"tokens_in":512,"tokens_out":1685,"duration_ms":9959,"temperature":1.0,"reasoning_tokens":1605,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:05:42.602581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 450 sentences through the same models with a matched group of fictional names whose assigned party labels are randomly permuted; if the political-alignment gradient in sentiment persists under a permutation that severs name from alignment, then the measured effect is not actually carried by political affiliation. Alternatively, have human raters score the same name-swapped sentences; if humans show the same left-right gradient, the 'bias' is partly a property of the sentence content rather than of the models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ParlGov party-position data used to map politicians onto the political compass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CC-News corpus used to select frequently mentioned politicians via entity linking."}],"review_version":1}