Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Benchmarking Multi-National Value Alignment for Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces NaVAB, the first benchmark for measuring how well LLMs align with the values of five nations—China, the US, the UK, France, and Germany—built from official news media and usable for improving alignment.

desk verdict A genuinely new multi-nation value-alignment benchmark with released code and data, but the AJ metric is sub-chance for base models and the paper's reliability claim does not survive contact with its own numbers. read the letter →

arxiv 2504.12911 v2 pith:MCQZI4JA submitted 2025-04-17 cs.CL cs.AI

Weijie Shi , Chengyi Ju , Chengzhong Liu , Jiaming Ji , Jipeng Zhang , Ruiyuan Zhang , Jia Zhu , Jiajie Xu
show 3 more authors
Yaodong Yang Sirui Han Yike Guo
This is my paper · ORCID
classification cs.CLcs.AI
keywords valuealignmentnationalvaluesLLMbenchmarknews-deriveddatasetConflictReductiondirectpreferenceoptimizationcross-culturalAIevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces NaVAB, a benchmark for measuring how well large language models align with the values of five nations: China, the United States, the United Kingdom, France, and Germany. The benchmark is built automatically from official news outlets through a pipeline that clusters news by topic, filters out non-value-sensitive content, and extracts position statements, each paired with an opposite statement. A graph-based Conflict Reduction step removes statements that contradict a nation's dominant stance. Experiments on open and closed models show that instruction-tuned and larger models generally align better, and that fine-tuning with DPO on NaVAB data improves average alignment by more than five percent. The paper's aim is to give researchers a scalable, reusable way to spot and correct country-specific value misalignment.

What carries the argument

The load-bearing mechanism is a three-stage data pipeline ending in a graph-based Conflict Reduction process. News articles are embedded with language-specific sentence encoders, reduced with UMAP, clustered with HDBSCAN, and outliers are recovered through instruction tagging; value-sensitive topics are then screened with GPT-4 and human checks. From the surviving articles, GPT-4 extracts value statements, which are organized into a graph whose edges express semantic, geospatial, and social links, and a cycle-detection path-finding procedure removes edges that deviate from the dominant stance. Each surviving statement is expanded into a <Q, S, RS> triple, and evaluation uses multiple-choice selection or GPT judge-based answer judgment to compare the model's position against the statement and its reverse.

What would settle it

Take a random sample of NaVAB statements and compare them with nationally representative public-opinion survey results on the same topics; if large fractions of statements contradict survey majorities, the benchmark would be shown to track media stances rather than national values.

Watch

Extended reading notes

Core claim

NaVAB claims to be the first multi-national benchmark that derives value-alignment tests from news media rather than hand-written questionnaires. For each nation it produces a set of test items of the form <question, statement, reverse statement>, split into statements quoted from named speakers and statements expressing the outlet's own official stance. On these items, LLM alignment is measured two ways: multiple-choice selection between the statement and its reverse, and answer judgment where a second model decides which reference a free-form answer matches. The headline empirical result is that value alignment is not uniform: base models score far below instruction-tuned models, mixture-of-experts models score highest, and DPO fine-tuning on NaVAB data raises alignment by over 5 percent on average for every tested model. The similarity of scores on quoted and official statements is taken as evidence that individual and institutional voices in a nation's media largely share values.

Load-bearing premise

The benchmark's ground truth for a nation's values is the positions of that nation's official and mainstream media outlets, as extracted and filtered by GPT-4.

Editorial extensions

If this is right

  • Model makers can use NaVAB as a reusable test suite to check where their models conflict with the values of a target country before deployment.
  • Because DPO fine-tuning on NaVAB improves alignment for every tested model and nation, the benchmark data can serve directly as preference pairs for alignment training.
  • Base models align markedly worse than instruction-tuned models, implying that a substantial part of apparent value alignment comes from instruction following rather than from pretraining values.
  • Quoted and official statements give similar scores, so a media-derived statement set can stand in for both individual and institutional national value positions.
  • Alignment gaps differ by nation and language, with German and French data generally harder, suggesting pretraining corpus language coverage shapes value alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If official media stances diverge from the views of the general public, NaVAB would measure alignment with media positions rather than with national values; validating a sample against public-opinion surveys could settle that.
  • Since GPT-4 is used both to extract statements and to judge answers, its own value tendencies may be baked into the benchmark; an independent human-annotated subset could quantify that contamination.
  • The same pipeline should transfer to other countries and languages, as long as a suitable news corpus and sentence encoder exist; the five-nation coverage is a proof of concept, not a ceiling.
  • A natural extension is to use NaVAB-style statements to audit models for value drift over time or across prompt variants, not only static evaluation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces NaVAB, a benchmark for evaluating how well large language models align with the values of five nations (China, the US, the UK, France, and Germany). The benchmark is constructed by a pipeline that collects news from official and mainstream media sources, clusters them via UMAP/HDBSCAN with GPT-4-assisted tagging, extracts value statements with GPT-4, filters conflicting statements through a graph-based Conflict Reduction procedure, and classifies statements as quoted or official. Two evaluation metrics are proposed: Multiple-Choice (MC), where a model selects between a statement S and its reverse RS, and Answer-Judgment (AJ), where GPT-4 judges whether a free-form answer aligns with S or RS. The authors report experiments on several open and closed models, find that base models align least well and MoE/closed models best, and show that DPO fine-tuning on NaVAB data improves alignment. The paper claims the two metrics are consistent and reliable and that NaVAB is the first multi-national value alignment benchmark.

Significance. If the benchmark construction and evaluation are valid, NaVAB would be a useful public resource: it provides a scalable, news-derived alternative to questionnaire-based political spectrum tests, covers five nations, and includes a conflict-reduction mechanism that is plausibly helpful for training data curation. The paper also releases code and data, which supports reproducibility. However, the central validity claims are currently undermined by two load-bearing problems: the AJ metric produces sub-chance scores for base models despite being a binary judgment with a 0.5 chance baseline, and the benchmark pipeline relies heavily on GPT-4 for extraction, classification, and judging, with only a weak human check that does not actually demonstrate that conflicts were removed. The construct itself is also narrower than advertised: it measures alignment with selected media outlets' positions rather than with national values, a gap the authors acknowledge only in the Limitations. These issues do not necessarily destroy the contribution, but they require substantial re-analysis and reframing before the benchmark's conclusions can be accepted.

major comments (3)
  1. [§3.1, Table 2, §3.3] The AJ metric as implemented appears unreliable: Table 2 shows Llama3.1-8B Base scoring about 0.27 on AJ for every nation and both statement sets, while its MC score is about 0.50. Since AJ is a forced binary judgment between Reference A (S) and Reference B (RS), the chance baseline is 0.5, so a score of 0.27 indicates systematic anti-alignment or, more plausibly, a biased or mis-scaled judge. The paper's claim in §3.3 and §3.5 that 'both evaluation methods are generally reliable and consistent' is contradicted by the large and systematic discrepancy between MC≈0.50 and AJ≈0.27 for the same models. No error bars, judge-calibration experiments, or chance-level baselines are reported, so every AJ-based ranking and all AJ improvements in Table 3 and Figure 6 are unsupported. The authors should either correct the AJ computation or judge prompt, validate the judge against human labels on a sample, and report confidence intervals; until then, AJ-based conclusions should be removed or substantially qualified.
  2. [§2.4, Table 8] The human verification of Conflict Reduction does not support the claim that conflicting value statements were removed. Table 8 reports that in a sample of 100 generated statements per nation, the average Align Rate is about 1% and the Conflict Rate is 0%, with the vast majority of statements classified as 'Unrelated' to a single selected reference statement. This protocol only shows that most sampled statements are unrelated to one chosen anchor; it does not measure whether conflicting statements exist within the retained set or whether the graph-based procedure (semantic similarity, geospatial distance, social network, 5-hop cycle detection, 5 rounds) successfully removes them. The conclusion in §2.4 and Appendix B that 'the Conflict Reduction process effectively removes conflicting statements while preserving aligned ones' is therefore not established. A meaningful validation should sample pairs or sets of retained statements and measure pairwise conflict rates, ideally comparing against a version of the pipeline without conflict reduction.
  3. [§2.1, §2.4, Limitations] The benchmark operationalizes 'national values' as the positions of a small set of selected media sources (e.g., CNN and NYT for the US; government-affiliated outlets for China), extracted and filtered by GPT-4. The abstract and title claim alignment with 'the values of five major nations,' but the data only represent the editorial or official stances of those outlets. The Limitations section acknowledges this gap, but the acknowledgment is insufficient because the gap affects the interpretation of every reported score and the benchmark's usefulness for actual value alignment. The authors should either (a) reframe the manuscript's central claim as 'alignment with mainstream/state media value positions' and adjust the title/abstract accordingly, or (b) provide evidence that the selected media positions correlate with broader public or institutional values, for example by comparing against existing cross-national value surveys or opinion polls.
minor comments (6)
  1. [Appendix A.3, Table 5] The model sizes in Table 5 appear to be swapped: Llama-3.1-8b is listed as 3.21B and Llama-3.2-3b as 8.03B, whereas the correct values are approximately 8.03B and 3.21B, respectively.
  2. [§3.4, Table 3] The ablation text states that removing Conflict Reduction decreases the correct rate by 'over 3% for the MC method,' but the numbers in Table 3 show smaller average decreases (for example, China Quoted MC: 0.515 vs 0.490, a 2.5 percentage-point drop; US Quoted MC: 0.498 vs 0.481, a 1.7 point drop). Please recheck the arithmetic or clarify whether the claim refers to a specific subset.
  3. [References] References [72] and [73] are identical (Rettenberger et al., 2024, 'Assessing political bias in large language models'); one should be removed or replaced.
  4. [Throughout] There are several typos and formatting issues: 'differnet' in the Figure 1 caption, 'Multiple-Choise' in the Table 2 caption, and the arrow notation 'MC↓ AJ↓' in Table 3 is not explained. Please proofread carefully.
  5. [Appendix A] The paper states that 'we conduct three independent trials and report the average results' but no variance, standard deviation, or significance tests are reported anywhere, including in Tables 2 and 3. Given the small differences that support some conclusions (e.g., the DPO improvements in Table 3), reporting error bars or at least per-trial ranges would strengthen the claims.
  6. [§2.1] The selection of 'official media' is heterogeneous: CNN and The New York Times are not official government outlets in the same sense as the Chinese Ministry of Foreign Affairs or People's Daily, while the UK, France, and Germany are represented by single aggregated sources. This asymmetry should be discussed explicitly, as it may affect cross-country comparability.

Circularity Check

1 steps flagged · score 6.0 of 10

The benchmark's ground truth (S/RS triples) and the AJ judge are both GPT-4, so the central 'value alignment' scores for GPT-4 reduce to GPT-4 agreeing with GPT-4-generated statements.

  1. self definitional [§2.4 Step IV (Evaluation Sample Construction) and §3.1 (Evaluation Metric); see also §2.4 Step I and Step II.]
    "For each validated value statement, we use LLM (GPT4) to construct a triple structure of <Q, S, RS> ... RS - Reverse Statement: a logically opposed position that maintains semantic coherence while inverting the original stance. ... GPT is then employed as a judge to determine whether the generated answer aligns more closely with Reference A: S or Reference B: RS."

    The benchmark's correct answers are not externally fixed labels. The value statements S are extracted by GPT-4, the reverse statements RS are generated by GPT-4, and the AJ metric uses GPT-4 as the judge comparing a model's answer with S versus RS. For GPT-4 itself, a high AJ score therefore measures whether GPT-4 endorses the statements that GPT-4 wrote and judges them as matching its own answer; it is self-consistency, not alignment with an independently verified national value. The MC metric uses the same GPT-4-constructed S/RS pairs as choices, so it inherits the same self-generated ground truth.

full rationale

This is a benchmark-construction paper rather than a formal derivation, so most of its pipeline (topic modeling, clustering, conflict reduction) is not circular. The circular step is concentrated in the evaluation data and metric: GPT-4 extracts the value statements (Section 2.4 Step I), constructs the S/RS triples (Section 2.4 Step IV), and serves as the AJ judge (Section 3.1). Consequently, the evaluation of GPT-4's alignment reduces to GPT-4 agreeing with GPT-4-generated statements, judged by GPT-4. Claude-3.5 is also benchmarked against GPT-4-generated ground truth, which weakens the independence of all model rankings. Human verification is present only for conflict reduction (Appendix B), not for validating that the extracted statements are correct national values or that the S/RS labels are unbiased. The Limitations section concedes that the data 'may not fully capture a nation's core values or the diverse perspectives of its people,' but it does not address the self-generation of the ground truth. No load-bearing self-citation chain or imported uniqueness theorem was found. The AJ near- or below-chance scores for base models are a reliability concern that I treat as correctness risk rather than circularity. Because the central measurement claim is substantially self-referential, but the paper still releases a reusable pipeline and dataset with independent news sources, the circularity score is 6 rather than higher.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the media-as-values assumption, the reliability of GPT-4 as an oracle, and several hand-set pipeline hyperparameters. No new physical or conceptual entities are introduced; the benchmark itself is an artifact, not an entity.

free parameters (4)
  • HDBSCAN min_cluster_size = 200
    Chosen for clustering news embeddings into topics; it affects the number and size of topic clusters and therefore the subsequent value screening. Fixed a priori, not tuned to a target score.
  • UMAP n_components = 5
    Dimensionality of reduced embeddings before clustering; set by hand in Table 4.
  • conflict_reduction_rounds = 5
    Number of iterative refinement rounds in graph-based conflict reduction; chosen by hand.
  • cycle_length = 5
    Hop length for cycle detection in the knowledge graph; set to 5 without a sensitivity analysis.
assumptions (4)
  • domain assumption Official media outlets' stances represent a nation's values.
    The pipeline builds the benchmark from official news sources; if media values diverge from public values, the benchmark measures media alignment. Acknowledged in Limitations.
  • domain assumption GPT-4 can reliably extract value statements, classify sources, and judge open answers.
    All stages of the pipeline and the AJ metric rely on GPT-4 as an oracle without broad human validation; the AJ sub-chance performance undercuts this.
  • domain assumption National values are internally coherent and conflicts are noise to be removed.
    Conflict Reduction removes statements that deviate from the dominant stance, presupposing a single coherent value stance per nation.
  • domain assumption A graph cycle of length 5 reliably indicates value conflicts.
    The path-finding conflict detection treats 5-hop cycles as conflict indicators; no sensitivity or ground-truth study is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Multi-National Value Alignment for Large Language Models." pith.science (2026). https://pith.science/paper/MCQZI4JA

@misc{pith2026250412911,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Multi-National Value Alignment for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCQZI4JA}},
  note         = {Machine review of arXiv:2504.12911}
}
read the original abstract

Do Large Language Models (LLMs) hold positions that conflict with your country's values? Occasionally they do! However, existing works primarily focus on ethical reviews, failing to capture the diversity of national values, which encompass broader policy, legal, and moral considerations. Furthermore, current benchmarks that rely on spectrum tests using manually designed questionnaires are not easily scalable. To address these limitations, we introduce NaVAB, a comprehensive benchmark to evaluate the alignment of LLMs with the values of five major nations: China, the United States, the United Kingdom, France, and Germany. NaVAB implements a national value extraction pipeline to efficiently construct value assessment datasets. Specifically, we propose a modeling procedure with instruction tagging to process raw data sources, a screening process to filter value-related topics and a generation process with a Conflict Reduction mechanism to filter non-conflicting values.We conduct extensive experiments on various LLMs across countries, and the results provide insights into assisting in the identification of misaligned scenarios. Moreover, we demonstrate that NaVAB can be combined with alignment techniques to effectively reduce value concerns by aligning LLMs' values with the target country.

Figures

Figures reproduced from arXiv: 2504.12911 by the authors.

Figure 1
Figure 1. A demonstration of differnet LLM’s responses [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of NaVAB. Each process is introduced in Section [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Two examples showing the clusters from different news data sources and the top 5 topics of the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: A comparison between traditional evaluation method and ours. MC and AJ denote Multiple-Choice and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: A case study comparing the LLM’s alignment before and after fine-tuning with DPO using NaVAB’s data. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The comparison of alignment results for various LLMs before and after DPO training, evaluated using the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Examples showing the clusters from different news data sources and the top topics of the corresponding [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fine-Grained Interpretation of Political Opinions in Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Four-dimensional political concept vectors learned from LLM internals can detect and partially steer political leanings better than a single left-right axis.

  2. HKGAI-V1: Towards Regional Sovereign Large Language Model for Hong Kong

    cs.CL 2025-07 reject novelty 5.0 of 10

    A DeepSeek-based model fine-tuned for Hong Kong outperforms general models on Hong Kong benchmarks, but most of those benchmarks are self-authored and unreleased.

Reference graph

Works this paper leans on

48 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    believe",

    Identify any indicators within the statement that suggest who might have said it. Look for verbs like "believe", "express", or specific references to individuals or positions

    work page

  2. [2]

    If there are clear indicators, state who said the statement and provide the reason based on the language cues within the statement

    work page

  3. [3]

    Unknown

    If the statement lacks clear indicators or attribution, state "Unknown." Notice that some statements may involving some people, but the statement does not provide enough information to determine who said it. In this case, you should also state "Unknown." Format example: reason [Name or Unknown] Examples Example 1: US President Barack Obama suffered racial...

    work page

  4. [26]

    arXiv preprint arXiv:2308.05374

    Trustworthy llms: A survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374

    arXiv

  5. [41]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  6. [42]

    Boanerges Aleman-Meza, Meenakshi Nagarajan, Cartic Ramakrishnan, Li Ding, Pranam Kolari, Amit P Sheth, I Budak Arpinar, Anupam Joshi, and 10 Tim Finin. 2006. Semantic analytics on social net- works: experiences in addressing the problem of con- flict of interest detection. In Proceedings of the 15th international conference on World Wide Web, pages 407–416

    work page 2006

  7. [43]

    David M Blei, Andrew Y Ng, and Michael I Jordan

    work page

  8. [44]

    Rod Brookes. 1999. Newspapers and national iden- tity: The bse/cjd crisis and the british press. Media, Culture & Society, 21(2):247–263

    work page 1999

Show all 48 references
  1. [45]

    Caroline Brun and Vassilina Nikoulina. 2024. Frenchtoxicityprompts: a large benchmark for eval- uating and mitigating toxicity in french texts. In LREC-COLING-2024, pages 105–114

    2024

  2. [46]

    Stephen Cushion. 2017. The democratic value of news: Why public service media matter. Bloomsbury Publishing

    2017

  3. [47]

    Ameet Deshpande, Vishvak Murahari, Tanmay Ra- jpurohit, Ashwin Kalyan, and Karthik Narasimhan

  4. [48]

    Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM confer- ence on fairness, accountability, and tr...

    2021

  5. [49]

    Simeon Djankov, Caralee McLiesh, Tatiana Nenova, and Andrei Shleifer. 2003. Who owns the media? The Journal of Law and Economics, 46(2):341–382

    2003

  6. [50]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiy- ong Wu, Tianyu Liu, et al. 2022. A survey on in- context learning. arXiv preprint arXiv:2301.00234

    2022 arXiv

  7. [51]

    Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to lan- guage models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. arXiv

    2023

  8. [52]

    Janell Fetterolf and Laura Clancy. 2024. Support for legal abortion is widespread in many places, es- pecially in europe

    2024

  9. [53]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462

    2020 arXiv

  10. [54]

    Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv

    2022

  11. [55]

    Jochen Hartmann, Jasper Schwenzow, and Maxi- milian Witte. 2023. The political ideology of con- versational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768

    2023 arXiv

  12. [56]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  13. [57]

    Chip Huyen. 2019. Evaluation metrics for language modeling. The Gradient, 40

    2019

  14. [58]

    Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap

  15. [59]

    Daniel Lee and H Sebastian Seung. 2000. Algo- rithms for non-negative matrix factorization. Ad- vances in neural information processing systems, 13

    2000

  16. [60]

    Li Li, Wei Ji, Yiming Wu, Mengze Li, You Qin, Lina Wei, and Roger Zimmermann. 2024. Panop- tic scene graph generation with semantics-prototype learning. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 3145–3153

    2024

  17. [61]

    Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. 2023. Winner: Weakly-supervised hierarchi- cal decomposition and alignment for spatio-temporal video grounding. In Proceedings of the IEEE/CVF conference on computer vision and patter...

    2023

  18. [62]

    Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xi- aoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li

  19. [63]

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. # instag: Instruction tagging for analyz- ing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations

    2023

  20. [64]

    Tinh Son Luong, Thanh-Thien Le, Linh Ngo Van, and Thien Huu Nguyen. 2024. Realistic evaluation of toxicity in large language models. arXiv

    2024

  21. [65]

    Leland McInnes, John Healy, Steve Astels, et al

  22. [66]

    Leland McInnes, John Healy, and James Melville

  23. [67]

    Fabio Motoki, Valdemar Pinho Neto, and Victor Ro- drigues. 2024. More human than human: measuring chatgpt political bias. Public Choice, 198(1):3–23

    2024

  24. [68]

    Matunda Nyanchama and Sylvia Osborn. 1999. The role graph model and conflict of interest. ACM Transactions on Information and System Security (TISSEC), 2(1):3–33

    1999

  25. [69]

    Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. 2021. Probing toxic content in large pre-trained language models. In ACL, pages 4262–4274

    2021

  26. [70]

    Yujin Potter, Shiyang Lai, Junsol Kim, James Evans, and Dawn Song. 2024. Hidden persuaders: Llms’ political leaning and their influence on voters. arXiv preprint arXiv:2410.24190

    2024 arXiv

  27. [71]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your lan- guage model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  28. [73]

    Luca Rettenberger, Markus Reischl, and Mark Schutera. 2024. Assessing political bias in large language models. arXiv preprint arXiv:2405.13041

    2024 arXiv

  29. [74]

    David Rozado. 2024. The political preferences of llms. arXiv

    2024

  30. [75]

    Michael Schudson. 1995. The power of news. Har- vard University Press

    1995

  31. [76]

    Shreya Shankar, JD Zamfirescu-Pereira, Björn Hart- mann, Aditya Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software ...

    2024

  32. [77]

    Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. arXiv preprint arXiv:2408.13006

    2024 arXiv

  33. [78]

    Jim Willis. 2007. The media effect: How the news influences politics and government. Bloomsbury Pub- lishing USA

    2007

  34. [79]

    Tao Wu, Mengze Li, Jingyuan Chen, Wei Ji, Wang Lin, Jinyang Gao, Kun Kuang, Zhou Zhao, and Fei Wu. 2024. Semantic alignment for multimodal large language models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3489– 3498

    2024

  35. [80]

    tag": str,

    John Zaller. 1991. Information, values, and opin- ion. American Political Science Review, 85(4):1215– 1237. A Experimental Details In this section, we provide a detailed description of the dataset used in this study, along with the ex- perimental procedures and configurations ...

    1991

  36. [2003]

    Journal of machine Learning research, 3(Jan):993–1022

    Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022

  37. [2017]

    hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11):205

  38. [2018]

    arXiv preprint arXiv:1802.03426

    Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426. 11

    arXiv

  39. [2023]

    Toxicity in chatgpt: Analyzing persona- assigned language models. arXiv

  40. [2024]

    Polyglotoxicityprompts: Multilingual evalua- tion of neural toxic degeneration in large language models. arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.