Representational Equality in Cross-country Value Simulation: A Systematic Analysis of Large Language Models

Chen Qian; Daisong Gong; Huimin Chen; Maosong Sun; Xiaowen Jian; Xinyi Mou

REVIEW 3 major objections 6 minor 108 references

Representational Equality in Cross-country Value Simulation: A Systematic Analysis of Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current open LLMs simulate human values unevenly across countries, systematically better for wealthier, better-governed, more individualist populations, and standard interventions do not reliably fix the gap.

desk verdict Solid, well-robustness-checked inequality finding; the 'simulation capability' framing is hostage to an unexamined benchmark-contamination premise, and the additional-information intervention is partly circular. read the letter →

arxiv 2608.08058 v1 pith:J2RA76NT submitted 2026-08-08 cs.CY

Xiaowen Jian , Xinyi Mou , Daisong Gong , Chen Qian , Huimin Chen , Maosong Sun This is my paper · ORCID

classification cs.CY

keywords representationalequalityvaluesimulationcross-countrybiaslargelanguagemodelsWorldValuesSurveyaccuracycontextualadaptationparametricmodification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that large language models, asked to simulate how people in different countries answer value-survey questions, perform far better for some populations than for others, and that this unevenness is systematic rather than random. Across nine open models and 59 countries, simulation accuracy is consistently higher for wealthier, more technologically advanced, better-governed, lower-power-distance, and more individualist populations, a pattern that survives across model families and scales. The paper introduces 'representational equality' as a metric distinct from average accuracy, and shows that interventions that raise average or target-group accuracy—native-language prompting, extra retrieved context, continued post-training, preference alignment—do not reliably shrink cross-country inequality; supplying retrieved survey context at inference is the one pathway that most consistently improves both accuracy and equality at once. The stakes: LLM-based opinion simulation, now proposed as a scalable proxy for cross-national social science, would quietly reproduce global structural inequalities in any downstream application that takes its outputs at face value.

What carries the argument

The load-bearing object is the Representational Equality Index, defined as the coefficient of variation of country-level simulation accuracies, $Eq_{CV} = \sigma_{A_c}/\mu_{A_c}$, selected after four fairness-style indices (max-min difference, min-max ratio, CV, Gini) were shown to rank models consistently. Underneath it sits the simulation pipeline: subpopulations are cells defined by country plus two demographic attributes (gender, age, education, income), kept when they contain at least ten WVS respondents, for 2,420 cells across 59 countries. Each cell's simulated value distribution comes from a first-token probability method, in which the model's log-probabilities over the top-20 first-position tokens are filtered to valid answer letters and normalized, with a capped residual-mass approximation for missing options; accuracy is $1 - \mathrm{JSD}$ between that distribution and the empirical WVS distribution, aggregated with equal weight across 11 value dimensions. The diagnostic arm runs Spearman correlations of country accuracy against PEST-family macro indicators (Political: six Worldwide Governance Indicators; Economic: GDP per capita; Socio-cultural: Hofstede's six dimensions; Technological: Internet use and the Global Innovation Index), with Benjamini–Hochberg false-discovery-rate correction. A supplementary accuracy-equality composite, the geometric mean of average JSD and $Eq_{CV}$, lets the paper rank models on both axes at once.

What would settle it

Run a contamination-controlled replication: measure n-gram or paraphrase overlap between the 160 benchmark questions and each model's training corpus, then re-score accuracy only on questions with no measurable overlap. If the GDP, Internet-use, and power-distance correlations with accuracy vanish or reverse on uncontaminated items, the inequality pattern is a memorization artifact; if they persist, it is a genuine property of cross-country simulation capability.

Watch

Extended reading notes

Core claim

The paper's central claim is that representational equality—the evenness of simulation accuracy across populations—is a measurable, structurally patterned property of current LLMs, and that current models fail it. Using first-token probabilities to approximate each model's response distribution over answer-option letters for 2,420 country-by-demography subpopulations, and scoring accuracy as $1-\mathrm{JSD}$ against World Values Survey Wave 7 ground truth, the authors find country-level accuracy dispersion with $Eq_{CV}$ values from 0.0286 to 0.0546 across nine models; GLM-4-9B is the most accurate (0.739) but not the most equal, while ChatGLM3-6B is the most equal. Country-level accuracy correlates positively with GDP per capita, Internet use, the Global Innovation Index, and the six Worldwide Governance Indicators, and negatively with Hofstede power distance, with individualism and indulgence positively associated. In a case study, GLM-4-9B's simulated Iraqi response distributions are more often closer to the US empirical distribution than to the Iraqi one, indicating anchoring toward better-represented countries. The paper also claims the two intervention pathways diverge: native-language prompting improves accuracy unevenly and can relocate inequality; retrieved additional information usually improves accuracy and equality together; language-specific continued post-training raises average accuracy for target languages but unevenly across countries and questions; and preference alignment via DPO or GRPO yields no systematic gains on either axis, with human-annotated preference data preserving accuracy better than AI-annotated data. An ISSP-based supplementary benchmark is offered to show the main patterns are not specific to one survey instrument.

Load-bearing premise

The evaluation assumes the WVS Wave 7 responses used as ground truth have not already been memorized by the evaluated models, and because no contamination check is reported, the country-accuracy gradient could partly measure training-data exposure rather than simulation capability.

Editorial extensions

If this is right

Cross-national LLM opinion simulation cannot be treated as representative of global populations unless it passes an equality audit alongside an accuracy check.
The highest-accuracy model is not the highest-ranked joint accuracy-equality model: GLM-4-9B leads on accuracy (0.739) yet trails ChatGLM3-6B and Mistral-7B-v0.3 on the composite metric.
Supplying retrieved empirical context at inference is the most reliable lever for raising accuracy and cutting cross-country inequality simultaneously.
Gains concentrated in particular language communities can relocate rather than reduce inequality, so interventions must be evaluated as distributions, not as group averages.
Preference alignment (DPO or GRPO, human- or AI-annotated) should not be expected to improve value simulation, and human-annotated preference data is the safer choice for preserving accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test the paper leaves implicit: if the wealth-accuracy gradient tracks training-corpus exposure, the same correlations should be predictable from country-level web-text frequencies, and equality audits could be run before any simulation by auditing pretraining data coverage.
The US-versus-Iraq anchoring suggests a general collapse of low-resource-country simulations toward the dominant global distribution; a per-question 'closeness to the best-represented country' diagnostic would make that mechanism visible in future audits.
The $Eq_{CV}$ framework transfers to other partitioning axes—language, dialect, rural/urban, within-country regions—so the same metric could audit representational equality below the country level.
Because the equality gains of additional-information retrieval likely scale with retrieved-item relevance, a testable extension is to measure how $Eq_{CV}$ changes with retrieval set size, retrieval quality, and the empirical support of retrieved subgroup distributions.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version. The core empirical claim — open LLMs simulate WVS value responses much more accurately for wealthy, well-governed, individualist countries — is well supported and looks real. EqCV across nine models and 59 countries, the extensive sensitivity analyses (LODO, weighting, ISSP replication, first-token validation), and the BH-FDR corrections all point the same way. The US/Iraq case study is concrete and informative. That is a meaningful addition to prior work that measured average accuracy or directional bias rather than cross-country dispersion as an equality property.

The soft spots are real but unequal in weight. First, contamination: WVS Wave 7 and ISSP 2020–2023 were public before the evaluated models' training cutoffs, and the paper provides no contamination analysis. That matters because the headline interprets accuracy as 'simulation capability' and concludes that LLM-based simulation reproduces structural inequalities. If models have memorized survey margins, the inequality could be an exposure artifact. This does not undermine the descriptive inequality result — whether memorized or not, the model's outputs are systematically uneven across countries — but it does undermine the simulation-capability interpretation. The omission is genuine; the authors should add a contamination check.

Second, the additional-information intervention feeds empirical WVS response distributions for the same demographic subgroup into the prompt via the historical-memory module. The reported accuracy and equality gains are therefore partly circular: the model is given answer distributions on related items from the same subgroup. That affects the intervention guidance, not the foundational inequality result.

Minor points: the AE composite takes a geometric mean of JSD and CV, which are on different scales; a short justification would help. The native-language results use official WVS translations, which is appropriate but also means the 'language cue' is confounded with the exact item wording that may appear in training data.

The paper is carefully done, and the conditional verdict is fair. It deserves a serious referee. I would send it out and ask for a contamination analysis plus a reframing of the intervention-level claims. The audience is computational social science and LLM fairness evaluation; they will get real value from the measurement framework even if the capability interpretation needs revising.

Referee Report

3 major / 6 minor

Summary. The paper introduces "representational equality" as a metric for LLM-based cross-country value simulation, defined as the evenness of simulation accuracy across countries and operationalized primarily through the coefficient of variation of country-level accuracies (EqCV). The authors evaluate nine open-source LLMs on 59 countries using World Values Survey Wave 7 as ground truth, report substantial cross-country inequality, and show that accuracy correlates positively with GDP per capita, Internet use, innovation, and governance quality and negatively with power distance. They then compare two intervention pathways—contextual adaptation (native-language prompting, additional information) and parametric modification (continued post-training, preference alignment)—and find that additional information most consistently improves both accuracy and equality, while preference alignment yields no systematic gains. Extensive sensitivity analyses include leave-one-dimension-out, alternative weighting, ISSP replication, and first-token versus repeated-generation validation.

Significance. If the central finding holds, this is a significant contribution to LLM-based social simulation and cross-cultural NLP, providing a formal evaluation framework, a broad empirical map of inequality, and a systematic comparison of intervention strategies. The paper's strengths are substantial: a 59-country, 2,420-subpopulation evaluation; nine open models spanning scales; four complementary equality indices; BH-FDR-corrected correlations; and careful robustness checks (LODO, weighting, ISSP, first-token validation). These elements make the descriptive inequality finding well supported. However, two premises are load-bearing and currently unverified: that the public WVS/ISSP ground truth has not leaked into training corpora, and that the additional-information intervention does not feed the target survey's own joint distribution into the prompt. Both affect the interpretation of the results rather than the internal consistency of the measurements.

major comments (3)

[Section 4.1, Section 5] The evaluation assumes that WVS Wave 7 (2017-2022) and ISSP 2020-2023 responses have not been memorized by the evaluated models, whose training cutoffs span 2023-2024. No contamination analysis is provided anywhere in the manuscript. This is load-bearing because the headline claim is about simulation capability; if models reproduce memorized survey margins, then the reported accuracy levels and their correlations with wealth, governance, and cultural indicators reflect training-data exposure rather than generalizable simulation. The ISSP robustness check in Appendix F does not settle this because those modules are equally public and fall inside the training windows. I request a contamination analysis, for example: evaluating on paraphrased question variants, comparing accuracy on items published after each model's cutoff, or running membership-inference probes on question-answer pairs from WVS/ISSP.
[Section 3.4, Appendix D.1, Section 6.2] The additional-information intervention feeds the model the empirical WVS response distributions of the same demographic subgroup for non-target questions via the historical-memory module. Because the target questions come from the same WVS questionnaire, this gives the model direct information about the joint response distribution for that subgroup. The reported improvements under this intervention (e.g., GLM-4-9B EqCV dropping from 0.0486 to 0.0255 under BM25) are therefore partly by construction: the model is given the very survey's subgroup-level answer patterns as context. This does not establish a generalizable inference-time intervention for realistic settings where such subgroup-level survey data are unavailable. Please either reframe the additional-information setting as an oracle or upper-bound analysis, or remove it from the main intervention claims and conclusions.
[Section 5.1, Table E.7] The claim that some models demonstrate 'higher' or 'lower' equality (e.g., ChatGLM3-6B with EqCV=0.0286 versus Qwen2.5-72B with EqCV=0.0546) is made without uncertainty quantification for the equality indices. Table E.7 reports 95% confidence intervals for country-level accuracy but not for EqCV, EqGini, or the AE composite. Given that country-level accuracies have overlapping confidence intervals across models, it is unclear whether these equality differences are statistically significant. Please provide bootstrap confidence intervals or a direct significance test for the equality indices, or temper the model-ranking claims to a descriptive level.

minor comments (6)

[Figure 3] The correlation heatmap cells are too small to read the bolding and correlation values; please enlarge the figure or split it into per-factor-family panels.
[Throughout] The notation for the primary index alternates between 'EqCV' and 'Eq CV'; please standardize to one form and use it consistently in the text, tables, and equations.
[Equation (2)] The AE composite metric multiplies a distance (JSD) by a dispersion (CV) and takes the geometric mean; the text would benefit from a clearer explanation of why this particular combination is appropriate and how to interpret its scale.
[Appendix B.1] The minimum-support threshold of 10 respondents is stated but not justified; please add a sensitivity analysis over this threshold or explain why 10 is sufficient for stable subgroup-level distributions.
[Section 8 (Limitations)] The limitations section is candid about several threats but does not mention the possibility of training-data contamination; please add a paragraph acknowledging this risk and the need for contamination checks in future work.
[References] Several reference entries contain typos or incomplete information (e.g., Rokeach 1973, 'Free peess'; some URLs in Appendix A.3). Please run a careful copyedit of the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline inequality is measured against external WVS/ISSP ground truth with no fitted parameters, and no derivation reduces to its inputs by construction.

full rationale

The central finding—that simulation accuracy varies across countries and correlates with GDP, governance, technology, and cultural indicators—is computed from Eq. 1, Acc(H,M,q)=1−JSD(D(H,q), D(M,H,q)), where D(H,q) is the empirical WVS/ISSP response distribution (Eq. 4) and D(M,H,q) is the model's first-token probability distribution. These are independent quantities: the survey distributions are external targets, and the model predictions come from prompted generation without any parameter fitted to those targets. The EqCV index (Appendix A.1, Eq. A.7) is a descriptive coefficient of variation over the resulting country-level accuracies, so the inequality pattern is not manufactured by the metric. The additional-information intervention includes empirical WVS response distributions for non-target questions of the same demographic subgroup in the prompt (Sections 3.4 and 3.6), and one might worry that this leaks the same survey into the target evaluation. However, the paper explicitly excludes the target question from retrieval ('The target question itself is excluded from every candidate ranking', Appendix A.4), and the accuracy of the target question is scored against the held-out target distribution. Thus the intervention is a legitimate transfer experiment—providing related empirical context and testing whether the model can predict the remaining target response—rather than a circular prediction. The paper's self-citations (e.g., Mou et al. 2026 for the contextual-adaptation versus parametric-modification taxonomy) frame the experimental design, but that taxonomy is also supported by external citations (Durmus et al. 2023; Santurkar et al. 2023; Ryan et al. 2024), and all intervention results are measured independently against WVS/ISSP benchmarks. Possible pretraining contamination of WVS/ISSP items is a real external-validity concern, but it is a factual premise about training data rather than an internal circularity, and the paper's derivations do not assume it. No load-bearing step reduces, by the paper's own equations or by self-citation, to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces a measurement construct, the Representational Equality Index, but no new physical or ontological entities. The free parameters are design choices in benchmark construction and evaluation aggregation, all disclosed in the paper. The main hidden assumption is the absence of benchmark contamination, which no experiment addresses. The additional-information intervention also relies on empirical WVS distributions from the same subgroups, which creates a partial circularity for that specific result.

free parameters (5)

Subpopulation cell minimum support = 10 respondents
Used to decide which demographic cells are retained; affects which countries and subgroups enter accuracy aggregation. Chosen after analyzing cell-size distributions, not derived.
Number of demographic attributes per subpopulation = 3 (country plus two of gender, age, education, income)
Chosen to balance value specificity and statistical support; more than 60% of resulting cells remain above the 10-respondent threshold.
Number of retrieved additional-information questions = 3
Fixed for all four retrieval strategies; no sensitivity analysis on the number of retrieved items is reported.
Dimension equal-weighting scheme = Equal weight per value dimension (11 dimensions)
Prevents question-dense dimensions from dominating; sensitivity check shows model-rank Spearman 0.933 for accuracy but only 0.567 for EqCV, so equality ranking is somewhat sensitive to this choice.
AE composite geometric mean = sqrt(JSD * EqCV)
Chosen as a partially compensatory summary of average accuracy and equality; not derived from first principles.

assumptions (5)

domain assumption WVS fixed-option questionnaire responses are a valid operationalization of human values for equality evaluation.
The paper acknowledges in Section 8 that findings may not generalize to open-ended discourse, moral reasoning, or behavioral enactments beyond structured questionnaires.
domain assumption First-token probabilities over answer-option letters faithfully approximate the full response distribution the model would produce.
Validated on four models in Appendix C.1 with mean JSD-based similarity 0.980, but only for fixed-option survey questions; this does not cover open-ended generation.
domain assumption Accuracy defined as 1 minus Jensen-Shannon divergence supports meaningful cross-country comparison of simulation quality.
JSD is a standard distributional divergence, but it depends on the number of options (2-10 across the benchmark), which varies across questions and dimensions.
domain assumption Country is a meaningful and sufficiently discriminative grouping unit for cross-cultural value comparison.
Supported by Appendix C.2 (country JSD 0.478 vs 0.214-0.288 for other demographics), but the paper acknowledges it masks within-country heterogeneity and cross-border linguistic minorities.
ad hoc to paper The evaluated models have not memorized the WVS/ISSP benchmark during pretraining.
No contamination test is provided; the interpretation of accuracy as simulation capability rather than memorization depends on this unstated premise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Representational Equality in Cross-country Value Simulation: A Systematic Analysis of Large Language Models." pith.science (2026). https://pith.science/paper/J2RA76NT

@misc{pith2026260808058,
  author       = {Pith},
  title        = {Pith review of: Representational Equality in Cross-country Value Simulation: A Systematic Analysis of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2RA76NT}},
  note         = {Machine review of arXiv:2608.08058}
}

read the original abstract

Traditional methods for studying human opinions often struggle to support representative and scalable research across countries. Large language models (LLMs) can serve as scalable proxies for simulating human opinions, enabling more efficient opinion analysis. However, this use of LLMs requires not only high average accuracy but also representational equality, that is, comparable simulation accuracy across populations. Uneven simulation accuracy may reproduce or amplify societal biases in downstream applications. This study systematically investigates country-level representational equality across 59 countries and finds substantial, systematic inequality. Populations from wealthier and more technologically advanced countries are simulated more accurately. We further compare two foundational intervention pathways, contextual adaptation and parametric modification, and show that improvements in average or target-group accuracy do not necessarily translate into greater representational equality. For contextual adaptation, native-language prompting generally improves accuracy but remains model-dependent, whereas additional information more often improves both accuracy and equality. For parametric modification, language-specific continued post-training improves accuracy for targeted language groups but unevenly, while preference alignment yields no systematic gains in accuracy or equality. Human-annotated preference data generally preserve accuracy better than AI-annotated data. These findings highlight the need for representational equality alongside accuracy and offer guidance for more inclusive, socially responsible LLM-based simulations.

Figures

Figures reproduced from arXiv: 2608.08058 by the authors.

**Figure 2.** Simulation accuracy for different LLMs or countries. (a) Violin plot showing [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

**Figure 3.** Country-level correlates of representational inequality across four macro-level [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Simulated value convergence toward the better-simulated country’s distri [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

**Figure 5.** Figure 5: Simulation accuracy changes (%) of LLMs under native-language prompting. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

**Figure 6.** Figure 6: Simulation results for Jordan on WVS Q150 (forced choice between freedom [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 7.** Figure 7: Simulation accuracy changes across LLMs under different additional [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

**Figure 8.** Figure 8: Changes in the correlations between simulation accuracy and economic, tech [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

**Figure 9.** Figure 9: Effect of additional information on GDP-accuracy association and country [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

**Figure 10.** Figure 10: Effects of multilingual CPT on simulation accuracy. (a) [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

**Figure 11.** Figure 11: Simulation accuracy changes across LLMs under different alignment settings. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

108 extracted references · 49 canonical work pages

[1]

2011 , publisher=

Survey methodology , author=. 2011 , publisher=

2011
[2]

2012 , publisher=

Social science research: Principles, methods, and practices , author=. 2012 , publisher=

2012
[3]

1999 , publisher=

Essential ethnographic methods: Observations, interviews, and questionnaires , author=. 1999 , publisher=

1999
[4]

2010 , publisher=

Participant observation: A guide for fieldworkers , author=. 2010 , publisher=

2010
[5]

Nature Reviews Methods Primers , volume=

Interviews in the social sciences , author=. Nature Reviews Methods Primers , volume=. 2022 , publisher=

2022
[6]

Conservation Biology , volume=

Conducting conservation social science surveys online , author=. Conservation Biology , volume=. 2021 , publisher=

2021
[7]

International Journal of Management, Technology, and Social Sciences (IJMTS) , volume=

Development and validation of survey questionnaire & experimental data--a systematical review-based statistical approach , author=. International Journal of Management, Technology, and Social Sciences (IJMTS) , volume=
[8]

Proceedings of the 34th International Conference on Neural Information Processing Systems , pages=

Language models are few-shot learners , author=. Proceedings of the 34th International Conference on Neural Information Processing Systems , pages=

Show all 108 references

[9]

OpenAI blog , volume=

Language models are unsupervised multitask learners , author=. OpenAI blog , volume=
[10]

Advances in neural information processing systems , volume=

Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=
[11]

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

Character-LLM: A Trainable Agent for Role-Playing , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

2023
[12]

Wang, Noah and Peng, Z.y. and Que, Haoran and Liu, Jiaheng and Zhou, Wangchunshu and Wu, Yuhan and Guo, Hongcheng and Gan, Ruitong and Ni, Zehao and Yang, Jian and Zhang, Man and Zhang, Zhaoxiang and Ouyang, Wanli and Xu, Ke and Huang, Wenhao and Fu, Jie and Peng, Junran , boo...

2024 doi
[13]

Political Analysis , volume=

Out of one, many: Using language models to simulate human samples , author=. Political Analysis , volume=. 2023 , publisher=

2023
[14]

Findings of the Association for Computational Linguistics: ACL 2024 , pages=

Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

2024
[15]

Proceedings of the 36th annual acm symposium on user interface software and technology , pages=

Generative agents: Interactive simulacra of human behavior , author=. Proceedings of the 36th annual acm symposium on user interface software and technology , pages=
[16]

International Conference on Machine Learning , pages=

Whose opinions do language models reflect? , author=. International Conference on Machine Learning , pages=. 2023 , organization=

2023
[17]

arXiv preprint arXiv:2306.16388 , year=

Towards measuring the representation of subjective global opinions in language models , author=. arXiv preprint arXiv:2306.16388 , year=

arXiv
[18]

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

Unintended impacts of llm alignment on global representation , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
[19]

Humanities and Social Sciences Communications , volume=

Performance and biases of Large Language Models in public opinion simulation , author=. Humanities and Social Sciences Communications , volume=. 2024 , publisher=

2024
[20]

Probing Pre-Trained Language Models for Cross-Cultural Differences in Values , year =

Arora, Arnav and Kaffee, Lucie-aim. Probing Pre-Trained Language Models for Cross-Cultural Differences in Values , year =. Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) , publisher =
[21]

arXiv preprint arXiv:2308.12014 , year=

From Instructions to Intrinsic Human Values--A Survey of Alignment Goals for Big Models , author=. arXiv preprint arXiv:2308.12014 , year=

arXiv
[22]

Free peess , year=

The nature of human values , author=. Free peess , year=
[23]

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

Investigating cultural alignment of large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
[24]

International conference on learning representations , volume=

Chateval: Towards better llm-based evaluators through multi-agent debate , author=. International conference on learning representations , volume=
[25]

The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation , volume =

Hartmann, Jochen and Schwenzow, Jasper and Witte, Maximilian , journal =. The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation , volume =
[26]

Instruction tuning with gpt-4 , volume =

Peng, Baolin and Li, Chunyuan and He, Pengcheng and Galley, Michel and Gao, Jianfeng , journal =. Instruction tuning with gpt-4 , volume =
[27]

Findings of the association for computational linguistics: ACL 2023 , pages=

Discovering language model behaviors with model-written evaluations , author=. Findings of the association for computational linguistics: ACL 2023 , pages=

2023
[28]

Alpacafarm: A simulation framework for methods that learn from human feedback , volume =

Dubois, Yann and Li, Chen Xuechen and Taori, Rohan and Zhang, Tianyi and Gulrajani, Ishaan and Ba, Jimmy and Guestrin, Carlos and Liang, Percy S and Hashimoto, Tatsunori B , journal =. Alpacafarm: A simulation framework for methods that learn from human feedback , volume =
[29]

Secrets of rlhf in large language models part ii: Reward modeling , volume =

Wang, Binghai and Zheng, Rui and Chen, Lu and Liu, Yan and Dou, Shihan and Huang, Caishuang and Shen, Wei and Jin, Senjie and Zhou, Enyu and Shi, Chenyu and others , journal =. Secrets of rlhf in large language models part ii: Reward modeling , volume =
[30]

Contrastive post-training large language models on data curriculum , volume =

Xu, Canwen and Rosset, Corby and Del Corro, Luciano and Mahajan, Shweti and McAuley, Julian and Neville, Jennifer and Awadallah, Ahmed Hassan and Rao, Nikhil , journal =. Contrastive post-training large language models on data curriculum , volume =
[31]

International Conference on Machine Learning , pages=

Using large language models to simulate multiple humans and replicate human subject studies , author=. International Conference on Machine Learning , pages=. 2023 , organization=

2023
[32]

2023 , institution=

Large language models as simulated economic agents: What can we learn from homo silicus? , author=. 2023 , institution=

2023
[33]

arXiv preprint arXiv:2402.18144 , year=

Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information , author=. arXiv preprint arXiv:2402.18144 , year=

arXiv
[34]

arXiv preprint arXiv:2310.05984 , year=

Simulating social media using large language models to evaluate alternative news feed algorithms , author=. arXiv preprint arXiv:2310.05984 , year=

arXiv
[35]

Annual Meeting of the Association for Computational Linguistics , year=

Unveiling the Truth and Facilitating Change: Towards Agent-based Large-scale Social Movement Simulation , author=. Annual Meeting of the Association for Computational Linguistics , year=
[36]

arXiv preprint arXiv:2305.05516 , year=

GPT in Game Theory Experiments , author=. arXiv preprint arXiv:2305.05516 , year=. 2305.05516 , archivePrefix=

arXiv
[37]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

Kiss up, kick down: Exploring behavioral changes in multi-modal large language models with assigned visual personas , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

2024
[38]

Towards Understanding and Mitigating Social Biases in Language Models , volume =

Paul Pu Liang and Chiyu Wu and Louis. Towards Understanding and Mitigating Social Biases in Language Models , volume =. Proc. of ICML , editor =
[39]

Nationality Bias in Text Generation , year =

Narayanan Venkit, Pranav and Gautam, Sanjana and Panchanadikar, Ruchi and Huang, Ting-Hao and Wilson, Shomir , booktitle =. Nationality Bias in Text Generation , year =
[40]

When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization , year =

Ladhak, Faisal and Durmus, Esin and Suzgun, Mirac and Zhang, Tianyi and Jurafsky, Dan and McKeown, Kathleen and Hashimoto, Tatsunori , booktitle =. When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization , year =
[41]

Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

Systematic biases in LLM simulations of debates , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

2024
[42]

ArXiv , year=

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , author=. ArXiv , year=
[43]

Political Science Research and Methods , year=

Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics , author=. Political Science Research and Methods , year=
[44]

BMC medical research methodology , volume=

Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups , author=. BMC medical research methodology , volume=. 2014 , publisher=

2014
[45]

International Journal of Public Opinion Quarterly , volume=

Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey , author=. International Journal of Public Opinion Quarterly , volume=. 2006 , publisher=

2006
[46]

Interacci

Examining Social Desirability Bias in Online and Offline Surveys , author=. Interacci. 2019 , url=

2019
[47]

Hague Journal on the Rule of Law , year=

The Worldwide Governance Indicators: Methodology and Analytical Issues , author=. Hague Journal on the Rule of Law , year=
[48]

Online readings in psychology and culture , volume=

Dimensionalizing cultures: The Hofstede model in context , author=. Online readings in psychology and culture , volume=. 2011 , publisher=

2011
[49]

Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

Aligning language models to user opinions , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

2023
[50]

Discourse Studies , year=

Identity and interaction: a sociocultural linguistic approach , author=. Discourse Studies , year=
[51]

PLoS Climate , volume=

Can large language models estimate public opinion about global warming? an empirical assessment of algorithmic fidelity and bias , author=. PLoS Climate , volume=. 2024 , publisher=

2024
[52]

ArXiv , year=

Language Models Trained on Media Diets Can Predict Public Opinion , author=. ArXiv , year=
[53]

PsyArXiv , year=

Which humans? , author=. PsyArXiv , year=. doi:10.31234/osf.io/5b26t , url=

doi
[54]

ACM Computing Surveys (CSUR) , year=

A Survey on Bias and Fairness in Machine Learning , author=. ACM Computing Surveys (CSUR) , year=
[55]

Computational Linguistics , year=

Bias and Fairness in Large Language Models: A Survey , author=. Computational Linguistics , year=
[56]

Online Readings in Psychology and Culture , year=

An Overview of the Schwartz Theory of Basic Values , author=. Online Readings in Psychology and Culture , year=
[57]

Political Analysis , year=

Synthetic Replacements for Human Survey Data? The Perils of Large Language Models , author=. Political Analysis , year=
[58]

Companion Proceedings of the ACM on Web Conference 2024 , year=

Simulating Human Society with Large Language Model Agents: City, Social Media, and Economic System , author=. Companion Proceedings of the ACM on Web Conference 2024 , year=

2024
[59]

Advances in Neural Information Processing Systems , volume=

Ali-agent: Assessing llms' alignment with human values via agent-based evaluation , author=. Advances in Neural Information Processing Systems , volume=
[60]

Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=

CrowS-pairs: A challenge dataset for measuring social biases in masked language models , author=. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=

2020
[61]

ArXiv , year=

DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models , author=. ArXiv , year=
[62]

Annual Meeting of the Association for Computational Linguistics , year=

Cultural Learning-Based Culture Adaptation of Language Models , author=. Annual Meeting of the Association for Computational Linguistics , year=
[63]

Specializing large language models to simulate survey response distributions for global populations , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Lon...

2025
[64]

Transactions of the Association for Computational Linguistics , year=

Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs , author=. Transactions of the Association for Computational Linguistics , year=
[65]

Journal of Information Technology & Politics , pages=

Beyond partisan leaning: a comparative analysis of political bias in large language models , author=. Journal of Information Technology & Politics , pages=. 2026 , doi=

2026
[66]

ArXiv , year=

Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas , author=. ArXiv , year=
[67]

Advances in Neural Information Processing Systems , volume=

Culturellm: Incorporating cultural differences into large language models , author=. Advances in Neural Information Processing Systems , volume=
[68]

Advances in Neural Information Processing Systems , volume=

Culturepark: Boosting cross-cultural understanding in large language models , author=. Advances in Neural Information Processing Systems , volume=
[69]

International Conference on Machine Learning , year=

MaxMin-RLHF: Alignment with Diverse Human Preferences , author=. International Conference on Machine Learning , year=
[70]

Proceedings of the fourth workshop on multilingual representation learning (MRL 2024) , pages=

Sambalingo: Teaching large language models new languages , author=. Proceedings of the fourth workshop on multilingual representation learning (MRL 2024) , pages=

2024
[71]

2010 , publisher=

Strategic Management: Awareness & Change , author=. 2010 , publisher=

2010
[72]

Annual Meeting of the Association for Computational Linguistics , year=

Navigating the Political Compass: Evaluating Multilingual LLMs across Languages and Nationalities , author=. Annual Meeting of the Association for Computational Linguistics , year=
[73]

Journal of comparative economics , volume=

Culture rules: The foundations of the rule of law and other norms of governance , author=. Journal of comparative economics , volume=. 2007 , publisher=

2007
[74]

ArXiv , year=

Llama 2: Open Foundation and Fine-Tuned Chat Models , author=. ArXiv , year=
[75]

ArXiv , year=

Mistral 7B , author=. ArXiv , year=
[76]

ArXiv , year=

ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools , author=. ArXiv , year=
[77]

ArXiv , year=

Qwen2.5 Technical Report , author=. ArXiv , year=
[78]

ArXiv , year=

The Llama 3 Herd of Models , author=. ArXiv , year=
[79]

ArXiv , year=

GLM-130B: An Open Bilingual Pre-trained Model , author=. ArXiv , year=
[80]

ACM Computing Surveys , volume=

From individual to society: A survey on social simulation driven by large language model-based agents , author=. ACM Computing Surveys , volume=. 2026 , publisher=

2026
[81]

Sociological Methods & Research , year=

Fairness in Criminal Justice Risk Assessments: The State of the Art , author=. Sociological Methods & Research , year=
[82]

2018 IEEE/ACM International Workshop on Software Fairness (FairWare) , year=

Fairness Definitions Explained , author=. 2018 IEEE/ACM International Workshop on Software Fairness (FairWare) , year=

2018
[83]

Econometric Reviews , year=

Gini Coefficient and the Relative Tax Burden , author=. Econometric Reviews , year=
[84]

2021 , howpublished=

A Guide to Formulating Equity and Fairness in an Optimization Model , author=. 2021 , howpublished=

2021
[85]

AIDBEI , year=

Characterizing Intersectional Group Fairness with Worst-Case Comparisons , author=. AIDBEI , year=
[86]

Journal of the Royal Society Interface , volume=

Expanding the measurement of culture with a sample of two billion humans , author=. Journal of the Royal Society Interface , volume=. 2022 , publisher=

2022
[87]

Findings of the Association for Computational Linguistics: ACL 2024 , pages=

``My answer is C'': First-token probabilities do not match text answers in instruction-tuned language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

2024
[88]

Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

The potential and challenges of evaluating attitudes, opinions, and values in large language models , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

2024
[89]

ACM Transactions on Asian and Low-Resource Language Information Processing , volume=

Morphologically Motivated Input Variations and Data Augmentation in Turkish-English Neural Machine Translation , author=. ACM Transactions on Asian and Low-Resource Language Information Processing , volume=. 2023 , publisher=

2023
[90]

arXiv preprint arXiv:2505.04531 , year=

Overcoming data scarcity in generative language modelling for low-resource languages: A systematic review , author=. arXiv preprint arXiv:2505.04531 , year=

arXiv
[91]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

When is multilinguality a curse? language modeling for 250 high-and low-resource languages , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

2024
[92]

Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

Targeted multilingual adaptation for low-resource language families , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

2024
[93]

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

MoVa: Towards generalizable classification of human morals and values , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

2025
[94]

Proceedings of the Twentieth European Conference on Computer Systems , pages=

Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=
[95]

Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations) , pages=

Llamafactory: Unified efficient fine-tuning of 100+ language models , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations) , pages=
[96]

arXiv preprint arXiv:2402.02680 , year=

Large language models are geographically biased , author=. arXiv preprint arXiv:2402.02680 , year=

arXiv
[97]

Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

On the dangers of stochastic parrots: Can language models be too big? , author=. Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

2021
[98]

ACM Journal of Data and Information Quality , volume=

Biases in large language models: origins, inventory, and discussion , author=. ACM Journal of Data and Information Quality , volume=. 2023 , publisher=

2023
[99]

Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

Impact of pretraining term frequencies on few-shot numerical reasoning , author=. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

2022
[100]

2023 , eprint=

Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 , author=. 2023 , eprint=

2023
[101]

Advances in neural information processing systems , volume=

Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=
[102]

arXiv preprint arXiv:2402.03300 , year=

Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

arXiv
[103]

Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization , pages=

The case for globalizing fairness: a mixed methods study on colonialism, AI, and health in Africa , author=. Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization , pages=
[104]

Journal of the Royal statistical society: series B (Methodological) , volume=

Controlling the false discovery rate: a practical and powerful approach to multiple testing , author=. Journal of the Royal statistical society: series B (Methodological) , volume=. 1995 , publisher=

1995
[105]

Scandinavian journal of statistics , pages=

A simple sequentially rejective multiple test procedure , author=. Scandinavian journal of statistics , pages=. 1979 , publisher=

1979
[106]

Psychological Bulletin , volume=

Tests for comparing elements of a correlation matrix , author=. Psychological Bulletin , volume=. 1980 , publisher=

1980
[107]

Psychological Bulletin , volume=

Dominance statistics: Ordinal analyses to answer ordinal questions , author=. Psychological Bulletin , volume=. 1993 , doi=

1993
[108]

Technometrics , volume=

Multiple comparisons using rank sums , author=. Technometrics , volume=. 1964 , publisher=

1964

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

2011 , publisher=

Survey methodology , author=. 2011 , publisher=

2011

[2] [2]

2012 , publisher=

Social science research: Principles, methods, and practices , author=. 2012 , publisher=

2012

[3] [3]

1999 , publisher=

Essential ethnographic methods: Observations, interviews, and questionnaires , author=. 1999 , publisher=

1999

[4] [4]

2010 , publisher=

Participant observation: A guide for fieldworkers , author=. 2010 , publisher=

2010

[5] [5]

Nature Reviews Methods Primers , volume=

Interviews in the social sciences , author=. Nature Reviews Methods Primers , volume=. 2022 , publisher=

2022

[6] [6]

Conservation Biology , volume=

Conducting conservation social science surveys online , author=. Conservation Biology , volume=. 2021 , publisher=

2021

[7] [7]

International Journal of Management, Technology, and Social Sciences (IJMTS) , volume=

Development and validation of survey questionnaire & experimental data--a systematical review-based statistical approach , author=. International Journal of Management, Technology, and Social Sciences (IJMTS) , volume=

[8] [8]

Proceedings of the 34th International Conference on Neural Information Processing Systems , pages=

Language models are few-shot learners , author=. Proceedings of the 34th International Conference on Neural Information Processing Systems , pages=