Pith. sign in

REVIEW 3 major objections 6 minor 165 references

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Exact token-level probability distributions, run through a fully crossed factorial ANOVA, isolate causal LLM biases that aggregate benchmarks miss.

desk verdict Worth a serious referee: the distributional ANOVA core is sound, but the multivariate Consensus formula is dimensionally wrong as written and the reported dispersion/direction 'exactness' depends on an arbitrary unacknowledged coupling choice. read the letter →

arxiv 2608.10503 v1 pith:5LGD5HNB submitted 2026-08-11 cs.CL

Davood Wadi , Mohsen Ghodrat , Matthew Philp This is my paper · ORCID
classification cs.CL
keywords LLMbehavioralevaluationtoken-levelprobabilitymassfunctionsfullycrossedfactorialdesigndistributionalANOVAHoeffdingdecompositionmultivariateconsensuscountry-of-originbiasexactinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to make LLM attitude and bias measurement exact rather than sampled. It replaces unstructured prompting with fully crossed factorial experiments and replaces Monte Carlo text generation with the model's own next-token probability mass functions, then processes those PMFs analytically. The central claim is Theorem 3.1: in any fully crossed design, the expectations of the isolated main-effect and interaction-effect distributions recover the classical unique Hoeffding/ANOVA decomposition, so a construct like country-of-origin bias can be measured as an interaction effect with baseline and main effects stripped away. If the claim holds, behavioral evaluation of LLMs becomes deterministic, free of sampling noise, and able to attribute a bias to baseline traits, contexts, or their interaction instead of to correlated prompt content. The paper demonstrates the pipeline on a five-model ethnocentrism case study in which aggregate benchmarks are directionally wrong for specific models.

What carries the argument

The load-bearing object is the distributional Hoeffding/ANOVA decomposition built from paired contrasts. Each effect distribution is formed by drawing from two marginal PMFs through a shared uniform quantile, the comonotone coupling, which preserves the marginals while minimizing the variance of the difference; this is what makes the contrast's expectation equal to the classical ANOVA effect while retaining the full PMF. Around that core, discrete convolution propagates the item-level PMFs into an exact composite-score distribution, and a multivariate generalization of the Consensus metric supplies the ordinal-aware certainty measure that Shannon entropy lacks.

What would settle it

Recompute the interaction summaries in Table 6 with independent or countermonotone pairing instead of the comonotone coupling; if the dPD and SNR of the US-model ingroup cells change materially, the reported directionality is a coupling artifact rather than a property of the model.

Watch

Extended reading notes

Core claim

The central discovery is that a fully crossed factorial experiment over exact token-level PMFs yields a distributional ANOVA whose expectations reproduce the classical unique Hoeffding decomposition. The paper builds a grand-mixture baseline and marginal-slice distributions, then defines main-effect and interaction-effect distributions as paired contrasts under a comonotone coupling, so the expectation of each effect equals the corresponding fixed-effects ANOVA parameter. This is what makes country-of-origin bias a well-defined interaction term: the US-developed models show positive own-country interactions (+3.21 for Gemma and +2.59 for Llama), while the aggregate target-country main effect would have hidden the sign reversals visible in the interaction table. The intended reading is that the pipeline is exact at the distributional level for means, with all aleatoric uncertainty propagated from tokens to the composite score.

Load-bearing premise

The reported spreads, signal-to-noise ratios, and directional probabilities depend on the paper's chosen pairing of the compared distributions, and if that pairing is not canonical, those strengths and directions can change even though the effect means remain fixed.

Editorial extensions

If this is right

  • Small effect parameters that flip sign 18% of the time at N=10 under text sampling are recovered exactly in one forward pass, so subtle interactions can be measured without Monte Carlo noise.
  • Aggregate target-country effects can be directionally wrong for individual models: the paper finds a negative aggregate France effect but a positive French interaction for Ministral.
  • Confounders such as prompt framing can be added as crossed factors, turning prompt sensitivity into an isolated main effect and model-by-framing interaction instead of an uncontrolled critique.
  • The same Theorem 3.1 guarantee applies to any fully crossed design, so any ordinal instrument can be decomposed into baseline, main, and interaction effect distributions with the same machinery.
  • Exact convolution of item PMFs propagates all aleatoric uncertainty from tokens to the final composite score, so downstream comparisons carry the model's full response distribution rather than point estimates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Exactness is proven for expectations; the spread and direction metrics inherit the chosen comonotone coupling, so an editor would want replications to report whether dPD and SNR are stable under independent or countermonotone pairing.
  • The grand-mixture baseline weights every experimental condition equally, so a deployment-realistic baseline would need prevalence-weighted conditions; the theorem still applies to those weights.
  • The same decomposition could separate a model that always disfavors a demographic (main effect) from one that disfavors it only in specific contexts (interaction), which is exactly the gender-bias question the introduction poses.
  • The framework's exactness is scoped to constrained single-token responses; extending it to chain-of-thought would require an integrated distribution over latent multi-token paths, not just a convolution of item-level PMFs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an exact-PMF framework for measuring LLM attitudes and biases, replacing Monte Carlo text sampling with direct token-level probability mass functions. It introduces a fully crossed factorial design, a multivariate ordinal Consensus metric, and a distributional ANOVA/Hoeffding decomposition that isolates baseline, main-effect, and interaction-effect distributions. A case study on the CETSCALE across five LLMs claims to expose country-of-origin interaction effects that aggregate benchmarks obscure, and an analysis of sampling cost shows that standard finite-sample estimators can flip the sign of small effects.

Significance. The core theoretical contribution—that the expectations of the construction's effect distributions recover the unique Hoeffding/ANOVA decomposition—is proved carefully in Appendix A.4 and appears correct. If the framework holds, it would give the NLP community a principled way to attribute behavioral differences to main effects versus interactions without sampling noise, using exact convolutions and closed-form contrast distributions. The detailed appendices, transparent case study, and explicit falsifiable predictions are strengths. However, two issues currently limit the paper: the published Consensus formula is dimensionally inconsistent, and the reported dispersion-based summaries (SD, SNR, dPD) depend on an arbitrary comonotone coupling rather than being intrinsic model properties. Both are fixable, so the result is not fundamentally unsound, but the exactness claims need to be qualified.

major comments (3)
  1. [3.3, Eq. (1); App. A.3, Eq. (12)] The multivariate Consensus as written divides a squared Euclidean distance by a linear distance: with dmax defined as a maximum distance on the scale, the argument 1 − ||y−µ||²/dmax can become negative, making the logarithm undefined. For K=17 and a 7-point scale, such negative arguments arise already for moderate deviations from the centroid, so the published formula cannot be what produced the values in Table 5. The normalization should presumably be by dmax², or the numerator should use the unsquared distance. Please correct the definition, restate the text, and recompute or confirm the affected Consensus values.
  2. [4.2, Table 6, Table 9; App. A.4.5–A.4.6, Def. A.8/A.10, Eqs. (42), (69)] Theorem 3.1 guarantees coupling-invariance only for expectations. The SD, SNR, and dPD values reported in Tables 2, 6, and 9 are computed under the comonotone (maximal-dependence) coupling, and any other coupling with the same marginals—independent, countermonotone, or otherwise—preserves the expectation-level theorem but changes these dispersion and directionality summaries. The dPD values of 0.63 and 0.58 cited as evidence of moderate US ingroup favoritism are therefore not intrinsic properties of the models' predictive distributions. The paper should report a coupling-sensitivity analysis or explicitly qualify every dispersion-based summary as conditional on the comonotone pairing.
  3. [4.2, Table 9] The comparison between the aggregate Target-Country main effect and the Model×Target interaction is presented as an empirical demonstration that aggregate benchmarks are 'directionally incorrect.' Because main effects and interactions in a fully crossed ANOVA decomposition are orthogonal by construction, the sign reversal between Panel A and Panel B is a mathematical necessity, not a data-dependent discovery. The text should state this explicitly; as written, it overstates the empirical content of the comparison.
minor comments (6)
  1. [App. A.1 vs. Sec. 3.2] The statement that |Vval| = |Y| contradicts the preceding description and Section 3.2, where multiple token surface forms (e.g., " 7" and "7") map to the same ordinal value. The mapping φ is surjective but not injective when tokenizer variants exist; please remove the cardinality equality.
  2. [Sec. 2] The phrase "As demonstrated in 3" should read "As demonstrated in Section 3."
  3. [Table 5] Several entropy values exceed log2(7) ≈ 2.807 (e.g., 7.315, 35.519), so the entropy is evidently summed over the 17 items. The table caption and the surrounding text should state this explicitly, because the Consensus values are not summed and the two metrics are otherwise not comparable.
  4. [Sec. 3.4 / App. A.4] The main text says Theorem 3.1 is proved in App. A.9, but App. A.9 concerns sampling estimators; the actual proof appears in App. A.4.4–A.4.6. Please correct the cross-reference.
  5. [Eq. (12)] The display for dmax is typeset ambiguously (the exponent on (y_max−y_min) is unclear), and the phrase "maximum diagonal distance on the Likert scale" is imprecise. Please write the formula explicitly and clarify that dmax is a distance in the K-dimensional response space.
  6. [App. A.4.11] The text compares SNR to Cohen's d but uses different definitions; please clarify that SNR is not Cohen's d and that the heuristic thresholds are only descriptive references, to avoid potential misinterpretation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Theorem 3.1 is proved in-appendix, the effect distributions are deterministic functions of observed PMFs, and no fitted parameter is presented as a prediction.

full rationale

The paper's central guarantee, Theorem 3.1, is not circular: the expectation-level Hoeffding/ANOVA decomposition is stated as Theorem A.5 with a complete proof including uniqueness via Möbius inversion (Rota 1964, an external classical reference), and Corollary A.11 proves E[EU(λU)] = µU(λU) directly from the definition of the comonotone-coupled contrast and the linearity of expectation. The effect distributions themselves are deterministic functions of the measured next-token PMFs ('Each paired-difference PMF in Eq. (43) is computed exactly via the mass-matching formula in Eq. (40). No estimation or approximation is used at any stage.', App. A.4.5), so no parameter is fitted to a subset of data and then re-predicted. The comparison to human CETSCALE samples is explicitly a 'historical magnitude anchor,' not a calibration of the model, and the sampling-cost analysis is a labeled simulation that treats the exact PMFs as ground truth by design. Two concerns must nevertheless be weighed. First, the conditional-independence assumption is attributed to Wadi and Fredette (2025), prior work sharing an author; but the paper also justifies it directly via the statelessness of LLM forward passes, so the self-citation is not load-bearing support for a contested result. Second, the reported SD, SNR, and dPD do depend on the comonotone coupling, exactly as the skeptic's analysis identifies; however, the paper explicitly acknowledges in App. A.4.4 that such contrasts 'depend on a chosen coupling' at the distributional level, and its exactness claims refer to sampling-free computation, not coupling invariance. The coupling is a disclosed convention justified by variance minimization and external optimal-transport theory (Villani 2008). Coupling-sensitivity of the dispersion summaries is a legitimate robustness concern for a correctness review, but it is not a circular reduction of an output to its input.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five stated assumptions, none of which is a fitted parameter. The most consequential are the conditional-independence factorization and the comonotone coupling choice, which shape all reported distributions.

assumptions (5)
  • domain assumption Item responses are conditionally independent given the experimental condition, so the joint item PMF factorizes and the composite score PMF is the discrete convolution of item PMFs.
    Invoked in Sec. 3.1 and App. A.2 (Eq. 8), justified by the statelessness of LLM forward passes, but never empirically tested. If item draws were correlated, the composite distribution and all downstream effect SDs would be wrong.
  • standard math The design space is finite, fully crossed, and endowed with a uniform measure, giving a symmetric ANOVA decomposition.
    Assumption A.1 in App. A.4.1; standard for fixed-effects factorial designs and needed for the Hoeffding decomposition uniqueness proof.
  • ad hoc to paper The comonotone coupling is the correct pairing for constructing contrast distributions.
    Defs. A.8 and A.10 and Eq. (42)-(43). The coupling preserves expectations for any pairing, but the shape of the effect distributions, and hence SD, SNR, and dPD, depend on this choice. No justification is given beyond wanting to minimize the variance of the paired difference.
  • domain assumption Lexical variants of scale tokens such as leading-space digits map to ordinal values, while semantic equivalents like 'seven' are excluded from the valid set.
    Sec. 3.2 and Eq. (7); the choice of V_val is a modeling decision that affects the renormalized item PMFs and hence every downstream number.
  • domain assumption A model's next-token distribution over the digits 1 through 7 under a constrained prompt is a valid measure of its latent attitude on the construct.
    Sec. 3.2 and the prompt template in Fig. 7; the entire case study treats forced digit responses as behavioral data, which presupposes construct validity for LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases." pith.science (2026). https://pith.science/paper/5LGD5HNB

@misc{pith2026260810503,
  author       = {Pith},
  title        = {Pith review of: Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5LGD5HNB}},
  note         = {Machine review of arXiv:2608.10503}
}
read the original abstract

As Large Language Models (LLMs) are increasingly deployed as autonomous agents, accurately evaluating their latent values and biases is critical. The NLP community typically evaluates models using large, unstructured benchmarks. While effective for general capabilities, these datasets fundamentally conflate causal mechanisms: even when an aggregate bias is detected, unstructured evaluations cannot disentangle whether it stems from baseline traits, contextual confounders, or complex interactions. To address this, we introduce an analytically exact framework for the controlled behavioral evaluation of LLMs. We bridge human psychometrics with LLM mechanics by resolving gaps in design, measurement, and analysis. First, we replace unstructured prompting with fully crossed factorial experiments to systematically isolate causal main and interaction effects. Second, we eliminate Monte Carlo text sampling noise by operating directly on exact, token-level Probability Mass Functions (PMFs). Third, we derive a multivariate ordinal consensus metric and a distributional ANOVA to process these PMFs analytically. We validate our framework with a case study on consumer ethnocentrism across five LLMs, demonstrating how our approach isolates systemic country-of-origin biases that aggregate benchmarks otherwise obscure.

Figures

Figures reproduced from arXiv: 2608.10503 by the authors.

Figure 1
Figure 1. High-level exact-PMF framework: Fully crossed experiment design [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Multivariate Consensus (Cns) for the CETSCALE instrument. Lower values indicate severe behavioral polarization. Construct. To contextualize the LLMs’ response to the CETSCALE, we compare each model’s pre￾dictive distribution for TARGET=USA against the human sample means and standard deviations re￾ported in the original CETSCALE validation stud￾ies (Shimp and Sharma, 1987; [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. Failure Rate across the evaluated LLMs. Consensus. Across all target countries, four mod￾els show exceptionally high internal agreement re￾garding the instrument, maintaining Cns > 0.88 ( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Distribution of CETSCALE for Aya Expanse [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Interactions of Model and Target Country. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The failure of aggregate Consensus in captur [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Prompt setup: a fixed system prompt enforces the response schema, while Target Country (sample shown [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Comparison of CETSCALE scores for the USA target condition between human populations re￾ported in Shimp and Sharma (1987) and LLMs evalu￾ated in this study. Points denote expected values, and error bars indicate standard deviation. All scores are based on the 17-item C…
Figure 9
Figure 9. Figure 9: Main Effect of Target Country The distortion is small for high-consensus mod￾els (Gemma: 0.18 points) but substantial for low￾consensus Ministral (M|Bias| = 1.38, reaching up to +2.79 points at low temperature and re￾versing sign at T=1.3; Figures 12–13), compa￾rable t…
Figure 10
Figure 10. Figure 10: Distribution of CETSCALE for Model and Target Country. [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Standard error of the sampling-based CETSCALE estimator versus per-condition sample size N, by model. Low-consensus models (e.g., Ministral) incur substantially larger error at every budget [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 13
Figure 13. Figure 13: Mean decoding bias versus top-p, colored by Consensus. Monte Carlo sampling paradigm, we ask a decision￾relevant question: what fraction of the causal ef￾fects we report could a sampling-based study get wrong (i.e., recover with the opposite sign) purely as an artifac…
Figure 12
Figure 12. Figure 12: Mean decoding bias versus temperature, colored by Consensus. Low-consensus models exhibit large, sign-changing distortions [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 14
Figure 14. Figure 14: Sign-flip probability of a sampling-based [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

165 extracted references · 32 canonical work pages

  1. [1]

    Frontiers of Computer Science , volume=

    A survey on large language model based autonomous agents , author=. Frontiers of Computer Science , volume=. 2024 , publisher=

    2024

  2. [2]

    Science China Information Sciences , volume=

    The rise and potential of large language model based agents: A survey , author=. Science China Information Sciences , volume=. 2025 , publisher=

    2025

  3. [3]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    arXiv 2009

  4. [4]

    Annals of the New York Academy of Sciences , volume=

    Holistic evaluation of language models , author=. Annals of the New York Academy of Sciences , volume=. 2023 , publisher=

    2023

  5. [5]

    Shopping By Algorithm: How Agentic AI Deploys Human Heuristics as a Surrogate Consumer , author=

  6. [6]

    Transactions of the Association for Computational Linguistics , volume=

    Causal inference in natural language processing: Estimation, prediction, interpretation and beyond , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    2022

  7. [7]

    Science Advances , volume=

    How to make causal inferences using texts , author=. Science Advances , volume=. 2022 , publisher=

    2022

  8. [8]

    Advances in methods and practices in psychological science , volume=

    Measurement schmeasurement: Questionable measurement practices and how to avoid them , author=. Advances in methods and practices in psychological science , volume=. 2020 , publisher=

    2020

Show all 165 references
  1. [9]

    2009 , publisher=

    Causality , author=. 2009 , publisher=

    2009

  2. [10]

    arXiv preprint arXiv:2510.26707 , year=

    Value Drifts: Tracing Value Alignment During LLM Post-Training , author=. arXiv preprint arXiv:2510.26707 , year=

    arXiv

  3. [11]

    arXiv preprint arXiv:2603.05909 , year=

    InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning , author=. arXiv preprint arXiv:2603.05909 , year=

  4. [12]

    Psychology & Marketing , volume=

    Using large language models to generate silicon samples in consumer and marketing research: Challenges, opportunities, and guidelines , author=. Psychology & Marketing , volume=. 2024 , publisher=

    2024

  5. [13]

    Journal of Electronic Commerce Research , volume=

    The interplay of altruism and financial incentives: Maximizing online reviews through effective messaging , author=. Journal of Electronic Commerce Research , volume=

  6. [14]

    Phalanx , volume=

    The Likert scale what it is and how to use it , author=. Phalanx , volume=. 2017 , publisher=

    2017

  7. [15]

    Are Multilingual LLM s Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings

    Cecilia Liu, Chen and Koto, Fajri and Baldwin, Timothy and Gurevych, Iryna. Are Multilingual LLM s Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings. Proceedings of the 2024 Conference of the North American Chapter of the Association for Co...

    2024 doi

  8. [16]

    Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge

    Tanwar, Eshaan and Chatterjee, Anwoy and Saxon, Michael and Albalak, Alon and Wang, William Yang and Chakraborty, Tanmoy. Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge. Proceedings of the 2025 Conference on...

    2025 doi

  9. [17]

    W orld V alues B ench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models

    Zhao, Wenlong and Mondal, Debanjan and Tandon, Niket and Dillion, Danica and Gray, Kurt and Gu, Yuling. W orld V alues B ench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models. Proceedings of the 2024 Joint International Conference on Compu...

    2024

  10. [18]

    On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena

    Naous, Tarek and Xu, Wei. On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologie...

    2025 doi

  11. [19]

    Tailored Emotional LLM -Supporter: Enhancing Cultural Sensitivity

    Liu, Chen Cecilia and Arnaout, Hiba and Kova c i \'c , Nils and Atzil-Slonim, Dana and Gurevych, Iryna. Tailored Emotional LLM -Supporter: Enhancing Cultural Sensitivity. Proceedings of the 19th Conference of the E uropean Chapter of the A ssociation for C omputational L ingui...

    2026 doi

  12. [20]

    Reading between the Lines: Can LLM s Identify Cross-Cultural Communication Gaps?

    Saha, Sougata and Pandey, Saurabh Kumar and Gupta, Harshit and Choudhury, Monojit. Reading between the Lines: Can LLM s Identify Cross-Cultural Communication Gaps?. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational L...

    2025 doi

  13. [21]

    Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLM s

    Kabir, Mohsinul and Abrar, Ajwad and Ananiadou, Sophia. Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLM s. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.2

    2025 doi

  14. [22]

    Musical Ethnocentrism in Large Language Models

    Kruspe, Anna. Musical Ethnocentrism in Large Language Models. Proceedings of the 3rd Workshop on NLP for Music and Audio (NLP4MusA). 2024

    2024

  15. [23]

    Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Morality

    Jung, Jana and Lutz, Marlene and Sen, Indira and Strohmaier, Markus. Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Morality. Proceedings of the 19th Conference of the E uropean Chapter of the A ssociation for C omputational L ...

    2026 doi

  16. [24]

    Psychometric Predictive Power of Large Language Models

    Kuribayashi, Tatsuki and Oseki, Yohei and Baldwin, Timothy. Psychometric Predictive Power of Large Language Models. Findings of the Association for Computational Linguistics: NAACL 2024. 2024. doi:10.18653/v1/2024.findings-naacl.129

    2024 doi

  17. [25]

    You don ' t need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments

    Shu, Bangzhao and Zhang, Lechen and Choi, Minje and Dunagan, Lavinia and Logeswaran, Lajanugen and Lee, Moontae and Card, Dallas and Jurgens, David. You don ' t need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on P...

    2024 doi

  18. [26]

    Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis

    Zheng, Hongyi and Saparov, Abulhair. Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.277

    2023 doi

  19. [27]

    Evaluating Language Model Character Traits

    Ward, Francis Rhys and Yang, Zejia and Jackson, Alex and Brown, Randy and Smith, Chandler and Colverd, Grace Beaney and Thomson, Louis Alexander and Douglas, Raymond and Bartak, Patrik and Rowan, Andrew. Evaluating Language Model Character Traits. Findings of the Association f...

    2024 doi

  20. [28]

    Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions

    Deas, Nicholas and McKeown, Kathleen. Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.981

    2025 doi

  21. [29]

    and Askell, Amanda and Grosse, Roger and Hernandez, Danny and Ganguli, Deep and Hubinger, Evan and Schiefer, Nicholas and Kaplan, Jared

    Perez, Ethan and Ringer, Sam and Lukosiute, Kamile and Nguyen, Karina and Chen, Edwin and Heiner, Scott and Pettit, Craig and Olsson, Catherine and Kundu, Sandipan and Kadavath, Saurav and Jones, Andy and Chen, Anna and Mann, Benjamin and Israel, Brian and Seethor, Bryan and M...

    2023

  22. [30]

    Journal of medical Internet research , volume=

    Token probabilities to mitigate large language models overconfidence in answering medical questions: quantitative study , author=. Journal of medical Internet research , volume=. 2025 , publisher=

    2025

  23. [31]

    arXiv preprint arXiv:2207.05221 , year=

    Language models (mostly) know what they know , author=. arXiv preprint arXiv:2207.05221 , year=

    arXiv

  24. [32]

    Navigating the Political Compass: Evaluating Multilingual LLM s across Languages and Nationalities

    Helwe, Chadi and Balalau, Oana and Ceolin, Davide. Navigating the Political Compass: Evaluating Multilingual LLM s across Languages and Nationalities. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.883

    2025 doi

  25. [33]

    Evaluating Psychological Safety of Large Language Models

    Li, Xingxuan and Li, Yutong and Qiu, Lin and Joty, Shafiq and Bing, Lidong. Evaluating Psychological Safety of Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.108

    2024 doi

  26. [34]

    Decoding Emergent Big Five Traits in Large Language Models: Temperature-Dependent Expression and Architectural Clustering

    Zacharopoulos, Christos Nikolaos and Kyriakoglou, Revekka. Decoding Emergent Big Five Traits in Large Language Models: Temperature-Dependent Expression and Architectural Clustering. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4...

    2025 doi

  27. [35]

    How Personality Traits Shape LLM Risk-Taking Behaviour

    Hartley, John and Hamill, Conor Brian and Seddon, Dale and Batra, Devesh and Okhrati, Ramin and Khraishi, Raad. How Personality Traits Shape LLM Risk-Taking Behaviour. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.1085

    2025 doi

  28. [36]

    Personality Vector: Modulating Personality of Large Language Models by Model Merging

    Sun, Seungjong and Baek, Seo Yeon and Kim, Jang Hyun. Personality Vector: Modulating Personality of Large Language Models by Model Merging. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.1253

    2025 doi

  29. [37]

    P ersona LLM : Investigating the Ability of Large Language Models to Express Personality Traits

    Jiang, Hang and Zhang, Xiajie and Cao, Xubo and Breazeal, Cynthia and Roy, Deb and Kabbara, Jad. P ersona LLM : Investigating the Ability of Large Language Models to Express Personality Traits. Findings of the Association for Computational Linguistics: NAACL 2024. 2024. doi:10...

    2024 doi

  30. [38]

    Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025) , pages=

    Demographics and Democracy: Benchmarking LLMs’ Gender Bias and Political Leaning in European Parliament , author=. Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025) , pages=

    2025

  31. [39]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Hidden persuaders: Llms’ political leaning and their influence on voters , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  32. [40]

    Multilingual Political Views of Large Language Models: Identification and Steering , author=. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguisti...

  33. [41]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Analyzing political bias in llms via target-oriented sentiment classification , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  34. [42]

    arXiv preprint arXiv:2509.08853 , year=

    POW: Political overton windows of large language models , author=. arXiv preprint arXiv:2509.08853 , year=

    arXiv

  35. [43]

    Findings of the Association for Computational Linguistics: EMNLP 2025 , pages=

    Probing Political Ideology in Large Language Models: How Latent Political Representations Generalize Across Tasks , author=. Findings of the Association for Computational Linguistics: EMNLP 2025 , pages=

    2025

  36. [44]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Measuring political bias in large language models: What is said and how it is said , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  37. [45]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in LLMs , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  38. [46]

    Computational Linguistics , volume=

    Lmlpa: Language model linguistic personality assessment , author=. Computational Linguistics , volume=. 2025 , publisher=

    2025

  39. [47]

    Likert , author=

    Decoding LLM personality measurement: Forced-choice vs. Likert , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  40. [48]

    Proceedings of the First Workshop on Large Language Model Memorization (L2M2) , pages=

    Better aligned with survey respondents or training data? unveiling political leanings of llms on us supreme court cases , author=. Proceedings of the First Workshop on Large Language Model Memorization (L2M2) , pages=

  41. [49]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    Codet5+: Open code large language models for code understanding and generation , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  42. [50]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  43. [51]

    2008 , publisher=

    Optimal transport: old and new , author=. 2008 , publisher=

    2008

  44. [52]

    Sensitivity estimates for nonlinear mathematical models , author=. Math. Model. Comput. Exp. , volume=

  45. [53]

    On the foundations of combinatorial theory: I

    Rota, Gian-Carlo , booktitle=. On the foundations of combinatorial theory: I. Theory of M. 1964 , publisher=

    1964

  46. [54]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  47. [55]

    2024 , eprint=

    Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier , author=. 2024 , eprint=

    2024

  48. [56]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

  49. [57]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  50. [58]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  51. [59]

    2003 , publisher=

    Behavioral game theory: Experiments in strategic interaction , author=. 2003 , publisher=

    2003

  52. [60]

    Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias

    Itzhak, Itay and Stanovsky, Gabriel and Rosenfeld, Nir and Belinkov, Yonatan. Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias. Transactions of the Association for Computational Linguistics. 2024. doi:10.1162/tacl_a_00673

    2024 doi

  53. [61]

    How Does Cognitive Bias Affect Large Language Models? A Case Study on the Anchoring Effect in Price Negotiation Simulations

    Takenami, Yoshiki and Huang, Yin Jou and Murawaki, Yugo and Chu, Chenhui. How Does Cognitive Bias Affect Large Language Models? A Case Study on the Anchoring Effect in Price Negotiation Simulations. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. d...

    2025 doi

  54. [62]

    Cognitive Biases, Task Complexity, and Result Interpretability in Large Language Models

    Mina, Mario and Ruiz-Fern \'a ndez, Valle and Falc \ a o, J \'u lia and Vasquez-Reina, Luis and Gonzalez-Agirre, Aitor. Cognitive Biases, Task Complexity, and Result Interpretability in Large Language Models. Proceedings of the 31st International Conference on Computational Li...

    2025

  55. [63]

    Benchmarking Cognitive Biases in Large Language Models as Evaluators

    Koo, Ryan and Lee, Minhwa and Raheja, Vipul and Park, Jong Inn and Kim, Zae Myung and Kang, Dongyeop. Benchmarking Cognitive Biases in Large Language Models as Evaluators. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.29

    2024 doi

  56. [64]

    Findings of the association for computational linguistics: EMNLP 2024 , pages=

    Cognitive bias in decision-making with LLMs , author=. Findings of the association for computational linguistics: EMNLP 2024 , pages=

    2024

  57. [65]

    Royal Society Open Science , volume=

    (Ir) rationality and cognitive biases in large language models , author=. Royal Society Open Science , volume=. 2024 , publisher=

    2024

  58. [66]

    Advances in Neural Information Processing Systems , volume=

    Capturing failures of large language models via human cognitive biases , author=. Advances in Neural Information Processing Systems , volume=

  59. [67]

    arXiv preprint arXiv:2408.02784 , year=

    Llm economicus? mapping the behavioral biases of llms via utility theory , author=. arXiv preprint arXiv:2408.02784 , year=

    arXiv

  60. [68]

    Computational Linguistics , volume=

    Bias and fairness in large language models: A survey , author=. Computational Linguistics , volume=. 2024 , publisher=

    2024

  61. [69]

    Toxicity in chatgpt: Analyzing persona-assigned language models

    Deshpande, Ameet and Murahari, Vishvak and Rajpurohit, Tanmay and Kalyan, Ashwin and Narasimhan, Karthik. Toxicity in chatgpt: Analyzing persona-assigned language models. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings...

    2023 doi

  62. [70]

    Realistic Evaluation of Toxicity in Large Language Models

    Luong, Tinh and Le, Thanh-Thien and Ngo, Linh and Nguyen, Thien. Realistic Evaluation of Toxicity in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.61

    2024 doi

  63. [71]

    Unveiling the Implicit Toxicity in Large Language Models

    Wen, Jiaxin and Ke, Pei and Sun, Hao and Zhang, Zhexin and Li, Chengfei and Bai, Jinfeng and Huang, Minlie. Unveiling the Implicit Toxicity in Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2...

    2023 doi

  64. [72]

    Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?

    An, Haozhe and Acquaye, Christabel and Wang, Colin and Li, Zongxia and Rudinger, Rachel. Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024 doi

  65. [73]

    Examining Gender and Racial Bias in Large Vision -- Language Models Using a Novel Dataset of Parallel Images

    Fraser, Kathleen and Kiritchenko, Svetlana. Examining Gender and Racial Bias in Large Vision -- Language Models Using a Novel Dataset of Parallel Images. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long...

    2024 doi

  66. [74]

    R eal T oxicity P rompts: Evaluating Neural Toxic Degeneration in Language Models

    Gehman, Samuel and Gururangan, Suchin and Sap, Maarten and Choi, Yejin and Smith, Noah A. R eal T oxicity P rompts: Evaluating Neural Toxic Degeneration in Language Models. Findings of the Association for Computational Linguistics: EMNLP 2020. 2020. doi:10.18653/v1/2020.findin...

    2020 doi

  67. [75]

    Synthese , volume=

    Algorithmic bias: on the implicit biases of social technology , author=. Synthese , volume=. 2021 , publisher=

    2021

  68. [76]

    2013 , publisher=

    Statistical power analysis for the behavioral sciences , author=. 2013 , publisher=

    2013

  69. [77]

    and Chen, S

    Makowski, Dominique and Ben-Shachar, Mattan S. and Chen, S. H. Annabel and Lüdecke, Daniel , TITLE=. Frontiers in Psychology , VOLUME=. 2019 , URL=. doi:10.3389/fpsyg.2019.02767 , ISSN=

    2019

  70. [78]

    and Lüdecke, Daniel , title =

    Makowski, Dominique and Ben-Shachar, Mattan S. and Lüdecke, Daniel , title =. 2019 , publisher =. doi:10.21105/joss.01541 , url =

    2019 doi

  71. [79]

    The 2025 Conference on Empirical Methods in Natural Language Processing , year=

    A Monte-Carlo Sampling Framework For Reliable Evaluation of Large Language Models Using Behavioral Analysis , author=. The 2025 Conference on Empirical Methods in Natural Language Processing , year=

    2025

  72. [80]

    2000 , publisher=

    Stated choice methods: analysis and applications , author=. 2000 , publisher=

    2000

  73. [81]

    Nature , volume=

    Detecting hallucinations in large language models using semantic entropy , author=. Nature , volume=. 2024 , publisher=

    2024

  74. [82]

    Journal of marketing research , volume=

    Consumer ethnocentrism: Construction and validation of the CETSCALE , author=. Journal of marketing research , volume=. 1987 , publisher=

    1987

  75. [83]

    Educational and psychological measurement , volume=

    Further evidence on response sets and test design , author=. Educational and psychological measurement , volume=. 1950 , publisher=

    1950

  76. [84]

    International Journal of Approximate Reasoning , volume=

    Consensus and dissention: A measure of ordinal dispersion , author=. International Journal of Approximate Reasoning , volume=. 2007 , publisher=

    2007

  77. [85]

    Environmental and Resource Economics , volume=

    A joint latent-class model: combining likert-scale preference statements with choice data to harvest preference heterogeneity , author=. Environmental and Resource Economics , volume=. 2011 , publisher=

    2011

  78. [86]

    Journal of rural social sciences , volume=

    Another look at Likert scales , author=. Journal of rural social sciences , volume=

  79. [87]

    , author=

    A Web-Based Survey to Assess Prospective Elementary School Teachers' Beliefs about Mathematics and Mathematics Learning: An Alternative to Likert Scales. , author=. International Group for the Psychology of Mathematics Education , volume=. 2003 , publisher=

    2003

  80. [88]

    British journal of applied science & technology , volume=

    Likert scale: Explored and explained , author=. British journal of applied science & technology , volume=. 2015 , publisher=

    2015

  81. [89]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  82. [90]

    Publications Manual , year = "1983", publisher =

    1983

  83. [91]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  84. [92]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  85. [93]

    Dan Gusfield , title =. 1997

    1997

  86. [94]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  87. [95]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  88. [96]

    Scientific Reports , year=

    A meta-analysis of the persuasive power of large language models , author=. Scientific Reports , year=

  89. [97]

    Artificial Intelligence Review , volume=

    Comparative analysis of methodologies and approaches in recommender systems utilizing large language models , author=. Artificial Intelligence Review , volume=. 2025 , publisher=

    2025

  90. [98]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  91. [99]

    PNAS nexus , volume=

    Cultural bias and cultural alignment of large language models , author=. PNAS nexus , volume=. 2024 , publisher=

    2024

  92. [100]

    Frontiers in Political Science , volume=

    Item response theory approach to ethnocentrism , author=. Frontiers in Political Science , volume=. 2023 , publisher=

    2023

  93. [101]

    Behavior Research Methods , volume=

    MacBehaviour: An R package for behavioural experimentation on large language models , author=. Behavior Research Methods , volume=. 2024 , publisher=

    2024

  94. [102]

    New Ideas in Psychology , volume=

    Large language models (LLMs) as research Subjects: Status, opportunities and challenges , author=. New Ideas in Psychology , volume=. 2025 , publisher=

    2025

  95. [103]

    Artificial Intelligence , volume=

    Exploring the psychology of LLMs’ moral and legal reasoning , author=. Artificial Intelligence , volume=. 2024 , publisher=

    2024

  96. [104]

    arXiv preprint arXiv:2411.02083 , year=

    Regress, Don't Guess--A Regression-like Loss on Number Tokens for Language Models , author=. arXiv preprint arXiv:2411.02083 , year=

    arXiv

  97. [105]

    arXiv preprint arXiv:2402.14903 , year=

    Tokenization counts: the impact of tokenization on arithmetic in frontier llms , author=. arXiv preprint arXiv:2402.14903 , year=

    arXiv

  98. [106]

    Journal of the academy of marketing science , volume=

    Domestic country bias, country-of-origin effects, and consumer ethnocentrism: a multidimensional unfolding approach , author=. Journal of the academy of marketing science , volume=. 2004 , publisher=

    2004

  99. [107]

    International marketing review , volume=

    Consumer ethnocentrism: an integrative review of its antecedents and consequences , author=. International marketing review , volume=. 2006 , publisher=

    2006

  100. [108]

    From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models

    Bhatia, Mehar and Ravi, Sahithya and Chinchure, Aditya and Hwang, EunJeong and Shwartz, Vered. From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    2024 doi

  101. [109]

    GD - COMET : A Geo-Diverse Commonsense Inference Model

    Bhatia, Mehar and Shwartz, Vered. GD - COMET : A Geo-Diverse Commonsense Inference Model. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.496

    2023 doi

  102. [110]

    Good Night at 4 pm?! Time Expressions in Different Cultures

    Shwartz, Vered. Good Night at 4 pm?! Time Expressions in Different Cultures. Findings of the Association for Computational Linguistics: ACL 2022. 2022. doi:10.18653/v1/2022.findings-acl.224

    2022 doi

  103. [111]

    2025 , eprint=

    Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4 , author=. 2025 , eprint=

    2025

  104. [112]

    arXiv preprint arXiv:2406.03930 , year=

    Culturally aware and adapted nlp: A taxonomy and a survey of the state of the art , author=. arXiv preprint arXiv:2406.03930 , year=

    arXiv

  105. [113]

    Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , pages=

    The social embeddedness of peer production: A comparative qualitative analysis of three Indian language Wikipedia editions , author=. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , pages=

    2022

  106. [114]

    arXiv preprint arXiv:2502.12057 , year=

    Culture is not trivia: Sociocultural theory for cultural nlp , author=. arXiv preprint arXiv:2502.12057 , year=

    arXiv

  107. [115]

    Language Sciences , volume=

    Methodological nationalism in linguistics , author=. Language Sciences , volume=. 2019 , publisher=

    2019

  108. [116]

    standard

    Monoglot “standard” in America: Standardization and metaphors of linguistic hegemony , author=. The matrix of language , pages=. 2018 , publisher=

    2018

  109. [117]

    Science , volume=

    Semantics derived automatically from language corpora contain human-like biases , author=. Science , volume=. 2017 , publisher=

    2017

  110. [118]

    Nature , volume=

    AI generates covertly racist decisions about people based on their dialect , author=. Nature , volume=. 2024 , publisher=

    2024

  111. [119]

    American Psychologist , volume=

    Multicultural minds: A dynamic constructivist approach to culture and cognition , author=. American Psychologist , volume=. 2000 , publisher=

    2000

  112. [120]

    Psychological Review , volume=

    Culture and systems of thought: Holistic versus analytic cognition , author=. Psychological Review , volume=. 2001 , publisher=

    2001

  113. [121]

    Psychological Bulletin , volume=

    Does culture influence what and how we think? Effects of priming individualism and collectivism , author=. Psychological Bulletin , volume=. 2008 , publisher=

    2008

  114. [122]

    Frontiers in Psychology , volume=

    Self-construal priming selectively modulates the scope of visual attention , author=. Frontiers in Psychology , volume=. 2015 , publisher=

    2015

  115. [123]

    Global Storytelling: Journal of Digital and Moving Images , volume=

    From Diaosi to Sang to Tangping: The Chinese DST Youth Subculture Online , author=. Global Storytelling: Journal of Digital and Moving Images , volume=. 2023 , publisher=. doi:10.3998/gs.5304 , url=

    2023 doi

  116. [124]

    China Perspectives , number =

    Xiaojing Zheng and Zitong Qiu , title =. China Perspectives , number =. 2023 , publisher =. doi:10.4000/chinaperspectives.15869 , url =

    2023 doi

  117. [125]

    2023 , url =

    Koh Ewe , title =. 2023 , url =

    2023

  118. [126]

    Continuum: Journal of Media & Cultural Studies , year =

    Su, Wendy , title =. Continuum: Journal of Media & Cultural Studies , year =. doi:10.1080/10304312.2023.2190059 , url =

    2023

  119. [127]

    Journal of Business Ethics , volume=

    Entering guanxi: A business ethical dilemma in Mainland China? , author=. Journal of Business Ethics , volume=. 2001 , publisher=

    2001

  120. [128]

    European Journal of Marketing , volume=

    A model of close business relationships in China (guanxi) , author=. European Journal of Marketing , volume=. 2001 , publisher=

    2001

  121. [129]

    arXiv preprint arXiv:2203.07785 , year=

    The Ghost in the Machine has an American accent: value conflict in GPT-3 , author=. arXiv preprint arXiv:2203.07785 , year=

    arXiv

  122. [130]

    First Conference on Language Modeling , year=

    Towards Measuring the Representation of Subjective Global Opinions in Language Models , author=. First Conference on Language Modeling , year=

  123. [131]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Whose Opinions Do Language Models Reflect? , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  124. [132]

    PNAS Nexus , volume =

    Tao, Yan and Viberg, Olga and Baker, Ryan S and Kizilcec, René F , title =. PNAS Nexus , volume =. 2024 , month =. doi:10.1093/pnasnexus/pgae346 , url =

    2024 doi

  125. [133]

    1984 , publisher=

    Culture's consequences: International differences in work-related values , author=. 1984 , publisher=

    1984

  126. [134]

    Wiley international encyclopedia of marketing , year=

    Social desirability bias , author=. Wiley international encyclopedia of marketing , year=

  127. [135]

    (No Title) , year=

    World values survey wave 7 (2017-2020) cross-national data-set , author=. (No Title) , year=

    2017

  128. [136]

    arXiv preprint arXiv:2303.17466 , year=

    Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study , author=. arXiv preprint arXiv:2303.17466 , year=

    arXiv

  129. [137]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    arXiv

  130. [138]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  131. [139]

    arXiv preprint arXiv:2306.03423 , year=

    I'm Afraid I Can't Do That: Predicting Prompt Refusal in Black-Box Generative Language Models , author=. arXiv preprint arXiv:2306.03423 , year=

    arXiv

  132. [140]

    Sociological Methods & Research , year =

    Julien Boelaert and Samuel Coavoux and Étienne Ollion and Ivaylo Petev and Patrick Präg , title =. Sociological Methods & Research , year =. doi:10.1177/00491241251330582 , URL =

    doi

  133. [141]

    arXiv preprint arXiv:2203.05794 , year=

    BERTopic: Neural topic modeling with a class-based TF-IDF procedure , author=. arXiv preprint arXiv:2203.05794 , year=

    arXiv

  134. [142]

    Behavioral and Brain Sciences , volume=

    The weirdest people in the world? , author=. Behavioral and Brain Sciences , volume=. 2010 , publisher=

    2010

  135. [143]

    1978 , publisher=

    Orientalism , author=. 1978 , publisher=

    1978

  136. [144]

    Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT) , pages=

    On the dangers of stochastic parrots: Can language models be too big? , author=. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT) , pages=

    2021

  137. [145]

    arXiv preprint arXiv:2309.08591 , year=

    Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings , author=. arXiv preprint arXiv:2309.08591 , year=

    arXiv

  138. [146]

    Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency , pages=

    Fairness and abstraction in sociotechnical systems , author=. Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency , pages=. 2019 , publisher=

    2019

  139. [147]

    arXiv preprint arXiv:2203.13722 , year=

    Probing Pre-Trained Language Models for Cross-Cultural Differences in Values , author=. arXiv preprint arXiv:2203.13722 , year=

    arXiv

  140. [148]

    arXiv preprint arXiv:2301.01768 , year=

    The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation , author=. arXiv preprint arXiv:2301.01768 , year=

    arXiv

  141. [149]

    Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022 , pages =

    HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models , author =. Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022 , pages =. 2022 , publisher =. doi:10.18653/v1/2022.findings-aacl.27 , url =

    2022 doi

  142. [150]

    It's a Funny Thing, Humour , publisher =

    Context and Ethnic Humour in Intergroup Relations , editor =. It's a Funny Thing, Humour , publisher =. 1977 , isbn =. doi:https://doi.org/10.1016/B978-0-08-021376-7.50050-4 , url =

    1977 doi

  143. [151]

    Borgella and Melanie S

    Michael Thai and Alex M. Borgella and Melanie S. Sanchez , keywords =. It's only funny if we say it: Disparagement humor is better received if it originates from a member of the group being disparaged , journal =. 2019 , issn =. doi:https://doi.org/10.1016/j.jesp.2019.103838 , url =

    2019

  144. [152]

    Advances in Neural Information Processing Systems , volume=

    Mitigating test-time bias for fair image retrieval , author=. Advances in Neural Information Processing Systems , volume=

  145. [153]

    2007 , publisher=

    Lies that bind: Chinese truth, other truths , author=. 2007 , publisher=

    2007

  146. [154]

    The small and big deceptions: In psychology and evolutionary sciences perspective , pages=

    How do others deceive? Cultural aspects of lying and cheating , author=. The small and big deceptions: In psychology and evolutionary sciences perspective , pages=. 2015 , publisher=

    2015

  147. [155]

    Society and Culture in South Asia , volume=

    Interfaith marriages and negotiated spaces , author=. Society and Culture in South Asia , volume=. 2018 , publisher=

    2018

  148. [156]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =

    Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense , author =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) ,...

    2024

  149. [157]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    Culturally Aware Natural Language Inference , author =. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =. 2023 , organization =

    2023

  150. [158]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages =

    Evaluating Cultural and Social Awareness of LLM Web Agents , author =. Findings of the Association for Computational Linguistics: NAACL 2025 , pages =. 2025 , address =

    2025

  151. [159]

    2024 , url =

    OpenAI , title =. 2024 , url =

    2024

  152. [160]

    Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =

    Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness , author =. Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =. 2025 , address =

    2025

  153. [161]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Having Beer after Prayer? Measuring Cultural Bias in Large Language Models , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , address =

    2024

  154. [162]

    Sociological Methods & Research , pages=

    Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization , author=. Sociological Methods & Research , pages=. 2025 , publisher=

    2025

  155. [163]

    2021 , publisher=

    The digital frontier: Infrastructures of control on the global web , author=. 2021 , publisher=

    2021

  156. [164]

    and China , author =

    Political biases and inconsistencies in bilingual GPT models—the cases of the U.S. and China , author =. Scientific Reports , volume =. 2024 , publisher =. doi:10.1038/s41598-024-76395-w , url =

    2024 doi

  157. [165]

    arXiv preprint arXiv:2501.16748 , year=

    Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions , author=. arXiv preprint arXiv:2501.16748 , year=. doi:10.48550/arXiv.2501.16748 , archivePrefix=

    doi arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.