Pith. sign in

REVIEW 4 major objections 5 minor 51 references

Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large language models consistently favor critically acclaimed films over commercially successful ones, a preference that grows with model scale.

desk verdict A well-executed study with a novel evaluative construct, but the headline B-vs-C preference is confounded with era and language, and the authors never test whether the effect survives within strata. read the letter →

arxiv 2608.06955 v1 pith:PIB5WW3D submitted 2026-08-07 cs.AI cs.CY

Jonghyun Jee , Aaron Shaw This is my paper · ORCID
classification cs.AIcs.CY
keywords largelanguagemodelscriticalacclaimfilmpreferenceculturalbiaspairedcomparisonBradley-Terrymodelevaluativehierarchyscale
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models reproduce the evaluative hierarchies of human culture rather than simply mirroring popularity. Using 160,000 forced-choice film comparisons across eight models from four families, it finds that critically acclaimed but commercially obscure films beat commercially successful but critically unrecognized films in 65.6% to 87.8% of direct matchups, with all differences statistically significant. The pattern strengthens with model size within every family. Nested regressions show that the preference is not explained by public visibility or popular reception; after controlling for those, critical acclaim carries its own positive signal. The paper concludes that LLMs exhibit a 'critical acclaim orientation' as a stable, cross-model behavioral regularity.

What carries the argument

The central machinery is the Bradley–Terry model, a paired-comparison model that assigns each film a latent strength parameter such that the probability one film is preferred over another is proportional to its strength; log-strengths are estimated via an MM algorithm. The 200-film benchmark is partitioned into three sets: dual-legitimacy (in both critical and commercial corpora), critical-only, and commercial-only. Nested OLS regressions then separate three signals — corpus set membership, era, public visibility (log IMDb votes, with log Wikipedia revisions as a robustness check), and popular reception (IMDb ratings) — to show that critical acclaim contributes to preference independently of visibility and reception.

What would settle it

Re-estimate the nested regressions using actual token or document frequencies of each film title from a model's public training corpus (rather than IMDb votes or Wikipedia revisions); if the Set B coefficient no longer reverses sign, the claimed critical acclaim signal is an artifact of the proxy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLM outputs, when forced to choose between two films, systematically favor critically consecrated works over commercially successful ones. This critical acclaim orientation holds for all eight tested models: Set B (critically acclaimed, commercially obscure) films win against Set C (commercially successful, critically unrecognized) films at rates between 65.6% and 87.8%, all p < .001. The effect grows with model scale in each family. In nested OLS regressions, the coefficient for Set B reverses from negative to positive once proxies for public visibility (IMDb vote counts) are added, indicating that critical acclaim is associated with higher preference strength net of visibility; adding popular reception (IMDb ratings) attenuates but does not eliminate the commercial-only penalty. The authors interpret this as evidence that LLMs encode the valence of critical discourse, not merely its volume.

Load-bearing premise

The load-bearing premise is that IMDb vote counts and Wikipedia revision counts validly measure how much exposure each film actually receives in the models' training corpora; if those proxies are biased, the regression sign reversal could be measurement error rather than evidence of a distinct critical acclaim signal.

Editorial extensions

If this is right

  • LLM-based recommender systems and cultural discovery tools may systematically steer users toward critically consecrated works even when users seek popular entertainment.
  • Within any model family, larger models show a stronger critical acclaim orientation, so capability scaling alone does not neutralize the bias; it amplifies it.
  • Evaluative and recommendation-oriented prompts produce divergent rankings, so the orientation is context-dependent and may surface in subtle, unprompted ways in real deployments.
  • Since the effect is not reducible to visibility, auditing cultural bias in LLMs requires measuring evaluative valence separately from mere exposure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would compile actual token counts of each film title in a model's public training corpus and re-run the regressions; if the critical acclaim signal vanishes, the effect is an artifact of proxy measurement rather than a distinct cultural orientation.
  • If the orientation reflects critical discourse in pretraining, similar patterns should appear for music, literature, and visual art, and should be detectable with analogous benchmark partitions.
  • The sign reversal on Set B hints at a possible 'commercial discount' — that models may treat commercial success as slightly negative once critical recognition is controlled; the paper flags this as speculative, and it could be tested with matched pairwise designs where only box-office status varies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a study of eight LLMs asked to choose between pairs of films from a 200-film benchmark partitioned into Set A (critical acclaim + commercial success), Set B (critical acclaim only), and Set C (commercial success only). Using 20,000 pairwise comparisons per model and Bradley-Terry aggregation, the authors find that Set B films are preferred over Set C films in all models (65.6–87.8% win rates), that this preference grows with model scale within each family, and that nested OLS regressions on the pooled log-strengths show a sign reversal for Set B once IMDb vote counts and ratings are added. They interpret this as "critical acclaim orientation" distinct from public visibility and popular reception, with a follow-up prompt-framing experiment showing divergence between evaluative and recommendation-oriented prompts.

Significance. The study is methodologically ambitious and addresses a genuinely novel question about whether LLMs reproduce prestige hierarchies rather than popularity signals. Strengths include the pre-registered validation criteria (H1a–H1c), the transparent release of code and data, the use of Bradley-Terry estimation with structural-coherence checks, and the consistency of the cross-model pattern. If the identification concerns below are resolved, the paper would be a valuable contribution to the cultural-bias literature and to debates about evaluative hierarchies in foundation models.

major comments (4)
  1. [Methods: Dataset / Table 1] Set B and Set C are nearly perfectly confounded with era and language. The paper states that 46/80 Set B films are pre-1980 and only 12 are English-language, while Set C has 61/80 post-2000 films and 69 English-language films. The headline H2 win rates (Table 1) are unadjusted; no within-era or within-language B-vs-C win rates are reported, and the regressions (M2–M4) control for era but not language. A model preference for older or non-English films would produce the same overall pattern without any distinct critical-acclaim signal. The authors should report B-vs-C win rates within the era and language strata (and ideally within the cross-strata cells), or otherwise demonstrate that the raw preference is not an artifact of these compositional differences. As written, the abstract's central claim is not identified.
  2. [Regression analysis / Table 2] The sign-reversal result that carries the paper's interpretive weight (M3/M4, Set B coefficient b=+0.638 and +0.554) depends on log IMDb votes and Wikipedia revision counts as proxies for training-corpus visibility. These are acknowledged as noisy (Appendix D), and measurement error in a control variable can induce bias in the coefficient of interest, particularly when the proxy undercounts obscure foreign films or overcounts franchises. In addition, the pooled OLS uses robust (HC3) but not clustered standard errors; since observations are film×model pairs, clustering by film is needed to avoid overstating precision. The authors should present cluster-robust standard errors (or a multi-level model) and a sensitivity analysis of the proxy assumption, for example using alternative visibility measures or bounding the measurement error.
  3. [Comparison Design] The adaptive three-phase sampling design (Phase 1 broad coverage, Phase 2 competitive pairing, Phase 3 upper-quartile concentration) means the B-vs-C win rate is computed over a non-random sample of film pairs, with an endogenous exposure effect: films that perform well early accrue more comparisons. The stability checks (H1b) show high inter-run Spearman correlations, but ranking stability does not imply that the marginal win rate is unbiased for a fixed comparison schedule. The paper should report the number (and proportion) of B-C pairs contributed by each phase and ideally a reweighted or phase-stratified estimate of the H2 win rate to show that the headline result is not an artifact of the adaptive allocation.
  4. [H4: Scale-Dependent Critical Acclaim Orientation] The scale claim rests on a comparison of two models per family (n=4 families), with no inferential test reported for the within-family deltas. The paper reports Δwin rates of +7.1 to +17.5 percentage points but no confidence intervals or p-values for these differences. Given the small number of families and the descriptive nature of the comparison, the claim that "the effect intensifies with model scale within each family" is not statistically supported; the authors should either provide a formal test across families (e.g., a mixed model or sign test) or moderate the claim accordingly.
minor comments (5)
  1. [Abstract] The abstract says "across 20,000 pairwise forced-choice comparisons per model," which is accurate for the main design, but the H1c prompt-frame invariance analyses use only four large-tier models at 4,000 comparisons per wording; consider clarifying that the 20,000 figure refers to the main elicitation procedure to avoid misleading the reader.
  2. [Methods: Dataset] Footnote markers for the TSPDT and BOM URLs are formatted as superscript '1' and '2' inline; they should be actual footnotes or links, otherwise the reference is ambiguous.
  3. [Methods: Elicitation Procedure / Equation (2)] Equation (2) uses 'w+i' for the regularized win total, but the exact value of the Dirichlet pseudo-count prior is not given; please specify the hyperparameter used.
  4. [Table 1 caption] The caption says "Unmarked win rates are non-significant (n.s.)" but the table marks all H2 results as significant; consider simplifying the note to avoid confusion about which entries are unmarked.
  5. [Analysis (H2–H4)] The term "critical acclaim orientation" is used throughout without a precise operational definition; please state explicitly in the analysis section that it is operationalized as the Set B vs. Set C win rate in the primary analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central measurements and regressions are self-contained empirical analyses, and no prediction reduces to a fitted input or self-citation.

full rationale

The paper is an empirical elicitation study rather than a derivation from first principles. The central B-versus-C win rates are raw observed proportions from forced-choice comparisons, tested against a 0.50 null with binomial tests, so the headline finding does not reduce to any fitted parameter. The Bradley-Terry strengths are estimated from the comparisons and then used as an outcome in nested OLS regressions with external film attributes (set membership, era, IMDb vote counts, IMDb ratings) as predictors; the sign reversal of the Set B coefficient is an interpretation of those regressions, not a quantity constructed from the predictors. The only predictive validation, H1d, uses five-fold cross-validation on held-out pairs, which is genuine out-of-sample prediction. The paper contains no load-bearing self-citations: the methodological references (Bradley-Terry, the MM algorithm, prompt-instability results) are external and do not supply the cultural claim. The skeptical concern about era and language confounding is a validity threat about whether Set B versus Set C differences are attributable to critical acclaim or to correlated covariates, but that does not make the test statistics circular. Similarly, the acknowledged noisiness of the IMDb and Wikipedia proxies is an assumption about measurement validity, not a reduction of the conclusion to its own inputs. No circular step meets the standard of quotable, equation-level equivalence, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on measurement design choices rather than fitted theoretical parameters. The main free parameter is a regularization constant; the key assumptions are the validity of external indices (TSPDT, BOM, IMDb) and the representativeness of the forced-choice protocol.

free parameters (1)
  • Dirichlet pseudo-count prior = 1/2
    Regularization constant in the MM algorithm for Bradley-Terry estimation to prevent divergence for undefeated or winless items; small enough to have minor impact.
assumptions (4)
  • domain assumption TSPDT Top 1,000 and Box Office Mojo rankings index critical acclaim and commercial success respectively.
    Used to partition the 200-film benchmark into Sets A, B, C; if these sources misclassify films, the set comparisons are affected.
  • domain assumption IMDb user ratings and vote counts proxy popular reception and public visibility in training corpora.
    Used as controls in nested OLS models M3-M4; validity is partially acknowledged as imperfect.
  • domain assumption Pairwise forced-choice responses at temperature 0 reflect stable latent preferences that Bradley-Terry can aggregate.
    Supported by H1a-H1d validation, but ultimately an assumption about what the outputs mean.
  • domain assumption Pooled OLS with robust standard errors is appropriate for the clustered film-by-model data.
    The authors rely on z-scoring to reduce model-level clustering, but film-level clustering (8 observations per film) is not addressed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation." pith.science (2026). https://pith.science/paper/PIB5WW3D

@misc{pith2026260806955,
  author       = {Pith},
  title        = {Pith review of: Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PIB5WW3D}},
  note         = {Machine review of arXiv:2608.06955}
}
read the original abstract

Large language models (LLMs) are trained on corpora that contain expressions of human judgment about films, books, music, and more. Yet whether LLMs systematically reproduce evaluative hierarchies remains unclear. Prior research on cultural bias in LLMs suggests competing expectations: models may mirror the popularity signals of internet texts, or may reproduce forms of prestige embedded in critical discourse. We probe this question through a study of film evaluations with eight models from four families (Anthropic, OpenAI, Alibaba, and Mistral), using a 200-film benchmark partitioned into critically acclaimed, commercially successful, and dual-legitimacy (critical acclaim + commercial success) films. Across 20,000 pairwise forced-choice comparisons per model analyzed with Bradley--Terry estimation, we observe a consistent critical acclaim orientation with all models: critically acclaimed yet commercially obscure films are selected over commercially successful yet critically unrecognized ones. This pattern grows with model scale within each family. In addition, nested OLS regression analyses show that evaluative orientation, public visibility, and popular reception distinctly help explain preferences. Adjusting for public visibility reverses the models' preference for dual-legitimacy films over critical acclaim-only films, while additionally accounting for popular reception attenuates much of the disadvantage of films with commercial success only. Finally, evaluative and recommendation-oriented prompt framings produce divergent rankings, suggesting that critical acclaim orientation may manifest indirectly in real-world LLM deployments.

Figures

Figures reproduced from arXiv: 2608.06955 by the authors.

Figure 1
Figure 1. λz distributions by corpus set and model (H2, H3, H4). Each panel shows the full distribution of standardized prefer￾ence strength for Sets A, B, and C. For each model-set pair, darker bars show the interquartile range and median values, while lighter shaded area visualizes the density. The per-set median is presented along the bottom of each panel. The consistent B > C separation in all panels supports H2; the mixe… view at source ↗
Figure 2
Figure 2. Fixed-effect coefficients from nested OLS models M1–M4 (outcome: standardized [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 22 canonical work pages

  1. [1]

    and Trager, Jackson and Park, Peter S

    Abdurahman, Suhaib and Atari, Mohammad and Karimi-Malekabadi, Farzan and Xue, Mona J. and Trager, Jackson and Park, Peter S. and Golazizian, Preni and Omrani, Ali and Dehghani, Morteza , year=. Perils and opportunities in using large language models in psychological research , volume=. PNAS Nexus , publisher=. doi:10.1093/pnasnexus/pgae245 , number=

    doi:10.1093/pnasnexus/pgae245

  2. [2]

    Machine Habitus: Toward a Sociology of Algorithms , publisher=

    Airoldi, Massimo , year=. Machine Habitus: Toward a Sociology of Algorithms , publisher=

    work page

  3. [3]

    The nested relationality of perceived legitimacy: Mapping taste hierarchies with granular digital traces , volume=

    Airoldi, Massimo , year=. The nested relationality of perceived legitimacy: Mapping taste hierarchies with granular digital traces , volume=. doi:10.1016/j.poetic.2024.101864 , journal=

    arXiv 2024

  4. [4]

    Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT

    Ali, Muhammad and Panda, Swetasudha and Shen, Qinlan and Wick, Michael and Kobren, Ari , year=. Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT , url=. 2407.21058 , archivePrefix=

    work page Pith review arXiv

  5. [5]

    Revealing Potential Biases in LLM-Based Recommender Systems in the Cold Start Setting , url=

    Andre, Alexandre and Roy, Gauthier and Dyer, Eva and Wang, Kai , year=. Revealing Potential Biases in LLM-Based Recommender Systems in the Cold Start Setting , url=. doi:10.48550/arXiv.2508.20401 , note=. 2508.20401 , archivePrefix=

    arXiv doi:10.48550/arxiv.2508.20401

  6. [6]

    Understanding Intrinsic Socioeconomic Biases in Large Language Models , volume=

    Arzaghi, Mina and Carichon, Florian and Farnadi, Golnoosh , year=. Understanding Intrinsic Socioeconomic Biases in Large Language Models , volume=. doi:10.1609/aies.v7i1.31616 , journal=

    work page doi:10.1609/aies.v7i1.31616

  7. [7]

    doi:10.31234/osf.io/5b26t , eprint=

    Which Humans? , url=. doi:10.31234/osf.io/5b26t , eprint=

    doi:10.31234/osf.io/5b26t

  8. [8]

    Bai, Xuechunzi and Wang, Angelina and Sucholutsky, Ilia and Griffiths, Thomas L. , year=. Explicitly unbiased large language models still form biased associations , volume=. Proceedings of the National Academy of Sciences , publisher=. doi:10.1073/pnas.2416228122 , number=

    doi:10.1073/pnas.2416228122

Show all 51 references
  1. [9]

    Culture, Class, Distinction , url=

    Bennett, Tony and Savage, Mike and Silva, Elizabeth Bortolaia and Warde, Alan and Gayo-Cal, Modesto and Wright, David , year=. Culture, Class, Distinction , url=. doi:10.4324/9780203930571 , publisher=

    doi

  2. [10]

    Language (Technology) is Power: A Critical Survey of "Bias" in NLP , url=

    Blodgett, Su Lin and Barocas, Solon and Daum. Language (Technology) is Power: A Critical Survey of "Bias" in NLP , url=. 2020 , pages=. doi:10.18653/v1/2020.acl-main.485 , booktitle=

    2020 doi

  3. [11]

    Trustworthy Social Bias Measurement , volume=

    Bommasani, Rishi and Liang, Percy , year=. Trustworthy Social Bias Measurement , volume=. doi:10.1609/aies.v7i1.31630 , booktitle=

    doi

  4. [12]

    Distinction: A Social Critique of the Judgement of Taste , publisher=

    Bourdieu, Pierre , year=. Distinction: A Social Critique of the Judgement of Taste , publisher=

  5. [13]

    Bradley, Ralph Allan and Terry, Milton E. , year=. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , volume=. Biometrika , publisher=. doi:10.2307/2334029 , number=

    doi

  6. [14]

    Language Models are Few-Shot Learners , volume=

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom a...

  7. [15]

    Anything But Heavy Metal

    Bryson, Bethany , year=. "Anything But Heavy Metal": Symbolic Exclusion and Musical Dislikes , volume=. American Sociological Review , publisher=. doi:10.2307/2096459 , number=

    doi

  8. [16]

    The Thirteenth International Conference on Learning Representations , author=

    DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life , url=. The Thirteenth International Conference on Learning Representations , author=

  9. [17]

    Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons , ISSN=

    DiGiuseppe, Matthew R and Flynn, Michael E , year=. Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons , ISSN=. doi:10.1093/poq/nfag013 , journal=

    doi

  10. [18]

    Theory and Society , author=

    Social class and arts consumption , volume=. Theory and Society , author=. 1978 , pages=. doi:10.1007/BF01702159 , number=

    1978 doi

  11. [19]

    A Scoping Review of Gender Stereotypes in Artificial Intelligence , url=

    Duan, Wen and Li, Lingyuan and Freeman, Guo and McNeese, Nathan , year=. A Scoping Review of Gender Stereotypes in Artificial Intelligence , url=. doi:10.1145/3706598.3713093 , booktitle=

  12. [20]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling , url=

    Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor , year=. The Pile: An 800GB Dataset of Diverse Text for Language Mo...

    arXiv

  13. [21]

    Generative AI and the politics of visibility , volume=

    Gillespie, Tarleton , year=. Generative AI and the politics of visibility , volume=. Big Data & Society , publisher=. doi:10.1177/20539517241252131 , number=

    doi

  14. [22]

    Behavioral and Brain Sciences , author=

    The weirdest people in the world? , volume=. Behavioral and Brain Sciences , author=. 2010 , pages=. doi:10.1017/S0140525X0999152X , number=

    2010 doi

  15. [23]

    Hunter, David R. , year=. MM algorithms for generalized Bradley-Terry models , volume=. The Annals of Statistics , publisher=. doi:10.1214/aos/1079120141 , number=

  16. [24]

    Linguistic Markers of Status in Food Culture: Bourdieu's Distinction in a Menu Corpus , volume=

    Jurafsky, Dan and Chahuneau, Victor and Routledge, Bryan and Smith, Noah , year=. Linguistic Markers of Status in Food Culture: Bourdieu's Distinction in a Menu Corpus , volume=. Journal of Cultural Analytics , publisher=. doi:10.22148/16.007 , number=

    doi

  17. [25]

    Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs , url=

    Khan, Ariba and Casper, Stephen and Hadfield-Menell, Dylan , year=. Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs , url=. doi:10.1145/3715275.3732147 , booktitle=

  18. [26]

    and Taddy, Matt and Evans, James A

    Kozlowski, Austin C. and Taddy, Matt and Evans, James A. , year=. The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings , volume=. American Sociological Review , publisher=. doi:10.1177/0003122419877135 , number=

    doi

  19. [27]

    Political Analysis , author=

    Mapping (A)Ideology: A Taxonomy of European Parties Using Generative LLMs as Zero-Shot Learners , volume=. Political Analysis , author=. 2025 , pages=. doi:10.1017/pan.2025.7 , number=

    2025 doi

  20. [28]

    and Song, Lesley Luyang and Zhang, Lu Doris , year=

    Lu, Jackson G. and Song, Lesley Luyang and Zhang, Lu Doris , year=. Cultural tendencies in generative AI , volume=. Nature Human Behaviour , publisher=. doi:10.1038/s41562-025-02242-1 , number=

    doi

  21. [29]

    and Plank, Barbara and Kreuter, Frauke , editor=

    Ma, Bolei and Wang, Xinpeng and Hu, Tiancheng and Haensch, Anna-Carolina and Hedderich, Michael A. and Plank, Barbara and Kreuter, Frauke , editor=. The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models , url=. 2024 , pages=. doi:1...

    2024 doi

  22. [30]

    doi:10.48550/arXiv.2506.21521 , eprint=

    Potemkin Understanding in Large Language Models , url=. doi:10.48550/arXiv.2506.21521 , eprint=

    doi arXiv

  23. [31]

    IEEE Internet Computing , author=

    Where Does Bias in Common Sense Knowledge Models Come From? , volume=. IEEE Internet Computing , author=. 2022 , pages=. doi:10.1109/MIC.2022.3170914 , number=

    2022

  24. [32]

    Public Choice , author=

    More human than human: measuring ChatGPT political bias , volume=. Public Choice , author=. 2023 , pages=. doi:10.1007/s11127-023-01097-2 , number=

    2023 doi

  25. [33]

    Having Beer after Prayer? Measuring Cultural Bias in Large Language Models , url=

    Naous, Tarek and Ryan, Michael J and Ritter, Alan and Xu, Wei , editor=. Having Beer after Prayer? Measuring Cultural Bias in Large Language Models , url=. 2024 , pages=. doi:10.18653/v1/2024.acl-long.862 , booktitle=

    2024 doi

  26. [34]

    Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , author=

    What Are Chatbots' Stereotypes About? A Data-Driven Analysis of Large Language Models' Content Associations with Social Categories , volume=. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , author=. 2025 , pages=. doi:10.1609/aies.v8i2.36682 , number=

    2025 doi

  27. [35]

    Training language models to follow instructions with human feedback , volume=

    Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and As...

  28. [36]

    and Kern, Roger M

    Peterson, Richard A. and Kern, Roger M. , year=. Changing Highbrow Taste: From Snob to Omnivore , volume=. American Sociological Review , publisher=. doi:10.2307/2096460 , number=

    doi

  29. [37]

    LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users , url=

    Poole-Dayan, Elinor and Roy, Deb and Kabbara, Jad , year=. LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users , url=. 2406.17737 , archivePrefix=

  30. [38]

    The political preferences of LLMs , volume=

    Rozado, David , year=. The political preferences of LLMs , volume=. PLOS ONE , publisher=. doi:10.1371/journal.pone.0306621 , number=

    doi

  31. [39]

    Normative Evaluation of Large Language Models with Everyday Moral Dilemmas , url=

    Sachdeva, Pratik and van Nuenen, Tom , year=. Normative Evaluation of Large Language Models with Everyday Moral Dilemmas , url=. doi:10.1145/3715275.3732044 , booktitle=

  32. [41]

    doi:10.48550/arXiv.2402.14875 , eprint=

    What's in a Name? Auditing Large Language Models for Race and Gender Bias , url=. doi:10.48550/arXiv.2402.14875 , eprint=

    doi arXiv

  33. [42]

    Anesthesia & Analgesia , author=

    Correlation Coefficients: Appropriate Use and Interpretation , volume=. Anesthesia & Analgesia , author=. 2018 , pages=. doi:10.1213/ANE.0000000000002864 , number=

    2018 doi

  34. [43]

    How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion , volume=

    Seth, Agrima and Choudhury, Monojit and Sitaram, Sunayana and Toyama, Kentaro and Vashistha, Aditya and Bali, Kalika , year=. How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion , volume=. doi:10.1609/aies.v8i3.36718 , booktitle=

    doi

  35. [44]

    You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments , url=

    Shu, Bangzhao and Zhang, Lechen and Choi, Minje and Dunagan, Lavinia and Logeswaran, Lajanugen and Lee, Moontae and Card, Dallas and Jurgens, David , editor=. You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Mod...

    2024 doi

  36. [45]

    Advances in Neural Information Processing Systems 36 , author=

    Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting , volume=. Advances in Neural Information Processing Systems 36 , author=. 2023 , pages=

    2023

  37. [46]

    and Stewart, Brandon M

    Waight, Hannah and Yang, Eddie and Yuan, Yin and Messing, Solomon and Roberts, Margaret E. and Stewart, Brandon M. and Tucker, Joshua A. , year=. State media control influences large language models , DOI=. Nature , publisher=

  38. [47]

    and Koyejo, Sanmi , year=

    Wang, Angelina and Ho, Daniel E. and Koyejo, Sanmi , year=. The inadequacy of offline large language model evaluations: A need to account for personalization in model behavior , volume=. Patterns , publisher=. doi:10.1016/j.patter.2025.101397 , number=

  39. [48]

    Language Machines: Cultural AI and the End of Remainder Humanism , publisher=

    Weatherby, Leif , year=. Language Machines: Cultural AI and the End of Remainder Humanism , publisher=

  40. [49]

    Advances in Neural Information Processing Systems , author=

    Jailbroken: How Does LLM Safety Training Fail? , volume=. Advances in Neural Information Processing Systems , author=. 2023 , pages=

    2023

  41. [50]

    2024 IEEE International Conference on Big Data (BigData) , author=

    Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scoring of Texts with Large Language Models , ISSN=. 2024 IEEE International Conference on Big Data (BigData) , author=. 2024 , pages=. doi:10.1109/BigData62323.2024.10825235 , note=

    2024

  42. [51]

    Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation , url=

    Zhang, Jizhi and Bao, Keqin and Zhang, Yang and Wang, Wenjie and Feng, Fuli and He, Xiangnan , year=. Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation , url=. doi:10.1145/3604915.3608860 , booktitle=

  43. [52]

    Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , author=

    Should LLMs Be WEIRD? Exploring WEIRDness and Human Rights in Large Language Models , volume=. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , author=. 2025 , pages=. doi:10.1609/aies.v8i3.36761 , number=

    2025 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.