How Large Language Models Systematically Misrepresent American Climate Opinions
Pith reviewed 2026-05-16 18:31 UTC · model grok-4.3
The pith
Large language models compress the diversity of U.S. climate opinions by shifting less-concerned groups toward higher worry and vice versa.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Prompted with profiles of 978 respondents from a nationally representative U.S. climate opinion survey, six LLMs produced answers that compressed the diversity of reported concern, moving less-concerned demographic groups upward and more-concerned groups downward across twenty questions. The compression is intersectional: LLMs apply gender assumptions that align with actual survey patterns for White and Hispanic Americans but misalign for Black Americans, where real gender differences in climate opinion are distinct from the uniform pattern the models generate.
What carries the argument
Prompting LLMs with respondent demographic profiles from an existing climate opinion survey and direct numerical comparison of generated answers to the original human responses on twenty questions.
If this is right
- Analyses of public climate opinion that rely on LLMs may understate real differences between demographic groups.
- Climate outreach or consultation strategies guided by LLM simulations could target subgroups inaccurately.
- Standard bias audits that examine only overall accuracy or single demographic slices may miss the intersectional compression pattern.
- Reliance on LLMs for policy simulation in contested domains like climate could reduce the perceived need for targeted engagement with distinct subgroups.
Where Pith is reading between the lines
- The same compression pattern could appear when LLMs are asked to simulate opinions on other polarized issues such as immigration or economic policy.
- Developers might reduce the effect by training or prompting LLMs on raw survey microdata rather than aggregated summaries.
- Policymakers could treat LLM-generated opinion profiles as hypotheses that require fresh human survey validation before use in outreach design.
Load-bearing premise
That feeding LLMs demographic profiles produces answers that can be compared fairly to real human survey responses without introducing artifacts from the prompting method or the models' training data.
What would settle it
Running the same prompt protocol on a fresh, independent nationally representative U.S. sample and finding no systematic compression of opinion diversity or intersectional gender mismatches across demographic groups.
Figures
read the original abstract
Federal agencies and researchers increasingly use large language models to analyze and simulate public opinion. When AI mediates between the public and policymakers, accuracy across intersecting identities becomes consequential; inaccurate group-level estimates may mislead outreach, consultation, and policy design. While research examines intersectionality in LLM outputs, few studies have compared these outputs against real human responses across intersecting identities. Climate policy is one such domain, and this is particularly urgent for climate change, where opinion is contested and diverse. We investigate how LLMs represent demographic and intersectional patterns in U.S. climate opinions. We prompted six LLMs with profiles of 978 respondents from a nationally representative U.S. climate opinion survey and compared AI-generated responses to actual human answers across 20 questions. We find that LLMs appear to compress the diversity of American climate opinions, predicting less-concerned groups as more concerned and vice versa. This compression is intersectional: LLMs appear to apply uniform gender assumptions that match reality for White and Hispanic Americans but may misrepresent Black Americans, where actual gender patterns differ. These patterns, which may be invisible to standard auditing approaches, could undermine equitable climate governance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript examines how six large language models represent U.S. climate opinions by prompting them with demographic profiles drawn from 978 respondents in a nationally representative survey. It compares the LLM-generated answers to the actual human responses across 20 questions and reports that LLMs compress opinion diversity, over-predicting concern among less-concerned demographic groups and under-predicting it among more-concerned groups. The compression is described as intersectional: LLMs apply uniform gender assumptions that align with observed patterns for White and Hispanic Americans but diverge for Black Americans.
Significance. If the reported patterns are robust, the work would provide concrete evidence that LLM-based opinion simulation can distort subgroup differences in a policy-relevant domain. The intersectional framing extends existing bias audits by linking model outputs to real survey benchmarks, which could inform guidelines for federal agencies using LLMs to model public opinion on contested issues such as climate policy.
major comments (2)
- [Abstract] Abstract: The central claim that LLMs 'compress the diversity of American climate opinions' rests entirely on the assumption that responses elicited by prompting LLMs with respondent demographic profiles are directly comparable to the 978 human survey answers. The abstract supplies no information on prompt wording, zero-shot versus few-shot structure, temperature or sampling parameters, or any debiasing procedures, leaving open the possibility that observed compressions reflect prompt-induced averaging or training-data stereotypes rather than model misrepresentation.
- [Abstract] Abstract: The intersectional result—that gender assumptions match reality for White and Hispanic Americans but misrepresent Black Americans—requires subgroup-specific statistical tests and controls for multiple comparisons across the 20 questions. Without any description of the exact questions, response scales, sample sizes per intersectional cell, or error analysis, it is impossible to determine whether the differential patterns are load-bearing or artifacts of small cell sizes.
minor comments (1)
- [Abstract] The abstract would be strengthened by naming the six LLMs and briefly characterizing the 20 questions (e.g., policy support versus risk perception) to allow readers to gauge the scope of the comparison.
Simulated Author's Rebuttal
We thank the referee for their constructive comments on the abstract. We have revised the abstract to include brief methodological details on prompting and analysis procedures. Below we respond point by point to the major comments.
read point-by-point responses
-
Referee: [Abstract] Abstract: The central claim that LLMs 'compress the diversity of American climate opinions' rests entirely on the assumption that responses elicited by prompting LLMs with respondent demographic profiles are directly comparable to the 978 human survey answers. The abstract supplies no information on prompt wording, zero-shot versus few-shot structure, temperature or sampling parameters, or any debiasing procedures, leaving open the possibility that observed compressions reflect prompt-induced averaging or training-data stereotypes rather than model misrepresentation.
Authors: We agree the abstract was overly concise and omitted key parameters. The full manuscript's Methods section specifies zero-shot prompting with a fixed template that replicates the exact survey question wording and response scales, temperature fixed at 0 for deterministic outputs, and no debiasing or few-shot examples applied. We have revised the abstract to note these choices explicitly. Direct comparability follows from using identical question text and scales for LLM and human responses; any compression therefore reflects model behavior rather than prompt artifacts. revision: yes
-
Referee: [Abstract] Abstract: The intersectional result—that gender assumptions match reality for White and Hispanic Americans but misrepresent Black Americans—requires subgroup-specific statistical tests and controls for multiple comparisons across the 20 questions. Without any description of the exact questions, response scales, sample sizes per intersectional cell, or error analysis, it is impossible to determine whether the differential patterns are load-bearing or artifacts of small cell sizes.
Authors: We acknowledge the abstract lacked these details. The manuscript reports subgroup sample sizes (including smaller cells for Black Americans), provides question wording and scales in an appendix, and applies subgroup-specific t-tests with Bonferroni correction for the 20 questions plus error bars and sensitivity checks for cell size. We have added a clause to the abstract referencing these analyses and included power notes for smaller intersections in the revised supplement. The reported gender divergence for Black respondents remains after these controls. revision: partial
Circularity Check
No circularity: empirical comparison to external survey data
full rationale
The paper's central claim rests on prompting six LLMs with demographic profiles drawn from an external nationally representative survey of 978 U.S. respondents and directly comparing the generated answers to the actual human responses across 20 questions. No equations, fitted parameters, self-referential predictions, or load-bearing self-citations are present in the available text. The observed patterns of opinion compression and intersectional gender assumptions are presented as empirical findings from this external benchmark rather than derived by construction from the paper's own inputs or prior author work. The derivation chain is therefore self-contained against the independent survey data.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
2024 Federal AI Use Case Inventory
Office of Management and Budget and Federal Chief Information Officers Council. 2024 Federal AI Use Case Inventory. Technical report, Office of Management and Budget (OMB); CIO.gov, Washington, DC, December 2024
work page 2024
-
[2]
Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christo- pher Rytting, and David Wingate. Out of One, Many: Using Language Models to Simulate Human Samples.Political Analysis, 31(3):337–351, July 2023. ISSN 1047-1987, 1476-4989. doi: 10.1017/pan.2023.2
-
[3]
Park, Rafael Valdece Sousa Bastos, and Philip E
Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy.Science Advances, 10(45):eadp1528, November 2024. doi: 10.1126/sciadv.adp1528
-
[4]
James N. Druckman and Mary C. McGrath. The evidence for moti- vated reasoning in climate change preference formation.Nature Climate Change, 9(2):111–119, February 2019. ISSN 1758-6798. doi: 10.1038/ s41558-018-0360-1
work page 2019
-
[5]
Matthew J. Hornsey, Emily A. Harris, Paul G. Bain, and Kelly S. Fielding. Meta-analyses of the determinants and outcomes of belief in climate change. Nature Climate Change, 6(6):622–626, June 2016. ISSN 1758-6798. doi: 10.1038/nclimate2943
-
[6]
Climate change through the lens of intersectionality.Environmental Politics, 23(3):417–433, May 2014
Anna Kaijser and Annica Kronsell. Climate change through the lens of intersectionality.Environmental Politics, 23(3):417–433, May 2014. ISSN 0964-4016, 1743-8934. doi: 10.1080/09644016.2013.835203
-
[7]
Amorim-Maia, Isabelle Anguelovski, Eric Chu, and James Connolly
Ana T. Amorim-Maia, Isabelle Anguelovski, Eric Chu, and James Connolly. Intersectional climate justice: A conceptual pathway for bridging adapta- tion planning, transformative action, and social equity.Urban Climate, 41: 101053, January 2022. ISSN 2212-0955. doi: 10.1016/j.uclim.2021.101053
-
[8]
Critical climate justice.The Geographical Journal, 188 (1):118–124, March 2022
Farhana Sultana. Critical climate justice.The Geographical Journal, 188 (1):118–124, March 2022. ISSN 0016-7398, 1475-4959. doi: 10.1111/geoj. 12417
-
[9]
Kimberle Crenshaw. Mapping the Margins: Intersectionality, Identity Pol- itics, and Violence against Women of Color.Stanford Law Review, 43(6): 1241, July 1991. ISSN 00389765. doi: 10.2307/1229039
-
[10]
Siri Eriksen, E. Lisa F. Schipper, Morgan Scoville-Simonds, Katharine Vin- cent, Hans Nicolai Adam, Nick Brooks, Brian Harding, Dil Khatri, Lutgart Lenaerts, Diana Liverman, Megan Mills-Novoa, Marianne Mosberg, Synne Movik, Benard Muok, Andrea Nightingale, Hemant Ojha, Linda Sygna, 24 Marcus Taylor, Coleen Vogel, and Jennifer Joy West. Adaptation inter- v...
-
[11]
Eric Chu, Isabelle Anguelovski, and JoAnn Carmin. Inclusive approaches to urban climate adaptation planning and implementation in the Global South.Climate Policy, 16(3):372–392, April 2016. ISSN 1469-3062. doi: 10.1080/14693062.2015.1019822
-
[12]
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and Fairness in Large Language Models: A Survey.Com- putational Linguistics, 50(3):1097–1179, September 2024. ISSN 0891-2017, 1530-9312. doi: 10.1162/coli a 00524
-
[13]
Llms, Virtual Users, and Bias: Predicting Any Survey Question Without Human Data
Enzo Sinacola, Arnault Pachot, and Thierry Petit. Llms, Virtual Users, and Bias: Predicting Any Survey Question Without Human Data. InPro- ceedings of 17th International Conference on Machine Learning and Com- puting, pages 396–407. Springer, Cham, 2025. ISBN 978-3-031-94892-3. doi: 10.1007/978-3-031-94892-3 29
-
[14]
Mahmud Omar, Shelly Soffer, Reem Agbareia, Nicola Luigi Bragazzi, Don- ald U. Apakama, Carol R. Horowitz, Alexander W. Charney, Robert Free- man, Benjamin Kummer, Benjamin S. Glicksberg, Girish N. Nadkarni, and Eyal Klang. Sociodemographic biases in medical decision making by large language models.Nature Medicine, 31(6):1873–1881, June 2025. ISSN 1546-170...
-
[15]
Understanding Intrinsic Socioeconomic Biases in Large Language Models
Mina Arzaghi, Florian Carichon, and Golnoosh Farnadi. Understanding Intrinsic Socioeconomic Biases in Large Language Models. InProceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society, pages 49–
work page 2024
-
[16]
AAAI Press, February 2025
work page 2025
-
[17]
A Survey on Intersectional Fairness in Ma- chine Learning: Notions, Mitigation, and Challenges
Usman Gohar and Lu Cheng. A Survey on Intersectional Fairness in Ma- chine Learning: Notions, Mitigation, and Challenges. InProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6619–6627, Macau, SAR China, August 2023. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-1-956792- 0...
-
[18]
Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change, May 2025
Murathan Kurfalı, Shorouq Zahra, Joakim Nivre, and Gabriele Messori. Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change, May 2025
work page 2025
-
[19]
Sanguk Lee, Tai-Quan Peng, Matthew H. Goldberg, Seth A. Rosenthal, John E. Kotcher, Edward W. Maibach, and Anthony Leiserowitz. Can large language models estimate public opinion about global warming? An empirical assessment of algorithmic fidelity and bias.PLOS Climate, 3 25 (8):e0000429, August 2024. ISSN 2767-3200. doi: 10.1371/journal.pclm. 0000429
-
[20]
Owen Gaffney, Amy Luers, Franklin Carrero-Martinez, Berna Oztekin- Gunaydin, Felix Creutzig, Virginia Dignum, Victor Galaz, Naoko Ishii, Francesca Larosa, Maria Leptin, and Ken Takahashi Guevara. The Earth alignment principle for artificial intelligence.Nature Sustainability, 8(5): 467–469, May 2025. ISSN 2398-9629. doi: 10.1038/s41893-025-01536-6
-
[21]
Salil Benegal, Fl´ avio Azevedo, and Mirya R Holman. Race, ethnicity, and support for climate policy.Environmental Research Letters, 17(11):114060, November 2022. ISSN 1748-9326. doi: 10.1088/1748-9326/aca0ac
-
[23]
David Schlosberg.Defining Environmental Justice: Theories, Movements, and Nature. OUP Oxford, May 2007. ISBN 978-0-19-153671-7
work page 2007
-
[24]
Jouni Paavola and W. Neil Adger. Fair adaptation to climate change. Ecological Economics, 56(4):594–609, April 2006. ISSN 0921-8009. doi: 10.1016/j.ecolecon.2005.03.015
-
[25]
Massimo Cattino and Diana Reckien. Does public participation lead to more ambitious and transformative local climate change planning?Current Opinion in Environmental Sustainability, 52:100–110, October 2021. ISSN 1877-3435. doi: 10.1016/j.cosust.2021.08.004
-
[26]
Farzaneh Shaikh Khatibi, Aysin Dedekorkut-Howes, Michael Howes, and Elnaz Torabi. Can public awareness, knowledge and engagement improve climate change adaptation policies?Discover Sustainability, 2(1):18, March
-
[27]
doi: 10.1007/s43621-021-00024-z
ISSN 2662-9984. doi: 10.1007/s43621-021-00024-z
-
[28]
Physics Letters B876, 140423 (2026) https://doi.org/10.1016/j
Jens Newig, Nicolas W. Jager, Edward Challies, and Elisa Kochsk¨ amper. Does stakeholder participation improve environmental governance? Ev- idence from a meta-analysis of 305 case studies.Global Environmental Change, 82:102705, September 2023. ISSN 0959-3780. doi: 10.1016/j. gloenvcha.2023.102705
work page doi:10.1016/j 2023
-
[29]
Anthony Leiserowitz, Jennifer Marlon, Seth Rosenthal, Matthew Ballew, Matthew Goldberg, John Kotcher, and Edward Maibach. Climate Change in the American Mind: National Survey Data on Public Opinion (2008- 2024), February 2020
work page 2008
-
[30]
Ollama. Ollama. Ollama, November 2025. 26
work page 2025
-
[31]
Ballew, Anthony Leiserowitz, Connie Roser-Renouf, Seth A
Matthew T. Ballew, Anthony Leiserowitz, Connie Roser-Renouf, Seth A. Rosenthal, John E. Kotcher, Jennifer R. Marlon, Erik Lyon, Matthew H. Goldberg, and Edward W. Maibach. Climate Change in the American Mind: Data, Tools, and Trends.Environment: Science and Policy for Sustainable Development, 61(3):4–18, May 2019. ISSN 0013-9157. doi: 10.1080/00139157.201...
-
[32]
Parrish Bergquist, David M Konisky, and John Kotcher. Energy policy and public opinion: Patterns, trends and future directions.Progress in Energy, 2(3):032003, July 2020. ISSN 2516-1083. doi: 10.1088/2516-1083/ab9592
-
[33]
Matthew John Cutler, Jennifer Marlon, Peter Howe, and Anthony Leis- erowitz. ‘Is global warming affecting the weather?’ Evidence for increased attribution beliefs among coastal versus inland US residents.Environmen- tal Sociology, 6(1):6–18, January 2020. ISSN null. doi: 10.1080/23251042. 2019.1690725
-
[34]
Emily A. Hurley and Micheal S. Molloy. How do people change their beliefs about climate change? A qualitative study on opinion shift in the U.S. Midwest.Climatic Change, 178(1):8, January 2025. ISSN 1573-1480. doi: 10.1007/s10584-024-03835-x
-
[35]
Jennifer R Marlon, Xinran Wang, Parrish Bergquist, Peter D Howe, An- thony Leiserowitz, Edward Maibach, Matto Mildenberger, and Seth Rosen- thal. Change in US state-level public opinion about climate change: 2008– 2020.Environmental Research Letters, 17(12):124046, December 2022. ISSN 1748-9326. doi: 10.1088/1748-9326/aca702
-
[36]
Evan Stewart, Katsyris Rivera-Kientz, and Timothy Dacey. Mapping Racial and Ethnic Variation in Climate Belief Networks.Sociology of Race and Ethnicity, page 23326492241259404, June 2024. ISSN 2332-6492. doi: 10.1177/23326492241259404
-
[37]
Questioning the survey responses of large language models
Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler- D¨ unner. Questioning the survey responses of large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, volume 37 ofNIPS ’24, pages 45850–45878, Red Hook, NY, USA, December 2024. Curran Associates Inc. ISBN 979-8-3313-1438-5
work page 2024
-
[38]
Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. BetterBench: Assessing AI Bench- marks, Uncovering Issues, and Establishing Best Practices, November 2024
work page 2024
-
[39]
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sid- ney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wil- son Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavl...
work page 2024
-
[40]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processing Sys- tems, NIPS ’22, pages 24824–24837, Red Hook, NY, USA, November 2022. Curran Ass...
work page 2022
-
[41]
Xuezhi Wang, Jason Wei, D. Schuurmans, Quoc Le, Ed H. Chi, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. InThe Eleventh International Conference on Learning Represen- tations, 2023
work page 2023
-
[42]
Riley E. Dunlap and Araon M. McCright. A Widening Gap: Republican and Democratic Views on Climate Change.Environment: Science and Policy for Sustainable Development, 50(5):26–35, September 2008. ISSN 0013-9157. doi: 10.3200/ENVT.50.5.26-35
-
[43]
Jenkins-Smith, Kim Klockow, Matthew C
Kevin Goebbert, Hank C. Jenkins-Smith, Kim Klockow, Matthew C. Nowlin, and Carol L. Silva. Weather, Climate, and Worldviews: The Sources and Consequences of Public Perceptions of Changes in Local Weather Patterns.Weather, Climate, and Society, 4:132–144, April 2012. doi: 10.1175/WCAS-D-11-00044.1
-
[44]
Gabriela Czarnek, Ma lgorzata Kossowska, and Paulina Szwed. Right-wing ideology reduces the effects of education on climate change beliefs in more developed countries.Nature Climate Change, 11(1):9–13, January 2021. ISSN 1758-6798. doi: 10.1038/s41558-020-00930-6
-
[45]
Douglas M. Jennewein, Johnathan Lee, Chris Kurtz, William Dizon, Ian Shaeffer, Alan Chapman, Alejandro Chiquete, Josh Burks, Amber Carlson, Natalie Mason, Arhat Kobawala, Thirugnanam Jagadeesan, Praful Bhar- gav Basani, Torey Battelle, Rebecca Belshe, Deb McCaffrey, Marisa Brazil, Chaitanya Inumella, Kirby Kuznia, Jade Buzinski, Dhruvil Deepakbhai Shah, S...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.