Fine-tuned GPT-3.5-turbo agrees with expert climate coders on social media claims as often as the experts agree with each other (alpha=0.89), but the study's open-source benchmark is weakened by a flawed prompt and random replacement of invalid responses.
ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Evaluating the accuracy of outputs generated by Large Language Models (LLMs) is especially important in the climate science and policy domain. We introduce the Expert Confidence in Climate Statements (ClimateX) dataset, a novel, curated, expert-labeled dataset consisting of 8094 climate statements collected from the latest Intergovernmental Panel on Climate Change (IPCC) reports, labeled with their associated confidence levels. Using this dataset, we show that recent LLMs can classify human expert confidence in climate-related statements, especially in a few-shot learning setting, but with limited (up to 47%) accuracy. Overall, models exhibit consistent and significant over-confidence on low and medium confidence statements. We highlight implications of our results for climate communication, LLMs evaluation strategies, and the use of LLMs in information retrieval systems.
citation-role summary
background 1
citation-polarity summary
fields
cs.CY 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Enhancing LLMs for Governance with Human Oversight: Evaluating and Aligning LLMs on Expert Classification of Climate Misinformation for Detecting False or Misleading Claims about Climate Change
Fine-tuned GPT-3.5-turbo agrees with expert climate coders on social media claims as often as the experts agree with each other (alpha=0.89), but the study's open-source benchmark is weakened by a flawed prompt and random replacement of invalid responses.