Pith. sign in

REVIEW 10 cited by

ClimateBert: A Pretrained Language Model for Climate-Related Text

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.12010 v3 pith:PMWWISBU submitted 2021-10-22 cs.CL

Nicolas Webersinke , Mathias Kraus , Julia Anna Bingler , Markus Leippold This is my paper
classification cs.CL
keywords languageclimate-relatedclimatebertcommonmodelpretrainedtexttexts
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Over the recent years, large pretrained language models (LM) have revolutionized the field of natural language processing (NLP). However, while pretraining on general language has been shown to work very well for common language, it has been observed that niche language poses problems. In particular, climate-related texts include specific language that common LMs can not represent accurately. We argue that this shortcoming of today's LMs limits the applicability of modern NLP to the broad field of text processing of climate-related texts. As a remedy, we propose CLIMATEBERT, a transformer-based language model that is further pretrained on over 2 million paragraphs of climate-related texts, crawled from various sources such as common news, research articles, and climate reporting of companies. We find that CLIMATEBERT leads to a 48% improvement on a masked language model objective which, in turn, leads to lowering error rates by 3.57% to 35.71% for various climate-related downstream tasks like text classification, sentiment analysis, and fact-checking.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Multimodal Dataset for Large Language Model Applications in the Energy Domain

    eess.SY 2026-07 accept novelty 6.5 of 10

    mAIEnergy is a harmonized, FAIR multimodal energy dataset (text, imagery, numerical series, geospatial graphs) purpose-built for LLM pre-training and retrieval-augmented generation.

  2. Unfit for stranding assessment: a panel-scale multimodal-LLM audit of building-decarbonisation disclosure (BeDA)

    cs.CY 2026-07 conditional novelty 6.0 of 10

    Only ~21.5% of built-environment firm-reports (45.5% of real-estate reports) disclose operational carbon intensity per m²/yr, the metric CRREM-style stranding assessment requires.

  3. Discourse-Aware Policy Analysis with Argumentation: A Hybrid LLM-Symbolic Framework for Disaster Governance

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A hybrid LLM-symbolic pipeline maps disaster-policy text to typed argumentation graphs using new frame-mediated relation subtypes, with a new 100-document dataset from USA, UK, Canada, and Australia.

  4. Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new human capital keyword lexicon, built by training word2vec on confirmed 10-K human capital disclosures, is released with the underlying text and code for measuring corporate human capital disclosure.

  5. EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EarthSE provides a two-level QA benchmark and an open-ended dialogue benchmark for Earth science and shows current LLMs perform poorly on both.

  6. What Transfers Under Source Shift? Definitions, Examples, and Fine-Tuning for Climate Disclosure Classification

    cs.CL 2026-07 conditional novelty 5.0 of 10

    In TCFD climate-disclosure classification, the strongest in-source adaptation strategies (similarity retrieval, LoRA fine-tuning) lose most of their advantage under source shift, while definitions and random few-shot ...

  7. ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queries

    cs.CL 2025-06 conditional novelty 5.0 of 10

    An automated pipeline creates climate instruction data, and fine-tuning a geoscience LLM on it improves climate question-answering accuracy over general instruction tuning.

  8. Multi-Agent Visual-Language Reasoning for Comprehensive Highway Scene Understanding

    cs.CV 2025-08 conditional novelty 4.0 of 10

    A two-agent vision-language system with GPT-4o-generated chain-of-thought prompts improves highway weather, wetness, and congestion classification on small curated video datasets, with the biggest gains when sensor da...

  9. Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applications

    cs.MA 2025-07 conditional novelty 4.0 of 10

    The paper defines urban LLM agents, surveys their sensing, memory, reasoning, execution, and learning workflows, and organizes their applications across planning, transportation, environment, safety, and society.

  10. Domain Specific Benchmarks for Evaluating Multimodal Large Language Models

    cs.LG 2025-06 conditional novelty 3.0 of 10

    A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.

Pith tools