REVIEW 6 cited by
ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper introduces ClimateGPT, a model family of domain-specific large language models that synthesize interdisciplinary research on climate change. We trained two 7B models from scratch on a science-oriented dataset of 300B tokens. For the first model, the 4.2B domain-specific tokens were included during pre-training and the second was adapted to the climate domain after pre-training. Additionally, ClimateGPT-7B, 13B and 70B are continuously pre-trained from Llama~2 on a domain-specific dataset of 4.2B tokens. Each model is instruction fine-tuned on a high-quality and human-generated domain-specific dataset that has been created in close cooperation with climate scientists. To reduce the number of hallucinations, we optimize the model for retrieval augmentation and propose a hierarchical retrieval strategy. To increase the accessibility of our model to non-English speakers, we propose to make use of cascaded machine translation and show that this approach can perform comparably to natively multilingual models while being easier to scale to a large number of languages. Further, to address the intrinsic interdisciplinary aspect of climate change we consider different research perspectives. Therefore, the model can produce in-depth answers focusing on different perspectives in addition to an overall answer. We propose a suite of automatic climate-specific benchmarks to evaluate LLMs. On these benchmarks, ClimateGPT-7B performs on par with the ten times larger Llama-2-70B Chat model while not degrading results on general domain benchmarks. Our human evaluation confirms the trends we saw in our benchmarks. All models were trained and evaluated using renewable energy and are released publicly.
Forward citations
Cited by 6 Pith papers
-
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Only gradient-based (LESS) representations make subset-to-query distance a reliable predictor of instruction-tuning performance; greedy round-robin helps most at small budgets, and random selection is surprisingly com...
-
The Curious Language Model: Strategic Test-Time Information Acquisition
CuriosiTree is a greedy tree-search policy that lets LLMs select cost-effective information-gathering actions at test time, outperforming baselines in a simulated clinical diagnosis environment.
-
ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queries
An automated pipeline creates climate instruction data, and fine-tuning a geoscience LLM on it improves climate question-answering accuracy over general instruction tuning.
-
Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change
ClimateEval unifies 25 climate-related NLP tasks into one benchmark and shows that open-source LLMs gain from few-shot examples but lag on misinformation and fine-grained entity recognition.
-
Multi-Agent Visual-Language Reasoning for Comprehensive Highway Scene Understanding
A two-agent vision-language system with GPT-4o-generated chain-of-thought prompts improves highway weather, wetness, and congestion classification on small curated video datasets, with the biggest gains when sensor da...
-
Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applications
The paper defines urban LLM agents, surveys their sensing, memory, reasoning, execution, and learning workflows, and organizes their applications across planning, transportation, environment, safety, and society.
Discussion (0). Sign in to comment.