REVIEW 20 cited by
Approaching Human-Level Forecasting with Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Approaching Human-Level Forecasting with Language Models
read the original abstract
Forecasting future events is important for policy and decision making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms. Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters, and in some settings surpasses it. Our work suggests that using LMs to forecast the future could provide accurate predictions at scale and help to inform institutional decision making.
Forward citations
Cited by 20 Pith papers
-
SocietyBench: Forecasting Counterfactual Social-World Evolution
A new benchmark measures LLM social-world forecasting on anonymized real events, finding the best model reaches 75/100 and agent scaffolding does not help.
-
Preference Optimization Drives Monoculture in LLM Prediction Markets
DPO fine-tuning causes LLM agents to share output distributions with pairwise error correlations of ρ=0.70, reducing ten agents to the effective power of ≈1.4 independent forecasters.
-
Nous: An Attempt to Extract and Inject the Cognition Behind Prediction-Market Behavior
Behavioral profiles from prediction-market traders are partially stable and identifiable but cannot be transmitted via prompts to reduce LLM forecast correlations or improve Brier scores.
-
Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles
Aggregating probability estimates from 15 LLMs with a learned linear model beats individual models and classical voting rules on clean questions, and the apparent cloud-vs-local capability gap largely vanishes under c...
-
Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
More capable LLMs produce worse distributional forecasts on superlinear growth time series with tail risks of regime change, with the error concentrated in the upper tail; this reverses on conventional threshold metrics.
-
Belief Propagation in LLM World Models: Measuring Strategic Information Bias with Prediction Markets
English news context systematically biases LLM predictions on Ukraine territorial markets toward Russian capture, and the bias originates in the text, not the model.
-
OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking
OracleProto is a reproducible framework that uses model-cutoff alignment, temporal masking, and leakage detection to create low-leakage benchmarks for LLM native forecasting from past events.
-
Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents
Foresight Arena is an on-chain benchmark using Brier and novel Alpha scores to evaluate AI forecasting agents on live prediction markets via Polygon smart contracts.
-
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
An agentic forecaster with linguistic belief states, logit-space multi-trial shrinkage, and hierarchical Platt calibration achieves SOTA Brier Index on ForecastBench binary questions.
-
WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament
A live, leakage-free benchmark of six frontier LLMs across all 104 World Cup matches finds they match, not beat, the betting market and herd on outcomes.
-
FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches
On 104 World Cup matches, four LLM forecasting agents make identical top picks in 92% of matches, none beats the betting market's Brier score, but their betting ROI spans -18% to +10%.
-
Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
More capable language models exhibit inverse scaling on distributional forecasting for superlinear-growth time series with tail risks, with errors concentrated in the upper tail across synthetic and real datasets.
-
Coordination as an Architectural Layer for LLM-Based Multi-Agent Systems
Coordination treated as a separable architectural layer in LLM multi-agent systems yields distinguishable Murphy-decomposed performance signatures on prediction-market tasks, with some configurations dominating a cost...
-
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
BLF achieves state-of-the-art binary forecasting on ForecastBench by using linguistic belief states updated in tool-use loops, hierarchical multi-trial logit averaging, and hierarchical Platt scaling calibration.
-
Harnessing Pre-Resolution Signals for Future Prediction Agents
Milkyway evolves a future prediction harness using internal feedback from repeated predictions on the same unresolved question, achieving top scores on FutureX (44.07 to 60.90) and FutureWorld (62.22 to 77.96).
-
Argumentative Large Language Models for Explainable and Contestable Claim Verification
ArgLLMs build argumentation frameworks from LLMs to support explainable and contestable formal reasoning for claim verification.
-
Towards Effective Long-Video Event Prediction via Multi-Level Event Semantics Mining
VISTA mines multi-level event semantics via visual prompts, knowledge-enhanced retrieval, and proposal integration to improve long-video event prediction over existing LVLMs.
-
Harnessing Pre-Resolution Signals for Future Prediction Agents
Milkyway uses pre-resolution signals from temporal contrasts in evolving evidence and repeated forecasts to evolve a harness and improve predictions before resolution, outperforming baselines on FutureX and FutureWorld.
-
Extrapolating Volition with Recursive Information Markets
Recursive information markets with forgetful LLM buyers can align information prices with true value and extend to scalable oversight in AI alignment.
-
The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission
Three independent LLMs exhibit correlated forecasting errors on 568 binary questions but human predictions show no activation of this shared bias.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.