REVIEW 20 cited by
FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
read the original abstract
Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce $\textbf{FutureX}$, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.
Forward citations
Cited by 20 Pith papers
-
PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
Only two of seven LLMs produce positive returns on live Polymarket data, with MiMo-V2-Flash at 17.6% CWR and Gemini-3-Flash at 6.2% CWR while the other five lose money.
-
Agentic Time Machine as an Infrastructure for Future-Event Forecasting
Agentic Time Machine reconstructs historical web states for offline evaluation of forecasting agents, with a multi-agent framework achieving top ranks on FutureX live and past benchmarks.
-
WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning
WorldReasoner supplies 345 resolved forecasting tasks built from 14,141 articles to score LM agents on outcome quality, evidence quality, and reasoning quality against time-bounded evidence and hindsight graphs.
-
Scientific reasoning does not reliably translate into scientific forecasting in frontier AI
Introduces the CUSP benchmark across 4760 events and finds frontier AI models can pick plausible directions but fail to predict whether or when scientific advances will occur, with performance varying by domain and in...
-
Scientific reasoning does not reliably translate into scientific forecasting in frontier AI
Frontier LLMs reason well about past science but are near chance at judging whether specific advances will happen, systematically late on when, and overconfident — even with extra pre-cutoff knowledge.
-
OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking
OracleProto is a reproducible framework that uses model-cutoff alignment, temporal masking, and leakage detection to create low-leakage benchmarks for LLM native forecasting from past events.
-
KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
KellyBench reveals that frontier language models lose money on average when making sequential betting decisions over a full soccer season, with the best model returning -8% and scoring only 26.5% on a human expert rub...
-
Decentralized Aggregation of LLM Predictions via Wagering Mechanisms
A leave-one-out wagering payout makes LLM aggregation weights equal expected score advantage, yielding DSIC predictions, decentralized wager learning, and performance matching centralized routers.
-
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
An agentic forecaster with linguistic belief states, logit-space multi-trial shrinkage, and hierarchical Platt calibration achieves SOTA Brier Index on ForecastBench binary questions.
-
Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry
InfoDelphi partitions evidence to induce information asymmetry in multi-agent LLM deliberation, yielding 12-18% Brier score gains and 4-8 pp accuracy gains on a 375-question benchmark.
-
LATTICE: Evaluating Decision Support Utility of Crypto Agents
LATTICE is a scalable LLM-judge benchmark for crypto agent decision support that reveals performance trade-offs among real-world copilots across dimensions and tasks.
-
Analytica: Soft Propositional Reasoning for Robust and Scalable LLM-Driven Analysis
Analytica uses soft propositional reasoning to decompose problems, ground facts with tools, and average outputs with linear models, yielding 15.84% average accuracy gains and lower variance on forecasting tasks.
-
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
BLF achieves state-of-the-art binary forecasting on ForecastBench by using linguistic belief states updated in tool-use loops, hierarchical multi-trial logit averaging, and hierarchical Platt scaling calibration.
-
Harnessing Pre-Resolution Signals for Future Prediction Agents
Milkyway evolves a future prediction harness using internal feedback from repeated predictions on the same unresolved question, achieving top scores on FutureX (44.07 to 60.90) and FutureWorld (62.22 to 77.96).
-
AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models
LLMs are unreliable when asked to emit buy/sell/hold actions, so this paper benchmarks them as code-writing quantitative researchers whose generated strategies are backtested deterministically.
-
Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery
DBench-Bio builds a dynamic biology benchmark from post-release abstracts, but LLM-generated gold answers and unverified per-model temporal separation undermine its claim to measure knowledge discovery.
-
FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
FutureOmni, a 919-video, 1,034-question audio-visual future-forecasting benchmark, shows top MLLMs reach only 64.8% accuracy, and OFF tuning improves open models.
-
CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency
CryptoBench is a new dynamic benchmark for LLM agents in cryptocurrency that reveals a retrieval-prediction imbalance in model performance.
-
ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment
ForeSci is a temporally controlled benchmark with 500 tasks for assessing LLM agents on forward-looking AI research judgments in four domains using cutoff-aligned knowledge bases.
-
Harnessing Pre-Resolution Signals for Future Prediction Agents
Milkyway uses pre-resolution signals from temporal contrasts in evolving evidence and repeated forecasts to evolve a harness and improve predictions before resolution, outperforming baselines on FutureX and FutureWorld.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.