GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation
read the original abstract
Time series foundation models excel in zero-shot forecasting, handling diverse tasks without explicit training. However, the advancement of these models has been hindered by the lack of comprehensive benchmarks. To address this gap, we introduce the General Time Series Forecasting Model Evaluation, GIFT-Eval, a pioneering benchmark aimed at promoting evaluation across diverse datasets. GIFT-Eval encompasses 23 datasets over 144,000 time series and 177 million data points, spanning seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts. To facilitate the effective pretraining and evaluation of foundation models, we also provide a non-leaking pretraining dataset containing approximately 230 billion data points. Additionally, we provide a comprehensive analysis of 17 baselines, which includes statistical models, deep learning models, and foundation models. We discuss each model in the context of various benchmark characteristics and offer a qualitative analysis that spans both deep learning and foundation models. We believe the insights from this analysis, along with access to this new standard zero-shot time series forecasting benchmark, will guide future developments in time series foundation models. Code, data, and the leaderboard can be found at https://github.com/SalesforceAIResearch/gift-eval .
This paper has not been read by Pith yet.
Forward citations
Cited by 42 Pith papers
-
TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Models
TSFMAudit detects pretraining contamination in time series foundation models via probe adaptation dynamics (faster loss drop, smaller backbone shift), tested on 6 models and 187 datasets against 10 LLM-derived baselines.
-
ISOMORPH: A Supply Chain Digital Twin for Simulation, Dataset Generation, and Forecasting Benchmarks
ISOMORPH is a modular digital twin simulator for supply chain networks that releases datasets exhibiting variance amplification and regime shifts for benchmarking forecasting models and performing forward uncertainty ...
-
TabArena: A Living Benchmark for Machine Learning on Tabular Data
TabArena launches a dynamic, updatable benchmarking system for tabular ML that shows boosted trees remain competitive, deep learning matches them under larger budgets with ensembling, foundation models excel on small ...
-
Beyond IID: How General Are Tabular Foundation Models, Really?
Tabular foundation models excel on tiny- to medium-sized IID data but are outperformed by traditional tree-based and deep learning models on non-IID, large, and high-dimensional datasets, based on evaluations across 1...
-
CloudCons: A Comprehensive End-to-End Benchmark for Cloud Resource Consolidation
CloudCons benchmark shows foundation models' superior zero-shot forecasting does not automatically yield better resource consolidation decisions, with predictive quantile choice acting as a key lever for efficiency-re...
-
Why Do Time Series Models Need Long Context Windows?
Long input windows are required to identify the generative process in time series forecasting even for short-memory processes, and decoupling identification from forecasting improves scalability.
-
Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density
Olivia harmonizes time series datasets via normalized power spectral density using a Harmonizer module and resonator-based HarmonicAttention, achieving state-of-the-art zero-shot, few-shot, and full-shot forecasting o...
-
ISOMORPH: A Supply Chain Digital Twin for Simulation, Dataset Generation, and Forecasting Benchmarks
ISOMORPH supplies the first public configurable digital-twin simulator for multi-echelon logistics networks together with generated datasets and zero-shot foundation-model benchmarks that exceed GIFT-Eval MASE at shor...
-
Explainable Load Forecasting with Covariate-Informed Time Series Foundation Models
Time series foundation models match the performance of specialized models for day-ahead load forecasting while providing explanations that match domain knowledge on weather and calendar effects.
-
TempusBench: An Evaluation Framework for Time-Series Forecasting
TempusBench is a new evaluation framework for time-series forecasting models that supplies fresh non-overlapping datasets, tasks beyond horizon and domain, consistent tuning across models, and visualization tools.
-
TS-Arena -- A Live Forecast Pre-Registration Platform
TS-Arena is a live pre-registration platform that evaluates time series forecasts on future data streams to eliminate information leakage.
-
TelecomTS: A Multi-Modal Observability Dataset for Time Series and Language Analysis
TelecomTS is a new observability dataset from 5G networks that preserves absolute scale and supports multi-modal tasks, showing that current time series and language models struggle with abrupt noisy dynamics.
-
Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecasting
Super-Linear introduces a pretrained MoE architecture using frequency-specialized linear experts and spectral gating for efficient general time series forecasting.
-
Aionoscope: Debugging Latent-State Accessibility in Time-Series Representations
Aionoscope shows that time-series representations recover coarse signal types reliably but expose dense latent states like phase and amplitude much less reliably, with best dense-probe R² at 0.689 versus oracle 0.999.
-
From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol
Presents a fail-closed certification protocol for determining when forecasting leaderboard winners are deployment-actionable, using a traffic dataset to show friction-induced reversals and an audit to prevent overclaiming.
-
Existence Precedes Value: Joint Modeling of Observational Existence and Evolving States in Time Series Forecasting
Timeflies reformulates time series forecasting as joint inference of future observability and value estimation using coupled observation and value streams.
-
TimeRouter: Efficient and Adaptive Routing of Time-Series Foundation Models
TimeRouter routes among time-series foundation models via discriminative routing, selective gating and ensemble fallback, reporting SOTA LB MASE 0.6765 on GIFT-EVAL.
-
Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining
Equal-weight mixture of synthetic generators matches or exceeds best single generator for time series foundation model pretraining and strengthens further with real data.
-
GITCO: Gated Inference-Time Context Optimization in TSFMs
GITCO delivers +1.95% average MASE reduction on TimesFM 2.5 across 53 datasets by gated inference-time suppression of anomalous patches, capturing 89.9% of the improvement upper bound.
-
Time Series as Language: A Universal Tokenizer for General-Purpose Time Series Foundation Models
UniTok tokenizes time series for an off-the-shelf LLM foundation model that unifies forecasting, generation, and classification through next-token prediction and training-free inference.
-
AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting
AME-TS is a structure-guided sparse MoE foundation model for time series that aligns expert routing with series-level temporal descriptors to achieve strong accuracy-efficiency tradeoffs on GIFT-Eval while improving s...
-
RareCP: Regime-Aware Retrieval for Efficient Conformal Prediction
RareCP improves interval efficiency for time series conformal prediction by retrieving and weighting regime-specific calibration examples while adapting to drift and maintaining coverage.
-
FETS Benchmark: Foundation Models Outperform Dataset-specific Machine Learning in Energy Time Series Forecasting
Foundation models outperform dataset-specific machine learning in energy time series forecasting across 54 datasets in 9 categories.
-
IDOBE: Infectious Disease Outbreak forecasting Benchmark Ecosystem
IDOBE compiles over 10,000 epidemiological outbreaks into a public benchmark and shows that MLP-based models deliver the most robust short-term forecasts while statistical methods hold a slight edge pre-peak.
-
WaveMoE: A Wavelet-Enhanced Mixture-of-Experts Foundation Model for Time Series Forecasting
WaveMoE uses a dual-path architecture with aligned time-series and wavelet tokens routed through shared experts to improve forecasting performance on diverse benchmarks.
-
Zero-shot Multivariate Time Series Forecasting Using Tabular Prior Fitted Networks
A framework recasts multivariate time series forecasting as scalar regression problems that tabular prior-fitted networks can solve zero-shot while addressing inter-channel interactions.
-
MICA: Multivariate Infini Compressive Attention for Time Series Forecasting
MICA adapts infini compressive attention to the channel dimension, enabling scalable cross-channel dependencies in Transformers and cutting forecast error by 5.4% on average versus channel-independent baselines.
-
MICA: Multivariate Infini Compressive Attention for Time Series Forecasting
MICA adds linearly scaling compressive cross-channel attention to Transformers, cutting average forecast error by 5.4% and ranking first among multivariate baselines.
-
A Foundation Model for Instruction-Conditioned In-Context Time Series Tasks
iAmTime is a time-series foundation model that uses instruction-conditioned in-context learning from demonstrations to perform zero-shot adaptation on forecasting, imputation, classification, and related tasks.
-
Forecasting Commencing Enrolments Under Data Sparsity: A Zero-Shot Time Series Foundation Models Framework for Higher Education Planning
Zero-shot TSFMs conditioned on leakage-safe covariates from Google Trends and an institutional index forecast commencing enrolments competitively with classical methods under data sparsity.
-
An AI system to help scientists write expert-level empirical software
ERA combines LLMs and tree search to produce expert-level empirical software that outperforms top human methods on single-cell analysis leaderboards and CDC COVID-19 forecasts.
-
An AI system to help scientists write expert-level empirical software
ERA is an AI system using LLMs and tree search to produce expert-level empirical software, generating methods that outperformed top human approaches in single-cell data analysis and COVID-19 forecasting tasks.
-
Benchmarking Deep Time Series Models for Equity Portfolios
Benchmark of 15 time-series architectures on equity portfolios finds no model dominates, with TransEnc-8 at 0.352 rank-1 acceptability and all promoted models showing negative net Sharpe at 20 bps costs under constraints.
-
Feature to Dynamics: Feature-space to Autoregression strategy for Zero-shot Time Series Forecasting
FSA learns a mapping from feature space to autoregressive strategy space to improve zero-shot univariate time series forecasting over Transformer baselines under matched pretraining conditions.
-
Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling
Falcon-X introduces a latent prototype space with Unified Prototype Diff-Attention and Latent Entity Attention for heterogeneous multivariate time series forecasting.
-
AION: Next-Generation Tasks and Practical Harness for Time Series
AION is a time series harness using agents, skills, rules, memory, evaluation, and protocols with temporal grounding, shown in a Kaggle Store Sales case study to produce more artifacts and reviews than direct agent use.
-
Investigating simple target-covariate relationships for Chronos-2 and TabPFN-TS
TabPFN-TS captures simple target-covariate relationships more effectively than Chronos-2 in controlled experiments, especially for short horizons.
-
TimeRFT: Stimulating Generalizable Time Series Forecasting for TSFMs via Reinforcement Finetuning
TimeRFT applies reinforcement learning with multi-faceted step-wise rewards and informative sample selection to improve generalization and accuracy in TSFM adaptation beyond supervised fine-tuning.
-
A Foundation Model for Instruction-Conditioned In-Context Time Series Tasks
iAmTime is a hierarchical transformer-based time series foundation model that uses semantic tokens and instruction-conditioned prompts to infer tasks from demonstrations, achieving improved zero-shot performance on fo...
-
Does Normalization Choice Matter for Causal Large Time-Series Models?
Normalization choice significantly influences training convergence and forecasting performance in causal large time-series models.
-
Assessing the Operational Viability of Foundation Models for Time Series Forecasting
Foundation models match or approach supervised performance in periodic and cold-start domains but lag in physically constrained systems, while a feature-based router improves accuracy and cuts inference cost versus al...
-
Don't Learn the Shape: Forecasting Periodic Time Series by Rank-1 Decomposition
A frozen average of the last two cycles matches or exceeds eight shape-learning alternatives on 97 GIFT-Eval configurations for periodic time series forecasting.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.