TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:MOJA3QA7record.jsonopen to challenge →
read the original abstract
Time series are generated in diverse domains such as economic, traffic, health, and energy, where forecasting of future values has numerous important applications. Not surprisingly, many forecasting methods are being proposed. To ensure progress, it is essential to be able to study and compare such methods empirically in a comprehensive and reliable manner. To achieve this, we propose TFB, an automated benchmark for Time Series Forecasting (TSF) methods. TFB advances the state-of-the-art by addressing shortcomings related to datasets, comparison methods, and evaluation pipelines: 1) insufficient coverage of data domains, 2) stereotype bias against traditional methods, and 3) inconsistent and inflexible pipelines. To achieve better domain coverage, we include datasets from 10 different domains: traffic, electricity, energy, the environment, nature, economic, stock markets, banking, health, and the web. We also provide a time series characterization to ensure that the selected datasets are comprehensive. To remove biases against some methods, we include a diverse range of methods, including statistical learning, machine learning, and deep learning methods, and we also support a variety of evaluation strategies and metrics to ensure a more comprehensive evaluations of different methods. To support the integration of different methods into the benchmark and enable fair comparisons, TFB features a flexible and scalable pipeline that eliminates biases. Next, we employ TFB to perform a thorough evaluation of 21 Univariate Time Series Forecasting (UTSF) methods on 8,068 univariate time series and 14 Multivariate Time Series Forecasting (MTSF) methods on 25 datasets. The benchmark code and data are available at https://github.com/decisionintelligence/TFB. We have also launched an online time series leaderboard: https://decisionintelligence.github.io/OpenTS/OpenTS-Bench/.
This paper has not been read by Pith yet.
Forward citations
Cited by 8 Pith papers
-
Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty
MSE-optimal multi-step forecasters cannot match the marginal distribution of realizations under nonzero conditional uncertainty, creating a quantifiable accuracy-realism Pareto frontier across benchmarks.
-
TempusBench: An Evaluation Framework for Time-Series Forecasting
TempusBench is a new evaluation framework for time-series forecasting models that supplies fresh non-overlapping datasets, tasks beyond horizon and domain, consistent tuning across models, and visualization tools.
-
Deep Time Series Models: A Comprehensive Survey and Benchmark
This survey and benchmark of deep time series models using the released TSLib library finds that models with specific structures perform well only on distinct analysis tasks.
-
How Good Can Linear Models Be for Time-Series Forecasting?
Tuned Ridge regression with series-specific preprocessing beats prior linear forecasters and complex neural baselines on most dataset-horizon pairs across eight benchmarks.
-
How Good Can Linear Models Be for Time-Series Forecasting?
Optimized Ridge regression with series-specific preprocessing beats prior linear forecasters and exceeds Transformer, MLP, and CNN baselines on six of eight time-series benchmarks.
-
NPMixer: Hierarchical Neighboring Patch Mixing for Time Series Forecasting
NPMixer improves multivariate time series forecasting accuracy by combining a data-adaptive wavelet decomposition with hierarchical neighboring patch mixing via MLPs and channel mixing on high-frequency components.
-
AlphaCast: A Human Wisdom-LLM Intelligence Co-Reasoning Framework for Interactive Time Series Forecasting
AlphaCast is a training-free LLM framework that performs interactive multi-stage reasoning for time series forecasting by integrating feature extraction, knowledge bases, case libraries, and contextual pools.
-
PMDformer: Patch-Mean Decoupling Information Transformer for Long-term Forecasting
PMDformer uses patch-mean decoupling, trend restoration attention, and proximal variable attention to improve accuracy and stability in long-term time series forecasting benchmarks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.