Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By cherry-picking just four datasets, researchers can make 46% of forecasting models appear best in class, and 77% appear in the top three, the paper's experiment on 13 models and 13 benchmarks shows.

desk verdict Useful warning about dataset cherry-picking in forecasting, but the headline numbers depend on an underspecified ranking rule and the '40% risk reduction' claim is not backed by the body. read the letter →

arxiv 2412.14435 v1 pith:UVSR75UI submitted 2024-12-19 cs.LG cs.AI

Luis Roque , Carlos Soares , Vitor Cerqueira , Luis Torgo This is my paper · ORCID
classification cs.LGcs.AI
keywords timeseriesforecastingdatasetselectionbiascherry-pickingmodelevaluationSMAPEbenchmarkdatasetsdeeplearningclassicalmethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that dataset selection bias — cherry-picking — can seriously distort the reported performance of time series forecasting models. Using 13 benchmark datasets and 13 forecasting methods, the authors show that with just four datasets (a number common in published work) 46% of methods could be presented as the best, and 77% as top-three. They also find that increasing the number of datasets from three to six reduces the chance of wrongly declaring an algorithm the best by about 40%. The result matters because most forecasting papers evaluate on only three to six datasets, so many published claims of superiority may be artifacts of selection rather than genuine quality.

What carries the argument

The machinery is a ranking-based evaluation framework built on SMAPE. For any subset of datasets of size n, each model is ranked by its average SMAPE across those datasets; the baseline is the ranking on the full 13-dataset collection. The framework then finds, for each model, the dataset subset of size n on which that model achieves its best rank, which operationalizes cherry-picking. Comparing the best possible rank with the baseline rank across subset sizes quantifies how much distortion small dataset counts can create.

What would settle it

Run the same 13 models on an independent collection of, say, 50 or more datasets spanning many domains. If the fraction of models that can be cherry-picked into the top-1 position with four datasets falls well below the 46% reported here — for instance, if no model can be made to look best unless it is genuinely best — then the paper's estimate of distortion does not generalize. If the fraction stays near 46% or higher, the claim that small dataset counts create widespread reporting bias is supported.

Watch

Extended reading notes

Core claim

The central discovery is that ranking-based conclusions in time series forecasting are highly sensitive to which datasets are included. On the full collection of 13 datasets, methods such as ETS and NHITS show the best median ranks, but most methods, especially deep learning ones like Informer and TCN, swing from rank 1 to rank 13 depending on the dataset. When the authors deliberately pick the four datasets that flatter each model, 46% of the 13 models can be reported as the single best method and 77% can be reported within the top three. The paper also reports that moving from three to six datasets lowers the risk of misidentifying the best algorithm by approximately 40%, and that classical methods are more stable to dataset selection than deep learning approaches.

Load-bearing premise

The whole analysis treats the collection of 13 datasets as the true universe of forecasting problems, so all cherry-picking distortion is measured against the ranking computed on that collection; if those 13 datasets are not representative of real-world benchmarks, the specific percentages and the dataset-count sensitivity curve apply only to this pool.

Editorial extensions

If this is right

  • Published comparisons that use only three to six datasets may present many models as top-tier simply because of dataset choice, so readers should treat 'best' claims from small-N studies with caution.
  • Increasing the number of evaluation datasets, at least from three to six, substantially reduces the chance of misidentifying the best algorithm, suggesting that evaluation frameworks should demand more than a handful of benchmarks.
  • Deep learning-based forecasting methods are more vulnerable to dataset-selection bias than classical methods, so their reported advantages need closer scrutiny relative to the datasets chosen.
  • The proposed framework gives researchers a way to measure the cherry-picking risk of any candidate set of datasets before publishing a comparison.
  • A model like ETS that ranks consistently well across many datasets is a more trustworthy baseline than a high-variance model even when their median ranks are similar.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 46% and 77% figures likely underestimate the true selection leverage, because real papers also choose the evaluation metric, the baseline set, and the hyperparameters, each of which adds more degrees of freedom beyond dataset choice.
  • The same framework could be applied to other degrees of freedom — metric choice, forecast horizon, or preprocessing — to map out the full space of reporting bias in forecasting papers.
  • If the 13 datasets are not representative of the broader benchmark landscape, the specific percentages may not transfer, but the qualitative pattern — small dataset counts create strong selection leverage — can be tested on larger independent benchmark pools.
  • The reported 40% risk reduction from 3 to 6 datasets suggests a concrete design rule for the field: require at least six datasets in forecasting evaluations to keep cherry-picking distortion within tolerable bounds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies dataset cherry-picking in time series forecasting evaluations. The authors propose a four-step framework: assemble a collection of benchmark datasets, select a diverse set of forecasting models, define a ranking function over dataset subsets, and compare subset-based rankings against a baseline ranking on the full collection. Applying this framework to 13 forecasting models and a collection of benchmark datasets, they report that with four selected datasets 46% of models could be made to appear best in class and 77% could appear in the top three, and that increasing the number of datasets from 3 to 6 reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%. The paper concludes that common small-N evaluations are highly susceptible to selection bias and calls for more comprehensive evaluation frameworks.

Significance. If the quantitative claims hold, the paper provides a timely and useful cautionary result for the time series forecasting community: typical evaluations on three to six datasets can support misleading claims of state-of-the-art performance for a large fraction of methods. The study is empirical and the main percentages are computed directly from the reported per-dataset rankings, rather than fitted to a predetermined conclusion, so the core analysis is not circular. The authors also make their code repository public, which strengthens reproducibility. The main value is in quantifying how widespread the cherry-picking problem can be, although the specific numbers are conditional on the chosen benchmark pool and on how the ranking function is defined.

major comments (4)
  1. [Framework for Evaluating Cherry-Picking, Step 3] The ranking function R(Mi, Dj) is underspecified. The text says that a rank is assigned 'based on its SMAPE values across the dataset subset Dj', but it does not state whether the subset score is the mean SMAPE across the datasets in Dj, the mean rank across datasets, or some other aggregate function. Since SMAPE scales differ substantially across datasets in Table 1 (e.g., hourly ETTh vs. daily M5), the resulting rankings, and therefore every headline percentage in Figures 3 and 4, are protocol-dependent until this aggregation is defined.
  2. [Experimental Setup, Datasets and Table 1] The manuscript says 'thirteen diverse benchmark datasets', but Table 1 enumerates 14 rows (Labour; M3 Monthly, Quarterly, Yearly; M4 Monthly, Quarterly, Yearly; M5; Tourism Monthly, Quarterly; Traffic; Wiki2; ETTh1; ETTh2). The subset combinatorics and the achievable maxima in the empirical analysis depend on whether the frequency splits of M3, M4, and Tourism are treated as separate datasets or as a single dataset. The text must reconcile this discrepancy and state the effective N used in the computations.
  3. [Abstract and Results and Discussion, Figure 4] The abstract's claim that increasing the number of datasets from 3 to 6 'reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%' is not defined or computed anywhere in the body. No quantity called 'risk' is introduced, and Figure 4 reports the fraction of models that can be placed in top positions by some cherry-picked subset, which is not a misidentification probability under any stated sampling protocol. Either define and compute this risk explicitly, or remove or rephrase the claim.
  4. [Framework for Evaluating Cherry-Picking, Step 4] The baseline ranking on the full collection is treated as the ground truth against which all cherry-picking distortion is measured. The specific percentages (46%, 77%, 40%) are therefore properties of the 14-row dataset pool used here, not universal facts about time series forecasting evaluation. The paper should explicitly scope the quantitative claims as conditional on this pool and its full-collection ranking, especially in the abstract and conclusions.
minor comments (5)
  1. [Equation (1)] Equation (1) defines the seasonal period m, but m does not appear in the formula; the notation should be cleaned up so that n and m are both used or m is removed.
  2. [Figure 2] Figure 2 contains many panels and red bars, but the caption does not explain whether the height or position of a bar encodes the rank, nor how the 'cherry-picked subset' is selected for each n; please clarify the visual encoding and the selection rule.
  3. [Figure 4] Figure 4 displays numerous percentages without a clear legend or axis mapping; the reader cannot easily tell which percentage corresponds to which n and which rank. Please add a legend or annotate the plotted points directly.
  4. [General presentation] The extracted text contains a title typo ('Make Y our Model Shine') and some spacing artifacts; please check the camera-ready version for such issues.
  5. [Experimental Setup, Reproducibility] The paper states that all experiments are fully reproducible and gives a repository URL, but it does not describe the training configuration, hyperparameters, or computational resources used for the deep learning models; a brief reproducibility statement would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline percentages are direct empirical counts over computed SMAPE rankings, not fitted to or defined by the conclusion.

full rationale

The paper's derivation chain is empirical and self-contained. The SMAPE values for each of the thirteen models on each dataset are external inputs; the ranking function R(Mi, Dj) is defined directly from those SMAPE values; and the headline percentages (46% best in class, 77% top three, and the sensitivity curve in Figure 4) are exhaustive combinatorial counts over dataset subsets, not quantities fitted to match the paper's conclusions. The 'baseline ranking' on the full collection is a definitional reference point used to measure deviation, not a parameter fitted from the target result; calling full-collection ranking the reference is a modeling choice, not circular reasoning. The only self-citation, to Roque, Soares, and Torgo (2024), appears in the Background as an example of a related robustness study and is not load-bearing for the cherry-picking framework or for any quantitative claim. The skeptical concerns about an underspecified subset-ranking aggregation rule, the 13-vs-14 dataset count, and the undefined 'risk' underlying the 40% abstract claim are internal-validity or reporting issues, not circularity: even if those numbers are ambiguous or unsupported, they are not equivalent by construction to the inputs. No step in the paper reduces a prediction to a fitted input or imports a uniqueness conclusion from the authors' own prior work.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted in this meta-analysis. The paper's assumptions are about representativeness of the benchmark pool, the definition of a baseline ranking, the choice of SMAPE as the evaluation metric, and fair model tuning. No new entities or mechanisms are introduced.

assumptions (4)
  • domain assumption The 13-dataset collection is representative of the broader time series forecasting benchmark landscape.
    All cherry-picking distortion is measured against the ranking on this full collection. If the collection is not representative, the quantified percentages are pool-specific. Invoked in Step 1 (dataset selection) and Table 1.
  • domain assumption The baseline ranking on the full collection defines true model performance.
    Cherry-picking is defined as selecting subsets that improve a model's rank relative to this baseline. This is a definitional choice without external ground truth. Introduced in Step 4.
  • domain assumption SMAPE is a valid metric for comparing forecasting accuracy across heterogeneous datasets.
    All rankings are based on SMAPE, following prior practice but not compared to alternative metrics such as MASE or RMSE. Invoked in Evaluation Metrics and Step 3.
  • domain assumption All 13 models are evaluated fairly and consistently, with no tuning advantage for any method.
    Model configurations and tuning effort are not reported. The robustness contrast between classical and deep learning models could be affected by default hyperparameters. This assumption underlies the sensitivity comparison in the results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine." pith.science (2026). https://pith.science/paper/UVSR75UI

@misc{pith2026241214435,
  author       = {Pith},
  title        = {Pith review of: Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UVSR75UI}},
  note         = {Machine review of arXiv:2412.14435}
}
read the original abstract

The importance of time series forecasting drives continuous research and the development of new approaches to tackle this problem. Typically, these methods are introduced through empirical studies that frequently claim superior accuracy for the proposed approaches. Nevertheless, concerns are rising about the reliability and generalizability of these results due to limitations in experimental setups. This paper addresses a critical limitation: the number and representativeness of the datasets used. We investigate the impact of dataset selection bias, particularly the practice of cherry-picking datasets, on the performance evaluation of forecasting methods. Through empirical analysis with a diverse set of benchmark datasets, our findings reveal that cherry-picking datasets can significantly distort the perceived performance of methods, often exaggerating their effectiveness. Furthermore, our results demonstrate that by selectively choosing just four datasets - what most studies report - 46% of methods could be deemed best in class, and 77% could rank within the top three. Additionally, recent deep learning-based approaches show high sensitivity to dataset selection, whereas classical methods exhibit greater robustness. Finally, our results indicate that, when empirically validating forecasting algorithms on a subset of the benchmarks, increasing the number of datasets tested from 3 to 6 reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%. Our study highlights the critical need for comprehensive evaluation frameworks that more accurately reflect real-world scenarios. Adopting such frameworks will ensure the development of robust and reliable forecasting methods.

Figures

Figures reproduced from arXiv: 2412.14435 by the authors.

Figure 1
Figure 1. Rank distribution of various forecasting models [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 3
Figure 3. Percentage of models that could be reported as top [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 2
Figure 2. Impact of cherry-picking on the rankings of NHITS (left), Informer (center), and TCN (right). Each subfigure (1 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Breakdown of the percentages for top 1, 2, and 3 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 19 canonical work pages

  1. [1]

    G.; Moffat, A.; Webber, W.; and Zobel, J

    Armstrong, T. G.; Moffat, A.; Webber, W.; and Zobel, J. 2009. Improvements that don't add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM '09, 601–610. New York, NY, USA: Association for Computing Machinery. ISBN 9781605585123

    work page 2009

  2. [2]

    Assimakopoulos, V.; and Nikolopoulos, K. 2000. The theta model: a decomposition approach to forecasting. International journal of forecasting, 16(4): 521--530

    work page 2000

  3. [3]

    J.; Song, H.; and Wu, D

    Athanasopoulos, G.; Hyndman, R. J.; Song, H.; and Wu, D. C. 2011. The tourism forecasting competition. International Journal of Forecasting, 27(3): 822--844

    2011

  4. [4]

    Bontempi, G.; Ben Taieb, S.; and Le Borgne, Y.-A. 2013. Machine learning strategies for time series forecasting. Business Intelligence: Second European Summer School, eBISS 2012, Brussels, Belgium, July 15-21, 2012, Tutorial Lectures 2, 62--77

    work page 2013

  5. [5]

    G.; Oreshkin, B

    Challu, C.; Olivares, K. G.; Oreshkin, B. N.; Ramirez, F. G.; Canseco, M. M.; and Dubrawski, A. 2023. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 6989--6997

    work page 2023

  6. [6]

    Das, A.; Kong, W.; Leach, A.; Mathur, S.; Sen, R.; and Yu, R. 2024. Long-term Forecasting with TiDE: Time-series Dense Encoder. arXiv:2304.08424

    arXiv 2024

  7. [7]

    Elman, J. L. 1990. Finding structure in time. Cognitive Science, 14(2): 179--211

    work page 1990

  8. [8]

    Gardner Jr, E. S. 1985. Exponential smoothing: The state of the art. Journal of forecasting, 4(1): 1--28

    1985

Show all 33 references
  1. [9]

    I.; Smyl, S.; and Bergmeir, C

    Godahewa, R.; Bandara, K.; Webb, G. I.; Smyl, S.; and Bergmeir, C. 2021. Ensembles of localised models for time series forecasting. Knowledge-Based Systems, 233: 107518

    2021

  2. [10]

    Hewamalage, H.; Ackermann, K.; and Bergmeir, C. 2023. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery, 37(2): 788--832

    2023

  3. [11]

    B.; Ord, J

    Hyndman, R.; Koehler, A. B.; Ord, J. K.; and Snyder, R. D. 2008. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media

    2008

  4. [12]

    J.; and Athanasopoulos, G

    Hyndman, R. J.; and Athanasopoulos, G. 2018. Forecasting: principles and practice. OTexts

    2018

  5. [13]

    J.; and Khandakar, Y

    Hyndman, R. J.; and Khandakar, Y. 2008. Automatic time series forecasting: the forecast package for R. Journal of statistical software, 27: 1--22

    2008

  6. [14]

    Januschowski, T.; Gasthaus, J.; Wang, Y.; Salinas, D.; Flunkert, V.; Bohlke-Schneider, M.; and Callot, L. 2020. Criteria for classifying forecasting methods. International Journal of Forecasting, 36(1): 167--177

    2020

  7. [15]

    Lea, C.; Vidal, R.; Reiter, A.; and Hager, G. D. 2016. Temporal Convolutional Networks: A Unified Approach to Action Segmentation. arXiv:1608.08242

    2016 arXiv

  8. [16]

    Makridakis, S.; and Hibon, M. 2000. The M3-Competition: results, conclusions and implications. International journal of forecasting, 16(4): 451--476

    2000

  9. [17]

    Makridakis, S.; Spiliotis, E.; and Assimakopoulos, V. 2018. The M4 Competition: Results, findings, conclusion and way forward. International Journal of forecasting, 34(4): 802--808

    2018

  10. [18]

    Makridakis, S.; Spiliotis, E.; and Assimakopoulos, V. 2022 a . M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting

    2022

  11. [19]

    Makridakis, S.; Spiliotis, E.; and Assimakopoulos, V. 2022 b . M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4): 1346--1364. Special Issue: M5 competition

    2022

  12. [20]

    Musgrave, K.; Belongie, S.; and Lim, S.-N. 2020. A Metric Learning Reality Check. arXiv:2003.08505

    2020 arXiv

  13. [21]

    G.; Meetei, O

    Olivares, K. G.; Meetei, O. N.; Ma, R.; Reddy, R.; Cao, M.; and Dicker, L. 2024. Probabilistic hierarchical forecasting with deep Poisson mixtures. International Journal of Forecasting, 40(2): 470--489

    2024

  14. [22]

    N.; Carpov, D.; Chapados, N.; and Bengio, Y

    Oreshkin, B. N.; Carpov, D.; Chapados, N.; and Bengio, Y. 2020. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv:1905.10437

    2020 arXiv

  15. [23]

    S.; Werner, L

    Rangapuram, S. S.; Werner, L. D.; Benidis, K.; Mercado, P.; Gasthaus, J.; and Januschowski, T. 2021. End-to-End Learning of Coherent Probabilistic Forecasts for Hierarchical Time Series. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Mac...

    2021

  16. [24]

    Roque, L.; Soares, C.; and Torgo, L. 2024. RHiOTS: A Framework for Evaluating Hierarchical Time Series Forecasting Algorithms. arXiv preprint arXiv:2408.03399

    2024 arXiv

  17. [25]

    Salinas, D.; Flunkert, V.; and Gasthaus, J. 2019. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. arXiv:1704.04110

    2019 arXiv

  18. [26]

    D.; Ord, J

    Snyder, R. D.; Ord, J. K.; and Beaumont, A. 2012. Forecasting the intermittent demand for slow-moving inventories: A modelling approach. International Journal of Forecasting, 28(2): 485--496

    2012

  19. [27]

    Trindade, A. 2015. ElectricityLoadDiagrams20112014 . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C58C86

    2015 doi

  20. [28]

    van den Oord, A.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals, O.; Graves, A.; Kalchbrenner, N.; Senior, A.; and Kavukcuoglu, K. 2016. WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499

    2016 arXiv

  21. [29]

    Wu, R.; and Keogh, E. J. 2023. Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress. IEEE Transactions on Knowledge and Data Engineering, 35(3): 2421--2429

    2023

  22. [30]

    Zeng, A.; Chen, M.; Zhang, L.; and Xu, Q. 2022. Are Transformers Effective for Time Series Forecasting? arXiv:2205.13504

    2022 arXiv

  23. [31]

    Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; and Zhang, W. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. arXiv:2012.07436

    2021 arXiv

  24. [32]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.