{"id":"608a43fd-a4e7-4edf-ab86-791823935a87","arxiv_id":"2412.14435","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Judiciously choosing just four datasets can make 46% of forecasting models appear best-in-class and 77% top-three, so dataset selection alone can distort reported performance.","lead":"This paper shows that in time series forecasting, selecting just four benchmark datasets can make 46% of models look like the best one and 77% look like top three. It quantifies how dataset choice alone distorts model rankings, and argues for more comprehensive evaluation frameworks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline percentages depend on an undefined subset-ranking rule and an undefined 'risk' quantity, and Table 1 lists 14 datasets while the text claims 13.","rationale":"The reader's weakest assumption was representativeness of the 13-dataset pool. That is a legitimate external-validity caveat. However, the more immediate threat to the central claim is internal: the paper does not specify how subset rankings are computed, the dataset count is inconsistent (13 vs 14), and the 40% risk-reduction claim in the abstract has no counterpart in the methods or results. If the subset aggregation is changed, percentages like 46% and 77% can shift; if the true N is 14 rather than 13, the subset search space changes; and the 40% figure may have no defined referent. These issues are fixable with clarifications and by pointing to the released code, which is why the conditional verdict remains appropriate rather than outright rejection. The qualitative conclusion that selective dataset choice can distort apparent rankings is supported by the rank distributions in Figure 1 and does not depend on the exact percentages. I therefore keep the reader's CONDITIONAL verdict but shift the focus of required revision to the internal protocol and the abstract's unsupported risk claim.","tokens_in":10749,"tokens_out":10573,"duration_ms":79711,"concrete_test":"Use the public repository to extract the per-dataset SMAPE matrix for all listed datasets and 13 models. Enumerate every subset of size n=3,4,6 and compute the rank of each model on each subset under two aggregation rules: (a) average SMAPE, (b) average per-dataset rank. Reproduce Figure 4 and the abstract's 46%/77% figures at n=4 under both rules, and the relative reduction in the top-1 share from n=3 to n=6. If the results differ between rules, or if the repo contains no computation of a 'risk' measure, the headline numbers are not protocol-independent and the 40% claim is unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims are not internally pinned down. First, the subset ranking function R(Mi, Dj) in Step 3 says only that rank is assigned 'based on SMAPE values across the dataset subset'; it does not state whether the subset score is mean SMAPE, mean rank, or some other aggregate. Because SMAPE scales differ across datasets, the resulting ranking, and hence every cherry-picking percentage, is protocol-dependent. Second, Section Experimental Setup says 'thirteen diverse benchmark datasets', but Table 1 enumerates 14 dataset entries (M3, M4, and Tourism are split by frequency). The subset combinatorics and achievable maxima depend on N, so the exact numbers are ambiguous. Third, the abstract's claim that increasing datasets from 3 to 6 reduces 'the risk of incorrectly identifying an algorithm as the best one by approximately 40%' is never defined or computed in the body; no quantity called 'risk' is introduced, and Figure 4 reports fractions of models that can be placed in top positions, not a misidentification probability under any sampling protocol. These are internal-validity problems: even within this benchmark pool the numbers are not uniquely determined by the text, and the abstract's strongest claim is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies dataset cherry-picking in time series forecasting evaluations. The authors propose a four-step framework: assemble a collection of benchmark datasets, select a diverse set of forecasting models, define a ranking function over dataset subsets, and compare subset-based rankings against a baseline ranking on the full collection. Applying this framework to 13 forecasting models and a collection of benchmark datasets, they report that with four selected datasets 46% of models could be made to appear best in class and 77% could appear in the top three, and that increasing the number of datasets from 3 to 6 reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%. The paper concludes that common small-N evaluations are highly susceptible to selection bias and calls for more comprehensive evaluation frameworks.","tokens_in":10950,"tokens_out":4659,"duration_ms":38880,"significance":"If the quantitative claims hold, the paper provides a timely and useful cautionary result for the time series forecasting community: typical evaluations on three to six datasets can support misleading claims of state-of-the-art performance for a large fraction of methods. The study is empirical and the main percentages are computed directly from the reported per-dataset rankings, rather than fitted to a predetermined conclusion, so the core analysis is not circular. The authors also make their code repository public, which strengthens reproducibility. The main value is in quantifying how widespread the cherry-picking problem can be, although the specific numbers are conditional on the chosen benchmark pool and on how the ranking function is defined.","major_comments":[{"comment":"The ranking function R(Mi, Dj) is underspecified. The text says that a rank is assigned 'based on its SMAPE values across the dataset subset Dj', but it does not state whether the subset score is the mean SMAPE across the datasets in Dj, the mean rank across datasets, or some other aggregate function. Since SMAPE scales differ substantially across datasets in Table 1 (e.g., hourly ETTh vs. daily M5), the resulting rankings, and therefore every headline percentage in Figures 3 and 4, are protocol-dependent until this aggregation is defined.","section":"Framework for Evaluating Cherry-Picking, Step 3"},{"comment":"The manuscript says 'thirteen diverse benchmark datasets', but Table 1 enumerates 14 rows (Labour; M3 Monthly, Quarterly, Yearly; M4 Monthly, Quarterly, Yearly; M5; Tourism Monthly, Quarterly; Traffic; Wiki2; ETTh1; ETTh2). The subset combinatorics and the achievable maxima in the empirical analysis depend on whether the frequency splits of M3, M4, and Tourism are treated as separate datasets or as a single dataset. The text must reconcile this discrepancy and state the effective N used in the computations.","section":"Experimental Setup, Datasets and Table 1"},{"comment":"The abstract's claim that increasing the number of datasets from 3 to 6 'reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%' is not defined or computed anywhere in the body. No quantity called 'risk' is introduced, and Figure 4 reports the fraction of models that can be placed in top positions by some cherry-picked subset, which is not a misidentification probability under any stated sampling protocol. Either define and compute this risk explicitly, or remove or rephrase the claim.","section":"Abstract and Results and Discussion, Figure 4"},{"comment":"The baseline ranking on the full collection is treated as the ground truth against which all cherry-picking distortion is measured. The specific percentages (46%, 77%, 40%) are therefore properties of the 14-row dataset pool used here, not universal facts about time series forecasting evaluation. The paper should explicitly scope the quantitative claims as conditional on this pool and its full-collection ranking, especially in the abstract and conclusions.","section":"Framework for Evaluating Cherry-Picking, Step 4"}],"minor_comments":[{"comment":"Equation (1) defines the seasonal period m, but m does not appear in the formula; the notation should be cleaned up so that n and m are both used or m is removed.","section":"Equation (1)"},{"comment":"Figure 2 contains many panels and red bars, but the caption does not explain whether the height or position of a bar encodes the rank, nor how the 'cherry-picked subset' is selected for each n; please clarify the visual encoding and the selection rule.","section":"Figure 2"},{"comment":"Figure 4 displays numerous percentages without a clear legend or axis mapping; the reader cannot easily tell which percentage corresponds to which n and which rank. Please add a legend or annotate the plotted points directly.","section":"Figure 4"},{"comment":"The extracted text contains a title typo ('Make Y our Model Shine') and some spacing artifacts; please check the camera-ready version for such issues.","section":"General presentation"},{"comment":"The paper states that all experiments are fully reproducible and gives a repository URL, but it does not describe the training configuration, hyperparameters, or computational resources used for the deep learning models; a brief reproducibility statement would help.","section":"Experimental Setup, Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile question and the empirical approach is not circular, but the central quantitative claims rest on an underspecified ranking aggregation and an undefined 'risk' measure. These are fixable within the paper's scope, so I recommend major revision rather than rejection. The 13-vs-14 dataset inconsistency also needs resolution before the numbers can be trusted. I would not raise concerns about the authors' motivations; the issue is purely one of internal specification and clarity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper gives concrete numbers on how much dataset choice alone can distort forecasting rankings. With 4 cherry-picked datasets, 46% of 13 methods can be presented as the best, and 77% as top-3. That's a striking, useful result for anyone who reviews forecasting papers, and it deserves a serious referee.\n\nWhat's actually new: earlier critiques (Zeng et al., Wu & Keogh, Musgrave et al.) pointed out specific evaluation flaws, but this one systematically enumerates dataset subsets and quantifies how many models can be made to look good. They ship code and data, which is real evidence. The finding that deep learning models are more sensitive to dataset selection than classical methods is plausible and consistent with Figure 1.\n\nThe soft spots are real, though. The ranking function R(Mi, Dj) is never pinned down. It says 'based on SMAPE values across the dataset subset', but is the subset score a mean SMAPE, a mean rank, a median? Since SMAPE scales differ across datasets, this choice can change the rankings and therefore every percentage in Figures 3 and 4. The paper needs to state the aggregation rule and ideally show sensitivity to it.\n\nSecond, the abstract's claim that going from 3 to 6 datasets reduces 'the risk of incorrectly identifying an algorithm as the best one by approximately 40%' is not supported anywhere in the body. No quantity called 'risk' is defined. Figure 4 reports the percentage of models that can be placed in top positions, which is an upper bound on manipulability, not a misidentification probability under any sampling scheme. The 3-vs-6 numbers in the text (54% vs 38% for top-1) would give a relative reduction of about 30%, not 40%, and an absolute drop of 16 points. The abstract overstates what was computed.\n\nThird, the text says 'thirteen benchmark datasets' but Table 1 lists fourteen rows when you count M3, M4, and Tourism by frequency. The combinatorics depend on N, so this needs to be resolved.\n\nFinally, the percentages are properties of this specific benchmark pool, not universal laws. That is an inherent limitation of a single-pool empirical study, and it should be stated clearly.\n\nWho this is for: people who design or review time series forecasting evaluations, and anyone working on benchmark design more broadly. It is a good discussion paper, but it needs a revision that defines R, defines or removes 'risk', fixes the dataset count, and tempers the abstract. I would send it to peer review rather than desk reject.\n\nRecommendation: send out, but flag these issues for major revision.","headline":"Useful warning about dataset cherry-picking in forecasting, but the headline numbers depend on an underspecified ranking rule and the '40% risk reduction' claim is not backed by the body.","tokens_in":11479,"tokens_out":4010,"would_cite":true,"duration_ms":27074,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By cherry-picking just four datasets, researchers can make 46% of forecasting models appear best in class, and 77% appear in the top three, the paper's experiment on 13 models and 13 benchmarks shows.","keywords":["time series forecasting","dataset selection bias","cherry-picking","model evaluation","SMAPE","benchmark datasets","deep learning","classical methods"],"falsifier":"Run the same 13 models on an independent collection of, say, 50 or more datasets spanning many domains. If the fraction of models that can be cherry-picked into the top-1 position with four datasets falls well below the 46% reported here — for instance, if no model can be made to look best unless it is genuinely best — then the paper's estimate of distortion does not generalize. If the fraction stays near 46% or higher, the claim that small dataset counts create widespread reporting bias is supported.","tokens_in":10550,"feed_emoji":"📈","tokens_out":8644,"duration_ms":62340,"temperature":0.7,"pith_summary":"This paper argues that dataset selection bias — cherry-picking — can seriously distort the reported performance of time series forecasting models. Using 13 benchmark datasets and 13 forecasting methods, the authors show that with just four datasets (a number common in published work) 46% of methods could be presented as the best, and 77% as top-three. They also find that increasing the number of datasets from three to six reduces the chance of wrongly declaring an algorithm the best by about 40%. The result matters because most forecasting papers evaluate on only three to six datasets, so many published claims of superiority may be artifacts of selection rather than genuine quality.","feed_headline":"Four datasets can make 46% of forecasters look best","feed_subtitle":"Four datasets let 77% of models appear in the top three; six datasets cut the error risk by 40%.","key_machinery":"The machinery is a ranking-based evaluation framework built on SMAPE. For any subset of datasets of size n, each model is ranked by its average SMAPE across those datasets; the baseline is the ranking on the full 13-dataset collection. The framework then finds, for each model, the dataset subset of size n on which that model achieves its best rank, which operationalizes cherry-picking. Comparing the best possible rank with the baseline rank across subset sizes quantifies how much distortion small dataset counts can create.","core_discovery":"The central discovery is that ranking-based conclusions in time series forecasting are highly sensitive to which datasets are included. On the full collection of 13 datasets, methods such as ETS and NHITS show the best median ranks, but most methods, especially deep learning ones like Informer and TCN, swing from rank 1 to rank 13 depending on the dataset. When the authors deliberately pick the four datasets that flatter each model, 46% of the 13 models can be reported as the single best method and 77% can be reported within the top three. The paper also reports that moving from three to six datasets lowers the risk of misidentifying the best algorithm by approximately 40%, and that classical methods are more stable to dataset selection than deep learning approaches.","pith_inferences":["The 46% and 77% figures likely underestimate the true selection leverage, because real papers also choose the evaluation metric, the baseline set, and the hyperparameters, each of which adds more degrees of freedom beyond dataset choice.","The same framework could be applied to other degrees of freedom — metric choice, forecast horizon, or preprocessing — to map out the full space of reporting bias in forecasting papers.","If the 13 datasets are not representative of the broader benchmark landscape, the specific percentages may not transfer, but the qualitative pattern — small dataset counts create strong selection leverage — can be tested on larger independent benchmark pools.","The reported 40% risk reduction from 3 to 6 datasets suggests a concrete design rule for the field: require at least six datasets in forecasting evaluations to keep cherry-picking distortion within tolerable bounds."],"forward_implications":["Published comparisons that use only three to six datasets may present many models as top-tier simply because of dataset choice, so readers should treat 'best' claims from small-N studies with caution.","Increasing the number of evaluation datasets, at least from three to six, substantially reduces the chance of misidentifying the best algorithm, suggesting that evaluation frameworks should demand more than a handful of benchmarks.","Deep learning-based forecasting methods are more vulnerable to dataset-selection bias than classical methods, so their reported advantages need closer scrutiny relative to the datasets chosen.","The proposed framework gives researchers a way to measure the cherry-picking risk of any candidate set of datasets before publishing a comparison.","A model like ETS that ranks consistently well across many datasets is a more trustworthy baseline than a high-variance model even when their median ranks are similar."],"supporting_citations":[{"why":"Establishes that widely used time-series anomaly detection benchmarks are flawed, motivating the concern that benchmark selection distorts apparent progress.","marker":"Wu and Keogh 2023"},{"why":"Shows simple linear models outperform Transformer-based forecasters on standard benchmarks, illustrating that experimental setup can drive reported results.","marker":"Zeng et al. 2022"},{"why":"DeepAR is cited as a model evaluated on only three public datasets, providing evidence that small-N evaluation is common practice.","marker":"Salinas, Flunkert, and Gasthaus 2019"},{"why":"Supplies the M4 competition datasets and the SMAPE metric used in the experiments.","marker":"Makridakis, Spiliotis, and Assimakopoulos 2018"},{"why":"NHITS, one of the deep learning methods in the study, and an example of the typical dataset selection in long-horizon forecasting papers.","marker":"Challu et al. 2023"},{"why":"Provides recommendations on forecast evaluation metrics, supporting the paper's use of SMAPE.","marker":"Hewamalage, Ackermann, and Bergmeir 2023"},{"why":"Informer model and ETT datasets, used as one of the high-variance deep learning methods.","marker":"Zhou et al. 2021"}],"fun_headline_variants":["46% of forecasters 'best' with 4 cherry-picked datasets","Pick 4 datasets, 77% of models crack top 3","Six datasets cut false 'best model' risk by 40%","Deep learning forecasters most skewed by dataset picks","Dataset cherry-picking inflates forecast rankings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole analysis treats the collection of 13 datasets as the true universe of forecasting problems, so all cherry-picking distortion is measured against the ranking computed on that collection; if those 13 datasets are not representative of real-world benchmarks, the specific percentages and the dataset-count sensitivity curve apply only to this pool.","fun_headline_variants_meta":{"raw":{"variants":["46% of forecasters 'best' with 4 cherry-picked datasets","Pick 4 datasets, 77% of models crack top 3","Six datasets cut false 'best model' risk by 40%","Deep learning forecasters most skewed by dataset picks","Dataset cherry-picking inflates forecast rankings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1551,"prompt_tokens":958,"completion_tokens":593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":508}},"tokens_in":574,"tokens_out":593,"duration_ms":5343,"temperature":1.0,"reasoning_tokens":508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:14:34.561983+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 13 models on an independent collection of, say, 50 or more datasets spanning many domains. If the fraction of models that can be cherry-picked into the top-1 position with four datasets falls well below the 46% reported here — for instance, if no model can be made to look best unless it is genuinely best — then the paper's estimate of distortion does not generalize. If the fraction stays near 46% or higher, the claim that small dataset counts create widespread reporting bias is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the M4 competition datasets and the SMAPE metric used in the experiments."},{"cited_title":"G.; Oreshkin, B","cited_arxiv_id":null,"evidence_quote":"NHITS, one of the deep learning methods in the study, and an example of the typical dataset selection in long-horizon forecasting papers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides recommendations on forecast evaluation metrics, supporting the paper's use of SMAPE."}],"review_version":1}