REVIEW 4 major objections 5 minor 1 cited by
Stratify: Unifying Multi-Step Forecasting Strategies
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A unified framework for multi-step forecasting strategies reveals that newly discovered strategy combinations beat all existing strategies in over 84% of 1,080 experiments.
desk verdict Stratify is a genuinely useful unification of multi-step forecasting strategies, but the headline 84% claim rests on a benchmark that only evaluates the full strategy space for one of five model classes, and on test-set selection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Stratify plane. For a horizon H, the framework takes the set of base strategies and the set of rectifier strategies, each drawn from the families RecMO, DirMO, and DirRecMO with parameter σ chosen from the divisors of H; σ is expressed as a percentage of H so that comparisons across horizons are fair. A strategy is the pair (base, rectifier): the base model produces a first forecast, a second model forecasts the residuals of that forecast, and the final prediction is the sum. All previously known strategies appear as special cases—for example, σ=1 recovers the original Rectify strategy and σ=H recovers MIMO-like behavior—which is what makes the new combinations genuinely new rather than a separate class.
What would settle it
Run the full Stratify search, including DirMO and DirRecMO rectifiers, for LSTM on the same 18 datasets and horizons; if the best full-plane strategy fails to beat the best existing strategy in roughly 84% of cases, the generality of the headline claim for non-MLP models is not supported.
Extended reading notes
Core claim
The paper's central claim is that every existing multi-step forecasting strategy—recursive, direct, DirRec, Rectify, MIMO, and their multi-output variants—occupies a single point in a two-dimensional space whose axes are the base strategy and the rectifying strategy, each chosen from the same three families and each parameterized by the fraction of the horizon produced per step. The newly accessible points in this space are genuine strategies rather than interpolations, and the best new point outperforms the best previously known strategy in over 84% of 1,080 experiments, with reported mean error reductions between 5% and 25% depending on function class. The same experiments show that no single strategy wins everywhere, so the value of the framework is that it turns strategy selection into a finite, visualizable search.
Load-bearing premise
The headline 'over 84%' rests on treating the RecMO-RecMO subregion, the only part of the Stratify space fully evaluated for the random forest, RNN, LSTM, and Transformer models, as representative of the whole framework for those model classes.
Editorial extensions
If this is right
- Practitioners can replace ad-hoc strategy selection with a finite search over the Stratify plane, and the paper's heat maps indicate that a small number of sampled points often suffices to locate a strong strategy.
- The best strategy remains task-dependent, so deploying Stratify means treating strategy choice as a hyperparameter rather than a fixed modeling assumption.
- All existing strategies remain accessible inside the framework, so adopting Stratify cannot make a practitioner worse off in terms of the strategy set they can express.
- Reported mean error reductions of 5–25% across function classes imply that the choice of strategy can matter as much as the choice of model architecture in multi-step forecasting.
Reading between the lines
- If the unexplored regions of the Stratify plane behave like the RecMO-RecMO region for RNN, LSTM, Transformer, and random forest models, the headline improvement is likely a lower bound; direct-based and DirRec-based rectifiers could yield further gains that were only measured for MLP.
- The percentage parameterization suggests a transfer test the paper does not run: a strategy identified as optimal at one horizon may remain near-optimal at other horizons on the same dataset, which would let practitioners search at short horizons and deploy at long ones.
- The smoothness of the plane invites a meta-learning extension where dataset features predict the best region, potentially removing the need to train every strategy in the space.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Stratify, a parameterised framework for multi-step forecasting (MSF) that represents a forecasting strategy as a pair consisting of a base strategy and a residual-rectifying strategy, each chosen from RecMO, DirMO, or DirRec and parameterised as a percentage of the forecast horizon. The framework is claimed to unify all existing MSF strategies while also defining novel ones. The authors evaluate the full Stratify space for MLP and a restricted RecMO-RecMO subregion for RF, RNN, LSTM, and Transformer on 18 datasets with horizons 10, 20, 40, and 80, and report that novel strategies outperform all existing strategies in over 84% of 1080 experiments, with 5–25% error reductions across function classes.
Significance. If the central claim is valid, Stratify would provide a useful unifying vocabulary for MSF strategies and a practical tool for discovering better-performing strategies, backed by a large benchmark suite and released code. The paper's breadth—18 datasets, five function classes, four horizons, and reproducible code—is a genuine strength. However, the headline quantitative claim rests on (a) selecting the best strategy on the test set and (b) comparing against an incomplete set of existing strategies for four of the five function classes. These are load-bearing for the abstract's 'over 84%' statement, so the significance of the empirical contribution is currently not established at the level claimed, although the framework itself remains conceptually interesting.
major comments (4)
- [Section 3.2, Table 3] The reported improvements are based on selecting, for each task, the lowest-MSE strategy from the set of novel strategies and dividing by the lowest-MSE strategy from the set of existing strategies, with MSE computed on the test set. This is an in-sample selection on the test set and is subject to winner's curse: even if no novel strategy is truly better than the best existing one, the minimum over many candidates will often appear better. The 'over 84% of 1080 experiments' claim in the abstract and Section 4 is therefore a statement about an oracle that knows test outcomes, not about a reproducible model-selection procedure. Please report results where the best strategy is selected on the validation set (or via nested cross-validation) and then evaluated on the test set, and also report the performance of a fixed strategy (e.g., the single best novel strategy overall) to quantify what a practitioner would actually obtain.
- [Section 3.1 and Table 5a] For RF, RNN, LSTM, and Transformer, the paper states that only the Stratify plane in the RecMO-RecMO region was evaluated. Yet Table 3 and Table 5a compare the best novel strategy in that subregion against the 'lowest MSE of existing strategies,' and the abstract claims improvement 'compared to all existing ones.' Existing strategies such as Direct (Eq. 4), DirRec (Eq. 5), DirMO (Eq. 10), DirRecMO (Eq. 11), Rectify, and RectifyMO all lie outside the RecMO-RecMO region and were not evaluated for these four function classes. The comparison is therefore restricted to RecMO-family baselines (recursive, RecMO variants, and MIMO at σ=H), so the headline claim is unsupported for four of the five function classes. At minimum, the full Stratify plane should be evaluated for at least one non-MLP class (or on a subset of datasets/horizons), or the claims should be explicitly restricted to MLP and the RecMO-RecMO subregion.
- [Figure 8 and Section 3.1] The MLP results in Figure 8 demonstrate that the best-performing strategies lie outside the RecMO-RecMO region (e.g., a δ:50% base with δ rectifiers, with ranking errors around 30 versus 40–60 in the RecMO-RecMO region). This shows that the RecMO-RecMO restriction is not a harmless simplification for the other function classes; it may systematically exclude the best regions of the strategy space. The computational justification (Section 3.1: 'we would need to train over 86,400 transformers') explains the restriction but does not support generalising the MLP-based full-plane findings to the other classes. The paper should either provide full-plane evidence for at least one additional function class or substantially weaken the generality of the 'novel strategies consistently outperform' claim.
- [Section 2.4.1] The theoretical consideration that 'using an unbiased strategy as the base makes a Stratify strategy unbiased for any rectifier selected' is stated without formal derivation or assumptions about finite-sample behaviour, convergence rates, or the effect of estimating the rectifier on residuals. This is not load-bearing for the empirical contribution, but if the claim is intended as a theoretical contribution, it needs a precise statement and proof; otherwise it should be presented as a heuristic remark.
minor comments (5)
- [Abstract and Section 4] The abstract and Section 4 report 'over 84% of 1080 experiments,' but the means in Table 5a (0.70, 0.92, 0.97, 0.91, 0.90) average to about 88%. Please reconcile these numbers or clarify how the 84% figure is computed.
- [Throughout] There are inconsistent spellings and capitalisations, e.g., 'parralel' (Section 2.2), 'DIRMO' vs 'DirMO', 'DIRRECMO' vs 'DirRecMO', and 'T able' in several table captions. Please proofread.
- [Table 3] Several entries show a standard error larger than the reported ratio, e.g., ExchangeRate RNN 0.59 ± 1.35. Such entries indicate high instability across seeds/horizons; consider reporting medians or interquartile ranges, and discuss the most unstable cases explicitly.
- [Section 3.2.1] The sentence 'With 95% confidence, we find that the ten novel strategies shown outperform 60% of existing strategies' mixes a statement about individual strategies with a statement about the proportion of strategies; please clarify whether 'outperform' refers to significant pairwise differences under the Nemenyi test and what '60% of existing strategies' means exactly.
- [Section 2.3] In the sentence 'The resulting forecast for is the summation of...', the word 'for' appears to be a leftover artifact; please revise for readability.
Circularity Check
Central improvement claim is a test-set minimum ratio, not an out-of-sample prediction; partial circularity.
-
fitted input called prediction
[Section 3.2, Table 3 caption and Table 5a]
"For each dataset and function family, we take the lowest MSE of a novel Stratify strategy and divide it by the lowest MSE of existing strategies. Values less than 1 show where strategies in Stratify outperform the best known previous methods."
The reported 'improvement' is computed by minimizing test MSE over all novel Stratify strategies and dividing by the minimum test MSE over existing strategies (Table 3 caption). The paper then describes this as 'the best strategy identified within the proposed Stratify framework' (Table 5a). Thus the strategy identity is selected on the same test set used to report its performance. The result is therefore an in-sample argmin statistic, not an out-of-sample evaluation of a fixed strategy: the numerator is the minimum of the novel family by construction, so the claim 'novel strategies improved performance' is a statement about a test-set selection, not a prediction.
full rationale
The paper contains no theoretical derivation chain and no load-bearing self-citation: the Stratify space is defined directly via Eq. 12 as combinations of RecMO, DirMO, and DirRec, and existing strategies are included in that space by construction. That part is not circular. The main circularity concern is empirical and located in the evaluation protocol: Table 3 and Table 5a report the ratio of the minimum test MSE among novel strategies to the minimum among existing strategies, and Table 5a explicitly uses 'the best strategy identified within the proposed Stratify framework'. Choosing the minimum on the test set means the reported improvement is not the out-of-sample performance of a prespecified strategy; it is an in-sample selection statistic. This fits the fitted-input-called-prediction pattern because the strategy identity is selected from the same test data used to claim superiority. Separately, Section 3.1 states that 'For the remaining functions we only evaluate the Stratify plane in the RecMO-RecMO region', while Direct, DirRec, DirMO, DirRecMO, Rectify, and RectifyMO lie outside that region; this undercuts the 'all existing ones' wording but is a scope and benchmarking limitation rather than a circularity. Overall, the framework itself is self-contained, but the central performance claim is partially circular because it is defined as a test-set minimum.
Assumptions & free parameters
free parameters (2)
- Best strategy indices (ρ, δ, ι percentages) =
per task selection, e.g., ρ:50% δ:50%
- Fixed model hyperparameters =
2 layers, 100 units, 1000 epochs, lr=0.01, batch size=1024, window w=160
assumptions (4)
- domain assumption The evaluation metric (MSE) and the benchmark datasets are appropriate proxies for forecasting quality.
- domain assumption Recursive forecasts are asymptotically biased and direct forecasts are unbiased.
- ad hoc to paper The restricted grid of percentages {10%, 20%, 50%, 100%} is sufficient to explore the strategy space.
- ad hoc to paper The RecMO-RecMO subregion is representative of the full Stratify space for non-MLP function classes.
Cite this review
Pith. "Pith review of Stratify: Unifying Multi-Step Forecasting Strategies." pith.science (2026). https://pith.science/paper/BDAKIGWS
@misc{pith2026241220510,
author = {Pith},
title = {Pith review of: Stratify: Unifying Multi-Step Forecasting Strategies},
year = {2026},
howpublished = {\url{https://pith.science/paper/BDAKIGWS}},
note = {Machine review of arXiv:2412.20510}
}
read the original abstract
A key aspect of temporal domains is the ability to make predictions multiple time steps into the future, a process known as multi-step forecasting (MSF). At the core of this process is selecting a forecasting strategy, however, with no existing frameworks to map out the space of strategies, practitioners are left with ad-hoc methods for strategy selection. In this work, we propose Stratify, a parameterised framework that addresses multi-step forecasting, unifying existing strategies and introducing novel, improved strategies. We evaluate Stratify on 18 benchmark datasets, five function classes, and short to long forecast horizons (10, 20, 40, 80). In over 84% of 1080 experiments, novel strategies in Stratify improved performance compared to all existing ones. Importantly, we find that no single strategy consistently outperforms others in all task settings, highlighting the need for practitioners explore the Stratify space to carefully search and select forecasting strategies based on task-specific requirements. Our results are the most comprehensive benchmarking of known and novel forecasting strategies. We make code available to reproduce our results.
Figures
Forward citations
Cited by 1 Pith paper
-
Neural operator discovery from heterogeneous trajectories
Trajectory grouping plus a low-dimensional latent bottleneck lets a neural operator discover each system's hidden governing factors and extrapolate to unseen systems.
Reference graph
Works this paper leans on
-
[1]
ACM Transactions on Management Information Systems 14(1), 1–29 (2023)
Morid, M.A., Sheng, O.R.L., Dunbar, J.: Time series prediction using deep learning methods in healthcare. ACM Transactions on Management Information Systems 14(1), 1–29 (2023)
work page 2023
-
[2]
IET Intelligent Transport Systems 12(9), 998–1004 (2018)
Nguyen, H., Kieu, L.-M., Wen, T., Cai, C.: Deep learning methods in trans- portation domain: a review. IET Intelligent Transport Systems 12(9), 998–1004 (2018)
work page 2018
-
[3]
Rajagukguk, R.A., Ramadhan, R.A., Lee, H.-J.: A review on deep learning mod- els for forecasting time series data of solar irradiance and photovoltaic power. Energies 13(24), 6623 (2020)
work page 2020
-
[4]
Applied soft computing 90, 106181 (2020)
Sezer, O.B., Gudelek, M.U., Ozbayoglu, A.M.: Financial time series forecast- ing with deep learning: A systematic literature review: 2005–2019. Applied soft computing 90, 106181 (2020)
work page 2020
-
[5]
Journal of economic surveys 37(1), 76–111 (2023)
Masini, R.P., Medeiros, M.C., Mendes, E.F.: Machine learning advances for time series forecasting. Journal of economic surveys 37(1), 76–111 (2023)
work page 2023
-
[6]
International Journal of Forecasting (2022)
Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M.Z., Barrow, D.K., Taieb, S.B., Bergmeir, C., Bessa, R.J., Bijak, J., Boylan, J.E., et al.: Forecasting: theory and practice. International Journal of Forecasting (2022)
work page 2022
-
[7]
Philosophical Transactions of the Royal Society A 379(2194), 20200209 (2021)
Lim, B., Zohren, S.: Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A 379(2194), 20200209 (2021)
work page 2021
-
[8]
Journal of Central South University 24, 1503–1512 (2017)
Ji, Y.-j., Gao, L., Chen, X., Guo, W.G.: Strategies for multi-step-ahead available parking spaces forecasting based on wavelet transform. Journal of Central South University 24, 1503–1512 (2017)
work page 2017
Show all 23 references
-
[9]
Machine Learning and Knowledge Extraction 6(3), 1633–1652 (2024)
Noa-Yarasca, E., Osorio Leyton, J.M., Angerer, J.P.: Extending multi-output methods for long-term aboveground biomass time series forecasting using con- volutional neural networks. Machine Learning and Knowledge Extraction 6(3), 1633–1652 (2024)
2024
-
[10]
Universit Libre de Bruxelles, Belgium, 75–86 (2014)
Taieb, S.B.: Machine learning strategies for multi-step-ahead time series forecast- ing. Universit Libre de Bruxelles, Belgium, 75–86 (2014)
2014
-
[11]
Expert systems with applications 39(8), 7067–7083 (2012)
Taieb, S.B., Bontempi, G., Atiya, A.F., Sorjamaa, A.: A review and comparison of strategies for multi-step ahead time series forecasting based on the nn5 forecasting competition. Expert systems with applications 39(8), 7067–7083 (2012)
2012
-
[12]
Ben Taieb, S., Hyndman, R.: Recursive and direct multi-step forecasting: the best of both worlds (19/12) (2012) 19
2012
-
[13]
Neurocomputing 73(10-12), 1950–1957 (2010)
Taieb, S.B., Sorjamaa, A., Bontempi, G.: Multiple-output modeling for multi- step-ahead time series forecasting. Neurocomputing 73(10-12), 1950–1957 (2010)
2010
-
[14]
In: 2015 International Conference on Advanced Computing and Applications (ACOMP), pp
An, N.H., Anh, D.T.: Comparison of strategies for multi-step-ahead prediction of time series using neural network. In: 2015 International Conference on Advanced Computing and Applications (ACOMP), pp. 142–149 (2015). https://doi.org/10. 1109/ACOMP.2015.24
2015
-
[15]
IEEE Transactions on Knowledge and Data Engineering (2024)
Shao, Z., Wang, F., Xu, Y., Wei, W., Yu, C., Zhang, Z., Yao, D., Sun, T., Jin, G., Cao, X., et al.: Exploring progress in multivariate time series forecasting: Comprehensive benchmarking and heterogeneity analysis. IEEE Transactions on Knowledge and Data Engineering (2024)
2024
-
[16]
Econometrica: Journal of the Econometric Society, 321–343 (1984)
Brown, B.W., Mariano, R.S.: Residual-based procedures for prediction and estimation in a nonlinear simultaneous system. Econometrica: Journal of the Econometric Society, 321–343 (1984)
1984
-
[17]
IEEE Transactions on neural networks 10(2), 402–409 (1999)
Atiya, A.F., El-Shoura, S.M., Shaheen, S.I., El-Sherif, M.S.: A comparison between neural-network forecasting techniques-case study: river flow forecasting. IEEE Transactions on neural networks 10(2), 402–409 (1999)
1999
-
[18]
IEEE Transactions on Neural Networks 10(5), 988–999 (1999) https://doi.org/10.1109/72.788640
Vapnik, V.N.: An overview of statistical learning theory. IEEE Transactions on Neural Networks 10(5), 988–999 (1999) https://doi.org/10.1109/72.788640
1999 doi
-
[19]
IEEE Access 9, 83105–83123 (2021)
Chandra, R., Goyal, S., Gupta, R.: Evaluation of deep learning models for multi- step ahead time series prediction. IEEE Access 9, 83105–83123 (2021)
2021
-
[20]
Journal of Machine Learning Research 12, 2825–2830 (2011)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Mach...
2011
-
[21]
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., K¨ opf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chin- tala, S.: PyTorch: An Imper...
2019
-
[22]
arXiv preprint arXiv:1412.6980 (2014)
Kingma, D.P.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
2014 arXiv
-
[23]
Demˇ sar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7(1), 1–30 (2006) 20 Appendix A Training time over Stratify space In Figure A1 we show the time taken to train each strategy in Stratify for the MLP on the mg 100...
2006
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.