{"id":"4b310ef9-fa31-451a-bf7f-f7e320e35958","arxiv_id":"2412.20510","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A unified parameterized framework for multi-step forecasting contains task-dependent novel strategy combinations that usually beat all existing strategies on benchmark datasets.","lead":"This paper introduces Stratify, a single framework that organizes different ways to forecast a time series many steps ahead, including known strategies and new combinations of them. Tests on 18 datasets suggest some new combinations forecast better than previous strategies, but the best choice depends on the task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"For four of five function classes, only the RecMO-RecMO subregion was evaluated, so the 'best existing' baseline excludes Direct, DirRec, DirMO, DirRecMO, Rectify, and RectifyMO; the 84% claim over 'all existing ones' is unsupported for those classes.","rationale":"The reader's weakest assumption identifies the RecMO-RecMO restriction as the key gap, and I agree. The paper's central claim is that novel Stratify strategies outperform all existing ones in over 84% of 1080 experiments across five function classes. For MLP, the full Stratify space was evaluated, giving some support for that class. For the other four classes, only the RecMO-RecMO subregion was evaluated, and existing strategies outside that subregion were not included as baselines. Since the paper's own MLP results show the best strategies in other regions (e.g., delta:50 base with delta rectifiers), it is entirely possible that existing strategies such as DirMO or RectifyMO would outperform the RecMO-RecMO novel strategies on RNN, LSTM, or Transformer. The abstract and Section 4 state the 84% result without this qualification, so the claim overreaches the evidence. I also note a secondary concern: the per-task selection of the best novel strategy on the test set inflates the reported improvements, though the critical difference diagrams provide some evidence for fixed novel strategies. The RecMO-RecMO gap is the more fundamental threat to the scope of the central claim. The concern is patchable by running the full comparison for one non-MLP function class or by restricting the claim, so the reader's CONDITIONAL verdict remains appropriate; no change is needed.","tokens_in":33594,"tokens_out":9544,"duration_ms":91535,"concrete_test":"Evaluate LSTM (or RNN) on a subset of 6 datasets across the four horizons with the full Stratify plane, including all existing strategies outside RecMO-RecMO (Direct, DirRec, DirMO, DirRecMO, Rectify, RectifyMO). Recompute the proportion of tasks where the best novel strategy beats the best existing strategy, using the paper's three-seed protocol. If the proportion drops below 84% or the mean relative MSE rises above 1.0, the headline claim fails for non-MLP classes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states: 'For the remaining functions we only evaluate the Stratify plane in the RecMO-RecMO region.' Thus, for RF, RNN, LSTM, and Transformer, only strategies whose base and rectifier are both RecMO were evaluated. Existing strategies such as Direct (Eq. 4), DirRec (Eq. 5), DirMO (Eq. 10), DirRecMO (Eq. 11), Rectify (Eq. 8), and RectifyMO (Sec. 2.3) use DirMO or DirRec components and lie outside this region. Nevertheless, Table 3 and Table 5a compare the best novel strategy against the 'lowest MSE of existing strategies,' and the abstract claims improvement 'compared to all existing ones.' For four of five function classes, the set of existing strategies actually evaluated is a restricted subset, so the comparison may be biased in favor of novel strategies. This is not merely hypothetical: for MLP, where the full space was evaluated, Figure 8 shows the best strategies lie outside RecMO-RecMO (e.g., a delta:50 base with delta rectifiers), demonstrating that other regions can contain substantially better strategies. Without evaluating the full set of existing strategies for at least one non-MLP class, the headline 84% figure is not established for those classes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Stratify, a parameterised framework for multi-step forecasting (MSF) that represents a forecasting strategy as a pair consisting of a base strategy and a residual-rectifying strategy, each chosen from RecMO, DirMO, or DirRec and parameterised as a percentage of the forecast horizon. The framework is claimed to unify all existing MSF strategies while also defining novel ones. The authors evaluate the full Stratify space for MLP and a restricted RecMO-RecMO subregion for RF, RNN, LSTM, and Transformer on 18 datasets with horizons 10, 20, 40, and 80, and report that novel strategies outperform all existing strategies in over 84% of 1080 experiments, with 5–25% error reductions across function classes.","tokens_in":33939,"tokens_out":5684,"duration_ms":51279,"significance":"If the central claim is valid, Stratify would provide a useful unifying vocabulary for MSF strategies and a practical tool for discovering better-performing strategies, backed by a large benchmark suite and released code. The paper's breadth—18 datasets, five function classes, four horizons, and reproducible code—is a genuine strength. However, the headline quantitative claim rests on (a) selecting the best strategy on the test set and (b) comparing against an incomplete set of existing strategies for four of the five function classes. These are load-bearing for the abstract's 'over 84%' statement, so the significance of the empirical contribution is currently not established at the level claimed, although the framework itself remains conceptually interesting.","major_comments":[{"comment":"The reported improvements are based on selecting, for each task, the lowest-MSE strategy from the set of novel strategies and dividing by the lowest-MSE strategy from the set of existing strategies, with MSE computed on the test set. This is an in-sample selection on the test set and is subject to winner's curse: even if no novel strategy is truly better than the best existing one, the minimum over many candidates will often appear better. The 'over 84% of 1080 experiments' claim in the abstract and Section 4 is therefore a statement about an oracle that knows test outcomes, not about a reproducible model-selection procedure. Please report results where the best strategy is selected on the validation set (or via nested cross-validation) and then evaluated on the test set, and also report the performance of a fixed strategy (e.g., the single best novel strategy overall) to quantify what a practitioner would actually obtain.","section":"Section 3.2, Table 3"},{"comment":"For RF, RNN, LSTM, and Transformer, the paper states that only the Stratify plane in the RecMO-RecMO region was evaluated. Yet Table 3 and Table 5a compare the best novel strategy in that subregion against the 'lowest MSE of existing strategies,' and the abstract claims improvement 'compared to all existing ones.' Existing strategies such as Direct (Eq. 4), DirRec (Eq. 5), DirMO (Eq. 10), DirRecMO (Eq. 11), Rectify, and RectifyMO all lie outside the RecMO-RecMO region and were not evaluated for these four function classes. The comparison is therefore restricted to RecMO-family baselines (recursive, RecMO variants, and MIMO at σ=H), so the headline claim is unsupported for four of the five function classes. At minimum, the full Stratify plane should be evaluated for at least one non-MLP class (or on a subset of datasets/horizons), or the claims should be explicitly restricted to MLP and the RecMO-RecMO subregion.","section":"Section 3.1 and Table 5a"},{"comment":"The MLP results in Figure 8 demonstrate that the best-performing strategies lie outside the RecMO-RecMO region (e.g., a δ:50% base with δ rectifiers, with ranking errors around 30 versus 40–60 in the RecMO-RecMO region). This shows that the RecMO-RecMO restriction is not a harmless simplification for the other function classes; it may systematically exclude the best regions of the strategy space. The computational justification (Section 3.1: 'we would need to train over 86,400 transformers') explains the restriction but does not support generalising the MLP-based full-plane findings to the other classes. The paper should either provide full-plane evidence for at least one additional function class or substantially weaken the generality of the 'novel strategies consistently outperform' claim.","section":"Figure 8 and Section 3.1"},{"comment":"The theoretical consideration that 'using an unbiased strategy as the base makes a Stratify strategy unbiased for any rectifier selected' is stated without formal derivation or assumptions about finite-sample behaviour, convergence rates, or the effect of estimating the rectifier on residuals. This is not load-bearing for the empirical contribution, but if the claim is intended as a theoretical contribution, it needs a precise statement and proof; otherwise it should be presented as a heuristic remark.","section":"Section 2.4.1"}],"minor_comments":[{"comment":"The abstract and Section 4 report 'over 84% of 1080 experiments,' but the means in Table 5a (0.70, 0.92, 0.97, 0.91, 0.90) average to about 88%. Please reconcile these numbers or clarify how the 84% figure is computed.","section":"Abstract and Section 4"},{"comment":"There are inconsistent spellings and capitalisations, e.g., 'parralel' (Section 2.2), 'DIRMO' vs 'DirMO', 'DIRRECMO' vs 'DirRecMO', and 'T able' in several table captions. Please proofread.","section":"Throughout"},{"comment":"Several entries show a standard error larger than the reported ratio, e.g., ExchangeRate RNN 0.59 ± 1.35. Such entries indicate high instability across seeds/horizons; consider reporting medians or interquartile ranges, and discuss the most unstable cases explicitly.","section":"Table 3"},{"comment":"The sentence 'With 95% confidence, we find that the ten novel strategies shown outperform 60% of existing strategies' mixes a statement about individual strategies with a statement about the proportion of strategies; please clarify whether 'outperform' refers to significant pairwise differences under the Nemenyi test and what '60% of existing strategies' means exactly.","section":"Section 3.2.1"},{"comment":"In the sentence 'The resulting forecast for is the summation of...', the word 'for' appears to be a leftover artifact; please revise for readability.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The conceptual contribution—a unified parameterised strategy space—is reasonable and likely of interest to the forecasting community. However, the empirical claims as stated are not yet supported: the test-set oracle selection and the incomplete set of existing baselines for four of the five function classes are both load-bearing for the abstract's headline. I would encourage the editor to request a revision that addresses these two points, ideally by adding a validation-based selection protocol and at least one full-plane non-MLP evaluation (even on a reduced dataset/horizon grid), and by recalibrating the generality of the claims to match the evidence. The paper is not fatally flawed, but the current form would overstate the strength of the findings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Stratify is a real contribution: it turns the set of known multi-step forecasting strategies into a two-dimensional parameter space—base strategy and rectifier strategy, each chosen from RecMO, DirMO, or DirRecMO variants with a horizon-normalized step size. The addition of RectifyMO (the multi-output version of Rectify) fills a genuine gap in the literature. For MLP, the full plane is evaluated and the visualizations are informative: best strategies tend to have a short base and a longer rectifier, and the error surface looks smooth enough to be searched.\n\nThe problem is the headline '84% of 1080 experiments' claim. For RF, RNN, LSTM, and Transformer, the authors evaluate only the RecMO-RecMO subregion (Section 3.1). That means the 'best existing strategy' those four classes are compared against is only the best RecMO variant. Direct, DirRec, DirMO, DirRecMO, Rectify, and RectifyMO are all absent from the baseline set for those classes. Since Figure 8 shows that for MLP the best strategies sit outside RecMO-RecMO (e.g., DirMO base with DirMO rectifier), it is quite likely the restricted baseline inflates the improvement. The 84% figure is simply not established for the non-MLP classes.\n\nSecond, the paper selects the best strategy on the test set (Section 3.2). That is an in-sample selection; it turns the benchmark into a best-case analysis rather than a predictive prescription. The improvement numbers ('5-25% error reductions') are therefore optimistic, and the paper should at least do a validation-based selection or nested evaluation to claim practical benefit.\n\nThese flaws are patchable, but the current abstract overstates what the evidence shows. The code link promised in the abstract was not present in the version I read, which is another check on the checklist.\n\nVerdict: the framework and the MLP analysis are worth a serious referee. The paper needs major revision—run the existing baselines in the full space for at least one non-MLP class, and change the selection procedure—before the headline claim can stand.\n\nWho gets value: practitioners who want a mental map of multi-step strategies, and researchers who want a testbed. I'd bring it to a reading group to discuss the gap between the framework and the evidence.","headline":"Stratify is a genuinely useful unification of multi-step forecasting strategies, but the headline 84% claim rests on a benchmark that only evaluates the full strategy space for one of five model classes, and on test-set selection.","tokens_in":34415,"tokens_out":3974,"would_cite":true,"duration_ms":36138,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A unified framework for multi-step forecasting strategies reveals that newly discovered strategy combinations beat all existing strategies in over 84% of 1,080 experiments.","keywords":["multi-step forecasting","time series forecasting","forecasting strategies","recursive strategy","direct strategy","rectify strategy","multi-output forecasting","Stratify"],"falsifier":"Run the full Stratify search, including DirMO and DirRecMO rectifiers, for LSTM on the same 18 datasets and horizons; if the best full-plane strategy fails to beat the best existing strategy in roughly 84% of cases, the generality of the headline claim for non-MLP models is not supported.","tokens_in":33415,"feed_emoji":"📈","tokens_out":6231,"duration_ms":56625,"temperature":0.7,"pith_summary":"The paper proposes Stratify, a parameterized framework that places all known multi-step forecasting strategies in one space and, in doing so, exposes combinations that had never been tested. The authors argue that these newly exposed strategies are not curiosities: the best of them beat the best existing strategy in over 84% of 1,080 experiments spanning 18 datasets, five model families, and horizons from 10 to 80 steps. The reason to care is practical: currently a practitioner chooses a strategy by folklore or trial and error, and the paper shows that the choice matters and that the space of reasonable choices is larger and more navigable than previously thought. A secondary claim is that no single strategy dominates, so the framework's main use is to support a systematic search for a task-specific strategy.","feed_headline":"New forecasting strategy space beats old best in 84% of tests","feed_subtitle":"Unified framework maps all known multi-step forecasting strategies and reveals novel combinations cutting error by 5-25%.","key_machinery":"The central object is the Stratify plane. For a horizon H, the framework takes the set of base strategies and the set of rectifier strategies, each drawn from the families RecMO, DirMO, and DirRecMO with parameter σ chosen from the divisors of H; σ is expressed as a percentage of H so that comparisons across horizons are fair. A strategy is the pair (base, rectifier): the base model produces a first forecast, a second model forecasts the residuals of that forecast, and the final prediction is the sum. All previously known strategies appear as special cases—for example, σ=1 recovers the original Rectify strategy and σ=H recovers MIMO-like behavior—which is what makes the new combinations genuinely new rather than a separate class.","core_discovery":"The paper's central claim is that every existing multi-step forecasting strategy—recursive, direct, DirRec, Rectify, MIMO, and their multi-output variants—occupies a single point in a two-dimensional space whose axes are the base strategy and the rectifying strategy, each chosen from the same three families and each parameterized by the fraction of the horizon produced per step. The newly accessible points in this space are genuine strategies rather than interpolations, and the best new point outperforms the best previously known strategy in over 84% of 1,080 experiments, with reported mean error reductions between 5% and 25% depending on function class. The same experiments show that no single strategy wins everywhere, so the value of the framework is that it turns strategy selection into a finite, visualizable search.","pith_inferences":["If the unexplored regions of the Stratify plane behave like the RecMO-RecMO region for RNN, LSTM, Transformer, and random forest models, the headline improvement is likely a lower bound; direct-based and DirRec-based rectifiers could yield further gains that were only measured for MLP.","The percentage parameterization suggests a transfer test the paper does not run: a strategy identified as optimal at one horizon may remain near-optimal at other horizons on the same dataset, which would let practitioners search at short horizons and deploy at long ones.","The smoothness of the plane invites a meta-learning extension where dataset features predict the best region, potentially removing the need to train every strategy in the space."],"forward_implications":["Practitioners can replace ad-hoc strategy selection with a finite search over the Stratify plane, and the paper's heat maps indicate that a small number of sampled points often suffices to locate a strong strategy.","The best strategy remains task-dependent, so deploying Stratify means treating strategy choice as a hyperparameter rather than a fixed modeling assumption.","All existing strategies remain accessible inside the framework, so adopting Stratify cannot make a practitioner worse off in terms of the strategy set they can express.","Reported mean error reductions of 5–25% across function classes imply that the choice of strategy can matter as much as the choice of model architecture in multi-step forecasting."],"supporting_citations":[{"why":"Supplies the variance-bias decomposition of recursive and direct strategies that motivates the framework and its claim that recursive base forecasts are biased.","marker":"[10]"},{"why":"Reviews and compares multi-step strategies; the paper extends this comparison to a unified space.","marker":"[11]"},{"why":"Introduces the Rectify strategy that Stratify generalizes into RectifyMO and the Stratify plane.","marker":"[12]"},{"why":"Defines the RecMO, DirMO, and DirRecMO multi-output strategies that form the base and rectifier families in Stratify.","marker":"[9]"},{"why":"Provides the benchmark datasets used in all 1,080 experiments.","marker":"[15]"},{"why":"Supplies the Friedman and Nemenyi significance tests used to claim that novel strategies significantly outperform existing ones.","marker":"[23]"}],"fun_headline_variants":["Stratify unifies forecasting strategies, improves 84% of tests","New strategy space cuts error 5-25%, wins 84% of trials","One framework maps all multi-step forecasting strategies","Stratify: map strategies, beat old best in 84% of cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 'over 84%' rests on treating the RecMO-RecMO subregion, the only part of the Stratify space fully evaluated for the random forest, RNN, LSTM, and Transformer models, as representative of the whole framework for those model classes.","fun_headline_variants_meta":{"raw":{"variants":["Stratify unifies forecasting strategies, improves 84% of tests","New strategy space cuts error 5-25%, wins 84% of trials","One framework maps all multi-step forecasting strategies","Stratify: map strategies, beat old best in 84% of cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3858,"prompt_tokens":889,"completion_tokens":2969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":2892}},"tokens_in":505,"tokens_out":2969,"duration_ms":22158,"temperature":1.0,"reasoning_tokens":2892,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:19:38.172109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full Stratify search, including DirMO and DirRecMO rectifiers, for LSTM on the same 18 datasets and horizons; if the best full-plane strategy fails to beat the best existing strategy in roughly 84% of cases, the generality of the headline claim for non-MLP models is not supported.","supporting_citations":[{"cited_title":"Universit Libre de Bruxelles, Belgium, 75–86 (2014)","cited_arxiv_id":null,"evidence_quote":"Supplies the variance-bias decomposition of recursive and direct strategies that motivates the framework and its claim that recursive base forecasts are biased."},{"cited_title":"Expert systems with applications 39(8), 7067–7083 (2012)","cited_arxiv_id":null,"evidence_quote":"Reviews and compares multi-step strategies; the paper extends this comparison to a unified space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Rectify strategy that Stratify generalizes into RectifyMO and the Stratify plane."},{"cited_title":"Machine Learning and Knowledge Extraction 6(3), 1633–1652 (2024)","cited_arxiv_id":null,"evidence_quote":"Defines the RecMO, DirMO, and DirRecMO multi-output strategies that form the base and rectifier families in Stratify."},{"cited_title":"IEEE Transactions on Knowledge and Data Engineering (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the benchmark datasets used in all 1,080 experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Friedman and Nemenyi significance tests used to claim that novel strategies significantly outperform existing ones."}],"review_version":1}