{"id":"57c7869c-7633-4436-9824-2e5f5ed4eceb","arxiv_id":"2510.04487","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Forking-sequences trains forecasting models on all forecast-creation dates jointly, reducing gradient and forecast variance relative to per-date window sampling.","lead":"This paper formalizes a neural forecasting training scheme, called forking-sequences, that encodes the whole history once and produces forecasts for every historical date at once. It claims this reduces forecast revision volatility, improves accuracy, and speeds up temporal cross-validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical gains likely confounded by inequitable WS baseline; headline percentages also inconsistent.","rationale":"The reader identified the same load-bearing concern: the WS baseline is handicapped by unmatched input size, no ensembling, unreported window lengths, and unmatched compute. My agreement is strong because the empirical section is the primary evidence for the method's practical value; the theoretical results are conditional and essentially a variance-decomposition identity, so they do not by themselves establish superiority over WS. A concrete rerun with matched baselines would settle whether the reported gains are due to forking-sequences or to confounds. The verdict should remain conditional: if the gains persist after matching, the method is useful; if not, the contribution is simply 'averaging helps,' which is already known. I also note the abstract/body percentage mismatch, but that is secondary to the baseline issue. The proposed concrete test directly addresses the weakest assumption and would be feasible given the paper's experimental setup.","tokens_in":21433,"tokens_out":3436,"duration_ms":29217,"concrete_test":"Reproduce Table 1 with a fair WS baseline: (a) set WS MLP input size to 2H to match FS; (b) apply the same moving-average ensemble at inference time to WS predictions; (c) report and/or tune WS window length L per dataset; (d) match compute by either increasing WS training steps multiplicatively (e.g., by T, the number of FCDs per series) or running to the same wall-clock time. If the average sCRPS improvement across the five encoders drops below, say, 10% or loses statistical significance, the central claim reduces to 'ensembling helps' rather than a benefit of the forking-sequences training scheme.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim—that forking-sequences training and inference improve sCRPS and sQPC over window-sampling—depends on the window-sampling baseline being representative of standard practice. That baseline is not matched in several material ways. In Appendix C Table 4, the MLP encoder has input size 2H for forking-sequences but input size 1 for window-sampling, so the WS MLP is strictly less expressive, not just differently trained. Section 2.4 and the experiments state that FS models receive a moving-average forecast ensemble at inference, while WS models do not, conflating the training scheme with ensembling. Additionally, both schemes are trained for 30K steps, but FS processes all FCDs per step, yielding many more gradient evaluations per step. The window length L for WS is never reported, and hyperparameters are explicitly not tuned for WS (Discussion admits 'skipped almost all hyperparameter tuning'). The headline percentages (e.g., 43.2% sCRPS improvement for LSTM) are averages across 16 datasets with high variance and include ensembling, so they do not isolate the forking-sequences training scheme. If a matched WS baseline—same context length, same ensembling, same compute—were used, the gains could shrink substantially; the reader's concern is precisely this. This is the load-bearing weakness because the theoretical results (Theorems 1–2) are correct but elementary (averaging reduces variance), so the practical contribution rests on the empirical comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes forking-sequences, a training and inference scheme for multi-horizon neural forecasting in which a shared encoder processes all FCDs jointly and the decoder produces forecasts for every FCD in parallel. The authors claim three benefits: (i) gradient variance reduction at O(1/T) under M-dependence (Theorem 1), (ii) forecast variance reduction via ensembling at O(1/|H|) (Theorem 2), and (iii) inference complexity O(T) instead of O(T^2) by reusing encoder computations. They validate these claims on 16 M1/M3/M4/Tourism datasets with MLP, RNN, LSTM, CNN, and Transformer encoders, reporting sCRPS and sQPC gains over window-sampling.","tokens_in":21770,"tokens_out":8874,"duration_ms":68806,"significance":"If the empirical claims hold, the paper addresses an important and under-studied property of forecasting systems: revision volatility across forecast creation dates. The formalization of forking-sequences and the computational-complexity argument for encoder reuse are useful contributions, and the paper is honest about the simplicity of the variance-reduction mechanism. The theoretical results are elementary but correct in spirit; the main value of the paper is therefore empirical. However, the current experiments do not provide a controlled comparison between forking-sequences and window-sampling, so the headline improvements are not yet established. The paper also contains inconsistent headline numbers between the abstract and the body. The central idea is defensible, but the empirical evidence needs substantial strengthening.","major_comments":[{"comment":"The FS-vs-WS comparison is not controlled. The WS MLP uses input size 1 while FS uses 2H (Table 4), so the WS MLP is a strictly different, weaker model. Section 2.4 states that FS models receive a moving-average forecast ensemble at inference, while WS models do not; the Discussion admits that 'almost all hyperparameter tuning' was skipped for WS; and the WS window length L is never reported. Because both schemes are trained for the same 30K steps while FS processes all FCDs per step, FS also receives far more gradient evaluations per step. Table 1/2 and the headline percentages therefore measure a combination of training scheme, model capacity, ensembling, and compute, not forking-sequences alone. Please rerun with matched context length, matched capacity, matched ensembling (WS forecasts can also be averaged via Eq. (6)), and matched total gradient evaluations or compute budget.","section":"§3, Appendix C, Table 4"},{"comment":"The headline numbers are internally inconsistent. The abstract reports median sCRPS improvements of 46.2%, 49.3%, 28.6%, 24.7%, and 6.4% for RNN, LSTM, CNN, Transformer, and State Space architectures; the body (Section 3.1) reports average improvements of 16.9% (RNN), 43.2% (LSTM), 19.2% (CNN), 1.8% (Transformer), and includes MLP (14.3%) while no State Space architecture appears in the experiments. The abstract also omits MLP from the list. The manuscript must present one consistent set of results, with average/median labeling explicit and every value traceable to Tables 1 and 2.","section":"Abstract vs §3.1/§3.2, Tables 1–2"},{"comment":"Theorem 2 is not proved. 'The proof is analogous to Theorem 1' is insufficient because the ensemble averages forecasts for the same target date made at different FCDs and horizons; these are not identically distributed, and the unbiasedness assumption on every member of H is strong and unverified. With horizon-dependent means, averaging can introduce bias, so convergence to the 'true value' requires a proof or a precise condition under which the average is unbiased. The same appendix defines Sigma ambiguously in the M-dependence definition; the Markov bound in Theorem 1 should use a long-run covariance rather than a per-lag covariance.","section":"§2.4, Theorem 2, Appendix D"},{"comment":"The text around Figs. 10a–10d is confusing. It says 'Ensembling during inference results in a marginal reduction in sCRPS across models, with substantial improvements in sQPC compared to no ensembling during inference, as shown in Figs. 10a and 10b' — but Figs. 10a/10b are labeled as 'without ensembling' versus window-sampling. Please report separately: (i) FS without ensembling vs WS (isolates the training scheme), (ii) FS with ensembling vs WS without ensembling (the current headline), and (iii) FS with ensembling vs WS with ensembling (isolates ensembling). The current presentation does not let the reader separate these effects.","section":"§3.2, Fig. 10"}],"minor_comments":[{"comment":"Reference [43] is 'Shumway and Stoffer', not 'Stuffer'.","section":"Appendix D"},{"comment":"Eq. (8) introduces an undefined function f(...) after the ensemble definition; either define it or remove it.","section":"§2.4, Eq. (8)"},{"comment":"The window length L for the window-sampling baseline is never reported, despite being a key hyperparameter. Please list it for every architecture and frequency.","section":"Appendix C, Tables 4–8"},{"comment":"The notation T is overloaded: it denotes the set of FCDs, its cardinality, and the total series length in Section 2.3. Use distinct symbols (e.g., cal T vs T).","section":"§2.2, Eq. (5)"},{"comment":"Several WS standard deviations are very large relative to FS (e.g., LSTM M1 Y in Table 1), and no significance tests or confidence intervals are reported across the five runs. Consider adding paired tests or at least error bars on the percentage improvements.","section":"Tables 1–2"},{"comment":"The bullet says accuracy is improved 'up to 43.2%', but Section 3.1 reports an average of 43.2% for LSTM. 'Up to' is inconsistent with an average.","section":"Contributions (Section 1)"},{"comment":"The y-axis on panels (b) and (d) ranges from -300 to 100, which is unusual for a percentage scale; check whether this is a typo and ensure all panels use the same scale.","section":"Fig. 10"}],"recommendation":"major_revision","confidential_remarks":"The abstract provided in the review materials differs materially from the full-text abstract (different architectures, different numbers, average vs median). This may be a versioning artifact, but the authors must ensure the final abstract and body report identical, traceable numbers. The empirical comparison is the main weakness; if the baseline can be matched on context, capacity, ensembling, and compute, and the gains persist, this would be a useful contribution. The theoretical section is elementary and needs proof cleanup, but the computational-efficiency argument for forking-sequences is the more compelling part."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, 'forking-sequences' is an old trick — the paper says so itself, citing MQCNN, MQT, SPADE and target-replication work — so the novelty is in the formalization and the benchmark, not in the idea. Second, the empirical headline numbers are not trustworthy as stated, because the window-sampling baseline is handicapped in several concrete ways. If you read only the abstract, you'll overestimate what the paper actually establishes.\n\nWhat's good: The complexity argument is solid and practical. Reusing encoder outputs across FCDs turns O(T^2) cross-validation inference into O(T) for conv/RNN/attention, and that's a real engineering win. The variance-reduction theorems (Theorems 1 and 2) are correct but elementary — they're just M-dependence variance bounds for a mean of T correlated variables, which is a standard result. The paper presents them cleanly and doesn't oversell them. The empirical study covers 16 datasets, five encoders, and multiple runs, which is a useful resource even if the comparison is unfair.\n\nThe soft spots are in the baseline. Appendix C shows the WS MLP has input size 1 where the FS MLP has 2H, so the WS model is strictly less expressive. WS gets no inference ensemble while FS does. Both train for 30K steps, but FS processes all FCDs per step, so it sees far more gradient signal per step. The WS window length L is never reported, and the Discussion admits 'we skipped almost all hyperparameter tuning.' So the headline gains (43.2% LSTM sCRPS, 37.9% sQPC) conflate the training scheme with ensembling and compute. The paper does show FS-without-ensemble still beats WS (Figure 10a/b), which suggests averaging isn't the whole story, but those figures are averages over 16 datasets with high variance, and the abstract's median percentages don't match the body's averages — there's also a 'State Space' in the abstract that isn't in the experiments. These are fixable, but right now the empirical claim is not a controlled comparison.\n\nWho's it for: practitioners who want a clear statement of the forking-sequences scheme and its variance properties, and researchers working on forecast revision metrics. It's a workshop-level contribution. I'd send it to review only conditionally: ask for a matched WS baseline (same context, same ensembling, same compute budget) and correct the abstract/body discrepancy. If the authors do that, the paper is a useful reference. Without it, the contribution reduces to 'averaging helps and reusing encoders is cheap,' which is true but not new.","headline":"A useful formalization of a known averaging trick, with an empirical comparison that needs a fair baseline before the accuracy claims can be trusted.","tokens_in":22268,"tokens_out":3337,"would_cite":true,"duration_ms":25807,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that forking-sequences—jointly encoding all forecast creation dates and decoding them in parallel—reduces gradient and forecast-revision variance at rate O(1/T), yielding more stable, more accurate forecasts.","keywords":["time series forecasting","forecast revisions","forecast stability","forking-sequences","gradient variance reduction","probabilistic forecasting","temporal cross-validation","multi-horizon forecasting"],"falsifier":"Record per-FCD gradient vectors during training and compute their autocorrelation at increasing lags; if correlations remain substantial at long lags, the O(1/T) bound in Theorem 1 does not hold. Equally decisive: retrain the window-sampling baseline with input size 2H, the same ensembling at inference, and the same number of FCDs per optimizer step (or matched total compute); if the reported sCRPS and sQPC gaps shrink substantially, the headline gains are due to underspecification of the baseline rather than to forking-sequences itself.","tokens_in":21339,"feed_emoji":"📉","tokens_out":7216,"duration_ms":57575,"temperature":0.7,"pith_summary":"Forking-sequences is a training and inference scheme for neural forecasters that treats all forecast creation dates jointly: one encoder pass over the observed series, then parallel decoders for every date-and-horizon cell of the forecast grid. The paper's central claim is that this architectural choice reduces the variance of stochastic gradients during training at rate O(1/T) under short-range dependence, so optimization converges faster, and that averaging the several forecasts pointing at the same target date reduces forecast-revision volatility at rate O(1/|H|). It also shows that reusing encoder outputs turns temporal cross-validation inference from quadratic into linear cost. Across 16 M-competition and Tourism datasets, keeping the architecture fixed and switching from window-sampling to forking-sequences improves scaled CRPS on average for all five encoder types, up to 43.2% for LSTM, and reduces a new symmetric quantile percentage change metric on average, up to 37.9% for LSTM. A sympathetic reader would care because erratic forecast revisions between creation dates are a real operational cost, and this work offers one mechanism that addresses accuracy, stability, and compute together.","feed_headline":"Training trick cuts forecast revision volatility by up to 38%","feed_subtitle":"A single encoder pass over all forecast creation dates also speeds training and cuts inference cost.","key_machinery":"The forking-sequences architecture itself: a single encoder consumes the full observed series and produces one hidden representation, and separate decoder heads emit forecasts for every forecast creation date and horizon simultaneously. The variance theorems ride on M-dependence—correlation between gradient samples or forecast errors vanishes beyond a finite lag—which lets the average over T dependent samples behave like an average of weakly correlated quantities and yields O(1/T) variance. The compute gain comes from reusing the encoder's activations across all FCDs, so the encoder runs once instead of once per FCD, changing inference cost from O(T^2) to O(T).","core_discovery":"The central discovery is that the forking-sequences design—already present in some production forecasters but never formalized—is not only a pragmatic trick but a variance-reduction mechanism with provable rates. The paper formalizes the gradient estimator that sums losses over all FCDs in a batch, proves under M-dependence (gradient samples whose correlations vanish beyond a finite lag) that its variance decays as O(1/T), and proves an analogous ensemble variance reduction O(1/|H|) for averaging forecasts that share a target date. It further shows that the encoder can be computed once for the entire series and reused across all FCDs, dropping cross-validation inference from O(T^2) to O(T).","pith_inferences":["If the O(1/T) argument is the whole story, forking-sequences is essentially a cheap way to enlarge the effective batch over correlated forecast dates; a window-sampling baseline matched on context length (2H), on ensembling, and on total FCDs processed per optimizer step would reveal how much of the reported gain is architectural reuse versus raw sample count.","The sQPC metric—symmetric quantile percentage change between consecutive forecast creation dates—could become a standard report for operational forecasting, where revision volatility is a direct cost in planning and inventory systems.","The encoder-reuse pattern (one context, many time-offset targets) appears beyond forecasting, for example in survival analysis and streaming anomaly detection, so the variance-reduction argument may transfer to those settings.","Averaging forecasts across FCDs weights all revisions equally; a testable extension is horizon-weighted or recency-weighted ensembling to tune the accuracy-stability tradeoff further."],"forward_implications":["Under M-dependent gradient samples, training with forking-sequences reduces gradient variance at O(1/T), which accelerates convergence; the paper demonstrates faster train-loss convergence across five deep encoders and in a convex autoregressive ablation.","Averaging the forecasts aimed at a single target date from earlier forecast creation dates reduces forecast variance at O(1/|H|), yielding more stable revisions with negligible extra encoder cost because the encoder runs once.","Cross-validated inference over many forecast creation dates drops from O(T^2) to O(T) for convolutional and recurrent encoders, making large-scale temporal backtesting practical.","On 16 M-competition and Tourism datasets, switching a fixed architecture from window-sampling to forking-sequences improves average sCRPS for all five encoder types (up to 43.2% for LSTM) and reduces average median-forecast sQPC (up to 37.9% for LSTM).","Ensembling during inference cuts forecast volatility by roughly 10-13% across architectures while maintaining accuracy, suggesting that stability can be improved without an accuracy penalty."],"fun_headline_variants":["Formalizing forking-sequences for stable, efficient forecasting","Forking-sequences: provably lower forecast volatility","Single-pass multi-horizon forecasting tames revision volatility","Cut forecast revision volatility with a smarter training design","Forking-sequences: faster, less volatile forecasts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that gradient samples (and forecast errors) across forecast creation dates are M-dependent, with correlations that vanish after a finite lag; if real gradients are long-range dependent, the O(1/T) variance reductions do not follow, and the empirical comparison also rests on the window-sampling baseline being a fair match—the paper's own appendix shows the WS MLP sees a context of length 1 instead of 2H, WS models are not given the inference-time en","fun_headline_variants_meta":{"raw":{"variants":["Formalizing forking-sequences for stable, efficient forecasting","Forking-sequences: provably lower forecast volatility","Single-pass multi-horizon forecasting tames revision volatility","Cut forecast revision volatility with a smarter training design","Forking-sequences: faster, less volatile forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2305,"prompt_tokens":898,"completion_tokens":1407,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":1327}},"tokens_in":642,"tokens_out":1407,"duration_ms":12194,"temperature":1.0,"reasoning_tokens":1327,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T11:27:38.309765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record per-FCD gradient vectors during training and compute their autocorrelation at increasing lags; if correlations remain substantial at long lags, the O(1/T) bound in Theorem 1 does not hold. Equally decisive: retrain the window-sampling baseline with input size 2H, the same ensembling at inference, and the same number of FCDs per optimizer step (or matched total compute); if the reported sCRPS and sQPC gaps shrink substantially, the headline gains are due to underspecification of the baseline rather than to forking-sequences itself.","supporting_citations":[],"review_version":1}