{"id":"82334307-f635-4775-9413-23d6d011dac3","arxiv_id":"2411.16728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-stage teacher forcing training method lets deep learning models outperform ECMWF-S2S forecasts at two to six weeks, reversing ChaosBench's finding that direct forecasting beats rolling.","lead":"This paper shows that a multi-stage teacher forcing training scheme can make rolling deep learning weather models beat the leading physics-based ECMWF system at two to six week forecasts. The authors argue the bottleneck was how models were trained, not the neural network design, and that rolling forecasts can beat direct ones when trained this way.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline skill gain over ECMWF-S2S is computed with different anomaly baselines for model and ECMWF; the paper's own §4.1.2 warns this can inflate results, so the 19–91% margin may be a scoring artifact rather than true skill.","rationale":"The reader's weakest assumption and my independent reading converge on the same point: the ECMWF comparison is not apples-to-apples because the anomaly baseline differs. The paper's own text in Section 4.1.2 is explicit that the model uses an 11-day centered rolling ERA5 climatology while ECMWF uses its own climatology, and it warns that climatology choice can make results 'appear inflated.' Since PCC and TCC are computed on anomalies, this is not a cosmetic detail but a direct determinant of the scores. The claimed 19–91% improvement over ECMWF is the headline result, repeated in the abstract, introduction, and conclusion, so if this comparison is invalid, the paper's central claim is unsupported. The concrete test I propose—recomputing both systems' skill with a shared climatology and a shared truth—would settle the issue. A secondary concern, the exclusion of 4 of 10 variables from the headline average, also weakens the claim but is less fundamental than the climatology mismatch; the variable subset could be fixed by reporting all variables without changing the comparison protocol. I see no reason to change the reader's conditional verdict: the paper makes an important, plausible claim that needs this verification before the magnitude and direction of the ECMWF comparison can be accepted.","tokens_in":13767,"tokens_out":4496,"duration_ms":42524,"concrete_test":"Obtain the ECMWF-S2S reforecasts used in the paper (cycle C47r3, 2015–2018) and recompute PCC/TCC for both ECMWF and the model against the same ERA5 truth with identical anomaly definitions: (a) the paper's 11-day centered rolling ERA5 climatology and (b) a fixed daily climatology from a common reference period (e.g., 1995–2014). Compare the resulting 15–42-day skill for the same six variables; if the 19–91% margin disappears or reverses under either shared baseline, the headline claim is a scoring artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that the proposed multi-stage optimization surpasses ECMWF-S2S by 19–91%—rests on comparing PCC/TCC values computed under two different anomaly definitions. Section 4.1.2 states that for the model and ERA5 they use an 11-day centered rolling climatology, while 'ECMWF employs its own climatology,' and then warns that 'using a longer average or fixed time period can weaken the climatology signal, making the results appear inflated.' Because PCC and TCC are correlations of anomalies, the choice of climatology directly changes the scores of both the forecast and the verifying observation. If ECMWF's own climatology is noisier or smoother than the ERA5 rolling climatology, the ECMWF skill numbers used as baseline are not comparable to the model's numbers. The figure of 19–91% therefore may measure the difference in anomaly definitions, not in forecast quality. The paper provides no sensitivity check with a shared climatology. This concern is load-bearing because the abstract, introduction, and conclusion all advance the ECMWF comparison as the headline result; without it, the paper's primary evidence for 'optimization, not architecture' is reduced to internal ablations that do not compare against a state-of-the-art NWP baseline on equal terms.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the poor performance of deep learning models at subseasonal-to-seasonal (S2S) forecasting is caused by the optimization procedure rather than by the model architecture. The authors propose a multi-stage teacher-forcing training scheme that first fine-tunes a pretrained 'Baguan' backbone to a 1-day forecast, then to a 7-day rolling horizon, and finally to a 42-day rolling horizon with adapter-based PEFT, using anomaly-based losses. They report that this scheme, with the same backbone, surpasses ECMWF-S2S by 19-91% in PCC and TCC, extends the skillful MJO forecast lead time from 23 to 30 days, and reverses the ChaosBench conclusion that direct forecasting outperforms rolling forecasting. A theoretical analysis based on deep linear models and Jacobian accumulation is included to explain why the curriculum-style training helps.","tokens_in":13967,"tokens_out":6030,"duration_ms":51588,"significance":"If the claims are substantiated, the paper would make an important contribution: it would show that a training recipe alone can push existing deep-learning architectures to S2S skill beyond a leading NWP system, and it would challenge a recent benchmark finding. The manuscript has concrete strengths: it ships an implementation, provides ablations over two backbones and two resolutions, and formulates a falsifiable claim about rolling versus direct forecasting. However, the headline results currently rest on a non-apples-to-apples evaluation (different climatology conventions), a selected variable subset, and unquantified run-to-run variability, so the significance cannot be fully assessed at this stage.","major_comments":[{"comment":"The headline comparison with ECMWF-S2S is computed under two different anomaly baselines: the model and ERA5 use an 11-day centered rolling climatology, while ECMWF uses its own climatology, and the paper itself warns that 'using a longer average or fixed time period can weaken the climatology signal, making the results appear inflated.' Because PCC and TCC are correlations of anomalies, this difference directly changes the scores; without a sensitivity experiment in which ECMWF skill is recomputed with the same ERA5 climatology (or the model with ECMWF's), the reported 19–91% improvement cannot be attributed to forecast quality rather than scoring convention. This is load-bearing because the abstract, introduction, and conclusion all present the ECMWF comparison as the primary evidence for the 'optimization, not architecture' thesis.","section":"§4.1.2 and §4.2"},{"comment":"The stated 'over 19–91%' improvement is averaged over only 6 of the 10 modeled variables: t2m, z500, t850, olr, u200, and u850. The paper says these are 'primarily reported' because of 'importance of the signals,' but total precipitation (tp) and the other humidity and wind variables are excluded from the headline skill numbers; this is a selection on the outcome and can inflate the perceived improvement. The authors should report skill for all variables they claim to forecast or provide a principled reason for the subset and show that the conclusion is not sensitive to the choice.","section":"§4.1.2 and §4.2"},{"comment":"The claim that the multi-stage rolling method outperforms direct forecasting by 'over 30%' is stated in the abstract and introduction, but the experimental section does not quantify this comparison: Figure 3 is illustrative, and the ablations in §5 compare only rolling variants (our method versus the naive method), not against direct forecasting models. Without a table or figure showing actual PCC and TCC values for direct forecasting under the same conditions, the challenge to the ChaosBench conclusion is not supported by the presented evidence.","section":"§3 (Figure 3) and §5"},{"comment":"The theoretical explanation rests on inequality (8), |∇ℓ(Θ)|² ≥ 4L² ℓ(Θ)(1 − σ_min(Θ)), which is asserted as 'easy to verify' with no proof and no explicit statement of which lemma from the original paper is being modified. This inequality drives the convergence bound (14); if it is not established, the theoretical rationale for why curriculum-style training reduces optimization difficulty is unsupported. The authors need to provide a complete proof or a precise citation that contains this exact inequality.","section":"Appendix B, Eqs. (8)–(14)"},{"comment":"The paper reports no error bars, confidence intervals, or significance tests for any of the skill metrics, and the claims of 'significantly improves' and 'surpassing' are based on single training runs for each configuration. The differences in Table 1 (e.g., ViT 0.1168 versus 0.0807) are small in absolute PCC units, so a statement of run-to-run variability is necessary to judge whether the reported gains are reproducible and not due to training noise.","section":"§4.2 and §5"}],"minor_comments":[{"comment":"The 'Baguan' backbone is described only through a non-archival blog link; the paper should include a full architectural description or cite a peer-reviewed/preprint source so that the base model is reproducible.","section":"§3.2 and Reference [1]"},{"comment":"For stage 1, the peak and minimum learning rates are both given as 1e−4, so the schedule is constant; please state this explicitly to avoid confusion.","section":"§4.1.1"},{"comment":"The 'Depth' column (51*42 and 15*42) is undefined; clarify how depth is computed and why it is reported as the product of network depth and rollout length.","section":"Table 1"},{"comment":"The MJO evaluation uses only the year 2015; state whether this is the standard verification period and whether the result is stable across other years.","section":"§4.3"},{"comment":"The captions contain a spelling error ('comparsion'); also, the figures would benefit from axis labels defining the lead time range and the metric unit.","section":"Figures 5 and 6"},{"comment":"The text in the loss-function paragraph says 'we use use A to denote anomaly values' — the duplicated 'use' should be removed.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's reliance on a non-archival blog citation for its pretrained backbone is a reproducibility concern, but the larger issue is the prominent '19–91%' claim that is not backed by a shared-climatology evaluation. The authors should also consider comparing with recent ML S2S models (e.g., FuXi-S2S) to better situate the contribution. The manuscript fits the journal's scope, but the evidentiary bar for the central claim needs to be raised before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper deserves a serious referee, but the headline 19-91% improvement over ECMWF-S2S is not established as stated. The stress-test note is correct: the model and ECMWF are scored under different anomaly definitions, and Section 4.1.2 itself warns that a longer or fixed climatology can make results look inflated. That makes the central claim a scoring-convention question, not just a forecasting-skill question, and no shared-climatology sensitivity check is shown.\n\nWhat is actually new: a concrete multi-stage training recipe—1-day fine-tune, then 7-day rolling, then 42-day rolling with six adapter sets—that allows a rolling model to beat direct forecasting on the authors' setup, directly contradicting the ChaosBench conclusion. The ablations across ViT and UNet backbones, the scaling comparison, and the MJO extension are useful empirical scaffolding. The theoretical motivation via Jacobian products and Lyapunov exponents is standard but appropriately connects the recipe to a real training-instability mechanism. Code is provided.\n\nThe soft spots are real and in proportion: no error bars or significance tests; the main comparison averages only 6 of 10 variables, leaving out harder ones like total precipitation; the last stage uses PEFT adapters, so the multi-stage effect is confounded with the adapter effect, since the naive baseline does not get adapters; and the MJO result is based on one year. The appendix proof adapts a deep linear network analysis and asserts some inequalities without full derivation, but the theory is supporting material, not the main load. The Baguan backbone is cited only through a blog link, another reproducibility caveat until weights are released.\n\nNone of these are fatal to the direction. The optimized rolling model may well beat naive rolling and be competitive with ECMWF on equal terms. But the 19-91% figure and the \"optimization, not architecture\" causal story are not supported by the evidence as presented. The paper is useful for the S2S/ML community if the authors redo the evaluation with a shared climatology, report all variables, add uncertainty quantification, and isolate the multi-stage effect from the adapter effect. I would bring this to a reading group and would send it out for review, with the expectation of heavy revision.","headline":"Worth refereeing, but the 19-91% ECMWF beat is not established while the anomaly baselines differ and the paper itself warns that this can inflate skill.","tokens_in":14576,"tokens_out":2479,"would_cite":false,"duration_ms":22585,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","86A10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that optimization, not network architecture, is why deep learning fails at the 2-6 week scale, and a multi-stage teacher forcing schedule lifts a fixed backbone past ECMWF-S2S by 19-91%.","keywords":["subseasonal-to-seasonal forecasting","deep learning weather prediction","multi-stage optimization","teacher forcing","rolling forecasting","direct forecasting","ECMWF-S2S","anomaly correlation skill"],"falsifier":"Recompute both systems' skill on a single shared anomaly baseline by applying the same 11-day centered rolling climatology from the same reference period to the model's forecasts, ERA5, and the ECMWF-S2S reforecasts, and check whether the 19-91% advantage in PCC and TCC survives; also recompute the headline averages over all ten modeled variables, including total precipitation, rather than the six reported ones.","tokens_in":13435,"feed_emoji":"🌦️","tokens_out":13139,"duration_ms":97886,"temperature":0.7,"pith_summary":"This paper argues that the reason deep learning weather models collapse to climatology-level skill at the subseasonal-to-seasonal scale, roughly 2 to 6 weeks ahead, is not their architecture but how they are optimized. The authors build a multi-stage training recipe that lengthens the rolling forecast horizon step by step, from a one-day model to a seven-day roller to a full 42-day roller, while injecting observed values during training to interrupt error accumulation, a form of teacher forcing. With the same backbone model throughout, this recipe raises anomaly-correlation skill on six key variables so that the model surpasses the ECMWF-S2S operational system by 19-91%. The paper also claims the opposite of a recent benchmark result: properly trained rolling forecasting beats direct forecasting for S2S prediction. If correct, the result implies that existing deep learning weather models could gain subseasonal skill without any architectural redesign.","feed_headline":"Training method, not model design, lifts AI weather forecasts 19-91%","feed_subtitle":"A staged teacher-forcing schedule lets rolling deep-learning forecasts beat a top operational S2S system.","key_machinery":"The load-bearing mechanism is the multi-stage teacher forcing training schedule, understood through error accumulation in rolling prediction. When a model rolls forward, the parameter gradient contains products of Jacobians, $\\prod_{k=j+1}^{T} J_k$, so any local parameter update is amplified or damped by the spectral growth of these products along the orbit, quantified by the maximum Lyapunov exponent $\\lambda_{\\max}$; in chaotic systems a positive $\\lambda_{\\max}$ produces diverging loss gradients. The recipe attacks this in three moves: it grows the rolling horizon in stages, $T = 1$, then $T = 7$, then $T = 42$, so each stage starts from a state with smaller Jacobians; it substitutes observed values for model predictions at controlled frequencies so the multiplicative path of the Jacobians is interrupted; and it freezes most parameters in the final stage, keeping only six adapter modules, one per seven-day block, trainable. The loss functions are computed on anomalies relative to an 11-day centered rolling climatology and combine latitude-weighted MSE with latitude-weighted Pearson correlation.","core_discovery":"The paper's central claim is that optimization, not network structure, is the deciding factor in whether deep learning succeeds at subseasonal-to-seasonal forecasting. Using a fixed backbone, a Siamese-MAE-pretrained vision transformer, the authors show that training a 42-day rolling predictor end to end is unstable because the backpropagation path contains products of Jacobian matrices along the forecast orbit, whose growth is controlled by the orbit's maximum Lyapunov exponent. Their remedy is a three-stage curriculum: fine-tune the model as a one-day forecaster, then as a seven-day roller, then as a 42-day roller with parameter-efficient adapters, while teacher forcing substitutes observed values for predictions at intermediate steps to break the multiplicative path of the Jacobians. On latitude-weighted anomaly correlations for six variables, t2m, z500, t850, olr, u200, and u850, the resulting model outperforms the ECMWF-S2S system by 19-91% over the 2-to-6-week window and extends skillful MJO prediction from 23 to 30 days. The paper further claims that rolling forecasting trained this way beats direct forecasting by over 30%, directly contesting the ChaosBench finding that direct forecasting is superior for S2S.","pith_inferences":["The reported 19-91% margin may owe part of its size to a scoring asymmetry: the paper's own anomaly skill is computed against a sharp 11-day rolling climatology while ECMWF's skill figures use a fixed climatology, and the paper's own evaluation section warns that fixed or longer climatologies weaken the anomaly signal and can inflate apparent gains; a fair head-to-head needs a shared anomaly basel","The headline averages exclude four of the ten modeled variables, including total precipitation, which is typically among the hardest subseasonal targets, so the claim that deep learning surpasses ECMWF-S2S should be read as applying to the six selected variables.","The Jacobian-product analysis justifies the training schedule through a linearized model, but the paper does not directly measure the spectral norm of the actual network Jacobians; a natural test is to record $\\| \\prod J_k \\|_2$ at each training stage and check that it decreases as the schedule progresses.","If the optimization-centered explanation is right, the current S2S skill ceiling is set by training difficulty rather than by expressivity or data, which predicts that applying this recipe to stronger backbones listed in the paper, such as GraphCast or Pangu-Weather, would yield further gains, a direct extension the paper leaves to future work."],"forward_implications":["Existing deep learning weather models that score at climatology level for 2-6 week leads could be upgraded to subseasonal skill by retraining with the multi-stage teacher forcing schedule, with no change to their architecture.","The benchmark conclusion that direct forecasting beats rolling forecasting for S2S would need revision, since rolling predictors trained this way win by over 30% on correlation skills and the comparison in that benchmark is confounded by how the rolling models were optimized.","Operational S2S forecasting could shift from running costly physics-based ensemble systems to much cheaper learned models, at least for the six large-scale variables studied.","The curriculum idea of gradually lengthening the prediction horizon while injecting ground truth transfers to any iterative spatiotemporal predictor, including other weather backbones and climate emulators."],"supporting_citations":[{"why":"The ChaosBench benchmark that reports deep learning models at climatology skill for S2S and concludes direct forecasting beats rolling forecasting; the result this paper reverses.","marker":"[26]"},{"why":"The teacher forcing method for chaotic dynamics that the multi-stage framework builds on to interrupt the accumulation of prediction errors.","marker":"[16]"},{"why":"The RNN training result that a positive maximum Lyapunov exponent of the Jacobian product leads to diverging loss gradients.","marker":"[24]"},{"why":"The deep linear network convergence analysis that the appendix adapts to prove that long rolling horizons need a small initial error.","marker":"[2]"},{"why":"The Baguan Siamese-MAE-pretrained ViT backbone on which the multi-stage optimization is demonstrated.","marker":"[1]"},{"why":"Siamese masked autoencoders, the pre-training method that produces the backbone's weather features.","marker":"[14]"},{"why":"The adapter-based parameter-efficient fine-tuning used in the final 42-day stage to keep optimization stable.","marker":"[17]"},{"why":"ERA5 reanalysis, the observational dataset used for training, testing, and computing climatology.","marker":"[15]"},{"why":"The RMM index used to verify the MJO forecast skill improvement from 23 to 30 days.","marker":"[40]"}],"fun_headline_variants":["Optimization, not architecture, boosts AI forecasts up to 91%","Multi-stage training beats ECMWF S2S by 19-91%","Teacher forcing fixes Jacobian blow-up in weather AI","Staged training lifts S2S forecasts past ECMWF by 91%","Rolling forecast fix: teacher forcing beats direct prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline claim that the model beats ECMWF-S2S by 19-91% assumes the two systems are scored on comparable anomaly baselines: the paper's own forecasts use an 11-day centered rolling climatology while ECMWF's published skill uses ECMWF's own climatology, and the paper itself warns that a different or longer climatology window can make apparent skill look inflated.","fun_headline_variants_meta":{"raw":{"variants":["Optimization, not architecture, boosts AI forecasts up to 91%","Multi-stage training beats ECMWF S2S by 19-91%","Teacher forcing fixes Jacobian blow-up in weather AI","Staged training lifts S2S forecasts past ECMWF by 91%","Rolling forecast fix: teacher forcing beats direct prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000938,"raw_usage":{"total_tokens":4061,"prompt_tokens":1049,"completion_tokens":3012,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":2922}},"tokens_in":665,"tokens_out":3012,"duration_ms":19992,"temperature":1.0,"reasoning_tokens":2922,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:13:43.650629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute both systems' skill on a single shared anomaly baseline by applying the same 11-day centered rolling climatology from the same reference period to the model's forecasts, ERA5, and the ECMWF-S2S reforecasts, and check whether the 19-91% advantage in PCC and TCC survives; also recompute the headline averages over all ten modeled variables, including total precipitation, rather than the six reported ones.","supporting_citations":[{"cited_title":"On the difficulty of learning chaotic dynamics with rnns","cited_arxiv_id":null,"evidence_quote":"The RNN training result that a positive maximum Lyapunov exponent of the Jacobian product leads to diverging loss gradients."},{"cited_title":"Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks","cited_arxiv_id":null,"evidence_quote":"The deep linear network convergence analysis that the appendix adapts to prove that long rolling horizons need a small initial error."},{"cited_title":"https://damo.alibaba","cited_arxiv_id":null,"evidence_quote":"The Baguan Siamese-MAE-pretrained ViT backbone on which the multi-stage optimization is demonstrated."},{"cited_title":"Siamese masked autoencoders","cited_arxiv_id":null,"evidence_quote":"Siamese masked autoencoders, the pre-training method that produces the backbone's weather features."},{"cited_title":"Parameter-efficient transfer learning for nlp","cited_arxiv_id":null,"evidence_quote":"The adapter-based parameter-efficient fine-tuning used in the final 42-day stage to keep optimization stable."},{"cited_title":"An all-season real-time multivariate mjo index: Development of an in- dex for monitoring and prediction","cited_arxiv_id":null,"evidence_quote":"The RMM index used to verify the MJO forecast skill improvement from 23 to 30 days."}],"review_version":1}