{"id":"e4c34848-acc8-47cc-ab99-909e34963cb7","arxiv_id":"2506.15217","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Expert aggregation algorithms, especially BOA and MLpol with a gradient trick, improved deterministic 2m temperature forecasts at 33 French stations and matched the best fixed convex combination in hindsight.","lead":"This paper tests online algorithms that combine many weather model forecasts into a single temperature prediction for the next hours or days. It reports small but consistent accuracy gains over any single model, with two algorithms matching the best possible fixed combination in hindsight.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 and Table 3 report different RMSEs for the same benchmarks: best convex combination 1.23 vs 1.24 and uniform aggregation 1.41 vs 1.40 despite identical expert sets.","rationale":"The reader's verdict was CONDITIONAL, citing missing uncertainty quantification and an unsupported National Blend of Models claim. My stress-test confirms the central claim should remain conditional, but for a more specific technical reason: the oracle benchmarks against which the claim is measured are internally inconsistent across tables. This is more load-bearing than the absence of confidence intervals because a confidence interval would not resolve a 1.23 vs 1.24 discrepancy; the benchmark value itself changes. The uniform aggregation row is a clean control: with weights fixed at 1/N, the gradient trick cannot alter predictions, so a change from 1.41 to 1.40 indicates either a typo, different data handling, or a silent change in what the row represents. The convex oracle likewise cannot depend on the aggregation's internal loss. The conclusion that BOA and MLpol equal the oracle is based on two-decimal rounded values, and full-precision comparison plus a clear statement of the oracle objective is needed. This is an internal correctness issue, not a disagreement with community consensus. The algorithms are specified and the data are promised, so the fix is straightforward: recompute the benchmark rows and report exact values. If the corrected oracle is 1.23, the headline claim weakens to 'close to, but not equal to, the oracle'; if it is 1.24, the paper must explain why Table 2 differs. Either way, conditional acceptance pending this verification remains appropriate. My agreement with the reader is partial because both readings focus on the reliability of the oracle comparison, but the reader attributed the risk to in-sample artifact, whereas the concrete cross-table inconsistency is a sharper and directly checkable problem.","tokens_in":22983,"tokens_out":9790,"duration_ms":97403,"concrete_test":"Recompute the benchmark rows in Tables 2 and 3 from the same stored per-station, per-lead-time forecast sequences: (i) RMSE of the uniform weight vector over all experts including PEARP quantiles; (ii) RMSE of the fixed convex weight vector minimizing the pooled square loss over the full period; and (iii) RMSE of the fixed convex weight vector minimizing the pooled linearized loss used in the gradient trick. If (i) and (ii) are not identical across the two tables, the reported tables contain an error. If (iii) differs from (ii), the Table 3 oracle is the linearized-loss oracle and must be relabeled. Then compare BOA and MLpol RMSEs to both oracles at full precision, and report per-sequence signed differences with a paired bootstrap interval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 3 is that BOA and MLpol attain the same RMSE as the best fixed convex combination of experts (Table 3, both 1.24°C). For this comparison to be meaningful, the oracle value must be computed identically whenever the expert set is the same. Tables 2 and 3 use the same expert set (with PEARP quantiles), but the reported oracle is 1.23°C in Table 2 and 1.24°C in Table 3. The best convex combination minimizes squared error over fixed weight vectors; it does not depend on whether the aggregation uses the gradient trick, so its RMSE cannot change between these tables. Similarly, the uniform aggregation row is fixed at weights 1/N and therefore must also be invariant to the gradient trick, yet it changes from 1.41°C in Table 2 to 1.40°C in Table 3. The only ways to reconcile the discrepancy are a silent redefinition of the oracle (for example, optimizing the linearized loss used in the gradient trick rather than the square loss), a data-processing difference, or a rounding artifact. If the oracle was redefined, then Table 3's 'best convex combination' is not the hindsight-optimal fixed mix under square loss, and the conclusion that BOA and MLpol match that mix is unsupported. If the discrepancy is an error or rounding effect, the reported equality is not trustworthy at the precision claimed. Since the headline empirical equality rests entirely on these two-decimal values, the paper must resolve this inconsistency before the central claim can be evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the use of online expert aggregation algorithms (EWA, BOA, MLprod, MLpol) for deterministic 2-meter temperature forecasting, combining raw and post-processed NWP forecasts (AROME, ARPEGE, IFS/CEP) with quantiles of the post-processed PEARP ensemble. Experiments are run on 33 French stations and lead times from 6 to 96 hours, with one aggregation per station and lead time. The paper reports RMSE improvements over the best single expert and the uniform average, and claims that, when the gradient trick is used, BOA and MLpol attain the same RMSE as the best fixed convex combination of experts in hindsight. A sliding-window variant is also studied to improve reactivity to biased experts.","tokens_in":23199,"tokens_out":5895,"duration_ms":61519,"significance":"If the central claim holds, the paper makes a useful applied contribution: it demonstrates that online, adaptive aggregation can reproduce the hindsight-optimal fixed mix of operational deterministic temperature forecasts without knowing that mix in advance, and that this is achieved with simple algorithms and no retraining. The paper has clear strengths: it uses a real multi-station, multi-lead-time operational dataset; it compares several state-of-the-art aggregation strategies; it provides a detailed discussion of their regret bounds and practical behavior; and it makes the data available on GitHub. The main caveat is that the empirical support is currently only in the form of two-decimal point estimates, and one of the key tables appears internally inconsistent, so the headline equality between the online algorithms and the oracle is not yet established.","major_comments":[{"comment":"The rows for the best convex combination and the uniform aggregation are not invariant under the gradient trick, although they must be. Table 2 (without the gradient trick, with PEARP quantiles) reports a best convex combination of 1.23°C and a uniform aggregation of 1.41°C; Table 3 (with the gradient trick, with PEARP quantiles) reports 1.24°C and 1.40°C for the same expert set. The best fixed convex combination minimizes cumulative squared error over the simplex and is independent of how the aggregation weights are updated, and the uniform aggregation is the fixed weight vector 1/N, so neither value can change between the two tables. If the oracle in Table 3 was instead computed using the linearized loss from the gradient trick, that is a different benchmark and must be stated explicitly. Otherwise one of the two tables is incorrect. Since the paper's central claim that BOA and MLpol 'have the same RMSE as this oracle' rests on the 1.24°C equality in Table 3, this inconsistency must be resolved before the claim can be evaluated.","section":"Section 3, Tables 1–3 and Figure 8"},{"comment":"All RMSE comparisons are point estimates pooled over stations and lead times, reported to two decimal places, with no confidence intervals, standard errors, or significance tests. Differences of 0.01°C (for example, 1.25 vs. 1.24) are interpreted as meaningful, and the text states that 'BOA has always been the best RMSE for all sliding windows' on the basis of such differences. The authors should add uncertainty quantification, for example by bootstrap over stations and lead times or by paired forecast-error tests. They should also state explicitly that the best convex combination oracle is computed on the same evaluation period used for the online algorithms; if so, the equality with BOA and MLpol should be framed as a comparison with an in-sample hindsight benchmark rather than as an out-of-sample claim.","section":"Abstract"},{"comment":"The abstract claims that the proposed methods 'show that they outperform the National Blend of Models,' but the National Blend of Models is never defined, described, or analyzed anywhere in Sections 3 or 4, and no corresponding row or table is provided. This claim is therefore unsupported by the manuscript as written. The authors should either add the NBM comparison and describe the data used for it, or remove the claim from the abstract and introduction.","section":"Section 3, Data and Figure 3"}],"minor_comments":[{"comment":"The lead-time description is inconsistent: the text lists the additional lead times as 57, 72 and 96 hours, while Figures 3, 5 and 6 and the discussion use an 84-hour lead time. Please align the data description with the figures and the text.","section":"Section 3, Data and Figure 3"},{"comment":"The captions of Figure 7 and Figure 9 contain the typo 'Weigths' instead of 'Weights', and the caption of Table 1 says 'MLpod' instead of 'MLprod'.","section":"Figure captions"},{"comment":"The notation for the sliding window is ambiguous: replacing a cumulative sum $X_t = \\sum_{s=1}^t x_s$ by $X_t = \\sum_{s=t-w}^t x_s$ is understandable, but the accompanying text 'each variable of the form $X_t = \\min(x_s, 1 \\le s \\le t)$ by $X_t = \\min(x_s, t-w \\le s \\le t)$' is not written in standard notation and should be clarified.","section":"Section 3, Sliding Window"},{"comment":"The manuscript contains duplicated passages: the discussion of the Chamonix weights appears twice around Figure 7 and Figure 9, and the conclusion text is repeated near the end of the file. These duplicates should be removed.","section":"General presentation"}],"recommendation":"major_revision","confidential_remarks":"The paper is a promising application of expert aggregation to operational temperature forecasting, and the dataset and experimental setup are valuable. However, the inconsistency between the oracle and uniform-aggregation rows in Tables 2 and 3 directly affects the paper's headline claim, and the abstract contains a National Blend of Models claim that is not supported by the body. These are fixable, but they require a careful re-run or redefinition of the benchmarks and an explicit uncertainty analysis. I therefore recommend major revision rather than acceptance at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful operational study, but the headline result has a hole in it. The paper applies classical expert aggregation (EWA, MLprod, MLpol, BOA) to deterministic 2-meter temperature forecasts, combining raw NWP models, their MOS post-processing, and quantiles of the PEARP ensemble. It shows real RMSE improvements (from 1.37°C for the best single expert to 1.24-1.28°C for the aggregations) on a substantial French station dataset. That part is solid and likely reproducible.\n\nThe new thing is the specific application, not the algorithms—all of them predate the paper and the regret bounds are imported from prior work. That's fine. The authors are honest about limitations, especially the aggregations' inability to quickly switch weight to biased experts that occasionally perform well.\n\nThe soft spot is a genuine internal inconsistency. In Table 2 (no gradient trick, with PEARP quantiles) the best convex combination oracle has RMSE 1.23°C and uniform aggregation 1.41°C. In Table 3 (with gradient trick, same expert set) the same oracle is reported as 1.24°C and the uniform aggregation as 1.40°C. Neither of these should depend on whether the aggregation uses the gradient trick. The oracle is defined by minimizing square loss over fixed weight vectors; the uniform aggregation is fixed at equal weights. So either the oracle was silently redefined (e.g., to the linearized loss) or there is an artifact in the computation. The paper's central claim—that BOA and MLpol match the best convex combination at 1.24°C—rests entirely on that number. If the true oracle is 1.23°C, the equality does not hold.\n\nBeyond that, the statistical analysis is thin. All comparisons are point estimates without confidence intervals or significance tests; differences of 0.01°C are treated as meaningful. The abstract mentions outperforming the National Blend of Models, but the body never compares to it. The 'best compound expert' oracle (0.41°C) is an in-sample hindsight bound, not a fair target.\n\nI'd send this to a serious referee anyway. The application is real, the data are described well enough for reproduction, and the main inconsistency is fixable. The authors need to state exactly how the oracle was computed, correct the tables, and add some uncertainty quantification. Until then, I wouldn't cite the oracle-matching claim, but the general finding that EA improves over individual forecasts is probably solid.","headline":"The paper's oracle-matching claim is undermined by a table inconsistency, but the study is worth a serious referee.","tokens_in":23814,"tokens_out":5789,"would_cite":false,"duration_ms":52108,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Online expert aggregation matches the best fixed model blend for temperature forecasts.","keywords":["expert aggregation","temperature forecasting","numerical weather prediction","online learning","model output statistics","gradient trick","second-order regret bounds","sliding window"],"falsifier":"Re-run the four aggregations on a temporal holdout (for instance, the final year of the dataset) or bootstrap the RMSE distribution across stations and lead times: if BOA and MLpol no longer tie the best fixed convex combination within sampling error, or if the aggregations no longer beat the best single expert, the central claim fails.","tokens_in":22713,"feed_emoji":"🌡️","tokens_out":13699,"duration_ms":124479,"temperature":0.7,"pith_summary":"This paper claims that online expert aggregation—simple algorithms that reweight weather-model forecasts as observations arrive—can produce deterministic two-metre temperature forecasts that beat every individual numerical weather prediction model, including post-processed ones, across 33 stations and lead times from 6 to 96 hours. With the gradient trick and five quantiles of the post-processed PEARP ensemble added as experts, EWA, BOA, MLprod, and MLpol reach root mean squared errors of 1.24 to 1.25 degrees Celsius, compared with 1.37 for the best single expert and 1.40 to 1.41 for a uniform average. BOA and MLpol attain the same RMSE as the best fixed convex combination of experts chosen in hindsight (1.24), without ever knowing that combination in advance. The paper also reports that the aggregations outperform the National Blend of Models and that a sliding window trades a small loss of average accuracy for greater reactivity. It acknowledges a limitation: the aggregations are too slow to switch weight to biased quantile experts during brief anomalous events such as the December 2022 cold spell at Chamonix.","feed_headline":"Online forecast blending matches the best hindsight mix at 1.24°C","feed_subtitle":"Online reweighting of raw and post-processed forecasts beats every single model, with no retraining.","key_machinery":"The machinery is convex online aggregation with regret guarantees. At each step the forecast is $\\hat y_t = \\sum_i w_{i,t}x_{i,t}$ with weights in the simplex, updated only from past losses, and the square loss $\\ell(x,y)=(x-y)^2$ is used throughout. The gradient trick replaces the loss by its linearization $\\nabla\\ell_t(\\hat y_t)\\cdot q = \\ell_t'(\\hat y_t)\\sum_i q_i x_{i,t}$, letting the aggregations compete against any fixed convex combination of experts rather than only the best single expert. The second-order algorithms (BOA, MLpol, MLprod) use the excess loss $\\ell_{i,t}^{\\mathrm{exc}} = \\ell_t(\\delta_i)-\\sum_j w_{j,t}\\ell_t(\\delta_j)$ and the variance of past losses to set per-expert learning rates, which is what makes them adaptive without knowing $T$ or loss bounds in advance. A squared-error decomposition into the experts' mean error minus their diversity motivates adding the biased PEARP quantile experts.","core_discovery":"The paper's central claim is that online expert aggregation, a framework designed for worst-case adversarial prediction, transfers to deterministic temperature forecasting, even when the experts are already post-processed forecasts whose errors are nearly random. On a dataset of 33 stations in France from March 2020 to September 2023 at lead times 6 to 96 hours, the paper reports that EWA, BOA, MLprod, and MLpol, run with the gradient trick and five post-processed PEARP quantiles as additional experts, all achieve RMSEs between 1.24 and 1.25 degrees Celsius. That is below the best single expert (1.37), the uniform average (1.40–1.41), and the National Blend of Models baseline; BOA and MLpol land exactly on the RMSE of the best fixed convex combination of experts computed in hindsight, 1.24, without knowing that combination during the forecast. The paper further argues that second-order aggregations are better suited than EWA to this non-adversarial setting, and that adding deliberately biased quantile experts helps the hindsight oracles but is hard for the online rules to exploit in real time.","pith_inferences":["If the equality between BOA and MLpol and the hindsight-optimal combination survives a genuine out-of-sample test, it would suggest that static weight optimization has little to add over online rules for this class of forecasts.","Because the aggregation rules require only a convex loss, a testable extension is probabilistic aggregation with a proper scoring rule such as the continuous ranked probability score (CRPS); the paper demonstrates deterministic square loss only.","The Chamonix December 2022 episode shows a concrete failure mode: cumulative-loss weights are too slow to promote experts that are good only during a short cold spell. A direct test of tracking-the-best-sequence or sleeping-expert aggregations would show whether reactivity can be restored without the sliding-window RMSE penalty."],"forward_implications":["Operational temperature guidance can be improved by roughly 0.12 to 0.13 degrees Celsius RMSE over the best single model simply by running an online reweighting scheme on forecasts that already exist.","Because the aggregations are online and adaptive, they can keep working across NWP model upgrades without retraining, which the paper notes is a weakness of neural-network post-processing.","With the gradient trick and quantile experts, BOA and MLpol recover the RMSE of the hindsight-optimal fixed convex combination (1.24), so the oracle is reachable without knowing the optimal weights in advance.","For short evaluation periods (under roughly 250 iterations) BOA and MLprod are the fastest learners; for long periods BOA is the most consistent of the four strategies.","Sliding windows make the weights visibly more reactive, but for BOA and MLpol they cost about 0.01 degrees Celsius in RMSE (1.25 vs 1.24), so reactivity and average score are in tension."],"supporting_citations":[{"why":"Supplies the EWA algorithm, its regret bound, and the gradient trick that permits competing with the best convex combination of experts.","marker":"[Cesa-Bianchi and Lugosi, 2006]"},{"why":"Supplies the MLprod and MLpol algorithms and their second-order excess-loss regret bounds used in the main experiments.","marker":"[Gaillard et al., 2014]"},{"why":"Supplies the BOA algorithm, the adaptive second-order aggregation that achieves the best RMSE over all sliding windows.","marker":"[Wintenberger, 2024]"},{"why":"Supplies the post-processed PEARP quantile forecasts used as biased experts; adding them improves the hindsight convex-combination oracle.","marker":"[Taillardat and Mestre, 2020]"},{"why":"Supplies the adaptive learning-rate scheme for EWA and the second-order regret framework that the other algorithms refine.","marker":"[Cesa-Bianchi et al., 2007]"},{"why":"Supplies the sliding-window modification tested for reactivity in the final section.","marker":"[Zamo et al., 2021]"}],"fun_headline_variants":["Adaptive forecast blending hits 1.24°C RMSE, matching offline optimum","Online blending ties best hindsight forecast mix at 1.24°C","No retraining: online blending matches best hindsight forecast mix","Forecast blend matches retroactive ideal, no retraining needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one pooled RMSE number per method, computed on the same data used to define the oracles, is a stable basis for comparing forecast methods; without confidence intervals or a temporal holdout, the small reported gaps and the equality with the hindsight-optimal mix could be in-sample artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive forecast blending hits 1.24°C RMSE, matching offline optimum","Online blending ties best hindsight forecast mix at 1.24°C","No retraining: online blending matches best hindsight forecast mix","Forecast blend matches retroactive ideal, no retraining needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001854,"raw_usage":{"total_tokens":7287,"prompt_tokens":952,"completion_tokens":6335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":6258}},"tokens_in":568,"tokens_out":6335,"duration_ms":42073,"temperature":1.0,"reasoning_tokens":6258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:40:39.428571+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four aggregations on a temporal holdout (for instance, the final year of the dataset) or bootstrap the RMSE distribution across stations and lead times: if BOA and MLpol no longer tie the best fixed convex combination within sampling error, or if the aggregations no longer beat the best single expert, the central claim fails.","supporting_citations":[{"cited_title":"Improved second-order bounds for prediction with expert advice","cited_arxiv_id":null,"evidence_quote":"Supplies the adaptive learning-rate scheme for EWA and the second-order regret framework that the other algorithms refine."},{"cited_title":"Sequential aggregation of probabilistic forecasts— Application to wind speed ensemble forecasts","cited_arxiv_id":null,"evidence_quote":"Supplies the sliding-window modification tested for reactivity in the final section."}],"review_version":1}