{"id":"5d388741-cbea-43a1-b293-7786671ae4bc","arxiv_id":"2412.02722","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding a pinball-MAPE plus normalized MSE loss and a destandardization component to N-BEATS improves monthly electricity load forecasting, with reported MAPE of 3.44% versus 3.78% for N-BEATS.","lead":"An enhanced N-BEATS model, called N-BEATS*, adds a new loss function and a destandardization step inside the network blocks for mid-term electricity load forecasting. On monthly data from 35 European countries, it reports lower prediction errors than the original N-BEATS and other baseline models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that N-BEATS* beats N-BEATS and all baselines rests on baseline numbers imported from the authors' prior paper [21] under a 'similar' but not identical protocol; a re-run under the exact N-BEATS* evaluation setup is needed before the central claim is supported.","rationale":"The reader's weakest assumption correctly identifies the load-bearing risk: baseline results, especially the N-BEATS predecessor, are imported from a prior paper under a 'similar' protocol rather than re-run identically. This is upstream of all other concerns. If that comparison fails, the headline improvement and the transitive argument against all baselines collapse. Secondary issues, such as missing error bars, the negligible 0.01 IQR APE difference, and the 20/35 country win count, are real but would not independently invalidate the central claim unless the baseline comparison is sound. The paper's own Section 4.3 wording and Section 5.6 limitations support caution. A conditional verdict with a requested baseline re-run is appropriate.","tokens_in":13705,"tokens_out":5894,"duration_ms":234273,"concrete_test":"Re-run the N-BEATS baseline from [21] using the released code and hyperparameters under the exact protocol of this paper: 2014 test year, 2013 validation year, same input normalization, same 100-trial bootstrap ensemble-of-64, and identical random seeds. Recompute the Table 2 N-BEATS row and the Diebold-Mariano statistic against N-BEATS*. If the MAPE gap narrows below roughly 5% relative or the DM statistic is no longer significant at α=0.01, the central claim is not established. Ideally also re-run at least one strong baseline, such as ETS+RD-LSTM, under the same protocol to verify the full ranking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 says baseline optimization and training procedures are 'similar to those used for N-BEATS*, with hyperparameter settings described in detail in [21]', and Section 4.2 says N-BEATS* follows the approach 'established for the N-BEATS* predecessor in [21]'. The central claim therefore inherits all N-BEATS baseline metrics from [21] without re-running them under the current protocol. If [21] differed in data split, lookback length, trial averaging, ensemble size, or hyperparameter search, then the reported 9% MAPE reduction and DM statistic of -3.05 could reflect protocol changes rather than the two proposed modifications (loss function and destandardization). The transitive argument that N-BEATS was already shown to beat every baseline in [21], plus N-BEATS* beats N-BEATS here, is not a direct significance test against each baseline under identical conditions. This is the weakest load-bearing assumption, and the paper's own wording ('similar') flags it explicitly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes N-BEATS*, a modification of N-BEATS for mid-term electricity load forecasting. Two changes are introduced: (i) a hybrid loss combining pinball-MAPE with variance-normalized MSE, and (ii) a destandardization step in each block that maps block forecasts and backcasts back to the level and scale of the block input. The model is trained globally on monthly ENTSO-E data from 35 European countries and evaluated on 2014 with 2013 as validation. The authors report that N-BEATS* achieves the lowest MAPE (3.44), RMSE (304), and IQR APE (3.29) among all methods, with a Diebold-Mariano statistic of -3.05 against N-BEATS, and they provide an ablation of the main components.","tokens_in":13990,"tokens_out":7845,"duration_ms":65101,"significance":"If the empirical claims are fully supported, the contribution is a modest but useful engineering improvement to an established architecture: a 9% average MAPE reduction over the predecessor on a 35-country benchmark, together with a clean ablation showing that the loss term and its variance normalization matter most. The modifications are simple and well motivated, the experimental design has a sensible temporal split, and the comparison against the direct predecessor uses a Diebold-Mariano test. The paper is weaker as a stand-alone benchmark because the baseline numbers and the significance of N-BEATS over those baselines are imported from the authors' prior work [21] rather than re-established under the current protocol; this weakens the 'superior to all baselines' headline until the protocols are shown to be identical or the baselines are re-run.","major_comments":[{"comment":"The baseline comparison is not self-contained. The N-BEATS baseline and all other baseline metrics are taken from [21] under a protocol described only as 'similar' (Section 4.3), while Section 4.2 fixes a specific protocol for N-BEATS* (lookback 12, 100 trials, 64-model ensembles from a 1024-model pool, retraining on train+validation after hyperparameter selection). If any of those settings differed for the baselines, the reported improvement of N-BEATS* over N-BEATS (MAPE 3.78 vs 3.44; DM -3.05) could be an artifact of protocol change rather than of the two proposed modifications. Please re-run N-BEATS and, ideally, all baselines under the exact N-BEATS* evaluation protocol, or show explicitly that the protocol in [21] is identical.","section":"Section 4.2/4.3, Table 2"},{"comment":"The Diebold-Mariano evidence is reported only as the single statistic -3.05 against N-BEATS, and the claim that N-BEATS* outperforms every baseline relies on the transitive argument from [21]. The paper should specify the DM test construction (what constitutes an observation, the loss differential, the variance estimator, and how dependence among the 35 countries is handled) and provide direct DM tests against the other baselines under the current protocol, or the 'best among all baselines' claim should be softened accordingly.","section":"Section 4.5"},{"comment":"Metrics are reported as point estimates without measures of variability even though Section 4.2 states that outcomes are averaged over 100 trials. This makes it impossible to judge whether small differences (e.g., IQR APE 3.29 vs 3.30, or the ablation gaps in Table 3) are meaningful. Add confidence intervals or standard deviations across trials and/or countries for the main metrics, and use significance tests for the ablation comparisons.","section":"Sections 4.5-4.6, Tables 2-3"}],"minor_comments":[{"comment":"The notation 'Linear(h^(m))' is used in both the backcast and forecast lines, but in practice these should be two separate linear layers; please clarify the notation.","section":"Section 3.1, Eq. (1)"},{"comment":"Please define Var(y_i) explicitly: is it the variance of the target vector for sample i over the forecast horizon, and is a sample or population variance used?","section":"Section 3.2, Eq. (2)"},{"comment":"The noDestd and noReLU variant descriptions refer to 'Fig. 1' for the Mean/Std and ReLU components; these references should be to Fig. 2.","section":"Section 4.6"},{"comment":"Please clarify whether the baselines from [21] also used the merged train+validation retraining step; if not, the final-model comparison is not apples-to-apples.","section":"Section 4.2/4.3"},{"comment":"The phrase 'reduces MAPE by up to 9%' is imprecise: the average reduction is about 9% (3.78 to 3.44), while the largest country-level reduction is 38.5%; please state both values explicitly.","section":"Section 4.5"},{"comment":"The 'lowest IQR APE' claim rests on a 0.01 percentage-point difference from N-BEATS (3.29 vs 3.30); please qualify this as not materially different unless a significance test supports it.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript depends heavily on the authors' own prior paper [21] for baseline results and significance claims. This is not inherently inappropriate, but the editor may wish to ask the authors to make the comparison self-contained, either by re-running the baselines under the current protocol or by providing explicit evidence of protocol equivalence. Releasing code and a reproducible evaluation script would substantially increase confidence in the reported DM statistic and ablation results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on 2412.02722. The paper proposes two small modifications to N-BEATS for mid-term load forecasting: a loss that adds normalized MSE to the pinball-MAPE from their earlier work, and a destandardization step that rescales each block's output by the input series' mean and standard deviation. Both are natural, clearly explained, and the ablation in Table 3 shows each component pulls its weight. The empirical setup is standard: 35 European countries, monthly data, train/validation/test split with 2013 as validation and 2014 as test.\n\nThe main result is that N-BEATS* gets MAPE 3.44 vs 3.78 for N-BEATS, with a Diebold-Mariano statistic of -3.05. That looks like a real gain. The soft spot is that the N-BEATS numbers are not re-run in this paper; they're taken from the authors' prior paper [21], and the text says the baseline procedures are 'similar' rather than identical. If the earlier paper used a different split, lookback, trial count, ensemble size, or hyperparameter search, the improvement could come from protocol changes rather than the two modifications. The transitive argument that N-BEATS already beat all baselines in [21], plus N-BEATS* beats N-BEATS here, does not replace a direct head-to-head under identical conditions. I'd want the authors to re-run N-BEATS and the other baselines under the exact N-BEATS* protocol, and report error bars or confidence intervals for the metrics. The IQR APE difference (3.29 vs 3.30) is negligible, so the dispersion claim is weak.\n\nNone of this kills the paper. The modifications are sensible, the ablation is honest, and the limitations section is candid. It's an incremental contribution to a practically important task, not a breakthrough. With the baseline re-run and some measure of variability, the main claim would be solid. As it stands, it's a conditional accept.\n\nI'd send it to peer review. A serious referee would catch the baseline issue, but the paper's central idea is testable and the authors have done the obvious ablation work. Reading group? Maybe, if anyone in the group cares about load forecasting or N-BEATS variants. Would I cite it? Not in my own work, but I can see people in the forecasting niche doing so.","headline":"A modest, honestly reported N-BEATS variant whose main comparison leans on baseline numbers inherited from the authors' prior paper; worth refereeing, but the authors need to re-run the baselines.","tokens_in":14500,"tokens_out":3212,"would_cite":false,"duration_ms":29136,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An enhanced N-BEATS architecture with a hybrid pinball-MAPE/nMSE loss and block-level destandardization achieves the lowest mid-term electricity load forecast errors across 35 European countries.","keywords":["N-BEATS","mid-term load forecasting","electricity demand","deep learning","neural networks","pinball loss","normalized MSE","time series forecasting"],"falsifier":"A head-to-head replication that reruns N-BEATS and N-BEATS* under one identical protocol on the same 35-country data and 2014 test year would settle it: if the MAPE gap disappears or the Diebold–Mariano statistic fails to fall below -2.576, the superiority claim fails.","tokens_in":13536,"feed_emoji":"⚡","tokens_out":7865,"duration_ms":64212,"temperature":0.7,"pith_summary":"This paper tries to establish that a modified N-BEATS network, called N-BEATS*, gives better mid-term electricity load forecasts than the original N-BEATS and than every statistical, machine-learning, and hybrid baseline tested. The two modifications are a loss function that adds variance-normalized MSE to pinball-MAPE, and a block-level destandardization step that makes each block predict standardized forecast and backcast vectors. On monthly demand from 35 European countries with 2014 as the test year, N-BEATS* reports the lowest MedAPE (2.20%), MAPE (3.44%), RMSE (304), and interquartile error spread, with a Diebold–Mariano statistic of -3.05 against N-BEATS. The practical significance is that a univariate, preprocessing-free model could serve as a reliable mid-term forecasting component for grid planning across many countries at once.","feed_headline":"N-BEATS* cuts mid-term load forecast error to 3.44% MAPE","feed_subtitle":"Hybrid loss and destandardized blocks beat N-BEATS and all baselines across 35 countries","key_machinery":"The carrying object is the N-BEATS* block stack, an extension of the original N-BEATS architecture in which each block contains fully connected layers, a fork into forecast and backcast paths, and a new destandardization step: block outputs are rescaled by the input series' standard deviation and recentered by its mean, so the network learns shape rather than level and variance. The second mechanism is the composite loss $L(y,\\hat y,\\tau)=\\mathrm{pMAPE}(y,\\hat y,\\tau)+\\lambda\\,\\mathrm{nMSE}(y,\\hat y)$, where $\\mathrm{pMAPE}$ is the pinball form of MAPE (with $\\tau$ controlling bias and enabling quantile forecasts) and $\\mathrm{nMSE}$ divides the squared error of each target by its variance, making the L2 term scale-free and equal to 1 for a mean-based baseline. These mechanisms allow one global model trained across heterogeneous country series to produce accurate, low-dispersion forecasts without explicit detrending or deseasonalization.","core_discovery":"The paper's central claim is that N-BEATS* outperforms its predecessor N-BEATS and all benchmark models on mid-term load forecasting, and that the gains come specifically from its two architectural and objective-function modifications. In the model's own terms, destandardizing the forecast and backcast predictions inside each block makes the network solve simpler, scale-free tasks across heterogeneous country series, while the composite loss $\\mathrm{pMAPE}(y,\\hat y,\\tau) + \\lambda\\,\\mathrm{nMSE}(y,\\hat y)$ balances bias control with heavy-error suppression. Across 35 countries with a 12-month test year, N-BEATS* attains the best MedAPE, MAPE, RMSE, and IQR APE, improves MAPE by about 9% over N-BEATS, and is the only tested model with a positive mean percentage error (0.56%), indicating a mild underprediction tendency rather than systematic overprediction. The ablation study shows that removing the normalized MSE term or its variance normalization degrades accuracy most, which the paper reads as evidence that the loss reweighting, not the architecture alone, drives the improvement.","pith_inferences":["The destandardization mechanism is transferable: any global deep forecaster facing heterogeneous series could adopt it, so the architecture change may benefit retail, traffic, or water-demand forecasting without the electricity-specific loss.","The evaluation covers a single test year (2014); a rolling-origin re-test over several years would reveal whether N-BEATS*' edge persists through different economic and weather regimes.","The reported positive MPE (0.56%) is tied to the chosen $\\tau=0.35$; tuning $\\tau$ per country or per season could reduce bias further than the fixed global value used here.","Because the model is trained on all countries jointly, countries with short histories may benefit more from cross-learning; a per-country sample-size analysis could identify where the gain concentrates."],"forward_implications":["Deployment becomes simpler: N-BEATS* needs only the historical demand series as input, so utilities can forecast a full year ahead without collecting weather or economic covariates.","The pinball parameter $\\tau$ gives operators a direct dial to shift forecasts toward over- or underprediction, matching the asymmetry of their error costs.","The ablation points to the loss design, especially variance-normalized L2, as the main accuracy driver; future work should concentrate on loss refinements before changing the network topology.","Because the model trains globally on 35 series, adding a new country is a matter of retraining the shared model, not building a per-series forecaster.","The ability to output quantiles through the pinball component means the same architecture could produce probabilistic mid-term forecasts, not just point predictions."],"supporting_citations":[{"why":"Supplies the original N-BEATS architecture whose block stack and residual connections N-BEATS* modifies.","marker":"[20]"},{"why":"Provides the predecessor N-BEATS model for mid-term load forecasting, the pinball-MAPE loss, the N-BEATS baseline numbers in Table 2, and the sampling scheme reused here.","marker":"[21]"},{"why":"Supplies the pattern-similarity baselines k-NNw+ETS, FNM+ETS, N-WE+ETS, and GRNN+ETS, plus the dataset visualizations.","marker":"[10]"},{"why":"Provides the ETS+RD-LSTM hybrid baseline and the hierarchical ensemble forecasting methodology.","marker":"[12]"},{"why":"Provides the auto.arima and ets implementations used for the ARIMA and ETS statistical baselines.","marker":"[15]"},{"why":"Supplies the MLP baseline model for pattern-based monthly load forecasting.","marker":"[24]"},{"why":"Supplies the LSTM baseline used in the comparison.","marker":"[25]"},{"why":"Supplies the ANFIS neuro-fuzzy baseline used in the comparison.","marker":"[23]"}],"fun_headline_variants":["N-BEATS*: 9% better load forecasts via hybrid loss","Hybrid loss and destandardized blocks slash forecast error","N-BEATS* beats all baselines in 35-country test","Mid-term load forecasts: N-BEATS* sets new MAPE low","Enhanced N-BEATS trims forecast error to 3.44% MAPE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on the assumption that the N-BEATS baseline numbers taken from the earlier study [21] were obtained under the same data split, trial count, ensemble configuration, and hyperparameter protocol as N-BEATS*; if those setups differed, the reported gain could come from the procedure rather than the model changes.","fun_headline_variants_meta":{"raw":{"variants":["N-BEATS*: 9% better load forecasts via hybrid loss","Hybrid loss and destandardized blocks slash forecast error","N-BEATS* beats all baselines in 35-country test","Mid-term load forecasts: N-BEATS* sets new MAPE low","Enhanced N-BEATS trims forecast error to 3.44% MAPE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1303,"prompt_tokens":979,"completion_tokens":324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":225}},"tokens_in":595,"tokens_out":324,"duration_ms":3437,"temperature":1.0,"reasoning_tokens":225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:00:57.364898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A head-to-head replication that reruns N-BEATS and N-BEATS* under one identical protocol on the same 35-country data and 2014 test year would settle it: if the MAPE gap disappears or the Diebold–Mariano statistic fails to fall below -2.576, the superiority claim fails.","supporting_citations":[{"cited_title":"N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, in: ICLR","cited_arxiv_id":null,"evidence_quote":"Supplies the original N-BEATS architecture whose block stack and residual connections N-BEATS* modifies."},{"cited_title":"A hybrid residual dilated LSTM and exponential smoothing model for midterm electric load forecast- ing","cited_arxiv_id":null,"evidence_quote":"Provides the ETS+RD-LSTM hybrid baseline and the hierarchical ensemble forecasting methodology."},{"cited_title":"Forecasting: Principles and Practice","cited_arxiv_id":null,"evidence_quote":"Provides the auto.arima and ets implementations used for the ARIMA and ETS statistical baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MLP baseline model for pattern-based monthly load forecasting."},{"cited_title":"Pattern-based forecasting monthly electricity demand using multilayer perceptron, in: Artificial Intelligence and Soft Computing, Springer International Publishing, Cham","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM baseline used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ANFIS neuro-fuzzy baseline used in the comparison."}],"review_version":1}