{"id":"2326307b-b4ae-4a41-8962-2b81cdaa59b9","arxiv_id":"2501.05000","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"For energy communities with under six months of data, persistence forecasting beats deep learning, and KNN achieves nearly the same battery cost savings as the best neural network.","lead":"This study compares deep learning load forecasters (LSTM, xLSTM, Transformer) with simple baselines (persistence, KNN) across energy communities of different sizes and data histories. It finds that simple models are competitive or better when data is scarce, and nearly match deep learning in financial savings from battery optimization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The six-month persistence-beats-DL threshold is validated only on Q4 2013; Appendix B's seasonal check uses future data and no training-size sweep, so the central threshold remains seasonally unverified.","rationale":"The reader's weakest assumption is the same one I would target: the persistence-vs-DL threshold is established on a single test season. I sharpen it with a concrete internal problem: Appendix B's seasonal robustness results are not a valid validation because Q1–Q3 test sets are trained on later quarters (future leakage), and because the three-quarter training split changes the training-size regime. The concern is load-bearing because the paper's headline 'less than six months' is a quantitative boundary used to advise ECs when DL is not worth the effort; if that boundary moves with season, the advice is wrong for a large share of the year. I do not see an internal inconsistency that would reject the paper; the Q4 results and public code are real evidence. The fix is a proper chronological seasonal sweep, which the dataset can partially support (Q1 max 6 months, Q2 max 9, Q3 max 12, Q4 max 15). Because this is exactly the kind of scope qualification the reader's CONDITIONAL verdict already calls for, I do not move the verdict; I leave it CONDITIONAL/UNCHANGED.","tokens_in":17815,"tokens_out":9818,"duration_ms":94793,"concrete_test":"Reproduce the training-size sweep for each 2013 quarter using only chronologically preceding data: Q1 2013 test with 2/4/6 months of training (Jul–Dec 2012), Q2 with 2/4/6/9 months, Q3 with 2/4/6/9/12 months, Q4 with the original 2/4/6/9/12/15 months; use the same 20 ECs, models, and hyperparameters. For each quarter and each training size, compute the paired difference in nMAE (20 ECs) between persistence and each deep learning model. If at any quarter a DL model has mean nMAE below persistence at 4 or 6 months with a paired confidence interval excluding zero, the six-month threshold is seasonal and the central claim must be qualified accordingly; if persistence remains lower in all quarters, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline quantitative claim—persistence outperforms LSTM/xLSTM/Transformer below six months of training data—is built exclusively on Q4 2013 as the test season (Section 2.3, Fig. 8, Table A.1 training-size rows). Appendix B is meant to establish seasonal robustness, but it does not test this claim. For Q1, Q2 and Q3 test quarters, the 'remaining three quarters' are used as training; for Q1 this means Q2–Q4 2013, i.e., future data relative to the test period, a leakage the authors themselves call unrealistic (Appendix B). Moreover, these seasonal runs use a fixed three-quarter training window (~9 months), not the 2/4/6-month sweep, so they cannot reveal whether the persistence advantage at small training sizes persists in other seasons. If within-season persistence is weaker in, say, Q1 or Q2, the 'less than six months' threshold could be an artifact of testing only on October–December. The claim as stated is therefore more fragile than the paper's tone suggests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks three deep-learning load forecasters (LSTM, xLSTM, Transformer) against persistence and KNN for day-ahead forecasting of virtual energy communities built from 4,500 London smart-meter profiles. It reports three main results: persistence beats all deep-learning models when training data are below six months; transfer learning from German synthetic load profiles improves mean nMAE by 1.97 percentage points when only two months of target data are available; and in a MILP battery-storage case study, KNN achieves financial savings (8.01% for 50 households) nearly identical to the best deep-learning model (8.06%). The paper also analyzes sensitivity to community size and model size, and it releases code and data for reproducibility.","tokens_in":18024,"tokens_out":10205,"duration_ms":93524,"significance":"If the headline threshold holds, the paper provides a practically useful rule for model selection in new energy communities, and it is one of the few studies that combine a broad benchmark, transfer learning, and an economic optimization case study in one framework. The transfer-learning figure is concrete and can be verified directly from Tables A.1 and A.2, and the KNN-versus-deep-learning savings comparison in Table A.3 is a valuable, credible result. The central limitation is that the persistence-beats-deep-learning threshold is established on a single test season, so the practical significance of the main claim depends on closing this seasonal-generalizability gap.","major_comments":[{"comment":"The central claim that persistence outperforms LSTM, xLSTM, and Transformer with less than six months of training data is established only on the Q4 2013 test period. Appendix B reports results for Q1-Q3 test quarters, but those runs use a fixed three-quarter training window of about nine months and, for Q1, Q2, and Q3, the training set includes future quarters (Q2-Q4, Q3-Q4, and Q4 of 2013, respectively), which the authors themselves describe as unrealistic. These runs therefore do not test whether the six-month threshold also holds in other seasons. Please either run the 2/4/6-month training-size sweep for other test quarters using only past data, or qualify the threshold claim as specific to the Q4 test period.","section":"Section 2.3, Fig. 8, Appendix B"}],"minor_comments":[{"comment":"The sentence 'with nine months of training data or more, the deep learning models outperform persistent prediction, with a difference in nMAE ranging from 1.8 percentage points to 2.0 percentage points' is not supported by Table A.1: at nine months the differences from persistence (12.79%) are 1.23, 1.23, and 1.59 percentage points for xLSTM, LSTM, and Transformer, respectively. The 1.8-2.0 range appears only at 12-15 months.","section":"Section 3.2, Fig. 8"},{"comment":"The abstract says 'less than six months' while the Conclusions say 'six months or less'; since the 6-month rows in Tables A.1 and A.2 still favor persistence, please choose one formulation and use it consistently.","section":"Abstract and Conclusions"},{"comment":"The xLSTM entries for 0.1k and 0.2k model sizes are marked '-' although Table C.5 lists configurations for these sizes; please clarify whether these runs were omitted or failed to converge.","section":"Table A.1"},{"comment":"The perfect-prediction scenario omits Eq. 7 while all forecast-based scenarios include it, so the savings gap between perfect and imperfect forecasts is not solely attributable to forecast quality; please state this explicitly when interpreting the perfect-prediction baseline.","section":"Section 2.4"},{"comment":"The caption says 'The column shows the mean over 20 ECs' but the figure appears to display distributions; please specify exactly what the boxes, whiskers, and markers represent.","section":"Figure 9"},{"comment":"For the decisive 6-month comparison, please report paired confidence intervals or a test statistic, because the LSTM-versus-persistence difference (13.46% vs. 12.79%) is small relative to the reported standard deviations.","section":"Section 3.2, Table A.1"},{"comment":"Contribution 2 calls the transfer-learning approach 'novel', but Section 2.2.2 states that the same approach was already applied in reference [25]; please clarify what is genuinely new in the present application.","section":"Introduction, Contribution 2 and Section 2.2.2"},{"comment":"The 10-household, 120 kWh case referenced in the Conclusions is not a separate row in Table A.4; adding a note that this configuration corresponds to the 10-household row of Table A.3 would prevent reader confusion.","section":"Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for Next Energy, and the public code/data release is a clear strength. The main technical gap is the seasonal generalizability of the headline six-month threshold; the Appendix B robustness check does not address it because it uses a fixed nine-month training window and, for three test quarters, future training data. If the authors can either add the training-size sweep for other quarters with past-only data or appropriately qualify the claim, the paper would be publishable. There are no concerns about attribution or citation practice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid, honestly-reported empirical benchmark, and the main results are worth knowing: for the London smart meter data, persistence beats LSTM/xLSTM/Transformer when training data is below about six months, transfer learning from German synthetic profiles cuts nMAE by ~2 points at two months of data, and in the battery optimization case study KNN matches the best deep learning model's cost savings within 0.05 points. The GitHub repo with code and data is a real plus; this is reproducible work.\n\nWhat's new: prior work on EC forecasting rarely sweeps community size, training length, and model capacity together, and no one had shown the data threshold or the KNN cost-parity result. The xLSTM short-term application is minor but fairly evaluated.\n\nThe soft spots are in proportion. First, the headline threshold is built entirely on Q4 2013 as test period. The appendix's seasonal check uses a 9-month training window and no training-size sweep, so it doesn't actually test whether persistence keeps its edge at 2/4/6 months in other seasons. The authors claim Q4 is 'representative' but that's an extrapolation. I don't think this sinks the paper—the threshold could easily hold year-round—but the claim is more fragile than the tone suggests.\n\nSecond, the text says the 9+ month nMAE improvement over persistence is 1.8-2.0 percentage points; from Table A.1 the 9-month gap is 1.2-1.6. The 1.8-2.0 figure only appears at 12-15 months. Minor overstatement, but it should be fixed. Third, no significance tests or confidence intervals. With 20 communities and overlapping standard deviations, some of the cross-model differences are probably noise. The paper should at least acknowledge this or run paired tests.\n\nThe KNN vs DL financial parity is the most robust practical finding, and it's well-supported by Table A.3.\n\nWho is this for? Practitioners setting up MPC in energy communities, and researchers benchmarking load forecasting. It deserves a serious referee; it's not a desk reject. My recommendation: send to review, require the authors to fix the 9-month claim, add a season-split training-size sweep or clearly label the threshold as Q4-specific, and put a significance test or an explicit error-bar comment on the cross-model comparisons.\n\nOverall, a worthwhile paper that does what it claims, with a few rough edges.","headline":"A useful, reproducible empirical benchmark with a practical data-threshold finding, but the headline six-month persistence advantage is tested on one season only and the text overstates the nine-month improvement.","tokens_in":18606,"tokens_out":3234,"would_cite":true,"duration_ms":29205,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For energy-community load forecasting, deep learning is not always worth the effort: with fewer than six months of training data, persistence forecasting is more accurate, and a simple KNN model nearly matches the best deep learning…","keywords":["load forecasting","energy communities","deep learning","transfer learning","persistence forecasting","K-nearest neighbors","battery storage optimization","model predictive control"],"falsifier":"Take a multi-year household dataset, repeat the training-size sweep (2, 4, 6, 9 months) with each season as the held-out test period, and check whether persistence still beats the deep learning models for every training length below six months in every season; if it does not, the six-month threshold claim is false.","tokens_in":17594,"feed_emoji":"⚡","tokens_out":6876,"duration_ms":57193,"temperature":0.7,"pith_summary":"This paper asks whether deep learning models are worth the effort for short-term load forecasting in households and energy communities. It claims that when fewer than six months of historical data are available, a simple persistence baseline—tomorrow's load equals the load of seven days ago—predicts better than LSTM, xLSTM, and Transformer models. It also claims that transfer learning from synthetic German load profiles reduces the normalized mean absolute error (nMAE) by about 1.97 percentage points when only two months of data are available, and that in a battery-optimization case study a KNN model achieves nearly the same cost savings as the best deep learning model (8.01% versus 8.06% for 50 households). The practical upshot is a data-availability threshold and a rule of thumb for when advanced models pay off.","feed_headline":"Persistence beats deep learning under six months of data","feed_subtitle":"Simple baselines match deep nets in energy forecasting, and KNN nearly ties the best model's battery savings.","key_machinery":"The argument is carried by a controlled sensitivity grid: each configuration is evaluated on 20 randomly drawn virtual energy communities, with community sizes of 1, 2, 10, 50, and 100 households, training windows of 2 to 15 months, model sizes from roughly 0.1k to 80k parameters, and a fixed Q4 2013 test period. The load-bearing baselines are persistence (a naive seasonal forecast repeating the load from seven days earlier) and KNN with k=40. Transfer learning pre-trains the deep models on synthetic German load profiles and fine-tunes on the target community. The financial leg is a mixed-integer linear program that optimizes charging and discharging of a shared battery, taking the forecast as input and settling deviations through the grid; this converts forecast error into a monetary savings number.","core_discovery":"The central claim is that for day-ahead load forecasting in energy communities, deep learning models are not universally superior to simple benchmarks. On the paper's test setup (Q4 2013, 20 virtual communities per configuration), persistence forecasting has lower nMAE than LSTM, xLSTM, and Transformer models for training windows of 2, 4, and 6 months; the deep models only overtake persistence once 9 to 12 months of training data are available, and the gain plateaus around 12 months. Transfer learning from publicly available synthetic profiles (German standard load profiles) improves the deep models' nMAE by 1.97 percentage points on average in the two-month data setting, but the benefit fades with more data. In the financial case study, a mixed-integer linear program scheduling a shared community battery under a day-ahead real-time price produces average cost savings of 8.06% for the best deep learning model and 8.01% for KNN at 50 households, with all models converging toward 8.3–8.4% at 100 households. The authors conclude that simple, robust models are competitive and often preferable for newly formed or data-scarce communities.","pith_inferences":["The six-month crossover was tested on an autumn quarter only; an obvious extension is to repeat the training-size sweep on winter, spring, and summer test sets to see whether the threshold is seasonal.","Because the financial case study uses a day-ahead real-time price, the price signal itself carries much of the scheduling information; under more prediction-sensitive objectives, such as peak-load minimization or self-sufficiency, the gap between KNN and deep learning could widen.","The transfer-learning recipe could be tested across countries and climates; the 1.97 percentage point gain from German synthetic profiles on UK data suggests larger gains or losses if source and target diverge more.","The same benchmark grid could be run with probabilistic forecasts (quantiles or intervals) since the MILP objective and the 15% grid-purchase constraint may respond differently to error distributions, not just mean error."],"forward_implications":["Energy communities with less than six months of operation should plan on persistence forecasting for model-predictive control; deep learning only pays off after roughly nine months of collected data.","Public synthetic load profiles are a viable pretraining source for data-scarce communities, cutting nMAE by about 2 percentage points when only two months of target data exist.","A simple KNN forecaster can capture nearly all the financial value of deep learning in a battery storage optimization, at a fraction of the complexity.","Transformer forecast accuracy is stable down to roughly 200 parameters, so model complexity can be reduced substantially for edge deployments without sacrificing accuracy.","Forecast error and its financial impact shrink as community aggregation grows; at 100 households all tested forecasters yield 8.28–8.39% savings."],"supporting_citations":[{"why":"Supplies the household smart meter load profiles used to construct the virtual energy communities.","marker":"[29]"},{"why":"Provides the synthetic German load profiles used for pretraining in the transfer-learning experiments.","marker":"[30]"},{"why":"Prior study by the same group establishing transfer learning from synthetic profiles for energy-community load forecasting; the present work extends it.","marker":"[25]"},{"why":"Supplies the LSTM architecture and regression-head design adopted in the compared deep models.","marker":"[13]"},{"why":"Introduces the xLSTM architecture that the paper compares against LSTM and Transformer baselines.","marker":"[21]"},{"why":"Provides the encoder-only Transformer architecture used in the comparison.","marker":"[33]"},{"why":"Defines the persistence and KNN benchmark methods and their setup.","marker":"[37]"},{"why":"Supplies the 12 kWh per-household battery capacity assumption used to size the storage in the case study.","marker":"[40]"},{"why":"Provides the day-ahead real-time price signal used in the case study tariff.","marker":"[39]"}],"fun_headline_variants":["Deep learning lags persistence under 6 months of data","KNN nearly matches best deep model's cost savings","Transfer learning adds 2% accuracy with scarce data","Simple models win when data is scarce for forecasting","Deep nets need 9+ months to beat persistence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The six-month crossover is measured on a single test season, Q4 2013; the paper assumes that this quarter represents the whole year, so if winter or summer load patterns shift the crossover, the threshold would move.","fun_headline_variants_meta":{"raw":{"variants":["Deep learning lags persistence under 6 months of data","KNN nearly matches best deep model's cost savings","Transfer learning adds 2% accuracy with scarce data","Simple models win when data is scarce for forecasting","Deep nets need 9+ months to beat persistence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1869,"prompt_tokens":1090,"completion_tokens":779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":703}},"tokens_in":706,"tokens_out":779,"duration_ms":7469,"temperature":1.0,"reasoning_tokens":703,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:21:03.304611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a multi-year household dataset, repeat the training-size sweep (2, 4, 6, 9 months) with each season as the held-out test period, and check whether persistence still beats the deep learning models for every training length below six months in every season; if it does not, the six-month threshold claim is false.","supporting_citations":[{"cited_title":"URL https://data.london.gov.uk/dataset/ smartmeter-energy-use-data-in-london-households 33","cited_arxiv_id":null,"evidence_quote":"Supplies the household smart meter load profiles used to construct the virtual energy communities."},{"cited_title":"Ceruti, B","cited_arxiv_id":null,"evidence_quote":"Provides the synthetic German load profiles used for pretraining in the transfer-learning experiments."},{"cited_title":"Improve Load Forecasting in Energy Communities through Transfer Learning using Open-Access Synthetic Profiles","cited_arxiv_id":"2407.08434","evidence_quote":"Prior study by the same group establishing transfer learning from synthetic profiles for energy-community load forecasting; the present work extends it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the persistence and KNN benchmark methods and their setup."},{"cited_title":"URL https://www.exaa.at/","cited_arxiv_id":null,"evidence_quote":"Provides the day-ahead real-time price signal used in the case study tariff."}],"review_version":1}