{"id":"f67383ea-fbab-4e5b-9eec-ac8ebf988eff","arxiv_id":"2501.07423","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"No single ML model wins year-round for student-residence energy forecasting; the proposed hypernetwork-LSTM and MiniAutoEncXGBoost do well on specific seasons and buildings.","lead":"This paper benchmarks 13 machine learning models for forecasting day-ahead electricity use in two university student residences and finds that no single model wins in every season. It introduces two hybrid models, a hypernetwork LSTM and a MiniRocket-autoencoder-XGBoost ensemble, that perform best in some building and seasonal cases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation's central claim rests on an unstated split rule: if the 70/10/20 split is not chronological, sliding-window samples leak across train/test, invalidating the reported forecast skill and seasonal rankings.","rationale":"I read the paper in good faith and identified the central claim as the empirical finding that no single model outperforms all others across seasons, with the proposed HyperNetLSTM and MiniAutoEncXGBoost being particularly adaptable. The most load-bearing concern is the ambiguous data split. In time series forecasting, a random split of sliding-window samples creates near-duplicate windows across train and test, which is a known form of data leakage. The paper never states that the split is chronological, and the wording 'partitioned' is consistent with either random or temporal assignment. If the split were random, the reported errors would be overly optimistic and the seasonal rankings would not represent true forecasting skill. This directly undermines the practical guidance derived from the results. I agree with the reader's weakest assumption. The missing spring analysis for Residence 1 and the absence of error bars are additional weaknesses but are secondary because they would remain concerns even under a correct temporal split. The paper does provide detailed hyperparameter tables and architecture descriptions, which is credit, but no released code or data to verify the split. The proposed concrete test—recomputing with a chronological split—would settle the concern. Since the reader already conditionalized on this issue, I recommend no change to the verdict. The paper should be accepted only if the authors clarify or demonstrate the split is chronological and provide error bars or multiple seeds.","tokens_in":10913,"tokens_out":4182,"duration_ms":41542,"concrete_test":"Re-run all experiments with a strictly chronological split: first 70% of timestamps for training, next 10% for validation, last 20% for testing, keeping the same hyperparameters and architecture. Compare per-model and per-season SMAPE/MAE/RMSE against Tables II-X. If any model ranking changes by more than one position or any SMAPE shifts by more than 0.5, the ambiguity in Section IV-A4 is load-bearing. Additionally, check the timestamps of the test windows: if any test window's date range overlaps a training window's date range, the original split was random and the reported forecasting skill is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A4 states only that the dataset 'was partitioned into training, validation, and test sets' with 70/10/20 percentages, without specifying whether the split preserves chronological order. This is load-bearing because the sliding window with stride 1 (Section IV-A3b) creates consecutive samples that differ by one hour; a random split would place windows from the same day or overlapping 24-hour spans in both training and test sets. The model would then be evaluated partly on interpolation of data it has effectively seen, producing optimistically low SMAPE/MAE/RMSE values and unstable seasonal rankings. The paper's central claim that 'no single model consistently outperforms others across all seasons' and the proposed models' 'strong adaptability' would lose its evidential basis if the split is random. Even if the split is chronological, the seasonal analysis for Residence 1 omits spring (Tables III–V only cover fall, winter, summer), and no error bars or multiple seeds are reported, so the differences in rankings could be within noise. The temporal split ambiguity is the most direct threat to the main empirical conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares 13 machine-learning models, including two proposed architectures (HyperNetLSTM and MiniAutoEncXGBoost), for day-ahead (24-hour) energy-consumption forecasting on two student-residence datasets at Western University spanning 2019–2023. Using SMAPE, MAE, and RMSE, it reports overall and season-wise rankings; the central claim is that no single model dominates all seasons, so season-specific model selection is needed. The two proposed models are reported to perform best overall on Residence 1 and Residence 2, respectively.","tokens_in":11175,"tokens_out":4600,"duration_ms":42165,"significance":"If the central empirical claim survives closer scrutiny, this is a useful applied benchmark: the comparison covers a broad model zoo, uses real building-level data, and documents seasonal heterogeneity in rankings. The paper's strength is its transparent tabular reporting of errors for every model in every season, which makes the 'no single best model' phenomenon easy to see. However, the claim currently rests on single-run point estimates from a split whose temporal ordering is never stated, and one seasonal block is missing; these issues prevent the results from being fully load-bearing as published.","major_comments":[{"comment":"The data-splitting description in Section IV-A4 states only that the dataset was partitioned 70/10/20 into training, validation, and test sets, without specifying whether the split is chronological. This matters because the sliding-window construction in Section IV-A3b uses a stride of one, so consecutive windows differ by a single hour; a random split would place overlapping or near-identical windows in both training and test sets, turning the evaluation into interpolation and inflating forecast skill. Please state explicitly that the split preserves temporal order (and, if it does not, re-run the evaluation with a chronological split and report the season/month boundaries of the test period).","section":"IV-A4 and IV-A3b"},{"comment":"The Residence 1 seasonal analysis in Section V-L reports Fall, Winter, and Summer (Tables III–V) but has no Spring table, even though the abstract and Section VI make claims about performance 'across all seasons' and the Residence 2 analysis in Section V-N includes a Spring table (Table IX). The absence of Residence 1 Spring results leaves the 'no single model across all seasons' claim incomplete for one of the two buildings; please add the missing table or explain why it is omitted.","section":"V-L, Tables III–V"},{"comment":"All error metrics are single-run point estimates with no standard deviations, confidence intervals, or significance tests. Several adjacent rankings are separated by less than 0.2 SMAPE (e.g., Table II: HyperNetLSTM 8.87 vs LSTM 9.12; Table VI: MiniAutoEncXGBoost 7.37 vs AttentionLSTM 7.51), so the season-by-season 'winner' assignments and the claim of 'strong adaptability' may be within seed/initialization noise. Run multiple seeds and report mean ± std or pairwise tests, or soften the claims to rankings-of-point-estimates.","section":"V, Tables II–X"},{"comment":"The proposed HyperNetLSTM and MiniAutoEncXGBoost are tuned via grid search on the same data that later produces the headline test-set comparisons, with no nested validation or separate model-selection phase. The reported superiority of the proposed models is therefore subject to selection-on-the-test-set optimism; please clarify the selection protocol or evaluate the final selected configurations on a truly held-out period.","section":"V-H and V-K"}],"minor_comments":[{"comment":"Section II-A contains a duplicated phrase: 'Additional feature extraction techniques were integrated additional feature extraction techniques were integrated.'; please remove the repetition.","section":"II-A"},{"comment":"Section III refers to 'Extended Short-Term Memory Networks (LSTMs)'; the standard expansion is 'Long Short-Term Memory.'","section":"III"},{"comment":"Reference [1] still contains the placeholder 'accessed: [Insert Access Date Here]'; fill in the access date or remove the placeholder.","section":"References"},{"comment":"The caption of Figure 8 says 'Hypernetwork-based MLP and primary network-based LSTM,' but the text describes the primary network as an LSTM; align the caption with the architecture description.","section":"Figure 8 caption"},{"comment":"There is no data or code availability statement; providing one, even with anonymized or aggregated data, would allow readers to reproduce the tables.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is an applied empirical comparison rather than a methodological advance; its incremental novelty lies in the two proposed architectures. The main technical obstacles are the unstated temporal split, missing Residence-1 spring results, and lack of uncertainty quantification. If these are fixed, I would support publication in a venue that accepts strong empirical benchmarks; the paper would not be a good fit for a methods-focused journal without a deeper analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you do building load forecasting: two new residence datasets, 13 models, seasonal breakdowns, and two new hybrids. The headline finding—no single model wins all seasons—is plausible but rests on a split rule the paper never states.\n\nWhat's actually new: the datasets are genuinely new, and the two hybrids (hypernetwork-LSTM, MiniRocket-autoencoder-XGBoost) are new combinations of known pieces. The seasonal tables are helpful; they show, for example, that the Transformer wins in fall for Residence 1 but is mediocre elsewhere. The two-building comparison is a plus—many such studies use one building.\n\nThe main soft spot is real: Section IV-A4 says the data were split 70/10/20 but does not say chronological. With a stride-1 sliding window, a random split puts overlapping 24-hour windows in both train and test, turning the reported errors into interpolation scores. That would deflate the reported SMAPE/MAE/RMSE and could reshuffle the seasonal rankings. The authors must clarify or re-run with a temporal split. This is not a manufactured issue; it is the first thing any referee should check.\n\nMinor but worth noting: no error bars or significance tests, so the differences between models in Tables III–X could be within noise. Residence 1 is missing its spring table—the text says \"Tables III, IV, and V\" but only fall, winter, and summer appear. No code or data are released, which makes it hard to verify the split or reproduce the numbers.\n\nThe qualitative conclusion—season-specific model selection matters—is not new; it is common in forecasting. But the evidence here is more detailed than most, and the two proposed models do appear to be competitive in several seasons. If the temporal split is confirmed, this is a solid applied benchmark for two buildings.\n\nSend to peer review, but require the authors to state the split rule, add uncertainty estimates, and release artifacts. The paper is honest applied work, not a breakthrough, and it deserves a serious referee to insist on those fixes.","headline":"A useful two-building seasonal benchmark, but the central claim hinges on a split rule the paper never states and the rankings may be noise without error bars.","tokens_in":11681,"tokens_out":1806,"would_cite":false,"duration_ms":18576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 13-model study of student residences finds that the best energy forecasting model changes with the season, and two adaptive designs lead overall.","keywords":["energy forecasting","seasonal variations","LSTM","hypernetwork","XGBoost","MiniRocket","student residences","model selection"],"falsifier":"Re-run the same 13-model comparison with a strictly chronological train/validation/test split—for example, training on data up to mid-2022 and testing on the final year—and compare the per-season SMAPE rankings; if a single model then wins every season or the proposed models' edge over LSTM disappears, the central claim fails.","tokens_in":10761,"feed_emoji":"⚡","tokens_out":4625,"duration_ms":40930,"temperature":0.7,"pith_summary":"This paper tests 13 machine learning models for day-ahead energy forecasting on two student residence buildings, using hourly data from 2019 to 2023. Its central finding is that no single model is consistently the most accurate across fall, winter, spring, and summer; the ranking of models changes with the season. The paper proposes two models—a Hypernetwork-based LSTM and a MiniRocket-plus-autoencoder-plus-XGBoost ensemble—and reports that both adapt particularly well to the abrupt consumption changes of summer. The takeaway is that energy forecasters should evaluate models per season or design adaptive architectures rather than commit to one universal model.","feed_headline":"13-model study: best energy forecaster changes by season","feed_subtitle":"Per-season rankings disagree, so forecasters should match model to season, not commit to one.","key_machinery":"The central objects are the two proposed architectures. HyperNetLSTM uses a hypernetwork—a small fully-connected network that takes the input window and dynamically produces the LSTM's weights, letting the recurrent model change its behavior as the input regime changes. MiniAutoEncXGBoost concatenates features extracted by MiniRocket (a fast deterministic convolution-based transform) and by a convolutional autoencoder, then feeds those features to an XGBoost regressor. The seasonal evaluation procedure—splitting the test predictions by calendar season and re-ranking the models—is what carries the main argument that model performance is not season-uniform.","core_discovery":"The paper's core claim is that seasonality changes which model should be trusted. On Residence 1, HyperNetLSTM has the lowest overall SMAPE (8.87), but in fall the Transformer wins (8.59), in winter HyperNetLSTM wins (6.11), and in summer HyperNetLSTM wins again (8.13). On Residence 2, MiniAutoEncXGBoost leads overall (7.37), yet LSTM wins in fall, ARFFNN in winter, AttentionLSTM in spring, and MiniAutoEncXGBoost in summer. The 'best model' is therefore season-dependent, and the proposed models' strength lies in tracking the irregular summer consumption drops tied to student vacations. The paper concludes that season-aware model selection or purpose-built adaptive designs are needed.","pith_inferences":["Because the data come from only two university residences, the season-specific rankings may not generalize to other building types; a testable extension is to run the same 13-model protocol on public load datasets with known calendar events.","The seasonal winners appear to line up with occupancy changes (e.g., summer vacations), so adding an academic-calendar feature directly might reduce the need for different architectures per season.","The hypernetwork's dynamic weight generation can be viewed as a soft form of season-conditioning; comparing it to feeding a season indicator into the LSTM would isolate what the hypernetwork actually adds.","The paper's central claim currently rests on point estimates; statistical significance tests across multiple train/test splits would show whether the seasonal ranking differences are real or within noise."],"forward_implications":["Practitioners should report seasonal error breakdowns, not only overall errors, before selecting a forecasting model.","A season-aware ensemble that switches among the top models per season would likely beat any single fixed model on these datasets.","The HyperNetLSTM design suggests a general mechanism for adapting recurrent networks to regime shifts beyond energy.","The MiniRocket-plus-autoencoder feature combination could transfer to other load-forecasting datasets with abrupt vacation or holiday effects.","The paper's conclusion points toward human-AI hybrid forecasters that encode occupancy or event information, since human behavior is the main driver of seasonal instability."],"supporting_citations":[{"why":"Supplies the N-BEATS state-of-the-art baseline the paper compares against.","marker":"[5]"},{"why":"Supplies the AR-Net auto-regressive feedforward baseline.","marker":"[6]"},{"why":"Defines the MiniRocket feature extractor used in the ensemble models.","marker":"[8]"},{"why":"Provides the Transformer architecture and self-attention mechanism adapted into the AttentionLSTM and Transformer baselines.","marker":"[11]"},{"why":"Motivates the convolutional autoencoder feature extractor used in MiniAutoEncXGBoost.","marker":"[16]"},{"why":"Provides the temporal convolutional baseline applied to energy-related forecasting.","marker":"[7]"},{"why":"Motivates using XGBoost for load forecasting with irregular patterns such as holidays.","marker":"[13]"}],"fun_headline_variants":["Best energy forecaster flips by season in dorms","13 models: none wins energy forecasting every season","Season decides which AI model forecasts energy best","No universal winner for student energy prediction","Season-aware model selection improves energy forecasts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the 70/10/20 split into training, validation, and test sets being chronological rather than random, because the paper does not state that the test period follows the training period in time.","fun_headline_variants_meta":{"raw":{"variants":["Best energy forecaster flips by season in dorms","13 models: none wins energy forecasting every season","Season decides which AI model forecasts energy best","No universal winner for student energy prediction","Season-aware model selection improves energy forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2438,"prompt_tokens":860,"completion_tokens":1578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":1510}},"tokens_in":476,"tokens_out":1578,"duration_ms":13586,"temperature":1.0,"reasoning_tokens":1510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:42:07.516023+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 13-model comparison with a strictly chronological train/validation/test split—for example, training on data up to mid-2022 and testing on the final year—and compare the per-season SMAPE rankings; if a single model then wins every season or the proposed models' edge over LSTM disappears, the central claim fails.","supporting_citations":[{"cited_title":"Minirocket: A very fast (almost) deterministic transform for time series classification,","cited_arxiv_id":null,"evidence_quote":"Defines the MiniRocket feature extractor used in the ensemble models."},{"cited_title":"Residential load profile clustering via deep convolutional autoencoder,","cited_arxiv_id":null,"evidence_quote":"Motivates the convolutional autoencoder feature extractor used in MiniAutoEncXGBoost."},{"cited_title":"Temporal convolutional networks applied to energy-related time series forecasting,","cited_arxiv_id":null,"evidence_quote":"Provides the temporal convolutional baseline applied to energy-related forecasting."},{"cited_title":"A hybrid prediction model based on pattern sequence-based matching method and extreme gradient boosting for holiday load forecasting,","cited_arxiv_id":null,"evidence_quote":"Motivates using XGBoost for load forecasting with irregular patterns such as holidays."}],"review_version":1}