{"id":"5177a8df-f904-4e8e-abb2-73bd50a539b6","arxiv_id":"2411.14421","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On two equal-sized ComStock-derived datasets differing in building-type diversity, architecture and heterogeneity affect building-energy forecast error more than model size, and fine-tuned TimesFM beats all trained-from-scratch baselines.","lead":"This paper measures how much the diversity of building types in a training dataset helps or hurts nine deep-learning forecasters and a foundation model. It finds that model architecture and dataset heterogeneity matter more than parameter count, and that a fine-tuned foundation model outperforms models trained from scratch.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Early-stopping contradiction in §4 vs Appendix D: if the test set was used for stopping, Table 1 and the central comparison are not independent.","rationale":"The reader’s weakest_assumption correctly lists both the dataset-comparability issue and the early-stopping contradiction, but emphasizes dataset comparability. I regard the early-stopping contradiction as more load-bearing because Table 1 is the sole evidence for the paper’s central quantitative claims. If early stopping used the test set, the comparison of models and datasets is not a valid evaluation, and no amount of re-interpretation of heterogeneity or parameter count can rescue the conclusions. If it used the validation set, there is still a confound between architecture and parameter count in Figure 2, and the two datasets differ in composition, not only in heterogeneity; these remain secondary concerns. The concrete test is feasible with the provided artifact and would settle the issue. I therefore keep the reader’s CONDITIONAL verdict: the paper’s conclusions are plausible but hinge on a resolvable inconsistency, and the final verdict should depend on the outcome of checking the early-stopping protocol. No ad hominem is intended; this is a technical inconsistency in the manuscript text.","tokens_in":13218,"tokens_out":3935,"duration_ms":41086,"concrete_test":"Inspect the released training code (APPFL/Building_load_forecasting) to determine whether the early-stopping callback evaluates the validation split or the test split. If the code is unavailable, rerun one representative condition from Table 1, e.g. LSTNet on IL-HET at L=512, T=4, using early stopping on the validation set only; compare the resulting test NMSE against the reported 0.0702. If the value changes materially or the model ranking shifts, the reported test results are not robust to the early-stopping protocol.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper contains two mutually incompatible statements about the early-stopping signal. Section 4 says: “our implementation of early-stopping (i.e. terminating training when continued improvements are no longer observed on the test set) yielded disparate results across IL-HET and IL-HOM.” Appendix D instead says: “we use an early stopping criteria of no improvement in the validation set metrics (also called patience) for 5 epochs,” and Figure 4 is explicitly labeled “Validation Set.” Table 10 shows that many models stop very early (e.g., PatchTST at epoch 1 on IL-HET T=4, LSTNet at epoch 2 on IL-HET T=4, TimesNet at epoch 1 on IL-HET T=96). If Section 4 is literal and the stopping signal was the test NMSE, then the test numbers in Table 1 are not held-out evaluations: they are the result of model selection directly on the test set. That would bias every reported comparison and would jeopardize the central claim that heterogeneity and architecture matter more than parameter count, because the rankings on which that claim rests would not be independent. Since the authors provide code and weights, this is directly checkable. If Appendix D is the correct description, the Table 1 results can stand on this point, but the contradiction must be resolved in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two curated building-energy datasets derived from ComStock, IL-HET and IL-HOM, which are equal in size (592 buildings) and region (Illinois) but differ in building-type diversity: IL-HET contains all 14 available building types, while IL-HOM contains only warehouses. The authors evaluate LSTM, LSTNet, Transformer, Autoformer, Informer, TimesNet, and PatchTST, together with TimesFM in zero-shot and fine-tuned modes, across lookaheads T = 4, 48, 96 with fixed lookback L = 512, and report NMSE and NMAE on held-out test splits. They conclude that dataset heterogeneity and model architecture have a larger impact on post-training forecasting performance than parameter count, and that fine-tuned foundation models are competitive despite higher computational cost. The paper releases the datasets, model weights, and code.","tokens_in":13603,"tokens_out":3805,"duration_ms":36372,"significance":"If its claims hold, the paper is a useful empirical contribution to short-term load forecasting and foundation-model evaluation: it provides two controlled, publicly released ComStock subsets, evaluates a broad architecture zoo under a unified protocol, and makes weights and code available. The finding that fine-tuned TimesFM achieves substantially lower NMSE than all base models (e.g., 0.0078 versus 0.0289 for TimesNet on IL-HET at T=4) is noteworthy and, if anything, understated in the abstract. However, the central claims about heterogeneity and parameter count are weakened by methodological confounds, and one explicit text contradiction about early stopping affects the independence of the reported test results. These issues are addressable but need to be resolved before the conclusions can be accepted at face value.","major_comments":[{"comment":"The paper contains two mutually inconsistent descriptions of the early-stopping signal. Section 4 states that early stopping terminates training \"when continued improvements are no longer observed on the test set,\" whereas Appendix D states that \"we use an early stopping criteria of no improvement in the validation set metrics (also called patience) for 5 epochs.\" If the Section 4 statement is literal, then the NMSE values in Table 1 are not independent held-out evaluations: the test set would have been used for model selection, biasing every ranking and undermining the central claim that heterogeneity and architecture matter more than parameter count. If Appendix D is the correct description, then Table 1 can stand on this point, but the contradiction must be removed in revision. Because the code and weights are public, the authors should verify which procedure was actually implemented and report the outcome.","section":"Appendix A / Section 2"},{"comment":"The heterogeneity manipulation is confounded. IL-HET and IL-HOM differ not only in building-type diversity but also in the marginal distribution of loads: Appendix A reports mean load 21.84 vs. 17.43 kWh and standard deviation 51.36 vs. 24.31 kWh for IL-HET versus IL-HOM. The two datasets also differ in building-type composition by construction, since IL-HOM is exclusively warehouses. Consequently, the observed performance differences cannot be attributed solely to \"heterogeneity\" as a construct; they may reflect scale, variance, or composition differences. The authors should either add intermediate heterogeneity levels that hold the load distribution approximately fixed, or reframe the contribution as a comparison of two naturally occurring dataset compositions rather than a controlled heterogeneity experiment.","section":"Figure 2, Tables 6-7"},{"comment":"The claim that model architecture matters more than parameter count is not supported by the evidence presented. Figure 2 plots NMSE against model size for the base models, but model size is completely entangled with architecture: the smallest models are LSTM and LSTNet, the largest are Transformer and Informer, and there is no within-architecture variation in parameter count. The scatterplot therefore cannot separate the effect of architecture from the effect of size, and no statistical test or controlled scaling experiment is provided. To make the claimed parameter-count conclusion load-bearing, the authors should vary width/depth within at least one architecture family, or explicitly weaken the conclusion to state that the evaluated architectures differ in both structure and size.","section":"Table 1, Table 10"},{"comment":"The reported results are based on single training runs, with no confidence intervals or repeated-seed analysis. This is particularly concerning because Table 10 shows that many models terminate after 1-4 epochs (e.g., PatchTST at epoch 1 on IL-HET at T=4, TimesNet at epoch 1 on IL-HET at T=96, LSTNet at epoch 2 on IL-HET at T=4), and Appendix D documents training instability on IL-HET. With such short and unstable training trajectories, a single run provides weak evidence for the ranking-based conclusions in Table 1 and Figure 2. The authors should report results over multiple seeds with confidence intervals, or at minimum demonstrate that the qualitative conclusions are stable across seeds.","section":"Appendix D"}],"minor_comments":[{"comment":"The Figure 5 caption says \"for T = 512, L= 48,\" which appears to reverse the lookback and lookahead values used in the experiments (L=512, T=48). This should be corrected.","section":"Table 6 caption"},{"comment":"The caption of Table 6 says \"Model sizes, parameter counts and structural hyperparameters for (T=512, L=48),\" but the experiments use L=512 and T=48. The values in the table also suggest the lookback/lookahead labels are swapped, so the caption should be corrected.","section":"Appendix D, paragraph on TimesFM"},{"comment":"The text refers to \"TimeFM\" in the sentence about learning-rate tuning; this should be \"TimesFM\" for consistency.","section":"Appendix A, paragraph on statistics"},{"comment":"The phrase \"the higher heterogeneity of IL-HET can be seen by compaing the means and standard deviations\" contains a typo (\"compaing\" should be \"comparing\").","section":"Section 2"},{"comment":"\"HV AC system\" should be \"HVAC system\" throughout.","section":"Section 4"},{"comment":"In the sentence \"the fined-tuned TimesFM is able to outperform the best base models,\" \"fined-tuned\" should be \"fine-tuned.\"","section":"Appendix E, introductory paragraph"},{"comment":"The text says the predictions are shown for \"the first 500 points of the 5th customer in both the datasets,\" while the caption says \"two selected buildings from IL-HOM and IL-HET.\" It would be clearer to state in the caption that the plots show two example buildings, one from each dataset.","section":"Figure 5 caption"}],"recommendation":"major_revision","confidential_remarks":"The early-stopping contradiction is the most serious correctness risk in the manuscript. If the Section 4 description is the literal implementation, the entire Table 1 comparison would be invalid; if Appendix D is accurate, the issue is a presentation error that can be fixed by rewriting Section 4. The other major concerns—confounded heterogeneity, confounded size-versus-architecture analysis, and absence of confidence intervals—are substantive but fixable within the manuscript's scope, so I do not recommend rejection. I would encourage the editor to require the authors to clarify the early-stopping protocol explicitly and, if needed, verify it from the released code before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a genuinely useful empirical contribution. They built two same-size, same-region ComStock subsets that differ in building-type diversity, and ran nine models including TimesFM in zero-shot and fine-tuned modes. Data and weights are released. The finding that fine-tuned TimesFM beats the best base models by a large margin on IL-HET (NMSE 0.0078 vs 0.0289 at T=4) is credible and probably robust. That alone is worth having on record.\n\nWhat's new: the controlled heterogeneity pair is a nice design. Earlier benchmarks like Buildings-900K don't isolate heterogeneity while holding size and region fixed. The correlation-based feature selection is reasonable. The paper is clearly written and the appendix supports the main tables.\n\nSoft spots, in order of severity.\n\nFirst, the early-stopping contradiction is real and needs a public fix. Section 4 says they stopped when improvements on the test set stopped; Appendix D says validation set with patience 5, and Figure 4 is labeled validation. If Section 4 is literal, every test number in Table 1 is selection on the test set and the main comparison is void. I suspect Appendix D is the correct description, but the paper must say so unambiguously. Because you release code, a referee can check this directly. Table 10 showing many models terminating at epoch 1-4 also suggests training instability, which ties into this.\n\nSecond, no confidence intervals or repeated runs. For a benchmark whose main claim is about which factors matter, single runs are thin. This is fixable and should be done.\n\nThird, the parameter-count conclusion is weak. Figure 2 plots NMSE against model size, but architecture and size are completely confounded—the large models are Transformers and the small ones are LSTM/TimesNet. You can't infer that size doesn't matter from that scatter. The data-composition point stands better than the 'architecture over size' point.\n\nAlso, heterogeneity is entangled with scale and composition: IL-HET has all 14 types, IL-HOM only warehouses, and the load std differs a lot. That's inherent to the design, but it limits causal language. They do say 'impact,' not 'cause,' so fine.\n\nWho it's for: applied load-forecasting researchers and people choosing models for building-level STLF. A serious referee should see it after the early-stopping issue is resolved in the text and ideally with uncertainty quantification added.\n\nRecommendation: engage with it—send it out if the authors fix the contradiction and add runs or at least clearly spec the early stopping. As is, conditional.","headline":"Useful controlled dataset pair and broad benchmark, but the early-stopping contradiction and confounded parameter-count argument need fixing before the headline claims are trustworthy.","tokens_in":13974,"tokens_out":1776,"would_cite":true,"duration_ms":16625,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dataset mix and architecture, not parameter count, decide building-load forecast accuracy.","keywords":["short-term load forecasting","building energy consumption","dataset heterogeneity","foundation models","time series forecasting","TimesFM","ComStock","model architecture"],"falsifier":"A concrete check is to rerun the model suite on ComStock subsets with identical building-type composition but synthetically varied load diversity; if the IL-HET versus IL-HOM gap vanishes, the heterogeneity effect is confounded with scale and composition. Separately, recomputing test scores under validation-based early stopping, as Appendix D describes, would confirm that the reported test numbers are independent of training choices.","tokens_in":13029,"feed_emoji":"⚡","tokens_out":6411,"duration_ms":60372,"temperature":0.7,"pith_summary":"The paper tests how dataset heterogeneity and model architecture affect short-term building energy forecasting using two ComStock subsets that are identical in size and region but differ in building-type diversity: IL-HET includes all 14 building types, while IL-HOM contains only warehouses. Across lookaheads of 4, 48, and 96 fifteen-minute steps, it compares six scratch-trained base architectures plus the TimesFM foundation model in zero-shot and fine-tuned modes. The central finding is that heterogeneity and architecture matter more than parameter count: fine-tuned TimesFM achieves the lowest normalized error on every setup, while small models such as TimesNet and PatchTST outperform larger transformer-based models. This matters because it points to where computational effort should go when building load forecasters for smart grids.","feed_headline":"Fine-tuned TimesFM tops building-load forecasts on every test","feed_subtitle":"On matched ComStock subsets, the fine-tuned foundation model beat all scratch-trained baselines, while zero-shot lagged.","key_machinery":"The controlled comparison is built on two same-size ComStock subsets, IL-HET and IL-HOM, that fix region, building count, lookback, and training protocol while varying building-type diversity. The evaluation machinery is normalized NMSE and NMAE with z-normalized inputs and a fixed lookback of 512 steps dictated by TimesFM's pretraining limit; the model zoo spans RNN, transformer, decomposition, 2D-backbone, and patch architectures, letting architecture and parameter count vary across models. This design is what lets the paper attribute performance differences to heterogeneity and architecture rather than to data volume.","core_discovery":"The paper claims that, holding dataset size fixed, the diversity of building types in the training pool and the choice of architecture influence post-training forecast accuracy more than the model's parameter count, and that fine-tuning a pretrained foundation model is the most reliable route to accuracy. On the IL-HET subset the best scratch-trained model at lookahead T=4 is TimesNet with NMSE 0.0289, whereas fine-tuned TimesFM reaches 0.0078, and the margin persists at longer lookaheads and on the homogeneous IL-HOM subset. Zero-shot TimesFM, by contrast, underperforms most base models, so the benefit depends on full-parameter fine-tuning. The paper also observes that patch-based models improve on the more heterogeneous dataset, and that training on IL-HET triggers earlier early stopping, indicating greater training instability.","pith_inferences":["If heterogeneity is the true driver, intermediate levels of building-type mixing should produce a gradual performance curve rather than a threshold effect; this is testable by constructing additional ComStock subsets with controlled diversity.","The fact that fine-tuned TimesFM also wins on the homogeneous IL-HOM set suggests pretrained priors help even where local statistics are simple, which could justify one globally fine-tuned foundation model for an entire building stock.","Parameter-efficient fine-tuning of TimesFM, which the paper lists as future work, would clarify how much of the full-parameter advantage is retained at a fraction of the compute cost.","Because IL-HET and IL-HOM differ in load scale and composition as well as diversity, the heterogeneity attribution should be confirmed by matching the two sets on load mean and variance before concluding that diversity itself drives the effect."],"forward_implications":["For short-term load forecasting, fine-tuning a pretrained time-series foundation model is a stronger accuracy lever than training a scratch model on the same data, despite higher compute cost.","Small architectures with temporal or periodic inductive biases, such as TimesNet and PatchTST, can beat larger transformer stacks, so parameter count is not the main driver of forecast quality.","Heterogeneous training pools are not uniformly harmful: patch-based models perform better on IL-HET than on IL-HOM, suggesting that mixture diversity can act as a regularizer.","Zero-shot deployment of a foundation model is not sufficient for building-load forecasting; full fine-tuning is needed to unlock its advantage.","Training on heterogeneous data is less stable, so early-stopping behavior should be tuned and reported per dataset rather than assumed to transfer."],"supporting_citations":[{"why":"Supplies the ComStock synthetic energy dataset and the building-type features used to construct IL-HET and IL-HOM.","marker":"(Parker et al., 2023)"},{"why":"Provides the TimesFM pretrained foundation model used in both zero-shot and fine-tuned experiments.","marker":"(Das et al., 2024)"},{"why":"Defines the TimesNet architecture, the best-performing scratch-trained base model in the comparison.","marker":"(Wu et al., 2023)"},{"why":"Defines the PatchTST architecture, the patch-based base model whose heterogeneity behavior is highlighted.","marker":"(Nie et al., 2023)"},{"why":"Supplies the LSTM baseline representing recurrent architectures in the model zoo.","marker":"(Hochreiter and Schmidhuber, 1997)"},{"why":"Supplies the LSTNet baseline combining RNN and convolutional layers.","marker":"(Lai et al., 2018)"},{"why":"Supplies the vanilla Transformer baseline whose attention-based temporal modeling is compared against others.","marker":"(Vaswani et al., 2017)"},{"why":"Supplies the Autoformer decomposition-based baseline.","marker":"(Wu et al., 2021)"},{"why":"Supplies the Informer transformer baseline with ProbSparse attention.","marker":"(Zhou et al., 2021)"}],"fun_headline_variants":["Fine-tuned foundation models win building energy forecasts","Dataset diversity beats model size in building load forecasting","TimesFM fine-tuned beats scratch models for commercial buildings","Heterogeneity, not parameter count, drives forecast accuracy","Zero-shot lags; fine-tuning lifts foundation models to top"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that IL-HET and IL-HOM are matched except for building-type diversity, and the paper's early-stopping description is inconsistent (Section 4 says test set, Appendix D says validation), which matters because test numbers must be independent of training decisions.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned foundation models win building energy forecasts","Dataset diversity beats model size in building load forecasting","TimesFM fine-tuned beats scratch models for commercial buildings","Heterogeneity, not parameter count, drives forecast accuracy","Zero-shot lags; fine-tuning lifts foundation models to top"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1205,"prompt_tokens":867,"completion_tokens":338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":483,"tokens_out":338,"duration_ms":3518,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:10:54.070030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to rerun the model suite on ComStock subsets with identical building-type composition but synthetically varied load diversity; if the IL-HET versus IL-HOM gap vanishes, the heterogeneity effect is confounded with scale and composition. Separately, recomputing test scores under validation-based early stopping, as Appendix D describes, would confirm that the reported test numbers are independent of training choices.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TimesFM pretrained foundation model used in both zero-shot and fine-tuned experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LSTNet baseline combining RNN and convolutional layers."},{"cited_title":"8 Contrary to conventional implementations, we use positional embeddings for temporal inputs (i.e","cited_arxiv_id":null,"evidence_quote":"Supplies the Informer transformer baseline with ProbSparse attention."}],"review_version":1}