Pith. sign in

REVIEW 4 major objections 7 minor 12 references

From RNNs to Foundation Models: An Empirical Study on Commercial Building Energy Consumption

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dataset mix and architecture, not parameter count, decide building-load forecast accuracy.

desk verdict Useful controlled dataset pair and broad benchmark, but the early-stopping contradiction and confounded parameter-count argument need fixing before the headline claims are trustworthy. read the letter →

arxiv 2411.14421 v2 pith:TZZ7TPY6 submitted 2024-11-21 cs.LG

Shourya Bose , Yijiang Li , Amy Van Sant , Yu Zhang , Kibaek Kim This is my paper · ORCID
classification cs.LG
keywords short-termloadforecastingbuildingenergyconsumptiondatasetheterogeneityfoundationmodelstimeseriesTimesFMComStockmodelarchitecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests how dataset heterogeneity and model architecture affect short-term building energy forecasting using two ComStock subsets that are identical in size and region but differ in building-type diversity: IL-HET includes all 14 building types, while IL-HOM contains only warehouses. Across lookaheads of 4, 48, and 96 fifteen-minute steps, it compares six scratch-trained base architectures plus the TimesFM foundation model in zero-shot and fine-tuned modes. The central finding is that heterogeneity and architecture matter more than parameter count: fine-tuned TimesFM achieves the lowest normalized error on every setup, while small models such as TimesNet and PatchTST outperform larger transformer-based models. This matters because it points to where computational effort should go when building load forecasters for smart grids.

What carries the argument

The controlled comparison is built on two same-size ComStock subsets, IL-HET and IL-HOM, that fix region, building count, lookback, and training protocol while varying building-type diversity. The evaluation machinery is normalized NMSE and NMAE with z-normalized inputs and a fixed lookback of 512 steps dictated by TimesFM's pretraining limit; the model zoo spans RNN, transformer, decomposition, 2D-backbone, and patch architectures, letting architecture and parameter count vary across models. This design is what lets the paper attribute performance differences to heterogeneity and architecture rather than to data volume.

What would settle it

A concrete check is to rerun the model suite on ComStock subsets with identical building-type composition but synthetically varied load diversity; if the IL-HET versus IL-HOM gap vanishes, the heterogeneity effect is confounded with scale and composition. Separately, recomputing test scores under validation-based early stopping, as Appendix D describes, would confirm that the reported test numbers are independent of training choices.

Watch

Extended reading notes

Core claim

The paper claims that, holding dataset size fixed, the diversity of building types in the training pool and the choice of architecture influence post-training forecast accuracy more than the model's parameter count, and that fine-tuning a pretrained foundation model is the most reliable route to accuracy. On the IL-HET subset the best scratch-trained model at lookahead T=4 is TimesNet with NMSE 0.0289, whereas fine-tuned TimesFM reaches 0.0078, and the margin persists at longer lookaheads and on the homogeneous IL-HOM subset. Zero-shot TimesFM, by contrast, underperforms most base models, so the benefit depends on full-parameter fine-tuning. The paper also observes that patch-based models improve on the more heterogeneous dataset, and that training on IL-HET triggers earlier early stopping, indicating greater training instability.

Load-bearing premise

The load-bearing premise is that IL-HET and IL-HOM are matched except for building-type diversity, and the paper's early-stopping description is inconsistent (Section 4 says test set, Appendix D says validation), which matters because test numbers must be independent of training decisions.

Editorial extensions

If this is right

  • For short-term load forecasting, fine-tuning a pretrained time-series foundation model is a stronger accuracy lever than training a scratch model on the same data, despite higher compute cost.
  • Small architectures with temporal or periodic inductive biases, such as TimesNet and PatchTST, can beat larger transformer stacks, so parameter count is not the main driver of forecast quality.
  • Heterogeneous training pools are not uniformly harmful: patch-based models perform better on IL-HET than on IL-HOM, suggesting that mixture diversity can act as a regularizer.
  • Zero-shot deployment of a foundation model is not sufficient for building-load forecasting; full fine-tuning is needed to unlock its advantage.
  • Training on heterogeneous data is less stable, so early-stopping behavior should be tuned and reported per dataset rather than assumed to transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If heterogeneity is the true driver, intermediate levels of building-type mixing should produce a gradual performance curve rather than a threshold effect; this is testable by constructing additional ComStock subsets with controlled diversity.
  • The fact that fine-tuned TimesFM also wins on the homogeneous IL-HOM set suggests pretrained priors help even where local statistics are simple, which could justify one globally fine-tuned foundation model for an entire building stock.
  • Parameter-efficient fine-tuning of TimesFM, which the paper lists as future work, would clarify how much of the full-parameter advantage is retained at a fraction of the compute cost.
  • Because IL-HET and IL-HOM differ in load scale and composition as well as diversity, the heterogeneity attribution should be confirmed by matching the two sets on load mean and variance before concluding that diversity itself drives the effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces two curated building-energy datasets derived from ComStock, IL-HET and IL-HOM, which are equal in size (592 buildings) and region (Illinois) but differ in building-type diversity: IL-HET contains all 14 available building types, while IL-HOM contains only warehouses. The authors evaluate LSTM, LSTNet, Transformer, Autoformer, Informer, TimesNet, and PatchTST, together with TimesFM in zero-shot and fine-tuned modes, across lookaheads T = 4, 48, 96 with fixed lookback L = 512, and report NMSE and NMAE on held-out test splits. They conclude that dataset heterogeneity and model architecture have a larger impact on post-training forecasting performance than parameter count, and that fine-tuned foundation models are competitive despite higher computational cost. The paper releases the datasets, model weights, and code.

Significance. If its claims hold, the paper is a useful empirical contribution to short-term load forecasting and foundation-model evaluation: it provides two controlled, publicly released ComStock subsets, evaluates a broad architecture zoo under a unified protocol, and makes weights and code available. The finding that fine-tuned TimesFM achieves substantially lower NMSE than all base models (e.g., 0.0078 versus 0.0289 for TimesNet on IL-HET at T=4) is noteworthy and, if anything, understated in the abstract. However, the central claims about heterogeneity and parameter count are weakened by methodological confounds, and one explicit text contradiction about early stopping affects the independence of the reported test results. These issues are addressable but need to be resolved before the conclusions can be accepted at face value.

major comments (4)
  1. [Appendix A / Section 2] The paper contains two mutually inconsistent descriptions of the early-stopping signal. Section 4 states that early stopping terminates training "when continued improvements are no longer observed on the test set," whereas Appendix D states that "we use an early stopping criteria of no improvement in the validation set metrics (also called patience) for 5 epochs." If the Section 4 statement is literal, then the NMSE values in Table 1 are not independent held-out evaluations: the test set would have been used for model selection, biasing every ranking and undermining the central claim that heterogeneity and architecture matter more than parameter count. If Appendix D is the correct description, then Table 1 can stand on this point, but the contradiction must be removed in revision. Because the code and weights are public, the authors should verify which procedure was actually implemented and report the outcome.
  2. [Figure 2, Tables 6-7] The heterogeneity manipulation is confounded. IL-HET and IL-HOM differ not only in building-type diversity but also in the marginal distribution of loads: Appendix A reports mean load 21.84 vs. 17.43 kWh and standard deviation 51.36 vs. 24.31 kWh for IL-HET versus IL-HOM. The two datasets also differ in building-type composition by construction, since IL-HOM is exclusively warehouses. Consequently, the observed performance differences cannot be attributed solely to "heterogeneity" as a construct; they may reflect scale, variance, or composition differences. The authors should either add intermediate heterogeneity levels that hold the load distribution approximately fixed, or reframe the contribution as a comparison of two naturally occurring dataset compositions rather than a controlled heterogeneity experiment.
  3. [Table 1, Table 10] The claim that model architecture matters more than parameter count is not supported by the evidence presented. Figure 2 plots NMSE against model size for the base models, but model size is completely entangled with architecture: the smallest models are LSTM and LSTNet, the largest are Transformer and Informer, and there is no within-architecture variation in parameter count. The scatterplot therefore cannot separate the effect of architecture from the effect of size, and no statistical test or controlled scaling experiment is provided. To make the claimed parameter-count conclusion load-bearing, the authors should vary width/depth within at least one architecture family, or explicitly weaken the conclusion to state that the evaluated architectures differ in both structure and size.
  4. [Appendix D] The reported results are based on single training runs, with no confidence intervals or repeated-seed analysis. This is particularly concerning because Table 10 shows that many models terminate after 1-4 epochs (e.g., PatchTST at epoch 1 on IL-HET at T=4, TimesNet at epoch 1 on IL-HET at T=96, LSTNet at epoch 2 on IL-HET at T=4), and Appendix D documents training instability on IL-HET. With such short and unstable training trajectories, a single run provides weak evidence for the ranking-based conclusions in Table 1 and Figure 2. The authors should report results over multiple seeds with confidence intervals, or at minimum demonstrate that the qualitative conclusions are stable across seeds.
minor comments (7)
  1. [Table 6 caption] The Figure 5 caption says "for T = 512, L= 48," which appears to reverse the lookback and lookahead values used in the experiments (L=512, T=48). This should be corrected.
  2. [Appendix D, paragraph on TimesFM] The caption of Table 6 says "Model sizes, parameter counts and structural hyperparameters for (T=512, L=48)," but the experiments use L=512 and T=48. The values in the table also suggest the lookback/lookahead labels are swapped, so the caption should be corrected.
  3. [Appendix A, paragraph on statistics] The text refers to "TimeFM" in the sentence about learning-rate tuning; this should be "TimesFM" for consistency.
  4. [Section 2] The phrase "the higher heterogeneity of IL-HET can be seen by compaing the means and standard deviations" contains a typo ("compaing" should be "comparing").
  5. [Section 4] "HV AC system" should be "HVAC system" throughout.
  6. [Appendix E, introductory paragraph] In the sentence "the fined-tuned TimesFM is able to outperform the best base models," "fined-tuned" should be "fine-tuned."
  7. [Figure 5 caption] The text says the predictions are shown for "the first 500 points of the 5th customer in both the datasets," while the caption says "two selected buildings from IL-HOM and IL-HET." It would be clearer to state in the caption that the plots show two example buildings, one from each dataset.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: empirical benchmark with external pretrained artifacts; one unresolved early-stopping contradiction is a validity risk, not demonstrated circularity.

full rationale

The paper's central claims are empirical benchmark results, not outcomes of a derivation chain. TimesFM is an external pretrained model (Das et al., 2024) with off-the-shelf zero-shot weights; base models are trained from scratch with standard train/validation/test splits (Section 3, 0.8:0.1:0.1); learning rates are selected on the validation set (Appendix D, Tables 8-12); and the test set is temporally held out. No equation reduces a reported NMSE or NMAE to its own input, and no load-bearing claim is justified by a self-citation. I specifically examined the protocol contradiction flagged in the manuscript: Section 4 states 'our implementation of early-stopping (i.e. terminating training when continued improvements are no longer observed on the test set)', while Appendix D states 'we use an early stopping criteria of no improvement in the validation set metrics (also calledpatience) for 5 epochs'. If the Section 4 statement is literal, the base-model test NMSEs in Table 1 are not independent of the stopping criterion, which would bias the architecture-versus-parameter-count comparison; this is a correctness and reproducibility risk. However, the paper itself describes validation-based early stopping in Appendix D, reports termination epochs (Table 10), and releases code and weights, so the contradiction is resolvable by inspection and does not demonstrate that any result is equivalent to its inputs by construction. Hence no circular step is established; score 1 reflects the minor unresolved evaluation-protocol inconsistency.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central heterogeneity claim rests on the approximate comparability of the two curated datasets, which is only partially controlled. The model-comparison claim rests on training choices (learning rates, epochs, early stopping) that are not held constant across models, and the parameter-count conclusion is inferred from a confounded design rather than a controlled experiment.

free parameters (4)
  • Learning rate per model-dataset-lookahead = Selected from {1e-3, 5e-4, 1e-4, 5e-5, 1e-5, 5e-6, 1e-6, 5e-7} on validation set; e.g., TimesFM IL-HET (512,4) lr=1e-6
    Chosen by validation performance, not derived; directly influences all reported test metrics.
  • Training epochs and early stopping patience = 20 epochs base models, 10 epochs TimesFM; patience 5 epochs (base) or 5 validation checks (TimesFM)
    Fixed training budgets differ across model classes, and early stopping terminates several base models at epoch 1-4, making the comparison unequal.
  • Lookback L and lookahead T = L=512; T=4, 48, 96
    L is capped by TimesFM's 512 pretraining limit; T values are standard forecasting horizons, but all are choices rather than derived quantities.
  • Normalization statistics = Per-feature z-normalization using train set mean and standard deviation
    The choice of train-set statistics for normalization is standard but affects metric scaling across datasets.
assumptions (3)
  • domain assumption ComStock synthetic energy data is an adequate proxy for real commercial building consumption.
    The study's conclusions about model ranking and heterogeneity are only as valid as the fidelity of the synthetic data; invoked in Section 2.
  • domain assumption IL-HET and IL-HOM differ only in building-type diversity, so performance differences are attributable to heterogeneity.
    The two datasets also differ in load distribution (std 51.4 vs 24.3 kWh) and building composition, conflating heterogeneity with scale and other factors; this assumption is central to the paper's main claim.
  • domain assumption TimesFM's pretraining corpus does not include the evaluation data, so fine-tuning gains are genuinely transferable.
    If ComStock-like synthetic data appeared in pretraining, the comparison would be partially circular; no leakage check is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From RNNs to Foundation Models: An Empirical Study on Commercial Building Energy Consumption." pith.science (2026). https://pith.science/paper/TZZ7TPY6

@misc{pith2026241114421,
  author       = {Pith},
  title        = {Pith review of: From RNNs to Foundation Models: An Empirical Study on Commercial Building Energy Consumption},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TZZ7TPY6}},
  note         = {Machine review of arXiv:2411.14421}
}
read the original abstract

Accurate short-term energy consumption forecasting for commercial buildings is crucial for smart grid operations. While smart meters and deep learning models enable forecasting using past data from multiple buildings, data heterogeneity from diverse buildings can reduce model performance. The impact of increasing dataset heterogeneity in time series forecasting, while keeping size and model constant, is understudied. We tackle this issue using the ComStock dataset, which provides synthetic energy consumption data for U.S. commercial buildings. Two curated subsets, identical in size and region but differing in building type diversity, are used to assess the performance of various time series forecasting models, including fine-tuned open-source foundation models (FMs). The results show that dataset heterogeneity and model architecture have a greater impact on post-training forecasting performance than the parameter count. Moreover, despite the higher computational cost, fine-tuned FMs demonstrate competitive performance compared to base models trained from scratch.

Figures

Figures reproduced from arXiv: 2411.14421 by the authors.

Figure 1
Figure 1. (a) Distribution of building type in IL-HET. (b) Boxplot of the spread of load values for each building type in IL-HET. (c) Load shapes from two buildings each from IL-HET and IL-HOM. base models and foundation models (FMs). Base models are trained from scratch on a given dataset, while FMs are pre-trained on a large corpora, and only need to be finetuned for the given downstream application. We sample across a wide… view at source ↗
Figure 2
Figure 2. Test set performance of base models for L = 512 and (a) T = 4, (b) T = 48, (c) T = 96. as well. On the other hand, the zero-shot performance of TimesFM is inadequate compared to other base models, highlighting the importance of full-parameter finetuning to unlock FM’s potentials. Finally, we analyze the importance of base model size on performance in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Distribution of building type in IL-HOM. (b) Boxplot of the spread of values in the time series of each building type in IL-HOM. Further evidence of the homogeneity of IL-HOM can be inferred from the equivalent figures to Figures 1a and 1b as presented in Figures 3a and 3b. While both figures contain only one element thanks to IL-HOM only containing buildings of the type Warehouse, the lower spread as well as va… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training instability in Transformer training on [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Predictions of the test set loads for two selected buildings from [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

  1. [4]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin

    DOI: https://doi.org/10.24432/C58C86. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems ,

    doi:10.24432/c58c86

  2. [7]

    Table 7: Model sizes, parameter counts and structural hyperparameters for TimesFM (for a comprehensive list, see (Das et al., 2024)) Model name Model hyperparameters Parameter count Model size TimesFM Token size: 1280 Num. Layers: 20 200m 854MB D Hyperparameter Tuning Given these models span a wide range of architectures, using the same value for the lear...

    work page 2024

  3. [9]

    8 Contrary to conventional implementations, we use positional embeddings for temporal inputs (i.e

    and Informer (Zhou et al., 2021), with the latter incorporating the ProbSparse attention mechanism. 8 Contrary to conventional implementations, we use positional embeddings for temporal inputs (i.e. u), alongside the standard positional embeddings. • Decomposition-based Architectures: These models replace the attention mechanism of the Transformer with ot...

    work page 2021

  4. [10]

    to generate the forecast. • Patch-based Architectures: These models split the input sequence into smaller sub-sequences called patches, followed by learning the interactions between different patches to better predict the forecast. In this category, we use PatchTST (Nie et al., 2023), which uses patching in addition to channel-independence to generate for...

    work page 2023

  5. [12]

    The metric is NMSE, and the best learning rates are highlighted

    10 Table 8: Learning rate tuning over the validation set on the dataset IL-HET. The metric is NMSE, and the best learning rates are highlighted. Model 10−3 5 × 10−4 10−4 5 × 10−5 10−5 5 × 10−6 10−6 5 × 10−7 L = 512, T= 4 LSTM 0.8258 0.2535 0.2020 0.1853 0.2804 0.2823 0.3878 0.4077 LSTNet 0.4564 0.2236 0.1417 0.1709 0.2202 0.3225 0.3207 0.3397 Transformer ...

    work page 2020

  6. [1997]

    with a dense decoder, and LSTNet (Lai et al., 2018), which combines RNNs with convolutional layers. • Transformer-based Architectures: These models use self-attention and cross-attention mechanisms in an encoder-decoder framework to parallelize the generation of the output sequence, given an input sequence. In this category, we use the vanilla Transformer...

    work page 2018

  7. [2007]

    Openstudio: An open source integrated analysis platform

    Rob Guglielmetti, Daniel Macumber, and Nicholas Long. Openstudio: An open source integrated analysis platform. In Proceedings of Building Simulation 2011: 12th Conf. of IBPSA, volume 12 of Building Simulation, pages 442–449, Sydney, Australia, November

    work page 2011

  8. [2013]

    UK Data Service, SN, 7857(2015):7857–7851,

    work page 2015

Show all 12 references
  1. [2015]

    Going deeper with convolutions

    5 Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1–9,

    2015

  2. [2017]

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long

    URL https://proceedings.neurips.cc/paper_ files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Proceedings of A...

    2017

  3. [2021]

    6 A Analysis of IL-HET and IL-HOM Table 2: Statistics for loads in IL-HOM and IL-HET

    URL https://ojs.aaai.org/index.php/AAAI/article/ view/17325. 6 A Analysis of IL-HET and IL-HOM Table 2: Statistics for loads in IL-HOM and IL-HET. Dataset Mean (loads) Standard deviation (loads) IL-HOM 17.4307 24.3129 IL-HET 21.8440 51.3635 Both IL-HET and IL-HOM contain load ...

    2015

  4. [2023]

    URL https://ojs.aaai.org/index.php/AAAI/ article/view/26317

    doi: 10.1609/aaai.v37i9.26317. URL https://ojs.aaai.org/index.php/AAAI/ article/view/26317. Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. InProcee...

    doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.