Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Real-E, a 39-country, 20-source electricity dataset, exposes why current forecasters fail to generalize, and the paper argues that the culprit is the non-stationary correlation structure between energy sources.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 04:56 UTC pith:VIX7WV6W

load-bearing objection Useful dataset resource, but the benchmark claims in this preprint rest on a two-country subset and some sloppy arithmetic. the 4 major comments →

arxiv 2509.05768 v1 pith:VIX7WV6W submitted 2025-09-06 cs.LG cs.AI

Real-E: A Foundation Benchmark for Advancing Robust and Generalizable Electricity Forecasting

Chen Shao , Yue Wang , Zhenyi Zhu , Zhanbo Huang , Sebastian P\"utz , Benjamin Sch\"afer , Tobais K\"afer , Michael F\"arber This is my paper
classification cs.LG cs.AI
keywords electricity forecastingmultivariate time seriesbenchmark datasetnon-stationary correlationgraph neural networkstransformersenergy mixgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces Real-E, a real-world electricity forecasting dataset spanning 39 European countries, 20 generation types, and up to ten years of 15-minute records. The authors argue that existing benchmarks, typically single-country and single-source series, hide how much the correlations between energy sources shift over time. To expose that, they define two metrics, TGV and GSD, and show Real-E has roughly twice the correlation volatility of earlier benchmarks. Benchmarking over 20 models, they find that Transformer-based and spectral-GNN forecasters degrade sharply on Real-E, while spatial-GNN models that explicitly adapt graph structure generalize best. The contribution is a dataset and a diagnosis: models that rely on stationary cross-series structure will not hold up in real grids.

Core claim

The paper's central discovery is that the cross-energy correlation structure of real European electricity systems is non-stationary at the scale of ordinary forecasting windows, and that this non-stationarity, not just noise or missing data, is what breaks state-of-the-art time series models. Using Real-E, a dataset built from the European network operators' transparency data, the authors construct time-windowed Pearson correlation graphs between generation types and quantify their evolution with two metrics: Temporal Graph Volatility, a Frobenius-norm difference between adjacency matrices of consecutive windows, and Graph Spectral Divergence, the Euclidean distance between sorted Laplacian

What carries the argument

The central object is Real-E, a multivariate time-series dataset built from the ENTSO-E transparency platform, organized by country and generation type with rich metadata. The paper's argument runs through two new diagnostic metrics: Temporal Graph Volatility (TGV), the Frobenius norm of the difference between adjacency matrices of time-windowed correlation graphs, and Graph Spectral Divergence (GSD), the Euclidean distance between sorted Laplacian spectra of those graphs. These metrics quantify how quickly and how structurally the cross-energy dependencies change, and they do the work of showing that Real-E is more volatile than existing benchmarks. The forecasting comparison itself is the

Load-bearing premise

The paper's conclusions assume that the Germany and France generation-by-type data at 15-minute resolution is representative enough of Real-E's full 39-country, 20-source collection that the measured model rankings and volatility metrics on this subset carry over to the whole dataset.

What would settle it

Compute TGV and GSD on the full Real-E collection (all 39 countries) rather than the Germany–France subset; if the subset's volatility is systematically higher or lower than the full dataset, the claim that real-world correlation dynamics are more volatile and therefore break transformers may be an artifact of the subset. Similarly, test a model that ranks well on the subset on a country with a different energy mix, such as hydro-dominated Norway; if its rank reverses, the generalizability claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If Real-E is representative of real grids, forecasting models intended for deployment should be validated on multi-country, multi-source data rather than single-series benchmarks.
  • Transformer and spectral-GNN architectures that assume stable cross-series correlations will not transfer to electricity systems; the paper says global attention mechanisms struggle with rapidly shifting dependencies.
  • Spatial GNNs that explicitly model and adapt the graph structure offer a stronger starting point for robust electricity forecasting.
  • TGV and GSD could be used as dataset selection tools: a benchmark with low volatility may give misleadingly optimistic generalization estimates.
  • The 85.4% average error increase across Transformer baselines quantifies the gap between benchmark performance and realistic grid conditions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's comparison uses a Germany–France subset of Real-E; if TGV/GSD computed across all 39 countries show even higher volatility, the reported model rankings may understate the generalization gap.
  • A practical extension would be to measure TGV/GSD at changing window lengths to test whether the volatility is an artifact of 15-minute sampling or reflects genuine regime shifts; the metric threshold that separates benign from breaking correlation dynamics is not yet established.
  • Because Real-E includes load, balancing, market, and transmission tables, the same benchmark protocol could be applied to forecasting prices or imbalances, not just generation; the paper's conclusion that spatial GNNs win on generation may or may not transfer to those targets.
  • The dataset's 15-minute resolution and ten-year span also invite long-horizon and hierarchical forecasting benchmarks (country to bidding zone to unit), which the paper only starts to explore.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Real-E, a multivariate electricity forecasting dataset assembled from ENTSO-E, claimed to be the largest such dataset to date, with 39 European countries, 20 generation types, up to 9.5 years of history, and multiple temporal resolutions. The authors propose two metrics, Temporal Graph Volatility (TGV) and Graph Spectral Divergence (GSD), to measure time-varying cross-energy correlation structure, and report that Real-E has higher volatility than existing benchmarks. They benchmark 20 forecasting models from several architectural families on a two-country subset, Real-E(Germany, France), and conclude that Transformer-based and spectral GNN methods degrade sharply compared with their performance on Electricity and Solar benchmarks, while spatial GNNs generalize best. The dataset and benchmark code are promised via external links.

Significance. If the empirical claims are supported, Real-E would be a valuable resource for the time-series forecasting community: multi-country, multi-energy, long-horizon data with associated metadata is genuinely missing from current public benchmarks. The paper's systematic comparison of 20 baselines across architectural families and the introduction of correlation-shift metrics are useful contributions. The public-domain licensing and the explicit research questions are also strengths. However, the central generalization claims currently rest entirely on a two-country subset, the proposed metrics are not shown to be comparable across datasets of different dimensions, and some reported performance-degradation numbers are inconsistent with the paper's own table. These issues must be resolved before the paper's conclusions can be accepted.

major comments (4)
  1. [§3, §4, §5] The paper's cross-dataset generalization conclusions are drawn entirely from the Real-E(Germany,France) subset. Section 3 states 'we focus on Germany and France as representative cases' and Section 4 states 'We choose a subset of proposed datasets, Real-E(Germany,France).' No evidence is provided that Germany and France are representative of the full 39-country, 20-energy-type dataset in terms of energy mix, correlation dynamics, regulatory setting, or data quality. The abstract and Section 5 nevertheless generalize to Real-E as a whole. Because RQ1 and RQ2 are framed as Real-E-wide questions, this is a load-bearing gap. Please either benchmark on the full dataset or provide a representativeness analysis (e.g., TGV/GSD distributions across countries, energy-mix diversity, missing-data patterns).
  2. [§3, Metrics in Practice, Eqs. (1)-(2)] TGV and GSD are not normalized for matrix dimension. TGV in Eq. (1) is a sum of Frobenius norms of adjacency matrices of size m×n; GSD in Eq. (2) is an Euclidean distance between Laplacian eigenvalue vectors. If Germany/France use a different number of time series than Electricity or Solar, the higher TGV/GSD values for Real-E may reflect dimension scaling rather than genuinely more complex correlation dynamics. The text reports averages 'on existing benchmarks' and 'our dataset attains higher values' without stating the number of series in each dataset or any normalization. This weakens the central data-analysis claim. Please report normalized metrics or dimension-matched comparisons.
  3. [§4, RQ1] The reported degradation rates are internally inconsistent with Table 2. For the four Transformer-based models, using the MAE values in Table 2 gives approximately (0.103+0.097+0.1105+0.156)/4 = 0.1166 on Electricity+Solar and (0.1845+0.2455+0.210+0.219)/4 = 0.2148 on Germany+France, an increase of about 84%, not 85.4%. For spectral GNNs, the category average MAE rises from about 0.1145 to 0.2235, an increase of about 95%, not 16.2%; even the best spectral model (FourierGNN) rises by about 20%. Please correct these numbers and clarify whether 'Spectral GNN' refers to a category or a specific model.
  4. [§2, §4, 'Real-E Link', 'Benchmark Link', 'Link', 'Github'] The manuscript relies on promised but absent artifacts. The dataset link is given as '[Real-E Link]', the benchmark link as '[Benchmark Link]', and preprocessing/splitting details are deferred to 'Link' and 'Github'. Section 2 only says 'we excluded time series with substantial missing values'; Section 4 reports a hyperparameter search range but not the actual per-model configurations, normalization, or validation procedure. For a benchmark paper, this prevents verification of the central empirical claims. Please include a complete data card, preprocessing script, and per-model hyperparameter settings in the submission or an accessible appendix.
minor comments (5)
  1. [Eq. (1)] The summation indices in Eq. (1) are inconsistent: the first sum runs from t=0 to T-1 and the second from t=1 to T. Please define T and align the index ranges.
  2. [Table 2] The top row of Table 2 (0.5391, 0.6245, 1.0969, 0.9698) is unlabeled. Please state explicitly whether these are TGV or GSD values and for which correlation-window settings.
  3. [References [22], [23]] Reference [22] and [23] are duplicate Informer entries; [22] has a malformed author field ('H. et al. Zhou'). Please merge and correct.
  4. [Abstract and Section 2] The abstract says 'over 74 power stations across 30+ European countries,' while Table 1 and Figure 1 report 39 countries and 20 energy types. Please reconcile these numbers.
  5. [Throughout] There are spacing/grammar typos, e.g., 'TheReal-E' in the abstract and Section 2, and 'our dataset lay a solid foundation' in Section 5. Please proofread.

Circularity Check

0 steps flagged

No significant circularity — benchmark results are measured, not derived from fitted inputs or self-citations.

full rationale

The paper's derivation chain is empirical rather than definitional. Real-E is constructed from ENTSO-E data; the TGV/GSD metrics (Eqs. 1–2) are defined directly on Pearson correlation matrices of observed time series and are computed from the data, not from model outputs or fitted parameters. The central benchmark conclusion—that Transformer-based and spectral GNN models struggle on Real-E—is a measured outcome of running 20 published baselines under a common protocol; no parameter is fitted to the claim and then renamed a prediction. The paper does not rely on load-bearing self-citation: the external baselines' best results are attributed to [16,19] (TimesNet, FourierGNN), and none of the cited authors overlap with the present authors. The main weaknesses—restricting data analysis and benchmarks to Germany/France while extrapolating to 39 countries, and possible arithmetic inconsistencies in reported degradation percentages—are external-validity and correctness risks, not circularity. There is no equation whose output equals its input by construction, no fitted parameter masquerading as a prediction, and no uniqueness theorem imported from the authors' prior work. Therefore no circular step can be exhibited with the required specificity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters are fit to make a central derivation work: the paper is an empirical benchmark. The main hidden assumptions are the choice of the Germany and France subset as representative and the reliability of the ENTSO-E source; these are domain assumptions rather than fitted parameters. The two correlation metrics are introduced without any independent benchmark showing they capture what they claim, but they are not used as fitted inputs.

pith-pipeline@v1.3.0-alltime-deepseek · 7261 in / 4660 out tokens · 42187 ms · 2026-08-05T04:56:05.344758+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Real-E: A Foundation Benchmark for Advancing Robust and Generalizable Electricity Forecasting." pith.science (2026). https://pith.science/paper/VIX7WV6W

@misc{pith2026250905768,
  author       = {Pith},
  title        = {Pith review of: Real-E: A Foundation Benchmark for Advancing Robust and Generalizable Electricity Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIX7WV6W}},
  note         = {Machine review of arXiv:2509.05768}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Energy forecasting is vital for grid reliability and operational efficiency. Although recent advances in time series forecasting have led to progress, existing benchmarks remain limited in spatial and temporal scope and lack multi-energy features. This raises concerns about their reliability and applicability in real-world deployment. To address this, we present the Real-E dataset, covering over 74 power stations across 30+ European countries over a 10-year span with rich metadata. Using Real- E, we conduct an extensive data analysis and benchmark over 20 baselines across various model types. We introduce a new metric to quantify shifts in correlation structures and show that existing methods struggle on our dataset, which exhibits more complex and non-stationary correlation dynamics. Our findings highlight key limitations of current methods and offer a strong empirical basis for building more robust forecasting models

Figures

Figures reproduced from arXiv: 2509.05768 by Benjamin Sch\"afer, Chen Shao, Michael F\"arber, Sebastian P\"utz, Tobais K\"afer, Yue Wang, Zhanbo Huang, Zhenyi Zhu.

Figure 1
Figure 1. Figure 1: Real-E is the largest electricity dataset to date, covering 10.18 million km2 across 39 countries (colored dots) and spanning over 20 generation types (yellow boxes). Each power plant is accompanied by detailed metadata (red boxes). It offers a rigorous foundation for advancing multivariate electricity forecasting research. data factors and their impact on the performance of current forecasting models have… view at source ↗
Figure 2
Figure 2. Figure 2: We use a polar plot (a) and a boxplot (b) to illus￾trate seasonal generation patterns of two complementary energy sources in Germany: solar and brown coal. (a) Solar generation shows a clear seasonal trend, with higher out￾put in spring and summer, peaking in June and reaching its minimum in December — primarily driven by solar irradiance. (b) In contrast, brown coal generation peaks in autumn and winter a… view at source ↗
Figure 3
Figure 3. Figure 3: We present a series of Pearson correlation ma [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages · 2 internal anchors

  1. [1]

    Uci machine learning repository, 2007

    Arthur Asuncion, David Newman, et al. Uci machine learning repository, 2007

    2007

  2. [2]

    Zico Kolter, and Vladlen Koltun

    Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018

    Pith/arXiv arXiv 2018

  3. [3]

    G. E. P. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung.Time Series Analysis: Forecasting and Control. Wiley, 2015

    work page 2015

  4. [4]

    George E. P. Box and Gwilym M. Jenkins.Time Series Analysis: Forecasting and Control. Holden-Day, San Francisco, 1970

    work page 1970

  5. [5]

    Spectral temporal graph neural network for multivariate time-series forecasting.Advances in Neural Information Processing Systems (NeurIPS), 2020

    Liangzu Cao, Yixian Chen, and Junbo Li. Spectral temporal graph neural network for multivariate time-series forecasting.Advances in Neural Information Processing Systems (NeurIPS), 2020

    work page 2020

  6. [6]

    Towards Expressive Spectral-Temporal Graph Neural Networks for Time Series Forecasting

    Ming Jin, Guangsi Shi, Yuan-Fang Li, Bo Xiong, Tian Zhou, Flora D. Salim, Liang Zhao, Lingfei Wu, Qingsong Wen, and Shirui Pan. Towards expres- sive spectral-temporal graph neural networks for time series forecasting. arXiv preprint arXiv:2305.06587, 2023. doi: 10.48550/arXiv.2305.06587. Re- vised version submitted 23 Feb 2025

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2305.06587 2023

  7. [7]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. InInternational Conference on Learning Representations, 2020

    work page 2020

  8. [8]

    Guokun Lai, Wei-Cheng Chang, and Yiming et al. Yang. Modeling long- and short-term temporal patterns with deep neural networks.ACM SIGIR, pages 95–104, 2018

    work page 2018

  9. [9]

    Multivariate time-series forecasting with tem- poral polynomial graph neural networks

    Yijing Liu, Qinxian Liu, Jian-Wei Zhang, Haozhe Feng, Zhongwei Wang, Zihan Zhou, and Wei Chen. Multivariate time-series forecasting with tem- poral polynomial graph neural networks. InAdvances in Neural Information Processing Systems, volume 36, 2022. URL https://papers.neurips.cc/paper/ 2022/file/7b102c908e9404dd040599c65db4ce3e-Paper-Conference.pdf

    work page 2022

  10. [10]

    Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio

    Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. InInternational Conference on Learning Representations (ICLR),

    work page

  11. [11]

    Entso-e electricity data repository, 2024

    ENTSO-E Transparency Platform. Entso-e electricity data repository, 2024. URL https://transparency.entsoe.eu

    work page 2024

  12. [12]

    Rajat Sen, Haekyu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. InAdvances in Neural Information Processing Systems, volume 32, 2019

    work page 2019

  13. [13]

    Christopher A. Sims. Macroeconomics and reality.Econometrica, 48(1): 1–48, 1980

    work page 1980

  14. [14]

    Zhang, and Jun Zhou

    Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting. InInternational Conference on Learning Representations (ICLR), 2024. Poster

    work page 2024

  15. [15]

    Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting.Ad- vances in Neural Information Processing Systems (NeurIPS), 2021

    Haixu Wu, Jiehui Xu, and Jianmin Wang. Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting.Ad- vances in Neural Information Processing Systems (NeurIPS), 2021

    work page 2021

  16. [16]

    Timesnet: Tem- poral 2d-variation modeling for general time series analysis.International Conference on Learning Representations (ICLR), 2023

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Timesnet: Tem- poral 2d-variation modeling for general time series analysis.International Conference on Learning Representations (ICLR), 2023

    work page 2023

  17. [17]

    Connecting the dots: Multivariate time series forecasting with graph neural networks

    Yingheng Wu, Kristjan Greenewald, Jonathan Hertrich, et al. Connecting the dots: Multivariate time series forecasting with graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020

    work page 2020

  18. [18]

    Graph wavenet for deep spatial-temporal graph modeling

    Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. InProceedings of International Joint Conference on Artificial Intelligence, pages 1907–1913, 2019

    work page 1907

  19. [19]

    FourierGNN: Rethinking multivariate time series forecasting from a pure graph perspective

    Kun Yi, Qi Zhang, Wei Fan, Hui He, Liang Hu, Pengyang Wang, Ning An, Longbing Cao, and Zhendong Niu. FourierGNN: Rethinking multivariate time series forecasting from a pure graph perspective. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  20. [20]

    Cliques of orders three and four in the Paley-type graphs

    Ailing Zeng, Wenqian Zhang, Yuxuan Zheng, Yixuan Xu, Jiajun Yan, Weinan Xu, Yong Zhang, and Xiaotian Ma. Dlinear: A baseline algorithm for time series forecasting.arXiv preprint arXiv:2301.07021, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    Aggarwal, and Guo-Jun Qi

    Liheng Zhang, Charu C. Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi-frequency trading patterns. InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 2141–2149. ACM, 2017. doi: 10.1145/3097983.3098061

    work page arXiv 2017

  22. [22]

    H. et al. Zhou. Informer: Beyond efficient transformer for long sequence time-series forecasting. InProceedings of AAAI Conference on Artificial Intelligence, pages 11106–11115, 2021

    work page 2021

  23. [23]

    Informer: Beyond efficient transformer for long sequence time-series forecasting.Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):11106–11115, 2020

    Haoyi Zhou, Shanghang Zhang, Jie Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting.Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):11106–11115, 2020

    work page 2020

  24. [24]

    Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting

    Tian Zhou, Ziqing Ma, Qingsong Wen, Ruixin Jin, Jianyuan Zhu, Xinyu Zhang, and Liang Sun Wang. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. InInternational Conference on Machine Learning (ICML), pages 11214–11225, 2022

    work page 2022

  25. [2020]

    URL https://openreview.net/forum?id=r1ecqn4YwB

    work page