REVIEW 3 major objections 5 minor 24 references
Flare data source, not model, flips forecast skill scores.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 16:25 UTC pith:4P7QGGIA
load-bearing objection Useful catalog audit and scaling-factor quantification, but the headline skill-score comparison is confounded because O and S models are scored on different test labels. the 3 major comments →
Defects and Inconsistencies in Solar Flare Data Sources: Implications for Machine Learning Forecasting
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the choice of flare-response catalog is a first-order determinant of measured machine-learning forecast skill, comparable in size to the choice of model family. The paper demonstrates this by training identical LSTM and logistic-regression models on combinations of two predictor products (near-real-time versus definitive active-region magnetic summary parameters) and two response catalogs (operational versus science-quality), then evaluating on fixed test intervals in Solar Cycle 25. It quantifies concrete defects: before December 2019 the operational catalog carried a scaling factor that depressed reported X-ray intensities by roughly 30 percent; the science-qualit
What carries the argument
The central objects are a pair of flare-response catalogs—the operational real-time event list and the reprocessed science-quality event list—and the active-region magnetic parameter series (SHARP) used as predictors. The argument is carried by cross-catalog event matching with tolerance windows on peak time and log-intensity, and by controlled training experiments that hold the model architecture fixed while swapping catalog and predictor versions.
Load-bearing premise
The comparisons assume that skill scores calculated with different test-label catalogs are directly comparable, even though the catalogs disagree about which events count as flares.
What would settle it
Reproduce the model comparison of Figure 15 but evaluate both catalog-trained models on a single consensus test set containing only flares that appear in both catalogs with consistent intensities after the scaling correction; if the TSS advantage of the operational catalog during solar minimum disappears, the reported effect is an artifact of label mismatch.
If this is right
- Skill scores reported in the literature can differ by as much as the catalog used, so studies using operational rather than science-quality labels are not directly comparable.
- Before the December 2019 satellite transition, operational flare intensities need a multiplicative correction of about 1/0.7 (or +0.15 in log10) to match science-quality values.
- No single catalog dominates: operational labels improve TSS during solar minimum, while science-quality labels are generally better during solar maximum.
- Near-real-time active-region parameters produce substantially more variable skill scores than definitive calibrated parameters, especially in high-activity periods.
- LSTM and logistic-regression models respond differently to data-source swaps, and the simpler model is more stable in these comparisons.
Where Pith is reading between the lines
- If catalog-dependent ranking flips with solar cycle, meta-analyses of flare-forecasting papers should include data provenance as a covariate; otherwise apparent model improvements may be catalog artifacts.
- A testable extension would train on one catalog and evaluate on the other, separating label-definition effects from true predictive signal; the paper's design matches training and test labels, which may inflate the operational catalog's apparent advantage.
- The 28 percent missing active-region annotation in the science-quality list is an annotation gap rather than a physics gap; better region-matching algorithms could change the cycle-dependent conclusions.
- The scaling-factor correction could be independently validated by recomputing flare-class thresholds from continuous X-ray flux data before and after the transition.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper audits common solar flare data sources (SWPC-FTP operational list, NCEI Science-Quality list, SSW catalog, SunPy-HEK) and SHARP predictor products, quantifying defects such as the pre-2019 SWPC scaling factor (~0.7), catalog mismatches, missing AR numbers, and missing HARP data. It then trains LSTM and logistic regression models on combinations of label catalogs and SHARP variants, reporting skill scores over different phases of Solar Cycle 25. The paper concludes that data-source selection materially changes forecast skill, with operational labels improving performance during solar minimum and degrading it during solar maximum.
Significance. If the central claim holds, the paper would be an important caution for the flare-forecasting community, showing that model comparison across studies is invalid without standardizing data provenance. The catalog audit itself is valuable: it provides reproducible tolerance-based matching, quantifies the 0.7 scaling bias, and releases a processing pipeline in Appendix C. However, the modeling evidence for the central claim is weakened by the evaluation design, where models are scored against the same catalog used for training, so differences in skill may reflect label difficulty rather than forecasting ability. The paper's descriptive analyses are strong, but the headline solar-cycle-dependent recommendation is not yet supported.
major comments (3)
- [Section 4.2, Tables 10-15, Figures 13-15] The central model comparison is confounded by the evaluation design. Models trained on SWPC-FTP labels are evaluated on SWPC-FTP test labels, while models trained on Science-Quality labels are evaluated on Science-Quality test labels. Because the catalogs differ in event membership, timings, and AR assignment, the two test sets are not the same. The higher TSS for O models during 2020-2021 (e.g., Table 13: Logistic-S-DEF 0.15 vs Logistic-O-DEF 0.37) may thus reflect that operational labels are easier to predict (e.g., fewer near-threshold events), not that the model better forecasts the underlying flare process. The Section 5 conclusion (item 3) that operational labels 'improve performance during solar minimum but degrade performance during solar maximum' is not separable from the change in evaluation set. Recommend evaluating both models on a common test set (e.g., matched flares with a
- [Section 2.4 and Section 4.1] The Science-Quality models use only the 72% of flares with valid AR numbers (Section 2.4). If the remaining 28% are not missing at random (e.g., near-limb or small flares), the training and test distributions for S models differ from O models. This compounds the cross-catalog confound. The paper should characterize the missingness (e.g., class, location, time) and show that results are robust to the augmentation procedure, for example by also training on the subset of SWPC-FTP flares with valid AR numbers.
- [Section 4.1, last paragraph] The claim 'We also demonstrate numerically that differences arising from data inconsistencies or quality are unaffected by the choice of sample construction method' is not supported by any presented results. No such numerical demonstration appears in the paper. Either provide the analysis or remove the claim.
minor comments (5)
- [Table 9] Typos: 'SPWC-FTP' should be 'SWPC-FTP'.
- [Figure 19] The caption says 'Marginal Distribution of SHARPs over different periods aggregated across Active Regions' but the figure actually appears to show something else (likely flare class counts). The dashed lines mentioned in the text are not described. Please correct the caption and integrate the figure with the text.
- [Appendix A] The 'Quiet Period Analysis' appendix appears to be a leftover section; it is not referenced from the main text and the two figures are not integrated into the narrative. Recommend moving this to a future paper or fully integrating it into Sections 3-4.
- [Tables 10-15 and Figure 15] Inconsistent test-period definitions: Tables 10-15 use 'Min (2020–2021), Evolving (2022), Max (2023–2024)' while Figure 15 uses '2020-01-01 to 2022-01-01' and '2023-01-01 to 2025-01-01'. Clarify whether 2025 data are included and align the labels.
- [References] Reference [Jin et al., 2026 in prep] is not a standard citation; use 'in preparation' or omit if not publicly available.
Circularity Check
No significant circularity; empirical audit with minor self-referential data augmentation.
full rationale
This is an empirical audit, not a derivation. The central quantities—catalog mismatch rates, the SWPC scaling ratio, and the model skill scores—are computed from independent data products and matched by explicitly stated tolerance criteria. The only self-referential element is Section 2.4, where Science-Quality flares are assigned AR numbers by matching to SWPC-FTP/SSW records, and the augmented list is then used in the Section 4 model comparisons. However, the flare class labels used as responses remain NCEI Science-Quality values, so the TSS comparisons do not reduce to the matching step by construction. The Section 5 recommendation that operational labels help at solar minimum but hurt at solar maximum is vulnerable to the confound that O and S models are evaluated on different test-label sets; that is a validity threat, not a circularity, and the paper itself discloses the 72% AR-assignment coverage limitation. Self-citations such as Jiao et al. (2020) and Chen et al. (2019b) are used for standard modeling choices and are not invoked as uniqueness theorems or as substitutes for evidence, so they are not load-bearing in a circular sense. No equation or fitted parameter is shown to be identical to its input; thus the circularity score is low.
Axiom & Free-Parameter Ledger
free parameters (5)
- Flare matching peak-time tolerance =
15 min
- Flare matching log10 intensity tolerance =
0.3
- AR assignment distance threshold =
250 arcsec
- Limb exclusion longitude =
±70° central meridian
- C-class exclusion =
excluded
axioms (5)
- domain assumption Peak-time and log-intensity proximity is sufficient to identify the same flare across catalogs.
- domain assumption Nearest active region in SRS by HPC distance is the correct AR for a flare with known location.
- domain assumption Binary M/X-vs-A/B classification with C-class removed is a valid proxy for operational flare forecasting.
- standard math Standard statistical/ML training assumptions (bootstrap ensembles, stratified split, PCA retention) preserve comparability across data sources.
- domain assumption SWPC scaling factor of 0.7 before GOES-16 is correctly removed from log intensities for comparison.
Cite this review
Pith. "Pith review of Defects and Inconsistencies in Solar Flare Data Sources: Implications for Machine Learning Forecasting." pith.science (2026). https://pith.science/paper/4P7QGGIA
@misc{pith2026251213417,
author = {Pith},
title = {Pith review of: Defects and Inconsistencies in Solar Flare Data Sources: Implications for Machine Learning Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/4P7QGGIA}},
note = {Machine review of arXiv:2512.13417}
}
read the original abstract
Machine learning models for forecasting solar flares have been trained and evaluated using a variety of data sources, including Space Weather Prediction Center (SWPC) operational and science-quality data. Typically, data from these sources is minimally processed before being used to train and validate a forecasting model. However, predictive performance can be affected if defects and inconsistencies between these data sources are ignored. For a set of commonly used data sources, along with the software that queries and outputs processed data, we identify their defects and inconsistencies, quantify their extent, and show how they can affect predictions from data-driven machine-learning forecasting models. We also outline procedures for fixing these issues or at least mitigating their impacts. Finally, based on thorough comparisons of the effects of data sources on the trained forecasting model's predictive skill scores, we offer recommendations for using different data products in operational forecasting.
Figures
Reference graph
Works this paper leans on
-
[7]
doi: 10.1007/s11214-017-0462-5. A. Hu, E. Camporeale, and B. Swiger. Multi-hour-ahead dst index prediction using multi-fidelity boosted neural networks.Space weather, 21(4):e2022SW003286,
-
[8]
doi: 10.1007/s11207-010-9624-2. D. Iong, Y. Chen, G. Toth, S. Zou, T. Pulkkinen, J. Ren, E. Camporeale, and T. Gombosi. New findings from explainable sym-h forecasting using gradient boosting machines.Space Weather, 20(8):e2021SW002928,
-
[10]
HARP - HMI Active Region Patches.http://jsoc.stanford.edu/jsocwiki/HARPDataSeries
JSOC. HARP - HMI Active Region Patches.http://jsoc.stanford.edu/jsocwiki/HARPDataSeries. Last edited by Todd Hoeksema on 2024-07-31. Accessed: 2024-08-08. S. Kasapis, L. Zhao, Y. Chen, X. Wang, M. Bobra, and T. I. I. Gombosi. Interpretable machine learning to forecast sep events for solar cycle 23.Space Weather,
2024
-
[11]
K. D. Leka and G. Barnes. Chapter 3 – Solar Flare Forecasting: Present Methods and Challenges. In N. Buzu- lukova, editor,Extreme Events in Geospace, pages 65–98. Elsevier, 2018b. doi: 10.1016/B978-0-12-812700-1. 00003-0. K. D. Leka, G. Barnes, and E. Wagner. The nwra classification infrastructure: description and extension to the discriminant analysis fl...
-
[13]
URLhttps://doi.org/10.1007/ s11207-010-9697-y
doi: 10.1007/s11207-010-9697-y. URLhttps://doi.org/10.1007/ s11207-010-9697-y. J. Mothersbaugh III, J. Machol, and E. Zetterlund. Readme for science-quality goes 8-15 xrs data, October
-
[14]
C. Pandey, R. A. Angryk, and B. Aydin. Unveiling the potential of deep learning models for solar flare prediction in near-limb regions.arXiv preprint arXiv:2309.14483,
-
[16]
doi: 10.1007/978-3-031-70381-2
-
[17]
doi: 10.3847/1538-4357/aaed40. W. D. Pesnell, B. Thompson, and P. Chamberlin.The Solar Dynamics Observatory (SDO). Springer,
-
[18]
doi: 10.1007/s11207-011-9841-3. P. H. Scherrer, J. Schou, R. Bush, A. Kosovichev, R. Bogart, J. Hoeksema, Y. Liu, T. Duvall, J. Zhao, A. Title, et al. The Helioseismic and Magnetic Imager (hmi) Investigation for the Solar Dynamics Observatory (SDO). Solar Physics, 275:207–227,
-
[21]
X. Wang, Y. Chen, G. Toth, W. B. Manchester, T. I. Gombosi, A. O. Hero, Z. Jiao, H. Sun, M. Jin, and Y. Liu. Predicting Solar Flares with Machine Learning: Investigating Solar Cycle Dependence.The Astrophysical Journal, 895(1):3, 2020b. doi: 10.3847/1538-4357/ab89ac. Z. Wang, S. Zou, H. Sun, and Y. Chen. Forecast global ionospheric tec: Apply modified u-n...
-
[22]
doi: 10.1007/s11207-009-9487-6. K. Yi, Y.-J. Moon, G. Shin, and D. Lim. Forecast of major solar x-ray flare flux profiles using novel deep learning models.The Astrophysical Journal Letters, 890(1):L5,
-
[1993]
doi: 10.1038/363426a0. A Quiet Period Analysis of Active Regions In this section, we aim to determine whether there are distributional differences between active regions during ongoing flares and during a “quiet period”. As an initial approach, we define the ”quiet period” of an active region as any period 24 hours removed from flare activity (i.e., more ...
-
[2007]
doi: 10.1086/511857. H. Sun, W. Manchester IV, and Y. Chen. Improved and interpretable solar flare predictions with spatial and topological features of the polarity inversion line masked magnetograms.Space weather, 19(12):e2021SW002837,
-
[2012]
doi: 10.1007/s11207-011-9804-8. L. E. Boucheron, A. Al-Ghraibah, and R. T. J. McAteer. Prediction of Solar Flare Size and Time-To-Flare Using Support Vector Machine Regression.The Astrophysical Journal, 812(1):51,
-
[2014]
doi: 10.1007/s11207-014-0529-3. M. G. Bobra, P. J. Wright, X. Sun, and M. J. Turmon. SMARPs and SHARPs: Two Solar Cycles of Active Region Data.The Astrophysical Journal Supplement Series, 256(2):26,
-
[2015]
doi: 10.1088/0004-637X/798/2/
-
[2017]
27 H. Liu, C. Liu, J. T. Wang, and H. Wang. Predicting solar flares using a long short-term memory network.The Astrophysical Journal, 877(2):121, 2019a. H. Liu, C. Liu, J. T. L. Wang, and H. Wang. Predicting solar flares using a long short-term memory network. The Astrophysical Journal, 877(2):121, 2019b. doi: 10.3847/1538-4357/ab1b3c. J. Machol, S. Codre...
-
[2018]
H. Chen, N. Sachdeva, Z. Huang, B. van der Holst, W. Manchester IV, A. Jivani, S. Zou, Y. Chen, X. Huan, and G. Toth. Decent estimate of cme arrival time from a data-assimilated ensemble in the alfv´ en wave solar atmosphere model (decade-awsom).Space Weather, 23(1):e2024SW004165, 2025a. H. Chen, G. Toth, Y. Chen, S. Zou, Z. Huang, and X. Huan. Geodgp: On...
2019
-
[2019]
doi: 10.1029/2018sw002061. E. Camporeale, S. Wing, and J. Johnson.Machine learning techniques for space weather. Elsevier,
-
[2020]
Y. Yu, Y. Chen, L. Zhao, K. Whitman, W. Manchester, and T. Gombosi. Solar energetic particle forecasting with multi-task deep learning: Sepnet.arXiv preprint arXiv:2512.12786,
-
[2021]
doi: 10.3847/1538-4365/ac1f1d. P. Boerner, C. Edwards, J. Lemen, A. Rausch, C. Schrijver, R. Shine, L. Shing, R. Stern, T. Tarbell, A. Title, C. J. Wolfson, R. Soufli, E. Spiller, E. Gullikson, D. McKenzie, D. Windt, L. Golub, W. Podgorski, P. Testa, 25 and M. Weber. Initial calibration of the atmospheric imaging assembly (AIA) on the solar dynamics obser...
-
[2022]
Readme file for using solar event lists.ftp://ftp.swpc.noaa.gov/pub/warehouse/1996/1996_events
SWPC. Readme file for using solar event lists.ftp://ftp.swpc.noaa.gov/pub/warehouse/1996/1996_events. tar.gz/readme. Accessed: 2024-09-19. D. Telloni, M. L. Schiavo, E. Magli, S. Fineschi, S. Guastavino, G. Nicolini, R. Susino, S. Giordano, F. Amadori, V. Candiani, et al. Prediction capability of geomagnetic events from solar wind data using neural networ...
1996
-
[2023]
Pandey, T
C. Pandey, T. Adeyeha, J. Hong, R. Angryk, and B. Aydin. Advancing solar flare prediction using deep learning with active region patches. In A. Bifet, T. Krilaviˇ cius, I. Miliou, and S. Nowaczyk, editors,Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track. ECML PKDD 2024, volume 14950 of Lecture Notes in Computer Science. Sp...
2024
-
[2024]
Z. Jiao, H. Sun, X. Wang, W. Manchester, T. Gombosi, A. Hero, and Y. Chen. Solar flare intensity prediction with machine learning models.Space weather, 18(7):e2020SW002440, 2020a. Z. Jiao, H. Sun, X. Wang, et al. Solar flare intensity prediction with machine learning models.Space Weather, 18(7):e2020SW002440, 2020b. doi: 10.1029/2020SW002440. E. Jonas, M....
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.