REVIEW 3 major objections 4 minor 18 references
Simulated Operational Testing of the Prototype Implementation of the SOFIE Model: The 2025 Space Weather Prediction Testbed Exercise
T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read A physics-based solar-particle model produced a 4-day radiation forecast in 5 hours during a 2025 operational exercise.
desk verdict Useful speed demonstration and grid-comparison, but the 2001 flux skill is partly fitted via a post-hoc injection scaling factor; the abstract overstates the accuracy-vs-speed tradeoff. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The speed comes from the way SOFIE is assembled and from the adaptive grid, not from any single new physical term. AWSoM-R supplies a precomputed, stream-aligned magnetohydrodynamic (MHD) solar wind; EEGGL inserts a Gibson–Low magnetic flux rope matched to the observed active-region location and CME speed; M-FLAMPA (the Multiple Field-Line-Advection Model for Particle Acceleration) simulates diffusive shock acceleration and transport of protons along magnetic field lines advected by that wind. The operational innovation tested here is a two-level grid strategy: the global solar-corona background is coarsened by a factor of two while adaptive mesh refinement stays on the heliospheric current
What would settle it
Take the same fast grid setup and M-FLAMPA parameters used here, fix the injection scaling factor at its default value of 1.0 before running, and apply the pipeline to a set of historical SEP events; if the predicted >10 MeV onset peaks and integral fluxes stop agreeing with spacecraft measurements to within the factor of 2–3 shown for 2001, then the accuracy demonstrated here was fitted rather than forecast.
Extended reading notes
Core claim
On its own terms, the paper establishes that the complete SOFIE chain can clear the operational bar: the full 4-day simulation of an SEP event finished in 4.86 hours of real time using the fast grid setup, and the earliest useful 10-hour forecast was available within 2.35 hours of eruption (leading time 7.65 hours). Accuracy at the operational channels: for >10 MeV, about 92% of predicted points fell within an order of magnitude of spacecraft measurements and the onset peak within a factor of 2–3; for >100 MeV, 92% within an order of magnitude and about 49% within a factor of 2. The paper also compares three grid designs and reports that the default high-resolution grid is the most accurate
Load-bearing premise
The load-bearing premise is that the particle-injection scaling factor — a free multiplier on how many protons are seeded at the shock — can be set before an event and left fixed; in this exercise it was raised from the default of 1.0 to 10.0 for the 4 November 2001 event specifically to make the modeled flux match the spacecraft measurements, and the 5-hour runtime also excludes the precomputed background solar wind.
Editorial extensions
If this is right
- For the 4 November 2001 event, the fast grid setup delivered the full 96-hour profile in 4.86 hours of real time, yielding a leading time of about 91 hours over the event timeline.
- About 92% of the fast setup's >10 MeV predicted points fell within an order of magnitude of spacecraft measurements, with onset peak within a factor of 2–3; the default grid raised this to over 99% and Spearman correlation above 0.9, but took about 21 hours.
- The coarser-corona grid removes the main computational bottleneck, which is the solar-corona domain with its small cells and high wave speeds; after the CME leaves that domain, the simulation runs faster than real time.
- The paper recommends a two-phase operational workflow: run the fast setup for the first roughly 5 hours for order-of-magnitude guidance, then switch to the high-resolution setup for refined flux profiles.
- The paper concludes that a physics-based SEP model can meet the latency and robustness requirements of operational space-weather prediction for crewed exploration missions.
Reading between the lines
- The 5-hour figure excludes the precomputed ambient solar wind; a full event-to-answer latency would include the daily background-solar-wind preparation, though that pipeline runs continuously and can be restarted from saved states.
- Because the injection scaling factor was tuned to 10 for the 2001 event after the fact, the cleanest prospective test of forecast skill is to fix that factor at its default before an event and measure the resulting bias across many events.
- The same fast pipeline could be run as an ensemble (varying CME speed, active-region location, and mean free path) to produce probabilistic radiation-dose maps rather than single deterministic profiles — an extension the paper does not develop.
- The Setup-2 ESP-phase timing error (onset about 7 hours early, peak reduced severalfold) suggests numerical diffusion from the coarser grid; a grid-refinement policy between the fast and intermediate setups might recover ESP accuracy without the full cost of the high-resolution setup.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports the first simulated operational deployment of the SOFIE physics-based SEP prediction suite during the May 2025 SWPT exercise at NOAA/SWPC. The team simulated two historical SEP events (10 September 2017 and 4 November 2001) using the AWSoM-R, EEGGL, and M-FLAMPA components and compared the resulting >10 MeV and >100 MeV proton time-intensity profiles with GOES observations. The main quantitative claim is that with a coarsened solar-corona grid (Setup 2), a 4-day simulation of the 2001 event completed in 4.86 hr on 1,000 CPU cores, catching up with real time at 1.19 hr, demonstrating faster-than-real-time forecasting. Post-exercise runs (Setups 1 and 3) compare three grid configurations; Setup 1 gives the best accuracy but takes 21.12 hr. The paper also presents synthetic white-light CME images, solar-wind plasma comparisons, and forecast metrics (Spearman correlation, factor-of-2 and order-of-magnitude hit rates). The authors explicitly disclose that the background solar wind was prepared in advance and that the M-FLAMPA injection scaling factor was tuned for the 2001 event.
Significance. The runtime result is a genuinely useful operational benchmark: the dated, internally consistent numbers in Table 2 and Fig. 9 (4.86/21.12/18.57 hr for Setups 2/1/3; catch-up at 1.19/10.87/4.11 hr) are concrete and reproducible, with all plotted data archived on Zenodo and the SWMF/SOFIE code publicly available. If the speed result generalizes, it directly addresses a frequently cited bottleneck for physics-based SEP forecasting. The paper also gives a valuable three-way grid-resolution comparison and is candid about parameter tuning and the exclusion of background-preparation time. However, the operational-accuracy claim is only partially supported: the 2001 Setup-2 flux agreement relies on a seed-injection scale factor set to 10.0 to match GOES, and the abstract's 'without compromising accuracy' is contradicted by the paper's own Table 2 metrics. The 2017 event, run with the default scale factor of 1.0, is the only event demonstrating genuine forecast-mode flux skill.
major comments (3)
- [§5.3, §6.2, Table 1] The Setup-2 flux agreement for the 4 November 2001 event is partly by construction. The M-FLAMPA injection scaling factor (Table 1) was set to 10.0 for this event 'to better reproduce the GOES measurements' (§6.2), whereas the default is 1.0. Since the predicted SEP intensity is approximately proportional to this factor, the claimed factor-of-2–3 onset agreement (§5.3) reflects post-hoc calibration, not a pre-event forecast. A factor-1.0 run would reduce the quoted >10 MeV onset peak from ~2,126 pfu to ~213 pfu, about 13× below the observed 2,804 pfu. Please quantify this sensitivity and either adopt a pre-specified/default scale or clearly reframe the 2001 flux skill as a calibrated demonstration. This does not affect the runtime conclusion, but it is load-bearing for the operational-accuracy claim.
- [§6.1, Table 2, Abstract] The abstract states that the coarser background grid with higher-resolution regions reduces computational cost 'without compromising accuracy.' Table 2 does not support this for Setup 2 relative to Setup 1: for >10 MeV the Spearman correlation drops from 0.929 to 0.841 and the within-factor-of-2 rate from 41.7% to 37.7%; for >100 MeV the Spearman correlation drops from 0.918 to 0.563 and the ESP peak falls from 132 to 28 pfu against 162 pfu observed. Setup 2 preserves the onset and decay phases reasonably but clearly compromises ESP-phase fidelity. Please revise the claim to specify which accuracy aspects are retained, or provide a statistical equivalence test.
- [§7, §3, Abstract] The headline '5 hours' is not an end-to-end forecast time. As the paper explicitly states, 'for both events, we prepared the background solar wind in advance, and the corresponding timing is not included' (§7). The AWSoM-R steady-state background is a necessary component of SOFIE, and its computational cost is not part of the 4.86-hr total. The abstract's 'completed ... within 5 hours' should be qualified as the eruption-triggered run assuming a precomputed background. Please report the end-to-end operational timeline (magnetogram ingestion, background restart, CME-parameter availability, M-FLAMPA run) or state in the abstract that the quoted time excludes the background.
minor comments (4)
- [§4.2] In the text near Figure 3(e), 'observed by ACR' appears to be a typo for 'ACE'.
- [Table 2] The five statistical metrics (Spearman correlation, percentage within an order of magnitude, percentage within a factor of 2, median logarithmic error, median absolute logarithmic error) are presented without definitions. Since they carry the quantitative comparison, please define them in the text or cite a published standard with explicit formulas.
- [§4.3] The claim that the 2017 event's onset and peak fluxes agree 'within a factor of 2' is not supported by a table or listed numerical values. Please provide the same quantitative summary used for the 2001 event (e.g., peak values and leading times) for the 2017 event.
- [§6.1, Fig. 8] The three setups are described in the text and in Figs. 2/5, but a compact table summarizing the exact AMR criteria (e.g., angular resolution, cone half-widths, refinement levels) for Setups 1–3 would improve reproducibility.
Circularity Check
Partial circularity: the 4 Nov 2001 flux skill claim is inflated by setting the M-FLAMPA seed injection scaling factor to 10 to match GOES; the 10 Sept 2017 prediction and the speed claim remain independent.
-
fitted input called prediction
[Section 3 (M-FLAMPA free parameters); Section 5.3; Section 6.2; Table 1; Table 2]
"the seed particle scaling factor, which sets the injected particle population at the shock front with a default value of 1.0 and can be adjusted to better reproduce observed profiles ... For example, we adopted a scaling factor of 10 for the 4 November 2001 event in the SWPT exercise to better reproduce the GOES measurements."
The seed injection scaling factor is a free multiplier on the injected proton population, so predicted proton flux is approximately proportional to it. For the 4 Nov 2001 event the paper explicitly set this factor to 10.0 'to better reproduce the GOES measurements', then presents Setup 2 as reproducing the GOES onset peaks within a factor of 2-3 and reports Table 2 accuracy metrics (e.g., >10 MeV onset peak 2,126 pfu vs 2,804 pfu observed; 92.66% within an order of magnitude). That amplitude agreement is partly assured by fitting the target data: with a factor of 1.0 the modeled >10 MeV onset peak would be about ten times lower, erasing the claimed factor-of-2-3 agreement. The circularity is partial because the 10 Sept 2017 event used the default factor of 1.0, and the runtime/timeliness c
full rationale
The paper's central claim is twofold: SOFIE can run faster than real time, and it reproduces observed SEP fluxes. The speed claim is supported by direct wall-clock measurements (Table 2, Figure 9) and is not circular, though the stated timing excludes pre-computed AWSoM-R backgrounds, a caveat the authors disclose. The 10 September 2017 event used the default injection scaling factor of 1.0, so its factor-of-2 agreement is a genuine, parameter-free prediction. However, for the headline 4 November 2001 Setup-2 result, the M-FLAMPA seed injection scaling factor was deliberately set to 10.0 'to better reproduce the GOES measurements' (Section 6.2, Table 1). Because this factor scales the predicted SEP flux, the quoted Setup-2 amplitude agreement and Table 2 accuracy metrics for that event reduce in part to a fitted parameter rather than a pre-event prediction. This is a real but partial circularity: it affects the 2001 flux-skill demonstration, while the 2017 result and the operational timing milestone retain independent content. There is no load-bearing self-citation chain or imported uniqueness theorem; the model equations and comparisons to GOES, ACE, and LASCO are independently computed, and prior SOFIE papers are cited as model background, not as a substitute for the testbed validation.
Assumptions & free parameters
free parameters (3)
- M-FLAMPA seed-particle injection scaling factor =
1.0 (2017-09-10); 10.0 (2001-11-04)
- AWSoM-R Poynting flux parameter (background-solar-wind heating) =
0.55 MW m^-2 T^-1 (2017); 0.20 MW m^-2 T^-1 (2001)
- M-FLAMPA upstream mean free path lambda_0 =
0.1 au for both events
assumptions (6)
- domain assumption AWSoM-R stream-aligned MHD plus Alfven-wave turbulence reproduces the ambient corona/solar-wind connectivity needed for SEP transport
- domain assumption EEGGL Gibson-Low spheromak flux rope (parameterized by AR location + measured CME speed) represents the real CME sufficiently for SEP production
- domain assumption M-FLAMPA focused-transport/field-line-advection treatment with isotropic pitch-angle scattering (lambda_0 = 0.1 au) captures shock acceleration and interplanetary proton transport
- domain assumption The seed-particle source is a uniform suprathermal tail at the shock front scaled by a constant factor
- ad hoc to paper Preceding CMEs/ICMEs before the 4 Nov 2001 event can be omitted from the background without materially biasing the SEP comparison
- domain assumption GOES >10 and >100 MeV proton channels, ACE in-situ plasma, GONG/MDI magnetograms, and SWPC-CAT CME speeds are reliable as used
Cite this review
Pith. "Pith review of Simulated Operational Testing of the Prototype Implementation of the SOFIE Model: The 2025 Space Weather Prediction Testbed Exercise." pith.science (2026). https://pith.science/paper/JNMQN4PS
@misc{pith2026251109716,
author = {Pith},
title = {Pith review of: Simulated Operational Testing of the Prototype Implementation of the SOFIE Model: The 2025 Space Weather Prediction Testbed Exercise},
year = {2026},
howpublished = {\url{https://pith.science/paper/JNMQN4PS}},
note = {Machine review of arXiv:2511.09716}
}
read the original abstract
The CLEAR Space Weather Center of Excellence's solar energetic particle (SEP) model, SOlar wind with FIeld lines and Energetic particles (SOFIE), was run and evaluated on-site during the Space Weather Prediction Testbed (SWPT) exercise at the National Oceanic and Atmospheric Administration's Space Weather Prediction Center (NOAA/SWPC) in May 2025. As a physics-based SEP model, SOFIE simulates the acceleration and transport of energetic particles by the coronal mass ejection (CME)-driven shock in the solar corona and inner heliosphere, and has been validated against historical events. However, questions remain regarding whether a physics-based model, traditionally considered computationally expensive, could meet operational needs. The SWPT exercise offered a valuable opportunity to evaluate SOFIE under simulated operational conditions. On-site interactive feedback from SWPC forecasters, Space Radiation Analysis Group (SRAG) console operators, Community Coordinated Modeling Center (CCMC) personnel, and Moon-to-Mars Space Weather Analysis Office (M2M SWAO) analysts led to significant strategic improvements in the model configuration. The simulation grid was optimized by combining a coarser background grid with higher-resolution regions along the CME path and toward Earth, reducing computational cost without compromising accuracy. In this work, we present the simulated operational performance of SOFIE and its capability to predict SEP fluxes significantly faster than real time. During the SWPT exercise, SOFIE completed a 4-day SEP simulation within 5 hours using 1,000 central processing unit cores, although the earliest SEP forecast was obtained a few hours after CME onset. This marks a milestone in demonstrating SOFIE's operational usefulness and robustness to support future human space exploration.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Afanasiev, A., Battarbee, M., & Vainio, R. (2015). Self-consistent Monte Carlo simulations of proton acceleration in coronal shocks: Effect of anisotropic pitch-angle scattering of particles.Astronomy & Astrophysics,584, A81. doi: 10.1051/0004-6361/201526750 Anastasiadis, A., Lario, D., Papaioannou, A., Kouloumvakos, A., & Vourlidas, A. (2019). Solar ener...
arXiv 2015
-
[3]
doi: 10.1007/s41116-016-0002-5 Georgoulis, M. K., Yardley, S. L., Guerra, J. A., Murray, S. A., Ahmadzadeh, A., Anastasiadis, A., . . . others (2024). Prediction of solar energetic events impacting space weather conditions.Advances in Space Research. doi: 10.1016/j.asr.2024.02.030 Gibson, S. E., & Low, B. (1998). A Time-dependent Three-dimensional Magneto...
-
[8]
doi: 10.3847/1538-4357/ad32ca –31– manuscript submitted toSpace Weather Illing, R., & Hundhausen, A. (1985). Observation of a coronal transient from 1.2 to 6 solar radii.Journal of Geophysical Research: Space Physics,90(A1), 275–
-
[14]
V., van der Holst, B., Oran, R., Downs, C., Roussev, I
doi: 10.3847/1538-4357/abc000 Sokolov, I. V., van der Holst, B., Oran, R., Downs, C., Roussev, I. I., Jin, M., . . . Gombosi, T. I. (2013). Magnetohydrodynamic Waves and Coronal Heating: Unifying Empirical and MHD Turbulence Models.The Astrophysical Journal, 764(1),
-
[23]
doi: 10.1088/0004-637X/764/1/23 Sokolov, I. V., Zhao, L., & Gombosi, T. I. (2022). Stream-aligned magnetohydrody- namics for solar wind simulations.The Astrophysical Journal,926(1),
-
[40]
I., Kordylewski, Z., Kudela, K., Kurt, V.,
doi: 10.3390/cells12010040 Kuznetsov, S., Bogomolov, A., Denisov, Y. I., Kordylewski, Z., Kudela, K., Kurt, V., . . . others (2003). The solar flare of November 4, 2001, and its manifestations in energetic particles from Coronas-F data.Solar System Research,37(2), 121–127. doi: 10.1023/A:1023384425209 Lee, D.-T., & Schachter, B. J. (1980). Two algorithms ...
-
[42]
doi: 10.1051/swsc/2021020 Gombosi, T. I., De Zeeuw, D. L., Powell, K. G., Ridley, A. J., Sokolov, I. V., Stout, Q. F., & T´ oth, G. (2003). Adaptive mesh refinement for global mag- netohydrodynamic simulation.Space Plasma Simulation, 247–274. doi: 10.1007/3-540-36530-3 12 Guo, F., & Giacalone, J. (2013). The acceleration of thermal protons at parallel col...
arXiv 2003
-
[81]
doi: 10.1088/0004-637X/782/2/81 Whitman, K., Egeland, R., Allison, C., Quinn, P., & Stegeman, L. (2025 (in prepa- ration)). Validation of Solar Energetic Particle Forecasting Models for Space Radiation Operations with SPHINX and VIVID.NASA Technical Paper. Wraback, E., Hoffmann, A., Manchester, W., Sokolov, I., van der Holst, B., & Carpenter, D. (2024). S...
Show all 18 references
-
[82]
V., Whitman, K., Gombosi, T
doi: 10.3847/1538-4357/adc4e3 Liu, W., Zhao, L., Sokolov, I. V., Whitman, K., Gombosi, T. I., Sachdeva, N., . . . Tegnell, K. L. (2025, November).Data based on the Study of Sim- ulated Real-Time Testing of the Prototype Implementation of the SOFIE Model: The 2025 Space Weather...
2025 doi
-
[88]
V., Manchester, W., Jin, M., & Gombosi, T
doi: 10.3847/1538-4357/aad68d Borovikov, D., Sokolov, I. V., Manchester, W., Jin, M., & Gombosi, T. I. (2017). Eruptive event generator based on the Gibson-Low magnetic configuration. Journal of Geophysical Research: Space Physics,122(8), 7979–7984. doi: 10.1002/2017JA024304 –...
2017
-
[91]
doi: 10.1088/0004-637X/783/2/91 Chen, H., Sachdeva, N., Huang, Z., van der Holst, B., Manchester IV, W., Ji- vani, A., . . . Toth, G. (2025). Decent Estimate of CME Arrival Time From a Data-Assimilated Ensemble in the Alfv´ en Wave Solar Atmosphere Model (DECADE-A WSoM).Space ...
2025 doi
-
[102]
T., Minyard, T., Ghattas, O., & Panda, D
doi: 10.3847/1538-4357/ac400f Stanzione, D., West, J., Evans, R. T., Minyard, T., Ghattas, O., & Panda, D. K. (2020). Frontera: The Evolution of Leadership Computing at the National Science Foundation. InPractice and experience in advanced research computing 2020: Catch the wa...
2020
-
[158]
F., Temmer, M., Lohf, H., Wang, Y.,
doi: 10.1088/0004-637X/773/2/158 Guo, J., Dumbovi´ c, M., Wimmer-Schweingruber, R. F., Temmer, M., Lohf, H., Wang, Y., . . . others (2018). Modeling the evolution and propagation of 10 September 2017 CMEs and SEPs arriving at Mars constrained by remote sensing and in situ meas...
2018
-
[172]
others (2023)
doi: 10.3847/1538-4357/834/2/172 Jivani, A., Sachdeva, N., Huang, Z., Chen, Y., van der Holst, B., Manchester, W., . . . others (2023). Global sensitivity analysis and uncertainty quantification for background solar wind using the alfv´ en wave solar atmosphere model.Space Wea...
2023 doi
-
[173]
Gombosi, T
doi: 10.3847/1538-4357/834/2/173 Jin, M., Manchester, W., van der Holst, B., Sokolov, I., T´ oth, G., Vourlidas, A., . . . Gombosi, T. (2017). Chromosphere to 1 AU simulation of the 2011 March 7th event: a comprehensive study of coronal mass ejection propagation.The Astrophysi...
2017 doi
-
[182]
doi: 10.3847/1538-4357/ad21fd Zank, G., Rice, W., & Wu, C. (2000). Particle acceleration and coronal mass ejec- tion driven shocks: A theoretical model.Journal of Geophysical Research: Space Physics,105(A11), 25079–25095. doi: 10.1029/1999JA000455 Zhao, L., Sokolov, I., Gombos...
2000 doi
-
[282]
Gombosi, T
doi: 10.1029/JA090iA01p00275 Jin, M., Manchester, W., van der Holst, B., Sokolov, I., T´ oth, G., Mullinix, R., . . . Gombosi, T. (2017). Data-constrained coronal mass ejections in a global magnetohydrodynamics model.The Astrophysical Journal,834(2),
2017 doi
-
[460]
I., Chen, Y., Glocer, A., Huang, Z., Jia, X., Liemohn, M
doi: 10.1086/305107 Gombosi, T. I., Chen, Y., Glocer, A., Huang, Z., Jia, X., Liemohn, M. W., . . . oth- –30– manuscript submitted toSpace Weather ers (2021). What Sustained Multi-disciplinary Research can Achieve: The Space Weather Modeling Framework.Journal of Space Weather ...
2021 doi
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.