REVIEW 4 major objections 5 minor 33 references
Three mature astrophysics codes, after targeted GPU porting, sustain 78–97% parallel efficiency at up to 2,048 GPUs on a pre-exascale system.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 07:50 UTC pith:34MULFBU
load-bearing objection Useful, mostly honest performance report on three astro codes for Leonardo, but the abstract overstates the 80%-at-1,024-GPU claim. the 4 major comments →
Towards Exascale Computing for Astrophysical Simulation Leveraging the Leonardo EuroHPC System
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms: preliminary tests show all three codes scale efficiently, reaching 80% scalability up to 1,024 GPUs. Specifically, gPLUTO shows 88% weak-scaling efficiency on the Orszag–Tang vortex and 97% on a circularly polarized Alfvén wave up to 2,048 GPUs, with strong scaling at 89–95%; iPIC3D reaches 78% weak-scaling efficiency up to 1,024 GPUs with 40× and 100× module speedups; OpenGadget3 stays above 80% strong-scaling efficiency for a 1024³ cosmology run up to a 32× resource increase, though whole-run speedup over CPU is redshift-dependent, about 3× at z=0. The authors interpret this as showing that their selective GPU-offloading approach works across three numerically dis
What carries the argument
The central mechanism is selective kernel offloading under a hybrid CPU-GPU execution model, informed by profiling. For the finite-volume MHD code, an accelerator-programming-model port with a templated array layer, managed memory, and non-blocking MPI ghost-cell exchange delivers coalesced GPU access and good scaling. For the particle-in-cell code, the compute-bound particle mover and moment gatherer are offloaded to GPUs while the matrix-free field solver stays on CPU, with asynchronous communication. For the N-body tree code, the load-bearing idea under development is regrouping particles by spatial locality within small Hilbert curve segments and using direct summation in a fixed radius,
Load-bearing premise
The load-bearing assumption is that the scaling numbers, gathered from short runs on carefully sized test problems, are representative of full production simulations with realistic physics and longer runtimes.
What would settle it
Run OpenGadget3 with full baryonic physics to redshift z=0 at 1,024–2,048 GPUs and measure the strong-scaling efficiency across the whole run; if the efficiency averaged over the run falls below 80%, the paper's over-80% claim for that code on production workloads is refuted. The paper itself flags the redshift dependence, so a time-resolved efficiency profile would settle whether the current optimistic figures survive late-time clustering.
If this is right
- If the reported efficiencies carry to production, the three physics communities can plan million-GPU-hour campaigns at resolutions previously inaccessible, without porting their science modules from scratch.
- The hybrid pattern—keep communication-heavy or memory-bound solvers on CPU while offloading compute-bound particle/mesh kernels to GPU—gives other legacy simulation codes a tested template for exascale adaptation.
- For gPLUTO, the boundary-exchange and ghost-cell design emerges as the scaling-critical element; any structured-grid MHD code sharing that architecture should copy the non-blocking multi-GPU exchange strategy.
- For OpenGadget3, the clear message is that the Barnes–Hut tree force, not hydrodynamics, is the barrier; the reported ~3× whole-run speedup and redshift dependence quantify the prize for completing the tree-traversal rework.
Where Pith is reading between the lines
- Because OpenGadget3's efficiency is measured at z~50 where matter is smooth and gravity ~10× faster, the quoted >80% scaling probably overstates the production low-redshift regime; a fair efficiency would be a time-weighted average over all redshifts in a run.
- iPIC3D's CPU-resident field solver plus per-step CPU-GPU data exchange implies a hidden serial bottleneck; one testable prediction is that at larger node counts the Field Solver's MPI latency will flatten the scaling curve, and overlapping its communication with the GPU kernels would restore scaling.
- For gPLUTO, the 88–97% figures on two smooth benchmark flows likely bound a harder case: turbulent flows with more boundary traffic could push efficiency down; a benchmark with strong small-scale structure would stress the ghost-cell exchange more realistically.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports work from the SPACE Center of Excellence on porting and scaling three astrophysical codes—gPLUTO, OpenGadget3, and iPIC3D—on the Leonardo EuroHPC Booster and DCGP partitions. The authors describe code development workflows, profiling with Extrae and Nsight, GPU offloading strategies, and CI/CD benchmarking with ReFrame. They present weak- and strong-scaling results for each code, with gPLUTO showing 88–97% weak scaling up to 2,048 GPUs, OpenGadget3 showing >80% strong-scaling efficiency for the 1024^3 case up to a 32× resource increase, and iPIC3D showing 78% weak scaling at 1,024 GPUs with large module-level GPU speedups. The abstract and summary claim that all three codes reach 80% scalability up to 1,024 GPUs.
Significance. If the performance numbers are correct, the paper provides valuable early evidence that three production astrophysics codes can be used efficiently on Leonardo's Booster partition at moderate to large scale, and it documents a reproducible methodology (ReFrame, CI/CD, profiling with standard tools). The tables are internally consistent, the module-level speedups for iPIC3D are concrete, and gPLUTO's weak-scaling results in particular appear to support efficient use at 2,048 GPUs. However, the headline claim about all three codes reaching 80% efficiency at 1,024 GPUs is not supported by the reported measurements as they stand: iPIC3D is explicitly at 78%, and the OpenGadget3 result is obtained in a high-redshift regime that the authors themselves state is unrepresentative of production runs. The significance of the paper would be materially improved by correcting these overstatements.
major comments (4)
- [Abstract and §4.4, §5] The abstract states that 'all three codes scale efficiently, reaching 80% scalability up to 1,024 GPUs.' This is contradicted by §4.4, where iPIC3D weak scaling is reported as 78% at 1,024 GPUs, and by §5, which says '78% weak scaling efficiency.' Either the threshold should be lowered to 'about 80%' or the iPIC3D claim should be qualified. As written, the central abstract claim is numerically overstated.
- [§4.3] The OpenGadget3 strong-scaling tests are run at z≈50. The manuscript itself states that at lower redshifts the Barnes–Hut tree becomes deeper and more costly, that gravitational force calculations are ~10× faster at high redshift, and that 'Most computational time occurs at a≳0.2, where efficiency drops.' Thus the observed >80% scaling efficiency in this high-redshift test does not provide evidence that the code will sustain 80% efficiency in a production cosmological run, where the workload is concentrated in the low-redshift regime. The claim should be restricted to the measured regime, or a lower-redshift scaling test should be added.
- [§4.3, Fig. 4] For the 1024^3 case, the text says 'maintaining over 80% efficiency up to a 32× resource increase' but does not state the baseline. From the figure axis, the smallest scale appears to be 16 GPUs, which would mean the claim extends only to 512 GPUs (128 nodes), not 1,024 GPUs. If the baseline is 32 GPUs, the claim reaches 1,024 GPUs. Please state the base node/GPU count explicitly. Without this, the abstract's '1,024 GPUs' for OpenGadget3 is unsupported by the reported number.
- [§2.1 and §4.2] The gPLUTO results are obtained with only ~65% of PLUTO modules implemented; AMR, cooling/heating, and dissipation are explicitly missing. While this is acknowledged, the summary's phrase 'closest to exascale readiness' should be tempered or explicitly tied to the current module coverage. The scaling results may not carry over once the missing physics modules (e.g., AMR) are added, because those modules will introduce new communication and load-balance patterns.
minor comments (5)
- [§2.5] The CPU profiling is limited to 'several units or tens of iterations' and for iPIC3D to four cycles. This is a reasonable practical compromise, but the paper should state more prominently that the profiling and scaling measurements may not capture startup, load-imbalance, or preemption effects that appear in full production runs.
- [§4.4, Table 3 caption] The caption states '20×20×20 particles per cell' while the text says '20×20×20 particles per cell'; the table caption in the PDF shows a slightly different format. Please ensure the particle-per-cell values are consistent between text, table, and figure captions.
- [§4.2, Table 2] The table groups are described as 'minimum obtained performance' but the meaning of the group columns (Base Res, OT, CPA) could be clearer. Consider adding a sentence explaining that each group is a strong-scaling campaign with fixed resolution and doubling nodes, and that the efficiencies are relative to the group's own base resolution.
- [Fig. 4] The figure shows speedup for OpenGadget3 but the legend uses 'Gravity only' and 'Gravity + Hydro' without marking which line corresponds to which particle number. Clarify the legend or describe it in the caption.
- [References] Reference [12] is a PhD thesis and several code implementations are cited as 'in prep.' or 'submitted'. For a reproducibility-focused paper, the authors should provide persistent DOIs or arXiv identifiers for the code versions used in the benchmarks where available.
Circularity Check
No circular derivation: scaling and speedup results are direct hardware measurements; self-citations are provenance, not load-bearing.
full rationale
The paper's central claims are empirical performance results obtained with standard profiling and benchmarking tools (Extrae, NVIDIA Nsight, ReFrame) on Leonardo. There is no fitted parameter that is later renamed as a prediction, no equation whose output is identical by construction to its input, and no derivation chain whose conclusion is presupposed by its setup. The scaling efficiencies for gPLUTO, OpenGadget3, and iPIC3D are direct wall-clock measurements. The cited prior work, including [12] for the gPLUTO port details, [21] for OpenACC-based OpenGadget3 offloading, and [11] for the SPACE-CoE project itself, describes implementation provenance and code background rather than supplying the scaling numbers reported here. Even where the authors benchmark their own codes, the measured values are externally reproducible hardware data, not consequences of a self-imposed ansatz or self-citation. Some explicitly disclosed limitations reduce confidence in the headline if read as production-scale readiness: §2.5 profiles only small iteration windows, §4.3 admits that OpenGadget3's scaling is measured at z≈50 and that efficiency drops at a≳0.2 where most production time is spent, and §4.4 reports iPIC3D at 78% weak scaling on 1024 GPUs while OpenGadget3's over-80% efficiency is documented up to a 32× resource increase. These are correctness/representativeness concerns, not circularity. The paper contains no equation-level self-reference and no prediction that reduces to its inputs by construction; the reported measurements stand as independent evidence. Score 1 reflects the presence of minor, non-load-bearing self-citations in the provenance narrative without any circular dependence in the central claims.
Axiom & Free-Parameter Ledger
free parameters (3)
- gPLUTO per-node weak-scaling resolution =
704×704×352 cells
- OpenGadget3 scaling-test redshift =
z ≈ 50
- iPIC3D benchmark problem and RoI =
Maxwellian 2D test; 180×180 particles/cell/species; second cycle as RoI
axioms (4)
- domain assumption POP3 hierarchical efficiency model (SerE/OCE/TE) correctly attributes observed scaling losses to specific routines
- domain assumption Runs limited to a few time steps (RoIs) are representative of steady-state production performance
- domain assumption A fixed per-node problem size that 'fully utilizes' one node is the correct weak-scaling unit, and 1 MPI rank per GPU with 8 OpenMP threads is the right deployment
- domain assumption The profiling tool stack (Extrae, Nsight, ReFrame) measures what it claims on this hardware/software stack
Cite this review
Pith. "Pith review of Towards Exascale Computing for Astrophysical Simulation Leveraging the Leonardo EuroHPC System." pith.science (2026). https://pith.science/paper/34MULFBU
@misc{pith2026251024175,
author = {Pith},
title = {Pith review of: Towards Exascale Computing for Astrophysical Simulation Leveraging the Leonardo EuroHPC System},
year = {2026},
howpublished = {\url{https://pith.science/paper/34MULFBU}},
note = {Machine review of arXiv:2510.24175}
}
read the original abstract
Developing and redesigning astrophysical, cosmological, and space plasma numerical codes for existing and next-generation accelerators is critical for enabling large-scale simulations. To address these challenges, the SPACE Center of Excellence (SPACE-CoE) fosters collaboration between scientists, code developers, and high-performance computing experts to optimize applications for the exascale era. This paper presents our strategy and initial results on the Leonardo system at CINECA for three flagship codes, namely gPLUTO, OpenGadget3 and iPIC3D, using profiling tools to analyze performance on single and multiple nodes. Preliminary tests show all three codes scale efficiently, reaching 80% scalability up to 1,024 GPUs.
Figures
Reference graph
Works this paper leans on
-
[1]
Sarkar, Is dark matter self-interacting?, Nat Astron 2 (2018) 856–857.doi:10.1038/s41550-018-0598-6
S. Sarkar, Is dark matter self-interacting?, Nat Astron 2 (2018) 856–857.doi:10.1038/s41550-018-0598-6
-
[2]
K. Schoeffler, et al, Can plasma physics establish a significant bound on long-range dark matter interactions?, Phys. Rev. D 111 (2025) L071701.doi:10.1103/PhysRevD.111.L071701
-
[3]
Romeo, et al, Simulations for 21 cm radiation lensing at eor redshifts, Mon
A. Romeo, et al, Simulations for 21 cm radiation lensing at eor redshifts, Mon. Not. R. Astron. Soc. 474 (2) (2017) 1787–1809.doi: 10.1093/mnras/stx2733
-
[4]
Croft, et al, Weak lensing of the lymanαforest, Mon
R. Croft, et al, Weak lensing of the lymanαforest, Mon. Not. R. Astron. Soc. 477 (2) (2018) 1814–1821
2018
-
[5]
Markidis, et al, Collisionless magnetic reconnection in a plasmoid chain, Nonlin
S. Markidis, et al, Collisionless magnetic reconnection in a plasmoid chain, Nonlin. Processes Geophys. 19 (2012) 145–153.doi:10.5194/ npg-19-145-2012
2012
-
[6]
Mattia, et al, Resistive relativistic mhd simulations of astrophysical jets, Astronomy & Astrophysics 679 (2023) A49.doi:10.1051/ 0004-6361/202347126
G. Mattia, et al, Resistive relativistic mhd simulations of astrophysical jets, Astronomy & Astrophysics 679 (2023) A49.doi:10.1051/ 0004-6361/202347126
2023
-
[7]
S. Ferro, et al, Comparative simulations of kelvin–helmholtz induced magnetic reconnection at the earth’s magnetospheric flanks, Phys. Plas- mas 31 (5) (2024) 052902.doi:10.1063/5.0191674
-
[8]
S. Borgani, et al, X-ray properties of galaxy clusters and groups from a cosmological hydrodynamical simulation, Mon. Not. R. Astron. Soc. 348 (3) (2004) 1078–1096.doi:10.1111/j.1365-2966.2004.07431.x
arXiv 2004
-
[9]
N. Shukla, et al, Conditions for the onset of the current filamentation instability in the laboratory, J. Plasma Phys. 84 (3) (2018) 905840302. doi:10.1017/S0022377818000314
-
[10]
E. Boella, et al, Interaction between electrostatic collisionless shocks generates strong magnetic fields, New J. Phys. 24 (2022) 063016. doi:10.1088/1367-2630/ac6ef1
-
[11]
N. Shukla, et al, Eurohpc space coe: Redesigning scalable parallel astrophysical codes for exascale, in: 22nd ACM Int. Conf. Comput. Front, ACM, New York, NY , USA, Cagliari, Italy, 2025, p. 7.doi:10.1145/3706594.3728892
arXiv 2025
-
[12]
Rossaza, Gpu porting of the pluto code for computational plasma physics using openacc, Ph.D
M. Rossaza, Gpu porting of the pluto code for computational plasma physics using openacc, Ph.D. thesis, University of Turin (2025). URLhttps://iris.unito.it/handle/2318/2065332
2025
-
[13]
F. Groth, et al, The cosmological simulation code OPENGADGET3 - implementation of meshless finite mass, Mon. Not. R. Astron. Soc. 526 (1) (2023) 616–644.arXiv:2301.03612,doi:10.1093/mnras/stad2717
Pith/arXiv arXiv 2023
-
[14]
Markidis, et al, Multi-scale simulations of plasma with iPIC3D, Math
S. Markidis, et al, Multi-scale simulations of plasma with iPIC3D, Math. Comput. Simul. 80 (7) (2010) 1509–1519.doi:10.1016/j. matcom.2009.08.038
doi:10.1016/j 2010
-
[15]
POP3: Performance Optimization and Productivity Centre of Excellence,https://www.pop-coe.eu/, european Centre of Excellence on HPC performance optimization, project duration 2024–2027, coordinated by Barcelona Supercomputing Center (2024)
2024
-
[16]
V . Springel, The cosmological simulation code GADGET-2, Monthly Notices of the Royal Astronomical Society 364 (4) (2005) 1105–1134. arXiv:astro-ph/0505010,doi:10.1111/j.1365-2966.2005.09655.x
Pith/arXiv arXiv 2005
-
[17]
J. Barnes, P. Hut, A hierarchical O(N log N) force-calculation algorithm, Nature 324 (6096) (1986) 446–449.doi:10.1038/324446a0
doi:10.1038/324446a0 1986
-
[18]
Program Generation, Optimization, and Platform Adaptation
M. Frigo, S. G. Johnson, The design and implementation of FFTW3, Proceedings of the IEEE 93 (2) (2005) 216–231, special issue on “Program Generation, Optimization, and Platform Adaptation”
2005
-
[19]
A. M. Beck, et al, An improved SPH scheme for cosmological simulations, Mon. Not. R. Astron. Soc. 455 (2) (2016) 2110–2130.arXiv: 1502.07358,doi:10.1093/mnras/stv2443
Pith/arXiv arXiv 2016
-
[20]
A. Ragagnin, N. Tchipev, M. Bader, K. Dolag, N. J. Hammer, Exploiting the Space Filling Curve Ordering of Particles in the Neighbour Search of Gadget3, in: Advances in Parallel Computing, 2016, pp. 411–420.doi:10.3233/978-1-61499-621-7-411
-
[21]
A. Ragagnin, K. Dolag, M. Wagner, C. Gheller, C. Roffler, D. Goz, D. Hubber, A. Arth, Gadget3 on GPUs with OpenACC, Parallel Computing: Technology Trends 27 (2020) 209–218.doi:10.3233/APC200043
-
[22]
Turisini, et al, Leonardo: A pan-european pre-exascale supercomputer for hpc and ai applications, J
M. Turisini, et al, Leonardo: A pan-european pre-exascale supercomputer for hpc and ai applications, J. Large-Scale Res. Facilities 9 (01 2024). doi:10.17815/jlsrf-8-186
-
[23]
T. Gamblin, et al, The spack package manager: Bringing order to hpc software chaos, in: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15), ACM, 2015, pp. 1–12.doi:10.1145/2807591.2807623
arXiv 2015
-
[24]
Barcelona Supercomputing Center, Extrae: The Extrae Tracing Tool, Barcelona Supercomputing Center, version 4.0, Available athttps: //www.ebrains.eu/tools/extrae(2025)
2025
-
[25]
Tornatore, et al., Code Modules and Kernels, SPACE-COE project document (2024)
L. Tornatore, et al., Code Modules and Kernels, SPACE-COE project document (2024). 12Shukla N., Romeo A., et al./Procedia Computer Science 00 (2025) 000–000
2024
-
[26]
Burtscher, K
M. Burtscher, K. Pingali, Chapter 6 - an efficient cuda implementation of the tree-based barnes hut n-body algorithm, in: GPU Computing Gems Emerald Edition, Applications of GPU Computing Series, Morgan Kaufmann, Boston, 2011, pp. 75–92.doi:https://doi.org/10. 1016/B978-0-12-384988-5.00006-1
2011
-
[27]
Commercon, et al., Code release (alpha), SPACE-COE project document (2024)
B. Commercon, et al., Code release (alpha), SPACE-COE project document (2024)
2024
-
[28]
S. Williams, et al, Roofline: An insightful visual performance model for multicore architectures, Commun. ACM 52 (4) (2009) 65–76.doi: 10.1145/1498765.1498785
arXiv 2009
-
[29]
Karakasis, et al, Enabling continuous testing of hpc systems using reframe, in: Tools and Techniques for High Performance Computing, Springer, Cham, 2020, pp
V . Karakasis, et al, Enabling continuous testing of hpc systems using reframe, in: Tools and Techniques for High Performance Computing, Springer, Cham, 2020, pp. 49–68
2020
-
[30]
Evans, et al, Simulation of Magnetohydrodynamic Flows: A Constrained Transport Model, Astrophys
C. Evans, et al, Simulation of Magnetohydrodynamic Flows: A Constrained Transport Model, Astrophys. J. 332 (1988) 659.doi:10.1086/ 166684
1988
-
[31]
P. Londrillo, L. D. Zanna, On the divergence-free condition in godunov-type schemes for ideal magnetohydrodynamics: the upwind constrained transport method, J. Comput. Phys. 195 (1) (2004) 17–48.doi:10.1016/j.jcp.2003.09.016
-
[32]
Dedner, et al, Hyperbolic Divergence Cleaning for the MHD Equations, J
A. Dedner, et al, Hyperbolic Divergence Cleaning for the MHD Equations, J. Comput. Phys. 175 (2002) 645–673.doi:10.1006/jcph. 2001.6961
arXiv 2002
-
[33]
A. Mignone, P. Tzeferacos, A second-order unsplit Godunov scheme for cell-centered MHD: The CTU-GLM scheme, J. Comput. Phys. 229 (2010) 2117–2138.arXiv:0911.3410,doi:10.1016/j.jcp.2009.11.026
Pith/arXiv arXiv 2010
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.