Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

RAMSES-yOMP: Performance Optimizations for the Astrophysical Hydrodynamic Simulation Code RAMSES

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding OpenMP threads to RAMSES delivers a factor-2 speedup at 1,536 cores.

desk verdict Useful, honest performance engineering with a public RAMSES fork; the speedup is credible, but the full-baryonic consistency claim is not yet supported because the test turns star formation off. read the letter →

arxiv 2411.14631 v1 pith:EMU5Y7JB submitted 2024-11-21 astro-ph.IM astro-ph.GA

San Han , Yohan Dubois , Jaehyun Lee , Juhan Kim , Corentin Cadiou , Sukyoung K. Yi This is my paper · ORCID
classification astro-ph.IMastro-ph.GA
keywords RAMSEShybridMPI/OpenMPadaptivemeshrefinementhydrodynamicsimulationparallelperformanceloadbalancingPoissonsolvermemoryefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the dominant bottleneck in large RAMSES simulations is not raw arithmetic but the cost of decomposing the volume into as many MPI domains as cores: load imbalance, redundant boundary data, and per-domain memory spikes all grow with domain count. RAMSES-yOMP therefore runs several OpenMP threads per MPI domain, so the same total cores are organized into fewer, larger domains. In a 1536-core benchmark with full baryonic physics, the modified code runs in about 53% of the original time, uses as little as 23% of the original memory, and writes about 70% of the original snapshot data. A consistency test on a zoom-in galaxy simulation shows the hybrid run matches the MPI-only run to round-off-level deviations once stochastic star formation is turned off. The authors conclude that these gains let existing facilities run larger or higher-resolution simulations than the MPI-only design would allow.

What carries the argument

The central object is the hybrid thread-team structure, defined by $N_{\mathrm{core}} = N_{\mathrm{MPI}} \times N_{\mathrm{OMP}}$: total cores are fixed while the balance between distributed-memory domains and shared-memory threads is swept. On top of that, two algorithmic changes do part of the work: a preconditioned pipelined conjugate gradient Poisson solver (PPCG) that halves the number of global-synchronization iterations, and a modified load-balancing cost function $C=\sum_\ell w(\ell)(k N_{\mathrm{grid}}(\ell)+N_{\mathrm{part}}(\ell))$ with $k$ reduced from 10 to 2.5, giving particles four times more relative weight in domain decomposition. OMP is applied fork-join style to the time-intensive grid and particle loops with static or dynamic scheduling depending on loop cost uniformity; the race-prone cloud-in-cell density estimation and mechanical supernova feedback are deliberately left single-threaded.

What would settle it

Run the same zoom-in galaxy test with star formation enabled and compare the hybrid run against two MPI-only runs with different vectorization sizes $N_{\mathrm{vec}}$; if the hybrid/MPI-only difference exceeds the $N_{\mathrm{vec}}$-induced spread and grows with thread count in a way that round-off accumulation cannot explain, the 'reasonably reproduces' claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the efficiency ceiling of RAMSES at large core counts is dominated by domain-decomposition side effects, not by the hydrodynamics itself. By keeping the same total number of cores but grouping them as $N_{\mathrm{MPI}}\times N_{\mathrm{OMP}}$ — a small number of MPI domains each running several OpenMP threads — the modified code reduces inter-domain boundary data, eases load imbalance, and lets memory be preallocated against a far smaller occupancy spread. In the 1536-core benchmark, execution time drops to $\sim$53% of the original (best at $N_{\mathrm{OMP}}=16$), memory to $\sim$23% at $N_{\mathrm{OMP}}=64$, and snapshot disk use to $\sim$70%; the gains come mostly from the particle and gravity categories, with hydro unchanged. The authors attribute the particle improvement to a retuned load-balancing weight ($k$: 10 to 2.5) and the gravity improvement to replacing conjugate gradient with a preconditioned pipelined CG solver that reaches $10^{-4}$ relative error in about half the iterations. A one-galaxy zoom-in test with star formation disabled shows hybrid and MPI-only runs agree within the scatter expected from round-off error.

Load-bearing premise

The load-bearing premise is that the consistency test with star formation turned off is enough to prove the hybrid code matches the MPI-only physics; if a hidden data race in the particle or feedback routines changes results by more than round-off error, the fidelity claim fails.

Editorial extensions

If this is right

  • At a fixed $N_{\mathrm{core}}=1536$, $N_{\mathrm{OMP}}=16$ gives the best wall-clock time, cutting total runtime to $\sim$53% of the original on Knights Landing and $\sim$45% on Ice Lake.
  • Memory use falls to $\sim$23--30% of the original and disk use to $\sim$70--75% at the same core count, because fewer MPI domains reduce domain-boundary redundancy and raise the particle occupation fraction.
  • Replacing CG with PPCG roughly halves the number of Poisson-solver iterations needed to reach $\varepsilon=10^{-4}$, removing a large share of global MPI communication.
  • The retuned load-balancing weight ($k=2.5$ instead of 10) and the smaller number of MPI domains make the particle and load-balancing categories the largest sources of saved time.
  • Because the thread count can be changed during a run, the code can adapt to resource availability in ways the MPI-only version cannot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit how the sweet spot scales: a weak-scaling study varying $N_{\mathrm{core}}$ from 1,536 up to tens of thousands would show whether $N_{\mathrm{OMP}}=16$ remains optimal or whether the optimum shifts with problem size and machine architecture.
  • Because cloud-in-cell density estimation and mechanical supernova feedback stay single-threaded, full-physics runs could in principle diverge from MPI-only runs by more than the round-off scatter shown in Appendix B; the consistency test with star formation off does not exercise those paths.
  • The calibrated value $k=2.5$ is specific to the NewHorizon-style baryonic prescription; simulations that add more chemical species, dust, or radiation would likely need to re-tune this weight, following the paper's category-timing procedure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents RAMSES-yOMP, a modified version of the AMR code RAMSES that adds OpenMP thread parallelization on top of the existing MPI domain decomposition, replaces the conjugate-gradient Poisson solver with a preconditioned pipelined CG (PPCG) solver, and changes the load-balance weight k for particles from 10 to 2.5. The authors benchmark the code on 1536 cores (Knights Landing and Ice Lake systems) against the original MPI-only code, reporting a wall-clock reduction to ~53% of the original at the best configuration (NOMP=16), a memory reduction to ~23-30% of the original, and a disk-space reduction to ~70-75%. They also perform a consistency test comparing MPI-only and hybrid runs of a zoom-in galaxy simulation over ~28 Myr, reporting gas-density differences at the few-percent level, which they attribute to round-off error. The paper concludes that yOMP offers a factor-of-2 speedup, large memory and disk savings, and reasonably reproduces MPI-only results.

Significance. If the claims hold, RAMSES-yOMP is a practically important contribution: it offers a factor-of-2 speedup at fixed core count, memory reduction by a factor of ~3-4, and ~1.3-1.4x disk savings, while making the code usable on a wider range of HPC systems. The paper's strengths include direct wall-clock measurements on two different processor architectures, a public code release (Zenodo/GitHub), a detailed category-level timing breakdown, and an explicit consistency test with difference maps. The cross-architecture reproducibility of the speedup (53% on Knights Landing, ~45% on Ice Lake) is a notable positive. However, the correctness validation is incomplete in a load-bearing way: the consistency test disables star formation, and the round-off analysis in Appendix B is performed in the same restricted setup. Because the race-prone subroutines identified in Section 4 (CIC density estimation, mechanical supernova feedback) are precisely those not exercised, the claim that the code 'reasonably reproduces' full baryonic MPI-only results is not yet established. The performance claims also rest on single runs without error bars.

major comments (4)
  1. [Sec. 3.4 and Sec. 4] The consistency test halts star formation, and the paper states that this was because its 'stochastic nature made it challenging to reproduce consistently in the OMP run.' The abstract and conclusion nevertheless claim that the benchmark uses 'full prescriptions for baryonic physics' and that the hybrid code 'reasonably reproduces' the MPI-only run. Section 4 explicitly identifies CIC density estimation and mechanical supernova feedback as subroutines that either require atomic/critical operations or are left single-threaded. Since star formation and mechanical feedback are precisely the untested components, the consistency test does not validate the correctness claim for the full baryonic model. The authors should either run a star-forming consistency test and compare statistical diagnostics (e.g., star formation rate, density PDF, power spectra), or explicitly restrict the correctness claim to the star-formation-free regime.
  2. [Appendix B] The round-off argument is not a substitute for testing the race-prone subroutines. The comparison between vectorization sizes Nvec=256 and Nvec=32 is made in the same star-formation-free setup as Fig. 7, so it only constrains round-off variability in the tested code paths. It does not cover the CIC density estimation or mechanical SN feedback subroutines, where a data race or a serialization difference could produce a systematic, non-round-off divergence. A test that exercises these subroutines (e.g., a run with star formation and mechanical SN feedback enabled, compared on statistical distributions) is needed to support the conclusion's claim.
  3. [Sec. 3.1 and Fig. 2] Each configuration is reported as a single wall-clock time with no estimate of run-to-run variability. On shared HPC systems, even deterministic executables show timing noise; the 53% and 45% figures and the identification of NOMP=16 as the 'sweet spot' would be more robust with at least three repeated runs or a statement of observed variance. The difference between NOMP=4 and NOMP=16 appears small in Fig. 2, so an error bar would materially change the strength of the claim that 16 threads is optimal.
  4. [Sec. 3.2 and Fig. 3] The claim that 'nearly half' of the time savings comes from the load-balance weight change and the other half from OMP is not directly supported by the data. The hatched bars compare the original code to yOMP with NOMP=1, which simultaneously includes the PPCG solver and k=2.5; the filled bars compare NOMP=1 to NOMP=16, which also changes NMPI. To support the stated decomposition, the authors should isolate the three effects (PPCG, k, OMP) by running the appropriate combinations (e.g., original CG with k=2.5, PPCG with k=10, PPCG with k=2.5, each with NOMP=1 and NOMP=16).
minor comments (5)
  1. [Abstract and Sec. 3.3] The phrase 'Memory usage has been reduced up to ~23%' is ambiguous; it should read 'reduced to ~23% of the original' or 'reduced by ~77%' to match the abstract's '75% less memory.'
  2. [Sec. 3.2] There are several typos, including 'redu c the ratio' and 'byyOMP' in the Fig. 3 caption, which should be corrected.
  3. [Sec. 3.1] The sentence 'In this performance test, we observe diminishing marginal performance gains with increasing NMPI and NOMP' is confusing because NMPI and NOMP trade off at fixed Ncore; the authors likely mean decreasing NMPI or increasing NOMP, and should clarify.
  4. [Fig. 6] The text references 'Panel (a)' and 'Panel (b)', but the figure panels do not appear to be labeled with (a) and (b); please add panel labels.
  5. [Appendix A] The grammar in the first paragraph is unclear, e.g., 'in which former is comparison in different number of threads and latter is the comparison of subroutines assigned to different categories'; this should be rewritten.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline results are measured benchmark comparisons, and the consistency test uses the original MPI-only code as an external control.

full rationale

This is a performance-engineering paper, not a derivation. The headline speedup, memory savings, and disk savings are reported as directly measured wall-clock times and resource-usage statistics from fixed-core benchmark runs (Secs. 3.1 and 3.3), so there is no fitted parameter that is then renamed as a prediction. The one tuned quantity, the load-balancing weight k = 2.5, is explicitly identified as an adjustable cost weight in Equation (1), and its effect is separately decomposed in Sec. 3.2 into a contribution from the improved cost function and a contribution from OMP threading; the total speedup is still a measured quantity, not a consequence of the tuning by construction. The consistency test in Sec. 3.4 compares the hybrid code against the paper's own MPI-only run started from the same initial condition, which is the standard and appropriate control for a code-replacement claim, and Appendix B strengthens it by showing that an MPI-only run with a different vectorization size deviates comparably, supporting the round-off explanation. Self-citations appear only as background (e.g., NewHorizon physics in Dubois et al. 2021) or as pointers to future work and code release (Han et al. 2024 in prep; Han et al. 2024), and none of them supplies the load-bearing evidence for the measured performance or the consistency result. The genuine weakness is that the consistency test halts star formation and leaves the race-prone mechanical supernova feedback single-threaded, so the fidelity claim for full baryonic physics is not fully covered; that is a validation gap or correctness risk, not circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims are empirical performance measurements, so the ledger is short. The only fitted quantity is the load-balance weight k, changed from 10 to 2.5. The load-bearing assumptions are that the cost function is a faithful proxy for runtime, that the observed divergence in the consistency test is purely round-off error, and that the comparison with the original code is fair at equal compiler and runtime settings.

free parameters (1)
  • load balance weight k = 2.5
    Relative cost weight between particles and grids in Eq. 1, changed from the default 10 to 2.5 to improve load balance for baryonic runs. This is a hand-tuned value, not derived from a model.
assumptions (3)
  • domain assumption The cost function (Eq. 1) with weights w(ell) proportional to fine-step counts is a valid proxy for true computational load.
    The load-balancing improvements and the retuned k=2.5 are justified only if this proxy captures actual runtime cost; the paper does not validate the proxy independently.
  • domain assumption Divergence between MPI-only and hybrid runs is caused by round-off error, not by race conditions or algorithmic differences.
    The paper supports this with an Nvec comparison (Appendix B), but star formation was disabled in the consistency test (Sec. 3.4), and mechanical SN feedback remains single-threaded (Sec. 4), so the assumption is not tested for the most race-prone subroutines.
  • domain assumption Compiler flags and build settings used for the original and yOMP benchmarks are equivalent and representative.
    The paper does not report compiler versions, optimization flags, or MPI/OpenMP runtime settings, so the fairness of the wall-clock comparison rests on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAMSES-yOMP: Performance Optimizations for the Astrophysical Hydrodynamic Simulation Code RAMSES." pith.science (2026). https://pith.science/paper/EMU5Y7JB

@misc{pith2026241114631,
  author       = {Pith},
  title        = {Pith review of: RAMSES-yOMP: Performance Optimizations for the Astrophysical Hydrodynamic Simulation Code RAMSES},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMU5Y7JB}},
  note         = {Machine review of arXiv:2411.14631}
}
read the original abstract

Developing an efficient code for large, multiscale astrophysical simulations is crucial in preparing the upcoming era of exascale computing. RAMSES is an astrophysical simulation code that employs parallel processing based on the Message Passing Interface (MPI). However, it has limitations in computational and memory efficiency when using a large number of CPU cores. The problem stems from inefficiencies in workload distribution and memory allocation that inevitably occur when a volume is simply decomposed into domains equal to the number of working processors. We present RAMSES-yOMP, which is a modified version of RAMSES designed to improve parallel scalability. Major updates include the incorporation of Open Multi-Processing into the MPI parallelization to take advantage of both the shared and distributed memory models. Utilizing this hybrid parallelism in high-resolution benchmark simulations with full prescriptions for baryonic physics, we achieved a performance increase of a factor of 2 in the total run-time, while using 75% less memory and 30% less storage compared to the original code, when using the same number of processors. These improvements allow us to perform larger or higher-resolution simulations than what was feasible previously.

Figures

Figures reproduced from arXiv: 2411.14631 by the authors.

Figure 1
Figure 1. shows an example of load balancing with MPI domain decomposition. The figure shows snapshots of a zoom-in simulation that focuses on a single MW-sized galaxy using RAMSES, with spatial resolution of 68 pc at z ∼ 0. Each row represents multiple snapshots of a thin slice passing through the central region of the galaxy, where the volume is divided into segments along the Peano-Hilbert space-filling curve. Nstep (indic… view at source ↗
Figure 2
Figure 2. represents the relative execution time of yOMP compared to the original RAMSES code. The first bar (labeled as “orig”) represents the result from the origi￾nal code, while the following bars show the results from yOMP with different NOMP = (1, 4, 16, 64), and ac￾cordingly NMPI = (1536, 384, 96, 24) for a fixed number of total working cores Ncore = 1536. The result with NOMP = 1 indicates the MPI-only run with the yO… view at source ↗
Figure 3
Figure 3. Performance improvements measured from [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Distribution of particles in the benchmark simu￾lation with two different numbers of MPI domains (NMPI). The x-axis shows the MPI rank ID for domains. The height of each bar indicates the number of particles owned in each domain, multiplied by the number of MPI domains…
Figure 6
Figure 6. Figure 6: RAM and disk space usage by a different number of OMP threads and MPI domains, with the number of cores fixed (Ncore = 1, 536). More threads lead to fewer domains, which reduces memory imbalance and redundancy. Up to ∼ 77% of RAM and ∼ 30% of disk space savings have be…
Figure 7
Figure 7. Figure 7: Comparison between MPI-only and hybrid parallelism (MPI + OMP) runs of a zoom-in simulation of a spiral galaxy, starting from the same initial state. Each row represents snapshots at different times since the initial state. The elapsed time (t) is shown in units of Myr…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Born to be Starless: Revisiting the Missing Satellite Problem

    astro-ph.GA 2025-06 conditional novelty 4.0 of 10

    Using the NewHorizon and NewHorizon2 simulations, starless subhalos are traced to birth regions with low matter accretion, where gas cannot self-shield from UV background heating before reionization.

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    2010, in 2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing, 583–592, doi: 10.1109/PDP.2010.51 1 https://github.com/sanhancluster/RAMSES-yOMP

    Ament, M., Knittel, G., Weiskopf, D., & Strasser, W. 2010, in 2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing, 583–592, doi: 10.1109/PDP.2010.51 1 https://github.com/sanhancluster/RAMSES-yOMP

    work page doi:10.1109/pdp.2010.51 2010

  2. [2]

    2019, A&A, 621, A96, doi: 10.1051/0004-6361/201834496

    Cadiou, C., Dubois, Y., & Pichon, C. 2019, A&A, 621, A96, doi: 10.1051/0004-6361/201834496

    doi:10.1051/0004-6361/201834496 2019

  3. [3]

    2012, MNRAS, 420, 2662, doi: 10.1111/j.1365-2966.2011.20236.x 11

    Dubois, Y., Devriendt, J., Slyz, A., & Teyssier, R. 2012, MNRAS, 420, 2662, doi: 10.1111/j.1365-2966.2011.20236.x 11

    arXiv 2012

  4. [4]

    2021, A&A, 651, A109, doi: 10.1051/0004-6361/202039429

    Dubois, Y., Beckmann, R., Bournaud, F., et al. 2021, A&A, 651, A109, doi: 10.1051/0004-6361/202039429

    doi:10.1051/0004-6361/202039429 2021

  5. [5]

    2024, A&A, 687, A240, doi: 10.1051/0004-6361/202449784

    Dubois, Y., Rodr ´ ıguez Montero, F., Guerra, C., et al. 2024, A&A, 687, A240, doi: 10.1051/0004-6361/202449784

    doi:10.1051/0004-6361/202449784 2024

  6. [6]

    L., Springel, V., et al

    Genel, S., Bryan, G. L., Springel, V., et al. 2019, ApJ, 871, 21, doi: 10.3847/1538-4357/aaf4bb

    doi:10.3847/1538-4357/aaf4bb 2019

  7. [7]

    2014, Parallel Computing, 40, 224, doi: https://doi.org/10.1016/j.parco.2013.06.001

    Ghysels, P., & Vanroose, W. 2014, Parallel Computing, 40, 224, doi: https://doi.org/10.1016/j.parco.2013.06.001

    work page doi:10.1016/j.parco.2013.06.001 2014

  8. [8]

    2024, RAMSES-yOMP: Public Release, Zenodo, doi: 10.5281/zenodo.14013854

    Han, S., Dubois, Y., Lee, J., et al. 2024, RAMSES-yOMP: Public Release, Zenodo, doi: 10.5281/zenodo.14013854

    work page doi:10.5281/zenodo.14013854 2024

Show all 18 references
  1. [9]

    W., Wadsley, J

    Keller, B. W., Wadsley, J. W., Wang, L., & Kruijssen, J. M. D. 2019, MNRAS, 482, 2244, doi: 10.1093/mnras/sty2859

    2019 doi

  2. [10]

    2014, ApJ, 788, 121, doi: 10.1088/0004-637X/788/2/121

    Kimm, T., & Cen, R. 2014, ApJ, 788, 121, doi: 10.1088/0004-637X/788/2/121

    2014 doi

  3. [11]

    2017, MNRAS, 466, 4826, doi: 10.1093/mnras/stx052

    Kimm, T., Katz, H., Haehnelt, M., et al. 2017, MNRAS, 466, 4826, doi: 10.1093/mnras/stx052

    2017 doi

  4. [12]

    V., Klypin, A

    Kravtsov, A. V., Klypin, A. A., & Khokhlov, A. M. 1997, ApJS, 111, 73, doi: 10.1086/313015

    1997 doi

  5. [13]

    N., et al

    Lee, J., Shin, J., Snaith, O. N., et al. 2021, ApJ, 908, 11, doi: 10.3847/1538-4357/abd08b

    2021 doi

  6. [14]

    M., Mobarry, C., de Fainchtein, R., & Packer, C

    MacNeice, P., Olson, K. M., Mobarry, C., de Fainchtein, R., & Packer, C. 2000, Computer Physics Communications, 126, 330, doi: 10.1016/S0010-4655(99)00501-9

    2000 doi

  7. [15]

    2018, MNRAS, 473, 4077, doi: 10.1093/mnras/stx2656

    Pillepich, A., Springel, V., Nelson, D., et al. 2018, MNRAS, 473, 4077, doi: 10.1093/mnras/stx2656

    2018 doi

  8. [16]

    2008, ApJS, 178, 179, doi: 10.1086/590370

    Prunet, S., Pichon, C., Aubert, D., et al. 2008, ApJS, 178, 179, doi: 10.1086/590370

    2008 doi

  9. [17]

    2002, A&A, 385, 337, doi: 10.1051/0004-6361:20011817 Thi´ ebaut, J., Pichon, C., Sousbie, T., Prunet, S., &

    Teyssier, R. 2002, A&A, 385, 337, doi: 10.1051/0004-6361:20011817 Thi´ ebaut, J., Pichon, C., Sousbie, T., Prunet, S., &

    2002 doi

  10. [18]

    Particle

    Pogosyan, D. 2008, MNRAS, 387, 397, doi: 10.1111/j.1365-2966.2008.13250.x 12 Han et al. APPENDIX A. RESULTS FROM DIFFERENT OPERATING SYSTEM In the main text of this paper, we employed a multi- core system based on the deprecated Knights Landing architecture, which is not direc...

    2008

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.