{"id":"df8461ea-f66a-47ee-a302-785f304a84c4","arxiv_id":"2411.14631","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A hybrid MPI+OpenMP rewrite of the RAMSES code achieves roughly a 2x speedup, 70-77% lower RAM use, and 25-30% lower storage use at fixed core count in galaxy-formation benchmarks.","lead":"RAMSES-yOMP is a modified version of the astrophysical simulation code RAMSES that adds OpenMP threading on top of MPI, plus a faster Poisson solver and retuned load balancing. On a 1536-core benchmark with realistic galaxy-formation physics, it runs about twice as fast and uses roughly a quarter of the memory of the original code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Consistency test omits star formation and mechanical SN feedback, so the claim that yOMP reproduces full baryonic MPI-only results is not yet established.","rationale":"The reader's weakest assumption is exactly the one I identified: the consistency test disables star formation, leaving the race-prone and stochastic baryonic subroutines unvalidated. The paper's own discussion in Sec. 4 names CIC density estimation and mechanical SN feedback as problematic, and these are not exercised in the test. Appendix B's round-off comparison is informative but confined to the same star-formation-free regime, so it does not settle the question. The performance and memory claims are separately supported by measurements on two architectures and a public code release, but the reproducibility of full-baryonic physics is a necessary condition for the code to be a drop-in replacement. The reader's CONDITIONAL verdict is appropriate: the central performance claim is plausible, but the physics-reproduction claim needs an additional test. My concern does not move the verdict; it reinforces the existing conditional status. I do not see an internal inconsistency or a fatal flaw in the performance measurements, so I am not proposing a stronger action.","tokens_in":13060,"tokens_out":4342,"duration_ms":45900,"concrete_test":"Re-run the Sec. 3.4 zoom-in consistency test with star formation and mechanical supernova feedback enabled, using identical initial conditions and a fixed random seed for the MPI-only and MPI+OMP runs. Measure the stellar mass, star formation rate, and gas-phase metallicity at t = 27.9 Myr, and compare the column density difference maps. If the hybrid run deviates from the MPI-only run by more than the run-to-run scatter of an ensemble of MPI-only runs with different random seeds, the deviation cannot be attributed to round-off alone and the reproducibility claim fails. Alternatively, instrument the CIC density and mechanical feedback subroutines to compare their outputs bitwise between MPI-only and hybrid modes given the same input state; bitwise equality would directly rule out algorithmic bias in those race-prone routines.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim includes both a factor-2 speedup and faithful reproduction of MPI-only physics. The performance side is empirically supported, but the correctness side rests on a consistency test (Sec. 3.4) that explicitly halts star formation because its stochastic nature was difficult to reproduce in the OMP run. This test therefore never exercises mechanical supernova feedback or the stochastic star-formation subroutines that Sec. 4 identifies as race-prone or left single-threaded. Appendix B argues that the observed divergence is within round-off variation by comparing against an MPI-only run with a different vectorization size, but that comparison is made in the same star-formation-free setup, so it does not cover the untested subroutines. If the hybrid implementation introduces a systematic difference in CIC density estimation or mechanical SN feedback (e.g., through different order of atomic updates or single-threaded serialization), the divergence in a full baryonic simulation could exceed round-off and grow with time, invalidating the claim that the new code 'reasonably reproduces' MPI-only results for realistic galaxy simulations. The paper itself acknowledges these subroutines as potential bottlenecks, so the omitted test is not a minor gap but a load-bearing limitation of the consistency argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents RAMSES-yOMP, a modified version of the AMR code RAMSES that adds OpenMP thread parallelization on top of the existing MPI domain decomposition, replaces the conjugate-gradient Poisson solver with a preconditioned pipelined CG (PPCG) solver, and changes the load-balance weight k for particles from 10 to 2.5. The authors benchmark the code on 1536 cores (Knights Landing and Ice Lake systems) against the original MPI-only code, reporting a wall-clock reduction to ~53% of the original at the best configuration (NOMP=16), a memory reduction to ~23-30% of the original, and a disk-space reduction to ~70-75%. They also perform a consistency test comparing MPI-only and hybrid runs of a zoom-in galaxy simulation over ~28 Myr, reporting gas-density differences at the few-percent level, which they attribute to round-off error. The paper concludes that yOMP offers a factor-of-2 speedup, large memory and disk savings, and reasonably reproduces MPI-only results.","tokens_in":13295,"tokens_out":9064,"duration_ms":80414,"significance":"If the claims hold, RAMSES-yOMP is a practically important contribution: it offers a factor-of-2 speedup at fixed core count, memory reduction by a factor of ~3-4, and ~1.3-1.4x disk savings, while making the code usable on a wider range of HPC systems. The paper's strengths include direct wall-clock measurements on two different processor architectures, a public code release (Zenodo/GitHub), a detailed category-level timing breakdown, and an explicit consistency test with difference maps. The cross-architecture reproducibility of the speedup (53% on Knights Landing, ~45% on Ice Lake) is a notable positive. However, the correctness validation is incomplete in a load-bearing way: the consistency test disables star formation, and the round-off analysis in Appendix B is performed in the same restricted setup. Because the race-prone subroutines identified in Section 4 (CIC density estimation, mechanical supernova feedback) are precisely those not exercised, the claim that the code 'reasonably reproduces' full baryonic MPI-only results is not yet established. The performance claims also rest on single runs without error bars.","major_comments":[{"comment":"The consistency test halts star formation, and the paper states that this was because its 'stochastic nature made it challenging to reproduce consistently in the OMP run.' The abstract and conclusion nevertheless claim that the benchmark uses 'full prescriptions for baryonic physics' and that the hybrid code 'reasonably reproduces' the MPI-only run. Section 4 explicitly identifies CIC density estimation and mechanical supernova feedback as subroutines that either require atomic/critical operations or are left single-threaded. Since star formation and mechanical feedback are precisely the untested components, the consistency test does not validate the correctness claim for the full baryonic model. The authors should either run a star-forming consistency test and compare statistical diagnostics (e.g., star formation rate, density PDF, power spectra), or explicitly restrict the correctness claim to the star-formation-free regime.","section":"Sec. 3.4 and Sec. 4"},{"comment":"The round-off argument is not a substitute for testing the race-prone subroutines. The comparison between vectorization sizes Nvec=256 and Nvec=32 is made in the same star-formation-free setup as Fig. 7, so it only constrains round-off variability in the tested code paths. It does not cover the CIC density estimation or mechanical SN feedback subroutines, where a data race or a serialization difference could produce a systematic, non-round-off divergence. A test that exercises these subroutines (e.g., a run with star formation and mechanical SN feedback enabled, compared on statistical distributions) is needed to support the conclusion's claim.","section":"Appendix B"},{"comment":"Each configuration is reported as a single wall-clock time with no estimate of run-to-run variability. On shared HPC systems, even deterministic executables show timing noise; the 53% and 45% figures and the identification of NOMP=16 as the 'sweet spot' would be more robust with at least three repeated runs or a statement of observed variance. The difference between NOMP=4 and NOMP=16 appears small in Fig. 2, so an error bar would materially change the strength of the claim that 16 threads is optimal.","section":"Sec. 3.1 and Fig. 2"},{"comment":"The claim that 'nearly half' of the time savings comes from the load-balance weight change and the other half from OMP is not directly supported by the data. The hatched bars compare the original code to yOMP with NOMP=1, which simultaneously includes the PPCG solver and k=2.5; the filled bars compare NOMP=1 to NOMP=16, which also changes NMPI. To support the stated decomposition, the authors should isolate the three effects (PPCG, k, OMP) by running the appropriate combinations (e.g., original CG with k=2.5, PPCG with k=10, PPCG with k=2.5, each with NOMP=1 and NOMP=16).","section":"Sec. 3.2 and Fig. 3"}],"minor_comments":[{"comment":"The phrase 'Memory usage has been reduced up to ~23%' is ambiguous; it should read 'reduced to ~23% of the original' or 'reduced by ~77%' to match the abstract's '75% less memory.'","section":"Abstract and Sec. 3.3"},{"comment":"There are several typos, including 'redu c the ratio' and 'byyOMP' in the Fig. 3 caption, which should be corrected.","section":"Sec. 3.2"},{"comment":"The sentence 'In this performance test, we observe diminishing marginal performance gains with increasing NMPI and NOMP' is confusing because NMPI and NOMP trade off at fixed Ncore; the authors likely mean decreasing NMPI or increasing NOMP, and should clarify.","section":"Sec. 3.1"},{"comment":"The text references 'Panel (a)' and 'Panel (b)', but the figure panels do not appear to be labeled with (a) and (b); please add panel labels.","section":"Fig. 6"},{"comment":"The grammar in the first paragraph is unclear, e.g., 'in which former is comparison in different number of threads and latter is the comparison of subroutines assigned to different categories'; this should be rewritten.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid performance-engineering contribution with a public code release, but the correctness validation gap is substantial: the consistency test disables star formation, which is exactly the component most likely to expose race conditions or algorithmic differences in the OMP implementation. The performance decomposition also needs to be made rigorous. I do not see a fatal flaw in the measured speedup, but the manuscript needs a full-baryonic consistency test and more careful quantitative reporting before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RAMSES-yOMP is a genuine engineering contribution. The hybrid MPI+OpenMP implementation, the PPCG solver, and the retuned load-balance cost are not individually new, but the authors integrate them into RAMSES, ship the code publicly, and give direct wall-clock measurements. On two architectures, they get a factor-2 speedup at fixed core count, a factor of ~3-4 memory reduction, and ~1.4x disk reduction. Those numbers are measured, not derived, and the code is released, so the result is checkable. The subroutine-category breakdown (Fig 3) and the memory-filling analysis (Fig 5) are clear and useful.\n\nThe consistency test is the right idea, but it has a hole the authors themselves point to. They halt star formation because its stochastic nature was hard to reproduce in the OMP run. That means the test never exercises mechanical supernova feedback or the race-prone CIC density estimation, both of which are listed in Sec 4 as unresolved bottlenecks. Appendix B shows the observed divergence is consistent with round-off, but only in the star-formation-free setup. So the claim that yOMP \"reasonably reproduces\" full baryonic MPI-only results is not yet demonstrated. That doesn't sink the performance story, but it should be either fixed with a test that includes star formation (maybe with a controlled random seed) or explicitly scoped out.\n\nTwo smaller issues. The performance numbers come from single runs with no error bars; that's common in this literature, but variance on a shared HPC system is real. And the abstract's \"75% less memory\" is for the 64-thread configuration, while the recommended 16-thread run gives ~67% less; the paper is transparent in Sec 3.3, but the abstract overstates the typical benefit. The claim that the code will scale to >10000 cores is speculation, since all benchmarks are at 1536 cores.\n\nAll that said, this is a solid paper with a public release and honest measurements. The main flaw is a scoping issue in the consistency claim, not a fundamental error. It deserves a serious referee and likely acceptance after the authors address the star-formation gap. I'd bring it to a reading group focused on simulation code development.","headline":"Useful, honest performance engineering with a public RAMSES fork; the speedup is credible, but the full-baryonic consistency claim is not yet supported because the test turns star formation off.","tokens_in":13851,"tokens_out":4072,"would_cite":true,"duration_ms":40189,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding OpenMP threads to RAMSES delivers a factor-2 speedup at 1,536 cores.","keywords":["RAMSES","hybrid MPI/OpenMP","adaptive mesh refinement","hydrodynamic simulation","parallel performance","load balancing","Poisson solver","memory efficiency"],"falsifier":"Run the same zoom-in galaxy test with star formation enabled and compare the hybrid run against two MPI-only runs with different vectorization sizes $N_{\\mathrm{vec}}$; if the hybrid/MPI-only difference exceeds the $N_{\\mathrm{vec}}$-induced spread and grows with thread count in a way that round-off accumulation cannot explain, the 'reasonably reproduces' claim fails.","tokens_in":12879,"feed_emoji":"⚡","tokens_out":10337,"duration_ms":82653,"temperature":0.7,"pith_summary":"This paper argues that the dominant bottleneck in large RAMSES simulations is not raw arithmetic but the cost of decomposing the volume into as many MPI domains as cores: load imbalance, redundant boundary data, and per-domain memory spikes all grow with domain count. RAMSES-yOMP therefore runs several OpenMP threads per MPI domain, so the same total cores are organized into fewer, larger domains. In a 1536-core benchmark with full baryonic physics, the modified code runs in about 53% of the original time, uses as little as 23% of the original memory, and writes about 70% of the original snapshot data. A consistency test on a zoom-in galaxy simulation shows the hybrid run matches the MPI-only run to round-off-level deviations once stochastic star formation is turned off. The authors conclude that these gains let existing facilities run larger or higher-resolution simulations than the MPI-only design would allow.","feed_headline":"OpenMP threads cut RAMSES runtime in half at 1,536 cores","feed_subtitle":"MPI+OpenMP rewrite also uses 75% less memory and 30% less storage at fixed core count.","key_machinery":"The central object is the hybrid thread-team structure, defined by $N_{\\mathrm{core}} = N_{\\mathrm{MPI}} \\times N_{\\mathrm{OMP}}$: total cores are fixed while the balance between distributed-memory domains and shared-memory threads is swept. On top of that, two algorithmic changes do part of the work: a preconditioned pipelined conjugate gradient Poisson solver (PPCG) that halves the number of global-synchronization iterations, and a modified load-balancing cost function $C=\\sum_\\ell w(\\ell)(k N_{\\mathrm{grid}}(\\ell)+N_{\\mathrm{part}}(\\ell))$ with $k$ reduced from 10 to 2.5, giving particles four times more relative weight in domain decomposition. OMP is applied fork-join style to the time-intensive grid and particle loops with static or dynamic scheduling depending on loop cost uniformity; the race-prone cloud-in-cell density estimation and mechanical supernova feedback are deliberately left single-threaded.","core_discovery":"The paper's central claim is that the efficiency ceiling of RAMSES at large core counts is dominated by domain-decomposition side effects, not by the hydrodynamics itself. By keeping the same total number of cores but grouping them as $N_{\\mathrm{MPI}}\\times N_{\\mathrm{OMP}}$ — a small number of MPI domains each running several OpenMP threads — the modified code reduces inter-domain boundary data, eases load imbalance, and lets memory be preallocated against a far smaller occupancy spread. In the 1536-core benchmark, execution time drops to $\\sim$53% of the original (best at $N_{\\mathrm{OMP}}=16$), memory to $\\sim$23% at $N_{\\mathrm{OMP}}=64$, and snapshot disk use to $\\sim$70%; the gains come mostly from the particle and gravity categories, with hydro unchanged. The authors attribute the particle improvement to a retuned load-balancing weight ($k$: 10 to 2.5) and the gravity improvement to replacing conjugate gradient with a preconditioned pipelined CG solver that reaches $10^{-4}$ relative error in about half the iterations. A one-galaxy zoom-in test with star formation disabled shows hybrid and MPI-only runs agree within the scatter expected from round-off error.","pith_inferences":["The paper leaves implicit how the sweet spot scales: a weak-scaling study varying $N_{\\mathrm{core}}$ from 1,536 up to tens of thousands would show whether $N_{\\mathrm{OMP}}=16$ remains optimal or whether the optimum shifts with problem size and machine architecture.","Because cloud-in-cell density estimation and mechanical supernova feedback stay single-threaded, full-physics runs could in principle diverge from MPI-only runs by more than the round-off scatter shown in Appendix B; the consistency test with star formation off does not exercise those paths.","The calibrated value $k=2.5$ is specific to the NewHorizon-style baryonic prescription; simulations that add more chemical species, dust, or radiation would likely need to re-tune this weight, following the paper's category-timing procedure."],"forward_implications":["At a fixed $N_{\\mathrm{core}}=1536$, $N_{\\mathrm{OMP}}=16$ gives the best wall-clock time, cutting total runtime to $\\sim$53% of the original on Knights Landing and $\\sim$45% on Ice Lake.","Memory use falls to $\\sim$23--30% of the original and disk use to $\\sim$70--75% at the same core count, because fewer MPI domains reduce domain-boundary redundancy and raise the particle occupation fraction.","Replacing CG with PPCG roughly halves the number of Poisson-solver iterations needed to reach $\\varepsilon=10^{-4}$, removing a large share of global MPI communication.","The retuned load-balancing weight ($k=2.5$ instead of 10) and the smaller number of MPI domains make the particle and load-balancing categories the largest sources of saved time.","Because the thread count can be changed during a run, the code can adapt to resource availability in ways the MPI-only version cannot."],"supporting_citations":[{"why":"Base RAMSES code whose MPI-only domain decomposition yOMP modifies.","marker":"Teyssier 2002"},{"why":"Peano-Hilbert space-filling curve decomposition that sets the boundary and load-balancing costs discussed throughout.","marker":"MacNeice et al. 2000; Prunet et al. 2008"},{"why":"Supplies the PPCG algorithm (Algorithm 4) that replaces the native conjugate-gradient Poisson solver.","marker":"Ghysels & Vanroose 2014"},{"why":"Supplies the preconditioner (Equation 24) generalized to the seven-point 3D stencil used by yOMP.","marker":"Ament et al. 2010"},{"why":"NewHorizon baryonic prescriptions (star formation, feedback, black holes) that the benchmark simulations inherit.","marker":"Dubois et al. 2021"},{"why":"Mechanical supernova feedback model singled out as a race-prone subroutine left single-threaded.","marker":"Kimm & Cen 2014"},{"why":"Thermo-turbulent star formation prescription whose stochasticity forced the consistency test to disable star formation.","marker":"Kimm et al. 2017"},{"why":"Tracer particle scheme included in the code and part of the particle-workload category.","marker":"Cadiou et al. 2019"},{"why":"Cited as the expected run-to-run stochastic deviation range against which hybrid/MPI-only differences are judged.","marker":"Thi´ebaut et al. 2008; Genel et al. 2019; Keller et al. 2019"}],"fun_headline_variants":["Hybrid MPI+OpenMP doubles RAMSES speed, cuts memory 75%","RAMSES-yOMP: 2x runtime gain, 75% less memory at 1536 cores","MPI+OpenMP rewrite cuts RAMSES memory to one quarter","RAMSES-yOMP: larger simulations with half the runtime","Hybrid parallelism: RAMSES runs 2x faster, uses 4x less memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the consistency test with star formation turned off is enough to prove the hybrid code matches the MPI-only physics; if a hidden data race in the particle or feedback routines changes results by more than round-off error, the fidelity claim fails.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid MPI+OpenMP doubles RAMSES speed, cuts memory 75%","RAMSES-yOMP: 2x runtime gain, 75% less memory at 1536 cores","MPI+OpenMP rewrite cuts RAMSES memory to one quarter","RAMSES-yOMP: larger simulations with half the runtime","Hybrid parallelism: RAMSES runs 2x faster, uses 4x less memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000369,"raw_usage":{"total_tokens":2013,"prompt_tokens":1016,"completion_tokens":997,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":885}},"tokens_in":632,"tokens_out":997,"duration_ms":8764,"temperature":1.0,"reasoning_tokens":885,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:04:31.482620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same zoom-in galaxy test with star formation enabled and compare the hybrid run against two MPI-only runs with different vectorization sizes $N_{\\mathrm{vec}}$; if the hybrid/MPI-only difference exceeds the $N_{\\mathrm{vec}}$-induced spread and grows with thread count in a way that round-off accumulation cannot explain, the 'reasonably reproduces' claim fails.","supporting_citations":[{"cited_title":"2014, Parallel Computing, 40, 224, doi: https://doi.org/10.1016/j.parco.2013.06.001","cited_arxiv_id":null,"evidence_quote":"Supplies the PPCG algorithm (Algorithm 4) that replaces the native conjugate-gradient Poisson solver."},{"cited_title":"2010, in 2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing, 583–592, doi: 10.1109/PDP.2010.51 1 https://github.com/sanhancluster/RAMSES-yOMP","cited_arxiv_id":null,"evidence_quote":"Supplies the preconditioner (Equation 24) generalized to the seven-point 3D stencil used by yOMP."}],"review_version":1}