REVIEW 2 major objections 3 minor 34 references
NUMA balancing hampering performance of spiking network simulations
T0 review · 2 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read With automatic NUMA balancing switched off, a benchmark simulation of the macaque visual cortex runs roughly 30 percent faster on dual-socket AMD EPYC nodes, with identical network output.
desk verdict Real runtime win from disabling NUMA balancing for NEST/MAM, but the energy savings claim is inferred from wall-clock and the headline 30% doesn't match the figures' 36%. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the automatic NUMA balancing heuristic of the Linux kernel, which scans memory pages on a periodic basis and migrates pages according to which NUMA domain faults on them first after unmapping. It interacts with NEST's simulation cycle, in which spike delivery and neuron update are thread-parallel across four NUMA domains per socket while collocation and communication are executed by a single master thread. The kernel samples one post-scan access per page per scan period, so its locality statistics reflect a coincidental simulation phase rather than the dominant one; the result is a stream of page migrations that cost time without delivering locality benefit. The pa
What would settle it
Measure the actual node-level power draw of the same multi-area model simulation on the same hardware with automatic NUMA balancing enabled and disabled. If the total energy consumed over the full run is not reduced in proportion to the roughly 30 percent wall-clock reduction—for example because the power draw during the long-tail cycles is lower—the paper's energy claim fails. Alternatively, run the same benchmark on a node with a single NUMA domain per socket or with a different NUMA-balancing implementation; if the runtime gap disappears, the effect is not general.
Extended reading notes
Core claim
The central discovery is that automatic NUMA balancing—the kernel feature that periodically unmaps pages and migrates them toward the NUMA domain that appears to access them most—systematically degrades the performance of the NEST simulation code on dual-socket AMD EPYC nodes. The degradation appears as a long tail of simulation-cycle times that forces every MPI process to wait for the slowest cycle, inflating the real-time factor. The mechanism is a mismatch of timescales: NUMA scan periods last seconds and thousands of simulation cycles pass between scans, so the kernel records only whichever thread happens to touch a page first after unmapping; in this code, that is often the master threa
Load-bearing premise
The paper equates the 30 percent runtime reduction with a 30 percent reduction in energy consumption, but no power or energy measurements are presented; if node power draw is not roughly flat across the phases affected by NUMA balancing, the energy saving could differ from the runtime saving.
Editorial extensions
If this is right
- If the claim holds, HPC centers can obtain a roughly 30 percent runtime reduction on affected simulation workloads simply by disabling automatic NUMA balancing per job, a change with no code modification and no correctness impact.
- The paper infers a matching energy reduction from the runtime drop; on systems with flat power draw, that energy saving would follow directly.
- Performance analyses of hybrid MPI/OpenMP codes that report noisy or unstable timing should check NUMA-balancing state before attributing variability to the application.
- The user-level scheduler switch introduced by the authors gives researchers a cheap, reversible way to determine the best setting for their own code.
- The result indicates that other OS-level memory-sampling heuristics could create similar hidden performance cliffs for codes with phase-alternating access patterns; such effects should be ruled out in benchmarking.
Reading between the lines
- The same mechanism could affect any hybrid MPI/OpenMP application with per-socket processes, full-core pinning, and serial phases in which one thread reads data written by all threads; codes other than brain simulators should be screened with cycle-resolved instrumentation.
- Direct power measurements would settle whether the runtime gain translates one-to-one to energy; in the absence of such data, the '30 percent energy' figure is a projection, not a measured quantity.
- The heatmap/variability diagnostic could be adopted as a general performance-analysis tool: plotting per-process cycle times against time and workload makes OS- and hardware-induced perturbations distinguishable from application-level imbalance.
- If the effect is common, the default kernel behavior may be wrong for HPC nodes with explicit thread pinning; a per-job switch could become a standard scheduler feature, and future kernel tuning of NUMA balancing scan rates could mitigate the problem without disabling the feature entirely.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates the impact of the Linux kernel's automatic NUMA balancing on the runtime of NEST simulations of the multi-area spiking network model (MAM) on the JURECA-DC supercomputer. Using cycle-resolved timers introduced into NEST 3.10, the authors show that with NUMA balancing enabled the distribution of simulation-cycle times has a long tail, and that disabling NUMA balancing removes this tail and reduces the real-time factor from 41.69 to 26.67 with jemalloc (Fig. 2A vs C) and from 36.35 to 29.81 with the system allocator (Fig. 2B vs D). The effect is reproducible across a strong-scaling sweep (4-32 nodes, three seeds, Fig. 7) and is presented as a machine-level interaction rather than an artifact of network dynamics. The paper also introduces a Slurm option for per-job control of automatic NUMA balancing and proposes a mechanism (Fig. 6) based on the alternation between thread-parallel and single-threaded phases in NEST's simulation cycle. The abstract and discussion interpret the runtime reduction as an approximately 30% reduction in energy consumption.
Significance. If the runtime effect is genuine, the paper's contribution is practically important: it identifies a single kernel parameter that can yield double-digit performance gains for a reference workload in computational neuroscience, and it provides tooling (cycle-resolved timers, time/process-resolved heatmaps) that can help performance engineers diagnose OS-level interference in other HPC codes. The study is methodologically strong in its use of controlled A/B comparisons, two memory allocators, a strong-scaling sweep, three independent seeds, and openly available code and data. The proposed mechanism in Fig. 6 is clearly labeled as a hypothesis, which is appropriate. However, the headline claim of a 30% energy reduction is not directly supported by the measurements, which are wall-clock only.
major comments (2)
- [Abstract and Section IV (Methods)] The abstract states that turning off automatic NUMA balancing 'may reduce energy consumption by 30%', but no power or energy measurements are presented anywhere in the paper. All performance metrics are wall-clock real-time factors (Figs. 2, 4, 7), and the Methods section describes no energy measurement. The energy claim is therefore inferred from runtime, implicitly assuming identical node power draw in the two configurations. Since the proposed mechanism (page migrations, memory traffic) could plausibly affect power draw independently of runtime, this inference is unverified. Please either measure energy directly or revise the abstract and Discussion to claim a runtime/performance improvement and describe energy savings only as a potential, non-quantified consequence.
- [Results, paragraph after Fig. 4] The text states 'The resulting reduction in RTF amounts to approximately 30%'. This does not match the displayed data. For jemalloc the RTF drops from 41.69 to 26.67 (Fig. 2A vs C), which is a 36% reduction; for the system allocator it drops from 36.35 to 29.81 (Fig. 2B vs D), which is an 18% reduction. No stated calculation gives 30%. Clarify which comparison or averaging justifies the 30% figure, or correct it to the actual values. This is important because the abstract's energy claim is built on this number.
minor comments (3)
- [Discussion] Typo: 'weekly correlates' should be 'weakly correlates'.
- [Discussion] Typo: 'wide spread' should be 'widespread'.
- [Throughout] The paper would benefit from an explicit statement that all experiments were performed on a single machine/kernel configuration (JURECA-DC, one Linux version). The authors acknowledge this in the Discussion, but a brief limitation note in the Results or Methods would strengthen the framing.
Circularity Check
No significant circularity: the central RTF reduction is directly measured in controlled experiments, not derived from fitted parameters or self-citations.
full rationale
The paper's central claim—that disabling automatic NUMA balancing reduces the real-time factor of NEST MAM simulations—is an empirical finding based on direct wall-clock measurements (Figs. 2, 4, 7). No free parameters are fitted, no quantity is defined in terms of another predicted quantity, and no uniqueness theorem or ansatz is imported from prior work. The proposed mechanistic explanation in Fig. 6 is explicitly presented as a hypothesis ('A possible scenario is the following...'), and the reported RTF values do not depend on that mechanism. The energy-saving statement in the abstract is an extrapolation from runtime reduction without power measurements; this is a measurement-support concern (correctness risk), not circularity, because runtime and energy are distinct measured/inferred quantities. Self-citations to NEST 3.10, CI-beNNch, and the authors' earlier RTF-related work are tool/method references, not load-bearing evidence for the experimental outcome; the experiment is self-contained and reproducible against the stated hardware and software configuration. The derivation chain therefore does not reduce to its own inputs.
Assumptions & free parameters
assumptions (4)
- domain assumption Energy consumption is proportional to wall-clock runtime on JURECA-DC compute nodes.
- domain assumption The Linux kernel automatic NUMA balancing behavior described (adaptive scan period 1-60 s, hinting faults, page migration) applies to the tested kernel and persists across runs.
- domain assumption The MAM benchmark is representative of spiking network simulation workloads.
- domain assumption Simulation results are unchanged by NUMA balancing.
Cite this review
Pith. "Pith review of NUMA balancing hampering performance of spiking network simulations." pith.science (2026). https://pith.science/paper/GYTGZOP7
@misc{pith2026260722275,
author = {Pith},
title = {Pith review of: NUMA balancing hampering performance of spiking network simulations},
year = {2026},
howpublished = {\url{https://pith.science/paper/GYTGZOP7}},
note = {Machine review of arXiv:2607.22275}
}
read the original abstract
Computing centers today mostly operate conventional CPU- and GPU-based systems, where the direct way of decreasing energy consumption is a reduction in the applications' runtime. Neuromorphic computing promises an alternative architecture with improved energy efficiency for artificial intelligence. In this endeavor, code for the simulation of large-scale spiking networks on conventional supercomputers is the reference. We show that turning off automatic NUMA balancing may reduce energy consumption by 30%. This dwarfs other attempts of increasing the energy efficiency of a computing center with respect to cost effectiveness. The memory access pattern of spiking network simulation code dynamically interacts with automatic NUMA balancing. This does not affect the correctness of simulation results and thus goes unnoticed in day-to-day neuroscience research. In performance analysis, however, time measurements fluctuate obstructing attempts to optimize simulation technology. A new time- and compute-node resolved performance display exposes the fine-grained temporal variability of distributed spiking network simulations. The analysis uncovers that automatic NUMA balancing is of disadvantage and affects the jemalloc library for thread-aware memory allocation in a transient manner. The method also allows developers to detect perturbations of the HPC system and target specific improvements to simulation technology. As a consequence, we have equipped our supercomputers with an option to turn on or off automatic NUMA balancing on a per-job basis on the user level. This gives researchers the opportunity to find the best setting for the application at hand. There are indications in the literature that the effect has been observed before, yet it does not seem common knowledge in scientific computing. It remains to be investigated how widespread the phenomenon is among scientific codes.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
T. C. Potjans and M. Diesmann, “The cell-type specific cortical microcircuit: Relating structure and activity in a full-scale spiking network model,”Cereb. Cortex, vol. 24, no. 3, pp. 785–806, Mar. 2014. [Online]. Available: https://doi.org/10.1093/cercor/bhs358
-
[2]
J. Senk, A. C. Kurth, S. Furber, T. Gemmeke, B. Golosio, A. Heittmann, J. C. Knight, E. M ¨uller, T. Noll, T. Nowotny, G. Peraza Coppola, L. Peres, O. Rhodes, A. Rowley, J. Schemmel, T. Stadtmann, T. Tetzlaff, G. Tiddia, S. J. van Albada, J. Villamar, and M. Diesmann, “Constructive community race: full-density spiking neural network model drives neuromorp...
doi:10.1088/2634-43 2026
-
[3]
M. Schmidt, R. Bakker, K. Shen, G. Bezgin, M. Diesmann, and S. J. van Albada, “A multi-scale layer-resolved spiking network model of resting-state dynamics in macaque visual cortical areas,”PLOS Comput. Biol., vol. 14, no. 10, p. e1006359, 2018. [Online]. Available: https://doi.org/10.1371/journal.pcbi.1006359
-
[4]
Multi-scale account of the network structure of macaque visual cortex,
M. Schmidt, R. Bakker, C. C. Hilgetag, M. Diesmann, and S. J. van Albada, “Multi-scale account of the network structure of macaque visual cortex,”Brain Struct. Funct., vol. 223, no. 3, pp. 1409–1435, Apr
-
[5]
Fast Simulation of a Multi-Area Spiking Network Model of Macaque Cortex on an MPI-GPU Cluster,
G. Tiddia, B. Golosio, J. Albers, J. Senk, F. Simula, J. Pronold, V . Fanti, E. Pastorelli, P. S. Paolucci, and S. J. van Albada, “Fast Simulation of a Multi-Area Spiking Network Model of Macaque Cortex on an MPI-GPU Cluster,”Front. Neuroinform., vol. 16, p. 883333, Jul
-
[6]
Scalable construction of spiking neural networks using up to thousands of gpus,
B. Golosio, G. Tiddia, J. Villamar, L. Pontisso, L. Sergi, F. Simula, P. Babu, E. Pastorelli, A. Morrison, M. Diesmann, A. Lonardo, P. Stanislao Paolucci, and J. Senk, “Scalable construction of spiking neural networks using up to thousands of gpus,”Neuromorphic Comput. Eng., vol. 6, no. 2, p. 024012, May 2026. [Online]. Available: http://dx.doi.org/10.108...
-
[7]
Larger GPU-accelerated brain simulations with procedural connectivity,
J. C. Knight and T. Nowotny, “Larger GPU-accelerated brain simulations with procedural connectivity,”Nat. Comput. Sci., vol. 1, no. 2, pp. 136– 142, 2021
2021
-
[8]
Large-scale, mixed-precision brain simulations on heterogeneous accelerators,
J. Knight, H. Zhu, and T. Nowotny, “Large-scale, mixed-precision brain simulations on heterogeneous accelerators,” in2026 34rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP). IEEE. [Online]. Available: https: //doi.org/10.23919/PDPCPS00030.2026.00031
arXiv 2026
Show all 34 references
-
[9]
Neuromorphic computing at scale,
D. Kudithipudi, C. Schuman, C. M. Vineyard, T. Pandit, C. Merkel, R. Kubendran, J. B. Aimone, G. Orchard, C. Mayr, R. Benosman, J. Hays, C. Young, C. Bartolozzi, A. Majumdar, S. G. Cardwell, M. Payvand, S. Buckley, S. Kulkarni, H. A. Gonzalez, G. Cauwenberghs, C. S. Thakur, A....
2025
-
[10]
NEST (NEural Simulation Tool),
M.-O. Gewaltig and M. Diesmann, “NEST (NEural Simulation Tool),” Scholarpedia J., vol. 2, no. 4, p. 1430, 2007. [Online]. Available: https://doi.org/10.4249/scholarpedia.1430
2007 doi
-
[11]
Exploiting network topology in brain-scale simulations of spiking neural networks,
M. Lober, M. Diesmann, and S. Kunkel, “Exploiting network topology in brain-scale simulations of spiking neural networks,”Neuromorphic Computing and Engineering, vol. 6, no. 2, p. 024024, jun 2026. [Online]. Available: https://doi.org/10.1088/2634-4386/ae762e
2026 doi
-
[12]
NUMA scheduling progress,
J. Corbet, “NUMA scheduling progress,” Linux Weekly News, Oct. 2013, accessed: 2026-07-23. [Online]. Available: https://lwn.net/ Articles/568870/
2013
-
[13]
Merge tag ‘balancenuma-v11’: Automatic NUMA balancing,
M. Gorman, R. van Riel, I. Molnar, and P. Zijlstra, “Merge tag ‘balancenuma-v11’: Automatic NUMA balancing,” Linux kernel git repository, commit3d59eebc5e13, 2013, merged into Linux 3.13. Accessed: 2026-07-23. [Online]. Available: https://git.kernel.org/pub/ scm/linux/kernel/g...
2013
-
[14]
Documentation for /proc/sys/kernel/,
The Linux Kernel Development Community, “Documentation for /proc/sys/kernel/,” The Linux Kernel Documentation, 2024, accessed: 2026-07-23. [Online]. Available: https://www.kernel.org/doc/ html/latest/admin-guide/sysctl/kernel.html
2024
-
[15]
Numa balancing hampering performance of spiking network simulations,
M. Lober, A. Inangu, G. Peraza Coppola, D. Terhorst, S. Gillessen, J. V ogelsang, H. E. Plesser, S. Kunkel, B. Wylie, B. Steinbusch, G. Trensch, and M. Diesmann, “Numa balancing hampering performance of spiking network simulations,” inNEST Conference 2026, 2026. [Online]. Avai...
2026
-
[16]
Numa balancing hampering performance of spiking network simulations,
M. Lober, A. Inangu, G. Peraza Coppola, D. Terhorst, S. Gillessen, J. V ogelsang, H. E. Plesser, B. Wylie, B. Steinbusch, G. Trensch, S. Kunkel, and M. Diesmann, “Numa balancing hampering performance of spiking network simulations,” inICNCE 2026, 2026, p152
2026
-
[17]
A Scalable Concurrent malloc(3) Implementation for FreeBSD,
J. Evans, “A Scalable Concurrent malloc(3) Implementation for FreeBSD,” inProceedings of the BSDCan Conference, April
-
[18]
Matplotlib: Visu- alization with Python,
J. D. Hunter and the Matplotlib development team, “Matplotlib: Visu- alization with Python,” https://matplotlib.org, 2007, version 2.0+ with default colormapviridis
2007
-
[19]
SLURM: Simple linux utility for resource management,
M. A. Jette, A. B. Yoo, and M. Grondona, “SLURM: Simple linux utility for resource management,” inProceedings of the 9th International Workshop on Job Scheduling Strategies for Parallel Processing, ser. Lecture Notes in Computer Science, vol. 2862. Springer, 2003, pp. 44–60
2003
-
[20]
Constructing neuronal network models in massively parallel environments,
T. Ippen, J. M. Eppler, H. E. Plesser, and M. Diesmann, “Constructing neuronal network models in massively parallel environments,”Front. Neuroinform., vol. 11, p. 30, 2017. [Online]. Available: https: //www.frontiersin.org/article/10.3389/fninf.2017.00030
2017
-
[21]
Challenges of memory management on modern NUMA systems,
F. Gaud, B. Lepers, J. R. Funston, M. Dashti, A. Fedorova, V . Qu ´ema, R. Lachaize, and M. Roth, “Challenges of memory management on modern NUMA systems,”Communications of the ACM, vol. 58, no. 12, pp. 59–66, 2015. [Online]. Available: https://dl.acm.org/doi/10.1145/2814328
2015 doi
-
[22]
High performance computing tuning guide for AMD EPYC™ 9004 Series Processors,
AMD, Inc., “High performance computing tuning guide for AMD EPYC™ 9004 Series Processors,” Advanced Micro Devices, Inc., Technical Report, 2023, aMD EPYC 9004 Series Tuning Guide. [Online]. Available: https://docs.amd.com/api/khub/documents/ NgOfoW49HKdzztTeLBbekA/content
2023
-
[23]
Optimizing linux for AMD EPYC™ 9004 Series Processors with SUSE Linux Enterprise Server 15 SP4,
M. Gorman and M. Jambor, “Optimizing linux for AMD EPYC™ 9004 Series Processors with SUSE Linux Enterprise Server 15 SP4,” SUSE Software Solutions Germany GmbH, SUSE Best Practices, 2024, accessed: 09 July 2026. [Online]. Available: https://documentation.suse.com/sbp/tuning-pe...
2024
-
[24]
Challenges of memory management on modern n-UMA systems,
F. Gaud, B. Lepers, J. R. Funston, M. Dashti, A. Fedorova, V . Qu ´ema, R. Lachaize, and M. Roth, “Challenges of memory management on modern n-UMA systems,”Com- munications of the ACM, Practice Section, Aug. 2023, online article. [Online]. Available: https://cacm.acm.org/pract...
2023
-
[25]
Global-state aware automatic NUMA balancing,
J. Liu and Z. Yu, “Global-state aware automatic NUMA balancing,” inProceedings of the 15th Asia-Pacific Symposium on Internetware (Internetware 2024). New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://dl.acm.org/doi/10.1145/ 3671016.3671380
2024
-
[26]
Software in science is ubiquitous yet overlooked,
A. Hocquet, F. Wieber, G. Gramelsberger, K. Hinsen, M. Diesmann, F. Pasquini Santos, C. Landstr¨om, B. Peters, D. Kasprowicz, A. Borrelli, P. Roth, C. A. L. Lee, A. Olteanu, and S. B ¨oschen, “Software in science is ubiquitous yet overlooked,”Nature Computational Science, vol....
2024
-
[27]
Continuous benchmarking: Keeping pace with an evolving ecosystem of models and technologies,
J. V ogelsang, M. Lober, C. M. Sch ¨ofmann, J. Villamar, D. Terhorst, J. Senk, H. E. Plesser, M. Diesmann, S. Kunkel, and A. C. Kurth, “Continuous benchmarking: Keeping pace with an evolving ecosystem of models and technologies,” 2026. [Online]. Available: https://arxiv.org/ab...
2026 arXiv
-
[28]
NEST 3.10,
C. M. Sch ¨ofmann, A. Benelhedi, J. Mitchell, S. Spreizer, P. Nagendra Babu, N. Haug, A. Morrison, D. Terhorst, A. Inangu, R. de Schepper, C. Linssen, J.-E. W. Skaar, S. Kunkel, J. M. Eppler, A. Korcsak-Gorzo, M. Lober, J. V ogelsang, G. Trensch, S. Graber, and H. E. Plesser, ...
2026 doi
-
[29]
JURECA: Data centric and booster modules implementing the modular supercomputing architecture at J ¨ulich Supercomputing Centre,
P. Th ¨ornig, “JURECA: Data centric and booster modules implementing the modular supercomputing architecture at J ¨ulich Supercomputing Centre,”Journal of large-scale research facilities JLSRF, vol. 7, Oct
-
[30]
Data for NUMA balancing hampering performance of spiking network simulations,
M. Lober, A. Inangu, G. Peraza Coppola, D. Terhorst, S. Gillessen, J. V ogelsang, H. E. Plesser, B. Wylie, B. Steinbusch, G. Trensch, S. Kunkel, and M. Diesmann, “Data for NUMA balancing hampering performance of spiking network simulations,” 2026. [Online]. Available: https://...
2026 doi
-
[2006]
Available: https://people.freebsd.org/ ∼jasone/jemalloc/ bsdcan2006/jemalloc.pdf
[Online]. Available: https://people.freebsd.org/ ∼jasone/jemalloc/ bsdcan2006/jemalloc.pdf
-
[2018]
Available: https://doi.org/10.1007/s00429-017-1554-4
[Online]. Available: https://doi.org/10.1007/s00429-017-1554-4
-
[2021]
Available: https://doi.org/10.17815/jlsrf-7-182
[Online]. Available: https://doi.org/10.17815/jlsrf-7-182
-
[2022]
Available: https://doi.org/10.3389/fninf.2022.883333
[Online]. Available: https://doi.org/10.3389/fninf.2022.883333
2022
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.