REVIEW 4 major objections 4 minor 13 references
Performance Comparison for Neuroscience Application Benchmarks
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read For neuroscience benchmark workloads, the Skylake node wins SIMD-friendly simulations, the POWER8 node wins GPU deep learning, and non-SIMD workloads run about the same.
desk verdict Useful benchmark data for neuroscience HPC, but the SIMD-width conclusion is undercut by a compiler-version confound; still worth refereeing as a practical procurement study. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the benchmark suite itself: five real neuroscience applications chosen to span simulation and data-analysis workloads. Each benchmark pins a distinct performance regime: Arbor tests SIMD and GPU compute throughput; NEST tests thread and task scaling plus cache behavior; TVB-HPC tests just-in-time compiled Python scaling; ASSET tests memory-bandwidth-bound statistical kernels; the deep-learning mini-app tests GPU data transport. Running the same suite on the two node types is what turns raw timings into architectural conclusions.
What would settle it
Run the Arbor CPU benchmark and the neuroimaging deep learning benchmark on both systems after installing identical compiler, CUDA, and runtime library versions; if the Skylake advantage on Arbor or the POWER8 advantage on deep learning shrinks materially or reverses, the hardware-attribution conclusion is not supported.
Extended reading notes
Core claim
On the paper's own terms, the discovery is an architecture–workload interaction. For Arbor, a multi-compartment neuron simulator that can vectorize, the Skylake-based system is significantly faster than the POWER8-based system on CPU-only nodes, with the gap roughly matching the ratio of floating-point throughput from the wider SIMD units. For NEST, which cannot exploit SIMD parallelism, the Skylake system still wins by making its larger core count usable. TVB-HPC and ASSET, neither able to exploit SIMD parallelism, perform similarly on both systems. The machine learning benchmark, a deep-learning neuroimaging mini-app, runs faster per sample on the POWER8/P100 system despite using an older GPU generation, which the authors attribute to better data transport. These results are presented as benchmark observations meant to guide future infrastructure choices for the brain-research community.
Load-bearing premise
The paper attributes the measured performance differences to hardware architecture even though the two systems ran different compilers, CUDA versions, and system software, so unequal software stacks could account for part of the gap.
Editorial extensions
If this is right
- A compute-limited neuroscience application that can vectorize will get more throughput from a wide-SIMD Skylake CPU node than from a POWER8 CPU node.
- For GPU deep-learning workflows where data transport is the bottleneck, a POWER8 node with NVLink-connected P100 GPUs can beat a Skylake node with V100 GPUs despite the V100's higher peak throughput.
- For spiking neural-network workloads like NEST, the number of physical cores matters more than SIMD width.
- Memory-bound and just-in-time compiled Python workloads such as TVB-HPC and ASSET are not strong discriminators between the two architectures.
- Hardware procurement for brain-research infrastructure should be workload-specific rather than based on a single architecture choice.
Reading between the lines
- Inference: the deep-learning result implies that interconnect and data-transport benchmarks, not GPU peak FLOPs, should be the deciding metric for neuroimaging deep learning; this could be tested by rerunning the same mini-app on a Skylake node with an improved GPU–CPU interconnect.
- Inference: the Arbor result suggests SIMD width is a useful proxy for performance portability; a natural extension would be to run the same Arbor benchmark on a POWER9 node with wider SIMD units to see whether the gap closes.
- Inference: the benchmark suite could be used predictively, classifying new brain-research workloads by bottleneck type (SIMD compute, core count, memory bandwidth, data transport) to anticipate which architecture will favor them before full benchmarking.
- Inference: because the two systems ran different compilers, CUDA versions, and runtime libraries, the architectural attribution remains provisional until a controlled toolchain comparison is done.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper benchmarks five neuroscience applications from the ICEI suite—NEST, Arbor, TVB-HPC, ASSET, and a deep-learning image-analysis workload—on two JSC systems: JURON (IBM POWER8 + NVIDIA P100) and JUWELS (Intel Skylake + NVIDIA V100). It reports execution times across MPI/OpenMP configurations, including GPU scaling for Arbor and the deep-learning benchmark, and summarizes the results as: JUWELS is faster for NEST and for the SIMD-exploiting CPU path of Arbor, the systems are similar for TVB-HPC and ASSET, and JURON is faster for the deep-learning workload. The paper interprets the Arbor result as evidence for the value of wide SIMD units and the deep-learning result as an indication of better data transport in JURON.
Significance. The work is a useful empirical data point: it uses real applications from a defined community benchmark suite, documents node-level hardware parameters, and explores process/thread binding configurations in some detail. Named versions of all software are provided, which makes the measurements reproducible in principle. If the conclusions survive a controlled re-run, the paper would support procurement decisions for brain-research HPC infrastructure. On the other hand, the causal claims about SIMD width and data transport are currently inferences from a comparison in which compiler, CUDA, and framework versions differ across systems; because these are exactly the software changes that affect vectorization and data movement, the headline conclusions are not yet established to the standard that the paper's language suggests.
major comments (4)
- [§4.2, Fig. 5] The central claim that Arbor is faster on JUWELS because of wide SIMD units is not supported by the presented evidence. The CPU runs used GCC 6.3.0 on JURON and GCC 8.2.0 on JUWELS, and no compiler flags, vectorization report, or assembly inspection is given; auto-vectorization behavior changed substantially between these versions. The fact that the runtime ratio roughly matches the FLOP-throughput ratio in Table 1 does not establish that the SIMD pipelines were exercised. Please rerun the CPU benchmark with the same compiler version on both systems (or provide a detailed vectorization report and exact compiler flags), and also match or justify the different CUDA versions (9.2.148 vs 9.2.88) for the GPU runs.
- [§4.1, Figs. 1–3 and Table 2] The NEST comparison is not based on a uniform configuration policy. On JURON the best-performing binding configuration was selected post hoc (Fig. 1 caption: 'the best performing configuration selected'), while on JUWELS a manually created hwloc pinning mask was used because the scheduler/MPI stack does not offer the same high-level options. The resulting advantage of JUWELS in Table 2 (44.50 s vs 25.15 s on one node) may therefore reflect the binding methodology rather than an intrinsic system difference. Please specify a fixed selection rule, report results for all binding configurations, or otherwise demonstrate that the comparison is fair. In addition, the absence of repeated runs means the magnitude of the reported differences is not quantified; include at least a few repetitions or an error estimate for the key configurations.
- [§4.5, Fig. 9] The deep-learning conclusion that JURON's better performance 'may be an indication of better data transport capabilities' is not isolated from software changes: TensorFlow versions are 1.4.1 on JURON and 1.8.0 on JUWELS, and the CUDA versions also differ. Since data-transport behavior is influenced by the framework version and its CUDA backend, the measured difference cannot be attributed to hardware without either matching the software stacks or collecting data-movement profiles (e.g., PCIe/NVLink utilization or copy-engine timings). Please provide such evidence or weaken the conclusion to a hypothesis.
- [§5 and Abstract] The abstract and conclusions present the SIMD-width and data-transport explanations as findings, but the paper itself states in §5 that 'We plan for further efforts to analyse the causes for the different performance behaviour.' The causal statements should either be explicitly labeled as hypotheses or be supported by the additional analysis requested above; as written, the conclusions overstate the strength of the evidence.
minor comments (4)
- [§3, first paragraph] The phrase 'during an the initial phase' is a typo and should read 'during the initial phase.'
- [§4.1, Fig. 1] The figure caption says 'a shaded bar denotes binding to socket, a solid bar denotes binding to core,' but the text refers to 'hatched bars' and 'solid bars'; please make the terminology consistent.
- [§4.1, Fig. 4] The performance-counter labels (CYCLES, L3, D$MISS, REJECT, LOAD, LSU, VSU, STALL) lack units and a description of the normalized quantity plotted; please add an explanatory sentence or axis label.
- [Table 1] Listing GPU throughput in 'Flop/cycle' is unclear because GPUs do not have a single conventional processor cycle; please clarify the unit or present the peak throughput in FLOP/s.
Circularity Check
No significant circularity: empirical benchmark with no derived predictions or fitted parameters.
full rationale
This paper is a purely empirical performance comparison of an application benchmark suite on two systems. There are no equations derived, no parameters fitted, and no quantity is predicted from an assumed input. The central claims are runtime measurements, and the causal attributions (SIMD width for Arbor, data transport for the deep learning application) are post-hoc interpretations of measured timings, not constructions that force the measured outcome. Although a reviewer might worry about compiler-version confounds (GCC 6.3.0 on JURON versus 8.2.0 on JUWELS for Arbor, or TensorFlow 1.4.1 versus 1.8.0), that concern is about experimental validity, not circularity: the reported runtimes are independent evidence regardless of how they are explained. The self-citations to Arbor and NEST code releases document the benchmark inputs and are not load-bearing arguments for any derived result. The paper itself states that causal analysis is planned future work, so no derivation chain is closed by an assumed input. No circular step reduces any claimed result to its own definition or to a fitted parameter renamed as a prediction.
Assumptions & free parameters
assumptions (4)
- domain assumption The ICEI Application Benchmark Suite is representative of the diverse scientific challenges within the Human Brain Project.
- domain assumption The selected problem sizes and simulation times are representative of typical production workloads.
- domain assumption The performance comparison is fair despite systematic differences in compilers, library versions, and system software between JURON and JUWELS.
- domain assumption Single measurements without repeated runs or error bars are sufficient to support the comparative conclusions.
Cite this review
Pith. "Pith review of Performance Comparison for Neuroscience Application Benchmarks." pith.science (2026). https://pith.science/paper/LGIPCNNR
@misc{pith2026190802702,
author = {Pith},
title = {Pith review of: Performance Comparison for Neuroscience Application Benchmarks},
year = {2026},
howpublished = {\url{https://pith.science/paper/LGIPCNNR}},
note = {Machine review of arXiv:1908.02702}
}
read the original abstract
Researchers within the Human Brain Project and related projects have in the last couple of years expanded their needs for high-performance computing infrastructures. The needs arise from a diverse set of science challenges that range from large-scale simulations of brain models to processing of extreme-scale experimental data sets. The ICEI project, which is in the process of creating a distributed infrastructure optimised for brain research, started to build-up a set of benchmarks that reflect the diversity of applications in this field. In this paper we analyse the performance of some selected benchmarks on an IBM POWER8 and Intel Skylake based systems with and without GPUs.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Akar, N.A., Biddiscombe, J., Cumming, B., Kabic, M., Karakasis, V., Klijn, W., Ksters, A., Martinez, I., Peyser, A., Yates, S.: arbor-sim/arbor: Version 0.1: First release (Oct 2018), https://doi.org/10.5281/zenodo.1459679
-
[2]
Akar, N.A., Cumming, B., Karakasis, V., K¨ usters, A., Klijn, W., Peyser, A., Yates, S.: Arbor - A morphologically-detailed neural network simulation library for contemporary high-performance computing architectures. In: 27th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2019, Pavia, Italy, February 13-15, 2...
-
[3]
Jirsa, V., McIntosh, R., Ritter, P., Mersmann, J., et al.: http://www.thevirtualbrain. org/
-
[4]
Archives italiennes de biologie 148(3), 189–205 (2010)
Jirsa, V.K., Sporns, O., Breakspear, M., Deco, G., McIntosh, A.R.: Towards The Virtual Brain: network modeling of the intact and the damaged brain. Archives italiennes de biologie 148(3), 189–205 (2010)
work page 2010
-
[5]
Frontiers in Neuroinformatics 12, 2 (2018)
Jordan, J., Ippen, T., Helias, M., Kitayama, I., Sato, M., Igarashi, J., Diesmann, M., Kunkel, S.: Extremely scalable spiking neuronal network simulation code: From laptops to exascale computers. Frontiers in Neuroinformatics 12, 2 (2018)
work page 2018
-
[6]
Kuramoto, Y.: Self-entrainment of a population of coupled non-linear oscillators. In: Araki, H. (ed.) International Symposium on Mathematical Problems in Theoretical Physics. pp. 420–422. Springer Berlin Heidelberg, Berlin, Heidelberg (1975)
work page 1975
-
[7]
In: Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC
Lam, S.K., Pitrou, A., Seibert, S.: Numba: A llvm-based python jit compiler. In: Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC. pp. 7:1–7:6. LLVM ’15, ACM, New York, NY, USA (2015), http://doi.acm.org/10. 1145/2833157.2833162
arXiv 2015
-
[8]
Linssen, C., Lepperød, M.E., Mitchell, J., Pronold, J., Eppler, J.M., Keup, C., Peyser, A., Kunkel, S., Weidel, P., Nodem, Y., Terhorst, D., Deepu, R., Deger, M., Hahne, J., Sinha, A., Antonietti, A., Schmidt, M., Paz, L., Garrido, J., Ippen, T., Riquelme, L., Serenko, A., K¨ uhn, T., Kitayama, I., Mørk, H., Spreizer, S., Jordan, J., Krishnan, J., Senden,...
Show all 13 references
-
[9]
In: 27th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2019, Pavia, Italy, February 13-15, 2019
Oden, L., Schiffer, C., Spitzer, H., Dickscheid, T., Pleiter, D.: IO challenges for human brain atlasing using deep learning methods - an in-depth analysis. In: 27th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2019, Pavia, Italy...
2019
-
[10]
Peyser, A., Sinha, A., Vennemo, S.B., Ippen, T., Jordan, J., Graber, S., Morrison, A., Trensch, G., Fardet, T., Mrk, H., Hahne, J., Schuecker, J., Schmidt, M., Kunkel, S., Dahmen, D., Eppler, J.M., Diaz, S., Terhorst, D., Deepu, R., Weidel, P., Kitayama, I., Mahmoudian, S., Ka...
2017 doi
-
[11]
Brain Connectivity 3(2), 121–145 (2013)
Ritter, P., Schirner, M., McIntosh, A.R., Jirsa, V.K.: The Virtual Brain integrates computational modeling and multimodal neuroimaging. Brain Connectivity 3(2), 121–145 (2013)
2013
-
[12]
In: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017)
Spitzer, H., Amunts, K., Harmeling, S., Dickscheid, T.: Parcellation of visual cortex on high-resolution histological brain sections using convolutional neural networks. In: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). pp. 920–923 (April 2017)
2017
-
[13]
PLOS Computational Biology 12(7), 1–34 (07 2016)
Torre, E., Canova, C., Denker, M., Gerstein, G., Helias, M., Gr¨ un, S.: ASSET: Analysis of sequences of synchronous events in massively parallel spike trains. PLOS Computational Biology 12(7), 1–34 (07 2016)
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.