REVIEW 5 major objections 5 minor 20 references
Projections of achievable performance for Weather & Climate Dwarfs, and for entire NWP applications, on hybrid architectures
T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This report shows how to predict time and energy of weather and climate codes on CPU+GPU systems, including hardware not yet built.
desk verdict A useful but thinly validated engineering deliverable: the in-sample GPU fitting and the perfect-overlap assumption undercut the headline claim that the models can predict performance on architectures yet to come. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the extended roofline model using W (amount of floating-point work) and Q (amount of data moved through the memory hierarchy), where each loop's execution time is max(W*Tflop, Q*Tmop) and total dwarf time is the sum over loops. For GPUs, the model instead decomposes time into parallel-execution, transfer, and inactive components derived from CUDA profiler counters and hardware peak rates. Energy is modelled as E = W*eflop + Q*emop + T*P0, implemented using measured package (PKG) and DRAM energy coefficients from RAPL, with a similar W/Q decomposition for multinode systems.
What would settle it
Run a weather dwarf (e.g. BiFFT or Spherical Harmonics) on a cluster with a small domain and frequent all-to-all communication, so that communication is deliberately not overlapped with computation, and compare measured time-to-solution and energy-to-solution to the simulator's projection for the same node and core counts. If the measured curve bends upward while the projected curve keeps decreasing with node count, the perfect-overlap assumption is falsified.
Extended reading notes
Core claim
The core discovery is that performance and energy of weather and climate dwarfs on hybrid architectures can be computed from per-kernel characteristics—instruction counts, memory requests, inactive time, and control-flow overhead—combined with hardware parameters such as clock speed, number of multiprocessors, memory bandwidth, and peak FLOP rates. For GPUs, the model decomposes kernel execution time into compute, transfer, inactive, control-flow, and miscellaneous components and sums them. For multinode CPUs, each loop's time is the larger of W*Tflop (work times time per FLOP) and Q*Tmop (bytes times time per byte moved), with internode communication added as an overlay. The paper verifies these models against measurements on three GPU generations and one Intel Xeon family, reporting dwarf-level errors mostly under 20% and energy errors around 8%.
Load-bearing premise
The projections assume that internode communication hides completely behind computation, so multinode time and energy curves do not include communication delays, and they also assume that instruction mixes and loop coefficients measured on one CPU and three GPUs stay valid for other and future chips.
Editorial extensions
If this is right
- If the models are correct, weather centres can rank future CPU/GPU configurations on time-to-solution and energy-to-solution before purchasing hardware, as the report does for Skylake versus Haswell and for several GPU choices.
- The same instruction-mix and loop coefficients measured on one CPU and three GPUs can be reused to project performance onto new chips without rerunning full workloads.
- The workflow extension means a whole NWP application like ALARO can be modelled as cooperating dwarfs on heterogeneous nodes, enabling architecture selection for the full forecast system.
- The energy projections make visible trade-offs that are otherwise hidden, such as a lower-power consumer GPU beating an HPC GPU on energy-to-solution when idle power is included.
Reading between the lines
- The perfect-overlap assumption likely makes multinode projections optimistic; real communication-bound runs will hit a scaling limit earlier than the simulated curves, so the reported balance point should be treated as a lower bound on node count.
- The method is transferable to non-weather stencil and FFT codes, but the fitted coefficients are tied to the benchmarked kernels and would need recalibration rather than direct reuse.
- A testable extension is to use the simulator prospectively for a not-yet-released GPU or CPU, publish the time and energy predictions, and compare against measured results once the hardware ships.
- The emphasis on idle and constant power suggests that future hardware selection for weather workloads should weigh idle power as heavily as peak performance, a principle the report demonstrates but does not generalise.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This ESCAPE project deliverable (arXiv:1908.06098) describes extensions to performance and energy models for Weather & Climate dwarfs on hybrid CPU/GPU architectures, implemented in the DCworms simulator. The GPU model predicts kernel execution times from profiler-derived kernel characteristics (instruction counts, parallelism and memory-efficiency metrics) combined with GPU clock, SM count, and bandwidth parameters. The CPU multinode model computes per-loop execution times from fitted Tmop/Tflop coefficients and memory-hierarchy bandwidths, and the energy model derives PKG and DRAM energy from RAPL measurements. These components are applied to the Spherical Harmonics, BiFFT, and ACRANEB2 dwarfs, and to an ALARO workflow mixing CPU and GPU execution. The central claim, stated in the Executive Summary, is that the extended models allow prediction of computational and energy performance on single-node and multinode systems equipped with CPUs and GPU accelerators, thereby enabling feasible performance projection at system scale.
Significance. If the central claim were established, the work would be valuable for hardware procurement and algorithm design in weather and climate computing: it would allow ranking of CPU/GPU configurations on time-to-solution and energy-to-solution before deployment. The manuscript has concrete strengths: the model equations are stated explicitly, the verification procedure is described in enough detail to be reproduced, the authors honestly report several limitations, and the DCworms integration is demonstrated on realistic workflows. However, the significance is currently limited by the fact that the central predictive claim rests on in-sample calibration: kernel characteristics and loop coefficients are derived from measurements on the same CPUs/GPUs used for verification, and no fully out-of-sample architecture transfer is demonstrated. The multinode component, which is central to the system-scale claim, shows a mean error of 52.4% when communication is included, and the GPU energy model is not validated at all.
major comments (5)
- [§3.3–3.5] The GPU model is calibrated from profiler counters averaged over the same three GPUs that are later used for verification (Tables 9, 12–20), so the reported 7–23% square errors are partly in-sample. The Fermi case is especially concerning: individual-kernel errors reach 32% (Table 14), and the Fermi-specific Ge correction in §3.4 is fitted post hoc to that architecture. This does not support the Executive Summary claim that the model predicts performance on different architectures, including architectures yet to come. An out-of-sample validation on a GPU not used in calibration (e.g., a Pascal or Volta device) is needed before this claim can be accepted.
- [§4.3, Figure 14] The multinode model assumes perfect overlap between computation and internode communication, and the only multinode validation shows mean difference 15.6% without communication and 52.4% with communication added. Since the communication-aware variant is the actual multinode model, the 52.4% mean error directly undermines the system-scale projection claim. The authors acknowledge in Section 7 that the model does not handle cases where communication is not perfectly overlapped; this limitation should be treated as a load-bearing restriction of the central claim, not a minor caveat, and the report should either validate a communication-aware model or clearly restrict the claim to communication-hidden workloads.
- [§5.5, Table 36] The GPU energy model E = T * (Epkg(idle) + S * Egpu) is proposed but never validated against measured GPU energy consumption. Table 36 reports only average power draws for two GPUs; no comparison of modeled versus measured energy for any GPU workload is provided. Because the hybrid ALARO projections in Figures 29–30 depend on this unvalidated GPU energy component, the energy-to-solution results for hybrid configurations should be presented as illustrative projections, not validated predictions, or a validation experiment should be added.
- [§6.1.1, §5.4] The CPU performance coefficients (Table 30) are fitted on a single Haswell Xeon E5-2697v3 and then used to project Skylake performance in §6.1.1 without any performance validation on Skylake. The only cross-CPU validation, Table 35, is energy on a different Haswell model (E5-2640v3), which does not test transfer across microarchitectures. The claim of predicting performance on 'architectures yet to come' is therefore unsupported for CPUs. At minimum, one measured Skylake run should be compared with the projection, or the projections should be explicitly labeled as extrapolations with unknown error.
- [§3.4, Tables 18–20] The model excludes small computational domains by assumption ('kernels have enough computations and parallelism'), and the verification tables show that the excluded small sizes have errors of −65% to −77%. This is a legitimate modeling decision if the target is operational resolutions, but the scope of the central claim should state this validity range explicitly. As written, the Executive Summary's 'feasible performance projection at system scale' could be read as covering all domain sizes, which the model does not support.
minor comments (5)
- [§5.2, Table 32] The hardware used for the energy characterization table is an Intel i7-4700EQ, while the multinode CPU model is built on Xeon E5-2697v3 and validated on E5-2640v3; the text should clarify which processor the table refers to and whether the coefficients are portable across these parts.
- [§6.1.2.2, Table 44] The quality metrics accompanying Table 44 are internally inconsistent: the maximum difference is reported as 6.08%, the minimum as 0.23%, and the mean as 8.59%; a mean cannot exceed the maximum. The numbers should be recomputed or corrected.
- [§5.5] The text says 'The measurements were taken running BiFFT at different grid sizes,' but Table 36 is titled 'ACRANEB2 energy consumption'; this mismatch should be fixed.
- [Throughout] Equations are not numbered, which makes it difficult to refer to specific model components; numbering the equations would improve reproducibility and readability.
- [§4.3, Figure 14] Figure 14 lacks a legend and axis labels explaining which curve is the real run and which is the simulation; the caption should also state the node count and problem size used in the validation.
Circularity Check
GPU model predictions are in-sample: kernel constants are averaged from the very GPUs later used for verification, and a Fermi-specific correction is fitted after observing Fermi errors.
-
fitted input called prediction
[Section 3.4, implementation of GPU model, page 16 (Fermi adjustment) and verification Tables 12-20]
"Moreover, we have observed that data transfer efficiency is lower for Fermi microarchitecture (Tesla 2070-q) and significantly lower when Geagv < 40%. This is why following adjustment has been proposed for this particular architecture: Ge(fermi) = { 1/4 Ge for Ge < 40%; 2/3 Ge for Ge >= 40% }"
The Fermi adjustment is introduced after observing the behavior of the same Tesla 2070-q measurements that are later reported as 'verification'. The corrected model is then used to simulate Tesla 2070-q, so the Fermi results in Tables 14 and 17 are a post-hoc fit to the target architecture, not an independent prediction. This makes the deliverable's claimed cross-architecture predictive power for Fermi in-sample.
-
fitted input called prediction
[Section 3.3-3.4, Tables 8-9 (kernel characteristics) and Tables 12-20 (verification)]
"Profiler counters are used to compute the characteristics of the kernels which are independent of the domain size. These variables are computed by dividing a counter value by the size of the domain (N). In order to obtain these characteristic one should also take the average of the kernel execution on different GPUs... The counters taken to compute characteristics of kernels were measured for the largest size of the domain that fitted all benchmarked GPU architectures i.e. 200x200x80."
The model constants Piavg, Geavg, iiavgn, Ieavgn and related values are averages of profiler counters taken from the three GPUs (GeForce 970, Tesla K20m, Tesla 2070-q) that are then simulated and compared in the verification tables. Because the target architectures themselves contribute to the fitted constants, the reported agreements are largely in-sample; the model does not demonstrate prediction of a held-out or unseen architecture.
full rationale
The report does contain some genuinely independent checks: the energy model is validated on a different CPU (Intel E5-2640v3, Section 5.4), BiFFT energy is compared against measured runs (Table 44), and the multinode SH simulation is compared with a real run (Figure 14). These parts give the deliverable independent content. However, the central GPU projection chain is not independent: kernel characteristics are obtained by averaging profiler data from the benchmarked GPUs, and the Fermi-specific memory-efficiency correction is added after observing Fermi errors. The Skylake CPU projection also reuses coefficients fitted on Haswell without out-of-sample microarchitecture validation, but that is extrapolation rather than circularity. Weighing the in-sample GPU fitting against the partly independent CPU/energy and multinode checks, a moderate partial circularity score of 5 is appropriate: some 'predictions on different architectures' reduce to parameters fitted to the same data used for verification.
Assumptions & free parameters
free parameters (6)
- GPU kernel characteristics (Piavg, Pmavg, Pcflavg, Geavg, Gravgn, instruction counts, IPCeavg) =
Table 9
- Fermi memory-efficiency correction =
1/4 * Ge if Ge < 40%, 2/3 * Ge if Ge >= 40%
- Per-loop Tmop/Tflop coefficients (V,X,Y,Z and U,S) =
Tables 30, 40, 41
- SH FLOP scale factor 0.833 =
0.833
- PKG and DRAM energy coefficients (U,S and X,Y) =
Tables 33-34, 42-43
- GPU power fraction S =
28.89% for Tesla K20, 58.71% for GeForce 970
assumptions (6)
- domain assumption Perfect overlap of computation and internode communication
- domain assumption GPU time is the sum of compute, transfer, and inactive components
- domain assumption Linear energy model E = W*eflop + Q*emop + T*P0
- ad hoc to paper Small computational domains can be ignored
- domain assumption Profiler-derived kernel characteristics transfer to other and future architectures
- domain assumption Calibrated energy coefficients generalize across CPUs
Cite this review
Pith. "Pith review of Projections of achievable performance for Weather & Climate Dwarfs, and for entire NWP applications, on hybrid architectures." pith.science (2026). https://pith.science/paper/Z5PRBMGJ
@misc{pith2026190806098,
author = {Pith},
title = {Pith review of: Projections of achievable performance for Weather & Climate Dwarfs, and for entire NWP applications, on hybrid architectures},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z5PRBMGJ}},
note = {Machine review of arXiv:1908.06098}
}
read the original abstract
This document is one of the deliverable reports created for the ESCAPE project. ESCAPE stands for Energy-efficient Scalable Algorithms for Weather Prediction at Exascale. The project develops world-class, extreme-scale computing capabilities for European operational numerical weather prediction and future climate models. This is done by identifying Weather & Climate dwarfs which are key patterns in terms of computation and communication (in the spirit of the Berkeley dwarfs). These dwarfs are then optimised for different hardware architectures (single and multi-node) and alternative algorithms are explored. Performance portability is addressed through the use of domain specific languages. This deliverable contains the description of the performance and energy models for the selected Weather & Climate dwarfs for different hardware architectures, multinode with GPU accelerators in particular. Presented performance models are extension to model provided in Deliverable 3.2. With some further enhancements, they are incorporated in the DCworms simulator. In particular, extended models allow to predict computational and energy performance on different architectures: single and multinodes, equipped with CPUs and GPUs accelerators. This allows to provide feasible performance projection at system scale.
Figures
Reference graph
Works this paper leans on
-
[1]
It is worth noting that profiler and simulator have been simplified not to include data exchange between GPU and CPU as well as computational time spent on CPU. 3.4 Implementation Taking into account ACRANABE2 dwarf and its kernels benchmarks, we propose model based on a set of counters provided by Nvidia CUDA profiler (see Table 7). ESCAPE 2018 D3.5 Proj...
work page 2018
-
[2]
ESCAPE 2018 D3.5 Projections of achievable performance for Weather & Climate Dwarfs and for NWP applications on hybrid architectures 39 6 Performance at scale This section outlines how DCworms simulator and aforementioned performance and energy models may be used to assess performance at scale. The simulator allows for estimating computational performance...
work page 2018
-
[5]
PKG and DRAM energy coefficients are presented in Table 42 andTable 43 respectively. CORES EPKG [J] EMPKG [J] U S IDLE 5.53 1 23.28 19.08 0.0427544 3.27 2 35.12 19.58 4 49.71 20.21 Table 42 PKG energy coefficients for BiFFT dwarf CORES EDRAM [J] EMDRAM [J] X Y IDLE 1.31 1 4.73 3.15 0.1814516 0.4144184 2 5.42 3.55 4 7.21 3.6 Table 43 DRAM energy coefficien...
-
[9]
Table 11 presents the proposed GPU model
Symbol Equation Description iavgn 𝑖 = ∑ 𝑖𝑥𝑥∈{𝑐𝑓,𝑚,𝑡,𝑓,𝑑,𝑏𝑐,𝑖} Sum of all instructions performed per thread (averaged between architectures and normalized to the size of the domain) Piavg 𝑃𝑖 = 1 − 𝑖 𝐼𝑒 ∗ 32 Percentage of time the kernel was inactive; Pmavg 𝑃𝑚 = 𝑖𝑚 𝐼𝑒 ∗ 32 Percentage of time the kernel was performing miscellaneous instructions Pcflavg 𝑃𝑐𝑓𝑙 ...
work page 2018
-
[13]
The computational complexity of FFT is ≈ O(N log N), where N is the cut off spectral truncation wavenumber. The Legendre transform has computational complexity of O(N2) which increases with horizontal resolution. Details on Spherical Harmonics dwarf can be found in Deliverable D1.1 (Section 4.1). ESCAPE 2018 D3.5 Projections of achievable performance for ...
work page 2018
-
[14]
as well as its quarter i.e. 100x100x80 (Tables 15 - 17). For other sizes only the cumulative time of dwarf execution was compared (Tables 18 - 20). The comparison is presented only in tabular form for clarity. Presented difference is the quotient of the simulated and profiling time minus one. Square error is computed on the values of difference. Figure 10...
work page 2018
-
[17]
We may then use multinode model proposed in Section 5.6
We assume that computational domain is equally distributed over nodes, so that W (amount of computations to be performed) and Q (amount of data to be sent within all levels of memory within a node) is also equally divided between nodes. We may then use multinode model proposed in Section 5.6. We use DCworms simulator to project performance and energy effi...
work page 2018
-
[18]
microarchitectures tells us that we should expect better time -to-solution for this new microarchitecture. Figure 17 Cache-aware roofline for Intel Skylake 6148 ESCAPE 2018 D3.5 Projections of achievable performance for Weather & Climate Dwarfs and for NWP applications on hybrid architectures 42 Figure 18 Cache-aware roofline for Intel E5-2697v3 DCworms s...
work page 2018
Show all 20 references
-
[19]
However, please have in mind that such simulation does not take into account internode communication, thus real differences between runs on different architectures may differ. ESCAPE 2018 D3.5 Projections of achievable performance for Weather & Climate Dwarfs and for NWP appli...
2018
-
[20]
Further details of the Bi - Fourier spectral transform and the implementation of the dwarf can be found in Deliverable D1.1 (Section 4.4). Figure 20 Gridpoint distribution of a rectangular LAM domain over 48 MPI tasks ESCAPE 2018 D3.5 Projections of achievable performance for ...
2018
-
[21]
BiFFT is run on many CPU nodes, while ACRANEB2 uses GPU accelerators
The projection is made ESCAPE 2018 D3.5 Projections of achievable performance for Weather & Climate Dwarfs and for NWP applications on hybrid architectures 53 on a hybrid architecture. BiFFT is run on many CPU nodes, while ACRANEB2 uses GPU accelerators. We made following arch...
2018
-
[22]
Applying the roofline model
Ofenbeck, G, R Steinmann, V Caparros, D G Spampinato, and M Puschel. "Applying the roofline model." Performance Analysis of Systems and Software (ISPASS), 2014: 76-85. J. W. Poulsen and P. Berg.Tuning the implementation of the radiation scheme ACRANEB2. Tech. rep. 17-22. Danis...
2014
-
[23]
To correlate the processing unit with a dwarf, different coefficients are to be found as described in Deliverable D3.2
ESCAPE 2018 D3.5 Projections of achievable performance for Weather & Climate Dwarfs and for NWP applications on hybrid architectures 46 Figure 23 BiFFT most time-consuming loops As seen in 6.1.2.1, computational intensity does not change for different testcases, number of fiel...
2018
-
[30]
The computational domain is divided between nodes, so that amount of work W and moved data Q lowers when adding more nodes – it is divided between MPI processes (i.e
SH TCO639 loop V X Y Z U ledir->dgemm_327 0.0381 0.1097 0.0201 0.0027 0.1988 leinv->dgemm_315 0.4100 5.5113E-05 0 0.9612 0.2683 Trmtol_134 2.4472E-05 1.8389E-05 1.8705E-05 0.3637 - Trltom_130 2.4471E-05 1.8389E-05 1.8705E-05 0.3582 - ftinv_ctl->fourier_in_54 2.1605E-07 0 0.391...
1988
-
[35]
Modelled energy for SH dwarf - The maximum difference is 7.7%; - The minimum difference is 0.6%; - The mean difference is 7.7%; - The standard deviation is 3.24%
cores Epkg [J] Edram [J] Time [s] Epkg model [J] Edram model [J] Epkg measured [J] Edram measured [J] Total energy model [J] Total energy measured [J] idle 31.82 3.71 1 44.89 11.1 139.9 5898.49 840.61 5614 1085 6739.11 6699 2 54.23 14.51 71.2 3389.71 518.67 3124 624.8 3908.38 ...
1955
-
[37]
Here we provide cache -aware roofline graphs for Skylake for comparison to older microarchitectures
CORES L1 [GB/S] L2 [GB/S] L3 [GB/S] DRAM [GB/S] PERFORMANCE [GFLOP/S] 1 123.78 77.19 33.41 26.61 76.8 2 24.,2 153.93 65.94 53.81 153.6 4 494.43 307.77 131.26 84.25 307.2 8 988.66 615.82 262.63 150.33 614.4 16 1976.89 1231.29 512.84 261.01 1228.8 20 2470.72 1538.71 633.32 292.8...
1976
-
[274]
Time-to-solution decreases with number of nodes used, but energy-to-solution remains the same
Figure 16 presents the results. Time-to-solution decreases with number of nodes used, but energy-to-solution remains the same. This is possible because our model makes an assumption, that internode and intranode communication is perfectly overlapped by computations. However, t...
2018
-
[970]
Exploiting hardware performance counters with flow and context sensitive profiling
Indices denote number of nodes. DCworms selected the Intel Skyla ke 6148 and Nvidia GeForce 970 as a best combination that minimises two criteria: time-to-solution and energy-to-solution. Figure 30 presents simulation results of different CPU+GPU configurations on one graph. E...
2018
-
[2009]
The performance of a simple von Neumann architecture that contains two levels of memory hierarchy can be predicted with the Roofline mod el
The application is modelled as a ratio of arithmetic operations to number of bytes sent through the memory hierarchy. The performance of a simple von Neumann architecture that contains two levels of memory hierarchy can be predicted with the Roofline mod el. The model can be e...
2010
-
[2014]
-O -xHost
to model energy on dual multi -core CPU with three levels of cache hierarchy. In this approach, the dynamic power management was modelled as a second degree polynomial, based on real benchmark data, that scales linearly with the number of active cores up t o the saturation poi...
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.