REVIEW 3 major objections 4 minor 5 references
Report on workflow analysis for specific LAM applications
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The forecast step of the RMI-EPS weather ensemble consumes up to 99% of the suite's energy, so energy optimizations there transfer almost one-for-one.
desk verdict Useful measured energy breakdown of an operational ensemble suite, but the headline 'up to 99%' overreaches; the data supports ~97% on measured nodes, with unmeasured parts excluded. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing measurement setup consists of PAPI Cray Power Management counters on the compute nodes, wrapped in a script that reads energy in joules and elapsed time in milliseconds before and after each job. Jobs are grouped into five categories, and energy is expressed as a linear formula in the number of control members $n$ and total members $N$: total energy $\approx 230.9\,n + 6639.6\,N + 1.7$ kJ, giving about $146{,}000$ kJ for $n=2$, $N=22$. This formula, together with the separate contribution of the Forecast job, is what carries the 99% and $3/2$ conclusions.
What would settle it
Measure the energy used by the unmeasured pre- and post-processing stages, communication, and cooling during a full 36-hour RMI-EPS run and add those totals to the measured compute-node energy; if the forecast's share drops well below 99%, the proportional-transfer conclusion would not hold.
Extended reading notes
Core claim
The paper's central discovery is that the Forecast job of the RMI-EPS ensemble prediction suite consumes up to 99% of the total energy of the entire workflow when measured on the compute nodes, while accounting for only about 35% of wall-clock time. As a direct consequence, an energy optimization that reduces the Forecast's energy by a fraction $x$ reduces total suite energy by almost the same fraction, whereas the wall-clock speedup of the whole suite from any Forecast-only optimization is bounded by about $1/(1-0.35)\approx 3/2$.
Load-bearing premise
The 99-percent energy figure rests on treating the compute-node energy measured during one particular run as the whole suite's energy bill, leaving out pre- and post-processing jobs on another cluster, communication, and cooling.
Editorial extensions
If this is right
- Energy optimizations applied to the forecast kernel will transfer almost proportionally to the whole suite: a 20% forecast energy cut yields roughly a 20% total energy cut.
- Wall-clock time cannot be improved by more than about a factor of $3/2$ through forecast-only optimization, because the forecast is only about 35% of elapsed time; further gains require optimizing data assimilation, lateral boundary conditions, or post-processing.
- Increasing the number of perturbed ensemble members keeps the forecast dominant in energy; the data-assimilation share shrinks because upper-air assimilation runs only on control members.
- The Kronos synthetic model, once built, can predict I/O and MPI behavior under scaled workloads, but it does not profile energy and cannot model future hardware.
Reading between the lines
- The paper leaves implicit that the $3/2$ wall-clock cap is an Amdahl-type bound derived from the 35% forecast share; applying the same arithmetic to other categories would identify which non-forecast stage is next most rewarding.
- A testable extension would be to run the same measurement on a GPU-accelerated version of the forecast: if the forecast energy share drops below 90%, the suite-level energy strategy must shift toward other stages.
- The reported energy formula allows extrapolation to intermediate ensemble sizes without re-running the suite, which could help planning energy budgets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This ESCAPE deliverable reports a workflow analysis and combined energy/wall-clock time measurement campaign for the RMI-EPS limited-area ensemble prediction suite. It defines five job categories, documents the cca/ecgate workflow, and presents Table 2 with per-job energy and wall-clock measurements. On this basis it concludes that the Forecast job dominates suite energy consumption, with an advertised share of up to 99% of total energy, and that the maximum theoretical speedup from optimizing the Forecast alone is about 3/2 in wall-clock time. The report also describes a Kronos-based synthetic workload model of the suite, presented explicitly as a proof of concept without simulation results.
Significance. If the quantitative claims are established, the paper has direct practical value for the ESCAPE project: it identifies where energy optimization effort should be concentrated and cautions that wall-clock optimization requires looking beyond the Forecast. The paper's strengths include a detailed, job-by-job description of a real operational ensemble workflow, the use of direct PAPI Cray PM counter measurements rather than estimates, and a clear separation of wall-clock and energy accounting, including the distinction between control and perturbed members. The Kronos proof of concept is honestly labeled as a workflow description without simulation results. The load-bearing quantitative claims, however, are currently stated more strongly than the measurements support, and one internal inconsistency in the speedup statement needs correction.
major comments (3)
- [Abstract / Executive Summary / Section 3.1 / Section 4 / Table 2] The headline claim that the Forecast job is responsible for 'up to 99% of the total energy consumption' is not supported by the presented measurements. Section 3.1 states that ecgate pre-processing and final post-processing are excluded from the measurements, and Section 4 states that the measurements 'only represent the consumption of energy due to the computational work of the compute nodes', with communication and cooling unmeasured. Table 2 additionally covers only a selection of 'most demanding' cca jobs, omitting several listed jobs such as Prepare_cycle, Climate, Prep_ini_surfex, Oulan, and Listen2Forecast from the measured total. Recomputing from Table 2's own formula for n=2, N=22 gives Forecast energy = 0.5 * 22 * (4957.4 + 7982.3) = 142,337 kJ and total energy ≈ 230.9 * 2 + 6639.6 * 22 + 1.7 = 146,535 kJ, i.e. a Forecast share of about 97.1%, not 99%. The claim should be revised to refer to the measured compute-node subset or to about 97%, and the excluded contributions need at least order-of-magnitude bounds before a whole-suite 'up to 99%' statement can be made.
- [Section 4, near 'Summarizing the results'] The maximum theoretical speedup from Forecast-only wall-clock optimization is stated inconsistently. The abstract, executive summary, and conclusion give a factor of about 3/2, which is consistent with Amdahl's law applied to the reported 35% Forecast wall-clock share (1/(1-0.35) ≈ 1.54). However, Section 4 states that 'the expected maximum theoretically achievable speedup for the RMI-EPS suite would be about 2'. This is not a minor wording difference: a factor of 2 corresponds to a Forecast wall-clock share of 50%, which is the value quoted for the perturbed members, not the suite-wide average. The text should either justify the factor of 2 with the specific scenario it applies to or correct it to the suite-level value.
- [Section 4, Table 2] The paper provides no measurement uncertainty or repetition information for the energy values in Table 2, despite acknowledging that PAPI Cray PM counters update at about 10 Hz and that jobs run on the ns queue 'probably have energy contributions that are overestimated' due to node contamination. Since the central quantitative conclusion is a ratio (Forecast share of total energy), the absence of error bars or a sensitivity analysis makes it impossible to assess whether the difference between the advertised 99% and the recomputed 97.1% is meaningful. At minimum, the authors should report the number of repeated measurements, the observed run-to-run variability, or a conservative error estimate for the ratios.
minor comments (4)
- [Section 3.3 vs Section 3.5, Table 1] The job name is given as 'Interpol_ec_sst' in Section 3.3 but as 'interpol_efc_sst' in Table 1; the spelling should be harmonized.
- [Section 4, Table 2] The table's 'Total for n=2 and N=22' is rounded to ≈146000 kJ, while the stated formula evaluates to about 146,535 kJ; this is a large rounding step for a headline number and should be made consistent.
- [Section 5.3] The text refers to 'about 20 most demanding jobs' selected for Kronos profiling, but the accompanying description and Figure 9 do not give a precise count; a list matching the profiled set would improve reproducibility.
- [Section 4, Figure 7 caption] The caption says the comparison is between 20 and 40 perturbed members, which with 2 control members yields N=22 and N=42; this is correct, but the parenthetical in the body text says 'cases (n=2, N=22) and (n=2, N=42)', which is redundant and could be simplified for clarity.
Circularity Check
No circularity: this is a measurement report whose headline percentages and Amdahl-style speedup bound follow from direct measurements, not from fitted inputs or self-citations.
full rationale
The report's central claims are empirical: PAPI Cray PM counters were used to measure energy and wall-clock time of selected RMI-EPS jobs on cca, and Table 2 records those measurements. The 'up to 99%' figure for the Forecast share is computed by summing the measured energy contributions in Table 2, and the 'maximum theoretical speed-up ... about 3/2' follows from the measured wall-clock fraction via Amdahl's law. There is no fitted parameter being renamed as a prediction, no quantity defined in terms of another quantity it is supposed to explain, and no load-bearing self-citation chain. The two cited references (Kronos and Smet 2017) are background material and do not carry the derivation. The report does contain a significant measurement-scope limitation: Section 4 states that 'the measurements only represent the consumption of energy due to the computational work of the compute nodes' and that communication and cooling were not measured, while Section 3.1 notes that ecgate pre- and post-processing stages were excluded from the energy measurements. These limitations weaken the support for claiming 99% of the 'total energy consumption' of the entire suite, but they are a correctness/coverage issue, not circularity. The derivation chain is self-contained: measurements in, arithmetic out. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Energy consumed on cca compute nodes is a sufficient proxy for total suite energy consumption, with ecgate pre-processing and post-processing treated as negligible.
- domain assumption The jobs selected in Table 2 represent enough of the suite workload to draw total-energy and total-time conclusions.
- domain assumption Wall-clock time can be averaged over members and energy can be summed over members assuming overlap does not alter the conclusion.
- standard math Amdahl's law applies to the measured wall-clock fractions for estimating the maximum speedup from forecast optimizations.
Cite this review
Pith. "Pith review of Report on workflow analysis for specific LAM applications." pith.science (2026). https://pith.science/paper/AFSLFPUJ
@misc{pith2026190806116,
author = {Pith},
title = {Pith review of: Report on workflow analysis for specific LAM applications},
year = {2026},
howpublished = {\url{https://pith.science/paper/AFSLFPUJ}},
note = {Machine review of arXiv:1908.06116}
}
read the original abstract
This document is one of the deliverable reports created for the ESCAPE project. ESCAPE stands for Energy-efficient Scalable Algorithms for Weather Prediction at Exascale. The project develops world-class, extreme-scale computing capabilities for European operational numerical weather prediction and future climate models. This is done by identifying Weather & Climate dwarfs which are key patterns in terms of computation and communication (in the spirit of the Berkeley dwarfs). These dwarfs are then optimised for different hardware architectures (single and multi-node) and alternative algorithms are explored. Performance portability is addressed through the use of domain specific languages. In this deliverable we focus on the RMI-EPS ensemble prediction suite. We first provide a detailed report on the workflow of the suite in which 5 main categories of jobs are defined; pre-processing, lateral boundary conditions (LBCs), data assimilation, forecast and post-processing. Combined Energy and wall-clock time measurements of the entire RMI-EPS suite were performed. They indicate that the wall-clock times are relatively spread between the various defined job categories, with the forecast accounting for the largest fraction at about 35%. As far as energy consumption is concerned, the forecast part dwarfs everything else and is responsible for up to 99% of the total energy consumption. This means that energy optimizations for the forecast part will translate almost proportionally into optimizations of the whole suite, while the maximum theoretical speed-up due to forecast optimizations cannot exceed a factor of about 3/2. Therefore, in terms of energy consumption, optimizations should first focus on the forecast part. For wall-clock time performance gains, however, optimizations (and possibly additional dwarfs) can be considered for the categories outside of the forecast part.
Figures
Reference graph
Works this paper leans on
-
[1]
Profiling: the code to be modeled is profiled with the help of a set of profiling tools (in our case Darshan and IPM)
-
[2]
Data ingestion: the profiling results are turned into Kronos-specific data format files
-
[3]
Schedule generation: schedules for the modeled code are built based on the dependencies of any individual sub-jobs and the results of step 2
-
[4]
The complete RMI -EPS ensemble suite contains hundreds of individual jobs
Execution: these abstract schedules are used by the Kronos Executor which submits concrete synthetic jobs to the scheduling queue of a real HPC system. The complete RMI -EPS ensemble suite contains hundreds of individual jobs. Trying to adapt each script to enable profiling would be prohibitively time -consuming. We instead chose to focus on a set of the ...
work page 2018
-
[5]
This workload was small enough to be run on the lxg server at ECMWF. The run completed successfully, but due to time constraints, we do not include specific results in this deliverable. Ultimately, the main result is that we were able to model the RMI -EPS suite using Kronos proving that Kronos can be a useful tool for the NWP community to assist in the d...
work page 2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.