pith. sign in

arxiv: 2605.22393 · v1 · pith:P5KDCLW5new · submitted 2026-05-21 · 💻 cs.DC

Nf-PEAK: Process-Based Energy Attribution for Nextflow Workflows on Kubernetes Clusters

Philipp Thamm , Somayeh Mohammadi , Kathleen West , Knut Reinert , Lauritz Thamsen , Ulf Leser This is my paper

Pith reviewed 2026-05-22 04:11 UTC · model grok-4.3

classification 💻 cs.DC
keywords energy attributionNextflow workflowsKubernetesRAPL countersprocess-based monitoringcontainerized energyworkflow optimizationsustainable computing
0
0 comments X

The pith

Nf-PEAK attributes CPU and DRAM energy to individual Nextflow tasks on Kubernetes by mapping pods to processes and applying a non-linear credit model to node-level RAPL data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces Nf-PEAK to solve the problem of estimating energy use at the level of single tasks inside scientific workflows that run on shared Kubernetes clusters. The method identifies workflow pods, links them to host processes through cgroup metadata, samples RAPL counters along with per-process performance data, and then distributes the node energy via a non-linear credit model before rolling results up to task level. A reader would care because workflows contain very different tasks whose individual energy footprints can now be measured and reduced separately, which matters for both running costs and environmental impact on multi-tenant clusters where direct measurement is blocked. The evaluation uses three nf-core workflows and shows the approach stays accurate when other jobs compete for CPU resources.

Core claim

Nf-PEAK identifies workflow pods, maps them to host processes via cgroup metadata, samples RAPL and per-process performance counters, and applies a non-linear energy-credit model to attribute CPU-package and DRAM energy to individual processes and Nextflow tasks. On a Kubernetes cluster the method reaches an average Mean Absolute Percentage Error of 6.6 percent in isolated runs and 10.9 percent when an unrelated workload saturates 8 of 32 hardware threads per node; accuracy remains stable from 2 to 8 nodes and is lower than that of the Kubernetes tool Kepler, especially under co-located load.

What carries the argument

Nf-PEAK, the containerized pipeline that combines cgroup-based pod-to-process mapping with a non-linear energy-credit model to apportion node-level RAPL readings to specific workflow tasks.

If this is right

  • Workflow developers can locate and rewrite the most energy-heavy tasks inside a pipeline.
  • Cluster operators obtain per-user energy accounting even when many jobs share the same nodes.
  • Nextflow and similar engines can add energy-aware scheduling that respects measured task costs.
  • Energy reports for scientific projects become finer-grained and therefore more actionable for sustainability goals.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same pod-mapping and credit-model approach could be adapted to other workflow engines that run in containers.
  • Combining the per-task numbers with carbon-intensity data would let users choose execution sites that minimize total emissions.
  • Extending the sampling to include GPU or I/O counters might allow attribution for workflows that are not purely CPU-bound.

Load-bearing premise

The non-linear energy-credit model together with cgroup-based pod mapping correctly divides node energy among concurrent processes despite resource contention and measurement noise.

What would settle it

Compare Nf-PEAK attributions against direct socket-level power meter readings on a test node while varying the amount and type of co-located CPU load.

Figures

Figures reproduced from arXiv: 2605.22393 by Kathleen West, Knut Reinert, Lauritz Thamsen, Philipp Thamm, Somayeh Mohammadi, Ulf Leser.

Figure 1
Figure 1. Figure 1: Overview of energy attribution with Nf-PEAK. Using pod information from Kubernetes, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Sensitivity of attributed Nf-PEAK energy and MAPE to the non-linearity exponent [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Actors and components of Nf-PEAK and their communication during attribution. Steps [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Isolated results on 2 nodes for RAPL and Nf-PEAK. For RNASeq, unattributed static [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Workflow-level energy and runtime on 4 nodes under increasing co-located CPU load. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Histogram of the energy consumption of each physical task in the Sarek workflow. Task [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
read the original abstract

Scientific workflows are pipelines of interdependent tasks. They are increasingly executed on shared Kubernetes clusters via workflow engines such as Nextflow. Their energy consumption matters for both cost and sustainability. It is necessary to examine and optimize workflow tasks individually, because they can be very heterogeneous. However, estimating task-level energy on clusters is difficult: Intel RAPL counters report only node-level energy, access to counters and host process information is typically restricted, and concurrent workloads introduce resource contention and measurement noise. We present Nf-PEAK, a containerized method to attribute CPU-package and DRAM energy to individual processes and Nextflow tasks. Nf-PEAK (i) identifies workflow pods, (ii) maps pods to host processes via cgroup metadata, (iii) samples RAPL and per-process performance counters, and (iv) applies a non-linear energy-credit model before aggregating results at task level. On a Kubernetes cluster, we evaluate three nf-core workflows under controlled co-located CPU load. Nf-PEAK reaches an average Mean Absolute Percentage Error of 6.6% in isolated runs and 10.9% when an unrelated workload saturates 8 of 32 hardware threads per node, and remains stable across 2, 3, 4, and 8 nodes. Compared to the state-of-the-art Kubernetes tool Kepler, Nf-PEAK yields lower error on average, particularly under co-located load.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Nf-PEAK, a containerized method for attributing CPU-package and DRAM energy (via RAPL) to individual processes and Nextflow tasks on Kubernetes clusters. It identifies workflow pods, maps them to host processes using cgroup metadata, samples RAPL and performance counters, and applies a non-linear energy-credit model to aggregate results at the task level. Evaluation on three nf-core workflows under controlled co-located CPU load reports average MAPE of 6.6% in isolated runs and 10.9% when an unrelated workload saturates 8 of 32 threads per node, with stability across 2–8 nodes and lower average error than the Kepler tool.

Significance. If the attribution accuracy holds under contention, the work would be significant for enabling task-level energy optimization in scientific workflows on shared clusters, supporting sustainability goals in distributed computing. Credit is due for the use of real hardware, external co-located workloads, and direct comparison to an existing Kubernetes tool rather than purely synthetic benchmarks.

major comments (2)
  1. [Abstract] Abstract: The 10.9% MAPE figure for co-located runs presupposes an independent ground-truth measure of each task's true energy consumption. If this reference is constructed by scaling or subtracting from isolated-run baselines, it does not account for contention-induced changes in per-task execution time, frequency scaling, and instantaneous power draw; the non-linear credit model is intended to compensate, but without an orthogonal validation (e.g., fine-grained per-process metering), the error metric risks understating real attribution error.
  2. [Abstract] Abstract: The non-linear energy-credit model is central to handling contention yet is described only at a high level with no equations, parameter definitions, or pseudocode; combined with the absence of error bars and raw data in the reported MAPE figures, this leaves the quantitative claims only moderately supported.
minor comments (1)
  1. [Abstract] The abstract would be clearer if it named the three specific nf-core workflows and the exact node hardware configuration used in the experiments.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments and for recognizing the potential significance of Nf-PEAK for task-level energy optimization in shared Kubernetes environments. We address each major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The 10.9% MAPE figure for co-located runs presupposes an independent ground-truth measure of each task's true energy consumption. If this reference is constructed by scaling or subtracting from isolated-run baselines, it does not account for contention-induced changes in per-task execution time, frequency scaling, and instantaneous power draw; the non-linear credit model is intended to compensate, but without an orthogonal validation (e.g., fine-grained per-process metering), the error metric risks understating real attribution error.

    Authors: We agree that establishing an independent ground truth under contention is challenging and that our evaluation relies on isolated-run baselines as the reference for each task's energy. The non-linear credit model, which incorporates per-process performance counters sampled during co-located execution, is intended to adjust attributions for contention effects such as frequency scaling and shared resource usage. However, we acknowledge that this does not constitute fully orthogonal validation (e.g., via dedicated per-process power meters). In the revision we will expand the evaluation section to explicitly discuss this limitation, include a sensitivity analysis of the credit model under varying contention levels, and add a forward-looking statement on future hardware-based validation. revision: partial

  2. Referee: [Abstract] Abstract: The non-linear energy-credit model is central to handling contention yet is described only at a high level with no equations, parameter definitions, or pseudocode; combined with the absence of error bars and raw data in the reported MAPE figures, this leaves the quantitative claims only moderately supported.

    Authors: The full manuscript (Section 3) provides the algorithmic description of the credit model, including the mapping from cgroup metadata to host processes and the aggregation at task level. To strengthen the presentation, we will add the explicit mathematical formulation of the non-linear credit function, definitions of all parameters (e.g., performance-counter weights and normalization constants), and pseudocode in a dedicated subsection. We will also augment the results figures with error bars (standard deviation across repeated runs) and release the raw per-task energy traces and MAPE calculations in a public repository linked from the paper. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical attribution method evaluated independently on hardware

full rationale

The paper describes a practical pipeline (pod identification via Kubernetes metadata, cgroup-based process mapping, RAPL and performance counter sampling, followed by a non-linear energy-credit model) and reports MAPE from controlled experiments on real clusters with isolated and co-located workloads. These results are compared directly to Kepler and measured across node counts; nothing in the abstract or described method reduces the reported accuracy figures to a fitted parameter, self-definition, or self-citation chain by construction. The evaluation uses external hardware measurements and an unrelated saturating workload, keeping the central claims independent of the attribution equations themselves.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no explicit free parameters, axioms, or invented entities; the non-linear credit model is mentioned but its internal parameters and assumptions are not detailed.

pith-pipeline@v0.9.0 · 5805 in / 1225 out tokens · 39679 ms · 2026-05-22T04:11:46.591536+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 1 internal anchor

  1. [1]

    Kepler: A Framework to Calculate the Energy Consump- tion of Containerized Applications

    Marcelo Amaral, Huamin Chen, Tatsuhiro Chiba, Rina Nakazawa, Sunyanan Choochotkaew, Eun Kyung Lee, and Tamar Eilam. Kepler: A Framework to Calculate the Energy Consump- tion of Containerized Applications. In2023 IEEE 16th International Conference on Cloud Computing (CLOUD), pages 69–71, Chicago, IL, USA, July 2023. IEEE. 13https://github.com/hubblo-org/sc...

    work page 2023

  2. [2]

    Process-Based Efficient Power Level Exporter

    Marcelo Amaral, Huamin Chen, Tatsuhiro Chiba, Rina Nakazawa, Sunyanan Choochotkaew, Eun Kyung Lee, and Tamar Eilam. Process-Based Efficient Power Level Exporter. In2024 IEEE 17th International Conference on Cloud Computing (CLOUD), pages 456–467, Shen- zhen, China, July 2024. IEEE

    work page 2024

  3. [3]

    Carbontracker: Tracking and predicting the carbon footprint of training deep learning models,

    Lasse F. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontracker: Track- ing and Predicting the Carbon Footprint of Training Deep Learning Models, July 2020. arXiv:2007.03051 [cs]

    work page arXiv 2020

  4. [4]

    Pay-back time: Increasing electricity prices and decreasing costs make renewable energy competitive.Energy Policy, 199:114523, April 2025

    Taimyra Batz Li˜ neiro and Felix M¨ usgens. Pay-back time: Increasing electricity prices and decreasing costs make renewable energy competitive.Energy Policy, 199:114523, April 2025

    work page 2025

  5. [5]

    Kubernetes Scheduling: Taxonomy, Ongoing Issues and Challenges.ACM Computing Surveys, 55(7):1–37, July 2023

    Carmen Carri´ on. Kubernetes Scheduling: Taxonomy, Ongoing Issues and Challenges.ACM Computing Surveys, 55(7):1–37, July 2023

    work page 2023

  6. [6]

    Hanebutte, Rahul Khanna, and Christian Le

    Howard David, Eugene Gorbatov, Ulf R. Hanebutte, Rahul Khanna, and Christian Le. RAPL: memory power estimation and capping. InProceedings of the 16th ACM/IEEE international symposium on Low power electronics and design, pages 189–194, Austin Texas USA, August

    work page

  7. [7]

    AI, Data Centers and Energy Demand: Reassessing and Exploring the Trends.Ifri Papers, February 2025

    Laure de Roucy-Rochegonde and Adrien Buffard. AI, Data Centers and Energy Demand: Reassessing and Exploring the Trends.Ifri Papers, February 2025

    work page 2025

  8. [8]

    Nextflow enables reproducible computational workflows.Nature Biotechnology, 35(4):316–319, April 2017

    Paolo Di Tommaso, Maria Chatzou, Evan W Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame. Nextflow enables reproducible computational workflows.Nature Biotechnology, 35(4):316–319, April 2017

    work page 2017

  9. [9]

    Fellows Yates, Thiseas C

    James A. Fellows Yates, Thiseas C. Lamnidis, Maxime Borry, Aida Andrades Valtue˜ na, Zandra Fagern¨ as, Stephen Clayton, Maxime U. Garcia, Judith Neukamm, and Alexander Peltzer. Reproducible, portable, and efficient ancient genome reconstruction with nf-core/eager.PeerJ, 9:e10947, March 2021

    work page 2021

  10. [10]

    SmartWatts: Self-Calibrating Software-Defined Power Meter for Containers

    Guillaume Fieni, Romain Rouvoy, and Lionel Seinturier. SmartWatts: Self-Calibrating Software-Defined Power Meter for Containers. In2020 20th IEEE/ACM International Sym- posium on Cluster, Cloud and Internet Computing (CCGRID), pages 479–488, May 2020. arXiv:2001.02505 [cs]

    work page arXiv 2020

  11. [11]

    Blair, and Adrian Friday

    Charlotte Freitag, Mike Berners-Lee, Kelly Widdicks, Bran Knowles, Gordon S. Blair, and Adrian Friday. The real climate and transformative impact of ICT: A critique of estimates, trends, and regulations.Patterns, 2(9):100340, September 2021

    work page 2021

  12. [12]

    Examining the Challenges of Scientific Workflows.Computer, 40(12):24–32, December 2007

    Yolanda Gil, Ewa Deelman, Mark Ellisman, Thomas Fahringer, Geoffrey Fox, Dennis Gan- non, Carole Goble, Miron Livny, Luc Moreau, and Jim Myers. Examining the Challenges of Scientific Workflows.Computer, 40(12):24–32, December 2007

    work page 2007

  13. [13]

    An Energy Efficiency Feature Survey of the Intel Haswell Processor

    Daniel Hackenberg, Robert Schone, Thomas Ilsche, Daniel Molka, Joseph Schuchart, and Robin Geyer. An Energy Efficiency Feature Survey of the Intel Haswell Processor. In2015 IEEE International Parallel and Distributed Processing Symposium Workshop, pages 896–904, Hyderabad, India, May 2015. IEEE

    work page 2015

  14. [14]

    EnergAt: Fine-Grained Energy Attribution for Multi-Tenancy.Energy Informatics Review, 4(3), 2024

    Hongyu H` e, Michal Friedman, and Theodoros Rekatsinas. EnergAt: Fine-Grained Energy Attribution for Multi-Tenancy.Energy Informatics Review, 4(3), 2024

    work page 2024

  15. [15]

    Nurminen, and Zhonghong Ou

    Kashif Nizam Khan, Mikael Hirki, Tapio Niemi, Jukka K. Nurminen, and Zhonghong Ou. RAPL in Action: Experiences in Using RAPL for Power Measurements.ACM Transactions on Modeling and Performance Evaluation of Computing Systems, 3(2):1–26, June 2018

    work page 2018

  16. [16]

    Quantifying the Carbon Emissions of Machine Learning

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the Carbon Emissions of Machine Learning, November 2019. arXiv:1910.09700 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  17. [17]

    Scientific Work- flows: Business as Usual? In Umeshwar Dayal, Johann Eder, Jana Koehler, and Hajo A

    Bertram Lud¨ ascher, Mathias Weske, Timothy McPhillips, and Shawn Bowers. Scientific Work- flows: Business as Usual? In Umeshwar Dayal, Johann Eder, Jana Koehler, and Hajo A. Reijers, editors,Business Process Management, volume 5701, pages 31–47. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009. Series Title: Lecture Notes in Computer Science. 14

    work page 2009

  18. [18]

    Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso, and Sven Nahnsen

    Philip A. Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso, and Sven Nahnsen. The nf-core framework for community-curated bioinformatics pipelines.Nature Biotechnology, 38(3):276–278, March 2020

    work page 2020

  19. [19]

    An Introduction to Docker and Analysis of its Performance.IJCSNS International Journal of Computer Science and Network Security, 17(3):228–235, 2017

    Babak Bashari Rad, Harrison John Bhatti, and Mohammad Ahmadi. An Introduction to Docker and Analysis of its Performance.IJCSNS International Journal of Computer Science and Network Security, 17(3):228–235, 2017

    work page 2017

  20. [20]

    Martin Sudmanns, Dirk Tiede, Hannah Augustin, and Stefan Lang. Assessing global Sentinel- 2 coverage dynamics and data availability for operational Earth observation (EO) applications using the EO-Compass.International Journal of Digital Earth, 13(7):768–784, July 2020

    work page 2020

  21. [21]

    Tanenbaum

    Maarten Van Steen and Andrew S. Tanenbaum. A brief introduction to distributed systems. Computing, 98(10):967–1009, October 2016

    work page 2016

  22. [22]

    METRION: A Framework for Ac- curate Software Energy Measurement, December 2025

    Benjamin Weigell, Simon Hornung, and Bernhard Bauer. METRION: A Framework for Ac- curate Software Energy Measurement, December 2025. arXiv:2512.06806 [cs]. 15

    work page arXiv 2025