Application Failures and Machine Computational Efficiency

Bethany Lusch; Carlo Graziani; O. E. Bronson Messer

REVIEW 3 major objections 4 minor 15 references

Application Failures and Machine Computational Efficiency

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Optimal checkpoints leave 95.7% of Frontier's node-hours for science.

desk verdict A clean usage-domain reformulation of checkpoint optimization with a transparent but unaudited Frontier efficiency estimate; the math holds up, the headline number should not be taken as a measurement. read the letter →

arxiv 2608.05408 v1 pith:6PVOHGT2 submitted 2026-08-05 cs.DC

Carlo Graziani , Bethany Lusch , O. E. Bronson Messer This is my paper · ORCID

classification cs.DC

keywords high-performancecomputingcheckpointingnodefailurescomputationalefficiencynode-hoursmeanusagebetweenfaulttoleranceFrontiersupercomputer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Exascale machines fail often enough that resilience costs shape how much of a supercomputer actually goes to science. This paper argues that failure rates and checkpoint intervals should be measured in node-hours of usage rather than in wall-clock time, because a 2000-node job running one hour carries the same node-fault risk as a 1000-node job running two hours. On that basis it derives the machine computational efficiency $e = 1 - L/U_{\rm tot}$, the expected fraction of allocated usage left for computation after checkpointing, failure, and restart losses. Applied to one year of production jobs on the Frontier supercomputer, the model yields $e = 0.957$, meaning a facility could overprovision projects by $e^{-1}-1 = 4.5\%$ instead of compensating each failed job. A reader should care because this turns resilience accounting into a measurable, per-machine quantity that allocation policies can use.

What carries the argument

The central machinery is the per-unit-usage failure rate $R_0$ (failures per node-hour) and its reciprocal, the mean usage between failures (MUBF). From it the model derives the optimal checkpoint usage cadence via the condition $1-R_0u_c = \exp(-R_0(u_c+u_{\rm chk}))$, the expected usage lost to a job failure under a truncated exponential distribution, and the expected restart cost including repeated failed restarts. The efficiency itself is defined by $e=1-L/U_{\rm tot}$ with $U_{\rm tot}$ the total expected usage of a job schedule and $L$ the summed expected losses, split into checkpoint loss $l_{\rm chk}$, failure loss $l_F$, and restart loss $l_R$. The key scaling behavior is that global checkpoints to shared storage cost $O(N^2)$ in node-hours for a job of $N$ nodes, while node-local checkpoints cost $O(N)$, which drives the bandwidth sensitivity of the results.

What would settle it

Compare the model's predicted distribution of usage lost per failed job, a truncated exponential with rate $R_0$, against a year of actually recorded job failures on Frontier; if a substantial share of lost usage comes from hangs or collective failures that the truncated exponential cannot produce, or if the observed shortfall between allocated and science node-hours exceeds the predicted $L$, then the $e=0.957$ estimate overstates usable capacity.

Watch

Extended reading notes

Core claim

The paper's central claim is that the right unit of failure accounting on scientific supercomputers is usage (node-hours), not elapsed time, and that a single per-unit-usage failure rate $R_0$ (failures per node-hour) is enough to compute both an optimal checkpoint cadence and a machine-level efficiency. It updates the classical checkpoint-interval result to the usage domain, with optimality condition $1-R_0u_c = \exp(-R_0(u_c+u_{\rm chk}))$, and derives expected per-job losses from failures, restarts, and checkpoints. Summing these over a job schedule gives the computational efficiency $e=1-L/U_{\rm tot}$, and Eq. (5.1) turns it into an allocation rule: a project needing $U$ node-hours of science must be allocated $U_{\rm alloc}=U/e$. For one year of Frontier production jobs the paper finds $e=0.957$ with $l_{\rm chk}=0.012$, $l_F=0.028$, and $l_R=0.003$, so the facility would overprovision by $e^{-1}-1=4.5\%$; without prophylactic checkpointing, efficiency drops to $0.914$.

Load-bearing premise

The whole estimate rests on treating every job failure as a random, independent failure of a single compute node with a fixed rate per node-hour; collective failures, application hangs, and software-only faults are left out.

Editorial extensions

If this is right

A facility that knows its job schedule and $R_0$ can compute a defensible overprovisioning factor: a project needing $U$ node-hours of science should be allocated $U/e$.
For Frontier, optimal checkpointing keeps efficiency at 95.7%, so the facility needs only 4.5% overprovisioning; dropping prophylactic checkpoints entirely would raise that to 9.4%.
Checkpointing cadence matters: displacing all checkpoint intervals by a factor of about 5 in either direction costs roughly 5% efficiency, and a factor of 10 too frequent costs about 10%.
Storage bandwidth to a shared checkpoint target is a major lever: efficiency falls sharply as bandwidth drops, and this penalty is worse at higher failure rates because of the $O(N^2)$ scaling for large jobs.
As node failure rates rise on future machines, the model predicts efficiency declines, for example to about 90.9% at $R_0=10^{-4}$ and 88.6% at $R_0=2\times10^{-4}$ for a Frontier-like load, making resilient design a capacity-planning issue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If $R_0$ actually varies by application, as the paper suspects, per-application efficiency accounting would give different overprovisioning factors than the machine-wide value; this is an extension the paper does not work out.
Because collective failures, application hangs, and software-only faults are excluded, real efficiency on production machines is likely lower than the paper's estimate; the paper itself flags this gap.
The same usage-based accounting could be applied to AI training clusters, where an analogous effective-training-time metric exists, but replacing a homogeneous workload with a scientific-job mix would be the needed adaptation.
A natural test is to rerun the analysis with per-application failure rates estimated from job exit statuses on another year or another machine; the model's predictions for loss terms could then be checked against observed lost node-hours.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Jim,

The paper is worth your time if you care about HPC resilience economics. It moves the failure-rate conversation from time to node-hours, which is the right variable for heterogeneous job sizes, and it does so cleanly. The core checkpoint equation (3.3) is Daly's (21) reparametrized, and the limiting cases recover Young and Daly correctly. I checked the algebra on (2.4), (2.6), (3.3), and the per-job loss formulas in Section 4; it all hangs together. The genuinely new pieces are the per-job discrete loss model with the ceiling on checkpoint count, the machine-level efficiency aggregate e = 1 - L/U_tot, and the application to one year of Frontier's schedule. The O(N^2) shared-storage checkpoint penalty is a concrete and useful design observation.

The soft spots are where the reader and the stress-test put them. The headline e=0.957, and the 4.5% overprovision rule, rest on R0 deriving from node-failure logs, a global r=0.46 runtime rescale, and the exclusion of everything that is not a node-local Poisson failure. Hang jobs and collective failures are not in the model at all, and Section 8 says so explicitly. That is an honest limitation, but it is also the main unknown. l_F + l_R is only 0.031; an omitted hang loss of one or two percentage points moves the overprovision recommendation materially. So treat 0.957 as an estimate under the model's assumptions, not a measured quantity. The paper would be stronger with uncertainty bars on R0 and a sensitivity analysis over the hang/collective-failure share. The lack of job termination status data (they only use the schedule, not outcomes) also means the efficiency is not validated against actual job failures.

None of that undermines the framework. The math is correct, the code and data are public, and the authors are transparent about the gaps. The paper is a solid contribution to the HPC operations literature. I would send it to a serious referee; the referee should push on R0 estimation and the failure-model coverage, but the framework itself deserves publication. For my own reading group, yes, this is worth a session.

Referee Report

3 major / 4 minor

Summary. This paper proposes a usage-based framework for quantifying the impact of node-local application failures on scientific high-performance computing systems. The authors recast checkpoint optimization from the time domain to the usage domain, introducing a per-unit-usage failure rate R0 and deriving an optimal checkpoint cadence, per-job expected losses (checkpointing, failure, and restart), and a machine-level computational efficiency e = 1 - L/U_tot. The framework is applied to one year of Frontier job-schedule data, yielding e = 0.957 and a corresponding overprovisioning factor of 4.5%. The paper also presents sensitivity analyses of efficiency to checkpoint cadence, failure rate, and storage bandwidth. The derivations are algebraic and the code and data are publicly available.

Significance. The usage-domain reformulation of checkpointing is a well-motivated and useful contribution: it makes failure rates and checkpoint intervals consistent across jobs of different sizes, which is important for heterogeneous scientific workloads. The framework produces closed-form expressions that can be evaluated from schedules and failure logs, and the paper ships reproducible code and the underlying Frontier schedule. If the model limitations are addressed quantitatively, the framework gives facilities a principled basis for overprovisioning node-hours. The headline numerical result is conditional on the node-local Poisson failure model, and the paper honestly identifies this limitation in Section 8; with appropriate quantification of that limitation, the contribution would be solid. The transparency of the derivation, the explicit statement of approximations, and the public release of code and data are notable strengths.

major comments (3)

[§7.4 and §8] The headline result e = 0.957 and the implied overprovisioning factor e^{-1}-1 = 4.5% are presented in §7.4 as a property of Frontier, but they follow only under the node-local Poisson failure model. Section 8 concedes that collective failures and hangs are not included and that hung jobs consume usage until killed. Since the omitted effects are not quantified, the number cannot be read as an efficiency estimate for Frontier as a whole. The authors should add a quantitative sensitivity analysis with a range of additional hang or collective-failure rates and show their effect on e, or explicitly state in §7.4 that the figure is an illustrative estimate conditional on the node-local Poisson assumption.
[§7.2.1] The estimate R0 = 2.0e-5 (node-hours)^{-1} is stated without supporting statistics. The paper should report the number of node-failure events, the total node-hours on which the rate is based, and, ideally, a confidence interval or a sensitivity range. Because all efficiency and loss results in §7.4-§7.8 scale with R0, the absence of any uncertainty measure is a load-bearing gap.
[§7.3] The runtime rescaling factor r = 0.46 is applied globally to requested times, under the assertion that most jobs terminate normally. The paper states it does not use or know termination status, so this assertion is unverified. If a material fraction of short runtimes are due to failures or early exits, the rescaled schedule will misrepresent the intended workload. The authors should justify the claim with data or treat r as an uncertain parameter in the analysis.

minor comments (4)

[Abstract] The abstract contains a duplicated article: 'translate exactly into the the types of challenges'.
[§6] The GitHub repository is a strength, but a DOI or versioned release would make the code and data more citable and stable.
[§7.2.2] The assumption that all applications checkpoint 20% of node memory (M = 200 GB) is a significant simplification; consider reporting how the results change with M, since Figure 7.5 already illustrates sensitivity to the analogous parameter (bandwidth).
[§2.1] Equation (2.2) assumes all nodes contend for a single shared storage bandwidth, which is pessimistic for modern parallel file systems with striped writes; a brief caveat would improve generality.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the efficiency estimate is a forward computation from stated failure, storage, and schedule inputs, not a fitted quantity renamed as a prediction.

full rationale

The derivation is self-contained and forward. Section 5 defines efficiency as e = (U_Tot - L) / U_Tot, where each loss term is computed from the failure model developed in Sections 2-4. The model inputs are the per-usage failure rate R0 (estimated from node failure logs in Section 7.2.1), storage and checkpoint parameters (Sections 7.2.2-7.2.3), and a job schedule from Frontier (Section 7.3). The optimal checkpoint cadence is obtained by minimizing the expected loss L with respect to u_c in Section 3 (Eq. 3.3), and the resulting efficiency e = 0.957 in Section 7.4 is a computed output rather than an input. The only empirical parameter, R0, comes from external node-failure data and is not reverse-engineered from the efficiency. The paper explicitly concedes in Section 8 that it has only considered node-local application failures and has omitted hangs, collective failures, and software faults; however, an acknowledged incompleteness of the input model is a robustness or correctness limitation, not circularity. Citations to Daly and Young are external and are used only for the standard optimal-checkpoint result, transposed from the time domain to the usage domain; there is no load-bearing self-citation chain. No fitted value is renamed as a prediction, and no quantity is defined in terms of the target efficiency. The central claim therefore does not reduce to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The framework depends on a small number of estimated or assumed system parameters, one global runtime-rescale factor, and several explicit domain assumptions about the failure process and storage behavior. The paper discloses the main limitations; the ledger below enumerates them.

free parameters (3)

R0 (per-unit-usage failure rate) = 2.0e-5 (node-hours)^-1
Estimated from Frontier node failure logs; central input to all loss and efficiency calculations; acknowledged as a slight underestimate of the true application failure rate (Section 7.2.1).
M (average checkpoint size) = 200 GB per node
Assumed average checkpoint size (20% of node memory) shared across all applications; a significant approximation acknowledged in Section 7.2.2.
r (runtime rescale factor) = 0.46
Ratio of mean runtime to mean requested time used to rescale all requested job times; derived from the job database but applied globally (Section 7.3).

assumptions (5)

domain assumption Application failures are caused exclusively by node-local faults, with a constant rate R0 per node-hour; collective failures and software-only failures are negligible.
Used throughout Sections 2-7; flagged as a gap in Section 8.
domain assumption Failure times are exponentially distributed (Poisson process in usage), so memoryless truncated-exponential forms apply in Eqs (2.3)-(2.4).
Standard reliability assumption, not validated against Frontier failure-time data.
domain assumption All applications checkpoint the same memory amount M = 200GB and have uncontended read/write bandwidth Bw=Br=6TB/s.
Assumed in Sections 2.1 and 7.2.2; called a 'significant approximation' in the text.
domain assumption The INCITE/ALCC job schedule, with single-node jobs removed and requested times rescaled by r=0.46, represents a typical scientific HPC load.
Section 7.3; the choice affects the headline efficiency.
standard math Standard results for geometric, negative binomial, and truncated exponential distributions are applied without proof.
References to Balakrishnan and Nevzorov [1].

invented entities (1)

None
purpose: No new physical or conceptual entities are postulated.
The paper introduces the per-unit-usage failure rate R0 and MUBF as defined accounting quantities, but these are not independent postulated entities with falsifiable handles outside the model; they are definitions in units of node-hours.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Application Failures and Machine Computational Efficiency." pith.science (2026). https://pith.science/paper/6PVOHGT2

@misc{pith2026260805408,
  author       = {Pith},
  title        = {Pith review of: Application Failures and Machine Computational Efficiency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PVOHGT2}},
  note         = {Machine review of arXiv:2608.05408}
}

read the original abstract

We present a framework for evaluating uptime efficiency of Exascale-class scientific computers when application failure rates are appreciable. This is the situation that confronts current leadership-class scientific computing platforms and large AI training installations. What distinguishes scientific computing platforms is the heterogeneity of their applications. We argue that this diversity requires that failure rates and mean intervals between failures should be specified in terms of \emph{usage} (e.g. node-hours) rather than time, as is currently customary. We consider the usage loss terms due to failures, to checkpointing, and to restart costs, and update the framework of Daly (2006) allowing users to specify optimal checkpointing usage intervals that minimize such losses. We derive the machine computational efficiency, which specifies the expected fractional resource allocation that is available for scientific computation. We illustrate the methodology using one year of production runtime data from the \emph{Frontier} supercomputer at Oak Ridge National Laboratory.

Figures

Figures reproduced from arXiv: 2608.05408 by the authors.

**Figure 3.1.** Optimal checkpointing. The figures show the relation between the dimensionless quantities [PITH_FULL_IMAGE:figures/full_fig_p004_3_1.png] view at source ↗

**Figure 7.1.** Distribution of job characteristics from [PITH_FULL_IMAGE:figures/full_fig_p008_7_1.png] view at source ↗

**Figure 7.4.** Efficiency as a function of failure rate. [PITH_FULL_IMAGE:figures/full_fig_p009_7_4.png] view at source ↗

Figures from the paper (2 more)

**Figure 7.3.** Figure 7.3: Efficiency and relative losses as a function [PITH_FULL_IMAGE:figures/full_fig_p009_7_3.png]

**Figure 7.5.** Figure 7.5: Efficiency as a function of bandwidth to [PITH_FULL_IMAGE:figures/full_fig_p009_7_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

[1]

Balakrishnan and V

N. Balakrishnan and V. Nevzorov , A Primer on Statistical Distributions , Wiley, 2004, https://books.google.com/books?id=JIfk5kBdLGIC

work page 2004
[2]

Chen , Algorithm-based recovery for iterative methods without checkpointing , in Proceedings of the 20th international symposium on High performance distributed computing, 2011, pp

Z. Chen , Algorithm-based recovery for iterative methods without checkpointing , in Proceedings of the 20th international symposium on High performance distributed computing, 2011, pp. 73--84

work page 2011
[3]

J. T. Daly , A higher order estimate of the optimum checkpoint interval for restart dumps , Future generation computer systems, 22 (2006), pp. 303--312

work page 2006
[4]

S. Di, H. Guo, E. Pershey, M. Snir, and F. Cappello , Characterizing and understanding hpc job failures over the 2k-day life of ibm bluegene/q system , in 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), IEEE, 2019, pp. 473--484

work page 2019
[5]

Erben and E

A. Erben and E. Erdil , Hardware failures won't limit ai scaling , 2024, https://epoch.ai/blog/hardware-failures-wont-limit-ai-scaling. Accessed: 2025-03-21

work page 2024
[6]

Heitmann, T

K. Heitmann, T. Uram, N. Frontiere, S. Habib, A. Pope, S. Rizzi, and J. Insley , The new worlds simulations: Large-scale simulations across three cosmologies , 2024, https://arxiv.org/abs/2406.07276

arXiv 2024
[7]

Kokolis, M

A. Kokolis, M. Kuchnik, J. Hoffman, A. Kumar, et al. , Revisiting reliability in large-scale machine learning research clusters , in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), IEEE, 2025, pp. 1259--1274

work page 2025
[8]

Nguyen-Cong, J

K. Nguyen-Cong, J. T. Willman, J. M. Gonzalez, A. S. Williams, et al. , Extreme metastability of diamond and its transformation to the bc8 post-diamond phase of carbon , The Journal of Physical Chemistry Letters, 15 (2024), pp. 1152--1160

work page 2024

Show all 15 references

[9]

Svobodova , Resilient distributed computing , IEEE Transactions on Software Engineering, (1984), pp

L. Svobodova , Resilient distributed computing , IEEE Transactions on Software Engineering, (1984), pp. 257--268

1984
[10]

Taherin, T

A. Taherin, T. Patel, G. Georgakoudis, I. Laguna, and D. Tiwari , Examining failures and repairs on supercomputers with multi-gpu compute nodes , in 2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), IEEE, 2021, pp. 305--313

2021
[11]

Taylor, P

M. Taylor, P. M. Caldwell, L. Bertagna, C. Clevenger, et al. , The simple cloud-resolving e3sm atmosphere model running on the frontier exascale system , in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '23, Ne...

2023
[12]

Verma, J

A. Verma, J. E. Huffman, A. Rodriguez, M. Conery, et al. , Diversity and scale: Genetic architecture of 2068 traits in the va million veteran program , Science, 385 (2024), p. eadj1182

2024
[13]

J. W. Young , A first order approximation to the optimum checkpoint interval , Communications of the ACM, 17 (1974), pp. 530--531

1974
[14]

X. Zhao, Z. Jiang, and J. Ma , A survey of deep anomaly detection for system logs , in 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, 2022, pp. 1--8

2022
[15]

Zheng, L

Z. Zheng, L. Yu, W. Tang, Z. Lan, R. Gupta, N. Desai, S. Coghlan, and D. Buettner , Co-analysis of ras log and job log on blue gene/p , in 2011 IEEE international parallel & distributed processing symposium, IEEE, 2011, pp. 840--851

2011

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Balakrishnan and V

N. Balakrishnan and V. Nevzorov , A Primer on Statistical Distributions , Wiley, 2004, https://books.google.com/books?id=JIfk5kBdLGIC

work page 2004

[2] [2]

Chen , Algorithm-based recovery for iterative methods without checkpointing , in Proceedings of the 20th international symposium on High performance distributed computing, 2011, pp

Z. Chen , Algorithm-based recovery for iterative methods without checkpointing , in Proceedings of the 20th international symposium on High performance distributed computing, 2011, pp. 73--84

work page 2011

[3] [3]

J. T. Daly , A higher order estimate of the optimum checkpoint interval for restart dumps , Future generation computer systems, 22 (2006), pp. 303--312

work page 2006

[4] [4]

S. Di, H. Guo, E. Pershey, M. Snir, and F. Cappello , Characterizing and understanding hpc job failures over the 2k-day life of ibm bluegene/q system , in 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), IEEE, 2019, pp. 473--484

work page 2019

[5] [5]

Erben and E

A. Erben and E. Erdil , Hardware failures won't limit ai scaling , 2024, https://epoch.ai/blog/hardware-failures-wont-limit-ai-scaling. Accessed: 2025-03-21

work page 2024

[6] [6]

Heitmann, T

K. Heitmann, T. Uram, N. Frontiere, S. Habib, A. Pope, S. Rizzi, and J. Insley , The new worlds simulations: Large-scale simulations across three cosmologies , 2024, https://arxiv.org/abs/2406.07276

arXiv 2024

[7] [7]

Kokolis, M

A. Kokolis, M. Kuchnik, J. Hoffman, A. Kumar, et al. , Revisiting reliability in large-scale machine learning research clusters , in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), IEEE, 2025, pp. 1259--1274

work page 2025

[8] [8]

Nguyen-Cong, J

K. Nguyen-Cong, J. T. Willman, J. M. Gonzalez, A. S. Williams, et al. , Extreme metastability of diamond and its transformation to the bc8 post-diamond phase of carbon , The Journal of Physical Chemistry Letters, 15 (2024), pp. 1152--1160

work page 2024