{"id":"cc51049d-ba57-4b03-b538-1ae5491366d7","arxiv_id":"2608.05408","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A usage-based checkpointing and computational efficiency framework, applied to Frontier's 2024 job schedule, yields an estimated machine efficiency of 0.957.","lead":"This paper proposes measuring supercomputer failure rates in node-hours rather than wall-clock time, and uses that to derive the fraction of computing capacity actually available for science. Applied to a year of Frontier data, the method puts the machine's computational efficiency at 95.7 percent under optimal checkpointing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frontier efficiency estimate hinges on unvalidated assumption that all failures are node-local Poisson events; omitted hangs/software failures could materially change the 4.5% overprovision conclusion.","rationale":"The reader's conditional verdict rests on the same weak point, and I agree. I checked the principal equations (3.3), (4.3), (4.6), (5.1) and found them internally consistent; the code and data are public, and the paper is transparent about limitations. The two other candidates—the global runtime rescale r=0.46 (Section 7.3) and the shared checkpoint-size assumption M=200GB (Section 7.2.2)—are acknowledged approximations whose effects are bounded by the sensitivity analysis in Figures 7.3-7.5; they would shift e by a few tenths of a percent for plausible parameter ranges, not by the several percent that an omitted failure mode could. The failure-model gap is therefore the single most load-bearing concern. It does not invalidate the framework, so the verdict should remain CONDITIONAL pending validation against job outcomes.","tokens_in":13153,"tokens_out":17310,"duration_ms":147374,"concrete_test":"Obtain job exit-status and scheduler accounting data for the same CY2024 Frontier jobs used in Section 7.3, classify each job as success, node-failure, software/application failure, timeout/hang, or killed, and compute (i) R0_true as total failure node-hours / total job node-hours, excluding only successful jobs, and (ii) a hang-loss term equal to node-hours consumed by timed-out jobs before termination. Recompute l_chk, l_F, l_R and e using R0_true and adding the hang loss to L in Eqs. (5.2)-(5.4). If the revised e is below 0.947 or the implied overprovision exceeds about 5.6%, the exclusion of non-node-local failures materially changes the central recommendation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical result, e=0.957 for Frontier (Section 7.4) and the implied 4.5% overprovision from Eq. (5.1), is only as good as the failure model it is built on. The model treats failures as a usage-Poisson process with rate R0 and attributes all loss to checkpointing (Eq. 4.3), failure loss (Eq. 2.4), and restart cost (Eq. 2.6). The sole empirical input, R0=2.0e-5 (node-hours)^-1, is derived from node-failure logs under the assumption that nodes unstressed by applications rarely fail, which equates application failures with node failures. Section 8 explicitly concedes that only node-local application failures are considered, and that hangs and collective failures are omitted. A hung job wastes its full requested usage until killed, which is not a Poisson event and is not captured by R0 at all. Since l_F+l_R is only 0.031, the conclusion 'overprovision by 4.5%' is sensitive: an omitted 1.5 percentage points of loss raises overprovision to about 7%; a substantial hang rate would invalidate the efficiency number. The omission is not a minor technicality but the main unknown controlling the headline quantity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a usage-based framework for quantifying the impact of node-local application failures on scientific high-performance computing systems. The authors recast checkpoint optimization from the time domain to the usage domain, introducing a per-unit-usage failure rate R0 and deriving an optimal checkpoint cadence, per-job expected losses (checkpointing, failure, and restart), and a machine-level computational efficiency e = 1 - L/U_tot. The framework is applied to one year of Frontier job-schedule data, yielding e = 0.957 and a corresponding overprovisioning factor of 4.5%. The paper also presents sensitivity analyses of efficiency to checkpoint cadence, failure rate, and storage bandwidth. The derivations are algebraic and the code and data are publicly available.","tokens_in":13431,"tokens_out":8969,"duration_ms":80185,"significance":"The usage-domain reformulation of checkpointing is a well-motivated and useful contribution: it makes failure rates and checkpoint intervals consistent across jobs of different sizes, which is important for heterogeneous scientific workloads. The framework produces closed-form expressions that can be evaluated from schedules and failure logs, and the paper ships reproducible code and the underlying Frontier schedule. If the model limitations are addressed quantitatively, the framework gives facilities a principled basis for overprovisioning node-hours. The headline numerical result is conditional on the node-local Poisson failure model, and the paper honestly identifies this limitation in Section 8; with appropriate quantification of that limitation, the contribution would be solid. The transparency of the derivation, the explicit statement of approximations, and the public release of code and data are notable strengths.","major_comments":[{"comment":"The headline result e = 0.957 and the implied overprovisioning factor e^{-1}-1 = 4.5% are presented in §7.4 as a property of Frontier, but they follow only under the node-local Poisson failure model. Section 8 concedes that collective failures and hangs are not included and that hung jobs consume usage until killed. Since the omitted effects are not quantified, the number cannot be read as an efficiency estimate for Frontier as a whole. The authors should add a quantitative sensitivity analysis with a range of additional hang or collective-failure rates and show their effect on e, or explicitly state in §7.4 that the figure is an illustrative estimate conditional on the node-local Poisson assumption.","section":"§7.4 and §8"},{"comment":"The estimate R0 = 2.0e-5 (node-hours)^{-1} is stated without supporting statistics. The paper should report the number of node-failure events, the total node-hours on which the rate is based, and, ideally, a confidence interval or a sensitivity range. Because all efficiency and loss results in §7.4-§7.8 scale with R0, the absence of any uncertainty measure is a load-bearing gap.","section":"§7.2.1"},{"comment":"The runtime rescaling factor r = 0.46 is applied globally to requested times, under the assertion that most jobs terminate normally. The paper states it does not use or know termination status, so this assertion is unverified. If a material fraction of short runtimes are due to failures or early exits, the rescaled schedule will misrepresent the intended workload. The authors should justify the claim with data or treat r as an uncertain parameter in the analysis.","section":"§7.3"}],"minor_comments":[{"comment":"The abstract contains a duplicated article: 'translate exactly into the the types of challenges'.","section":"Abstract"},{"comment":"The GitHub repository is a strength, but a DOI or versioned release would make the code and data more citable and stable.","section":"§6"},{"comment":"The assumption that all applications checkpoint 20% of node memory (M = 200 GB) is a significant simplification; consider reporting how the results change with M, since Figure 7.5 already illustrates sensitivity to the analogous parameter (bandwidth).","section":"§7.2.2"},{"comment":"Equation (2.2) assumes all nodes contend for a single shared storage bandwidth, which is pessimistic for modern parallel file systems with striped writes; a brief caveat would improve generality.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the derivations are careful. The main risk is that the headline e=0.957 will be cited without its model caveats. I would ask the authors to add the sensitivity analysis and reframing described in major comments 1-2 before publication. The code and data release are commendable and should be highlighted in the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jim,\n\nThe paper is worth your time if you care about HPC resilience economics. It moves the failure-rate conversation from time to node-hours, which is the right variable for heterogeneous job sizes, and it does so cleanly. The core checkpoint equation (3.3) is Daly's (21) reparametrized, and the limiting cases recover Young and Daly correctly. I checked the algebra on (2.4), (2.6), (3.3), and the per-job loss formulas in Section 4; it all hangs together. The genuinely new pieces are the per-job discrete loss model with the ceiling on checkpoint count, the machine-level efficiency aggregate e = 1 - L/U_tot, and the application to one year of Frontier's schedule. The O(N^2) shared-storage checkpoint penalty is a concrete and useful design observation.\n\nThe soft spots are where the reader and the stress-test put them. The headline e=0.957, and the 4.5% overprovision rule, rest on R0 deriving from node-failure logs, a global r=0.46 runtime rescale, and the exclusion of everything that is not a node-local Poisson failure. Hang jobs and collective failures are not in the model at all, and Section 8 says so explicitly. That is an honest limitation, but it is also the main unknown. l_F + l_R is only 0.031; an omitted hang loss of one or two percentage points moves the overprovision recommendation materially. So treat 0.957 as an estimate under the model's assumptions, not a measured quantity. The paper would be stronger with uncertainty bars on R0 and a sensitivity analysis over the hang/collective-failure share. The lack of job termination status data (they only use the schedule, not outcomes) also means the efficiency is not validated against actual job failures.\n\nNone of that undermines the framework. The math is correct, the code and data are public, and the authors are transparent about the gaps. The paper is a solid contribution to the HPC operations literature. I would send it to a serious referee; the referee should push on R0 estimation and the failure-model coverage, but the framework itself deserves publication. For my own reading group, yes, this is worth a session.","headline":"A clean usage-domain reformulation of checkpoint optimization with a transparent but unaudited Frontier efficiency estimate; the math holds up, the headline number should not be taken as a measurement.","tokens_in":13978,"tokens_out":2021,"would_cite":true,"duration_ms":17325,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimal checkpoints leave 95.7% of Frontier's node-hours for science.","keywords":["high-performance computing","checkpointing","node failures","computational efficiency","node-hours","mean usage between failures","fault tolerance","Frontier supercomputer"],"falsifier":"Compare the model's predicted distribution of usage lost per failed job, a truncated exponential with rate $R_0$, against a year of actually recorded job failures on Frontier; if a substantial share of lost usage comes from hangs or collective failures that the truncated exponential cannot produce, or if the observed shortfall between allocated and science node-hours exceeds the predicted $L$, then the $e=0.957$ estimate overstates usable capacity.","tokens_in":12960,"feed_emoji":"🖥️","tokens_out":12967,"duration_ms":96578,"temperature":0.7,"pith_summary":"Exascale machines fail often enough that resilience costs shape how much of a supercomputer actually goes to science. This paper argues that failure rates and checkpoint intervals should be measured in node-hours of usage rather than in wall-clock time, because a 2000-node job running one hour carries the same node-fault risk as a 1000-node job running two hours. On that basis it derives the machine computational efficiency $e = 1 - L/U_{\\rm tot}$, the expected fraction of allocated usage left for computation after checkpointing, failure, and restart losses. Applied to one year of production jobs on the Frontier supercomputer, the model yields $e = 0.957$, meaning a facility could overprovision projects by $e^{-1}-1 = 4.5\\%$ instead of compensating each failed job. A reader should care because this turns resilience accounting into a measurable, per-machine quantity that allocation policies can use.","feed_headline":"Optimal checkpoints leave 95.7% of Frontier node-hours for science","feed_subtitle":"Node-hour accounting says facilities should overprovision by 4.5%; skipping checkpoints costs 9.4%.","key_machinery":"The central machinery is the per-unit-usage failure rate $R_0$ (failures per node-hour) and its reciprocal, the mean usage between failures (MUBF). From it the model derives the optimal checkpoint usage cadence via the condition $1-R_0u_c = \\exp(-R_0(u_c+u_{\\rm chk}))$, the expected usage lost to a job failure under a truncated exponential distribution, and the expected restart cost including repeated failed restarts. The efficiency itself is defined by $e=1-L/U_{\\rm tot}$ with $U_{\\rm tot}$ the total expected usage of a job schedule and $L$ the summed expected losses, split into checkpoint loss $l_{\\rm chk}$, failure loss $l_F$, and restart loss $l_R$. The key scaling behavior is that global checkpoints to shared storage cost $O(N^2)$ in node-hours for a job of $N$ nodes, while node-local checkpoints cost $O(N)$, which drives the bandwidth sensitivity of the results.","core_discovery":"The paper's central claim is that the right unit of failure accounting on scientific supercomputers is usage (node-hours), not elapsed time, and that a single per-unit-usage failure rate $R_0$ (failures per node-hour) is enough to compute both an optimal checkpoint cadence and a machine-level efficiency. It updates the classical checkpoint-interval result to the usage domain, with optimality condition $1-R_0u_c = \\exp(-R_0(u_c+u_{\\rm chk}))$, and derives expected per-job losses from failures, restarts, and checkpoints. Summing these over a job schedule gives the computational efficiency $e=1-L/U_{\\rm tot}$, and Eq. (5.1) turns it into an allocation rule: a project needing $U$ node-hours of science must be allocated $U_{\\rm alloc}=U/e$. For one year of Frontier production jobs the paper finds $e=0.957$ with $l_{\\rm chk}=0.012$, $l_F=0.028$, and $l_R=0.003$, so the facility would overprovision by $e^{-1}-1=4.5\\%$; without prophylactic checkpointing, efficiency drops to $0.914$.","pith_inferences":["If $R_0$ actually varies by application, as the paper suspects, per-application efficiency accounting would give different overprovisioning factors than the machine-wide value; this is an extension the paper does not work out.","Because collective failures, application hangs, and software-only faults are excluded, real efficiency on production machines is likely lower than the paper's estimate; the paper itself flags this gap.","The same usage-based accounting could be applied to AI training clusters, where an analogous effective-training-time metric exists, but replacing a homogeneous workload with a scientific-job mix would be the needed adaptation.","A natural test is to rerun the analysis with per-application failure rates estimated from job exit statuses on another year or another machine; the model's predictions for loss terms could then be checked against observed lost node-hours."],"forward_implications":["A facility that knows its job schedule and $R_0$ can compute a defensible overprovisioning factor: a project needing $U$ node-hours of science should be allocated $U/e$.","For Frontier, optimal checkpointing keeps efficiency at 95.7%, so the facility needs only 4.5% overprovisioning; dropping prophylactic checkpoints entirely would raise that to 9.4%.","Checkpointing cadence matters: displacing all checkpoint intervals by a factor of about 5 in either direction costs roughly 5% efficiency, and a factor of 10 too frequent costs about 10%.","Storage bandwidth to a shared checkpoint target is a major lever: efficiency falls sharply as bandwidth drops, and this penalty is worse at higher failure rates because of the $O(N^2)$ scaling for large jobs.","As node failure rates rise on future machines, the model predicts efficiency declines, for example to about 90.9% at $R_0=10^{-4}$ and 88.6% at $R_0=2\\times10^{-4}$ for a Frontier-like load, making resilient design a capacity-planning issue."],"supporting_citations":[{"why":"The classical first-order optimum checkpoint interval result whose usage-domain limit appears as Eq. (3.4).","marker":"[13]"},{"why":"The higher-order optimum checkpoint interval result that the paper re-derives in usage units; Eq. (3.3) is its usage-domain analog.","marker":"[3]"},{"why":"The AI-cluster 'effective training time ratio' that the paper adapts into an average over heterogeneous scientific workloads.","marker":"[7]"},{"why":"The 'FLOPS per MTBF' notion that motivates measuring failures per unit of work rather than per unit of time.","marker":"[10]"},{"why":"The long-term job-failure study that supports treating application failures as a dominant reliability concern on leadership machines.","marker":"[4]"},{"why":"The standard distributional results used for expected failure counts and expected restart attempts.","marker":"[1]"}],"fun_headline_variants":["Node-hour fault metric yields optimal checkpointing for supercomputers","Usage-based failures refine Frontier efficiency to 95.7%","Checkpoint intervals tuned to node-hours boost supercomputer efficiency","Failure accounting by node-hours, not time, improves supercomputer uptime","Frontier data shows 95.7% efficiency with usage-based checkpointing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole estimate rests on treating every job failure as a random, independent failure of a single compute node with a fixed rate per node-hour; collective failures, application hangs, and software-only faults are left out.","fun_headline_variants_meta":{"raw":{"variants":["Node-hour fault metric yields optimal checkpointing for supercomputers","Usage-based failures refine Frontier efficiency to 95.7%","Checkpoint intervals tuned to node-hours boost supercomputer efficiency","Failure accounting by node-hours, not time, improves supercomputer uptime","Frontier data shows 95.7% efficiency with usage-based checkpointing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1476,"prompt_tokens":962,"completion_tokens":514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":578,"tokens_out":514,"duration_ms":4285,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:43:17.672469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the model's predicted distribution of usage lost per failed job, a truncated exponential with rate $R_0$, against a year of actually recorded job failures on Frontier; if a substantial share of lost usage comes from hangs or collective failures that the truncated exponential cannot produce, or if the observed shortfall between allocated and science node-hours exceeds the predicted $L$, then the $e=0.957$ estimate overstates usable capacity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The classical first-order optimum checkpoint interval result whose usage-domain limit appears as Eq. (3.4)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The higher-order optimum checkpoint interval result that the paper re-derives in usage units; Eq. (3.3) is its usage-domain analog."},{"cited_title":"Kokolis, M","cited_arxiv_id":null,"evidence_quote":"The AI-cluster 'effective training time ratio' that the paper adapts into an average over heterogeneous scientific workloads."},{"cited_title":"Taherin, T","cited_arxiv_id":null,"evidence_quote":"The 'FLOPS per MTBF' notion that motivates measuring failures per unit of work rather than per unit of time."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The long-term job-failure study that supports treating application failures as a dominant reliability concern on leadership machines."},{"cited_title":"Balakrishnan and V","cited_arxiv_id":null,"evidence_quote":"The standard distributional results used for expected failure counts and expected restart attempts."}],"review_version":1}