Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Modeling Spatially Correlated Failure-time Data Under Two Distance Functions with an Application to Titan GPU Data

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that GPU failure correlations in the Titan supercomputer arise from two distinct spatial structures—physical cabinet layout and folded-torus cable connections—and that a failure-time model with both sets of spatial random e

desk verdict Useful two-distance spatial survival framework, but the applied claim about logical distance rests on an undefended assumption that DBE censoring is independent. read the letter →

arxiv 2509.05444 v1 pith:VHQW2VTU submitted 2025-09-05 stat.AP

Jared M. Clark , Jie Min , Yueyao Wang , Yili Hong , George Ostrouchov This is my paper · ORCID
classification stat.AP MSC 62N0562M3062F1562P30
keywords acceleratedfailuretimemodelGPUlifetimelogicalconnectionsphysicalspatialsurvivaldatasupercomputerreliabilitytorusdistanceBayesianinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes an accelerated failure-time (AFT) model for lifetime data in which correlation can act along two different spatial geometries at once: the physical Euclidean distance between cabinets and the logical distance along the cable network. The motivating data are 19,319 GPUs from the Titan supercomputer, where physical distance is tied to heat dissipation and logical distance to job scheduling. The paper argues that accounting for only one distance, as earlier spatial survival models do, is inadequate for this system. Simulations show the estimators recover the true parameters, and a Bayes factor comparison on the Titan data strongly favors the two-distance model over a model with only physical spatial effects. If correct, the result means that both the room layout and the network fabric shape GPU reliability, so reliability models and supercomputer design should treat them as separate correlation sources.

What carries the argument

The load-bearing object is the two-set random-effects AFT model y = Xβ + Z_v v + Z_w w + ε, with v and w independent Gaussian vectors whose correlation matrices are powered exponential. For the logical component, distances are circle distances along rows and columns of a folded torus; the paper proves the resulting torus correlation matrix is positive definite by writing it as the Kronecker product B⊗A of two circle correlation matrices, valid for 0<κ_w≤1. The covariance of the response is therefore the sum Σ = Z_v Σ_v Z_vᵀ + Z_w Σ_w Z_wᵀ + Σ_ε, a positive definite matrix. The physical component uses ordinary absolute distances with 0<κ_v≤2; the logical component uses circular distances with

What would settle it

Fit a version of the model that allows a nonzero covariance between the physical and logical spatial effects—for example a shared latent factor or a cross-covariance parameter—on either the Titan data or the paper's simulation design, and inspect the posterior of that cross-covariance. If its credible interval excludes zero, the additive variance decomposition and the Bayes factor favoring the two-distance model are artifacts of an untested assumption.

Watch

Extended reading notes

Core claim

The central claim is that failure-time correlation in spatially structured engineered systems can be driven by more than one distance function, and that the extra structure is identifiable from data. Concretely, the paper's model writes log failure time as fixed effects plus two independent Gaussian spatial random-effect vectors: v with powered-exponential correlation in physical row-column distance, and w with powered-exponential correlation in logical distances measured on the torus formed by the folded cable topology. On the Titan GPU data the model estimates sigma2_v greater than sigma2_w with posterior probability 0.725, finds stronger anisotropy in logical than physical correlation len

Load-bearing premise

The physical and logical spatial random effects are assumed independent, so their variances simply add; if the two sources share drivers, the estimated variance split and the model comparison would be misspecified.

Editorial extensions

If this is right

  • On Titan, physical proximity and cable adjacency each carry information about GPU failure time; reliability analyses that use only cabinet location understate the role of the network fabric.
  • The same model template extends to other supercomputers or data centers with folded-torus or otherwise periodic interconnects, where logical distance is circular.
  • Posterior estimates of logical correlation lengths indicate anisotropic job-scheduling effects—long correlation along rows, short along columns—which could guide scheduling policies.
  • The positive-definiteness result for powered-exponential torus correlations widens the toolbox for spatial statistics on grids with wrap-around topology.
  • A simpler exponential correlation structure for the logical component, since κ_w is near 1, would likely suffice and ease MCMC convergence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The independence of v and w is the untested hinge; a natural extension is a shared latent factor or cross-covariance term, and the current data may not be able to distinguish true cross-talk from the assumed additive split.
  • A direct practical read: if both distances matter, physically moving cabinets closer to cooling while keeping cable adjacency constant will not change job-scheduling-induced failure dependence—design and scheduling interventions target different correlation sources.
  • The torus correlation proof suggests that other separable products of circle-valid correlation functions could be used in the same framework; positivity essentially requires the exponent constraint κ≤1.
  • A testable extension would be comparing the two-distance model against a physical-distance-only model on other HPC systems with different topologies to see whether logical correlation strength scales with degree of job contiguity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a Bayesian accelerated failure-time (AFT) model with two independent sets of spatial random effects: one based on physical (Euclidean) distance and one based on logical distance defined on a torus. A powered exponential correlation is used for both components, and the authors prove that the torus correlation matrix is positive definite when the smoothness parameter satisfies 0<κ_w≤1 by writing it as a Kronecker product of two positive definite circle correlation matrices. A simulation study reports accurate estimation with moderate censoring (~50%). The method is applied to the Titan GPU failure data, focusing on old-batch off-the-bus (OTB) failures, with double-bit error (DBE) failures treated as right-censoring. The authors report Bayes factors favoring the model with logical-distance random effects and conclude that cable connections induce correlation in the Titan dataset.

Significance. If valid, the paper makes a useful methodological contribution: a parametric AFT model with multiple spatial random effect components and a separable torus correlation function, together with a correct positive-definiteness result. The proof of Theorem 1 is sound and clearly presented, and the simulation framework is reasonable. The substantive application to Titan GPU data is interesting and could inform supercomputer reliability modeling. However, the central applied claim that logical-distance random effects are needed rests on assumptions that are not supported by the reported analysis, especially the treatment of DBE failures as independent censoring and the assumed independence of the two spatial random vectors. The paper would be strengthened by sensitivity analyses and a joint or cause-specific treatment of the competing failure modes.

major comments (3)
  1. [Sections 5.1, 5.2, and 3.1] The headline conclusion that logical-distance random effects are needed depends on treating DBE failures as right-censoring. This is valid only if DBE is non-informative for the OTB failure process. DBE is a competing failure mode of the same hardware, not administrative loss to follow-up. With 94.16% censoring, even modest spatial dependence of DBE intensity could dominate the censored contributions. The paper's own Figure 6 shows that censoring rates vary substantially by location, and the assumption in Section 5.2 that censoring is 'not a function of the location or fixed effect structure' is asserted without evidence. The simulation study uses roughly 50% censoring and does not include informative censoring or competing risks. I request a sensitivity analysis or a joint/cause-specific model for OTB and DBE, or at least a clear empirical justification for independent censoring.
  2. [Section 2.1, Eq. (1)] The covariance decomposition Σ = Z_v Σ_v Z_v^T + Z_w Σ_w Z_w^T + Σ_ε relies on the assumption that v and w are independent. This is a load-bearing assumption for the estimated variance components and for the Bayes factor comparing the model with and without logical random effects. The only justification is the domain argument that job scheduling and heat dissipation are independent. No diagnostic, alternative model, or sensitivity check is provided. I recommend testing the sensitivity of the variance component estimates and the Bayes factor to a non-zero cross-covariance between v and w, or fitting a model that allows dependence.
  3. [Section 5.3] The Bayes factor BF12 = 5.88e-6 is the primary evidence for the paper's central claim that logical-distance random effects are needed. However, the manuscript does not report how the marginal likelihoods were computed, the number of MCMC chains, or any prior sensitivity analysis. The inference is based on a single chain of 4,000 draws after a 4,000-draw burn-in. With heavy censoring and diffuse priors on several covariance parameters, Bayes factors can be unstable. Please report the computational method for BF12, convergence diagnostics (e.g., R-hat, effective sample size), and a prior sensitivity check for the covariance parameters.
minor comments (5)
  1. [Eq. (6)] In the definition of b^L_{sl}, the second circular distance term uses n_r instead of n_c: it should be min{|c^*_s - c^*_t|, n_c - |c^*_s - c^*_t|}. The proof in Eq. (10) uses n_c, so this appears to be a typo.
  2. [Table 3] The 95% interval for Node 2 (β_11) is identical to that of Node 1, which is likely a typo. The reported mean and SD (-0.302, 0.020) would imply a different interval.
  3. [Section 5.1] The analysis is restricted to old-batch GPUs to avoid bias, but no sensitivity analysis is given for this restriction. A brief comparison or discussion of how batch might interact with the spatial effects would strengthen the application.
  4. [Section 5.3] The statement 'Posterior diagnostics suggest convergence' is vague. Please report trace plots, R-hat values, or effective sample sizes, especially because only one chain is used.
  5. [Section 4.1] The simulation study uses a single four-level factor and fixed hyperparameters. It would be useful to state explicitly that the simulation does not cover informative censoring or model misspecification under dependent random effects.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model-comparison and positive-definiteness results are not constructed from their own conclusions.

full rationale

The paper's central derived claims are the positive definiteness of the torus correlation matrix (Theorem 1, Section 2.3) and the need for logical-distance random effects (Bayes factor comparison, Section 5.3). Neither reduces to its own inputs by construction. Theorem 1 follows from the separability of the powered-exponential correlation, external results of Gneiting (2013) for circle distances, and the algebraic fact that the Kronecker product of positive definite matrices is positive definite (Schott 2016). The Bayes factor BF12 = 5.88e-6 is a likelihood-based comparison among models M0, M1, and M2; M2 is not assumed to be true, and the data could in principle have favored M1. The simulation study generates data from the same model and checks estimator accuracy; this is a computational consistency check, not a fitted input renamed as a prediction. The paper cites prior work by overlapping authors (Ostrouchov et al. 2020; Min et al. 2025) for motivation and context, but the derivation chain does not rely on those citations to establish its mathematical or empirical conclusions. The most substantial concern is the treatment of DBE failures as independent right-censoring (Sections 5.1-5.2), which is a real identifiability and validity limitation but not a circular step; the paper itself acknowledges this limitation by noting the model 'could likely be extended to handle competing risks data.' Accordingly, there is no specific equation or fitted parameter that is equivalent by construction to the paper's conclusions, so the circularity score is 0.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The covariance parameters are fitted to the Titan data via MCMC and are the quantitative basis for the claim that both physical and logical correlations exist. The main structural axioms are the independence of the two random effects, the normal error assumption, non-informative censoring, and the external positive-definiteness results used to build the torus correlation. No new physical entities are introduced; the torus distance is a mathematical relabeling of the network topology.

free parameters (9)
  • sigma2 (error variance) = 0.182
    Estimated from Titan data; governs the noise scale in the AFT model.
  • sigma2_v (physical variance) = 0.022
    Estimated from Titan data; the claim that physical location matters depends on this being non-zero.
  • sigma2_w (logical variance) = 0.012
    Estimated from Titan data; the claim that logical connections matter depends on this being non-zero.
  • nuP_r (physical row length scale) = 0.964
    Estimated from Titan data; controls physical correlation decay along rows.
  • nuP_c (physical column length scale) = 0.964
    Estimated from Titan data; controls physical correlation decay along columns.
  • kappa_v (physical smoothness) = 1.297
    Estimated from Titan data; constrained to (0,2] for Euclidean distance.
  • nuL_r (logical row length scale) = 1.876
    Estimated from Titan data; controls logical correlation decay along rows.
  • nuL_c (logical column length scale) = 0.598
    Estimated from Titan data; controls logical correlation decay along columns.
  • kappa_w (logical smoothness) = 0.945
    Estimated from Titan data; constrained to (0,1] for circle distance.
assumptions (6)
  • ad hoc to paper Independence between the two spatial random vectors v and w
    Section 2.1: 'One key assumption of this model is that v and w are independent random vectors.' Needed for the additive covariance decomposition; justified only by a domain argument about energy input vs dissipation, not tested.
  • domain assumption Normal (log-normal) error distribution in the AFT model
    Section 5.3: 'We also assume a normal distribution for the error vector, epsilon, when applying the mixed effects model to the GPU dataset.'
  • domain assumption Non-informative censoring
    Section 5.2: 'we assume that the censoring mechanism is the same for the entire dataset. That is, censoring times are not a function of the location or fixed effect structure.' DBE failures are treated as right-censoring.
  • standard math Gneiting's circle positive-definiteness result
    Section 2.3: 'Gneiting (2013) showed that the powered exponential covariance matrix is positive definite for circles when 0<kappa_w<=1.'
  • standard math Kronecker product of positive definite matrices is positive definite
    Section 2.3, citing Schott (2016); used to prove the torus correlation matrix is PD.
  • domain assumption Old-batch-only subset is representative
    Section 5.1: 'our analysis will focus soley on the old batch data in order to avoid large biases in estimation.' Assumes the subset reflects the same failure mechanisms relevant to the spatial components.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modeling Spatially Correlated Failure-time Data Under Two Distance Functions with an Application to Titan GPU Data." pith.science (2026). https://pith.science/paper/VHQW2VTU

@misc{pith2026250905444,
  author       = {Pith},
  title        = {Pith review of: Modeling Spatially Correlated Failure-time Data Under Two Distance Functions with an Application to Titan GPU Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHQW2VTU}},
  note         = {Machine review of arXiv:2509.05444}
}
read the original abstract

One common approach to statistical analysis of spatially correlated data relies on defining a correlation structure based solely on unknown parameters and the physical distance between the locations of observed values. However, some data have a complex spatial structure that cannot be adequately described with the physical distance alone. In this work, the spatial failure-time data of focus contains information on GPUs that are connected through a network fabric topology that differs from their physical layout and that is expected to introduce additional correlations. The proposed lifetime regression model includes random effects capturing the dependency due to physical location as well as random effects explaining the dependency due to logical connections between GPUs. The analysis of this GPU dataset serves as an example of models with multiple spatial random effects and the ideas presented can be extended to other applications with complex spatial structures. A Bayesian modeling scheme is recommended for this class of analyses. The examples in this work use the software package, Stan, to produce Markov chain Monte Carlo draws for parameter estimation. This modeling effort is validated through simulation which demonstrates accuracy in statistical inference. We also apply the developed framework to the large-scale Titan GPU failure time data.

Figures

Figures reproduced from arXiv: 2509.05444 by the authors.

Figure 1
Figure 1. The physical organization of Titan supercomputer inside a cabinet. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the logical connections among cabinets in the same column. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Illustration of distance on torus. In both the row and column dimensions we are working with locations on a circle. Note that the Cartesian product of two circles is a torus (a doughnut-shaped surface). Therefore, it has become necessary to define distance on a torus [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Illustration of the physical distance. where νw = (ν L r , νL c ) ⊤ and 0 < κw ≤ 1. We want to point out that since the labeling schemes of the physical and logical spatial structures are different, Zv and Zw are not identical when model (1) is applied to the GPU data.…
Figure 5
Figure 5. Figure 5: RMSE in estimation plotted against number of locations. Here [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Failure rates based on physical location (left panel) and censoring rates based on [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Kaplan-Meier estimates for the survival curve, stratified by cage. Cage 3, which is [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Kaplan-Meier estimates stratified by slot (top panel) and Kaplan-Meier estimates [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Estimated correlation function based on physical distance (left panel) and estimated [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    Ezell, M. (2013). Understanding the impact of interconnect failures on system operation. In Proceedings of Cray User Group Conference, Napa Valley, CA

    work page 2013

  2. [2]

    Geng, L. and G. Hu (2022). Bayesian spatial homogeneity pursuit for survival data with an application to the SEER respiratory cancer data. Biometrics\/ 78 , 536--547

    work page 2022

  3. [3]

    Gneiting, T. (2013). Strictly and non-strictly positive definite functions on spheres. Bernoulli\/ 19 , 1327--1349

    work page 2013

  4. [4]

    Hanson, T. E., A. Jara, and L. Zhao (2012). A Bayesian semiparametric temporally-stratified proportional hazards model with spatial frailties. Bayesian Analysis\/ 7 , 147--188

    work page 2012

  5. [5]

    Shimakura, and D

    Henderson, R., S. Shimakura, and D. Gorst (2002). Modeling spatial variation in Leukemia survival data. Journal of the American Statistical Association\/ 97 , 965--972

    work page 2002

  6. [6]

    Brezger, and L

    Hennerfeind, A., A. Brezger, and L. Fahrmeir (2006). Geoadditive survival models. Journal of the American Statistical Association\/ 101 , 1065--1075

    work page 2006

  7. [7]

    Mahmoudi, A

    Hesam, S., M. Mahmoudi, A. R. Foroushani, M. Yaseri, and M. A. Mansournia (2018). A spatial survival model in presence of competing risks for Iranian gastrointestinal cancer patients. Asian Pacific Journal of Cancer Prevention: APJCP\/ 19\/ (10), 2947

    work page 2018

  8. [8]

    Li, J., Y. Hong, R. Thapa, and H. E. Burkhart (2015). Survival analysis of loblolly pine trees with spatially correlated random effects. Journal of the American Statistical Association\/ 110 , 486--502

    work page 2015

Show all 26 references
  1. [9]

    Li, Y. and L. Ryan (2002). Modeling spatial survival data using semiparametric frailty models. Biometrics\/ 58 , 287--297

    2002

  2. [10]

    Min, J., Y. Hong, W. Q. Meeker, and G. Ostrouchov (2025). A spatially correlated competing risks time-to-event model for supercomputer GPU failure data. Technometrics\/ 67 , 531--545

    2025

  3. [11]

    Momenyan, S. (2021). Joint analysis of longitudinal measurements and spatially clustered competing risks HIV/AIDS data. Statistics in Medicine\/ 40\/ (28), 6459--6477

    2021

  4. [12]

    Ahmadi, and J

    Momenyan, S., F. Ahmadi, and J. Poorolajal (2022). Competing risks model for clustered data based on the subdistribution hazards with spatial random effects. Journal of Applied Statistics\/ 49\/ (7), 1802--1820

    2022

  5. [13]

    Mohammadzadeh, and A

    Motarjem, K., M. Mohammadzadeh, and A. Abyar (2019). Bayesian analysis of spatial survival model with non- G aussian random effect. Journal of Mathematical Sciences\/ 237 , 692--701

    2019

  6. [14]

    Mohammadzadeh, and A

    Motarjem, K., M. Mohammadzadeh, and A. Abyar (2020). Geostatistical survival model with Gaussian random effect. Statistical Papers\/ 61 , 85--107

    2020

  7. [15]

    Maxwell, R

    Ostrouchov, G., D. Maxwell, R. A. Ashraf, C. Engelmann, M. Shankar, and J. H. Rogers (2020). GPU lifetimes on Titan supercomputer: Survival analysis and reliability. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pp.\ 1--14. IEEE

    2020

  8. [16]

    Pan, C., B. Cai, L. Wang, and X. Lin (2014). Bayesian semiparametric model for spatially correlated interval-censored survival data. Computational Statistics & Data Analysis\/ 74 , 198--208

    2014

  9. [17]

    Bevilacqua, and M

    Porcu, E., M. Bevilacqua, and M. G. Genton (2016). Spatio-temporal covariance and cross-covariance functions of the great circle distance on a sphere. Journal of the American Statistical Association\/ 111\/ (514), 888--898

    2016

  10. [18]

    Schott, J. R. (2016). Matrix Analysis for Statistics\/ (Third ed.). New Jersey, Hoboken: John Wiley & Sons, Inc

    2016

  11. [19]

    Wang, M., Z. Li, J. Lu, L. Zhang, Y. Li, and L. Zhang (2024). Spatial-temporal Bayesian accelerated failure time models for survival endpoints with applications to prostate cancer registry data. BMC Medical Research Methodology\/ 24\/ (1), 86

    2024

  12. [20]

    Zhang, and A

    Wang, S., J. Zhang, and A. B. Lawson (2016). A Bayesian normal mixture accelerated failure time spatial model and its application to prostate cancer. Statistical Methods in Medical Research\/ 25 , 793--806

    2016

  13. [21]

    Wendland, H. (2004). Scattered Data Approximation . Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press

    2004

  14. [22]

    Wang, Z.-S

    Wu, T., Y. Wang, Z.-S. Ye, and N. Chen (2024). Spatio-temporal analysis and prediction of mass telecommunication base station failure events. Technometrics\/ 66\/ (1), 77--89

    2024

  15. [23]

    Yao, K., J. Zhu, D. J. O'Brien, and D. Walsh (2023). Bayesian spatio-temporal survival analysis for all types of censoring with application to a wildlife disease study. Environmetrics\/ 34\/ (8), e2823

    2023

  16. [24]

    Zhang, J. and A. B. Lawson (2011). Bayesian parametric accelerated failure time spatial model and its application to prostate cancer. Journal of applied statistics\/ 38\/ (3), 591--603

    2011

  17. [25]

    Zhou, H. and T. Hanson (2018). A unified framework for fitting Bayesian semiparametric models to arbitrarily censored survival data, including spatially referenced data. Journal of the American Statistical Association\/ 113 , 571--581

    2018

  18. [26]

    Hanson, and J

    Zhou, H., T. Hanson, and J. Zhang (2017). Generalized accelerated failure time spatial frailty model for arbitrarily censored data. Lifetime data analysis\/ 23\/ (3), 495--515

    2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.