REVIEW 3 major objections 4 minor 1 cited by
SIFBench: An Extensive Benchmark for Fatigue Analysis
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read SIFBench introduces a benchmark of over five million finite-element-derived stress intensity factors across 37 crack scenarios, with baselines and a unified Python interface.
desk verdict SIFBench is a genuinely useful benchmark resource for ML-for-fatigue work, but the headline 'over 5 million geometries' claim is internally contradicted by the paper's own per-dataset counts, and the lack of FEM validation and baseline variance needs to be addressed before it can be fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the benchmark database itself, whose records pair normalized crack and component geometry parameters (such as $a/t$, $a/c$, $c/b$, $W/R$, and $R/t$) with SIF values sampled along the crack front at parametric angle $\phi$. The data were produced by finite element and hp-finite element simulations, with SIFs extracted by displacement- and integral-based post-processing, and the geometric parameter ranges and sampling resolutions for each scenario are documented in appendix tables so users know the boundaries of the data. A unified Python interface reads the CSV files and provides training and validation templates, making the dataset the reusable instrument that carries the paper's argument.
What would settle it
Re-run the finite-element computation for one scenario (e.g., the semi-elliptical surface crack in a rectangular plate under tension) with progressively refined meshes and with quarter-point elements, and compare the computed SIF values at matched crack-front angles. If the values shift by more than a few percent across refinements, or if they disagree with the analytical handbook solution in the regime where that solution is valid, then the database's ground-truth quality would be in question for every model trained on it.
Extended reading notes
Core claim
SIFBench is an open-source benchmark database containing over five million distinct configurations of cracks, component geometries, and loading conditions, generated by high-fidelity finite element simulations and organized into 37 scenarios. The scenarios split into single-crack cases (semi-elliptical surface cracks in plates, quarter-elliptical and through-thickness corner cracks at straight and countersunk bolt holes, each under tension, bending, or bearing) and twin-crack cases (pairs of corner cracks at the same hole types). Each record stores the dimensionless geometric parameters of the configuration together with the stress intensity factor values along the crack front, parameterized by angle $\phi$. The paper's central claim is that this standardized, openly accessible resource, with its unified Python interface, baseline models, and evaluation metrics, removes the data bottleneck that has held back machine-learning approaches to SIF prediction.
Load-bearing premise
The FEM-computed SIF values are accurate enough to serve as ground truth for the represented geometries; the paper does not report mesh-convergence or validation checks against analytical solutions, so any systematic error in the simulations propagates into every model trained on the benchmark.
Editorial extensions
If this is right
- Research groups can train and compare SIF-prediction models on a common, openly hosted dataset instead of each generating their own FEM data, making results directly comparable.
- Baseline results establish a reference: feedforward networks typically perform best on single-crack scenarios, random forests on twin-crack scenarios, while the Fourier neural operator lags in accuracy on most of these tasks.
- The documented parameter ranges define the current scope of validity for any model trained on the benchmark, so users know where interpolation is supported and where it is not.
- The paper states that the benchmark will be expanded to more complex component shapes, broader loading spectra, and experimental data, so the current 37 scenarios are an initial standardized core.
Reading between the lines
- If the community adopts SIFBench as a standard evaluation set, stress-intensity-factor prediction could accumulate the kind of direct model-to-model comparability seen in mature ML benchmark suites, accelerating progress that the lack of shared data currently blocks.
- Because the benchmark's ground truth is entirely FEM-generated, its ceiling is the accuracy of those simulations; adding mesh-convergence studies or comparisons with analytical solutions for a subset of scenarios would let users know how much of the reported model error is model error versus data error.
- The current tabular format (geometric parameters plus angle) suits the baselines tested, but the same database could be reorganized as crack-front coordinate fields or images to enable convolutional and geometry-aware neural operators, a testable extension the paper does not explore.
- Since the scenarios center on fastener holes and plates common in aircraft structures, transferring the benchmark to automotive or energy-sector components would require generating new scenarios; the paper's pipeline and interface could be reused for that purpose.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SIFBench, an open benchmark dataset of finite-element-derived stress intensity factors (SIFs) covering single and twin cracks in plates and around straight and countersunk holes, under tension, bending, and bearing loadings. The authors report baseline results for random forests, support vector regression, feedforward neural networks, and Fourier neural operators, together with evaluation metrics (NAE, relative L2 error, CCDF-NAE) and template Python code. The central claim is that SIFBench contains over five million unique crack and component geometries across 37 scenarios, derived from high-fidelity FEM simulations.
Significance. If the scale and FEM-accuracy claims hold, SIFBench would be a valuable community resource for ML-based fatigue analysis: it aggregates 37 scenarios, provides a unified Python interface, ships baseline code, and includes a neural-operator baseline, all of which lower the entry barrier for researchers and engineers. The paper is honest about limitations such as the absence of dynamic, thermal, and multi-axial loadings. However, the dataset's value as ground truth depends on FEM validation that is not supplied, and the headline scale claim is not supported by the paper's own enumeration. These issues must be resolved before the benchmark can be trusted as a reference standard.
major comments (3)
- [Abstract, §3.1–3.2, Conclusion] The headline scale claim is not supported by the paper's own counts. Summing the per-dataset numbers in §3.1–3.2 yields 543,806 single-crack configurations (2,956 + 28,781 + 5,426 + 114,442 + 127,884 + 85,360 + 72,275 + 22,417 + 84,265) and 1,169,194 twin-crack configurations (72,000 + 3,055 + 1,091,419 + 2,720), for a total of 1,713,000, yet §3 states 'roughly 1.6 million and 3.3 million geometries' and the abstract and conclusion claim 'over 5 million different crack and component geometries.' The 5M figure cannot be obtained from the enumeration unless angle-sampled SIF rows are counted as distinct geometries, which contradicts the wording 'unique plate and crack geometries' used throughout §3.1–3.2. Please reconcile the counts and state exactly what is measured: unique geometry configurations, geometry-loading pairs, or SIF rows.
- [§3.1.1–3.1.3, Appendix A] The benchmark's value as ground truth rests on FEM accuracy, but the paper provides no validation for its own datasets. Sections 3.1.1–3.1.3 describe 'high-fidelity FEM simulations' (Abaqus/FRANC3D for the surface-crack dataset, hp-FEM from CAStLE for the hole datasets) without mesh-refinement or convergence studies, and without comparison to available analytical or handbook solutions such as Raju-Newman or Fawaz-Andersson, both cited in the paper. Because the datasets are intended to train ML models, any systematic FEM error in a scenario would be inherited by all downstream models. Please add, for each geometry family (or at least one representative case per Appendix Table 3–15), a convergence check and a validation subset against a known solution, reporting maximum and mean relative deviations.
- [§3.3, Appendix Tables 16–17] The reported mean NAE values for the TT CC (Countersunk, b/t=0.5) datasets are in the tens to hundreds (e.g., RFR 25.9 under tension, SVR 214.9) while the corresponding relative L2 errors in Table 1 are modest (0.12–0.30). This is likely because some crack-front locations have SIF values near zero, making NAE = |y−ŷ|/|y| ill-conditioned or unbounded. This is not necessarily an arithmetic mistake, but it means mean NAE is not a stable evaluation metric for these sign-changing SIF fields. The paper should state this behavior explicitly, consider reporting median NAE or a thresholded NAE, and advise users of the benchmark accordingly.
minor comments (4)
- [Listing 1] The test block in Listing 1 loads SURFACE_CRACK_TEST.csv but then calls rfr.predict(d[:,:-1]), i.e., predictions are made on the training rows rather than the test rows, and no comparison is made to the test labels; line 9 also contains an extra closing parenthesis. The template should be corrected so that users' first-run experience evaluates on held-out data.
- [Tables 1, 2, 16, 17; Appendix B] Baseline tables report a single run per model with no standard deviations or repeated seeds, while Appendix B describes random subsampling (e.g., 'randomly selected subset ... capped at 100,000') without a fixed seed. Please provide seeds or repeated-run statistics so the baseline numbers are reproducible.
- [Appendix Tables 5 and 13] The entries '1,05' in the a/t and a1/t rows should read '1.05'; the paper should use a single decimal separator throughout.
- [§3.1–3.2, §3.3] The paper states that SIF values are sampled at 'a resolution of 0.024 radians' but never reports the number of angular points per crack front. Reporting the number of angle samples per scenario would let readers convert geometry counts into SIF-row counts and better interpret the stated dataset sizes.
Circularity Check
No significant circularity: SIFBench is an empirical data benchmark whose SIF values come from external FEM simulations; self-citations appear only as dataset provenance.
full rationale
SIFBench makes no derivation claim that reduces to its inputs. The central object is a collection of SIF values generated by FEM (Abaqus, FRANC3D, and CAStLE hp-FEM meshes), which are external, independently established computational methods; the paper does not fit a parameter and then relabel it as a prediction. The baseline ML results in Tables 1 and 2 are measured against these FEM outputs, so they are empirical evaluations, not circular constructions. The only self-citations (Merrell et al. 2024 for the surface-crack data and Gautam et al. 2025 as motivation) are provenance and background, and they are not load-bearing for the benchmark's scale claim or for any numerical result. The apparent mismatch between the headline "over 5 million" geometries and the sum of per-dataset counts in Sections 3.1 and 3.2 is a consistency and arithmetic defect, but it is not circularity: it does not make any output equal to its input by definition. Accordingly, no circular step can be exhibited by quote, and the paper should be regarded as essentially free of circularity aside from minor self-citation.
Assumptions & free parameters
free parameters (4)
- Random forest max_depth =
None
- SVR hyperparameters =
scikit-learn defaults (C=1.0, epsilon=0.1, RBF kernel)
- FNN architecture =
5 hidden layers, 15 neurons each, leaky ReLU
- FNO architecture =
64 Fourier modes, width 64, 4 spectral layers
assumptions (4)
- domain assumption FEM-computed SIF values in the database are accurate ground truth for the represented geometries.
- domain assumption The dimensionless feature sets (a/c, a/t, c/b, W/R, R/t, phi) uniquely and completely characterize each crack and component configuration.
- domain assumption Source databases from CAStLE (Fawaz and Andersson) and Merrell et al. are reliable and were correctly processed.
- standard math Linear elastic fracture mechanics provides the correct framework for the SIF values.
Cite this review
Pith. "Pith review of SIFBench: An Extensive Benchmark for Fatigue Analysis." pith.science (2026). https://pith.science/paper/HDEKHXVL
@misc{pith2026250601173,
author = {Pith},
title = {Pith review of: SIFBench: An Extensive Benchmark for Fatigue Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/HDEKHXVL}},
note = {Machine review of arXiv:2506.01173}
}
read the original abstract
Fatigue-induced crack growth is a leading cause of structural failure across critical industries such as aerospace, civil engineering, automotive, and energy. Accurate prediction of stress intensity factors (SIFs) -- the key parameters governing crack propagation in linear elastic fracture mechanics -- is essential for assessing fatigue life and ensuring structural integrity. While machine learning (ML) has shown great promise in SIF prediction, its advancement has been severely limited by the lack of rich, transparent, well-organized, and high-quality datasets. To address this gap, we introduce SIFBench, an open-source, large-scale benchmark database designed to support ML-based SIF prediction. SIFBench contains over 5 million different crack and component geometries derived from high-fidelity finite element simulations across 37 distinct scenarios, and provides a unified Python interface for seamless data access and customization. We report baseline results using a range of popular ML models -- including random forests, support vector machines, feedforward neural networks, and Fourier neural operators -- alongside comprehensive evaluation metrics and template code for model training, validation, and assessment. By offering a standardized and scalable resource, SIFBench substantially lowers the entry barrier and fosters the development and application of ML methods in damage tolerance design and predictive maintenance.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
When is the combined load identifiable from a stress-intensity profile? A coupled forward-inverse study on SIFBench finite-element data
For fixed crack geometry, combined tension–bending–bearing load is identifiable from the SIF profile iff the three elementary load profiles are affinely independent; recovery error is controlled by the minimum singula...
Reference graph
Works this paper leans on
-
[1]
Albrecht, P. and Yamada, K. (1977). Rapid calculation of stress intensity factors. Journal of the Structural Division , 103(2):377--389
work page 1977
-
[2]
Aliabadi, M., Rooke, D., and Cartwright, D. (1987). An improved boundary element formulation for calculating stress intensity factors: application to aerospace structures. The Journal of Strain Analysis for Engineering Design , 22(4):203--207
work page 1987
-
[3]
Anderson, T. L. and Anderson, T. L. (2005). Fracture mechanics: fundamentals and applications . CRC press
work page 2005
-
[4]
Azizzadenesheli, K., Kovachki, N., Li, Z., Liu-Schiaffini, M., Kossaifi, J., and Anandkumar, A. (2024). Neural operators for accelerating scientific simulations and design. Nature Reviews Physics , 6(5):320--328
work page 2024
-
[5]
Babu s ka, I. and Suri, M. (1990). The p-and hp versions of the finite element method, an overview. Computer methods in applied mechanics and engineering , 80(1-3):5--26
work page 1990
-
[6]
Bathe, K.-J. (2006). Finite element procedures . Klaus-Jurgen Bathe
work page 2006
-
[7]
Callister Jr, W. D. and Rethwisch, D. G. (2020). Materials science and engineering: an introduction . John wiley & sons
work page 2020
-
[8]
Courtin, S., Gardin, C., Bezine, G., and Hamouda, H. B. H. (2005). Advantages of the j-integral approach for calculating stress intensity factors when using the commercial finite element software abaqus. Engineering Fracture Mechanics , 72(14):2174--2185
work page 2005
Show all 35 references
-
[9]
Dehning, P., Thiede, S., Mennenga, M., and Herrmann, C. (2017). Factors influencing the energy intensity of automotive manufacturing plants. Journal of cleaner production , 142:2305--2314
2017
-
[10]
and Pook, L
Dixon, J. and Pook, L. (1969). Stress intensity factors calculated generally by the finite element technique. Nature , 224(5215):166--167
1969
-
[11]
and Andersson, B
Fawaz, S. and Andersson, B. (2004). Accurate stress intensity factor solutions for corner cracks at a hole. Engineering Fracture Mechanics , 71(9-10):1235--1254
2004
-
[12]
M., Settgast, R
Fu, P., Johnson, S. M., Settgast, R. R., and Carrigan, C. R. (2012). Generalized displacement correlation method for estimating stress intensity factors. Engineering Fracture Mechanics , 88:90--107
2012
-
[13]
Gali \'c , I., C ular, I., Vu c kovi \'c , K., and Tonkovi \'c , Z. (2018). Comparison of sif solutions obtained by xfem and conventional fem for cracks in complex geometries like valve body. Procedia structural integrity , 13:2109--2113
2018
-
[14]
M., and Lindgren, E
Gautam, T., Hochhalter, J., Zhe, S., Kirby, R. M., and Lindgren, E. (2025). Developing robust stress intensity factor models using fourier-based data analysis to guide machine learning method selection and training. Available at SSRN 5226340
2025
-
[15]
and Shen, G
Glinka, G. and Shen, G. (1991). Universal features of weight functions for cracks in mode i. Engineering Fracture Mechanics , 40(6):1135--1146
1991
-
[16]
Han, Q., Wang, Y., Yin, Y., and Wang, D. (2015). Determination of stress intensity factor for mode i fatigue crack based on finite element analysis. Engineering Fracture Mechanics , 138:118--126
2015
-
[17]
Hou, J., Lv, J., Ricoeur, A., Hu, Y., Zuo, H., Chen, Y., and Li, Q. (2022). The m-integral in fracture and damage mechanics: A review of developments and applications. Engineering Fracture Mechanics , 273:108741
2022
-
[18]
Irwin, G. R. (1957). Analysis of stresses and strains near the end of a crack traversing a plate
1957
-
[19]
Launey, M. E. and Ritchie, R. O. (2009). On the fracture toughness of advanced materials. Advanced Materials , 21(20):2103--2110
2009
-
[20]
Liu, X., Xiao, Q., and Karihaloo, B. L. (2004). Xfem for direct evaluation of mixed mode sifs in homogeneous and bi-materials. International Journal for Numerical Methods in Engineering , 59(8):1103--1118
2004
-
[21]
M., and Hochhalter, J
Merrell, J., Emery, J., Kirby, R. M., and Hochhalter, J. (2024). Stress intensity factor models using mechanics-guided decomposition and symbolic regression. Engineering Fracture Mechanics , 310:110432
2024
-
[22]
Mitl \"o hner, J., Neumaier, S., Umbrich, J., and Polleres, A. (2016). Characteristics of open data csv files. In 2016 2nd International Conference on Open and Big Data (OBD) , pages 72--79. IEEE
2016
-
[23]
Newman, J. C. (2000). Irwin's stress intensity factor--a historical perspective. ASTM special technical publication , 1389:39--53
2000
-
[24]
and Raju, I
Newman Jr, J. and Raju, I. (1981). An empirical stress-intensity factor equation for the surface crack. Engineering fracture mechanics , 15(1-2):185--192
1981
-
[25]
Perez, N. (2004). Linear elastic fracture mechanics . Springer
2004
-
[26]
Pommier, S., Sakae, C., and Murakami, Y. (1999). An empirical stress intensity factor set of equations for a semi-elliptical crack in a semi-infinite body subjected to a polynomial stress distribution. International Journal of Fatigue , 21(3):243--251
1999
-
[27]
and Newman, J
Raju, I. and Newman, J. (1979). Stress-intensity factors for two symmetric corner cracks. In Fracture Mechanics: Proceedings of the Eleventh National Symposium on Fracture Mechanics: Part I . ASTM International
1979
-
[28]
Spencer, A. J. M. (2004). Continuum mechanics . Courier Corporation
2004
-
[29]
Wang, C. H. (1996). Introduction to fracture mechanics . DSTO Aeronautical and Maritime Research Laboratory Melbourne, Australia
1996
-
[30]
and Lambert, S
Wang, X. and Lambert, S. (1995). Local weight functions for semi-elliptical surface cracks in finite thickness plates. Theoretical and applied fracture mechanics , 23(3):199--208
1995
-
[31]
Wawrzynek, P., Carter, B., Hwang, C.-Y., and Ingraffea, A. (2010). Advances in simulation of arbitrary 3d crack growth using franc3dv5. Journal of the Computational Structural Engineering Institute of Korea , 23(6):607--613
2010
-
[32]
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA
2006
-
[33]
Xia, B., Ma, Z., Hu, H., Li, Y., and Zhao, W. (2022). A prediction method of stress intensity factor for mode-i crack in coal rock based on deep learning. Theoretical and Applied Fracture Mechanics , 122:103645
2022
-
[34]
Xu, T., Ding, S., Zhou, H., and Li, G. (2022). Machine learning-based efficient stress intensity factor calculation for aeroengine disk probabilistic risk assessment under polynomial stress fields. Fatigue & Fracture of Engineering Materials & Structures , 45(2):451--465
2022
-
[35]
Zhang, X., Zhao, T., Liu, Y., Chen, Q., Wang, Z., and Wang, Z. (2023). A data-driven model for predicting the mixed-mode stress intensity factors of a crack in composites. Engineering Fracture Mechanics , page 109385
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.