REVIEW 4 major objections 7 minor 1 cited by
RESMETRIC: Analyzing Resilience to Enable Research on Antifragility
T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read ResMetric turns a single quality-of-service time series into a battery of resilience metrics plus a computable degree of antifragility, with no system model required.
desk verdict Solid, honest tool paper with a credible artifact, but the antifragility index is unvalidated and demonstrably unstable across dip-detection choices, so the central 'enables antifragility research' claim needs retuning. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the dip: a segment of the performance curve Q(t) that falls below its undisturbed behavior, with a start, a threshold crossing, a recovery crossing, and a settling point. Everything else is organized around detecting and measuring dips: three detection algorithms (maxima-based, threshold-based, and linear-regression with Bayesian optimization of the segment count), the dip-dependent metrics R, RR, and RL, and the integrated metric IRM. The antifragility score α_u then averages the chosen metric's change across the detected dips. The choice of detector is pivotal, because it decides how many dips exist — and the case study shows a system with a single detected dip receives no antifragility score at all.
What would settle it
Feed ResMetric two synthetic QoS traces with identical dip shapes, one produced by a working adaptation strategy and one by the disturbance simply ending: the tool would return identical resilience and antifragility scores, exposing that the curve alone cannot distinguish adaptive recovery from passive recovery. A related, already-observed symptom is reported in the paper itself: with linear-regression dip detection, the lifelong-learning system has just one detected dip, and the calculation of its antifragility degree is reported as not feasible.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that resilience and antifragility can be measured from a single normalized performance curve Q(t) over time, without any model of the system's internals or adaptation logic. To that end ResMetric implements literature metrics — AUC, robustness R (the minimum performance reached), recovery rate RR (the reciprocal of dip length), recovery level RL (adaptive capacity or recovery ability), and an integrated resilience metric IRM that repairs a defect found in the literature's GR metric by incrementing the time-averaged performance loss so it stays positive when a system improves after a dip. The paper's own definition of antifragility is a degree α_u per resilience metric u: 0 for strict monotone decline, a monotonicity ratio in (0,1) for mixed up-and-down behavior, and average improvement rate plus one for monotone increase. Deployed on four classifiers in the gas-delivery case study, the tool yields rankings the authors call non-expected — the offline model with the worst overall accuracy has the highest degree of antifragility, while the lifelong-learning system has the lowest or, with a different dip detector, no score at all — which the paper reads as evidence that a naive average-change notion of antifragility is insufficient and needs community refinement.
Load-bearing premise
The load-bearing premise is that a single normalized quality-of-service time series contains all the information needed to quantify resilience and antifragility; if resilience depends on the system's architecture, adaptation logic, or context that the curve does not show, then the metrics and α_u rankings computed by ResMetric are not interpretable.
Editorial extensions
If this is right
- Any system with measurable quality-of-service data can be scored for resilience without choosing or building a system model, so resilience reporting becomes routine rather than bespoke.
- Antifragility becomes a concrete, comparable number, letting researchers rank adaptation strategies by whether and how fast their resilience improves across repeated disruptions.
- Because the choice of dip detector changes the ranking, published resilience comparisons should now state which detector and which metrics were used; the paper's case study makes that dependency visible.
- The modular architecture invites researchers to implement their own resilience metrics and antifragility definitions and run them on the same input, turning debates about definitions into experiments.
- Practitioners get interactive plots of dips, robustness, recovery, and antifragility from a JSON file or a Plotly graph, without writing analysis code.
Reading between the lines
- The paper's own non-expected result — the worst-performing offline model scoring highest on antifragility — suggests α_u may reward any downward-then-upward movement, conflating instability with improvement; a natural test is to require monotone improvement in the resilience metric across dips and see whether the rankings change.
- Dip detection is in essence a change-point detection problem, and the paper compares three heuristics; plugging in statistically grounded segmenters (such as PELT or Bayesian online change-point detection) would show whether antifragility rankings stabilize across detector choices.
- Before trusting the metrics on real systems, a calibration pass on synthetic traces with planted dip depths, durations, recovery slopes, and improvement trends would show whether each metric recovers the known quantities — an experiment the paper does not run.
- The paper flags an implicit weighting inside its integrated metric, since Rapidity is unbounded while Robustness, Recovery, and TAPL lie in [0,1]; making that weighting a user-chosen parameter would be a direct extension that lets different communities tune IRM to their domain.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents RESMETRIC, a Python package and command-line tool that computes resilience metrics from a normalized quality-of-service time series, including AUC, threshold dwell time, derivatives, robustness, recovery rate, recovery level, and an integrated resilience metric, plus a newly proposed antifragility index alpha_u that aggregates changes in a selected resilience metric across detected dips. The tool is demonstrated on a gas-detection lifelong learning system with four models: an offline SVC, an online SVC, the LLL system, and a new ensemble model. The authors use the case study to show that dip-detection choices and metric choices materially affect resilience and antifragility assessments, and they position RESMETRIC as a reusable, model-agnostic basis for community discussion of antifragility.
Significance. The artifact-oriented contribution is valuable: RESMETRIC packages several literature metrics into one reproducible, model-agnostic tool with visualization, and the case study makes the metric-selection sensitivity concrete. The availability of the artifact and the fact that the displayed plots are generated by the tool are strengths. The authors also honestly report that their antifragility index is a naive proposal and can give counterintuitive or non-computable results. However, the antifragility index is not validated against any ground truth or synthetic benchmark, and the case study is a single-run demonstration, so the abstract's claim that the tool 'enables research on antifragility' is stronger than the evidence supports. The paper would be a solid tool presentation after the antifragility claims are calibrated and the metric is defined formally.
major comments (4)
- [Section III-C, IV-E, IV-F] The antifragility index alpha_u is the paper's only antifragility measure, and it is defined in prose as an aggregate over detected dips. The case study demonstrates that the resulting rankings are not stable: with max-dip detection (Fig. 7) the offline SVC has the highest alpha_u_bar and the LLL system the lowest; with linear-regression dip detection (Fig. 8) the LLL system's alpha_u is not computable, the state-of-the-art and ensemble systems have alpha_u_bar = 0, and the offline system's value nearly doubles. Section IV-F concedes that 'the choice of dip detection algorithm significantly influences the results,' and Section VI calls the approach 'naive.' No external or synthetic ground truth is provided to show that alpha_u measures improvement in resilience. This is a load-bearing gap for the abstract's claim that RESMETRIC enables antifragility research; it should be addressed by validating the index on data with known fragility/antifragility behavior or by explicitly restricting the claim to a research prototype.
- [Section IV] The comparison is based on a single performance trace per system (Figs. 3-8). No multiple runs, confidence intervals, or sensitivity analysis other than the dip-detection choice are reported. Statements such as 'the ensemble and LLL systems have fairly similar scores' (Section IV-C) are therefore not quantitatively supported. Please state whether the models are deterministic and add at least a replication over random seeds for the ensemble and any stochastic components, or clearly label the demonstration as illustrative of tool features rather than as a model comparison.
- [Section IV-A] The fourth curve is new but described only as 'Gradient Boosting' integrated 'within the MAPE-K loop'; no hyperparameters, training/test split, number of estimators, or integration details are given. This makes the case-study curve non-reproducible and weakens the comparison. Provide the configuration or a pointer to a script that generates the curve.
- [Section III-C and III-B] The paper never gives equations for alpha_u, alpha_u_bar, or the IRM variant. The description of alpha_u as a 'membership function for monotonicity' with values calculated as 'the average rate of improvement, incremented by one' is ambiguous, and the Bayesian-optimized linear-regression dip detection is not specified (objective, bounds, iterations). Since dip detection and metric formulas determine the reported numbers, these should be stated precisely or referenced to a documented code version.
minor comments (7)
- [Section IV-C] Quantify the claimed similarity of ensemble and LLL, for example, by tabulating the R, RR, and RL values rather than relying on visual inspection.
- [Section III-C] The phrase 'mean antifragilty' should be 'mean antifragility.'
- [Section V] In the paragraph on Yang et al., 'MircoRes' should be 'MicroRes' to match the cited work.
- [Section III-B vs IV-D] Section III-B reports roughly 5 minutes per quality signal for linear regression dip detection, while Section IV-D reports an average of 175 seconds per system; reconcile these timings.
- [Fig. 4] The dashed lines for time below threshold are difficult to distinguish from the solid AUC lines; consider distinct markers or a separate panel.
- [Section III] The phrase 'Fig, 2' should be 'Fig. 2.'
- [Section III-A] Mention explicitly that normalizing AUC by window length makes metrics comparable only if window boundaries are chosen consistently across systems.
Circularity Check
No significant circularity: RESMETRIC's metrics are imported from the literature and applied to the input trace, not derived by fitting; the few self-citations are non-load-bearing.
full rationale
The paper's central product is a tool that computes published resilience metrics (AUC, robustness, recovery rate/level, IRM) from a normalized QoS time series, and the case study applies these metrics to four classifier traces. The calculations are definitional in the ordinary sense: each metric is explicitly defined (e.g., robustness as minimum performance, recovery rate as reciprocal of dip length, IRM as a product of published components), and none is fitted to a label or to an outcome to be predicted. The antifragility index α_u is the only author-proposed quantity. It is defined as an aggregate of changes in a user-selected resilience metric across detected dips (Section III-C). It is not fitted to the case-study results and is explicitly presented as one proposal: the authors state in Section VI that they 'supply one proposal for antifragility' and concede in Section IV-F that 'the choice of dip detection algorithm significantly influences the results' and in Section VI that 'a naive approach to antifragility might not be sufficient to fully capture the desired properties.' These are validity/robustness limitations, not circular reductions: the rankings in Figs. 7-8 depend on the dip-detection preprocessing choice, which is a correctness risk for the antifragility claim but does not make a computed metric equal to its input. Self-citations are minor and non-load-bearing: Ref [9] is the tool's own repository DOI and Ref [7] is an exemplar reference with overlapping authors; neither supplies the paper's central derivation. The tool is self-contained in the sense that the metrics are standard or explicitly defined, and no claimed prediction is obtained by construction from fitted values. Score 1 reflects the minor self-citation presence without any material circular dependency.
Assumptions & free parameters
free parameters (3)
- Threshold theta =
0.8 in the case study; user-set otherwise
- Number of segments in linear regression dip detection =
Chosen by Bayesian optimization per signal; values not reported
- Exponential kernel half-time =
User-set; not used in the case study
assumptions (5)
- domain assumption Input time series are normalized to [0,1] and represent quality of service over time.
- domain assumption Resilience can be inferred from a single performance-over-time trace without a system model.
- domain assumption The gas detection curves, including the newly added ensemble curve, accurately represent the compared systems.
- ad hoc to paper The piecewise alpha_u index is a meaningful operationalization of antifragility.
- domain assumption Preprocessing filters remove noise without systematically distorting dip metrics.
invented entities (2)
-
Antifragility index alpha_u
-
IRM variant with incremented TAPL
Cite this review
Pith. "Pith review of RESMETRIC: Analyzing Resilience to Enable Research on Antifragility." pith.science (2026). https://pith.science/paper/DKSJRVWA
@misc{pith2026250118245,
author = {Pith},
title = {Pith review of: RESMETRIC: Analyzing Resilience to Enable Research on Antifragility},
year = {2026},
howpublished = {\url{https://pith.science/paper/DKSJRVWA}},
note = {Machine review of arXiv:2501.18245}
}
read the original abstract
A key feature in self-adaptive systems is resilience, which is an ongoing research topic. Recently, the community started to explore antifragility, which describes the improvement of resilience over time. While there are model-agnostic resilience metrics, there is currently no out-of-the-box tool for researchers and practitioners to determine to which degree their system is resilient. To facilitate research on antifragility, we present ResMetric, a model-agnostic tool that calculates and visualizes various resilience metrics based on the quality of service over time. With ResMetric, researchers can evaluate their definition of resilience and antifragility. This paper highlights how ResMetric can be employed by demonstrating its use in a case study on gas detection.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Beyond Resilience: Antifragility in Critical Infrastructure Cybersecurity
A bounded Jensen-gain/post-disruption-gain definition of antifragility is paired with evidence that sector-level fragility burden and process-level perturbation observability are measurable in public datasets.
Reference graph
Works this paper leans on
-
[1]
Benchmarking the resilience of self-adaptive software systems: perspectives and challenges,
R. Almeida and M. Vieira, “Benchmarking the resilience of self-adaptive software systems: perspectives and challenges,” in Proceedings of the 6th international symposium on software engineering for adaptive and self-managing systems , 2011, pp. 190–195
work page 2011
-
[2]
Resilience and antifragility of autonomous systems (dagstuhl seminar 24182),
S. Burton, R. Calinescu, and R. Mirandola, “Resilience and antifragility of autonomous systems (dagstuhl seminar 24182),” Dagstuhl Reports , vol. 14, no. 4, pp. 142–163, 2024
work page 2024
-
[3]
S. Mahdavi-Hezavehi, V . H. Durelli, D. Weyns, and P. Avgeriou, “A systematic literature review on methods that handle multiple quality attributes in architecture-based self-adaptive systems,” Information and Software Technology, vol. 90, pp. 1–26, 2017
work page 2017
-
[4]
Body sensor network: A self-adaptive system exemplar in the healthcare domain,
E. B. Gil, R. Caldas, A. Rodrigues, G. L. G. da Silva, G. N. Rodrigues, and P. Pelliccione, “Body sensor network: A self-adaptive system exemplar in the healthcare domain,” in 2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). IEEE, 2021, pp. 224–230
work page 2021
-
[5]
Self-adaptive systems: A system- atic literature review across categories and domains,
T. Wong, M. Wagner, and C. Treude, “Self-adaptive systems: A system- atic literature review across categories and domains,” Information and Software Technology, vol. 148, p. 106934, 2022
work page 2022
-
[6]
Evaluating the effectiveness of the rainbow self-adaptive system,
S.-W. Cheng, D. Garlan, and B. Schmerl, “Evaluating the effectiveness of the rainbow self-adaptive system,” in 2009 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems . IEEE, 2009, pp. 132–141
work page 2009
-
[7]
Formal synthesis of uncertainty reduction controllers,
M. Carwehl, C. Imrie, T. V ogel, G. Rodrigues, R. Calinescu, and L. Grunske, “Formal synthesis of uncertainty reduction controllers,” in Proceedings of the 19th International Symposium on Software Engineer- ing for Adaptive and Self-Managing Systems , 2024, pp. 2–13
work page 2024
-
[8]
A conceptual and architectural characterization of antifragile systems,
V . Grassi, R. Mirandola, and D. Perez-Palacin, “A conceptual and architectural characterization of antifragile systems,” 2024
work page 2024
Show all 23 references
-
[9]
Resmetric: A python module for visualizing resilience and antifragility,
F. Koenig, “Resmetric: A python module for visualizing resilience and antifragility,” Jan. 2025. [Online]. Available: https://doi.org/10.5281/ zenodo.14724651
2025
-
[10]
Seismic resilience of a hospital system,
Gian Paolo Cimellaro, Andrei M. Reinhorn, and Michel Bruneau, “Seismic resilience of a hospital system,” Structure and Infrastructure Engineering, vol. 6, no. 1-2, pp. 127–144, 2010
2010
-
[11]
Engineering Resilience in Critical Infrastructures,
G. Sansavini, “Engineering Resilience in Critical Infrastructures,” in Re- silience and Risk , I. Linkov and J. M. Palma-Oliveira, Eds. Dordrecht: Springer Netherlands, 2017, pp. 189–203
2017
-
[12]
System resilience distribution identification and analysis based on performance processes after disruptions,
Y . Song and R. Li, “System resilience distribution identification and analysis based on performance processes after disruptions,” PLOS ONE, vol. 17, no. 11, p. e0276908, Nov. 2022
2022
-
[13]
A metric and frameworks for resilience anal- ysis of engineered and infrastructure systems,
R. Francis and B. Bekera, “A metric and frameworks for resilience anal- ysis of engineered and infrastructure systems,” Reliability Engineering & System Safety , vol. 121, pp. 90–103, Jan. 2014
2014
-
[14]
Fostering resilience to extreme events within infrastructure systems: Characterizing decision contexts for mitigation and adaptation,
T. McDaniels, S. Chang, D. Cole, J. Mikawoz, and H. Longstaff, “Fostering resilience to extreme events within infrastructure systems: Characterizing decision contexts for mitigation and adaptation,” Global Environmental Change, vol. 18, no. 2, pp. 310–318, May 2008
2008
-
[15]
Chemical gas sensor drift compensation using classifier ensembles,
A. Vergara, S. Vembu, T. Ayhan, M. A. Ryan, M. L. Homer, and R. Huerta, “Chemical gas sensor drift compensation using classifier ensembles,” Sensors and Actuators B: Chemical , vol. 166–167, pp. 320– 329, 2012
2012
-
[16]
Gas sensor array drift at different concentrations,
A. Vergara, “Gas sensor array drift at different concentrations,” UCI Machine Learning Repository, 2012
2012
-
[17]
Lifelong self-adaptation: Self-adaptation meets lifelong machine learning,
O. Gheibi and D. Weyns, “Lifelong self-adaptation: Self-adaptation meets lifelong machine learning,” in Proceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems , ser. SEAMS ’22. New York, NY , USA: Association for Computing Machinery, 2...
2022
-
[18]
MicroRes: Versatile Resilience Profiling in Microservices via Degradation Dissemination Indexing,
T. Yang, C. Lee, J. Shen, Y . Su, C. Feng, Y . Yang, and M. R. Lyu, “MicroRes: Versatile Resilience Profiling in Microservices via Degradation Dissemination Indexing,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA ...
2024
-
[19]
An empirical study of software architecture resilience evaluation methods,
J. Pan, Z. Liu, D. Li, L. Wang, and B. Li, “An empirical study of software architecture resilience evaluation methods,” Journal of Systems and Software, vol. 202, p. 111726, Aug. 2023
2023
-
[20]
Robustness-Driven Resilience Evaluation of Self-Adaptive Software Systems,
J. C ´amara, R. de Lemos, N. Laranjeiro, R. Ventura, and M. Vieira, “Robustness-Driven Resilience Evaluation of Self-Adaptive Software Systems,” IEEE Transactions on Dependable and Secure Computing , vol. 14, no. 1, pp. 50–64, Jan. 2017
2017
-
[21]
Empirical resilience evaluation of an architecture-based self-adaptive software system,
J. C ´amara, P. Correia, R. de Lemos, and M. Vieira, “Empirical resilience evaluation of an architecture-based self-adaptive software system,” in Proceedings of the 10th International ACM Sigsoft Conference on Quality of Software Architectures , ser. QoSA ’14. New York, NY , U...
2014
-
[22]
Architecture-based resilience evaluation for self-adaptive systems,
J. C ´amara, R. de Lemos, M. Vieira, R. Almeida, and R. Ventura, “Architecture-based resilience evaluation for self-adaptive systems,” Computing, vol. 95, no. 8, pp. 689–722, Aug. 2013
2013
-
[23]
Evaluation of resilience in self-adaptive systems using probabilistic model-checking,
J. C ´amara and R. de Lemos, “Evaluation of resilience in self-adaptive systems using probabilistic model-checking,” in 2012 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) , Jun. 2012, pp. 53–62
2012
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.