Pith. sign in

REVIEW 3 major objections 5 minor 76 references

Can We Recycle Our Old Models? An Empirical Evaluation of Model Selection Mechanisms for AIOps Solutions

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Old AIOps models that are normally discarded after retraining can be recycled: ranking them by performance on the most recent labeled period selects a top model that tends to beat the periodically retrained baseline.

desk verdict A careful empirical study whose headline claim is partly undercut by a leakage/recency-assumption issue in TBM/rTBM; still worth serious peer review. read the letter →

arxiv 2505.02961 v1 pith:5VAHUMYG submitted 2025-05-05 cs.SE

Yingzhe Lyu , Hao Li , Heng Li , Ahmed E. Hassan This is my paper · ORCID
classification cs.SE
keywords AIOpsmodelselectionhistoricalmodelsconceptdrifttemporaladjacencyperiodicalretrainingfailurepredictionrecycling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the historical models that AIOps (artificial intelligence for IT operations) teams accumulate during periodic retraining, and usually discard, can be recycled for future predictions. The authors evaluate six model selection mechanisms on three large public operation datasets (Google cluster traces, Backblaze disk stats, and Alibaba GPU traces) by using each mechanism to pick a top model from the history for every new testing period. They find that mechanisms exploiting temporal adjacency, especially the revised temporal-adjacency selector (rTBM) and the revised similarity selector (rSBM), tend to outperform the standard periodically retrained model in AUC and can even match a hypothetical oracle that knows the future labels in one configuration. If this holds, AIOps maintenance could shift from mandatory retraining to selecting among accumulated historical models, avoiding the cost of constant updates. The paper also documents a persistent gap between current mechanisms and the theoretical upper bound, which it offers as the target for future work.

What carries the argument

The central mechanism is temporal-adjacency-based ranking over a history of models. For each upcoming unlabeled period, the selector scores every previously trained model by its prediction performance (AUC) on the most recent labeled period—under the assumption that adjacent periods have the closest distributions—and deploys the top-ranked model; the revised rTBM variant specifically assumes that if the second-most-recent model wins, the untested newest model would be even better and promotes it to the top. The similarity-based sibling SBM/rSBM instead picks the most similar historical period using Hausdorff distance, a measure of how far two sample sets are from each other, between the testing period's samples and each candidate period's samples. The paper measures these mechanisms against periodical retraining and against a hypothetical oracle that cheats by using the true labels of the test period.

What would settle it

Run the six selection mechanisms on an operation trace where concept drift events are explicitly annotated and occur inside, rather than at the boundary of, the fixed time periods; if the temporal-adjacency mechanisms no longer beat the periodically retrained model, the last-period proxy is not the active ingredient.

Watch

Extended reading notes

Core claim

The paper claims that the pile of historical AIOps models normally thrown away after periodic retraining can be reused as a candidate pool: instead of always deploying the newest model, a maintenance system can rank the old models and deploy the top-ranked one for the next period. On Google cluster traces, Backblaze disk stats, and Alibaba GPU traces, selection mechanisms that exploit temporal adjacency—ranking each historical model by its performance on the most recent labeled period (TBM), optionally promoting the newest model (rTBM), or using Hausdorff distance to find the most similar past period (SBM/rSBM)—tend to beat the periodically retrained baseline in AUC, and rTBM is statistically indistinguishable from the oracle in one configuration. The paper also shows that the same mechanisms' rankings agree with the oracle's ranking (best for rSBM) and are mostly stable across repeated runs, while a gap remains between any real mechanism and the oracle upper bound.

Load-bearing premise

The load-bearing premise is that the most recent labeled period is the best available preview of the next unlabeled period, so ranking old models by their performance on that last labeled period predicts how they will rank on the upcoming one.

Editorial extensions

If this is right

  • On the Google and Backblaze datasets, several selection mechanisms, especially rTBM, rSBM, TBM, and CRC, achieve statistically higher AUC than the periodically retrained model for at least one base classifier.
  • The oracle baseline beats periodical retraining for most models on Google and Backblaze, showing that there is real headroom in the discarded historical models that current selectors do not fully capture.
  • Mechanisms based on temporal adjacency generally beat their non-temporal counterparts, and the revised versions that promote the most recent model outperform the originals.
  • rSBM produces rankings most aligned with the oracle by Kendall's $\tau$ and top-3 Jaccard similarity across datasets and models.
  • Most mechanisms produce stable rankings across 100 random-seed runs, with strong Kendall's $W$ agreement, except some LaF and rTBM combinations that show weak agreement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the ranking itself, not the retrained weights, is the reusable asset: a production AIOps system could archive every trained checkpoint, score it on the most recent labeled window, and deploy the top scorer, retraining only when no archived model clears a quality bar.
  • Because each dataset is tested at only one fixed period length (daily, weekly, or monthly), the temporal-adjacency effect might be period-length dependent; sweeping period granularity or aligning periods to detected drift events would test whether the advantage survives.
  • The gap between the mechanisms and the oracle suggests that combining temporal-adjacency ranking with label-free confidence signals such as CRC or LaF, rather than using either family alone, is a natural next step.
  • The strong ranking agreement of rSBM with the oracle could be exploited directly to weight a time-based ensemble of historical models, a direction the paper names as possible future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates whether model selection mechanisms can choose among previously trained historical AIOps models as an alternative to periodically retraining a single model. Using Google cluster trace, Backblaze disk stats, and Alibaba GPU cluster trace datasets, the authors train sliding-window models and compare six selection mechanisms (TBM, rTBM, SBM, rSBM, CRC, LaF) plus stationary and periodic-retraining baselines and a hypothetical oracle. The main reported finding is that temporal-adjacency-based mechanisms, particularly rTBM and rSBM, outperform periodic retraining on Google and Backblaze, with rTBM sometimes matching the oracle, while a performance gap remains between all mechanisms and the oracle.

Significance. If the central claim holds, the paper would provide a practical alternative to expensive periodic retraining in AIOps: instead of discarding historical models, practitioners could select among them based on temporal adjacency or similarity. The study has notable strengths: it uses three large public datasets, 100 repeated runs with different random seeds, Scott-Knott statistical grouping, a replication package, and an explicit threats-to-validity section. However, the empirical claim is currently undermined by internal-validity concerns about data leakage in the TBM/rTBM and rSBM evaluation protocols, and by an apparent inconsistency in the definition of rTBM that makes its reported advantage over retraining difficult to interpret. The work is a reasonable contribution to an under-studied problem, but the headline conclusion is not yet cleanly supported.

major comments (3)
  1. [Section 3.2.1 and Section 3.2.5] The definition of rTBM appears to make it equivalent to the periodical retraining baseline. TBM ranks candidates by their performance on the last available period, and rTBM promotes the model from the latest historical time period to the top whenever the highest-ranked model is from a second-latest period. Since the latest historical model is exactly the sliding-window model trained through the previous period, and the periodical retraining baseline selects exactly that model, rTBM as written always selects the same model as retraining. In that case, the reported statistically significant advantages of rTBM over Retrain in Section 4.1 (e.g., Google LR, Backblaze CART and LR) cannot arise. Please clarify the actual rTBM procedure, or correct the definition; as written, the paper is internally inconsistent on a load-bearing point.
  2. [Section 3.2.1 with Section 3.1.5] TBM's scoring of candidates on the 'last available period' (period t-1) is in-sample for the newest candidate, whose sliding training window ends at t-1, while it is out-of-sample for older candidates. This systematically inflates the newest model's estimated performance relative to older models, independent of its true generalization ability. Because the comparison baseline is precisely the newest model, the reported TBM/rTBM advantage over periodic retraining is confounded by evaluation fairness rather than being a clean demonstration of selection among historical models. The authors should re-run the evaluation with a validation period disjoint from every candidate's training window, or exclude the newest model from the scored candidate set and report how the rankings change.
  3. [Section 3.2.2] The rSBM mechanism 'ignores the data leakage issue' by ranking all historical models together, including models whose training window intersects the selected similar historical period. Since rSBM is one of the mechanisms the abstract credits with prevailing over periodic retraining, its reported advantage and its high alignment with the oracle ranking may be artifacts of in-sample scoring on the selected validation period. The paper should quantify this by comparing rSBM against the leakage-aware SBM and by reporting results when models with intersecting training windows are excluded, so that the similarity-based result is not driven by leakage.
minor comments (5)
  1. [Abstract and Section 1] The word 'mechnisms' is misspelled twice in the abstract and once in the introduction; it should be 'mechanisms'.
  2. [Section 3.3.2] The notation for the Jaccard similarity coefficient uses intersection and union over the same index set in a way that is ambiguous; please define the top-k sets of each ranking explicitly before giving the formula.
  3. [Section 3.1.5] The sentence 'we do not applied downsampling' should be 'we did not apply downsampling'.
  4. [Section 4.1] The Scott-Knott results in Figure 3 are presented in a dense ASCII format that is difficult to read; a table with explicit group labels for each mechanism/model/dataset would improve reproducibility and clarity.
  5. [Section 3.1.4] It would be helpful to state explicitly how many candidate historical models are available at each testing period for each dataset (for example, at most 14 for Google), since the ranking metrics and the Alibaba limitation depend on this number.

Circularity Check

0 steps flagged · score 2.0 of 10

Empirical study with held-out test data; no derivation reduces to its inputs, though minor self-citations and an untested recency prior add small burden.

full rationale

This is an empirical evaluation, not a derivation. The central claim—that temporal-adjacency selection mechanisms (TBM/rTBM, SBM/rSBM) can outperform periodical retraining—is measured by applying the mechanisms' top-ranked models to held-out future test periods and comparing AUC against baselines (Section 4.1). No equation in Section 3 defines the outcome in terms of the input. TBM's assumption that the last labeled period is the best proxy (Section 3.2.1: 'We consider the samples from the last period of time to be the most similar to the testing samples') is a testable hypothesis; the paper's own results show it fails on Alibaba and for several model/dataset combinations, so the conclusion is not forced. The rTBM rule ('when the highest ranked model is from the second latest historical time periods, we assume the model from the latest historical time period (which was not tested due to the concern of data leakage) would be better and bring it to the top') is an untested recency prior, and TBM scoring has an acknowledged leakage concern for the newest model, but these are internal-validity threats: rTBM can still select older models and can underperform retraining, so its edge is not guaranteed by construction. The only self-citations are motivational: 'Our prior works [40,39] find that the operation datasets are subject to a considerable scale of concept drift' (Section 1). That premise is not load-bearing for the measured comparisons, which are self-contained against the three public datasets. No fitted parameter is renamed as a prediction and no uniqueness theorem is imported. Overall, no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the temporal-adjacency assumption and on hand-chosen training and evaluation constants (window length, downsampling ratio, feature window, top-k). No new physical or mathematical entities are introduced. The concept-drift premise is imported from the authors' own prior work, which is a modest circularity burden but not a tautology.

free parameters (4)
  • Sliding window training length = Half of total time periods
    Set separately per dataset (e.g., 14 days for Google, 18 months for Backblaze, 4 weeks for Alibaba) in Section 3.1.5. It determines which models exist and how much history the selection mechanisms can draw on.
  • Majority downsampling ratio = 10:1 success-to-fail for Google and Backblaze
    Used to rebalance training sets in Section 3.1.5. Chosen by hand, it alters the learned models and, indirectly, the results of selection.
  • Feature extraction window = First five minutes of job execution
    Defined in Sections 3.1.1 and 3.1.3 for job failure prediction. Follows prior work but is a hand-chosen constant that shapes all temporal features.
  • Jaccard top-k cutoff = k = 3
    Set in Section 3.3.2 based on prior work; other cutoffs are deferred to the replication package. Affects how ranking agreement is reported.
assumptions (4)
  • domain assumption Gradual drift assumption: the most recent labeled period is the best proxy for the next period.
    Underlies TBM/rTBM in Section 3.2.1; if drift is abrupt, last-period ranking is a poor proxy and the main conclusion weakens.
  • domain assumption Concept drift materially affects model performance in the studied operation datasets.
    The study's motivation relies on this, supported by the authors' prior work [39,40] rather than re-tested here.
  • domain assumption Hausdorff distance is an appropriate similarity measure between multivariate distributions of different sample sizes.
    Used by SBM in Section 3.2.2; other distances are rejected for practical reasons, but the choice is not validated against alternatives.
  • standard math AUC, Kendall's tau, Jaccard, and Kendall's W are valid measures for the stated comparisons.
    Standard metrics with citable backing; their use matches prior model selection studies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can We Recycle Our Old Models? An Empirical Evaluation of Model Selection Mechanisms for AIOps Solutions." pith.science (2026). https://pith.science/paper/5VAHUMYG

@misc{pith2026250502961,
  author       = {Pith},
  title        = {Pith review of: Can We Recycle Our Old Models? An Empirical Evaluation of Model Selection Mechanisms for AIOps Solutions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VAHUMYG}},
  note         = {Machine review of arXiv:2505.02961}
}
read the original abstract

AIOps (Artificial Intelligence for IT Operations) solutions leverage the tremendous amount of data produced during the operation of large-scale systems and machine learning models to assist software practitioners in their system operations. Existing AIOps solutions usually maintain AIOps models against concept drift through periodical retraining, despite leaving a pile of discarded historical models that may perform well on specific future data. Other prior works propose dynamically selecting models for prediction tasks from a set of candidate models to optimize the model performance. However, there is no prior work in the AIOps area that assesses the use of model selection mechanisms on historical models to improve model performance or robustness. To fill the gap, we evaluate several model selection mechanisms by assessing their capabilities in selecting the optimal AIOps models that were built in the past to make predictions for the target data. We performed a case study on three large-scale public operation datasets: two trace datasets from the cloud computing platforms of Google and Alibaba, and one disk stats dataset from the BackBlaze cloud storage data center. We observe that the model selection mechnisms utilizing temporal adjacency tend to have a better performance and can prevail the periodical retraining approach. Our findings also highlight a performance gap between existing model selection mechnisms and the theoretical upper bound which may motivate future researchers and practitioners in investigating more efficient and effective model selection mechanisms that fit in the context of AIOps.

Figures

Figures reproduced from arXiv: 2505.02961 by the authors.

Figure 1
Figure 1. Data schema for our studied datasets. Each colored box represents [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. The average AUC performance of model selection mechanisms in each [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Scott-Knott test results of the AUC performance from different multi [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The average Kendall’s τ correlation between model selection mecha￾nisms and the oracle ranking in each testing period. test. Similarly, for the Jaccard similarity coefficient, [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Scott-Knott test results of the Kendall’s [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: The Jaccard similarity coefficient (k = 3) between model selection mechanisms and the oracle ranking in each testing period. ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 rSBM/LR rSBM/NN rTBM/RF CRC/LR CRC/NN LaF/LR rSBM/RF rTBM/LR rTBM/NN T…
Figure 7
Figure 7. Figure 7: Scott-Knott test results of the Jaccard similarity coefficient ( [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: The Kendall’s W correlation among rankings from the same model [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Scott-Knott test results of the Kendall’s W correlation from different [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 73 canonical work pages

  1. [1]

    In: 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Network, DSN 2023, Porto, Portugal, June 27-30, 2023, pp

    Alharthi, K.A., Jhumka, A., Di, S., Gui, L., Cappello, F., McIntosh-Smith, S.: Time machine: Generative real-time model for failure (and lead time) prediction in HPC systems. In: 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Network, DSN 2023, Porto, Portugal, June 27-30, 2023, pp. 508–521. IEEE (2023)

    work page 2023

  2. [2]

    IEEE Trans

    Almodovar, C., Sabrina, F., Karimi, S., Azad, S.A.: Logfit: Log anomaly detection using fine-tuned language models. IEEE Trans. Netw. Serv. Manag. 21(2), 1715–1723 (2024)

    work page 2024

  3. [3]

    In: 2020 USENIX Conference on Operational Machine Learning, OpML 2020, July 28 - August 7, 2020 (2020)

    Banerjee, A., Chen, C., Hung, C., Huang, X., Wang, Y., Chevesaran, R.: Challenges and experiences with mlops for performance diagnostics in hybrid-cloud enterprise software deployments. In: 2020 USENIX Conference on Operational Machine Learning, OpML 2020, July 28 - August 7, 2020 (2020)

    work page 2020

  4. [4]

    In: ICSE-SEIP 2020: 42nd International Conference on Software Engineering, Software Engineering in Practice, Seoul, South Korea, 27 June - 19 July, 2020 (2020)

    Bansal, C., Renganathan, S., Asudani, A., Midy, O., Janakiraman, M.: Decaf: diagnosing and triaging performance issues in large-scale cloud services. In: ICSE-SEIP 2020: 42nd International Conference on Software Engineering, Software Engineering in Practice, Seoul, South Korea, 27 June - 19 July, 2020 (2020)

    work page 2020

  5. [5]

    Bendimerad, A., Remil, Y., Mathonat, R., Kaytoue, M.: On-premise AIOps infrastruc- ture for a software editor SME: an experience report. In: S. Chandra, K. Blincoe, P. Tonella (eds.) Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, De...

    work page 2023

  6. [6]

    Journal of Machine Learning Research 13(10), 281–305 (2012)

    Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. Journal of Machine Learning Research 13(10), 281–305 (2012)

    work page 2012

  7. [7]

    In: System Modeling and Optimization: Proceedings of the 22nd IFIP TC7 Conference held from July 18–22, 2005, in Turin, Italy 22, pp

    Birsan, T., Tiba, D.: One hundred years since the introduction of the set distance by dimitrie pompeiu. In: System Modeling and Optimization: Proceedings of the 22nd IFIP TC7 Conference held from July 18–22, 2005, in Turin, Italy 22, pp. 35–39. Springer (2006) 3 https://github.com/EmpyreanKnight/suppmaterial-25-yingzhe-AIOpsSelection Title Suppressed Due ...

    work page 2006

  8. [8]

    In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pp

    Botezatu, M.M., Giurgiu, I., Bogojeska, J., Wiesmann, D.: Predicting disk replacement towards reliable data centers. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pp. 39–48 (2016)

    work page 2016

Show all 76 references
  1. [9]

    In: IEEE/ACM International Workshop on Cloud Intelligence & AIOps, AIOps@ICSE 2023, Melbourne, Australia, May 15, 2023, pp

    Chaudhary, G., Mebratu, D., Lewis, B., Khanna, R., Jin, J., Hossain, M.: Monitor- ing workload performance in noisy neighborhoods using performance monitoring units. In: IEEE/ACM International Workshop on Cloud Intelligence & AIOps, AIOps@ICSE 2023, Melbourne, Australia, May 1...

    2023

  2. [10]

    In: 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019 (2019)

    Chen, J., He, X., Lin, Q., Zhang, H., Hao, D., Gao, F., Xu, Z., Dang, Y., Zhang, D.: Continuous incident triage for large-scale online service systems. In: 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2...

    2019

  3. [11]

    In: 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourne, Australia, September 21-25, 2020 (2020)

    Chen, J., Zhang, S., He, X., Lin, Q., Zhang, H., Hao, D., Kang, Y., Gao, F., Xu, Z., Dang, Y., Zhang, D.: How incidental are the incidents? characterizing and prioritizing incidents for large-scale online service systems. In: 35th IEEE/ACM International Conference on Automated...

    2020

  4. [12]

    In: Proceedings of the 2014 IEEE International Symposium on Software Reliability Engineering Workshops, ISSREW ’14, pp

    Chen, X., Lu, C.D., Pattabiraman, K.: Failure prediction of jobs in compute clouds: A google cluster case study. In: Proceedings of the 2014 IEEE International Symposium on Software Reliability Engineering Workshops, ISSREW ’14, pp. 341–346 (2014)

    2014

  5. [13]

    In: ESEC/FSE ’20: 28th ACM Joint European Software Engineering Confer- ence and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020 (2020)

    Chen, Y., Yang, X., Dong, H., He, X., Zhang, H., Lin, Q., Chen, J., Zhao, P., Kang, Y., Gao, F., Xu, Z., Zhang, D.: Identifying linked incidents in large-scale online service systems. In: ESEC/FSE ’20: 28th ACM Joint European Software Engineering Confer- ence and Symposium on ...

    2020

  6. [14]

    In: Proceedings of the 2019 World Wide Web Conference, WWW ’19, pp

    Chen, Y., Yang, X., Lin, Q., Zhang, H., Gao, F., Xu, Z., Dang, Y., Zhang, D., Dong, H., Xu, Y., Li, H., Kang, Y.: Outage prediction and diagnosis for cloud service systems. In: Proceedings of the 2019 World Wide Web Conference, WWW ’19, pp. 2659–2665 (2019)

    2019

  7. [15]

    In: Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings, ICSE-Companion ’19, pp

    Dang, Y., Lin, Q., Huang, P.: AIOps: Real-world challenges and research innovations. In: Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings, ICSE-Companion ’19, pp. 4–5 (2019)

    2019

  8. [16]

    Diffenderfer, J., Bartoldson, B.R., Chaganti, S., Zhang, J., Kailkhura, B.: A winning hand: Compressing deep networks can improve out-of-distribution robustness. In: M. Ranzato, A. Beygelzimer, Y.N. Dauphin, P. Liang, J.W. Vaughan (eds.) Advances in Neural Information Processi...

    2021

  9. [17]

    IEEE Signal Process

    Ding, J., Tarokh, V., Yang, Y.: Model selection techniques: An overview. IEEE Signal Process. Mag. 35(6), 16–34 (2018). DOI 10.1109/MSP.2018.2867638. URL https: //doi.org/10.1109/MSP.2018.2867638

    2018

  10. [18]

    In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ASE ’12, pp

    Ding, R., Fu, Q., Lou, J.G., Lin, Q., Zhang, D., Shen, J., Xie, T.: Healing online service systems via mining historical issue repositories. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ASE ’12, pp. 318–321 (2012)

    2012

  11. [19]

    In: Proceedings of the 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN ’14, pp

    Ding, R., Fu, Q., Lou, J.G., Lin, Q., Zhang, D., Xie, T.: Mining historical issue repos- itories to heal large-scale online service systems. In: Proceedings of the 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN ’14, pp. 311–322 (2014)

    2014

  12. [20]

    In: Proceedings of the 37th IEEE International Conference on Distributed Computing Systems, ICDCS ’17, pp

    El-Sayed, N., Zhu, H., Schroeder, B.: Learning from failure across multiple clusters: A trace-driven approach to understanding, predicting, and mitigating job terminations. In: Proceedings of the 37th IEEE International Conference on Distributed Computing Systems, ICDCS ’17, p...

    2017

  13. [21]

    https://github.com/alibaba/clusterdata (2021)

    Group, A.: Alibaba cluster trace program. https://github.com/alibaba/clusterdata (2021)

    2021

  14. [22]

    In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE ’18, pp

    He, S., Lin, Q., Lou, J.G., Zhang, H., Lyu, M.R., Zhang, D.: Identifying impactful service system problems via log analysis. In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering,...

    2018

  15. [23]

    ACM Trans

    Hu, Q., Guo, Y., Xie, X., Cordy, M., Papadakis, M., Traon, Y.L.: Laf: Labeling-free model selection for automated deep neural network reusing. ACM Trans. Softw. Eng. Methodol. 33(1), 25:1–25:28 (2024)

    2024

  16. [24]

    Backblaze B2 Cloud Storage (2020)

    Inc., B.: Backblaze hard drive stats. Backblaze B2 Cloud Storage (2020). Posted at https://www.backblaze.com/b2/hard-drive-test-data.html

    2020

  17. [25]

    In: 2014 IEEE 7th Interna- tional Conference on Cloud Computing, Anchorage, AK, USA, June 27 - July 2, 2014 (2014)

    Jehangiri, A.I., Yahyapour, R., Wieder, P., Yaqub, E., Lu, K.: Diagnosing cloud perfor- mance anomalies using large time series dataset analysis. In: 2014 IEEE 7th Interna- tional Conference on Cloud Computing, Anchorage, AK, USA, June 27 - July 2, 2014 (2014)

    2014

  18. [26]

    Jiang, J., Lu, W., Chen, J., Lin, Q., Zhao, P., Kang, Y., Zhang, H., Xiong, Y., Gao, F., Xu, Z., Dang, Y., Zhang, D.: How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems. In: Proceedings of the 28th ACM Joint Eur...

    2020

  19. [27]

    In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’18, pp

    Li, H., Chen, T.H.P., Hassan, A.E., Nasser, M., Flora, P.: Adopting autonomic com- puting capabilities in existing large-scale systems: An industrial experience report. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practi...

    2018

  20. [28]

    In: 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2014, Atlanta, GA, USA, June 23-26, 2014 (2014)

    Li, J., Ji, X., Jia, Y., Zhu, B., Wang, G., Li, Z., Liu, X.: Hard drive failure prediction using classification and regression trees. In: 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2014, Atlanta, GA, USA, June 23-26, 2014 (2014)

    2014

  21. [29]

    Li, J., Stones, R.J., Wang, G., Liu, X., Li, Z., Xu, M.: Hard drive failure prediction using decision trees. Reliab. Eng. Syst. Saf. (2017)

    2017

  22. [30]

    ACM Transactions on Software Engineering and Methodology 29(2), 1–24 (2020)

    Li, Y., Jiang, Z.M., Li, H., Hassan, A.E., He, C., Huang, R., Zeng, Z., Wang, M., Chen, P.: Predicting node failures in an ultra-large-scale cloud computing platform: An AIOps solution. ACM Transactions on Software Engineering and Methodology 29(2), 1–24 (2020)

    2020

  23. [31]

    In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pp

    Liao, L., Eismann, S., Li, H., Bezemer, C.P., Costa, D.E., van Hoorn, A., Shang, W.: Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models . In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pp. 317...

    2025

  24. [32]

    In: Proceedings of the 2014 IEEE International Conference on Data Mining, ICDM ’14, pp

    Lim, M.H., Lou, J.G., Zhang, H., Fu, Q., Teoh, A.B.J., Lin, Q., Ding, R., Zhang, D.: Identifying recurrent and unknown performance issues. In: Proceedings of the 2014 IEEE International Conference on Data Mining, ICDM ’14, pp. 320–329 (2014)

    2014

  25. [33]

    In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE ’18, pp

    Lin, Q., Hsieh, K., Dang, Y., Zhang, H., Sui, K., Xu, Y., Lou, J.G., Li, C., Wu, Y., Yao, R., et al.: Predicting node failure in cloud service systems. In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundatio...

    2018

  26. [34]

    In: Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016 (2016)

    Lin, Q., Lou, J., Zhang, H., Zhang, D.: idice: problem identification for emerging issues. In: Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016 (2016)

    2016

  27. [35]

    In: 2022 3rd International Conference on Pattern Recognition and Machine Learning (PRML), pp

    Liu, B.: Consistent relative confidence and label-free model selection for convolutional neural networks. In: 2022 3rd International Conference on Pattern Recognition and Machine Learning (PRML), pp. 375–379 (2022)

    2022

  28. [36]

    In: Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, ASE ’13, pp

    Lou, J., Lin, Q., Ding, R., Fu, Q., Zhang, D., Xie, T.: Software analytics for inci- dent management of online services: An experience report. In: Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, ASE ’13, pp. 475–485 (2013)

    2013

  29. [37]

    Automated Software Engineering 24(4), 905–941 (2017)

    Lou, J.G., Lin, Q., Ding, R., Fu, Q., Zhang, D., Xie, T.: Experience report on apply- ing software analytics in incident management of online service. Automated Software Engineering 24(4), 905–941 (2017)

    2017

  30. [38]

    In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, pp

    Luo, C., Lou, J.G., Lin, Q., Fu, Q., Ding, R., Zhang, D., Wang, Z.: Correlating events with time series for incident diagnosis. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, pp. 1583– 1592 (2014) Title Suppress...

    2014

  31. [39]

    ACM Trans

    Lyu, Y., Li, H., Jiang, Z.M.J., Hassan, A.E.: On the model update strategies for su- pervised learning in aiops solutions. ACM Trans. Softw. Eng. Methodol. 33(7), 184:1– 184:38 (2024)

    2024

  32. [40]

    ACM Transactions on Software Engineering and Methodology (2021)

    Lyu, Y., Li, H., Sayagh, M., Jiang, Z.M.J., Hassan, A.E.: An empirical study of the im- pact of data splitting decisions on the performance of aiops solutions. ACM Transactions on Software Engineering and Methodology (2021)

    2021

  33. [41]

    ACM Trans

    Lyu, Y., Rajbahadur, G.K., Lin, D., Chen, B., Jiang, Z.M.J.: Towards a consistent interpretation of aiops models. ACM Trans. Softw. Eng. Methodol. 31(1), 16:1–16:38 (2022)

    2022

  34. [42]

    In: Proceedings of the 2017 USENIX Annual Technical Con- ference, ATC ’17, pp

    Mahdisoltani, F., Stefanovici, I., Schroeder, B.: Proactive error prediction to improve storage system reliability. In: Proceedings of the 2017 USENIX Annual Technical Con- ference, ATC ’17, pp. 391–402 (2017)

    2017

  35. [43]

    In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp

    Meng, L., Li, Y., Chen, L., Wang, Z., Wu, D., Zhou, Y., Xu, B.: Measuring discrimination to boost comparative testing for multiple deep learning models. In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 385–396. I...

    2021

  36. [44]

    Misiakos, P., Wendler, C., P¨ uschel, M.: Learning dags from data with few root causes. In: Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 (2023)

    2023

  37. [45]

    In: Discov- ery Science, pp

    Nishida, K., Yamauchi, K.: Detecting concept drift using statistical testing. In: Discov- ery Science, pp. 264–269. Springer (2007)

    2007

  38. [46]

    ACM Trans

    Openja, M., Khomh, F., Foundjem, A., Jiang, Z.M.J., Abidi, M., Hassan, A.E.: An em- pirical study of testing machine learning in the wild. ACM Trans. Softw. Eng. Methodol. 34(1), 7:1–7:63 (2025)

    2025

  39. [47]

    Neural Networks 113, 54–71 (2019)

    Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Networks 113, 54–71 (2019). DOI 10.1016/J. NEUNET.2019.01.012. URL https://doi.org/10.1016/j.neunet.2019.01.012

    2019 doi

  40. [48]

    In: Pro- ceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI, pp

    Poenaru-Olaru, L., Karpova, N., Cruz, L., Rellermeyer, J.S., van Deursen, A.: Is your anomaly detector ready for change? adapting aiops solutions to the real world. In: Pro- ceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI, pp...

    2024

  41. [49]

    Gartner Research (2018)

    Prasad, P., Rich, C.: Market guide for AIOps platforms. Gartner Research (2018). Posted at https://www.gartner.com/doc/3892967/market-guide-aiops-platforms

    2018

  42. [50]

    In: IEEE Trans- actions on Software Engineering (2021)

    Rajbahadur, G.K., Wang, S., Oliva, G.A., Kamei, Y., Hassan, A.E.: The impact of feature importance methods on the interpretation of defect classifiers. In: IEEE Trans- actions on Software Engineering (2021)

    2021

  43. [51]

    In: Proceedings of the 2015 IEEE 23rd International Symposium on Quality of Service, IWQoS ’15, pp

    Ros` a, A., Chen, L.Y., Binder, W.: Catching failures of failures at big-data clusters: A two-level neural network approach. In: Proceedings of the 2015 IEEE 23rd International Symposium on Quality of Service, IWQoS ’15, pp. 231–236 (2015)

    2015

  44. [52]

    In: Proceedings of the 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid ’15, pp

    Ros` a, A., Chen, L.Y., Binder, W.: Predicting and mitigating jobs failures in big data clusters. In: Proceedings of the 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid ’15, pp. 221–230 (2015)

    2015

  45. [53]

    Biometrics 30(3), 507–512 (1974)

    Scott, A.J., Knott, M.: A cluster analysis method for grouping means in the analysis of variance. Biometrics 30(3), 507–512 (1974)

    1974

  46. [54]

    In: Proceedings of the 9th International Conference on Predictive Models in Software Engineering, PROMISE ’13 (2013)

    Song, L., Minku, L.L., Yao, X.: The impact of parameter tuning on software effort estimation using learning machines. In: Proceedings of the 9th International Conference on Predictive Models in Software Engineering, PROMISE ’13 (2013)

    2013

  47. [55]

    In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’18, pp

    Tantithamthavorn, C., Hassan, A.E.: An experience report on defect modelling in prac- tice: Pitfalls and challenges. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’18, pp. 286–295 (2018)

    2018

  48. [56]

    In: Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, pp

    Tantithamthavorn, C., McIntosh, S., Hassan, A.E., Matsumoto, K.: Automated parame- ter optimization of classification techniques for defect prediction models. In: Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, pp. 321–332 (2016)

    2016

  49. [57]

    IEEE Transactions on Software Engineering 45(7), 683–711 (2018) 30 Yingzhe Lyu et al

    Tantithamthavorn, C., McIntosh, S., Hassan, A.E., Matsumoto, K.: The impact of au- tomated parameter optimization on defect prediction models. IEEE Transactions on Software Engineering 45(7), 683–711 (2018) 30 Yingzhe Lyu et al

    2018

  50. [58]

    Computer Science Department, Trinity College Dublin 106(2), 58 (2004)

    Tsymbal, A.: The problem of concept drift: Definitions and related work. Computer Science Department, Trinity College Dublin 106(2), 58 (2004)

    2004

  51. [59]

    In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’03, pp

    Wang, H., Fan, W., Yu, P.S., Han, J.: Mining concept-drifting data streams using en- semble classifiers. In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’03, pp. 226–235 (2003)

    2003

  52. [60]

    Wang, H., Yu, P.S., Han, J.: Mining Concept-Drifting Data Streams, pp. 789–802. Springer (2010)

    2010

  53. [61]

    In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pp

    Wang, L., Zhang, C., Ding, R., Xu, Y., Chen, Q., Zou, W., Chen, Q., Zhang, M., Gao, X., Fan, H., Rajmohan, S., Lin, Q., Zhang, D.: Root cause analysis for microservice systems via hierarchical reinforcement learning from human feedback. In: Proceedings of the 29th ACM SIGKDD C...

    2023

  54. [62]

    In: 19th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 22) (2022)

    Weng, Q., Xiao, W., Yu, Y., Wang, W., Wang, C., He, J., Li, Y., Zhang, L., Lin, W., Ding, Y.: MLaaS in the wild: Workload analysis and scheduling in large-scale hetero- geneous GPU clusters. In: 19th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 22) (2022)

    2022

  55. [63]

    Technical report, Google Inc

    Wilkes, J.: Google cluster-usage traces v3. Technical report, Google Inc. (2020). Posted at https://github.com/google/cluster-data/tree/master

    2020

  56. [64]

    Wu, X., Li, H., Khomh, F.: On the effectiveness of log representation for log-based anomaly detection. Empir. Softw. Eng. 28(6), 137 (2023)

    2023

  57. [65]

    In: Proceedings of the 2018 USENIX Annual Technical Conference, ATC ’15, pp

    Xu, Y., Sui, K., Yao, R., Zhang, H., Lin, Q., Dang, Y., Li, P., Jiang, K., Zhang, W., Lou, J.G., Chintalapati, M., Zhang, D.: Improving service availability of cloud systems by predicting disk error. In: Proceedings of the 2018 USENIX Annual Technical Conference, ATC ’15, pp. ...

    2018

  58. [66]

    In: Proceedings of the 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN ’16, pp

    Xue, J., Birke, R., Chen, L.Y., Smirni, E.: Managing data center tickets: Prediction and active sizing. In: Proceedings of the 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN ’16, pp. 335–346 (2016)

    2016

  59. [67]

    IEEE Transactions on Network and Service Manage- ment 15(1), 39–52 (2018)

    Xue, J., Birke, R., Chen, L.Y., Smirni, E.: Spatial-temporal prediction models for active ticket managing in data centers. IEEE Transactions on Network and Service Manage- ment 15(1), 39–52 (2018)

    2018

  60. [68]

    Yang, F., Yin, W., Wang, L., Li, T., Zhao, P., Liu, B., Wang, P., Qiao, B., Liu, Y., Bj¨ orkman, M., Rajmohan, S., Lin, Q., Zhang, D.: Diffusion-based time series data im- putation for cloud failure prediction at microsoft 365. In: Proceedings of the 31st ACM Joint European So...

    2023

  61. [69]

    In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp

    Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., Zhang, W.: Semi- supervised log-based anomaly detection via probabilistic label estimation. In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 1448–1460....

    2021

  62. [70]

    Yao, H., Tang, X., Wei, H., Zheng, G., Li, Z.: Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Arti- ficial Intelli...

    2019

  63. [71]

    In: 45th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp

    Zeng, Z., Zhang, Y., Xu, Y., Ma, M., Qiao, B., Zou, W., Chen, Q., Zhang, M., Zhang, X., Zhang, H., Gao, X., Fan, H., Rajmohan, S., Lin, Q., Zhang, D.: Traceark: Towards ac- tionable performance anomaly alerting for online service systems. In: 45th IEEE/ACM International Confer...

    2023

  64. [72]

    In: 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp

    Zhang, C., Peng, X., Sha, C., Zhang, K., Fu, Z., Wu, X., Lin, Q., Zhang, D.: Deeptralog: Trace-log combined microservice anomaly detection through graph-based deep learning. In: 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA...

    2022

  65. [73]

    Zhang, J., Li, J., Yang, Z.: Dynamic robustness evaluation for automated model selection in operation. Inf. Softw. Technol. 178, 107603 (2025) Title Suppressed Due to Excessive Length 31

    2025

  66. [74]

    arXiv preprint arXiv:2406.11213 (2024)

    Zhang, L., Jia, T., Jia, M., Yang, Y., Wu, Z., Li, Y.: A survey of AIOps for failure management in the era of large language models. arXiv preprint arXiv:2406.11213 (2024)

    2024 arXiv

  67. [75]

    In: 2005 International Conference on Dependable Systems and Networks (DSN 2005), 28 June - 1 July 2005, Yokohama, Japan, Proceedings (2005)

    Zhang, S., Cohen, I., Goldszmidt, M., Symons, J., Fox, A.: Ensembles of models for automated diagnosis of system performance problems. In: 2005 International Conference on Dependable Systems and Networks (DSN 2005), 28 June - 1 July 2005, Yokohama, Japan, Proceedings (2005)

    2005

  68. [76]

    Zhao, N., Chen, J., Wang, Z., Peng, X., Wang, G., Wu, Y., Zhou, F., Feng, Z., Nie, X., Zhang, W., Sui, K., Pei, D.: Real-time incident prediction for online service systems. In: Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Fou...

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.