{"id":"f5fc8272-38ca-45eb-8ccf-34ad2fbb41ad","arxiv_id":"2505.02961","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Selecting from previously discarded historical models, especially using temporal adjacency, can match or beat periodic retraining for AIOps failure prediction on cloud data.","lead":"This paper tests whether old, previously trained AIOps models that were scheduled for discard can be reused to predict future failures better than retraining a model from scratch. On three public cloud operation datasets, selection methods that favor recent-time models often match or beat the standard periodic retraining strategy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TBM/rTBM validation may be biased: the last labeled period overlaps the newest model's training window, and rTBM's recency promotion is an untested assumption that could explain its edge over retraining.","rationale":"The reader's weakest assumption is that the last labeled period is a reliable proxy for the next period; my concern sharpens this into a concrete internal-validity flaw in how the proxy is measured. The TBM/rTBM protocol either scores the newest model on its own training data (leakage) or excludes it and then inserts it via an untested recency assumption. Both paths mean the observed rTBM advantage over retraining is not a fair comparison of model selection against always choosing the newest model. This is load-bearing because the abstract and conclusion specifically claim that temporal-adjacency mechanisms can prevail periodic retraining. I do not call the paper fraudulent or claim the result is false; the empirical wins may survive a leakage-free check, and other mechanisms (CRC, rSBM) also show wins. But the central claim currently rests on a protocol that conflates evaluation fairness with the treatment. A single targeted re-analysis on the two datasets where the wins were observed would settle whether the concern lands. I therefore recommend CONDITIONAL rather than REJECT: the paper is otherwise well-structured, uses public data, and provides a replication package, so the appropriate response is to condition acceptance on verifying that the temporal-adjacency advantage survives a leakage-free validation protocol.","tokens_in":24131,"tokens_out":5375,"duration_ms":57443,"concrete_test":"Re-run the Google and Backblaze experiments with a leakage-free validation protocol for TBM/rTBM: for each target period t, evaluate every candidate model on a validation period that is out-of-sample for that candidate. In particular, train the would-be newest model on data ending at t-2 (so t-1 is out-of-sample) and use its t-1 AUC as its selection score; then compare rTBM and TBM against the periodical retraining baseline using the same Scott-Knott procedure as in Section 4.1. If the statistically significant advantage of rTBM/TBM over retraining disappears or drops below significance, the central claim is not supported by the current protocol.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that temporal-adjacency selection mechanisms 'can prevail the periodical retraining approach.' The evidence for this rests mainly on rTBM and rSBM wins on Google and Backblaze (Section 4.1). However, the TBM/rTBM validation protocol has an internal-validity weakness. Per Section 3.1.5, a new sliding-window model is trained after each period, so at target period t the newest candidate was trained on data through t-1. TBM (Section 3.2.1) scores candidates on the 'last available period,' i.e., t-1, which is in-sample for the newest model and out-of-sample for all older ones. If the newest model is included, its score is inflated by leakage. If it is excluded, as the rTBM parenthetical 'which was not tested due to the concern of data leakage' suggests, then rTBM promotes the newest model to the top whenever the second-latest candidate wins, based not on evidence but on an assumption that recency implies superiority. In either reading, the rTBM advantage over retraining is not a clean demonstration that selection among historical models beats always picking the newest model: it is either a leakage artifact or an untested recency prior that reduces to retraining. The same leakage issue affects SBM/rSBM, where Section 3.2.2 acknowledges that rSBM 'ignores the data leakage issue.' Because the headline conclusion is specifically that temporal-adjacency mechanisms outperform periodic retraining, this confounding of evaluation fairness with the treatment itself is the most load-bearing concern: if corrected, the reported statistically significant wins may shrink or disappear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates whether model selection mechanisms can choose among previously trained historical AIOps models as an alternative to periodically retraining a single model. Using Google cluster trace, Backblaze disk stats, and Alibaba GPU cluster trace datasets, the authors train sliding-window models and compare six selection mechanisms (TBM, rTBM, SBM, rSBM, CRC, LaF) plus stationary and periodic-retraining baselines and a hypothetical oracle. The main reported finding is that temporal-adjacency-based mechanisms, particularly rTBM and rSBM, outperform periodic retraining on Google and Backblaze, with rTBM sometimes matching the oracle, while a performance gap remains between all mechanisms and the oracle.","tokens_in":24491,"tokens_out":6388,"duration_ms":69698,"significance":"If the central claim holds, the paper would provide a practical alternative to expensive periodic retraining in AIOps: instead of discarding historical models, practitioners could select among them based on temporal adjacency or similarity. The study has notable strengths: it uses three large public datasets, 100 repeated runs with different random seeds, Scott-Knott statistical grouping, a replication package, and an explicit threats-to-validity section. However, the empirical claim is currently undermined by internal-validity concerns about data leakage in the TBM/rTBM and rSBM evaluation protocols, and by an apparent inconsistency in the definition of rTBM that makes its reported advantage over retraining difficult to interpret. The work is a reasonable contribution to an under-studied problem, but the headline conclusion is not yet cleanly supported.","major_comments":[{"comment":"The definition of rTBM appears to make it equivalent to the periodical retraining baseline. TBM ranks candidates by their performance on the last available period, and rTBM promotes the model from the latest historical time period to the top whenever the highest-ranked model is from a second-latest period. Since the latest historical model is exactly the sliding-window model trained through the previous period, and the periodical retraining baseline selects exactly that model, rTBM as written always selects the same model as retraining. In that case, the reported statistically significant advantages of rTBM over Retrain in Section 4.1 (e.g., Google LR, Backblaze CART and LR) cannot arise. Please clarify the actual rTBM procedure, or correct the definition; as written, the paper is internally inconsistent on a load-bearing point.","section":"Section 3.2.1 and Section 3.2.5"},{"comment":"TBM's scoring of candidates on the 'last available period' (period t-1) is in-sample for the newest candidate, whose sliding training window ends at t-1, while it is out-of-sample for older candidates. This systematically inflates the newest model's estimated performance relative to older models, independent of its true generalization ability. Because the comparison baseline is precisely the newest model, the reported TBM/rTBM advantage over periodic retraining is confounded by evaluation fairness rather than being a clean demonstration of selection among historical models. The authors should re-run the evaluation with a validation period disjoint from every candidate's training window, or exclude the newest model from the scored candidate set and report how the rankings change.","section":"Section 3.2.1 with Section 3.1.5"},{"comment":"The rSBM mechanism 'ignores the data leakage issue' by ranking all historical models together, including models whose training window intersects the selected similar historical period. Since rSBM is one of the mechanisms the abstract credits with prevailing over periodic retraining, its reported advantage and its high alignment with the oracle ranking may be artifacts of in-sample scoring on the selected validation period. The paper should quantify this by comparing rSBM against the leakage-aware SBM and by reporting results when models with intersecting training windows are excluded, so that the similarity-based result is not driven by leakage.","section":"Section 3.2.2"}],"minor_comments":[{"comment":"The word 'mechnisms' is misspelled twice in the abstract and once in the introduction; it should be 'mechanisms'.","section":"Abstract and Section 1"},{"comment":"The notation for the Jaccard similarity coefficient uses intersection and union over the same index set in a way that is ambiguous; please define the top-k sets of each ranking explicitly before giving the formula.","section":"Section 3.3.2"},{"comment":"The sentence 'we do not applied downsampling' should be 'we did not apply downsampling'.","section":"Section 3.1.5"},{"comment":"The Scott-Knott results in Figure 3 are presented in a dense ASCII format that is difficult to read; a table with explicit group labels for each mechanism/model/dataset would improve reproducibility and clarity.","section":"Section 4.1"},{"comment":"It would be helpful to state explicitly how many candidate historical models are available at each testing period for each dataset (for example, at most 14 for Google), since the ranking metrics and the Alibaba limitation depend on this number.","section":"Section 3.1.4"}],"recommendation":"major_revision","confidential_remarks":"The rTBM definitional issue is serious: if rTBM truly reduces to periodic retraining, a key piece of the paper's headline evidence vanishes, and the remaining support depends on leakage-prone evaluation protocols. I would not accept the paper without a clarified rTBM definition and a leakage-free re-analysis. I do not see concerns about novelty disclosure or citation behavior; the concept-drift premise drawn from the authors' prior work is acceptable for an empirical study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is the first to evaluate model selection mechanisms on historical AIOps models, and the empirical scaffolding is solid: three public datasets, four classifiers, 100 repetitions, Scott-Knott grouping, a replication package, and a clear threats section. I agree with the reader's accept-with-moderate-confidence posture, but the stress-test note is right and more load-bearing than the reader's own weakest-assumption framing suggests.\n\nThe problem is in TBM/rTBM. At target period t, the newest candidate model is trained on data through t-1, and TBM scores candidates on the last available period, t-1. That is in-sample for the newest model and out-of-sample for everyone else. rTBM excludes the newest model from testing and then promotes it to the top whenever the second-latest model wins. So rTBM's advantage over periodical retraining is either leakage or an untested recency prior that is essentially retraining with a different label. The paper acknowledges the leakage in Sections 3.2.1 and 3.2.2, but it still draws the abstract conclusion that temporal-adjacency mechanisms 'can prevail' retraining. That conclusion is not cleanly supported for TBM/rTBM.\n\nThat said, the paper is not hollow. SBM, which controls for leakage by placing no-leakage models above leaked ones, still beats retraining on some model/dataset pairs (e.g., LR on Google and Backblaze). So the idea that old models can be recycled has some genuine support, just weaker and narrower than the abstract implies. The Alibaba arm adds little: four test periods, no significant differences, as the authors admit. The citation pattern is fine—the self-cites to [39,40] are prior work on drift in these datasets, not padding.\n\nBottom line: this deserves a serious referee. The authors should rerun TBM/rTBM without the leakage-confounded validation (e.g., exclude the newest model and don't promote it by assumption), report what survives, and soften the abstract accordingly. If the result shrinks to 'SBM sometimes helps,' that is still a publishable empirical finding.","headline":"A careful empirical study whose headline claim is partly undercut by a leakage/recency-assumption issue in TBM/rTBM; still worth serious peer review.","tokens_in":25018,"tokens_out":4184,"would_cite":false,"duration_ms":43769,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Old AIOps models that are normally discarded after retraining can be recycled: ranking them by performance on the most recent labeled period selects a top model that tends to beat the periodically retrained baseline.","keywords":["AIOps","model selection","historical models","concept drift","temporal adjacency","periodical retraining","failure prediction","model recycling"],"falsifier":"Run the six selection mechanisms on an operation trace where concept drift events are explicitly annotated and occur inside, rather than at the boundary of, the fixed time periods; if the temporal-adjacency mechanisms no longer beat the periodically retrained model, the last-period proxy is not the active ingredient.","tokens_in":23964,"feed_emoji":"♻️","tokens_out":6936,"duration_ms":69634,"temperature":0.7,"pith_summary":"This paper asks whether the historical models that AIOps (artificial intelligence for IT operations) teams accumulate during periodic retraining, and usually discard, can be recycled for future predictions. The authors evaluate six model selection mechanisms on three large public operation datasets (Google cluster traces, Backblaze disk stats, and Alibaba GPU traces) by using each mechanism to pick a top model from the history for every new testing period. They find that mechanisms exploiting temporal adjacency, especially the revised temporal-adjacency selector (rTBM) and the revised similarity selector (rSBM), tend to outperform the standard periodically retrained model in AUC and can even match a hypothetical oracle that knows the future labels in one configuration. If this holds, AIOps maintenance could shift from mandatory retraining to selecting among accumulated historical models, avoiding the cost of constant updates. The paper also documents a persistent gap between current mechanisms and the theoretical upper bound, which it offers as the target for future work.","feed_headline":"Recency-based selection beats periodic retraining","feed_subtitle":"Ranking past AIOps models by last-period performance wins on Google, Alibaba, and Backblaze traces.","key_machinery":"The central mechanism is temporal-adjacency-based ranking over a history of models. For each upcoming unlabeled period, the selector scores every previously trained model by its prediction performance (AUC) on the most recent labeled period—under the assumption that adjacent periods have the closest distributions—and deploys the top-ranked model; the revised rTBM variant specifically assumes that if the second-most-recent model wins, the untested newest model would be even better and promotes it to the top. The similarity-based sibling SBM/rSBM instead picks the most similar historical period using Hausdorff distance, a measure of how far two sample sets are from each other, between the testing period's samples and each candidate period's samples. The paper measures these mechanisms against periodical retraining and against a hypothetical oracle that cheats by using the true labels of the test period.","core_discovery":"The paper claims that the pile of historical AIOps models normally thrown away after periodic retraining can be reused as a candidate pool: instead of always deploying the newest model, a maintenance system can rank the old models and deploy the top-ranked one for the next period. On Google cluster traces, Backblaze disk stats, and Alibaba GPU traces, selection mechanisms that exploit temporal adjacency—ranking each historical model by its performance on the most recent labeled period (TBM), optionally promoting the newest model (rTBM), or using Hausdorff distance to find the most similar past period (SBM/rSBM)—tend to beat the periodically retrained baseline in AUC, and rTBM is statistically indistinguishable from the oracle in one configuration. The paper also shows that the same mechanisms' rankings agree with the oracle's ranking (best for rSBM) and are mostly stable across repeated runs, while a gap remains between any real mechanism and the oracle upper bound.","pith_inferences":["An implication the authors leave implicit is that the ranking itself, not the retrained weights, is the reusable asset: a production AIOps system could archive every trained checkpoint, score it on the most recent labeled window, and deploy the top scorer, retraining only when no archived model clears a quality bar.","Because each dataset is tested at only one fixed period length (daily, weekly, or monthly), the temporal-adjacency effect might be period-length dependent; sweeping period granularity or aligning periods to detected drift events would test whether the advantage survives.","The gap between the mechanisms and the oracle suggests that combining temporal-adjacency ranking with label-free confidence signals such as CRC or LaF, rather than using either family alone, is a natural next step.","The strong ranking agreement of rSBM with the oracle could be exploited directly to weight a time-based ensemble of historical models, a direction the paper names as possible future work."],"forward_implications":["On the Google and Backblaze datasets, several selection mechanisms, especially rTBM, rSBM, TBM, and CRC, achieve statistically higher AUC than the periodically retrained model for at least one base classifier.","The oracle baseline beats periodical retraining for most models on Google and Backblaze, showing that there is real headroom in the discarded historical models that current selectors do not fully capture.","Mechanisms based on temporal adjacency generally beat their non-temporal counterparts, and the revised versions that promote the most recent model outperform the originals.","rSBM produces rankings most aligned with the oracle by Kendall's $\\tau$ and top-3 Jaccard similarity across datasets and models.","Most mechanisms produce stable rankings across 100 random-seed runs, with strong Kendall's $W$ agreement, except some LaF and rTBM combinations that show weak agreement."],"supporting_citations":[{"why":"Establishes that operation datasets undergo concept drift that degrades deployed AIOps model performance, motivating the search for better maintenance strategies.","marker":"[39]"},{"why":"Supplies the periodical retraining practice that the paper uses as its main baseline for comparison.","marker":"[30]"},{"why":"Provides the labeling-free CRC model selection mechanism evaluated as one of the historical-model selectors.","marker":"[35]"},{"why":"Provides the labeling-free LaF model selection mechanism evaluated as another historical-model selector.","marker":"[23]"},{"why":"Defines the Google job-failure prediction task, features, and preprocessing reused in the Google case study.","marker":"[20]"},{"why":"Defines the Backblaze disk-failure prediction task, SMART features, and evaluation protocol reused in the disk case study.","marker":"[42]"},{"why":"Supplies the Google cluster trace dataset used as one of the three case-study subjects.","marker":"[63]"},{"why":"Supplies the Backblaze disk stats dataset used as another case-study subject.","marker":"[24]"},{"why":"Supplies the Alibaba GPU cluster trace dataset used as the third case-study subject.","marker":"[21]"}],"fun_headline_variants":["Recycling old AIOps models beats periodic retraining","Past AIOps models outrank fresh retrains","Recency-based model reuse tops baseline","Old AIOps models win via recency selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the most recent labeled period is the best available preview of the next unlabeled period, so ranking old models by their performance on that last labeled period predicts how they will rank on the upcoming one.","fun_headline_variants_meta":{"raw":{"variants":["Recycling old AIOps models beats periodic retraining","Past AIOps models outrank fresh retrains","Recency-based model reuse tops baseline","Old AIOps models win via recency selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1565,"prompt_tokens":984,"completion_tokens":581,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":523}},"tokens_in":600,"tokens_out":581,"duration_ms":6771,"temperature":1.0,"reasoning_tokens":523,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:38:31.232894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the six selection mechanisms on an operation trace where concept drift events are explicitly annotated and occur inside, rather than at the boundary of, the fixed time periods; if the temporal-adjacency mechanisms no longer beat the periodically retrained model, the last-period proxy is not the active ingredient.","supporting_citations":[{"cited_title":"Backblaze B2 Cloud Storage (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the Backblaze disk stats dataset used as another case-study subject."},{"cited_title":"https://github.com/alibaba/clusterdata (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the Alibaba GPU cluster trace dataset used as the third case-study subject."},{"cited_title":"ACM Trans","cited_arxiv_id":null,"evidence_quote":"Establishes that operation datasets undergo concept drift that degrades deployed AIOps model performance, motivating the search for better maintenance strategies."},{"cited_title":"ACM Transactions on Software Engineering and Methodology 29(2), 1–24 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the periodical retraining practice that the paper uses as its main baseline for comparison."},{"cited_title":"In: 2022 3rd International Conference on Pattern Recognition and Machine Learning (PRML), pp","cited_arxiv_id":null,"evidence_quote":"Provides the labeling-free CRC model selection mechanism evaluated as one of the historical-model selectors."},{"cited_title":"ACM Trans","cited_arxiv_id":null,"evidence_quote":"Provides the labeling-free LaF model selection mechanism evaluated as another historical-model selector."},{"cited_title":"In: Proceedings of the 37th IEEE International Conference on Distributed Computing Systems, ICDCS ’17, pp","cited_arxiv_id":null,"evidence_quote":"Defines the Google job-failure prediction task, features, and preprocessing reused in the Google case study."},{"cited_title":"In: Proceedings of the 2017 USENIX Annual Technical Con- ference, ATC ’17, pp","cited_arxiv_id":null,"evidence_quote":"Defines the Backblaze disk-failure prediction task, SMART features, and evaluation protocol reused in the disk case study."},{"cited_title":"Technical report, Google Inc","cited_arxiv_id":null,"evidence_quote":"Supplies the Google cluster trace dataset used as one of the three case-study subjects."}],"review_version":1}