{"id":"7dad6dc1-6bd7-4123-8895-1f6233b71e87","arxiv_id":"2506.13838","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Retraining failure prediction models on recent data only, and only when drift detectors signal a change, reduces energy consumption while mostly preserving accuracy, but no single retraining strategy is best across all datasets.","lead":"This paper measures how different retraining strategies for machine learning models affect energy consumption and accuracy in failure prediction systems. It finds that retraining on recent data only and triggering retraining only when data-change detectors flag a need can cut energy use substantially, but the best strategy depends on the dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KS trigger rule is unspecified; the drift-retraining results, including the 40% headline, are controlled by this hidden parameter.","rationale":"The paper's central empirical claim about drift-based retraining is conditional on the behavior of the KS detector, and the detector's sensitivity is the single largest control variable in the experiment. The reader identified exactly this gap: the trigger threshold and aggregation rule are never specified. I find that concern load-bearing because the paper's own results demonstrate that detector sensitivity flips the conclusion: for Google, sensitive detectors make informed retraining more expensive than periodic retraining, while for Alibaba, an insensitive detector makes informed retraining equivalent to no retraining in accuracy. The headline 40% figure is produced by one detector-dataset combination that is neither too sensitive nor too insensitive under the authors' unreported settings. Without knowing alpha and the aggregation rule, a reader cannot tell whether the reported savings are a robust property of the method or an artifact of one threshold choice. The concern does not invalidate the sliding-window result, which is well supported, nor does it suggest misconduct; it is an under-specified control parameter that affects the primary novel comparison. Because the issue is addressable by reporting the trigger rule and a sensitivity analysis, the appropriate verdict remains CONDITIONAL, which matches the reader's original verdict, so no adjustment is needed.","tokens_in":15564,"tokens_out":5480,"duration_ms":58355,"concrete_test":"Run the Alibaba and Google experiments from the replication package with the KS significance level varied over alpha = 0.01, 0.05, and 0.1, keeping all other settings fixed, and report the number of triggered retraining events, cumulative training energy, and ROC AUC for KS-ALL, KS-PCA, and KS-FI under both sliding-window and full-history data. If the ranking of detectors or the reported energy savings changes by more than 10% across alpha values, the central informed-retraining claim is not robust to the unspecified trigger rule and must be reported as conditional on that rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The informed-retraining comparison is governed entirely by when the Kolmogorov-Smirnov detector decides to retrain, but that decision rule is never stated. Section V.C defines KS-ALL, KS-PCA, and KS-FI but does not give the significance level alpha, the per-feature p-value aggregation rule, the minimum sample size for the test, or the cadence at which drift is evaluated. The sensitivity of this trigger is not a minor implementation detail: the paper's own results show the two failure modes. For Google, KS-PCA and KS-FI are 'too sensitive' and consume more energy than periodic retraining; for Alibaba, KS-ALL is 'not able to identify any drift' and accuracy falls to the Static baseline. The reported 40% saving comes from KS-FI on Alibaba, a detector that sits between these modes only under the authors' chosen, unreported threshold. A small change in alpha could move any detector from the under-triggering regime to the over-triggering regime, changing the number of retraining events, the total energy, and the ranking of detectors. Because the energy consumed by retraining dominates the detector overhead, the entire informed-vs-periodic comparison and the headline 'up to 40%' claim depend on a parameter the paper never reports. This is also a reproducibility gap: Table II cannot be independently reconstructed without the trigger rule, even with the replication package, since the rule determines which configurations are compared.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of energy consumption in retraining strategies for failure-prediction ML models on three public datasets (Backblaze, Google, Alibaba). It compares sliding-window versus full-history retraining and periodic versus informed (drift-detection-based) retraining, measuring training, drift-detection, and inference energy with CodeCarbon across multiple random seeds, and evaluating accuracy with ROC AUC. The paper claims that sliding-window retraining reduces energy by up to 25% with negligible accuracy loss, and that informed retraining with a well-chosen drift detector can reduce annual energy by up to 40% based on a one-year extrapolation. It concludes that there is no universally optimal retraining strategy, and that the choice of drift detector is decisive.","tokens_in":15834,"tokens_out":6782,"duration_ms":65950,"significance":"If the findings hold, the paper makes a useful empirical contribution to Green AI: it provides direct measurements of retraining energy in a realistic AIOps domain, quantifies the overhead of unsupervised drift detectors (Table I shows under 4%), and demonstrates that the sustainability of informed retraining depends heavily on detector choice. The sliding-window versus full-history result is the most solid contribution, supported by direct measurements, multiple seeds, and Wilcoxon tests. The drift-based savings claim is more fragile because it depends on an unreported detector trigger specification and on a linear one-year extrapolation. The replication package and reliance on public datasets are strengths; the paper is empirically grounded rather than a derivation.","major_comments":[{"comment":"The trigger rule for the KS drift detector is not specified. The text describes KS-ALL, KS-PCA, and KS-FI in terms of feature handling, but it never states the significance level alpha, how per-feature p-values are combined into a retraining decision, the sample size used for each test, or the cadence at which drift is evaluated. Since the number of retraining events determines training energy, and Table I shows detector overhead is under 4%, the entire informed-versus-periodic comparison and Table II depend on this hidden parameter. The reported results already show the sensitivity: KS-PCA and KS-FI are described as too sensitive on Google and consume more energy than periodic retraining, while KS-ALL detects no drift on Alibaba and yields accuracy matching the Static baseline. Without the trigger specification, Table II cannot be reproduced from the manuscript alone, and small changes in alpha could move any detector between under- and over-triggering regimes. Please report the exact trigger rule (including p-value aggregation and evaluation cadence) and include a sensitivity analysis over the threshold and aggregation rule.","section":"Section V.C and Table II"},{"comment":"The 'up to 40%' energy saving is a one-year extrapolation that assumes constant energy and drift frequency, as stated in the text. The measured periods are short (6 months for Backblaze, 1 month for Alibaba, 2 weeks for Google), and the 40% figure is driven by a single detector-dataset combination (KS-FI on Alibaba). Because retraining energy is roughly proportional to the number of drift-triggered retraining events, the estimate is only as good as the assumption that drift frequency in the measured window continues unchanged. The authors should present this as a bounded estimate, report the number of retraining events per configuration, and either justify the stationarity assumption or provide a range under alternative drift-frequency scenarios. The abstract and conclusion currently state the 40% value without this caveat.","section":"Section VI.A.3 and Table II"},{"comment":"The claim that informed retraining saves up to 40% 'provided a reliable data change detector is in place' is based on selecting, after seeing the results, the best detector for each dataset (KS-FI for Backblaze and Alibaba, KS-ALL for Google). The paper offers no a priori method to choose a detector that is neither too sensitive nor too insensitive; the condition 'reliable detector' is therefore outcome-defined rather than operationally specified. As a result, the headline claim is not yet actionable for practitioners. Please add a selection heuristic or validation procedure (for example, using a held-out period to estimate retraining frequency), or soften the claim to state that savings were observed for at least one detector per dataset and that the choice is currently hindsight-based.","section":"Section VI.A.3 and Finding 2"}],"minor_comments":[{"comment":"There is a duplicated word: 'the distribution of the data data learned during the training process' should read 'the distribution of the data learned during the training process.'","section":"Section II.B"},{"comment":"The caption is incomplete: 'over the period of one .' should be 'over the period of one year.' Also, '1.000' and '1.000.000' should be written as '1,000' and '1,000,000' for consistency.","section":"Table II caption"},{"comment":"The sentence 'followed by Alibaba (approx. 1 megajoules)' has a singular-plural agreement error; it should be 'approx. 1 megajoule.'","section":"Section VI.A.3"},{"comment":"The paper uses 'accuracy' interchangeably with ROC AUC. Although the authors define this choice, using 'ROC AUC' consistently in the abstract and findings would avoid potential confusion among readers.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of the paper is worth publishing after the trigger parameters and sensitivity analysis are added. I would support acceptance after a successful revision, but the 40% claim should be reworded to reflect the extrapolation and the hindsight-based detector selection. The reliance on the first author's own earlier work in motivating retraining techniques is understandable, but the contribution over those papers should be stated more explicitly in the introduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing: the paper is worth reading. It's the first controlled measurement I know of that compares sliding-window vs full-history retraining and periodic vs unsupervised-drift retraining on the same failure-prediction models, across three public datasets, with a replication package. The headline 25% energy savings for sliding window is well supported: 30 seeds for two datasets, 5 for Google, Wilcoxon tests, energy measured with CodeCarbon. The accuracy loss is negligible. That's a genuinely useful result for practitioners.\n\nThe second headline, up to 40% savings from drift-triggered retraining, is softer. The comparison depends entirely on when the KS detector fires, and the paper never reports the significance level, how per-feature p-values are combined, or the evaluation cadence. That's not a minor implementation detail: as the paper's own results show, KS-PCA and KS-FI are too sensitive on Google (costing more energy than periodic), while KS-ALL misses drift entirely on Alibaba. The 40% figure comes from KS-FI on Alibaba, which sits between those regimes only under the chosen, unreported threshold. Without the trigger rule, Table II can't be independently reconstructed. The one-year extrapolation also assumes constant energy and drift frequency, and the detector per dataset is selected after seeing the results. All fixable: report the exact trigger, add a sensitivity sweep, and frame the drift claim as conditional on detector choice.\n\nThere's also a useful negative result: retraining strategy barely affects inference energy, which is worth knowing.\n\nWho is this for? People working on Green AI or model maintenance in production. It won't reorganize the field, but it gives a solid empirical baseline. It's the kind of paper that deserves a serious referee: the methodology is mostly careful, the evidence is real, and the missing KS parameters are fixable. My advice: send it to review, and ask the authors to report the full trigger specification and a sensitivity analysis over it.","headline":"Solid empirical study of retraining energy; the sliding-window claim holds up, but the 40% drift-triggered savings rest on an unreported KS trigger rule that should be fixed before acceptance.","tokens_in":16367,"tokens_out":1754,"would_cite":true,"duration_ms":17845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retraining on recent data or on drift signals cuts model-upkeep energy by up to 40 percent without meaningful accuracy loss.","keywords":["sustainable machine learning","model retraining","concept drift","energy efficiency","drift detection","failure prediction","sliding window","green AI"],"falsifier":"Re-run the experiments while sweeping the KS test's significance level between 0.01 and 0.10 and count retraining events; if the one-year energy gap between informed and periodic retraining falls below the reported 7 to 40 percent at any setting in that range, the headline savings are trigger-threshold dependent rather than a stable property of drift-based retraining.","tokens_in":1498,"feed_emoji":"⚡","tokens_out":1705,"duration_ms":66205,"temperature":0.7,"pith_summary":"The paper asks whether the way a machine-learning model is retrained matters for energy, not just for accuracy, in production systems. It reports two quantitative findings: retraining on only the most recent data instead of all accumulated data lowers training energy by up to 25 percent with negligible accuracy change, and retraining only when an unsupervised drift detector signals a distribution shift lowers estimated one-year energy by up to 40 percent relative to fixed periodic retraining, provided the detector is chosen well. The study is built on three failure-prediction applications using random-forest classifiers, measuring CPU, RAM, and GPU energy across eight retraining configurations. The consequence of interest is practical: energy-conscious ML maintenance can avoid sacrificing accuracy, but the choice of drift detector is load-bearing.","feed_headline":"Drift-triggered retraining cuts yearly ML energy by up to 40 percent","feed_subtitle":"An empirical study of failure-prediction models shows retraining decisions shape energy use far more than inference does.","key_machinery":"The machinery is the pairing of a data-selection rule with a retraining trigger. The data rule is the sliding window, which bounds the training set to the most recent period and is compared against full-history training. The trigger is an unsupervised Kolmogorov-Smirnov test, a statistical test that compares two distributions, which is applied to training data versus incoming inference data and fires retraining on significant divergence. Three variants differ in how features enter the comparison: all features, PCA-reduced features, and features selected by Gini importance. The KS test is what converts retrain-when-needed into an executable policy, and the variants are what make the energy comparison concrete. Energy is measured per pipeline stage with the CodeCarbon tool using RAPL and NVML.","core_discovery":"On the paper's own terms, the central discovery is empirical: over real-world failure-prediction datasets, the dominant driver of lifecycle energy is not the model itself but the retraining policy. Sliding-window retraining, which keeps the training set at a fixed recent size, consumes up to 25 percent less training energy than full-history retraining while matching or nearly matching ROC AUC. Drift-based informed retraining, when the detector is neither blind nor hypersensitive, is estimated to reduce one-year training-plus-drift energy by roughly 7 percent on Backblaze, 40 percent on Alibaba, and 10 percent on Google compared with periodic retraining. Inference energy is essentially unaffected by the retraining strategy. No single configuration wins everywhere, so the authors frame the contribution as a quantitative guide for selecting retraining techniques.","pith_inferences":["If the same relationship holds beyond random forests and failure prediction, retraining policy, not model architecture, should be the first lever in green-ML audits, and studies reporting only training energy may miss the dominant lifecycle decision.","The ranking of the three drift detectors is likely tied to the unstated KS significance threshold, so sweeping that threshold would show whether the reported 40 percent saving is robust or an artifact of one trigger sensitivity.","A testable extension is to run the drift detector on a subsample or at lower frequency than every inference batch, since detector energy is small but nonzero, which could improve net savings or reveal a floor where detection cost outweighs retraining saved.","The experiments exhibit both failure modes of drift detectors, missing drift and crying wolf, so a practical decision rule is to choose the least sensitive detector that still catches the drifts that actually degrade AUC."],"forward_implications":["A model maintained with a sliding window should consume up to 25 percent less retraining energy than one retrained on full history, with ROC AUC differences of at most a few percentage points on these datasets.","Deploying a well-chosen unsupervised drift detector instead of a fixed schedule is estimated to cut one-year training-plus-drift energy by about 7 to 40 percent depending on the dataset.","Detector overhead is small, under 4 percent of combined training and detection energy, so the main cost of a poor detector is unnecessary retraining rather than detection itself.","Retraining strategy does not materially change inference energy, so energy optimization should focus on training and retraining frequency.","There is no universally best technique; the recommendation is to pair drift-based triggering with sliding-window data and to validate detector sensitivity per application."],"supporting_citations":[{"why":"Supplies the AIOps failure-prediction pipelines, datasets, and periodic retraining convention that the experiments replicate.","marker":"[6]"},{"why":"Establishes the sliding-window versus full-history comparison and the AIOps model maintenance setup that this study extends to energy.","marker":"[7]"},{"why":"Provides the informed-retraining precedent and the datasets and design used for the Alibaba and Google failure-prediction models.","marker":"[14]"},{"why":"Prior proposal that sliding-window and drift-based retraining are more sustainable, which this paper validates with energy measurements.","marker":"[10]"},{"why":"Motivates the choice of Kolmogorov-Smirnov unsupervised drift detection and warns that detector sensitivity drives retraining behavior.","marker":"[19]"},{"why":"Source for the Kolmogorov-Smirnov test as an unsupervised drift detection method and for concept-drift degradation background.","marker":"[3]"},{"why":"Frames the periodic versus informed retraining perspectives from the concept-drift adaptation literature.","marker":"[24]"},{"why":"Documents energy consumption of drift detectors and trade-offs with accuracy, informing the drift-detection overhead analysis.","marker":"[18]"},{"why":"The energy measurement tool used to produce the training, drift-detection, and inference energy numbers.","marker":"[39]"}],"fun_headline_variants":["Retraining policy, not model, drives ML energy use","Smart retraining cuts ML energy up to 40%","Sliding-window retraining saves up to 25% energy","Drift-aware retraining: 40% less energy for ML","Retraining choice: up to 40% energy savings"],"cache_read_input_tokens":18560,"weakest_assumption_plain":"The findings for drift-based retraining rest on the trigger rule of the KS detector, yet the paper never states the significance threshold or how per-feature p-values are combined, and a different threshold changes how often the model retrains and therefore how large the reported savings are.","fun_headline_variants_meta":{"raw":{"variants":["Retraining policy, not model, drives ML energy use","Smart retraining cuts ML energy up to 40%","Sliding-window retraining saves up to 25% energy","Drift-aware retraining: 40% less energy for ML","Retraining choice: up to 40% energy savings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1555,"prompt_tokens":914,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":530,"tokens_out":641,"duration_ms":5405,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:00:55.476358+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiments while sweeping the KS test's significance level between 0.01 and 0.10 and count retraining events; if the one-year energy gap between informed and periodic retraining falls below the reported 7 to 40 percent at any setting in that range, the headline savings are trigger-threshold dependent rather than a stable property of drift-based retraining.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AIOps failure-prediction pipelines, datasets, and periodic retraining convention that the experiments replicate."},{"cited_title":"Towards a consistent interpretation of aiops models","cited_arxiv_id":null,"evidence_quote":"Establishes the sliding-window versus full-history comparison and the AIOps model maintenance setup that this study extends to energy."},{"cited_title":"Poenaru-Olaru, J","cited_arxiv_id":null,"evidence_quote":"Prior proposal that sliding-window and drift-based retraining are more sustainable, which this paper validates with energy measurements."},{"cited_title":"Poenaru-Olaru, L","cited_arxiv_id":null,"evidence_quote":"Motivates the choice of Kolmogorov-Smirnov unsupervised drift detection and warns that detector sensitivity drives retraining behavior."},{"cited_title":"Learning under concept drift: A review","cited_arxiv_id":null,"evidence_quote":"Frames the periodic versus informed retraining perspectives from the concept-drift adaptation literature."},{"cited_title":"How to sustainably monitor ml-enabled systems? accuracy and energy efficiency tradeoffs in concept drift detection","cited_arxiv_id":null,"evidence_quote":"Documents energy consumption of drift detectors and trade-offs with accuracy, informing the drift-detection overhead analysis."},{"cited_title":"de Araújo, JPW, and Minerva Books","cited_arxiv_id":null,"evidence_quote":"The energy measurement tool used to produce the training, drift-detection, and inference energy numbers."}],"review_version":2}