{"id":"c32948dd-b90a-46a1-819e-23a07c17f087","arxiv_id":"2505.13693","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"HarmonE couples a MAPE-K self-adaptation loop with MLOps to cut energy use by up to 54.5% while keeping 95% of predictive accuracy in a traffic-flow digital twin.","lead":"This paper introduces HarmonE, an architecture that lets machine learning pipelines monitor their own accuracy and energy use, and adapt by switching models or retraining only when needed. Its tests on a simulated traffic-flow system show large energy savings with a small accuracy loss.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline energy reduction is enforced by the unreported E_ref threshold in Eq. (3); without parameter values and sensitivity analysis, the 54.5% savings could be a tuned artifact rather than a robust property.","rationale":"The paper's central claim is empirical: HarmonE saves 54.5% energy over LSTM+PRT while keeping 95% of its accuracy. The control rule in Eq. (3) makes the energy-saving direction a direct consequence of the unreported design-time reference E_ref and delta, so the magnitude of the savings is not a free discovery but an enforced constraint. Without reporting these parameters and testing sensitivity, the headline number is not interpretable as a general property of the architecture. The retraining-data ambiguity compounds this by leaving unclear whether retraining can actually respond to drift, which is essential to the claimed accuracy preservation. These are the same weaknesses the reader identified, and they land on the central claim rather than on peripheral details. The missing error bars and the ill-specified Switch baseline are secondary and would not, by themselves, overturn the large energy gap, but the unreported controller parameters could. The reader's CONDITIONAL verdict is appropriate; no verdict change is needed, but the requested revisions should include full configuration disclosure and a sensitivity analysis over E_ref.","tokens_in":12528,"tokens_out":5524,"duration_ms":51988,"concrete_test":"Obtain the configuration (E_ref, delta, beta, gamma, S_min, tau_drift, and retraining window definition) from the linked repository or authors. Fix all parameters except E_ref and run the test set for a grid of normalized E_ref values (e.g., 0.3, 0.5, 0.7, 0.9 of max training energy), reporting energy, R2, and adaptation counts. If the (20.62 mJ, R2=0.8628) point changes by more than ~20% in energy or ~0.02 in R2 across the grid, the headline result is an artifact of E_ref choice. Separately, inspect the retraining code to determine whether retraining uses original training data or a sliding window; if it is a sliding window, rerun LSTM+PRT with the same sliding window of 1200 samples to confirm the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (3) in §3.3 makes the energy-saving direction a design choice: when E_i > E_ref, the dynamic threshold tau_E is tightened, forcing the planner to switch to a lower-energy model. The reported 54.5% reduction vs. LSTM+PRT is therefore not an emergent property of HarmonE; it is a consequence of the unreported reference level E_ref and decay factor delta (plus beta, gamma, S_min, tau_drift). If E_ref was set low, or chosen after inspecting test energy, the headline comparison is circular. The paper gives none of these values, and Table 2 reports no error bars or sensitivity, so we cannot distinguish a robust operating point from a tuned one. The retraining-data ambiguity in §4.2 ('1200 samples, consistent with the original training window size') is a second leg of the same concern: if those are the original 1200 training samples, retraining cannot adapt to induced drift; if they are a sliding window, the comparison to periodic retraining needs to be stated explicitly. Both issues directly determine the central 54.5%/95.0% numbers, making this the most load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HarmonE, an architectural approach that augments an MLOps pipeline with a MAPE-K self-adaptation loop to manage sustainability trade-offs, primarily energy consumption versus prediction accuracy. The approach defines sustainability goals and adaptation thresholds at design time via a Decision Map, monitors runtime metrics (accuracy, normalized energy, drift), and triggers model switching, selective retraining, or versioned-model reuse. The evaluation uses a traffic-flow prediction scenario on PeMS data with a spectrum of three models (LR, SVM, LSTM) and compares HarmonE against eight baselines including periodic retraining and a switching-based approach. The headline results are a 54.5% energy reduction versus LSTM+PRT, 95% of LSTM+PRT's R2 accuracy, and 53.4% lower inference time. The paper also reports that the MAPE-K loop itself consumes only about 1.37% of total system energy. The central claims are that HarmonE achieves a favorable accuracy-energy trade-off while triggering fewer, more context-aware adaptations than periodic retraining.","tokens_in":12779,"tokens_out":3541,"duration_ms":33138,"significance":"If the reported results are robust and reproducible, HarmonE would be a useful contribution to sustainable MLOps: it combines an architectural Decision Map with a control-theoretic dynamic threshold, and it provides open-source code and a public dataset. The paper's framing around multi-dimensional sustainability (technical, environmental, economic, social) is relevant and timely. However, the significance currently rests on an evaluation whose main quantitative claims are not yet supported: the energy-saving mechanism is partially engineered through an unreported reference level, the strongest baseline is described as defaulting to the heaviest model, and the retraining protocol is ambiguous. Because these issues directly affect the 54.5%/95% headline numbers, the contribution cannot yet be assessed as a general property of the architecture rather than a tuned artifact.","major_comments":[{"comment":"The central energy-saving claim is not yet established as an emergent property of HarmonE because the dynamic threshold update in Eq. (3) enforces a cap near the reference level E_ref: whenever E_i > E_ref, the threshold τ_E is tightened, forcing adaptation to a lower-energy model. The paper does not report E_ref, δ, β, γ, S_min, τ_drift, ε, or the monitoring interval, and Table 2 contains no sensitivity analysis. Without these values and a study varying E_ref (and δ), the 54.5% reduction versus LSTM+PRT could be a direct consequence of choosing a low reference level rather than a robust property of the architecture. Please report the full parameter configuration and show how the energy-accuracy trade-off changes across a range of E_ref values.","section":"§3.3, Eq. (3) and §5, RQ1"},{"comment":"The Switch baseline is described in §4.2 as switching among LR, SVM, and LSTM based on uncertainties detected by the Managing System, but §5, RQ2 explains that it 'defaults to the heavier LSTM model without retraining smaller models when drift occurs.' This makes Switch a weak comparator: it is essentially HarmonE with retraining and versioned-model reuse disabled, and with the decision logic possibly degraded. A fairer baseline would be an energy-aware switching strategy that also has access to the same model spectrum but without HarmonE's design-time thresholds and versioning, to isolate the contribution of the proposed control mechanism. As reported, the comparison to Switch and Switch+PRT does not separate the benefits of threshold-based control from the benefits of simply having a model-switching capability.","section":"§4.2 and §5, RQ2"},{"comment":"The retraining procedure is ambiguous and could confound the comparison with periodic retraining. The paper states: 'Retraining, whether periodic or triggered (HarmonE), is performed using 1200 samples, consistent with the original training window size.' If those 1200 samples are the same original training data, retraining cannot adapt to the induced drift and HarmonE's drift-triggered retraining is not faithfully implemented. If they are a sliding window of the latest 1200 test samples, then the number of samples used differs from the original 1200-sample training set, and the amount of data consumed by retraining should be reported in the comparison. Please clarify which data are used and how the comparison to periodic retraining handles the different data usage.","section":"§4.2 and §5"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the quantitative claims. Each approach was run five times, and Figure 5 shows some run-to-run variation (explicitly for Switch+PRT), but Table 2 reports only point estimates. Given that the difference between HarmonE (20.62 mJ) and Switch+PRT (25.32 mJ) is about 18.6%, the reader cannot assess whether this gap is meaningful relative to measurement noise. Please report standard deviations or confidence intervals for energy, R2, and inference time, and, if possible, a statistical comparison across the five runs.","section":"§5, Table 2 and Figure 5"},{"comment":"The adaptation counts in Table 2 are not broken down by type (switching events S, retraining events R, version-repo accesses V), although Figure 4 shows these as separate categories. This makes it difficult to evaluate RQ3: for example, HarmonE's 12 adaptations are described as efficient, but the reader cannot tell how many were model switches versus retrains versus versioned-model reuses, nor how many of the 12 were triggered by energy violations versus drift. A breakdown would strengthen the claim that adaptation decisions are selective and context-aware.","section":"§5, RQ3 and Table 2"}],"minor_comments":[{"comment":"Table 2 leaves the '# Adaptations' column empty for LR, LR+PRT, SVM, SVM+PRT, LSTM, and LSTM+PRT. Either fill in the values (e.g., '0' or 'N/A') or add a footnote explaining why they are omitted, for example that periodic retraining runs are counted as four retraining events.","section":"§4.2 and Table 2"},{"comment":"The notation for accuracy is inconsistent: the text uses 'R2' and 'R2 score' while the tables and figures use 'R2'. Please standardize the notation (e.g., 'R²' or 'R2') throughout.","section":"§3.3 and §5"},{"comment":"The paper says KL divergence is computed between recently observed true values and a reference distribution, following [18], but it does not specify how the reference distribution is chosen or how the 5-timestep input window is incorporated into the drift metric. A brief operational definition would help reproducibility.","section":"§3.3 and §4.1"},{"comment":"Figure 6 includes a 'Reference Energy Level' red line, but the actual value of E_ref is not reported in the text or captions. Since this value is central to the energy control mechanism, it should be disclosed (even in a reproducibility appendix).","section":"§5, Figure 6"},{"comment":"The sentence 'To ensure consistent and reliable results, each baseline experiment is repeated independently five times, following established guidelines for empirical software engineering [29]' is a reasonable practice, but the paper should specify whether the five runs used different random seeds or the same seed, and whether stochastic elements (e.g., ε-greedy exploration) were rerandomized across runs.","section":"§4.2"},{"comment":"The paper discusses internal, construct, and external validity but does not address the main threat: the sensitivity of the results to the unreported controller parameters. A paragraph on parameter sensitivity (or a pointer to a sensitivity analysis) would directly address the concern that the reported trade-off is tuned.","section":"§6, Threats to Validity"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea—using a MAPE-K loop with a control-theoretic threshold to balance sustainability goals in MLOps—is relevant and may be publishable after substantial revision. My main concern is that the evaluation as written does not demonstrate the claimed trade-off as a robust property: the energy cap is enforced by the controller design, the reference level and other parameters are not disclosed, and the strongest comparator (Switch) is described as degraded. These issues are fixable within the manuscript's scope by reporting parameters, adding sensitivity analysis, strengthening the baselines, and reporting variance. I do not see a fundamental flaw that would require rejection, but the current form is not yet acceptable. I would also encourage the authors to clarify the retraining data semantics, because the present ambiguity undermines confidence in the comparison against periodic retraining."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading HarmonE: the architecture is a well-integrated design for energy-aware MLOps, but the headline 54.5% energy saving is not an emergent property of that design — it is enforced by an unreported reference level E_ref in Eq. (3), and the evaluation does not provide the parameter values or sensitivity analysis needed to know whether that number is robust or tuned.\n\nWhat is new and good: HarmonE combines MAPE-K self-adaptation, decision maps, model switching, versioned model reuse, and a control-theoretic energy threshold into a single MLOps architecture. That integration is genuinely absent from the cited EcoMLS and QoS-aware switching work. The paper is clearly written, honest about its single use case, and the authors ship code and use a real traffic dataset with a sensible model spectrum (LR, SVM, LSTM). The RQ structure is fine, and the overhead measurement (MAPE-K consumes ~1.4% of total energy) is a nice touch.\n\nThe soft spots are concentrated where the headline numbers live. The energy cap is written into the control law: when E_i exceeds E_ref, Eq. (3) tightens the dynamic threshold, forcing a switch to a lighter model. So it is unsurprising that HarmonE's cumulative energy stays below the reference line in Figure 6. The paper never reports E_ref, beta, gamma, delta, S_min, tau_drift, or the monitoring interval, and there is no sensitivity analysis. If E_ref was set low or chosen after inspecting test energy, the 54.5% reduction would be a tuned artifact, not a general property of the architecture. That is the load-bearing weakness.\n\nSecond, the retraining procedure is under-specified. \"Retraining ... is performed using 1200 samples, consistent with the original training window size\" — that could mean the original training data or a sliding window. If it is always the same 1200 old samples, periodic retraining cannot adapt to drift, which would confound the comparison with HarmonE's selective retraining. Third, the Switch baseline is described as defaulting to the heavier LSTM, making it a weak comparator; a smarter adaptive baseline would test the architecture more fairly. Fourth, Table 2 has no error bars or significance tests despite five runs; the small spread in Figure 5 is suggestive but not statistical evidence.\n\nNone of this kills the architecture. The idea is worth pursuing and the paper is a reasonable first validation. But the current evidence does not support the broad \"ensures long-term sustainability\" claim. Engage with it, but a referee should demand: all design-time parameters and a sensitivity analysis, a clear description of the retraining data window, error bars and tests, and a credible adaptive baseline. The GitHub repo should be pinned with a commit hash.\n\nThis is a paper for the self-adaptive software engineering and green AI communities. A serious referee could shape the weaknesses into a solid revision. I would accept it for peer review, but I would not cite it as evidence yet.","headline":"HarmonE is a sensible integrated architecture for energy-aware MLOps, but the headline 54.5% energy saving is partly enforced by an unreported reference threshold, and the evaluation lacks the parameter reporting and baseline rigor to back it.","tokens_in":13352,"tokens_out":3326,"would_cite":false,"duration_ms":29942,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A control loop lets MLOps pipelines cut energy use by 54.5% while keeping 95% of accuracy.","keywords":["self-adaptation","MLOps","sustainability","green AI","MAPE-K","model switching","drift detection","energy efficiency"],"falsifier":"Run HarmonE on the same PeMS traffic-flow task with the controller parameters ($\\bar{E}_{ref}$, $\\beta$, $\\gamma$, $\\delta$, $S_{min}$, $\\tau_{drift}$) set to neutral values chosen before inspecting the test set, and measure the energy-accuracy trade-off. If the energy saving relative to LSTM+PRT falls substantially below 54.5% or the accuracy ratio drops below 95%, the central claim fails. A simpler check is to read the actual values from the paper's companion materials; if $\\bar{E}_{ref}$ was set near the LSTM baseline's consumption, the comparison is biased.","tokens_in":12290,"feed_emoji":"🌱","tokens_out":6821,"duration_ms":55212,"temperature":0.7,"pith_summary":"The paper argues that MLOps pipelines can be made sustainable without sacrificing accuracy by wrapping them in a self-adaptive layer. HarmonE, the proposed architecture, uses a monitor-analyze-plan-execute-knowledge (MAPE-K) loop to track prediction accuracy, energy consumption, and data-distribution drift, and triggers adaptation only when thresholds are crossed. In a traffic-flow forecasting scenario with three models of differing cost, the paper reports that HarmonE consumes 54.5% less energy than the accurate LSTM+PRT baseline while retaining 95.0% of its $R^2$ accuracy, and cuts inference time by 53.4%. The intended contribution is architectural: a way to encode sustainability goals at design time and enforce them at runtime with low overhead.","feed_headline":"Self-adapting MLOps cuts energy 54.5% at 95% accuracy","feed_subtitle":"HarmonE switches models, retrains selectively, and reuses old versions only when drift or energy thresholds are breached.","key_machinery":"The load-bearing mechanism is the dynamic energy threshold of Eq. (3), $\\tau_{E_{i+1}}=\\tau_{E_i}+\\delta(\\bar{E}_{ref}-\\bar{E}_i)$, which converts the architect's sustainability goal into an evolving adaptation boundary. When observed normalized energy $\\bar{E}_i$ stays above the reference $\\bar{E}_{ref}$, the threshold drops and the next violation forces a model switch or retraining; when energy is below the reference, the boundary loosens and transient spikes are tolerated. This is paired with the composite performance score $S_i$ (Eq. 1) and its exponential moving average (Eq. 2), which rank candidate models during adaptation, and with a drift threshold $\\tau_{drift}$ that triggers versioned-model reuse or retraining. Together these pieces turn a static MLOps pipeline into a closed-loop controller.","core_discovery":"On the paper's own terms, HarmonE shows that a self-adaptive MLOps pipeline can meet explicit sustainability goals by treating energy as a controlled variable rather than a side effect. At each monitoring interval the system computes a composite score $S_i=\\beta A_i+(1-\\beta)(1-\\bar{E}_i)$, smooths it as an exponential moving average, and maintains a dynamic energy threshold $\\tau_{E_{i+1}}=\\tau_{E_i}+\\delta(\\bar{E}_{ref}-\\bar{E}_i)$ that tightens when energy exceeds a design-time reference. Exceeding this threshold, or a drift threshold, triggers one of three tactics: switching to a lighter or more accurate model from a repository, retraining, or reusing a versioned model trained on a similar distribution. In the reported traffic-flow experiment, this mechanism yields an average energy of 20.62 mJ and $R^2=0.8628$, versus 45.35 mJ and $R^2=0.9085$ for LSTM+PRT; the paper states this as 54.5% lower energy at 95.0% of the baseline's accuracy, with inference time reduced from 4.05 ms to 1.89 ms. It also reports that the adaptation logic itself consumes only 1.37% of total system energy.","pith_inferences":["If the dynamic-threshold mechanism generalizes, the same control-theoretic update could be applied to other sustainability dimensions, such as memory, monetary cost, or carbon emissions, by substituting each dimension's normalized metric for $\\bar{E}_i$ and keeping a reference level.","The reported 54.5% saving is relative to a reference energy level; a stronger evaluation would report the controller parameters and a sensitivity analysis of the saving as the reference moves, since the threshold update can be tuned to force a low-energy operating point.","The drift-and-reuse idea suggests a testable extension: in domains with recurring distributions (e.g., daily or weekly patterns in sensing data), versioned model reuse may dominate retraining, and the savings could be predicted from the periodicity of the drift signal.","Because the evaluation uses one sensor node and one time-series task, applying HarmonE to vision or NLP would require redefining accuracy, energy, and drift metrics; the architecture is domain-agnostic but each new domain needs its own decision map."],"forward_implications":["Fixed-interval retraining (LR+PRT, SVM+PRT, LSTM+PRT, Switch+PRT) retrains four times per run regardless of need; HarmonE's triggered adaptations achieve comparable accuracy with fewer, well-timed interventions.","The approach keeps energy below the reference level over long horizons; in the experiment HarmonE's cumulative energy curve stays under the design-time reference while single-model baselines either overshoot it or sacrifice accuracy.","Model switching alone (Switch) is not enough: without retraining, the system defaults to the heavy LSTM and reaches only $R^2=0.8445$; HarmonE's combination of switching, selective retraining, and versioned reuse is what sustains both accuracy and energy goals.","The adaptation overhead is small enough for practical use: about 17.9 ms per MAPE-K invocation and 1.37% of total energy, so the sustainability gain is not offset by the controller's own cost."],"supporting_citations":[{"why":"Supplies the MAPE-K loop structure that HarmonE instantiates for monitoring, analyzing, planning, and executing adaptations.","marker":"[12]"},{"why":"Defines adaptation intent as a sustainability goal with boundaries, the conceptual basis for HarmonE's design-time goals.","marker":"[9]"},{"why":"Provides the control-theoretic update rule that HarmonE adapts for the dynamic energy threshold in Eq. (3).","marker":"[8]"},{"why":"Supplies the KL-divergence drift detection and the drift-induction procedure used to create and detect test-time distribution shifts.","marker":"[18]"},{"why":"Defines the sustainability dimensions and the decision-map notation used to capture goals in the architecture.","marker":"[16]"},{"why":"The closest prior self-adaptive energy-aware switching approach; HarmonE positions itself against it as a full-lifecycle alternative.","marker":"[27]"},{"why":"Established empirical-software-engineering guidelines that justify the five repeated runs and cooldown periods in the evaluation.","marker":"[29]"}],"fun_headline_variants":["Self-adapting MLOps cuts energy 54.5%","Energy-aware MLOps: 54.5% less power, 95% accuracy","HarmonE adaptive MLOps cuts energy 54.5%","MLOps self-tuning trims 54.5% energy, keeps accuracy","Adaptive MLOps pipeline slashes energy 54.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the reference energy level and the threshold parameters were fixed at design time without tuning on the test data; the paper does not report their values, so the headline 54.5% saving could be an artifact of calibration rather than a general property of the architecture.","fun_headline_variants_meta":{"raw":{"variants":["Self-adapting MLOps cuts energy 54.5%","Energy-aware MLOps: 54.5% less power, 95% accuracy","HarmonE adaptive MLOps cuts energy 54.5%","MLOps self-tuning trims 54.5% energy, keeps accuracy","Adaptive MLOps pipeline slashes energy 54.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3995,"prompt_tokens":1058,"completion_tokens":2937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2836}},"tokens_in":674,"tokens_out":2937,"duration_ms":19036,"temperature":1.0,"reasoning_tokens":2836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:11:04.660005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HarmonE on the same PeMS traffic-flow task with the controller parameters ($\\bar{E}_{ref}$, $\\beta$, $\\gamma$, $\\delta$, $S_{min}$, $\\tau_{drift}$) set to neutral values chosen before inspecting the test set, and measure the energy-accuracy trade-off. If the energy saving relative to LSTM+PRT falls substantially below 54.5% or the accuracy ratio drops below 95%, the central claim fails. A simpler check is to read the actual values from the paper's companion materials; if $\\bar{E}_{ref}$ was set near the LSTM baseline's consumption, the comparison is biased.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MAPE-K loop structure that HarmonE instantiates for monitoring, analyzing, planning, and executing adaptations."},{"cited_title":"In: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (2022)","cited_arxiv_id":null,"evidence_quote":"Defines adaptation intent as a sustainability goal with boundaries, the conceptual basis for HarmonE's design-time goals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the control-theoretic update rule that HarmonE adapts for the dynamic energy threshold in Eq. (3)."},{"cited_title":"In: 2022 IEEE/ACM 1st International Workshop on Software Engineering for Responsible Artificial Intelligence (SE4RAI) (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the KL-divergence drift detection and the drift-induction procedure used to create and detect test-time distribution shifts."},{"cited_title":"In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS)","cited_arxiv_id":null,"evidence_quote":"Defines the sustainability dimensions and the decision-map notation used to capture goals in the architecture."},{"cited_title":"Springer Science & Business Media (2012)","cited_arxiv_id":null,"evidence_quote":"Established empirical-software-engineering guidelines that justify the five repeated runs and cooldown periods in the evaluation."}],"review_version":1}