{"id":"ea67f777-a49e-43c8-b7de-8ec7155f8a5b","arxiv_id":"2507.22702","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Ecoscape provides a configurable chaos-injection benchmark with a weighted SLO violation score for comparing Kubernetes remediation strategies in edge ML inference.","lead":"Ecoscape is a new benchmark that injects simulated faults into Kubernetes edge deployments and scores how well automated remediation strategies keep services within their latency, accuracy, and energy targets. It gives software teams a way to compare fault-recovery strategies in edge machine learning without building a physical edge testbed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1 assumes larger-is-worse SLOs, but the case study's accuracy SLO is larger-is-better; as written, accuracy violations below 75% contribute zero to the score, so the reported totals cannot support comparative claims without an undocumented inversion.","rationale":"I read Ecoscape in good faith as a benchmark proposal whose central claim is that it facilitates comparative analysis of Kubernetes remediation strategies through a configurable fault-injection environment and a quantitative SLO violation score. The paper has real independent support: a public GitHub repository, a Zenodo replication package for the SLIs, a documented declarative configuration, and integration with established tools (Chaos Mesh, Prometheus, Kepler). Those count as genuine evidence that the artifact exists and is usable. However, the reader's weakest-assumption choice, physical-testbed fidelity, is not the only or most immediate risk. The scoring formula itself appears inconsistent with the flagship case study. Eq. 1 applies positive weight only when the SLI exceeds its threshold, yet the accuracy SLO is stated as a lower bound, so the raw accuracy metric violates the formula's direction assumption. If the implementation does not invert accuracy into an error rate, the reported scores cannot have been computed as described; if it does invert accuracy, that transformation is absent from the paper and from the documented configuration model. Either way, the benchmark's quantifiable comparison mechanism is not yet correctly specified for one of its three demonstrated SLOs. This is an internal correctness concern, not a disagreement with external consensus, and it is more load-bearing than the simulation-fidelity concern because it undermines the validity of the score even in the simulated environment. The reader's concern remains valid and should be part of a conditional acceptance, but the scoring directionality should be settled first. I therefore recommend keeping the reader's CONDITIONAL verdict, with the added condition that the authors either correct the SLO definitions and configuration schema to support violation direction, or document and implement the inversion explicitly. The proposed concrete test, checking the code path for the accuracy SLI and recomputing one scenario's score, would settle whether this concern lands.","tokens_in":9228,"tokens_out":5635,"duration_ms":69532,"concrete_test":"Inspect the Ecoscape replication package and public repository to determine what SLI value is actually fed into Eq. 1 for the accuracy SLO, and whether the configuration schema supports a violation direction or comparator. Then rerun the CPU-stress scenario from Section V with the published configuration, ensuring accuracy falls below 75% during the chaos phase, and recompute the total score two ways: once with raw accuracy in Eq. 1 and once with an inverted error-rate formulation (v = 1 - accuracy, tau = 0.25). If raw accuracy is used, the accuracy term is zero for every violating timestep and positive for compliant timesteps, and the reported score 0.011 will not match.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central artifact is the weighted SLO violation score defined by Eqs. 1-2, and that score is what Ecoscape uses to compare remediation strategies. Eq. 1 assigns a positive violation term only when v(t) > tau, and the paper states in Section IV that \"SLOs must be formulated so that SLI values larger than the threshold indicate an SLO violation.\" However, Section V defines the object recognition accuracy SLO as \"accuracy ... must exceed 75%,\" i.e., the violation condition is v(t) < tau. If the raw accuracy ratio is fed into Eq. 1, the score penalizes high accuracy and is exactly zero for every timestep in which accuracy violates the SLO. The reported total scores of 0.011 and 0.042, and the accuracy traces in Figures 2-3, are therefore not reproducible from the formula as stated unless the implementation silently uses a transformed SLI such as error rate with an inverted threshold. That transformation is not described anywhere, and the benchmark's configuration format is not documented as supporting a violation direction or comparator. Because the score is the quantitative basis for the paper's comparative claim, this is a load-bearing correctness issue that precedes the reader's concern about physical-testbed fidelity: if the score cannot represent one of the three flagship SLOs, the comparison is invalid even inside the simulated environment. The issue is fixable (e.g., define the accuracy SLO as error rate below 25% or add an explicit direction field), but as written the central metric is internally inconsistent with its own case study.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Ecoscape, a Kubernetes-based benchmark for evaluating remediation strategies in edge machine learning environments. The benchmark injects faults via Chaos Mesh, monitors SLOs via Prometheus and Kepler, and computes a weighted SLO violation score defined by Eqs. (1) and (2). The authors demonstrate the benchmark on an object-recognition case study with two fault scenarios (CPU stress and increased network latency), reporting SLO violation scores of 0.011 and 0.042. The stated goal is to provide a standardized, configurable method for comparing remediation strategies without a physical edge testbed.","tokens_in":9512,"tokens_out":4394,"duration_ms":52947,"significance":"If the central claim holds, Ecoscape would be a useful open-source tool for fair comparison of Kubernetes-based remediation strategies, a gap the paper argues is currently unfilled. Strengths include the public GitHub repository and Zenodo replication package, the explicit use of established components (Chaos Mesh, Prometheus, Kepler), and the simple, parameter-free scoring formula that is transparent and configurable. However, the paper's experimental evidence does not yet establish the benchmark's comparative power: it reports single runs of two scenarios, includes no no-remediation baseline and no alternative strategies, and uses a scoring formula that is inconsistent with one of its own SLO definitions. The physical-testbed fidelity limitation is acknowledged but not addressed with validation. These issues are load-bearing for the paper's main claim and require revision.","major_comments":[{"comment":"Eq. (1) assigns a positive violation score only when v(t) > tau, and the surrounding text states that SLOs must be formulated so that SLI values larger than the threshold indicate an SLO violation. The case study, however, defines the object-recognition accuracy SLO as 'must exceed 75%', i.e., the violation condition is v(t) < tau. If the raw accuracy ratio is fed into Eq. (1), accuracy violations contribute exactly zero and high accuracy contributes a positive penalty, so the reported total scores 0.011 and 0.042 are not reproducible from the stated formula. The implementation must either transform accuracy into an error rate with an inverted threshold or extend Eq. (1) to support a configurable violation direction; this must be documented and fixed.","section":"Section IV, Eq. (1); Section V, SLO definitions"},{"comment":"The plotted quantity is not derived from Eq. (1). The y-axis is labelled 'Ratio of SLO violation' and includes values below zero (down to -0.6), but Eq. (1) is non-negative by construction. No definition of the plotted ratio, the 'slo baseline' line, or the three vertical red lines is provided beyond the qualitative descriptions in the text. Without a definition of the plotted quantity, the figures cannot be used to verify the reported scores or the claimed internal consistency in Section VI.","section":"Section V, Figures 2 and 3"},{"comment":"The evaluation does not support the central comparative claim. Each scenario is presented as a single run with no repetitions, no confidence intervals, and no baseline run without remediation, and the two remediation actions are not compared against each other or against alternatives. Section VI also concedes that no autonomous remediator was used. Consequently, the claim that Ecoscape 'facilitates the comparative analysis of diverse actions' (Section I) is not demonstrated by the reported experiments; the paper needs repeated trials, a no-remediation control, and at least two alternative remediation strategies scored under identical conditions.","section":"Section V and Section VI"},{"comment":"The portability of scores from a Kubernetes cluster with Chaos Mesh injections to physical edge hardware is a load-bearing premise of the benchmark, since the stated goal is benchmarking 'without needing a physical edge testbed' (Abstract). The discussion only lists arguments for and against the abstraction. The paper should provide some validation evidence, such as a comparison of resource-limit and stress-ng behavior against a physical edge device, or a clearly stated threat-to-validity analysis; otherwise the benchmark score's predictive value for real edge deployments remains unsubstantiated.","section":"Section VI, feasibility of simulated testbed"}],"minor_comments":[{"comment":"Specify whether the SLO weights are required to sum to 1; otherwise Vtotal is not bounded by 0 and 1 as claimed in the text.","section":"Section IV, Eq. (2)"},{"comment":"The accuracy SLO is defined as a percentage while latency and energy are defined as thresholds; define the SLI scale consistently so that Eq. (1) can be applied mechanically.","section":"Section V, SLO list"},{"comment":"The three vertical red lines are described in the text but should be explicitly labeled in the captions (e.g., warm-up-to-evaluation, remediation start, reconfiguration completion).","section":"Section V, Figures 2 and 3"},{"comment":"The phrase 'operating under under multiple SLOs' contains a duplicated word; remove the second 'under'.","section":"Section VII"},{"comment":"The description of Frisbee states that it 'does not provide quantitative performance metrics'; this claim should be substantiated with a citation to the Frisbee paper or softened, as it is used to motivate Ecoscape's scoring contribution.","section":"Section II, related work"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a tool demonstration; its central scoring formula is simple but vulnerable to a straightforward definitional inconsistency with the accuracy SLO. The evaluation is closer to a feasibility study than to a benchmark validation. The problems are correctable within the manuscript's scope, but the authors need to address the scoring direction issue, provide a reproducible definition of the plotted metrics, and strengthen the experimental design before the comparative claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a serious, early-stage benchmark proposal with a real open-source artifact, and the authors are honest about its maturity. Second, the central scoring formula has an internal inconsistency with the case study's accuracy SLO; this is fixable but needs to be resolved before the comparative claims can be trusted.\n\nWhat is genuinely new: Ecoscape combines Chaos Mesh, Kubernetes, Prometheus/Kepler, and a configurable weighted SLO violation score into a single framework aimed at evaluating remediation strategies in edge ML without a physical testbed. The declarative JSON configuration, the four-phase run structure, and the openly published replication package are concrete contributions. The case study, with CPU stress and network latency faults, shows the intended workflow and produces plots that are internally consistent with the narrative. That is real value for a subfield that lacks a standard evaluation harness.\n\nThe soft spots are substantial but not fatal. Most importantly, Eq. (1) defines a violation only when the SLI is larger than the threshold, and the paper explicitly states SLOs must be formulated that way. Yet the accuracy SLO in Section V is \"accuracy must exceed 75%,\" meaning a violation occurs when accuracy is below 75%. Feeding raw accuracy into Eq. (1) would yield zero violation contribution exactly when accuracy violates the SLO. The reported total scores (0.011 and 0.042) are therefore not reproducible from the formula as written unless the implementation silently uses an inverted SLI such as error rate. That transformation is not documented. This is a load-bearing issue because the score is the entire quantitative basis for the benchmark's comparison claim. The fix is straightforward—state the accuracy SLO as error rate below 25% or add a comparator/direction field in the configuration—but it cannot be left unaddressed.\n\nThe other weaknesses are ones the authors themselves flag: single runs with no repetitions or confidence intervals, no baseline without remediation, no alternative remediation strategy for comparison, and no autonomous remediator. The absence of a baseline is particularly limiting for a benchmark whose purpose is comparative evaluation. The concern about Kubernetes resource limits faithfully emulating edge device behavior is real, but the authors acknowledge it explicitly and it is a secondary matter.\n\nOverall, this is a coherent, honest proposal from people who know the problem space. The math is simple and correctly stated, the artifact is open, and the limitations are mostly acknowledged. It deserves a serious referee, not a desk rejection. I would send it to peer review with a clear request: fix the metric direction issue, add at least a no-remediation baseline and repeated runs, and preferably compare one alternative strategy. For a reading group on empirical systems evaluation, it is a good discussion piece; but I would not cite it in my own work until the score is made unambiguous.","headline":"A useful open-source benchmark proposal for Kubernetes remediation strategies in edge ML, but the scoring formula as written contracts the accuracy SLO in the case study, and the evaluation lacks baselines, repetitions, and a real remediator.","tokens_in":10076,"tokens_out":1545,"would_cite":false,"duration_ms":20215,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Ecoscape, a benchmark that injects faults into Kubernetes and scores SLO violations so edge-machine-learning remediation strategies can be compared without a physical testbed.","keywords":["fault tolerance","benchmark","Kubernetes","edge computing","machine learning inference","remediation strategies","chaos engineering","SLO violation score"],"falsifier":"Run the same set of remediation strategies through Ecoscape and on a heterogeneous physical edge testbed using the same fault profiles; if the ranking of total violation scores differs, the simulated abstraction does not faithfully transfer. A cheaper check is to measure whether CPU stress injected into a Kubernetes node produces the same latency, accuracy, and energy signature as a resource-constrained edge device.","tokens_in":9012,"feed_emoji":"🧪","tokens_out":4563,"duration_ms":51622,"temperature":0.7,"pith_summary":"The paper tries to establish that remediation strategies for keeping edge machine-learning services within service level objectives can be compared fairly by running them in a simulated Kubernetes environment with injected faults and scoring SLO violations. The central claim is a new benchmark, Ecoscape, that turns each SLO into a normalized violation score and combines weighted scores into one number. A sympathetic reader would care because today there is no standard way to compare remediation actions such as rescheduling workloads or reducing model depth. If Ecoscape works, teams can evaluate the fault tolerance of edge-ML systems without a physical testbed and can tune SLO priorities to their domain.","feed_headline":"Benchmark scores edge-ML remediators by weighted SLO violations","feed_subtitle":"Ecoscape injects CPU and network faults on Kubernetes, then reduces SLO compliance to one comparable score.","key_machinery":"The load-bearing object is the weighted SLO violation score: for each SLO, the metric $\\hat{V}_{\\text{SLO}} = \\frac{1}{|T|} \\sum_{t=1}^{T} \\left(1 - \\frac{\\tau}{v(t)}\\right)$ when $v(t) > \\tau$, otherwise zero, aggregated as $V_{\\text{total}} = \\sum_i w_i \\cdot \\hat{V}_{\\text{SLO}_i}$. This score, inspired by autoscaling performance evaluation, turns SLO compliance into one number a remediator is graded against. Around it, the benchmark machinery is the pipeline that makes the score meaningful: JSON scenario definitions that become Kubernetes manifests, location labels that guide fault injection, Kafka-based load generation with event-time latency measurement, and Kepler for energy monitoring.","core_discovery":"Ecoscape's central claim is that a configurable, reproducible benchmark procedure can quantify how well a Kubernetes remediator maintains predefined service level objectives under injected faults. Declarative JSON configurations are translated into Kubernetes deployments, chaos-injection tools simulate network latency and CPU stress, and Prometheus and Kepler monitor latency, accuracy, and energy consumption. The resulting SLO violation score, computed as a normalized per-SLO violation aggregated by user-chosen weights, lets different remediation actions be compared on a single scale. In the object-recognition case study, two fault scenarios (CPU stress on edge nodes and increased network latency) produced total violation scores of 0.011 and 0.042 as the remediator restored SLO compliance through rescheduling and model-depth reduction.","pith_inferences":["The paper leaves implicit that the same scoring scheme could be used online, not just offline: a remediator could pick the action with the lowest predicted violation score at runtime.","Because resource provisioning is modeled as an SLO, a natural extension is adding deployment cost or carbon footprint as weighted objectives, letting the weights encode business priorities.","The current benchmark covers performance-style faults; extending the chaos definitions to message loss and network partitioning would test whether the score remains discriminative under byzantine conditions.","A direct validation step the paper does not run is to compare Ecoscape's ranking of strategies with the ranking observed on a heterogeneous physical edge testbed under the same fault profiles."],"forward_implications":["A Kubernetes remediator can be graded by one number instead of by ad hoc latency, accuracy, and energy logs, allowing direct comparison of strategies.","Because scenarios are declarative, benchmark runs are reproducible and configurable to domain-specific SLOs and weights.","Teams can evaluate remediation actions without physical edge hardware, lowering the entry barrier for fault-tolerance testing.","The case study suggests rescheduling and model-depth reduction are measurable responses to CPU stress and network latency faults, respectively.","Publishing the measured SLIs as a replication package means other remediators can be rerun against the same fault scenarios."],"supporting_citations":[{"why":"Shows that reconfiguration actions such as scheduling adjustments and parameter changes can substantially impact SLO compliance, motivating the benchmark.","marker":"[11]"},{"why":"Frisbee is a prior fault-tolerance benchmark that injects faults but does not address edge computing or provide quantitative remediation metrics.","marker":"[12]"},{"why":"µChaos applies chaos engineering at the edge but requires physical hardware, contrasting with Ecoscape's simulated cluster approach.","marker":"[13]"},{"why":"EdgeCloudSim emulates the edge-cloud continuum but lacks portability to real-world environments compared with a Kubernetes-based implementation.","marker":"[14]"},{"why":"Defines the benchmark requirements Ecoscape claims to satisfy, including reproducibility, execution-environment description, and measurable quality attributes.","marker":"[24]"},{"why":"Supplies the weighted-sum scoring idea that Ecoscape adapts into its SLO violation score.","marker":"[25]"},{"why":"Kepler provides the energy-consumption monitoring data used as one of the three SLOs in the case study.","marker":"[26]"},{"why":"Defines event-time latency, the metric behind the latency SLO that drives the remediation evaluation.","marker":"[28]"}],"fun_headline_variants":["Ecoscape: score-based benchmark for edge-ML remediation","Fault tolerance benchmark for Kubernetes edge-ML remediators","Quantify remediation strategies in edge ML with Ecoscape","Chaos engineering benchmark scores edge-ML recovery tactics","Compare edge-ML remediation strategies via SLO score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that limiting CPU and memory in a Kubernetes pod and injecting stress with chaos tooling reproduces how a real edge device would behave, so scores measured in the cluster predict what a remediator would achieve on physical hardware.","fun_headline_variants_meta":{"raw":{"variants":["Ecoscape: score-based benchmark for edge-ML remediation","Fault tolerance benchmark for Kubernetes edge-ML remediators","Quantify remediation strategies in edge ML with Ecoscape","Chaos engineering benchmark scores edge-ML recovery tactics","Compare edge-ML remediation strategies via SLO score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1423,"prompt_tokens":894,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":449}},"tokens_in":510,"tokens_out":529,"duration_ms":6365,"temperature":1.0,"reasoning_tokens":449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:21:59.796571+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same set of remediation strategies through Ecoscape and on a heterogeneous physical edge testbed using the same fault profiles; if the ranking of total violation scores differs, the simulated abstraction does not faithfully transfer. A cheaper check is to measure whether CPU stress injected into a Kubernetes node produces the same latency, accuracy, and energy signature as a resource-constrained edge device.","supporting_citations":[{"cited_title":"Process-Based Efficient Power Level Exporter,","cited_arxiv_id":null,"evidence_quote":"Kepler provides the energy-consumption monitoring data used as one of the three SLOs in the case study."},{"cited_title":"Investigating quality attributes of machine learning inference on the edge-cloud continuum,","cited_arxiv_id":null,"evidence_quote":"Shows that reconfiguration actions such as scheduling adjustments and parameter changes can substantially impact SLO compliance, motivating the benchmark."},{"cited_title":"Frisbee: A suite for benchmarking systems recovery,","cited_arxiv_id":null,"evidence_quote":"Frisbee is a prior fault-tolerance benchmark that injects faults but does not address edge computing or provide quantitative remediation metrics."},{"cited_title":"Kalka and T","cited_arxiv_id":null,"evidence_quote":"µChaos applies chaos engineering at the edge but requires physical hardware, contrasting with Ecoscape's simulated cluster approach."},{"cited_title":"Edgecloudsim: An environment for performance evaluation of edge computing systems,","cited_arxiv_id":null,"evidence_quote":"EdgeCloudSim emulates the edge-cloud continuum but lacks portability to real-world environments compared with a Kubernetes-based implementation."},{"cited_title":"Benchmarking as empirical standard in software engineering research,","cited_arxiv_id":null,"evidence_quote":"Defines the benchmark requirements Ecoscape claims to satisfy, including reproducibility, execution-environment description, and measurable quality attributes."},{"cited_title":"Autoscaler evaluation and configuration: A practitioner’s guideline,","cited_arxiv_id":null,"evidence_quote":"Supplies the weighted-sum scoring idea that Ecoscape adapts into its SLO violation score."}],"review_version":1}