{"id":"1a7654dc-03ef-4ef2-aafb-d4fbd0d846d8","arxiv_id":"2501.11107","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A chain of LLM agents can autonomously define, run, analyze, and fix Kubernetes chaos engineering experiments at low time and API cost on small and large test systems.","lead":"This paper presents ChaosEater, a system that chains LLM agents to run a full chaos engineering cycle on Kubernetes systems with almost no human input. In case studies on two systems, it completed cycles in 11 to 25 minutes for $0.21 to $0.84, fixing planted resiliency flaws in most runs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The system defines its own pass/fail thresholds anchored to normal behavior, so 'reasonable CE cycle' is unvalidated when thresholds are lax; the SockShop stress failure may not affect the chosen replica-count steady states.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the meaningfulness of LLM-generated steady states and failure scenarios. My reading of the system prompts and case-study outputs supports this concern and adds a concrete mechanism: the threshold-definition prompt (Agent #1-2) explicitly requires the current value to satisfy the threshold and asks for tolerance that makes the threshold easier to satisfy, so the system is calibrating the pass criterion to the normal state. In addition, the SockShop case shows a mismatch between the chosen failures and the chosen steady states: StressChaos on cart-db is paired with steady states about replica counts, which are unlikely to be affected by CPU/memory stress unless the stress is severe enough to crash the container. The paper's own Appendix B.4 admits the tool fails to find hidden issues in systems that already have some resilience, which is a direct acknowledgment that the 'reasonable CE cycle' claim is limited to obvious, pre-existing weaknesses. These are empirical validation gaps rather than internal contradictions; the system clearly runs and produces coherent reconfigurations in the two presented cases. The correct disposition remains conditional: the paper is a promising proof of concept, but the central claim about completing reasonable CE cycles should be tempered until the generated tests are validated against independent, human-defined resilience criteria.","tokens_in":52226,"tokens_out":5269,"duration_ms":56231,"concrete_test":"Re-run the SockShop experiment with only the StressChaos fault on carts-db (omitting the front-end PodChaos), and run an independent human-defined SLO check, for example front-end p95 latency below 500 ms or http_req_failed rate below 0.1%, alongside the generated VaC scripts. If the VaC scripts pass while the independent SLO fails, the LLM-generated steady states are too lax to constitute a meaningful resilience test.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"To support the claim that CHAOS EATER 'stably completes reasonable single CE cycles,' the LLM-generated steady states and failure injections must constitute a meaningful resilience test. The design undercuts this: Agent #1-2 (Section 2.2) is instructed that the current value must satisfy the threshold and to 'include reasonable tolerance that makes the threshold being more easily satisfied,' so thresholds are anchored to normal operation rather than to an externally defined SLO. Agent #1-5 is told to prioritize fault injections targeting the steady states already defined, coupling the failure injection to the easy-to-satisfy test. In the reported SockShop cycle (Figure 2), the steady states are replica-count thresholds (cart-db: at least 1 ready replica 100% of the time and 2 ready replicas 80%; front-end: at least 1 ready replica 100%), while the Black Friday scenario includes StressChaos on carts-db. CPU/memory stress can severely degrade service latency without changing ready-replica counts, so the VaC scripts would pass while the user-facing service is degrades. The two planted faults (restartPolicy: Never; single front-end replica) are obvious enough that almost any reasonable availability threshold would detect them. Appendix B.4 concedes that on systems with existing resilience, CHAOS EATER fails to find hidden issues. Thus the central claim is not established as a general statement: the five successful runs show pipeline execution on two easy cases, not that the completed cycles test real resilience.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ChaosEater is an LLM-based agentic system that aims to automate the full chaos engineering (CE) cycle for Kubernetes systems. Given a folder of K8s manifests and a Skaffold configuration, it performs preprocessing, hypothesis definition (steady states plus a failure scenario), experiment planning and execution via Chaos Mesh, analysis of failed validations, and reconfiguration of K8s manifests, repeating the experiment/analysis/improvement loop until the hypothesis is satisfied. Steady states are encoded as unit-test scripts (\"Validation as Code\"). The evaluation consists of two case studies, NGINX and Sock Shop, with five runs each; the paper reports time and API costs, plus qualitative ratings of one selected run per system by two human engineers and three LLM judges. The central claim is that ChaosEater \"stably completes reasonable single CE cycles\" at low cost.","tokens_in":52472,"tokens_out":4536,"duration_ms":46665,"significance":"If the central claim holds, this is a meaningful step toward end-to-end automation of chaos engineering: it removes manual hypothesis definition, experiment planning, failure analysis, and K8s-manifest reconfiguration, and it does so with an open-source implementation, published prompt templates, and a reproducible temperature-0/seed-42 setup. The Validation-as-Code idea, in particular, is a useful contribution because it makes steady-state checking explicit, transparent, and machine-executable. The significance is currently bounded by evaluation validity: the evidence consists of five runs on two small systems with deliberately planted, easily detectable faults, the steady-state thresholds are generated by the same LLM family that later judges the results, and the paper itself concedes in Appendix B.4 that the system fails to find hidden issues in already-resilient systems. The strength of the paper is its architecture and open artifact, not yet the demonstrated generality of the behavioral claim.","major_comments":[{"comment":"The steady states and failure injections that define a 'reasonable CE cycle' are generated by the same LLM pipeline that is later judged, and the prompts explicitly instruct the threshold agent to make thresholds 'more easily satisfied' and to ensure the current value satisfies the threshold. This anchors pass/fail criteria to normal operation rather than to an externally defined SLO. The Sock Shop example in Figure 2 illustrates the risk: the chosen steady states are ready-replica counts, while the Black Friday scenario includes StressChaos on carts-db; CPU/memory stress can degrade latency and user-facing availability without changing ready-replica counts, so the VaC scripts can pass while the service is substantially degraded. To support the abstract's claim, the paper needs an independent validation that the generated thresholds and failure scenarios constitute a meaningful resilience test, for example a human-audited SLO benchmark or a comparison against pre-specified thresholds.","section":"Section 2.2 (Agent #1-2 and Agent #1-5 prompts)"},{"comment":"The stability claim rests on five runs per system, and one of the five Sock Shop runs did not perform the expected front-end reconfiguration. The paper states that a valid cycle was completed without reconfiguration, but it does not report what steady states, failure scenario, and VaC results that run produced, nor does it define what counts as a 'successful' cycle when no issue is found. With n=5 and one divergent run, the statement 'stably completes reasonable single CE cycles' is not established. The authors should report per-run variation, define success criteria independent of whether reconfiguration happens, and ideally increase the number of runs or use multiple seeds.","section":"Section 3, 'Costs and stability'"},{"comment":"The paper's own limitation statement says that for systems with existing resilience, ChaosEater 'fails to find new hidden issues through a CE cycle.' This directly bounds the central claim: the two case-study systems were configured with simple, planted faults (restartPolicy: Never and a single front-end replica), so the successful runs demonstrate execution on easy cases, not general ability to discover meaningful vulnerabilities. The abstract and conclusion should either scope the claim to systems with known simple resiliency issues or provide evidence on systems where the issues are not pre-planted and obvious.","section":"Appendix B.4 and Conclusion"},{"comment":"The qualitative validation uses a self-authored rubric, only one selected run per system, two external human engineers, and three LLM judges of which one is GPT-4o, the same model family used to run ChaosEater. All evaluators rated every phase above the positive threshold, but no inter-rater reliability or disagreement analysis is reported, and the risk of circularity from having the same model family both generate and judge the cycle is not addressed. The authors should report per-evaluator scores, a sensitivity analysis excluding the GPT-4o judge, and a clearer justification for why the selected runs are representative of the five-run set.","section":"Section 3 and Appendix C.3, qualitative validation"}],"minor_comments":[{"comment":"The text says the workflow goes through 'five divided phases' but then lists six phase names: pre-processing, hypothesis, experiment, analysis, improvement, and post-processing. Please correct the count or the list.","section":"Section 2.2"},{"comment":"The column header 'NGNIX' is a typo and should read 'NGINX'.","section":"Table 1"},{"comment":"The model identifier 'gpt-4o-2024-0908-06' appears inconsistent with 'gpt-4o-2024-08-06' used in Section 3; please unify the naming.","section":"Appendix C.3"},{"comment":"The criterion for a score of 1 says 'The experiment plan does serve to validate the hypothesis at all'; this should read 'does not serve'.","section":"Table 2, Experiment score 1"}],"recommendation":"major_revision","confidential_remarks":"This is a promising systems paper with a strong open artifact and a clear architecture, but the evaluation does not yet support the abstract's general claim about 'reasonable single CE cycles.' The threshold-meaningfulness problem and the small sample size are fixable with additional experiments, a benchmark against externally defined SLOs, and more precise scoping of the claims; I would not reject the paper on the current evidence, but the authors should address these points before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, know this: the paper is a proof of concept, not a demonstrated product. It builds a 20-agent workflow that takes a K8s project, defines steady states and failure injections, runs them via Chaos Mesh, analyzes the logs, and edits the manifests. That is genuinely new—prior work either automated experiment execution or used LLMs for single phases. The VaC idea (turning LLM-judged hypotheses into unit-test scripts) is the cleanest part. And they ship everything: prompts, code, GUI, and a demo.\n\nWhat is good: the workflow design is thoughtful, the phase separation is sensible, and the cost figures (11-25 min, $0.21-0.84) are plausible for the workloads shown. The qualitative evaluation, while not a benchmark, is documented in enough detail to see what happened. The authors also admit in Appendix B that the system fails to find hidden issues on already-resilient systems, which is the right kind of honesty.\n\nThe soft spots are real. The main one: the system defines its own pass bar. Agent #1-2 is told to add \"reasonable tolerance that makes the threshold being more easily satisfied\" and the current value must satisfy it. So thresholds are anchored to normal operation, not an external SLO. That makes a \"successful\" CE cycle somewhat circular for resilience validation—the cycle passes because the LLM set easy goals, not because the system is actually resilient. The Sock Shop example shows this: steady states are replica-count checks, while the Black Friday scenario includes CPU stress on carts-db that may degrade latency without touching replica counts. The stress-test note on this seems right.\n\nThe empirical base is also thin: two systems, five runs each, one of ten runs did not reconfigure Sock Shop as expected. No human baseline on time or cost. One of the three LLM judges is GPT-4o, the same family that generated the work, which weakens that part of the validation. These issues are addressable: add externally defined SLOs, use a held-out judge, run more systems, ideally with a benchmark.\n\nIs the central claim load-bearing? I do not think the system design is broken; the pipeline executes real injections and real reconfigurations. But \"stably completes reasonable single CE cycles\" is not established beyond easy planted-fault cases. I would want the claim scaled back to \"completes CE cycles for simple, detectable issues\" until the evaluation is broader.\n\nWho this is for: researchers and practitioners working on LLM agents for infrastructure operations, or CE tooling. It deserves a serious referee—it is a legitimate new system with reusable artifacts—but I would push for major revision on evaluation rigor.","headline":"A well-built, fully open proof of concept for LLM-driven chaos engineering, but the central claim rests on a thin, partly self-referential evaluation.","tokens_in":53033,"tokens_out":2748,"would_cite":true,"duration_ms":27923,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pipeline of LLM agents can complete a full chaos-engineering cycle on Kubernetes systems autonomously, finding and fixing fragility in minutes for under a dollar.","keywords":["chaos engineering","Kubernetes","large language models","LLM agents","infrastructure as code","fault injection","resiliency testing","Validation as Code"],"falsifier":"Run CHAOS EATER on a Kubernetes system with a planted, non-obvious vulnerability, such as a single replica whose readiness probe fails only under partial network partition, with no hint about the defect, and check whether the generated VaC thresholds and failure injections detect it. If the cycle completes without reconfiguration because all LLM-chosen thresholds are satisfied by construction, the claim that the pipeline performs meaningful resilience testing is not supported.","tokens_in":51992,"feed_emoji":"🧪","tokens_out":6360,"duration_ms":60304,"temperature":0.7,"pith_summary":"The paper proposes CHAOS EATER, a system that uses a fixed workflow of LLM agents to carry out an entire chaos-engineering cycle on Kubernetes systems managed as code: it defines steady states and failure scenarios, plans and executes fault injection, analyzes whether the system stayed healthy, and reconfigures the Kubernetes manifests when it did not. The central claim is that this full loop completes stably and cheaply, at roughly $0.21 to $0.84 in API cost and 11 to 25 minutes per cycle in the two case studies, and that human engineers and three LLM judges rate the resulting cycles as reasonable. If true, the work matters because it turns resilience testing from a manual, expert-only activity into an automated software-engineering task.","feed_headline":"LLM agents close the loop on chaos engineering at $0.21 a cycle","feed_subtitle":"ChaosEater writes its own hypothesis, injects faults, analyzes results, and patches Kubernetes manifests with no human in the loop.","key_machinery":"The carrying object is a predefined agentic workflow of twenty LLM agents chained through the phases pre-processing, hypothesis, experiment, analysis, improvement, and post-processing, with rule-based verification loops at script-generation and manifest-application points. The load-bearing idea inside it is Validation as Code: each steady state is a pair of a measurable state and a threshold, and the threshold is turned into a unit-test script (Python against the Kubernetes API, or k6 JavaScript for communication metrics) whose assertions decide whether the hypothesis held. These scripts and the chosen failure injections are compiled by a hierarchical node-grouping algorithm into a Chaos Mesh workflow manifest, which schedules pre-validation, failure-injection, and post-validation stages automatically. That conversion is what lets the LLM define intent while the execution and judgment remain deterministic.","core_discovery":"On its own terms, the paper's discovery is that the entire systematic chaos-engineering cycle can be delegated to LLMs by fixing the workflow and subdividing operations among twenty role-specialized agents, and that this delegation is reliable enough to complete single cycles on both a two-manifest NGINX system and the 29-manifest Sock Shop system without runtime errors, reconfiguring the system appropriately in five of five and four of five runs respectively. The key mechanism for trustworthy validation is 'Validation as Code' (VaC): steady states are encoded as executable unit tests with threshold assertions, so the experiment's pass/fail judgment is made by code rather than by an LLM reading logs. The paper validates the cycles qualitatively with two external human engineers and three LLMs, all of whom rated every phase at or above the positive threshold.","pith_inferences":["A natural next step the paper leaves implicit is a hidden-vulnerability benchmark: plant a defect that is not detectable by manifest inspection alone and check whether the self-chosen thresholds and failures actually expose it; without such a benchmark, a completed cycle could be vacuous.","The paper itself notes that on already-resilient systems the current system does not find hidden issues; this suggests the pipeline's present value is catching simple configuration fragility, and that long-term multiple cycles need history management and continuous learning to reach deeper fault discovery.","The same workflow design could be ported to other code-defined infrastructure, such as Terraform or Helm, or to non-Kubernetes targets, by swapping the manifest parser and the fault-injection backend.","Automatic prompt tuning and workflow optimization, which the paper identifies as future work, could make the system model-agnostic and reduce the manual prompt-management cost of the twenty-agent design."],"forward_implications":["A single operator can hand a folder of Kubernetes manifests to CHAOS EATER and receive, in under half an hour, a summary plus a hardened manifest set, replacing the manual hypothesis-definition, experiment-planning, and reconfiguration work.","Cost grows slowly with system size: roughly quadrupling API cost and doubling time when moving from a two-manifest NGINX system to the 29-manifest Sock Shop, suggesting the approach is feasible beyond toy deployments.","Because steady-state validation is encoded as unit tests, the same judgment is reproducible run-to-run; the LLM proposes, and the code disposes.","The reconfiguration actions the system takes are standard resilience fixes, such as replacing a bare Pod with a Deployment and raising replica counts, so the pipeline can implement a meaningful subset of resiliency improvements without human intervention.","Qualitative agreement between human engineers and LLM judges on all phases suggests the cycle output is understandable and auditable by people."],"supporting_citations":[{"why":"Defines the systematic chaos-engineering cycle and steady-state principles that CHAOS EATER encodes as its workflow.","marker":"Basiri et al., 2016"},{"why":"Provides the code-defined failure-injection engine and workflow manifests through which CHAOS EATER schedules faults and VaC scripts.","marker":"Chaos Mesh, 2021"},{"why":"Supplies k6, used to write load-test VaC scripts that measure communication steady states such as availability and error rate.","marker":"Grafana Labs, 2021"},{"why":"Defines the target platform and manifest format that the agents summarize, analyze, and reconfigure.","marker":"Kubernetes, 2014"},{"why":"Supplies GPT-4o, the underlying LLM used in all case-study runs.","marker":"OpenAI, 2024"},{"why":"Provides the Sock Shop microservice application used as the large-scale case study.","marker":"Weaveworks, 2023"},{"why":"Provides Skaffold, which defines and automates deployment of the input Kubernetes systems in the pre-processing phase.","marker":"Google, 2019"}],"fun_headline_variants":["LLM agents close the chaos engineering loop automatically","ChaosEater: LLMs run the entire chaos engineering cycle","Chaos engineering end-to-end with LLMs, validated by code","Validation as Code makes LLM chaos engineers reliable","ChaosEater: LLM agents automate chaos testing and fixing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the LLM-chosen steady-state thresholds and failure scenario being a real test of the system; the threshold agent is told to set thresholds that the current, healthy state already satisfies, so a cycle can pass or yield only trivial fixes if the LLM picks an easy target.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents close the chaos engineering loop automatically","ChaosEater: LLMs run the entire chaos engineering cycle","Chaos engineering end-to-end with LLMs, validated by code","Validation as Code makes LLM chaos engineers reliable","ChaosEater: LLM agents automate chaos testing and fixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00101,"raw_usage":{"total_tokens":4263,"prompt_tokens":933,"completion_tokens":3330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":3247}},"tokens_in":549,"tokens_out":3330,"duration_ms":23747,"temperature":1.0,"reasoning_tokens":3247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:37:22.286474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CHAOS EATER on a Kubernetes system with a planted, non-obvious vulnerability, such as a single replica whose readiness probe fails only under partial network partition, with no hint about the defect, and check whether the generated VaC thresholds and failure injections detect it. If the cycle completes without reconfiguration because all LLM-chosen thresholds are satisfied by construction, the claim that the pipeline performs meaningful resilience testing is not supported.","supporting_citations":[],"review_version":1}