{"id":"0ec0affe-d29f-486d-8e75-22d3ae9e7df0","arxiv_id":"2506.14281","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper proposes a compliance-aware, four-phase chaos engineering framework for Softtech using LitmusChaos and a standard monitoring stack, but provides no implementation or validation.","lead":"This paper is a design plan for introducing chaos engineering, the practice of deliberately breaking software to find weak points, at Softtech, a Turkish financial software company. It matters because it tries to show how such testing can fit inside strict banking regulations when no public playbook exists.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Non-intrusive reversible controls (§3.2.1, §3.3) may suppress the very failure observations needed to support the 'enhanced resilience' claim; no evidence yet that meaningful experiments survive compliance constraints.","rationale":"I read the paper in good faith: it is a clearly written design proposal for using Jernberg's four-phase model with LitmusChaos on Softtech's Plateau platform, and it honestly defers empirical validation. The central assertion in §5 is modest ('aims to enhance'), but even that aim requires that the proposed experiments retain enough disruptive force to provide information. The weakest link is therefore not Jernberg's model or LitmusChaos per se; it is the combination of non-intrusive, reversible controls (§3.2.1) and automated rollback (§3.3) with the need to measure resilience. If these controls are strict enough to satisfy financial regulation, they may suppress the failure conditions needed to observe recovery behavior, making the framework compliant but uninformative. This is internally tractable: a pilot with and without the controls would show whether the compliance layer preserves experimental signal. The reader's verdict of CONDITIONAL is appropriate; I would keep it unchanged because the concern can be resolved by such a pilot or by an explicit restatement of the paper as a non-evaluated design proposal. My disagreement with the reader's weakest assumption is only partial: they name the same non-intrusive-controls issue, but I see the more load-bearing version as the internal tension that controls can prevent informative experiments, not merely the question of whether regulators approve them.","tokens_in":6186,"tokens_out":4495,"duration_ms":45606,"concrete_test":"Run a pilot on a Plateau staging cluster: pick one Discovery-phase experiment, e.g., 500ms network latency injected into Plateau Process Manager. Execute it twice: (A) with the §3.2.1 non-intrusive controls and §3.3 auto-rollback active, (B) with standard blast-radius controls and no auto-rollback, with a longer observation window. Compare the deviations in latency, error rate, and availability, and whether the recovery/MTTR path is observed. If (A) detects no weakness or never reaches a steady-state deviation that (B) detects, then the compliance controls suppress the evidence on which the 'enhanced resilience' claim rests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5 is that the customized framework 'aims to enhance system resilience through progressive and iterative improvements.' For this to be true, the framework must not merely operate safely in Softtech's regulated environment; it must generate observations that can reveal resilience weaknesses. The design's own controls may prevent that. §3.2.1 mandates 'non-intrusive controls' so experiments have 'minimal impact, are reversible, safe, and transparent,' and §3.3 adds 'automated rollback mechanisms for stability when systems don't recover as expected.' Under these constraints, a fault injection could be rolled back precisely when it triggers the degraded behavior that chaos engineering is meant to expose, and 'minimal impact' experiments may never push the system off its steady state. The paper offers no argument or evidence that regulators will accept failures large enough to produce measurable deviations in MTTR, availability, or error rates—the metrics named in §3.3 as the future validation. This is an internal tension in the design, not simply a missing field study: the compliance and safety mechanisms can systematically select for experiments that cannot falsify Plateau's resilience, in which case the claimed improvement would be unsupported by the framework's own logic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This short paper (EASE ’25 format, 4 pages) proposes a customized chaos engineering framework for Softtech, a software company serving the financial sector. The design is based on Jernberg's four activity phases (Discovery, Implementation, Sophistication, Expansion), uses LitmusChaos as the orchestration tool, and integrates monitoring (Prometheus, Grafana, Kiali, New Relic), automated rollback, reporting, and compliance controls. The paper answers two research questions: (RQ1) how Softtech's infrastructure, business priorities, and organizational context shape the customization, and (RQ2) what key activities and components are needed. The authors state explicitly that empirical validation and implementation are future work (§1, §5), and the conclusion frames the framework as a design that 'aims to enhance system resilience through progressive and iterative improvements.' The contribution is therefore a design proposal rather than an evaluated framework.","tokens_in":6372,"tokens_out":3454,"duration_ms":40411,"significance":"If the proposed framework were implemented and empirically validated, it could provide rare practical guidance for adopting chaos engineering in highly regulated financial environments, an area the authors correctly identify as under-documented (§2.4). The paper is clearly organized, cites relevant principles and tools, and honestly discloses both the absence of validation and the use of AI for language polishing. Its main value is as a starting point for a real deployment study, not as evidence that the described framework currently enhances resilience. The central claim, however, is not yet supported by the manuscript's own content.","major_comments":[{"comment":"The paper's central claim that the framework 'enhances system resilience' (abstract and §5: 'aims to enhance system resilience through progressive and iterative improvements') is not supported by any measurements or qualitative evaluation. The authors explicitly defer 'empirical validation and implementation' to future work (§1) and state that metrics such as MTTR, availability, and error rates are only planned for future validation (§3.3). The conclusion's assertion of 'improved resilience testing and incident reduction' is thus beyond what the paper can establish. Please either include a pilot evaluation or reframe the title, abstract, and conclusion to present this as an unevaluated design proposal whose effectiveness remains to be tested.","section":"Abstract; §5 Conclusion; §1"},{"comment":"There is an internal tension between the compliance constraints and the experimental value of the proposed chaos exercises. Section 3.2.1 mandates 'non-intrusive controls' ensuring experiments 'have minimal impact, are reversible, safe, and transparent,' and §3.3 adds 'automated rollback mechanisms for stability when systems don't recover as expected.' These mechanisms can suppress the very failure observations that chaos engineering is meant to produce: if a fault is rolled back as soon as the system degrades, the system never exhibits the measurable deviation that would expose weaknesses, and 'minimal impact' experiments may never push the system off its steady state. The paper should address this tension explicitly, for example by defining non-intrusiveness with respect to business-impact thresholds rather than fault magnitude, or by specifying safety-trigger criteria that allow experiments to run to completion while still protecting production workloads.","section":"§3.2.1 and §3.3"},{"comment":"The paper adopts Jernberg's four-phase activity model with little justification for why those particular phases are necessary or sufficient for Softtech, and it does not compare this choice with the alternative frameworks surveyed in §2.2 (e.g., Chess, ChaosTwin). Because RQ1 and RQ2 are answered entirely descriptively, the reader cannot tell which design decisions follow from Softtech's specific requirements (zero downtime, regulatory compliance, on-premise deployments) and which are generic defaults. A more convincing customization would map each requirement to a concrete design decision or evaluation criterion, and would explain how the proposed framework improves on existing frameworks rather than merely relabeling them.","section":"§3.2; §4"}],"minor_comments":[{"comment":"Plateau is described in §3.1.1 as an 'open-source, low-code microservice framework on Kubernetes,' but §4 calls it an 'open-source, cloud-based Kubernetes platform.' Please align the wording to avoid an apparent inconsistency.","section":"§3.1.1 vs. §4"},{"comment":"The informal contraction 'we'll' appears in the Discovery phase description ('we'll define a backlog'); use 'we will' or restructure the sentence for a formal proceedings style.","section":"§3.2"},{"comment":"Several references are informal blog posts or vendor pages (e.g., [7], [9], [27]); while such sources are common in practitioner-oriented writing, the paper would be stronger if the tool taxonomy also included peer-reviewed sources or if the authors stated why these sources were chosen.","section":"References"},{"comment":"The ACM reference formatting contains placeholder values ('https://doi.org/XXXXXXX.XXXXXXX') and the paper's copyright/ISBN lines contain 'XXXX-X'; these must be completed or removed before publication.","section":"References"},{"comment":"The survey of existing frameworks is brief and descriptive; consider a small comparison table summarizing each framework's target environment, fault-injection types, and suitability for regulated production systems.","section":"§2.2 and §2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible design-proposal contribution for a short EASE paper, and the authors are transparent about the absence of validation. The main issue is the mismatch between the title/conclusion ('Enhanced System Resilience', 'improved resilience testing and incident reduction') and the explicitly deferred evaluation. A revised version that narrows the claims or includes even a small pilot study (e.g., one LitmusChaos experiment on a staging environment with before/after metrics) would change my assessment substantially. I also note that the tension between 'non-intrusive, reversible' controls and the informative power of chaos experiments is a substantive design issue, not just a missing field study; it needs to be resolved in the design itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it is not a research result. It is a four-page design proposal from Softtech, the IT arm of Turkey's Isbank, for introducing chaos engineering on their Kubernetes-based Plateau platform. It is clearly written and honest: the authors state upfront that empirical validation is future work, and the related-work section correctly says there are few concrete examples of chaos engineering in regulated financial environments. That gap is real, and this paper is one attempt to fill it.\n\nWhat is new is narrow. The framework itself is Jernberg's four phases (Discovery, Implementation, Sophistication, Expansion) re-staged with LitmusChaos, Prometheus, Grafana, and MongoDB. The only real original element is the application context: on-premise, mainframe-to-mobile, compliance-heavy banking. That context shapes their choices, e.g., non-intrusive controls, audits, and compliance reporting. As a template for a practitioner in a similar environment, it is plausible.\n\nThe soft spots are not hidden. There is no data, no pilot, no comparison against alternative frameworks. The central claim in Section 5 is only that the framework “aims to enhance resilience,” which is honest but unsupported. More interesting is an internal tension the paper does not address. Section 3.2.1 mandates non-intrusive, reversible, safe, transparent controls, and Section 3.3 adds automated rollback when systems don't recover. Those are sensible compliance measures, but they can systematically select for experiments that never push the system off its steady state. If an experiment gets rolled back exactly when it reveals a recovery problem, the exercise may produce compliance theater rather than resilience insights. The paper offers no argument that regulators would accept failures large enough to move the metrics (MTTR, error rates) they say they plan to track. This is a real design flaw, not just a missing field study. It doesn't sink the paper as a proposal, but it needs to be addressed.\n\nWho is this for? Practitioners in regulated or legacy environments who want a starting point for their own chaos engineering adoption. Researchers won't find new technique or evidence. The bibliography is appropriate and the writing is transparent about the use of AI for language polishing, which is fine.\n\nI'd send it to peer review as a short industry design paper. It deserves a referee's time, and a good referee would push the authors on the safety-vs-diagnostic-power tension and on which specific design decisions the regulatory context forced them to make. Just don't expect it to change any research directions.","headline":"A clear, honest design proposal for chaos engineering in a regulated financial context, but it is a proposal: no data, no validation, and the safety controls may undercut the experiments' diagnostic power.","tokens_in":6919,"tokens_out":2722,"would_cite":false,"duration_ms":26755,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A four-phase chaos engineering framework with compliance controls aims to harden a regulated bank's platform.","keywords":["Chaos Engineering","System Resilience","Software Reliability","Financial sector compliance","Kubernetes","LitmusChaos","Fault injection","Design proposal"],"falsifier":"A pilot on the Plateau platform that runs Discovery-Implementation experiments and compares MTTR, availability, or error rates against a control period, or a regulatory review that rejects the non-intrusive control design, would settle whether the framework delivers the promised resilience improvement.","tokens_in":5975,"feed_emoji":"🧪","tokens_out":4786,"duration_ms":43711,"temperature":0.7,"pith_summary":"This paper proposes a custom chaos engineering framework for Softtech, a software company whose main client is a large bank. The design rests on a four-phase activity cycle—Discovery, Implementation, Sophistication, Expansion—run against Softtech's Kubernetes-based Plateau platform, with LitmusChaos injecting faults and compliance controls governing every experiment. The stated purpose is to improve system resilience and reduce downtime while meeting financial-sector regulations. The paper is explicitly a design proposal; empirical validation and pilot implementation are flagged as future work.","feed_headline":"Four phases to bring chaos engineering to a regulated bank","feed_subtitle":"The design pairs LitmusChaos fault injection with compliance controls to reduce downtime while staying within financial rules.","key_machinery":"The central object is the four-phase activity model (Discovery, Implementation, Sophistication, Expansion), taken from prior chaos engineering work and customized to Softtech. Discovery produces an experiment backlog, Implementation runs fault-injection experiments, Sophistication automates and matures them, and Expansion broadens their scope. Around this cycle the framework wraps LitmusChaos as the Kubernetes-native fault-injection orchestrator, monitoring tools for steady-state validation, automated rollbacks, and the compliance controls that keep experiments reversible, transparent, and auditable. The compliance layer is the piece that lets the proposal work in a heavily regulated financial environment.","core_discovery":"The paper's central claim is that Softtech's infrastructure, business priorities, and organizational context require a tailored chaos engineering approach, and that its two research questions can be answered by a framework built on four phases. Discovery builds a prioritized backlog of chaos experiments by analyzing Plateau's architecture and dependencies. Implementation executes selected experiments through LitmusChaos inside CI/CD pipelines. Sophistication refines and automates experiments using a chaos maturity model, and Expansion widens coverage to more components and teams. Compliance controls—non-intrusive reversible experiments, logging and monitoring, audits, and reporting—are the mechanism that makes the proposal viable in a regulated financial setting.","pith_inferences":["A testable extension is that teams running the full cycle for several quarters would show lower MTTR and fewer high-severity incidents; the paper does not yet provide such data.","The compliance constraints may limit fault severity and randomness, so the most informative production chaos experiments could be the hardest to approve; whether the remaining tests preserve enough signal is an open question.","The same four-phase pattern with LitmusChaos could transfer to other regulated Kubernetes platforms, but each organization would need its own compliance and approval workflow."],"forward_implications":["If the framework is implemented as designed, Softtech can run controlled failure experiments on Plateau and catch resilience weaknesses before they cause real incidents.","Tracking metrics such as MTTR, service availability, and error rates during experiments would give Softtech measurable feedback for meeting its SLAs and zero-downtime goal.","Integrating the phases with CI/CD pipelines would make resilience testing a continuous activity rather than an occasional exercise.","The Expansion phase would let the practice spread from the initial Plateau components to additional applications and teams across Softtech."],"supporting_citations":[{"why":"Supplies the four-phase activity model (Discovery, Implementation, Sophistication, Expansion) that the framework is built on.","marker":"[11]"},{"why":"Provides LitmusChaos, the Kubernetes-native fault-injection orchestrator chosen for the framework's core experimentation.","marker":"[15]"},{"why":"Defines the Chaos Maturity Model used in the Sophistication phase to refine and automate experiments.","marker":"[3]"},{"why":"Establishes the core principles of chaos engineering (steady state, hypothesis, monitoring, minimizing impact) that ground the proposal.","marker":"[2]"},{"why":"Documents the origin and practice of chaos engineering, justifying the approach for system resilience.","marker":"[26]"}],"fun_headline_variants":["Four-phase chaos plan for compliant financial resilience","Four-phase chaos for finance, compliant by design","A four-phase path to chaos-resilient finance","Chaos with compliance: a four-phase approach for banks","Four-phase chaos engineering for regulated finance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plan assumes that non-intrusive, reversible, compliance-supervised chaos experiments still produce meaningful resilience insights and that regulators will accept the controls as sufficient.","fun_headline_variants_meta":{"raw":{"variants":["Four-phase chaos plan for compliant financial resilience","Four-phase chaos for finance, compliant by design","A four-phase path to chaos-resilient finance","Chaos with compliance: a four-phase approach for banks","Four-phase chaos engineering for regulated finance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001276,"raw_usage":{"total_tokens":5135,"prompt_tokens":779,"completion_tokens":4356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":4285}},"tokens_in":395,"tokens_out":4356,"duration_ms":31595,"temperature":1.0,"reasoning_tokens":4285,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:05.927279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A pilot on the Plateau platform that runs Discovery-Implementation experiments and compares MTTR, availability, or error rates against a control period, or a regulatory review that rejects the non-intrusive control design, would settle whether the framework delivers the promised resilience improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the four-phase activity model (Discovery, Implementation, Sophistication, Expansion) that the framework is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LitmusChaos, the Kubernetes-native fault-injection orchestrator chosen for the framework's core experimentation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Chaos Maturity Model used in the Sophistication phase to refine and automate experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the core principles of chaos engineering (steady state, hypothesis, monitoring, minimizing impact) that ground the proposal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the origin and practice of chaos engineering, justifying the approach for system resilience."}],"review_version":1}