Designing a Custom Chaos Engineering Framework for Enhanced System Resilience at Softtech

Burak Tuzlutas; Burak Yesiltas; Ethem Utku Aktas

REVIEW 3 major objections 5 minor 27 references

Designing a Custom Chaos Engineering Framework for Enhanced System Resilience at Softtech

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A four-phase chaos engineering framework with compliance controls aims to harden a regulated bank's platform.

desk verdict A clear, honest design proposal for chaos engineering in a regulated financial context, but it is a proposal: no data, no validation, and the safety controls may undercut the experiments' diagnostic power. read the letter →

arxiv 2506.14281 v1 pith:NLU7WX4K submitted 2025-06-17 cs.SE

Ethem Utku Aktas , Burak Tuzlutas , Burak Yesiltas This is my paper · ORCID

classification cs.SE

keywords ChaosEngineeringSystemResilienceSoftwareReliabilityFinancialsectorcomplianceKubernetesLitmusFaultinjectionDesignproposal

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a custom chaos engineering framework for Softtech, a software company whose main client is a large bank. The design rests on a four-phase activity cycle—Discovery, Implementation, Sophistication, Expansion—run against Softtech's Kubernetes-based Plateau platform, with LitmusChaos injecting faults and compliance controls governing every experiment. The stated purpose is to improve system resilience and reduce downtime while meeting financial-sector regulations. The paper is explicitly a design proposal; empirical validation and pilot implementation are flagged as future work.

What carries the argument

The central object is the four-phase activity model (Discovery, Implementation, Sophistication, Expansion), taken from prior chaos engineering work and customized to Softtech. Discovery produces an experiment backlog, Implementation runs fault-injection experiments, Sophistication automates and matures them, and Expansion broadens their scope. Around this cycle the framework wraps LitmusChaos as the Kubernetes-native fault-injection orchestrator, monitoring tools for steady-state validation, automated rollbacks, and the compliance controls that keep experiments reversible, transparent, and auditable. The compliance layer is the piece that lets the proposal work in a heavily regulated financial environment.

What would settle it

A pilot on the Plateau platform that runs Discovery-Implementation experiments and compares MTTR, availability, or error rates against a control period, or a regulatory review that rejects the non-intrusive control design, would settle whether the framework delivers the promised resilience improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that Softtech's infrastructure, business priorities, and organizational context require a tailored chaos engineering approach, and that its two research questions can be answered by a framework built on four phases. Discovery builds a prioritized backlog of chaos experiments by analyzing Plateau's architecture and dependencies. Implementation executes selected experiments through LitmusChaos inside CI/CD pipelines. Sophistication refines and automates experiments using a chaos maturity model, and Expansion widens coverage to more components and teams. Compliance controls—non-intrusive reversible experiments, logging and monitoring, audits, and reporting—are the mechanism that makes the proposal viable in a regulated financial setting.

Load-bearing premise

The plan assumes that non-intrusive, reversible, compliance-supervised chaos experiments still produce meaningful resilience insights and that regulators will accept the controls as sufficient.

Editorial extensions

If this is right

If the framework is implemented as designed, Softtech can run controlled failure experiments on Plateau and catch resilience weaknesses before they cause real incidents.
Tracking metrics such as MTTR, service availability, and error rates during experiments would give Softtech measurable feedback for meeting its SLAs and zero-downtime goal.
Integrating the phases with CI/CD pipelines would make resilience testing a continuous activity rather than an occasional exercise.
The Expansion phase would let the practice spread from the initial Plateau components to additional applications and teams across Softtech.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is that teams running the full cycle for several quarters would show lower MTTR and fewer high-severity incidents; the paper does not yet provide such data.
The compliance constraints may limit fault severity and randomness, so the most informative production chaos experiments could be the hardest to approve; whether the remaining tests preserve enough signal is an open question.
The same four-phase pattern with LitmusChaos could transfer to other regulated Kubernetes platforms, but each organization would need its own compliance and approval workflow.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know about this paper is that it is not a research result. It is a four-page design proposal from Softtech, the IT arm of Turkey's Isbank, for introducing chaos engineering on their Kubernetes-based Plateau platform. It is clearly written and honest: the authors state upfront that empirical validation is future work, and the related-work section correctly says there are few concrete examples of chaos engineering in regulated financial environments. That gap is real, and this paper is one attempt to fill it.

What is new is narrow. The framework itself is Jernberg's four phases (Discovery, Implementation, Sophistication, Expansion) re-staged with LitmusChaos, Prometheus, Grafana, and MongoDB. The only real original element is the application context: on-premise, mainframe-to-mobile, compliance-heavy banking. That context shapes their choices, e.g., non-intrusive controls, audits, and compliance reporting. As a template for a practitioner in a similar environment, it is plausible.

The soft spots are not hidden. There is no data, no pilot, no comparison against alternative frameworks. The central claim in Section 5 is only that the framework “aims to enhance resilience,” which is honest but unsupported. More interesting is an internal tension the paper does not address. Section 3.2.1 mandates non-intrusive, reversible, safe, transparent controls, and Section 3.3 adds automated rollback when systems don't recover. Those are sensible compliance measures, but they can systematically select for experiments that never push the system off its steady state. If an experiment gets rolled back exactly when it reveals a recovery problem, the exercise may produce compliance theater rather than resilience insights. The paper offers no argument that regulators would accept failures large enough to move the metrics (MTTR, error rates) they say they plan to track. This is a real design flaw, not just a missing field study. It doesn't sink the paper as a proposal, but it needs to be addressed.

Who is this for? Practitioners in regulated or legacy environments who want a starting point for their own chaos engineering adoption. Researchers won't find new technique or evidence. The bibliography is appropriate and the writing is transparent about the use of AI for language polishing, which is fine.

I'd send it to peer review as a short industry design paper. It deserves a referee's time, and a good referee would push the authors on the safety-vs-diagnostic-power tension and on which specific design decisions the regulatory context forced them to make. Just don't expect it to change any research directions.

Referee Report

3 major / 5 minor

Summary. This short paper (EASE ’25 format, 4 pages) proposes a customized chaos engineering framework for Softtech, a software company serving the financial sector. The design is based on Jernberg's four activity phases (Discovery, Implementation, Sophistication, Expansion), uses LitmusChaos as the orchestration tool, and integrates monitoring (Prometheus, Grafana, Kiali, New Relic), automated rollback, reporting, and compliance controls. The paper answers two research questions: (RQ1) how Softtech's infrastructure, business priorities, and organizational context shape the customization, and (RQ2) what key activities and components are needed. The authors state explicitly that empirical validation and implementation are future work (§1, §5), and the conclusion frames the framework as a design that 'aims to enhance system resilience through progressive and iterative improvements.' The contribution is therefore a design proposal rather than an evaluated framework.

Significance. If the proposed framework were implemented and empirically validated, it could provide rare practical guidance for adopting chaos engineering in highly regulated financial environments, an area the authors correctly identify as under-documented (§2.4). The paper is clearly organized, cites relevant principles and tools, and honestly discloses both the absence of validation and the use of AI for language polishing. Its main value is as a starting point for a real deployment study, not as evidence that the described framework currently enhances resilience. The central claim, however, is not yet supported by the manuscript's own content.

major comments (3)

[Abstract; §5 Conclusion; §1] The paper's central claim that the framework 'enhances system resilience' (abstract and §5: 'aims to enhance system resilience through progressive and iterative improvements') is not supported by any measurements or qualitative evaluation. The authors explicitly defer 'empirical validation and implementation' to future work (§1) and state that metrics such as MTTR, availability, and error rates are only planned for future validation (§3.3). The conclusion's assertion of 'improved resilience testing and incident reduction' is thus beyond what the paper can establish. Please either include a pilot evaluation or reframe the title, abstract, and conclusion to present this as an unevaluated design proposal whose effectiveness remains to be tested.
[§3.2.1 and §3.3] There is an internal tension between the compliance constraints and the experimental value of the proposed chaos exercises. Section 3.2.1 mandates 'non-intrusive controls' ensuring experiments 'have minimal impact, are reversible, safe, and transparent,' and §3.3 adds 'automated rollback mechanisms for stability when systems don't recover as expected.' These mechanisms can suppress the very failure observations that chaos engineering is meant to produce: if a fault is rolled back as soon as the system degrades, the system never exhibits the measurable deviation that would expose weaknesses, and 'minimal impact' experiments may never push the system off its steady state. The paper should address this tension explicitly, for example by defining non-intrusiveness with respect to business-impact thresholds rather than fault magnitude, or by specifying safety-trigger criteria that allow experiments to run to completion while still protecting production workloads.
[§3.2; §4] The paper adopts Jernberg's four-phase activity model with little justification for why those particular phases are necessary or sufficient for Softtech, and it does not compare this choice with the alternative frameworks surveyed in §2.2 (e.g., Chess, ChaosTwin). Because RQ1 and RQ2 are answered entirely descriptively, the reader cannot tell which design decisions follow from Softtech's specific requirements (zero downtime, regulatory compliance, on-premise deployments) and which are generic defaults. A more convincing customization would map each requirement to a concrete design decision or evaluation criterion, and would explain how the proposed framework improves on existing frameworks rather than merely relabeling them.

minor comments (5)

[§3.1.1 vs. §4] Plateau is described in §3.1.1 as an 'open-source, low-code microservice framework on Kubernetes,' but §4 calls it an 'open-source, cloud-based Kubernetes platform.' Please align the wording to avoid an apparent inconsistency.
[§3.2] The informal contraction 'we'll' appears in the Discovery phase description ('we'll define a backlog'); use 'we will' or restructure the sentence for a formal proceedings style.
[References] Several references are informal blog posts or vendor pages (e.g., [7], [9], [27]); while such sources are common in practitioner-oriented writing, the paper would be stronger if the tool taxonomy also included peer-reviewed sources or if the authors stated why these sources were chosen.
[References] The ACM reference formatting contains placeholder values ('https://doi.org/XXXXXXX.XXXXXXX') and the paper's copyright/ISBN lines contain 'XXXX-X'; these must be completed or removed before publication.
[§2.2 and §2.3] The survey of existing frameworks is brief and descriptive; consider a small comparison table summarizing each framework's target environment, fault-injection types, and suitability for regulated production systems.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a design proposal that imports an external framework and tools; no prediction or derivation reduces to its own inputs.

full rationale

This paper does not present a quantitative derivation or empirical prediction. It is a design proposal, and Section 1 explicitly states that 'empirical validation and implementation are planned for future work.' The four-phase activity set (Discovery, Implementation, Sophistication, Expansion) is taken from Jernberg's external framework [11], and LitmusChaos, Prometheus, Grafana, Kiali, New Relic, and MongoDB are adopted as external tools. There is no fitted parameter, no equation, and no quantity defined in terms of another quantity of the same paper, so there is nothing that could reduce to its own inputs by construction. The central claim in Section 5 is that the framework 'aims to enhance system resilience through progressive and iterative improvements'; this is an aspiration, not a measured outcome, and the paper does not claim to have validated it. The skeptical concern that non-intrusive, reversible controls may suppress the failure signal needed to falsify resilience is a genuine threat to future validity, but it is not a circularity: no result is being derived from the controls themselves. There are no self-citations at all, and the cited external work (Jernberg, LitmusChaos, Netflix, Gremlin) is independent of the authors. Accordingly, the circularity burden is minimal and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on qualitative assumptions: that established chaos engineering activities transfer to Softtech's regulated environment, and that the selected commodity tooling integrates with Plateau. No empirical evidence is supplied in the paper.

assumptions (4)

domain assumption Adopting chaos engineering principles as described by [2,22] improves resilience.
The design assumes the benefits claimed in the literature, such as lower MTTR/MTTD and fewer outages, transfer to Softtech; no local evidence is provided.
domain assumption Jernberg's four activity phases (Discovery, Implementation, Sophistication, Expansion) are the correct decomposition for Softtech.
Section 3.2 adopts them without deriving or validating them for this context.
domain assumption Non-intrusive controls (minimal impact, reversible, safe) can satisfy financial regulations while producing meaningful experiments.
Section 3.2.1 introduces these controls but provides no evidence that regulators accept them or that they preserve experimental value.
domain assumption LitmusChaos works effectively with the Plateau platform and supports the required failure injection and rollback.
Section 3.3 asserts suitability without integration testing.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Designing a Custom Chaos Engineering Framework for Enhanced System Resilience at Softtech." pith.science (2026). https://pith.science/paper/NLU7WX4K

@misc{pith2026250614281,
  author       = {Pith},
  title        = {Pith review of: Designing a Custom Chaos Engineering Framework for Enhanced System Resilience at Softtech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLU7WX4K}},
  note         = {Machine review of arXiv:2506.14281}
}

read the original abstract

Chaos Engineering is a discipline which enhances software resilience by introducing faults to observe and improve system behavior intentionally. This paper presents a design proposal for a customized Chaos Engineering framework tailored for Softtech, a leading software development company serving the financial sector. It outlines foundational concepts and activities for introducing Chaos Engineering within Softtech, while considering financial sector regulations. Building on these principles, the framework aims to be iterative and scalable, enabling development teams to progressively improve their practices. The study addresses two primary questions: how Softtech's unique infrastructure, business priorities, and organizational context shape the customization of its Chaos Engineering framework and what key activities and components are necessary for creating an effective framework tailored to Softtech's needs.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 26 canonical work pages

[1]

Merishani Arsecularatne and Ruwan Wickramarachchi. 2023. Adoptability of Chaos Engineering with DevOps to Stimulate the Software Delivery Performance. In 2023 International Research Conference on Smart Computing and Systems Engi- neering (SCSE), Vol. 6. IEEE, 1–8

work page 2023
[2]

Ali Basiri, Niosha Behnam, Ruud De Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos engineering. IEEE Software 33, 3 (2016), 35–41

work page 2016
[3]

Ali Basiri, Aaron Blohowiak, Lorin Hochstein, Nora Jones, and Casey Rosen- thal. 2017. Chaos Engineering: Building Confidence in System Behavior Through Experiments. O’Reilly Media, Inc

work page 2017
[4]

Ali Basiri, Lorin Hochstein, Nora Jones, and Haley Tucker. 2019. Automating chaos experiments in production. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 31–40

work page 2019
[5]

Aaron Blohowiak, Ali Basiri, Lorin Hochstein, and Casey Rosenthal. 2016. A platform for automating chaos experiments. In2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) . IEEE, 5–8

work page 2016
[6]

Carlos Camacho, Pablo C Cañizares, Luis Llana, and Alberto Núñez. 2022. Chaos as a Software Product Line—a platform for improving open hybrid-cloud systems resiliency. Software: Practice and Experience 52, 7 (2022), 1581–1614

work page 2022
[7]

Navdeep Singh Gill. 2024. Chaos engineering: tools, principles and best practices. https://www.xenonstack.com/insights/chaos-engineering. Accessed: 2025-02-23

work page 2024
[8]

Grafana Labs. 2023. Grafana. https://grafana.com/

work page 2023

Show all 27 references

[9]

Gremlin. 2023. Chaos engineering: the history, principles, and prac- tice. https://www.gremlin.com/community/tutorials/chaos-engineering-the- history-principles-and-practice. Accessed: 2024-11-18

2023
[10]

Gremlin Inc. 2025. Gremlin. https://www.gremlin.com/. Accessed: 2025-11-12

2025
[11]

Hugo Jernberg. 2020. Building a Framework for Chaos Engineering. LU-CS-EX (2020)

2020
[12]

Hugo Jernberg, Per Runeson, and Emelie Engström. 2020. Getting Started with Chaos Engineering-design of an implementation framework in practice. In Pro- ceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . 1–10

2020
[13]

Dominik Kesim, André van Hoorn, Sebastian Frank, and Matthias Häussler. 2020. Identifying and prioritizing chaos experiments by using established risk analysis techniques. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 229–240

2020
[14]

Kiali Authors. 2023. Kiali. https://kiali.io/

2023
[15]

LitmusChaos. 2020. LitmusChaos - open source chaos engineering platform. https://litmuschaos.io/. Accessed: 2024-12-10

2020
[16]

Sehrish Malik, Moeen Ali Naqvi, and Leon Moonen. 2023. Chess: A framework for evaluation of self-adaptive systems based on chaos engineering. In2023 IEEE/ACM 18th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). IEEE, 195–201

2023
[17]

Chaos Mesh. 2025. Chaos Mesh: a powerful chaos engineering platform for kubernetes. https://chaos-mesh.org/docs. Accessed: 2025-03-02

2025
[18]

MongoDB, Inc. 2023. MongoDB. https://www.mongodb.com/

2023
[19]

Moeen Ali Naqvi, Sehrish Malik, Merve Astekin, and Leon Moonen. 2022. On evaluating self-adaptive and self-healing systems using chaos engineering. In 2022 IEEE international conference on autonomic computing and self-organizing systems (ACSOS). IEEE, 1–10

2022
[20]

Netflix. 2025. Chaos Monkey. https://netflix.github.io/chaosmonkey/. Accessed: 2025-01-10

2025
[21]

New Relic, Inc. 2023. New Relic. https://newrelic.com/

2023
[22]

Joshua Owotogbe, Indika Kumara, Willem-Jan Van Den Heuvel, and Damian An- drew Tamburri. 2024. Chaos Engineering: A Multi-Vocal Literature Review.arXiv preprint arXiv:2412.01416 (2024)

2024 arXiv
[23]

Filippo Poltronieri, Mauro Tortonesi, and Cesare Stefanelli. 2021. Chaostwin: A chaos engineering and digital twin approach for the design of resilient it services. In 2021 17th International Conference on Network and Service Management (CNSM). IEEE, 234–238

2021
[24]

Filippo Poltronieri, Mauro Tortonesi, and Cesare Stefanelli. 2022. A chaos en- gineering approach for improving the resiliency of it services configurations. In NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium . IEEE, 1–6

2022
[25]

Prometheus Authors. 2023. Prometheus. https://prometheus.io/

2023
[26]

Casey Rosenthal and Nora Jones. 2020. Chaos engineering: system resiliency in practice. O’Reilly Media

2020
[27]

Rahul Singh. 2024. Chaos Engineering tools in 2024. https://www.devopsschool. com/blog/chaos-engineering-tools-in-2024/. Accessed: 2024-12-21

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Merishani Arsecularatne and Ruwan Wickramarachchi. 2023. Adoptability of Chaos Engineering with DevOps to Stimulate the Software Delivery Performance. In 2023 International Research Conference on Smart Computing and Systems Engi- neering (SCSE), Vol. 6. IEEE, 1–8

work page 2023

[2] [2]

Ali Basiri, Niosha Behnam, Ruud De Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos engineering. IEEE Software 33, 3 (2016), 35–41

work page 2016

[3] [3]

Ali Basiri, Aaron Blohowiak, Lorin Hochstein, Nora Jones, and Casey Rosen- thal. 2017. Chaos Engineering: Building Confidence in System Behavior Through Experiments. O’Reilly Media, Inc

work page 2017

[4] [4]

Ali Basiri, Lorin Hochstein, Nora Jones, and Haley Tucker. 2019. Automating chaos experiments in production. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 31–40

work page 2019

[5] [5]

Aaron Blohowiak, Ali Basiri, Lorin Hochstein, and Casey Rosenthal. 2016. A platform for automating chaos experiments. In2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) . IEEE, 5–8

work page 2016

[6] [6]

Carlos Camacho, Pablo C Cañizares, Luis Llana, and Alberto Núñez. 2022. Chaos as a Software Product Line—a platform for improving open hybrid-cloud systems resiliency. Software: Practice and Experience 52, 7 (2022), 1581–1614

work page 2022

[7] [7]

Navdeep Singh Gill. 2024. Chaos engineering: tools, principles and best practices. https://www.xenonstack.com/insights/chaos-engineering. Accessed: 2025-02-23

work page 2024

[8] [8]

Grafana Labs. 2023. Grafana. https://grafana.com/

work page 2023