Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering

Joshua Owotogbe

REVIEW 5 major objections 5 minor 43 references

Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes a chaos engineering framework that deliberately injects faults—component, network, dependency—into LLM-based multi-agent systems to surface and fix emergent failures like hallucinations and cascading errors.

desk verdict A well-organized PhD research proposal on applying chaos engineering to LLM-based multi-agent systems, but with no results, no framework specification, and no data to evaluate. read the letter →

arxiv 2505.03096 v1 pith:PYRKMYZP submitted 2025-05-06 cs.MA cs.AIcs.SE

Joshua Owotogbe This is my paper · ORCID

classification cs.MAcs.AIcs.SE

keywords chaosengineeringLLM-basedmulti-agentsystemsrobustnesstestingfailureinjectionhallucinationcascadingfaultssystemreliabilitydesignscience

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model-based multi-agent systems (LLM-MAS) promise to handle complex tasks, but they fail in ways that are hard to predict: agents hallucinate, misunderstand each other, and let small errors cascade. This paper proposes bringing chaos engineering—the discipline of deliberately injecting failures into production-like systems—to LLM-MAS, so those weak points are found before they cause real damage. The central claim is that a systematic loop of fault injection, monitoring, and adaptation can both assess and improve the robustness of LLM-MAS. The paper lays out a three-phase design-science plan to catalogue failure modes, build the framework, and validate it in industrial settings; as of writing, only the first literature-review results exist. If the claim holds, teams deploying LLM agents in critical tasks would gain a concrete, measurable way to certify reliability.

What carries the argument

The load-bearing mechanism is the chaos engineering loop: a chaos module injects controlled disruptions—component faults, network faults, dependency faults—into a running LLM-MAS; a monitoring module gathers metrics such as response time, fault detection rate, error rate, and resource use; an adaptation module applies mitigation strategies; and feedback closes the loop. This mirrors the fault-injection loop proven in distributed systems, and it is the piece that carries the argument: robustness is treated as an experimentally observable system property rather than a property of individual prompts or models. The framework also contains a baseline and comparative analysis step and an audit dimension, but the loop is the engine.

What would settle it

Run the proposed chaos framework against a standard LLM-MAS benchmark with fixed tasks, repeatedly inject the same three fault types, and check whether fault detection rates and task-success rates are reproducible and causally linked. If a single injected agent-communication fault never changes end-task success, or if repeated identical injections give wildly different detection results, the premise that these failures are injectable, observable, and isolatable fails.

Watch

Extended reading notes

Core claim

The paper's claim is that chaos engineering—the practice of deliberately injecting failures into running systems to uncover weaknesses—can be systematically adapted to Large Language Model-based Multi-Agent Systems (LLM-MAS). It argues that the same controlled-experiment mindset that hardens distributed services can make LLM-MAS robust against the failure modes that make them risky in production: hallucinations, agent-to-agent communication breakdowns, resource contention, and cascading faults. To that end it specifies a three-phase research program: (1) a review of literature and open-source tools to catalogue LLM-MAS failure modes, (2) construction of a framework whose chaos module injects component, network, and dependency faults, with monitoring and adaptation modules observing metrics and applying mitigation strategies, and (3) validation through controlled experiments and action research with an industry partner, feeding an audit and certification process. The author presents this as a proposal; evaluation is planned and expected to complete by December 2028.

Load-bearing premise

The load-bearing premise is that the semantic failure modes of LLM-MAS—hallucinations, miscommunication, cascading faults—can be injected, observed, and measured in controlled experiments in the same way network, CPU, or dependency faults are injected into distributed systems; that premise is asserted, not yet demonstrated.

Editorial extensions

If this is right

Before deployment, teams could run chaos experiments in sandboxed environments to identify which parts of an agent network are single points of failure.
Robustness becomes measurable with shared metrics such as fault detection rate, error rate, and recovery time, allowing different LLM-MAS architectures to be compared and tracked over time.
Organizations could use chaos experiments as part of certification audits for industrial LLM-based applications, turning 'we think it is reliable' into 'we have seen it recover from injected failures.'
The approach complements existing safety techniques such as refusal training and cross-examination by testing emergent, system-level behavior instead of only model-level behavior.
Open-sourcing the fault-injection tools would let the community stress-test agents in development pipelines and share failure catalogs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension not spelled out in the paper: chaos experiments could become regression tests in CI/CD pipelines for LLM agents, with a resilience budget that blocks deployment if fault-recovery metrics drop below a threshold.
A testable extension in today's open frameworks is to inject semantic faults—adversarial instructions, truncated context, or contradictory messages from one agent—and measure whether downstream agents detect or compound them, which would directly quantify cascade paths.
The certification idea implies a shift from static benchmarks to dynamic resilience scoring; a concrete first step would be a standardized fault catalog with severities, so robustness reports are comparable across organizations.
The framework's split between infrastructure faults and semantic faults suggests that a chaos taxonomy for LLM-MAS is a prerequisite; the paper leaves that taxonomy to later phases.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

What you should know: this is not a research paper with results. It is a PhD research proposal, clearly labeled as such, with a main research question, three sub-questions, a design-science methodology, and an evaluation plan that ends in December 2028. The abstract's claim to "propose a chaos engineering framework" is a statement of intent, not a demonstrated contribution.

To its credit, the paper is honest and well-structured. The author explicitly identifies the project as PhD work, lays out three concrete research questions, and grounds the plan in design science and action research. The related work section shows awareness that chaos engineering has already been applied to ML and AI systems (refs 33-35, 37-38), which is a mature touch. The idea of injecting hallucinations, communication failures, and cascading faults into LLM-MAS is a plausible extension of existing chaos engineering practice, and the plan to validate with industry partners is reasonable.

The soft spots are the load-bearing ones. There is no implemented framework, no formal specification of the chaos module, no fault-injection interface, no failure models, and no metrics. The "first results" are a literature review that is under review at ACM Computing Surveys and not included here, plus a promise that GitHub analysis is ongoing. The core premise—that semantic, natural-language failures like hallucination can be injected, isolated, measured, and mitigated in the same way as CPU or network faults—is asserted but not examined. Without an operational definition of what it means to "inject a hallucination," there is no way to assess whether the framework is feasible or even falsifiable. The paper itself acknowledges this by presenting everything as future work, but that means there is no scientific claim to referee.

Who is this for? A research supervisor or a workshop on research proposals might find it useful for discussing methodology. A reader looking for a new technique, a benchmark, or evidence will get nothing.

Recommendation: desk reject for a research venue, but with encouragement to resubmit when the framework is actually specified and at least some pilot results exist. Not a serious candidate for peer review in its current form.

Referee Report

5 major / 5 minor

Summary. The manuscript arXiv:2505.03096 (cs.MA), "Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering," is a short position/research-plan paper for a Ph.D. project. It poses one main research question (RQ) and three subquestions (SQ1-SQ3) about systematically applying chaos engineering to LLM-based multi-agent systems (LLM-MAS). The proposed method is organized into three design-science phases: a conceptual phase combining a multivocal literature review with GitHub repository mining; a framework-development phase that includes a chaos module, monitoring module, and adaptation module applied to LLM-MAS; and an empirical-validation phase using controlled experiments and action research with Deloitte. Section IV reports as first results only that a multivocal review is under review at ACM Computing Surveys and that GitHub analysis is ongoing. Section V describes planned quantitative and qualitative metrics and states that the Ph.D. project is expected to be completed by December 2028.

Significance. If the proposed framework were realized and validated, the topic would be timely: adapting chaos engineering to LLM-MAS could give practitioners an explicit method for fault injection, resilience measurement, and certification of agent systems, complementing current robustness studies. The paper usefully identifies a genuine gap between conventional chaos-engineering practice and the semantic, emergent failure modes of LLM-MAS. However, the manuscript as submitted contains no implemented artifact, no experimental data, no failure-mode taxonomy, and no formal or empirical validation; every load-bearing element is deferred to future work or to an under-review literature review. The scientific contribution is therefore a research proposal rather than a demonstrated result, and the current text provides no falsifiable predictions or measurable claims that a reader can check.

major comments (5)

[Abstract and Section I (Introduction)] The central claim that the paper "proposes a chaos engineering framework" to identify vulnerabilities and "ensure reliable performance" is not supported by any concrete specification in the manuscript. A framework proposal needs at least an operational description of the fault-injection interface, the set of injectable failure models, the observability/metrics contract, and the adaptation loop. None of these is defined; Figure 1 is a schematic without formal semantics.
[Section III (Research Method and Contributions) and Figure 1] Figure 1 labels components such as "Chaos Module," "Component," "Network fault," and "Dependency Fault," but the text never explains how these are instantiated for LLM-MAS. In particular, the paper does not specify what "injecting a hallucination" or "injecting an agent communication failure" means: is it a prompt perturbation, a response override, a message-drop at the framework layer, or something else? Without an operational definition, the proposed experiments are not reproducible and the framework is not falsifiable.
[Section IV (First Results)] Section IV reports that the multivocal review "is being reviewed" in ACM Computing Surveys and that GitHub repository analysis is "currently" ongoing, but it reports no synthesized findings, no tool list, no failure-mode taxonomy, and no concrete results from those repositories. Since the paper claims Contribution 1 as a contribution, the absence of any content from that review or repository mining means the only claimed result is not present in the manuscript.
[Section V (Evaluation Plan)] The evaluation plan is entirely prospective. It lists metrics (response time, fault detection rates, error rates, resource utilization) and qualitative measures, but it does not define the baseline against which the framework will be compared, the specific LLM-MAS architectures to be tested, the fault scenarios to be instantiated, or the success thresholds. The stated completion date of December 2028 confirms that no validation exists yet; as written, the paper offers a plan rather than evidence.
[Self-referential validation chain] The manuscript validates its framework using (a) the author's own multivocal review, which is under review and not included, and (b) action research with an industry partner that will use the framework to audit systems. This makes the validation loop depend on entities external to the paper, with no public benchmark or independent artifact. Even as a research proposal, the paper should state explicitly what would count as failure of the framework and which comparisons would falsify its effectiveness.

minor comments (5)

[Throughout] The paper has several small grammatical and typographical issues, e.g., "Bing" in the list of LLMs is imprecise (the reference points to Microsoft Copilot), and phrases like "the review synthesizes insights" should be checked for tense consistency.
[Section II (Related Work)] Several citations are to arXiv preprints and blogs; for a robustness-testing proposal, it would be helpful to cite peer-reviewed chaos engineering and LLM-agent evaluation work more systematically, and to clearly distinguish practitioner sources from academic sources.
[Section I, RQ/SQ structure] The three subquestions overlap: SQ2 and SQ3 both concern robustness assurance, and SQ3's "audit and certify" function is not clearly separated from SQ2's "detecting and mitigating failures." A sentence explaining the distinction would improve clarity.
[Figure 1] Figure 1's text is very small and the boxes are not introduced in the body text; please enlarge the figure and add a caption that defines each module and arrow.
[References] Reference [33] is a non-peer-reviewed blog post, and references [34] and [35] are also non-archival sources; consider replacing them with peer-reviewed fault-injection studies for ML systems if available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an unfinalized research proposal, and its planned validation does not reduce to its inputs by construction.

full rationale

The manuscript contains no mathematical derivation, fitted-parameter prediction, or uniqueness theorem to audit. The central claim is a proposed design-science plan (Section III, Figure 1), and the only reported artifacts are an in-progress multivocal review and ongoing GitHub repository analysis (Section IV). The sole potentially self-referential item is footnote 2, which links to arXiv:2412.01416 as the under-review survey; even if that preprint is the author's own work, it is used only as a status report ('This work is being reviewed in ACM Computing Surveys'), not as load-bearing evidence that the chaos-engineering framework succeeds. The actual validation is explicitly deferred to controlled experiments, case studies, and action research with Deloitte (Section V), with completion expected by December 2028. This makes the central claim unverified rather than circular: no equation or definition in the paper makes the predicted robustness outcome equal to an input, and no validity claim is imported from a self-authored uniqueness theorem. Under the hard rule requiring a quoted reduction to the paper's own inputs, no circularity can be exhibited, so an honest non-finding is appropriate.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper provides no derivations, so the only load-bearing inputs are assumptions about the transferability of chaos engineering to LLM-MAS and the validity of planned qualitative validation.

assumptions (3)

domain assumption Chaos engineering principles developed for distributed systems can be adapted to LLM-based multi-agent systems.
The paper assumes chaos engineering's fault injection and recovery concepts carry over to semantic failures in LLM-MAS, including hallucinations and communication breakdowns, without demonstrating that these failures are measurable and injectable. This assumption underpins the entire framework.
domain assumption The design science and action research methodology is an accepted way to establish that a proposed framework is effective.
The author adopts design science (refs. 39, 40) and action research (refs. 41-43) as the validation path, which relies on the assumption that qualitative case studies can certify robustness, a methodological commitment rather than a measured result.
ad hoc to paper The author's multivocal literature review, currently under review, accurately identifies all relevant chaos engineering tools and failure modes.
Section IV states the review is the foundation for the framework but does not include it, so the paper depends on an unavailable and as yet unverified self-authored source.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering." pith.science (2026). https://pith.science/paper/PYRKMYZP

@misc{pith2026250503096,
  author       = {Pith},
  title        = {Pith review of: Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PYRKMYZP}},
  note         = {Machine review of arXiv:2505.03096}
}

read the original abstract

This study explores the application of chaos engineering to enhance the robustness of Large Language Model-Based Multi-Agent Systems (LLM-MAS) in production-like environments under real-world conditions. LLM-MAS can potentially improve a wide range of tasks, from answering questions and generating content to automating customer support and improving decision-making processes. However, LLM-MAS in production or preproduction environments can be vulnerable to emergent errors or disruptions, such as hallucinations, agent failures, and agent communication failures. This study proposes a chaos engineering framework to proactively identify such vulnerabilities in LLM-MAS, assess and build resilience against them, and ensure reliable performance in critical applications.

Figures

Figures reproduced from arXiv: 2505.03096 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 27 canonical work pages

[1]

Copilot

Microsoft, “Copilot.” https://copilot.cloud.microsoft/?auth=2, November 2024

work page 2024
[2]

Google AI for Developers

Google, “Google AI for Developers.” https://ai.google.dev/, November 2024

work page 2024
[3]

ChatGPT

OpenAI, “ChatGPT.” https://chatgpt.com/?ref=dotcom, November 2024

work page 2024
[4]

Attention is all you need,

A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

2017
[5]

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

2020
[6]

WebGPT: Browser- assisted question-answering with human feedback,

R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V . Kosaraju, W. Saunders, et al. , “WebGPT: Browser- assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332, 2021

arXiv 2021
[7]

Examining zero-shot vulnerability repair with large language models,

H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in 2023 IEEE Symposium on Security and Privacy (SP) , pp. 2339–2356, IEEE, 2023

work page 2023
[8]

A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,

Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, et al. , “A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023 , 2023

arXiv 2023

Show all 43 references

[9]

Why Agents are the next frontier of generative AI,

M. C. Lareina Yee and R. Roberts, “Why Agents are the next frontier of generative AI,” 2024. https: //www.mckinsey.com/capabilities/mckinsey-digital/our-insights/ why-agents-are-the-next-frontier-of-generative-ai[Accessed: December 2024]

2024
[10]

LLM multi-agent systems: Challenges and open problems,

S. Han, Q. Zhang, Y . Yao, W. Jin, Z. Xu, and C. He, “LLM multi-agent systems: Challenges and open problems,” arXiv preprint arXiv:2402.03578, 2024

2024 arXiv
[11]

Exploring autonomous agents through the lens of large language models: A review,

S. Barua, “Exploring autonomous agents through the lens of large language models: A review,” arXiv preprint arXiv:2404.04442 , 2024

2024 arXiv
[12]

Why solving multi-agent path finding with large language model has not succeeded yet,

W. Chen, S. Koenig, and B. Dilkina, “Why solving multi-agent path finding with large language model has not succeeded yet,” arXiv preprint arXiv:2401.03630, 2024

2024 arXiv
[13]

Metagpt: Meta programming for multi-agent collaborative framework,

S. Hong, X. Zheng, J. Chen, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, et al., “Metagpt: Meta programming for multi-agent collaborative framework,” arXiv preprint arXiv:2308.00352, 2023

2023 arXiv
[14]

Chaos engineering for resilience assessment of digital twins,

M. Fogli, C. Giannelli, F. Poltronieri, C. Stefanelli, and M. Tortonesi, “Chaos engineering for resilience assessment of digital twins,” IEEE Transactions on Industrial Informatics , vol. 20, no. 2, pp. 1134–1143, 2023

2023
[15]

Chaos engineering,

A. Basiri, N. Behnam, R. De Rooij, L. Hochstein, L. Kosewski, J. Reynolds, and C. Rosenthal, “Chaos engineering,” IEEE Software , vol. 33, no. 3, pp. 35–41, 2016

2016
[16]

Chaos engineering of ethereum blockchain clients,

L. Zhang, J. Ron, B. Baudry, and M. Monperrus, “Chaos engineering of ethereum blockchain clients,” Distributed Ledger Technologies: Re- search and Practice , vol. 2, no. 3, pp. 1–18, 2023

2023
[17]

Chaos engineering: At the age of AI and ML,

S. Chatterjee and N. Rauschmayr, “Chaos engineering: At the age of AI and ML,” in Conf42 Chaos Engineering 2022 , Conf42, March 2022. Online conference session

2022
[18]

The role of theory and theorising in design science research,

J. Venable, “The role of theory and theorising in design science research,” in Proceedings of the 1st international conference on design science in information systems and technology (DESRIST 2006) , pp. 1– 18, Citeseer, 2006

2006
[19]

Action research,

D. E. Avison, F. Lau, M. D. Myers, and P. A. Nielsen, “Action research,” Communications of the ACM , vol. 42, no. 1, pp. 94–97, 1999

1999
[20]

A survey on evaluating large language models in code generation tasks,

L. Chen, Q. Guo, H. Jia, Z. Zeng, X. Wang, Y . Xu, J. Wu, Y . Wang, Q. Gao, J. Wang, et al., “A survey on evaluating large language models in code generation tasks,” arXiv preprint arXiv:2408.16498 , 2024

2024 arXiv
[21]

LLMs for code: The potential, prospects, and problems,

T. Sharma, “LLMs for code: The potential, prospects, and problems,” in 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C), pp. 373–374, IEEE, 2024

2024
[22]

Weaknesses in LLM- generated code for embedded systems networking,

M. Dunne, K. Schram, and S. Fischmeister, “Weaknesses in LLM- generated code for embedded systems networking,” in 2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS), pp. 250–261, IEEE, 2024

2024
[23]

A survey on LLM- based multi-agent systems: workflow, infrastructure, and challenges,

X. Li, S. Wang, S. Zeng, Y . Wu, and Y . Yang, “A survey on LLM- based multi-agent systems: workflow, infrastructure, and challenges,” Vicinagearth, vol. 1, no. 1, p. 9, 2024

2024
[24]

System for systematic literature review using multiple AI agents: Concept and an empirical evaluation,

A. M. Sami, Z. Rasheed, K.-K. Kemell, M. Waseem, T. Kilamo, M. Saari, A. N. Duc, K. Syst ¨a, and P. Abrahamsson, “System for systematic literature review using multiple AI agents: Concept and an empirical evaluation,” arXiv preprint arXiv:2403.08399 , 2024

2024
[25]

Synchromesh: Reliable code generation from pre-trained language models,

G. Poesia, O. Polozov, V . Le, A. Tiwari, G. Soares, C. Meek, and S. Gulwani, “Synchromesh: Reliable code generation from pre-trained language models,” arXiv preprint arXiv:2201.11227 , 2022

2022 arXiv
[26]

Assessing the quality of GitHub Copilot’s code generation,

B. Yetistiren, I. Ozsoy, and E. Tuzun, “Assessing the quality of GitHub Copilot’s code generation,” in Proceedings of the 18th international con- ference on predictive models and data analytics in software engineering, pp. 62–71, 2022

2022
[27]

Can LLM replace Stack Overflow? a study on robustness and reliability of large language model code genera- tion,

L. Zhong and Z. Wang, “Can LLM replace Stack Overflow? a study on robustness and reliability of large language model code genera- tion,” in Proceedings of the AAAI Conference on Artificial Intelligence , pp. 21841–21849, 2024

2024
[28]

Improving alignment and robustness with circuit breakers,

A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, J. Z. Kolter, M. Fredrikson, and D. Hendrycks, “Improving alignment and robustness with circuit breakers,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

2024
[29]

Robust LLM safeguarding via refusal feature adversarial training,

L. Yu, V . Do, K. Hambardzumyan, and N. Cancedda, “Robust LLM safeguarding via refusal feature adversarial training,” arXiv preprint arXiv:2409.20089, 2024

2024 arXiv
[30]

Netsafe: Exploring the topological safety of multi-agent networks,

M. Yu, S. Wang, G. Zhang, J. Mao, C. Yin, Q. Liu, Q. Wen, K. Wang, and Y . Wang, “Netsafe: Exploring the topological safety of multi-agent networks,” arXiv preprint arXiv:2410.15686 , 2024

2024 arXiv
[31]

Trustagent: Towards safe and trustworthy LLM-based agents through agent constitution,

W. Hua, X. Yang, M. Jin, Z. Li, W. Cheng, R. Tang, and Y . Zhang, “Trustagent: Towards safe and trustworthy LLM-based agents through agent constitution,” in Trustworthy Multi-modal Foundation Models and AI Agents (TiFA), 2024

2024
[32]

Cut the crap: An economical communication pipeline for LLM-based multi-agent systems,

G. Zhang, Y . Yue, Z. Li, S. Yun, G. Wan, K. Wang, D. Cheng, J. X. Yu, and T. Chen, “Cut the crap: An economical communication pipeline for LLM-based multi-agent systems,” arXiv preprint arXiv:2410.02506, 2024

2024 arXiv
[33]

Chaos engineering in machine learning: Embracing the unpredictable to enhance system robustness,

E. Gomede, “Chaos engineering in machine learning: Embracing the unpredictable to enhance system robustness,” AI Monks, 2023. Updated: March 10, 2024

2023
[34]

Introducing chaos engineering to machine learning deploy- ments,

V . Raja, “Introducing chaos engineering to machine learning deploy- ments,” March 29 2022. Microsoft Blog

2022
[35]

Embracing disruption: Applying machine learning to chaos engineering,

P. Pushp, “Embracing disruption: Applying machine learning to chaos engineering,” August 18 2023. Published on LinkedIn

2023
[36]

An empirical investigation of the acceptance of chaos engineering,

L. Petersson, “An empirical investigation of the acceptance of chaos engineering,” 2022

2022
[37]

Harnessing chaos: The role of chaos engineering in cloud applications and impacts on site reliability engineering,

R. Yadav, “Harnessing chaos: The role of chaos engineering in cloud applications and impacts on site reliability engineering,” International Journal of Computer Trends and Technology , vol. 72, pp. 25–30, June 2024

2024
[38]

A chaos engineering system for live analysis and falsification of exception- handling in the jvm,

L. Zhang, B. Morin, P. Haller, B. Baudry, and M. Monperrus, “A chaos engineering system for live analysis and falsification of exception- handling in the jvm,” IEEE Transactions on Software Engineering , vol. 47, no. 11, pp. 2534–2548, 2019

2019
[39]

The design science paradigm as a frame for empirical software engineering,

P. Runeson, E. Engstr ¨om, and M.-A. Storey, “The design science paradigm as a frame for empirical software engineering,” Contemporary empirical methods in software engineering , pp. 127–147, 2020

2020
[40]

How software engineering research aligns with design science: a review,

E. Engstr ¨om, M.-A. Storey, P. Runeson, M. H¨ost, and M. T. Baldassarre, “How software engineering research aligns with design science: a review,”Empirical Software Engineering, vol. 25, pp. 2630–2660, 2020

2020
[41]

Managing risk in software process improvement: an action research approach,

J. H. Iversen, L. Mathiassen, and P. A. Nielsen, “Managing risk in software process improvement: an action research approach,” Mis Quarterly, pp. 395–433, 2004

2004
[42]

Data quality certification using iso/iec 25012: Industrial experiences,

F. Gualo, M. Rodr ´ıguez, J. Verdugo, I. Caballero, and M. Piattini, “Data quality certification using iso/iec 25012: Industrial experiences,” Journal of Systems and Software , vol. 176, p. 110938, 2021

2021
[43]

Action research as research methodology in software engineering,

M. Staron and M. Staron, “Action research as research methodology in software engineering,” Action Research in Software Engineering: Theory and Applications, pp. 15–36, 2020

2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Copilot

Microsoft, “Copilot.” https://copilot.cloud.microsoft/?auth=2, November 2024

work page 2024

[2] [2]

Google AI for Developers

Google, “Google AI for Developers.” https://ai.google.dev/, November 2024

work page 2024

[3] [3]

ChatGPT

OpenAI, “ChatGPT.” https://chatgpt.com/?ref=dotcom, November 2024

work page 2024

[4] [4]

Attention is all you need,

A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

2017

[5] [5]

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

2020

[6] [6]

WebGPT: Browser- assisted question-answering with human feedback,

R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V . Kosaraju, W. Saunders, et al. , “WebGPT: Browser- assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332, 2021

arXiv 2021

[7] [7]

Examining zero-shot vulnerability repair with large language models,

H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in 2023 IEEE Symposium on Security and Privacy (SP) , pp. 2339–2356, IEEE, 2023

work page 2023

[8] [8]

A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,

Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, et al. , “A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023 , 2023

arXiv 2023