Pith. sign in

REVIEW 4 major objections 5 minor 62 references

AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AgentChaos injects API faults into running agents and finds every tested system loses accuracy, up to 50 points.

desk verdict Field-level HTTP injection is a real contribution; the crash faults aren't real crashes in the measurements, so the crash-vs-omission story doesn't hold as written. read the letter →

arxiv 2608.06790 v1 pith:YFYX72JN submitted 2026-08-07 cs.SE

Gou Tan , Zhensu Sun , Jieke Shi , Ting Zhang , Zilong He , Qingfu Wu , Shuai Liang , Weifeng Sun
show 5 more authors
Junda He Pengfei Chen Chuanfu Zhang Lwin Khin Shar David Lo
This is my paper · ORCID
classification cs.SE
keywords agentsystemsfaultinjectionchaosengineeringLLMAPIreliabilitytaxonomyrobustnessevaluationdiagnosis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes AgentChaos, a chaos engineering framework for systematically testing how agent systems built on LLMs handle faulty API responses, and uses it to measure robustness across architectures, benchmarks, and models. The central claim is that every agent system degrades under fault injection—pass@1 drops by up to 50 percentage points—and that the ranking of which systems are most fragile stays consistent across backbone LLMs, meaning robustness is a property of system design rather than model strength. The paper also shows that the most harmful faults, such as truncated or corrupted responses, look like normal output and are hardest to diagnose, with existing diagnosis methods topping out below 56% accuracy. A sympathetic reader would care because the result identifies a practical gap: agent developers who invest in stronger models may not be fixing the real weakness, which is how the system handles response-level failures. The paper also contributes a reusable injection mechanism at the HTTP layer that requires no source code modification, and a 65-configuration fault space with trigger verification so measurements reflect only actually triggered faults.

What carries the argument

The key machinery is an HTTP-layer fault injection wrapper that sits between any agent system and the LLM API: it monkey-patches the HTTP client at runtime, intercepts each response to the Chat Completions endpoint, decides via an injection policy whether to fire, and deterministically modifies fields inside `choices[0].message` according to a fault taxonomy of crash (error, timeout), omission (empty, truncate), and value (corrupt, schema) faults on `content` and `tool_calls`. Because all agent systems talk to LLMs through this shared HTTP interface, the wrapper is non-intrusive and system-agnostic. A separate trigger verification pass inspects execution traces after each task and only counts tasks where the configured fault actually fired, preventing dilution of the measured degradation by untriggered tasks.

What would settle it

A direct check would run original, unmodified versions of AutoGen, MAD, MapCoder, EvoMAC, and Mini-SE through the same 65 fault configurations using their native implementation code, and compare the resulting Δpass@1 and robustness ranking against the ADK reimplementations reported here. If the ranking or the magnitude of drops changes materially on the original implementations, the claim that robustness depends on architecture rather than implementation would be undermined.

Watch

Extended reading notes

Core claim

AgentChaos establishes that all agent systems evaluated—AutoGen (conversation), MAD (debate), MapCoder (pipeline), EvoMAC (evolutionary), and Mini-SE (single-agent)—lose task-solving accuracy under injected LLM API faults, with Δpass@1 ranging from 0.87% to 49.66%. The drop is strikingly consistent across four backbone LLMs (Claude-Sonnet-4.5, GPT-5.2, DeepSeek-V3.2, Seed-1.8), with MapCoder always suffering the largest degradation, suggesting that fault handling is a property of system architecture, not model capability. The paper also finds a separation between fault severity and harm: omissions such as truncation cause drops comparable to crash faults but are nearly invisible to diagnosis, with rule-based type accuracy as low as 4.3% on truncation. Fault impact depends on configuration; persistent injection is worst (up to 62.39% drop), pipeline systems are most sensitive to injection position (up to 83.87% drop when the first call fails), and content-field faults hurt more than tool-call faults.

Load-bearing premise

The five agent systems were reimplemented on the Google ADK framework rather than run as the original released code, and the paper assumes these reimplementations preserve each original system's interaction logic closely enough that the measured Δpass@1 values reflect the real systems. The paper itself flags this in Section 6.2, noting that behavioral differences may affect absolute pass@1 and that only one system per architectural pattern was tested.

Editorial extensions

If this is right

  • If the central claim holds, swapping in a more capable backbone LLM will not fix robustness problems caused by faulty API responses; developers must instead harden the system's handling of those responses.
  • Pipeline-style agent architectures, which feed each stage's output into the next, should add output validation and recovery at every stage boundary, because a single early fault can propagate through the whole chain.
  • Frameworks should log structured response metadata, such as token usage, `finish_reason`, and response length, so that truncation and other omission faults become detectable during post-hoc analysis.
  • Persistent and burst fault patterns are the most damaging injection strategies, so production systems should budget for, and practice recovering from, sustained API failure rather than only transient glitches.
  • Fault diagnosis should route cases by fault category: rule-based matching for crash and empty faults, LLM-based inference for truncate faults, since the two approaches are complementary rather than interchangeable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The consistency of the robustness ranking across models implies that model-specific response characteristics are not the dominant factor in fault propagation, and this same measurement could be applied to newer architectures like RAG-based or retrieval-augmented agents to test whether the pattern generalizes beyond the five evaluated patterns.
  • A practical implication not fully drawn by the paper: the differential harm between content and tool-call faults suggests that agents should validate tool-call JSON schema more aggressively than they validate free-form text, since tool-call faults are usually caught but content faults silently corrupt downstream reasoning.
  • The trigger-verification design implies that future work on agent robustness should always pair fault injection with a check of whether the intended injection actually occurred, otherwise robustness numbers can be inflated by dilution from untriggered runs.
  • An obvious testable extension is to use AgentChaos as a screening tool before deployment: inject a persistent truncation or empty fault against a candidate agent system and use the resulting Δpass@1 to decide whether the system needs output-validation defenses before rollout.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents AgentChaos, a framework for runtime, non-intrusive fault injection into the LLM API response layer of agent systems. It defines a taxonomy of crash, omission, and value faults on content and tool-call fields, combines these with injection strategies and compound scenarios into 65 configurations, and evaluates five agent systems (reimplemented on Google ADK) across seven benchmarks and four backbone LLMs. The main reported findings are that all systems degrade under fault injection (up to about 50 percentage points of pass@1), that the robustness ranking is consistent across backbone LLMs, that injection position and strategy strongly affect outcome, and that rule-based and LLM-based fault diagnosis methods stay below 56% accuracy. The paper also proposes trigger verification to filter tasks where the configured fault did not actually fire.

Significance. The core idea is well motivated and the shared HTTP-layer interception point is a clean, non-intrusive design that could be practically useful for agent-system reliability testing. The paper also contributes a systematic taxonomy grounded in dependability theory, a reproducible injection policy, trigger verification, and an evaluation across multiple architectures, benchmarks, and models. The artifact links and the explicit treatment of some threats to validity are strengths. However, the current submission has a load-bearing mismatch between the crash-fault taxonomy and its implementation, and the statistical basis for several quantitative conclusions is thin. If the crash-fault implementation is corrected or re-labeled, and the quantitative claims are appropriately qualified, the framework and findings would be a solid contribution to the community.

major comments (4)
  1. [§4.3; §6.1; Table 7] The Error and Timeout crash faults are not implemented as transport-level or HTTP-status-level faults. The wrapper forwards the request to the real provider, receives a complete successful response, and then replaces message.content with error-like or timeout-like text (or clears tool_calls). No change to the HTTP status code, no connection drop, and no transport delay are described for these two fault types. Therefore the 'crash' rows in Table 7, and the §6.1 distinction between crash faults ('trigger errors and automatic retries') and omission faults ('bypass error handling'), are not supported by the described implementation: under the reported injection, Error and Timeout also return a 200-OK response with modified text and also bypass real HTTP-level error handling and retry paths. This affects the per-fault-type conclusions and the 'most severe faults are not the most harmful' finding. The authors should either implement genuine HTTP status-code errors and transport-level timeouts, or re-label these injections as content-level simulations and remove or substantially qualify the crash-fault claims.
  2. [§5.1; §6.2; Table 3] The statistical support for the quantitative claims is weak. The 65 fault configurations are distributed over 300 examples, yielding approximately 4.6 triggered tasks per configuration, as the paper itself notes in §6.2. Table 3 contains multiple negative Δpass@1 cells attributed to noise (e.g., AutoGen -2.24% on MMLU-Pro under DeepSeek-V3.2 and Mini-SE -2.71% on SWE-bench Pro under GPT-5.2), yet no confidence intervals, no per-configuration sample sizes, and no numerical results for the claimed three-repeat stability are reported. The statement in §5.2 that 'the robustness ranking stays similar' is therefore not backed by any visible supporting data. Since RQ2's per-configuration comparisons (Table 7) are each based on a handful of tasks, conclusions about which fault types are most harmful are fragile. Please report per-configuration N, confidence intervals or the full three-run results, and treat single-configuration cells as indicative rather than definitive.
  3. [§5.1; §6.2] The central conclusion that 'robustness depends on system implementation rather than model capability' rests on comparisons among five systems that are all reimplementations on Google ADK, with only one reimplementation per architectural pattern. As §6.2 admits, behavioral differences from the original implementations may affect absolute pass@1 values, and one system per pattern cannot separate the effect of the pattern from the effect of the specific system. Because the abstract and §6.1 state the conclusion strongly, this limitation should be moved from a threat to a boundary condition of the claims. The authors should either validate the ranking on at least one original implementation, or explicitly restrict the conclusion to the tested reimplementations and patterns.
  4. [§4.4; Table 5; §5.2] Trigger verification excludes untriggered tasks, and for AutoGen only 48.30% of tasks overall (12.41% for tool_calls) are retained in the triggered set. The retained subset is likely to consist of the longer call chains, so the conditional Δpass@1 for AutoGen may not represent the behavior of a typical AutoGen run, and cross-system comparisons in §5.2 may partially reflect this selection difference rather than pure robustness differences. The paper acknowledges the low trigger rate but does not quantify the selection bias. A sensitivity analysis, or a comparison on matched task lengths, would be needed to support the cross-system ranking.
minor comments (5)
  1. [Figure 3] Several labels in Figure 3 misspell words: 'tiggered', 'untiggered', 'w/faut injection', and 'w/ofaut injection' should read 'triggered', 'untriggered', 'w/fault injection', and 'w/o fault injection'.
  2. [Table 3] In the AutoGen row under Seed-1.8, the entry '20.098.63' appears to be a formatting error: the delta value 20.09 and the next w/o value 98.63 are missing a separator and should be displayed as distinct columns.
  3. [Table 7] Several taxonomically valid fault-type and target-field combinations (Timeout Tool call, Empty Tool call, Corrupt Tool call) are absent from Table 7 without explanation; the paper should state whether these were excluded due to low trigger counts or omitted for space, and if they were measured, the values should be reported.
  4. [§4.3] The phrase 'which is evaluated independently per call for reproducibility' is ambiguous: independent random decisions are not reproducible unless a seeded random generator is used. Please state explicitly whether a fixed seed or a deterministic policy is used for the intermittent strategy.
  5. [Abstract; §5.2] The abstract and introduction say pass@1 drops by 'up to 50 percentage points', while Table 3 reports a maximum of 49.66%; using the exact value consistently would avoid the appearance of rounding up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AgentChaos is an empirical fault-injection benchmark whose central claims are directly measured pass@1 differences, not derived predictions.

full rationale

The paper's central claims are empirical measurements: pass@1 with and without fault injection across systems, benchmarks, and backbone LLMs, plus accuracy of existing diagnosis methods. No parameter is fitted and no quantity is 'predicted' from an input; the robustness ranking is read off Table 3, not derived from the fault taxonomy. The fault taxonomy itself is explicitly adapted from the classical dependability classification of Avizienis et al. [4] and applied to the fixed response fields described in Section 2.1; the claim that the enumeration is complete is a definitional statement about the resulting configuration grid, not a circular derivation of the experimental conclusions. Trigger verification (Section 4.4) filters untriggered tasks before computing pass@1 with fault injection; this is an explicit, transparent metric definition and does not force the observed degradation values or their cross-system ordering. The diagnosis baseline (Section 5.4) is anchored to public API error documentation and states that patterns are 'written from public API error documentation rather than from the content we inject', which is the opposite of a circular fit. Self-citations appear in related work and background (e.g., references 38, 39, 51) but none carries the load-bearing claim that all systems degrade or that the ranking is model-consistent. The paper's own threats-to-validity section flags the ADK reimplementation and the one-system-per-pattern limitation (Section 6.2), which are honest scope caveats rather than circular reasoning. The skeptic concern that Error and Timeout faults are implemented by modifying the response body rather than the HTTP status or transport behavior is a construct-validity threat: the measured quantity remains an empirical pass@1 difference, and the paper's conclusions about robustness to the injected conditions do not reduce by definition to the injected strings. Accordingly, no load-bearing circular step is exhibited, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted to data; the framework's thresholds are hand-chosen defaults, though they set fault severity and so determine the observed drops. The central assumptions are that OpenAI-compatible HTTP interception captures all LLM traffic, that ADK reimplementations preserve the original systems' behavior, and that excluding untriggered tasks does not bias the comparison. No new theoretical entities are introduced.

free parameters (4)
  • intermittent injection probability = 0.3
    Default probability for intermittent strategy; sensitivity checked at 0.1, 0.3, 0.5 in Table 8 with stable results.
  • burst count = 3
    Number of consecutive fault injections for burst strategy; sensitivity checked at 1, 3, 5 in Table 8.
  • truncation ratio = 0.3 (keep first 30%)
    Truncate fault keeps first 30% of the target field; a hand-chosen threshold that sets fault severity.
  • corruption ratio = about 20% of characters
    Corrupt fault replaces about 20% of characters with random Unicode symbols; arbitrary but deterministic, affecting measured impact.
assumptions (5)
  • domain assumption All agent systems access LLMs through OpenAI-compatible HTTP APIs that can be intercepted by monkey-patching httpx.AsyncClient.
    Section 2.1 and 4.3: 'Most agent frameworks use OpenAI-compatible APIs' and the implementation uses httpx monkey patching. Non-Python or non-httpx clients or non-OpenAI formats are not directly supported.
  • domain assumption LLM API responses contain mutually exclusive content and tool_calls fields in the OpenAI Chat Completions format.
    Section 2.1, Figure 2: used to define target fields and modification functions.
  • domain assumption The reimplementations of AutoGen, MAD, MapCoder, EvoMAC, and Mini-SE on Google ADK preserve each system's original interaction logic.
    Section 5.1 and 6.2: authors reimplement all systems on ADK 'preserving each system's original interaction logic'; acknowledged as a threat to internal validity.
  • domain assumption Tasks without a recorded fault event can be safely excluded without biasing the robustness estimate.
    Section 4.4: untriggered tasks are filtered to measure 'true impact'; assumes triggered tasks are representative of fault impact.
  • domain assumption The fault taxonomy derived from dependability theory plus the fixed response fields is complete.
    Section 3: 'this enumeration covers all fault types rather than selecting them by hand'; completeness is asserted from enumeration over a fixed schema.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection." pith.science (2026). https://pith.science/paper/YFYX72JN

@misc{pith2026260806790,
  author       = {Pith},
  title        = {Pith review of: AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YFYX72JN}},
  note         = {Machine review of arXiv:2608.06790}
}
read the original abstract

Agent systems rely on LLM APIs for every response, but these APIs can return server errors, truncated responses, or corrupted content that propagates through downstream agents and causes task failure. Evaluating robustness under these faults is crucial for reliable deployment. Existing fault injection methods are offline, require source code modification, or cannot modify specific response fields. A comprehensive evaluation also requires a systematic fault taxonomy because different fault types affect downstream agents differently. We propose AgentChaos, a chaos engineering framework for controlled, runtime, non-intrusive LLM API fault injection. Since all agent systems access LLMs through the same HTTP interface, we inject faults at this shared layer without modifying source code. We define crash, omission, and value faults on content and tool call fields, intercept and modify LLM API responses at runtime, and verify whether each fault is triggered to filter untriggered tasks and avoid underestimating fault impact. Evaluations across agent systems, benchmarks, and backbone LLMs under 65 fault configurations show that all systems degrade under fault injection, with pass@1 dropping by up to 50 percentage points. The ranking is consistent across models, suggesting that robustness depends on system implementation rather than model capability. Existing fault diagnosis methods achieve below 53% accuracy on fault type and below 56% on fault step, leaving room for improvement. We further reveal practical findings for agent system developers.

Figures

Figures reproduced from arXiv: 2608.06790 by the authors.

Figure 1
Figure 1. A truncated LLM API response at the Planner pro [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. OpenAI Chat Completions API response. content and tool_calls are mutually exclusive. Inactive fields are gray. finish_reason: finish metadata [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of AgentChaos. The framework takes an agent system, tasks, and a fault configuration as input, injects [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Fault injection wrapper. It intercepts LLM API re [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 32 canonical work pages

  1. [1]

    Introducing claude sonnet 4.5

    2025. Introducing claude sonnet 4.5. https://www.anthropic.com/news/claude- sonnet-4-5. Accessed: 2026-08-03

    work page 2025

  2. [2]

    Anthropic. 2025. Claude Code GitHub Actions. https://code.claude.com/docs/en/ github-actions. Accessed: 2026-08-03

    work page 2025

  3. [3]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models.CoRR abs/2108.07732 (2021). arXiv:2108.07732 https://arxiv.org/abs/2108.07732

    arXiv 2021

  4. [4]

    Landwehr

    Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl E. Landwehr

  5. [5]

    Pan, Shuyi Yang, Lakshya A

    Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya G. Parameswaran, Dan Klein, Kannan Ramchandran, Matei A. Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail?. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Pro...

    work page 2025

  6. [6]

    ChaosBlade. 2026. Chaosblade: An Easy to Use and Powerful Chaos Engineering Toolkit. https://github.com/chaosblade-io/chaosblade. Accessed: 2026-08-03

    work page 2026

  7. [7]

    Hongyang Chen, Pengfei Chen, Guangba Yu, Xiaoyun Li, and Zilong He. 2024. MicroFI: Non-Intrusive and Prioritized Request-Level Fault Injection for Microser- vice Applications.IEEE Trans. Dependable Secur. Comput.21, 5 (2024), 4921–4938. doi:10.1109/TDSC.2024.3363902

    work page arXiv 2024

  8. [8]

    Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, and David Lo. 2025. SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios.CoRRabs/2509.22097 (2025). arXiv:2509.22097 doi:10.48550/ARXIV.2509.22097

    arXiv doi:10.48550/arxiv.2509.22097 2025

Show all 62 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  2. [10]

    Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. 2025. SWE-Exp: Experience- Driven Software Issue Resolution.CoRRabs/2507.23361 (2025). arXiv:2507.23361 doi:10.48550/ARXIV.2507.23361

    2025 doi

  3. [11]

    Yinfang Chen, Xudong Sun, Suman Nath, Ze Yang, and Tianyin Xu. 2023. Push- Button Reliability Testing for Cloud-Backed Applications with Rainmaker. In 20th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2023, Boston, MA, April 17-19, 2023, Mahesh Balakri...

    2023

  4. [12]

    DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, and et al

  5. [13]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 20...

    2025 doi arXiv

  6. [14]

    Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kan- nappan, and Rebecca Qian. 2025. TRAIL: Trace Reasoning and Agentic Issue Localization.CoRRabs/2505.08638 (2025). arXiv:2505.08638 doi:10.48550/ARXIV. 2505.08638

    2025 doi arXiv

  7. [15]

    Yu Gao, Wensheng Dou, Dong Wang, Wenhan Feng, Jun Wei, Hua Zhong, and Tao Huang. 2023. Coverage Guided Fault Injection for Cloud Systems. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2211–2223. doi:10...

    2023

  8. [16]

    Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang. 2025. Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysis.CoRRabs/2509.13782 (2025). arXiv:2509.13782 doi:10.48550/ARXIV. 2509.13782

    2025 doi

  9. [17]

    Google. 2026. Agent Development Kit (ADK). https://github.com/google/adk- python. Accessed: 2026-08-03

    2026

  10. [18]

    Junjie He, Shenao Wang, Yanjie Zhao, Xinyi Hou, Zhao Liu, Quanchen Zou, and Haoyu Wang. 2026. TaintP2X: Detecting Taint-Style Prompt-to-Anything Injection Vulnerabilities in LLM-Integrated Applications. (2026)

    2026

  11. [19]

    Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. 2025. Self-Evolving Multi-Agent Collaboration Net- works for Software Development. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapor...

    2025

  12. [20]

    Haojia Huang, Pengfei Chen, Guangba Yu, Haiyu Huang, Jia Chang, Jun Li, and Jian Han. 2025. Conan: Uncover Consensus Issues in Distributed Databases Using Fuzzing-Driven Fault Injection. InIEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2...

    2025

  13. [21]

    Lyu, and Maarten Sap

    Jen-tse Huang, Jiaxu Zhou, Tailin Jin, Xuhui Zhou, Zixi Chen, Wenxuan Wang, Youliang Yuan, Michael R. Lyu, and Maarten Sap. 2025. On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents. InForty-second In- ternational Conference on Machine Learning, ICML 20...

    2025

  14. [22]

    Ashraful Islam, Mohammed Eunus Ali, and Md

    Md. Ashraful Islam, Mohammed Eunus Ali, and Md. Rizwan Parvez. 2024. Map- Coder: Multi-Agent Code Generation for Competitive Problem Solving. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2024, Bangko...

    2024 doi

  15. [23]

    Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng. 2026. MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems. arXiv:2602.19843 [cs.SE] https://arxiv.org/abs/2602.19843

    2026

  16. [24]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. InProceedings of the 2024 Conference on Empirical Methods in Natural Languag...

    2024 doi

  17. [25]

    Fengyu Liu, Yuan Zhang, Jiaqi Luo, Jiarun Dai, Tian Chen, Letian Yuan, Zhengmin Yu, Youkun Shi, Ke Li, Chengyuan Zhou, Hao Chen, and Min Yang. 2025. Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM- based Agents. In34th USENIX Security Symposi...

    2025

  18. [26]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. InAdvances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Infor...

    2023

  19. [27]

    Minghua Ma, Jackson Clark, and Shenglin Zhang. 2025. AIOpsLab in Action: An Open Platform for AIOps Research. InProceedings of the 33rd ACM International AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection ASE ’26, October 12–16, 2026, Munich, Germ...

    2025

  20. [28]

    Microsoft. 2026. Content streaming. https://learn.microsoft.com/en-us/azure/ foundry/openai/concepts/content-streaming. Accessed: 2026-08-03

    2026

  21. [29]

    Yuqing Niu, Jieke Shi, Ruidong Han, Ye Liu, Chengyan Ma, Yunbo Lyu, and David Lo. 2026. What You Trust is Insecure: Demystifying How Developers (Mis)Use Trusted Execution Environments in Practice. InIEEE International Conference on Software Analysis, Evolution and Reengineerin...

    2026

  22. [30]

    OpenAI. 2025. ChatGPT. https://chatgpt.com. Accessed: 2026-08-03

    2025

  23. [31]

    OpenAI. 2025. Introducing ChatGPT agent: bridging research and action. https: //openai.com/index/introducing-chatgpt-agent/. Accessed: 2026-08-03

    2025

  24. [32]

    Bytedance Seed. 2025. Seed1.8 Model Card: Towards Generalized Real-World Agency. https://seed.bytedance.com/en/seed1_8. Accessed: 2026-08-03

    2025

  25. [33]

    Yuchen Shao, Yuheng Huang, Jiawei Shen, Lei Ma, Ting Su, and Chengcheng Wan

  26. [34]

    Yuchen Shao, Yuheng Huang, Jiazhen Zou, Yuling Shi, Long Yang, Lei Ma, Ting Su, and Chengcheng Wan. 2026. Comfrey: Mitigating Integration Failures in LLM-enabled Software at Run-Time. (2026)

    2026

  27. [35]

    Akshey Sigdel and Rista Baral. 2026. Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance.CoRR abs/2603.13404 (2026). arXiv:2603.13404 doi:10.48550/ARXIV.2603.13404

    2026 doi

  28. [36]

    In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025

    Are LLMs Correctly Integrated into Software Systems?. In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1178–1190. doi:10.1109/ICSE55347.2025.00204

    2025

  29. [37]

    Bissyandé, Andrey Boytsov, Ulrick Ble, and Anne Goujon

    Yewei Song, Xunzhu Tang, Cedric Lothritz, Saad Ezzini, Jacques Klein, Tegawendé F. Bissyandé, Andrey Boytsov, Ulrick Ble, and Anne Goujon. 2025. CallNavi, A challenge and empirical study on LLM function calling and routing. InProceedings of the 29th International Conference on...

    2025

  30. [38]

    Gou Tan, Zilong He, Min Li, Pengfei Chen, Jieke Shi, Zhensu Sun, Ting Zhang, Danwen Chen, Lwin Khin Shar, Chuanfu Zhang, and David Lo. 2026. LIDL: LLM Integration Defect Localization via Knowledge Graph-Enhanced Multi-Agent Analysis.CoRRabs/2601.05539 (2026). arXiv:2601.05539 ...

    2026 doi

  31. [39]

    Ron Solomon, Yarin Yerushalmi Levi, Lior Vaknin, Eran Aizikovich, Amit Baras, Etai Ohana, Amit Giloni, Shamik Bose, Chiara Picardi, Yuval Elovici, and Asaf Shabtai. 2025. LumiMAS: A Comprehensive Framework for Real-Time Monitoring and Enhanced Observability in Multi-Agent Syst...

    2025 doi

  32. [40]

    AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection

    Gou Tan, Zhensu Sun, Jieke Shi, Ting Zhang, Zilong He, Qingfu Wu, Shuai Liang, Weifeng Sun, Junda He, Pengfei Chen, Chuanfu Zhang, Lwin Khin Shar, and David Lo. 2026. Artifact of Paper “AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection”. doi:10.5...

    2026 doi

  33. [41]

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2025. Mixture-of-Agents Enhances Large Language Model Capabilities. InThe Thir- teenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openr...

    2025

  34. [42]

    Gou Tan, Zilong He, Min Li, Haiyu Huang, Yilun Wang, Pengfei Chen, Giuliano Casale, and Chuanfu Zhang. 2026. LLMRCA: Multilevel Root Cause Analysis for LLM Applications Using Multimodal Observability Data.ACM Trans. Softw. Eng. Methodol.(April 2026). Just Accepted. doi:10.1145/3806200

    2026 doi

  35. [43]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language ...

    2024

  36. [44]

    Yilun Wang, Guangba Yu, Haiyu Huang, Zirui Wang, Yujie Huang, Pengfei Chen, and Michael R. Lyu. 2026. Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems.CoRRabs/2603.00468 (2026). arXiv:2603.00468 doi:10.48550/ARXIV.2603.00468

    2026 doi

  37. [45]

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-Shepherd: Verify and Reinforce LLMs Step- by-step without Human Annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024 doi

  38. [46]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling Next- Gen LLM Applications via Multi-Agent Conversations. InFirst...

    2024

  39. [47]

    Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, and Pinjia He. 2025. Scalable Supervising Software Agents with Patch Reasoner.CoRR abs/2510.22775 (2025). arXiv:2510.22775 doi:10.48550/ARXIV.2510.22775

    2025 doi

  40. [48]

    Zirui Wang, Guangba Yu, and Michael R. Lyu. 2026. AI-NativeBench: An Open- Source White-Box Agentic Benchmark Suite for AI-Native Systems.CoRR abs/2601.09393 (2026). arXiv:2601.09393 doi:10.48550/ARXIV.2601.09393

    2026 doi

  41. [49]

    Zhou Yang, Chenyu Wang, Jieke Shi, Thong Hoang, Pavneet Singh Kochhar, Qinghua Lu, Zhenchang Xing, and David Lo. 2023. What Do Users Ask in Open- Source AI Repositories? An Empirical Study of GitHub Issues. In20th IEEE/ACM International Conference on Mining Software Repositori...

    2023

  42. [50]

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2026. A Survey on Evaluation of LLM- based Agents. InFindings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, Jul...

    2026 doi

  43. [51]

    Ziluo Xue, Yanjie Zhao, Shenao Wang, Kai Chen, and Haoyu Wang. 2025. A Char- acterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks. In40th IEEE/ACM International Conference on Automated Software Engineer- ing, ASE 2025, Seoul, Korea, Republic of, November 1...

    2025

  44. [52]

    Guangba Yu, Zirui Wang, Yujie Huang, Renyi Zhong, Yuedong Zhong, Yilun Wang, and Michael R. Lyu. 2026. Why Does the LLM Stop Computing: An Empir- ical Study of User-Reported Failures in Open-Source LLMs.CoRRabs/2601.13655 (2026). arXiv:2601.13655 doi:10.48550/ARXIV.2601.13655

    2026 doi

  45. [53]

    Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. 2025. AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?CoRRabs/2509.03312 (2025). arXiv:2509.03312 doi:10.48550/ARXIV. 2509.03312

    2025 doi arXiv

  46. [54]

    Guangba Yu, Gou Tan, Haojia Huang, Zhenyu Zhang, Pengfei Chen, Roberto Natella, Zibin Zheng, and Michael R. Lyu. 2026. A Survey on Failure Analysis and Fault Injection in AI Systems.ACM Trans. Softw. Eng. Methodol.35, 1 (2026), 28:1–28:42. doi:10.1145/3732777

    2026 doi

  47. [55]

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. 2025. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. InForty-second Internati...

    2025

  48. [56]

    Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Yuhui Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, and Daben Liu. 2026. RAFFLES: Reasoning- based Attribution of Faults for LLM Systems. InProceedings of the 19th Conference of the European Chapter of the Association for...

    2026 doi

  49. [57]

    Ruixin Zhang, Wuyang Dai, Hung Viet Pham, Gias Uddin, Jinqiu Yang, and Song Wang. 2026. Engineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLI. InProceedings of the 34th ACM International Conference on the Foundations of Softw...

    2026

  50. [58]

    Xinxue Zhu, Jiacong Wu, Xiaoyu Zhang, Tianlin Li, Yanzhou Mu, Juan Zhai, Chao Shen, Chunrong Fang, and Yang Liu. 2026. An Empirical Study of Bugs in Modern LLM Agent Frameworks.CoRRabs/2602.21806 (2026). arXiv:2602.21806 doi:10.48550/ARXIV.2602.21806 Received 2026-03-26; accep...

    2026 doi arXiv

  51. [60]

    Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You

  52. [61]

    arXiv:2509.25370 doi:10.48550/ARXIV.2509.25370

    Where LLM Agents Fail and How They can Learn From Failures.CoRR abs/2509.25370 (2025). arXiv:2509.25370 doi:10.48550/ARXIV.2509.25370

    2025 doi

  53. [2004]

    Dependable Secur

    Basic Concepts and Taxonomy of Dependable and Secure Computing.IEEE Trans. Dependable Secur. Comput.1, 1 (2004), 11–33. doi:10.1109/TDSC.2004.2

    2004 doi

  54. [2025]

    arXiv:2512.02556 [cs.CL] https://arxiv.org/abs/2512.02556

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 [cs.CL] https://arxiv.org/abs/2512.02556

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.