{"id":"9eac9ca4-21d5-4856-ab8b-b3408bcdc642","arxiv_id":"2608.06790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An HTTP-layer fault injection framework shows that LLM API faults can cut agent task success by up to 50 percentage points, and current diagnosis methods stay below 56% accuracy.","lead":"This paper introduces AgentChaos, a tool that deliberately breaks LLM API responses at the HTTP layer to see how AI agent systems cope. It finds that all tested agent systems lose up to half their task success rate under such faults, and that system design matters more than the underlying model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Crash faults are not injected as crashes: Error/Timeout modify the 200-OK response body into error-like text, so measured 'crash' robustness tests text handling, not real HTTP 5xx/timeout behavior.","rationale":"The reader's weakest assumption concerns the fidelity of the Google ADK reimplementations. That is a genuine external-validity threat, but the crash-fault implementation issue I identify is more load-bearing because it is an internal inconsistency: the injected 'crash' faults are not crash faults at the transport layer. Even if the original agent systems were used unchanged, Error and Timeout as described in Section 4.3 would still be delivered as valid HTTP responses with modified content. This undermines the interpretation of the crash-fault results, the crash-versus-omission contrast in Section 6.1, and any practical guidance about retry handling. It does not necessarily invalidate the broad finding that all systems degrade under fault injection, since omission and value faults are implemented faithfully, and the ranking across systems may survive; therefore I do not move the verdict from CONDITIONAL. The correct verdict remains conditional, pending verification of the injection semantics, so the reader's verdict is unchanged. The concrete status-code/exception probe would settle the concern cheaply and would also clarify whether the quantitative headline needs adjustment.","tokens_in":25468,"tokens_out":5547,"duration_ms":59060,"concrete_test":"Run the released AgentChaos wrapper on one task under Error-content and one under Timeout-content while logging the response object exactly as the agent system receives it, including status_code, headers, and whether the underlying HTTP client raises an exception or enters a retry path. If both cases show status 200 with no exception/retry, recompute Table 3's Δpass@1 after excluding Error and Timeout configurations (or re-implement them as actual 5xx responses and connection timeouts and rerun a representative subset). If the maximum drop and the system ranking survive, the central claim stands; if they change materially, the crash-fault portion of the claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The taxonomy in Section 3 defines Error as 'the LLM API returns an error message' and Timeout as 'the LLM API does not respond within the allowed time.' Section 4.3 implements these by replacing choices[0].message.content with an error string like 'HTTP 500 Internal Server Error' or a timeout notification after the real provider's complete response has been received; no change to the HTTP status code or transport behavior is described. The agent's HTTP client therefore sees a successful 200 response whose body contains error-like text. This is a value/omission fault in disguise, not a crash fault. It is load-bearing because Section 6.1 and Table 7 rely on the crash-versus-omission distinction: crash faults are said to 'trigger errors and automatic retries,' while omission faults 'bypass error handling.' Under the described implementation, Error and Timeout also bypass real HTTP error handling and retry paths. The headline maximum Δpass@1 and the per-fault-type and per-configuration conclusions include these configurations, so the paper's quantitative claims about robustness to real LLM API crashes are not supported by the reported measurements.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AgentChaos, a framework for runtime, non-intrusive fault injection into the LLM API response layer of agent systems. It defines a taxonomy of crash, omission, and value faults on content and tool-call fields, combines these with injection strategies and compound scenarios into 65 configurations, and evaluates five agent systems (reimplemented on Google ADK) across seven benchmarks and four backbone LLMs. The main reported findings are that all systems degrade under fault injection (up to about 50 percentage points of pass@1), that the robustness ranking is consistent across backbone LLMs, that injection position and strategy strongly affect outcome, and that rule-based and LLM-based fault diagnosis methods stay below 56% accuracy. The paper also proposes trigger verification to filter tasks where the configured fault did not actually fire.","tokens_in":25712,"tokens_out":7285,"duration_ms":72705,"significance":"The core idea is well motivated and the shared HTTP-layer interception point is a clean, non-intrusive design that could be practically useful for agent-system reliability testing. The paper also contributes a systematic taxonomy grounded in dependability theory, a reproducible injection policy, trigger verification, and an evaluation across multiple architectures, benchmarks, and models. The artifact links and the explicit treatment of some threats to validity are strengths. However, the current submission has a load-bearing mismatch between the crash-fault taxonomy and its implementation, and the statistical basis for several quantitative conclusions is thin. If the crash-fault implementation is corrected or re-labeled, and the quantitative claims are appropriately qualified, the framework and findings would be a solid contribution to the community.","major_comments":[{"comment":"The Error and Timeout crash faults are not implemented as transport-level or HTTP-status-level faults. The wrapper forwards the request to the real provider, receives a complete successful response, and then replaces message.content with error-like or timeout-like text (or clears tool_calls). No change to the HTTP status code, no connection drop, and no transport delay are described for these two fault types. Therefore the 'crash' rows in Table 7, and the §6.1 distinction between crash faults ('trigger errors and automatic retries') and omission faults ('bypass error handling'), are not supported by the described implementation: under the reported injection, Error and Timeout also return a 200-OK response with modified text and also bypass real HTTP-level error handling and retry paths. This affects the per-fault-type conclusions and the 'most severe faults are not the most harmful' finding. The authors should either implement genuine HTTP status-code errors and transport-level timeouts, or re-label these injections as content-level simulations and remove or substantially qualify the crash-fault claims.","section":"§4.3; §6.1; Table 7"},{"comment":"The statistical support for the quantitative claims is weak. The 65 fault configurations are distributed over 300 examples, yielding approximately 4.6 triggered tasks per configuration, as the paper itself notes in §6.2. Table 3 contains multiple negative Δpass@1 cells attributed to noise (e.g., AutoGen -2.24% on MMLU-Pro under DeepSeek-V3.2 and Mini-SE -2.71% on SWE-bench Pro under GPT-5.2), yet no confidence intervals, no per-configuration sample sizes, and no numerical results for the claimed three-repeat stability are reported. The statement in §5.2 that 'the robustness ranking stays similar' is therefore not backed by any visible supporting data. Since RQ2's per-configuration comparisons (Table 7) are each based on a handful of tasks, conclusions about which fault types are most harmful are fragile. Please report per-configuration N, confidence intervals or the full three-run results, and treat single-configuration cells as indicative rather than definitive.","section":"§5.1; §6.2; Table 3"},{"comment":"The central conclusion that 'robustness depends on system implementation rather than model capability' rests on comparisons among five systems that are all reimplementations on Google ADK, with only one reimplementation per architectural pattern. As §6.2 admits, behavioral differences from the original implementations may affect absolute pass@1 values, and one system per pattern cannot separate the effect of the pattern from the effect of the specific system. Because the abstract and §6.1 state the conclusion strongly, this limitation should be moved from a threat to a boundary condition of the claims. The authors should either validate the ranking on at least one original implementation, or explicitly restrict the conclusion to the tested reimplementations and patterns.","section":"§5.1; §6.2"},{"comment":"Trigger verification excludes untriggered tasks, and for AutoGen only 48.30% of tasks overall (12.41% for tool_calls) are retained in the triggered set. The retained subset is likely to consist of the longer call chains, so the conditional Δpass@1 for AutoGen may not represent the behavior of a typical AutoGen run, and cross-system comparisons in §5.2 may partially reflect this selection difference rather than pure robustness differences. The paper acknowledges the low trigger rate but does not quantify the selection bias. A sensitivity analysis, or a comparison on matched task lengths, would be needed to support the cross-system ranking.","section":"§4.4; Table 5; §5.2"}],"minor_comments":[{"comment":"Several labels in Figure 3 misspell words: 'tiggered', 'untiggered', 'w/faut injection', and 'w/ofaut injection' should read 'triggered', 'untriggered', 'w/fault injection', and 'w/o fault injection'.","section":"Figure 3"},{"comment":"In the AutoGen row under Seed-1.8, the entry '20.098.63' appears to be a formatting error: the delta value 20.09 and the next w/o value 98.63 are missing a separator and should be displayed as distinct columns.","section":"Table 3"},{"comment":"Several taxonomically valid fault-type and target-field combinations (Timeout Tool call, Empty Tool call, Corrupt Tool call) are absent from Table 7 without explanation; the paper should state whether these were excluded due to low trigger counts or omitted for space, and if they were measured, the values should be reported.","section":"Table 7"},{"comment":"The phrase 'which is evaluated independently per call for reproducibility' is ambiguous: independent random decisions are not reproducible unless a seeded random generator is used. Please state explicitly whether a fixed seed or a deterministic policy is used for the intermittent strategy.","section":"§4.3"},{"comment":"The abstract and introduction say pass@1 drops by 'up to 50 percentage points', while Table 3 reports a maximum of 49.66%; using the exact value consistently would avoid the appearance of rounding up.","section":"Abstract; §5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central framework is promising, but the crash-fault implementation mismatch is a substantive correctness issue that the authors need to address head-on, not merely as a threat to validity. The small per-configuration sample sizes and the ADK reimplementation issue are also important for the paper's headline claims. These concerns are fixable in a revision, so I would not reject; a major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the field-level HTTP injection is a real new capability, and the paper deserves a serious referee. But the crash faults are not actually injected as crashes—Error and Timeout overwrite the body of an otherwise successful 200 response—so the headline crash-versus-omission contrast doesn't hold in these measurements.\n\nWhat's genuinely new: a runtime wrapper that intercepts the HTTP client and modifies content/tool_calls without touching source. That's not in AgenTracer, AutoInject, MAS-FIRE, or Rainmaker. The 65-configuration map, the trigger-verification filter, and the diagnosis baseline are all useful. They also ship artifacts, and the evaluation spans five architectures, four backbone LLMs, and seven benchmarks. That's broad and mostly careful.\n\nThe main soft spot is the crash-fault implementation, and it's load-bearing. Section 4.3 says Error replaces content with 'HTTP 500 Internal Server Error' and Timeout replaces content with a timeout notification. The agent's HTTP client still sees a 200. Real retry and error-handling paths never fire. Section 6.1 then claims crash faults 'trigger errors and automatic retries' while omission faults bypass error handling—the measurements can't support that. They show how agents handle error-looking text in the content field, which is much closer to a value fault. The fix is straightforward: inject real HTTP status codes and timeouts, or relabel these configurations and soften the crash claims.\n\nThe other issues are real but acknowledged: about 4.6 triggered tasks per configuration, no error bars, negative deltas in some cells; one reimplementation per architecture; and AutoGen loses more than half its tasks to trigger filtering. These cap the precision of the quantitative claims, but the authors put them in the threats section and don't oversell the ranking.\n\nThis is for agent-reliability researchers and framework designers who want a cheap, non-intrusive injection point. I'd read it as conditional: the framework and taxonomy stand, the crash-vs-omission finding does not as reported. If I cite it, it would be for the field-level injection technique, not the crash result.","headline":"Field-level HTTP injection is a real contribution; the crash faults aren't real crashes in the measurements, so the crash-vs-omission story doesn't hold as written.","tokens_in":26268,"tokens_out":3293,"would_cite":true,"duration_ms":31405,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentChaos injects API faults into running agents and finds every tested system loses accuracy, up to 50 points.","keywords":["agent systems","fault injection","chaos engineering","LLM API reliability","fault taxonomy","robustness evaluation","fault diagnosis"],"falsifier":"A direct check would run original, unmodified versions of AutoGen, MAD, MapCoder, EvoMAC, and Mini-SE through the same 65 fault configurations using their native implementation code, and compare the resulting Δpass@1 and robustness ranking against the ADK reimplementations reported here. If the ranking or the magnitude of drops changes materially on the original implementations, the claim that robustness depends on architecture rather than implementation would be undermined.","tokens_in":25299,"feed_emoji":"🛠️","tokens_out":3547,"duration_ms":27032,"temperature":0.7,"pith_summary":"This paper proposes AgentChaos, a chaos engineering framework for systematically testing how agent systems built on LLMs handle faulty API responses, and uses it to measure robustness across architectures, benchmarks, and models. The central claim is that every agent system degrades under fault injection—pass@1 drops by up to 50 percentage points—and that the ranking of which systems are most fragile stays consistent across backbone LLMs, meaning robustness is a property of system design rather than model strength. The paper also shows that the most harmful faults, such as truncated or corrupted responses, look like normal output and are hardest to diagnose, with existing diagnosis methods topping out below 56% accuracy. A sympathetic reader would care because the result identifies a practical gap: agent developers who invest in stronger models may not be fixing the real weakness, which is how the system handles response-level failures. The paper also contributes a reusable injection mechanism at the HTTP layer that requires no source code modification, and a 65-configuration fault space with trigger verification so measurements reflect only actually triggered faults.","feed_headline":"Injected API faults drop agent accuracy by up to 50 points","feed_subtitle":"All five agent systems degrade under runtime faults, and the ranking holds across every model tested.","key_machinery":"The key machinery is an HTTP-layer fault injection wrapper that sits between any agent system and the LLM API: it monkey-patches the HTTP client at runtime, intercepts each response to the Chat Completions endpoint, decides via an injection policy whether to fire, and deterministically modifies fields inside `choices[0].message` according to a fault taxonomy of crash (error, timeout), omission (empty, truncate), and value (corrupt, schema) faults on `content` and `tool_calls`. Because all agent systems talk to LLMs through this shared HTTP interface, the wrapper is non-intrusive and system-agnostic. A separate trigger verification pass inspects execution traces after each task and only counts tasks where the configured fault actually fired, preventing dilution of the measured degradation by untriggered tasks.","core_discovery":"AgentChaos establishes that all agent systems evaluated—AutoGen (conversation), MAD (debate), MapCoder (pipeline), EvoMAC (evolutionary), and Mini-SE (single-agent)—lose task-solving accuracy under injected LLM API faults, with Δpass@1 ranging from 0.87% to 49.66%. The drop is strikingly consistent across four backbone LLMs (Claude-Sonnet-4.5, GPT-5.2, DeepSeek-V3.2, Seed-1.8), with MapCoder always suffering the largest degradation, suggesting that fault handling is a property of system architecture, not model capability. The paper also finds a separation between fault severity and harm: omissions such as truncation cause drops comparable to crash faults but are nearly invisible to diagnosis, with rule-based type accuracy as low as 4.3% on truncation. Fault impact depends on configuration; persistent injection is worst (up to 62.39% drop), pipeline systems are most sensitive to injection position (up to 83.87% drop when the first call fails), and content-field faults hurt more than tool-call faults.","pith_inferences":["The consistency of the robustness ranking across models implies that model-specific response characteristics are not the dominant factor in fault propagation, and this same measurement could be applied to newer architectures like RAG-based or retrieval-augmented agents to test whether the pattern generalizes beyond the five evaluated patterns.","A practical implication not fully drawn by the paper: the differential harm between content and tool-call faults suggests that agents should validate tool-call JSON schema more aggressively than they validate free-form text, since tool-call faults are usually caught but content faults silently corrupt downstream reasoning.","The trigger-verification design implies that future work on agent robustness should always pair fault injection with a check of whether the intended injection actually occurred, otherwise robustness numbers can be inflated by dilution from untriggered runs.","An obvious testable extension is to use AgentChaos as a screening tool before deployment: inject a persistent truncation or empty fault against a candidate agent system and use the resulting Δpass@1 to decide whether the system needs output-validation defenses before rollout."],"forward_implications":["If the central claim holds, swapping in a more capable backbone LLM will not fix robustness problems caused by faulty API responses; developers must instead harden the system's handling of those responses.","Pipeline-style agent architectures, which feed each stage's output into the next, should add output validation and recovery at every stage boundary, because a single early fault can propagate through the whole chain.","Frameworks should log structured response metadata, such as token usage, `finish_reason`, and response length, so that truncation and other omission faults become detectable during post-hoc analysis.","Persistent and burst fault patterns are the most damaging injection strategies, so production systems should budget for, and practice recovering from, sustained API failure rather than only transient glitches.","Fault diagnosis should route cases by fault category: rule-based matching for crash and empty faults, LLM-based inference for truncate faults, since the two approaches are complementary rather than interchangeable."],"supporting_citations":[{"why":"Supplies the classical crash/omission/value fault classification from dependability theory that the paper adapts to LLM API responses.","marker":"[4]"},{"why":"AutoGen, the conversation-pattern agent system reimplemented and evaluated under fault injection.","marker":"[46]"},{"why":"MapCoder, the pipeline-pattern system that shows the largest robustness degradation in the study.","marker":"[22]"},{"why":"MAD, the debate-pattern system whose moderator fault propagation is analyzed as a key result.","marker":"[24]"},{"why":"EvoMAC, the evolutionary pattern system used to show that iterative refinement recovers from earlier faults.","marker":"[19]"},{"why":"Mini-SE, the single-agent system used for the fault-diagnosis evaluation on SWE-bench Pro.","marker":"[47]"},{"why":"The LLM-based fault attribution method that AgentChaos benchmarks its diagnosis results against.","marker":"[55]"},{"why":"EvalPlus (HumanEval+ and MBPP+), the stricter benchmark extensions that produced the largest measured drops.","marker":"[26]"},{"why":"Empirical study of bugs in AI coding tools, cited as evidence that truncation, timeouts and API errors are common in production.","marker":"[54]"},{"why":"Microsoft content streaming documentation, used to ground the Empty fault type in real safety-filter behavior.","marker":"[28]"}],"fun_headline_variants":["AgentChaos injects API faults, cutting agent accuracy by up to 50 percent","Agent fault handling is architecture, not model capability","API fault injection reveals system architecture determines agent robustness","Fault injection drops agent accuracy by up to 50 points regardless of model","Diagnosis of API faults under 56% accuracy leaves agents vulnerable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The five agent systems were reimplemented on the Google ADK framework rather than run as the original released code, and the paper assumes these reimplementations preserve each original system's interaction logic closely enough that the measured Δpass@1 values reflect the real systems. The paper itself flags this in Section 6.2, noting that behavioral differences may affect absolute pass@1 and that only one system per architectural pattern was tested.","fun_headline_variants_meta":{"raw":{"variants":["AgentChaos injects API faults, cutting agent accuracy by up to 50 percent","Agent fault handling is architecture, not model capability","API fault injection reveals system architecture determines agent robustness","Fault injection drops agent accuracy by up to 50 points regardless of model","Diagnosis of API faults under 56% accuracy leaves agents vulnerable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3783,"prompt_tokens":1028,"completion_tokens":2755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":2666}},"tokens_in":644,"tokens_out":2755,"duration_ms":19819,"temperature":1.0,"reasoning_tokens":2666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:47:59.873051+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check would run original, unmodified versions of AutoGen, MAD, MapCoder, EvoMAC, and Mini-SE through the same 65 fault configurations using their native implementation code, and compare the resulting Δpass@1 and robustness ranking against the ADK reimplementations reported here. If the ranking or the magnitude of drops changes materially on the original implementations, the claim that robustness depends on architecture rather than implementation would be undermined.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mini-SE, the single-agent system used for the fault-diagnosis evaluation on SWE-bench Pro."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The LLM-based fault attribution method that AgentChaos benchmarks its diagnosis results against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EvalPlus (HumanEval+ and MBPP+), the stricter benchmark extensions that produced the largest measured drops."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Microsoft content streaming documentation, used to ground the Empty fault type in real safety-filter behavior."}],"review_version":1}