Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Targeted repair of agentic workflows works better when failures are first turned into checkable symbolic specs than when systems only optimize trajectories.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:23 UTC pith:6BVA6XAH

load-bearing objection Solid multi-platform repair system for agentic workflows; the diagnosis-driven story is useful but rests on unmeasured LLM-generated assertions. the 4 major comments →

arxiv 2607.02882 v1 pith:6BVA6XAH submitted 2026-07-03 cs.SE

Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inference

Xuyan Ma , Yawen Wang , Junjie Wang , Xiaofei Xie , Boyu Wu , Mingyang Li , Dandan Wang , Qing Wang This is my paper
classification cs.SE
keywords agentic workflowsymbolic inferencefailure diagnosisautomated repairroot cause analysisbehavioral specificationspre-execution assessment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agentic workflows on platforms like Dify, Coze, and n8n fail in ways that are hard to fix because LLM outputs are uncertain, nodes depend on one another, and tools are heterogeneous. Most existing repair and agent-enhancement methods only look at whole trajectories or final scores, so they change things without knowing which node failed or why. FlowFixer instead turns each failed run into a unified symbolic trace, has an LLM write executable behavioral assertions about existence, order, and cause for every node, checks those assertions against the actual outputs, and uses the violations to name the responsible node and a root-cause type. From that diagnosis it builds a small set of atomic edits, filters the bad candidates with a cheap four-way static check, and only then re-runs the survivors. On hundreds of real failures the method repairs 71.3 percent of cases and raises both attribution and root-cause accuracy over strong baselines, showing that diagnosis-first symbolic modeling can make low-code agent pipelines maintainable.

Core claim

When platform-orchestrated agentic workflows fail, converting their executions into symbolic traces, inferring executable node-level behavioral specifications along existence, temporal, and causal dimensions, and verifying those specifications yields diagnosis evidence that supports more accurate failure attribution and root-cause analysis and, in turn, higher repair success than trajectory-level optimization alone.

What carries the argument

Symbolic inference of node behavioral specifications: an LLM, given node context, task goal, and workflow topology, emits assertable constraints in a small DSL; static checking of those assertions supplies the structured evidence used for both diagnosis and pre-execution filtering of repair patches.

Load-bearing premise

The method assumes an LLM can write complete and correct executable behavioral assertions for every node from context and task description alone; if those specs systematically miss or invent constraints, the diagnosis evidence and the repairs built on it both collapse.

What would settle it

On a held-out set of the same platform failures, replace the inferred symbolic assertions with empty or random constraints and measure whether repair success rate and attribution accuracy fall to the level of the trajectory-only baselines; if they do not, the symbolic-diagnosis claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. FlowFixer is a diagnosis-driven automated repair framework for platform-orchestrated agentic workflows (Dify, Coze, n8n). It normalizes failed executions into a unified symbolic trace, uses an LLM to synthesize executable node-level behavioral assertions along existence, temporal, and causal dimensions (DSL in Table I), and verifies those assertions to produce evidence for failure attribution and root-cause analysis against a 16-type taxonomy. Root-cause-aware repair strategies then guide atomic edit patches (insert/remove/replace/append/swap), which are filtered by a four-dimensional pre-execution assessment before dynamic verification; an experience pool accumulates online feedback and historical repair knowledge. On AgentFail plus 136 n8n failures, FlowFixer reports 71.3% repair success rate (RSR), 84.4% failure attribution accuracy (FAA), and 87.9% root-cause accuracy (RCA), outperforming traditional APR, agent-enhancement, and attribution baselines (Table II), with ablations of symbolization, external knowledge, and the experience pool (Table III).

Significance. If the results hold under stronger validation of the symbolic layer, the paper offers a useful shift from trajectory-level agent optimization toward explicit, node-level diagnosis for low-code agentic workflows—an increasingly practical SE setting that traditional APR cannot address directly. Strengths include multi-platform evaluation, a reasonably broad baseline suite spanning APR and agent enhancement, systematic ablations, a concrete pre-execution filter with reported 99.7% precision / 84.6% recall, and illustrative end-to-end case studies. The combination of a lightweight assertion DSL, root-cause taxonomy, and atomic workflow edits is a clear engineering contribution for maintainability of heterogeneous node pipelines.

major comments (4)
  1. [Sec. III-B2, Fig. 2, Table III] Sec. III-B2 and Fig. 2: The central thesis—that symbolic inference yields reliable diagnosis that then drives targeted repair—depends on LLM-synthesized existence/temporal/causal assertions being sufficiently correct. The manuscript never reports assertion precision/recall, false-positive/false-negative rates, or any human audit of generated specs against ground-truth node contracts. Table III’s “w/o Symbol” ablation removes the entire symbolic pipeline and therefore cannot isolate assertion quality from the mere presence of a structured intermediate representation. Without a direct quality measure (or a controlled study showing that noisy specs still produce causal, not merely correlational, gains), the RSR/FAA/RCA improvements in Table II overstate the value of “diagnosis-driven” symbolic inference.
  2. [Sec. III-C1] Sec. III-C1 (Failure Attribution): The suspicious score is described only qualitatively as combining assertion-violation rate with structural/propagation position. No formula, weights, normalization, or ranking procedure is given, nor is sensitivity to those choices reported. Because FAA (84.4%) is a primary claim and feeds root-cause analysis and patch generation, the attribution mechanism must be specified precisely enough to be reproduced and stress-tested (e.g., violation-only vs. structure-only vs. combined).
  3. [Sec. IV-B, Table II, Fig. 4] Sec. IV-B and Table II: Results from AgentFail (Dify/Coze) and the authors’ 136 n8n cases are merged because “methods have similar results,” but no per-platform or per-dataset breakdown of RSR/FAA/RCA is provided. Given that AgentFail is prior work by the same authors and supplies both trajectories and root-cause annotations aligned with the taxonomy used in Sec. III-C2/Fig. 4, platform- and source-disaggregated metrics (and a short statement on annotation independence for RCA) are needed to support the multi-platform generality claim.
  4. [Sec. III-D3, Sec. IV-E] Sec. III-D3 and Sec. IV-E: Dynamic verification success is defined as the modified workflow executing successfully on original test inputs, yet agentic nodes are stochastic. The paper does not state how many execution trials, what temperature/decoding settings, or what pass criterion (single success vs. k-of-n) is used for RSR. Free parameters that affect the loop—retry/test budget and the offset-rationality magnitude threshold in pre-execution assessment—are also left unspecified. These choices are load-bearing for the 71.3% RSR figure and for fair comparison to iterative baselines.
minor comments (6)
  1. [Sec. IV-E] Sec. IV-E: Backbone model is listed as “GPT-5.2”; please confirm the exact model identifier and API settings used for all FlowFixer components and baselines so results can be reproduced.
  2. [Fig. 4] Fig. 4: Multiple distinct root causes share the same strategy labels (e.g., several map to R1). A short clarifying sentence that strategies are many-to-one with root causes would avoid confusion.
  3. [Sec. IV-D, Table II] Metrics definition of RCA (Sec. IV-D) is conditional on correct failure attribution; state this explicitly in Table II’s caption so absolute root-cause recovery rate is not misread.
  4. [Sec. VII-A] Sec. VII-A: Pre-execution assessment reports 99.7% precision and 84.6% recall; define the positive class (predicted-fail vs. predicted-pass) and the sample size of candidates evaluated so the rates are interpretable.
  5. [Sec. VIII] Related Work (Sec. VIII) could more clearly separate workflow-structure repair from prompt-only evolution when positioning against Maestro, CE-Graph, SCOPE, and SelfHeal, matching the baseline categories used in Sec. IV-C.
  6. [Fig. 1, throughout] Minor presentation: unify hyphenation of “agentic workflow(s)” vs. “agentic-workflow”; ensure Fig. 1 stage labels match the body text (“Failure Diagnosis” / “Workflow Repair”).

Circularity Check

1 steps flagged

No derivation-by-construction circularity; empirical RSR/FAA/RCA numbers are measured on annotated logs, with only minor non-load-bearing self-citation of the authors' AgentFail dataset and taxonomy.

specific steps
  1. self citation load bearing [Sec. IV-B Dataset; also Sec. III-C2 Root Cause Analysis / Fig. 4]
    "We adopt the publicly available AgentFail [34] dataset as the primary experimental benchmark... FlowFixer constructs a root cause taxonomy from prior studies on agent and workflow failures [28], [34]–[36]"

    Reference [34] is the authors' own prior arXiv paper that supplies both the majority of the evaluation logs and part of the root-cause taxonomy used by the method. This is ordinary self-citation of a dataset paper and does not force the measured RSR/FAA/RCA numbers by construction; the annotations remain external labels and the performance deltas are still empirical comparisons against baselines. It is therefore only a minor, non-load-bearing instance.

full rationale

FlowFixer is an empirical systems paper whose central claims are measured repair success, failure-attribution accuracy and root-cause accuracy on held-out failure logs (AgentFail + newly collected n8n cases). There is no equation, uniqueness theorem, fitted free parameter, or ansatz that is later re-presented as a prediction. The root-cause taxonomy and repair strategies are imported from prior literature (including the authors' own AgentFail paper) as structured prior knowledge; they are not fitted to the test outcomes, and the reported accuracies are computed against independent expert annotations rather than being forced by construction. The LLM-generated behavioral assertions (Sec. III-B2) are an internal methodological step whose quality is unmeasured, but that is a validity/correctness concern, not circularity of the derivation chain. The single self-citation of AgentFail supplies the evaluation corpus and part of the taxonomy; it does not make the performance numbers tautological. Hence the circularity score is 1 (minor self-citation that is not load-bearing).

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 3 invented entities

The central performance claims rest on a small set of modeling choices (three constraint dimensions, a fixed 16-type taxonomy, five atomic edits, four assessment axes) and on the unproved but standard assumption that an LLM can emit usable formal assertions and patches. No continuous free parameters are fitted to the test metrics; the main invented artifacts are the intermediate symbolic representation and the assessment procedure itself.

free parameters (2)
  • retry budget / test budget
    Number of diagnosis-repair iterations before declaring failure; value not stated numerically yet controls whether a case counts as repaired.
  • offset-rationality magnitude threshold
    Unspecified numeric bound that decides whether a patch is 'too large' or 'too small' in the pre-execution filter (Sec. III-D2).
axioms (4)
  • domain assumption Workflow failures are adequately captured by violations of existence, temporal-order and causal/semantic constraints expressible in the paper's assertion DSL.
    Stated as the design rationale for the three-dimensional decomposition (Sec. III-B2); if many real failures lie outside these three classes, diagnosis evidence is incomplete.
  • ad hoc to paper An LLM given node context, connections and task goal can synthesize correct executable assertions and later correct patches.
    Core of symbolic inference and of patch generation; never independently validated beyond end-to-end metrics.
  • domain assumption The sixteen-type root-cause taxonomy (node capability / orchestration / execution) is complete enough for the evaluated platforms.
    Imported from prior studies and used as structured prior knowledge (Fig. 4); coverage is assumed rather than measured.
  • standard math Workflows are directed graphs of typed nodes with explicit configs and data/control edges (Eqs. 1–2).
    Standard platform model restated in Sec. II; uncontroversial.
invented entities (3)
  • Unified symbolic trace + assertion DSL (BNF in Table I) no independent evidence
    purpose: Normalize heterogeneous platform logs and make node correctness, order and causality machine-checkable.
    New intermediate representation introduced by the paper; independent evidence is only the downstream empirical gains.
  • Multi-dimensional pre-execution assessment (structure/semantics/consistency/offset) no independent evidence
    purpose: Filter infeasible patches before costly dynamic runs.
    Novel filter whose 99.7% precision is measured only inside this paper's loop.
  • Experience pool (online feedback + accumulated repair experience) no independent evidence
    purpose: Supply short- and long-term memory across repair iterations.
    Standard memory idea specialized to this pipeline; ablation shows contribution but no external validation.

pith-pipeline@v1.1.0-grok45 · 22394 in / 2991 out tokens · 35108 ms · 2026-07-12T06:23:51.116280+00:00 · methodology

0 comments
read the original abstract

Platform-orchestrated agentic workflows have become a popular paradigm for developing LLM-based applications. However, their reliability remains a major challenge due to the uncertainty of LLM outputs, complex inter-node dependencies, and heterogeneous tool interactions. Existing agentic workflow optimization and agent enhancement methods primarily rely on trajectory-level feedback. Without explicitly identifying the underlying failure root causes, their resulting repair plans are often insufficiently targeted. We propose FlowFixer, a diagnosis-driven automated repair framework for agentic workflows. FlowFixer first transforms workflow executions into unified symbolic traces and performs symbolic inference to derive executable behavioral specifications that capture node correctness, temporal dependencies, and causal relationships. Based on specification verification, it conducts failure attribution and root cause analysis, and then generates targeted repair patches. To reduce verification costs, FlowFixer further employs a multi-dimensional pre-execution assessment to filter infeasible repairs before dynamic verification. We evaluate FlowFixer on workflow failures collected from three popular development platforms: Dify, Coze and n8n. Results show that FlowFixer achieves a repair success rate of 71.3%, outperforming state-of-the-art baselines by 11.9% to 27.6%. It also improves failure attribution accuracy by 4.8% to 33.1% and root cause analysis accuracy by 15.3% to 38.8%. This work offers a new perspective on reliable diagnosis and repair of agentic workflows through symbolic modeling and inference.

Figures

Figures reproduced from arXiv: 2607.02882 by Boyu Wu, Dandan Wang, Junjie Wang, Mingyang Li, Qing Wang, Xiaofei Xie, Xuyan Ma, Yawen Wang.

Figure 1
Figure 1. Figure 1: Overview of FlowFixer. violations and collect node-level verification evidence. Based on such evidence and workflow structure, FlowFixer locates the failure-responsible node and analyzes its root cause. It then generates repair patches through a set of atomic edit operations and applies them to the original workflow to obtain a modified workflow. The modified workflow undergoes a lightweight pre￾execution … view at source ↗
Figure 2
Figure 2. Figure 2: Symbolic Inference Example. “node” represents the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multi-dimension Pre-execution Assessment. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Root Cause Taxonomy and Repair Strategies. Root [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example of the process of failure diagnosis and repair. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of the process of repair. workflow structure. Consequently, this repair candidate is rejected before expensive dynamic execution. FlowFixer then generates a new repair patch by inserting “Checker” between “Begin” and “Classification”, preserving the original execution dependencies while introducing the required input validation. The repaired workflow successfully passes the pre-execution assessment… view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of repair operators and repaired node. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 17 linked inside Pith

  1. [1]

    From language to action: a review of large language models as autonomous agents and tool users,

    S. S. Chowa, R. Alvi, S. S. Rahman, M. A. Rahman, M. A. K. Raiaan, M. R. Islam, M. Hussain, and S. Azam, “From language to action: a review of large language models as autonomous agents and tool users,” Artificial Intelligence Review, 2026

    2026

  2. [2]

    Dify Contributors, “Dify,” https://dify.ai, 2023

    2023

  3. [3]

    Coze Contributors, “Coze,” https://www.coze.com/, 2025

    2025

  4. [4]

    n8n Contributors, “n8n,” https://n8n.io, 2019

    2019

  5. [5]

    Workshop: Building ai applications with dify. ai: A hands-on workshop,

    A. Leatherwood and V . Matta, “Workshop: Building ai applications with dify. ai: A hands-on workshop,” 2025

    2025

  6. [6]

    (2025) What is dify? build powerful ai apps & agents with ease

    WorkFlows.so. (2025) What is dify? build powerful ai apps & agents with ease. Accessed: 2026-05-30. [Online]. Available: https://workflows. so/blog/what-is-dify-build-powerful-ai-apps-agents-with-ease

    2025

  7. [7]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,

    L. Wang, W. Xu, Y . Lan, Z. Hu, Y . Lan, R. K.-W. Lee, and E.-P. Lim, “Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,” inProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), 2023, pp. 2609–2634

    2023

  8. [8]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Conget al., “Chatdev: Communicative agents for software development,” inProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024, pp. 15 174–15 186

    2024

  9. [9]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhouet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 247–23 275

    2024

  10. [10]

    Ai agents and agentic systems: A multi-expert analysis,

    L. Hughes, Y . K. Dwivedi, T. Malik, M. Shawosh, M. A. Albashrawi, I. Jeon, V . Dutot, M. Appanderanda, T. Crick, R. De’et al., “Ai agents and agentic systems: A multi-expert analysis,”Journal of Computer Information Systems, vol. 65, no. 4, pp. 489–517, 2025

    2025

  11. [11]

    Textual bayes: Quantifying uncertainty in llm-based systems,

    B. L. Ross, N. V ouitsis, A. A. Ghomi, R. Hosseinzadeh, J. Xin, Z. Liu, Y . Sui, S. Hou, K. K. Leung, G. Loaiza-Ganemet al., “Textual bayes: Quantifying uncertainty in llm-based systems,”arXiv preprint arXiv:2506.10060, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Tools in the loop: Quantifying uncertainty of llm question answering systems that use tools,

    P. Lymperopoulos and V . Sarathy, “Tools in the loop: Quantifying uncertainty of llm question answering systems that use tools,”arXiv preprint arXiv:2505.16113, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” 2023, publisher Copyright: © 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.; 11th International Conference on Learning Representations, ICLR 2023 ; Conference date: 01-05-2023...

    2023

  14. [14]

    Automatic programming: Large language models and beyond,

    M. R. Lyu, B. Ray, A. Roychoudhury, S. H. Tan, and P. Thongtanunam, “Automatic programming: Large language models and beyond,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–33, 2025

    2025

  15. [15]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  16. [16]

    Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying llm-based software engineering agents,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715754

    doi:10.1145/3715754 2025

  17. [17]

    Repairagent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press, 2025, p. 2188–2200. [Online]. Available: https://doi.org/10.1109/ICSE55347.2025.00157

    doi:10.1109/icse55347.2025.00157 2025

  18. [18]

    Understanding software engineering agents: A study of thought-action-result trajectories,

    I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,”arXiv preprint arXiv:2506.18824, 2025

    arXiv 2025

  19. [19]

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems,

    S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y . Chenet al., “Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems,”arXiv preprint arXiv:2505.00212, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Scope: Prompt evolution for enhancing agent effectiveness,

    Z. Pei, H.-L. Zhen, S. Kai, S. J. Pan, Y . Wang, M. Yuan, and B. Yu, “Scope: Prompt evolution for enhancing agent effectiveness,”arXiv preprint arXiv:2512.15374, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Recreate: Reasoning and creating domain agents driven by experience,

    Z. Hao, H. Wang, J. Luo, J. Zhang, Y . Zhou, Q. Lin, C. Wang, H. Dong, and J. Chen, “Recreate: Reasoning and creating domain agents driven by experience,” 2026. [Online]. Available: https://arxiv.org/abs/2601.11100

    Pith/arXiv arXiv 2026

  22. [22]

    Aegis: Taxonomy and optimizations for overcoming agent-environment failures in llm agents,

    K. Song, A. Jayarajan, Y . Ding, Q. Su, Z. Zhu, S. Liu, and G. Pekhimenko, “Aegis: Taxonomy and optimizations for overcoming agent-environment failures in llm agents,”arXiv preprint arXiv:2508.19504, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Maestro: Joint graph & config optimization for reliable ai agents,

    W. Wang, P. Kattakinda, and S. Feizi, “Maestro: Joint graph & config optimization for reliable ai agents,”arXiv preprint arXiv:2509.04642, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Reducing cost of llm agents with trajectory reduction,

    Y .-A. Xiao, P. Gao, C. Peng, and Y . Xiong, “Reducing cost of llm agents with trajectory reduction,” 2026. [Online]. Available: https://arxiv.org/abs/2509.23586

    arXiv 2026

  25. [25]

    Stop wasting your tokens: Towards efficient runtime multi-agent systems,

    F. Lin, S. Chen, R. Fang, H. Wang, and T. Lin, “Stop wasting your tokens: Towards efficient runtime multi-agent systems,”arXiv preprint arXiv:2510.26585, 2025

    arXiv 2025

  26. [26]

    Autogen: Enabling next-gen llm applications via multi-agent conversation,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liuet al., “Autogen: Enabling next-gen llm applications via multi-agent conversation,”arXiv preprint arXiv:2308.08155, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,

    Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,”Advances in Neural Information Processing Systems, vol. 36, pp. 38 154–38 180, 2023

    2023

  28. [28]

    Where llm agents fail and how they can learn from failures,

    K. Zhu, Z. Liu, B. Li, M. Tian, Y . Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhanget al., “Where llm agents fail and how they can learn from failures,”arXiv preprint arXiv:2509.25370, 2025

    arXiv 2025

  29. [29]

    From spark to fire: Modeling and mitigating error cascades in llm-based multi-agent collaboration,

    Y . Xie, C. Zhu, X. Zhang, T. Zhu, D. Ye, M. Qi, H. Chen, and W. Zhou, “From spark to fire: Modeling and mitigating error cascades in llm-based multi-agent collaboration,”arXiv preprint arXiv:2603.04474, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    Reagent: Reversible multi-agent reasoning for knowledge-enhanced multi-hop qa,

    Z. Xinjie, F. Gao, X. Song, Y . Chen, R. Yang, Y . Fu, Y . Wang, Y . Iwasawa, Y . Matsuo, and I. Li, “Reagent: Reversible multi-agent reasoning for knowledge-enhanced multi-hop qa,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 4067–4089

    2025

  31. [31]

    Agentkit: structured llm reasoning with dynamic graphs,

    Y . Wu, Y . Fan, S. Y . Min, S. Prabhumoye, S. McAleer, Y . Bisk, R. Salakhutdinov, Y . Li, and T. Mitchell, “Agentkit: structured llm reasoning with dynamic graphs,”arXiv preprint arXiv:2404.11483, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Webtestpilot: Agentic end-to-end web testing against natural language specification by inferring oracles with symbolized gui elements,

    X. Teoh, Y . Lin, D.-M. Nguyen, R. Ren, W. Zhang, and J. S. Dong, “Webtestpilot: Agentic end-to-end web testing against natural language specification by inferring oracles with symbolized gui elements,”arXiv preprint arXiv:2602.11724, 2026

    Pith/arXiv arXiv 2026

  33. [33]

    Logicskills: A structured benchmark for formal reasoning in large language models,

    B. Rabern, P. Mondorf, and B. Plank, “Logicskills: A structured benchmark for formal reasoning in large language models,”arXiv preprint arXiv:2602.06533, 2026

    arXiv 2026

  34. [34]

    Demystifying the lifecycle of failures in platform-orchestrated agentic workflows,

    X. Ma, X. Xie, Y . Wang, J. Wang, B. Wu, M. Li, and Q. Wang, “Demystifying the lifecycle of failures in platform-orchestrated agentic workflows,”arXiv preprint arXiv:2509.23735, 2026

    arXiv 2026

  35. [35]

    Why do multi-agent llm systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandranet al., “Why do multi-agent llm systems fail?”Advances in Neural Information Processing Systems, vol. 38, 2026

    2026

  36. [36]

    Exploring autonomous agents: A closer look at why they fail when completing tasks,

    R. Lu, Y . Li, and Y . Huo, “Exploring autonomous agents: A closer look at why they fail when completing tasks,” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 3856–3860

    2025

  37. [37]

    Selfheal: Empirical fix pattern analysis and bug repair in llm agents,

    N. Islam, M. A. Raza, and M. Wardat, “Selfheal: Empirical fix pattern analysis and bug repair in llm agents,” 2026. [Online]. Available: https://arxiv.org/abs/2604.17699

    Pith/arXiv arXiv 2026

  38. [38]

    Agentfixer: From failure detection to fix recommendations in agentic systems,

    H. Mulian, S. Zeltyn, I. Levy, L. Galanti, A. Yaeli, and S. Shlomov, “Agentfixer: From failure detection to fix recommendations in agentic systems,” inProceedings of the 2026 International Workshop on Agentic Engineering, ser. AGENT ’26. New York, NY , USA: Association for Computing Machinery, 2026, p. 96–103. [Online]. Available: https://doi.org/10.1145/...

    doi:10.1145/3786167.3788427 2026

  39. [39]

    Gepa: Reflective prompt evolution can outperform reinforcement learning,

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab, “Gepa: Reflective prompt evolution can outperform reinforcement learning,” inInternational Conference on Learning Representations (ICLR). OpenReview, 2026, oral P...

    2026

  40. [40]

    Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis,

    Y . Ge, L. Xie, Z. Li, Y . Pei, and T. Zhang, “Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis,”arXiv preprint arXiv:2509.13782, 2025

    arXiv 2025

  41. [41]

    Dover: Intervention-driven auto debugging for llm multi-agent systems, 2025,

    M. Ma, J. Zhang, F. Yang, Y . Kang, Q. Lin, T. Yang, S. Rajmohan, and D. Zhang, “Dover: Intervention-driven auto debugging for llm multi-agent systems, 2025,”URL https://arxiv. org/abs/2512.06749

    arXiv 2025

  42. [42]

    Failure-driven workflow refinement,

    J. Zhang, K. Cai, Q. Zeng, N. Liu, S. Fan, Z. Chen, and K. Wang, “Failure-driven workflow refinement,” 2025. [Online]. Available: https://arxiv.org/abs/2510.10035

    arXiv 2025

  43. [43]

    Instruction-level weight shaping: A framework for self- improving ai agents,

    R. Costa, “Instruction-level weight shaping: A framework for self- improving ai agents,”arXiv preprint arXiv:2509.00251, 2025

    arXiv 2025

  44. [44]

    Agentdevel: Reframing self-evolving llm agents as release engineering,

    D. Zhang, “Agentdevel: Reframing self-evolving llm agents as release engineering,” 2026. [Online]. Available: https://arxiv.org/abs/2601.04620

    arXiv 2026

  45. [45]

    Trail: Trace reasoning and agentic issue localization,

    D. Deshpande, V . Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian, “Trail: Trace reasoning and agentic issue localization,”arXiv preprint arXiv:2505.08638, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Traceability and accountability in role-specialized multi-agent llm pipelines,

    A. Barrak, “Traceability and accountability in role-specialized multi-agent llm pipelines,” 2025. [Online]. Available: https://arxiv.org/abs/2510.07614

    arXiv 2025

  47. [47]

    Correct: Condensed error recognition via knowledge transfer in multi-agent systems,

    Y . Yu, M. Li, S. Xu, J. Fu, X. Hou, F. Lai, and B. Wang, “Correct: Condensed error recognition via knowledge transfer in multi-agent systems,”arXiv preprint arXiv:2509.24088, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Where did it all go wrong? a hierarchical look into multi-agent error attribution,

    A. Banerjee, A. Nair, and T. Borogovac, “Where did it all go wrong? a hierarchical look into multi-agent error attribution,”arXiv preprint arXiv:2510.04886, 2025

    arXiv 2025

  49. [49]

    Raffles: Reasoning-based attribution of faults for llm systems,

    C. Zhu, S. Hong, J. Wu, K. Chawla, Y . Tang, Y . Yin, N. Wolfe, E. Babinsky, and D. Liu, “Raffles: Reasoning-based attribution of faults for llm systems,” inProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 7659–7688

    2026

  50. [50]

    Agentracer: Who is inducing failure in the llm agentic systems?

    G. Zhang, J. Wang, J. Chen, W. Zhou, K. Wang, and S. Yan, “Agentracer: Who is inducing failure in the llm agentic systems?”arXiv preprint arXiv:2509.03312, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Graphtracer: Graph-guided failure tracing in llm agents for robust multi-turn deep search,

    H. Zhang, Y . Shi, X. Gu, H. You, Z. Zhang, L. Gan, Y . Yuan, and J. Huang, “Graphtracer: Graph-guided failure tracing in llm agents for robust multi-turn deep search,”arXiv preprint arXiv:2510.10581, 2025

    arXiv 2025

  52. [52]

    Aegis: Automated error generation and attribution for multi-agent systems,

    F. Kong, R. Zhang, H. Yin, G. Zhang, X. Zhang, Z. Chen, Z. Zhang, X. Zhang, S.-C. Zhu, and X. Feng, “Aegis: Automated error generation and attribution for multi-agent systems,”arXiv preprint arXiv:2509.14295, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Seeing the whole elephant: A benchmark for failure attribution in llm-based multi-agent systems,

    M. Chen, J. Wang, F. Mu, Y . Wang, Z. Liu, H. Feng, and Q. Wang, “Seeing the whole elephant: A benchmark for failure attribution in llm-based multi-agent systems,” 2026. [Online]. Available: https://arxiv.org/abs/2604.22708

    Pith/arXiv arXiv 2026