Pith

open record

sign in

arxiv: 2605.08717 · v2 · pith:UHF6IZ5R · submitted 2026-05-09 · cs.SE · cs.AI

Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents

Chenyu Zhao , Shenglin Zhang , Yihang Lin , Wenwei Gu , Zhimin Chen , Yongqian Sun , Dan Pei , Chetan Bansal
show 2 more authors
Saravan Rajmohan Minghua Ma
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-30 23:08 UTCgrok-4.3pith:UHF6IZ5Rrecord.jsonopen to challenge →

classification cs.SE cs.AI
keywords software engineering agentsfailure recoverystructured diagnosisrecovery guidancesoftware repairAIOpstelemetry analysis
0
0 comments X

The pith

PROBE turns failed software engineering agent runs into evidence-grounded recovery guidance that raises success rates.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Software engineering agents frequently fail on tasks, but existing approaches leave recovery to manual or ad hoc processes that do not systematically use runtime evidence. PROBE introduces a three-part structure that preserves detailed telemetry from failed runs, fuses signals into diagnoses, and then filters output through a gate to produce only bounded, actionable guidance an agent can execute on the next attempt. Evaluation on 257 initially unresolved cases shows this yields 65.37 percent top-1 diagnosis accuracy and a 21.79 percent recovery rate. The results establish that diagnosis alone leaves a gap; only when diagnosis is converted into scoped guidance does recovery improve. The framework attaches as a side channel without altering the underlying agent policy or tools.

Core claim

PROBE organizes failed-run telemetry into structured evidence, structured diagnosis, and bounded recovery guidance through a Telemetry Layer that preserves fine-grained runtime signals, a Diagnosis Layer that fuses cross-signal evidence into grounded diagnoses, and a Guidance Gate that produces diagnosis-derived guidance only when it is evidence-grounded, actionable, and within the scope of agent-side behavior. On 257 initially unresolved cases across repository-level repair, enterprise workflow recovery, and AIOps mitigation, PROBE reaches 65.37 percent Top-1 diagnosis accuracy and a 21.79 percent recovery rate, outperforming the strongest non-PROBE baseline by 43.58 and 12.45 percentage po

What carries the argument

The Guidance Gate, which produces diagnosis-derived guidance only when evidence-grounded, actionable, and within the scope of agent-side behavior.

If this is right

  • On 257 initially unresolved cases, PROBE reaches 65.37 percent Top-1 diagnosis accuracy.
  • PROBE reaches a 21.79 percent recovery rate on the same cases.
  • These figures exceed the strongest non-PROBE baseline by 43.58 and 12.45 percentage points.
  • Accurate diagnosis alone is necessary but insufficient without translation into bounded, executable guidance.
  • The framework attaches as a non-intrusive side channel to existing workflows without changing agent policy, toolset, or budget.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The diagnosis-recovery gap implies that agent architectures may need dedicated components for generating executable next steps rather than stopping at diagnosis.
  • Non-intrusive attachment to existing systems suggests the approach can be tested on other agent platforms without full reimplementation.
  • The emphasis on bounded guidance may extend to domains where agents must recover from failures under fixed execution budgets.

Load-bearing premise

The Guidance Gate can reliably produce bounded recovery guidance that is evidence-grounded and executable by the agent in subsequent attempts.

What would settle it

Running the same 257 unresolved cases with PROBE guidance disabled and measuring whether the recovery rate drops back to baseline levels.

Figures

Figures reproduced from arXiv: 2605.08717 by Chenyu Zhao, Chetan Bansal, Dan Pei, Minghua Ma, Saravan Rajmohan, Shenglin Zhang, Wenwei Gu, Yihang Lin, Yongqian Sun, Zhimin Chen.

Figure 1
Figure 1. Figure 1: Prior work treats observability, diagnosis, and in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the PROBE framework. PROBE organizes failed-run telemetry into a typed telemetry bundle (T ), constructs structured evidence (E), derives structured diagnosis (D), and produces bounded recovery guidance (G), forming a recovery loop for subsequent attempts. Step 1. Failure Localization Repeated Tool Failures Execution Errors Unstable Intent Transitions Inefficient Action Patterns Claimed vs. Obs… view at source ↗
Figure 3
Figure 3. Figure 3: Detailed workflow of the Diagnosis Layer. workflow state, and evaluator-side observations. It helps distinguish agent-side behavioral failures from environment-side constraints. Optional external outcome evidence. 𝑇outcome is included when evaluator-side feedback is available, such as benchmark ver￾dicts, test results, incident-resolution checks, or task-specific scor￾ing scripts. It grounds telemetry agai… view at source ↗
Figure 4
Figure 4. Figure 4: AIOpsLab case illustrating how PROBE preserves failed-run evidence, derives a structured diagnosis, and pro￾duces bounded recovery guidance for the subsequent at￾tempt. tasks overall, but their Top-1 accuracy remains much lower than PROBE, at 14.79% and 21.79%. This suggests that generic trace sum￾maries provide useful runtime context, but often fail to preserve the signal-specific anchors needed to identi… view at source ↗
read the original abstract

Software engineering agents are increasingly deployed in evaluable engineering environments, yet post-failure recovery remains costly, manual, and ad hoc. Existing systems expose traces or generate follow-up feedback, but they do not convert heterogeneous runtime evidence into grounded, bounded recovery guidance for a subsequent attempt. We present PROBE, a failure-anchored framework for structured recovery in software engineering agents. PROBE organizes failed-run telemetry into structured evidence, structured diagnosis, and bounded recovery guidance through a Telemetry Layer, a Diagnosis Layer, and a Guidance Gate. The Telemetry Layer preserves fine-grained runtime signals, the Diagnosis Layer fuses cross-signal evidence into grounded diagnoses, and the Guidance Gate produces diagnosis-derived guidance only when it is evidence-grounded, actionable, and within the scope of agent-side behavior. We evaluate PROBE across three settings: repository-level software repair, enterprise workflow recovery, and AIOps service mitigation. On 257 initially unresolved cases, PROBE achieves 65.37% Top-1 diagnosis accuracy and a 21.79% recovery rate, outperforming the strongest non-PROBE baseline by 43.58 and 12.45 percentage points. The results reveal a diagnosis-recovery gap: accurate diagnosis is necessary but insufficient unless translated into bounded guidance that a subsequent attempt can execute and verify. Beyond controlled evaluation, a Microsoft IcM prototype shows that PROBE can attach as a non-intrusive side channel to existing service-diagnosis workflows without changing the agent policy, toolset, or execution budget. These results suggest that telemetry-grounded, failure-anchored recovery can improve post-failure recoverability under realistic engineering constraints.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces PROBE, a failure-anchored framework for structured recovery in software engineering agents. It organizes failed-run telemetry into structured evidence, structured diagnosis, and bounded recovery guidance through a Telemetry Layer, a Diagnosis Layer, and a Guidance Gate. The framework is evaluated across repository-level software repair, enterprise workflow recovery, and AIOps service mitigation. On 257 initially unresolved cases, PROBE reports 65.37% Top-1 diagnosis accuracy and a 21.79% recovery rate, outperforming the strongest non-PROBE baseline by 43.58 and 12.45 percentage points. A Microsoft IcM prototype demonstrates non-intrusive attachment to existing workflows.

Significance. If the reported performance numbers hold, the work addresses a practically important gap between diagnosis and actionable recovery in SE agents by enforcing evidence-grounded, bounded guidance. The empirical results on a non-trivial set of cases and the prototype for real-world integration without altering agent policy or budget are strengths that could influence agent design in engineering environments.

major comments (2)
  1. [Evaluation] The central empirical claims rest on results for 257 cases, but the manuscript provides no description of the experimental setup, baseline implementations, case selection criteria, or statistical significance tests. This absence prevents verification of the 65.37% and 21.79% figures and the reported outperformance margins.
  2. [Guidance Gate] The Guidance Gate is presented as the mechanism that converts diagnosis into executable guidance and is credited with closing the diagnosis-recovery gap, yet the manuscript does not specify the criteria or procedure used to enforce boundedness and executability or how the gap is quantified across the three settings.
minor comments (1)
  1. A table breaking down the 257 cases by the three evaluation settings (with per-setting accuracy and recovery rates) would improve clarity of the aggregate results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and the recommendation for major revision. We address each major comment below and will revise the manuscript to provide the requested details and clarifications.

read point-by-point responses

  1. Referee: [Evaluation] The central empirical claims rest on results for 257 cases, but the manuscript provides no description of the experimental setup, baseline implementations, case selection criteria, or statistical significance tests. This absence prevents verification of the 65.37% and 21.79% figures and the reported outperformance margins.

    Authors: We agree that the current manuscript would benefit from expanded detail on the experimental setup to support reproducibility and verification of the reported results. In the revised version, we will substantially expand the Evaluation section (and add an appendix if needed) to include: explicit case selection criteria for the 257 unresolved cases across the three settings, implementation details for all baselines, and any statistical significance tests applied to the diagnosis accuracy and recovery rates. These additions will directly enable verification of the 65.37% Top-1 accuracy, 21.79% recovery rate, and the outperformance margins over the strongest baseline. revision: yes

  2. Referee: [Guidance Gate] The Guidance Gate is presented as the mechanism that converts diagnosis into executable guidance and is credited with closing the diagnosis-recovery gap, yet the manuscript does not specify the criteria or procedure used to enforce boundedness and executability or how the gap is quantified across the three settings.

    Authors: We acknowledge that the manuscript should more explicitly articulate the operational criteria and procedures of the Guidance Gate. In the revision, we will add a dedicated subsection describing: (i) the precise criteria used to decide whether guidance is emitted (evidence-grounded, actionable within agent behavior, and bounded in scope), (ii) the step-by-step procedure for enforcing these constraints, and (iii) how the diagnosis-recovery gap is quantified in each setting (repository-level repair, enterprise workflow recovery, and AIOps mitigation) by relating diagnosis accuracy to subsequent recovery success. This will clarify the mechanism's contribution to closing the gap. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper describes an empirical framework (PROBE) with Telemetry Layer, Diagnosis Layer, and Guidance Gate, then reports direct evaluation metrics (65.37% Top-1 accuracy, 21.79% recovery rate) on 257 cases. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text. The central claims rest on observable performance differences versus baselines rather than any self-referential reduction. This is a standard empirical software-engineering result with no load-bearing definitional or predictive circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 3 invented entities

Abstract-only review provides no information on free parameters or axioms; the framework introduces three new conceptual layers as entities without external validation mentioned.

invented entities (3)
  • Telemetry Layer no independent evidence
    purpose: Preserves fine-grained runtime signals from failed runs
    Core component of the proposed PROBE framework.
  • Diagnosis Layer no independent evidence
    purpose: Fuses cross-signal evidence into grounded diagnoses
    Core component of the proposed PROBE framework.
  • Guidance Gate no independent evidence
    purpose: Produces diagnosis-derived guidance only when evidence-grounded, actionable, and within scope
    Core component of the proposed PROBE framework.

pith-pipeline@v0.9.1-grok · 5858 in / 1271 out tokens · 37266 ms · 2026-06-30T23:08:07.393737+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation

    cs.SE 2026-07 conditional novelty 6.0

    AgentTether repairs 69% of initially failed LLM agent tasks on τ-bench by combining graph-guided root-cause diagnosis, cross-iteration repair memory, and guarded runtime intervention, improving over blind retry by 26 ...

Reference graph

Works this paper leans on

51 extracted references · 51 canonical work pages · cited by 1 Pith paper · 8 internal anchors

  1. [1]

    Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Saravan Rajmohan. 2023. Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1737–1749. doi:10.1109/ICSE48619.2...

    work page doi:10.1109/icse48619.2023.00149 2023

  2. [2]

    C., Arun Iyer, Suresh Parthasarathy, Sriram K

    Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D. C., Arun Iyer, Suresh Parthasarathy, Sriram K. Rajamani, Balasubramanyan Ashok, and Shashank Shet. 2024. CodePlan: Repository-Level Coding using LLMs and Plan- ning.Proc. ACM Softw. Eng.1, FSE (2024), 675–698. doi:10.1145/3643757

    work page doi:10.1145/3643757 2024

  3. [3]

    Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal. 2026. AgentRx: Diagnosing AI Agent Failures from Execution Trajectories. arXiv:2602.02475 [cs.AI] https://arxiv.org/abs/2602.02475

    work page arXiv 2026

  4. [4]

    BerriAI. 2024. LiteLLM. https://docs.litellm.ai/. Unified interface for calling multiple large language model providers. Accessed: 2026-04-09

    work page 2024

  5. [5]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. arXiv:2403.17134 [cs.SE] https://arxiv.org/abs/2403.17134

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Islem Bouzenia and Michael Pradel. 2025. Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories. In40th IEEE/ACM Inter- national Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 2846–2857. doi:10.1109/ASE63991.2025. 00234

    work page doi:10.1109/ase63991.2025 2025

  7. [7]

    Harrison Chase. 2022. LangChain. https://github.com/langchain-ai/langchain. Open-source framework for building LLM applications and agents. Accessed: 2026-04-09

    work page 2022

  8. [8]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv:2304.05128 [cs.CL] https://arxiv. org/abs/2304.05128

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, and Saravan Rajmohan. 2025. AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Au- tonomous Clouds. InProceedings of the Eighth Conference on Machine Learn- ing and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025, ...

    work page 2025

  10. [10]

    Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, Jun Zeng, Supriyo Ghosh, Xuchao Zhang, Chaoyun Zhang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Tianyin Xu. 2024. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. InProceedings of the Nineteenth European Confer...

    work page doi:10.1145/3627703.3629553 2024

  11. [11]

    Zhi Chen, Wei Ma, and Lingxiao Jiang. 2026. Beyond Final Code: A Process- Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios. arXiv:2503.12374 [cs.SE] https://arxiv.org/abs/2503.12374

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [12]

    Liming Dong, Qinghua Lu, and Liming Zhu. 2024. AgentOps: Enabling Observ- ability of LLM Agents. arXiv:2411.05285 [cs.AI] https://arxiv.org/abs/2411.05285

    work page arXiv 2024

  13. [13]

    Drishti Goel, Fiza Husain, Aditya Singh, Supriyo Ghosh, Anjaly Parayil, Chetan Bansal, Xuchao Zhang, and Saravan Rajmohan. 2024. X-Lifecycle Learning for Cloud Incident Management using LLMs. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, Porto de Galinhas, Brazil, July 15-19, 2024, M...

    work page doi:10.1145/3663529.3663861 2024

  14. [14]

    2025.Agents Companion

    Antonio Gulli, Lavi Nigam, Julia Wiesinger, Vladimir Vuskovic, Irina Sigler, Ivan Nardini, Nicolas Stroppa, Sokratis Kartakis, Narek Saribekyan, and Alan Bount. 2025.Agents Companion. Technical Report. Google. https:// www.kaggle.com/whitepaper-agent-companion Google Whitepaper. Avail- able at: https://cdn.jsdelivr.net/gh/abncharts/abncharts.public.1/abna...

    work page 2025

  15. [15]

    Zikang Guo, Benfeng Xu, Chiwei Zhu, Wentao Hong, Xiaorui Wang, and Zhen- dong Mao. 2026. MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools. InFortieth AAAI Conference on Artifi- cial Intelligence, Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advanc...

    work page doi:10.1609/aaai.v40i37 2026

  16. [16]

    Songqiao Han, Xiyang Hu, Hailiang Huang, Mingqi Jiang, and Yue Zhao. 2022. ADBench: Anomaly Detection Benchmark. arXiv:2206.09426 [cs.LG] https: //arxiv.org/abs/2206.09426

    work page arXiv 2022

  17. [17]

    Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang

    work page

  18. [18]

    InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024

    Xpert: Empowering Incident Management with Query Recommendations via Large Language Models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 92:1–92:13. doi:10.1145/3597503.3639081

    work page doi:10.1145/3597503.3639081 2024

  19. [19]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

    work page 2024

  20. [20]

    LangChain. 2026. LangSmith: A Platform for Observability and Evaluation of LLM-based Systems. https://docs.langchain.com/langsmith/observability. Ac- cessed: 2026-03

    work page 2026

  21. [21]

    Han Li, Yifan Yao, Letian Zhu, Rili Feng, Hongyi Ye, Jiaming Wang, Yancheng He, Pengyu Zou, Lehan Zhang, Xinping Lei, Haoyang Huang, Ken Deng, Ming Sun, Zhaoxiang Zhang, He Ye, and Jiaheng Liu. 2026. CodeTracer: Towards Traceable Agent States. arXiv:2604.11641 [cs.SE] https://arxiv.org/abs/2604.11641

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [22]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, Decem- ber 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (...

    work page doi:10.18653/v1/2023.emnlp-main.187 2023

  23. [23]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651 [cs.CL] https://arxiv.or...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair, Sagar Davasam, Aman Tiwari, Sathwik Tejaswi Madhusudhan, Sridhar Krishna Nemala, Srinivas Sunkara, and Sai Rajeswar. 2026. EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings. arXiv:2603.13594 [cs.AI] https://arxiv.org/abs/2603.13594

    work page arXiv 2026

  25. [25]

    Model Context Protocol Contributors. 2025. Model Context Protocol Specification. https://modelcontextprotocol.io/specification/2025-06-18. Accessed: 2026-04-09

    work page 2025

  26. [26]

    Rahul Nanda, Chandra Maddila, Smriti Jha, Euna Mehnaz Khan, Matteo Paltenghi, and Satish Chandra. 2026. Wink: Recovering from Misbehaviors in Coding Agents. arXiv:2602.17037 [cs.SE] https://arxiv.org/abs/2602.17037

    work page arXiv 2026

  27. [27]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is Self-Repair a Silver Bullet for Code Generation? arXiv:2306.09896 [cs.CL] https://arxiv.org/abs/2306.09896

    work page arXiv 2024

  28. [28]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez

    work page

  29. [29]

    Gorilla: Large Language Model Connected with Massive APIs. InAdvances in Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurIPS 2024, Vancou- ver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang ...

    work page 2024

  30. [30]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. InThe Twelfth International Conference on Le...

    work page 2024

  31. [31]

    Kauffmann, Robert A

    Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Gregoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Muller

    work page

  32. [32]

    IEEE 109, 5 (May 2021), 756–795

    A Unifying Review of Deep and Shallow Anomaly Detection.Proc. IEEE 109, 5 (May 2021), 756–795. doi:10.1109/jproc.2021.3052449

    work page doi:10.1109/jproc.2021.3052449 2021

  33. [33]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023, New Or...

    work page 2023

  34. [34]

    Manish Shetty, Chetan Bansal, Sumit Kumar, Nikitha Rao, Nachiappan Nagappan, and Thomas Zimmermann. 2021. Neural Knowledge Extraction From Cloud Service Incidents. In43rd IEEE/ACM International Conference on Software Engi- neering: Software Engineering in Practice, ICSE (SEIP) 2021, Madrid, Spain, May 25-28, 2021. IEEE, 218–227. doi:10.1109/ICSE-SEIP52600...

    work page doi:10.1109/icse-seip52600.2021.00031 2021

  35. [35]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. doi:10.48550/ARXIV.2303.11366

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.11366 2023

  36. [36]

    Gou Tan, Zilong He, Min Li, Haiyu Huang, Yilun Wang, Pengfei Chen, Giuliano Casale, and Chuanfu Zhang. 2026. LLMRCA: Multilevel Root Cause Analysis for LLM Applications Using Multimodal Observability Data.ACM Trans. Softw. Eng. Methodol.(April 2026). doi:10.1145/3806200 Just Accepted

    work page doi:10.1145/3806200 2026

  37. [37]

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian

    work page

  38. [38]

    arXiv preprint arXiv:2407.18901 , year=

    AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. arXiv:2407.18901 [cs.SE] https://arxiv.org/abs/2407. ASE ’26, October 12–16, 2026, Munich, Germany Zhao et al. 18901

    work page arXiv 2026

  39. [39]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompt- ing Elicits Reasoning in Large Language Models. InAdvances in Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA,...

    work page 2022

  40. [40]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework.CoRR abs/2308.08155 (2023). arXiv:2308.08155 doi:10.48550/ARXIV.2308.08155

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2308.08155 2023

  41. [41]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying LLM-Based Software Engineering Agents.Proc. ACM Softw. Eng.2, FSE (2025), 801–824. doi:10.1145/3715754

    work page doi:10.1145/3715754 2025

  42. [42]

    Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’24). ACM, 819–831. doi:10.1145/3650212.3680323

    work page doi:10.1145/3650212.3680323 2024

  43. [43]

    Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu

    Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. 2023. OpenAgents: An Open Platform for Language Agents in the Wild.CoRRabs/2310.10634 (2023). arXiv:2310.10634 doi:10.48550/ARXIV.2310.10634

    work page doi:10.48550/arxiv.2310.10634 2023

  44. [44]

    Zhe Xie, Shenglin Zhang, Yitong Geng, Yao Zhang, Minghua Ma, Xiaohui Nie, Zhenhe Yao, Longlong Xu, Yongqian Sun, Wentao Li, and Dan Pei. 2024. Mi- croservice Root Cause Analysis With Limited Observability Through Intervention Recognition in the Latent Space. InProceedings of the 30th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining, KDD 2024...

    work page doi:10.1145/3637528.3671530 2024

  45. [45]

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer In- terfaces Enable Automated Software Engineering. arXiv:2405.15793 [cs.SE] https://arxiv.org/abs/2405.15793

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview. net/forum?id=WE_vluYUL-X

    work page 2023

  47. [47]

    Las-Casas, Rodrigo Fonseca, and Saravan Rajmohan

    Dylan Zhang, Xuchao Zhang, Chetan Bansal, Pedro Henrique B. Las-Casas, Rodrigo Fonseca, and Saravan Rajmohan. 2023. PACE-LM: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud In- cident Root Cause Analysis.CoRRabs/2309.05833 (2023). arXiv:2309.05833 doi:10.48550/ARXIV.2309.05833

    work page doi:10.48550/arxiv.2309.05833 2023

  48. [48]

    Las-Casas, Ro- drigo Fonseca, and Saravan Rajmohan

    Dylan Zhang, Xuchao Zhang, Chetan Bansal, Pedro Henrique B. Las-Casas, Ro- drigo Fonseca, and Saravan Rajmohan. 2024. LM-PACE: Confidence Estimation by Large Language Models for Effective Root Causing of Cloud Incidents. In Companion Proceedings of the 32nd ACM International Conference on the Founda- tions of Software Engineering, FSE 2024, Porto de Galin...

    work page doi:10.1145/3663529.3663858 2024

  49. [49]

    Xuchao Zhang, Supriyo Ghosh, Chetan Bansal, Rujia Wang, Minghua Ma, Yu Kang, and Saravan Rajmohan. 2024. Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4. arXiv:2401.13810 [cs.CL] https://arxiv. org/abs/2401.13810

    work page arXiv 2024

  50. [50]

    Le, and Ed H

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.n...

    work page 2023

  51. [51]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview....

    work page 2024