Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TrajAudit claims to pinpoint the earliest decisive error step in coding-agent trajectories, reaching 56.6% exact step-level accuracy versus 31.9% for the best baseline in the with-reference setting.

desk verdict Useful benchmark and a plausible method, but the paper's own numbers contradict each other and the headline accuracy claims rest on an unvalidated counterfactual ground truth. read the letter →

arxiv 2605.26563 v3 pith:SEHWXHFR submitted 2026-05-26 cs.SE

Minxing Wang , Xiaofei Xie , Yintong Huo This is my paper · ORCID
classification cs.SE
keywords agenticsystemsfailurediagnosistrajectoryanalysisfaultlocalizationLLMagentssoftwaremaintenancelong-contextreasoningbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing failure-diagnosis methods for LLM coding agents fail on long, noisy trajectories because they passively consume entire execution logs. TrajAudit instead uses a three-part agent: a prior-reasoning module reads the test error to form a preliminary diagnosis, a folding module compresses failure-irrelevant observations, and an investigator agent can unfold specific steps on demand. On RootSE—a new benchmark of 93 failed repository-level coding runs—TrajAudit localizes the ground-truth earliest decisive error step exactly in 56.6% of cases with a reference patch and 50.9% without, versus 31.9% and 26.9% for the best baseline. It also uses fewer tokens. The paper's goal, if correct, is a practical diagnostic tool and a reusable benchmark for agent failure analysis.

What carries the argument

Semantic saliency folding: a compression rule that keeps only observations matching patch-header patterns (e.g., '--- a', '+++ b', '@@ -N,M +P,Q @@') or failure-keyword dictionaries (e.g., 'exception', 'traceback', 'fail'), replacing the rest with a placeholder that the investigator agent can expand. Paired with prior failure reasoning, which generates a preliminary diagnosis from test code and error descriptions, the folding step directly attacks the two named problems—observational noise and excessive length—by reducing what the LLM must attend to while keeping full information one API call away.

What would settle it

Take a failed trajectory, pick the annotated decisive step, replay the agent but with only that step's action swapped for the correct one while keeping all later steps exactly as actually executed, and check whether the task still fails; or ask two independent annotation teams to label the same 93 trajectories and see whether agreement approaches chance on trajectories where failure symptoms are spread across multiple steps. Either result would test whether the ground truth, and therefore the accuracy numbers, is meaningful.

Watch

Extended reading notes

Core claim

On its own terms, TrajAudit is an automated failure-diagnosis framework that identifies the first step in a failed agent trajectory at which the agent commits to a flawed direction—the 'earliest decisive error step'—and gives a natural-language justification. The method combines prior failure reasoning (an LLM prompted on the test code and error message to predict the likely failure phase), semantic saliency folding (compressing observations that contain neither code-diff patches nor failure keywords into a single placeholder token), and an investigator agent that probes folded observations through an API when context is insufficient. The paper also constructs RootSE, a benchmark of 93 real-

Load-bearing premise

The benchmark's ground truth is the 'earliest decisive error step'—defined counterfactually as a step such that, if corrected while all later steps run optimally, the task would succeed—and this assumes that a single well-defined decisive step exists in every failed trajectory and that annotators can reliably find it (Cohen's kappa 0.78 before arbitration).

Editorial extensions

If this is right

  • If the reported accuracy holds, TrajAudit can localize the decisive mistake in roughly half of long repository-level coding failures, a level that existing methods do not approach on the same benchmark.
  • The accuracy gap over baselines widens as trajectory length grows, reaching its largest margin on trajectories above 80 steps, which supports the claim that the compression-and-probe design specifically mitigates long-context degradation.
  • Token consumption is lower than all baselines—about 10 to 26 percent below the All-at-Once method and far below stepwise processing—so the approach is practical to run repeatedly on real agent logs.
  • The RootSE benchmark of 93 annotated instances with length and phase distributions provides a common testbed for future failure-diagnosis research on repository-level coding agents.
  • Because failures are annotated across five workflow phases, the framework yields phase-level insight, not just a step index, which can guide where to intervene in an agent's design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the folding logic is code-specific (patch headers and failure keywords); adapting TrajAudit to other agent domains such as web browsing or data-analysis agents would require redefining what counts as salient, but the active-probing architecture should transfer.
  • Editorial: the counterfactual ground truth assumes a single decisive step; if failures often arise from two interacting errors, the annotation itself may have no stable answer, and the 56.6% figure would be an upper bound on what a human can agree on.
  • Editorial: one testable extension is to use TrajAudit's diagnosis as feedback to the coding agent in a repair loop, measuring whether re-running the agent with the flagged step's context corrected increases task success rate.
  • Editorial: because the prior-failure-reasoning module already points to a phase, the framework could be extended to predict not just the step but a repair suggestion—connecting localization to automated fixing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses automated failure diagnosis for long, noisy trajectories generated by repository-level coding agents. It proposes TrajAudit, which combines semantic saliency folding, prior failure reasoning from test failure reports, and an investigator agent that can dynamically unfold compressed trajectory observations. The authors also introduce RootSE, a benchmark of real-world failed agent trajectories annotated with the earliest decisive error step and a justification. Experiments on RootSE report substantial gains over the strongest baseline (e.g., 56.6% vs. 31.9% exact step-level accuracy with reference, 50.9% vs. 26.9% without reference), lower token consumption, an ablation study, and robustness across backbone LLMs. The abstract, however, reports 102 instances and gains of 10.8%/21.6%, while the body reports 93 instances and gains of 24.7/24.0 percentage points.

Significance. If the empirical results hold, the paper makes a useful contribution: it identifies a genuine limitation of existing failure-diagnosis methods on long coding-agent trajectories, provides a new benchmark with richer annotations than prior work, and demonstrates a plausible mechanism for improving localization accuracy via context folding and prior reasoning. Strengths include the use of real trajectories from three agentic systems, a documented multi-stage annotation protocol with reported inter-rater agreement, an ablation of the two main components, token-efficiency analysis, and a public-release commitment. However, the central claim is an empirical effect size, and currently the numeric reporting is inconsistent, the counterfactual ground-truth definition is not behaviorally validated, and no uncertainty quantification is provided. The contribution is promising but needs substantial revision before acceptance.

major comments (4)
  1. [Abstract vs. §3 and Table 2] The abstract reports a benchmark of 102 instances and gains of 10.8%/21.6% in exact failure localization accuracy, whereas the body (§3, §6.1, Table 2) reports 93 instances and gains of 24.7/24.0 percentage points. These are materially different claims. Since the headline numbers are what readers and the community will rely on, the discrepancy must be reconciled and the correct values reported with the appropriate setting.
  2. [§3.3.1–3.3.2 and §3.2.3] The core metric depends on the 'Earliest Decisive Error Step,' defined counterfactually: a step whose correction, followed by optimal subsequent execution, would make the task succeed. This property is not observable from the trajectory, and §3.2.3 filters the benchmark to trajectories 'clearly attributed to a single decisive step.' The reported Cohen's kappa (0.78) shows inter-annotator agreement under a shared guideline, but it does not show that annotators are recovering a real, single decisive step. This is load-bearing because every headline accuracy (Table 2) is measured against these labels. Please provide behavioral validation (e.g., for a sample, correct the predicted step and verify success under optimal subsequent behavior, or have independent experts resolve the counterfactual from the trajectory), or explicitly bound the claim to 'steps that expert annotators judge decisive'
  3. [§5.5, Tables 2–6] All conclusions are based on three runs each, but no variance, confidence intervals, or significance tests are reported anywhere; Tables 2–7 give only point averages. Given known LLM nondeterminism (acknowledged in §8) and the moderate differences in RQ3, the 24.7/24.0-point gaps need uncertainty quantification (e.g., paired bootstrap or a randomization test) to establish that they are not within run-to-run noise. At minimum, report per-run standard deviations and the number of runs.
  4. [§4.2, Algorithm 1, §5.4] The semantic saliency folding module and evaluation protocol contain manually chosen components—threshold tau, keyword dictionary K, the five-phase taxonomy, and the LLM-as-a-judge rubric—all constructed on RootSE. The paper provides an ablation but no sensitivity analysis. Because the benchmark is also built by the authors, this raises the risk that reported gains partly reflect hand-tuning to RootSE. Please report how accuracy varies with tau and K (e.g., leave-one-out keyword removal or a range of tau), and consider a held-out trajectory source to test generalization.
minor comments (5)
  1. [§4.1] The text says a 'predefined set of five phases' but then lists only four (requirement analysis, planning, code implementation, and final verification), omitting exploration/localization used elsewhere. Please fix.
  2. [References] Reference [4] is a placeholder ('Anonymous. 2026. Reference. doi:...'). If this is the prompt/artifact reference, cite the actual repository or artifact.
  3. [Table 4] Table 4 reports per-length-level justification accuracy but not the number of instances per level; add counts so step-level percentages can be interpreted.
  4. [Table 5] Token consumption is reported only for exact successes; state how many instances that subset contains and whether selection bias affects the comparison.
  5. [§5.2, Table 2] The Random Attribution row reports a single 5.4% for both settings; explain whether this is the mean over random draws and why no setting split is given.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's contribution is empirical (a diagnosis framework plus a benchmark), and no fitted parameter is renamed as a prediction or derived from the ground-truth labels by construction.

full rationale

TrajAudit contains no mathematical derivation chain whose outputs could equal its inputs by construction. The target label, Earliest Decisive Error Step (§3.3.1), is also the evaluation metric, but that is a task definition, not a circular dependency: the method is not fitted to the human annotations, and the baseline methods are evaluated under the same definition. The authors do build RootSE and evaluate TrajAudit on it, but this is a generalizability/validity limitation rather than circularity unless specific components are shown to be tuned to the ground-truth labels; the paper does not report such tuning. The failure-keyword dictionary and phase taxonomy are manually constructed, but they are not claimed as predictions, and there is no exhibited reduction of the reported step-level accuracy to those design choices. The counterfactual nature of the ground truth is an annotation-validity concern, not evidence that TrajAudit's outputs are constructed from its inputs. The only self-citation, reference [32], supports background motivation and is not load-bearing. The abstract/body inconsistencies in instance counts and improvement percentages are factual inconsistencies, not circular steps. Overall, no significant circularity is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or ontological entities are postulated; all introduced components are software modules or agents described in the paper. The load-bearing assumptions are empirical claims about trajectory noise, long-context LLM behavior, and the annotatability of a counterfactual failure step.

free parameters (4)
  • tau (length threshold for folding) = not reported
    Algorithm 1 folds observations when length exceeds tau or when no keyword is present, but the value of tau is never given. It directly controls how much of the trajectory is compressed and therefore influences every TrajAudit result.
  • Failure keyword dictionary K = not fully specified
    Section 4.2 says K was 'constructed through LLM generation and manual refinement' and gives only examples. The exact dictionary determines which observations are preserved and which are folded, so the method cannot be replicated without it.
  • Five-phase workflow taxonomy = n/a
    The prior failure reasoning module uses a predefined phase set, but Section 4.1 lists five phases while naming only four. The mapping from trajectories to phases is not algorithmically specified.
  • LLM-as-a-judge rubric for justification accuracy = not reported
    Justification accuracy is checked by the same default LLM family used to produce diagnoses, but the judging prompt and equivalence rubric are not included. This hand-set evaluation threshold is a free choice that affects the reported justification scores.
assumptions (5)
  • domain assumption Keyword and patch-header patterns are a reliable proxy for failure-relevant trajectory content.
    Section 4.2 assumes that observations containing diff headers or words like 'exception' and 'fail' carry the failure signal, while other long observations can be folded without losing decisive information. If a decisive error appears only in verbose content without these markers, folding could hide it.
  • domain assumption LLM reasoning degrades on long contexts, so compressing and focusing improves diagnosis.
    This is a cited empirical claim from prior work (e.g., Liu et al., Chaudhury et al.) and motivates both modules. TrajAudit's design depends on it, but the paper does not independently verify it on RootSE.
  • domain assumption The Earliest Decisive Error Step is well-defined and annotatable via the counterfactual 'if corrected and subsequent steps optimal, the task succeeds'.
    Section 3.3.1 formalizes this definition. It assumes a unique earliest step exists in every failed trajectory and that human annotators can recover it; the reported Cohen's kappa of 0.78 indicates substantial but not perfect agreement.
  • domain assumption Test code and error description contain enough signal to produce a useful preliminary diagnosis.
    Section 4.1 feeds only test code and error description into the prior failure reasoning module. If the error message is misleading or uninformative, the prior could direct the investigator to the wrong phase.
  • ad hoc to paper The manually constructed keyword set and threshold tau are not overfit to RootSE.
    Section 4.2 and Algorithm 1 introduce these choices without sensitivity analysis. Since the method is evaluated only on RootSE, the reported accuracy may depend on tuning performed on that same benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems." pith.science (2026). https://pith.science/paper/SEHWXHFR

@misc{pith2026260526563,
  author       = {Pith},
  title        = {Pith review of: TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SEHWXHFR}},
  note         = {Machine review of arXiv:2605.26563}
}
read the original abstract

Agentic systems have been widely studied to automate coding tasks such as bug fixing and feature implementation. As these systems increasingly operate on complex codebases, understanding where and why they fail becomes essential for iterative refinement and operational reliability. Existing automated failure diagnosis approaches leverage \textit{task execution trajectories}, yet they struggle with trajectories produced by repository-level coding agents due to two key properties. First, these trajectories are often long, spanning many execution steps, making it difficult for LLMs to track the causal chain of failure over the execution history. Second, these trajectories are laden with noise, containing substantial low-signal observations such as redundant program structures and verbose code context, which can interfere with LLM reasoning. To address these challenges, we propose \textit{TrajAudit}, an automated failure diagnosis framework specifically for trajectories produced by repository-level coding agents. TrajAudit employs an investigator agent supported by two modules: one reduces failure-irrelevant noisy context through semantic saliency folding, and the other derives preliminary diagnostic guidance from test failure reports as prior knowledge to help LLMs focus on likely failure regions. The investigator agent can further invoke tools to inspect folded content on demand, enabling a focused investigation without losing access to the full trajectory context. We also introduce \textit{RootSE}, a benchmark of 102 real-world instances from repository-level coding tasks, each annotated with the earliest decisive error step and a justification. Experiments on RootSE show that TrajAudit outperforms the strongest baselines by 10.8\% and 21.6\% in exact failure localization accuracy in the with- and without-reference settings, respectively, demonstrating its effectiveness.

Figures

Figures reproduced from arXiv: 2605.26563 by the authors.

Figure 1
Figure 1. Failure diagnosis in agentic systems. refer to the information returned by tools invoked by the agent, often accounting for over 70% of the total trajectory content. How￾ever, most observations are not relevant to failure localization, such as redundant program structures and verbose code context, which can interfere with LLM reasoning [44]. (2) Excessive length. These trajectories often span from 20 to over 100 ste… view at source ↗
Figure 2
Figure 2. The agent workflow and execution trajectory in a coding task. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of baseline methods under varying tra [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: RootSE Annotation Guideline. to a single decisive step due to system limitations rather than to ambiguous task descriptions or misaligned test code. 3.3 Annotation Following prior work [57], we adopt the Earliest Decisive Error Step as the failure point definition for …
Figure 5
Figure 5. Figure 5: Phase-wise Failure Distribution in RootSE. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The overall workflow of TrajAudit by dynamically inspecting folded observations and probing for additional context when the compressed trajectory provides insuf￾ficient information [43]. Through the complementary strengths of targeted information extraction and active …
Figure 7
Figure 7. Figure 7: Exact Step-Level Accuracy across Varying Trajectory [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: A Worked Example of TrajAudit. most suspicious failure region. The diagnosis is based on the obser￾vation that the test code explicitly specifies the expected port, yet the generated patch does not address this inconsistency, suggesting that the agent failed to correct…

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AfterVibe: What Remains When the Conversation Ends

    cs.SE 2026-07 conditional novelty 7.5 of 10

    AfterVibe extracts natural-language specs from vibe-coding trajectories, validates them via blind regeneration scored by a three-tier verifier, and reaches mean scores of 5.06–5.74/6 on 72 industrial tasks.

  2. Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Across 1,794 CLI agent trajectories, failures are mostly epistemic, start by median step 7, and often stay silent until after lock-in.

  3. What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents

    cs.SE 2026-07 conditional novelty 6.0 of 10

    TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.

Reference graph

Works this paper leans on

60 extracted references · 9 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007. On the accuracy of spectrum-based fault localization. InTesting: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007). IEEE, 89–98

    2007

  2. [2]

    Elena Akik, Marko Vještica, Vladimir Dimitrieski, Slavica Kordić, and Sonja Ristić. 2025. Architecture of Multi-agent System for Automatic Code Template Maintenance. InEuropean Conference on Advances in Databases and Information Systems. Springer, 296–310

    2025

  3. [3]

    Stefano V Albrecht and Peter Stone. 2018. Autonomous agents modelling other agents: A comprehensive survey and open problems.Artificial Intelligence258 (2018), 66–95

    2018

  4. [4]

    2026.Reference

    Anonymous. 2026.Reference. doi:10.5281/zenodo.19230090

    doi:10.5281/zenodo.19230090 2026

  5. [5]

    Amine Barrak. 2025. Traceability and Accountability in Role-Specialized Multi- Agent LLM Pipelines. In2025 40th IEEE/ACM International Conference on Auto- mated Software Engineering Workshops (ASEW). IEEE, 315–322

    2025

  6. [6]

    Islem Bouzenia and Michael Pradel. 2025. Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2846– 2857

    2025

  7. [7]

    Subhajit Chaudhury, Payel Das, Sarathkrishna Swaminathan, Georgios Kollias, Elliot Nelson, Khushbu Pahwa, Tejaswini Pedapati, Igor Melnyk, and Matthew Riemer. 2025. EpMAN: Episodic Memory AttentioN for Generalizing to Longer Contexts. InProceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers). 11696–11708

    2025

  8. [8]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and psychological measurement20, 1 (1960), 37–46

    1960

Show all 60 references
  1. [9]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. 2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941(2025)

    2025 arXiv

  2. [10]

    Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kan- nappan, and Rebecca Qian. 2025. TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638 [cs.AI] https://arxiv.org/abs/2505.08638

    2025 arXiv

  3. [11]

    Liming Dong, Qinghua Lu, and Liming Zhu. 2024. Agentops: Enabling observ- ability of llm agents.arXiv preprint arXiv:2411.05285(2024)

    2024 arXiv

  4. [12]

    Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. InProceedings of the 2017 ACM SIGSAC conference on computer and communications security. 1285–1298

    2017

  5. [13]

    Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi. 2025. Interactive debugging and steering of multi- agent ai systems. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–15

    2025

  6. [14]

    Stan Franklin and Art Graesser. 1996. Is it an Agent, or just a Program?: A Taxonomy for Autonomous Agents. InInternational workshop on agent theories, architectures, and languages. Springer, 21–35

    1996

  7. [15]

    Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang. 2025. Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis.arXiv preprint arXiv:2509.13782(2025)

    2025

  8. [16]

    Haixuan Guo, Shuhan Yuan, and Xintao Wu. 2021. Logbert: Log anomaly detec- tion via bert. In2021 international joint conference on neural networks (IJCNN). IEEE, 1–8

    2021

  9. [17]

    Shanshan Han, Qifan Zhang, Weizhao Jin, and Zhaozhuo Xu. 2024. LLM multi- agent systems: Challenges and open problems.arXiv preprint arXiv:2402.03578 (2024)

    2024 arXiv

  10. [18]

    Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu. 2016. Experience re- port: System log analysis for anomaly detection. In2016 IEEE 27th international symposium on software reliability engineering (ISSRE). IEEE, 207–218

    2016

  11. [19]

    Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. [n. d.]. L2MAC: Large Language Model Automatic Computer for Extensive Code Generation. In The Twelfth International Conference on Learning Representations

  12. [20]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al

  13. [21]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  14. [22]

    Li Hu, Guoqiang Chen, Xiuwei Shang, Shaoyin Cheng, Benlong Wu, LiGangyang LiGangyang, Xu Zhu, Weiming Zhang, and Nenghai Yu. 2025. CompileAgent: Automated real-world repo-level compilation with tool-integrated LLM-based agent system. InProceedings of the 63rd Annual Meeting of...

    2025

  15. [23]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    2023 arXiv

  16. [24]

    James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the taran- tula automatic fault-localization technique. InProceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 273–282

    2005

  17. [25]

    Satyadhar Joshi. 2025. LLMOps, AgentOps, and MLOps for Generative AI: A Comprehensive Review. (2025)

    2025

  18. [26]

    Max Landauer, Sebastian Onder, Florian Skopik, and Markus Wurzenberger. 2023. Deep learning for anomaly detection in log data: A survey.Machine Learning with Applications12 (2023), 100470

    2023

  19. [27]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.biometrics(1977), 159–174

    1977

  20. [28]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society.Advances in neural information processing systems36 (2023), 51991–52008

    2023

  21. [29]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large language model-based agents for software en- gineering: A survey.ACM Transactions on Software Engineering and Methodology (2024)

    2024

  22. [30]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics12 (2024), 157–173

    2024

  23. [31]

    Tianyang Liu, Canwen Xu, and Julian McAuley. [n. d.]. RepoBench: Benchmark- ing Repository-Level Code Auto-Completion Systems. InThe Twelfth Interna- tional Conference on Learning Representations

  24. [32]

    Ruofan Lu, Yichen Li, and Yintong Huo. 2025. Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 3856– 3860

    2025

  25. [33]

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. 2025. Large language model agent: A survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460(2025)

    2025 arXiv

  26. [34]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. InThe Twelfth Inter- national Conference on Learning Representations

    2023

  27. [35]

    Niels Mündler, Mark N Müller, Jingxuan He, and Martin Vechev. 2024. Swt-bench: Testing and validating real-world bug-fixes with code agents.Advances in Neural Information Processing Systems37 (2024), 81857–81887

    2024

  28. [36]

    Eugene W Myers. 1986. An O (ND) difference algorithm and its variations. Algorithmica1, 1 (1986), 251–266

    1986

  29. [37]

    2023.OpenAI API

    OpenAI. 2023.OpenAI API. https://openai.com/blog/openai-api [Online; accessed 1 Aug 2023]

    2023

  30. [38]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation.ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–28

    2025

  31. [39]

    Melissa Z Pan, Mert Cemri, Lakshya A Agrawal, Shuyi Yang, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, et al. 2025. Why do multiagent systems fail?. InICLR 2025 Workshop on Building Trust in Language Models and Applications. ...

    2025

  32. [40]

    Chris Parnin and Alessandro Orso. 2011. Are automated debugging techniques actually helping programmers?. InProceedings of the 2011 international symposium on software testing and analysis. 199–209

    2011

  33. [41]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2024. Chatdev: Communicative agents for software development. InProceedings of the 62nd annual meeting of the association for computational linguistics (vol...

    2024

  34. [42]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789(2023)

    2023 arXiv

  35. [43]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems36 (2023), 68539–68551

    2023

  36. [44]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. InInternational Conference on Machine Learning. PMLR, 31210–31227

    2023

  37. [45]

    Yuan Tian and Tianyi Zhang. 2025. Selective Prompt Anchoring for Code Gener- ation. InInternational Conference on Machine Learning. PMLR, 59528–59551

    2025

  38. [46]

    Maria Trofimova, Anton Shevtsov, Badertdinov Ibragim, Konstantin Pyaev, Simon Karasik, and Alexander Golubev. 2025. OpenHands Trajectories with Qwen3- Coder-480B-A35B-Instruct.Nebius blog(2025)

    2025

  39. [47]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741(2024)

    2024 arXiv

  40. [48]

    Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. 2025. Agents in soft- ware engineering: Survey, landscape, and vision.Automated Software Engineering 32, 2 (2025), 70

    2025

  41. [49]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  42. [50]

    Mark Weiser. 1984. Program slicing.IEEE Transactions on software engineering4 (1984), 352–357

    1984

  43. [51]

    W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization.IEEE Transactions on Software Engineering 42, 8 (2016), 707–740

    2016

  44. [52]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. InFirst conference on language modeling

    2024

  45. [53]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1482–1494

    2023

  46. [54]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

    2024

  47. [55]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations

    2022

  48. [56]

    Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. 2025. AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?arXiv preprint arXiv:2509.03312(2025)

    2025 arXiv

  49. [57]

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. 2025. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. InForty-second Internati...

    2025

  50. [58]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- tocoderover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604

    2024

  51. [59]

    Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. 2025. GSM: How Do your LLMs Behave over Infinitely Increasing Reasoning Complex- ity and Context Length?. InForty-second International Conference on Machine Learning. Received 20 February 2007; revised 12 Mar...

    2025

  52. [2023]

    InThe twelfth international conference on learning representations

    MetaGPT: Meta programming for a multi-agent collaborative framework. InThe twelfth international conference on learning representations

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.