Large-scale trajectory analysis of 19 coding agents on 500 tasks finds that LLM choice drives outcomes more than framework design and that context-gathering plus validation behaviors improve success beyond task difficulty predictions.
An empirical study on failures in automated issue solving,
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.SE 6roles
background 1polarities
background 1representative citing papers
TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.
A small SFT-trained critic provides intra-trajectory steering to frozen code agents, delivering +3 to +5 point gains on SWE-bench Verified at 30-92x lower cost than a strong teacher.
Graphectory turns stochastic agent trajectories into analyzable graphs, showing that stronger models and successful fixes follow coherent localization-validation steps while failures are chaotic, and online detection plus rollback improves resolution rates by 6.9-23.5%.
ContraFix uses contrastive runtime analysis plus a dual-track skill base to reach 92% resolution on SEC-Bench and 73.8% on PatchEval while improving semantic correctness of patches.
Agent-generated tests mainly act as observational feedback channels and do not meaningfully improve issue resolution success in current LLM software engineering agents.
citing papers explorer
-
Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
Large-scale trajectory analysis of 19 coding agents on 500 tasks finds that LLM choice drives outcomes more than framework design and that context-gathering plus validation behaviors improve success beyond task difficulty predictions.
-
What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents
TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.
-
Steer, Don't Solve: Training Small Critic Models for Large Code Agents
A small SFT-trained critic provides intra-trajectory steering to frozen code agents, delivering +3 to +5 point gains on SWE-bench Verified at 30-92x lower cost than a strong teacher.
-
Process-Centric Analysis of Agentic Software Systems
Graphectory turns stochastic agent trajectories into analyzable graphs, showing that stronger models and successful fixes follow coherent localization-validation steps while failures are chaotic, and online detection plus rollback improves resolution rates by 6.9-23.5%.
-
ContraFix: Skill-Enhanced Contrastive Runtime Analysis for Vulnerability Repair
ContraFix uses contrastive runtime analysis plus a dual-track skill base to reach 92% resolution on SEC-Bench and 73.8% on PatchEval while improving semantic correctness of patches.
-
Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
Agent-generated tests mainly act as observational feedback channels and do not meaningfully improve issue resolution success in current LLM software engineering agents.