Archer automates agentic code review for LLVM optimizations and reports finding semantic bugs in 21% of recent open PRs and 11% of closed PRs.
Process-Centric Analysis of Agentic Software Systems
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
Agentic systems are modern software systems: they consist of orchestrated modules, expose interfaces, and are deployed in software pipelines. Unlike conventional programs, their execution, i.e., trajectories, is inherently stochastic and adaptive to the problems they solve. Evaluation of such systems is often outcome-centric. This narrow focus overlooks detailed insights, failing to explain how agents reason, plan, act, or change their strategies. Inspired by the structured representation of conventional software systems as graphs, we introduce Graphectory to systematically encode the temporal and semantic relations in such systems. Using Graphectory, we automatically analyze 4000 trajectories of two dominant agentic programming workflows, SWE-agent and OpenHands, with four backbone Large Language Models (LLMs), attempting to resolve SWE-bench issues. Our automated analyses (completed within four minutes) reveal that: (1) agents using richer prompts or stronger LLMs exhibit more complex Graphectory, reflecting deeper exploration, broader context gathering, and more thorough validation; (2) agents' strategies vary with problem difficulty and the underlying LLM - for resolved issues, strategies often follow coherent localization-patching-validation steps, while unresolved ones exhibit chaotic or backtracking behaviors; and (3) even successful agentic systems often display inefficient processes. We also implement a novel technique for real-time construction and analysis of Graphectory and Langutory during agent execution to flag trajectory issues. Upon detecting such issues, the technique notifies the agent with a diagnostic message and, when applicable, rolls back the trajectory. Experiments show that online monitoring and interventions improve resolution rates by 6.9%-23.5% across models for problematic instances, while significantly shortening trajectories with near-zero overhead.
citation-role summary
background 3
citation-polarity summary
years
2026 7roles
background 3polarities
background 3representative citing papers
AgentSZZ is an LLM-agent framework that identifies bug-inducing commits with up to 27.2% higher F1 scores than prior methods by enabling adaptive exploration and causal tracing, especially for cross-file and ghost commits.
Large-scale trajectory analysis of 19 coding agents on 500 tasks finds that LLM choice drives outcomes more than framework design and that context-gathering plus validation behaviors improve success beyond task difficulty predictions.
TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.
TraceGraph constructs shared state graphs from multi-model trajectories to expose productive cores and trap regions, then uses them to diagnose navigation differences across benchmarks and to drive a recovery pipeline that improves SWE-bench resolved rates by 3-4 points on fired instances.
citing papers explorer
-
Archer: Towards Agentic Review for Compiler Optimizations
Archer automates agentic code review for LLVM optimizations and reports finding semantic bugs in 21% of recent open PRs and 11% of closed PRs.
-
AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits
AgentSZZ is an LLM-agent framework that identifies bug-inducing commits with up to 27.2% higher F1 scores than prior methods by enabling adaptive exploration and causal tracing, especially for cross-file and ghost commits.
-
Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
Large-scale trajectory analysis of 19 coding agents on 500 tasks finds that LLM choice drives outcomes more than framework design and that context-gathering plus validation behaviors improve success beyond task difficulty predictions.
-
What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents
TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.
-
TraceGraph: Shared Decision Landscapes for Diagnosing and Improving Agent Trajectories
TraceGraph constructs shared state graphs from multi-model trajectories to expose productive cores and trap regions, then uses them to diagnose navigation differences across benchmarks and to drive a recovery pipeline that improves SWE-bench resolved rates by 3-4 points on fired instances.
- From Plan to Action: How Well Do Agents Follow the Plan?
- ReCodeAgent: A Multi-agent Workflow for Language-Agnostic Translation and Validation of Large-Scale Repositories