Pith

open record

sign in

arxiv: 2607.06184 · v1 · pith:K32DAGUI · submitted 2026-07-07 · cs.SE

What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents

Rui Shu , Chun Yong Chong , Xin Zhou , Yun Peng , Zihan Wu , Xu Han , Zeyang Zhuang , Guowen Yuan
show 1 more author
Yuan Wang
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 13:57 UTCglm-5.2pith:K32DAGUIrecord.jsonopen to challenge →

classification cs.SE
keywords coding agentstrajectory analysissoftware engineeringLLM agentsanti-pattern detectionSWE-Benchresolve rateprocess diagnostics
0
0 comments X

The pith

Pass/fail hides how coding agents actually work—trajectory structure reveals it

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that resolve rate—the binary pass/fail metric used to rank coding agents—is a coarse signal that hides meaningful process differences between runs. Two agents can solve the same task, but one might reach a targeted fix in 10 steps while the other takes 49 steps with repeated failures and recovery. The authors build TraceProbe, a framework that normalizes raw agent trajectories into a canonical nine-type action taxonomy with deterministic effect labels (survived, failed, reverted, justified, off-anchor), then applies two rule-based modules: Insight, which detects single-trajectory anti-patterns like search loops and verification skips, and Converge, which aligns pairs of runs and classifies where their behavior diverges. Applying this to 2,500 trajectories from five production agent settings on SWE-Bench Verified, the paper establishes three findings. First, file-level comparison is too coarse to separate success from failure, but function-level selection and completion behavior localize where runs go wrong. Second, anti-pattern detectors act mainly as corpus-level difficulty clues rather than reliable failure predictors, with search loops being the most stable signal across benchmarks. Third, even resolved runs differ substantially in how quickly they reach relevant code and how much failed work they incur. The paper positions trajectory structure as auditable diagnostic context that complements—not replaces—resolve rate.

Core claim

The central discovery is that coding agent trajectories contain structured, comparable process evidence that resolve rate systematically discards. The paper shows that this evidence can be recovered deterministically through a canonical action taxonomy with effect labels, enabling three kinds of diagnosis: single-run anti-pattern detection (Insight), pairwise run alignment and divergence classification (Converge), and oracle-grounded milestone timing. The most concrete finding is that function-level divergence between failed and resolved runs on the same task appears early enough (median ~20 steps before failure) to guide inspection, and that search loops are the one anti-pattern whose fire-

What carries the argument

TraceProbe normalizes heterogeneous agent traces into a nine-type canonical action taxonomy (file read, file write, search, command, sub-agent spawn, plan, navigate, fetch, reason) with deterministic effect labels (survived, failed, reverted, justified, recorded, off-anchor, reasoning). Two rule-based modules operate on this substrate: Insight applies frozen-threshold predicates to single trajectories to detect named anti-patterns (search loops, re-read churn, tool oscillation, verification skips, etc.), while Converge performs longest-common-subsequence alignment between a compared run and a reference run, then classifies divergence spans across three layers: file selection, edit stability,

If this is right

  • Agent developers could use trajectory diagnostics to triage which resolved runs warrant review for wasted effort, even when the patch passes all tests.
  • Function-level divergence timing could serve as an early-warning signal in development environments, flagging runs that are heading toward failure before they complete.
  • The distinction between corpus-level difficulty clues and failure-specific signals suggests that benchmark leaderboards should report process profiles alongside resolve rates to avoid rewarding inefficient or fragile solving strategies.
  • The finding that scaffold-driven process differences are model-dependent (visible under GPT-5.4 but not under Opus 4.6) implies that scaffold comparisons should always report which model was held fixed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If trajectory structure diagnostics were integrated into continuous integration pipelines for agent development, they could automatically flag regressions in agent behavior (e.g., increased search loops or off-anchor reads) even when resolve rate holds steady, catching quality decay before it manifests as test failures.
  • The benchmark-specificity of detector thresholds suggests a natural learning problem: rather than manually freezing thresholds per benchmark, one could derive data-driven thresholds that calibrate to each corpus's difficulty distribution while preserving the deterministic predicate structure.
  • The separation between oracle-free structural detectors and oracle-grounded milestones implies a tiered deployment strategy: structural detectors can run on any agent trace without benchmark patches, making them immediately usable in production settings where ground-truth patches are unavailable.

Load-bearing premise

The detector thresholds (e.g., 'at least 10 consecutive search/read actions' for search loops, 'at least 3 reads within a 10-action window' for re-read churn) were calibrated on SWE-Bench Verified and then frozen. When transferred to SWE-Bench Pro, prevalence rates shift substantially and several detectors change direction entirely, meaning the specific numeric cutoffs are benchmark-calibrated and the central claims about which patterns are stable or failure-associated depend

What would settle it

If the canonical action taxonomy fails to normalize traces across different scaffolds in a way that preserves comparable semantics, or if the LCS alignment between runs is so sensitive to reference choice that divergence classifications are unreliable, the framework's core diagnostic value collapses. The paper's own transfer check on SWE-Bench Pro already shows that several detector thresholds do not generalize, weakening the portability claim.

Figures

Figures reproduced from arXiv: 2607.06184 by Chun Yong Chong, Guowen Yuan, Rui Shu, Xin Zhou, Xu Han, Yuan Wang, Yun Peng, Zeyang Zhuang, Zihan Wu.

Figure 1
Figure 1. Figure 1: In this example, Claude Code (Opus 4.6) and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: TRACEPROBE architecture. Raw trajectories are normalized into a shared canonical representation, then analyzed by INSIGHT for single-run patterns and by CONVERGE for comparison against another run chosen as the reference trajectory. TABLE I: Step-level label schema in TRACEPROBE. Deterministic canonical actions, effect labels, and workflow phases support the main analyses, while optional LLM semantic tags … view at source ↗
Figure 3
Figure 3. Figure 3: Step-count distributions by outcome. Split violins use [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Effect-label distribution by agent setting. Each stacked [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: First anchor write timing (M2) by setting. Points mark [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
read the original abstract

Coding agents are ranked almost entirely by resolve rate: whether their final patch passes the target tests. Yet two agents can reach the same outcome through very different processes, and a single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens. This process evidence lives in the trajectory, which records a run's searches, reads, edits, tool calls, validation, and reversions. However, raw traces are heterogeneous and hard to compare across runs. We present TraceProbe, a trajectory-diagnostic framework that recovers what resolve rate hides. TraceProbe normalizes each raw run into a canonical nine-type action taxonomy with deterministic effect labels, then applies two rule-based modules: Insight names single-trajectory anti-patterns adapted from established debugging practice (e.g., search loops, verification skips), while Converge aligns pairs of runs and classifies where their behavior diverges under controlled references. Applying TraceProbe to 2,500 trajectories from five production settings on SWE-Bench Verified, we find that (i) file choice is too coarse to separate success from failure, whereas function selection and completion behavior localize it; (ii) Insight anti-patterns act mainly as corpus-level difficulty clues, with search loops the most stable; and (iii) even resolved runs differ in how quickly they reach relevant code and how much failed work they incur. Trajectory structure thus adds auditable diagnostic context to outcomes by localizing inspection targets, suggesting failure hypotheses, and prioritizing runs for review.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 6 minor

Summary. The paper presents TraceProbe, a trajectory-diagnostic framework for coding agents that normalizes raw traces into a canonical nine-type action taxonomy with deterministic effect labels, then applies two rule-based modules: INSIGHT (single-trajectory anti-pattern detection) and CONVERGE (cross-trajectory alignment and divergence classification). The framework is applied to 2,500 trajectories from five production settings on SWE-Bench Verified, with a cross-benchmark transfer check on SWE-Bench Pro. The study addresses three diagnostic questions: which anti-patterns act as failure clues, whether effect labels reveal process differences beyond raw telemetry, and whether settings with similar resolve rates differ in process profiles. The central claim is that trajectory structure adds auditable diagnostic context to resolve rate by localizing inspection targets, suggesting failure hypotheses, and prioritizing runs for review. The methodology is clearly specified with frozen thresholds (Table II), same-task controls, same-model contrasts, and a cross-benchmark transfer check. Data and code are publicly released.

Significance. The paper makes a solid contribution to the growing area of coding-agent evaluation by proposing a deterministic, rule-based diagnostic framework that complements resolve rate rather than replacing it. Key strengths include: (1) the detector catalog (Table II) with explicitly frozen predicates and thresholds, making all measurements reproducible and auditable; (2) the separation of oracle-free structural detectors from anchor-grounded milestones and divergence evidence, which clarifies the epistemic status of each signal; (3) the cross-benchmark transfer check on SWE-Bench Pro that honestly reports which detectors generalize and which do not; (4) the reference-change sensitivity analysis (Table VII) showing that file-selection divergence is nearly reference-invariant; and (5) public release of data and code. The study design with same-scaffold and same-model contrasts is well-matched to the diagnostic claims. The paper is appropriately scoped as a diagnostic framework, not a causal oracle or remediation method.

major comments (1)
  1. §IV.A and §V.C (RQ3): The paper states that 750 additional trajectories were collected from a 50-task stratified subsample run three times under each setting to estimate within-task variance for process-profile metrics. However, no variance estimates are reported anywhere in §V (Results). Table VI reports per-setting medians for steps, survived %, failed %, off-anchor %, etc., but each task/setting pair in the main corpus is run only once. The RQ3 claim that settings with similar resolve rates differ in process profiles (e.g., OpenCode/Opus 4.6 vs. OpenCode/GLM-5.4, both ~71% resolve rate, but differing in failed % at 4.4 vs. 17.6) is load-bearing for the paper's central thesis that trajectory structure reveals what resolve rate hides. Without any variance estimate, the reader cannot assess whether the between-setting differences in Table VI are large relative to within-task run-to-run噪声
minor comments (6)
  1. Table III: 'No Formal Tail Validation' shows 63.2% prevalence in resolved runs vs. 58.2% in failed runs, meaning it is more common in resolved runs. The text in §V.A notes this ('no formal tail validation occurs more often in resolved runs'), but the table caption or a footnote could make this counterintuitive direction more prominent to avoid misreading.
  2. §III.D: The LCS alignment compatibility rules are described qualitatively but the exact scoring function (match score, mismatch penalty, gap penalty) is not specified. Adding these parameters would improve reproducibility, especially since the Needleman–Wunsch variant is mentioned as a robustness check but its parameters are also unspecified.
  3. The semantic layer (LLM-based tags) is used by two detectors in Table II (Phase Oscillation, Semantic Fruitless Exploration) and reported in Table III, but the paper does not report inter-rater or test-retest reliability for the LLM labeler. A brief note on label stability would strengthen the credibility of these exploratory detectors.
  4. Figure 1 references 'Opus 4.6' and 'GLM-5' as model identifiers. These appear to be fictionalized or future-dated model names. If these are anonymized or placeholder names, a footnote clarifying the naming convention would aid reproducibility.
  5. §V.C, Table VI: The 'Harmful ratio' metric is defined as 'the share of a run's non-reasoning actions that failed or were reverted,' but this definition appears only in the prose, not in Table II's catalog. Adding it to Table II would make the metric definition consistent with the other detectors.
  6. The paper would benefit from a brief discussion of computational cost: how long does TraceProbe take to process a single trajectory and to align a pair? This is relevant for practitioners considering adoption.

Simulated Author's Rebuttal

1 responses · 0 unresolved

The referee raises one major concern: the 750-trajectory repeated-run subsample (§IV.A) was collected to estimate within-task variance for process-profile metrics, but no variance estimates appear in §V (Results), leaving the reader unable to assess whether the between-setting differences in Table VI (e.g., OpenCode/Opus 4.4% failed vs. OpenCode/GPT 17.6% failed at similar ~71% resolve rates) are large relative to run-to-run noise. We agree this is a genuine gap between what the paper promises and what it reports, and we will revise to include the variance estimates.

read point-by-point responses

  1. Referee: §IV.A and §V.C (RQ3): The paper states that 750 additional trajectories were collected from a 50-task stratified subsample run three times under each setting to estimate within-task variance for process-profile metrics. However, no variance estimates are reported anywhere in §V (Results). Table VI reports per-setting medians for steps, survived %, failed %, off-anchor %, etc., but each task/setting pair in the main corpus is run only once. The RQ3 claim that settings with similar resolve rates differ in process profiles (e.g., OpenCode/Opus 4.6 vs. OpenCode/GLM-5.4, both ~71% resolve rate, but differing in failed % at 4.4 vs. 17.6) is load-bearing for the paper's central thesis that trajectory structure reveals what resolve rate hides. Without any variance estimate, the reader cannot assess whether the between-setting differences in Table VI are large relative to within-task run-to-run噪声

    Authors: The referee is correct. The paper states in §IV.A that 750 additional trajectories were collected from a 50-task stratified subsample run three times under each setting to estimate within-task variance for process-profile metrics, yet §V reports no variance estimates from this data. This is a genuine gap between the data collection described and the results presented, and it directly affects the interpretability of the RQ3 claim that settings with similar resolve rates differ in process profiles. We will address this in revision by adding a variance analysis from the repeated-run subsample to §V.C. Specifically, we will report within-task standard deviations or bootstrap confidence intervals for the key process-profile metrics in Table VI (at minimum: failed %, survived %, off-anchor %, and harmful ratio), and we will explicitly state whether the between-setting differences highlighted in RQ3 (e.g., OpenCode/Opus 4.4% failed vs. OpenCode/GPT 17.6% failed) exceed the within-task run-to-run variance. If some differences do not exceed the noise floor, we will say so and qualify the corresponding claims. We will also add a sentence to §IV.A clarifying that the repeated-run data is used in the revised §V.C variance analysis. We note that the repeated-run subsample covers 50 tasks across difficulty strata, so the variance estimates will be approximate rather than exhaustive; we will state this limitation explicitly. We do not believe the overall thesis is undermined—the deterministic representation, frozen detector catalog, same-task divergence localization, and cross-benchmark transfer check all stand independently of the variance question—but the referee is right that the specific RQ3 between-setting comparison needs variance context to be properly evaluated by the reader. revision: yes

Circularity Check

0 steps flagged

No circularity found: the framework's representation, detectors, and comparisons are deterministic mappings from observable trajectory events, with no self-citation chain or definitional reduction.

full rationale

The paper's derivation chain is self-contained and non-circular. (1) The canonical action representation (Table I) is a deterministic mapping from raw trajectory events via scaffold adapters—no fitting to outcomes. (2) Effect labels (SURVIVED, FAILED, REVERTED, etc.) are derived from observable state transitions, not from the target outcomes they are used to analyze. (3) INSIGHT detectors (Table II) are rule-based predicates with frozen thresholds; the paper transparently states thresholds were calibrated on SWE-Bench Verified and then frozen, and explicitly frames them as 'deterministic pattern definitions and step-level evidence for inspection, rather than a list of failure predictors' (§V.A). (4) CONVERGE uses LCS alignment between trajectories—a standard algorithm, not a self-defined construct. (5) Milestones use the benchmark's gold-patch anchor set, which is external to the paper. (6) The cross-benchmark transfer check (Table V) applies the same frozen detectors to SWE-Bench Pro, an external benchmark, providing genuine out-of-sample validation. (7) No self-citations appear in the reference list; the paper builds entirely on external prior work (SWE-Bench, AgentBoard, TrajEval, etc.). The skeptic's concern about the unreported 750 repeated-run trajectories is a validity/completeness issue (missing evidence for a variance claim), not circularity—the claim itself does not reduce to its inputs by construction. The threshold calibration on SWE-Bench Verified is acknowledged and partially validated via the Pro transfer check; while the thresholds are benchmark-calibrated, the resulting measurements are not tautological with the outcomes they describe.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or dimensions. The nine-type action taxonomy and the detector catalog are methodological constructs, not ontological commitments. The free parameters are detector thresholds calibrated on SWE-Bench Verified. The axioms are domain assumptions about benchmark validity and representation sufficiency, plus the ad-hoc choice of action taxonomy and alignment method.

free parameters (7)
  • Search Loop threshold = 10 consecutive SEARCH/READ actions
    Chosen on SWE-Bench Verified; the paper states thresholds were chosen there and frozen before evaluation.
  • Re-read Churn threshold = 3 reads within 10-action window
    Numeric threshold for the re-read churn detector, frozen before evaluation but calibrated on the target benchmark.
  • Tool Oscillation threshold = 2 READ-WRITE-READ cycles
    Threshold for the tool oscillation detector.
  • Off-anchor Exploration threshold = 0.85 off-anchor ratio
    Threshold for classifying off-anchor exploration divergence.
  • Rapid Rewrite step window = 3 steps
    Window within which a second write to the same file triggers rapid rewrite classification.
  • Phase Oscillation threshold = 3 transitions within 6-action window
    Threshold for the semantic phase oscillation detector.
  • Semantic Fruitless Exploration threshold = 5 code-read actions, 4 never reused
    Threshold for the semantic fruitless exploration detector.
axioms (5)
  • domain assumption SWE-Bench Verified gold-patch changed files define the correct anchor set for task relevance.
    §III.B: 'the modified files define the trajectory's anchor set, or gold-patch file set.' This is standard for SWE-Bench but assumes gold patches are the correct relevance oracle.
  • ad hoc to paper The nine-type action taxonomy (FILE READ, FILE WRITE, SEARCH, COMMAND, PLAN, NAVIGATE, FETCH, AGENT SPAWN, REASON) is sufficient to capture diagnostically relevant trajectory structure.
    §III.B, Table I. The taxonomy is proposed by the paper; its completeness is not independently validated.
  • ad hoc to paper LCS alignment between canonical actions is a meaningful basis for comparing agent trajectories.
    §III.D: Converge uses monotonic LCS alignment. The paper checks against Needleman-Wunsch but does not validate against human judgment of alignment quality.
  • domain assumption Deterministic effect labels (SURVIVED, FAILED, REVERTED, etc.) correctly capture observable action outcomes from trajectory data alone.
    §III.B: effect labels are derived from observed state transitions. This assumes the trajectory records sufficient information to determine outcomes deterministically.
  • domain assumption SWE-Bench Verified and SWE-Bench Pro Python tasks are representative enough for the framework's conclusions to be meaningful.
    §VI: 'Both benchmarks are Python issue to PR repair, so results may not transfer to other languages or task types.'

pith-pipeline@v1.1.0-glm · 23735 in / 3306 out tokens · 467354 ms · 2026-07-08T13:57:38.286341+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 65 canonical work pages · 23 internal anchors

  1. [1]

    Claude Code,

    Anthropic, “Claude Code,” https://claude.com/product/claude-code, 2025, agentic coding assistant by Anthropic, accessed 2026

    work page 2025

  2. [2]

    OpenAI, “Codex,” https://openai.com/codex/, 2025, cloud-based soft- ware engineering agent by OpenAI, accessed 2026

    work page 2025

  3. [3]

    OpenCode: An open-source AI coding agent,

    OpenCode Contributors, “OpenCode: An open-source AI coding agent,” https://opencode.ai/, 2025, open-source coding agent supporting multiple model backends, accessed 2026

    work page 2025

  4. [4]

    Utboost: Rigorous evaluation of coding agents on swe-bench,

    B. Yu, Y . Zhu, P. He, and D. Kang, “Utboost: Rigorous evaluation of coding agents on swe-bench,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Association for Computational Ling...

    work page doi:10.18653/v1/2025.acl-long.189 2025

  5. [5]

    Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study

    Y . Wang, M. Pradel, and Z. Liu, “Are ”solved issues” in swe-bench really solved correctly? an empirical study,”CoRR, vol. abs/2503.15223,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study

    [Online]. Available: https://doi.org/10.48550/arXiv.2503.15223

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.15223

  7. [7]

    The swe-bench illusion: When state-of-the-art llms remember instead of reason,

    S. Liang, S. Garg, and R. Z. Moghaddam, “The swe-bench illusion: When state-of-the-art llms remember instead of reason,”CoRR, vol. abs/2506.12286, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2506.12286

    work page arXiv 2025

  8. [8]

    Quantifying infrastructure noise in agentic coding evals,

    Anthropic, “Quantifying infrastructure noise in agentic coding evals,” https://www.anthropic.com/engineering/infrastructure-noise, 2026, an- thropic Engineering Blog

    work page 2026

  9. [9]

    Why SWE-bench Verified no longer measures frontier coding capabilities,

    OpenAI, “Why SWE-bench Verified no longer measures frontier coding capabilities,” https://openai.com/index/ why-we-no-longer-evaluate-swe-bench-verified/, 2025, openAI Blog

    work page 2025

  10. [10]

    Agentboard: An analytical evaluation board of multi-turn LLM agents,

    C. Ma, J. Zhang, Z. Zhu, C. Yang, Y . Yang, Y . Jin, Z. Lan, L. Kong, and J. He, “Agentboard: An analytical evaluation board of multi-turn LLM agents,” inAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Macke...

    work page 2024

  11. [11]

    Available: http://papers.nips.cc/paper files/paper/2024/ hash/877b40688e330a0e2a3fc24084208dfa-Abstract-Datasets and Benchmarks Track.html

    [Online]. Available: http://papers.nips.cc/paper files/paper/2024/ hash/877b40688e330a0e2a3fc24084208dfa-Abstract-Datasets and Benchmarks Track.html

    work page 2024

  12. [12]

    Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

    M. Kim, D. Wang, S. Cui, F. Farmahinifarahani, S. Garg, B. Ray, T. Y . Zhuo, R. Mukherjee, and V . Kumar, “TRAJEV AL: decomposing code agent trajectories for fine-grained diagnosis,”CoRR, vol. abs/2603.24631, 2026. [Online]. Available: https://doi.org/10.48550/ arXiv.2603.24631

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  13. [13]

    Process-Centric Analysis of Agentic Software Systems

    S. Liu, Y . Chen, R. Krishna, S. Sinha, J. Ganhotra, and R. Jabbarvand, “Process-centric analysis of agentic software systems,”CoRR, vol. abs/2512.02393, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2512.02393

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    Understanding code agent behaviour: An empirical study of success and failure trajectories,

    O. Majgaonkar, Z. Fei, X. Li, F. Sarro, and H. Ye, “Understanding code agent behaviour: An empirical study of success and failure trajectories,”CoRR, vol. abs/2511.00197, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2511.00197

    work page doi:10.48550/arxiv.2511.00197 2025

  15. [15]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66

    work page 2024

  16. [16]

    On randomness in agentic evals,

    B. H. Bjarnason, A. Silva, and M. Monperrus, “On randomness in agentic evals,”CoRR, vol. abs/2602.07150, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2602.07150

    work page doi:10.48550/arxiv.2602.07150 2026

  17. [17]

    Swe-bench goes live!

    L. Zhang, S. He, C. Zhang, Y . Kang, B. Li, C. Xie, J. Wang, M. Wang, Y . Huang, S. Fu, E. Nallipogu, Q. Lin, Y . Dang, S. Rajmohan, and D. Zhang, “Swe-bench goes live!” inAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diago, CA, USA, December 2-7, 2025 / Mexico City...

    work page 2025

  18. [18]

    Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents

    I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel, “Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,”CoRR, vol. abs/2505.20411,

    work page arXiv

  19. [19]

    Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents

    [Online]. Available: https://doi.org/10.48550/arXiv.2505.20411

    work page doi:10.48550/arxiv.2505.20411

  20. [20]

    SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

    X. Deng, J. Da, E. Pan, Y . Y . He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler, “Swe-bench pro: Can AI agents solve long-horizon software engineering tasks?”CoRR, vol. abs/2509.16941, 2025. [Online]. Available: https://doi.org/10...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2509.16941 2025

  21. [21]

    Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving

    D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li, S. Liu, Y . Xiao, L. Chen, Y . Zhang, J. Su, T. Liu, R. Long, K. Shen, and L. Xiang, “Multi-swe-bench: A multilingual benchmark for issue resolving,”CoRR, vol. abs/2504.02605, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2504.02605

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.02605 2025

  22. [22]

    SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents

    M. S. Rashid, C. Bock, Y . Zhuang, A. Buchholz, T. B. Esler, S. Valentin, L. Franceschi, M. Wistuba, P. T. Sivaprasad, W. J. Kim, A. Deoras, G. Zappella, and L. Callot, “Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents,”CoRR, vol. abs/2504.08703, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2504.08703

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.08703 2025

  23. [23]

    Swe-compass: Towards unified evaluation of agentic coding abilities for large language models,

    J. Xu, K. Deng, W. Li, S. Yu, H. Tang, H. Huang, Z. Lai, Z. Zhan, Y . Wu, C. Zhang, K. Lei, Y . Yao, X. Lei, W. Zhu, Z. Feng, H. Li, J. Xiong, D. Li, Z. Gao, K. Wu, W. Xiang, Z. Zhan, Y . Zhang, W. Gong, Z. Gao, G. Wang, Y . Xue, M. Li, M. Xie, X. Zhang, J. Wang, W. Zhuang, Z. Lin, H. Wang, Z. Zhang, Y . Zhang, H. Zhang, B. Chen, and J. Liu, “Swe-compass:...

    work page doi:10.48550/arxiv.2511.05459 2025

  24. [24]

    SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

    J. Chen, H. Huang, Y . Lyu, J. An, J. Shi, C. Yang, T. Zhang, H. Tian, Y . Li, Z. Li, X. Zhou, X. Hu, and D. Lo, “Secureagentbench: Benchmarking secure code generation under realistic vulnerability scenarios,”CoRR, vol. abs/2509.22097, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2509.22097

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2509.22097 2025

  25. [25]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons,...

    work page 2024

  26. [26]

    Openhands: An open platform for AI software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, and et al., “Openhands: An open platform for AI software developers as generalist agents,” in The Thirteenth International Conference on Learning Representations, ICLR 2...

    work page 2025

  27. [27]

    Agentless: Demystifying LLM-based Software Engineering Agents

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying llm-based software engineering agents,”CoRR, vol. abs/2407.01489,

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    Agentless: Demystifying LLM-based Software Engineering Agents

    [Online]. Available: https://doi.org/10.48550/arXiv.2407.01489

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.01489

  29. [29]

    Traject-bench:a trajectory-aware benchmark for evaluating agentic tool use,

    P. He, Z. Dai, B. He, H. Liu, X. Tang, H. Lu, J. Li, J. Ding, S. Mukherjee, S. Wang, Y . Xing, J. Tang, and B. Dumoulin, “Traject-bench:a trajectory-aware benchmark for evaluating agentic tool use,”CoRR, vol. abs/2510.04550, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.04550

    work page doi:10.48550/arxiv.2510.04550 2025

  30. [30]

    Process-level trajectory evaluation for environment configuration in software engineering agents,

    J. Kuang, Y . Li, X. Zhang, Y . Li, D. Yin, X. Sun, Y . Shen, and P. S. Yu, “Process-level trajectory evaluation for environment configuration in software engineering agents,”CoRR, vol. abs/2510.25694, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.25694

    work page doi:10.48550/arxiv.2510.25694 2025

  31. [31]

    Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure

    T. Mehtiyev and W. Assunc ¸ ˜ao, “Beyond resolution rates: Behavioral drivers of coding agent success and failure,”CoRR, vol. abs/2604.02547,

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [32]

    Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure

    [Online]. Available: https://doi.org/10.48550/arXiv.2604.02547

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.02547

  33. [33]

    Understanding software engineering agents: A study of thought-action-result trajectories,

    I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,” in40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 2025, pp. 2846–2857. [Online]. Available: https://doi.org/10.1109/ASE63991. 2025.00234

    work page doi:10.1109/ase63991 2025

  34. [34]

    Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study

    I. Ceka, S. Pujar, S. Ramji, L. Buratti, G. E. Kaiser, and B. Ray, “Understanding software engineering agents through the lens of traceability: An empirical study,”CoRR, vol. abs/2506.08311, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2506.08311

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.08311 2025

  35. [35]

    TRAIL: Trace Reasoning and Agentic Issue Localization

    D. Deshpande, V . Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian, “TRAIL: trace reasoning and agentic issue localization,” CoRR, vol. abs/2505.08638, 2025. [Online]. Available: https://doi.org/ 10.48550/arXiv.2505.08638

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.08638 2025

  36. [36]

    Interpretable anomaly detection in event sequences via sequence matching and visual comparison,

    S. Guo, Z. Jin, Q. Chen, D. Gotz, H. Zha, and N. Cao, “Interpretable anomaly detection in event sequences via sequence matching and visual comparison,”IEEE Trans. Vis. Comput. Graph., vol. 28, no. 12, pp. 4531–4545, 2022. [Online]. Available: https: //doi.org/10.1109/TVCG.2021.3093585

    work page doi:10.1109/tvcg.2021.3093585 2022

  37. [37]

    Agentrx: Diagnosing AI agent failures from execution trajectories,

    S. Barke, A. Goyal, A. Khare, A. Singh, S. Nath, and C. Bansal, “Agentrx: Diagnosing AI agent failures from execution trajectories,” CoRR, vol. abs/2602.02475, 2026. [Online]. Available: https://doi.org/ 10.48550/arXiv.2602.02475

    work page doi:10.48550/arxiv.2602.02475 2026

  38. [38]

    TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems

    M. Wang, X. Xie, and Y . Huo, “Trajaudit: Automated failure diagnosis for agentic coding systems,”CoRR, vol. abs/2605.26563, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2605.26563

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.26563 2026

  39. [39]

    Agentdiagnose: An open toolkit for diagnosing LLM agent trajectories,

    T. Ou, W. Guo, A. Gandhi, G. Neubig, and X. Yue, “Agentdiagnose: An open toolkit for diagnosing LLM agent trajectories,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025 - System Demonstrations, Suzhou, China, November 4-9, 2025, I. Habernal, P. Schulam, and J. Tiedemann, Eds. Association for Computation...

    work page doi:10.18653/v1/2025.emnlp-demos.15 2025

  40. [40]

    Which agent causes task failures and when? on automated failure attribution of LLM multi- agent systems,

    S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y . Chen, and Q. Wu, “Which agent causes task failures and when? on automated failure attribution of LLM multi- agent systems,” inForty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, ser. Proceedings of Machine Learning ...

    work page 2025

  41. [41]

    AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

    G. Zhang, J. Wang, J. Chen, W. Zhou, K. Wang, and S. Yan, “Agentracer: Who is inducing failure in the LLM agentic systems?” CoRR, vol. abs/2509.03312, 2025. [Online]. Available: https://doi.org/ 10.48550/arXiv.2509.03312

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2509.03312 2025

  42. [42]

    Where LLM agents fail and how they can learn from failures,

    K. Zhu, Z. Liu, B. Li, M. Tian, Y . Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhang, X. Ma, X. Yu, G. Ramesh, J. Wu, Z. Liu, P. Lu, J. Zou, and J. You, “Where LLM agents fail and how they can learn from failures,”CoRR, vol. abs/2509.25370, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2509.25370

    work page doi:10.48550/arxiv.2509.25370 2025

  43. [43]

    Why do multi-agent LLM systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. G. Parameswaran, D. Klein, K. Ramchandran, M. A. Zaharia, J. E. Gonzalez, and I. Stoica, “Why do multi-agent LLM systems fail?” inAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diago, C...

    work page 2025

  44. [44]

    Detecting silent failures in multi-agentic AI trajectories,

    D. Pathak, H. Kumar, A. Roy, F. George, M. Verma, and P. Moogi, “Detecting silent failures in multi-agentic AI trajectories,”CoRR, vol. abs/2511.04032, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2511.04032

    work page arXiv 2025

  45. [45]

    TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents

    V . Mittapalli, S. J. Dani, S. S. Pilli, S. Ansu, M. Teymoorianfard, F. Dernoncourt, H. Chen, Y . Wang, R. A. Rossi, and N. K. Ahmed, “Trace: Trajectory reasoning through adaptive cross-step evidence aggregation for llm agents,” 2026. [Online]. Available: https://arxiv.org/abs/2606.07054

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  46. [46]

    Agent trajectory explorer: Visualizing and providing feedback on agent trajectories,

    M. Desmond, J. Y . Lee, I. Ibrahim, J. M. Johnson, A. Sil, J. MacNair, and R. Puri, “Agent trajectory explorer: Visualizing and providing feedback on agent trajectories,” inThirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in...

    work page doi:10.1609/aaai.v39i28.35350 2025

  47. [47]

    SeaView: Software Engineering Agent Visual Interface for Enhanced Workflow

    T. Bula, S. Pujar, L. Buratti, M. A. Bornea, and A. Sil, “Seaview: Software engineering agent visual interface for enhanced workflow,”CoRR, vol. abs/2504.08696, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2504.08696

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.08696 2025

  48. [48]

    Illuminating LLM Coding Agents: Visual Analytics for Deeper Understanding and Enhancement

    J. Wang, Y . Chen, M. Pan, C. M. Yeh, and M. Das, “Illuminating LLM coding agents: Visual analytics for deeper understanding and enhancement,”CoRR, vol. abs/2508.12555, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2508.12555

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.12555 2025

  49. [49]

    CodeTracer: Towards Traceable Agent States

    H. Li, Y . Yao, L. Zhu, R. Feng, H. Ye, J. Wang, Y . He, P. Zou, L. Zhang, X. Lei, H. Huang, K. Deng, M. Sun, Z. Zhang, H. Ye, and J. Liu, “Codetracer: Towards traceable agent states,”CoRR, vol. abs/2604.11641, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2604.11641

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.11641 2026

  50. [50]

    Agentprm: Process reward models for LLM agents via step-wise promise and progress,

    Z. Xi, C. Liao, G. Li, Z. Zhang, W. Chen, B. Wang, S. Jin, Y . Zhou, J. Guan, W. Wu, T. Ji, T. Gui, Q. Zhang, and X. Huang, “Agentprm: Process reward models for LLM agents via step-wise promise and progress,” inProceedings of the ACM Web Conference 2026, WWW 2026, Dubai, United Arab Emirates, originally scheduled for April 13- 17, 2026, rescheduled for Ju...

    work page doi:10.1145/3774904.3792551 2026

  51. [51]

    SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution

    H. Wang, C. T. Leong, J. Wang, J. Wang, and W. Li, “SPA-RL: reinforcing LLM agents via stepwise progress attribution,”CoRR, vol. abs/2505.20732, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2505.20732

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  52. [52]

    When agents go astray: Course-correcting SWE agents with prms,

    S. Gandhi, J. Tsay, J. Ganhotra, K. Kate, and Y . Rizk, “When agents go astray: Course-correcting SWE agents with prms,”CoRR, vol. abs/2509.02360, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2509.02360

    work page arXiv 2025

  53. [53]

    Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials,

    Y . Xu, D. Lu, Z. Shen, J. Wang, Z. Wang, Y . Mao, C. Xiong, and T. Yu, “Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials,” inThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    work page 2025

  54. [54]

    [Online]

    OpenReview.net, 2025. [Online]. Available: https://openreview. net/forum?id=EEgYUccwsV

    work page 2025

  55. [55]

    Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills

    J. Ni, Y . Liu, X. Liu, Y . Sun, M. Zhou, P. Cheng, D. Wang, E. Zhao, X. Jiang, and G. Jiang, “Trace2skill: Distill trajectory-local lessons into transferable agent skills,”CoRR, vol. abs/2603.25158, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2603.25158

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2603.25158 2026

  56. [56]

    Trajectory-informed memory generation for self-improving agent systems,

    G. Fang, V . Isahagian, K. R. Jayaram, R. Kumar, V . Muthusamy, P. Oum, and G. Thomas, “Trajectory-informed memory generation for self-improving agent systems,”CoRR, vol. abs/2603.10600, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2603.10600

    work page doi:10.48550/arxiv.2603.10600 2026

  57. [57]

    Shepherd: Pattern- guided trajectory selection for coding agents on SWE-bench,

    A. Cuadron, A. Desai, L. G. Schroeder, X. Wang, W. Ma, D. Li, Y . Wang, I. Stoica, G. Neubig, and J. E. Gonzalez, “Shepherd: Pattern- guided trajectory selection for coding agents on SWE-bench,” 2026. [Online]. Available: https://openreview.net/forum?id=ZBOFr4ryBk

    work page 2026

  58. [58]

    Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

    J. Lin, S. Liu, C. Pan, L. Lin, S. Dou, X. Huang, H. Yan, Z. Han, and T. Gui, “Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses,”CoRR, vol. abs/2604.25850, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2604.25850

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.25850 2026

  59. [59]

    Fowler,Refactoring - Improving the Design of Existing Code, ser

    M. Fowler,Refactoring - Improving the Design of Existing Code, ser. Addison Wesley object technology series. Addison-Wesley, 1999. [Online]. Available: http://martinfowler.com/books/refactoring.html

    work page 1999

  60. [60]

    Refactoring software, architectures, and projects in crisis,

    W. J. Brown, R. C. Malveau, H. W. McCormick III, and T. J. Mowbray, “Refactoring software, architectures, and projects in crisis,”Google Scholar Google Scholar Digital Library Digital Library, 1998

    work page 1998

  61. [61]

    Social debt in software engineering: insights from industry,

    D. A. Tamburri, P. Kruchten, P. Lago, and H. van Vliet, “Social debt in software engineering: insights from industry,”J. Internet Serv. Appl., vol. 6, no. 1, pp. 10:1–10:17, 2015. [Online]. Available: https://doi.org/10.1186/s13174-015-0024-6

    work page doi:10.1186/s13174-015-0024-6 2015

  62. [62]

    In: 2023 IEEE/ACM 45th Inter- national Conference on Software Engineering (ICSE)

    A. Mailach and N. Siegmund, “Socio-technical anti-patterns in building ml-enabled software: Insights from leaders on the forefront,” in45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 690–702. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023. 00067

    work page doi:10.1109/icse48619.2023 2023

  63. [63]

    An empirical characterization of bad practices in continuous integration,

    F. Zampetti, C. Vassallo, S. Panichella, G. Canfora, H. C. Gall, and M. D. Penta, “An empirical characterization of bad practices in continuous integration,”Empir. Softw. Eng., vol. 25, no. 2, pp. 1095–1135, 2020. [Online]. Available: https://doi.org/10.1007/s10664-019-09785-8

    work page doi:10.1007/s10664-019-09785-8 2020

  64. [64]

    An empirical study on failures in automated issue solving,

    S. Liu, F. Liu, L. Li, X. Tan, Y . Zhu, X. Lian, and L. Zhang, “An empirical study on failures in automated issue solving,”CoRR, vol. abs/2509.13941, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2509.13941

    work page arXiv 2025

  65. [65]

    Introducing SWE-bench Verified,

    OpenAI, “Introducing SWE-bench Verified,” https://openai.com/index/ introducing-swe-bench-verified/, 2024, human-verified 500-task subset of SWE-bench; curated by OpenAI in collaboration with the SWE-bench authors

    work page 2024