{"id":"73e9a7b5-f7f2-44e5-9f5e-fbb5da00882f","arxiv_id":"2605.26563","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TrajAudit pinpoints the earliest wrong step in long AI-coding-agent logs with 50.9% exact accuracy on the new RootSE benchmark, beating prior methods by about 24 percentage points.","lead":"This paper introduces TrajAudit, a system that automatically finds the first wrong step in long records of an AI coding agent's work. The authors also release RootSE, a benchmark of real failed coding-agent runs, and report that TrajAudit beats existing methods by around 24 percentage points while using fewer tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical claim hinges on a counterfactual ground truth that is never behaviorally validated; if annotators cannot reliably resolve the 'earliest decisive error step,' the 24.7-point accuracy gap is not a stable finding.","rationale":"The paper proposes a plausible system and a reusable benchmark, and the ablation study, three-backbone robustness check, and token-efficiency analysis are useful supporting evidence. However, the central empirical claim is only as strong as the benchmark labels. All headline numbers reduce to exact step-level agreement with a counterfactually defined label. The reader identified this as the weakest assumption, and I agree: the definition in Section 3.3.1 is not directly checkable from the trajectory, and the reported annotation reliability does not validate the counterfactual itself. The benchmark construction further selects for trajectories where a single decisive step is clear, which means the hard cases are excluded; this may make the benchmark easier, not harder, for all methods. The concrete re-annotation test would directly probe whether the labels are reproducible by independent annotators and whether disagreement would change the ranking. Other concerns—missing FAMAS/TRAIL baselines, inconsistent abstract/body numbers, and lack of significance tests—are real but secondary; they affect the strength of the comparison, not the meaningfulness of the measured quantity. For these reasons I do not move the reader's conditional verdict: the paper should be revised to validate the ground truth, reconcile the numbers, and add statistical rigor before the central claim can be accepted.","tokens_in":18527,"tokens_out":8855,"duration_ms":104341,"concrete_test":"Re-annotation audit: sample 30 RootSE instances; give three independent annotators the published guideline (Figure 4) plus the trajectory, test, and error information (without gold labels) and have them label the Earliest Decisive Error Step. Report exact-match agreement with the published ground truth and pairwise Cohen's kappa. If exact-match agreement falls below ~75% or kappa below 0.6, the step-level accuracy metric is too unstable to support the 24.7-point claim; if agreement is high, the concern is substantially mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RootSE's ground-truth label—Earliest Decisive Error Step (Section 3.3.1)—is defined counterfactually: step t is decisive if, with t corrected and all later steps executed optimally, the task would succeed. This counterfactual is not observable from the trajectory, and the paper provides no behavioral validation of it. Instead, reliability rests on two annotators reaching Cohen's kappa 0.78, with all remaining disagreements arbitrated to 100% consensus (Section 3.3.2). That procedure tests agreement among people who share the same guideline, not whether the label corresponds to a recoverable, single decisive step. Moreover, Section 3.2.3 explicitly filters out trajectories not 'clearly attributed to a single decisive step,' so the benchmark contains only the cases where the definition is easiest, and the reported exact step-level accuracies (Table 2) may be measuring how well TrajAudit matches annotator intuition rather than a meaningful property. If the counterfactual is not reliably resolvable—e.g., for interacting errors or an underspecified 'optimal strategy'—the 24.7-point gap is measuring noise. The paper also contains unresolved numeric contradictions between the abstract (102 instances, 10.8%/21.6%) and the body (93 instances, 24.4%), but the load-bearing issue is the unvalidated ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses automated failure diagnosis for long, noisy trajectories generated by repository-level coding agents. It proposes TrajAudit, which combines semantic saliency folding, prior failure reasoning from test failure reports, and an investigator agent that can dynamically unfold compressed trajectory observations. The authors also introduce RootSE, a benchmark of real-world failed agent trajectories annotated with the earliest decisive error step and a justification. Experiments on RootSE report substantial gains over the strongest baseline (e.g., 56.6% vs. 31.9% exact step-level accuracy with reference, 50.9% vs. 26.9% without reference), lower token consumption, an ablation study, and robustness across backbone LLMs. The abstract, however, reports 102 instances and gains of 10.8%/21.6%, while the body reports 93 instances and gains of 24.7/24.0 percentage points.","tokens_in":18841,"tokens_out":5686,"duration_ms":55332,"significance":"If the empirical results hold, the paper makes a useful contribution: it identifies a genuine limitation of existing failure-diagnosis methods on long coding-agent trajectories, provides a new benchmark with richer annotations than prior work, and demonstrates a plausible mechanism for improving localization accuracy via context folding and prior reasoning. Strengths include the use of real trajectories from three agentic systems, a documented multi-stage annotation protocol with reported inter-rater agreement, an ablation of the two main components, token-efficiency analysis, and a public-release commitment. However, the central claim is an empirical effect size, and currently the numeric reporting is inconsistent, the counterfactual ground-truth definition is not behaviorally validated, and no uncertainty quantification is provided. The contribution is promising but needs substantial revision before acceptance.","major_comments":[{"comment":"The abstract reports a benchmark of 102 instances and gains of 10.8%/21.6% in exact failure localization accuracy, whereas the body (§3, §6.1, Table 2) reports 93 instances and gains of 24.7/24.0 percentage points. These are materially different claims. Since the headline numbers are what readers and the community will rely on, the discrepancy must be reconciled and the correct values reported with the appropriate setting.","section":"Abstract vs. §3 and Table 2"},{"comment":"The core metric depends on the 'Earliest Decisive Error Step,' defined counterfactually: a step whose correction, followed by optimal subsequent execution, would make the task succeed. This property is not observable from the trajectory, and §3.2.3 filters the benchmark to trajectories 'clearly attributed to a single decisive step.' The reported Cohen's kappa (0.78) shows inter-annotator agreement under a shared guideline, but it does not show that annotators are recovering a real, single decisive step. This is load-bearing because every headline accuracy (Table 2) is measured against these labels. Please provide behavioral validation (e.g., for a sample, correct the predicted step and verify success under optimal subsequent behavior, or have independent experts resolve the counterfactual from the trajectory), or explicitly bound the claim to 'steps that expert annotators judge decisive'","section":"§3.3.1–3.3.2 and §3.2.3"},{"comment":"All conclusions are based on three runs each, but no variance, confidence intervals, or significance tests are reported anywhere; Tables 2–7 give only point averages. Given known LLM nondeterminism (acknowledged in §8) and the moderate differences in RQ3, the 24.7/24.0-point gaps need uncertainty quantification (e.g., paired bootstrap or a randomization test) to establish that they are not within run-to-run noise. At minimum, report per-run standard deviations and the number of runs.","section":"§5.5, Tables 2–6"},{"comment":"The semantic saliency folding module and evaluation protocol contain manually chosen components—threshold tau, keyword dictionary K, the five-phase taxonomy, and the LLM-as-a-judge rubric—all constructed on RootSE. The paper provides an ablation but no sensitivity analysis. Because the benchmark is also built by the authors, this raises the risk that reported gains partly reflect hand-tuning to RootSE. Please report how accuracy varies with tau and K (e.g., leave-one-out keyword removal or a range of tau), and consider a held-out trajectory source to test generalization.","section":"§4.2, Algorithm 1, §5.4"}],"minor_comments":[{"comment":"The text says a 'predefined set of five phases' but then lists only four (requirement analysis, planning, code implementation, and final verification), omitting exploration/localization used elsewhere. Please fix.","section":"§4.1"},{"comment":"Reference [4] is a placeholder ('Anonymous. 2026. Reference. doi:...'). If this is the prompt/artifact reference, cite the actual repository or artifact.","section":"References"},{"comment":"Table 4 reports per-length-level justification accuracy but not the number of instances per level; add counts so step-level percentages can be interpreted.","section":"Table 4"},{"comment":"Token consumption is reported only for exact successes; state how many instances that subset contains and whether selection bias affects the comparison.","section":"Table 5"},{"comment":"The Random Attribution row reports a single 5.4% for both settings; explain whether this is the mean over random draws and why no setting split is given.","section":"§5.2, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The core idea and the RootSE benchmark are worthwhile contributions. The main risk is that the benchmark labels and manually constructed design choices drive the reported gains. I would ask specifically for uncertainty quantification, reconciliation of the abstract/body numbers, and a direct validation or careful bounding of the counterfactual ground-truth definition. If the authors can provide those, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. TrajAudit is a sensible new combination of existing ideas — prior failure reasoning from test reports, semantic saliency folding of noisy observations, and an investigator agent that can unfold on demand — aimed at a real problem: localizing the first wrong decision in long repository-level coding agent trajectories. And RootSE, the benchmark of 93 failed trajectories, is genuinely the first of its kind for this setting.\n\nI think that's a real contribution. The annotation guideline is careful, the failure taxonomy is useful, and the authors are honest about filtering out ambiguous cases. If the code and data are actually released as stated, that's another plus.\n\nHowever, the empirical case is not as clean as the abstract implies. The abstract says 102 instances and 10.8%/21.6% gains; the body says 93 instances and 24.0/24.7 percentage-point gains. That's a numeric contradiction that has to be resolved. There are also no variance or significance tests for the three runs, and the most directly relevant prior work — FAMAS (Ge et al.) and TRAIL (Deshpande et al.) — is cited but not included as baselines. I don't think that's fatal, but it weakens the claim that TrajAudit is state-of-the-art.\n\nThe load-bearing issue is the ground truth. The Earliest Decisive Error Step is defined counterfactually: if this step were corrected and all later steps were optimal, the task would succeed. That's a reasonable idealized definition, and the 0.78 Cohen's kappa is respectable. But the paper doesn't behaviorally validate that annotators can actually recover this counterfactual. And the filtering step — keeping only trajectories where failure is 'clearly attributed to a single decisive step' — means the benchmark only contains the cases where the definition is easiest. So the 24-point gap may be a real effect, or it may partly reflect that the task is easier than the benchmark's stated difficulty.\n\nStill, the direction is plausible and the components make sense; the ablation shows both modules help. This is a paper worth engaging with, but it needs a serious revision to reconcile the numbers, add statistical rigor, include or justify excluding FAMAS and TRAIL, and ideally do some behavioral validation of the ground truth.\n\nI'd send it to peer review — it deserves referee time — but I'd expect heavy revision.","headline":"Useful benchmark and a plausible method, but the paper's own numbers contradict each other and the headline accuracy claims rest on an unvalidated counterfactual ground truth.","tokens_in":19342,"tokens_out":1625,"would_cite":true,"duration_ms":17170,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TrajAudit claims to pinpoint the earliest decisive error step in coding-agent trajectories, reaching 56.6% exact step-level accuracy versus 31.9% for the best baseline in the with-reference setting.","keywords":["agentic systems","failure diagnosis","trajectory analysis","fault localization","LLM agents","software maintenance","long-context reasoning","benchmark"],"falsifier":"Take a failed trajectory, pick the annotated decisive step, replay the agent but with only that step's action swapped for the correct one while keeping all later steps exactly as actually executed, and check whether the task still fails; or ask two independent annotation teams to label the same 93 trajectories and see whether agreement approaches chance on trajectories where failure symptoms are spread across multiple steps. Either result would test whether the ground truth, and therefore the accuracy numbers, is meaningful.","tokens_in":18373,"feed_emoji":"🐞","tokens_out":4533,"duration_ms":46810,"temperature":0.7,"pith_summary":"The paper argues that existing failure-diagnosis methods for LLM coding agents fail on long, noisy trajectories because they passively consume entire execution logs. TrajAudit instead uses a three-part agent: a prior-reasoning module reads the test error to form a preliminary diagnosis, a folding module compresses failure-irrelevant observations, and an investigator agent can unfold specific steps on demand. On RootSE—a new benchmark of 93 failed repository-level coding runs—TrajAudit localizes the ground-truth earliest decisive error step exactly in 56.6% of cases with a reference patch and 50.9% without, versus 31.9% and 26.9% for the best baseline. It also uses fewer tokens. The paper's goal, if correct, is a practical diagnostic tool and a reusable benchmark for agent failure analysis.","feed_headline":"TrajAudit pinpoints the decisive error step in coding-agent runs","feed_subtitle":"On a 93-case benchmark, it beats existing methods by 24 points and cuts token use, giving agent maintainers a concrete place to intervene.","key_machinery":"Semantic saliency folding: a compression rule that keeps only observations matching patch-header patterns (e.g., '--- a', '+++ b', '@@ -N,M +P,Q @@') or failure-keyword dictionaries (e.g., 'exception', 'traceback', 'fail'), replacing the rest with a placeholder that the investigator agent can expand. Paired with prior failure reasoning, which generates a preliminary diagnosis from test code and error descriptions, the folding step directly attacks the two named problems—observational noise and excessive length—by reducing what the LLM must attend to while keeping full information one API call away.","core_discovery":"On its own terms, TrajAudit is an automated failure-diagnosis framework that identifies the first step in a failed agent trajectory at which the agent commits to a flawed direction—the 'earliest decisive error step'—and gives a natural-language justification. The method combines prior failure reasoning (an LLM prompted on the test code and error message to predict the likely failure phase), semantic saliency folding (compressing observations that contain neither code-diff patches nor failure keywords into a single placeholder token), and an investigator agent that probes folded observations through an API when context is insufficient. The paper also constructs RootSE, a benchmark of 93 real-","pith_inferences":["Editorial: the folding logic is code-specific (patch headers and failure keywords); adapting TrajAudit to other agent domains such as web browsing or data-analysis agents would require redefining what counts as salient, but the active-probing architecture should transfer.","Editorial: the counterfactual ground truth assumes a single decisive step; if failures often arise from two interacting errors, the annotation itself may have no stable answer, and the 56.6% figure would be an upper bound on what a human can agree on.","Editorial: one testable extension is to use TrajAudit's diagnosis as feedback to the coding agent in a repair loop, measuring whether re-running the agent with the flagged step's context corrected increases task success rate.","Editorial: because the prior-failure-reasoning module already points to a phase, the framework could be extended to predict not just the step but a repair suggestion—connecting localization to automated fixing."],"forward_implications":["If the reported accuracy holds, TrajAudit can localize the decisive mistake in roughly half of long repository-level coding failures, a level that existing methods do not approach on the same benchmark.","The accuracy gap over baselines widens as trajectory length grows, reaching its largest margin on trajectories above 80 steps, which supports the claim that the compression-and-probe design specifically mitigates long-context degradation.","Token consumption is lower than all baselines—about 10 to 26 percent below the All-at-Once method and far below stepwise processing—so the approach is practical to run repeatedly on real agent logs.","The RootSE benchmark of 93 annotated instances with length and phase distributions provides a common testbed for future failure-diagnosis research on repository-level coding agents.","Because failures are annotated across five workflow phases, the framework yields phase-level insight, not just a step index, which can guide where to intervene in an agent's design."],"fun_headline_variants":["TrajAudit finds the first wrong move in coding agents","Pinpoint the decisive error: TrajAudit for coding agents","TrajAudit: where coding agents first go wrong","TrajAudit isolates the critical failure step in agent runs","TrajAudit diagnoses coding-agent failures at the decisive step"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's ground truth is the 'earliest decisive error step'—defined counterfactually as a step such that, if corrected while all later steps run optimally, the task would succeed—and this assumes that a single well-defined decisive step exists in every failed trajectory and that annotators can reliably find it (Cohen's kappa 0.78 before arbitration).","fun_headline_variants_meta":{"raw":{"variants":["TrajAudit finds the first wrong move in coding agents","Pinpoint the decisive error: TrajAudit for coding agents","TrajAudit: where coding agents first go wrong","TrajAudit isolates the critical failure step in agent runs","TrajAudit diagnoses coding-agent failures at the decisive step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2421,"prompt_tokens":829,"completion_tokens":1592,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1516}},"tokens_in":573,"tokens_out":1592,"duration_ms":12024,"temperature":1.0,"reasoning_tokens":1516,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:03:48.601834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a failed trajectory, pick the annotated decisive step, replay the agent but with only that step's action swapped for the correct one while keeping all later steps exactly as actually executed, and check whether the task still fails; or ask two independent annotation teams to label the same 93 trajectories and see whether agreement approaches chance on trajectories where failure symptoms are spread across multiple steps. Either result would test whether the ground truth, and therefore the accuracy numbers, is meaningful.","supporting_citations":[],"review_version":3}