{"id":"4be8c526-6bcc-4f3a-987c-a559f1f4b891","arxiv_id":"2607.06184","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"TraceProbe normalizes coding agent trajectories into canonical actions and applies rule-based detectors to localize failure patterns and behavioral divergences that resolve rate hides.","lead":"This paper introduces TraceProbe, a framework that analyzes coding agent trajectories to reveal process differences hidden by resolve rate. A smart generalist might read it to understand how to audit and compare AI coding agents beyond pass/fail outcomes.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The paper collects 750 repeated-run trajectories to estimate within-task variance but never reports the results, leaving the RQ3 claim that settings with similar resolve rates differ in process profiles unvalidated against run-to-run noise.","rationale":"The reader correctly identifies threshold generalization as a concern, and the paper handles it with reasonable honesty — the SWE-Bench Pro transfer check (Table V) provides partial evidence, and the paper explicitly recommends threshold audits. My concern is different and more internal: the paper collected 750 trajectories specifically to estimate run-to-run variance for process-profile metrics but never reports the results. This directly affects whether the RQ3 setting-level process differences (Table VI) are real or within noise. However, I recommend UNCHANGED rather than a harsher verdict because: (1) the paper is already CONDITIONAL, which is appropriate; (2) the paper's claims are framed as diagnostic/observational rather than causal, which provides some buffer; (3) the framework's value as an inspection tool does not fully depend on the process differences being systematic — even noisy process profiles can guide inspection; (4) the data and code are released, so the variance analysis can be run by others. The condition the reader identified (threshold generalization and human validation) should be supplemented with: 'report the within-task variance analysis from the already-collected 750-trajectory repeated-run data.' The paper would be substantially strengthened by adding a paragraph or table reporting these variance estimates, and if the between-setting differences survive the variance check, the RQ3 claim becomes much more convincing. If they do not survive, the paper should reframe RQ3 as 'observed process profiles differ' rather than 'settings differ in process profiles.'","tokens_in":20801,"tokens_out":4998,"duration_ms":383830,"concrete_test":"Compute within-task coefficient of variation (CV) for each process-profile metric in Table VI (steps, survived %, failed %, off-anchor %, harmful ratio) using the 750-trajectory repeated-run data (50 tasks × 5 settings × 3 runs). For each metric, check whether the between-setting differences in Table VI exceed the within-setting variance bands (e.g., whether the median difference between any two settings exceeds 2× the pooled within-task standard deviation). If the CV for key metrics like failed % or off-anchor % exceeds ~30% and between-setting differences fall within 1-2 SD of within-task noise, the RQ3 claim that process profiles systematically differ across settings weakens substantially. Report these variance estimates in a supplementary table.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'trajectory structure adds auditable diagnostic context' rests on showing that process profiles differ meaningfully across agent settings, even when resolve rates are similar. The RQ3 results (Table VI) report per-setting medians for steps, survived %, failed %, off-anchor %, etc., but each task/setting pair is run only once in the main corpus. The paper acknowledges this threat in §IV.A: 'To measure run-to-run variability, we repeat a 50-task stratified random subsample three independent times under each setting, producing 750 additional trajectories.' This is the right experimental design. The problem is that the results of this variance analysis are never reported in the paper. There is no table, figure, or sentence in §V (Results) that reports within-task variance estimates for any process-profile metric. The paper asserts that the repeated-run check 'keeps the main results from depending on single-trace attribution' but provides no evidence for this claim. This is load-bearing because: (1) run-to-run variance in LLM coding agents is a documented concern (the paper itself cites refs [7], [8], [14] on infrastructure noise and run-to-run variability); (2) if the within-setting, within-task coefficient of variation for metrics like survived % or failed % is large relative to the between-setting differences in Table VI, then the observed process profile differences could be noise rather than systematic setting effects; (3) the paper's own RQ3 claim — 'settings with similar resolve rates can still differ in durable edits, failed work, and task-scope focus' — requires that these differences exceed run-to-run noise. The data to check this was collected but the analysis is absent from the paper. The reader's concern about threshold generalization across benchmarks is valid but is already partially addressed by the SWE-Bench Pro transfer check (Table V) and the paper's explicit scoping. The unreported variance data is more directly load-bearing bec","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper presents TraceProbe, a trajectory-diagnostic framework for coding agents that normalizes raw traces into a canonical nine-type action taxonomy with deterministic effect labels, then applies two rule-based modules: INSIGHT (single-trajectory anti-pattern detection) and CONVERGE (cross-trajectory alignment and divergence classification). The framework is applied to 2,500 trajectories from five production settings on SWE-Bench Verified, with a cross-benchmark transfer check on SWE-Bench Pro. The study addresses three diagnostic questions: which anti-patterns act as failure clues, whether effect labels reveal process differences beyond raw telemetry, and whether settings with similar resolve rates differ in process profiles. The central claim is that trajectory structure adds auditable diagnostic context to resolve rate by localizing inspection targets, suggesting failure hypotheses, and prioritizing runs for review. The methodology is clearly specified with frozen thresholds (Table II), same-task controls, same-model contrasts, and a cross-benchmark transfer check. Data and code are publicly released.","tokens_in":21579,"tokens_out":1220,"duration_ms":220890,"significance":"The paper makes a solid contribution to the growing area of coding-agent evaluation by proposing a deterministic, rule-based diagnostic framework that complements resolve rate rather than replacing it. Key strengths include: (1) the detector catalog (Table II) with explicitly frozen predicates and thresholds, making all measurements reproducible and auditable; (2) the separation of oracle-free structural detectors from anchor-grounded milestones and divergence evidence, which clarifies the epistemic status of each signal; (3) the cross-benchmark transfer check on SWE-Bench Pro that honestly reports which detectors generalize and which do not; (4) the reference-change sensitivity analysis (Table VII) showing that file-selection divergence is nearly reference-invariant; and (5) public release of data and code. The study design with same-scaffold and same-model contrasts is well-matched to the diagnostic claims. The paper is appropriately scoped as a diagnostic framework, not a causal oracle or remediation method.","major_comments":[{"comment":"§IV.A and §V.C (RQ3): The paper states that 750 additional trajectories were collected from a 50-task stratified subsample run three times under each setting to estimate within-task variance for process-profile metrics. However, no variance estimates are reported anywhere in §V (Results). Table VI reports per-setting medians for steps, survived %, failed %, off-anchor %, etc., but each task/setting pair in the main corpus is run only once. The RQ3 claim that settings with similar resolve rates differ in process profiles (e.g., OpenCode/Opus 4.6 vs. OpenCode/GLM-5.4, both ~71% resolve rate, but differing in failed % at 4.4 vs. 17.6) is load-bearing for the paper's central thesis that trajectory structure reveals what resolve rate hides. Without any variance estimate, the reader cannot assess whether the between-setting differences in Table VI are large relative to within-task run-to-run噪声","section":null}],"minor_comments":[{"comment":"Table III: 'No Formal Tail Validation' shows 63.2% prevalence in resolved runs vs. 58.2% in failed runs, meaning it is more common in resolved runs. The text in §V.A notes this ('no formal tail validation occurs more often in resolved runs'), but the table caption or a footnote could make this counterintuitive direction more prominent to avoid misreading.","section":null},{"comment":"§III.D: The LCS alignment compatibility rules are described qualitatively but the exact scoring function (match score, mismatch penalty, gap penalty) is not specified. Adding these parameters would improve reproducibility, especially since the Needleman–Wunsch variant is mentioned as a robustness check but its parameters are also unspecified.","section":null},{"comment":"The semantic layer (LLM-based tags) is used by two detectors in Table II (Phase Oscillation, Semantic Fruitless Exploration) and reported in Table III, but the paper does not report inter-rater or test-retest reliability for the LLM labeler. A brief note on label stability would strengthen the credibility of these exploratory detectors.","section":null},{"comment":"Figure 1 references 'Opus 4.6' and 'GLM-5' as model identifiers. These appear to be fictionalized or future-dated model names. If these are anonymized or placeholder names, a footnote clarifying the naming convention would aid reproducibility.","section":null},{"comment":"§V.C, Table VI: The 'Harmful ratio' metric is defined as 'the share of a run's non-reasoning actions that failed or were reverted,' but this definition appears only in the prose, not in Table II's catalog. Adding it to Table II would make the metric definition consistent with the other detectors.","section":null},{"comment":"The paper would benefit from a brief discussion of computational cost: how long does TraceProbe take to process a single trajectory and to align a pair? This is relevant for practitioners considering adoption.","section":null}],"recommendation":"major_revision","confidential_remarks":"The missing variance analysis is the primary concern. The experimental design is correct — the 750-run subsample is the right approach — but the results must be reported. If the variance turns out to be small relative to between-setting differences, this strengthens the paper considerably. If it is large, the RQ3 claims need qualification. Either way, the paper cannot ship without this analysis given that it is explicitly promised in §IV.A and is load-bearing for RQ3. The reader's report and stress-test note both correctly identify this as the central issue. I would also note that the threshold calibration concern (raised by the reader) is real but is already partially addressed by the SWE-Bench Pro transfer check and the honest discussion in §VI. The paper's scoping is appropriate — it does not overclaim — and the framework is a genuine contribution to the evaluation toolkit."},"author_rebuttal":{"model":"glm-5.2","summary":"The referee raises one major concern: the 750-trajectory repeated-run subsample (§IV.A) was collected to estimate within-task variance for process-profile metrics, but no variance estimates appear in §V (Results), leaving the reader unable to assess whether the between-setting differences in Table VI (e.g., OpenCode/Opus 4.4% failed vs. OpenCode/GPT 17.6% failed at similar ~71% resolve rates) are large relative to run-to-run noise. We agree this is a genuine gap between what the paper promises and what it reports, and we will revise to include the variance estimates.","responses":[{"response":"The referee is correct. The paper states in §IV.A that 750 additional trajectories were collected from a 50-task stratified subsample run three times under each setting to estimate within-task variance for process-profile metrics, yet §V reports no variance estimates from this data. This is a genuine gap between the data collection described and the results presented, and it directly affects the interpretability of the RQ3 claim that settings with similar resolve rates differ in process profiles. We will address this in revision by adding a variance analysis from the repeated-run subsample to §V.C. Specifically, we will report within-task standard deviations or bootstrap confidence intervals for the key process-profile metrics in Table VI (at minimum: failed %, survived %, off-anchor %, and harmful ratio), and we will explicitly state whether the between-setting differences highlighted in RQ3 (e.g., OpenCode/Opus 4.4% failed vs. OpenCode/GPT 17.6% failed) exceed the within-task run-to-run variance. If some differences do not exceed the noise floor, we will say so and qualify the corresponding claims. We will also add a sentence to §IV.A clarifying that the repeated-run data is used in the revised §V.C variance analysis. We note that the repeated-run subsample covers 50 tasks across difficulty strata, so the variance estimates will be approximate rather than exhaustive; we will state this limitation explicitly. We do not believe the overall thesis is undermined—the deterministic representation, frozen detector catalog, same-task divergence localization, and cross-benchmark transfer check all stand independently of the variance question—but the referee is right that the specific RQ3 between-setting comparison needs variance context to be properly evaluated by the reader.","revision_made":"yes","referee_comment":"§IV.A and §V.C (RQ3): The paper states that 750 additional trajectories were collected from a 50-task stratified subsample run three times under each setting to estimate within-task variance for process-profile metrics. However, no variance estimates are reported anywhere in §V (Results). Table VI reports per-setting medians for steps, survived %, failed %, off-anchor %, etc., but each task/setting pair in the main corpus is run only once. The RQ3 claim that settings with similar resolve rates differ in process profiles (e.g., OpenCode/Opus 4.6 vs. OpenCode/GLM-5.4, both ~71% resolve rate, but differing in failed % at 4.4 vs. 17.6) is load-bearing for the paper's central thesis that trajectory structure reveals what resolve rate hides. Without any variance estimate, the reader cannot assess whether the between-setting differences in Table VI are large relative to within-task run-to-run噪声"}],"tokens_in":20547,"tokens_out":712,"duration_ms":117401,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper introduces TraceProbe, a deterministic trajectory-diagnostic framework for coding agents, and it's a solid piece of work with one load-bearing gap. The framework normalizes raw agent traces into a nine-type canonical action taxonomy with deterministic effect labels, then applies two rule-based modules — INSIGHT for single-trajectory anti-patterns and CONVERGE for cross-trajectory divergence via LCS alignment. The detector catalog (Table II) has frozen predicates with explicit thresholds, and the paper ships code and data. This is genuinely useful infrastructure for agent developers who need to understand why runs fail, not just whether they pass.","headline":"Useful diagnostic framework with honest scoping, but the unreported variance data is a real gap that needs addressing before the central RQ3 claim holds.","tokens_in":21622,"tokens_out":712,"would_cite":false,"duration_ms":45852,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Pass/fail hides how coding agents actually work—trajectory structure reveals it","keywords":["coding agents","trajectory analysis","software engineering","LLM agents","anti-pattern detection","SWE-Bench","resolve rate","process diagnostics"],"falsifier":"If the canonical action taxonomy fails to normalize traces across different scaffolds in a way that preserves comparable semantics, or if the LCS alignment between runs is so sensitive to reference choice that divergence classifications are unreliable, the framework's core diagnostic value collapses. The paper's own transfer check on SWE-Bench Pro already shows that several detector thresholds do not generalize, weakening the portability claim.","tokens_in":21052,"feed_emoji":"🔍","tokens_out":1183,"duration_ms":241001,"temperature":0.7,"pith_summary":"This paper argues that resolve rate—the binary pass/fail metric used to rank coding agents—is a coarse signal that hides meaningful process differences between runs. Two agents can solve the same task, but one might reach a targeted fix in 10 steps while the other takes 49 steps with repeated failures and recovery. The authors build TraceProbe, a framework that normalizes raw agent trajectories into a canonical nine-type action taxonomy with deterministic effect labels (survived, failed, reverted, justified, off-anchor), then applies two rule-based modules: Insight, which detects single-trajectory anti-patterns like search loops and verification skips, and Converge, which aligns pairs of runs and classifies where their behavior diverges. Applying this to 2,500 trajectories from five production agent settings on SWE-Bench Verified, the paper establishes three findings. First, file-level comparison is too coarse to separate success from failure, but function-level selection and completion behavior localize where runs go wrong. Second, anti-pattern detectors act mainly as corpus-level difficulty clues rather than reliable failure predictors, with search loops being the most stable signal across benchmarks. Third, even resolved runs differ substantially in how quickly they reach relevant code and how much failed work they incur. The paper positions trajectory structure as auditable diagnostic context that complements—not replaces—resolve rate.","feed_headline":"Pass/fail hides how coding agents actually work—trajectory structure reveals it","feed_subtitle":"A diagnostic framework normalizes 2,500 agent runs into comparable action sequences, showing that resolved tasks can still differ sharply in","key_machinery":"TraceProbe normalizes heterogeneous agent traces into a nine-type canonical action taxonomy (file read, file write, search, command, sub-agent spawn, plan, navigate, fetch, reason) with deterministic effect labels (survived, failed, reverted, justified, recorded, off-anchor, reasoning). Two rule-based modules operate on this substrate: Insight applies frozen-threshold predicates to single trajectories to detect named anti-patterns (search loops, re-read churn, tool oscillation, verification skips, etc.), while Converge performs longest-common-subsequence alignment between a compared run and a reference run, then classifies divergence spans across three layers: file selection, edit stability,","core_discovery":"The central discovery is that coding agent trajectories contain structured, comparable process evidence that resolve rate systematically discards. The paper shows that this evidence can be recovered deterministically through a canonical action taxonomy with effect labels, enabling three kinds of diagnosis: single-run anti-pattern detection (Insight), pairwise run alignment and divergence classification (Converge), and oracle-grounded milestone timing. The most concrete finding is that function-level divergence between failed and resolved runs on the same task appears early enough (median ~20 steps before failure) to guide inspection, and that search loops are the one anti-pattern whose fire-","pith_inferences":["If trajectory structure diagnostics were integrated into continuous integration pipelines for agent development, they could automatically flag regressions in agent behavior (e.g., increased search loops or off-anchor reads) even when resolve rate holds steady, catching quality decay before it manifests as test failures.","The benchmark-specificity of detector thresholds suggests a natural learning problem: rather than manually freezing thresholds per benchmark, one could derive data-driven thresholds that calibrate to each corpus's difficulty distribution while preserving the deterministic predicate structure.","The separation between oracle-free structural detectors and oracle-grounded milestones implies a tiered deployment strategy: structural detectors can run on any agent trace without benchmark patches, making them immediately usable in production settings where ground-truth patches are unavailable."],"forward_implications":["Agent developers could use trajectory diagnostics to triage which resolved runs warrant review for wasted effort, even when the patch passes all tests.","Function-level divergence timing could serve as an early-warning signal in development environments, flagging runs that are heading toward failure before they complete.","The distinction between corpus-level difficulty clues and failure-specific signals suggests that benchmark leaderboards should report process profiles alongside resolve rates to avoid rewarding inefficient or fragile solving strategies.","The finding that scaffold-driven process differences are model-dependent (visible under GPT-5.4 but not under Opus 4.6) implies that scaffold comparisons should always report which model was held fixed."],"fun_headline_variants":["Resolve rate hides coding agent failures—trajectory structure finds them","What pass/fail misses: comparable diagnostics from agent trajectories","Function-level divergence separates failed agent runs 20 steps early","2,500 agent runs show resolved tasks still differ in cost and behavior","TraceProbe recovers process evidence resolve rate discards"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The detector thresholds (e.g., 'at least 10 consecutive search/read actions' for search loops, 'at least 3 reads within a 10-action window' for re-read churn) were calibrated on SWE-Bench Verified and then frozen. When transferred to SWE-Bench Pro, prevalence rates shift substantially and several detectors change direction entirely, meaning the specific numeric cutoffs are benchmark-calibrated and the central claims about which patterns are stable or failure-associated depend","fun_headline_variants_meta":{"raw":{"variants":["Resolve rate hides coding agent failures—trajectory structure finds them","What pass/fail misses: comparable diagnostics from agent trajectories","Function-level divergence separates failed agent runs 20 steps early","2,500 agent runs show resolved tasks still differ in cost and behavior","TraceProbe recovers process evidence resolve rate discards","Search loops are the one anti-pattern that tracks agent run difficulty","File choice can't separate agent success from failure—function selection can","Trajectory structure turns agent traces into auditable diagnostics","Pass/fail outcomes hide how much failed work resolved agent runs incur","Canonical action taxonomy makes 2,500 coding agent runs comparable"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":777,"prompt_tokens":589,"completion_tokens":188,"prompt_tokens_details":null},"tokens_in":589,"tokens_out":188,"duration_ms":17997,"temperature":1.0,"reasoning_tokens":36,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T13:57:38.286341+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the canonical action taxonomy fails to normalize traces across different scaffolds in a way that preserves comparable semantics, or if the LCS alignment between runs is so sensitive to reference choice that divergence classifications are unreliable, the framework's core diagnostic value collapses. The paper's own transfer check on SWE-Bench Pro already shows that several detector thresholds do not generalize, weakening the portability claim.","supporting_citations":[],"review_version":1}