{"id":"c24a09ec-b2cb-48ea-952c-58da7959ed15","arxiv_id":"2505.08638","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"TRAIL provides a public benchmark of agent execution traces annotated with errors and shows that even the best LLM judges achieve only 11% joint accuracy at localizing them.","lead":"This paper introduces TRAIL, a dataset of 148 human-annotated agent execution traces with a taxonomy of errors, and finds that top AI models can only correctly locate about 11% of the errors. It matters because debugging complex AI agents currently relies on slow human analysis, and a public benchmark could accelerate automatic trace debugging.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth reliability is asserted, not demonstrated: §A.7 reports only post-hoc span-modification rates, not an inter-annotator agreement measure, so the 11% headline is not anchored to a stable gold standard.","rationale":"The single most load-bearing assumption is that the human-annotated gold set is accurate and consistent. Without a proper inter-annotator reliability measure, the benchmark's headline absolute score is not anchored: if annotation is noisy or subjective, model scores measure agreement with one annotator team's choices rather than trace-debugging ability. The paper's own evidence for reliability is insufficient, since the reported percentage of spans modified after verification is compatible with either high or low initial agreement. The internal inconsistencies in trace counts (114+30 vs 118+31) and error counts (841 vs 835) deepen the concern that the released gold files do not match the paper's description, so the 11% figure is not currently reproducible from the tables. This is a real, testable concern rather than a rejection: the dataset is public, so independent re-annotation and rescoring can determine whether the conclusion survives. The reader's conditional verdict is appropriate; the concern does not move the verdict, but it does require the authors to provide actual agreement statistics and reconcile the reported numbers before the headline claim can be relied upon.","tokens_in":27446,"tokens_out":5271,"duration_ms":57202,"concrete_test":"Take a random subset of 30 traces (15 GAIA, 15 SWE-Bench), have a fresh annotator team label them independently using the published taxonomy and rubric, with no access to the released gold labels, and compute Cohen's kappa (error presence per span) and a weighted kappa for category and impact. Then rescore Gemini-2.5-pro and o3 against (a) the original gold and (b) the new majority-vote gold. If per-span kappa is below 0.6, or if model joint accuracy shifts materially between the two gold sets, the ground truth is not stable enough to support the 11% headline. At minimum, report a true inter-annotator statistic and reconcile the trace and error counts before releasing final scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SOTA LLMs score 11% joint accuracy on TRAIL presumes that the gold annotations are correct, complete, and reproducible. §4.2 says \"high inter-annotator agreement\" but defers to §A.7; §A.7 reports only that after a consensus verification pass 5.63% of SWE-Bench spans and 5.31% of GAIA spans were modified. A modification rate is not an agreement coefficient: it does not measure independent-annotator consistency, and post-hoc consensus can encode one dominant annotator's judgment. Many taxonomy categories (Language-only, Resource Abuse, Formatting Errors) are subjective, and the task demands exhaustive detection; any gold label that a reasonable annotator would omit directly depresses model F1 and joint accuracy. The headline result could therefore reflect disagreement with particular annotation choices rather than an inability to localize genuine failures. The discrepancy between 114+30 traces in §4.3 and 118+31 in Table 5, and between 841 errors in the abstract and 835 in Table 5, further obscures what the gold set actually contains. This makes the \"11%\" figure unverifiable from the released counts alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TRAIL, a benchmark of 148 (or 149, see major comments) human-annotated OpenTelemetry traces drawn from GAIA and SWE-Bench Lite, together with a hierarchical taxonomy of agentic errors spanning reasoning, system execution, and planning/coordination. The authors evaluate eight long-context LLMs as trace judges, asking them to localize error spans and classify error categories, and report that the best model (Gemini-2.5-Pro) achieves only 11% combined joint accuracy (18% on GAIA, 5% on SWE-Bench). They also study how input length, reasoning effort, and model family affect performance, and release the dataset and code publicly.","tokens_in":27675,"tokens_out":4191,"duration_ms":41232,"significance":"If the central result holds, the paper makes a useful contribution: it provides a publicly available, ecologically grounded trace-level benchmark with a fine-grained taxonomy, and it documents a consistent and striking failure of current LLMs at structured trace debugging. The strength of the paper is that the main finding is empirical and not circular: ground truth comes from human annotation, the evaluated models are not used to construct labels, and no parameters are fitted to produce the 11% figure. The reasoning-effort ablation and the analysis of context-length limits are also valuable. However, the benchmark's credibility rests on ground-truth reliability and on reproducible dataset statistics, and both are currently under-supported. The taxonomies and the public release are likely to be useful to the agent-evaluation community once these issues are addressed.","major_comments":[{"comment":"The claim of 'high inter-annotator agreement' is not supported by the reported statistics. §A.7 states only that 5.63% of SWE-Bench spans and 5.31% of GAIA spans were modified after a consensus verification pass, which is a post-hoc modification rate, not an agreement coefficient between independent annotators. Please report per-annotator agreement (e.g., Cohen's kappa or Krippendorff's alpha) for error category, span location, and impact, and clarify whether annotations were produced independently before the consensus rounds. Without this, the gold labels and the 11%/18%/5% headline numbers are not anchored to a demonstrably stable ground truth.","section":"§4.2, §A.7"},{"comment":"The dataset counts are internally inconsistent. The abstract and §4 say 148 traces and 841 errors, but Table 5 lists 118 GAIA + 31 SWE-Bench = 149 traces and 579 + 256 = 835 errors; §4.3 further says errors were found in 114 GAIA and 30 SWE-Bench traces. Please reconcile these numbers and state exactly which files constitute the released benchmark, since the reported error totals are used to compute F1 and joint accuracy.","section":"Abstract, §4, Table 5"},{"comment":"The headline '11% combined joint accuracy' is not reproducible from the reported per-split joint accuracies. Gemini-2.5-Pro achieves 0.183 on GAIA and 0.050 on SWE-Bench; weighting by the trace counts in Table 5 gives roughly 14–15%, not 11%. Please define the combined metric precisely and report the exact computation, or the central claim of the abstract cannot be verified.","section":"§5, Table 1, abstract"},{"comment":"The evaluation reports averages of three runs without variance, confidence intervals, or significance tests, and the paper includes no human-expert baseline on the benchmark task. Since the central claim is that 11% is 'poor', the absence of a human ceiling and of uncertainty estimates leaves the model rankings and the difficulty claim uncalibrated. Please add per-run variation (or confidence intervals) and at least a small human-expert evaluation on a subset of traces.","section":"§5, Table 1, §A.3"}],"minor_comments":[{"comment":"The text in §3 refers to 'Figure 3' when summarizing the taxonomy, but Figure 3 is captioned 'TRAIL Dataset Statistics'; the taxonomy diagram appears in Figure 1. Please correct the cross-reference.","section":"§3 and Figure 3"},{"comment":"The Security correlation is reported as 1.00 for every model on both splits, which suggests the human security scores are constant; please report the underlying score distributions or remove the row if it is uninformative.","section":"Table 6"},{"comment":"The phrase 'four independent verification rounds' is ambiguous; if the same annotators revised labels after seeing other annotators' judgments, these are not independent measurements and should be described as consensus rounds.","section":"§A.7"},{"comment":"The correlation table reports Pearson and Spearman coefficients without p-values or confidence intervals; given the modest number of traces, please add uncertainty estimates or significance levels.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The central result is plausible and the benchmark is potentially valuable, but the headline number is currently not reproducible from the paper's own tables, and the ground-truth reliability is asserted rather than demonstrated. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection. No circularity or novelty concerns beyond the internal inconsistencies noted above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TRAIL is worth a serious look. It is the first benchmark I know of that combines a fine-grained error taxonomy with human-annotated OpenTelemetry traces from GAIA and SWE-Bench, and the structured-trace format plus execution/planning error categories are genuinely new relative to MAST and ACPBench. The paper also does a good job showing that current LLMs are bad at this task: the best model, Gemini-2.5-Pro, scores 11% joint accuracy, and the failure is consistent across eight models. That result is believable and important.\n\nThe main soft spot is annotation validity. The paper claims high inter-annotator agreement, but the appendix only reports that 5.31% (GAIA) and 5.63% (SWE-Bench) of spans were modified after a consensus verification pass. A modification rate is not an agreement statistic; it does not measure independent-annotator consistency, and post-hoc consensus can encode one annotator's judgment. Since the task asks for exhaustive detection and several categories are subjective, noisy gold labels could move the headline numbers. This is the load-bearing weakness, and it needs to be addressed.\n\nThere are also internal inconsistencies: the abstract says 148 traces and 841 errors, but Table 5 sums to 149 traces and 835 errors, and §4.3 reports 114+30 while Table 5 says 118+31. These are easy to fix but should be fixed before the dataset is cited as ground truth. I would also like a human baseline or at least an explicit estimate of label noise, and error bars on the three runs.\n\nNone of this undercuts the central conclusion—models are far from useful on trace debugging. But it does mean TRAIL's value as a benchmark depends on gold-label quality, and the paper currently asserts rather than demonstrates that quality.\n\nThe paper deserves peer review, not desk rejection. The resource is new, the problem is real, and the headline finding is robust. A good referee will push the authors to report proper agreement measures, reconcile the counts, and add a human baseline.","headline":"TRAIL is a genuinely useful new benchmark for agentic trace debugging, but its headline 11% accuracy figure rests on an annotation quality claim the paper does not actually demonstrate.","tokens_in":28163,"tokens_out":2511,"would_cite":true,"duration_ms":23097,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modern long-context LLMs are poor at debugging agent workflow traces: the best model scores 11% joint accuracy on TRAIL.","keywords":["agentic workflows","error taxonomy","trace debugging","issue localization","LLM-as-a-judge","long-context reasoning","OpenTelemetry traces","AI evaluation"],"falsifier":"Re-annotate a random sample of TRAIL traces with an independent team using the same taxonomy and measure agreement on category-plus-span; if agreement is low, the ground truth is too unstable to support the 11% figure. Alternatively, give models a multiple-choice version with the same spans and categories; a large score jump would show the bottleneck is output format, not trace-debugging skill.","tokens_in":27284,"feed_emoji":"🐛","tokens_out":8805,"duration_ms":81782,"temperature":0.7,"pith_summary":"The paper argues that evaluating agentic systems needs fine-grained trace analysis, not just end-to-end task success, and supplies the missing infrastructure: a formal taxonomy of agent failures and a human-annotated benchmark, TRAIL, of 148 OpenTelemetry-formatted traces from GAIA and SWE-Bench-Lite. On each trace, an LLM judge must find every error, name its category, and point to the exact span where it occurred. The paper reports that state-of-the-art long-context models do this poorly: Gemini-2.5-Pro, the best tested, reaches 18% joint accuracy on the GAIA split and 5% on the SWE-Bench split, for a combined 11%, and three of eight models cannot even fit the full trace in context. If this result holds, current LLMs cannot reliably audit or debug the agentic workflows they are increasingly asked to operate, making TRAIL a concrete target for improving trace reasoning.","feed_headline":"Agent trace debugging stumps top LLMs at 11 percent","feed_subtitle":"TRAIL's 148 annotated traces ask models to find and classify errors; even the best model mostly fails.","key_machinery":"The central machinery is the TRAIL error taxonomy—a three-branch hierarchy covering Reasoning Errors, System Execution Errors, and Planning and Coordination Errors, with leaf categories such as Formatting Errors, Tool Selection Errors, Rate Limiting, Context Handling Failures, and Task Orchestration—paired with a span-level annotation task in which every error must be assigned to an OpenTelemetry span ID. A model's answer is jointly accurate only when the predicted category and the predicted span location both match the human annotation. The taxonomy converts vague impressions of agent failure into a reproducible scoring target, while the span-location requirement forces the judge to localize, not just diagnose, each failure.","core_discovery":"TRAIL is a benchmark for trace-level evaluation of agentic systems. It pairs a three-branch taxonomy of agentic errors—reasoning errors, system execution errors, and planning and coordination errors—with 148 expert-annotated execution traces totaling 1,987 OpenTelemetry spans, of which 575 contain at least one of the 841 annotated errors, drawn from GAIA and SWE-Bench-Lite. The evaluation task is to take a raw trace and output, for every error, its fine-grained category, the span ID where it occurs, evidence, description, and impact level, plus rubric scores for reliability, security, instruction adherence, and plan optimality. Under this protocol, the authors find current models fall far short of human-level trace debugging: Gemini-2.5-Pro achieves 11% combined joint accuracy, model rankings on TRAIL largely track long-context leaderboards, performance degrades with input length, and higher reasoning effort improves scores.","pith_inferences":["Beyond the paper: if the taxonomy's categories overlap in practice, exact-match joint accuracy may understate models' ability to localize the right span; a hierarchical or partial-credit metric could produce a different ranking.","Beyond the paper: because errors were partly induced through prompt constraints, TRAIL's error mix may not match naturally occurring agent failures, so transfer to production logs is untested.","Beyond the paper: the same span-level protocol could extend to multimodal traces and to streaming observability data, where finding the failing component in a live trace is an operational need."],"forward_implications":["No tested model exceeds 11% joint accuracy, so relying on any current LLM judge for agentic trace debugging would misclassify or miss the majority of errors.","Three of the eight evaluated models cannot process the full trace within their context window, meaning trace length itself is a bottleneck for trace reasoning.","Raising reasoning effort improves trace-debugging scores, so test-time reasoning is a lever for better agent evaluation.","The error taxonomy supplies a shared vocabulary for reporting agent failures, making evaluations more actionable than end-to-end success rates.","The dataset and protocol can serve as a target for training and tuning models specifically for trace reasoning and issue localization."],"supporting_citations":[{"why":"Provides the GAIA benchmark tasks whose agent executions become the open-world information-retrieval half of TRAIL.","marker":"Mialon et al., 2023"},{"why":"Provides SWE-Bench-Lite tasks whose CodeAct agent executions become the software-engineering half of TRAIL.","marker":"Jimenez et al., 2024"},{"why":"Defines the standardized trace format used to record all agent executions in the dataset.","marker":"OpenTelemetry, 2025"},{"why":"Supplies the openinference span schema that structures the trace JSON with span IDs, attributes, and tool events.","marker":"Arize AI, 2025"},{"why":"Prior agent failure taxonomy whose reasoning and coordination categories TRAIL extends with execution and planning categories.","marker":"Cemri et al., 2025"},{"why":"The CodeAct agent used to generate the SWE-Bench traces.","marker":"Wang et al., 2024c"},{"why":"Hierarchical agent orchestration design that TRAIL follows for the GAIA multi-agent traces.","marker":"Liu et al., 2023b"},{"why":"The open deep-research agent implementation used to run the GAIA traces.","marker":"Hugging Face, 2024"},{"why":"The o3-mini backbone model chosen for the GAIA manager and search agents.","marker":"OpenAI, 2025d"},{"why":"The Claude 3.7 Sonnet backbone model for the SWE-Bench CodeAct agent.","marker":"Anthropic, 2025"}],"fun_headline_variants":["Best LLMs score 11% on new agent trace benchmark","Agent debugging: top models fail at 11% accuracy","New benchmark shows LLMs struggle with agent traces","TRAIL benchmark: Gemini best at 11%, humans far ahead","Agentic trace errors: LLMs score just 11% on TRAIL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human-annotated labels are accurate and consistent enough to serve as ground truth; if the labels are noisy or the taxonomy categories are ambiguous, the model rankings and the headline 11% figure could change materially.","fun_headline_variants_meta":{"raw":{"variants":["Best LLMs score 11% on new agent trace benchmark","Agent debugging: top models fail at 11% accuracy","New benchmark shows LLMs struggle with agent traces","TRAIL benchmark: Gemini best at 11%, humans far ahead","Agentic trace errors: LLMs score just 11% on TRAIL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2762,"prompt_tokens":959,"completion_tokens":1803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1716}},"tokens_in":575,"tokens_out":1803,"duration_ms":11063,"temperature":1.0,"reasoning_tokens":1716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:48:54.324227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of TRAIL traces with an independent team using the same taxonomy and measure agreement on category-plus-span; if agreement is low, the ground truth is too unstable to support the 11% figure. Alternatively, give models a multiple-choice version with the same spans and categories; a large score jump would show the bottleneck is output format, not trace-debugging skill.","supporting_citations":[],"review_version":1}