{"id":"584e604f-2eb8-4ff2-987e-02057fff3e13","arxiv_id":"2509.03312","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AgenTracer automatically builds failure-annotated multi-agent trajectories and trains an 8B tracer that outperforms large proprietary models at locating the decisive error step.","lead":"This paper builds AgenTracer, an automated pipeline that creates training data for finding which agent or step caused a failure in a multi-agent LLM system, and uses it to train an 8B-parameter tracer model. The model reportedly beats frontier proprietary LLMs on the Who&When attribution benchmark and provides feedback that improves existing agent frameworks by 4.8-14.2%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Counterfactual replay labels depend on an unspecified success evaluator Ω; without evidence that interventions are actually re-executed, TracerTraj's training and held-out labels may be arbitrary.","rationale":"The reader's weakest assumption identifies the identical load-bearing concern: the counterfactual-replay annotation process does not specify how Ω is evaluated. This is the most critical issue because the entire method—the TracerTraj dataset, the multi-granular RL training, and the held-out TracerTraj evaluation—depends on the validity of the decisive-error labels. If Ω is not a genuine re-execution, the labels are essentially the analyzer's opinion, and the paper's central claim loses its empirical foundation. The external Who&When benchmark is a mitigating factor, but it cannot rescue the training data if the labels are noisy or arbitrary; the model's performance there could stem from other cues, and the feedback-improvement experiments in Section 5.3 would lack a grounded explanation. The sign error in Eq. (4) is a further sign that the formalization is not carefully checked, but the main attack remains the unspecified Ω. The reader's verdict of CONDITIONAL is appropriate: the paper should be accepted only if the authors clarify and verify the annotation procedure. My stress test does not change that verdict, so I recommend UNCHANGED. The concrete test—re-annotating a sample with actual re-execution—would settle the concern definitively.","tokens_in":19923,"tokens_out":4816,"duration_ms":47859,"concrete_test":"Select a random subset of 50–100 failed trajectories from TracerTraj and re-run the annotation pipeline with actual environment re-execution: for each candidate corrected action at each step, roll out the original multi-agent system at temperature 0 with fixed random seeds, and determine success/failure using benchmark-specific verifiers (unit tests for MBPP+/KodCode/Blackjack; exact-match or programmatic checks for MATH/GSM8K; a specified verifier for GAIA). Compare the resulting decisive-error pairs with the stored TracerTraj labels. If agreement is below, say, 90%, the annotation pipeline is not grounded. If the authors cannot run this check, they should at minimum specify exactly how Ω was computed and how stochasticity was handled in the original annotation process.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that AgenTracer-8B achieves frontier-level failure attribution and that its feedback improves downstream agentic systems—rests entirely on the quality of the decisive-error annotations in TracerTraj-2.5K. Section 4.1 and Algorithm 1 define the annotation process via Ω(R(τ, t, a'_t)): after the analyzer proposes a corrected action, the intervention is applied and the outcome is evaluated. The paper never states whether Ω is computed by actually re-running the multi-agent system (e.g., executing unit tests, checking exact-match answers, or running a verifier) or by an LLM judgment. This is not a minor implementation detail: if Ω is an LLM's guess about whether the corrected trajectory would succeed, then the 'decisive error' labels are just the analyzer's counterfactual speculation. The analyzer is DeepSeek-R1 (Section 5.1), which prior work (Zhang et al., 2025c) shows is poor at failure attribution (<10% accuracy). The held-out TracerTraj evaluation in Table 2 then becomes circular: the model is trained and evaluated on labels produced by the same unverified pipeline, so high accuracy on TracerTraj may only reflect learning the annotation process's biases, not genuine failure attribution. The Who&When benchmark is external and provides some independent grounding, but the training data quality remains foundational: if the labels are arbitrary, the model's Who&When success requires an alternative explanation (e.g., learning surface cues rather than causal attribution), and the self-evolution gains in Section 5.3 lose their claimed mechanism. Additionally, Eq. (4) contains a sign error: it defines C(τ) using Ω(τ)=1 ∧ Ω(R(...))=0, which is the opposite of the 'failure→success' criterion stated in the text and correctly implemented in Algorithm 1. This formal inconsistency, while not directly affecting the empirical pipeline, underscores that the paper's specification of the core annotation procedure is unreliable. The decisive issue remains the unverifi","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgenTracer, an automated pipeline for annotating failed LLM multi-agent trajectories, and AgenTracer-8B, a Qwen3-8B model trained with multi-granular GRPO to perform failure attribution. The pipeline constructs TracerTraj-2.5K by counterfactual replay of failed trajectories and programmatic fault injection into successful ones. The authors report that AgenTracer-8B outperforms large proprietary models such as Gemini-2.5-Pro and Claude-Sonnet-4 on the external Who&When benchmark, and that its feedback improves downstream systems such as MetaGPT and MaAS by 4.8–14.2%.","tokens_in":20289,"tokens_out":3931,"duration_ms":41602,"significance":"If the claims hold, this is a practically valuable contribution: a lightweight 8B model that attributes agentic failures at or above frontier-model level would enable cheap, scalable debugging and self-improvement for multi-agent systems. The external Who&When evaluation and the downstream performance gains are concrete and falsifiable, and the availability of a 2.5K trajectory annotation dataset could be useful to the community. However, the contribution's validity depends on the quality and grounding of the automatically generated labels, and on the correctness of the formal definition of 'decisive error'.","major_comments":[{"comment":"Equation (4) defines C(τ) with the condition Ω(τ)=1 ∧ Ω(R(τ,t,a'_t))=0, i.e., a step whose oracle correction turns a successful trajectory into a failure. Since the task concerns failed trajectories, for which Ω(τ)=0, this set is empty for every input of interest. The text and Algorithm 1 use the opposite condition — the original trajectory fails and the corrected trajectory succeeds. This is not a local typo: Eq. (4) is the formal foundation for Eq. (5) and for the entire annotation procedure. It must be corrected to Ω(τ)=0 ∧ Ω(R(τ,t,a'_t))=1 and propagated consistently.","section":"§3, Eq. (4)"},{"comment":"The decisive-error annotations are produced by evaluating Ω(R(τ,t,a'_t)) and Ω(τ˜), but the manuscript never states what Ω is. Is it obtained by actually re-running the multi-agent system (e.g., executing unit tests, running a verifier, checking exact-match answers), or is it an LLM's judgment about whether the corrected trajectory would succeed? Stochasticity is also not addressed: if the system is rolled out with temperature > 0, a single re-run is not a reliable indicator of success. This is load-bearing because if Ω is an LLM guess, the 'decisive error' labels are the analyzer's counterfactual speculation, and all training and held-out evaluations on TracerTraj inherit that speculation. The paper must specify the concrete evaluation protocol for each benchmark and how randomness is handled.","section":"§4.1, Algorithm 1"},{"comment":"The held-out TracerTraj evaluation is taken from a 9:1 split of the same automatically annotated dataset. The labels were generated by the DeepSeek-R1-driven analyzer and fault-injection process, so high accuracy on TracerTraj partly measures the model's agreement with the annotation pipeline rather than independent failure-attribution quality. The external Who&When benchmark provides independent grounding, but the paper presents TracerTraj results as equally strong evidence. I suggest reframing TracerTraj as a pipeline-consistency check or supplementing it with human validation, inter-annotator agreement, or a separate human-annotated test set.","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"The observation '❶' is numbered twice; the second should be renumbered (e.g., 'Observation ❷').","section":"§5.2"},{"comment":"The reference to dataset statistics is left as '??' instead of pointing to Table 3 in Appendix A.","section":"§4.1"},{"comment":"Typo: 'Conversly' should be 'Conversely'.","section":"§5.3"},{"comment":"The model name is inconsistent: the abstract says 'Claude-4-Sonnet' while Table 1 and the case study use 'Claude-Sonnet-4'. Please align the naming.","section":"§5.1 / Abstract"},{"comment":"The claim that ground-truth supervision 'may mislead' attribution is based on a few isolated cells in Tables 1 and 2; no systematic analysis or error bars are provided. A brief discussion of variability would help.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The external Who&When results and the downstream improvement experiments are the strongest parts of the paper. The formal definition error in Eq. (4) is fixable, but the unspecified Ω evaluation is a deeper validity concern: if the authors cannot demonstrate that interventions are actually re-executed in the environment, the TracerTraj labels lose their grounding, and the main training-data claim collapses. I would recommend requiring a precise, benchmark-by-benchmark description of Ω and, if possible, human validation of a sample of TracerTraj labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nAgenTracer has a genuinely new idea: instead of hand-labeling failed multi-agent trajectories, they automatically generate annotations by replaying trajectories with oracle-corrected actions and by programmatically injecting faults into successful ones. That gives them a 2.5K dataset for training a failure tracer, and their 8B model beats frontier models on the external Who&When benchmark and produces feedback that improves downstream systems. If the annotation pipeline is sound, this is a real contribution to agent debugging.\n\nThe external results are the strongest part. AgenTracer-8B's step-level accuracy on Who&When (automated) is 42.86% vs. Claude-4-Sonnet's 40.65% and DeepSeek-R1's 31.32%, and it generalizes to unseen trajectories. The self-improvement gains (4.8–14.2% on MetaGPT, MaAS, OWL) are plausible and useful.\n\nBut there are soft spots that need attention.\n\nFirst, Equation (4) defines the decisive-error set with Ω(τ)=1 ∧ Ω(R(...))=0, which reverses the definition in the text and Algorithm 1 (failure→success). That's a clear sign error, even if Algorithm 1 is what was actually used.\n\nSecond, and more important, the paper never specifies what Ω is when evaluating a counterfactual intervention. After the analyzer proposes a corrected action, do they actually re-run the multi-agent system (e.g., executing code, running a verifier) or is success/failure judged by an LLM? For the fault-injection half, \"programmatic\" suggests execution, but for general agentic tasks it's unclear. If Ω is an LLM guess, then the training labels are just the analyzer's speculation, and the held-out TracerTraj results are circular. The external Who&When benchmark partially mitigates this, but the training-data quality is foundational.\n\nThird, the claimed \"up to 18.18%\" improvement over Gemini-2.5-Pro/Claude-4-Sonnet doesn't make sense from Table 1—the biggest gap I can find is 17.4 points on handcrafted agent-level w/G, not 18.18%. They need to point to the exact cell.\n\nFourth, no error bars or release artifacts (just a project page), which makes it hard to assess stability.\n\nOverall, the core idea is solid and the external results are encouraging, but the unspecified evaluator is a load-bearing gap. The authors should clarify Ω, fix the sign error, and justify the 18.18% claim. Then this would be a strong paper. As is, it deserves peer review but with major revisions.","headline":"A genuinely new pipeline for auto-annotating agent failures, with strong external benchmark results, but an unspecified success evaluator and a sign error undercut the central claims until clarified.","tokens_in":20865,"tokens_out":4313,"would_cite":false,"duration_ms":37741,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Counterfactual replay can label agentic failures at scale, and an 8B tracer trained on those labels finds the decisive error step more accurately than frontier LLMs while its feedback improves downstream agentic systems.","keywords":["multi-agent systems","failure attribution","LLM agents","counterfactual replay","fault injection","reinforcement learning","TracerTraj","Who&When"],"falsifier":"Re-run the counterfactual-annotation protocol with Omega computed by genuinely re-executing each repaired trajectory in the original multi-agent system, with stochasticity controlled by seeds, and compare the decisive-step labels to TracerTraj-2.5K; a large mismatch would show the reported accuracies reflect agreement with guessed labels rather than correct attribution.","tokens_in":19855,"feed_emoji":"🔍","tokens_out":7742,"duration_ms":74641,"temperature":0.7,"pith_summary":"The paper tries to establish two linked results. First, failure attribution in LLM-based multi-agent systems—finding the earliest action whose correction flips a failed run to success—can be automated at scale using counterfactual replay and programmatic fault injection, yielding a curated dataset called TracerTraj-2.5K. Second, a lightweight 8B model, AgenTracer-8B, fine-tuned with multi-granular reinforcement learning on that dataset outperforms much larger proprietary reasoning models on the Who&When benchmark, with step-level gains of up to about 18 percentage points. Its natural-language feedback also improves off-the-shelf multi-agent systems such as MetaGPT and MaAS by 4.8 to 14.2 percent over three rounds. If correct, failure attribution becomes a tractable, model-level service rather than a manual or frontier-model-only activity.","feed_headline":"An 8B model beats frontier LLMs at tracing agent failures","feed_subtitle":"Counterfactual replay trains it on 2,000+ labeled failures; its feedback lifts MetaGPT and MaAS up to 14 percent.","key_machinery":"The central mechanism is counterfactual replay with an oracle rectification operator: for each step in a failed trajectory, an analyzer agent substitutes a corrected action and the outcome determines whether that step is decisive. Complementing this, programmatic fault injection corrupts actions in successful trajectories to synthesize failures with known labels. Training then uses a multi-granular reward inside GRPO, combining strict format gating, binary agent-level credit, and a Gaussian step-level proximity reward, so the model learns both who failed and where.","core_discovery":"At the center of the paper is the decisive error: the earliest action in a failed trajectory whose replacement with an oracle-correct action would turn failure into success. The paper claims this counterfactual notion can be operationalized automatically. For failed trajectories, an analyzer agent proposes corrected actions step by step, and the outcome of each intervention is used to find the earliest step that fixes the run; for successful trajectories, the pipeline programmatically corrupts an action to create a failure whose responsible step is known by construction. The resulting dataset, TracerTraj-2.5K, is used to train AgenTracer-8B with GRPO-based reinforcement learning whose reward","pith_inferences":["If the success/failure outcome Omega is implemented as an LLM judgment rather than a deterministic re-simulation, TracerTraj labels may inherit the judge's own blind spots; validating a sample by actual re-execution would settle this.","Because the same DeepSeek-R1 model proposes oracle corrections and attack injections, both halves of the dataset could share a systematic bias; using independent oracle models would test robustness.","A natural extension is online tracing, flagging the decisive step before the episode completes, whereas the current method diagnoses finished trajectories; this would make the feedback loop faster.","The 4.8 to 14.2 percent gains are measured over three feedback rounds; an open question is whether further rounds converge, oscillate, or overfit to the tracer's own preferences."],"forward_implications":["Automated annotation replaces manual labeling: thousands of failed trajectories can be converted into training pairs without human inspection.","A compact 8B open-weight model can perform failure attribution at or above the level of frontier proprietary reasoning models, making inline tracing practical.","Feeding the tracer's reasoning trace back into a failing multi-agent system improves its next-round accuracy by 4.8 to 14.2 percent, even for systems the tracer was not trained on.","Step-level attribution accuracy becomes the meaningful metric; in the harder no-ground-truth setting the model stays robust while frontier baselines drop.","The pipeline covers handcrafted, partially automated, and fully automated frameworks, so a single tracer can serve diverse agentic architectures."],"supporting_citations":[{"why":"Documents failure rates up to 86.7 percent in mainstream multi-agent frameworks, motivating the need for automated failure attribution.","marker":"Cemri et al., 2025"},{"why":"Supplies the Who&When benchmark, the decisive-error formalization, and the observation that even strong reasoning models fall below ten percent accuracy.","marker":"Zhang et al., 2025c"},{"why":"Provides DeepSeek-R1, used both as the analyzer/oracle agent for annotation and as a baseline, plus the GRPO reinforcement learning algorithm underlying training.","marker":"Guo et al., 2025"},{"why":"The Qwen3 technical report; Qwen3-8B is the base model fine-tuned into AgenTracer-8B.","marker":"Yang et al., 2025"},{"why":"MetaGPT is one of the source multi-agent systems for trajectory collection and one of the downstream systems improved by AgenTracer-8B feedback.","marker":"Hong et al., 2023"},{"why":"MaAS is the primary downstream system where AgenTracer feedback yields up to 14.2 percent performance gain.","marker":"Zhang et al., 2025b"},{"why":"Self-Refine is the classical self-feedback baseline compared against AgenTracer in the downstream improvement experiment.","marker":"Madaan et al., 2023"},{"why":"CRITIC is the second baseline, showing that tool-interactive critique often degrades agentic performance without accurate attribution.","marker":"Gou et al., 2024"}],"fun_headline_variants":["AgenTracer: tiny 8B model outtraces giant LLMs","Counterfactual replay pinpoints the agent step that failed","Automated fault injection exposes agentic failure causes","8B tracer beats Claude and Gemini at failure attribution","From counterfactual replay: who really broke your agent?"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The entire annotation pipeline depends on the outcome function Omega being a genuine evaluation of what happens after a corrected action is substituted; if Omega is only a language model's guess rather than an actual re-run of the system, then every decisive-error label in TracerTraj—and therefore the reported attribution accuracies—could be measuring agreement with that guess.","fun_headline_variants_meta":{"raw":{"variants":["AgenTracer: tiny 8B model outtraces giant LLMs","Counterfactual replay pinpoints the agent step that failed","Automated fault injection exposes agentic failure causes","8B tracer beats Claude and Gemini at failure attribution","From counterfactual replay: who really broke your agent?"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1412,"prompt_tokens":787,"completion_tokens":625,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":542}},"tokens_in":531,"tokens_out":625,"duration_ms":6830,"temperature":1.0,"reasoning_tokens":542,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:59:37.700499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the counterfactual-annotation protocol with Omega computed by genuinely re-executing each repaired trajectory in the original multi-agent system, with stochasticity controlled by seeds, and compare the decisive-step labels to TracerTraj-2.5K; a large mismatch would show the reported accuracies reflect agreement with guessed labels rather than correct attribution.","supporting_citations":[],"review_version":1}