{"id":"35391e7d-7ce2-4d49-a55e-f12f543ea07a","arxiv_id":"2607.07989","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AgentLocate localizes multi-agent LLM failures to a responsible agent and earliest decisive step via judge hypotheses, confidence-weighted multi-evaluator verification, and LoRA refinement.","lead":"AgentLocate finds which agent and which earliest step first made a multi-agent LLM run fail, using a judge model checked by several evaluators and then lightly fine-tuned. It matters because multi-agent systems are hard to debug when errors cascade across long tool-using trajectories.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Ground-truth decisive steps may not match the paper's own counterfactual definition, so reported gains could be overstated relative to the causal claim.","rationale":"The Reader correctly isolates the non-execution of counterfactuals as the load-bearing premise. I sharpen it one notch: the risk is not only that the Judge approximates R, but that the evaluation labels themselves may never have been validated against R, so the reported outperformance is agreement with an unvalidated proxy rather than recovery of the defined causal pair. That is the single most load-bearing concern for the strongest claim; everything else (small test sets, missing error bars, unreleased code) is secondary. The concrete re-simulation test would settle the issue cleanly. Until it is done, CONDITIONAL remains the right verdict; I do not move it to REJECT because the pipeline is coherent, ablations are informative, and efficiency results are solid. Agreement with the Reader is therefore agree on the core weakness, with only a modest tightening of the target of the concern.","tokens_in":25860,"tokens_out":636,"duration_ms":6812,"concrete_test":"On a stratified sample of ~20 Who&When test trajectories, re-simulate R(τ,t) for every candidate t (or at least for the annotated t★ and AgentLocate's prediction) using the original multi-agent policies and environment; check whether the annotated t★ is the earliest step that actually flips the outcome. If fewer than ~70% of annotations satisfy Eq. 8, or if AgentLocate's accuracy collapses when scored only on verified flip points, the causal claim is not supported by the tables.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that AgentLocate better recovers (i★(τ), t★(τ)), where t★ is defined in Appendix B (Eqs. 7–9) as the earliest step whose single idealized correction R(τ,t) flips failure to success. Who&When and Aegis-Bench supply the labels used for accuracy, yet the manuscript never shows that those annotations were obtained by actually constructing and re-simulating R(τ,t) under the same policies and scheduler. Section 3.1 explicitly states the Judge does not execute R and only approximates the criterion by reasoning over the logged trajectory; the same approximation almost certainly underlies the human/algorithmic labels. If the benchmark t★ is only a plausible earliest-error annotation rather than a verified counterfactual flip point, then Tables 1–3 measure agreement with that annotation, not recovery of the causal object the paper defines. The LoRA stage then further amplifies whatever systematic bias is present in the evaluator-aggregated labels. Absolute step-level numbers already remain modest (e.g., 38% on Algorithm-Generated), so any label–definition gap would directly weaken the strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies failure localization in LLM-based multi-agent systems: given a failed trajectory, recover both the responsible agent and the earliest decisive step. It defines the decisive step t★(τ) as the earliest time whose single idealized counterfactual correction R(τ,t) would flip failure to success (Appendix B, Eqs. 7–9), and proposes AgentLocate, a Judge–Evaluator pipeline that (i) has an LLM Judge hypothesize (î,t̂) under all-at-once or step-by-step protocols, (ii) verifies with multiple independent Evaluators under diverse prompts and confidence-weighted aggregation, and (iii) adapts the Judge via LoRA on evaluator-enriched instances. Experiments on Who&When (Algorithm-Generated and Hand-Crafted) and Aegis-Bench report consistent gains over WhichAgent, AgenTracer, ECHO, AEGIS, and two poisoning-forensics baselines, with ablations on variants, evaluator count/model, refinement rounds, trajectory length, and efficiency (tokens, cost, runtime).","tokens_in":26177,"tokens_out":1732,"duration_ms":20120,"significance":"If the results hold under a well-aligned evaluation of the causal object the paper defines, this is a useful systems contribution for multi-agent reliability: a practical, relatively efficient localization pipeline with multi-model evidence, complementary benchmarks (short vs long trajectories; agent+step vs agent+error-mode), and explicit comparison to both attribution and forensics methods. Strengths include a clear problem statement, multi-perspective verification rather than one-shot judging, ablations of the refinement cycle (Variants I–III), and efficiency tables that make the method usable for debugging. The work is timely for multi-agent deployment and debugging, even if absolute step-level accuracy remains modest.","major_comments":[{"comment":"Appendix B, Eqs. (7)–(9) define t★(τ) as the earliest step for which the counterfactual trajectory R(τ,t) succeeds. Section 3.1 states that the Judge does not execute R and only approximates this criterion by reasoning over the logged trajectory. The manuscript never shows that Who&When / Aegis-Bench ground-truth labels were obtained by constructing and re-simulating R under the same policies and scheduler. Tables 1–3 therefore measure agreement with benchmark annotations, not necessarily recovery of the causal object defined in Appendix B. This is load-bearing for the central claim. Please either (a) validate a subset of labels by actual counterfactual replay, (b) restate claims as agreement with human/algorithmic earliest-error annotations rather than verified counterfactual flip points, or (c) provide evidence that the annotations match the R-based definition.","section":"Appendix B, Eqs. (7)–(9); §3.1; Tables 1–3"},{"comment":"The adaptive stage fine-tunes the Judge on evaluator-aggregated labels (Eqs. 5–6), often with Evaluators from the same model family as the Judge (default Qwen-7B). Table 13 shows Variant II (evaluator aggregation alone) is already strong, so the LoRA stage may largely amplify shared model biases rather than inject independent causal signal. The paper should quantify agreement between Judge, individual Evaluators, and ground truth before fine-tuning, report inter-evaluator agreement, and include a cross-family setting (e.g., Judge Qwen, Evaluators Llama/Mistral only) as a primary rather than secondary result (Table 9 is partial). Without this, the reported gains risk partial circular reinforcement of the same approximation used for labels.","section":"§3.2–3.3; Table 9; Table 13"},{"comment":"Who&When test splits are small (~40% of 126 Algorithm-Generated and of 58 Hand-Crafted cases). Point estimates such as 69.05% / 38.10% (Table 1, Qwen-7B all-at-once) are reported without confidence intervals, bootstrap standard errors, or significance tests against the strongest baseline. On Hand-Crafted (Table 2), step-level accuracy is low for all methods and differences can be a few cases. Please add uncertainty estimates and, where possible, paired tests; otherwise the claim of consistent outperformance is overstated relative to sample size.","section":"Tables 1–2; §4.1 dataset splits"},{"comment":"AEGIS is agent-level only and ECHO is omitted on Aegis-Bench; pair-level gains on Aegis-Bench (Table 3) are modest (e.g., 10.50% vs 7.33% for AgenTracer under Qwen all-at-once). The abstract and conclusion claim strong localization of both agent and step across benchmarks. Please qualify claims for Aegis-Bench (agent vs pair) and avoid treating pair-level as equivalent to step-level localization. Also clarify whether AEGIS “–” step cells are excluded fairly from averages when summarizing “consistently outperforms.”","section":"Abstract; Table 3; §4.2 / Appendix G"}],"minor_comments":[{"comment":"Figure 1 is a useful failure-propagation case; make the decisive step (Step 6) and agent names visually easier to parse (e.g., bold the decisive action and align step numbers with the narrative).","section":"Figure 1"},{"comment":"Notation: ρ(t) for the scheduler and i★=ρ(t★) is clear, but the main text sometimes uses (î,t̂) and (ieval,teval) without restating domains; a short notation table would help.","section":"§2.1; §3"},{"comment":"Appendix C/D prompts are valuable; state temperature / decoding settings and whether Judge and Evaluators use identical decoding hyperparameters.","section":"Appendix C–D; §4.1"},{"comment":"Tables 6–8 include training overhead for AgentLocate and AgenTracer; explicitly state whether baseline inference-only costs exclude any offline training those methods require, for a fair comparison.","section":"Tables 6–8; §4.2 efficiency"},{"comment":"Typos / polish: “AgentLocatecombines” spacing issues appear in the abstract and introduction; unify hyphenation of multi-agent and step-by-step throughout.","section":"Abstract; §1"},{"comment":"Related work on poisoning forensics is appropriately distinguished in Appendix A; a one-sentence pointer in the main related-work section would help readers who skip the appendix.","section":"§2.2; Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The empirical package is above average for multi-agent attribution papers, but the causal definition vs. annotation gap is the main reason I recommend major rather than minor revision. If the authors reframe claims around annotation agreement and add uncertainty estimates plus a clearer cross-model evaluator study, this could become a solid COLM-level contribution. Scope fits cs.CR / multi-agent reliability; no ethics red flags beyond standard LLM-as-judge use on public benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: AgentLocate is a clean engineering contribution for multi-agent failure attribution. It does not invent the who/when problem—WhichAgent, AgenTracer, AEGIS, and ECHO already frame agent-plus-step localization—but the concrete verify-and-adapt loop (Judge hypothesis, multi-style Evaluators, confidence-weighted vote, LoRA refinement) is new enough, and the numbers against named baselines are consistently better.\n\nWhat it does well. Problem setup is clear. Experiments cover four models, all-at-once and step-by-step modes, Who&When (both subsets) and Aegis-Bench, plus trajectory-length splits, evaluator count, refinement rounds, evaluator model swaps, efficiency (tokens/cost/time), and a useful negative result that poisoning-forensics methods do not transfer. The ablation that evaluator-enriched supervision beats one-shot fine-tuning on ground-truth labels is the most interesting internal result. Discussion of late-stage agents being over-blamed is honest.\n\nSoft spots, in proportion. Absolute step-level accuracy remains modest (e.g. ~38% on Algorithm-Generated). Who&When test sets are small and tables lack uncertainty. Code is not released yet. The stress-test concern is partly right: Appendix B defines t* via a single-step counterfactual re-simulation R(τ,t), but Section 3.1 says the Judge never runs R and only reasons over the log; the benchmark labels almost certainly share that approximation. So the tables measure agreement with existing annotations more than recovery of the idealized causal object. That weakens the causal rhetoric without erasing the relative gains over baselines. Mild circularity risk exists when same-family models produce the LoRA supervision.\n\nWho it is for: people building or debugging multi-agent LLM systems who need a practical attribution tool, not a new theory of causality. The central argument holds as an empirical systems result. It deserves a serious referee; I would engage with it, especially once artifacts land.","headline":"Practical Judge–Evaluator–LoRA pipeline that beats existing multi-agent failure localizers on the usual benchmarks; relative gains look real, absolute step accuracy stays modest, and the causal definition is only approximated.","tokens_in":26809,"tokens_out":523,"would_cite":true,"duration_ms":16458,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"AgentLocate localizes failures in LLM multi-agent systems to the responsible agent and the earliest decisive step by combining an LLM judge, multi-perspective evaluators, and evaluator-guided fine-tuning.","keywords":["multi-agent systems","failure localization","LLM-as-judge","AgentLocate","decisive failure step","confidence-weighted voting","parameter-efficient fine-tuning"],"falsifier":"On a held-out set of failed trajectories whose ground-truth decisive steps are known by actual counterfactual re-simulation, measure whether correcting the step predicted by AgentLocate reverses the failure more often than the steps predicted by the strongest baselines; if it does not, the localization claim fails.","tokens_in":26756,"feed_emoji":"🕵️","tokens_out":887,"duration_ms":20447,"temperature":0.7,"pith_summary":"When an LLM multi-agent system fails on a task, the distributed, long-horizon nature of the run makes it hard to say which agent first pushed the trajectory past recovery. This paper introduces AgentLocate, a practical localization pipeline that first has an LLM Judge propose a responsible agent and step, then has several independent Evaluators re-examine the same log under different prompting styles, aggregates their answers with confidence-weighted voting, and finally uses that verified feedback to lightly fine-tune the Judge. On the Who&When and Aegis-Bench suites the method raises both agent-level and step-level accuracy above prior localization techniques and above repurposed poisoning-forensics tools, while keeping token use and wall-clock time modest. The result is a concrete debugging aid: developers can point to the earliest decisive mis-action rather than guessing among many intertwined agents.","feed_headline":"AgentLocate pins multi-agent LLM failures to agent and step","feed_subtitle":"Judge-evaluator loop with fine-tuning beats prior methods on Who&When and Aegis-Bench","key_machinery":"The Judge-Evaluator refinement cycle: an LLM Judge emits a candidate pair (agent, step); independent Evaluators, each with a distinct prompt style, re-score the same trajectory and report location, rationale and confidence; a confidence-weighted vote yields a verified label; that label, together with the rationales, becomes a LoRA training instance that adapts the Judge for subsequent localization.","core_discovery":"Treating failure localization as a verifiable cycle—Judge hypothesis, multi-Evaluator critique with confidence-aware aggregation, and LoRA adaptation of the Judge—produces more accurate identification of both the responsible agent and the earliest decisive step than one-shot judges, counterfactual-replay tracers, taxonomy matchers, or poisoning-forensics methods, across short and long multi-agent trajectories.","pith_inferences":["The same verify-and-adapt loop could be applied to non-LLM multi-agent systems whose logs admit a similar decisive-step definition.","If Evaluator diversity is the main source of signal, cheaper non-LLM critics might later replace some of the LLM Evaluators without large accuracy loss.","Longer trajectories that currently dilute the failure signal may become easier once the Judge is trained to ignore post-decision noise rather than simply fine-tuned on final labels."],"forward_implications":["Debugging multi-agent workflows can begin at the earliest decisive mis-action rather than at the final wrong answer.","Poisoning-forensics tools that look for adversarial traces are the wrong instrument for ordinary coordination and reasoning failures.","A single Judge-Evaluator refinement round already captures most of the accuracy gain, so the method can be used as a lightweight post-mortem step.","Visibility bias toward late-stage verifier or retrieval agents can be measured and later mitigated by explicit error-propagation modeling."],"fun_headline_variants":["AgentLocate localizes multi-agent LLM failures to agent and step","Judge-evaluator loop finds which agent and step fail LLM systems","AgentLocate attributes failures to agent and earliest decisive step","Adaptive judge pinpoints responsible agent and step in multi-agent LLMs","AgentLocate outperforms priors on agent-and-step failure localization"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"LLM reasoning over the logged trajectory can stand in for the true counterfactual test that defines the earliest decisive step, and the aggregated Evaluator labels are reliable enough to serve as supervision for improving the Judge.","fun_headline_variants_meta":{"raw":{"variants":["AgentLocate localizes multi-agent LLM failures to agent and step","Judge-evaluator loop finds which agent and step fail LLM systems","AgentLocate attributes failures to agent and earliest decisive step","Adaptive judge pinpoints responsible agent and step in multi-agent LLMs","AgentLocate outperforms priors on agent-and-step failure localization"]},"model":"grok-4.5","effort":"low","cost_usd":0.00594,"raw_usage":{"total_tokens":1518,"prompt_tokens":744,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":59400000,"prompt_tokens_details":{"text_tokens":744,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":704,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":744,"tokens_out":70,"duration_ms":6119,"temperature":1.0,"reasoning_tokens":704,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T14:07:43.296897+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out set of failed trajectories whose ground-truth decisive steps are known by actual counterfactual re-simulation, measure whether correcting the step predicted by AgentLocate reverses the failure more often than the steps predicted by the strongest baselines; if it does not, the localization claim fails.","supporting_citations":[],"review_version":1}