Standard agent telemetry can detect most failures while hiding the decision and provenance evidence needed to identify the fault's origin; five frontier LLMs drop to near-zero origin-step accuracy on restricted views.
REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces
1 Pith paper cite this work. Polarity classification is still indexing.
abstract
Large language model (LLM) agents now solve complex tasks through long plan-and-execution traces, yet the ability to locate errors in a completed traces still lags far behind, especially in the \emph{silent failure} regime. Existing approaches predict suspect steps via classifiers or LLM judges, or recover correct answers via retry, but none feed the intervention outcome back to \emph{refine the attribution itself}. We propose \methodname, a method that closes this gap by diagnosing a candidate error step, testing it through controlled replay with a diagnosis-specific patch, and using the verified outcome flip as contrastive evidence to refine the final attribution. Across four localization benchmarks spanning multi-hop reasoning across domains, \methodname achieves the highest localization accuracy among same-auditor methods across all four benchmarks, with the largest gains on structured tool-use traces, while providing actionable localization even when ground-truth answers are unavailable.
citation-role summary
citation-polarity summary
fields
cs.AI 1years
2026 1verdicts
ACCEPT 1roles
other 1polarities
unclear 1representative citing papers
citing papers explorer
-
TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?
Standard agent telemetry can detect most failures while hiding the decision and provenance evidence needed to identify the fault's origin; five frontier LLMs drop to near-zero origin-step accuracy on restricted views.