Pith. sign in

REVIEW 3 major objections 2 minor

Train only on successful agent runs, then flag the exact steps that made later runs fail.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 03:38 UTC pith:6LBEVEEZ

load-bearing objection Abstract-only: OAT is a coherent one-class N-CDE framing for unsupervised agent failure attribution with large claimed F1/speed gains, but causal step isolation is unverifiable without methods. the 3 major comments →

arxiv 2607.12747 v1 pith:6LBEVEEZ submitted 2026-07-14 cs.AI cs.CL

Tracing Agentic Failure from the Flow of Success

Samuel Yeh , Yiwen Zhu , Shaleen Deep , Sharon Li This is my paper
classification cs.AI cs.CL
keywords failure attributionLLM agentsone-class learningneural controlled differential equationsanomaly detectiontrajectory diagnosisunsupervised learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM agent systems often fail in multi-step trajectories, yet spotting which step actually caused the failure is hard. Prompting the model itself is slow and expensive; supervised methods need costly step-level error labels on failed runs. This paper argues you can skip failure labels entirely. OAT trains a one-class model solely on successful trajectories, learns their continuous dynamical pattern in latent space with a neural controlled differential equation, and at inference time scores every step of a failed trajectory by how far it deviates from that learned success flow. The highest-scoring steps become the attributed errors. With as few as about 100 successful trajectories, the method is hundreds to thousands of times faster than prompting baselines and improves F1 by roughly 20 points in-domain and 7 points under distribution shift. The practical payoff is a lightweight diagnostic that scales with ordinary success logs instead of expert annotations.

Core claim

Unsupervised failure attribution is solvable by modeling only successful trajectories as a continuous dynamical system with a neural controlled differential equation; step-wise anomaly scores against that learned flow reliably recover the causal error steps inside failed trajectories, even under distribution shift and without any failure supervision.

What carries the argument

OAT (one-class attribution via neural CDE): a neural controlled differential equation trained exclusively on successful trajectories to capture their latent-space dynamics; at test time each step of a failure trajectory receives an anomaly score equal to its deviation from those dynamics, and high-scoring steps form the error set.

Load-bearing premise

The continuous dynamical pattern of successful trajectories, once learned by a neural CDE, is complete enough that any step-wise deviation from it marks a true causal error even when the failure occurs under distribution shift.

What would settle it

On a held-out suite of agent trajectories that contain known ground-truth error steps (including out-of-distribution tasks), check whether ranking steps by OAT anomaly score recovers those error steps at F1 competitive with or better than strong prompting baselines; a clear drop below the reported +7% OOD margin would refute the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Failure diagnosis becomes feasible from ordinary success logs alone, removing the need for step-level failure annotations.
  • Attribution latency falls by two to three orders of magnitude relative to multi-prompt pipelines, enabling near-real-time debugging.
  • The same success-only model can be reused across in-domain and out-of-distribution agent tasks without retraining on failures.
  • Practitioners can bootstrap a usable error detector with roughly 100 successful trajectories rather than large labeled failure corpora.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same one-class dynamical framing could be applied to other multi-step LLM workflows (tool-use chains, multi-agent debates) where success logs are abundant and failure labels are scarce.
  • If the neural-CDE latent flow proves stable, online monitoring of live agent runs becomes possible: flagging anomalous steps before the full trajectory finishes.
  • Combining OAT anomaly scores with lightweight human review of only the top-ranked steps may yield a hybrid annotation loop that further reduces labeling cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes OAT for unsupervised failure attribution in LLM-based agentic systems: identify which steps in a failure trajectory caused the failure, while training exclusively on successful trajectories (as few as ~100) and without step-level error annotations. It casts the problem as one-class learning with neural controlled differential equations (N-CDEs) that model the dynamical pattern of successful trajectories in latent space; at inference, each step of a failure trajectory receives an anomaly score from its deviation from those learned dynamics, and the scores are used to form a set of error steps. The abstract reports that OAT is 200–5000× faster than prompting-based baselines and outperforms them by +20% F1 in-domain and +7% F1 out-of-distribution.

Significance. If the central claim holds under full experimental scrutiny, the work would be a practically important contribution: a lightweight, annotation-free alternative to expensive prompting pipelines and to supervised post-training on failure data with step-level labels. Training on only ~100 successes and claiming both large speedups and consistent F1 gains (including under distribution shift) would make OAT a useful direction for debugging agentic systems. The one-class N-CDE framing is a coherent modeling choice for success-only dynamics; those strengths, however, remain conditional on verification of method details, evaluation protocol, and baselines that the abstract alone cannot supply.

major comments (3)
  1. Abstract, central modeling claim: the load-bearing premise is that step-wise anomaly scores from an N-CDE fit only on successful trajectories mark causal error steps inside failure trajectories (including OOD). The abstract does not define how latent states are obtained, how the controlled path is built from discrete agent steps, how the per-step anomaly score is computed, or how scores are thresholded/ranked into an error-step set. Without those definitions, it is impossible to judge whether the scores isolate causal steps rather than coarser trajectory-level failure signals common in one-class sequential models.
  2. Abstract, evaluation claims (+20% / +7% F1): the reported gains presuppose external ground-truth causal error-step labels and a clear F1 protocol (set vs. ranking, threshold selection, error bars). None of these are specified. Absent the evaluation design, datasets, baselines, and ablations, the F1 numbers cannot be treated as evidence that OAT performs genuine causal attribution rather than success/failure discrimination correlated with overall failure.
  3. Abstract, free parameters and unsupervised setup: architecture/training hyperparameters, trajectory embedding, and the anomaly-score threshold or set-formation rule are free parameters. How they are chosen without any failure supervision (or leakage from validation failures) is load-bearing for the 'train only on successes' claim and is not stated. This must be specified and stress-tested before the unsupervised claim can be accepted.
minor comments (2)
  1. Abstract wording is generally clear, but 'form a set of error steps' is underspecified relative to how F1 is computed; a one-sentence clarification of set construction would help even at abstract length.
  2. The speedup range 200–5000× is very wide; when the full text is available, pin the range to concrete baseline configurations and hardware so the claim is interpretable.

Circularity Check

0 steps flagged

No circularity detectable from abstract: success-only N-CDE training evaluated against external error-step labels on failures.

full rationale

Only the abstract is available, so no equations, self-citations, uniqueness theorems, or ansatzes can be inspected. From the abstract alone the claimed derivation is not circular by construction: OAT is trained exclusively on successful trajectories (as few as ~100) via a neural controlled differential equation that models their latent dynamical pattern; at inference, steps of a failure trajectory receive anomaly scores measuring deviation from those learned success dynamics, and the scores are used to form a set of error steps. Evaluation is reported as F1 against (external) error-step labels on both in-domain and OOD failure data, with comparisons to prompting baselines. The prediction target (error steps inside failures) is therefore not definitionally identical to the training input (success trajectories), nor is any fitted parameter simply renamed as the reported F1. Residual methodological risks (threshold choice, whether anomaly scores isolate causal steps versus generic failure signals) are ordinary one-class-learning concerns, not circularity. Score 0 is the honest finding under the hard rule that circularity requires a quotable reduction; none exists here.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

From the abstract alone, the claim rests on standard one-class / anomaly-detection assumptions plus the modeling choice that neural CDEs capture success dynamics well enough for step-level error localization. Free parameters (architecture, thresholds, embedding) and the success-as-normality axiom are load-bearing; no new physical entities are invented.

free parameters (3)
  • neural CDE architecture and training hyperparameters
    Latent dimension, vector-field network, solver, learning rate, and related choices are not fixed by theory and must be set to fit success trajectories; they shape the anomaly scores.
  • anomaly score threshold / set-formation rule
    Converting continuous per-step deviation scores into a discrete set of error steps requires a cutoff or selection rule that is not determined a priori and affects F1.
  • trajectory embedding / latent representation
    How steps are mapped into the latent path that the N-CDE models is a design choice that defines what 'deviation from success dynamics' means.
axioms (3)
  • domain assumption Successful trajectories share a learnable dynamical pattern in latent space that is a valid model of normal agent behavior.
    Core one-class premise of OAT; if success modes are highly multimodal or failures look like near-successes, anomaly scores need not mark causal errors.
  • domain assumption Neural controlled differential equations can represent the continuous-time dynamics of discrete agent trajectories well enough for step-level anomaly detection.
    Methodological commitment stated in the abstract; alternative sequence models might fit equally well or better.
  • ad hoc to paper Step-wise deviation from success dynamics is a useful proxy for causal error steps in failure trajectories.
    Links the anomaly score to the failure-attribution task; correlation with true root-cause steps is assumed rather than derived.
invented entities (1)
  • OAT anomaly scoring over N-CDE success dynamics no independent evidence
    purpose: Assign each step in a failure trajectory a score measuring deviation from learned success flow, then form a set of error steps.
    The scoring procedure is the paper's operational construct; independent evidence would be external root-cause labels and ablations, which are only claimed, not inspectable, in the abstract.

pith-pipeline@v1.1.0-grok45 · 6151 in / 2586 out tokens · 26238 ms · 2026-07-15T03:38:59.794689+00:00 · methodology

0 comments
read the original abstract

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 $\times$ faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.