Pith. sign in

REVIEW 4 major objections 5 minor 92 references

This paper claims agent failures can be localized to a single interaction edge and fault side, and that independent judges recover its labels at κ=0.76.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:19 UTC pith:PDCG3UXE

load-bearing objection A genuinely useful organizational taxonomy for agent failures, but the validation does not establish the fault-side labels as measured structure—the attribution rule is unfalsifiable and the judges were handed an answer key. the 4 major comments →

arxiv 2607.28802 v1 pith:PDCG3UXE submitted 2026-07-30 cs.AI

Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

Harsh Raj , Vipul Gupta , Anas Mahmoud , Razvan-Gabriel Dumitru , Darvin Yi , Aakash Sabharwal , Yunzhong He This is my paper
classification cs.AI
keywords agent failuresfault localizationfailure taxonomyLLM agentsroot-cause attributioninteraction edgeagent-as-a-judgerepair assignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when an agent system fails, the visible outcome is not enough: the same failure may need a model fix, a harness fix, or an environment fix. It introduces a taxonomy that assigns every failure to an interaction between two components (an edge) and names the responsible component (the fault side), yielding 41 failure modes. The taxonomy is designed to be architecture-agnostic, covering single agents, multi-agent systems, coding assistants, and long-horizon personal assistants. To show the labels are not just one annotator's intuition, the authors ran four frontier LLMs as independent judges and found the strongest judge agrees with human category labels at Cohen's κ=0.76.

Core claim

The central claim is that failures in LLM agents are best described as COMP1—COMP2·fault:SIDE: the edge names where the failure originated and the fault side names which component must be repaired. This makes failure labels actionable—model-side failures identify post-training targets, harness-side failures point to scaffolding and tool-integration fixes, and environment or grader failures signal evaluation redesign. The paper organizes 41 failure modes into this structure and grounds them in 40 worked examples from benchmarks, system cards, reports, and logged trajectories. Independent reasoning agents, given only the taxonomy definitions and the original sources, recover the human-assigned

What carries the argument

The edge-and-fault-side representation: each failure is written as an interaction edge between two components (e.g., TOOL—MODEL, OWNER—MODEL, EXTERNAL ENVIRONMENT—MODEL) together with the responsible component (fault side). The attribution rule traces the observed outcome backward to the earliest failure from which execution does not recover, and assigns the fault side by asking whether a more capable model could have prevented or recovered from the failure. This representation is what makes the taxonomy actionable and what is being tested in the agent-as-a-judge validation.

Load-bearing premise

A failure is model-side whenever a more capable model could have prevented or recovered from it; this counterfactual is untestable and drives 36 of the 41 failure modes onto the model side.

What would settle it

Take the 40 curated examples that are labeled model-side and run each with a frontier model known to be more capable on the same harness and evidence; if a majority of the failures persist, the 'more capable model' rule misattributes those faults. More directly, compare the paper's fault-side labels with labels from a minimal-intervention criterion (change exactly one component's behavior and see if the outcome changes); substantial disagreement would refute the localization claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Repair routing: model-side labels direct effort to post-training, harness-side labels to scaffolding changes, and environment-side labels to benchmark or service redesign.
  • Cross-architecture applicability: the same edge vocabulary works for coding agents, personal assistants, and multi-agent systems without rebinding failure modes to one benchmark.
  • Reproducibility: independent reasoning agents recover human category labels with best κ=0.76, and agreement among judges reaches κ=0.84, suggesting the labels are not idiosyncratic.
  • Selective ensembling can raise precision to 0.96 at 68% coverage, at the cost of abstaining on the hardest cases.
  • The taxonomy is descriptive, not quantitative—it does not estimate how often each failure mode occurs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'more capable model' counterfactual could be replaced by an empirical repair test: if swapping in a strictly stronger model on the same harness makes a 'model-side' failure disappear, the label is confirmed; the paper's own E32 comparison shows such tests are already feasible.
  • The strong model-side skew (36 of 41 modes) may be an artifact of the counterfactual attribution rule rather than a fact about where agent failures really originate; a prevalence study using causal-necessity labels would likely shift more failures toward harness and environment sides.
  • The 40 worked examples, with human labels and judge disagreements, could serve as a ground-truth benchmark for failure-localization systems; current judge accuracy (~80%) suggests headroom for better attribution methods.
  • The edge-and-fault-side representation generalizes beyond LLM agents to any interactive autonomous system—robots, embedded controllers, or hybrid human-AI teams—wherever an interaction graph can be defined.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an interaction-centric taxonomy for localizing failures in LLM agent systems. Failures are assigned to an edge between two components (model, owner, grader, third party, context, memory, tool, other models, local/external environment) and a fault side indicating which component should be repaired. The taxonomy organizes 41 failure modes, 36 of which are model-side. The authors ground the taxonomy in 40 worked examples from public benchmarks, system cards, reports, and logged trajectories, and validate it by asking four frontier LLMs to recover the human-assigned labels from the source material and the frozen taxonomy definitions; the best judge achieves Cohen's κ=0.76 against the human labels. The paper claims this agreement shows the categories capture shared structure rather than annotator-specific preferences, and concludes that model-side failures identify post-training targets while harness/environment-side failures point elsewhere.

Significance. A reliable, shared taxonomy that maps a visible agent failure to the component whose repair would prevent it would be genuinely useful for agent development and evaluation. The paper's component vocabulary and edge-based notation are clear, and the appendix provides unusually detailed worked examples and verbatim definitions, which is a real strength. However, the central evidentiary claim—that the fault-side labels are reproducible measurements of where the failure originated—is not supported by the current validation. The attribution rule in §7 is unfalsifiable, the judge-based validation in §6 is partly circular because judges receive author-written disambiguation rules that pre-decide the contested boundaries, and the single human annotator belongs to the group that built the taxonomy. The paper's own case study in Appendix A.2 demonstrates the fragility of the attribution rule. The taxonomy may still be useful as a descriptive organizing framework, but the manuscript does not establish that its labels locate faults rather than encode conventions.

major comments (4)
  1. [§7, Fig. 2, Appendix A.2] The fault-side attribution rule is load-bearing and unfalsifiable as stated: 'a failure is model-side when a more capable model could have prevented it or recovered from it' (§7). No reference model or capability class is specified, so for almost any failure one can posit a sufficiently capable model, making the 36/41 model-side distribution (§5, Fig. 2) a convention rather than a measured property. The paper's own case study (Appendix A.2, Fig. 5) shows a judge using exactly this 'best-possible agent would have...' logic to mislabel a known environment failure as a model-side Observation Failure. The rule needs a concrete operationalization—e.g., a specified reference model class, or evidence-based recoverability checks—before the labels can support the paper's repair-assignment claims.
  2. [§6, Appendix A.1, Fig. 4] The validation loop does not independently test the taxonomy's boundaries. Turn 2 instructs judges to 'assign fault to the component whose own behavior failed, not to whoever could have prevented it' (Appendix A.1), which is a different criterion from the §7 counterfactual. Turn 3 then supplies author-written disambiguation rules (Fig. 4) that resolve exactly the ambiguous cases the taxonomy claims to validate—for example, pre-deciding that a dropped constraint guarding a high-rollback-cost action is Unauthorized Irreversible Action rather than Context Rationale Erosion, and that a hard external block is Service Failure rather than instruction-following. The reported κ=0.76 (and judge–judge κ=0.84) therefore measures how well the judges follow the authors' answer key, not whether independent analysts recover the taxonomy's structure on their own. The paper should report a no-reflection/n
  3. [§5.2, E4, E18, E21, Fig. 4] There is an internal inconsistency in root-cause attribution. In E18 and E21, context compaction that drops a constraint is labeled as Context Rationale Erosion with fault:CONTEXT, even when the model subsequently acts wrongly. In E4, the same mechanism—context compaction dropping the instruction not to act—is labeled as OWNER—MODEL Unauthorized Irreversible Action with fault:MODEL. The distinction is justified only by the Fig. 4 rule that a dropped constraint guarding a high-rollback-cost action should be attributed to the model, regardless of the erosion mechanism that produced it. That is a normative policy choice, not a causal localization, and it contradicts the paper's stated principle of labeling 'the earliest failure from which execution does not recover' (§3). The attribution rule should be applied consistently, or the paper should explicitly distinguish causal origin from a sep
  4. [Appendix A.2, E40, §6] The case study in Appendix A.2 is more damaging than the paper acknowledges. The human annotator labels a bug in the evaluation harness as EXTERNAL ENVIRONMENT—MODEL Stale State Delivery because the taxonomy has 'no dedicated' edge for a pure harness bug, so the failure is mapped to 'the nearest available edge.' This means the fault-side label is an approximation chosen to fit the schema, not a localization of the actual responsible component. Combined with the small evaluation set (40 examples, single human annotator from the author group, no confidence intervals on κ in Table 2/Fig. 3), the evidence is insufficient for the abstract's claim that the categories 'capture shared structure rather than annotator-specific labeling preferences.' The paper should either add the missing harness component/edge or explicitly present the fault-side mapping as a pragmatic convention with known limit
minor comments (5)
  1. [§2, Fig. 2] The figure appears to list only two MODEL—MODEL failure modes, while §5.2 defines four (Delegation Failure and Communication Failure separately for peer and subagent roles). Please clarify whether the figure is schematic or exhaustive, and make the count of 41 consistent with the displayed leaves.
  2. [§1, §5.2, E24] Edge notation is inconsistent: the text uses both TOOL—MODEL·fault:TOOL (§1) and MODEL—TOOL·fault:TOOL (Fig. 2, E24). Pick one ordering and apply it throughout the taxonomy and examples.
  3. [§6, Fig. 3] The text says 'Figure 3 shows pairwise Cohen’s κ' but the figure is numbered 3 in the manuscript; earlier references to 'Figure 2' for the hierarchy are correct, but the cross-reference style should be checked globally. Also, report confidence intervals or at least the number of examples per category for the κ values.
  4. [Abstract, §6] The abstract states the strongest judge reaches κ=0.76 'against human category labels,' but the human labels come from a single annotator. Please state this explicitly in the abstract or soften the claim to agreement with the reference annotations.
  5. [§7, Limitations] The limitations paragraph is candid about judge accuracy and coverage, but it does not mention the unfalsifiability of the attribution rule. A sentence acknowledging this limitation would help readers calibrate the taxonomy's prescriptive claims.

Circularity Check

2 steps flagged

Partial circularity: the judge validation is scaffolded by author-written disambiguation rules, and the fault-side rule makes the model-side split definitional.

specific steps
  1. fitted input called prediction [§6 (Validating the Taxonomy with an Agent-as-a-Judge), Turn 3; Fig. 4; §4 (Categorization Methodology)]
    "The judge checks its proposed label against the predefined disambiguation rules and either confirms or revises it. The final label is used for evaluation."

    The 'predefined disambiguation rules' in Fig. 4 are author-written answers to the exact ambiguous boundaries the taxonomy is supposed to validate, e.g., 'A constraint honored, then violated only after a context summary, is CONTEXT—MODEL Context Rationale Erosion ..., not OWNER—MODEL Over-initiative.' The human labels were produced by the same researchers who froze the taxonomy (§4: 'we froze the taxonomy and used that version for all reported labels and for the validation in §6'). Thus the judge's final label is not an independent recovery of structure; it is the output of the taxonomy plus an author-supplied answer key. The reported κ therefore partly measures rule-following by LLMs, not evidence that the categories capture independent shared structure.

  2. self definitional [§7 Discussion, first paragraph; also §1 contribution]
    "This imbalance partly reflects our attribution rule: a failure is model-side when a more capable model could have prevented it or recovered from it."

    The fault side is the paper's central output ('fault side indicating where the repair belongs'). Defining model-side as 'a more capable model could have prevented it or recovered from it' makes the 36/41 model-side distribution a direct consequence of the definition: with no specified reference model, the counterfactual is unfalsifiable, and the localization claim ('model-side failures identify targets for post-training') is true by stipulation rather than by measurement. The §6 validation does not test this counterfactual; in A.1 the judges were instead instructed to 'assign fault to the component whose own behavior failed, not to whoever could have prevented it,' a different criterion.

full rationale

The paper is a taxonomy-construction and internal-reproducibility paper, not a formal derivation, so most of its content is honestly presented as categorization rather than as a theorem. The self-citations to the authors' own trajectory logs and prior papers are evidence sources and are not load-bearing in a circular way. However, the central empirical claim — that independent judges recover human labels with κ=0.76 and that this 'suggests that the categories capture shared structure' — is partially circular. The judges are given the frozen taxonomy, then an additional author-written list of disambiguation rules that pre-decide the hardest boundary cases; the gold labels were assigned by the same authors using the same stabilized taxonomy. This makes the agreement partly an artifact of the prompt. Separately, the fault-side attribution rule in §7 defines model-side by an untestable counterfactual, so the distribution of failures across fault sides is not an empirical finding about where repairs belong. These issues warrant a 6 rather than a higher score because there is still substantial independent content: the worked examples are drawn from many external sources, the judges do reconstruct evidence from original traces, and the taxonomy itself is a useful organizational structure even if the fault-side validation is weaker than claimed.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

No numeric free parameters are fitted. The load-bearing assumptions are the root-cause uniqueness rule, the counterfactual fault-attribution rule, the use of LLM judges as independent validators, and the correctness of a single human annotator's labels. The only invented analytical object is the 'fault side' label itself.

axioms (4)
  • domain assumption A unique earliest unrecovered failure can be identified for each trajectory.
    §3: 'trace the causal chain backward... label the earliest failure from which execution does not recover.' This assumes a well-defined causal root cause in noisy partial traces; the paper's own E4 and A.2 show cases where the source evidence does not pin a unique root cause.
  • ad hoc to paper Fault can be assigned to the model when 'a more capable model could have avoided or recovered' from the failure.
    §1 and §7. This counterfactual cannot be falsified and drives 36/41 modes to the model side; it is a convention, not an empirical fact.
  • domain assumption LLM judges that follow the provided taxonomy definitions and disambiguation rules provide independent evidence of taxonomy validity.
    §6: judges are run as independent analysts, but they share the author-written reflection rules, so agreement conflates instruction-following with independent discovery.
  • domain assumption The human-assigned labels used as gold are correct.
    §6: a single human annotator from the taxonomy-building group defines ground truth; no inter-human agreement is reported.
invented entities (1)
  • fault side (component-at-fault label) no independent evidence
    purpose: Assigns responsibility for a failure to one endpoint of an interaction edge, e.g., fault:MODEL vs fault:TOOL.
    It is a labeling convention introduced by this paper; the only evidence offered is agreement among annotators/judges who were given the same convention, so it has no falsifiable handle outside the framework.

pith-pipeline@v1.3.0-alltime-deepseek · 39726 in / 10323 out tokens · 104030 ms · 2026-08-03T00:19:44.121489+00:00 · methodology

0 comments
read the original abstract

Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would improve the agent system. This creates a repair-assignment problem: the same visible failure may call for model post-training, harness engineering, environment redesign, or benchmark repair depending on its source. Because agent behavior emerges from interactions among models, harnesses, users, tools, memory, and environments, outcome-level labels are often insufficient for improvement. Most failure taxonomies do little to resolve this problem because they are benchmark-specific and lack a shared structure. We introduce an interaction-centric taxonomy that localizes failures to the interactions in which they originate and identifies the responsible component. It organizes 41 failure modes by assigning each to an edge between two components and a fault side indicating where the repair belongs. This makes the taxonomy actionable: model-side failures identify targets for post-training, harness-side failures point to scaffolding and tool-integration fixes, and environment or grader failures reveal evaluation conditions requiring redesign. The schema applies across agent architectures, from coding assistants to long-horizon personal assistants and multi-agent systems. We ground the taxonomy in worked examples from public benchmarks, model system cards, published reports, and logged agent trajectories, and evaluate its reproducibility using independent reasoning agents as judges. Across four frontier models, the strongest judge reaches Cohen's $\kappa=0.76$ against human category labels, suggesting that the categories capture shared structure rather than annotator-specific preferences.

Figures

Figures reproduced from arXiv: 2607.28802 by Aakash Sabharwal, Anas Mahmoud, Darvin Yi, Harsh Raj, Razvan-Gabriel Dumitru, Vipul Gupta, Yunzhong He.

Figure 1
Figure 1. Figure 1: Radial interaction map. The model is the hub, the User, Harness, and Environment families form the inner ring, and their components appear on the outer ring. Each failure is represented by an edge between two components. Since a model interacts with another model through its own harness, model–model interactions are grouped under Harness alongside Context, Memory, and Tool. The outer node is another model,… view at source ↗
Figure 2
Figure 2. Figure 2: Interaction-centric taxonomy of 41 failure modes. Failures are organized by the family of the component interacting with the model: User, Harness, or Environment, and then by the specific component within that family. Each branch represents an interaction edge between the model and that component. The hierarchy is organizational, where the model and the interacting component form the two endpoints of each … view at source ↗
Figure 3
Figure 3. Figure 3: Pairwise Cohen’s κ among the four LLM judges and the human annotator, with category labels on the left and complete failure-mode labels on the right. The bold outline marks agreement with the human annotator. the correct category first, so a category error can lead to an incorrect failure-mode label. When given the gold category, accuracy improves for the Opus models ( [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 4
Figure 4. Figure 4: Disambiguation rules applied in the reflection step (Turn 3) to confirm or revise the Turn 2 label. Pipeline. Three turns share one session. Turn 1 (extraction) reconstructs the rollout into a neutral, chronological dossier of what the agent did. Turn 2 (classification) localizes the root-cause failure to one category and failure mode, given the taxonomy definitions of Appendix B and a short instruction to… view at source ↗
Figure 5
Figure 5. Figure 5: A Harbor-Mix rollout where the judge blames the model for a failure that belongs to the environment. Panel (a) shows the actual exchange: after the agent finishes the first phase, it waits for a scripted reply that never comes, and every notification check returns empty. Panel (b) contrasts the two labels for this trace. The judge (Claude Opus-4.7) calls it an Observation Failure and faults the model for n… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

92 extracted references · 37 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2509.16941 , year=

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? , author=. arXiv preprint arXiv:2509.16941 , year=

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2505.00212 , year=

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems , author=. arXiv preprint arXiv:2505.00212 , year=

    Pith/arXiv arXiv

  3. [3]

    From Failed Trajectories to Reliable

    Chen, Mengzhuo and Wang, Junjie and Liu, Zhe and Wang, Yawen and Wang, Qing , journal=. From Failed Trajectories to Reliable

  4. [4]

    arXiv preprint arXiv:2503.13657 , year=

    Why do multi-agent llm systems fail? , author=. arXiv preprint arXiv:2503.13657 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2509.25370 , year=

    Where llm agents fail and how they can learn from failures , author=. arXiv preprint arXiv:2509.25370 , year=

    arXiv

  6. [6]

    2025 , month = apr, type =

    Taxonomy of Failure Mode in Agentic. 2025 , month = apr, type =

    2025

  7. [7]

    arXiv preprint arXiv:2603.06847 , year=

    Characterizing faults in agentic AI: A taxonomy of types, symptoms, and root causes , author=. arXiv preprint arXiv:2603.06847 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2603.04259 , year=

    When AI Fails, What Works? A Data-Driven Taxonomy of Real-World AI Risk Mitigation Strategies , author=. arXiv preprint arXiv:2603.04259 , year=

    arXiv

  9. [9]

    arXiv preprint arXiv:2511.19933 , year=

    Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications , author=. arXiv preprint arXiv:2511.19933 , year=

    arXiv

  10. [10]

    arXiv preprint arXiv:2605.17467 , year=

    VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems , author=. arXiv preprint arXiv:2605.17467 , year=

    Pith/arXiv arXiv

  11. [11]

    Claude Agent SDK for Python , year =

  12. [12]

    arXiv preprint arXiv:2404.18796 , year=

    Replacing judges with juries: Evaluating llm generations with a panel of diverse models , author=. arXiv preprint arXiv:2404.18796 , year=

    Pith/arXiv arXiv

  13. [13]

    The thirteenth international conference on learning representations , year=

    Openrca: Can large language models locate the root cause of software failures? , author=. The thirteenth international conference on learning representations , year=

  14. [14]

    arXiv preprint arXiv:2602.02475 , year=

    AgentRx: Diagnosing AI Agent Failures from Execution Trajectories , author=. arXiv preprint arXiv:2602.02475 , year=

    arXiv

  15. [15]

    arXiv preprint arXiv:2510.07593 , year=

    AgentAsk: Multi-agent systems need to ask , author=. arXiv preprint arXiv:2510.07593 , year=

    arXiv

  16. [16]

    2025 , eprint =

    Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows , author =. 2025 , eprint =

    2025

  17. [17]

    arXiv preprint arXiv:2604.09408 , year=

    HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help? , author=. arXiv preprint arXiv:2604.09408 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2510.16380 , year=

    Morebench: Evaluating procedural and pluralistic moral reasoning in language models, more than outcomes , author=. arXiv preprint arXiv:2510.16380 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2502.13295 , year=

    Demonstrating specification gaming in reasoning models , author=. arXiv preprint arXiv:2502.13295 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2509.21054 , year=

    Disagreements in Reasoning: How a Model's Thinking Process Dictates Persuasion in Multi-Agent Systems , author=. arXiv preprint arXiv:2509.21054 , year=

    arXiv

  21. [21]

    arXiv preprint arXiv:2510.04374 , year=

    Gdpval: Evaluating ai model performance on real-world economically valuable tasks , author=. arXiv preprint arXiv:2510.04374 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2606.05806 , year=

    When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents , author=. arXiv preprint arXiv:2606.05806 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2507.15296 , year=

    Butterfly effects in toolchains: A comprehensive analysis of failed parameter filling in llm tool-agent systems , author=. arXiv preprint arXiv:2507.15296 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2603.05637 , year=

    Real faults in model context protocol (mcp) software: a comprehensive taxonomy , author=. arXiv preprint arXiv:2603.05637 , year=

    arXiv

  25. [25]

    arXiv preprint arXiv:2604.28093 , year=

    What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design , author=. arXiv preprint arXiv:2604.28093 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2507.02825 , year=

    Establishing best practices for building rigorous agentic benchmarks , author=. arXiv preprint arXiv:2507.02825 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2605.26667 , year=

    MemFail: Stress-Testing Failure Modes of LLM Memory Systems , author=. arXiv preprint arXiv:2605.26667 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2605.06527 , year=

    STALE: Can LLM Agents Know When Their Memories Are No Longer Valid? , author=. arXiv preprint arXiv:2605.06527 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2603.11768 , year=

    Governing evolving memory in llm agents: Risks, mechanisms, and the stability and safety governed memory (ssgm) framework , author=. arXiv preprint arXiv:2603.11768 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2606.13177 , year=

    MemRefine: LLM-Guided Compression for Long-Term Agent Memory , author=. arXiv preprint arXiv:2606.13177 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2505.16067 , year=

    How memory management impacts llm agents: An empirical study of experience-following behavior , author=. arXiv preprint arXiv:2505.16067 , year=

    arXiv

  32. [32]

    arXiv preprint arXiv:2606.22388 , year=

    PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems , author=. arXiv preprint arXiv:2606.22388 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2510.03285 , year=

    WAREX: Web Agent Reliability Evaluation on Existing Benchmarks , author=. arXiv preprint arXiv:2510.03285 , year=

    arXiv

  34. [34]

    arXiv preprint arXiv:2607.11098 , year=

    AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP , author=. arXiv preprint arXiv:2607.11098 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2507.09063 , year=

    SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments , author=. arXiv preprint arXiv:2507.09063 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2606.31174 , year=

    ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents , author=. arXiv preprint arXiv:2606.31174 , year=

    Pith/arXiv arXiv

  37. [37]

    Forty-third International Conference on Machine Learning , year=

    Aorchestra: Automating sub-agent creation for agentic orchestration , author=. Forty-third International Conference on Machine Learning , year=

  38. [38]

    arXiv preprint arXiv:2601.13295 , year=

    CooperBench: Why Coding Agents Cannot be Your Teammates Yet , author=. arXiv preprint arXiv:2601.13295 , year=

    arXiv

  39. [39]

    International Conference on Learning Representations , volume=

    Swe-bench: Can language models resolve real-world github issues? , author=. International Conference on Learning Representations , volume=

  40. [40]

    2026 , howpublished =

    Why. 2026 , howpublished =

    2026

  41. [41]

    2025 , howpublished =

    Piloting. 2025 , howpublished =

    2025

  42. [42]

    Effective Harnesses for Long-Running Agents , year =

  43. [43]

    2025 , howpublished =

    Project Vend: Can. 2025 , howpublished =

    2025

  44. [44]

    Can Coding Agents Tackle Early-Stage Drug Discovery? , year =

  45. [45]

    2026 , howpublished =

    2026

  46. [46]

    2026 , howpublished =

    Cramer, Jude , title =. 2026 , howpublished =

    2026

  47. [47]

    2026 , howpublished =

    Can. 2026 , howpublished =

    2026

  48. [48]

    2025 , howpublished =

    2025

  49. [49]

    2025 , howpublished =

    Aider issue \#3713:. 2025 , howpublished =

    2025

  50. [50]

    2026 , howpublished =

    Silent Empty Response when. 2026 , howpublished =

    2026

  51. [51]

    arXiv preprint arXiv:2607.18754 , year=

    AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents , author=. arXiv preprint arXiv:2607.18754 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2605.10286 , year=

    AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks , author=. arXiv preprint arXiv:2605.10286 , year=

    Pith/arXiv arXiv

  53. [53]

    2026 , howpublished =

    Raj, Harsh , title =. 2026 , howpublished =

    2026

  54. [54]

    2026 , howpublished =

    Wang, Ruiyang , title =. 2026 , howpublished =

    2026

  55. [55]

    arXiv preprint arXiv:2602.00933 , year=

    Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers , author=. arXiv preprint arXiv:2602.00933 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint , year=

    LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks , author=. arXiv preprint , year=

  57. [57]

    dataclaw-peteromallet , year =

  58. [58]

    pi-playdate , year =

  59. [59]

    IEEE transactions on dependable and secure computing , volume=

    Basic concepts and taxonomy of dependable and secure computing , author=. IEEE transactions on dependable and secure computing , volume=. 2004 , publisher=

    2004

  60. [60]

    Safety science , volume=

    A new accident model for engineering safer systems , author=. Safety science , volume=. 2004 , publisher=

    2004

  61. [61]

    Findings of the association for computational linguistics: ACL 2023 , pages=

    Discovering language model behaviors with model-written evaluations , author=. Findings of the association for computational linguistics: ACL 2023 , pages=

    2023

  62. [62]

    International Conference on Learning Representations , volume=

    Towards understanding sycophancy in language models , author=. International Conference on Learning Representations , volume=

  63. [63]

    Specification Gaming: The Flip Side of

    Krakovna, Victoria and Uesato, Jonathan and Mikulik, Vladimir and Rahtz, Matthew and Everitt, Tom and Kumar, Ramana and Kenton, Zac and Leike, Jan and Legg, Shane , year =. Specification Gaming: The Flip Side of

  64. [64]

    Advances in Neural Information Processing Systems , volume=

    Defining and characterizing reward gaming , author=. Advances in Neural Information Processing Systems , volume=

  65. [65]

    arXiv preprint arXiv:1606.06565 , year=

    Concrete problems in AI safety , author=. arXiv preprint arXiv:1606.06565 , year=

    Pith/arXiv arXiv

  66. [66]

    Proceedings of the 16th ACM workshop on artificial intelligence and security , pages=

    Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection , author=. Proceedings of the 16th ACM workshop on artificial intelligence and security , pages=

  67. [67]

    arXiv preprint arXiv:2505.23836 , year=

    Large language models often know when they are being evaluated , author=. arXiv preprint arXiv:2505.23836 , year=

    Pith/arXiv arXiv

  68. [68]

    Advances in Neural Information Processing Systems , volume=

    Gorilla: Large language model connected with massive apis , author=. Advances in Neural Information Processing Systems , volume=

  69. [69]

    arXiv preprint arXiv:2505.02709 , year=

    Technical report: Evaluating goal drift in language model agents , author=. arXiv preprint arXiv:2505.02709 , year=

    Pith/arXiv arXiv

  70. [70]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  71. [71]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    Pith/arXiv arXiv

  72. [72]

    International Conference on Learning Representations , volume=

    Identifying the risks of lm agents with an lm-emulated sandbox , author=. International Conference on Learning Representations , volume=

  73. [73]

    , author=

    MemGPT: towards LLMs as operating systems. , author=. 2023 , publisher=

    2023

  74. [74]

    ACM Transactions on Information Systems , volume=

    A survey on the memory mechanism of large language model-based agents , author=. ACM Transactions on Information Systems , volume=. 2025 , publisher=

    2025

  75. [75]

    arXiv preprint arXiv:2307.03172 , year=

    Lost in the middle: How language models use long contexts , author=. arXiv preprint arXiv:2307.03172 , year=

    Pith/arXiv arXiv

  76. [76]

    2025 , note =

    Zharmagambetov, Arman and Guo, Chuan and Evtimov, Ivan and Pavlova, Maya and Salakhutdinov, Ruslan and Chaudhuri, Kamalika , booktitle =. 2025 , note =

    2025

  77. [77]

    2601.06663 , archivePrefix =

    Zhou, Kaiwen and Jangam, Shreedhar and Nagarajan, Ashwin and Polu, Tejas and Oruganti, Suhas and Liu, Chengzhi and Kuo, Ching-Chen and Zheng, Yuting and Narayanaraju, Sravana and Wang, Xin Eric , year =. 2601.06663 , archivePrefix =

    arXiv

  78. [78]

    arXiv preprint arXiv:2606.27154 , year=

    OpenRCA 2.0: From Outcome Labels to Causal Process Supervision , author=. arXiv preprint arXiv:2606.27154 , year=

    Pith/arXiv arXiv

  79. [79]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  80. [80]

    2024 , month = sep, howpublished =

    Learning to Reason with. 2024 , month = sep, howpublished =

    2024

Showing first 80 references.