REVIEW 22 cited by
AgentOps: Enabling Observability of LLM Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
AgentOps: Enabling Observability of LLM Agents
read the original abstract
Large language model (LLM) agents have demonstrated remarkable capabilities across various domains, gaining extensive attention from academia and industry. However, these agents raise significant concerns on AI safety due to their autonomous and non-deterministic behavior, as well as continuous evolving nature . From a DevOps perspective, enabling observability in agents is necessary to ensuring AI safety, as stakeholders can gain insights into the agents' inner workings, allowing them to proactively understand the agents, detect anomalies, and prevent potential failures. Therefore, in this paper, we present a comprehensive taxonomy of AgentOps, identifying the artifacts and associated data that should be traced throughout the entire lifecycle of agents to achieve effective observability. The taxonomy is developed based on a systematic mapping study of existing AgentOps tools. Our taxonomy serves as a reference template for developers to design and implement AgentOps infrastructure that supports monitoring, logging, and analytics. thereby ensuring AI safety.
Forward citations
Cited by 22 Pith papers
-
How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study
Building SE agents reorganizes software process around evaluation-driven iteration and dual-audience specs, while creating new bottlenecks in review, evaluation trust, and model-update instability.
-
How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study
SE-agent development follows a recurring seven-stage loop where evaluation drives iteration, and challenges such as unreliable evaluation signals and comprehension debt emerge.
-
FlowSteer: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems
FlowSteer is a prompt-only attack that biases multi-agent LLM workflow planning to propagate malicious signals, raising success rates by up to 55%, with FlowGuard as an input-side defense reducing it by up to 34%.
-
Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents
PROBE structures runtime telemetry into diagnoses and evidence-grounded guidance, raising recovery rates by 12.45 points over baselines on 257 unresolved software repair and AIOps cases.
-
AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
An open-source agent-debugging loop attributes failures to the responsible step and uses the diagnosis to repair failed runs, recovering 13 of 73 GAIA tasks.
-
Harnessing Agent Skills: Architectural Patterns and a Reference Architecture for Skill-Mediated LLM Agents
Catalogs ten patterns and synthesizes a four-layer reference architecture for skill harnessing in LLM agents, evaluated via cross-instantiation on eight systems.
-
TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
TrajAudit pinpoints the earliest wrong step in long AI-coding-agent logs with 50.9% exact accuracy on the new RootSE benchmark, beating prior methods by about 24 percentage points.
-
TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
TrajAudit diagnoses failures in repository-level agentic coding trajectories by filtering noise and injecting test-failure priors, achieving >24.4 pp higher localization accuracy and 18% lower token use on the new Roo...
-
TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
TrajAudit localizes the earliest decisive failure step in long, noisy coding-agent trajectories via saliency folding, test-report priors, and an investigator agent, beating baselines on RootSE.
-
In-IDE Toolkit for Developers of AI-Based Features
Presents an AI Toolkit plugin for JetBrains IDEs that integrates trace capture and evaluation into the Run/Debug loop, guided by practitioner needs and showing early adoption signals in PyCharm.
-
Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents
PROBE turns runtime telemetry from failed software engineering agent runs into evidence-grounded diagnoses and actionable recovery guidance, achieving 65.37% diagnosis accuracy and 21.79% recovery rate on 257 cases.
-
Towards Security-Auditable LLM Agents: A Unified Graph Representation
Agent-BOM is a unified hierarchical attributed directed graph that models static capability bases and dynamic semantic states of LLM agents for path-level security auditing and risk assessment.
-
ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation
ClawTrace enables cost-aware LLM agent skill distillation by tracing per-step costs and generating preserve, prune, and repair patches, with ablations showing reduced regressions and prune rules transferring to cut co...
-
Auditable Agents
No agent system can be accountable without auditability, which requires five dimensions (action recoverability, lifecycle coverage, policy checkability, responsibility attribution, evidence integrity) and mechanisms f...
-
Toward Continuous Assurance for the Democratization of AI Agent Creation in Industry
A proposed framework and prototype auditor aim to keep citizen-created AI agents operationally ready by checking dependencies and contracts on a schedule.
-
GRADE: Graph Representation of LLM Agent Dependency and Execution
GRADE models any LLM agent run as a graph with execution and graded dependency edge layers to enable failure prediction and fault localization across tool, coding, and web agent corpora.
-
From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents
A survey that introduces a taxonomy for evidence tracing and execution provenance in LLM agents and reviews methods for building provenance-aware, auditable agent systems.
-
From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents
This survey defines execution provenance as a typed graph of agent execution and evidence tracing as its projection onto evidence-support relations, then reviews methods, taxonomy, benchmarks, and challenges for audit...
-
A Query Engine for the Agents
Hyperparam supplies under-70KB JS libraries (Hyparquet, Squirreling, Icebird) for async-native SQL over Parquet/Iceberg with per-cell LLM UDFs, claiming 300x speedup versus DuckDB-WASM on filter queries and two-thirds...
-
From Assistance to Agency: Rethinking Autonomy and Control in CI/CD Pipelines
The central challenge in AI-augmented CI/CD is designing authority transfer from humans to agents under constraints, as current systems remain limited to bounded data-plane autonomy backed by external governance.
-
Agent Behavior Mining: Generative AI Agent Governance in Business Processes
Introduces Agent Behavior Mining to translate generative AI agent activities into standardized process logs, enabling process mining for policy deviation detection and variability quantification in business processes,...
-
Agent System Operations: Categorization, Challenges, and Future Directions
This survey categorizes anomalies in agent systems into intra-agent and inter-agent types and introduces the AgentOps framework with four operational stages.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.