Pith. sign in

REVIEW 3 major objections 4 minor 278 references

Agentic AI cannot be validated by component tests alone: what must be assessed is the full action trajectory in context, the central claim of this 257-paper survey.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:36 UTC pith:ZSEUPTBZ

load-bearing objection A well-structured survey with a genuinely useful five-axis taxonomy and an honest limitations section; the quantitative gap claims would be more convincing if the corpus and coding trace were released, but this deserves serious peer review. the 3 major comments →

arxiv 2607.29405 v1 pith:ZSEUPTBZ submitted 2026-07-31 cs.AI cs.MAcs.SE

Beyond Component Testing: Validating Agentic AI Systems

Fabio Orazio Mirto , Luca D'Agati , Giuseppe Tricomi , Stefano Silvestri , Francesco Longo , Antonio Puliafito , Giovanni Merlino This is my paper
classification cs.AI cs.MAcs.SE
keywords agentic AItrajectory validationruntime monitoringAI assurancesoftware testingmulti-agent systemstemporal validityregulatory evidence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that agentic AI systems—software that pursues goals through planning, tool use, memory, and adaptation—cannot be trusted on the strength of component-level tests or one-shot input–output evaluation. What must be validated instead is the realized trajectory: the whole sequence of actions, tool calls, memory updates, escalations, and handoffs as it unfolds in a changing environment. To make this case, the authors synthesize 257 papers across software testing, agent evaluation, cyber-physical systems, runtime monitoring, and regulatory guidance, and organize them with a five-dimension taxonomy covering behavioral, safety, temporal, regulatory, and multi-agent concerns. The survey's core finding is asymmetric maturity: behavioral evaluation is comparatively developed, while temporal validity, runtime evidence maintenance, regulatory legibility, and open-ended multi-agent assurance remain structurally under-addressed. If the claim is right, assurance for agentic systems must become a lifecycle activity—monitoring, revalidation triggers, and audit-ready evidence—rather than a release-time test.

Core claim

On its own terms, the paper's discovery is a reframing of the assurance target. For an agentic system the relevant unit of behavior is the trajectory—a temporally ordered sequence of states, actions, and execution annotations—and acceptable behavior must be judged over that sequence, including tool ordering, escalation timing, recovery, and stopping conditions, not merely at the final answer. From a systematic screening of 7,197 records down to 257 papers, the authors derive five validation dimensions—behavioral, safety, temporal, regulatory, and multi-agent—and a coverage matrix showing behavioral work comparatively mature while temporal validity, runtime evidence maintenance, regulatory le

What carries the argument

The load-bearing instrument is the trajectory formalism paired with a five-dimension taxonomy. The trajectory is a deliberately light sequence of states, actions, and execution annotations (tool calls, retrieved evidence, delegations, human handoffs) rich enough to reconstruct why a step happened and what it triggered. The taxonomy converts that object into five validation dimensions—behavioral, safety, temporal, regulatory, multi-agent—each with characteristic failure modes and measurable metrics (trajectory consistency, unsafe-action rate, evidence freshness, trace completeness, coordination success). The taxonomy does the argumentative work: five classical testing mismatches (determinacy,

Load-bearing premise

The directional gap claims—behavioral evaluation mature, temporal and multi-agent assurance under-developed—rest on the 257-paper corpus being representative; the paper itself notes in its limitations section that a single reviewer performed screening and that one database supplied 64% of single-source records, and the sensitivity analysis covers only the eight coding discordances, not the selection step.

What would settle it

Two concrete checks. First, re-run the screening and dimension coding with multiple independent reviewers on the same retrieved records; if the coverage matrix no longer shows behavioral maturity with temporal, regulatory, and multi-agent under-development, the directional claims collapse. Second, a controlled deployment study where an agentic system passes all component and endpoint benchmark tests yet fails trajectory-level criteria—stale-data decisions, mistimed escalation, coordination deadlock—would confirm the central claim, while a study showing component scores predict real incidents j

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Validation practice must add trajectory-level criteria—tool ordering, intermediate states, escalation timing, recovery behavior—on top of unit, integration, and benchmark tests.
  • Temporal validity becomes first-class: evidence packages need explicit expiry conditions and revalidation triggers when tools, memory, or environments drift.
  • Regulatory legibility becomes a continuous runtime property: audit-log coverage and provenance completeness must be maintained, not assembled once at release.
  • Multi-agent systems cannot be certified from single-agent results; interaction-level properties such as deadlock, duplication, and conflict require explicitly coordinated test scenarios.
  • Operational deployments need monitors keyed to concrete trajectory signals—evidence freshness, unsafe-action rate, coordination success rate—that can block or escalate before hazards propagate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the trajectory thesis is right, endpoint benchmark rankings systematically overstate agent trustworthiness; a testable corollary is that scoring intermediate steps would reorder today's leaderboards.
  • Editorial extension: the temporal–regulatory intersection implies a concrete audit mechanism the paper leaves implicit—attaching an 'evidence expiry' timestamp to each assurance claim, so stale claims become mechanically detectable by reviewers.
  • Editorial extension: the paper's own limitation analysis (single-reviewer screening; one database supplying 64% of single-source records) means the maturity asymmetry is itself testable—re-running screening with multiple independent reviewers would show whether the gaps are field structure or selection artifacts.
  • Editorial extension: the multi-agent asymmetry predicts that in deployed agent teams, per-agent success rates will be weak predictors of system-level failure; coordination metrics (handoff latency, conflict frequency) should predict better—an empirical claim a deployment study could settle.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This survey argues that trustworthy deployment of agentic AI requires validating trajectories in context rather than assessing isolated components. It synthesizes 257 papers across software engineering, agent evaluation, CPS validation, runtime assurance, and regulatory guidance through a PRISMA-inspired process, and organizes the field with a five-dimension taxonomy: behavioral, safety, temporal, regulatory, and multi-agent validation. The paper reports a quantified gap analysis claiming behavioral evaluation is comparatively mature while temporal validity, runtime evidence maintenance, regulatory legibility, and open-ended multi-agent assurance are under-developed. It illustrates the taxonomy with three constructed trajectory-level case studies (medical, industrial, smart mobility) and proposes a four-direction research agenda: bounded-autonomy specifications, adversarial trajectory generation, temporal runtime monitoring, and oversight/evidence legibility.

Significance. If the results hold, this is a timely and useful synthesis. The five testing-mismatch argument in §1.2/§5 is clear and internally consistent; the taxonomy gives the field a shared vocabulary for validation objects rather than capabilities; the three case studies operationalize the abstraction at step granularity; and the proposed research agenda is concrete. The paper also deserves credit for reporting its limitations explicitly, including single-reviewer screening, the IEEE Xplore skew, and the classification-only sensitivity analysis. However, the quantified gap claims are a stated contribution, and they rest on the representativeness of a 257-paper corpus produced by a single-reviewer screening pipeline whose audit trail is not released. That makes the directional conclusions currently unverifiable, even though the central conceptual thesis is well supported.

major comments (3)
  1. [§2.8, Table 2] The adversarial sensitivity analysis in Table 2 is limited to reassigning the eight Pass 1/Pass 2 classification discordances among the 257 already-included papers. It does not address selection bias at Stages 1–2, where 7,197 records were reduced to 561 on title/abstract screening, nor the 64% IEEE Xplore share of single-source records reported in §2.2. Since the directional gap claims in §7 (behavioral mature; temporal/regulatory/multi-agent under-developed) and the research agenda in §9 are presented as consequences of the coded corpus, these claims are not yet auditable. A second reviewer screening a random sample of Stage 1–2 exclusions, or an explicit selection-bias model, would materially strengthen the paper; alternatively, the directional conclusions should be substantially softened. This is load-bearing because the quantified gap analysis is a stated contribution.
  2. [Declarations; §§2.4–2.5] The statement 'Data availability: Not applicable' is in tension with the PRISMA-inspired method and with the paper's own references to a 'decision trace' and 'coding trace' in §§2.4–2.5. Without the included/excluded record list, the screening decisions, and the primary-dimension coding, a reader cannot verify the 257-paper corpus composition or the coverage matrix in Tables 6–7. For a survey whose novelty rests on systematic corpus coding and quantified gaps, the underlying data should be released as supplementary material. At minimum, the screening decision log and the per-paper primary-dimension coding for all 257 papers should be provided.
  3. [§7, Table 6] Table 6 is the evidentiary core of the gap analysis and is cited repeatedly in §9 to justify specific research directions (e.g., 'Behavioral×Assurance Cases ... only 4 coded appearances', 'Temporal×CPS ... only 3 coded appearances'). However, the table's content is not present in the manuscript as provided; only the caption appears. If this is an artifact of the review copy, please ensure the published version contains the matrix. If it is genuinely absent, the research agenda is currently anchored to cell counts the reader cannot check.
minor comments (4)
  1. [§2.4, §2.6] Figure 1 and Figure 2 are referenced but not embedded in the provided text; ensure the PRISMA flow diagram and the year-band composition figure are included in the final version.
  2. [§7.2] The sentence beginning 'From verifiable safeguards...' appears incomplete: 'From verifiable safeguards and safety-requirement derivation. Explainable and unified-safeguard approaches...' should be merged or rewritten for grammatical continuity.
  3. [§7] The relationship between Table 6 and Table 7 is explained only after both are referenced. Consider introducing the two counting layers before the first citation of either table, so the reader knows that primary-dimension counts and appearance counts measure different things.
  4. [References] The reference list is uneven: many IEEE conference entries lack titles or full page ranges, while arXiv entries include DOIs. A consistent style would improve readability and verifiability.

Circularity Check

0 steps flagged

No significant circularity: the central claim is an interpretive synthesis supported by external literature, and the coded-corpus gap analysis is descriptive, not a fitted prediction.

full rationale

The paper's central claim — that trustworthy deployment of agentic AI depends on validating trajectories in context rather than isolated components alone — is argued from the five testing mismatches in §1.2/Table 1 and the structural analysis in §5, supported by cited external evidence (e.g., Yehudai et al. 2025; Luo et al. 2025; Greshake et al. 2023; FDA/MDCG guidance). No equation or fitted parameter re-enters the derivation: the taxonomy in §6 is presented as an organizing scheme, and the coverage matrix in Tables 6–7 is a descriptive summary of the coded corpus, not a quantity predicted from a fitted model. Table 2 is an explicit sensitivity audit of eight coding discordances, and the paper itself concedes the main selection-bias risk in §2.8: 'A single reviewer conducted title, abstract, and full-text screening, creating a risk of systematic selection bias' and 'Internal checks cannot exclude systematic interpretation bias in the single-reviewer selection stage, which may have shifted corpus composition.' The only mildly self-referential loop is that the five-dimension taxonomy is said to be 'derived from the primary assurance burdens that recur across the corpus' (§6) and is then used to code that same corpus (§2.5); however, this is an in-sample qualitative organizing device, not a derivation that reduces a predicted output to its own inputs. The case-study traces are explicitly labeled constructed illustrations: 'they are not incident reports or independent empirical evidence' (§8), so they are not presented as independent predictions. No load-bearing self-citations appear: the authors do not justify the central premise by citing their own prior work, and the cited uniqueness/assurance frameworks are external. Corpus representativeness and single-reviewer screening are genuine validity threats, but they are acknowledged limitations about evidence quality, not circularity in the derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no fitted constants or new physical/technical entities. Its load-bearing assumptions are about the adequacy of the trajectory abstraction, the completeness of the taxonomy, corpus representativeness, and the validity of the constructed case-study traces.

axioms (4)
  • domain assumption A finite trajectory τ=((s_t,a_t,m_t), ...) is the appropriate unit of analysis for validation.
    Section 4.1 defines trajectory as the core validation object; if this lightweight model cannot capture the relevant state annotations, the taxonomy and gap analysis could miss failures.
  • domain assumption The five validation dimensions (behavioral, safety, temporal, regulatory, multi-agent) are comprehensive and analytically distinct.
    Section 6 presents the taxonomy as complete, but the paper acknowledges intersections and does not prove exhaustiveness. Completeness is assumed rather than derived.
  • domain assumption The screened 257-paper corpus is representative enough to support directional gap claims.
    Sections 2.2-2.8 describe single-reviewer screening, IEEE Xplore dominance, and a frozen codebook. The sensitivity analysis in Table 2 only tests classification discordances, not selection bias, so representativeness is assumed.
  • domain assumption Cited failure patterns in medical, industrial, and mobility settings generalize to the constructed trajectory traces.
    Section 8 explicitly labels the traces as illustrative constructions grounded in the reviewed literature; the transfer from cited response-level or domain studies to agentic trajectory defects is an interpretive inference.

pith-pipeline@v1.3.0-daily-deepseek · 43069 in / 8066 out tokens · 96593 ms · 2026-08-03T07:36:41.100729+00:00 · methodology

0 comments
read the original abstract

Agentic AI systems act through multi-step trajectories that combine planning, tool use, memory, interaction, and adaptation. This behavior stretches validation practice beyond component testing and one-shot input--output evaluation, because acceptable system behavior now depends on how decisions unfold over time and under changing environmental conditions. This survey synthesizes 257 papers spanning agent evaluation, software assurance, cyber-physical systems, runtime monitoring, and regulatory guidance in order to characterize the validation problem for agentic systems. The review is organized around a five-dimension taxonomy covering behavioral, safety, temporal, regulatory, and multi-agent concerns, and uses that taxonomy to map current approaches and expose recurrent coverage gaps. The analysis shows that behavioral evaluation is comparatively mature, while temporal validity, runtime evidence maintenance, regulatory legibility, and open-ended multi-agent systems assurance remain under-developed. Three cross-domain case studies (medical care, industrial operations, smart-mobility systems) provide operational illustrations of how the five taxonomy dimensions recur in safety-critical settings, grounded in the failure patterns documented in the reviewed literature. The paper concludes with a lifecycle-oriented research agenda centered on bounded-autonomy specifications, adversarial trajectory generation, runtime monitoring, and audit-ready evidence structures. The central claim is that trustworthy deployment of agentic AI depends on validating trajectories in context rather than assessing isolated components alone.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

278 extracted references · 2 canonical work pages

  1. [1]

    2021 , type =

    Artificial Intelligence/Machine Learning (. 2021 , type =

    2021

  2. [2]

    2021 , type =

    Good Machine Learning Practice for Medical Device Development: Guiding Principles , institution =. 2021 , type =

    2021

  3. [3]

    2017 , note =

    Regulation (. 2017 , note =

    2017

  4. [4]

    Collaco, B. G. and Haider, S. A. and Prabha, S. and others , title =. npj Digital Medicine , year =

  5. [5]

    and Liu, S

    Zhao, L. and Liu, S. and Xin, T. and others , title =. npj Artificial Intelligence , year =. doi:10.1038/s44387-026-00076-4 , url =

    work page doi:10.1038/s44387-026-00076-4

  6. [6]

    2025 , eprint =

    Luo, Junyu and Zhang, Weizhi and Yuan, Ye and Zhao, Yusheng and others , title =. 2025 , eprint =. doi:10.48550/arXiv.2503.21460 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.21460 2025

  7. [7]

    and Burger, Doug and Wang, Chi , title =

    Wu, Qingyun and Bansal, Gagan and Zhang, Jiale and Wu, Yiran and Li, Beibin and Zhu, Erkang and Jiang, Li and Zhang, Xiaoyun and Zhang, Chi and Liu, Shaokun and Awadallah, Ahmed Hassan and White, Ryen W. and Burger, Doug and Wang, Chi , title =. Proceedings of the First Conference on Language Modeling (. 2024 , publisher =. doi:10.48550/arXiv.2308.08155 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.08155 2024

  8. [8]

    Yu , title =

    Henry Peng Zou and Wei-Chieh Huang and Yaozu Wu and Jizhou Guo and Yankai Chen and Chunyu Miao and Hoang Nguyen and Yue Zhou and Weizhi Zhang and Liancheng Fang and Hanrong Zhang and Fangxin Wang and Pengfei Zhang and Huacan Wang and Langzhou He and Yangning Li and Dongyuan Li and Renhe Jiang and Xue Liu and Philip S. Yu , title =. 2026 , eprint =. doi:10...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.00753 2026

  9. [9]

    and Sandler, Corey and Badgett, Tom , title =

    Myers, Glenford J. and Sandler, Corey and Badgett, Tom , title =

  10. [10]

    Beizer, Boris , title =

  11. [11]

    Ammann, Paul and Offutt, Jeff , title =

  12. [12]

    Draelos, R. L. and Afreen, S. and Blasko, B. and others , title =. npj Digital Medicine , year =

  13. [13]

    and Meshkat, S

    Sharma, D. and Meshkat, S. and Perivolaris, A. and others , title =. npj Digital Medicine , year =

  14. [14]

    Scientific Reports , year =

    Alelyani, Tareq , title =. Scientific Reports , year =

  15. [15]

    and Li, X

    Yu, M. and Li, X. and He, X. and others , title =. Proceedings of the 31st. 2025 , publisher =

    2025

  16. [16]

    2025 , eprint =

    Yehudai, Avshalom and Eden, Lior and Li, Andrew and Uziel, Guy and Zhao, Yifei and Bar-Haim, Roy and Cohan, Arman and Shmueli-Scheuer, Michal , title =. 2025 , eprint =. doi:10.48550/arXiv.2503.16416 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.16416 2025

  17. [17]

    and Ray, Baishakhi , title =

    Chen, Shuo and Singh, Shreshth and Pusarla, Prasoon and Hou, Bowen and Bhat, Sheshadri and Fields, Benjamin A. and Ray, Baishakhi , title =. 2025 , doi =

    2025

  18. [18]

    2025 , eprint =

    Jha, Shashi and Bhat, Praneeth and Khanduja, Anmol and Shenoy, Pushkar and Otte, Matt and Sreekanti, Vikram and Tumanov, Alexey and Posada, Jose and Mytkowicz, Todd and Sarma, Sanjay , title =. 2025 , eprint =. doi:10.48550/arXiv.2502.05352 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.05352 2025

  19. [19]

    2025 , eprint =

    Badertdinov, Ibragim and Golubev, Alexander and Nekrashevich, Maksim and Shevtsov, Anton and Karasik, Simon and Andriushchenko, Andrei and Trofimova, Maria and Litvintseva, Daria and Yangel, Boris , title =. 2025 , eprint =. doi:10.48550/arXiv.2505.20411 , note =

    doi:10.48550/arxiv.2505.20411 2025

  20. [20]

    , title =

    Zhu, Kaijie and Zhang, Junwei and Xiao, Yifei and Liu, Xin and Song, Kai and Tan, Chuan and Zeng, Bohan and Wang, Jialong and Xing, Eric P. , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

  21. [21]

    2025 , eprint =

    Cemri, Mert and Lyu, Yiran and Lo, Yatong and Swaminathan, Adith and Nandwani, Yatin and Wang, Zihan and Gupta, Shashank , title =. 2025 , eprint =. doi:10.48550/arXiv.2503.13657 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.13657 2025

  22. [22]

    Proceedings of the 16th

    Lin, Vivian and Brown, Ryan and Jafarpour, Saman and Mangharam, Rahul , title =. Proceedings of the 16th. 2025 , pages =

    2025

  23. [23]

    and Xiang, Weiming and Wolf, Marilyn and Tran, Hoang-Dung , title =

    Brown, Ryan and Nguyen, Luan V. and Xiang, Weiming and Wolf, Marilyn and Tran, Hoang-Dung , title =. Proceedings of the 16th. 2025 , pages =

    2025

  24. [24]

    Systems, Software and Services Process Improvement (

    Veledar, Olivera and Damjanovic-Behrendt, Violeta and Macher, Georg , title =. Systems, Software and Services Process Improvement (. 2019 , pages =

    2019

  25. [25]

    and Chen, Shangwu and Wang, Ruian and McCall, Matthew and Titzer, Ben L

    Zhong, Pengyuan Y. and Chen, Shangwu and Wang, Ruian and McCall, Matthew and Titzer, Ben L. and Miller, Heather and Gibbons, Phillip B. , title =. 2025 , eprint =. doi:10.48550/arXiv.2502.08966 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.08966 2025

  26. [26]

    2023 , eprint =

    Greshake, Kai and Abdelnabi, Sahar and Mishra, Shailesh and Endres, Christoph and Holz, Thorsten and Fritz, Mario , title =. 2023 , eprint =. doi:10.48550/arXiv.2302.12173 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.12173 2023

  27. [27]

    AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for

    Debenedetti, Edoardo and Zhang, Jiawei and Balunovic, Mislav and Beurer-Kellner, Luca and Fischer, Marc and Tram. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for. 2024 , eprint =. doi:10.48550/arXiv.2406.13352 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.13352 2024

  28. [28]

    Zico and Fredrikson, Matt and Winsor, Emma and Wynne, Jonathan and Gal, Yarin and Davies, Xander , title =

    Andriushchenko, Maksym and Souly, Alexandra and Dziemian, Maksymilian and Duenas, Daniel and Lin, Max and Wang, Jiawei and Hendrycks, Dan and Zou, Andy and Kolter, J. Zico and Fredrikson, Matt and Winsor, Emma and Wynne, Jonathan and Gal, Yarin and Davies, Xander , title =. 2024 , eprint =. doi:10.48550/arXiv.2410.09024 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.09024 2024

  29. [29]

    and Paterson, Colin and Picardi, Claudia and Jia, Yue and Calinescu, Radu and Habli, Ibrahim , title =

    Hawkins, Richard D. and Paterson, Colin and Picardi, Claudia and Jia, Yue and Calinescu, Radu and Habli, Ibrahim , title =. 2021 , type =. doi:10.48550/arXiv.2102.01564 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2102.01564 2021

  30. [30]

    and Zhang, Y

    Wang, S. and Zhang, Y. and Feng, D. and Ge, Y. and Zhou, M. and Liu, Y. and Zhang, N. and Wang, H. , title =. 2025 , eprint =. doi:10.48550/arXiv.2502.11127 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.11127 2025

  31. [31]

    2025 , eprint =

    Koohestani, Reza , title =. 2025 , eprint =. doi:10.48550/arXiv.2509.23864 , note =

    doi:10.48550/arxiv.2509.23864 2025

  32. [32]

    2025 , type =

    Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations , institution =. 2025 , type =

    2025

  33. [33]

    2025 , type =

    Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions , institution =. 2025 , type =

    2025

  34. [34]

    SyntheaTM and synthetic patient generation for healthcare simulation and evaluation , year =

  35. [35]

    and Finfer, Simon and others , title =

    Kuo, Nicholas I-Hsien and Polizzotto, Mark N. and Finfer, Simon and others , title =. Scientific Data , year =. doi:10.1038/s41597-022-01784-7 , url =

    doi:10.1038/s41597-022-01784-7

  36. [36]

    2023 , eprint=

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author=. 2023 , eprint=

    2023

  37. [37]

    2018 , eprint=

    AI Safety via Debate , author=. 2018 , eprint=

    2018

  38. [38]

    doi:10.1109/ICAIDES67265.2025.11404085 , note =

    COSTAR: Cloud-Observed Safety and Trust-Aware Agentic Reasoning for Enterprise Workflows , year =. doi:10.1109/ICAIDES67265.2025.11404085 , note =

    arXiv 2025

  39. [39]

    Machine Learning Operations (

    Kreuzberger, Dominik and K. Machine Learning Operations (. IEEE Access , year =

  40. [40]

    Proceedings of the 2024

    Leest, Joran and Raibulet, Claudia and Gerostathopoulos, Ilias and Lago, Patricia , title =. Proceedings of the 2024. 2024 , pages =. doi:10.1145/3639476.3639771 , note =

    arXiv 2024

  41. [41]

    Acharya, D. B. and Kuppan, K. and Divya, B. , title =. IEEE Access , year =

  42. [42]

    and others , title =

    Abou Ali, N. and others , title =. 2025 , eprint =. doi:10.48550/arXiv.2510.25445 , note =

    doi:10.48550/arxiv.2510.25445 2025

  43. [44]

    A Comprehensive Study and Implementation of Agentic AI via MCP Servers , doi =

    Kumar, Nitin and Sagar, Vikas and Jain, Garima , booktitle =. A Comprehensive Study and Implementation of Agentic AI via MCP Servers , doi =. 2025 , month =

    2025

  44. [46]

    doi:10.1109/ACCESS.2026.3651218 , note =

    Agentic AI in Healthcare and Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-Based Agents , year =. doi:10.1109/ACCESS.2026.3651218 , note =

    arXiv 2026

  45. [47]

    doi:10.1109/CCNC65079.2026.11366609 , note =

    Guardrailing LLM and Agentic Decisions for 6G AI-RAN , year =. doi:10.1109/CCNC65079.2026.11366609 , note =

    arXiv 2026

  46. [48]

    doi:10.1109/ICMLA66185.2025.00213 , note =

    LLMZ+: Contextual Prompt Whitelist Principles for Agentic LLMs , year =. doi:10.1109/ICMLA66185.2025.00213 , note =

    arXiv 2025

  47. [49]

    Toward Constitutional Autonomy in AI Systems: A Theoretical Framework for Aligned Agentic Intelligence , year=

    Torgbi Agbemabiese, William , journal=. Toward Constitutional Autonomy in AI Systems: A Theoretical Framework for Aligned Agentic Intelligence , year=

  48. [50]

    Jose, Joysh Jayaron and Jose, Joshika Annie and Bamini, A. M. Anusha , booktitle=. Evolving Data, Evolving Models: Analyzing Concept Drift with MLOps , year=

  49. [51]

    NeuroMetaQDS: An Interpretable Rule Based Quantized Decision System for Robust Regression Under Concept Drift , year=

    R, Sasirekha and Abrar, Shaik Mohammad and Pasha, Shaik Adil , booktitle=. NeuroMetaQDS: An Interpretable Rule Based Quantized Decision System for Robust Regression Under Concept Drift , year=

  50. [52]

    Updating Machine Learning Models in Real-World Applications: Finding Best Deployment Settings , year=

    Poray, Julia and Jungmann, Paul and Franczyk, Bogdan and Heller, Thomas , booktitle=. Updating Machine Learning Models in Real-World Applications: Finding Best Deployment Settings , year=

  51. [53]

    Extending the TMMi Framework for Secure Testing of AI Agents , year=

    Peri, Lakshmi Vidya , booktitle=. Extending the TMMi Framework for Secure Testing of AI Agents , year=

  52. [54]

    and Anderegg, Alfred H

    Ferrell, Uma D. and Anderegg, Alfred H. Andy , booktitle=. Validation of Assurance Case for Dynamic Systems , year=

  53. [55]

    Cognitive SOC: Evidence-Backed Narrative Generation for Security Operations with Multi-Agent LLM Architecture , year=

    Sheikhi, Saeid and Kostakos, Panos and Loven, Lauri , booktitle=. Cognitive SOC: Evidence-Backed Narrative Generation for Security Operations with Multi-Agent LLM Architecture , year=

  54. [56]

    doi:10.1109/VIS60296.2025.00037 , note =

    The Agentopia Times: Understanding and Mitigating Hallucinations in Multi-Agent LLM Systems via Data Journalism Gameplay , year =. doi:10.1109/VIS60296.2025.00037 , note =

    arXiv 2025

  55. [57]

    2023 , eprint=

    Reactive Multi-agent Coordination using Auction-based Task Allocation and Behavior Trees , author=. 2023 , eprint=

    2023

  56. [58]

    2025 , eprint=

    Towards Engineering Multi-Agent LLMs: A Protocol-Driven Approach , author=. 2025 , eprint=

    2025

  57. [59]

    Learning Multi-Agent Coordination for Replenishment At Sea , year=

    Han, Byeolyi and Cho, Minwoo and Chen, Letian and Paleja, Rohan and Wu, Zixuan and Ye, Sean and Seraj, Esmaeil and Sidoti, David and Gombolay, Matthew , journal=. Learning Multi-Agent Coordination for Replenishment At Sea , year=

  58. [61]

    2307.13854 , archivePrefix =

    WebArena: A Realistic Web Environment for Building Autonomous Agents , year =. 2307.13854 , archivePrefix =

    Pith/arXiv arXiv

  59. [62]

    2406.12045 , archivePrefix =

    \ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , year =. 2406.12045 , archivePrefix =

    Pith/arXiv arXiv

  60. [63]

    2401.13178 , archivePrefix =

    AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents , year =. 2401.13178 , archivePrefix =

    Pith/arXiv arXiv

  61. [64]

    2412.14161 , archivePrefix =

    TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks , year =. 2412.14161 , archivePrefix =

    Pith/arXiv arXiv

  62. [65]

    and Li, Y

    Mohammadi, M. and Li, Y. and others , title =. Proceedings of the 31st. 2025 , publisher =

    2025

  63. [66]

    2308.10848 , archivePrefix =

    AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors , year =. 2308.10848 , archivePrefix =

    Pith/arXiv arXiv

  64. [67]

    Sculley, D. and Holt, Gary and Golovin, Daniel and Davydov, Eugene and Phillips, Todd and Ebner, Dietmar and Chaudhary, Vinay and Young, Michael and Crespo, Jean-Francois and Dennison, Dan , title =. Advances in Neural Information Processing Systems 28 (. 2015 , publisher =

    2015

  65. [68]

    , title =

    Breck, Eric and Cai, Shanqing and Nielsen, Eric and Salib, Michael and Sculley, D. , title =. 2017. 2017 , publisher =. doi:10.1109/BigData.2017.8258038 , numpages =

    arXiv 2017

  66. [69]

    2023 , eprint =

    Zha, Daochen and Bhat, Zaid Pervaiz and Lai, Kwei-Herng and Yang, Fan and Jiang, Zhimeng and Zhong, Shaochen and Hu, Xia , title =. 2023 , eprint =. doi:10.48550/arXiv.2303.10158 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.10158 2023

  67. [70]

    , title =

    Quionero-Candela, Joaquin and Sugiyama, Masashi and Schwaighofer, Anton and Lawrence, Neil D. , title =

  68. [71]

    Concrete Problems in AI Safety , year =

    Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Man. Concrete Problems in AI Safety , year =. doi:10.48550/arXiv.1606.06565 , note =. 1606.06565 , archivePrefix =

    Pith/arXiv arXiv doi:10.48550/arxiv.1606.06565

  69. [72]

    2022 , eprint =

    Kadavath, Saurabh and others , title =. 2022 , eprint =. doi:10.48550/arXiv.2207.05221 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2207.05221 2022

  70. [73]

    and Steinhardt, Jacob , title =

    Lipton, Zachary C. and Steinhardt, Jacob , title =. 2018 , eprint =. doi:10.48550/arXiv.1807.03341 , note =

    Pith/arXiv arXiv doi:10.48550/arxiv.1807.03341 2018

  71. [74]

    CHI Conference on Human Factors in Computing Systems (

    Sambasivan, Nithya and Kapania, Shivani and Highfill, Hannah and Akrong, Diana and Paritosh, Praveen Kumar and Aroyo, Lora Mois , title =. CHI Conference on Human Factors in Computing Systems (. 2021 , publisher =. doi:10.1145/3411764.3445518 , note =

    arXiv 2021

  72. [75]

    2016 , howpublished =

    Zinkevich, Martin , title =. 2016 , howpublished =

    2016

  73. [76]

    Liang, Percy and Bommasani, Rishi and Lee, Tony and Tsipras, Dimitris and Soylu, Dilara and Yasunaga, Michihiro and Zhang, Yian and Narayanan, Deepak and Wu, Yuhuai and Kumar, Ananya and Newman, Benjamin and Yuan, Binhang and Yan, Bobby and Zhang, Ce and Cosgrove, Christian and Manning, Christopher D. and R. Holistic Evaluation of Language Models , year =...

    Pith/arXiv arXiv doi:10.48550/arxiv.2211.09110

  74. [78]

    Reliability-by-Design for Agentic GenAI: Turning the AI Risk Atlas Into an EU AI Act-Ready Assurance Case , volume =

    Ray, Partha , year =. Reliability-by-Design for Agentic GenAI: Turning the AI Risk Atlas Into an EU AI Act-Ready Assurance Case , volume =. IEEE Reliability Magazine , doi =

  75. [80]

    doi:10.1109/EDUCON62633.2025.11016653 , note =

    Harnessing Multi-Agent LLMs for Complex Engineering Problem-Solving: A Framework for Senior Design Projects , year =. doi:10.1109/EDUCON62633.2025.11016653 , note =

    arXiv 2025

  76. [81]

    doi:10.1109/ACCESS.2026.3650833 , note =

    Agentic Intelligence for Unified Cyber Defense: A Self-Adaptive Framework for Threat Detection Across Cloud, Edge, and IoT Systems , year =. doi:10.1109/ACCESS.2026.3650833 , note =

    arXiv 2026

  77. [82]

    doi:10.1109/ICICNIS66685.2025.11315712 , note =

    Autonomous Agentic AI for Clinical Workflow Orchestration: Self-Managing Healthcare Operations , year =. doi:10.1109/ICICNIS66685.2025.11315712 , note =

    arXiv 2025

  78. [83]

    doi:10.1109/ISSREW63542.2024.00053 , note =

    Automated Interpretation of Fleet Incidents to Enable System Level Runtime Assurance , year =. doi:10.1109/ISSREW63542.2024.00053 , note =

    arXiv 2024

  79. [84]

    doi:10.1109/ACCESS.2026.3675554 , note =

    Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges , year =. doi:10.1109/ACCESS.2026.3675554 , note =

    arXiv 2026

  80. [86]

    doi:10.1109/ISWTA68114.2025.11329530 , note =

    AI-Driven Model-Retraining Architecture to Sustain Operational Accuracy in Data-Drifting Environments , year =. doi:10.1109/ISWTA68114.2025.11329530 , note =

    arXiv 2025

Showing first 80 references.