Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper defines eight recurring defect types in LLM-based agent code and presents Agentable, a static analyzer combining Code Property Graphs and LLMs that reports 88.79% precision and 91.03% recall.

desk verdict A useful first taxonomy and detection tool for LLM-agent code defects, but the headline recall figure is not independent because the test ground truth was validated with the tool itself. read the letter →

arxiv 2412.18371 v2 pith:SWQYJUFY submitted 2024-12-24 cs.SE

Kaiwen Ning , Jiachi Chen , Jingwen Zhang , Wei Li , Zexu Wang , Yuming Feng , Weizhe Zhang , Zibin Zheng This is my paper · ORCID
classification cs.SE
keywords LLMagentsagentdefecttaxonomystaticanalysisCodePropertyGraphLLM-baseddetectionempiricalsoftwareengineeringfaulttolerancetoolintegrationdefects
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the code wiring LLMs into autonomous agents suffers from a small, nameable set of recurring defects that can be caught before runtime. The authors analyzed 6,854 developer Q&A posts to define eight defect types spanning tool registration, LLM output parsing, trigger-word handling, fault tolerance, API calls, and package dependencies. They built Agentable, a static analyzer that combines Code Property Graphs for code structure with LLM calls for semantic judgment, and evaluated it on 84 real-world agent projects and 78 deliberately defective ones. On the annotated set it reached 88.79% precision and 91.03% recall, and it flagged 889 suspected defects in the real projects. If correct, this gives agent developers a concrete defect checklist and a working detector instead of an open-ended debugging problem.

What carries the argument

The machinery is a three-module detection pipeline. A Code Property Graph is abstracted into a Unified Node Relationship Tree (UNRT) that treats classes and functions as equivalent nodes, so searches for LLM initialization, agent initialization, and tool instances work across different framework styles. A semantic enrichment module pulls in source code and abstract syntax trees to recover runtime-state information the graph loses. An LLM invocation module performs the generalization-heavy judgments, such as whether a tool's name, description, and implementation agree or whether fault tolerance wraps a call. Each of the eight defect types has its own detection oracle that decides which module combination to run.

What would settle it

Have an independent team audit the base agent project and the 78 AgentTest variants; if the base still contains a defect after the three fixes, or if any test project contains a defect beyond the one injected, then the 91.03% recall estimate is not a valid measure of true recall.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-based agent code has its own defect taxonomy, distinct from both ordinary bugs and security vulnerabilities. The eight types are: Adaptation Defect between Agent and LLM, Insufficient External Tool Information, LLM Output Parsing Error, Tool Return Error, Action Listener Setting, Missing Necessary Fault Tolerance, LLM API-related Defect, and External Package Dependency Defect. Each type is grounded in reports from developer Q&A posts and tied to a component of the agent workflow. The authors further claim that Agentable, a static analyzer combining Code Property Graphs with LLM-based semantic checks, detects these defects at the reported precision and recall, and that the 889 defects found in 84 real projects show the taxonomy matters in practice.

Load-bearing premise

The reported 91.03% recall presupposes that the base agent project used to build AgentTest is defect-free after three manual fixes and that each of the 78 test projects contains exactly one true defect.

Editorial extensions

If this is right

  • Agent developers can use the eight-type taxonomy as a code-review checklist before deploying an agent.
  • The same detection patterns can be wired into continuous integration so tool registration, output parsing, and fault-tolerance mistakes are caught before release.
  • The defect distribution, dominated by LLM API-related and external-package-dependency issues, tells framework authors where to focus documentation and guardrails.
  • Each defect type comes with a proposed mitigation, so a detection report points toward a fix rather than just a warning.
  • The design's reliance on graph abstraction and LLM judgment is intended to transfer to agent frameworks beyond the ones that appear in its training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its taxonomy is anchored in projects and posts written mostly in Python, so agent ecosystems in other languages may need additional defect categories.
  • A practical extension is to turn the warning-only External Package Dependency Defect reports into real checks by adding version-resolution analysis at build time.
  • The same detection oracles could be adapted to multi-agent systems, where inter-agent message mismatches may form a whole new defect class.
  • An empirical test of the taxonomy's completeness would be to run Agentable on a fresh sample of agent projects and measure how many reported anomalies fall outside the eight types.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents the first systematic study of code defects in LLM-based autonomous agents. It derives a taxonomy of eight defect types (ADAL, IETI, LOPE, TRE, ALS, MNFT, LARD, EPDD) from 331 StackOverflow posts using open card sorting, and then proposes Agentable, a static analysis tool combining Code Property Graphs and LLM reasoning to detect these defects. The tool is evaluated on AgentSet (84 real-world agent projects, reporting 889 detections with 88.79% precision estimated from a randomly sampled subset) and AgentTest (78 hand-injected defect projects, reporting 91.03% recall). The paper claims that this is the first work to define and detect LLM-agent code defects and to provide a large-scale empirical analysis of their prevalence in real projects.

Significance. If the evaluation is sound, this is a useful and timely contribution to the emerging area of LLM-agent reliability. The proposed taxonomy is grounded in developer posts and covers components that prior security-focused work has not systematically addressed. The design of Agentable—using CPGs for structural analysis and LLMs for semantic/generalization checks—is a reasonable and transferable approach, and the construction of a real-world dataset of 84 agent projects is a valuable resource for the community. However, the central recall claim rests on a benchmark whose negative ground truth is verified by the very tool being evaluated, which substantially weakens the credibility of the 91.03% figure as an independent measure of detection ability.

major comments (4)
  1. [Section V-A, paragraph 'Dataset Construction of AgentTest'] The claim that the base project is defect-free after fixing the three manually identified defects is validated by 'ran Agentable for detection, confirming that no defects were present.' This is circular for the purpose of measuring recall: if Agentable has any false negatives, residual defects remain in the base project, so the denominator of 78 (one injected defect per project) is incorrect and detections of residual defects would be misclassified as false positives. Please re-validate the negative ground truth with an independent method (e.g., multiple independent human inspections or a different detection tool) or, if no such validation is feasible, explicitly analyze and report how this circularity affects the reported recall rate.
  2. [Section V-A, paragraph 'Dataset Construction of AgentTest', and Table IV] The positive scenarios in AgentTest are constructed 'based on the detection results of RQ1,' meaning they are derived from the outputs of Agentable itself. This aligns the benchmark with the specific patterns the tool was designed to recognize, so the measured recall of 91.03% reflects the tool's ability to find defects similar to those it already finds, not necessarily its ability to detect the full range of defect manifestations in the wild. I recommend constructing positive cases from independent sources—for example, real defect reports from GitHub issues or a held-out set of projects not used in RQ1—and reporting how the injection scenarios were selected.
  3. [Abstract, Section I, and Section V-C] The abstract and introduction repeatedly state that Agentable achieved an 'overall accuracy of 88.79%', but the computation in Section V-C and Table III is a precision (the fraction of reported defects that are true positives), and no true negatives are measured. Accuracy and precision are different metrics; using 'accuracy' without a definition that includes true negatives is misleading. Please replace 'accuracy' with 'precision' throughout, or explicitly define the metric used.
  4. [Section IV-D-2] The detection oracles are described at a fairly high level, and the details needed for independent replication—such as the exact LLM prompts (beyond stating that chain-of-thought is used), the specific CPG query patterns for each defect type, and the value of the grouping parameter n—are not provided. Since the source code is not yet released, the evaluation is not reproducible as presented. Please include the prompts and pattern specifications in an appendix or make the code and datasets available with the submission, rather than only after acceptance.
minor comments (5)
  1. [Entire manuscript] Acronym usage is inconsistent: the text alternates between 'ADAL' and 'AAL', 'IETI' and 'ETE', 'LOPE' and 'LPE', 'LARD' and 'ARD', and 'EPDD' and 'EPDC'. Please standardize to the definitions in Table II.
  2. [Section III-D(5) and (6), Figure 8] The same code example in Figure 8 is used to illustrate both the ALS and MNFT defects; it would be clearer to explicitly annotate which lines correspond to which defect in the figure caption or in a separate listing.
  3. [Section V-D, paragraph after Table IV] The sentence 'in an TRE defect detection, the LLM sometimes flagged non-essential fault-tolerance aspects missing in the Tool code as IETI defects' appears to contain a typo ('TRE' should likely be 'IETI'), and the surrounding discussion of IETI errors should be checked for consistency.
  4. [Section V-C, subsection heading 'Accuracy'] The subsection heading 'Accuracy' should be renamed 'Precision' to match the metric actually reported in Table III and the text.
  5. [Section V-C, 'Overhead' paragraph] The reported LLM invocation cost of $24.2 and average cost of $0.027 per defect are useful details; consider also reporting the number of LLM calls made in total, which would help readers assess scalability.

Circularity Check

2 steps flagged · score 6.0 of 10

Recall estimate is partly circular: AgentTest's negative ground truth is confirmed by Agentable itself and its positive scenarios are built from Agentable's RQ1 detections.

  1. self definitional [Section V-A, Dataset Construction of AgentTest]
    "Next, we manually fixed these defects and ran Agentable for detection, confirming that no defects were present."

    The negative ground truth of the recall benchmark is established by running the very tool whose recall is being measured. If Agentable has any false negatives, the base project still contains defects after the fixes, and those residual defects are invisible by construction. The reported recall of 91.03% assumes exactly 78 true defects across the 78 projects in Table IV, so any undetected residual defect enlarges the true positive denominator and lowers the true recall. Thus the recall measurement is not an independent estimate of detection ability; for the negative cases, the ground truth is defined by Agentable's own output.

  2. fitted input called prediction [Section V-A, Dataset Construction of AgentTest]
    "Furthermore, based on the detection results of RQ1, we constructed specific Agent scenarios containing individual defects, ensuring that the total number still met the 95% confidence level and a confidence interval of 10."

    The positive cases in AgentTest are not an independent sample of the eight defect types; they are scenarios built from the detections Agentable already produced in RQ1, with the per-type allocation also based on RQ1's distribution. Recall is therefore measured on a benchmark whose positive and distributional structure is aligned with the tool's own true positives. This makes the 91.03% recall figure a measure of how well Agentable detects the patterns it already detects, rather than an independent test against the StackOverflow-derived defect definitions.

full rationale

The paper has real independent content: the eight defect types are derived from 6,854 StackOverflow posts via card sorting, and the precision estimate on AgentSet is based on manual labeling of a random sample, so the taxonomy and precision claim are not circular. However, the headline recall of 91.03% is not an independent measurement. AgentTest's base project is declared defect-free only after running Agentable, which is the system whose false-negative rate the benchmark is supposed to estimate; any Agentable false negative is therefore removed from the ground truth by construction. In addition, the injected positive scenarios and their distribution are constructed from Agentable's RQ1 detections, so the benchmark is fitted to the tool's own output. These two steps make the recall estimate partially self-referential, but they do not make the entire derivation circular, since the defect definitions, the CPG+LLM detection design, and the manually labeled precision evaluation stand apart. Score 6 reflects partial circularity in the central recall claim.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claims rest on the completeness and correctness of the eight-defect taxonomy, the quality of the self-built ground truth, and the accuracy of the LLM used in the detection pipeline. None of these is verified against an external standard, and the datasets are not yet available for independent assessment.

free parameters (1)
  • Node grouping size n = 10
    Set to 10 to prevent context overflow in LLM calls (Section V-A). This hand-chosen parameter affects how much code is sent to the LLM per invocation and could influence detection performance, but it is not fitted against data.
assumptions (4)
  • domain assumption The eight defect types derived from 331 StackOverflow posts form a complete taxonomy of LLM-agent code defects.
    Used to define all detection oracles; no external validation against other defect sources or benchmarks is provided (Section III-C, III-D).
  • domain assumption The AgentTest base project (LLM Agents by mpaepper) is defect-free after fixing three identified defects.
    The recall calculation treats any detection in the modified base as a false positive and any injected defect as the sole true defect, so this premise is load-bearing (Section V-A).
  • domain assumption GPT-4o-mini provides sufficiently accurate judgments for semantic checks such as tool description-name consistency and fault-tolerance assessment.
    Most oracles rely on LLM outputs; errors in the LLM directly translate into detection false positives and negatives (Section IV-D).
  • domain assumption Joern's CPG construction preserves enough structure to locate agent components in Python projects.
    The code abstraction module depends on Joern to build the graph; if the CPG omits relevant nodes, the tool cannot find the classes and functions needed for analysis (Section IV-C).
invented entities (1)
  • LLM-agent defect taxonomy (eight types: ADAL, IETI, LOPE, TRE, ALS, MNFT, LARD, EPDD)
    purpose: Provides a classification scheme for code-level defects in LLM-based agents and serves as the target labels for the detection tool.
    The taxonomy is derived solely from the authors' manual analysis of StackOverflow posts and is not independently benchmarked or reproduced by other researchers; it is a conceptual construct introduced in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents." pith.science (2026). https://pith.science/paper/SWQYJUFY

@misc{pith2026241218371,
  author       = {Pith},
  title        = {Pith review of: Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SWQYJUFY}},
  note         = {Machine review of arXiv:2412.18371}
}
read the original abstract

AI agents are systems capable of perceiving their environment, autonomously planning and executing tasks. Recent advancements in LLM have introduced a transformative paradigm for AI agents, enabling them to interact with external resources and tools through prompts. In such agents, the workflow integrates developer-written code, which manages framework construction and logic control, with LLM-generated natural language that enhances dynamic decision-making and interaction. However, discrepancies between developer-implemented logic and the dynamically generated content of LLMs in terms of behavior and expected outcomes can lead to defects, such as tool invocation failures and task execution errors. These issues introduce specific risks, leading to various defects in LLM-based AI Agents, such as service interruptions. Despite the importance of these issues, there is a lack of systematic work that focuses on analyzing LLM-based AI Agents to uncover defects in their code. In this paper, we present the first study focused on identifying and detecting defects in LLM Agents. We collected and analyzed 6,854 relevant posts from StackOverflow to define 8 types of agent defects. For each type, we provided detailed descriptions with an example. Then, we designed a static analysis tool, named Agentable, to detect the defects. Agentable leverages Code Property Graphs and LLMs to analyze Agent workflows by efficiently identifying specific code patterns and analyzing natural language descriptions. To evaluate Agentable, we constructed two datasets: AgentSet, consists of 84 real-world Agents, and AgentTest, which contains 78 Agents specifically designed to include various types of defects. Our results show that Agentable achieved an overall accuracy of 88.79% and a recall rate of 91.03%. Furthermore, our analysis reveals the 889 defects of the AgentSet, highlighting the prevalence of these defects.

Figures

Figures reproduced from arXiv: 2412.18371 by the authors.

Figure 1
Figure 1. An example of LLM-based Agent. B. Code Property Graphs Graph-based code analysis methods typically transform source code into various graph structures, such as Abstract Syntax Trees (ASTs), Control Flow Graphs (CFGs), and Pro￾gram Dependency Graphs (PDGs), to efficiently capture local structures and dependencies within the code [30]. However, each type of graph focuses on different aspects, and no single graph can f… view at source ↗
Figure 2
Figure 2. An example of the motivation. III. DEFINE THE DEFECT OF LLM-BASED AGENT In this section, we conduct an empirical study of real-world posts on StackOverflow to define and categorize code defects in LLM-based Agents. StackOverflow is a developer Q&A platform that allows developers to communicate and share so￾lutions to software development issues they encounter [32]. In line with previous work [20], [33], StackOverflo… view at source ↗
Figure 3
Figure 3. An example of a card. The cause of this defect is the failure to accurately describe the functionality of the Tool and properly pass this information to the LLM. This leads to the Agent being unable to invoke the correct tool, resulting in unexpected behavior. In the process of open card sorting, we ensure the represen￾tativeness of defects by considering the reproducibility of the code. Some issues may be specific … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: An example of ADAL Defect. (2) Insufficient External Tool Information (IETI). Agents extend the capabilities and knowledge of LLMs by preparing a series of external tools, such as calculators or weather infor￾mation retrieval services. In an Agent system, these externa…
Figure 5
Figure 5. Figure 5: An example of IETI Defect. (3) LLM Output Parsing Error (LOPE). When generating content, an LLM may produce outputs that do not meet the intended requirements due to various reasons, such as hal￾lucinations [49]. During runtime, the Agent must continually parse the LLM…
Figure 6
Figure 6. Figure 6: An example of LOPE Defect. (4) Tool Return Error (TRE). After the Agent successfully commands the LLM to invoke an external tool, it must wait for the tool to execute and return results to the LLM. However, tools may involve chained function calls during their implemen…
Figure 8
Figure 8. Figure 8: An example of ALS and MNFT Defect. output of external tools or data invocations, resulting in service interruptions. Example: In the example from [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: An example of LARD Defect. (8) External Package Dependency Defect (EPDD). The implementation of tools in an Agent is decoupled from the Agent framework, allowing users to implement and register tools according to their specific needs. However, if there is an overlap be…
Figure 11
Figure 11. Figure 11: The overview of Agentable. Algorithm 1 Unified Node Relationship Tree (UNRT) Con￾struction Process 1: Input: CPG from Joern 2: Output: UNRT 3: Initialize an empty class inheritance tree T 4: for each class Ci in the source code do 5: Add Ci to T with its inheritance r…
Figure 12
Figure 12. Figure 12: Defect distribution of Agents. components of the Agent. Therefore, in addition to presenting Agentable, we also provide possible solutions for each defect type in Table VI. These solutions are summarized from the discussion sections of the StackOverflow posts we colle…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Specification and Detection of LLM Code Smells

    cs.SE 2025-12 conditional novelty 6.0 of 10

    A catalog of five LLM code smells and a static detection tool find that 60.5% of 200 open-source LLM-using Python projects exhibit at least one smell.

Reference graph

Works this paper leans on

62 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Expel: Llm agents are experiential learners,

    A. Zhao, D. Huang, Q. Xu, M. Lin, Y .-J. Liu, and G. Huang, “Expel: Llm agents are experiential learners,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 17, pp. 19 632–19 642, Mar. 2024. [Online]. Available: https://ojs.aaai.org/ index.php/AAAI/article/view/29936

    work page 2024

  2. [2]

    Exploring large language model based intelligent agents: Definitions, methods, and prospects,

    Y . Cheng, C. Zhang, Z. Zhang, X. Meng, S. Hong, W. Li, Z. Wang, Z. Wang, F. Yin, J. Zhao, and X. He, “Exploring large language model based intelligent agents: Definitions, methods, and prospects,” 2024. [Online]. Available: https://arxiv.org/abs/2401.03428

    arXiv 2024

  3. [3]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    arXiv 2022

  4. [4]

    Language models don’t always say what they think: unfaithful explanations in chain- of-thought prompting,

    M. Turpin, J. Michael, E. Perez, and S. Bowman, “Language models don’t always say what they think: unfaithful explanations in chain- of-thought prompting,” Advances in Neural Information Processing Systems, vol. 36, 2024

    work page 2024

  5. [5]

    Large language model based multi-agents: A survey of progress and challenges,

    T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,” 2024. [Online]. Available: https://arxiv.org/abs/2402.01680

    arXiv 2024

  6. [6]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y . Zhou, W. Wang, C. Jiang, Y . Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y . Zheng, X. Qiu, X. Huang, and T. Gui, “The rise and potential of large language model based agents: A survey,” 2023. [Online]. Avail...

    arXiv 2023

  7. [7]

    A survey on the memory mechanism of large language model based agents,

    Z. Zhang, X. Bo, C. Ma, R. Li, X. Chen, Q. Dai, J. Zhu, Z. Dong, and J.-R. Wen, “A survey on the memory mechanism of large language model based agents,” 2024. [Online]. Available: https://arxiv.org/abs/2404.13501

    arXiv 2024

  8. [8]

    Large language model-based agents for software engineering: A survey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large language model-based agents for software engineering: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2409.02977

    arXiv 2024

Show all 62 references
  1. [9]

    Understanding the planning of llm agents: A survey,

    X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y . Wang, R. Tang, and E. Chen, “Understanding the planning of llm agents: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2402.02716

    2024 arXiv

  2. [10]

    Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases,

    Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases,”

  3. [11]

    Intelligent agents with llm-based process automation,

    Y . Guan, D. Wang, Z. Chu, S. Wang, F. Ni, R. Song, and C. Zhuang, “Intelligent agents with llm-based process automation,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 5018–5027

    2024

  4. [12]

    Prompt- to-sql injections in llm-integrated web applications: Risks and defenses,

    R. Pedro, M. E. Coimbra, D. Castro, P. Carreira, and N. Santos, “Prompt- to-sql injections in llm-integrated web applications: Risks and defenses,” in 2025 IEEE/ACM 47th International Conference on Software Engi- neering (ICSE). IEEE Computer Society, 2024, pp. 76–88

    2025

  5. [13]

    Personal llm agents: Insights and survey about the capability, efficiency and security,

    Y . Li, H. Wen, W. Wang et al. , “Personal llm agents: Insights and survey about the capability, efficiency and security,” 2024. [Online]. Available: https://arxiv.org/abs/2401.05459

    2024 arXiv

  6. [14]

    Prioritizing safeguarding over autonomy: Risks of llm agents for science,

    X. Tang, Q. Jin, K. Zhu, T. Yuan, Y . Zhang, W. Zhou, M. Qu, Y . Zhao, J. Tang, Z. Zhang, A. Cohan, Z. Lu, and M. Gerstein, “Prioritizing safeguarding over autonomy: Risks of llm agents for science,” 2024. [Online]. Available: https://arxiv.org/abs/2402.04247

    2024 arXiv

  7. [15]

    Fabian both,

    why we no longer use LangChain for building our AI agents, “Fabian both,” 2024. [Online]. Available: https://www.octomind.dev/ blog/why-we-no-longer-use-langchain-for-building-our-ai-agents

    2024

  8. [16]

    A dynamic LLM-powered agent network for task-oriented agent collaboration,

    Z. Liu, Y . Zhang, P. Li, Y . Liu, and D. Yang, “A dynamic LLM-powered agent network for task-oriented agent collaboration,” in First Conference on Language Modeling , 2024. [Online]. Available: https://openreview.net/forum?id=XII0Wp1XA9

    2024

  9. [17]

    “it’s a fair game

    Z. Zhang, M. Jia, H.-P. Lee, B. Yao, S. Das, A. Lerner, D. Wang, and T. Li, ““it’s a fair game”, or is it? examining how users navigate dis- closure risks and benefits when using llm-based conversational agents,” in Proceedings of the CHI Conference on Human Factors in Computi...

    2024

  10. [18]

    BadAgent: Inserting and activating backdoor attacks in LLM agents,

    Y . Wang, D. Xue, S. Zhang, and S. Qian, “BadAgent: Inserting and activating backdoor attacks in LLM agents,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangk...

    2024

  11. [19]

    Making it tangible: hybrid card sorting within qualitative interviews,

    L. Y . Conrad and V . M. Tucker, “Making it tangible: hybrid card sorting within qualitative interviews,” Journal of Documentation, vol. 75, no. 2, pp. 397–416, 2019

    2019

  12. [20]

    Identifying smart contract security issues in code snippets from stack overflow,

    J. Chen, C. Chen, J. Hu, J. Grundy, Y . Wang, T. Chen, and Z. Zheng, “Identifying smart contract security issues in code snippets from stack overflow,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis . New York, NY , USA: Associa...

    2024

  13. [21]

    Llm agent operating system,

    K. Mei, Z. Li, S. Xu, R. Ye, Y . Ge, and Y . Zhang, “Llm agent operating system,” arXiv preprint arXiv:2403.16971 , 2024

    2024 arXiv

  14. [22]

    Llm-based multi-agent systems for software engineering: Vision and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Vision and the road ahead,” 2024. [Online]. Available: https://arxiv.org/abs/2404.04834

    2024 arXiv

  15. [23]

    Finmem: A performance-enhanced llm trading agent with layered memory and character design,

    Y . Yu, H. Li, Z. Chen, Y . Jiang, Y . Li, D. Zhang, R. Liu, J. W. Suchow, and K. Khashanah, “Finmem: A performance-enhanced llm trading agent with layered memory and character design,” in Proceedings of the AAAI Symposium Series , vol. 3, no. 1, 2024, pp. 595–597

    2024

  16. [24]

    Knowagent: Knowledge-augmented planning for llm-based agents,

    Y . Zhu, S. Qiao, Y . Ou, S. Deng, N. Zhang, S. Lyu, Y . Shen, L. Liang, J. Gu, and H. Chen, “Knowagent: Knowledge-augmented planning for llm-based agents,” arXiv preprint arXiv:2403.03101 , 2024

    2024 arXiv

  17. [25]

    Graphspd: Graph-based security patch detection with enriched code semantics,

    S. Wang, X. Wang, K. Sun, S. Jajodia, H. Wang, and Q. Li, “Graphspd: Graph-based security patch detection with enriched code semantics,” in 2023 IEEE Symposium on Security and Privacy (SP) . IEEE, 2023, pp. 2409–2426

    2023

  18. [26]

    mpaepper,

    L. Agents, “mpaepper,” 2023. [Online]. Available: https://github.com/ mpaepper/llm agents

    2023

  19. [27]

    Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents,

    H. Zhang, J. Huang, K. Mei, Y . Yao, Z. Wang, C. Zhan, H. Wang, and Y . Zhang, “Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents,” 2024. [Online]. Available: https://arxiv.org/abs/2410.02644

    2024 arXiv

  20. [28]

    Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents,

    E. Debenedetti, J. Zhang, M. Balunovi ´c, L. Beurer-Kellner, M. Fischer, and F. Tram `er, “Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents,” 2024. [Online]. Available: https://arxiv.org/abs/2406.13352

    2024 arXiv

  21. [29]

    Llm powered autonomous agents,

    L. Weng, “Llm powered autonomous agents,” 2023. [Online]. Available: https://lilianweng.github.io/posts/2023-06-23-agent/

    2023

  22. [30]

    Adcpg: Classifying javascript code property graphs with explanations for ad and tracker blocking,

    C. Lee and S. Son, “Adcpg: Classifying javascript code property graphs with explanations for ad and tracker blocking,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, 2023, pp. 3505–3518

    2023

  23. [31]

    Modeling and discover- ing vulnerabilities with code property graphs,

    F. Yamaguchi, N. Golde, D. Arp, and K. Rieck, “Modeling and discover- ing vulnerabilities with code property graphs,” in 2014 IEEE symposium on security and privacy . IEEE, 2014, pp. 590–604

    2014

  24. [32]

    Time to separate from stackoverflow and match with chatgpt for encryption,

    E. Firouzi and M. Ghafari, “Time to separate from stackoverflow and match with chatgpt for encryption,” Journal of Systems and Software , p. 112135, 2024

    2024

  25. [33]

    Demystifying and detecting cryptographic defects in ethereum smart contracts,

    J. Zhang, Y . Shen, J. Chen, J. Su, Y . Wang, T. Chen, J. Gao, and Z. Chen, “Demystifying and detecting cryptographic defects in ethereum smart contracts,” arXiv preprint arXiv:2408.04939 , 2024

    2024 arXiv

  26. [34]

    Analysis of modern release engineering topics:–a large-scale study using stackoverflow–,

    M. Openja, B. Adams, and F. Khomh, “Analysis of modern release engineering topics:–a large-scale study using stackoverflow–,” in 2020 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 2020, pp. 104–114

    2020

  27. [35]

    Cebench: A benchmarking toolkit for the cost-effectiveness of llm pipelines,

    W. Sun, J. Wang, Q. Guo, Z. Li, W. Wang, and R. Hai, “Cebench: A benchmarking toolkit for the cost-effectiveness of llm pipelines,” arXiv preprint arXiv:2407.12797, 2024

    2024

  28. [36]

    Creating large language model appli- cations utilizing langchain: A primer on developing llm apps fast,

    O. Topsakal and T. C. Akinci, “Creating large language model appli- cations utilizing langchain: A primer on developing llm apps fast,” in International Conference on Applied Engineering and Natural Sciences , vol. 1, no. 1, 2023, pp. 1050–1056

    2023

  29. [37]

    Autogen: Enabling next-gen llm applications via multi-agent conversation framework,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen llm applications via multi-agent conversation framework,” arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  30. [38]

    Adapting llms for efficient, personalized information retrieval: Methods and implications,

    S. Ghodratnama and M. Zakershahrak, “Adapting llms for efficient, personalized information retrieval: Methods and implications,” in Inter- national Conference on Service-Oriented Computing . Springer, 2023, pp. 17–26. 15

    2023

  31. [39]

    Using flowise to streamline biomedical data discovery and analysis,

    J. A. Reis, J. R. Almeida, T. M. Almeida, and J. L. Oliveira, “Using flowise to streamline biomedical data discovery and analysis,” in 2024 IEEE 22nd Mediterranean Electrotechnical Conference (MELECON) . IEEE, 2024, pp. 695–700

    2024

  32. [40]

    Defining smart contract defects on ethereum,

    J. Chen, X. Xia, D. Lo, J. C. Grundy, X. Luo, and T. Chen, “Defining smart contract defects on ethereum,” IEEE Trans. Software Eng., vol. 48, no. 2, pp. 327–345, 2022

    2022

  33. [41]

    Self-admitted technical debt practices: a comparison between industry and open- source,

    F. Zampetti, G. Fucci, A. Serebrenik, and M. Di Penta, “Self-admitted technical debt practices: a comparison between industry and open- source,” Empirical Software Engineering , vol. 26, pp. 1–32, 2021

    2021

  34. [42]

    Hyperion: Unveiling dapp inconsistencies using llm and dataflow-guided symbolic execution,

    S. Yang, X. Lin, J. Chen, Q. Zhong, L. Xiao, R. Huang, Y . Wang, and Z. Zheng, “Hyperion: Unveiling dapp inconsistencies using llm and dataflow-guided symbolic execution,” 2024. [Online]. Available: https://arxiv.org/abs/2408.06037

    2024 arXiv

  35. [43]

    An empirical study on challenges for llm developers,

    X. Chen, C. Gao, C. Chen, G. Zhang, and Y . Liu, “An empirical study on challenges for llm developers,” arXiv preprint arXiv:2408.05002, 2024

    2024 arXiv

  36. [44]

    Definition and detection of defects in nft smart contracts,

    S. Yang, J. Chen, and Z. Zheng, “Definition and detection of defects in nft smart contracts,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 373–384

    2023

  37. [45]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  38. [46]

    How does gpt-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model,

    M. Hanna, O. Liu, and A. Variengien, “How does gpt-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  39. [47]

    Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles,

    C. Cui, Y . Ma, X. Cao, W. Ye, and Z. Wang, “Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles,” IEEE Intelligent Transportation Systems Magazine , 2024

    2024

  40. [48]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023

    2023 arXiv

  41. [49]

    Cognitive mirage: A review of hallucinations in large language models,

    H. Ye, T. Liu, A. Zhang, W. Hua, and W. Jia, “Cognitive mirage: A review of hallucinations in large language models,” arXiv preprint arXiv:2309.06794, 2023

    2023 arXiv

  42. [50]

    Demystifying rce vulnerabilities in llm-integrated apps,

    T. Liu, Z. Deng, G. Meng, Y . Li, and K. Chen, “Demystifying rce vulnerabilities in llm-integrated apps,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , 2024, pp. 1716–1730

    2024

  43. [51]

    Large model agents: State-of-the-art, cooperation paradigms, security and privacy, and future trends,

    Y . Wang, Y . Pan, Q. Zhao, Y . Deng, Z. Su, L. Du, and T. H. Luan, “Large model agents: State-of-the-art, cooperation paradigms, security and privacy, and future trends,” arXiv preprint arXiv:2409.14457, 2024

    2024 arXiv

  44. [52]

    Ai agents under threat: A survey of key security challenges and future pathways,

    Z. Deng, Y . Guo, C. Han, W. Ma, J. Xiong, S. Wen, and Y . Xiang, “Ai agents under threat: A survey of key security challenges and future pathways,” arXiv preprint arXiv:2406.02630 , 2024

    2024

  45. [53]

    Privacyasst: Safeguarding user privacy in tool-using large language model agents,

    X. Zhang, H. Xu, Z. Ba, Z. Wang, Y . Hong, J. Liu, Z. Qin, and K. Ren, “Privacyasst: Safeguarding user privacy in tool-using large language model agents,” IEEE Transactions on Dependable and Secure Computing, 2024

    2024

  46. [54]

    Utilizing precise and complete code context to guide llm in automatic false positive mitigation,

    J. Chen, H. Xiang, L. Li, Y . Zhang, B. Ding, and Q. Li, “Utilizing precise and complete code context to guide llm in automatic false positive mitigation,” arXiv preprint arXiv:2411.03079 , 2024

    2024 arXiv

  47. [55]

    Detecting software vulnerabilities based on source code analysis using gcn transformer,

    V .-C. Bui and X.-C. Do, “Detecting software vulnerabilities based on source code analysis using gcn transformer,” in 2023 RIVF International Conference on Computing and Communication Technologies (RIVF) . IEEE, 2023, pp. 112–117

    2023

  48. [56]

    Toward extracting learning pattern: A comparative study of gpt-4o-mini and bert models in predicting cvss base vectors,

    S. Isogai, S. Ogata, Y . Kashiwa, S. Yazawa, K. Okano, T. Okubo, and H. Washizaki, “Toward extracting learning pattern: A comparative study of gpt-4o-mini and bert models in predicting cvss base vectors,” in 2024 IEEE 35th International Symposium on Software Reliability Engine...

    2024

  49. [57]

    Moral alignment for llm agents,

    E. Tennant, S. Hailes, and M. Musolesi, “Moral alignment for llm agents,” 2024. [Online]. Available: https://arxiv.org/abs/2410.01639

    2024 arXiv

  50. [58]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig et al. , “Swe-agent: Agent-computer interfaces enable automated software engineering,” 2024. [Online]. Available: https://arxiv.org/abs/2405.15793

    2024 arXiv

  51. [59]

    From llms to llm- based agents for software engineering: A survey of current, challenges and future,

    H. Jin, L. Huang, H. Cai, J. Yan, B. Li, and H. Chen, “From llms to llm- based agents for software engineering: A survey of current, challenges and future,” 2024. [Online]. Available: https://arxiv.org/abs/2408.02479

    2024 arXiv

  52. [60]

    Enhancing fault localization through ordered code analysis with llm agents and self- reflection,

    M. N. Rafi, D. J. Kim, T.-H. Chen, and S. Wang, “Enhancing fault localization through ordered code analysis with llm agents and self- reflection,” 2024. [Online]. Available: https://arxiv.org/abs/2409.13642

    2024 arXiv

  53. [61]

    Agentless: Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying llm-based software engineering agents,” 2024. [Online]. Available: https://arxiv.org/abs/2407.01489

    2024 arXiv

  54. [2024]

    Available: https://arxiv.org/abs/2407.12784

    [Online]. Available: https://arxiv.org/abs/2407.12784

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.