Pith. sign in

REVIEW 2 major objections 1 minor 27 references

Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer

T0 review · 2 major / 1 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A static auditing framework for local LLM agent runtimes raises recall on source-level vulnerabilities from under 22 percent to 66-75 percent.

desk verdict The paper builds a static-analysis framework for local LLM agent runtimes and shows solid recall gains on OpenClaw advisories, but the evaluation stays inside that single codebase. read the letter →

arxiv 2606.21071 v1 pith:LAAFVJEV submitted 2026-06-19 cs.CR cs.AI

Zhengsong Zhang , Zongze Li , Jiawei Guo , Haipeng Cai This is my paper
classification cs.CRcs.AI
keywords localLLMagentssourcecodeauditstaticanalysisvulnerabilitydetectionSemgrepQLagentruntimesecuritySTRIDEtaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines the source code of local LLM agents that translate natural-language goals into actions on the host shell, filesystem, and other resources. It introduces CLAWAUDIT to audit the prompt builder, parser, tool dispatcher, and permission components that had not been checked before. The framework creates a STRIDE-derived taxonomy and writes custom rules for two static-analysis engines that target patterns unique to these agents. When tested on a temporally split set of 446 real advisories, the rules detect far more issues than the engines' built-in security checks while maintaining performance on code written after the rules were created. This matters because these agents run with direct access to user data and credentials, turning their internal implementation into a privileged attack surface.

What carries the argument

CLAWAUDIT, the static auditing framework that pairs a five-category STRIDE-derived taxonomy with custom Semgrep and CodeQL rules written for agent runtime patterns such as prompt construction and permission gating.

What would settle it

Applying the same 47 Semgrep rules and 30 CodeQL queries to the full source tree of a second local LLM agent and obtaining recall below 30 percent on its documented vulnerabilities would show the rules do not transfer.

Watch

Extended reading notes

Core claim

CLAWAUDIT derives a five-category vulnerability taxonomy from STRIDE and instantiates it as 47 Semgrep YAML rules plus 30 CodeQL queries that target agent-specific patterns absent from standard rule sets. On the held-out test portion of OPENCLAWBENCH (217 advisories), the Semgrep rules reach 66.8 percent recall versus 21.7 percent for the Pro baseline, while the CodeQL queries reach 75.1 percent recall versus 13.8 percent for the security-extended baseline. Performance on the train and test splits differs by at most four percentage points, showing that the rules generalize beyond the examples used to write them. A live-code audit indicates that the rules still require manual triage for produ

Load-bearing premise

The 446 advisories drawn from one agent repository capture the vulnerability distribution that would appear in other local LLM agent runtimes.

Editorial extensions

If this is right

  • Developers of local agents can integrate the rules into their build process to surface flaws in the prompt builder, memory writer, and network client before release.
  • The small train-test gap shows that rules written against older code continue to find newer implementation weaknesses in the same agent.
  • Standard security rule sets for Semgrep and CodeQL miss the majority of the implementation-level issues that arise when agents mediate between models and host resources.
  • Automated detection must be followed by manual review because the rules are tuned for recall rather than precision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same taxonomy and rule-writing approach could be reapplied to other agent codebases to produce comparable coverage numbers for their runtime layers.
  • Treating the agent runtime as an operating-system-level component implies that future agents may need formal verification or language-level isolation in addition to static checks.
  • The remaining manual-triage step points to a possible next engineering target: lightweight semantic filters that reduce false positives while preserving the observed recall gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper presents CLAWAUDIT, a static auditing framework for local LLM agent runtimes. It derives a five-category vulnerability taxonomy from STRIDE and implements it via 47 Semgrep YAML rules and 30 CodeQL queries targeting agent-specific patterns. The framework is evaluated on OPENCLAWBENCH, a dataset of 446 source-code-level advisories from the OpenClaw repository temporally split into 229 for rule derivation and 217 held-out for testing. On the test set, it reports substantial recall improvements over baselines: Semgrep from 21.7% to 66.8% and CodeQL from 13.8% to 75.1%, with train/test gaps under 4 percentage points.

Significance. If the results hold, this work offers the first source-code audit of the runtime layer in local LLM agents and demonstrates that custom static rules can significantly enhance detection of implementation-level vulnerabilities in these systems. The use of a temporal held-out split and small generalization gap provide solid support for the recall claims within the OpenClaw codebase. The acknowledgment of the need for manual triage in live audits is a realistic assessment of practical deployment.

major comments (2)
  1. [Abstract] Abstract: The abstract positions CLAWAUDIT as applicable to local LLM agent runtimes in general (explicitly referencing Nanobot alongside OpenClaw), yet the evaluation, rule development, and OPENCLAWBENCH benchmark are performed exclusively on advisories from the OpenClaw repository. This single-project scope means the reported recall gains cannot be separated from OpenClaw-specific coding idioms, so the headline improvements do not yet establish the framework's utility for the broader class of runtimes claimed.
  2. [Abstract] Abstract and benchmark description: No detail is supplied on the process used to label the 446 advisories or on safeguards ensuring that rule writing did not involve post-hoc adjustments based on test-set observations. Without this information the temporal split's independence and the reproducibility of the 66.8 % / 75.1 % recall figures cannot be verified.
minor comments (1)
  1. [Abstract] Abstract: The phrase 'preliminary live-code audit' is introduced without any quantitative scope, number of issues examined, or concrete findings beyond the general statement that manual triage is required.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting limitations in scope and reproducibility. We address each major comment below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The abstract positions CLAWAUDIT as applicable to local LLM agent runtimes in general (explicitly referencing Nanobot alongside OpenClaw), yet the evaluation, rule development, and OPENCLAWBENCH benchmark are performed exclusively on advisories from the OpenClaw repository. This single-project scope means the reported recall gains cannot be separated from OpenClaw-specific coding idioms, so the headline improvements do not yet establish the framework's utility for the broader class of runtimes claimed.

    Authors: We agree that the abstract and claims overstate generality given the single-project evaluation. The taxonomy is derived from STRIDE and intended to apply broadly, but the rules and recall figures are specific to OpenClaw idioms. We will revise the abstract, introduction, and conclusion to explicitly state that empirical results are from OpenClaw, note that applicability to other runtimes such as Nanobot requires further validation, and frame the work as an initial case study rather than a general proof. revision: yes

  2. Referee: [Abstract] Abstract and benchmark description: No detail is supplied on the process used to label the 446 advisories or on safeguards ensuring that rule writing did not involve post-hoc adjustments based on test-set observations. Without this information the temporal split's independence and the reproducibility of the 66.8 % / 75.1 % recall figures cannot be verified.

    Authors: We will add a dedicated subsection in the benchmark description detailing the labeling process (advisories were identified via keyword search and manual review of OpenClaw GitHub issues and commits, with each mapped to specific source locations by two authors with inter-rater agreement measured). We confirm and will document that all rule development and tuning occurred exclusively on the 229 temporally earlier train advisories, with the 217 test advisories held completely out of view until final evaluation; no post-hoc adjustments were made. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: held-out temporal split supplies independent test set for rule evaluation

full rationale

The paper derives 47 Semgrep rules and 30 CodeQL queries from a 229-advisory train split drawn from OpenClaw, then measures recall lift on a temporally separated 217-advisory test split from the same repository. This construction is standard supervised evaluation on unseen instances rather than any self-definitional, fitted-input, or self-citation reduction; the reported deltas (Semgrep 21.7 % → 66.8 %, CodeQL 13.8 % → 75.1 %) are computed against external baselines on data the rules were never tuned to. No load-bearing premise collapses to a prior self-citation or to the input advisories by definition. The evaluation therefore remains self-contained against its stated benchmark.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the applicability of a STRIDE-derived taxonomy to agent runtime components and on the representativeness of the single-repository benchmark; no numerical free parameters or new physical entities are introduced.

assumptions (1)
  • domain assumption STRIDE threat categories can be mapped to implementation patterns in prompt builders, tool dispatchers, skill loaders, memory writers, and permission gates of local LLM agents
    The five-category taxonomy is derived directly from STRIDE and instantiated in the custom rules.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer." pith.science (2026). https://pith.science/paper/LAAFVJEV

@misc{pith2026260621071,
  author       = {Pith},
  title        = {Pith review of: Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LAAFVJEV}},
  note         = {Machine review of arXiv:2606.21071}
}
read the original abstract

Local LLM agents such as OpenClaw and Nanobot run on end-user machines and act on host resources - the shell, filesystem, browser, stored credentials, and messaging applications - through natural-language goals. These agents have become privileged software runtimes that mediate between user intent, model outputs, and host-level actions. Existing research characterizes the landscape through prompt injection, malicious skills, marketplace risks, or black-box evaluation of agents. But the implementation layer that performs this mediation, the prompt builder, parser, tool dispatcher, skill loader, memory writer, network client, and permission gate, has remained an unexamined safety boundary. To our knowledge, no prior work has examined the agent's source tree to audit these components for implementation-level security weaknesses. We present CLAWAUDIT, a static auditing framework for measuring vulnerability exposure in local LLM agent runtimes. CLAWAUDIT derives a five-category vulnerability taxonomy from STRIDE and develops custom static-analysis rules that target agent-specific patterns absent from established rule sets for vulnerability analysis. We instantiate the taxonomy in two backends, 47 Semgrep YAML rules and 30 CodeQL queries, and evaluate on OPENCLAWBENCH, a benchmark of 446 source-code-level advisories from the OpenClaw repository and split temporally into 229 rule-derivation (train) and 217 held-out (test) advisories. On the held-out test, CLAWAUDIT raises Semgrep recall from 21.7% (Pro baseline) to 66.8%, and CodeQL recall from 13.8% (security-extended) to 75.1%. Train/test gaps remain within 4 percentage points for all four configurations, indicating that the rules generalize to vulnerabilities unseen during rule writing. A preliminary live-code audit shows that these recall-oriented rules require manual triage, motivating semantic filtering before production deployment.

Figures

Figures reproduced from arXiv: 2606.21071 by the authors.

Figure 1
Figure 1. Overview of CLAWAUDIT’s taxonomy→rule→benchmark pipeline. The framework derives an agent-runtime taxonomy, encodes domain-specific static-analysis rules, and evaluates them against disclosed source-code-level advisories. CodeQL evaluates declarative predicates over a relational program graph and supports limited inter-procedural data-flow reasoning. Implementing the same five-category taxonomy in both backends lets … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 6 canonical work pages

  1. [1]

    SetupBench: Assessing software engineering agents’ ability to bootstrap development environments.arXiv preprint arXiv:2507.09063, 2025

    Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. SetupBench: Assessing software engineering agents’ ability to bootstrap development environments.arXiv preprint arXiv:2507.09063, 2025

    work page arXiv 2025

  2. [2]

    ToxicSkills: Snyk Finds Malware and Prompt Injection in 36% of AI Agent Skills

    Luca Beurer-Kellner et al. ToxicSkills: Snyk Finds Malware and Prompt Injection in 36% of AI Agent Skills. Snyk Blog, February 2026. https://snyk.io/blog/ toxicskills-malicious-ai-agent-skills-clawhub

    2026

  3. [3]

    280+ leaky skills: How openclaw & clawhub are exposing api keys and pii.Snyk Blog, February, 2026

    Luca Beurer-Kellner, Aleksei Kudrinskii, Marco Milanta, Kristian Bonde Nielsen, Hemang Sarkar, and Liran Tal. 280+ leaky skills: How openclaw & clawhub are exposing api keys and pii.Snyk Blog, February, 2026. 20

    2026

  4. [4]

    Technical Report: Exploring the Emerging Threats of the Agent Skill Ecosystem

    Luca Beurer-Kellner, Aleksei Kudrinskii, Marco Milanta, Kristian Bonde Nielsen, Hemang Sarkar, and Liran Tal. Technical report: Exploring the emerging threats of the agent skill ecosystem.arXiv preprint arXiv:2605.28588, 2026

    work page Pith review arXiv 2026

  5. [5]

    Openclaw security: Risks of exposed ai agents explained

    Bitsight. Openclaw security: Risks of exposed ai agents explained. https://www.bitsight. com/blog/openclaw-ai-security-risks-exposed-instances , 2026. Accessed: 2026-05-11

    2026

  6. [6]

    ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents

    Hwan Chang, Yonghyun Jun, and Hwanhee Lee. Chatinject: Abusing chat templates for prompt injection in llm agents.arXiv preprint arXiv:2509.22830, 2025

    work page Pith review arXiv 2025

  7. [7]

    Securing the Agent Skill Ecosystem: How Snyk and Vercel Are Protecting Developers

    Randall Degges. Securing the Agent Skill Ecosystem: How Snyk and Vercel Are Protecting Developers. Snyk Blog, February 2026. https://snyk.io/blog/ snyk-vercel-securing-agent-skill-ecosystem/

    2026

  8. [8]

    CodeQL: Semantic code analysis engine

    GitHub. CodeQL: Semantic code analysis engine. https://codeql.github.com, 2025. Ac- cessed: 2026-06-16

    2025

Show all 27 references
  1. [9]

    GitHub advisory database

    GitHub. GitHub advisory database. https://github.com/advisories, 2026. Accessed: 2026-06-16

    2026

  2. [10]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. InProceedings of the 16th ACM workshop on artificial intelligenc...

    2023

  3. [11]

    Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers.arXiv preprint arXiv:2506.13538, 2025

    Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers.arXiv preprint arXiv:2506.13538, 2025

    2025 arXiv

  4. [12]

    Nanobot: Lightweight, open-source AI agent for tools, chats, and workflows

    HKUDS. Nanobot: Lightweight, open-source AI agent for tools, chats, and workflows. https: //github.com/HKUDS/nanobot, 2026. Accessed: 2026-06-16

    2026

  5. [13]

    ClawHavoc: 341 malicious clawed skills found by the bot they were targeting, 2026

    Koi Security. ClawHavoc: 341 malicious clawed skills found by the bot they were targeting, 2026. Accessed: 2026-05-11

    2026

  6. [14]

    LangChain, October 2022

    LangChain AI. LangChain, October 2022. https://github.com/langchain-ai/ langchain

    2022

  7. [15]

    LlamaIndex, 11 2022.https://github.com/jerryjliu/llama_index

    Jerry Liu. LlamaIndex, 11 2022.https://github.com/jerryjliu/llama_index

    2022

  8. [16]

    Agent skills in the wild: An empirical study of security vulnerabilities at scale.arXiv preprint arXiv:2601.10338, 2026

    Yi Liu, Weizhe Wang, Ruitao Feng, Yao Zhang, Guangquan Xu, Gelei Deng, Yuekang Li, and Leo Zhang. Agent skills in the wild: An empirical study of security vulnerabilities at scale.arXiv preprint arXiv:2601.10338, 2026

    2026 arXiv

  9. [17]

    The STRIDE threat model

    Microsoft. The STRIDE threat model. https://learn.microsoft.com/en-us/ previous-versions/commerce-server/ee823878(v=cs.20), 2009. Accessed: 2026- 06-16

    2009

  10. [18]

    Evaluating and comparing memory error vulnerability detectors.Information and Software Technology, 137:106614, 2021

    Yu Nong, Haipeng Cai, Pengfei Ye, Li Li, and Feng Chen. Evaluating and comparing memory error vulnerability detectors.Information and Software Technology, 137:106614, 2021. 21

    2021

  11. [19]

    Open science in software engineering: A study on deep learning-based vulnerability detection.IEEE Transactions on Software Engineering, 49(4):1983–2005, 2022

    Yu Nong, Rainy Sharma, Abdelwahab Hamou-Lhadj, Xiapu Luo, and Haipeng Cai. Open science in software engineering: A study on deep learning-based vulnerability detection.IEEE Transactions on Software Engineering, 49(4):1983–2005, 2022

    1983

  12. [20]

    DualGauge: Automated joint security-functionality benchmarking of specification- only code generation by LLMs and coding agents.arXiv preprint arXiv:2511.20709, 2026

    Rupam Patir, Keyan Guo, Suvadra Barua, Abhijeet Pathak, Dinesh Gudimetla, Jiawei Guo, Hongxin Hu, and Haipeng Cai. DualGauge: Automated joint security-functionality benchmarking of specification- only code generation by LLMs and coding agents.arXiv preprint arXiv:2511.20709, 2026

    2026

  13. [21]

    Semgrep: Lightweight static analysis for many languages

    Semgrep, Inc. Semgrep: Lightweight static analysis for many languages. https://semgrep.dev,

  14. [22]

    Accessed: 2026-06-16

    2026

  15. [23]

    John Wiley & Sons, Indianapolis, IN, 2014

    Adam Shostack.Threat Modeling: Designing for Security. John Wiley & Sons, Indianapolis, IN, 2014

    2014

  16. [24]

    PicoClaw: Tiny, fast, and deployable anywhere — an ultra-lightweight personal AI assistant

    Sipeed. PicoClaw: Tiny, fast, and deployable anywhere — an ultra-lightweight personal AI assistant. https://github.com/sipeed/picoclaw, 2026. Accessed: 2026-06-16

    2026

  17. [25]

    agent-scan: Security Scanner for AI Agents, MCP Servers, and Agent Skills

    Snyk. agent-scan: Security Scanner for AI Agents, MCP Servers, and Agent Skills. GitHub repository, 2026.https://github.com/snyk/agent-scan

    2026

  18. [26]

    OpenClaw: Personal AI assistant

    Peter Steinberger and OpenClaw Contributors. OpenClaw: Personal AI assistant. https://github. com/openclaw/openclaw, 2026. Accessed: 2026-06-16

    2026

  19. [27]

    CWE-74"] category:

    Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 7101–7117, 2025. A Representative CLAWAUDITR...

    2025

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.