{"id":"a3bfc701-d0b4-4ef0-a3e9-088e10ed7f91","arxiv_id":"2606.21071","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CLAWAUDIT applies a STRIDE-derived taxonomy and 47 Semgrep plus 30 CodeQL rules to local LLM agent code, lifting recall on held-out OpenClaw advisories from 21.7% and 13.8% baselines to 66.8% and 75.1%.","lead":"The paper introduces CLAWAUDIT, a static auditing framework with custom Semgrep and CodeQL rules to find security issues in the source code of local LLM agent runtimes such as OpenClaw. A smart generalist might read it because these agents control host resources like shells and files, making their implementation a key security boundary.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Benchmark and rules derived exclusively from OpenClaw; no cross-project evaluation on other runtimes (e.g. Nanobot)","rationale":"The reader's weakest assumption matches the single point that prevents the reported recall numbers from supporting the broader claim about local LLM agent runtimes. The temporal split and small train/test gap address overfitting within OpenClaw but leave the cross-runtime representativeness untested; that is the load-bearing assumption for any claim beyond this one repository.","tokens_in":1864,"tokens_out":357,"duration_ms":18578,"concrete_test":"Port the 47 Semgrep YAML rules and 30 CodeQL queries to Nanobot (or another open local-LLM-agent codebase), collect or manually label a set of source-level advisories in that project, and compute recall; if the lift over the same baselines falls below 30 points, the generalization claim does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim reports recall lifts (Semgrep 21.7%→66.8%, CodeQL 13.8%→75.1%) on a temporally split held-out set of 217 advisories drawn from the same OpenClaw repository used to derive the 47+30 custom rules. The taxonomy and patterns target agent-specific constructs (prompt builder, tool dispatcher, permission gate, etc.) that may be implemented differently in other local LLM runtimes. The temporal split only guarantees that test advisories were not seen during rule writing; it does not establish that the vulnerability distribution itself is representative of the broader class of runtimes the framework claims to address. Without at least one additional codebase, the measured recall cannot be separated from OpenClaw-specific coding idioms.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper presents CLAWAUDIT, a static auditing framework for local LLM agent runtimes. It derives a five-category vulnerability taxonomy from STRIDE and implements it via 47 Semgrep YAML rules and 30 CodeQL queries targeting agent-specific patterns. The framework is evaluated on OPENCLAWBENCH, a dataset of 446 source-code-level advisories from the OpenClaw repository temporally split into 229 for rule derivation and 217 held-out for testing. On the test set, it reports substantial recall improvements over baselines: Semgrep from 21.7% to 66.8% and CodeQL from 13.8% to 75.1%, with train/test gaps under 4 percentage points.","tokens_in":2050,"tokens_out":526,"duration_ms":40777,"significance":"If the results hold, this work offers the first source-code audit of the runtime layer in local LLM agents and demonstrates that custom static rules can significantly enhance detection of implementation-level vulnerabilities in these systems. The use of a temporal held-out split and small generalization gap provide solid support for the recall claims within the OpenClaw codebase. The acknowledgment of the need for manual triage in live audits is a realistic assessment of practical deployment.","major_comments":[{"comment":"Abstract: The abstract positions CLAWAUDIT as applicable to local LLM agent runtimes in general (explicitly referencing Nanobot alongside OpenClaw), yet the evaluation, rule development, and OPENCLAWBENCH benchmark are performed exclusively on advisories from the OpenClaw repository. This single-project scope means the reported recall gains cannot be separated from OpenClaw-specific coding idioms, so the headline improvements do not yet establish the framework's utility for the broader class of runtimes claimed.","section":"Abstract"},{"comment":"Abstract and benchmark description: No detail is supplied on the process used to label the 446 advisories or on safeguards ensuring that rule writing did not involve post-hoc adjustments based on test-set observations. Without this information the temporal split's independence and the reproducibility of the 66.8 % / 75.1 % recall figures cannot be verified.","section":"Abstract"}],"minor_comments":[{"comment":"Abstract: The phrase 'preliminary live-code audit' is introduced without any quantitative scope, number of issues examined, or concrete findings beyond the general statement that manual triage is required.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback highlighting limitations in scope and reproducibility. We address each major comment below and will incorporate revisions to strengthen the manuscript.","responses":[{"response":"We agree that the abstract and claims overstate generality given the single-project evaluation. The taxonomy is derived from STRIDE and intended to apply broadly, but the rules and recall figures are specific to OpenClaw idioms. We will revise the abstract, introduction, and conclusion to explicitly state that empirical results are from OpenClaw, note that applicability to other runtimes such as Nanobot requires further validation, and frame the work as an initial case study rather than a general proof.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The abstract positions CLAWAUDIT as applicable to local LLM agent runtimes in general (explicitly referencing Nanobot alongside OpenClaw), yet the evaluation, rule development, and OPENCLAWBENCH benchmark are performed exclusively on advisories from the OpenClaw repository. This single-project scope means the reported recall gains cannot be separated from OpenClaw-specific coding idioms, so the headline improvements do not yet establish the framework's utility for the broader class of runtimes claimed."},{"response":"We will add a dedicated subsection in the benchmark description detailing the labeling process (advisories were identified via keyword search and manual review of OpenClaw GitHub issues and commits, with each mapped to specific source locations by two authors with inter-rater agreement measured). We confirm and will document that all rule development and tuning occurred exclusively on the 229 temporally earlier train advisories, with the 217 test advisories held completely out of view until final evaluation; no post-hoc adjustments were made.","revision_made":"yes","referee_comment":"[Abstract] Abstract and benchmark description: No detail is supplied on the process used to label the 446 advisories or on safeguards ensuring that rule writing did not involve post-hoc adjustments based on test-set observations. Without this information the temporal split's independence and the reproducibility of the 66.8 % / 75.1 % recall figures cannot be verified."}],"tokens_in":1616,"tokens_out":465,"duration_ms":30101,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is a source-level audit of local LLM agent runtimes. They take the prompt builder, tool dispatcher, skill loader, and permission gate components, map them to a STRIDE-derived taxonomy, and write 47 Semgrep rules plus 30 CodeQL queries that target patterns not covered by standard security rule sets. On a temporally split held-out set of 217 advisories from the OpenClaw repo, the custom rules lift Semgrep recall from 21.7% to 66.8% and CodeQL from 13.8% to 75.1%, with train/test gaps under 4 points. That temporal split and the small gap are the strongest parts of the evidence; they show the rules are not just memorizing the training advisories.\n\nWhat the work actually supplies is a concrete way to look inside the runtime layer that sits between the model output and host actions. The benchmark construction from real advisories in one popular agent is useful as a starting point, and the claim that no prior work has done source-tree audits of these specific components holds up in the abstract.\n\nThe soft spot is clear and not minor: every rule and every test item comes from OpenClaw. The stress-test note is right that this leaves open whether the patterns transfer to other runtimes such as Nanobot. The authors themselves flag that the rules still need manual triage, which limits immediate practical use. Labeling details for the 446 advisories are not spelled out in the abstract, so it is hard to judge how much post-hoc adjustment went into the rules.\n\nThis is the kind of paper that belongs in a security or AI-systems venue. Readers who work on agent safety or static analysis tooling will get value from the taxonomy and the rule examples even if they have to adapt them. It is coherent on its own terms and engages the right prior work on prompt injection and agent evaluation. I would send it to peer review; the core method is worth referee scrutiny on generalizability and on how the advisories were collected.","headline":"The paper builds a static-analysis framework for local LLM agent runtimes and shows solid recall gains on OpenClaw advisories, but the evaluation stays inside that single codebase.","tokens_in":2534,"tokens_out":493,"would_cite":false,"duration_ms":10423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A static auditing framework for local LLM agent runtimes raises recall on source-level vulnerabilities from under 22 percent to 66-75 percent.","keywords":["local LLM agents","source code audit","static analysis","vulnerability detection","Semgrep","CodeQL","agent runtime security","STRIDE taxonomy"],"falsifier":"Applying the same 47 Semgrep rules and 30 CodeQL queries to the full source tree of a second local LLM agent and obtaining recall below 30 percent on its documented vulnerabilities would show the rules do not transfer.","tokens_in":2776,"feed_emoji":"🔍","tokens_out":807,"duration_ms":29205,"temperature":0.7,"pith_summary":"The paper examines the source code of local LLM agents that translate natural-language goals into actions on the host shell, filesystem, and other resources. It introduces CLAWAUDIT to audit the prompt builder, parser, tool dispatcher, and permission components that had not been checked before. The framework creates a STRIDE-derived taxonomy and writes custom rules for two static-analysis engines that target patterns unique to these agents. When tested on a temporally split set of 446 real advisories, the rules detect far more issues than the engines' built-in security checks while maintaining performance on code written after the rules were created. This matters because these agents run with direct access to user data and credentials, turning their internal implementation into a privileged attack surface.","feed_headline":"Custom rules lift LLM agent vulnerability recall to 66-75 percent","feed_subtitle":"Static checks written for prompt builders and permission gates outperform standard tools on held-out agent code.","key_machinery":"CLAWAUDIT, the static auditing framework that pairs a five-category STRIDE-derived taxonomy with custom Semgrep and CodeQL rules written for agent runtime patterns such as prompt construction and permission gating.","core_discovery":"CLAWAUDIT derives a five-category vulnerability taxonomy from STRIDE and instantiates it as 47 Semgrep YAML rules plus 30 CodeQL queries that target agent-specific patterns absent from standard rule sets. On the held-out test portion of OPENCLAWBENCH (217 advisories), the Semgrep rules reach 66.8 percent recall versus 21.7 percent for the Pro baseline, while the CodeQL queries reach 75.1 percent recall versus 13.8 percent for the security-extended baseline. Performance on the train and test splits differs by at most four percentage points, showing that the rules generalize beyond the examples used to write them. A live-code audit indicates that the rules still require manual triage for produ","pith_inferences":["The same taxonomy and rule-writing approach could be reapplied to other agent codebases to produce comparable coverage numbers for their runtime layers.","Treating the agent runtime as an operating-system-level component implies that future agents may need formal verification or language-level isolation in addition to static checks.","The remaining manual-triage step points to a possible next engineering target: lightweight semantic filters that reduce false positives while preserving the observed recall gains."],"forward_implications":["Developers of local agents can integrate the rules into their build process to surface flaws in the prompt builder, memory writer, and network client before release.","The small train-test gap shows that rules written against older code continue to find newer implementation weaknesses in the same agent.","Standard security rule sets for Semgrep and CodeQL miss the majority of the implementation-level issues that arise when agents mediate between models and host resources.","Automated detection must be followed by manual review because the rules are tuned for recall rather than precision."],"fun_headline_variants":["STRIDE taxonomy guides audit of local LLM agent runtimes","Semgrep rules reach 66.8 percent recall on held-out agent advisories","CodeQL queries reach 75.1 percent recall on held-out agent code","Custom rules target prompt builders and tool dispatchers in agents","Agent runtime components audited for security weaknesses via static analysis"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The 446 advisories drawn from one agent repository capture the vulnerability distribution that would appear in other local LLM agent runtimes.","fun_headline_variants_meta":{"raw":{"variants":["STRIDE taxonomy guides audit of local LLM agent runtimes","Semgrep rules reach 66.8 percent recall on held-out agent advisories","CodeQL queries reach 75.1 percent recall on held-out agent code","Custom rules target prompt builders and tool dispatchers in agents","Agent runtime components audited for security weaknesses via static analysis"]},"model":"grok-4.3","cost_usd":0.009797,"raw_usage":{"total_tokens":4459,"prompt_tokens":865,"num_sources_used":0,"completion_tokens":87,"cost_in_usd_ticks":97974500,"prompt_tokens_details":{"text_tokens":865,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3507,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":865,"tokens_out":87,"duration_ms":29348,"temperature":1.0,"reasoning_tokens":3507,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T13:59:34.404762+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Applying the same 47 Semgrep rules and 30 CodeQL queries to the full source tree of a second local LLM agent and obtaining recall below 30 percent on its documented vulnerabilities would show the rules do not transfer.","supporting_citations":[],"review_version":1}