Pith. sign in

REVIEW 2 major objections 4 minor 38 references

SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents

T0 review · 2 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SkillGate claims a two-stage regex-plus-LLM gateway can screen AI coding-agent skill files at install time, detecting malicious skills with F1 0.817 and a 1.13% false-positive rate while cutting LLM input tokens by 77% compared with full-fi

desk verdict Well-executed, honestly-evaluated skill-file detector; benchmark numbers hold, but the regex prefilter's 90% recall ceiling is the real external-validity risk. read the letter →

arxiv 2607.25619 v1 pith:7WDZBONU submitted 2026-07-28 cs.SE cs.CR

Rui Yang , Michael Fu , Kla Tantithamthavorn , Chetan Arora , Joey Chua This is my paper · ORCID
classification cs.SEcs.CR
keywords agentskillfilessupply-chainsecurityruntimescreeningLLMjudgeregexprefilterfalse-positiveratecodingagentsinstall-timedefense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that malicious skill files—Markdown instructions that coding agents load from registries—can be detected cheaply enough to screen every install. It proposes a gateway that first applies a large regex ruleset; files with no matches pass without any language-model call, and flagged files send only their matched snippet windows to a small LLM judge instead of the whole file. On a benchmark of 1,650 skill files with 9.1% malicious prevalence, the system reports F1 0.817, false-positive rate 1.13%, and 77% fewer LLM tokens than full-file review, with a weighted per-file latency around 0.8 seconds. The point is that a deployable pre-install defense is feasible where static-only scanners produce unusable false-positive rates and full-file LLM review is too slow and costly.

What carries the argument

The load-bearing mechanism is the two-stage gate: (1) a RuleEngine of 530 regex patterns derived from a public attack-technique taxonomy and community detection rules, which returns hit positions but never a final verdict; and (2) a snippet assembler that extracts up to 20 windows of 500 characters around each match, prepends a short context preamble, and sends this bounded payload to an LLM judge that returns SAFE, SUSPICIOUS, or MALICIOUS with a confidence score. The design separates gating from adjudication: the regex decides where to look, the LLM decides what the evidence means. Because 67% of files have zero hits and bypass the model, and flagged files ship only their suspicious region

What would settle it

Run the published system on a fresh collection of malicious skill files harvested from public registries after the paper's release and compare recall to the benchmark's 0.769; if recall drops materially, or if the regex prefilter's ceiling (0.900 on the benchmark) is far lower on that collection, the deployment claim is falsified. A narrower experiment: measure false positives on a documentation-heavy corpus with base64 examples—if they exceed the reported 1.13%, the FPR claim does not generalize.

Watch

Extended reading notes

Core claim

The central claim is that a hybrid pipeline—a deterministic regex prefilter acting as a gate, not a classifier, plus an LLM judge that sees only the flagged snippet windows—resolves the accuracy-cost tradeoff for skill-file screening. The prefilter alone catches 135 of 150 malicious files but flags 27% of benign files; adding the judge cuts false positives to 1.13% at recall 0.769. Compared with two prior static scanners, the system reports a 5–6x higher threshold-independent precision-recall area, and compared with sending every file in full to the same judge, it uses 77% fewer tokens while reaching similar recall. The paper frames this as closing the gap at the agent-instruction layer, bef

Load-bearing premise

The load-bearing premise is that the benchmark's hand-crafted malicious skills represent real attacks; if real-world malicious skills are more varied or better obfuscated, the reported recall and false-positive rate will not transfer.

Editorial extensions

If this is right

  • A coding agent wired through the proxy can screen every skill installation with roughly 0.8 seconds of weighted overhead, making inline pre-install defense feasible within a normal registry fetch.
  • Developers would see roughly 17 false positives out of 1,500 benign skills, versus hundreds for static-only scanners, so the tool does not erode trust with spurious blocks.
  • Operators can lower the confidence threshold to about 0.14 to push recall to 0.824 while keeping false positives near 1.3%, without per-deployment tuning.
  • Sending the full file to the same judge instead of snippet windows adds only about 3.5 percentage points of recall but roughly triples the number of LLM calls and raises false positives, so the snippet design is the better default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gate-decides-where, judge-decides-what split should transfer to other LLM context channels—runtime tool responses, fetched documentation, even email—where adversarial text can arrive as data but act as instructions.
  • The pipeline's ceiling is set by the 530-pattern prefilter: any attack whose text contains none of those patterns is invisible to the system no matter how good the judge is. A natural next study is measuring recall against obfuscation techniques designed to avoid those patterns.
  • Token savings will vary with file length: for very short skill files the fixed context preamble dominates each request, so deployments with mostly tiny skills will see smaller cost reductions than the 77% reported here.
  • Because the benchmark's malicious files are hand-crafted exemplars, the strongest test of the paper's claim is an in-the-wild evaluation on freshly observed registry attacks, which would also reveal whether the reported false-positive categories shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents SkillGate, a hybrid detection gateway for malicious AI agent skill files. A 530-pattern regex prefilter (428 MITRE ATT&CK-derived + 102 Sigma rules) scans skill-file content; files with zero hits are classified SAFE without an LLM call, while files with hits are escalated to an LLM judge that sees only the matched snippet windows (default cw=500, ms=20), not the full file. The system is evaluated on SkillsBench (n=1,650, 9.1% malicious) against ClawVet and SkillScanner, with ablations (pure regex, full-file LLM, snippet-budget variants). The headline results are F1=0.817, recall=0.769, FPR=1.13%, MCC=0.803, AUPRC=0.830, 76.9% LLM-token reduction versus full-file screening, and weighted average latency ~818ms. The paper also analyzes false-positive categories and reports three-run means with standard deviations.

Significance. If the reported benchmark results transfer to practice, SkillGate addresses a real and growing supply-chain attack surface with a cost profile that makes install-time screening plausible. The work has several genuine strengths: the implementation and ruleset are open source; the confidence threshold and context-window parameters were pre-specified on a held-out development set; baselines are evaluated at oracle-optimal thresholds, which is conservative with respect to SkillGate; and the ablation study isolates the contribution of each pipeline stage. The main risk to significance is external validity: SkillsBench is a single benchmark whose 150 malicious files are hand-crafted attack exemplars, and the prefilter's zero-hit fast path imposes a structural recall ceiling that the LLM judge cannot bypass. These limitations are acknowledged honestly in Section VI, but they are load-bearing for the paper's broader 'deployable defense' claim.

major comments (2)
  1. [Section III.B, Table VI (B1)] The prefilter blind spot is a load-bearing limitation, not just a routine external-validity caveat. Files with zero regex hits are classified SAFE with no LLM call (Section III.B), so the 15 malicious SkillsBench files that fail to match any of the 530 patterns are structurally invisible to the judge. Table VI (B1) shows the prefilter recall ceiling is 0.900 (135/150); the RQ1 recall of 0.769 is therefore capped by regex coverage before the judge is even consulted. The paper should quantify how many of these 15 zero-hit malicious files the LLM would catch if they were escalated (e.g., by running the full-file judge on this subset), and discuss a concrete fallback policy — for example, escalating zero-hit files with high entropy, encoding markers, or suspicious metadata. The B2 ablation (full-file LLM, recall 0.804) suggests the LLM alone cannot fully recover the 15, but the overlap betwe
  2. [Section VI, Section VII] The conclusion states that SkillGate is a 'practical first line of defense' and the abstract emphasizes 'deployable,' but the evaluation corpus contains only hand-crafted malicious skills sampled from eight fixed categories. The paper concedes in Section VI that live ClawHavoc samples were unavailable and that measured recall may not transfer to organically evolving campaigns. This is an honest statement, but the conclusion does not temper the deployability claim accordingly. I would ask the authors to add a robustness analysis — for example, applying simple transformations (tokenization changes, base64/hex encoding, synonym substitutions, splitting payloads across multiple code blocks) to the 150 malicious files and measuring how many fall below the prefilter threshold. Such an experiment would directly test the structural blind spot and give practitioners a realistic sense of the defen
minor comments (4)
  1. [Section V, Figure 5] Typographical errors: 'realisitc' and 'deployabile' should be 'realistic' and 'deployable'. The formatting of Table VI also has missing spaces in 'SKILLGATEms=5snippets' and 'SKILLGATE(default,ms=20)'.
  2. [Table V] In Table V, the SKILLGATE row appears to have a formatting issue: '173 0 13 13' should likely read '17, 3, 0, 13, 13' (FPs = 17, block-grade = 3, critical = 0, high = 13, medium = 13). Please verify the column alignment.
  3. [Section III.D] The policy mapping in Stage 4 is ambiguous: 'SUSPICIOUS (confidence≥0.5) → QUARANTINE; SUSPICIOUS (confidence<0.5) → WARN.' Since the label SUSPICIOUS already carries a meaning, clarify how the numeric confidence interacts with the label (e.g., is SUSPICIOUS with confidence 0.4 treated differently from SAFE with confidence 0.4?).
  4. [References] Reference [1] is a LinkedIn post; for a journal version, consider replacing it with a peer-reviewed or at least a citable measurement study if one becomes available, or clearly mark it as a non-archival industry report. The same applies to the 'ClawHavoc' claim, which is central to the motivation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark is external, patterns derive from MITRE ATT&CK/Sigma, configuration is pre-specified on a held-out dev set, and reported limitations are external-validity caveats rather than fitted inputs.

full rationale

SkillGate is an empirical system paper rather than a derivation from first principles. The benchmark (SkillsBench, n=1,650) is external ([23], X. Chen), and the 530 regex patterns come from the MITRE ATT&CK framework [15] and Sigma rules [16], not from the benchmark labels or from the authors' own prior work. The operating point (cw=500, threshold θ=0.70, ms=20) is stated as pre-specified on a held-out development set; the ms=20 choice is explicitly justified on that dev set 'without consulting benchmark labels.' The one structural property—files with zero prefilter hits are classified SAFE and never reach the LLM judge—is transparently reported: ablation B1 shows the pure-regex prefilter ceiling is recall 0.900 (135/150), and Section VI concedes that 'SkillsBench's 150 malicious files are hand-crafted attack exemplars rather than captured in-the-wild samples' and that performance on live adversarial packages 'may differ from our benchmark results.' These are external-validity limitations, not circular reductions of the headline result to its inputs. The AUPRC scoring choice (prefilter-safe files receive score 0) reflects the deployed pipeline's actual behavior and is not a fitted artifact. No load-bearing self-citations, no imported uniqueness theorems, no ansatz-smuggling via citation, and no fitted quantities relabeled as predictions appear in the derivation chain. The paper is self-contained against an external benchmark and reports its own ablations honestly, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a small number of manually chosen hyperparameters (snippet budget, context window, confidence threshold) — all chosen on a dev set or by design, not fitted to the benchmark test set. The main structural assumptions are benchmark representativeness and the concentration-of-malice assumption behind snippet windows.

free parameters (4)
  • maximum snippet windows (ms) = 20
    Chosen on a held-out development set because it dominates ms=5 (recall 0.972 vs 0.928 at precision 1.0); controls token savings and accuracy.
  • context window (cw) = 500 characters
    Default context window around each regex match; system hyperparameter affecting snippet content and LLM accuracy.
  • confidence threshold (θ) = 0.70
    Pre-specified on a held-out development set; maps LLM verdict+confidence to BLOCK/QUARANTINE/WARN/ALLOW. Test-set F1-optimal threshold is ~0.14, reported but not used as the headline.
  • preamble length = ~700 characters
    Fixed preamble with package name/path/metadata prepended to snippet; affects token count, especially for short files.
assumptions (5)
  • domain assumption MITRE ATT&CK and Sigma rule sets cover the relevant attack techniques for skill files.
    The prefilter is built from 428 MITRE ATT&CK patterns and 102 Sigma rules; the paper acknowledges it cannot guarantee coverage of all security patterns.
  • domain assumption The hostile content in a malicious skill file is concentrated in small regions, so bounded snippet windows (cw=500, ms=20) capture the evidence.
    Design assumption behind snippet savings; B2 full-file ablation shows snippet windows lose 3.5pp recall vs full-file, confirming the assumption holds only partially.
  • domain assumption SkillsBench (n=1,650, 9.1% malicious) is representative of the real-world skill-file distribution.
    The paper flags in Threats to Validity that the 150 malicious files are hand-crafted exemplars, not in-the-wild samples, so measured recall may not transfer.
  • domain assumption An LLM-as-a-judge (gpt-5.4-mini) can correctly adjudicate SAFE/SUSPICIOUS/MALICIOUS from snippet windows.
    The final label comes from the LLM; results may shift across API versions or sampling settings, acknowledged in Threats to Validity.
  • standard math Standard evaluation metrics (F1, MCC, AUPRC) are appropriate for imbalanced skill-file detection.
    Metrics are standard; AUPRC is computed from a per-sample likelihood score defined in Section III-G.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents." pith.science (2026). https://pith.science/paper/7WDZBONU

@misc{pith2026260725619,
  author       = {Pith},
  title        = {Pith review of: SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7WDZBONU}},
  note         = {Machine review of arXiv:2607.25619}
}
read the original abstract

Software engineering teams now deploy AI coding agents (Cursor, Claude Code, GitHub Copilot) as first-class productivity tools, installing domain-specific skill files to tailor agent behavior to project APIs, framework conventions, and organizational workflows. These complex Markdown files are easily downloaded from public registries with a single npx skills add command and no real security screening, representing a novel supply-chain attack surface: a malicious skill file can silently reprogram agent behavior, exfiltrating credentials, injecting backdoors into generated code, or redirecting agent actions to attacker-controlled endpoints. The threat is not hypothetical: recent reports document hundreds of malicious skill packages in public registries, including organized campaigns that distributed credential-stealing infostealers via fake productivity skills. No systematic toolchain defense exists for this attack surface. We present SkillGate, a deployable security gateway that screens AI skill packages before coding agent installation. SkillGate uses a hybrid regex-prefilter + LLM-judge pipeline: safe-signal files bypass the LLM entirely (skip savings); flagged files have only their matched snippet windows sent to the judge, not the full content (snippet savings). We answer four research questions covering detection effectiveness, screening cost, runtime overhead, and false positive behavior on the SkillsBench benchmark against two existing tools. On SkillsBench (n=1,650, 9.1% malicious), SkillGate achieves F1=0.817, FPR=1.13% while reducing LLM input tokens by 77% vs. full-file screening, and outperforming existing tools by 5-6x on threshold-independent AUPRC (0.830 vs. 0.144/0.162).

Figures

Figures reproduced from arXiv: 2607.25619 by the authors.

Figure 1
Figure 1. A malicious skill file installs with no screening and can silently exfiltrate [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. End-to-end overview of SKILLGATE. prompts and responses against a fixed harm taxonomy (violence, self-harm, hate, and similar categories). These guardrails are built for conversational systems: they screen free-form user– assistant text, not the tool definitions, workflow instructions, and scripts that constitute a skill file, and carry no notion of an agent installing and acting on third-party instructions. SKILLGA… view at source ↗
Figure 3
Figure 3. Precision-Recall curves on SkillsBench. Filled circles mark current [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: False positive categorization of SKILLGATE on SkillsBench threshold). We additionally report the block-grade false-positive rate—the share of benign files each tool flags at HIGH or CRITICAL severity, which a severity-gated policy would hard￾block. Results: SKILLGATE p…
Figure 5
Figure 5. Figure 5: F1, Recall, and Precision of SKILLGATE on SkillsBench as the confidence threshold θ varies. o200k_base over the exact payloads the deployed system sends (preamble, skill header, and snippet windows) rather than simplified estimates, so the reported reductions match pro…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 10 linked inside Pith

  1. [1]

    Clawhavoc poisons openclaw’s clawhub with 1,184 malicious skills,

    C. Press, “Clawhavoc poisons openclaw’s clawhub with 1,184 malicious skills,” https://www.linkedin.com/pulse/clawhavoc-poisons-openclaws- clawhub-1184-malicious-skills-f5xqc, Feb. 2026

    2026

  2. [2]

    Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic ai,

    R. Sapkota, K. I. Roumeliotis, and M. Karkee, “Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic ai,”arXiv preprint arXiv:2505.19443, 2025

    arXiv 2025

  3. [3]

    Usage, effects and requirements for ai coding assistants in the enterprise: An empirical study,

    M. Vukovic, R. Pan, T. K. Ho, R. Krishna, R. Pavuluri, and M. Merler, “Usage, effects and requirements for ai coding assistants in the enterprise: An empirical study,”arXiv preprint arXiv:2601.20112, 2026

    arXiv 2026

  4. [4]

    Stack overflow developer survey 2024,

    Stack Overflow, “Stack overflow developer survey 2024,” https://survey.stackoverflow.co/2024/, 2024, 76% of respondents use or plan to use AI coding tools in their workflow. Accessed 2026-06

    2024

  5. [6]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66

    2024

  6. [7]

    Model Context Protocol specification,

    Anthropic, “Model Context Protocol specification,” https://modelcontextprotocol.io/, 2024, accessed 2026-06

    2024

  7. [8]

    Backstabber’s knife collection: A review of open source software supply chain attacks,

    M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstabber’s knife collection: A review of open source software supply chain attacks,” inProceedings of the 17th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA), 2020

    2020

  8. [9]

    Small world with high risks: A study of security threats in the npm ecosystem,

    M. Zimmermann, C. Staicu, C. Tenny, and M. Pradel, “Small world with high risks: A study of security threats in the npm ecosystem,” in28th USENIX Security Symposium, USENIX Security 2019, Santa Clara, CA, USA, August 14-16, 2019, N. Heninger and P. Traynor, Eds. USENIX Association, 2019, pp. 995–1010. [Online]. Available: https://www.usenix.org/conference...

    2019

Show all 38 references
  1. [10]

    Sok: Taxonomy of attacks on open-source software supply chains,

    P. Ladisa, H. Plate, M. Martinez, and O. Barais, “Sok: Taxonomy of attacks on open-source software supply chains,” in2023 IEEE Symposium on Security and Privacy (SP). IEEE, May 2023, p. 1509–1526. [Online]. Available: http://dx.doi.org/10.1109/sp46215.2023.10179304

    2023

  2. [11]

    Semgrep: Static analysis at ludicrous speed,

    Semgrep, Inc., “Semgrep: Static analysis at ludicrous speed,” https://semgrep.dev/, 2024, accessed 2026-06

    2024

  3. [12]

    Snyk: Security platform for developers,

    Snyk Ltd., “Snyk: Security platform for developers,” https://snyk.io/, 2024, accessed 2026-06

    2024

  4. [13]

    Malicious agent skills in the wild: A large-scale security empirical study,

    Y . Liu, Z. Chen, Y . Zhang, G. Deng, Y . Li, J. Ning, Y . Zhang, and L. Y . Zhang, “Malicious agent skills in the wild: A large-scale security empirical study,”arXiv preprint arXiv:2602.06547, 2026

    2026 arXiv

  5. [14]

    Skill-inject: Measuring agent vulnerability to skill file attacks,

    D. Schmotz, L. Beurer-Kellner, S. Abdelnabi, and M. Andriushchenko, “Skill-inject: Measuring agent vulnerability to skill file attacks,”arXiv preprint arXiv:2602.20156, 2026

    2026 arXiv

  6. [15]

    MITRE ATT&CK: Adversarial tactics, techniques, and common knowledge,

    MITRE Corporation, “MITRE ATT&CK: Adversarial tactics, techniques, and common knowledge,” https://attack.mitre.org, 2024, accessed: 2024

    2024

  7. [16]

    Sigma: Generic signature format for SIEM systems,

    SigmaHQ, “Sigma: Generic signature format for SIEM systems,” https://github.com/SigmaHQ/sigma, 2024, accessed 2026-06

    2024

  8. [17]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” inThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [Online]...

    2023

  9. [18]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”CoRR, vol. abs/2302.04761,

    arXiv

  10. [19]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qian, S. Zhao, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, “Toolllm: Facilitating large language models to master 16000+ real-world apis,”CoRR, vol. abs/2307.16789, 2023. [On...

    2023 doi arXiv

  11. [20]

    Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,”Communications of the ACM, vol. 68, no. 2, pp. 96–105, 2025

    2025

  12. [21]

    Do users write more insecure code with ai assistants?

    N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” inProceedings of the 2023 ACM SIGSAC conference on computer and communications security, 2023, pp. 2785–2799

    2023

  13. [22]

    Practical automated detection of malicious npm packages,

    A. Sejfia and M. Sch ¨afer, “Practical automated detection of malicious npm packages,” in44th IEEE/ACM International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25–27, 2022. ACM, 2022, pp. 1681–1692

    2022

  14. [23]

    SkillsBench-1650: A benchmark for AI agent skill safety detection,

    X. Chen, “SkillsBench-1650: A benchmark for AI agent skill safety detection,” https://huggingface.co/datasets/zenith6888/SkillsBench-1650, 2026, hugging Face dataset, CC-BY-4.0

    2026

  15. [24]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” inProceedings of the 16th ACM workshop on artificial intelligence and security, 2023, pp. 79–90

    2023

  16. [25]

    Open Source Security Foundation: Dependency review and SCA tooling,

    Open Source Security Foundation (OpenSSF), “Open Source Security Foundation: Dependency review and SCA tooling,” https://owasp.org/www-project-dependency-check/, 2022

    2022

  17. [26]

    Ignore previous prompt: Attack techniques for language models,

    F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,”arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  18. [27]

    Formalizing and benchmarking prompt injection attacks and defenses,

    Y . Liu, Y . Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024, D. Balzarotti and W. Xu, Eds. USENIX Association, 2024. ...

    2024

  19. [28]

    Benchmarking and defending against indirect prompt injection attacks on large language models,

    J. Yi, Y . Xie, B. Zhu, E. Kiciman, G. Sun, X. Xie, and F. Wu, “Benchmarking and defending against indirect prompt injection attacks on large language models,” inProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 1, 2025, pp. 1809–1820

    2025

  20. [29]

    Defending against indirect prompt injection attacks with spotlighting,

    K. Hines, G. Lopez, M. Hall, F. Zarfati, Y . Zunger, and E. Kiciman, “Defending against indirect prompt injection attacks with spotlighting,” inProceedings of the Conference on Applied Machine Learning in Information Security (CAMLIS 2024), Arlington, Virginia, USA, October 24...

    2024

  21. [30]

    Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails,

    T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen, “Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Ass...

    2023 doi

  22. [31]

    Prompt injection and privilege separation in LLM applica- tions,

    S. Willison, “Prompt injection and privilege separation in LLM applica- tions,” https://simonwillison.net/2023/Apr/25/dual-llm-pattern/, 2023

    2023

  23. [32]

    Llama guard: LLM- based input-output safeguard for human-AI conversations,

    H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa, “Llama guard: LLM- based input-output safeguard for human-AI conversations,”arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  24. [33]

    OpenAI moderation API,

    OpenAI, “OpenAI moderation API,” https://platform.openai.com/docs/guides/moderation, 2022, accessed 2026-06

    2022

  25. [34]

    ClawVet: Static analysis for MCP skill security,

    M. Shaikhet al., “ClawVet: Static analysis for MCP skill security,” https://github.com/MohibShaikh/clawvet, 2024

    2024

  26. [35]

    skill-scanner: MCP skill security scanner,

    Cisco AI Defense, “skill-scanner: MCP skill security scanner,” https://github.com/cisco-ai-defense/skill-scanner, 2024, v2.0.11. Accessed 2026-06

    2024

  27. [36]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,”CoRR, vol. abs/2306.05685, 2023. [Online]. Available: https://doi.org/10.48550/arXi...

    2023 doi arXiv

  28. [37]

    The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,

    D. Chicco and G. Jurman, “The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,”BMC Genomics, vol. 21, no. 1, p. 6, 2020

    2020

  29. [2021]

    Available: https://arxiv.org/abs/2107.03374

    [Online]. Available: https://arxiv.org/abs/2107.03374

    arXiv

  30. [2023]

    Available: https://doi.org/10.48550/arXiv.2302.04761

    [Online]. Available: https://doi.org/10.48550/arXiv.2302.04761

    doi arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.