Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SkillSec-Eval establishes that reusable agent skills are vulnerable at every lifecycle stage, not just execution, and quantifies the gaps with a 327-skill benchmark.

desk verdict A genuinely useful lifecycle taxonomy for agent-skill security, but every empirical claim in Section 6 should be treated as provisional — one internal inconsistency (planner model differs between baseline and defended setups) directly confounds the headline planner-defense numbers. read the letter →

arxiv 2607.13987 v1 pith:ECWFIHNP submitted 2026-07-15 cs.CR

Sanket Badhe , Priyanka Tiwari This is my paper · ORCID
classification cs.CR
keywords LLMagentsreusableskillsagentsupplychainsecuritylifecyclethreatmodelingpromptinjectionretrievalattackstainttrackingbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that reusable agent skills — packaged capabilities that let LLM agents call tools and workflows — are not just threatened at runtime. It decomposes the skill's life into distinct trust boundaries and shows that each one can be attacked: malicious skills slip through repository checks, retrieval can be gamed by Sybil publication and keyword stuffing, planners trust fabricated recommendations, runtime monitors miss paraphrased exfiltration, and updates inherit stale trust. To make this measurable, the paper introduces SkillSec-Eval, a benchmark built from 327 real-world skills, and reports attack success rates and defense scores per stage. The point is that defending agents requires lifecycle-aware security, not just prompt-injection or runtime filtering; the paper itself notes the setting is a controlled environment rather than a production deployment.

What carries the argument

The carrying mechanism is the lifecycle decomposition of a skill into six stages — authoring, storage, retrieval, selection, execution, evolution — each treated as a trust boundary, and its instantiation in the SkillSec-Eval framework as five operational components. The framework's key novelty is evaluating attacks at the stage where the violated trust assumption lives, with isolated defenses: a hybrid admission controller with LLM semantic validation, retrieval validation (diversity filtering, metadata consistency, permission justification), planner-side metadata sanitization, a runtime monitor with dynamic taint tracking and policy enforcement, and update revalidation via provenance checks

What would settle it

A production deployment that logs attack attempts before and after enabling these defenses would settle the generalization claim: if novel attacks outside the taxonomy succeed at rates near the undefended baselines, the benchmark is not representative; if the defended rates reproduce, it is.

Watch

Extended reading notes

Core claim

The central claim is that vulnerabilities arise at multiple lifecycle stages beyond execution, and that a lifecycle-aware evaluation can systematically expose them. Defining a skill as a tuple of metadata, workflow, permissions, tools, and provenance, the paper organizes attacks into repository admission, semantic retrieval, planner selection, runtime execution, and evolution. Its benchmark shows undefended retrieval succumbs to Sybil publication 93.20% of the time, planner fake-recommendation attacks succeed at 45.64%, and even with dynamic taint tracking and policy enforcement, 23.0% of runtime attacks still reach the privileged sink because LLM paraphrase destroys string-based taint. Defe

Load-bearing premise

The load-bearing premise is that the 327 filtered benign skills plus the synthetic attack transformations are representative of real skill-marketplace threats, which the paper itself concedes in Section 7 is a controlled evaluation environment rather than a production deployment.

Editorial extensions

If this is right

  • Repository admission needs LLM-based semantic validation, not just signatures and rules; a hybrid defense reduced the malicious admission rate from 52.9% to 7.9%.
  • Semantic retrieval is a major attack surface: Sybil publication put a malicious skill in the top five for 93.20% of queries, and post-retrieval filtering lowered this to 26.59% but not to zero.
  • Planner attacks are mostly metadata persuasion: fake recommendations succeeded 45.64% at baseline and dropped to 8.72% after sanitization, while prompt injection was surprisingly weak at 4.69%.
  • Runtime monitoring is a necessary backstop but not sufficient: it blocked 87.0% of policy violations yet allowed 23.0% of attacks because paraphrase defeats string-based taint tracking.
  • Treating skill updates as new admission events detects 92.5% of malicious updates but rejects 37.0% of benign ones, revealing a trade-off between evolution security and developer friction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same lifecycle lens should apply to MCP servers and plugin marketplaces, suggesting that LLM-ecosystem supply-chain defenses need semantic validation analogous to code signing.
  • Editorial inference: the 23% residual runtime attack success rate implies that taint tracking for LLM workflows needs semantic information-flow models, such as paraphrase-resistant embeddings, rather than string matching.
  • Editorial inference: because planner attacks exploit social-proof metadata, agent frameworks should treat all reputation signals as untrusted input and validate them cryptographically.
  • Editorial inference: a concrete extension would be to run the same attack families against different embedding models and planners to test whether the effectiveness ordering of these defenses transfers across components.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SkillSec-Eval, a lifecycle-aware framework and benchmark for security evaluation of reusable LLM agent skills. It defines a six-stage lifecycle (authoring, storage, retrieval, selection, execution, evolution) and a corresponding threat taxonomy, then instantiates five operational evaluation boundaries: repository admission, semantic retrieval, planner selection, runtime execution, and skill evolution. Using a repository of 327 benign skills derived from SkillMCP, the authors generate attacks at each boundary, evaluate a suite of defenses (hybrid admission control, retrieval validation, metadata validation, runtime taint tracking, and update revalidation), and report attack success rates and defense metrics in Tables 2–6. The central claim is that vulnerabilities arise at multiple lifecycle stages beyond execution, so security analysis of agent skills should be lifecycle-aware.

Significance. If the results hold, the paper makes a useful contribution by providing a shared vocabulary and evaluation scaffold for an emerging problem: secure reuse of agent skills. The taxonomy is reasonable, the modular separation of lifecycle boundaries is a genuine improvement over end-to-end agent benchmarks, the use of a real-world-derived skill corpus is a strength, and the appendix includes exact prompts for reproducibility. The qualitative conclusion—that repository admission, retrieval, planner selection, and evolution are each attack surfaces—is credible and aligns with trends in LLM-agent supply-chain security. However, the empirical support is not yet strong enough to support the specific quantitative claims, and one section-level inconsistency directly undermines the planner experiment, a key pillar of the cross-stage argument.

major comments (3)
  1. [§5.3 vs §6.3, Table 4] The planning configuration is internally inconsistent. §5.3 states 'Planner experiments employ Gemini 3.1 pro as the planning model,' while §6.3 states the baseline planner 'uses Gemini 1.5 Flash in a zero-shot configuration.' The defended condition in §6.3 does not specify its model. If the baseline is Gemini 1.5 Flash and the defended condition is Gemini 3.1 Pro, then the large reductions in Table 4 (e.g., Fake Recommendation 45.64%→8.72%, Misleading Description 27.27%→0.00%) are confounded by model capability and cannot be attributed to the proposed Metadata Validation Layer. If both conditions actually use the same model, then one of the two sections is inaccurate, which harms reproducibility. The authors must name the model used in each condition and, ideally, report both conditions under both models or otherwise show that the defense effect is not a model effect.
  2. [Tables 2–6 and §5.4] All reported metrics are single-point estimates with no sample sizes, confidence intervals, standard deviations, or significance tests. Table 3 is based on 440 queries, but Tables 2, 4, 5, and 6 do not state N. Attack generation itself is described as including 'LLM-assisted adversarial optimization' (§3.4) and automatic transformations (§5.2), which likely introduce stochasticity; without intervals, differences such as Semantic Camouflage Defended ASR 9.82% vs. Trigger Poisoning 17.50% cannot be interpreted. Additionally, the pipeline uses several free thresholds—semantic diversity filtering at 85% similarity, retrieval Top-50→Top-5, evolution behavioral consistency threshold—and no sensitivity analysis is provided for any of them. The cross-stage comparison at the core of the paper rests on these point estimates, so the empirical section should report per-query or per-instance variabil
  3. [§3.4, §4, §5.2, §7] The attacks and the defenses are instantiated from the same taxonomy: the defense components in §4.2–4.6 detect the specific transformations used to generate attacks in §5.2 (e.g., retrieval defenses target Sybil publication, metadata deception, and permission inflation; the admission validator checks exactly the consistency criteria used to define malicious skills). The paper itself acknowledges in §7 that this is 'a controlled evaluation environment rather than a production deployment.' The qualitative conclusion that multiple lifecycle stages are attack surfaces can survive this circularity, but the quantitative ASR and defense-success figures should be presented as within-taxonomy estimates, not as generalizable rates. A stronger version would evaluate against a held-out or independently sourced set of attacks, or at least ablate the defense rules against attack variants that are not
minor comments (6)
  1. [Abstract/Keywords] The keyword line reads 'Keywords: Yes, you must specify some keywords.' This is a placeholder that should be replaced with actual keywords.
  2. [Table 2 / §5.3] The experimental setup for repository admission mentions an 'undefended baseline' in addition to the rule-based and hybrid pipelines, but Table 2 has no undefended row. Either add the baseline (presumably MAR=100%) or state that it is omitted by construction.
  3. [§4.3] There is a typo: 'the planner validator performs three consistency checks., First,'—the period before 'First' is misplaced.
  4. [§3.4 / Figure 1 caption] 'as summarized in in Figure 1' repeats 'in'. Also 'T rust Boundary' contains an odd space in the figure/caption text that should be corrected.
  5. [References] The OWASP reference is typeset with a spurious space ('OW ASP' in two places); other references use inconsistent arXiv stylistic conventions (e.g., 'arXiv' vs. 'arXiv preprint').
  6. [§5.4 / Table 3] Table 3 labels the third metric 'Avg. Malicious'; the text defines it as 'average number of malicious skills presented to the planner.' Clarify in the column header (e.g., 'Avg. Malicious in Top-5') and, if the metric is a count, note its range (0–5).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: central claim rests on measured attack success rates; the planner-model mismatch is a confounding/reproducibility concern, not circularity.

full rationale

SkillSec-Eval's central claim is empirical rather than derivational. The paper builds a 327-skill repository from SkillMCP, generates attacks from a lifecycle taxonomy, and measures baseline and defended success rates at five boundaries. The headline results (e.g., Sybil 93.20% ASR, Fake Recommendation 45.64%, runtime baseline 100% Execution ASR) are measured outcomes, not quantities fitted from the model or implied by construction. The defense evaluations are controlled experiments: thresholds such as the 85% similarity cutoff, cosine behavioral bounds, and LLM-based sanitizers are not fitted to the reported success rates. The main caveat is external validity, which the paper itself concedes in Section 7 ('it represents a controlled evaluation environment rather than a production deployment'); this is a generalizability limitation, not circularity. The only self-citation (Badhe 2025, §2.1) is illustrative in related work and is not load-bearing for any contribution. The apparent inconsistency between §5.3 ('Planner experiments employ Gemini 3.1 pro') and §6.3 ('The baseline planner uses Gemini 1.5 Flash') is a serious reproducibility and potential confounding issue for the planner defense comparison, but it is a correctness risk, not a circular-derivation risk. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no claimed result reduces by definition to its own inputs. Therefore the paper receives a low circularity score despite the noted experimental concerns.

Assumptions & free parameters 3 free parameters · 5 assumptions · 4 invented entities

The paper rests mostly on domain assumptions about how real agent ecosystems behave; its numeric knobs are hand-set thresholds whose sensitivity is not explored. The named framework components are internally validated only.

free parameters (3)
  • Semantic diversity filtering similarity threshold = 0.85 (85%)
    Hand-set in §6.2 for near-duplicate removal; no sensitivity analysis, though Sybil defense outcomes (0.27 avg malicious in Top-5) depend on it.
  • Retrieval candidate pool size and final Top-k = Top-50 retrieved, Top-5 presented
    Configuration in §5.3; baseline compares direct Top-5 with a defense that uses Top-50 plus filtering, so the comparison is confounded by pool size.
  • Evolution behavioral consistency threshold = not reported
    The evolution defense uses an unreported cosine-similarity bound (§6.5); the MDR/FPR trade-off (92.5%/37%) is a function of this threshold.
assumptions (5)
  • domain assumption The underlying LLM, embedding model, runtime, and OS are correct and outside the attacker's control.
    Stated in §3.3 to isolate skill-lifecycle vulnerabilities; if the model itself is compromised, results may not transfer.
  • domain assumption The skill abstraction S=(M,W,P,T,V) and the four-component manifest faithfully model real agent skills.
    §3.1/§4.1; the taxonomy and framework are built on this representation.
  • domain assumption Planners select skills almost exclusively from natural-language metadata rather than executable workflow content.
    §3.2 and §6.3; this premise drives the planner-deception attack model and defense design.
  • domain assumption The 327 filtered SkillMCP skills and the synthetic transformations are representative of real-world skill-marketplace content and attacks.
    §5.1–5.2; the paper later concedes it is a controlled environment (§7).
  • domain assumption Vector retrieval with all-MiniLM-L6-v2 and cosine similarity represents production agent retrieval.
    §5.3; no justification that this encoder matches enterprise embedding stacks.
invented entities (4)
  • SkillSec-Eval
    purpose: Lifecycle-aware evaluation framework and benchmark
    Introduced in §4; evaluated only on the authors' own synthetic attack set, no external validation or code release.
  • LifecycleGuard
    purpose: Runtime monitor using dynamic taint tracking and policy enforcement
    Introduced in §5.3/§6.4; internal results only, no independent implementation.
  • Retrieval Validation Layer
    purpose: Post-retrieval diversity, metadata, and permission filtering
    Proposed in §4.3/§6.2; no ablation against external baselines.
  • Metadata Validation Layer
    purpose: Pre-planner sanitization
    Proposed in §4.4/§6.3; no independent evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation." pith.science (2026). https://pith.science/paper/ECWFIHNP

@misc{pith2026260713987,
  author       = {Pith},
  title        = {Pith review of: Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ECWFIHNP}},
  note         = {Machine review of arXiv:2607.13987}
}
read the original abstract

Reusable skills are becoming a fundamental building block of Large Language Model (LLM) agents, enabling capabilities to be packaged, shared, and reused across diverse applications. However, existing security research primarily focuses on prompt injection and runtime execution, leaving security risks throughout the broader skill lifecycle largely unexplored. In this paper, we present SkillSec-Eval, a lifecycle-aware framework for systematically evaluating the security of reusable agent skills. We first characterize the skill lifecycle and develop a threat taxonomy spanning repository admission, semantic retrieval, planner selection, execution, and skill evolution. We then instantiate this taxonomy in SkillSec-Eval and conduct a comprehensive empirical evaluation using a repository of 327 real-world skills. Our study demonstrates that vulnerabilities arise at multiple lifecycle stages beyond execution, highlighting the need for lifecycle-aware security analysis of reusable agent skills.

Figures

Figures reproduced from arXiv: 2607.13987 by the authors.

Figure 1
Figure 1. The lifecycle of reusable agent skills, illustrating the sequence of trust boundaries and the corresponding threat taxonomy evaluated in SkillSec-Eval. 3.4. Lifecycle Threat Taxonomy To enable systematic evaluation, we organize attacks according to the lifecycle stage whose trust assumptions are violated, as summarized in in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ElasticBack: Stealthy Conditional Backdoor in LLM-Agent Skills via Coupled Trigger-Rule Optimization

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A single poisoned agent skill can carry a weight-free conditional backdoor, with the rule in the skill file and the trigger in the query, so the payload fires only when both appear.

Reference graph

Works this paper leans on

33 extracted references · 7 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Guyon and A

    I. Guyon and A. Elisseeff. An Introduction to Variable and Feature Selection. JMLR

  2. [2]

    , title =

    He, Feng and Zhu, Tianqing and Ye, Dayong and Liu, Bo and Zhou, Wanlei and Yu, Philip S. , title =. ACM Comput. Surv. , month = dec, articleno =. 2025 , issue_date =. doi:10.1145/3773080 , abstract =

    doi:10.1145/3773080 2025

  3. [3]

    SkillMCP: Agent Skills Marketplace , year =

  4. [4]

    Proceedings of the 33rd USENIX Security Symposium (USENIX Security) , year =

    Liu, Yupei and others , title =. Proceedings of the 33rd USENIX Security Symposium (USENIX Security) , year =

  5. [5]

    Proceedings of the 34th USENIX Security Symposium (USENIX Security) , year =

    Chen, Sizhe and others , title =. Proceedings of the 34th USENIX Security Symposium (USENIX Security) , year =

  6. [6]

    Proceedings of the Network and Distributed System Security Symposium (NDSS) , year =

    Li, Zichuan and Cui, Jian and Liao, Xiaojing and Xing, Luyi , title =. Proceedings of the Network and Distributed System Security Symposium (NDSS) , year =

  7. [7]

    arXiv preprint arXiv:2604.03081 , year =

    Qu, Yubin and Liu, Yi and Geng, Tongcheng and Deng, Gelei and Li, Yuekang and Zhang, Leo Yu and Zhang, Ying and Ma, Lei , title =. arXiv preprint arXiv:2604.03081 , year =

    arXiv

  8. [8]

    arXiv preprint arXiv:2605.14460 , year =

    Liu, Xinyu and Zhao, Yukai and Hu, Xing and Xia, Xin , title =. arXiv preprint arXiv:2605.14460 , year =

    arXiv

Show all 33 references
  1. [9]

    Proceedings of the 17th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA) , year =

    Ohm, Marc and Plate, Henrik and Sykosch, Arnold and Meier, Michael , title =. Proceedings of the 17th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA) , year =

  2. [10]

    Proceedings of the 2025 Conference on Applied Machine Learning for Information Security , pages =

    ScamAgents: How AI Agents Can Simulate Human-Level Scam Calls , author =. Proceedings of the 2025 Conference on Applied Machine Learning for Information Security , pages =. 2025 , editor =

    2025

  3. [11]

    SoK: Taxonomy of Attacks on Open-Source Software Supply Chains , year=

    Ladisa, Piergiorgio and Plate, Henrik and Martinez, Matias and Barais, Olivier , booktitle=. SoK: Taxonomy of Attacks on Open-Source Software Supply Chains , year=

  4. [12]

    and Wan, Sheng and Zhang, Sheng and Lee, Wang-Chien and Sima, Xiaoyuan and Zhao, Siyuan and Wang, Chi , title =

    Wu, Qingyun and Bansal, Gagan and Zhang, Jieyu and Wu, Yiran and Li, Beibin and Zhu, Erkang and White, Li and White, Ryen W. and Wan, Sheng and Zhang, Sheng and Lee, Wang-Chien and Sima, Xiaoyuan and Zhao, Siyuan and Wang, Chi , title =. Proceedings of the Conference on Langua...

  5. [13]

    and Cao, Yuan , title =

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian ... and Cao, Yuan , title =. Proceedings of the International Conference on Learning Representations (ICLR) , year =

  6. [14]

    Toolformer: Language Models Can Teach Themselves to Use Tools , booktitle =

    Schick, Timo and Dwivedi-Yu, Jane and Dess. Toolformer: Language Models Can Teach Themselves to Use Tools , booktitle =. 2023 , url =

    2023

  7. [15]

    LangGraph: Building Stateful, Multi-Actor Applications with LLMs , howpublished =

  8. [16]

    CrewAI: Orchestrating Autonomous AI Agents , howpublished =

  9. [17]

    OpenAI Agent SDK: Build Agentic Applications , howpublished =

  10. [18]

    34th USENIX Security Symposium (USENIX Security 25) , year =

    Daniel Ayzenshteyn and Roy Weiss and Yisroel Mirsky , title =. 34th USENIX Security Symposium (USENIX Security 25) , year =

  11. [19]

    Introducing the Model Context Protocol , howpublished =

  12. [20]

    AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for

    Edoardo Debenedetti and Jie Zhang and Mislav Balunovic and Luca Beurer-Kellner and Marc Fischer and Florian Tram. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for. The Thirty-eight Conference on Neural Information Processing Systems Datase...

  13. [21]

    AgentBound: Securing Execution Boundaries of AI Agents , journal =

    B. AgentBound: Securing Execution Boundaries of AI Agents , journal =. 2026 , url =

    2026

  14. [22]

    arXiv preprint arXiv:2604.05969 , year =

    Acharya, Nirajan and Gupta, Gaurav Kumar , title =. arXiv preprint arXiv:2604.05969 , year =

    arXiv

  15. [23]

    Proceedings of the ACM CAIS Workshop on Agent Skills (AgentSkills) , year =

    Xu, Renjun and Yan, Yang , title =. Proceedings of the ACM CAIS Workshop on Agent Skills (AgentSkills) , year =

  16. [24]

    A Practical Guide for Securely Using Third-Party MCP Servers , howpublished =

  17. [25]

    and Wan, Sheng and Zhang, Sheng and Lee, Wang-Chien and Sima, Xiaoyuan and Zhao, Siyuan and Wang, Chi , title =

    Wu, Qingyun and Bansal, Gagan and Zhang, Jieyu and Wu, Yiran and Li, Beibin and Zhu, Erkang White, Li and White, Ryen W. and Wan, Sheng and Zhang, Sheng and Lee, Wang-Chien and Sima, Xiaoyuan and Zhao, Siyuan and Wang, Chi , title =. Proceedings of the Conference on Language M...

  18. [26]

    Preprints.org , year =

    Shen, Jinhao and Yi, Huahui and Hu, Wentao and Jiang, Yiyang and Zhang, Wengyu and Wei, Xiao-Yong and Li, Qing , title =. Preprints.org , year =

  19. [27]

    and others , title =

    Li, X. and others , title =. arXiv preprint arXiv:2602.12670 , year =

    arXiv

  20. [28]

    and Beurer-Kellner, L

    Schmotz, D. and Beurer-Kellner, L. and Abdelnabi, S. and Andriushchenko, M. , title =. arXiv preprint arXiv:2602.20156 , year =

    arXiv

  21. [29]

    and Wang, W

    Liu, Y. and Wang, W. and Feng, R. and Zhang, Y. and Xu, G. and Deng, G. and Li, Y. and Zhang, L. , title =. arXiv preprint arXiv:2601.10338 , year =

    arXiv

  22. [30]

    and Chen, Z

    Liu, Y. and Chen, Z. and Zhang, Y. and Deng, G. and Li, Y. and Ning, J. and Zhang, Y. and Zhang, L. Y. , title =. arXiv preprint arXiv:2602.06547 , year =

    arXiv

  23. [31]

    and others , title =

    Liang, Y. and others , title =. arXiv preprint arXiv:2603.04448 , year =

  24. [32]

    Bhardwaj, V. P. , title =. arXiv preprint arXiv:2603.00195 , year =

  25. [33]

    and Chen, Z

    Guo, Z. and Chen, Z. Nie, X. and Lin, J. and Zhou, Y. and Zhang, W. , title =. arXiv preprint arXiv:2603.21019 , year =

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.