Cryptography and Security · arXiv:2608.08468 v1 · submitted 9 Aug 2026 · 4 references
SkillsMetric: Mapping the Detection Boundary of Static Analysis for Malicious Agent Skills
The paper says
“Agent Skills—structured packages of instructions and scripts that augment LLM-based agents—are rapidly proliferating, yet their security properties remain under-explored.”
more of the abstract
We present \textscSkillsMetric, a five-stage static analysis framework that scores skill packages along pattern density, statistical anomaly, dataflow taint, import anomaly, and capability mismatch dimensions. We construct an adversarial evaluation dataset of 2,266 skills spanning 16 attack types across code-level, system-level, and semantic-level threats, and evaluate on the full SkillMD-138K corpus. Our framework achieves an AUC of 0.93 and 5-fold cross-validated F1 of 73.4%±0.5%, with strong detection of data exfiltration (93%) and steganographic payloads (93%). Crucially, we identify fundamental blind spots: host destruction attacks using common shell commands evade all five stages (0% detection), and prompt injection via natural-language manipulation achieves only 42% detection. These findings establish that static analysis alone is insufficient for skill security, motivating defense-in-depth architectures that combine fast static pre-screening with semantic review.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Aug 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Useful first map of what static analysis can and cannot catch in agent skills, but the headline 0% blind spot rests on one sample and the boundary claims outrun the evidence; the direction is plausible, the precision is not. This line is wrong
5 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 4 references say.
The paper says, at Table 1 / §4.2
The review did not quote this passage.
Pith says: Objection 1 of 5 · major
The headline 0% detection for host destruction is based on a single sample (N=1 in Table 1). With zero detections out of one trial, the two-sided 95% binomial confidence interval spans approximately [0%, 97.5%], so the data cannot distinguish a genuinely undetectable attack category from a single sample that happens to fall outside SkillsMetric's pattern set.
more of objection 1
The paper's own limitation (4) acknowledges small per-type counts, but the abstract and §7 nevertheless state that host destruction is a 'fundamental blind spot'; that claim goes beyond what an N=1 result can support.
The paper says, at §4.2 / Figure 1
The review did not quote this passage.
Pith says: Objection 2 of 5 · major
The text states that host destruction triggers no stage at all, but Figure 1 appears to report mean stage scores of 101 for S1 and 9 for S4 for this category, and the figure's values run up to 287, suggesting these are not the normalized stage scores \hat{s}_i used in Eq.
more of objection 2
(1). This is an internal inconsistency: if S1 actually fires with a mean near 101, then the 0% detection is a threshold/weight artifact rather than the absence of signal, and the boundary claim must be restated in terms of score thresholds. Please clarify the scaling of the values in Figure 1 and reconcile them with the 'no stage fires' statement.
The paper says, at §3 Eq. (1) / Table 2
The review did not quote this passage.
Pith says: Objection 3 of 5 · major
The weights w=(0.15,0.20,0.30,0.15,0.10), the interaction-bonus coefficients, and the detection threshold tau=25 are all set with reference to the same benign and adversarial corpora used for the reported AUC (0.93) and F1 (73.4%). The 5-fold cross-validation in Table 2 therefore does not provide an unbiased estimate for a detector whose weights and threshold were manually fitted on the full dataset; the risk of in-sample optimism in the quantitative boundary claims is real.
more of objection 3
A held-out validation set, nested cross-validation, or a sensitivity analysis over the manually chosen weights is needed before the reported detection rates can be interpreted as predictive of performance on new skills.
The paper says, at §6 / §7
The review did not quote this passage.
Pith says: Objection 4 of 5 · major
No external static-analysis baseline is run: the paper mentions skill-security-analyzer but does not execute it on the 266 malicious samples, and no stage ablation is reported. Without a baseline or ablation, the claim that the identified blind spots are properties of 'static analysis' rather than of SkillsMetric's particular feature set is unsupported.
more of objection 4
For example, a path-aware rule targeting deletion of home-rooted or absolute paths outside build directories could plausibly detect much of the host-destruction category at acceptable false-positive cost; the paper does not test such a rule against the 2,000 benign skills or the 138K corpus.
The paper says, at §4.1 / §7
The review did not quote this passage.
Pith says: Objection 5 of 5 · major
The adversarial dataset is constructed by injecting attack payloads into benign host skills, and the paper acknowledges that this cannot capture the obfuscation depth or contextual mimicry of a determined real-world attacker. Yet the abstract and §7 generalize from these synthetic samples to 'static analysis alone is insufficient for skill security.'
more of objection 5
The synthetic-construction limitation should be carried into the wording of the central claim, or the evaluation should be supplemented with real-world malicious skills or a red-team exercise; otherwise the boundary is a property of the synthetic corpus rather than of the threat landscape.
5 minor objections
The paper says, at §3
Pith says: minor objection 1 of 5
The 50+ regular expressions in Stage 1 are never enumerated; please provide the pattern list or a code release so that the detector's feature set is reproducible and auditable.
The paper says, at Table 2
Pith says: minor objection 2 of 5
Please clarify why the single-run precision (63.4%) is so much lower than the 5-fold precision (89.4%) when the recall values are nearly identical; reporting a precision-recall curve would help readers interpret the operating point.
show 3 more
The paper says, at §4.4
Pith says: minor objection 3 of 5
The cross-model LLM comparison is explicitly preliminary, but the '7× canary-leakage gap' in the text and Table 3 should be accompanied by confidence intervals or a stronger caveat that the two models were run on partially different sample sets.
The paper says, at References
Pith says: minor objection 4 of 5
Reference [4] is typeset as 'DIMV A' and should be 'DIMVA'; please correct the venue name.
The paper says, at Figure 3
Pith says: minor objection 5 of 5
The x-axis of the score distribution ends at 80, so the CRITICAL tail above 70 is only partially visible; please use an axis break or a log-scaled x-axis to show the full distribution.
What would settle it
Build an expanded host-destruction set of 20-50 samples authored independently of SkillsMetric's pattern set (varied targets: home directory, dotfiles, /etc, project directories; varied primitives: rm -rf, shutil.rmtree, os.chmod, find -delete; some plausibly disguised as cleanup) and score them with SkillsMetric, a naive keyword matcher, the external skill-security-analyzer tool named in §6, and a simple path-aware rule (home-rooted or absolute-path deletions), all at τ=25 against the same 2,000 benign skills. If any detector exceeds roughly 50% detection at under 5% benign flag rate, the headline 0% is scanner-specific and the conclusion should be weakened from 'static analysis cannot detect host destruction' to 'SkillsMetric's current pattern set misses host destruction.' If all detectors stay near 0%, the fundamental-blind-spot claim is supported.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Aug 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 5 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper is worth reading. It is the first serious attempt I know to map what static analysis can and cannot catch for LLM agent skills: a 16-type taxonomy, a 266-sample adversarial dataset (prior work used 11), a five-stage scorer, a scan of the 138K SkillMD corpus, and a sandboxed comparison of two LLM models. The central qualitative finding is persuasive: syntactic analysis alone cannot distinguish a destructive shutil.rmtree call from a cleanup script, and prompt injection disguised as natural language carries almost no structural signal. That static analysis alone is insufficient for skill security probably survives, even if the exact numbers are fragile.
Where I push back is the precision of the claimed boundary. Host destruction at 0% detection rests on N=1. With one sample and zero detections, the 95% confidence interval reaches roughly 97.5%; the data cannot distinguish a fundamental blind spot from one sample outside this scanner's pattern set. Prompt injection at 42% is N=12, with an interval roughly 15-72%. The stage weights in Eq. 1 are manually set on the same adversarial corpus used for evaluation, which is a real in-sample-fitting risk for the positive metrics like AUC 0.93. The stress-test note is right that this cannot explain the 0%, because an additive score cannot lift a sample with no firing stage. The bigger problem is generalizing from one scanner with unenumerated regexes to 'static analysis' in general. No baseline is run against skill-security-analyzer or any other detector. The authors disclose most of this in the limitations section, which is to their credit, but the abstract and conclusion state the boundary as established.
Also, the claim that destructive commands are 'too common to flag without massive false positives' is asserted, not tested. No candidate rule is evaluated against the benign corpus. A path-aware rule might catch part of this category at acceptable false-positive cost, so calling the blind spot fundamental is premature.
Who this is for: agent platform maintainers, AI security researchers, and tool builders. The taxonomy and dataset, once released, are citable. I would want revision before accepting: release code and data, enumerate the regex patterns, expand per-category samples, run at least one baseline, and rephrase the boundary claims as scanner-specific findings.
Recommendation: send it to peer review, but conditional on revision and more careful claims.
The strongest objection
The central claim's load-bearing content is the boundary it draws: host destruction at 0% and prompt injection at 42% (Table 1), generalized in §7 to 'static analysis alone is insufficient for skill security.' The 42% rests on N=12; the headline 0% rests on N=1. With one sample and zero detections, the two-sided 95% binomial interval spans roughly [0%, 97.5%], so the data cannot distinguish a genuinely undetectable category from one sample that falls outside SkillsMetric's pattern set. The paper never enumerates the 50+ Stage-1 regexes, never ablates stages, and names skill-security-analyzer in §6 without running it. The asserted mechanism for the blind spot — destructive commands like rm are 'too common to flag without massive false positives' — is untested: no candidate host-destruction rule is measured against the 2,000 benign skills or the 138,133-skill corpus. A path-aware rule (flag deletion of home-rooted or absolute paths outside build directories, chmod on dotfiles) could plausibly catch much of this category at acceptable false-positive cost, which would make the 0% a feature-set artifact of this scanner rather than a property of static analysis. The manual weights in Eq. 1 are a genuine in-sample-fitting risk for the positive metrics (AUC 0.93; per-type rates in Table 1), but they cannot explain the 0%: an additive weighted sum cannot lift a sample whose every stage is silent, so the blind spot itself is robust to weight tuning. The load-bearing fragility is instead the single sample plus the unvalidated scanner-to-category generalization. The qualitative direction is still plausible — 42% prompt-injection detection at N=12 (95% CI roughly [15%, 72%]) suffices to show static analysis alone is insufficient — but the quantitative boundary is not established at the precision claimed.
I read the paper's central claim as a boundary mapping: attacks leaving structural artifacts are detectable; host destruction and prompt injection evade the five stages, which establishes that static analysis alone cannot secure agent skills. For the boundary numbers to be reliable, two conditions must hold: the adversarial samples must represent their attack categories, and the evaluated pipeline must stand in for 'static analysis.' The host-destruction row fails the first condition at the headline point — N=1 gives a 95% interval of roughly [0%, 97.5%] — and no baseline or ablation supports the second. The paper's own §5 limitations disclose small per-category counts and missing head-to-head baselines, which correctly caps the verdict at CONDITIONAL rather than ACCEPT or REJECT; the disclosed limitation is also why I do not escalate to REJECT. I depart from the reader's weakest_assumption in one respect: the in-sample weights concern is real but secondary. Because the composite score is an additive weighted sum (Eq. 1), a sample whose five stages are all silent cannot be pushed across τ=25 by any weight vector; weight fitting inflates the positive metrics (AUC 0.93, per-type detection rates) but cannot manufacture or explain the 0% blind spot. The vulnerability that matters is the N=1 sample and the untested assertion that destructive commands are too common to flag — I could not find any measurement of the false-positive cost of a host-destruction rule in §3 or §4. Independent support: the population scan is honestly scoped (SKILL.md-only, flagged as a lower bound in §4.3/§5), the 5-fold CV adds some robustness to the overall metrics, and the qualitative insufficiency conclusion is directionally robust even if every contested number shifted, because 42% detection at N=12 (CI roughly [15%, 72%]) already leaves a large evasion fraction. My concrete test — an independent, multi-detector evaluation on an expanded host-destruction set — settles whether the 0% is a property of the attack category or an artifact of this scanner's feature set. If the test shows a path-aware baseline detects host destruction, the paper's distinctive finding is overstated; if all detectors fail, the finding stands. This is consistent with the reader's CONDITIONAL verdict, so verdict_should_be is UNCHANGED, with agreement_with_reader = partial.
The referee's report
If the quantitative boundary claims held, this would be a useful contribution: the paper provides a systematic 16-type attack taxonomy, a five-stage scoring framework, and the largest adversarial skill dataset to date, while also reporting a population-level scan of the SkillMD-138K corpus. The explicit attempt to characterize where static analysis fails, rather than only reporting aggregate accuracy, is valuable for practitioners designing skill-screening pipelines. The authors also state several limitations honestly, including synthetic construction, partial population-scan coverage, manually assigned weights, and small per-category counts. However, the central boundary claim currently rests on very thin evidence, and one apparent internal inconsistency in the reported stage scores needs to be resolved before the headline conclusions can be accepted.
Pith's reading of the paper (a machine's summary, not the authors')
The paper sets out to map what static analysis can and cannot catch in malicious Agent Skills—packages of instructions and scripts that give LLM agents new capabilities. It builds a five-stage scoring pipeline that looks for dangerous code patterns, statistical anomalies, dataflow taint chains, unusual imports, and mismatches between a skill's claimed and actual behavior. On an evaluation set with 266 malicious samples across 16 attack types, the pipeline catches code-level attacks such as data exfiltration and steganography at about 93%, but finds two blind spots: host-destruction commands like file deletion score 0% detection, and natural-language prompt injection only 42%. The paper's central claim is that static analysis alone is insufficient for skill security, so platforms should layer semantic review and sandboxing on top of fast static pre-screening.
What carries the argument. The carrying object is the five-stage composite risk score S, a weighted sum of normalized stage scores with interaction bonuses. Stage 3 (dataflow taint: source–transform–sink chains) carries the highest weight because complete chains are rare in benign code; Stage 5 (capability mismatch) carries the lowest because metadata declarations are noisy. The score's additive form matters: a multiplicative aggregator would zero out attacks that leave signal in only one stage, and the paper deliberately keeps weights manually assigned rather than learned so that reviewers can read off each stage's contribution.
Core claim. The central claim is that the detection boundary of static analysis for agent skills is set by whether an attack leaves structural artifacts in code. Attacks that need network exfiltration, obfuscation, or unusual imports produce strong signals and are reliably caught; attacks that use semantically ordinary code for abnormal purposes are invisible to all five stages. The paper demonstrates this with two headline numbers: host destruction using common shell calls such as shutil.rmtree and os.chmod is detected 0% of the time, and prompt injection phrased as ordinary natural language only 42%. Because these attacks exploit the skill runner's trust elevation—SKILL.md content is injected with user-level authority—the paper argues that no syntactic screening can close the gap, motivating defense-in-depth.
The premise it rests on. The reported blind-spot rates rest on the assumption that the synthetic malicious samples—including a single host-destruction case—stand in for real-world attacks, and that the manually chosen stage weights, set after inspecting per-stage signals on those same samples, do not amount to fitting the test set.
If the paper is right (Pith's inferences, not the authors')
Platforms that rely on static pre-screening alone will miss file-destroying skills and many prompt-injection skills, so they need an LLM-based semantic review tier for skills above a low threshold.
A population scan of 138K real-world skills flags about 1.75% at the detection threshold, and because companion scripts are not scanned, this flag rate is a lower bound on risky content.
The additive score composition lets operators shift the threshold to balance false positives: raising the auto-block tier to the HIGH boundary reduces the need for LLM review to roughly 2% of skills, cutting cost versus LLM-only review.
Skill runners should treat SKILL.md content as untrusted user-level input and add sandboxing, since static analysis cannot distinguish a malicious rmtree call from a legitimate cleanup script.
If the 0% host-destruction result generalizes, detection of intent-level attacks will require behavioral containment—for example, asking the LLM to justify destructive operations before execution—rather than better pattern matching.
The 42% prompt-injection detection suggests a testable extension: an LLM-based classifier scoring natural-language instruction semantics could close much of the gap, and the paper's tiered architecture already reserves a slot for exactly that.
The lower-bound flag rate implies that a companion-script scan of the flagged skills would likely raise the observed prevalence of risky content, making the ecosystem less healthy than the 82.4% SAFE fraction suggests.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agent Skills—structured packages of instructions and scripts that augment LLM-based agents—are rapidly proliferating, yet their security properties remain under-explored. We present \textscSkillsMetric, a five-stage static analysis framework that scores skill packages along pattern density, statistical anomaly, dataflow taint, import anomaly, and capability mismatch dimensions. We construct an adversarial evaluation dataset of 2,266 skills spanning 16 attack types across code-level, system-level, and semantic-level threats, and evaluate on the full SkillMD-138K corpus. Our framework achieves an AUC of 0.93 and 5-fold cross-validated F1 of 73.4%±0.5%, with strong detection of data exfiltration (93%) and steganographic payloads (93%). Crucially, we identify fundamental blind spots: host destruction attacks using common shell commands evade all five stages (0% detection), and prompt injection via natural-language manipulation achieves only 42% detection. These findings establish that static analysis alone is insufficient for skill security, motivating defense-in-depth architectures that combine fast static pre-screening with semantic review.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“These findings establish that static analysis alone is insufficient for skill security, motivating defense-in-depth architectures that combine fast static pre-screening with semantic review.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.98 · human participants 0.04 · controlled comparison 0.10 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.10 · mentions code or data availability 0.05
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agent Skills---structured packages of instructions and scripts that augment LLM-based agents---are rapidly proliferating, yet their security properties remain under-explored. | background 0.78 | 0.05 | 0.02 | asserted 1.00 |
| We present \textsc{SkillsMetric}, a five-stage static analysis framework that scores skill packages along pattern density, statistical anomaly, dataflow taint, import anomaly, and capability mismatch dimensions. | method 0.62 | 0.92 | 0.04 | asserted 0.99 |
| We construct an adversarial evaluation dataset of 2{,}266 skills spanning 16~attack types across code-level, system-level, and semantic-level threats, and evaluate on the full SkillMD-138K corpus. | method 0.86 | 0.96 | 0.03 | asserted 0.99 |
| Our framework achieves an AUC of 0.93 and 5-fold cross-validated F1 of 73.4\%$\pm$0.5\%, with strong detection of data exfiltration (93\%) and steganographic payloads (93\%). | result 1.00 | 0.97 | 0.08 | asserted 1.00 |
| Crucially, we identify fundamental blind spots: \emph{host destruction} attacks using common shell commands evade all five stages (0\% detection), and \emph{prompt injection} via natural-language manipulation achieves only 42\% detection. | result 0.82 | 0.96 | 0.24 | asserted 1.00 |
| These findings establish that static analysis alone is insufficient for skill security, motivating defense-in-depth architectures that combine fast static pre-screening with semantic review. | interpretation 0.92 | 0.48 | 0.89 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,214 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
Showing 2 of the paper's 3 figures. Figures are the authors', reproduced from the paper.
References (4)
- [1] Z. Faye et al. SkillMD-138K: A Large-Scale Dataset of Agent Skills. HuggingFace Datasets, 2026.https://huggingface.co/datasets/FayeZC/ SkillMD-138K on Pith
- [2] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. InProc. AISec, 2023 on Pith
- [3] S. Perez, F. Ribeiro, et al. Ignore This Title and HackAPrompt: Exposing Systemic Weaknesses of LLMs through a Global Scale Prompt Hacking Competition. InProc. EMNLP, 2023 on Pith
- [4] M. Ohm, H. Plate, A. Syber, and M. Maier. Backstabber’s Knife Collec- tion: A Review of Open Source Software Supply Chain Attacks. InProc. DIMV A, 2020. 6 on Pith
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment