Cryptography and Security · arXiv:2609.01487 v4 · submitted 1 Sep 2026
Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents
The paper says
“Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable channel for steering future actions.”
more of the abstract
Such skills may leak secrets, corrupt code, bypass approvals, or stage data for exfiltration only after a concrete user task and workspace state make the unsafe action appear useful. This makes pre-install vetting insufficient and calls for runtime, task-conditioned protection. We propose Defense-as-Skill, a defense paradigm that implements the runtime guard itself as an installable, inspectable, and editable skill. Our guard, SkillSonar, runs alongside untrusted task skills and checks sensitive actions against the user's task boundary, routing each action to an allow, replan, or confirmation decision without modifying the underlying agent runtime. To study this setting, we construct SCOPE-R, a task-conditioned dataset covering 6 risk families and 21 sub-categories, with 206 attack-confirmed malicious instances and 43 benign tasks. We then improve SkillSonar on the SCOPE-R training subset using runtime guard-skill evolution, a Monte-Carlo Tree Search procedure that evolves the on-disk guard skill from feedback on the rollouts. Across Claude Code and OpenClaw, the evolved guard substantially reduces attack success while maintaining a favorable safety-utility trade-off; on repeated GLM-5 runs, ID ASR drops from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115, with gains transferring across victim models, held-out risk families, and external benchmarks. These results show that runtime guards can be skill-native, transparent, and evolvable policy artifacts for skill-augmented agents.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 8 Sep 2026 · of version 1 of the paper; version 4 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Competent empirical systems paper on an interesting new idea—runtime safety as an evolvable skill—but its central measured gains collapse without a soft invocation instruction, and the evidence base is still small and unshared. This line is wrong
4 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 4.2 and Figure 4 (ablation)
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
The paper's key safety gains are conditional on an explicit instruction that requires the agent to call SkillSonar before any action (Section 4.2, Appendix D.1). The ablation in Figure 4(a) shows that without this instruction, ID ASR rises from 0.104 to 0.400 and OOD ASR from 0.109 to 0.582, both near the no-guard baseline.
more of objection 1
This is a collapse, not a minor degradation. The paper treats the explicit invocation as part of the intended deployment configuration, but provides no evidence that such an instruction can be reliably injected in realistic, heterogeneous deployment settings (e.g., where the user supplies the task prompt, or the system prompt is fixed by a platform). The guard's protection is therefore not an emergent property of the skill mechanism or evolved policy; it is a prompt-enforced behavior that fails when the prompt is absent or overridden. The paper's own ablation proves this. To support the central claim of a deployable runtime guard, the authors must either demonstrate an alternative invocation mechanism (e.g., system hook, forced skill dependency, automatic retrieval) that does not rely on a discretionary instruction, or provide evidence that the explicit instruction can be reliably injected in realistic heterogeneous settings with the same effectiveness.
The paper says, at Section 5.1, SCOPE-R dataset size
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The dataset contains only 206 attack-confirmed malicious instances and 43 benign tasks, split further into train/ID/OOD. The main repeated-run results (Table 2) are on a small test set (e.g., 52 ID malicious instances). While the paper demonstrates cross-family and cross-benchmark generalization, the statistical power for fine-grained family-level conclusions (Table 10) is low: some families have as few as a handful of instances.
more of objection 2
The authors should report standard errors or confidence intervals for the family-level breakdown, and discuss the limitations of the dataset size for diagnosing specific failure modes.
The paper says, at Section 4.3, Equation (1) – optimization objective
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The scalar reward used in MCTS (Eq. 1) uses a small epsilon to keep safety as the dominant objective. The exact choice of epsilon and lambda weights is not justified; a sensitivity analysis over these hyperparameters is missing. Since the search may be sensitive to the relative weight of e.g., confirmation count, the reader cannot assess whether the reported trade-off is robust or an artifact of the particular weight setting.
The paper says, at Section 5.2, Table 2 – AcceptEdits comparison
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
AcceptEdits achieves lower OOD ASR than SkillSonar on GLM-5 (0.073 vs. 0.115) and equal task utility (0.715). The paper argues SkillSonar provides a more favorable safety-utility trade-off, but this is not uniformly true. The claim of favorable trade-off should be nuanced, especially given that AcceptEdits is a simple, non-learned platform feature that does not require explicit invocation instructions. The paper should clarify under what conditions SkillSonar is preferable to simpler permission gating.
5 minor objections
The paper says, at Section 4.3 heading
Pith says: minor objection 1 of 5
The heading of Section 4.3 contains a typo: 'alrorithm' should be 'algorithm'.
The paper says, at Table 3 caption
Pith says: minor objection 2 of 5
Table 3 and Table 4 both report 'additional victim models' and 'additional runtime environment' respectively, but the naming is confusing. Clarify which table corresponds to which harness (Claude Code vs OpenClaw) and why the open-claw results are presented separately.
show 3 more
The paper says, at Figure 5
Pith says: minor objection 3 of 5
The x-axis and y-axis labels are generic ('Score'). The figure caption explains the axes, but the figure itself would be more readable if the axes were labeled explicitly as 'ASR' and 'Task Utility'.
The paper says, at Appendix A, Table 7
Pith says: minor objection 4 of 5
The taxonomy is clear, but the table would benefit from indicating which families are held out as OOD (C and P) directly in the table caption or within the table.
The paper says, at General
Pith says: minor objection 5 of 5
Some citations in the reference list appear to be from the same group or preprint series; ensure that the paper appropriately distinguishes peer-reviewed publications from non-archival sources.
What would settle it
On 20 randomly sampled OOD test instances (10 from each held-out family), replace the explicit invocation instruction with a system-level rule embedded in the base system prompt (e.g., 'Always consult skill-sonar before any action'). Do not alter the task prompt. Measure ASR over 3 repeated runs per instance. If the resulting mean ASR exceeds 0.25, the guard's effectiveness depends on the specific invocking prompt placement and is not robust to naturally occurring prompt structures.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 8 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
First, the good news. This paper identifies a real gap: skill-augmented agents lack a deployable, inspectable runtime defense that lives inside the same skill mechanism. Packaging the guard as a skill and evolving it via MCTS on concrete rollouts is a genuinely new design point. The authors do several things right: repeated runs with standard deviations on the primary model, human-audited judges, ablations that isolate the effect of the skill representation versus policy content, and a candid limitations section. The cross-model and cross-benchmark transfer experiments, while preliminary, suggest the approach generalizes beyond its training configuration.
Now the soft spots, which are real but not fatal for a systems paper at this stage. The decisive issue is the explicit invocation requirement. The paper’s own ablation shows that without the instruction, ASR jumps from 0.104 to 0.400 (ID) and 0.115 to 0.582 (OOD)—essentially back to no-guard levels. The authors argue this is part of the intended deployment configuration, but they provide zero evidence that such an instruction can be reliably injected in realistic settings where the system prompt is fixed or the user controls the task prompt. Until that dependency is demonstrated with a different mechanism (e.g., a system hook or forced skill dependency), the claim of a deployable runtime guard is only half-supported.
Other concerns: the dataset has 206 malicious instances, all self-constructed. Some headline transfer numbers are point estimates without error bars. No code or data is released at submission. The MCTS budget (10 iterations) is small enough that the evolved guard may not be close to optimal. These are all addressable, and the paper’s honest limitations section acknowledges several of them.
Who gets value from this paper: researchers working on agent safety, particularly those who think about deployable, editable defenses rather than external classifiers. It would be a fine start-of-area reading group paper.
Recommendation: conditional acceptance. The authors should release the dataset and code, add repeated runs and error bars to cross-model tables, and either demonstrate a more robust invocation mechanism or clearly bound the deployment scenarios where explicit instruction is feasible. The empirical core is plausible and worth referee time.
The strongest objection
The paper's central measured gains (ID ASR 0.104, OOD ASR 0.115) are conditional on prepending an explicit instruction requiring the agent to call SkillSonar before any action (Section 4.2, Appendix D.1). The ablation shows that without this instruction, ID ASR rises to 0.400 and OOD ASR to 0.582, both near the no-guard baseline. This is not a minor degradation - it is a collapse. The paper treats the explicit invocation as part of the 'intended deployment configuration', but it provides no evidence that such an instruction can be reliably injected in realistic, heterogeneous deployment settings (e.g., where the user provides the task prompt, or where the system prompt is fixed by the platform). The guard's protection is therefore not an emergent property of the skill mechanism or the evolved policy; it is a prompt-enforced behavior that fails when the prompt is absent or overridden. The paper's own ablation proves this. Unless the authors demonstrate an alternative invocation mechanism (e.g., system hook, forced skill dependency, automatic retrieval) that does not rely on a discretionary instruction, the central claim of a deployable runtime guard is not supported by the reported measurements.
The reader's weakest assumption correctly identifies the explicit invocation instruction as load-bearing. The paper's own ablation confirms that without it, the guard provides negligible benefit. My stress test reinforces this point with a call for a specific empirical check: whether alternative placements or rewordings of the instruction preserve the low ASR. This does not change the reader's verdict because the reader already based the CONDITIONAL rating partly on this assumption. The paper's transparency about the deployment setting is commendable, but the evidence does not yet demonstrate that the instruction can be reliably injected outside the authors' controlled setup. The central claim is well-supported within the paper's assumptions, but those assumptions require validation beyond the current set of experiments.
The referee's report
If the reported effectiveness holds in realistic deployments, the work contributes a practical, inspectable, and evolvable runtime protection mechanism for a growing class of skill-augmented agents. The SCOPE-R dataset addresses a gap in task-conditioned, skill-induced risk evaluation. The idea of treating the guard as an optimizable skill opens a new direction for agent-side safety pipelines that can be updated without platform-level changes. The cross-model and cross-benchmark transfer results strengthen the generalizability claim. However, the central dependence on a load-bearing explicit invocation instruction (which collapses protection in the ablation) and the limited dataset scale temper the strength of the immediate deployment claim.
Pith's reading of the paper (a machine's summary, not the authors')
Skill-augmented agents load reusable skill packages that stay in context for the whole task, and a malicious skill can wait until a concrete user task makes an unsafe action look useful. This paper argues that pre-install vetting cannot catch such task-conditioned attacks, and proposes Defense-as-Skill: build the runtime guard itself as an installable, inspectable, editable skill that checks each proposed action against the user's task boundary and routes it to allow, replan, or confirm. The authors build SCOPE-R, a dataset of 206 attack-confirmed malicious skill instances across six risk families and 21 sub-categories, and evolve the guard with a Monte-Carlo Tree Search that edits the on-disk skill from real rollout feedback. On repeated runs with a primary victim model, the evolved guard reduces in-distribution attack success from 0.482 to 0.104 and out-of-distribution attack success from 0.606 to 0.115, while retaining most of the no-guard task utility and transferring to other victim models, held-out risk families, and external benchmarks. The deployment condition is load-bearing: the guard only works when an explicit instruction tells the agent to consult it before acting, and the paper's own ablation shows that without that instruction, in-distribution attack success stays at 0.400, close to the no-guard 0.482.
What carries the argument. The central object is the guard skill itself: a policy written as a Markdown skill, named SkillSonar, installed through the same skill mechanism as the untrusted task skills, acting as an in-agent policy layer that assigns each proposed action a risk level from R0 (allow) through R3 (block or replan) based on whether the action stays inside the user's task boundary. Two constructions carry the argument around it. SCOPE-R is a task-conditioned dataset built by an attacker-victim-judge loop that rewrites benign task skills until a judge confirms a target unsafe behavior actually occurred in a rollout, yielding 206 attack-confirmed malicious instances across six risk families and 21 sub-categories plus 43 benign tasks. Runtime guard-skill evolution is a Monte-Carlo Tree Search that treats each on-disk guard version as a search node, selects candidates by an upper-confidence bound computed on a cheap rollout set, full-evaluates selected candidates on a disjoint set, and uses a safety-first additive score plus judge-written failure rationales to generate edited child guards. The mechanism that makes the scheme portable is that the guard is a first-class skill artifact, so it can be measured, edited, and evolved in the same runtime where the attacks actually occur.
Core claim. SkillSonar is a runtime guard packaged as a skill: it is installed and loaded by the same skill-augmented agent that executes the user task, and for each sensitive action it decides, from the user request, the loaded skill instructions, and the execution history, whether to allow the action, narrow or replan it, or require explicit user confirmation, all without modifying the underlying agent runtime. Because the guard is an ordinary skill artifact, it can be inspected, edited, and optimized like any other skill: the proposed runtime guard-skill evolution treats candidate guard versions as nodes in a Monte-Carlo Tree Search, evaluates them on real task-skill rollouts using attack success, task utility, confirmation count, and token cost, and generates refined on-disk policies from judge-written feedback about which attacks slipped through. The evolved guard reduces measured attack success in repeated runs on the primary victim model from 0.482 to 0.104 in-distribution and from 0.606 to 0.115 out-of-distribution, at task utility comparable to no-guard levels and token overhead in the same order of magnitude. The paper further claims that the protection transfers zero-shot to other victim models, to entirely held-out risk families, and to independently built benchmarks, that it retains substantial protection against an adaptive attacker re-run against the final guard, and that neither the system-prompt baseline nor rule-based or semantic baselines reach its safety-utility operating point in the authors' matched comparisons.
The premise it rests on. The defense's measured power rests on an explicit instruction prepended to the task prompt that tells the agent to call the guard skill before any action, and without that instruction the paper's own ablation shows in-distribution attack success stays at 0.400, essentially at no-guard levels.
If the paper is right (Pith's inferences, not the authors')
Runtime guards for skill-augmented agents can ship as ordinary, inspectable, editable skill artifacts rather than external classifiers, hosted guard models, or platform patches.
A guard optimized on one victim model transfers zero-shot to other victim models and to two entirely held-out risk families (Capability Control and Privacy & Data Flow), so the protection is not tied to the evolution-time model or families.
The deployment contract includes an explicit invocation instruction: without it, in-distribution attack success stays at 0.400, close to the no-guard 0.482, so installing a safety skill alone is not enough.
The skill-native modular representation beats flattening the same policy text into a system prompt, cutting ID ASR from 0.353 to 0.104 while raising task utility and cutting token cost by about 21%.
SkillSonar is positioned as one layer in a safety stack that should complement, not replace, mechanism-level controls such as permission systems and sandboxing.
Where the task prompt is user-controlled or untrusted, the prepended invocation instruction cannot be relied on; a mechanism-level first-step hook that forces a call to the guard is the natural fallback, and testing it would show whether the measured gains survive that deployment shift.
The paper treats confirmation count and token cost as penalties, but stops short of a rule for when to confirm versus block; tuning the R2/R3 thresholds per risk family, using the confirmation budget as a resource, is a concrete next experiment that could move the frontier toward the low-ASR, high-utility corner.
If the skill-native representation advantage is genuine, other agent-side artifacts such as memory files, hook configurations, and manifest entries may be load-bearing for safety in the same way, and the same feedback-driven MCTS editing loop could be pointed at them.
The judge audit found that every observed disagreement was a false-positive success label, so no-guard ASR may be inflated on un-audited samples; scaling the human audit from the reported 18% of evaluation cases to the full test set is the cheapest check on the headline numbers.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable channel for steering future actions. Such skills may leak secrets, corrupt code, bypass approvals, or stage data for exfiltration only after a concrete user task and workspace state make the unsafe action appear useful. This makes pre-install vetting insufficient and calls for runtime, task-conditioned protection. We propose Defense-as-Skill, a defense paradigm that implements the runtime guard itself as an installable, inspectable, and editable skill. Our guard, SkillSonar, runs alongside untrusted task skills and checks sensitive actions against the user's task boundary, routing each action to an allow, replan, or confirmation decision without modifying the underlying agent runtime. To study this setting, we construct SCOPE-R, a task-conditioned dataset covering 6 risk families and 21 sub-categories, with 206 attack-confirmed malicious instances and 43 benign tasks. We then improve SkillSonar on the SCOPE-R training subset using runtime guard-skill evolution, a Monte-Carlo Tree Search procedure that evolves the on-disk guard skill from feedback on the rollouts. Across Claude Code and OpenClaw, the evolved guard substantially reduces attack success while maintaining a favorable safety-utility trade-off; on repeated GLM-5 runs, ID ASR drops from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115, with gains transferring across victim models, held-out risk families, and external benchmarks. These results show that runtime guards can be skill-native, transparent, and evolvable policy artifacts for skill-augmented agents.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.99
“Across Claude Code and OpenClaw, the evolved guard substantially reduces attack success while maintaining a favorable safety-utility trade-off.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.16 · human participants 0.06 · controlled comparison 0.72 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.61 · mentions code or data availability 0.07
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable channel for steering future actions. | background 1.00 | 0.10 | 0.05 | asserted 1.00 |
| Such skills may leak secrets, corrupt code, bypass approvals, or stage data for exfiltration only after a concrete user task and workspace state make the unsafe action appear useful. | background 0.99 | 0.08 | 0.04 | hedged possibility 1.00 |
| This makes pre-install vetting insufficient and calls for runtime, task-conditioned protection. | gap 0.84 | 0.12 | 0.13 | asserted 0.94 |
| We propose Defense-as-Skill, a defense paradigm that implements the runtime guard itself as an installable, inspectable, and editable skill. | contribution 0.91 | 0.81 | 0.13 | asserted 1.00 |
| Our guard, SkillSonar, runs alongside untrusted task skills and checks sensitive actions against the user's task boundary, routing each action to an allow, replan, or confirmation decision without modifying the underlying agent runtime. | method 0.90 | 0.89 | 0.09 | asserted 1.00 |
| To study this setting, we construct SCOPE-R, a task-conditioned dataset covering 6 risk families and 21 sub-categories, with 206 attack-confirmed malicious instances and 43 benign tasks. | method 0.79 | 0.96 | 0.02 | asserted 1.00 |
| We then improve SkillSonar on the SCOPE-R training subset using runtime guard-skill evolution, a Monte-Carlo Tree Search procedure that evolves the on-disk guard skill from feedback on the rollouts. | method 0.98 | 0.97 | 0.04 | asserted 0.98 |
| Across Claude Code and OpenClaw, the evolved guard substantially reduces attack success while maintaining a favorable safety-utility trade-off. | result 1.00 | 0.95 | 0.63 | asserted 1.00 |
| On repeated GLM-5 runs, SkillSonar reduces ID ASR from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115. | result 1.00 | 0.95 | 0.11 | asserted 1.00 |
| Further analyses demonstrate transfer across victim models, held-out risk families, and external benchmarks, as well as retained protection against adaptive attackers. | result 0.98 | 0.94 | 0.16 | asserted 1.00 |
| Ablations further show that explicit safety responsibility assignment and the skill-native representation are both important to the observed gains. | result 0.52 | 0.95 | 0.07 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,383 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment