REVIEW 3 major objections 5 minor 52 references
This paper argues that benign execution trajectories can expose proprietary agent skills, and that a black-box attacker can reconstruct usable skill instructions from paired skill-enabled and skill-disabled runs.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 02:03 UTC pith:J2GO3ZWV
load-bearing objection Genuinely new attack on proprietary agent skills, but the contrastive control that carries the empirical claim is unvalidated — plausible, not yet proven. the 3 major comments →
Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectories
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Skill Leakage is defined as unauthorized inference of proprietary skill content from user-visible trajectories, using only benign task queries. The central empirical discovery is that skills leave recurring, identifiable behavioral signatures—how the agent interprets requirements, inspects resources, selects tools, recovers from errors, and constructs outputs—and that comparing a probe executed with the skill enabled against the same probe executed with the skill suppressed isolates those signatures from task structure and base-agent behavior. SigLeak operationalizes this with a Generator that builds diverse, decision-rich probes and a Skill Synthesizer that extracts and consolidates per-pai
What carries the argument
Skill signatures—recurring observable behavior through which a skill's procedural knowledge shows up in trajectories—are the object that carries the argument. The paired execution protocol is what isolates them: the same probe is run with the skill available and with an explicit 'do not use any skills' instruction, holding the task fixed, so task requirements and base-agent behavior cancel out of the comparison. A Generator produces probes high in task diversity and decision density; a Skill Synthesizer applies a Delta operator to each trajectory pair, classifies observable steps, initializes a skill from recurring cross-task rules, and conservatively updates it while leaving uncontradicted
Load-bearing premise
The load-bearing premise is that prepending 'Do not use any skills' genuinely prevents the target skill from being invoked and changes nothing else about the agent's behavior; if the suppression is partial or alters caution, tool use, or reasoning depth, the contrast attributes instruction effects to the skill.
What would settle it
Instrument the target agent to log whether the skill file is actually loaded in the skill-disabled condition. If the suppression prompt ever fails—or if disabling by prompt yields trajectories different from disabling by uninstalling the skill entirely—then the paired contrast is not cleanly attributable to the skill, and the reported recovery gains would need to be re-measured against true skill removal.
If this is right
- An attacker who can query a deployed skill-equipped agent can functionally recover the skill with queries that never ask for it.
- The leakage channel is behavioral, not textual: it persists even when direct prompt-stealing queries are blocked by a defense.
- Inference is target-specific, not generic: inferred skills for three same-domain workflows align with their own targets, not with sibling skills.
- Bounded refinement is enough: Stage 2 gains peak within the first two rounds, so a small probe budget already captures most recoverable procedure.
- Skill-disabled trajectories are a useful reference signal, but their value depends on the suppression instruction actually being obeyed.
Where Pith is reading between the lines
- This is a claim about any hidden procedural component with observable behavior, not only skills; the same contrastive probing could apply to system prompts, tool policies, or embedded workflows, though the paper does not test that.
- The results suggest a defense direction: if behavioral traces are the leak, providers could randomize or degrade trajectory-level cues, at the cost of the human oversight that motivates exposing trajectories in the first place.
- One testable extension is to vary the suppression instruction—wording, placement, or an actual skill removal—and measure how much the attribution and recovered skill change; this would bound the side channel independent of SigLeak's particular prompts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper defines Skill Leakage as the inference of proprietary agent skills from user-visible execution trajectories elicited by benign queries, and proposes SigLeak, a black-box framework that generates diagnostic probes, contrasts matched skill-enabled and skill-disabled trajectories, extracts recurring skill signatures, and iteratively refines a reconstructed SKILL.md. The authors evaluate SigLeak across five scenarios, three model families, and three agent frameworks, reporting that it outperforms or matches direct generation, naive trace summarization, and a defended BBS prompt-stealing baseline in nearly every setting. The headline result is a 6.88 percentage-point average improvement in downstream success rate over the skill-disabled reference, together with the highest SkillSim among inference methods. The paper concludes that benign execution trajectories can expose proprietary procedural knowledge.
Significance. If the central attribution claim can be made to stand, this is a timely and useful contribution to the security of agent skill marketplaces and Agent-as-a-Service platforms. The paper opens a new threat model—behavioral side-channel inference from trajectories—and proposes a concrete end-to-end pipeline with a clear two-stage design. The evaluation is broad (five benchmarks, three model families, three agent frameworks), and the release of code and full prompts is a reproducible standard worth crediting. The comparison against explicit prompt-stealing baselines is also well motivated. However, the empirical support for the headline 6.88 pp gain and for the skill-specificity of the extracted signatures currently rests on a confounded skill-disabled control; the significance is therefore conditional on the validation proposed below.
major comments (3)
- [§3.2 (Eqs. 4–6), Appendix B.3] The contrast on which the whole method rests is confounded. Eq. (4) defines c_off by the shorthand “Do not use any skills,” but Appendix B.3 shows the true prompt is a long instruction that forbids accessing any skill “even if the task succeeds,” imposes an “EXPERIMENT CONDITION” frame, and restricts work to the current directory. These extra constraints can change caution, tool-use scope, and verification behavior independently of skill suppression. The paper gives no evidence that c_off prevents skill invocation or that it is otherwise behavior-neutral. Since Eq. (6) attributes every difference between τ_on and τ_off to the skill, the extracted signatures and the reconstructed skill may include instruction-induced artifacts, and the 6.88 pp gain over the skill-disabled reference overstates recovery of the target skill. Please validate the control, e.g., by comparing c_off with a true u
- [Table 1 and Appendix F.1] The headline quantitative claims are averages over three runs without variance in Table 1. In the single configuration where three-run standard deviations are reported (GPT-5.4-mini/Codex), differences as large as the ALFWorld SigLeak gain (79.52±3.94 vs 76.67±3.30) are within one standard deviation. Many scenario and cell gains are small (+1.62 pp ALFWorld, +5.75 pp LiveMath). The “best or tied-best in nearly every setting” and the 6.88 pp average should be accompanied by per-cell variance or confidence intervals / paired tests; at present the improvement claim is not robustly quantified.
- [§4.2, §4.4, Appendix E] The abstract’s claim of the “highest overall SkillSim” relies on an LLM-judged metric with no validation of the judge. Coarse SkillSim is a single 1–5 LLM rating; fine-grained SkillSim uses LLM-based SBC extraction and one-to-one soft alignment. No human agreement study, alternate judge model, or sensitivity analysis is reported. Given that SkillSim is the primary evidence for semantic fidelity, the paper should demonstrate that the metric is stable under judge choice or correlate it with human judgments on a sample, otherwise the semantic-fidelity ranking may reflect prompt/format preferences rather than true procedural recovery.
minor comments (5)
- [Figure 2 caption] The caption text appears garbled: “G PT-5.4-miniG LM-5.2M inimax-M350556...”. Please fix the typesetting.
- [Figure 2 vs Appendix A.1] Inconsistent sample size: Figure 2 says a 100-instance subset of SpreadsheetBench, while Appendix A.1 says a 200-instance subset. Please unify.
- [§4.1] The sentence “The same task description is provided to all compared inference methods, with the descriptions included in Appendix D” appears twice in consecutive paragraphs. Remove the duplicate.
- [§4.4] The heading “SigLeakdistinguishesskillswithinthesamedomain” is followed by the fragment “main.” before the next sentence. Please clean up.
- [References] Spelling inconsistency: “ClawMart” in the abstract/introduction vs “Claw Mart” in the references. Use one form consistently.
Circularity Check
No significant circularity: SigLeak's reconstructed skill is evaluated on held-out benchmark tasks, and the 6.88 pp SR gain is an empirical contrast against an explicitly labeled experimental control, not a quantity derived from the method's own inputs.
full rationale
The paper's derivation chain is empirical rather than definitional. SigLeak constructs diagnostic probes and extracts skill signatures by contrasting matched skill-enabled and skill-disabled trajectories (Eqs. 4-6), then evaluates the resulting skill on separate official benchmark evaluators (SR) and on SBC-based semantic matching (SkillSim). The evaluation data and metrics are not provided to the Generator or Skill Synthesizer, so the reported gains are measurements rather than identities. The skill-disabled condition is explicitly framed as 'an experimental control rather than a component of the inferred skill' (Appendix B.3), and the paper provides an additional comparison against Naive Trace Summarization, which does not receive the paired contrast; SigLeak's advantage there gives independent evidence that the contrast carries real signal. The cited prior methods (SkillOpt, Trace2Skill, BBS, SBC, etc.) are external works, and no load-bearing claim rests on a self-citation or an imported uniqueness theorem. The main threats to the paper's conclusion are experimental-validity issues rather than circularity: the c_off control prompt may alter caution, tool-use style, or reasoning depth beyond merely suppressing skill invocation, and the paper does not verify that it leaves all non-skill behavior unchanged; also, capping Stage 2 at two rounds based on the ablation's peak is a selection/overfitting concern. These are legitimate correctness risks, but they do not make the output equal to the input by construction, and no equation or fitted parameter in the paper reduces the claimed result to its own assumptions. Therefore no circular step meeting the quote-and-reduction requirement is present.
Axiom & Free-Parameter Ledger
free parameters (3)
- Probe budget (6 Stage-1 probes, 3 per round, max 2 rounds) =
6/3/2
- SigLeak inference model =
DeepSeek-V4-Flash
- SkillSim match weights =
1.00 / 0.75 / 0.50 / 0.25
axioms (5)
- domain assumption Skill selection is model-mediated and a user instruction such as 'Do not use any skills' reliably suppresses skill invocation without otherwise altering behavior.
- domain assumption User-visible trajectories expose agent messages, tool calls, and environment observations (Definition 1).
- domain assumption Skills systematically shape agent execution into recurring observable patterns ('skill signatures').
- domain assumption An LLM can reliably generate diverse, decision-rich probes and synthesize reusable skill text from trajectory deltas.
- domain assumption LLM-based judges can reliably score behavioral similarity between skills and extract meaningful Skill Behavior Constraints.
read the original abstract
Agent skills package reusable procedures that improve downstream performance. Their lightweight, portable form enables marketplace monetization and private deployment behind cloud-hosted agent interfaces, giving providers incentives to keep high-value skills proprietary. Yet hiding the artifacts does not conceal their behavioral effects, which remain observable in execution trajectories and form a behavioral side channel. We define this exposure as Skill Leakage: reconstructing proprietary skills from trajectories elicited by benign queries, without reference answers or success labels. We introduce SigLeak, a black-box framework that exploits recurring skill signatures in agent behavior. It constructs diverse, decision-rich diagnostic tasks, contrasts matched skill-enabled and skill-disabled trajectories, and iteratively refines a reconstructed skill from the isolated patterns. Across five scenarios, three model families, and three agent frameworks, SigLeak outperforms or matches three baselines in nearly every setting. It raises the success rate by 6.88 percentage points over the skill-disabled reference on average and achieves the highest overall SkillSim, our metric for coarse- and fine-grained semantic similarity. These results show that benign execution trajectories can expose proprietary procedural knowledge. The code is available at https://anonymous.4open.science/r/SigLeak-D1DB.
Figures
Reference graph
Works this paper leans on
-
[1]
2023 , url =
Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url =
2023
-
[2]
Transactions on Machine Learning Research , year=
Voyager: An Open-Ended Embodied Agent with Large Language Models , author=. Transactions on Machine Learning Research , year=
-
[3]
and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle =
Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle =. 2024 , doi =
2024
-
[4]
Science China Information Sciences , volume =
The Rise and Potential of Large Language Model Based Agents: A Survey , author =. Science China Information Sciences , volume =. 2025 , publisher =
2025
-
[5]
arXiv preprint arXiv:2602.12430 , year=
Agent skills for large language models: Architecture, acquisition, security, and the path forward , author=. arXiv preprint arXiv:2602.12430 , year=
-
[6]
2025 , note =
Anthropic , title =. 2025 , note =
2025
-
[7]
2026 , note =
Claude Code Plugins, Skills & MCP Servers , howpublished =. 2026 , note =
2026
-
[8]
2026 , note =
Agent Skills Marketplace , howpublished =. 2026 , note =
2026
-
[9]
arXiv preprint arXiv:2603.02766 , year=
Evoskill: Automated skill discovery for multi-agent systems , author=. arXiv preprint arXiv:2603.02766 , year=
-
[10]
arXiv preprint arXiv:2605.08670 , year=
MIND-Skill: Quality-Guaranteed Skill Generation via Multi-Agent Induction and Deduction , author=. arXiv preprint arXiv:2605.08670 , year=
-
[11]
arXiv preprint arXiv:2603.25158 , year=
Trace2skill: Distill trajectory-local lessons into transferable agent skills , author=. arXiv preprint arXiv:2603.25158 , year=
-
[12]
arXiv preprint arXiv:2605.23904 , year=
Skillopt: Executive strategy for self-evolving agent skills , author=. arXiv preprint arXiv:2605.23904 , year=
-
[13]
arXiv preprint arXiv:2605.10999 , year=
Skillgen: Verified inference-time agent skill synthesis , author=. arXiv preprint arXiv:2605.10999 , year=
-
[14]
arXiv preprint arXiv:2605.10500 , year=
Skillevolver: Skill learning as a meta-skill , author=. arXiv preprint arXiv:2605.10500 , year=
-
[15]
arXiv preprint arXiv:2604.21829 , year=
Black-Box Skill Stealing Attack from Proprietary LLM Agents: An Empirical Study , author=. arXiv preprint arXiv:2604.21829 , year=
-
[16]
2026 , howpublished =
Yusuf Karaaslan , title =. 2026 , howpublished =
2026
-
[17]
SKILL: Automated AI Skill Generation via Expert Knowledge Distillation , author=
COLLEAGUE. SKILL: Automated AI Skill Generation via Expert Knowledge Distillation , author=. arXiv preprint arXiv:2605.31264 , year=
-
[18]
2024 , doi =
Ma, Zeyao and Zhang, Bohan and Zhang, Jing and Yu, Jifan and Zhang, Xiaokang and Zhang, Xiaohan and Luo, Sijia and Wang, Xi and Tang, Jie , booktitle =. 2024 , doi =
2024
-
[19]
2026 , howpublished =
Daniel Miessler , title =. 2026 , howpublished =
2026
-
[20]
2026 , howpublished =
Andrej Karpathy , title =. 2026 , howpublished =
2026
-
[21]
arXiv preprint arXiv:2603.08655 , year=
Officeqa pro: An enterprise benchmark for end-to-end grounded reasoning , author=. arXiv preprint arXiv:2603.08655 , year=
-
[22]
arXiv preprint arXiv:2506.01062 , year=
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models , author=. arXiv preprint arXiv:2506.01062 , year=
-
[23]
Forty-second International Conference on Machine Learning , year=
The Berkeley Function Calling Leaderboard (bfcl): From tool use to agentic evaluation of large language models , author=. Forty-second International Conference on Machine Learning , year=
-
[24]
2026 , eprint=
LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches , author=. 2026 , eprint=
2026
-
[25]
2021 , url =
Mohit Shridhar and Xingdi Yuan and Marc-Alexandre C\^ot\'e and Yonatan Bisk and Adam Trischler and Matthew Hausknecht , booktitle =. 2021 , url =
2021
-
[26]
arXiv preprint arXiv:2603.04448 , year=
Skillnet: Create, evaluate, and connect ai skills , author=. arXiv preprint arXiv:2603.04448 , year=
-
[27]
arXiv preprint arXiv:2603.01145 , year=
Autoskill: Experience-driven lifelong learning via skill self-evolution , author=. arXiv preprint arXiv:2603.01145 , year=
-
[28]
arXiv preprint arXiv:2604.04804 , year=
Skillx: Automatically constructing skill knowledge bases for agents , author=. arXiv preprint arXiv:2604.04804 , year=
-
[29]
arXiv preprint arXiv:2603.18743 , year=
Memento-skills: Let agents design agents , author=. arXiv preprint arXiv:2603.18743 , year=
-
[30]
arXiv preprint arXiv:2605.16986 , year=
Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents , author=. arXiv preprint arXiv:2605.16986 , year=
-
[31]
arXiv preprint arXiv:2602.12670 , year=
SkillsBench: Benchmarking how well agent skills work across diverse tasks , author=. arXiv preprint arXiv:2602.12670 , year=
-
[32]
arXiv preprint arXiv:2604.03964 , year=
Skillfoundry: Building self-evolving agent skill libraries from heterogeneous scientific resources , author=. arXiv preprint arXiv:2604.03964 , year=
-
[33]
arXiv preprint arXiv:2604.16911 , year=
Skilldex: A Package Manager and Registry for Agent Skill Packages with Hierarchical Scope-Based Distribution , author=. arXiv preprint arXiv:2604.16911 , year=
-
[34]
arXiv preprint arXiv:2603.00195 , year=
Formal analysis and supply chain security for agentic AI skills , author=. arXiv preprint arXiv:2603.00195 , year=
-
[35]
arXiv preprint arXiv:2606.01311 , year=
SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories , author=. arXiv preprint arXiv:2606.01311 , year=
-
[36]
Reinforcement Learning for Self-Improving Agent with Skill Library
Wang, Jiongxiao and Yan, Qiaojing and Wang, Yawei and Tian, Yijun and Mishra, Soumya Smruti and Xu, Zhichao and Gandhi, Megha and Xu, Panpan and Cheong, Lin Lee. Reinforcement Learning for Self-Improving Agent with Skill Library. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). 2026. doi:1...
-
[37]
arXiv preprint arXiv:2602.08234 , year=
SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning , author=. arXiv preprint arXiv:2602.08234 , year=
-
[38]
arXiv preprint arXiv:2605.28791 , year=
Skill-Conditioned Gated Self-Distillation for LLM Reasoning , author=. arXiv preprint arXiv:2605.28791 , year=
-
[39]
2026 , eprint=
SoK: Agentic Skills -- Beyond Tool Use in LLM Agents , author=. 2026 , eprint=
2026
-
[40]
2026 , eprint=
Agent Skill Evaluation and Evolution: Frameworks and Benchmarks , author=. 2026 , eprint=
2026
-
[41]
2026 , eprint=
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents , author=. 2026 , eprint=
2026
-
[42]
2026 , eprint=
SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? , author=. 2026 , eprint=
2026
-
[43]
2026 , eprint=
SkillTester: Benchmarking Utility and Security of Agent Skills , author=. 2026 , eprint=
2026
-
[44]
2025 , howpublished =
Cloudflare , title =. 2025 , howpublished =
2025
-
[45]
2026 , howpublished =
Host Agents or Tools with. 2026 , howpublished =
2026
-
[46]
2026 , eprint=
Skill Coverage: A Test Adequacy Metric for Agent Skills , author=. 2026 , eprint=
2026
-
[47]
Large Language Model-based Human-Agent Collaboration for Complex Task Solving
Feng, Xueyang and Chen, Zhi-Yuan and Qin, Yujia and Lin, Yankai and Chen, Xu and Liu, Zhiyuan and Wen, Ji-Rong. Large Language Model-based Human-Agent Collaboration for Complex Task Solving. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.72
-
[48]
LLM -Based Human-Agent Collaboration and Interaction Systems: A Survey
Zou, Henry Peng and Huang, Wei-Chieh and Wu, Yaozu and Guo, Jizhou and Chen, Yankai and Miao, Chunyu and Nguyen, Hoang H and Zhou, Yue and Zhang, Weizhi and Fang, Liancheng and Zhang, Hanrong and Wang, Fangxin and Zhang, Pengfei and He, Langzhou and Li, Yangning and Li, Dongyuan and Jiang, Renhe and Yu, Philip S. LLM -Based Human-Agent Collaboration and I...
-
[49]
First Conference on Language Modeling , year =
Effective Prompt Extraction from Language Models , author =. First Conference on Language Modeling , year =
-
[50]
Hui, Bo and Yuan, Haolin and Gong, Neil and Burlina, Philippe and Cao, Yinzhi , title =. 2024 , isbn =. doi:10.1145/3658644.3670370 , booktitle =
arXiv 2024
-
[51]
Unveiling Privacy Risks in LLM Agent Memory
Wang, Bo and He, Weiyi and Zeng, Shenglai and Xiang, Zhen and Xing, Yue and Tang, Jiliang and He, Pengfei. Unveiling Privacy Risks in LLM Agent Memory. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.1227
-
[52]
2026 , eprint=
RedAct: Redacting Agent Capability Traces for Procedural Skill Protection , author=. 2026 , eprint=
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.