{"id":"6448230a-0def-497f-8da1-69489b8165c2","arxiv_id":"2607.25560","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A black-box attacker can recover a functional approximation of a hidden agent skill from paired skill-enabled and skill-disabled execution trajectories elicited by benign queries.","lead":"The paper shows that an attacker can reconstruct a proprietary AI agent skill by watching how the agent behaves on crafted tasks, comparing runs where the agent is told to use the skill against runs where it is told not to. The finding matters because agent skill marketplaces are commercializing these artifacts, and this is the first systematic demonstration of passive behavioral skill leakage from benign queries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The skill-disabled control in Eq. 4–5 is unvalidated; if 'Do not use skills' alters caution or tool use, the 6.88 pp SR gain and extracted signatures conflate instruction effects with skill effects.","rationale":"The reader's weakest_assumption exactly matches my analysis: the paired-protocol validity depends on c_off being behavior-neutral with respect to everything except skill invocation. This is the most load-bearing concern because it underpins both the signature-extraction mechanism and the headline 6.88 pp SR improvement. The paper provides extensive detail on the control prompt (Appendix B.3) but no empirical validation that it is a clean counterfactual. The concern is specific and testable: an oracle comparison with a truly uninstalled skill would settle it. I agree with the reader's CONDITIONAL verdict: the attack is plausible and well-structured, and there is corroborating evidence from SkillSim and the within-domain discrimination analysis (Figure 4b) that is independent of the disabled control, but the cleanest demonstration of the central claim is not yet fully validated. No ad hominem, no manufactured concern; this is a genuine methodological soft spot that the paper itself flags as critical without resolving.","tokens_in":24158,"tokens_out":3677,"duration_ms":41286,"concrete_test":"Compare the skill-disabled condition induced by c_off against a true oracle: an identical agent with the target skill physically uninstalled from the backend. Run the same set of diagnostic probes under three conditions: (a) skill enabled (τ_on), (b) skill disabled by c_off (τ_off), and (c) skill uninstalled (τ_uninst). Compute Delta(q; τ_on, τ_off) and Delta(q; τ_on, τ_uninst) for each probe and measure pairwise agreement (e.g., overlap of extracted signature spans or behavioral features like tool-call frequency, validation steps, output verbosity). Also measure skill-invocation rate under c_off (e.g., via log inspection in a controlled environment). If τ_off and τ_uninst diverge substantially, or if c_off fails to suppress invocation, the paired-protocol attribution is confounded and the headline SR gain must be re-evaluated against the uninstalled baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—that SigLeak improves SR over the skill-disabled reference by 6.88 pp and that extracted signatures are skill-specific—rests entirely on the paired-protocol contrast Delta(q; τ_on, τ_off) defined in §3.2 (Eqs. 4–6). This contrast treats the prompted non-invocation condition c_off as a behavior-neutral way to remove the skill. Appendix B.3 shows c_off is a long, explicit instruction: 'Do not discover, open, invoke, summarize, or rely on any installed skill… Accessing one fails the experimental condition.' Such an instruction could alter the agent's behavior beyond suppressing skill invocation: it may make the agent more cautious (e.g., avoiding tools, doing extra verification), change reasoning depth, or induce literal compliance that suppresses legitimate tool use. The paper provides no check that c_off actually prevents skill invocation, nor that it leaves all other behavior unchanged. If c_off has any non-skill effect, Delta is confounded: signatures extracted from the contrast may include instruction-induced artifacts, and the headline SR gain relative to the disabled reference partly reflects recovery from an artificially degraded baseline rather than recovery of the target skill. This is the single load-bearing assumption: without a valid skill-disabled control, the method's claimed advantage over Naive Trace Summarization (which does not use the contrast) and the central claim that 'benign execution trajectories can expose proprietary procedural knowledge' are not established. The paper even acknowledges in B.3 that 'its wording directly affects the validity of the paired comparison,' yet reports no validation of the control itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines Skill Leakage as the inference of proprietary agent skills from user-visible execution trajectories elicited by benign queries, and proposes SigLeak, a black-box framework that generates diagnostic probes, contrasts matched skill-enabled and skill-disabled trajectories, extracts recurring skill signatures, and iteratively refines a reconstructed SKILL.md. The authors evaluate SigLeak across five scenarios, three model families, and three agent frameworks, reporting that it outperforms or matches direct generation, naive trace summarization, and a defended BBS prompt-stealing baseline in nearly every setting. The headline result is a 6.88 percentage-point average improvement in downstream success rate over the skill-disabled reference, together with the highest SkillSim among inference methods. The paper concludes that benign execution trajectories can expose proprietary procedural knowledge.","tokens_in":24466,"tokens_out":6518,"duration_ms":65634,"significance":"If the central attribution claim can be made to stand, this is a timely and useful contribution to the security of agent skill marketplaces and Agent-as-a-Service platforms. The paper opens a new threat model—behavioral side-channel inference from trajectories—and proposes a concrete end-to-end pipeline with a clear two-stage design. The evaluation is broad (five benchmarks, three model families, three agent frameworks), and the release of code and full prompts is a reproducible standard worth crediting. The comparison against explicit prompt-stealing baselines is also well motivated. However, the empirical support for the headline 6.88 pp gain and for the skill-specificity of the extracted signatures currently rests on a confounded skill-disabled control; the significance is therefore conditional on the validation proposed below.","major_comments":[{"comment":"The contrast on which the whole method rests is confounded. Eq. (4) defines c_off by the shorthand “Do not use any skills,” but Appendix B.3 shows the true prompt is a long instruction that forbids accessing any skill “even if the task succeeds,” imposes an “EXPERIMENT CONDITION” frame, and restricts work to the current directory. These extra constraints can change caution, tool-use scope, and verification behavior independently of skill suppression. The paper gives no evidence that c_off prevents skill invocation or that it is otherwise behavior-neutral. Since Eq. (6) attributes every difference between τ_on and τ_off to the skill, the extracted signatures and the reconstructed skill may include instruction-induced artifacts, and the 6.88 pp gain over the skill-disabled reference overstates recovery of the target skill. Please validate the control, e.g., by comparing c_off with a true u","section":"§3.2 (Eqs. 4–6), Appendix B.3"},{"comment":"The headline quantitative claims are averages over three runs without variance in Table 1. In the single configuration where three-run standard deviations are reported (GPT-5.4-mini/Codex), differences as large as the ALFWorld SigLeak gain (79.52±3.94 vs 76.67±3.30) are within one standard deviation. Many scenario and cell gains are small (+1.62 pp ALFWorld, +5.75 pp LiveMath). The “best or tied-best in nearly every setting” and the 6.88 pp average should be accompanied by per-cell variance or confidence intervals / paired tests; at present the improvement claim is not robustly quantified.","section":"Table 1 and Appendix F.1"},{"comment":"The abstract’s claim of the “highest overall SkillSim” relies on an LLM-judged metric with no validation of the judge. Coarse SkillSim is a single 1–5 LLM rating; fine-grained SkillSim uses LLM-based SBC extraction and one-to-one soft alignment. No human agreement study, alternate judge model, or sensitivity analysis is reported. Given that SkillSim is the primary evidence for semantic fidelity, the paper should demonstrate that the metric is stable under judge choice or correlate it with human judgments on a sample, otherwise the semantic-fidelity ranking may reflect prompt/format preferences rather than true procedural recovery.","section":"§4.2, §4.4, Appendix E"}],"minor_comments":[{"comment":"The caption text appears garbled: “G PT-5.4-miniG LM-5.2M inimax-M350556...”. Please fix the typesetting.","section":"Figure 2 caption"},{"comment":"Inconsistent sample size: Figure 2 says a 100-instance subset of SpreadsheetBench, while Appendix A.1 says a 200-instance subset. Please unify.","section":"Figure 2 vs Appendix A.1"},{"comment":"The sentence “The same task description is provided to all compared inference methods, with the descriptions included in Appendix D” appears twice in consecutive paragraphs. Remove the duplicate.","section":"§4.1"},{"comment":"The heading “SigLeakdistinguishesskillswithinthesamedomain” is followed by the fragment “main.” before the next sentence. Please clean up.","section":"§4.4"},{"comment":"Spelling inconsistency: “ClawMart” in the abstract/introduction vs “Claw Mart” in the references. Use one form consistently.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The first major comment is the key gate: without a validated skill-disabled control, the central claim cannot be assessed. The revision should prioritize the control-validation experiment and a more complete uncertainty analysis. If those are addressed, the paper could become a solid contribution; the current version is not yet there."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper identifies a genuinely new attack surface: inferring hidden agent skills from benign, unlabeled execution trajectories, with active probing. That's a real gap relative to BBS (adversarial extraction) and RedAct (passive corpus redaction). Second, the empirical centerpiece — the contrastive enabled/disabled comparison — is confounded, and the paper's own appendix admits the risk without resolving it. So the central quantitative claim is currently unproven, even though the threat is plausible.\n\nWhat's good. The formulation of Skill Leakage is clean and the method is sensible: generate decision-rich probes, compare matched trajectories with and without skill invocation, synthesize a skill, refine iteratively. The evaluation spans five scenarios, three model families, three frameworks, and the code is released. The in-domain arXiv experiment, showing that inferred skills are most similar to their targets, is a nice discrimination test. The qualitative trajectory examples in the appendix genuinely show distinct skill signatures across skills. The paper also honestly reports the ALFWorld case where naive summarization matches SigLeak.\n\nThe soft spot. The 'skill-disabled' condition is not actual skill removal; it's an explicit user instruction telling the agent not to access any skill. Appendix B.3 prints the full prompt, which is long and emphatic: 'Accessing one fails the experimental condition even if the task succeeds.' The paper notes its wording 'directly affects the validity of the paired comparison' but gives no check that the instruction suppresses only skill use. It could plausibly make the agent more cautious, avoid legitimate tool use, or add a mandatory verification step. If so, the delta between enabled and disabled trajectories is inflated by instruction effects, and the 6.88 pp gain over the skill-disabled reference is partly recovery from an artificially degraded baseline. The extracted signatures could also carry instruction artifacts. This is the load-bearing assumption, and it needs an experimental fix: a no-instruction control, or an oracle condition with the skill actually uninstalled.\n\nMinor issues: the main Table 1 reports means over three runs without variance (the appendix gives standard deviations for one model family only), and the SkillSim judge is an unnamed LLM. Neither is fatal, but both deserve attention in a revision.\n\nWho should read this: AI security researchers and anyone building agent marketplaces or Agent-as-a-Service. The attack is likely real in spirit; the demonstration needs tightening. I'd send it to reviewers — the topic is important and the flaws are fixable — but the decision should be conditional on validating the control or softening the central claim.","headline":"Genuinely new attack on proprietary agent skills, but the contrastive control that carries the empirical claim is unvalidated — plausible, not yet proven.","tokens_in":25031,"tokens_out":4210,"would_cite":true,"duration_ms":41578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that benign execution trajectories can expose proprietary agent skills, and that a black-box attacker can reconstruct usable skill instructions from paired skill-enabled and skill-disabled runs.","keywords":["agent skills","skill leakage","execution trajectories","black-box inference","behavioral side channel","skill signatures","contrastive trajectory comparison","LLM agent security"],"falsifier":"Instrument the target agent to log whether the skill file is actually loaded in the skill-disabled condition. If the suppression prompt ever fails—or if disabling by prompt yields trajectories different from disabling by uninstalling the skill entirely—then the paired contrast is not cleanly attributable to the skill, and the reported recovery gains would need to be re-measured against true skill removal.","tokens_in":24025,"feed_emoji":"🕵️","tokens_out":4740,"duration_ms":50076,"temperature":0.7,"pith_summary":"Skills package reusable procedures, and providers hide high-value skill files while exposing only an agent interface. This paper argues that hiding the artifact does not hide the behavior: execution trajectories produced by benign queries are a behavioral side channel, and a black-box attacker can reconstruct a usable approximation of a proprietary skill without ever seeing the skill file, reference answers, or success labels. The proposed framework, SigLeak, generates decision-rich diagnostic probes, contrasts matched trajectories in which the skill is or is not invoked, and iteratively refines the reconstructed skill from the recurring differences. Across five task scenarios, three model families, and three agent frameworks, the inferred skills improve downstream success rate by 6.88 percentage points over the skill-disabled reference on average and score highest on the paper's skill-similarity metric. The point of the paper is that secrecy of the artifact is not secrecy of the procedure.","feed_headline":"Benign queries reconstruct hidden agent skills","feed_subtitle":"Contrasting skill-on and skill-off runs recovers enough procedure to lift success by 6.88 points on average.","key_machinery":"Skill signatures—recurring observable behavior through which a skill's procedural knowledge shows up in trajectories—are the object that carries the argument. The paired execution protocol is what isolates them: the same probe is run with the skill available and with an explicit 'do not use any skills' instruction, holding the task fixed, so task requirements and base-agent behavior cancel out of the comparison. A Generator produces probes high in task diversity and decision density; a Skill Synthesizer applies a Delta operator to each trajectory pair, classifies observable steps, initializes a skill from recurring cross-task rules, and conservatively updates it while leaving uncontradicted","core_discovery":"Skill Leakage is defined as unauthorized inference of proprietary skill content from user-visible trajectories, using only benign task queries. The central empirical discovery is that skills leave recurring, identifiable behavioral signatures—how the agent interprets requirements, inspects resources, selects tools, recovers from errors, and constructs outputs—and that comparing a probe executed with the skill enabled against the same probe executed with the skill suppressed isolates those signatures from task structure and base-agent behavior. SigLeak operationalizes this with a Generator that builds diverse, decision-rich probes and a Skill Synthesizer that extracts and consolidates per-pai","pith_inferences":["This is a claim about any hidden procedural component with observable behavior, not only skills; the same contrastive probing could apply to system prompts, tool policies, or embedded workflows, though the paper does not test that.","The results suggest a defense direction: if behavioral traces are the leak, providers could randomize or degrade trajectory-level cues, at the cost of the human oversight that motivates exposing trajectories in the first place.","One testable extension is to vary the suppression instruction—wording, placement, or an actual skill removal—and measure how much the attribution and recovered skill change; this would bound the side channel independent of SigLeak's particular prompts."],"forward_implications":["An attacker who can query a deployed skill-equipped agent can functionally recover the skill with queries that never ask for it.","The leakage channel is behavioral, not textual: it persists even when direct prompt-stealing queries are blocked by a defense.","Inference is target-specific, not generic: inferred skills for three same-domain workflows align with their own targets, not with sibling skills.","Bounded refinement is enough: Stage 2 gains peak within the first two rounds, so a small probe budget already captures most recoverable procedure.","Skill-disabled trajectories are a useful reference signal, but their value depends on the suppression instruction actually being obeyed."],"fun_headline_variants":["SigLeak infers hidden agent skills from benign queries","Agent skills leak through behavior: SigLeak reconstructs them","Proprietary agent skills recoverable from execution traces","Benign probes expose proprietary agent skill content","Skill leakage: black-box reconstruction from trajectories"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that prepending 'Do not use any skills' genuinely prevents the target skill from being invoked and changes nothing else about the agent's behavior; if the suppression is partial or alters caution, tool use, or reasoning depth, the contrast attributes instruction effects to the skill.","fun_headline_variants_meta":{"raw":{"variants":["SigLeak infers hidden agent skills from benign queries","Agent skills leak through behavior: SigLeak reconstructs them","Proprietary agent skills recoverable from execution traces","Benign probes expose proprietary agent skill content","Skill leakage: black-box reconstruction from trajectories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2213,"prompt_tokens":722,"completion_tokens":1491,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":1416}},"tokens_in":466,"tokens_out":1491,"duration_ms":11719,"temperature":1.0,"reasoning_tokens":1416,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:03:15.477128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument the target agent to log whether the skill file is actually loaded in the skill-disabled condition. If the suppression prompt ever fails—or if disabling by prompt yields trajectories different from disabling by uninstalling the skill entirely—then the paired contrast is not cleanly attributable to the skill, and the reported recovery gains would need to be re-measured against true skill removal.","supporting_citations":[],"review_version":1}