{"id":"8bc5f51d-4290-4270-8854-3987b7193442","arxiv_id":"2607.13987","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reusable agent skills are vulnerable at every lifecycle stage—repository admission, retrieval, planner selection, execution, and evolution—and SkillSec-Eval provides a benchmark to measure those attacks and defenses.","lead":"An evaluation framework, SkillSec-Eval, maps security threats in reusable LLM-agent skills across the full skill lifecycle and reports that attacks succeed at retrieval, planning, execution, and update stages, not just at runtime. It gives security teams a common way to benchmark where skill ecosystems break and which defenses matter.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Planner model differs between baseline and defended setups (§5.3 vs §6.3), confounding the planner-defense gains and weakening a key empirical pillar of the lifecycle claim.","rationale":"The reader's verdict is CONDITIONAL, citing irreproducibility, missing artifacts, and a circular attack/defense design. The planner-model inconsistency is a concrete internal flaw that directly affects the empirical validity of the planner-stage results, which are central to the paper's lifecycle claim. I consider this more load-bearing than the representativeness limitation, because it is a verifiable internal inconsistency rather than a generalizability caveat. The reader did mention the planner model inconsistency in passing, so my agreement is partial. The concern is fixable via a controlled re-run, so the CONDITIONAL verdict remains appropriate.","tokens_in":13656,"tokens_out":3332,"duration_ms":35386,"concrete_test":"Re-run the planner experiments of Table 4 with the planner model held fixed across baseline and defended conditions—e.g., both using Gemini 3.1 Pro and both using Gemini 1.5 Flash. If the defended ASR for Fake Recommendation is not substantially below the baseline in the same-model comparison (e.g., not below 20%), the metadata-validation defense effect is not established. Additionally, obtain from the authors the exact planner model used for each row of Table 4.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 states 'Planner experiments employ Gemini 3.1 pro as the planning model.' Section 6.3 states 'The baseline planner uses Gemini 1.5 Flash in a zero-shot configuration.' If the defended configuration uses Gemini 3.1 Pro while the baseline uses Gemini 1.5 Flash, then the reported planner robustness improvements in Table 4—e.g., Fake Recommendation dropping from 45.64% to 8.72%—are confounded by model capability rather than by the proposed Metadata Validation Layer. Conversely, if both use the same model, then §5.3 is inaccurate, undermining reproducibility. Either way, the causal attribution that metadata validation improves planner robustness is not supported by the current text. The central claim that 'vulnerabilities arise at multiple lifecycle stages beyond execution' relies on the planner selection stage being both vulnerable and defensible; a confounded comparison leaves that stage's evidence ambiguous. This is an internal inconsistency, not a matter of external generalization, and it is directly checkable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SkillSec-Eval, a lifecycle-aware framework and benchmark for security evaluation of reusable LLM agent skills. It defines a six-stage lifecycle (authoring, storage, retrieval, selection, execution, evolution) and a corresponding threat taxonomy, then instantiates five operational evaluation boundaries: repository admission, semantic retrieval, planner selection, runtime execution, and skill evolution. Using a repository of 327 benign skills derived from SkillMCP, the authors generate attacks at each boundary, evaluate a suite of defenses (hybrid admission control, retrieval validation, metadata validation, runtime taint tracking, and update revalidation), and report attack success rates and defense metrics in Tables 2–6. The central claim is that vulnerabilities arise at multiple lifecycle stages beyond execution, so security analysis of agent skills should be lifecycle-aware.","tokens_in":13933,"tokens_out":4127,"duration_ms":45818,"significance":"If the results hold, the paper makes a useful contribution by providing a shared vocabulary and evaluation scaffold for an emerging problem: secure reuse of agent skills. The taxonomy is reasonable, the modular separation of lifecycle boundaries is a genuine improvement over end-to-end agent benchmarks, the use of a real-world-derived skill corpus is a strength, and the appendix includes exact prompts for reproducibility. The qualitative conclusion—that repository admission, retrieval, planner selection, and evolution are each attack surfaces—is credible and aligns with trends in LLM-agent supply-chain security. However, the empirical support is not yet strong enough to support the specific quantitative claims, and one section-level inconsistency directly undermines the planner experiment, a key pillar of the cross-stage argument.","major_comments":[{"comment":"The planning configuration is internally inconsistent. §5.3 states 'Planner experiments employ Gemini 3.1 pro as the planning model,' while §6.3 states the baseline planner 'uses Gemini 1.5 Flash in a zero-shot configuration.' The defended condition in §6.3 does not specify its model. If the baseline is Gemini 1.5 Flash and the defended condition is Gemini 3.1 Pro, then the large reductions in Table 4 (e.g., Fake Recommendation 45.64%→8.72%, Misleading Description 27.27%→0.00%) are confounded by model capability and cannot be attributed to the proposed Metadata Validation Layer. If both conditions actually use the same model, then one of the two sections is inaccurate, which harms reproducibility. The authors must name the model used in each condition and, ideally, report both conditions under both models or otherwise show that the defense effect is not a model effect.","section":"§5.3 vs §6.3, Table 4"},{"comment":"All reported metrics are single-point estimates with no sample sizes, confidence intervals, standard deviations, or significance tests. Table 3 is based on 440 queries, but Tables 2, 4, 5, and 6 do not state N. Attack generation itself is described as including 'LLM-assisted adversarial optimization' (§3.4) and automatic transformations (§5.2), which likely introduce stochasticity; without intervals, differences such as Semantic Camouflage Defended ASR 9.82% vs. Trigger Poisoning 17.50% cannot be interpreted. Additionally, the pipeline uses several free thresholds—semantic diversity filtering at 85% similarity, retrieval Top-50→Top-5, evolution behavioral consistency threshold—and no sensitivity analysis is provided for any of them. The cross-stage comparison at the core of the paper rests on these point estimates, so the empirical section should report per-query or per-instance variabil","section":"Tables 2–6 and §5.4"},{"comment":"The attacks and the defenses are instantiated from the same taxonomy: the defense components in §4.2–4.6 detect the specific transformations used to generate attacks in §5.2 (e.g., retrieval defenses target Sybil publication, metadata deception, and permission inflation; the admission validator checks exactly the consistency criteria used to define malicious skills). The paper itself acknowledges in §7 that this is 'a controlled evaluation environment rather than a production deployment.' The qualitative conclusion that multiple lifecycle stages are attack surfaces can survive this circularity, but the quantitative ASR and defense-success figures should be presented as within-taxonomy estimates, not as generalizable rates. A stronger version would evaluate against a held-out or independently sourced set of attacks, or at least ablate the defense rules against attack variants that are not","section":"§3.4, §4, §5.2, §7"}],"minor_comments":[{"comment":"The keyword line reads 'Keywords: Yes, you must specify some keywords.' This is a placeholder that should be replaced with actual keywords.","section":"Abstract/Keywords"},{"comment":"The experimental setup for repository admission mentions an 'undefended baseline' in addition to the rule-based and hybrid pipelines, but Table 2 has no undefended row. Either add the baseline (presumably MAR=100%) or state that it is omitted by construction.","section":"Table 2 / §5.3"},{"comment":"There is a typo: 'the planner validator performs three consistency checks., First,'—the period before 'First' is misplaced.","section":"§4.3"},{"comment":"'as summarized in in Figure 1' repeats 'in'. Also 'T rust Boundary' contains an odd space in the figure/caption text that should be corrected.","section":"§3.4 / Figure 1 caption"},{"comment":"The OWASP reference is typeset with a spurious space ('OW ASP' in two places); other references use inconsistent arXiv stylistic conventions (e.g., 'arXiv' vs. 'arXiv preprint').","section":"References"},{"comment":"Table 3 labels the third metric 'Avg. Malicious'; the text defines it as 'average number of malicious skills presented to the planner.' Clarify in the column header (e.g., 'Avg. Malicious in Top-5') and, if the metric is a count, note its range (0–5).","section":"§5.4 / Table 3"}],"recommendation":"major_revision","confidential_remarks":"The planner model inconsistency (§5.3 vs §6.3) is a checkable issue and should be the first thing the authors address; if it reflects a genuine model switch, the planner experiments must be rerun with matched models. The statistical-reporting gap is also important but likely fixable. I see no fundamental reason why the framework cannot be salvaged, but the current manuscript does not yet support the quantitative claims as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper gives the agent-security community something it has been missing: a single lifecycle vocabulary (repository admission → retrieval → planner → execution → evolution) and an evaluation harness that spans all five boundaries. That is a real contribution, and I expect the taxonomy to get cited. The idea that attacks occur well before execution is plausible and consistent with prior work on supply-chain and retrieval manipulation, and the framework is a sensible way to bring those fragmented threads together.\n\nThe empirical evaluation, however, is the soft underbelly. The most pressing problem is a direct internal inconsistency: Section 5.3 says planner experiments use Gemini 3.1 pro, while Section 6.3 says the baseline planner uses Gemini 1.5 Flash. That means the Table 4 improvements (Fake Recommendation dropping from 45.64% to 8.72%) are confounded by model capability, not just by the metadata validation layer. This is checkable and must be fixed — either both conditions use the same model or the comparison is meaningless. The stress-test note is right.\n\nOther issues are the usual but worth naming. No code or data are released, so the numbers (93.20% Sybil ASR, 23.0% execution ASR, etc.) cannot be reproduced or even given error bars. Attack set sizes are missing. The attacks and defenses are both generated from the same taxonomy, which makes the loop self-consistent rather than independently validated — the 92.5% MDR in the evolution experiment means little if the attacks are just the inverse of the detector's rules. The limitations section honestly concedes this is a controlled environment, but the abstract still claims a \"comprehensive\" evaluation, which overreaches. Minor: the paper contains a placeholder \"Keywords: Yes, you must specify some keywords.\" and several garbled citations/references, suggesting a rushed compile.\n\nWho is this for? Researchers working on agent-skill supply-chain security, especially those who need a shared vocabulary for comparing attacks and defenses. It is not a benchmark you should trust with its current evidence, but it is a framework worth refining.\n\nRecommendation: send to peer review. A serious referee will push the authors to fix the planner confound, release artifacts, add uncertainty quantification, and either include an independent attack set or explicitly reframe as a framework demonstration rather than a hard benchmark. As is, it should not be accepted, but desk-rejecting it would throw away a useful organizing structure.","headline":"A genuinely useful lifecycle taxonomy for agent-skill security, but every empirical claim in Section 6 should be treated as provisional — one internal inconsistency (planner model differs between baseline and defended setups) directly confounds the headline planner-defense numbers.","tokens_in":14380,"tokens_out":2389,"would_cite":true,"duration_ms":25427,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkillSec-Eval establishes that reusable agent skills are vulnerable at every lifecycle stage, not just execution, and quantifies the gaps with a 327-skill benchmark.","keywords":["LLM agents","reusable skills","agent supply chain security","lifecycle threat modeling","prompt injection","retrieval attacks","taint tracking","security benchmark"],"falsifier":"A production deployment that logs attack attempts before and after enabling these defenses would settle the generalization claim: if novel attacks outside the taxonomy succeed at rates near the undefended baselines, the benchmark is not representative; if the defended rates reproduce, it is.","tokens_in":13549,"feed_emoji":"🛡️","tokens_out":5963,"duration_ms":55139,"temperature":0.7,"pith_summary":"This paper tries to establish that reusable agent skills — packaged capabilities that let LLM agents call tools and workflows — are not just threatened at runtime. It decomposes the skill's life into distinct trust boundaries and shows that each one can be attacked: malicious skills slip through repository checks, retrieval can be gamed by Sybil publication and keyword stuffing, planners trust fabricated recommendations, runtime monitors miss paraphrased exfiltration, and updates inherit stale trust. To make this measurable, the paper introduces SkillSec-Eval, a benchmark built from 327 real-world skills, and reports attack success rates and defense scores per stage. The point is that defending agents requires lifecycle-aware security, not just prompt-injection or runtime filtering; the paper itself notes the setting is a controlled environment rather than a production deployment.","feed_headline":"Attacks hit reusable AI-agent skills at every lifecycle stage","feed_subtitle":"A 327-skill benchmark shows security must be enforced from repository to update, not just at runtime.","key_machinery":"The carrying mechanism is the lifecycle decomposition of a skill into six stages — authoring, storage, retrieval, selection, execution, evolution — each treated as a trust boundary, and its instantiation in the SkillSec-Eval framework as five operational components. The framework's key novelty is evaluating attacks at the stage where the violated trust assumption lives, with isolated defenses: a hybrid admission controller with LLM semantic validation, retrieval validation (diversity filtering, metadata consistency, permission justification), planner-side metadata sanitization, a runtime monitor with dynamic taint tracking and policy enforcement, and update revalidation via provenance checks","core_discovery":"The central claim is that vulnerabilities arise at multiple lifecycle stages beyond execution, and that a lifecycle-aware evaluation can systematically expose them. Defining a skill as a tuple of metadata, workflow, permissions, tools, and provenance, the paper organizes attacks into repository admission, semantic retrieval, planner selection, runtime execution, and evolution. Its benchmark shows undefended retrieval succumbs to Sybil publication 93.20% of the time, planner fake-recommendation attacks succeed at 45.64%, and even with dynamic taint tracking and policy enforcement, 23.0% of runtime attacks still reach the privileged sink because LLM paraphrase destroys string-based taint. Defe","pith_inferences":["Editorial inference: the same lifecycle lens should apply to MCP servers and plugin marketplaces, suggesting that LLM-ecosystem supply-chain defenses need semantic validation analogous to code signing.","Editorial inference: the 23% residual runtime attack success rate implies that taint tracking for LLM workflows needs semantic information-flow models, such as paraphrase-resistant embeddings, rather than string matching.","Editorial inference: because planner attacks exploit social-proof metadata, agent frameworks should treat all reputation signals as untrusted input and validate them cryptographically.","Editorial inference: a concrete extension would be to run the same attack families against different embedding models and planners to test whether the effectiveness ordering of these defenses transfers across components."],"forward_implications":["Repository admission needs LLM-based semantic validation, not just signatures and rules; a hybrid defense reduced the malicious admission rate from 52.9% to 7.9%.","Semantic retrieval is a major attack surface: Sybil publication put a malicious skill in the top five for 93.20% of queries, and post-retrieval filtering lowered this to 26.59% but not to zero.","Planner attacks are mostly metadata persuasion: fake recommendations succeeded 45.64% at baseline and dropped to 8.72% after sanitization, while prompt injection was surprisingly weak at 4.69%.","Runtime monitoring is a necessary backstop but not sufficient: it blocked 87.0% of policy violations yet allowed 23.0% of attacks because paraphrase defeats string-based taint tracking.","Treating skill updates as new admission events detects 92.5% of malicious updates but rejects 37.0% of benign ones, revealing a trade-off between evolution security and developer friction."],"fun_headline_variants":["SkillSec-Eval: Vulnerabilities emerge in every AI-agent skill stage","Benchmark: AI-agent skills are insecure at all 5 lifecycle stages","Skill attacks go beyond runtime: 327-skills lifecycle benchmark","Lifecycle-wide security failures in reusable AI-agent skills","New benchmark: AI-agent skill security fails at every lifecycle step"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 327 filtered benign skills plus the synthetic attack transformations are representative of real skill-marketplace threats, which the paper itself concedes in Section 7 is a controlled evaluation environment rather than a production deployment.","fun_headline_variants_meta":{"raw":{"variants":["SkillSec-Eval: Vulnerabilities emerge in every AI-agent skill stage","Benchmark: AI-agent skills are insecure at all 5 lifecycle stages","Skill attacks go beyond runtime: 327-skills lifecycle benchmark","Lifecycle-wide security failures in reusable AI-agent skills","New benchmark: AI-agent skill security fails at every lifecycle step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3523,"prompt_tokens":656,"completion_tokens":2867,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":2779}},"tokens_in":400,"tokens_out":2867,"duration_ms":21578,"temperature":1.0,"reasoning_tokens":2779,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:05:40.232452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A production deployment that logs attack attempts before and after enabling these defenses would settle the generalization claim: if novel attacks outside the taxonomy succeed at rates near the undefended baselines, the benchmark is not representative; if the defended rates reproduce, it is.","supporting_citations":[],"review_version":1}