{"id":"6b0832f5-b779-45ff-88f6-20a14ebe0661","arxiv_id":"2607.25619","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A regex prefilter plus a snippet-only LLM judge detects malicious skill files with F1=0.817 and 1.13% false positives while using 77% fewer LLM tokens than full-file screening.","lead":"SkillGate is a security gateway that checks AI coding-agent skill files for malicious instructions before they are installed, using a fast pattern filter followed by an AI judge that looks only at suspicious snippets. On a public benchmark it catches most attacks with few false alarms while cutting AI-processing costs by about three-quarters, which could make pre-install screening affordable for everyday developer workflows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prefilter blind spot: 10% of benchmark malicious files never reach the LLM judge, so real-world recall is capped by regex coverage; this is the load-bearing external-validity risk, not the snippet window.","rationale":"The paper's central benchmark claim is internally consistent: the F1, FPR, MCC, token, and latency numbers are well-supported by the experimental setup and ablations. I considered whether the AUPRC computation over prefilter-safe files (score 0) introduces a tie-breaking artifact; standard average-precision calculation groups tied scores, so this is not a strong concern. The load-bearing issue is the prefilter skip path. The 15/150 benchmark malicious files that the regex misses are never adjudicated by the LLM, and the paper's own ablation exposes this as a hard recall ceiling. This matters because the cost-saving design trusts the regex gate completely: files with zero pattern hits are treated as safe without any LLM oversight. The reader identified the same general external-validity gap, but the sharper formulation is not just that hand-crafted samples may not transfer; it is that the architecture structurally cannot detect any skill file that avoids the 530 patterns. The paper does disclose this in Section VI, and the benchmark-scoped claim 'On SkillsBench' is accurate. However, the conclusion that SkillGate makes 'inline malicious-skill detection both accurate and inexpensive enough to run at install time' overreaches without an evasion check. I therefore recommend CONDITIONAL acceptance: require the proposed evasion experiment or explicitly narrow the claim to the benchmark and threat model. This is not a rejection; the work is a solid engineering contribution with transparent evaluation, but the security claim hinges on a blind spot that should be empirically bounded.","tokens_in":16642,"tokens_out":16229,"duration_ms":172111,"concrete_test":"Run SkillGate unchanged on an evasion set built from the 135 prefilter-matched malicious SkillsBench files: apply minimal mutations so none triggers the 530 patterns (e.g., split quoted commands, replace 'curl' with synonyms, base64-encode the payload, insert zero-width characters), and include the 15 already-unmatched files. Measure end-to-end recall. If recall drops toward 0 on mutated samples, the prefilter ceiling is the binding constraint and the RQ1 claim should be restated as conditional on regex coverage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The default pipeline (Section III.B–C) classifies zero-hit files as SAFE with no LLM call. Table VI (B1) shows the regex prefilter alone matches only 135/150 malicious SkillsBench files (recall ceiling 0.900); the 15 non-matched malicious files are never seen by the LLM judge, no matter how capable the judge is. The RQ1 recall (0.769) is further reduced by the judge overriding genuine hits, but the 0.900 ceiling is structural: a malicious skill that avoids all 530 patterns bypasses the system entirely. This is not only an external-validity limitation; it is a testable property of the architecture. Because the ruleset is open-source and the threat model (Section II.D) excludes only adaptive adversaries, even a non-adaptive attacker using tokenization, encoding, or synonym substitution can move a file below the regex threshold. Section VI concedes the benchmark's attack vocabulary may not cover such obfuscation and that live ClawHavoc samples are unavailable. The headline F1/AUPRC therefore depends on prefilter coverage of the benchmark's specific attack taxonomy, and the skip path prevents the LLM from compensating.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SkillGate, a hybrid detection gateway for malicious AI agent skill files. A 530-pattern regex prefilter (428 MITRE ATT&CK-derived + 102 Sigma rules) scans skill-file content; files with zero hits are classified SAFE without an LLM call, while files with hits are escalated to an LLM judge that sees only the matched snippet windows (default cw=500, ms=20), not the full file. The system is evaluated on SkillsBench (n=1,650, 9.1% malicious) against ClawVet and SkillScanner, with ablations (pure regex, full-file LLM, snippet-budget variants). The headline results are F1=0.817, recall=0.769, FPR=1.13%, MCC=0.803, AUPRC=0.830, 76.9% LLM-token reduction versus full-file screening, and weighted average latency ~818ms. The paper also analyzes false-positive categories and reports three-run means with standard deviations.","tokens_in":16965,"tokens_out":5312,"duration_ms":50646,"significance":"If the reported benchmark results transfer to practice, SkillGate addresses a real and growing supply-chain attack surface with a cost profile that makes install-time screening plausible. The work has several genuine strengths: the implementation and ruleset are open source; the confidence threshold and context-window parameters were pre-specified on a held-out development set; baselines are evaluated at oracle-optimal thresholds, which is conservative with respect to SkillGate; and the ablation study isolates the contribution of each pipeline stage. The main risk to significance is external validity: SkillsBench is a single benchmark whose 150 malicious files are hand-crafted attack exemplars, and the prefilter's zero-hit fast path imposes a structural recall ceiling that the LLM judge cannot bypass. These limitations are acknowledged honestly in Section VI, but they are load-bearing for the paper's broader 'deployable defense' claim.","major_comments":[{"comment":"The prefilter blind spot is a load-bearing limitation, not just a routine external-validity caveat. Files with zero regex hits are classified SAFE with no LLM call (Section III.B), so the 15 malicious SkillsBench files that fail to match any of the 530 patterns are structurally invisible to the judge. Table VI (B1) shows the prefilter recall ceiling is 0.900 (135/150); the RQ1 recall of 0.769 is therefore capped by regex coverage before the judge is even consulted. The paper should quantify how many of these 15 zero-hit malicious files the LLM would catch if they were escalated (e.g., by running the full-file judge on this subset), and discuss a concrete fallback policy — for example, escalating zero-hit files with high entropy, encoding markers, or suspicious metadata. The B2 ablation (full-file LLM, recall 0.804) suggests the LLM alone cannot fully recover the 15, but the overlap betwe","section":"Section III.B, Table VI (B1)"},{"comment":"The conclusion states that SkillGate is a 'practical first line of defense' and the abstract emphasizes 'deployable,' but the evaluation corpus contains only hand-crafted malicious skills sampled from eight fixed categories. The paper concedes in Section VI that live ClawHavoc samples were unavailable and that measured recall may not transfer to organically evolving campaigns. This is an honest statement, but the conclusion does not temper the deployability claim accordingly. I would ask the authors to add a robustness analysis — for example, applying simple transformations (tokenization changes, base64/hex encoding, synonym substitutions, splitting payloads across multiple code blocks) to the 150 malicious files and measuring how many fall below the prefilter threshold. Such an experiment would directly test the structural blind spot and give practitioners a realistic sense of the defen","section":"Section VI, Section VII"}],"minor_comments":[{"comment":"Typographical errors: 'realisitc' and 'deployabile' should be 'realistic' and 'deployable'. The formatting of Table VI also has missing spaces in 'SKILLGATEms=5snippets' and 'SKILLGATE(default,ms=20)'.","section":"Section V, Figure 5"},{"comment":"In Table V, the SKILLGATE row appears to have a formatting issue: '173 0 13 13' should likely read '17, 3, 0, 13, 13' (FPs = 17, block-grade = 3, critical = 0, high = 13, medium = 13). Please verify the column alignment.","section":"Table V"},{"comment":"The policy mapping in Stage 4 is ambiguous: 'SUSPICIOUS (confidence≥0.5) → QUARANTINE; SUSPICIOUS (confidence<0.5) → WARN.' Since the label SUSPICIOUS already carries a meaning, clarify how the numeric confidence interacts with the label (e.g., is SUSPICIOUS with confidence 0.4 treated differently from SAFE with confidence 0.4?).","section":"Section III.D"},{"comment":"Reference [1] is a LinkedIn post; for a journal version, consider replacing it with a peer-reviewed or at least a citable measurement study if one becomes available, or clearly mark it as a non-archival industry report. The same applies to the 'ClawHavoc' claim, which is central to the motivation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically careful for a single-benchmark study, and the pre-specified thresholds and conservative baseline comparisons are notable strengths. My main concern is the structural prefilter blind spot: 10% of benchmark malicious files never reach the LLM judge, and the paper does not analyze this subset or propose a mitigation. This is fixable within the manuscript's scope — an isolation analysis and a small robustness/evasion experiment would address it. I do not see a fatal flaw in the reported benchmark numbers, but the deployability claim in the title and abstract needs to be either tempered or supported by additional evidence. Hence major revision rather than accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, carefully-evaluated defense for skill-file attacks, and the headline results hold on the benchmark. The one thing to keep in mind is that the regex prefilter is a hard recall ceiling — 15 of the 150 malicious files never reach the LLM judge — so real-world effectiveness depends on pattern coverage more than on the snippet design.\n\nThe design contribution is genuine. A 530-pattern regex gate lets 67% of files skip the LLM entirely, and the files that do escalate send only matched snippet windows instead of full content. That combination buys a 77% token reduction while keeping F1 at 0.817 and FPR at 1.13% on SkillsBench, against static baselines that either flag half of benign content or, in SkillScanner+LLM's case, spend three times the LLM calls for notably worse recall. The evaluation is also unusually careful: three runs with standard deviations, a pre-specified threshold, oracle-optimal baseline operating points that favor the competition, a clean ablation separating prefilter-only, full-file, and snippet-budget configurations, and a threats-to-validity section that doesn't hide the hand-crafted nature of the malicious samples or the known coverage gaps. The implementation, ruleset, and harness are open-source, which makes the empirical claims independently checkable.\n\nThe stress-test note lands. Table VI (B1) puts the prefilter recall ceiling at 0.900, and the default pipeline classifies zero-hit files as SAFE without consulting the LLM. So the judge never sees 10% of the benchmark's malicious files, and anything that sidesteps the 530 patterns bypasses SkillGate entirely. That isn't just an external-validity caveat; it is an architectural property. The paper acknowledges this in Section VI, but the implication deserves more weight: the F1 and AUPRC numbers are conditional on SkillsBench's attack vocabulary, and a non-adaptive attacker using ordinary obfuscation outside that vocabulary gets a free pass. The snippet-window loss (0.769 vs 0.804 recall for full-file) is a smaller, well-understood trade-off; the prefilter blind spot is the load-bearing one.\n\nThis paper is for researchers and practitioners working on agent supply-chain security. It deserves a serious referee. The central claim — that this hybrid design achieves good detection at deployable cost on a public benchmark — is supported. The external-validity concern is real and should be pushed in revision, ideally with a sensitivity analysis over regex coverage, but it doesn't undermine the claim as stated.\n\nRecommendation: send it to peer review.","headline":"Well-executed, honestly-evaluated skill-file detector; benchmark numbers hold, but the regex prefilter's 90% recall ceiling is the real external-validity risk.","tokens_in":17430,"tokens_out":3687,"would_cite":true,"duration_ms":33576,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkillGate claims a two-stage regex-plus-LLM gateway can screen AI coding-agent skill files at install time, detecting malicious skills with F1 0.817 and a 1.13% false-positive rate while cutting LLM input tokens by 77% compared with full-fi","keywords":["agent skill files","supply-chain security","runtime screening","LLM judge","regex prefilter","false-positive rate","coding agents","install-time defense"],"falsifier":"Run the published system on a fresh collection of malicious skill files harvested from public registries after the paper's release and compare recall to the benchmark's 0.769; if recall drops materially, or if the regex prefilter's ceiling (0.900 on the benchmark) is far lower on that collection, the deployment claim is falsified. A narrower experiment: measure false positives on a documentation-heavy corpus with base64 examples—if they exceed the reported 1.13%, the FPR claim does not generalize.","tokens_in":16562,"feed_emoji":"🛡️","tokens_out":6685,"duration_ms":53901,"temperature":0.7,"pith_summary":"The paper claims that malicious skill files—Markdown instructions that coding agents load from registries—can be detected cheaply enough to screen every install. It proposes a gateway that first applies a large regex ruleset; files with no matches pass without any language-model call, and flagged files send only their matched snippet windows to a small LLM judge instead of the whole file. On a benchmark of 1,650 skill files with 9.1% malicious prevalence, the system reports F1 0.817, false-positive rate 1.13%, and 77% fewer LLM tokens than full-file review, with a weighted per-file latency around 0.8 seconds. The point is that a deployable pre-install defense is feasible where static-only scanners produce unusable false-positive rates and full-file LLM review is too slow and costly.","feed_headline":"SkillGate catches malicious skills at 1% false positives, 77% fewer tokens","feed_subtitle":"A pre-install proxy uses a regex gate plus a small LLM judge to block malicious skill files at ~0.8s per file.","key_machinery":"The load-bearing mechanism is the two-stage gate: (1) a RuleEngine of 530 regex patterns derived from a public attack-technique taxonomy and community detection rules, which returns hit positions but never a final verdict; and (2) a snippet assembler that extracts up to 20 windows of 500 characters around each match, prepends a short context preamble, and sends this bounded payload to an LLM judge that returns SAFE, SUSPICIOUS, or MALICIOUS with a confidence score. The design separates gating from adjudication: the regex decides where to look, the LLM decides what the evidence means. Because 67% of files have zero hits and bypass the model, and flagged files ship only their suspicious region","core_discovery":"The central claim is that a hybrid pipeline—a deterministic regex prefilter acting as a gate, not a classifier, plus an LLM judge that sees only the flagged snippet windows—resolves the accuracy-cost tradeoff for skill-file screening. The prefilter alone catches 135 of 150 malicious files but flags 27% of benign files; adding the judge cuts false positives to 1.13% at recall 0.769. Compared with two prior static scanners, the system reports a 5–6x higher threshold-independent precision-recall area, and compared with sending every file in full to the same judge, it uses 77% fewer tokens while reaching similar recall. The paper frames this as closing the gap at the agent-instruction layer, bef","pith_inferences":["The gate-decides-where, judge-decides-what split should transfer to other LLM context channels—runtime tool responses, fetched documentation, even email—where adversarial text can arrive as data but act as instructions.","The pipeline's ceiling is set by the 530-pattern prefilter: any attack whose text contains none of those patterns is invisible to the system no matter how good the judge is. A natural next study is measuring recall against obfuscation techniques designed to avoid those patterns.","Token savings will vary with file length: for very short skill files the fixed context preamble dominates each request, so deployments with mostly tiny skills will see smaller cost reductions than the 77% reported here.","Because the benchmark's malicious files are hand-crafted exemplars, the strongest test of the paper's claim is an in-the-wild evaluation on freshly observed registry attacks, which would also reveal whether the reported false-positive categories shift."],"forward_implications":["A coding agent wired through the proxy can screen every skill installation with roughly 0.8 seconds of weighted overhead, making inline pre-install defense feasible within a normal registry fetch.","Developers would see roughly 17 false positives out of 1,500 benign skills, versus hundreds for static-only scanners, so the tool does not erode trust with spurious blocks.","Operators can lower the confidence threshold to about 0.14 to push recall to 0.824 while keeping false positives near 1.3%, without per-deployment tuning.","Sending the full file to the same judge instead of snippet windows adds only about 3.5 percentage points of recall but roughly triples the number of LLM calls and raises false positives, so the snippet design is the better default."],"fun_headline_variants":["Regex gate plus LLM judge spots malicious skills with 1% false positives","SkillGate filters skill files cheaply: 77% fewer LLM tokens, 1% FPR","Hybrid prefilter and LLM judge: 0.817 F1 at 1.13% false positives","SkillGate: regex gate skips safe skills, LLM judge checks only flagged snippets","SkillGate cuts token cost 77% while matching full-file recall on skill malware"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the benchmark's hand-crafted malicious skills represent real attacks; if real-world malicious skills are more varied or better obfuscated, the reported recall and false-positive rate will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Regex gate plus LLM judge spots malicious skills with 1% false positives","SkillGate filters skill files cheaply: 77% fewer LLM tokens, 1% FPR","Hybrid prefilter and LLM judge: 0.817 F1 at 1.13% false positives","SkillGate: regex gate skips safe skills, LLM judge checks only flagged snippets","SkillGate cuts token cost 77% while matching full-file recall on skill malware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3569,"prompt_tokens":863,"completion_tokens":2706,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":2588}},"tokens_in":607,"tokens_out":2706,"duration_ms":16377,"temperature":1.0,"reasoning_tokens":2588,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:53:48.607276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published system on a fresh collection of malicious skill files harvested from public registries after the paper's release and compare recall to the benchmark's 0.769; if recall drops materially, or if the regex prefilter's ceiling (0.900 on the benchmark) is far lower on that collection, the deployment claim is falsified. A narrower experiment: measure false positives on a documentation-heavy corpus with base64 examples—if they exceed the reported 1.13%, the FPR claim does not generalize.","supporting_citations":[],"review_version":1}