Cryptography and Security · arXiv:2610.08098 v2 · submitted 6 Oct 2026
Surviving the Router: Optimizing Skill Injections for Retrieval and Execution
The paper says
“AI agents increasingly rely on modular third-party "skills" that are dynamically selected by skill routers to execute complex tasks.”
more of the abstract
While recent studies highlight the threat of prompt injections embedded in these skills, existing evaluations often assume settings where the malicious skill is already selected for execution. We show that this assumption can substantially overestimate attack success. In realistic multi-skill environments, injected skills must first compete for retrieval, reducing the effective attack success rate (ASR) of existing injections by 87-97%. To address this limitation, we introduce CORSA (Cluster Optimization for Router-Aware Skill Attacks), a router-aware attack that optimizes skill injections for both retrieval and execution across clusters of related tasks. We evaluate skill injection attacks under router-managed multi-skill settings by extending the benchmark introduced by SkillRouter with eight malicious payload categories. CORSA uses successive optimization stages to first improve retrieval and then optimize end-to-end attack success, while we evaluate user utility and injection naturalism separately. Our experiments show that CORSA substantially improves both retrieval and end-to-end attack success over existing skill injections while preserving user utility, and that the resulting attacks transfer across different router architectures and LLM backbones.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 7 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Router-aware skill injection is a real and under-studied problem, and CORSA is a solid first attack that makes the evaluation point stick; the headline numbers need error bars, a router-free baseline, and a decision on top-k before I'd trust the absolute magnitudes. This line is wrong
3 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 3.1, Eq. (1), Section 4.3
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
Section 3.1 defines the router as returning the k highest-scoring candidates and defines attack success as membership in that set, S' in R_k(t,D); Eq. (1), however, sets Hit(t,S) = 1[rank_R(t,S,D) = 1], and Section 4.3 requires rank exactly 1 for both Hit@1 and ASR.
more of objection 1
The paper never fixes k in the threat model, never reports sensitivity to k, and never specifies how the agent selects among the returned candidates. This mismatch is load-bearing: with 51 of the 75 benchmark tasks requiring multiple skills, multi-candidate retrieval is the natural setting, and a skill ranked 2-10 remains a plausible candidate for the agent to load. All headline quantities, namely the 87-97% ASR reduction, the 11.0-to-32.3 Hit@1 gain, and the 10.0-to-22.0 ASR gain, are computed under this silent k=1 assumption; under a top-k-then-select protocol the reduction could shrink, and CORSA's advantage could narrow if it largely promotes already-retrieved skills from rank 2-10 to rank 1. I request a k-sensitivity analysis (Hit@k for k in {1,3,5,10}) and an ASR variant that counts success whenever the injected skill is in the top-k and the agent selects it under a stated selection policy.
The paper says, at Section 5.1, Tables 1-3
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
All headline numbers are single-run measurements without seeds or error bars, although GEPA is a stochastic evolutionary search. The main gaps (11.0 vs 32.3 Hit@1; 10.0 vs 22.0 ASR) are large, but the utility and naturalism comparisons (38.3 vs 38.8 for utility; 0.10 vs 0.17 for naturalism) are small enough to be indistinguishable from noise without variance information.
more of objection 2
Please report means and standard deviations over at least three independent optimization runs for the main GPT-5.4 configuration and, where feasible, for the DeepSeek-V4-Pro attacker, and state which qualitative claims are robust across seeds.
The paper says, at Section 5.1 vs Section 5.2, Eqs. (3)-(4)
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The headline ASR of 22.0% is measured on the same 75 tasks whose rewards Eqs. (3)-(4) explicitly maximize (Hit and Hit times Exec over each cluster), so it is in part an optimized objective value rather than an independent prediction.
more of objection 3
To the paper's credit, Table 1 labels this set as In-domain and Section 5.2 supplies independent paraphrase and synthetic evaluations, where CORSA's ASR drops to 11.5% and 5.4% and its utility drops to 15.8% and 11.8%, the former below the SkillJect paraphrase utility of 25.0%. The abstract and Section 5.1, however, present the in-domain numbers as the principal evidence, and the utility-preservation claim in Section 5.1 holds only in-domain. I recommend re-anchoring the presentation so that in-domain numbers are labeled as optimization results and the independent generalization and transfer numbers (Tables 1-3) carry the effectiveness claims.
7 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 7
The 87-97% reduction range is not tied to a specific comparison: in-domain SkillJect ASR (10.0%) against the prior 100%-selection assumption is a 90% reduction, and the larger endpoint appears to come from the paraphrase and synthetic sets (Table 1: 1.0% and 0.2% ASR). Please state explicitly which comparison yields each endpoint.
The paper says, at Section 4.4
Pith says: minor objection 2 of 7
The GEPA optimization budget (population size, number of iterations, reflection model, and approximate cost per cluster) is not reported. Since GEPA is the core of CORSA, these values are needed to judge the attack's cost and the sensitivity of the results to the optimization budget.
show 5 more
The paper says, at Section 4.2
Pith says: minor objection 3 of 7
The eight-cluster partition is generated by GPT-5.4; please provide the clustering prompt and the task-to-cluster assignment, or confirm that they are included in the code release, since the cluster definition is a free design choice that can affect the headline results.
The paper says, at Figure 3
Pith says: minor objection 4 of 7
The benign-skill utility of 38.7% is low, so 'preserving user utility' should be read as matching the baseline in-domain rather than maintaining a high absolute completion rate; the paper should state this explicitly, and likewise note that CORSA's naturalism (0.17) is higher than SkillJect's (0.10) but far below the benign skill's 0.67.
The paper says, at Throughout
Pith says: minor objection 5 of 7
The baseline name is rendered inconsistently as SKILLJECT, SKILLJECT, and SkillJect across the abstract, figures, tables, and text; please unify the name, and consistently distinguish SKILL-INJECT (the benchmark) from SKILLJECT (the attack).
The paper says, at Section 6, Table 5
Pith says: minor objection 6 of 7
The defense evaluation is restricted to a single indirect payload, as the text discloses, but the statements 'Both static scanners miss all attack packages' and the reported recall values should be explicitly labeled as payload-specific, since detection rates may differ across the other seven harm categories.
The paper says, at Appendix B, Table 8
Pith says: minor objection 7 of 7
SkillJect's Hit@1 and ASR increase from 11.0%/10.0% in the original set to 16.0%/15.0% in the distractor set, which is counterintuitive; please add a sentence explaining how the hard-tier pool composition changes retrieval scores for the injected skill.
What would settle it
Re-run the main SkillJect-vs-CORSA comparison from Section 5.1 on the same clusters, payloads, routers, and optimized skills, but define retrieval as rank<=k for k in {1,3,5,10} and execute the agent under two selection policies (always first candidate; uniform over the returned top-k). Report Hit@k and ASR@k. If SkillJect's ASR@5 approaches CORSA's or the 87-97% drop falls below roughly 50%, the top-1 metric drives the headline; if the drop and relative gain persist at larger k, the concern is resolved.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 7 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Short version: this paper earns a serious look. Prior skill-injection work mostly assumes the poisoned skill is already selected. The authors show that once a router with roughly 2,000 competing skills is in the loop, a stock SkillJect injection drops to 11% Hit@1 / 10% ASR, and a two-stage cluster optimization (retrieve first, then execute) raises those to 32.3% / 22.0% while leaving benign utility flat. The empirical package is broader than most: eight payload categories, eight clusters, five routers, transfer across victim models and scaffolds, and an honest defense evaluation. The central correction—routing must be part of attack success—is important, and it lands.
Soft spots, in order of severity. The headline 22.0% ASR is measured on the same 75 tasks that the attack optimized on, so it is partly an optimized objective value, not an independent prediction. The paraphrase and synthetic test sets give it some teeth, but the in-domain numbers should not be read as deployment expectations. The 87-97% reduction claim is stated without a router-free baseline ASR in the paper; I can infer it from SkillJect's numbers elsewhere, but it is not reported or cited here, so the magnitude cannot be checked. There are no seeds or error bars. And the optimization budget (GEPA iterations, population, reflection model) is not in the paper text, only promised in the repo.
On the stress-test concern: yes, there is a real mismatch. The threat model defines retrieval as membership in the router's top-k, but Eq. 1 and all metrics use rank exactly 1. The missing k-sensitivity is a legitimate request, and the absolute ASRs are conditioned on that choice. But I don't think it breaks the paper. The SkillJect-vs-CORSA comparison is under identical metrics, so the relative conclusion survives. If anything, rank-1 is a conservative proxy: if deployed agents often take the top candidate, the numbers are close to reality, and if they browse, both methods' retrieval would improve, not just CORSA. Still, the paper should either justify top-1 as the deployment assumption or include a rank sweep.
Who this is for: anyone working on agent-skill security, skill marketplaces, or retrieval-based prompt injection. It deserves a serious referee; the right outcome is probably major revision with the missing baseline, seeds, k-sensitivity, and GEPA budgets, not rejection.
The strongest objection
The paper's own threat model defines retrieval as membership in the router's top-k output: Section 3.1 states the router 'returns the k highest-scoring candidates' and defines attack success as S' in R_k(t,D). The evaluation, however, silently takes k=1. Eq. (1) defines Hit(t,S)=1[rank_R(t,S,D)=1], and Section 4.3 equates Hit@1 with 'the attack passes retrieval,' while ASR additionally requires rank exactly 1. No sensitivity over k is reported, and no agent-selection policy among returned candidates is specified. This is not a cosmetic mismatch: 51 of the 75 benchmark prompts require multiple skills, so requiring rank 1 is especially unnatural for those tasks. If deployed routers surface several candidates and the agent selects among them, SkillJect's retrieval would be higher than the reported 11.0%, the headline 87-97% reduction would shrink, and CORSA's advantage—which may consist largely of promoting an already-retrieved skill from rank 2-10 to rank 1—could narrow. All retrieval and ASR magnitudes in Figure 3, Tables 1-4, and the abstract are therefore conditioned on an untested k=1 assumption.
The reader's conditional verdict already identifies the top-k versus rank-1 mismatch, and I agree that it is the most load-bearing concern because it sits inside the definition of the attack objective itself. Eq. (1) is used both as the Stage A/B optimization reward and as the evaluation metric, so the k=1 assumption is baked into CORSA's construction, not just into the reported numbers. The missing router-free baseline is a separate, real gap: without it, the '87-97% reduction' in the abstract is not directly substantiated, since only the routed SkillJect ASR (10.0%) is tabulated. I treat that as secondary because it mainly affects the framing of the drop, whereas the top-k issue affects both methods' absolute and relative scores. The paper otherwise has independent support: the cross-model, cross-scaffold, and cross-router transfer experiments are genuine evidence, and the reproducibility statement promises code and configurations; those do not, however, resolve the k sensitivity. The central qualitative claim—that routing constrains naive injections and that router-aware optimization helps—is likely to survive a k-sensitivity check, but the specific magnitudes (87-97%, 11.0 to 32.3 Hit@1, 10.0 to 22.0 ASR) are not yet supported for the stated threat model. The verdict should therefore remain conditional rather than being upgraded to acceptance.
The referee's report
If the findings hold, the paper makes a useful and timely contribution: it identifies a real evaluation gap, namely that prior skill-injection work largely assumes the poisoned skill is already selected, and it supplies a method that explicitly optimizes for competitive retrieval as well as execution. The manuscript is honest in ways that strengthen it: Table 1 labels the in-domain set as such and shows substantial degradation on paraphrase and synthetic tasks; Section 6 reports defense results in which scanners catch most but not all packages; and the transfer experiments cover five router configurations, five victim models, and two scaffolds. The promised code release and the falsifiable quantitative claim (an 87-97% reduction of effective ASR under routing) are appropriate for an experimental security paper. That said, the stress-test concern lands: the top-k definition in the threat model conflicts with the rank-1 evaluation metrics, so the magnitude of every headline number is conditioned on an untested assumption.
Pith's reading of the paper (a machine's summary, not the authors')
Existing skill-injection attacks assume the poisoned skill is already handed to the agent. This paper argues that in realistic skill marketplaces the injected skill must first win retrieval against thousands of benign competitors, and shows that ordinary injections lose that race so often that their effective attack success rate falls by 87-97%. To survive the router, the paper introduces CORSA, a two-stage optimization that rewrites a skill's name, description, and instructions so the skill ranks first for a whole cluster of related tasks and then executes its payload. In their experiments CORSA raises top-rank retrieval from 11.0% to 32.3% and end-to-end attack success from 10.0% to 22.0% while keeping the user's original task completed roughly as often (38.8% vs 38.3%). The claim that matters is that attack evaluations for skill-based agents should include competitive retrieval, because ignoring the router dramatically overstates how dangerous injected skills are.
What carries the argument. The machinery is the injected skill file itself, treated as an editable tuple of name, description, and instruction body plus a fixed payload, and the two-stage reward it is optimized against. Stage A maximizes $\text{Hit}(t,S) = \mathbf{1}[\text{rank}_R(t,S,\mathcal{D}) = 1]$ summed over a cluster of related tasks, so the skill is tuned purely to win retrieval; Stage B, seeded by the Stage A winner, maximizes $\text{Hit}(t,S) \cdot \text{Exec}(t,S)$, requiring both top-rank retrieval and execution of the payload's helper script. This staging separates forces that pull the text in opposite directions: retrieval wants task-relevant content, while execution wants clear instructions that invoke the payload. The cluster is the unit of generalization: one published skill is meant to compromise many downstream users, and optimizing over related tasks rather than individual queries avoids overfitting while staying relevant.
Core claim. The paper's central claim is that the router, not the executor, is the gate that decides whether a skill injection works: an injected skill must first be retrieved from a competitive pool of thousands of benign skills before its payload can even be attempted. Measuring the existing router-unaware approach under this realistic condition, the authors find its effective attack success rate falls by 87-97%, with top-rank retrieval stuck near 11% and end-to-end success near 10%. CORSA is their response: a two-stage optimizer that rewrites a single poisoned skill's name, description, and instructions so that it ranks first for a cluster of 3-13 related tasks, then, starting from that retrieval-winning candidate, keeps it retrieved while maximizing the chance the agent executes the payload. Across eight harm-category payloads and two attacker models, CORSA lifts Hit@1 to 32.3% and ASR to 22.0%, at utility essentially unchanged (38.8% vs 38.3%) and with higher judged naturalism. The authors further show that the optimized skills generalize to paraphrased and synthetic task variants, transfer to unseen victim LLMs and a second agent scaffold, and transfer asymmetrically across five router architectures.
The premise it rests on. The load-bearing premise is that retrieval success means being ranked exactly first, although the stated threat model lets the router return a top-k list and the agent choose from it; if agents in practice browse several candidates, the measured 87-97% drop and CORSA's advantage would both shrink.
If the paper is right (Pith's inferences, not the authors')
Skill-injection evaluations that hand the poisoned skill to the agent overstate attack success by a large factor; realistic benchmarks must include competitive retrieval.
Optimizing for retrieval first and execution second recovers a large share of that lost success, tripling top-rank retrieval and more than doubling end-to-end ASR.
Freezing the optimized skill without re-optimization still yields ASR of 11.5% on paraphrased tasks and 5.4% on new same-cluster tasks, while the router-unaware baseline collapses below 1%.
Optimized injections transfer to unseen victim models, to a second agent scaffold, and across router architectures, so changing any single system component does not by itself neutralize the attack.
Static skill scanners flag none of the optimized packages, and LLM-augmented scanners miss at least five of 24, so router-aware attacks remain a live supply-chain risk.
Going beyond the paper: if deployed assistants present users with several candidate skills, the effective attack surface is probably larger than Hit@1 suggests, because the agent may pick the injected skill even when it is not ranked first; a top-k evaluation would map that additional exposure.
Going beyond the paper: the same two-stage objective could be used defensively, e.g., to red-team a router before release by generating cluster-optimized probes and measuring how often they reach rank one.
Going beyond the paper: because the optimization is text-only and requires no model weights, any future router built on the same embedding or reranking families should be tested against similarly optimized injections, not just against hand-written ones.
Going beyond the paper: cluster-level optimization predicts that marketplaces with many near-duplicate skills are the highest-value target, since one published poisoned skill can capture an entire family of tasks; tracking clusters, not just individual skills, may help defenders prioritize review.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
AI agents increasingly rely on modular third-party "skills" that are dynamically selected by skill routers to execute complex tasks. While recent studies highlight the threat of prompt injections embedded in these skills, existing evaluations often assume settings where the malicious skill is already selected for execution. We show that this assumption can substantially overestimate attack success. In realistic multi-skill environments, injected skills must first compete for retrieval, reducing the effective attack success rate (ASR) of existing injections by 87-97%. To address this limitation, we introduce CORSA (Cluster Optimization for Router-Aware Skill Attacks), a router-aware attack that optimizes skill injections for both retrieval and execution across clusters of related tasks. We evaluate skill injection attacks under router-managed multi-skill settings by extending the benchmark introduced by SkillRouter with eight malicious payload categories. CORSA uses successive optimization stages to first improve retrieval and then optimize end-to-end attack success, while we evaluate user utility and injection naturalism separately. Our experiments show that CORSA substantially improves both retrieval and end-to-end attack success over existing skill injections while preserving user utility, and that the resulting attacks transfer across different router architectures and LLM backbones.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-07
The authors' main claim, as the classifier picked it out 0.60
“Our experiments show that CORSA substantially improves both retrieval and end-to-end attack success over existing skill injections while preserving user utility, and that the resulting attacks transfer across different router architectures and LLM backbones.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.22 · human participants 0.05 · controlled comparison 0.80 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.97 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| AI agents increasingly rely on modular third-party "skills" that are dynamically selected by skill routers to execute complex tasks. | background 1.00 | 0.04 | 0.02 | asserted 0.99 |
| While recent studies highlight the threat of prompt injections embedded in these skills, existing evaluations often assume settings where the malicious skill is already selected for execution. | gap 0.90 | 0.05 | 0.04 | asserted 0.93 |
| We show that this assumption can substantially overestimate attack success. | result 0.58 | 0.76 | 0.23 | asserted 0.95 |
| In realistic multi-skill environments, injected skills must first compete for retrieval, reducing the effective attack success rate (ASR) of existing injections by 87-97%. | result 0.99 | 0.90 | 0.48 | asserted 1.00 |
| To address this limitation, we introduce CORSA (Cluster Optimization for Router-Aware Skill Attacks), a router-aware attack that optimizes skill injections for both retrieval and execution across clusters of related tasks. | contribution 0.87 | 0.89 | 0.12 | asserted 0.99 |
| We evaluate skill injection attacks under router-managed multi-skill settings by extending the benchmark introduced by SkillRouter with eight malicious payload categories. | method 0.91 | 0.91 | 0.03 | asserted 0.99 |
| CORSA uses successive optimization stages to first improve retrieval and then optimize end-to-end attack success, while we evaluate user utility and injection naturalism separately. | method 0.99 | 0.91 | 0.10 | asserted 0.94 |
| Our experiments show that CORSA substantially improves both retrieval and end-to-end attack success over existing skill injections while preserving user utility, and that the resulting attacks transfer across different router architectures and LLM backbones. | result 1.00 | 0.96 | 0.64 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-07: 1 request, 6,259 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
- PROVISO: Provenance-Informed Action Selection for Secure LLM Agents cs.AI · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment