{"id":"d7845632-ee1a-4c17-a556-bb952e533602","arxiv_id":"2610.06401","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RAISED trains LLM agents to reproduce their clean behavior on injected tool outputs, cutting AgentDojo attack success to about one percent on Gemma without the utility losses of SFT and preference-based defenses.","lead":"A new training method, RAISED, teaches tool-using AI agents to ignore malicious instructions hidden inside tool outputs while still following the legitimate ones they need to finish their task. On standard security benchmarks it blocks most attacks with almost no drop in ordinary usefulness, where earlier training-based defenses typically trade capability for safety.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Invariance learned on self-authored injections is tested on three benchmark corpora but not on adversarial spans outside those distributions; re-attacking the released checkpoints with independently rewritten static injections would settle whether ASR ~1% is channel-level or corpus-level.","rationale":"Good-faith reading: the central engineering claim is that training a student to match the frozen base model's clean-context next-token distribution on both clean and injected versions of its own trajectories (Eq. 3) realizes the invariance target of Eq. (1), B-theta(c-tilde) congruent B-theta0(c), thereby reducing a static attacker's success while preserving benign utility. That claim is corroborated where it is directly testable: three independently authored benchmark corpora, three model families, three seeds, and a controlled ablation (Section E) holding data and inputs fixed, in which self-distillation dominates SFT and DPO on utility at comparable ASR. The KL-drift figures (0.0010 vs 0.044/0.029, Fig. 2; longer-schedule values in B.1) are consistent with the mechanism, and the internal-state analysis (D.7) shows reduced injection sensitivity. Acknowledged limitations, teacher-bounded quality (G), illustrative rather than prevalence-estimating failure mode (B.2), and static-only threat model (F), are stated candidly and are not treated as defects. The remaining soft spot is the one the reader flagged, sharpened here: within the static threat model, the training distribution of adversarial spans is generated entirely by the defended model itself (C.5.1), and the evaluation corpora, while independent, are exactly the fixed corpora against which the claim is made. What is never varied is the attack-writing process itself: style register, embedding depth, multi-part structure. Eq. (3) would reward a corpus-level shortcut as much as genuine span-insensitivity, and the controlled measurements cannot distinguish these. This is not an internal inconsistency; it is a precisely delimited generalization gap at the boundary of the paper's claims, which the paper's candor covers on the adaptive side but not on the fresh-static-attack side. The paper also discloses the SecAlign++ budget asymmetry and quantifies the 625-step trade-off on Nemotron (D.8), though only qualitatively for Gemma; that is a secondary, disclosed fairness issue rather than the primary threat to the central claim. If the proposed re-attack test shows ASR remains near 1% with U_atk near U across the three models, the central claim is robust within its scope and the paper warrants acceptance-type handling with the reader's requested code release and matched-budget tables. If ASR rises materially, the headline should be narrowed to the tested corpora and the verdict should move toward restricted-scope handling. Without running the test, the reader's CONDITIONAL disposition is the right verdict, so no change is made.","tokens_in":34681,"tokens_out":19438,"duration_ms":263981,"concrete_test":"Take the released RAISED checkpoints (Gemma, Nemotron, Qwen) and the public AgentDojo/AgentDyn episode harness, unchanged. Before any interaction with the defended checkpoints, have a different strong model, given only the episode's tool outputs as the attacker channel, write fresh static injections with varied styles, mid-document embedding, and multi-part instructions, inserted at the original positions. Run the frozen checkpoints once, with no retraining, and compare ASR and U_atk to Table 1 and D.6. If ASR stays near 1% with U_atk near U, channel-level invariance is confirmed and the concern fails; if ASR rises above roughly 10%, the headline overstates robustness to unseen static attacks and the transfer claim as stated should be narrowed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to be more than a report about three fixed corpora, the student must acquire channel-level invariance: for any adversarial span inserted into a tool observation, next-token behavior equals clean-context behavior. Training-time span diversity comes exclusively from self-authored attacks: the defended model writes them post-hoc, with knowledge of the exact tool outputs and attacker tooling, in its own linguistic register (Section C.5.1). Eq. (3) punishes deviation from the clean distribution on injected contexts equally whether the student learns true span-insensitivity or a corpus-level shortcut, e.g., recognizing its own prose style, edge placement, or scenario-local goals. The three evaluation corpora (InjecAgent, AgentDojo, AgentDyn) are genuine and independent mitigations: their attacks are human-authored, static, and placed in realistic content, and ASR falls to roughly 1%. The untested middle ground lies between these corpora and the adaptive attacks excluded in Section F [b669-b680]: fresh static injections written by a different author with the same environment access, using other styles, deeper embedding, or multi-part instructions. The paper candidly delimits the adaptive frontier, acknowledges the teacher-quality bound (Section G) and the illustrative scope of the failure-mode examples (B.2), but no passage addresses this static distribution-shift gap, so the abstract's broad phrasing, 'substantially reduces the attack success rate of prompt injections in tool responses,' exceeds what the current evidence directly covers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces RAISED, a training-time defense against indirect prompt injection in tool-using LLM agents. RAISED has two stages: a frozen copy of the base model generates its own executable tool-use scenarios (steered toward cases where task completion requires acting on legitimate tool-output guidance), solves and verifies them, and writes its own injections; the student is then trained via KL divergence to match the teacher's clean-context next-token distribution on both clean and injected variants of the same trajectories (Eq. 3). The paper's central claims are: (i) existing training-based defenses (SFT, SecAlign++'s DPO) induce measurable output-distribution drift on benign inputs and a behavioral failure mode in which the defended model refrains from tool-guided steps needed to finish authorized tasks (Sections 4.1-4.2, B.1-B.2); (ii) RAISED substantially reduces attack success rate on InjecAgent, AgentDojo, and AgentDyn while preserving benign utility and general capabilities across Gemma, Nemotron, and Qwen (Tables 1-3, D.6); and (iii) a controlled ablation (Section E) isolating the training objective from the data shows self-distillation preserves more agentic utility than SFT or DPO at similar ASR. Robustness claims are explicitly scoped to the static threat model (Section 3.2, Section F).","tokens_in":34909,"tokens_out":9907,"duration_ms":74622,"significance":"If the results hold, the paper makes two useful contributions to the prompt-injection defense literature. First, it provides concrete evidence, including a clean controlled ablation (Section E) that holds the trajectories, inputs, and budget fixed while varying only the objective, that clean-context distribution matching preserves more agentic utility than SFT or the DPO objective used by SecAlign++ at similar attack success rates; this is the kind of controlled comparison the area needs and it is done properly with three seeds. Second, the evaluation is grounded externally: ASR and utility are measured on three established benchmarks (InjecAgent, AgentDojo, AgentDyn) whose attacks are human-authored and independent of the defended model, so there is no circularity in the robustness claim. The paper is also commendably honest about scope: it explicitly excludes adaptive attacks (Section F), acknowledges the teacher-quality bound (Section G), and labels the behavioral-drift examples as illustrative rather than prevalence estimates (Section B.2). The reproducibility statement and released code support verification.","major_comments":[{"comment":"The abstract and Section 7.1 claim that RAISED 'substantially reduces the attack success rate of prompt injections in tool responses,' and the method is framed around the channel-level invariance of Eq. (1), i.e., insensitivity to any inserted adversarial span. The supporting evidence is restricted to three fixed corpora (InjecAgent, AgentDojo, AgentDyn), while all training-time injections are written by the defended model itself, post-hoc, in its own linguistic register, with knowledge of the exact tool outputs and attacker tooling (Section C.5.1). The paper candidly delimits the adaptive frontier in Section F [b669-b680], but no passage addresses the middle ground within the static threat model: fresh static injections authored by independent writers, in other styles, deeper embedding positions, or multi-part forms. Because Eq. (3) penalizes deviation from the clean distribution equally whether the student learns true span-insensitivity or a corpus-level shortcut (e.g., recognizing its own prose style or scenario-local goals), the ~1% ASR values in Table 1 do not by themselves establish the general invariance the abstract and Eq. (1) claim; the diversity across the three benchmark corpora mitigates but does not close this gap. I ask the authors either to (a) re-attack the released RAISED checkpoints with a held-out set of fresh, independently authored static injections (e.g., different styles, deeper embedding, multi-part instructions) and report ASR/U/U_atk in the Table 1 format, or (b) explicitly scope the abstract and Section 1 claims to the evaluated attack distributions and state the static-distribution-shift transfer as an open question, in the same manner Section F handles adaptive attacks.","section":"Section 3.2 / Eq. (1); Section 5.1 / C.5.1; Section F; Section 7.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The distribution-drift analysis measures next-token KL on 500 standalone non-tool QA questions, whereas the paper's utility argument (Contribution 1) concerns agentic tool-use behavior. The paper correctly hedges drift as only a 'potential mechanism' for utility loss, but a sentence noting that drift on tool-use contexts was not measured, and pointing to the Section B.2 examples as the qualitative evidence on that channel, would tighten the link between the two parts of Contribution 1.","section":"Section 4.1 / B.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The training input h'_i is drawn from {h_i, ~h_i}, but the 0.5 injection ratio is stated only in Section C.5.1; a one-line note in Section 5.2 would let readers of the main text see the clean/injected balance without consulting the appendix.","section":"Section 5.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract's 'unlike prior training-based defenses' is supported for the SFT and SecAlign++ baselines actually evaluated, but other training-time defenses (e.g., Jatmo, instruction-hierarchy training) are not compared; consider hedging to 'the training-based defenses we evaluate' to match the evidence.","section":"Abstract / Section 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The main-comparison SecAlign++ checkpoint uses 313 steps over 5,008 examples while RAISED trains on 10,000 trajectories. The paper addresses this asymmetry in Sections C.4, D.8, and the Section E ablation, but a single pointer sentence near the Table 1 discussion would prevent readers from misreading the main comparison as precisely budget-matched.","section":"Section 7.1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"RAISED is a solid training-time defense paper. The central claim — that clean-context self-distillation cuts indirect prompt-injection ASR to roughly one percent on three independent benchmarks while preserving benign agentic and general utility — holds up on the evidence shown. It deserves a serious referee.\n\nWhat is actually new: the controlled distribution-drift analysis (Section 4.1 / B.1), the named behavioral failure mode where defended models stop acting on legitimate tool-output guidance (4.2 / B.2), and the application of consistency training to multi-turn tool-use trajectories with self-generated scenarios and attacks. The strongest evidence is Section E, which holds the data and input representation fixed and varies only the objective: self-distillation preserves more agentic utility than SFT or SecAlign++'s DPO at similar ASR, and the self-generated trajectories add a further utility gain. That is the right way to separate the idea from the data pipeline.\n\nThe robustness numbers are externally grounded. ASR is measured on InjecAgent, AgentDojo, and AgentDyn, whose attacks are human-authored static injections the defended model never saw, so the circularity burden is genuinely low. The paper is also unusually candid about scope: adaptive attacks are explicitly out of scope, the failure-mode examples are labeled illustrative, and the teacher-quality bound is stated.\n\nSoft spots, in proportion. First, a real generalization gap: training attacks are written by the defended model itself, so the learned invariance is demonstrated on three corpus-specific attack distributions, not shown to be channel-level. The paper scopes honestly to static attacks, but nothing addresses fresh static injections written by a different author in a different style. The abstract's \"substantially reduces the attack success rate of prompt injections in tool responses\" slightly over-covers that gap. I read this as a generalization caveat rather than a broken central claim, but a third-party re-attack of the released checkpoints with independently rewritten injections would settle it, and I would ask for that.\n\nSecond, the SecAlign++ comparison uses the 313-step checkpoint against a longer RAISED budget. This is disclosed and quantified in D.8, and the utility-damage story justifies the choice, but the headline table still reads as asymmetric. A matched-budget row in the main table would fix it. Third, smaller items: the failure mode has no prevalence estimate, which the authors admit; the per-block internal-state analysis (D.7) is suggestive but light; and the code link has no commit hash, so pin it.\n\nWho this is for: anyone working on agent security or training-based defenses. The drift measurement and the controlled ablation are worth citing even independently of the method. My recommendation: accept for peer review. The central claim is supported, the limitations are disclosed, and the remaining asks are concrete rather than structural.","headline":"A solid, honestly-scoped training-time defense with a clean controlled ablation; the open question is how far the invariance learned on self-authored injections generalizes beyond the three benchmark corpora used for scoring.","tokens_in":35636,"tokens_out":3057,"duration_ms":39999},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"For tool-using language models, prompt-injection resistance can be trained by imitating the model's own clean behavior under attack — attack success on AgentDojo falls from 39.5% to 1.0% while benign utility loses only one point.","keywords":["indirect prompt injection","self-distillation","LLM agents","tool use","attack success rate","utility under attack","output-distribution drift","consistency training"],"falsifier":"Two decisive checks. First, run the RAISED-trained model on an injection benchmark it was never trained or tuned on, with attacks authored by a different model or by human red-teamers: if attack success climbs well above the roughly 1% level reported on AgentDojo, the learned invariance does not transfer to independently written injections. Second, give an adversary black-box query access to the deployed model and let it optimize an injection string against observed behavior: any single optimized injection that reliably diverts the agent, or a utility drop on real tool-guidance tasks the model never generated, would falsify the invariance claim outside the static threat model the paper restricts itself to.","tokens_in":36118,"feed_emoji":"🛡️","tokens_out":14780,"duration_ms":153247,"temperature":0.7,"pith_summary":"Tool-using language-model agents must act on content they pull from emails, webpages, and APIs, and attackers can hide instructions inside that content to redirect the agent. Prior attempts to train away this vulnerability made models more resistant but also changed their ordinary behavior, often leaving authorized tasks unfinished. This paper tries to establish that the right training target is the model's own clean behavior: in RAISED, a frozen copy of the model writes its own tool-use tasks and its own attack strings, and a trainable copy is trained to make the same next-token decisions the frozen copy would make on the clean task even when an injection has been dropped into a tool response. On the AgentDojo benchmark with Gemma, the paper reports attack success falling from 39.5% to 1.0% and utility under attack rising from 64.0% to 80.5%, with benign utility down just one point — matching a leading preference-tuning defense without its seven-point utility loss — and it reports the same pattern across three model families. If the learned invariance transfers to attacks and content the model never wrote itself, agents can be hardened against injected instructions without paying the usual capability tax.","feed_headline":"Self-distilled agents cut injection attack success from 39.5% to 1.0%","feed_subtitle":"RAISED trains an agent to mimic its own clean behavior, matching top defenses' security gains without the utility drop.","key_machinery":"The load-bearing mechanism is two-stage. Stage one, self-generation, has the frozen base model create the training distribution: an executable scenario with 3–6 tools, a user request needing a chain of dependent calls, and a success check, with generation steered toward \"surfaced guidance,\" \"recovery/completion,\" and \"follow-through\" structures so that many benign tasks require acting on legitimate tool-output instructions; the model then solves and verifies each scenario and, in a separate attacker role, writes an injection and chooses where in its own tool output to insert it. Stage two, injection-augmented self-distillation, trains the student by minimizing $\\mathbb{E}_{\\tau \\sim \\mathcal{D}} \\sum_{i \\in I(\\tau)} D_{\\mathrm{KL}}(q_i \\| p_\\theta(\\cdot|h'_i))$ with $h'_i$ either the clean history $h_i$ or the injected history $\\tilde{h}_i$, where $q_i$ is the teacher's clean-context next-token distribution. On clean contexts the objective anchors the student to the base model; on injected contexts the same target makes the student insensitive to the adversarial span. Because the legitimate tool content is unchanged, the model learns to distrust the injected portion rather than the tool-output channel as a whole.","core_discovery":"The central claim is that prompt-injection resistance and benign utility are not locked in a trade-off when the defense is framed as invariance rather than retraining. The paper's target is the identity $B_\theta(\tilde{c}) \\equiv B_{\\theta_0}(c)$: the defended model's behavior on an injected context should equal the base model's behavior on the corresponding clean context, so attack success falls to the level the base model reaches by accident while utility under attack matches clean utility. The discovery is that this can be realized by having the model itself supply the training signal. A frozen copy of the base model acts as generator, solver, and attacker: it writes an executable tool-use scenario steered toward cases where finishing the authorized task requires acting on legitimate guidance returned by a tool, solves the scenario through its normal tool-calling interface, verifies success against a generated check, then writes its own injection string and plants it in one of its own tool outputs. A trainable student is then trained with a KL objective — a standard measure of how far two probability distributions are apart — to match the frozen model's clean-context next-token distribution (the top 256 token probabilities plus one residual bucket) on both clean and injected versions of the same trajectory. The paper reports per-token forward and reverse KL from the base model of about 0.0010, versus 0.044 for SFT and 0.029 for DPO, and reports that on AgentDojo with Gemma attack success falls from 39.5% to 1.0% while benign utility drops from 84.5% to 83.5%.","pith_inferences":["Beyond the paper: because the training attacks are written by the model being defended, the reported roughly 1% ASR may understate performance against injection styles the base model would never write; a natural stress test is to evaluate RAISED on an injection suite authored by a different model or by humans without further fine-tuning.","Beyond the paper: the same clean-context self-distillation could be applied to other untrusted-channel threats — phishing instructions in email text, web content in browsing agents, or poisoned tool descriptions — wherever a trustworthy clean behavior already exists to be preserved by matching the base distribution.","Beyond the paper: the measured KL-to-utility relationship suggests a testable rule of thumb — defenses keeping per-token KL from the base model below roughly 0.002 should show little benign utility loss, while drift above roughly 0.03 should show measurable degradation — which could be checked across other models and benchmarks.","Beyond the paper: the authors restrict their empirical claims to static attacks, so an adaptive attacker with query access and an optimizer over injection strings sits outside the reported numbers; the roughly 1% figure should not be quoted for adversarial red-teaming settings."],"forward_implications":["Attack success rate alone can mislead: the filtering baseline PromptGuard2 cuts AgentDojo ASR from 39.5% to 16.1% yet drops utility under attack from 64.0% to 42.9%, so ASR, benign utility, and utility under attack should be reported and read together.","Per-token KL divergence from the base model is a cheap early-warning proxy for capability drift: SFT and DPO sit at 0.044 and 0.029 while self-distillation sits at 0.0010, and the two preference- and SFT-based methods drift further with more training while self-distillation does not.","Defensive training targets should come from the model's own distribution whenever the desired behavior already exists: externally authored SFT targets and DPO preference pairs carry a likelihood-displacement risk that preserving clean-context behavior avoids.","Because RAISED preserves rather than improves the base model, the defended model inherits the base model's errors, so the method is a defense against context influence rather than a general alignment or safety fix.","The method also helps in low-ASR regimes: on Qwen, where the undefended model already achieves 5.8% ASR on AgentDojo, RAISED still pushes it to 0.9% while keeping general and agentic scores close to the Vanilla checkpoint."],"supporting_citations":[{"why":"SecAlign++, the closest training-based baseline: supplies the preference-optimization objective RAISED replaces, and the robustness numbers RAISED matches without the utility drop.","marker":"Chen et al. (2026)"},{"why":"The earlier SecAlign defense whose DPO objective the drift analysis examines and contrasts with distribution matching.","marker":"Chen et al. (2025)"},{"why":"AgentDojo, the benchmark that yields the headline numbers: ASR from 39.5% to 1.0% and utility under attack from 64.0% to 80.5%.","marker":"Debenedetti et al. (2024)"},{"why":"InjecAgent, one of the three robustness benchmarks on which RAISED is measured.","marker":"Zhan et al. (2024)"},{"why":"AgentDyn, the dynamic benchmark where prior training defenses show their largest utility loss and where RAISED preserves clean utility.","marker":"Li et al. (2026)"},{"why":"Defines indirect prompt injection, the threat model the defense is built against.","marker":"Greshake et al. (2023)"},{"why":"Supports the premise that self-generated training targets limit distribution drift during fine-tuning.","marker":"Yang et al. (2024)"},{"why":"Supplies the likelihood-displacement mechanism used to explain why a DPO-style preference objective need not preserve the base model's behavior.","marker":"Razin et al. (2025)"},{"why":"DPO, the preference objective that RAISED replaces with clean-context distribution matching.","marker":"Rafailov et al. (2024)"}],"fun_headline_variants":["Self-distillation beats prompt injection without utility loss","RAISED: match clean behavior, cut attack success 39.5% to 1.0%","Injection-proof agents via self-distillation: no utility trade-off","Self-distillation keeps agent utility while blocking prompt injection"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The claim stands or falls on transfer: resistance learned from scenarios and attacks that the base model writes for itself must generalize to attacks written independently by real adversaries, and preserved willingness to act on self-generated tool-guidance cases must predict preserved utility on real workloads — if the model-written attacks are easier than genuine ones, or the self-written tasks are closer to the benchmarks than to deployment traffic, the headline attack-success and utility numbers overstate real-world robustness.","fun_headline_variants_meta":{"raw":{"variants":["Self-distillation beats prompt injection without utility loss","RAISED: match clean behavior, cut attack success 39.5% to 1.0%","Injection-proof agents via self-distillation: no utility trade-off","Self-distillation keeps agent utility while blocking prompt injection"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"cost":0.000559048,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000559048,"upstream_inference_prompt_cost":0.000126168,"upstream_inference_completions_cost":0.00043288},"total_tokens":2652,"prompt_tokens":1106,"completion_tokens":1546,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1467}},"tokens_in":1106,"tokens_out":1546,"cost_source":"openrouter_usage","duration_ms":9006,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1467,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-06T13:23:15.124254+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Two decisive checks. First, run the RAISED-trained model on an injection benchmark it was never trained or tuned on, with attacks authored by a different model or by human red-teamers: if attack success climbs well above the roughly 1% level reported on AgentDojo, the learned invariance does not transfer to independently written injections. Second, give an adversary black-box query access to the deployed model and let it optimize an injection string against observed behavior: any single optimized injection that reliably diverts the agent, or a utility drop on real tool-guidance tasks the model never generated, would falsify the invariance claim outside the static threat model the paper restricts itself to.","supporting_citations":[{"cited_title":"APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets","cited_arxiv_id":null,"evidence_quote":"InjecAgent, one of the three robustness benchmarks on which RAISED is measured."},{"cited_title":"APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets","cited_arxiv_id":null,"evidence_quote":"Supports the premise that self-generated training targets limit distribution drift during fine-tuning."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}