{"id":"736aacb2-46b1-43e0-a03d-ba0d168027a0","arxiv_id":"2505.18333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"StruQ, SecAlign, Instruction Hierarchy, PromptGuard, and Attention Tracker are substantially less effective and utility-preserving than claimed when evaluated with diverse prompts and adaptive attacks.","lead":"This paper re-tests six published defenses against prompt injection in large language models and finds they fail under broader, adaptive attacks and lose general-purpose utility. It argues that prior defense papers used narrow tests, such as a single injected prompt, and overclaimed success.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Instruction Hierarchy verdict rests on treating GPT-4o-mini as equivalent to the defense; the paper does not validate that attribution, so the specific claim that Instruction Hierarchy is ineffective is not supported.","rationale":"The paper's central contribution is a critical evaluation framework and its application to six defenses. The most fragile empirical pillar is Section 5.2, where the model evaluated is not the defense itself but a proprietary API model claimed to deploy it. The paper states that Instruction Hierarchy 'has thus been deployed on GPT-4o-mini' and then says, 'To reassess its utility and effectiveness, we evaluate GPT-4o-mini.' For the conclusion 'Instruction Hierarchy is not effective' to be valid, GPT-4o-mini's vulnerability must be attributable to Instruction Hierarchy. That attribution is not established; GPT-4o-mini is a black box with unknown additional safety measures, and the paper offers no evidence that the behavior measured is caused by IH rather than by other components. This is an internal attribution problem, not merely a disagreement with the original IH paper. The concern is load-bearing because the paper explicitly claims IH remains vulnerable and uses that as evidence that existing defenses are overrated. However, the other results are based on open-source released models: StruQ and SecAlign weights are downloaded, PromptGuard and Attention Tracker weights are obtained, and the GCG/adaptive attacks are run locally. Those findings stand independently. I also considered the lack of error bars and seed reporting for the 50/25-tuple GCG subsets, but the adaptive attacks reaching ASV 1.00 and the qualitative agreement across two benchmarks make that a secondary reproducibility issue rather than the primary logical weakness. The appropriate verdict remains CONDITIONAL, requiring either a stronger basis for the GPT-4o-mini/IH attribution or a softened claim that limits the conclusion to the proprietary deployment rather than to Instruction Hierarchy as a method. Since the reader already identified this assumption, the verdict is unchanged.","tokens_in":14107,"tokens_out":6472,"duration_ms":52500,"concrete_test":"Test the attribution directly by reproducing Instruction Hierarchy on an open model (e.g., fine-tune Llama-3-8B-Instruct with the IH recipe from arXiv:2404.13208 or use a public IH checkpoint) and run the identical OpenPromptInjection and MMLU-PI evaluations with the same Combined Attack and ASV metric. If the open IH model shows substantially lower ASVs than the reported GPT-4o-mini values (0.68 and 0.75), then the paper's conclusion that IH itself is ineffective is not supported. Alternatively, run a set of IH-specific behavioral probes on GPT-4o-mini and on the open IH model, comparing how they prioritize conflicting system, user, and injected instructions; if GPT-4o-mini's behavior is consistent with additional or different defenses, the attribution should be weakened or the claim rephrased to target GPT-4o-mini only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is in Section 5.2. The paper needs to establish that GPT-4o-mini's behavior is attributable to Instruction Hierarchy (IH). It cites an OpenAI blog deployment claim and then evaluates GPT-4o-mini on OpenPromptInjection and MMLU-PI, concluding that 'Instruction Hierarchy is not effective when evaluating on diverse injected prompts.' But GPT-4o-mini is a proprietary black-box model with undisclosed safety mechanisms, prompt filtering, RLHF, and possibly a modified or partial version of IH. Without isolating IH, the observed ASVs of 0.68 and 0.75 could be caused by unrelated components or by interactions between IH and other defenses. The paper provides no control, no ablation, and no behavioral evidence that GPT-4o-mini's instruction-priority behavior matches the published IH method. Thus the paper's deployed-defense case study is unsupported. This does not invalidate the StruQ, SecAlign, PromptGuard, or Attention Tracker results, which use released models, but it does weaken the broad abstract claim that all evaluated defenses are less successful than previously reported, since IH is one of the six evaluated defenses and the only one examined through a proprietary proxy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that recent defenses against prompt injection attacks have been evaluated with an insufficient methodology, and proposes a two-dimensional evaluation framework: effectiveness against both existing and adaptive attacks over diverse target/injected prompts, and preservation of general-purpose utility measured by absolute task metrics. The authors re-evaluate six defenses: StruQ and SecAlign (prevention, via released Llama-3 models), Instruction Hierarchy (via GPT-4o-mini as a proxy), and PromptGuard and Attention Tracker (detection, via released models). Their results show that StruQ and SecAlign suffer measurable utility drops, that GCG-based attacks achieve substantially higher ASVs than originally reported (e.g., 0.80–1.00 against StruQ), that Instruction Hierarchy appears ineffective on GPT-4o-mini, that PromptGuard has very high FPRs despite high AUC, and that Attention Tracker's FNR rises sharply under their adaptive attack. The paper concludes that existing defenses are not as successful as previously reported.","tokens_in":14239,"tokens_out":3395,"duration_ms":25383,"significance":"If the empirical findings hold, this is a valuable contribution to the security evaluation literature: it demonstrates concrete failure modes of existing evaluation practices, provides a reusable benchmark extension (MMLU-PI) and methodology for adaptive attacks, and ships code and data. The direct comparisons on released models for StruQ, SecAlign, PromptGuard, and Attention Tracker are convincing, and the adaptive-attack results for those defenses are a useful stress test. The paper's main limitation is that the Instruction Hierarchy conclusion rests on an unvalidated attribution of GPT-4o-mini's behavior to that specific defense; since Instruction Hierarchy is one of the six evaluated defenses, this weakens the broad abstract claim. With appropriate re-framing or additional evidence for Section 5.2, the core methodology and the other case studies would be publishable.","major_comments":[{"comment":"The claim that 'Instruction Hierarchy is not effective when evaluating on diverse injected prompts' is not supported by the experiments as presented. The paper evaluates GPT-4o-mini and attributes its behavior to Instruction Hierarchy solely on the basis of an OpenAI blog deployment claim. No evidence is provided that GPT-4o-mini actually implements Instruction Hierarchy as described in [37], rather than a modified version, an additional defense, or unrelated safety mechanisms. There is no control model, no ablation, and no behavioral probe (e.g., testing known instruction-priority behaviors) to validate the attribution. The observed ASVs of 0.68 and 0.75 could arise from other components. This is load-bearing because Instruction Hierarchy is one of the six defenses in the paper's scope, and the abstract's sweeping conclusion that 'existing defenses are not as successful as previously reported' depends in part on this example. The paper should either remove the Instruction Hierarchy case study, re-frame it as an evaluation of GPT-4o-mini as a deployed system (without attributing the outcome to a specific defense), or provide a validated open-source implementation of Instruction Hierarchy and evaluate that instead.","section":"Section 5.2"},{"comment":"The headline GCG-based ASV numbers are computed on small subsets (50 tuples from OpenPromptInjection and 25 from MMLU-PI), but the paper omits the tuple-selection protocol and all GCG hyperparameters (number of iterations, top-K, candidate set size, suffix length, random seed). Without these details, the reader cannot assess whether the selected subsets are representative of the full T sets or whether the optimization was run to convergence. Given that these numbers are the primary quantitative evidence that StruQ and SecAlign remain vulnerable to existing optimization-based attacks, the paper should specify the sampling procedure, state the hyperparameters, and ideally report the variance across multiple runs or subsets. The released code mitigates but does not replace in-paper protocol reporting, especially for a critical-evaluation paper whose goal is methodological rigor.","section":"Section 5.1, Tables 2 and 5"},{"comment":"The text states that 'all contaminated data samples that successfully evade Attention Tracker's detection also make the LLM generate the attacker-desired response re.' This is a strong joint claim about both detection evasion and attack success, but Table 3(b) reports only FNR. No ASV values are reported for the joint outcome, and the connection between FNR and the claimed adversarial task completion is not demonstrated in any table or equation. If this claim is meant to support the adaptive attack's effectiveness, the ASV should be reported alongside the FNR (e.g., as a joint metric or as separate columns). As written, the claim is unverifiable from the presented data.","section":"Section 5.3, Table 3(b) and surrounding text"}],"minor_comments":[{"comment":"Typo: 'AlpacaEval on AlphaFarm' should read 'AlpacaEval on AlpacaFarm'.","section":"Section 5.1"},{"comment":"Typo: 'Slimilarly' should be 'Similarly'.","section":"Appendix A"},{"comment":"The MMLU-PI benchmark construction is described only briefly; the paper should state the sampling seed, the exact number of tasks/subjects, and how T and Xc were generated to enable reproduction.","section":"Section 4.2.3 and Section 5"},{"comment":"The original papers' reported ASV/ASR values (e.g., 0.04 for StruQ and 0 for SecAlign under optimization-based attacks) are not shown alongside the new results; adding a row or a pointer in the table would make the comparison more immediate.","section":"Tables 2 and 5"},{"comment":"The balancing coefficient alpha in Equation (6) is set to 0.01 without any sensitivity analysis; a short sentence explaining the choice or referencing a sensitivity check would be helpful.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core methodology and the case studies on released models (StruQ, SecAlign, PromptGuard, Attention Tracker) are sound and potentially influential. The main weakness is Section 5.2, where the inference from GPT-4o-mini to Instruction Hierarchy is not justified; this should be dealt with before publication. The paper reuses the authors' own OpenPromptInjection benchmark, which is appropriate for a critical evaluation but should be disclosed more prominently (it already is, though the potential conflict is worth noting to the editor). Given that the code and data are released, the reproducibility concern is partially addressed, but the missing GCG protocol details remain a significant omission for a methodology-focused paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of arXiv:2505.18333. The paper's big result is that several published prompt-injection defenses look much weaker when tested with diverse prompts and absolute utility metrics. On released models, the evidence is strong: StruQ and SecAlign hit ASVs of 0.80–1.00 under GCG, not the 0.04 claimed in the original papers. PromptGuard has AUC 0.92 but FPR 0.89; Attention Tracker has FNR 0.69 on MMLU-PI, rising to 1.00 under an adaptive attack. These are concrete, reproducible findings, and the code and data are public. The authors also build a sensible new benchmark, MMLU-PI, and propose adaptive attacks that are genuinely useful. The methodological critique of win-rate-only utility and AUC-only detection is sound, even if the general point about rigorous evaluation has been made before.\n\nThe soft spots are real but mostly addressable. The GCG numbers come from only 50 and 25 tuples, with no seeds or error bars. The effect sizes are large enough that I don't doubt the qualitative direction, but a referee should ask for variance. Attack hyperparameters are under-specified: iterations, top-K, candidate set size, and the balancing coefficient alpha for the detection adaptive attack are not fully reported. The open code likely fills this in, but the paper should state it.\n\nThe bigger issue is Section 5.2. The paper treats GPT-4o-mini as \"Instruction Hierarchy\" based only on an OpenAI blog post, then concludes that Instruction Hierarchy is not effective. That attribution is not validated. GPT-4o-mini is a proprietary black box with unknown safety filtering, RLHF, and possibly a partial or modified version of the defense. There is no ablation or behavioral evidence isolating the instruction-hierarchy mechanism. This weakens the abstract's sweeping claim that \"existing defenses are not as successful as previously reported,\" because IH is one of six evaluated defenses. The StruQ, SecAlign, PromptGuard, and Attention Tracker results stand on their own; the IH case study does not.\n\nOverall, this is a useful and honest paper. The released-model findings are significant and will force a re-read of the original defense papers. My recommendation: send it to peer review, and require a revision with error bars, full attack hyperparameters, and a reworked IH claim—either soften it to \"GPT-4o-mini, claimed to deploy instruction hierarchy, remains vulnerable\" or evaluate an open instruction-hierarchy model directly. As it stands, the paper is worth engaging, and a serious referee can push it into shape.","headline":"Released-model results on StruQ, SecAlign, PromptGuard, and Attention Tracker are convincing and useful; the GPT-4o-mini proxy for Instruction Hierarchy overreaches, but the core findings stand.","tokens_in":14893,"tokens_out":2547,"would_cite":true,"duration_ms":19000,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Existing defenses against prompt injection attacks are not as successful as previously reported when evaluated with diverse prompts and adaptive attacks.","keywords":["prompt injection","LLM security","defense evaluation","adaptive attacks","attack success value","absolute utility","instruction hierarchy","GCG attack"],"falsifier":"Run the same benchmark suite on the open-source Instruction Hierarchy model from reference [37] rather than the black-box GPT-4o-mini. If its Combined Attack ASVs on OpenPromptInjection and MMLU-PI are near zero, the paper's claim that Instruction Hierarchy is ineffective against diverse injected prompts would be directly contradicted; if they are high, the claim would be confirmed.","tokens_in":13782,"feed_emoji":"🛡️","tokens_out":7982,"duration_ms":56497,"temperature":0.7,"pith_summary":"Recent defenses against prompt injection attacks look much stronger in their own papers than they do under a broader test. The paper argues that prior evaluations used unrepresentative injected prompts, relative win-rate utility metrics, AUC-only detector metrics, and no adaptive attacks. Re-evaluating StruQ, SecAlign, Instruction Hierarchy (as deployed in GPT-4o-mini), PromptGuard, and Attention Tracker on two benchmarks with diverse target and injected prompts, the paper finds substantially higher attack success values and false negative rates than previously reported, along with utility losses that win-rate comparisons had missed. The upshot is a call for a two-axis evaluation standard—effectiveness against existing and adaptive attacks plus absolute general-purpose utility—that future defenses should meet.","feed_headline":"Prompt-injection defenses fail under adaptive attacks","feed_subtitle":"A systematic re-evaluation of five recent defenses finds high attack success and hidden utility losses.","key_machinery":"The load-bearing object is the evaluation harness, not a new defense. It builds a set $T$ of tuples $(p_t, r_t, p_e, r_e)$ from OpenPromptInjection and a new MMLU-PI benchmark, sampling diverse target and injected prompts with differing ground-truth responses, and measures Attack Success Value (ASV)—the fraction of tuples where the LLM produces the injected response. For detectors it measures false positive rate and false negative rate at a deployment threshold rather than AUC alone, and it defines adaptive attacks: for StruQ/SecAlign, tokens whose embeddings are closest to the filtered delimiter tokens structure the injected prompt; for detectors, a combined loss of detector evasion plus injection success is optimized with GCG. Absolute utility is measured by task-specific scores such as accuracy, ROUGE-1, and GLEU on the same benchmarks.","core_discovery":"On the paper's own terms, the discovery is that five recent defenses do not hold up when evaluated with diverse prompt pairs and attacks they were not specifically tuned against. On OpenPromptInjection and the new MMLU-PI benchmark, GCG-based attacks achieve ASVs of 0.80 and 0.88 against StruQ, rising to 1.00 with an adaptive attack; SecAlign allows ASVs of 0.72 on MMLU-PI; and the deployed Instruction Hierarchy (GPT-4o-mini) permits Combined Attack ASVs of 0.68 and 0.75. Detection defenses show comparable fragility: PromptGuard has FPRs of 0.89 and 0.84 despite high AUCs, and Attention Tracker has an FNR of 0.69 on MMLU-PI, rising to 1.00 under an adaptive attack. The paper's conclusion is that the reported success of these defenses was an artifact of limited test prompts, optimistic metrics, and missing adaptive attacks.","pith_inferences":["A natural next step the paper does not take is to apply its adaptive-attack framework for detectors (a weighted combination of detector-evasion and injection-success losses) to other detection methods; doing so may reveal similar fragility.","Because the GCG evaluations used a 50-tuple (or 25-tuple) subset of T for computational efficiency, the exact ASV numbers for StruQ and SecAlign would likely shift if the full 4,900-tuple OpenPromptInjection set were used; the qualitative conclusion of high attack success would probably remain.","The Instruction Hierarchy case study inherits an attribution caveat: if GPT-4o-mini contains additional defenses beyond Instruction Hierarchy, then the measured ASVs characterize the deployed model, not Instruction Hierarchy itself.","The paper's two-axis template—diverse prompts, adaptive attacks, absolute utility, and threshold-based detector metrics—could serve as a reusable standard for future defense papers beyond the five defenses examined."],"forward_implications":["Future prevention defenses should report Attack Success Value across diverse target and injected prompt pairs rather than relying on a single fixed injected prompt.","Detection defenses should be evaluated at a concrete decision threshold with FPR and FNR reported, since high AUC does not guarantee usable classification in deployment.","Evaluations of prompt-injection defenses should include adaptive attacks, which in these experiments push StruQ's ASV to 1.00 and Attention Tracker's FNR to 1.00.","General-purpose utility of a defended LLM should be measured as absolute task performance, not only as a relative win rate against a reference model.","The version of Instruction Hierarchy deployed in GPT-4o-mini does not confer the robustness that its deployment suggests, based on Combined Attack ASVs of 0.68 and 0.75."],"supporting_citations":[{"why":"Supplies the StruQ defense and its fine-tuned model weights, the subject of the first case study.","marker":"[6]"},{"why":"Supplies the SecAlign defense and its fine-tuned model weights, the second prevention-based case study.","marker":"[7]"},{"why":"The Instruction Hierarchy defense whose claimed deployment in GPT-4o-mini is evaluated in Section 5.2.","marker":"[37]"},{"why":"The provider's blog post claiming GPT-4o-mini uses Instruction Hierarchy; it is the sole basis for attributing the model's behavior to the defense.","marker":"[25]"},{"why":"Provides the PromptGuard detection model whose AUC is high but whose FPR is re-measured as 0.89 and 0.84.","marker":"[22]"},{"why":"Provides the Attention Tracker detection method whose AUC is 1.00 but whose FNR is re-measured and adaptively attacked.","marker":"[15]"},{"why":"Provides the OpenPromptInjection benchmark (seven NLP tasks, tuple set T, clean/contaminated pairs) and the Combined Attack heuristic.","marker":"[19]"},{"why":"Source of the prompt-response pairs used to construct the new MMLU-PI benchmark.","marker":"[13]"},{"why":"The GCG optimization-based attack used to compute existing and adaptive attack success values and to optimize the detectors' evasion loss.","marker":"[44]"},{"why":"The AlpacaFarm benchmark whose lack of ground-truth responses motivates the paper's critique of relative win-rate utility.","marker":"[8]"}],"fun_headline_variants":["Adaptive attacks break five prompt-injection defenses","Defenses against prompt injection crumble when tested adaptively","Prompt-injection defenses not so robust after all","Study finds prompt-injection defenses fail against adaptive threats","Re-evaluation shows defenses flop under realistic attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that GPT-4o-mini's benchmark behavior is fully attributable to the Instruction Hierarchy defense, based only on a deployment claim by the model's provider; if GPT-4o-mini includes other defenses or a modified version of the method, then the case study's conclusion about Instruction Hierarchy itself is not supported by these experiments.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive attacks break five prompt-injection defenses","Defenses against prompt injection crumble when tested adaptively","Prompt-injection defenses not so robust after all","Study finds prompt-injection defenses fail against adaptive threats","Re-evaluation shows defenses flop under realistic attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00045,"raw_usage":{"total_tokens":2241,"prompt_tokens":890,"completion_tokens":1351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1276}},"tokens_in":506,"tokens_out":1351,"duration_ms":8759,"temperature":1.0,"reasoning_tokens":1276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:32:55.632663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same benchmark suite on the open-source Instruction Hierarchy model from reference [37] rather than the black-box GPT-4o-mini. If its Combined Attack ASVs on OpenPromptInjection and MMLU-PI are near zero, the paper's claim that Instruction Hierarchy is ineffective against diverse injected prompts would be directly contradicted; if they are high, the claim would be confirmed.","supporting_citations":[{"cited_title":"Struq: Defending against prompt injection with structured queries","cited_arxiv_id":null,"evidence_quote":"Supplies the StruQ defense and its fine-tuned model weights, the subject of the first case study."},{"cited_title":"Gpt-4o mini: advancing cost-efficient intelli gence","cited_arxiv_id":null,"evidence_quote":"The provider's blog post claiming GPT-4o-mini uses Instruction Hierarchy; it is the sole basis for attributing the model's behavior to the defense."},{"cited_title":"PromptGuard Prompt Injection Guardrail","cited_arxiv_id":null,"evidence_quote":"Provides the PromptGuard detection model whose AUC is high but whose FPR is re-measured as 0.89 and 0.84."},{"cited_title":"Formalizing and bench- marking prompt injection attacks and defenses","cited_arxiv_id":null,"evidence_quote":"Provides the OpenPromptInjection benchmark (seven NLP tasks, tuple set T, clean/contaminated pairs) and the Combined Attack heuristic."},{"cited_title":"Measuring massive multitask language underst anding","cited_arxiv_id":null,"evidence_quote":"Source of the prompt-response pairs used to construct the new MMLU-PI benchmark."},{"cited_title":"Alpacaf arm: A simulation framework for meth- ods that learn from human feedback","cited_arxiv_id":null,"evidence_quote":"The AlpacaFarm benchmark whose lack of ground-truth responses motivates the paper's critique of relative win-rate utility."}],"review_version":1}